Focused deterrence strategies effects on crime: A systematic review

A relatively small number of people, often involved in gangs and criminally‐active groups, are responsible for a disproportionate share of crime. Focused deterrence strategies attempt to reduce offending behavior for specific types of crime. Our review suggests that these strategies are associated with moderate overall reductions in crime. Crime is not displaced to other areas, rather it is more likely that there is a diffusion of crime control benefits to adjacent areas and socially‐connected groups of offenders.


| The review in brief
A relatively small number of people, often involved in gangs and criminally-active groups, are responsible for a disproportionate share of crime. Focused deterrence strategies attempt to reduce offending behavior for specific types of crime. Our review suggests that these strategies are associated with moderate overall reductions in crime.
Crime is not displaced to other areas, rather it is more likely that there is a diffusion of crime control benefits to adjacent areas and socially-connected groups of offenders.

| What is this review about?
Crime is highly concentrated amongst a small number of highly-active offenders. Focused deterrence strategies combine law enforcement, community mobilization, and social services in an attempt to reduce offending behavior for specific crime types. A key feature of this crime control strategy involves the direct communications of the consequences of continued criminal offending and the availability of social services to targeted subjects. This review examines whether focused deterrence reduces crime and considers how observed crime reduction effects may vary by the different types of focused deterrence strategies and program evaluation designs.  Yes. The available evidence suggests an overall reduction in crime when focused deterrence strategies are used. The largest reductions are generated by focused deterrence strategies that target criminally active gangs or groups, followed by programs that target individual chronic offenders and drug market interventions.

| Do some programs work better than others?
Yes. Gang/group intervention programs generate the largest effects, followed by programs targeting high-risk individuals, with the smallest effects generated by drug market intervention (DMI) programs. DMI programs are most likely to suffer implementation problems which reduce effectiveness.

| Does crime get displaced to other areas?
No. No studies found significant crime displacement effects into surrounding areas. There is some evidence of the diffusion of crime control benefits.

| What do the findings of this review mean?
Findings from this review support the growing use of focused deterrence as a proactive crime reduction strategy. Practitioners and policy-makers should continue to implement focused deterrence programs to address serious crime problems.
The number of studies included in the updated review is more than double the number of studies included in the previous iteration of the review. However, despite the increase in eligible studies, no evaluations utilized a randomized controlled trial design. The growth of focused deterrence warrants more methodologically rigorous program evaluations and further exploration into the specific components of the strategy in order to improve our understanding of how the program reduces crime.

| How up-to-date is this review?
The review authors searched for studies up to October 2015. 2 | BRIEF ABSTRACT

| Background
Focused deterrence strategies, also known as "pulling levers" policing programs, have been increasingly implemented in the United States and other countries to reduce serious violent crime committed by gangs and other criminally active groups, recurring offending by highly active individual offenders, and crime and disorder problems generated by overt street-level drug markets.

| Objectives
To synthesize the extant evaluation literature and assess the effects of focused deterrence strategies on crime.

| Search methods
Multiple search strategies were used to identify eligible studies.
These strategies included a keyword search of online abstract databases, hand searches of relevant journals, consultation with policing experts, and searches of bibliographies of past narrative, empirical, and systematic reviews of police crime prevention efforts.

| Selection criteria
Eligible studies had to evaluate programs with the core elements of a focused deterrence strategy present, use quasi-experimental or randomized experimental designs, and report at least one crime outcome.

| Data collection and analysis
Twenty-four studies evaluating focused deterrence interventions were identified and full narratives of these studies were reported. A formal meta-analysis was conducted to determine the crime prevention effects of the eligible studies.

| Results
Our meta-analysis suggests that focused deterrence strategies are associated with an overall statistically significant, moderate crime reduction effect. However, effect sizes varied by program type and were smaller for evaluations with more rigorous research designs.

| Authors' conclusions
The available empirical evidence suggests these strategies may generate crime reduction impacts. However, more rigorous program evaluations are needed.

| Background
Focused deterrence strategies, also known as "pulling levers" policing programs, have been increasingly implemented in the United States and other countries to reduce serious violent crime committed by gangs and other criminally active groups, recurring offending by highly active individual offenders, and crime and disorder problems generated by overt street-level drug markets. These strategies are framed by an action research model that is common to both problemoriented policing and public health interventions to reduce violence.
Briefly, focused deterrence strategies seek to change offender behavior by understanding underlying crime-producing dynamics and conditions that sustain recurring crime problems and implementing an appropriately focused blended strategy of law enforcement, community mobilization, and social service actions. Direct communications of increased enforcement risks and the availability of social service assistance to target groups and individuals are defining characteristics of focused deterrence strategies.

| Objectives
To synthesize the extant evaluation literature and assess the effects of pulling levers focused deterrence strategies on crime.

| Search methods
Several strategies were used to perform an exhaustive search for literature fitting the eligibility criteria. First, a keyword search was performed on an array of online abstract databases. Second, we reviewed the bibliographies of past narrative and empirical reviews of literature that examined the effectiveness of pulling levers focused deterrence programs. Third, we performed forward searches for works that have cited the original focused deterrence review and seminal focused deterrence studies. Fourth, we searched bibliographies of narrative reviews of police crime prevention efforts and past completed Campbell systematic reviews of police crime prevention efforts. Fifth, we performed hand searches of leading journals in the field.

| Selection criteria
Eligible studies had to meet three criteria: (a) the program had to have the core elements of a focused deterrence strategy present; (b) a comparison group was included, or a one-group-only interrupted time-series design was used; (c) at least one crime outcome was reported. The units of analysis had to be people or places.

| Data collection and analysis
Twenty-four studies evaluating focused deterrence interventions were identified and full narratives of these studies were reported. All selected studies used quasi-experimental designs. A formal metaanalysis was conducted to determine the crime prevention effects of the eligible studies. Random effects models were used to calculate mean effect sizes.

| Results
Nineteen of the 24 evaluations of focused deterrence strategies reported at least one noteworthy crime reduction effect associated with the approach. It is important to note here that, even with the addition of 14 recent evaluations to this updated review, none employed a randomized controlled trial design to evaluate the intervention. Our meta-analysis suggests that pulling levers focused deterrence strategies are associated with an overall statistically significant, moderate crime reduction effect. However, program effect sizes varied by program type and was smaller for evaluations with more rigorous research designs.

| Authors' conclusions
The available empirical evidence suggests these strategies may generate crime reduction impacts. These encouraging results suggests that policymakers and practitioners should continue to implement these programs to control serious crime problems.
However, investments still need to be made to strengthen the overall rigor of program evaluations and improve our understanding of key program activities associated with observed crime reduction impacts.

| The intervention
The focused deterrence approach is consistent with recent theorizing about police innovation, which suggests approaches that seek to both create more focus in the application of crime prevention programs and expand the tools of policing that are likely to be most successful in controlling crime (Weisburd & Eck, 2004). Focused deterrence interventions are aimed at influencing the criminal behavior of individuals through the strategic application of enforcement, community, and social service resources to facilitate desirable behaviors. These strategies are often framed as problem-oriented exercises where specific recurring crime problems are analyzed, and responses are highly customized to local conditions and operational capacities. Focused deterrence allows police to increase the certainty, swiftness, and severity of punishment in innovative ways.
In an earlier version of this Campbell Collaboration systematic review, three basic kinds of focused deterrence programs were identified (Braga & Weisburd, 2012, 2011. The first type draws on the model of the Boston Operation Ceasefire experience during the 1990s (see Braga, Kennedy, Waring, and Piehl, 2001;Kennedy, Piehl, & Braga, 1996). This approach is focused on gang and criminally active group violence reduction strategies. It joins criminal justice agencies, social service organizations, and community members to engage directly with violent groups, communicate credible moral and law enforcement messages against violence clearly, make genuine offers of help for those who want it, and launch strategic enforcement campaigns against those who continue their violent behavior.
The second type of focused deterrence strategy is intended to reduce crime driven by street-level drug markets and is generally called a "drug market intervention" (DMI) program. DMI-focused BRAGA ET AL. | 3 of 65 deterrence strategies are used to identify street-level dealers, immediately apprehend violent drug offenders, and suspend criminal cases for nonviolent dealers (Kennedy, 2008). DMI strategies then bring together nonviolent drug dealers, their families, law enforcement and criminal justice officials, service providers, and community leaders for a meeting that communicates directly to offenders that their drug dealing has to stop, the community cares for them but rejects their conduct, services, and job opportunities are available, and renewed dealing will result in the activation of the existing case (Kennedy & Wong, 2009).
Finally, some focused deterrence programs are aimed at preventing repeat offending by high-risk individuals. In general, these strategies address the most dangerous offenders with a wide range of legal tools, put offenders on formal notice that their "next offense" will bring extraordinary legal attention, and focus community "moral voices" on such offenders to set a clear standard that violence is unacceptable (Deuchar, 2013;Kennedy, 2008;Papachristos, Meares, & Fagan, 2007).

| How the intervention might work
There are several theoretical mechanisms underlying focused deterrence that shed insights on how the intervention might work in practice. There is ample skepticism in the literature regarding "person-focused" approaches in policing (Weisburd, 2008). Such skepticism is rooted in evaluations of the standard model of policing dominant in the last century (National Research Council, 2004). In the standard model, the police focused on investigating and apprehending offenders. But the results of studies examining the crime prevention effects of strategies such as rapid response to calls for service (e.g., see Spelman & Brown, 1984), and investigations of crime after its occurrence (e.g., see Eck, 2002), led scholars to conclude that generalized person-focused approaches were ineffective (National Research Council, 2004;Sherman et al., 1997;Telep & Weisburd, 2012;Weisburd & Eck, 2004). Even in the case of interventions directed at individuals, and including focused deterrence, Weisburd and Eck (2004, p. 53) concluded that the evidence for effectiveness was "weak." The theory is important to provide a strong logic model for effectiveness, especially when drawing a conclusion on the basis of non-experimental evidence of program impacts. The strong theoretical model for the effectiveness of focused deterrence adds weight to the empirical evidence that we present in this article. Even though focused deterrence programs vary, they share common prevention mechanisms that are believed to influence crime. Although we do not evaluate these mechanisms directly, findings from evaluations of the associated programs provide insight into the effectiveness of these prevention mechanisms, which in turn yields knowledge that can aid in designing effective programs (Ludwig, Kling, & Mullainathan, 2011).

| Deterrence
Deterrence theory suggests that crime can be prevented when the costs of committing the crime are perceived by the offender to outweigh the benefits (Gibbs, 1975;Zimring & Hawkins, 1973). Most discussions of the deterrence mechanism distinguish between "general" and "special" deterrence (Cook, 1980). General deterrence is the idea that the general population is dissuaded from committing the crime when it sees that punishment necessarily follows the commission of a crime. Special deterrence involves punishment administered to criminals with the intent to discourage them from committing crimes in the future. Much of the literature evaluating deterrence has been focused on the effect of changing certainty, swiftness, and severity of punishment associated with certain acts on the prevalence of those crimes (Apel & Nagin, 2011;Nagin, 1998;Paternoster, 1987).
In addition to any increases in certainty, swiftness, and severity of sanctions associated with gun violence, focused deterrence strategies are intended to prevent crime through the advertising of the law enforcement strategy and the personalized nature of its application.
The effective operation of general deterrence is dependent on the communication of punishment threats to relevant audiences. As Zimring and Hawkins (1973, p. 142) observed, "the deterrence threat may best be viewed as a form of advertising." A key element of focused deterrence strategies involves the delivery of a direct and explicit "retail deterrence" message to a small target audience regarding what kind of behavior would provoke a special response and what that response would be. For instance, beyond the particular groups subjected to gang violence reduction interventions, the deterrence message was applied to a smaller specific audience (e.g., all gang-involved youth in a particular city) rather than to a larger general audience, and it was operated by making explicit cause-andeffect connections between the behavior of the target population and the behavior of the authorities. Knowledge of what happened to others in the target population was intended to prevent further acts of violence by gangs in the jurisdiction.
The results of available research reveal that deterrent effects are ultimately determined by offender perceptions of sanction risk and certainty (Nagin, 1998). Durlauf and Nagin (2011, p. 40) observed that " [S]trategies that result in large and visible shifts in apprehension risk are most likely to have deterrent effects that are large enough not only to reduce crime but also apprehensions," and they identified focused deterrence strategies as having these characteristics. As described earlier, focused deterrence strategies are targeted on specific behaviors by a small number of chronic offenders who are highly vulnerable to criminal justice sanctions.
The approach directly confronts offenders and informs them that continued offending will not be tolerated and how the system will respond to violations of these new behavior standards. Face-to-face meetings with offenders are an important first step in altering their perceptions about sanction risk (Horney & Marshall, 1992;Nagin, 1998). As McGarrell, Chermak, Wilson, and Corsaro (2006) suggested, direct communications and affirmative follow-up responses are the types of new information that may cause offenders to reassess the risks of continuing their criminal behavior.
In focused deterrence strategies, deterrent messages are framed to address the group context from which many crime problems emerge. The groups themselves can act as another internal communication vehicle for transmitting the actual sanction risk to other offenders. Sanctions for individual noncompliance are applied to groups; all communications to offenders focus on this group concept, with the thought that peer pressure will change individual and group behavior. As Braga and Kennedy (2012) described, meaningful enforcement actions and scrutiny by law enforcement agencies can leverage the rationality of group members to no longer encourage norms that provoke the outbreaks of violence. The citywide communication of the antiviolence message, coupled with meaningful examples of the consequences that will be brought to bear on groups that break the rules, can weaken or eliminate the "kill or be killed" norm as individuals recognize that their enemies will be operating under the new rules as well.
Changes in-group norms and in objective risks associated with particular forms of misbehavior may, for example, make it more difficult to recruit peers for particular instances of co-offending.

Ethnographic research findings on illicit gun markets in Chicago have
shown that gangs' assessment of the law enforcement responses to gun violence leads them to withhold access to firearms for younger and more impulsive members (Cook, Ludwig, Venkatesh, & Braga, 2007). DMI's goal of fundamentally disrupting overt drug markets can greatly enhance the difficulty of drug dealing: when buyers no longer routinely "cruise" once active markets, even a motivated street dealer may find it impossible to do business.

| Other theoretical perspectives
Many scholars have suggested there are other complementary violence reduction mechanisms at work in the focused deterrence strategies described here that need to be highlighted and better understood (Braga, 2012;Brunson, 2015;Corsaro & Engel, 2015). In Durlauf and Nagin's (2011) article, their focus is on the possibilities for increasing perceived risk and deterrence by increasing police presence. Nevertheless, in the focused deterrence approach, the emphasis is not only on increasing the risks associated with offending, but it is also on decreasing opportunity structures for crime, deflecting offenders away from crime, increasing the collective efficacy of communities, and increasing the legitimacy of police actions. Indeed, program designers and implementers sought to generate large crime reduction impacts from the multifaceted ways in which this strategy influences targeted offenders (Kennedy, 2011). Discouragement emphasizes reducing the opportunities for crime and increasing alternative opportunity structures for offenders (Clarke, 1997). In this context, situational crime prevention techniques are often implemented as part of the core pulling levers work in focused deterrence strategies (Braga & Kennedy, 2012). Extending guardianship, assisting natural surveillance, strengthening formal surveillance, reducing the anonymity of offenders, and using place managers can greatly enhance the range and the quality of the varying enforcement and regulatory levers that can be pulled on offending groups and key actors in criminal networks. The focused deterrence approach also is aimed at redirecting offenders away from crime through the provision of social services and opportunities.
Treated individuals are offered job training, employment, substance abuse treatment, housing assistance, and a variety of other services and opportunities. Sampson, Raudenbush, and Earls (1997) emphasized the capacity of a community to realize common values and regulate behavior within it through cohesive relationships and mutual trust among residents. They argued that the key factor determining whether crime will flourish is a sense of the "collective efficacy" of a community. A community with strong collective efficacy is characterized as having high capacities for collective action for the public good. The use of focused deterrence strategies enhances collective efficacy in communities by emphasizing the importance of engaging and enlisting community members in the strategies developed. Implementation of the High Point DMI strategy, for example, drew on collective efficacy principles by engaging family, friends, and other "influential" community members in addressing the criminal behaviors of local drug dealers (Kennedy & Wong, 2009).
Community-based action in focused deterrence strategies helps remove the justifications used by offenders to explain away their responsibility for the targeted behavior. In call-ins and on the street, community members effectively invalidate the excuses for criminal behavior by challenging the norms and narratives that point to racism, poverty, injustice, and the like. In Boston, for example, Black clergy challenged gang members who attempted to use these excuses by countering that poverty, racism, and injustice were not linked to their decisions to fire shots in their neighborhoods and kill other young people who have experienced the same societal ills and life difficulties (Braga, Kennedy, Waring, & Piehl, 2001). Community members also work with law enforcement and social service agencies to (a) set basic rules for group-involved offenders such as "don't shoot guns" and (b) alert the conscience of these offenders by appealing to moral values inherent in taking the life of another, causing harm to their neighborhood, or the pain that would be experienced by their mothers if they were killed or sent to prison for a long time in a far-away location (Kennedy, 2011).
Finally, use of the focused deterrence approach takes advantage of recent theorizing regarding procedural justice and legitimacy. The effectiveness of policing is dependent on public perceptions of the legitimacy of police actions (Tyler, 2004). Legitimacy is the public belief that there is a responsibility and obligation to accept and defer voluntarily to the decisions made by authorities (Tyler, 2006).
Findings from recent studies reveal that when procedural justice approaches are used by the police, citizens will not only evaluate the legitimacy of the police more highly, but they will also be more likely to obey the law in the future (Paternoster, Brame, Bachman, & Sherman, 1997; but see Nagin & Telep, 2017). Advocates of focused deterrence strategies argue that targeted offenders should be BRAGA ET AL. | 5 of 65 treated with respect and dignity (Kennedy, 2008(Kennedy, , 2011, reflecting procedural justice principles. The Chicago Project Safe Neighborhood (PSN) strategy, for instance, was aimed at increasing the likelihood that the offenders would "buy-in" and comply voluntarily with the prosocial, antiviolence norms being advocated by interacting with offenders in ways that enhance procedural justice in their communication sessions (Papachristos et al., 2007).

| Why it is important to do the review
In the previous version of this Campbell Collaboration systematic review, 10 quasi-experimental evaluations of the crime control impacts of focused deterrence programs were identified based on a search for eligible studies completed in 2010 (Braga & Weisburd, 2012, 2011. In that review, researchers found that focused deterrence strategies were associated with significant reductions in targeted crime problems. Although the authors concluded that the available evidence was highly supportive of crime reduction impacts (Braga & Weisburd, 2012, 2011, they noted the absence of randomized experiments and the fact that, in several of the included evaluations, weaker designs were used with nonequivalent comparisons. The small number of studies and the preponderance of weaker evaluation designs, however, contribute to some healthy ongoing skepticism regarding the crime control benefits associated with focused deterrence programs among practitioners and crime policy scholars. The evaluation of the best-known focused deterrence strategy, Boston's Operation Ceasefire (Braga et al., 2001;Piehl, Cooper, Braga, & Kennedy, 2003), has been greeted with both a healthy dose of skepticism (Fagan, 2002;Rosenfeld, Fornango, & Baumer, 2005) and some support (Cook & Ludwig, 2006;Morgan & Winship, 2007). The National Academy of Sciences' report on firearms data and research concluded that the Ceasefire quasi-experimental evaluation was "compelling" in associating the intervention with a 63% reduction in youth homicide in Boston (National Research Council, 2005, p. 10); however, the report also stated that the lack of a randomized controlled trial left some doubt over how much of the decline was due to Ceasefire relative to other rival causal factors. Despite this uncertainty over the impact of the Boston Ceasefire strategy on youth homicide, the focused deterrence framework has been applied in many U.S. cities through federally sponsored violence prevention programs such as the Strategic Alternatives to Community Safety Initiative and Project Safe Neighborhoods (Dalton, 2002).
Former New York City Mayor Rudy Giuliani criticized the "Boston Model" as not leading to lasting crime control gains in his 2001 farewell address (The New York Times, 2001). In an article published in The New Yorker, well-respected deterrence scholar Professor Franklin Zimring is quoted as lamenting the lack of rigorous evaluations of focused deterrence programs and, when assessing the Boston experience, suggested, "Ceasefire is more of a theory of treatment rather than a proven strategy" (Seabrook, 2009, p. 37). Other criminologists seem unaware of the existing empirical evidence. For instance, in his 2013 summary of the crime prevention value of focused deterrence programs, former National Council on Crime and Delinquency president Barry Krisberg reported, "It certainly hasn't been effective so far, and there is no information suggesting it is effective" (as interviewed by KTVU, 2013).
Recently, more cities have tested the focused deterrence approach to control gang violence, disorderly drug markets, and repeat offender problems. The National Network for Safe Communities, an applied research project of the John Jay College of Criminal Justice, provides support to some 42 U.S. cities who are implementing some version of a focused deterrence strategy. 1 A few other countries have started to test the approach. For instance, a focused deterrence program has been implemented targeting youth violence in Glasgow, Scotland (Deuchar, 2013). Police executives and other public officials in Eastern European and South American countries, such as Turkey and Brazil, have also explored the possibility of implementing focused deterrence strategies to control gang and group-related violence in their cities (National Network for Safe Communities, 2013).
Given the growing popularity of focused deterrence programs and conflicting scholarly views on the crime reduction value associated with the approach, ongoing systematic review of rigorous program evaluations is necessary to keep policy and practice debates rooted in the most up-to-date and comprehensive scientific evidence.

| OBJECTIVES
The objective of this review is to synthesize the existing published and non-published empirical evidence on the effects of pulling levers focused deterrence strategies on crime and to provide a systematic assessment of the preventive value of this approach.

Types of studies
To be eligible for this review, interventions had to include the key components of a focused deterrence strategy as described above.
Randomized experimental and quasi-experimental (nonrandomized) designs that compared pre and postintervention measures were eligible for inclusion in this review, though we did not identify any randomized experiments in our search (Campbell & Stanley, 1966;Shadish, Cook, & Campbell, 2002). Eligible quasi-experiments used a comparison group or one-group-only interrupted time-series design 1 For a complete list of cities supported by the National Network for Safe Communities, go to https://nnscommunities.org/impact/cities (last accessed May 24, 2019). that controlled for extraneous factors to analyze variations in crime trends pre and postintervention. 2

Types of units of analysis
The units of analysis could be areas, such as cities, neighborhoods, or police beats, or persons.

Types of interventions
To be eligible for this review, interventions had to be identified as a focused deterrence strategy. As described by Kennedy (2006, pp. 156-157), pulling levers operations have tended to follow this basic framework: • Selection of a specific crime problem, such as youth homicide or street drug dealing.
• Assembling an interagency enforcement group, typically including police, probation, parole, state and federal prosecutors, and sometimes federal enforcement agencies.
• Conducting research, usually relying heavily on the field experience of front-line police officers, to identify key offenders-and frequently groups of offenders, such as street gangs, drug crews, and the like-and the context of their behavior.
• Framing a special enforcement operation directed at those offenders and groups of offenders, and designed to substantially influence that context, for example by using any and all legal tools (or levers) to sanction groups such as crack crews whose members commit serious violence.
• Matching those enforcement operations with parallel efforts to direct services and the moral voices of affected communities to those same offenders and groups.
• Communicating directly and repeatedly with offenders and groups to let them know that they are under particular scrutiny, what acts (such as shootings) will get special attention when that has in fact happened to particular offenders and groups, and what they can do to avoid enforcement action. One form of this communication is the "forum," "notification," or "call-in," in which offenders are invited or directed (usually because they are on probation or parole) to attend face-to-face meetings with law enforcement officials, service providers, and community figures.
We used this basic framework to assist in our determination of whether particular programs followed the focused deterrence approach. It is important to note here, however, that certain programs that were determined to be eligible for this review did not necessarily follow the very specific pulling levers steps identified by Kennedy (2006). Focused deterrence strategies are often framed as problem-oriented exercises where specific recurring crime problems are analyzed and responses are highly customized to local conditions and operational capacities. As such, we fully anticipated a variety of focused deterrence strategies to be identified by our systematic review.

Types of outcome measures
Eligible studies had to measure the effects of the focused deterrence intervention on officially recorded levels of crime at places or crime by individuals. Appropriate crime measures included crime incident reports, citizen emergency calls for service, and arrest data. Particular attention was paid to studies that measured crime displacement effects and diffusion of crime control benefit effects (Clarke & Weisburd, 1994;Reppetto, 1976). The review considered all forms of displacement and diffusion reported by the studies (e.g., spatial, temporal, target, modus operandi). Also, assessed was the quality of the methodologies used to measure displacement and diffusion effects.
After finishing the above searches and reviewing the studies as described later, we emailed the list of studies meeting our eligibility criteria in December 2015 to leading criminal justice scholars knowledgeable in the area of focused deterrence strategies (see Appendix A). These 100 scholars were defined as those who authored at least one study which appeared on our inclusion list, anyone involved with U.S. National Research Council (2004 The previous iteration of this systematic review did not include studies with one-group-only interrupted time-series design designs. As will be shown below, the updated review identified only one such study (Delaney, 2006). reviews of police research and firearms research, and other leading scholars identified by the authors. This helped us identify unpublished studies that did not appear in conventional databases or other reviews. Finally, we consulted with an information retrieval specialist at the outset of our review and at points along the way in order to ensure that appropriate search strategies were used to identify the studies meeting the criteria of this review. 4 The following 15 databases were searched:

| Data collection and analysis
Two authors (Braga and Turchan) executed the varied search strategies to identify eligible studies. Abstracts that appeared to have a chance of fitting the eligibility criteria were added to a centralized list for further consideration. Once the initial search strategies were completed, the list of abstracts was jointly reviewed by two authors (Braga and Turchan). For abstracts that both reviewers believed had a reasonable likelihood of meeting the eligibility criteria, full-text reports, journal articles, and books were obtained and analysed in-depth. The third author (Weisburd) weighed in when there were any disagreements about the potential eligibility of a particular study.

| Details of study coding categories
All eligible studies were coded (see coding protocol in Appendix B) on a variety of criteria including: a. Reference information (title, authors, publication, etc.) b. Nature of description of a selection of site, problems, and so forth.

| Statistical procedures and conventions
Analysis of outcome measures across studies was carried out in a uniform manner and, when appropriate and possible, involved quantitative analytical methods. We conducted meta-analyses of program effects to determine the size and direction of the effects and to weight effect sizes based on the variance of the effect size and the study sample size (Lipsey & Wilson, 2001). In this systematic review, the standardized mean difference effect size (also known as Cohen's d; see Rosenthal, 1994)  Biostat's Comprehensive Meta-Analysis Version 2.2 was then used to conduct the meta-analysis of effect sizes. Computation of effect sizes in the studies was not always direct. The goal was to convert all observed effects into a standardized mean difference effect size 4 Ms. Phyllis Schultze of the Gottfredson Library at the Rutgers University School of Criminal Justice assisted with the initial abstract search and was consulted throughout on our search strategies. 5 Maintained by the Gottfredson Library at the Rutgers University School of Criminal Justice. 6 The now-defunct Campbell Collaboration Social, Psychological, Educational and Criminological Trials Register was consulted for the original version of this systematic review. 7 https://www.campbellcollaboration.org/effect-size-calculato.html. metric. Indeed, it was sometimes difficult to develop precise effect size metrics from published materials. This reflects a more general problem in crime and justice with "reporting validity" (Farrington, 2006;Lösel & Köferl, 1989) and has been documented in reviews of reporting validity in crime and justice studies (see Perry & Johnson, 2008;Perry, Weisburd, & Hewitt, 2010).
Some studies reported Cohen's d as a key outcome measure (see, e.g., Saunders, Lundberg, Braga, Ridgeway, & Miles, 2015) and, after confirming that the studies used the appropriate methods, these effects sizes were included. In other evaluations, treatment and control group crime counts were used to calculate effect sizes for each study contrast. From these raw counts, Odds Ratios (ORs) were first calculated. The log of this OR was then multiplied by √3/π in order to attain the final effect size expressed as Cohen's d (see Hasselblad & Hedges, 1995). We then made an adjustment for over-dispersion using the method in Farrington, Gill, Waples, and Argomaniz (2007). In a few studies, counts were not provided or could not be reconstructed from information in the study report.
This was most often in papers that reported Incidence Rate Ratios (IRRs) in order to estimate treatment effects conditional on the use of covariates. In such cases, ORs were obtained by taking the product of the IRR and a ratio of the pretest means in the control and treatment group (OR = IRR × [mean_pre_C/mean pre_T]). This then allows d to be calculated from log OR using conventional methods. The standard error of this IRR is squared to obtain the variance. In the interrupted time series designs, we used standards and methods to estimate d as outlined by the Cochrane Group. 8

| Determination of independent findings
One problem in conducting meta-analyses in crime and justice is that investigators often do not prioritize outcomes examined. This is common in studies in the social sciences in which authors view the good practice as demanding that all relevant outcomes be reported.
However, the lack of prioritization of outcomes in a study raises the question of how to derive an overall effect of treatment. For example, the reporting of one significant result may reflect a type of "creaming" in which the authors focus on one large and significant finding while ignoring the less positive results of other outcomes. But authors commonly view the presentation of multiple findings as a method for identifying the specific contexts in which the treatment is effective. When the number of such comparisons is small and therefore unlikely to affect the error rates for specific comparisons such an approach is often valid.
All studies were analyzed using three approaches. The first approach is conservative; we calculated an overall mean effect size for each study that combined all reported outcomes. The second represents the largest effect reported in the studies and offers an upper bound to the review findings. It is important to note that in some of the studies with more than one outcome reported, the largest outcome reflected what authors thought would be the most direct program effect. Finally, the smallest effect size for each study was analyzed. This approach is the most conservative and likely underestimates the effect of disorder policing programs on crime. It was used here primarily to provide a lower bound to the review findings. In short, using three approaches to testing program effects served as a sensitivity analysis to evaluate the effects of input variation on the output variation 6 | RESULTS 6.1 | Selection of studies 6.1.1 | Results of the search Search strategies in the systematic review process generate a large number of citations and abstracts for potentially relevant studies that must be closely screened to determine whether the studies meet the eligibility criteria (Farrington & Petrosino, 2001).
The screening process yields a much smaller pool of eligible studies for inclusion in the review. Combined with the results from the original review, the search strategies produced 62,541 distinct abstracts. The contents of these abstracts were reviewed for any suggestion of an evaluation of focused deterrence interventions. A total of 473 distinct abstracts were selected for further consideration. A joint review of these initially identified abstracts determined 131 abstracts had a reasonable likelihood of meeting the eligibility criteria and warranted in-depth examination. For these 131 abstracts, full-text reports, journal articles, and books were acquired then carefully assessed to determine whether the interventions involved focused deterrence strategies and whether the studies used randomized controlled trial designs or nonrandomized quasi-experimental designs (excluded studies are reported in Appendix D). Twenty-four eligible studies were identified and included in the updated review 1. Operation Ceasefire in Massachusetts (Braga et al., 2001) 2. Indianapolis Violence Reduction Partnership in Indianapolis, Indiana (McGarrell et al., 2006) 3. Operation Peacekeeper in Stockton, California (Braga, 2008b) 4. Project Safe Neighborhoods in Lowell, Massachusetts (Braga, Pierce, McDevitt, Bond, & Cronin, 2008) 5. Cincinnati Initiative to Reduce Violence in Cincinnati, Ohio (Engel, Corsaro, & Tillyer, 2010) 6. Operation Ceasefire in Newark, New Jersey (Boyle, Lanterman, Pascarella, & Cheng, 2010) 7. Operation Ceasefire in Los Angeles, California (Tita, Riley, & Greenwood, 2003) 8. Operation Ceasefire in Rochester, New York (Delaney, 2006) 9. Project Safe Neighborhoods in Chicago, Illinois (Papachristos et al., 2007) 8 https://methods.cochrane.org/sites/methods.cochrane.org.bias/files/public/uploads/EPOC %20Data%20Collection%20Checklist.pdf. 10. Drug Market Intervention in Nashville, Tennessee  11. Drug Market Intervention in Rockford, Illinois (Corsaro, Brunson, & McGarrell, 2009) 12. Drug Market Intervention in High Point, North Carolina (Corsaro, Hunt, Hipple, & McGarrell, 2012) 13. Drug Market Intervention in Peoria, Illinois (Corsaro & Brunson, 2013) 14. Operation Ceasefire II in Boston, MA  15. Community Initiative to Reduce Violence in Glasgow, Scotland (Williams, Currie, Linden, & Donnelly, 2014) 16. Group Violence Reduction Strategy in Chicago, Illinois (Papachristos & Kirk, 2015) 17. Group Violence Reduction Strategy in New Orleans, Louisiana (Corsaro & Engel, 2015) 18. No Violence Alliance in Kansas City, Missouri (Fox, Novak, & Yaghoub, 2015) 19. Project Longevity in New Haven, Connecticut (Sierra-Arevalo, Charette, & Papachristos, 2015) 20. Drug Market Intervention in Roanoke, Virginia (Saunders, Kilmer, & Ober, 2015) 21. Drug Market Intervention in Montgomery County, Maryland  22. Drug Market Intervention in Guntersville, Alabama  23. Drug Market Intervention in Seattle, Washington  24. Drug Market Intervention in Ocala, Florida  6.2 | Characteristics of eligible studies The 14 newly identified studies represent a large increase in eligible studies (140%) over the 10 evaluations considered in the previous systematic review. Table 1 summarizes the characteristics of the 24 selected studies. The selected studies examined focused deterrence interventions that were implemented in small, medium, and large cities. Only one study evaluated a focused deterrence program implemented in a jurisdiction outside the United States (Scotland).
More than one-third (N = 9, 37.5%) of the eligible studies were acquired through "grey literature" sources 9 at the time the review of abstracts was completed. 10   The grey literature is a term applied to sources of information that are not commercially published and is typically composed of technical reports, working papers, government and agency reports, and conference proceedings. Wilson (2009) has argued that there is often little difference in methodological quality between published and unpublished studies, suggesting the importance of searching the grey literature. 10 During the development of this report, the New Haven study was accepted for publication at Crime & Delinquency and the Roanoke study was accepted for publication at Journal of the American Statistical Association. Violence reduction strategy targeting individual gang members described as a "hybrid" between the Boston Ceasefire pulling levers strategy and the Chicago Ceasefire street worker program. 85-week postintervention period (May 11, 2005-December 31, 2006. No threats to integrity of treatment noted during program implementation.
Intervention implemented in two square mile area that experienced elevated levels of gun violence. Outcome measure was the weekly number of gunshot wound incidents.
Near-equivalent quasi-experiment comparing gunshot wound trends in the targeted area relative to gunshot wound trends in a comparison area selected based on similar levels of gun violence, geographic size, and demographic characteristics. ARIMA models controlling for trends and seasonal variations used to estimate impact of intervention on time series. Used dual kernel density spatial analyses to examine the distribution of gunshot wound hot spots around target and comparison zones before and after the intervention was implemented.
(Continues) BRAGA ET AL.  Fox et al. (2015) Strategy focused on reducing serious violence by street gangs and criminally active groups. 12-month postintervention period. Early implementation was plagued by poor leadership and communication which delayed full implementation until nearly 1 year after the originally intended start day and those problems have been rectified.
Citywide intervention. Outcome measures include monthly counts of homicide and aggravated assault with a firearm.
One-group-only interrupted time series evaluation used to compare citywide trends in the preintervention period to 1, 3, 6, and 12 months postintervention time periods.
No threats to integrity of treatment noted during program implementation.
Citywide intervention. Outcome measures were the monthly counts of citywide total fatal and nonfatal shootings, group-memberinvolved shootings, and nongroup member-involved shootings.
Non-equivalent quasi-experimental design comparing shooting trends in New Haven to a similar city (Hartford, CT). ARIMA models controlling for trends and seasonal variations used to estimate the impact of the intervention on the time series.

Drug Market
Intervention. Roanoke, VA. Saunders et al. (2015) Strategy focused on reducing crime driven by street-level drug market. 12-month postintervention period (Beginning December 2011 for Hurt Park and January 2013 for Melrose-Rugby). The treatment was undermined due to a history of poor police-community relations and lack of faith among residents in police despite the program's efforts to engage the community.
Intervention was implemented in the Hurt Park and Melrose-Rugby neighborhoods.
Outcome measures were 12-month counts of total crime, violent crime, property crime, and drug crime.
Quasi-experimental evaluation comparing trends in the targeted neighborhood to trends in comparison neighborhoods matched via synthetic control methods. Negative binomial regression models controlling for trends were used to estimate the impact of the intervention on the time series.
Drug Market Intervention Montgomery County, MD Saunders et al. (2015) Strategy focused on reducing crime driven by street-level drug market 12-month postintervention period (March 2011-February 2012) The treatment was undermined due to a lack of community engagement Intervention was implemented in the Damascus Gardens one-square block apartment complex Outcome measures were 12-month counts of total crime, violent crime, property crime, and drug crime Quasi-experimental evaluation comparing trends in the targeted neighborhood to trends in comparison neighborhoods matched via synthetic control methods Negative binomial regression models controlling for trends were used to estimate the impact of the intervention on the time series Drug Market Intervention. Guntersville, AL. Saunders et al. (2015) Strategy focused on reducing crime driven by street-level drug market. 12-month postintervention period (December 2011-November 2012).
Intervention was implemented in the Lakeview neighborhood. Outcome measures were 12-month counts of total crime, violent crime, property crime, and drug crime.
Quasi-experimental evaluation comparing trends in the targeted neighborhood to trends in comparison neighborhoods matched via synthetic control methods.
(Continues) The lack of randomized controlled trials is concerning, as well implemented randomized studies provide the strongest evidence of the causal impacts of programs or practices. Nonetheless, our review suggests that, in recent years, program evaluators have increasingly used more rigorous quasi-experimental designs with matched comparison groups to estimate focused deterrence impacts. The previous iteration of this Campbell review (Braga & Weisburd, 2012, 2011 found that only 30% ( Table 3, the Boston Ceasefire I intervention was associated with a 63% decrease in youth homicides that was distinct from youth homicide trends in the comparison cities.
The Boston Ceasefire I evaluation has been reviewed by a number of researchers and the relationship between program implementation and the subsequent trajectory of youth homicide in Boston during the 1990s has been closely scrutinized (see Fagan, 2002;Ludwig, 2005

Study Treatment Units of Analysis Research Design
The treatment was undermined due to a lack of citizen involvement, as well as community distrust for police and faith leaders involved in the intervention.
Negative binomial regression models controlling for trends were used to estimate the impact of the intervention on the time series.

Drug Market
Intervention. Seattle, WA. Saunders et al. (2015) Strategy focused on reducing crime driven by street-level drug market. 12-month postintervention period (Beginning December 2009 for 23rd Street and January 2013 for International District).
No threats to integrity of treatment noted during program implementation.
Intervention was implemented in the areas of the 23rd Street Corridor and International District. Outcome measures were 12-month counts of total crime, violent crime, property crime, and drug crime.
Quasi-experimental evaluation comparing trends in the targeted neighborhood to trends in comparison neighborhoods matched via synthetic control methods. Negative binomial regression models controlling for trends were used to estimate the impact of the intervention on the time series.

Drug Market
Intervention. Ocala, FL. Saunders et al. (2015) Strategy focused on reducing crime driven by street-level drug market. 12-month postintervention period (Beginning November 2009 for Second Chance and October 2010 for First Avenue).
No threats to integrity of treatment noted during program implementation.
Intervention was implemented in the "Second Chance" neighborhood and the First Avenue housing project. Outcome measures were 12-month counts of total crime, violent crime, property crime, and drug crime.
Quasi-experimental evaluation comparing trends in the targeted neighborhood to trends in comparison neighborhoods matched via synthetic control methods. Negative binomial regression models controlling for trends were used to estimate the impact of the intervention on the time series.
Abbreviation: ARIMA, autoregressive integrated moving average. 11 Table 2 summarizes the characteristics of the key focused deterrence evaluation identified through the varied search processes. Five evaluations had companion quasi-experimental analyses that supported the program impact conclusions presented here: Boston Ceasefire I (Piehl et al., 2003), Boston Ceasefire II (Braga, Apel, & Welsh, 2013), Chicago PSN (Wallace, Papachristos, Meares, & Fagan, 2016), High Point DMI (Corsaro, 2013), and Indianapolis ). In addition to the Corsaro et al. (2012) evaluation, the RAND Corporation completed an independent evaluation of the High Point DMI using a synthetic control quasi-experimental design . The RAND evaluation found a slightly stronger impact of the DMI program on targeted outcomes. They find that in the year following a DMI, calls for service decreased 16% and violent crimes decreased 34%, on average, compared to synthetic control markets. The RAND evaluation also found no evidence of statistically significant crime displacement or diffusion effects after a DMI was implemented. No statistically significant reduction in gunshot wound victims in target zone The results of the displacement/diffusion analysis were inconclusive Operation Ceasefire. Los Angeles, CA. Tita et al. (2004) In Boyle Heights, gang crime decreased significantly compared with other regions of Hollenbeck during the suppression period of the intervention, and violent, gang, and gun crime all decreased significantly in the deterrence period. In the five targeted police reporting districts, violent crime decreased significantly in comparison with the rest of Boyle Heights in the suppression and the deterrence periods, and gang crime decreased significantly in the suppression period. In the Census block groups overlapping the targeted reporting districts, violent crime decreased significantly compared with the matched blocks.
Analyses suggested strong diffusion of crime control benefits into Census block groups immediately surrounding targeted area and a reduction in gang crime associated with the "socially tied" gangs.  (2015) Statistically significant 32% reduction in shooting victimization among targeted gangs relative to matched comparisons. Marginally significant 23% reduction in total shooting involvement among targeted gangs relative to matched comparisons.

Displacement/diffusion effects not measured
Group Violence Reduction Strategy. New Orleans, LA. Corsaro and Engel (2015) Statistically significant 17% reduction in total homicides, 32% reduction in gang-member-involved homicides, 17% reduction in firearm homicides, and 17% reduction in nonfatal firearm assaults. No statistically significant relationship with nongangmember-involved homicides.

Displacement/diffusion effects not measured
No Violence Alliance. Kansas City, MO. Fox et al. (2015) Statistically significant homicide reductions of 40% at one month, 34% at three months, and 29% at 6 months. Statistically significant gun-involved aggravated assault reductions of 19% at one month and 14% at 3 months No statistically significant relationship with homicide at 12 months or gun-involved aggravated assault at 6 and 12 months.

Displacement/diffusion effects not measured
Project Longevity. New Haven, CT. Sierra-Arevalo et al. (2015) Statistically significant 37% reduction in total shootings and homicides and 73% reduction in group-memberinvolved homicides and shootings Displacement/diffusion effects not measured Drug Market Intervention. Roanoke, VA.  In the Hurt Park neighborhood, statistically significant 30% reduction in total crime at 3 months, 19% at 6 months, 28% at 9 months, and 23% at 12 months; statistically significant 45% reduction in property crime at 6 months, 57% at 9 months, and 50% at 12 months, as well as significant 24% reduction in violent crime at 3 months and 29% at 9 months. In the Melrose-Rugby neighborhood, statistically significant 15% reduction in violent crime at 3 months and 34% at 6 months.

Displacement/diffusion effects not measured
Drug Market Intervention. Montgomery County, MD. Saunders et al. (2015) No statistically significant reduction in total crime, violent crime, property crime, or drug crime Displacement/diffusion effects not measured Drug Market Intervention. Guntersville, AL. Saunders et al. (2015) No statistically significant reduction in total crime, violent crime, property crime, or drug crime Displacement/diffusion effects not measured Drug Market Intervention. Seattle, WA.  In the International District, statistically significant 15% reduction in total crime at 3 months and 6 months; statistically significant 8% reduction in property crime at 3 months and marginally significant 17% reduction at 6 months; statistically significant 53% reduction in violent crime at 3 months and marginally significant 40% reduction at 6 months, 34% reduction at 9 months, and 34% reduction at 12 months; statistically  groups (Delaney, 2006). DMI programs in Guntersville, Montgomery County, Peoria, and Roanoke were noted to suffer from a lack of community involvement in the targeted areas (Corsaro & Brunson, 2013;Saunders, Ober, Kilmer, & Greathouse, 2016). 12 T A B L E 3 (Continued)

Study
Crime outcomes Displacement/diffusion significant 29% reduction in drug crime at 3 months and marginally significant 17% reduction at 6 months. In the 23rd Street Corridor neighborhood, no statistically significant reduction in total crime, violent crime, property crime, or drug crime.
Drug Market Intervention.
Ocala, FL. Saunders et al. (2015) No statistically significant reduction in total crime, violent crime, property crime, or drug crime for either intervention site Displacement/diffusion effects not measured 12 The National Network for Safe Communities raised concerns to the RAND Corporation over the treatment fidelity of the DMI programs that were sponsored by the U.S. Bureau of Justice Assistance and implemented under the guidance of a technical assistance team from Michigan State University. One key concern centered on the absence of reconciliation efforts between police and affected communities on perceived harms associated with prior drug control tactics. Reconciliation is viewed as a critical component of developing the necessary community support needed to exert informal social control over drug sellers in targeted overt drug markets. Further concerns involved unclear definitions of the drug market areas to be targeted for intervention, a lack of opportunity and resources available Table 3 summarizes the main effects of the intervention on crime outcomes and, when measured, crime displacement and diffusion of crime control benefits effects. A more detailed narrative review of the focused deterrence strategies contained in the eligible studies is provided in Appendix C.
Nineteen of the twenty-four studies (79.2%) reported noteworthy crime reductions associated with the focused deterrence approach. (Table 3). All 12 studies evaluating the impacts of focused deterrence strategies on violence by gangs and criminally active groups reported at least one statistically significant crime control impact associated with program implementation. While a nonstatistically significant reduction in gunshot wound victimization in the target zone was noted, the evaluation of Newark's Operation Ceasefire did not report any statistically significant crime prevention benefits generated by focusing on individual violent gang members.
The other four studies that did not report any noteworthy crime control impacts were DMI programs implemented in Guntersville, Montgomery County, Ocala, and Peoria.
Five studies conducted six tests of possible crime displacement and diffusion of crime control benefits associated with the evaluated focused deterrence programs (Table 3). These studies included gang/ group violence reduction strategies (Boston II, Los Angeles), DMI programs (High Point, Nashville), and individual repeat offender strategies (Newark). Two of the four studies that measured whether crime levels were impacted in areas immediate proximate to treatment areas reported noteworthy diffusion of crime control benefits associated with the focused deterrence intervention (Nashville, Los Angeles); none reported significant crime displacement effects into surrounding areas.
Two focused deterrence studies investigated the existence of displacement and diffusion effects on the criminal behavior of gangs that were socially connected to targeted groups. The Los Angeles intervention targeted two rival gangs operating out of the same area (Hollenbeck). Criminal activity (e.g., violent, gang, and gun crimes) was substantially reduced among the two gangs over a 6-month pre-post period. Slightly larger reductions in these crimes were evident among four nontargeted, rival gangs in surrounding areas during the same time period. Part of the explanation for the diffusion effects may rest with fewer feuds between the targeted and nontargeted gangs. Tita et al. (2003) also speculated that diffusion effects may have been influenced by social ties among the targeted and rival gangs. This seemed to be especially the case for gang crimes involving guns. In a companion paper to the main effects program evaluation, Braga et al. (2013) found that the Boston Ceasefire II strategy also created spillover deterrent effects onto other gangs that were socially connected to targeted gangs through rivalries and alliances. Total shootings involving these "vicariously treated" gangs decreased by 24% relative to total shootings involving matched comparison gangs. Table 4 presents our assessment of the risk of bias in the N = 24 included focused deterrence studies. We assessed the level of risk of bias along with six sources of potential bias for each study ("Low" or  (Papachristos & Kirk, 2015). Twelve quasi-experimental studies (50.0%) used treatment and control units that were not the same.

| Risk of bias in included studies
For instance, the Indianapolis quasi-experimental evaluation compared homicide trends in the city relative to homicide trends in other nonequivalent cities (McGarrell et al., 2006). Two studies used interrupted time series analyses without equivalent comparisons (Delaney, 2006;Fox et al., 2015).
None of the included studies reported any evidence of contamination of control conditions during the intervention period.
Further, none of the included studies reported evidence suggestive that the evaluators were only selecting those crime types that showed an effect. Finally, only seven studies (29.2%) presented any other evidence of possible bias. As described earlier, these studies suggested that there were serious threats to the integrity of the treatments applied. The internal validity of the included studies was mixed. Among the eligible quasi-experimental designs, the strength of the research design varied. Therefore, we conducted sensitivity analyses that tested the moderating effects of research design on the relationship between focused deterrence programs and crime outcomes. BRAGA ET AL.

| 19 of 65
Because the studies vary in their contexts and approaches, which is indicated by a significant Q statistic (Q = 122.568, df = 23, p < .05, Tau 2 = 0.053), we used a random effects model to estimate the overall mean effect size. The meta-analysis of effect sizes suggests a statistically significant effect in favor of focused deterrence strategies. The overall effect size for these studies was 0.383 (p < .05; 95% CI = 0.264, 0.503). This is below Cohen's (1988) standard of 0.50 for a medium effect size. Nonetheless, the overall effect size is relatively large compared to assessments of interventions in crime and justice work more generally (Lipsey, 2000;MacKenzie & Hickman, 1998;Weisburd, 1993;Weisburd et al., 2008).  To what degree did this study engage in selective reporting? f How much did other reported risks of bias threaten the integrity of this study? 13 Random effects models were used to estimate the overall standardized mean effect sizes. For the largest effect size meta-analysis, Q = 152.740, df = 23, p < .05, Tau 2 = 0.093. For the smallest effect size meta-analysis, Q = 109.537, df = 23, p < .05, Tau 2 = 0.062.

| Program type and research design as effect size moderators
Focused deterrence strategies have been directed at reducing crime by street gangs and criminally active groups, overt drug markets, and high-risk individuals. These programs represent differing applications of focused deterrence strategies to control distinct types of problems. The inclusion of moderator variables, such as program and research design types, help to explain and understand differences across studies in the outcomes observed (Lipsey, 2003). Figure 4 presents a random effects model examining the mean effect sizes for the three different program types. It is important to note that the Q-statistic associated with the between-group variation was large and statistically significant (Q = 41.555, df = 2, p < .05), suggesting that program type was influential in determining effect sizes. The gang/group intervention programs were associated with the largest within-group effect size (0.657, p < .05), followed by the high-risk individuals programs (0.204, p < .05) and the DMI programs (0.091, p < .05). When program type was included as a moderator, the metaanalysis estimated a more modest overall effect size (0.229, p < .05).
The smaller mean effect size associated with the DMI programs was influenced by the noteworthy share of programs with reported threats to the integrity of the focused deterrence treatment. Not surprisingly, DMI programs that were implemented with higher treatment fidelity generated larger overall crime reduction impacts.
As mentioned earlier, four of the nine (44.4%) eligible DMI programs suffered from implementation difficulties centered on securing the necessary community involvement in targeted drug market areas (Guntersville, Montgomery County, Peoria, and Roanoke). When treatment integrity was included as an effect size moderator for the nine DMI studies, programs with noted implementation issues had a smaller nonstatistically significant mean effect size (0.053). In contrast, the mean effect size for DMI programs without implementation difficulties suggested a modest, statistically significant crime reduction impact (0.184, p < .05).
The meta-analysis in the previous iteration of the Campbell systematic review estimated a larger overall mean effect size (0.604) relative to the current meta-analysis (0.383). This difference is primarily due to the greater prevalence of more rigorous quasiexperimental designs with higher levels of internal validity among the studies included in the current systematic review. It is well known among social scientists that program evaluations with more rigorous research designs tend to report null effects. As Peter H. Rossi's Iron Law of Evaluation states, "The expected value of any net impact assessment of any large scale social program is zero" (Rossi, 1987, p

Mean Effect Sizes for Study Outcomes
F I G U R E 1 Mean effect sizes for study outcomes designed the impact assessment of a social program, the more likely is the resulting estimate of net impact to be zero" (Rossi, 1987, p. 3).
Given the important distinction in methodological quality between the nonequivalent and matched quasi-experimental studies, we examined research design as a moderator variable. Inference, 1999), recent reviews in crime and justice suggest that weaker research designs often lead to more positive outcomes (Weisburd, Lum, & Petrosino, 2001;Welsh, Peel, Farrington, Elffers, & Braga, 2011).

| Publication bias
Publication bias, generally defined as the concern that the collection of studies easily available to a reviewer represents those studies most likely to have statistically significant results, presents a strong challenge to any review of evaluation studies (Rothstein, 2008). The credibility of a review arguably depends more heavily on the collection of studies reviewed than on which statistical methods of synthesis are used (Wilson, 2009). Similar to the problem of a biased study sample leading to biased results in an individual study, a biased collection of studies will potentially lead to biased conclusions in a systematic review (Rothstein & Hopewell, 2009). As reported earlier, our search strategies were designed to mitigate the potential effects of publication bias on our analyses. Indeed, it is encouraging that more than one-third of the eligible studies (9 of 24, 37.5%) were acquired through grey literature sources such as published reports, F I G U R E 2 Largest mean effect sizes for study outcomes 14 These findings are similar to the previous iteration of the Campbell focused deterrence systematic review. In the prior meta-analysis, the nonequivalent quasi-experimental designs had an effect size of 0.766 (p < .05), the matched quasi-experimental designs had an effect size of 0.196 (p < .05) and, when research design type was included as a moderator, the overall effect size was 0.312 (p < .05). However, as noted earlier, the current review has a larger share of matched quasi-experimental designs relative to nonequivalent quasiexperimental designs. As such, the overall mean effect size estimated in the current metaanalysis is smaller (0.383) as compared to the original review (0.604). theses, dissertations, unpublished reports, and unpublished working papers. The studies identified through grey literature sources reported a much smaller overall mean effect size (0.156, p < .05) when compared to the overall mean effect size (0.475, p < .05) reported by studies in published journal articles, suggesting that our search strategies were successful in identifying a range of focused deterrence studies with varying effects on crime outcomes. 15 Like many systematic reviews, our meta-analyses used the trimand-fill procedure to explore whether publication bias might be affecting the results and to estimate how the reported effects would change if the bias were to be removed (Duval, 2005;Duval & Tweedie, 2000). The diagnostic funnel plot is based on the idea that, in the absence of bias, the plot of study effect sizes should be symmetric about the mean effect size. If there is asymmetry, the trim-and-fill procedure imputes the missing studies, adds them to the analysis, and then recomputes the mean effect size. Trim-and-fill procedures do suffer from some well-known limitations that could result in the underestimation or overestimation of publication bias (Rothstein, 2008;Simonsohn, Nelson, & Simmons, 2014). 16 Nonetheless, this approach does provide reviewers with a well-understood measure of the possible influence of bias on their meta-analytic results.
A visual inspection of the resulting funnel plot indicated some asymmetry with more studies with a large effect and a large standard error to the right of the mean than the left of the mean. The trimand-fill procedure determined that nine studies should be added to create symmetry. The funnel plot with imputed studies is presented in Figure 6. These additional studies modestly altered the mean effect size estimate. The mean random effect decreased from 0.383 (95% CI [0.264, 0.503]) to 0.215 (95% CI [0.098, 0.332]). Indeed, the 95% confidence intervals substantially overlap, suggesting that the underlying parameters may not be different. Nevertheless, the trimand-fill result suggests mild publication selection bias. However, the adjusted mean effect size remained a similar statistically significant F I G U R E 3 Smallest mean effect sizes for study outcomes 15 For grey literature studies, Q = 23.204, df = 8, p < .05. For journal article studies, Q = 46.913, df = 14, p < .05. The between Q = 10.079, df = 1, p < .05, suggesting that the publication type produced statistically significant differences in observed crime outcomes. The moderated overall effect size was 0.296 (p < .05). 16 As discussed by Rothstein (2008, p. 69), the trim-and-fill procedure is based on the notion that, in the absence of bias, a funnel plot of study effect sizes will be symmetric about the mean effect. If there are more small studies on one side than on the other side of the bottom of the funnel plot, there is concern that some studies may have been censored from the meta-analysis. The trim-and-fill approach imputes the missing studies, adds them to the analysis, and then recomputes the mean effect size. The most notable limitation is that this approach assumes the observed asymmetry is a result of publication bias rather than of true differences in the results of the small studies compared with the larger ones. small size and, as such, the observed publication bias does not appear to be sufficient to nullify the results (as suggested by the funnel plot in Figure 6).

| Summary of main results
The results of our review suggest focused deterrence strategies may generate noteworthy crime control impacts. In 19 of the 24 eligible studies, evaluators reported that the implementation of the evaluated program was associated with a statistically significant crime reduction effect on a targeted crime problem. The results of our meta-analysis of effect sizes estimated a statistically significant, moderate overall mean effect in favor of focused deterrence strategies. When these second-order effects were measured, focused deterrence programs did not result in significant crime displacement impacts. Rather, focused deterrence programs tended to generate diffusion of crime control benefits that extended into proximate areas and socially connected groups that did not receive direct treatments.
The magnitude of the impact of focused deterrence varied by program type. The strongest crime reduction impacts were associated with focused deterrence programs designed to reduce serious violence generated by ongoing conflicts among gangs and criminally active groups. Even when the integrity of the treatment applied was considered, DMI programs generated the smallest crime reduction impacts associated with the three different kinds of focused deterrence strategies.

| Limitations and potential biases in the review process
Outcome measured by studies included in this review relied exclusively on official records and did not include alternative measures such as self-reported victimization. While more than onethird of the eligible studies came from grey literature sources, our trim-and-fill procedures suggests the possibility of modest publication bias. Nevertheless, the possible publication bias suggested by the trim-and-fill analysis was not enough to nullify our overall findings that focused deterrence programs generate modest crime control impacts.

| Agreements and disagreements with other studies or reviews
The results of this systematic review support the conclusion of the National Academies' Committee on Proactive Policing report that focused deterrence can be effective in preventing crime (Weisburd & Majmundar, 2018). As described at the outset of this report, there have been scholarly disagreements over the crime control efficacy of focused deterrence. This is particularly true of varying assessments of the well-known Boston Ceasefire strategy implemented during the 1990s. The results of this review are congruent with the perspectives of scholars who supported the violence reduction value of Boston Ceasefire (e.g., Cook & Ludwig, 2006;Morgan & Winship, 2007) and diverge with the perspective of the skeptics (see, e.g., Fagan, 2002;Rosenfeld et al., 2005).

Implications for practice and policy
The positive findings of our systematic review and meta-analysis, in combination with the strong theoretical literature supporting the mechanisms of focused deterrence, provide solid support for the adoption of such programs by cities suffering from serious crime problems. Indeed, it is unusual that an overwhelming majority of program effect sizes included in our meta-analysis favored treatment over control conditions. It is also noteworthy that the strongest crime reduction impacts were associated with focused deterrence programs designed to reduce serious violence generated by ongoing conflicts among gangs and criminally active groups. Similarly, the National Academies' Committee on Proactive Policing concluded that "evaluations of focused deterrence programs show consistent crime control impacts in reducing gang violence" (Weisburd & Majmundar, 2018, p. 175). These encouraging findings suggest that focused deterrence might be particularly effective at controlling violence emanating from recurring group-based conflicts relative to crime problems driven by disorderly street drug markets or individual repeat offenders.
DMI programs were associated with the smallest crime reduction impacts of the three focused deterrence program types.
Given the large body of research that has shown the ineffectiveness of many police crime prevention efforts (Visher & Weisburd, 1998), the overall crime reduction impact generated by DMI programs is still noteworthy. However, the smaller crime control benefits suggested by our meta-analysis are different from effectiveness claims made from early applications of the approach. For instance, reflecting on several short term, simple pretest versus posttest onegroup-only comparisons of violent crime incidents in treated areas, Kennedy and Wong (2009, p. 43) suggest "that it may be possible to close overt community drug markets and substantially reduce violent and drug-related crime." In their more rigorous quasiexperimental evaluation of the High Point DMI, Corsaro et al.
(2012) suggest a more modest 14% reduction in violent crime incidents associated with the approach.
It is interesting to note that these findings follow those that have been generated in studies of developmental prevention. Summarizing systematic reviews in this area, Farrington, Ttofi, and Lösel (2016) find that programs that focus on higher risk youth are more likely to be successful. In correctional evaluations, the importance of focusing on high-risk offenders has also been a key element for predicting program success (e.g., Andrews & Bonta, 2006). Our finding that the largest impacts are found for programs that focus on the most violent offenders is congruent with what has been observed in treatment programs more generally. This insight could further explain the stronger impacts of focused deterrence on the violent behavior of high-risk groups and repeat offenders relative to the smaller impacts when implemented to control the behaviors of a broader range of offenders participating in street-level drug sales.
The existing empirical evidence suggests that "person focused" policing interventions associated with the standard model of policing, such as programs designed to arrest and prosecute repeat offenders, were not effective in controlling crime (National Research Council, 2005). In contrast, the evaluation evidence reviewed here suggests that focused deterrence strategies, designed to change offender behavior through a blended enforcement, social service and opportunity provision, and community-based action approach, is effective in controlling crime. Other key programmatic elements include strategic analyses of targeted crime problems and a welldeveloped communications plan designed to make targeted offenders understand the new regime that is being imposed on them.
The available literature further suggests that focused deterrence strategies, especially DMI programs, may be difficult to implement and these challenges can undermine its crime control efficacy in certain jurisdictions. It is important to recognize that successful focused deterrence programs follow a deliberate strategy development process rather than the simple adoption of tactics applied in other jurisdictions. Consistent with its problem-oriented policing roots, the focused deterrence framework requires local jurisdictions to conduct careful upfront research on the nature of targeted crime problems to customize a response to identified underlying conditions and dynamics that fits both local community contexts and the operational capacities of criminal justice, social service, and community-based agencies. The successful implementation of focused deterrence strategies requires the establishment of a "network of capacity" consisting of dense and productive relationships among these diverse partnering agencies (see Braga & Winship, 2006). Cities without robust networks in place have found it difficult to implement and sustain focused deterrence strategies.

Implications for research
More than half of the eligible studies included in this updated systematic review were completed after the original Campbell review (Braga & Weisburd, 2012, 2011. Nonetheless, it seems particularly important to assess how elements of procedural justice and collective efficacy influence program outcomes. In recent years there has been growing concern not just about whether policing impacts on crime but also upon how it affects communities. The President's Task Force on 21st Century Policing (2015), for example, identified public trust in the police to be the "first pillar" of policing. While we do not have robust evidence on the mechanisms underlying prevention in most focused deterrence evaluations, the Chicago PSN quasi-experiment provides encouraging evidence for prevention mechanisms that would enhance public evaluations of legitimacy (Papachristos et al., 2007). The Chicago PSN evaluation suggests that direct communications with offenders in a procedurally just manner in the context of maintaining an enforcement environment enhances program effectiveness. This suggests potential for focused deterrence policing to be implemented in ways that are likely to increase legitimacy among offenders. We need more studies that examine this and other potential mechanisms that may improve community outcomes. Cluster randomized experiments represent a variation of the classic randomized controlled trial design in which clusters (groups) of subjects, rather than individual subjects, are randomly allocated to treatment and control conditions. This design allows better control of treatment "contamination" across individual subjects. In the case of gang violence, this contamination is the stable unit treatment valuation assumption (SUTVA) problem generated by social connections among gangs. In a multisite cluster randomized trial, clusters of subjects are randomly allocated to treatment and control conditions in two or more sites. Randomly allocating distinct clusters of gangs connected by rivalries and alliances to treatment and control conditions limits the treatment contamination problem.
Researchers in each participating city would need to identify gang conflict and alliance networks and apply social network analysis techniques to specify distinct socially connected cliques of gangs. Researchers would also need to track shootings by specific gangs during preintervention and postintervention time periods in participating cities.

BRAGA ET AL.
| 27 of 65 developing such capacities in very violent disadvantaged neighborhoods in the United States, it seems possible. Obviously, it would be highly problematic for corrupt, violent, and incompetent police forces to lead the implementation of focused deterrence strategies. Barring that concerning possibility, the flexible problem-solving framework undergirding focused deterrence strategies suggests that the approach can be appropriately tailored to varying urban contexts.
At this point in time, the potential violence reduction efficacy of these approaches in other countries is largely based on speculation rather than empirical facts and practical experience. However, experimentation with focused deterrence strategies to control crime problems beyond U.S. settings is clearly warranted by the available scientific evidence.
While the evaluation evidence needs to be strengthened with additional rigorous randomized experimental field trials, and more developed study of the theoretical mechanisms underlying its impacts, our review suggests that jurisdictions suffering from gang and group-related violence problems should add focused deterrence strategies to their existing portfolio of prevention and control interventions. The effects of focused deterrence on crime problems generated by repeat offenders and street-level drug markets seem to be smaller and, as such, more caution should be applied when implementing these kinds of programs. Jurisdictions looking to implement focused deterrence programs need guidance on the key operational elements of these varied approaches. As evaluation evidence and practical experience continues to accumulate, a premium must be placed on identifying these complementary crime control mechanisms and isolating their impacts on targeted crime problems.

PLANS FOR UPDATING THE REVIEW
In accordance with Campbell Collaboration guidelines, Braga, Weisburd, and Turchan will update this review once every 5 years. A study must meet the following criteria in order to be eligible. Answer each question with a "yes" or a "no." a. The study is an evaluation of a pulling levers focused deterrence intervention._____ b. The study includes a comparison group (or a preintervention comparison period in the case of pre-post studies), which did not receive the treatment condition (focused deterrence  The trajectory of the Boston Gun Project, and the resulting Operation Ceasefire intervention, is by now well-known and extensively documented (Braga et al., 2001;Kennedy, 1997Kennedy, , 2006Kennedy et al., 1996). Briefly, a working group of law enforcement personnel, youth workers, and Harvard researchers diagnosed the youth violence problem in Boston as one of patterned, largely vendetta-like hostility amongst a small population of chronic offenders, and particularly among those involved in 61 loose, informal, mostly neighborhood-based "gangs." These 61 gangs consisted of some 1,300 members, representing less than 1% of the city's youth between the ages of 14 and 24. Although small in number, these gangs were responsible for more than 60% of youth homicide in Boston.
The Operation Ceasefire focused deterrence strategy was designed to prevent violence by reaching out directly to gangs, saying explicitly that violence would no longer be tolerated, and backing up that message by "pulling every lever" legally available when violence occurred (Kennedy 1997). The chronic involvement of gang members in a wide variety of offenses made them, and the gangs they formed, vulnerable to a coordinated criminal justice response. The authorities could disrupt street drug activity, focus police attention on low-level street crimes such as trespassing and public drinking, serve outstanding warrants, cultivate confidential informants for medium-and long-term investigations of gang activities, deliver strict probation and parole enforcement, seize drug proceeds and other assets, ensure stiffer plea bargains and sterner prosecutorial attention, request stronger bail terms (and enforce them), and bring potentially severe federal investigative and prosecutorial attention to gang-related drug and gun activity.
Simultaneously, youth workers, probation and parole officers, and later churches and other community groups offered gang members services and other kinds of help.
These partners also delivered an explicit message that violence was unacceptable to the community and that "street" justifications for violence were mistaken. The Ceasefire Working Group delivered this message in formal meetings with gang members (known as "forums" or "call-ins"), through individual police and probation contacts with gang members, through meetings with inmates at secure juvenile facilities in the city, and through gang outreach workers. The deterrence message was not a deal with gang members to stop violence. Rather, it was a promise to gang members that violent behavior would evoke an immediate and intense response. If gangs committed other crimes but refrained from violence, the normal workings of police, prosecutors, and the rest of the criminal justice system dealt with these matters. But if gang members hurt people, the Working Group concentrated its enforcement actions on their gangs.
The Ceasefire "crackdowns" were not designed to eliminate gangs or stop every aspect of gang activity, but to control and deter serious violence. To do this, the Working Group explained its actions against targeted gangs to other gangs, as in "this gang did violence, we responded with the following actions, and here is how to prevent anything similar from happening to you." The ongoing Working Group process regularly watched the city for outbreaks of gang violence and framed any necessary responses in accord with the Ceasefire strategy. As the strategy unfolded, the Working Group Controlling for the other covariates, the evaluation reported that Ceasefire was associated with a 63% reduction in the monthly count of youth homicides, a 25% reduction in the monthly count of citywide gun assault incidents, a 32% reduction in the monthly count of citywide shots-fired calls for service, and a 44% reduction in the monthly count of youth gun assaults in selected high-risk district (Braga et al., 2001; see also Piehl et al., 2000). In a companion paper, Piehl et al. (2003)  The DOJ-sponsored evaluation then conducted a comparative analysis of youth homicide trends in 39 of the most populous cities in the United States and 29 New England cities with populations of more than 60,000 residents (Braga et al., 2001). Using count regression models that controlled for trends, seasonal variations, and serial autocorrelation, the research found that only three cities The National Academies' Panel on Improving Information and Data on Firearms (National Research Council, 2005) concluded that the Ceasefire evaluation was compelling in associating the intervention with the subsequent decline in youth homicide. However, the Panel also suggested that many complex factors affect youth homicide trends and it was difficult to specify the exact relationship between the Ceasefire intervention and subsequent changes in youth offending behaviors. While the DOJ-sponsored evaluation controlled for existing violence trends and certain rival causal factors such as changes in the youth population, drug markets, and employment in Boston, there could be complex interaction effects among these factors not measured by the evaluation that could account for some meaningful portion of the decrease. The evaluation was not a randomized, controlled experiment. Therefore, the nonrandomized control group research design cannot rule out these internal threats to the conclusion that Ceasefire was the key factor in the youth homicide decline.

Indianapolis Violence Reduction Partnership (IVRP) in Indianapolis, IN
The IVRP working group was comprised of Indiana University researchers and federal, state, and local law enforcement agencies (McGarrell & Chermak, 2003). During the problem analysis phase, the researchers examined 258 homicides from 1997 and the first 8 months of 1998 and found that a majority of homicide victims (63%) and offenders (75%) had criminal and/or juvenile records. Those with a prior record often had a substantial number of arrests. The working group members followed the structured qualitative data gathering exercises used in Boston to gain insight on the nature of homicide incidents. The qualitative exercise revealed that 59% of the incidents involved "groups of known chronic offenders" and 53% involved drug-related motives such as settling business and turf disputes (McGarrell & Chermak, 2003). It is worth noting that the terminology "groups of known chronic offenders" was initially used because, at that point in time, there was not a consensual definition of "gang" and the reality of much gang activity in Indianapolis was of a relatively loose structure (McGarrell & Chermak, 2003).
The working group developed two sets of overlapping strategies.
First, the most violent chronic offenders in Indianapolis were identified and targeted for heightened arrest, prosecution, and incarceration (McGarrell & Chermak, 2003). Second, the working group engaged the pulling levers approach to reduce violent behavior by gangs and groups of known chronic offenders (McGarrell & Chermak, 2003). The IVRP strategy implemented by the Indianapolis working group closely resembled the Boston version of pulling levers.
The communications strategy, however, differed in an important way.
The deterrence and social services message was delivered in meetings with high-risk probationers and parolees organized by neighborhoods. Similarly, home visits by probation and parolees were generally organized by neighborhood. As the project progressed, when a homicide or series of homicides involved certain groups or gangs, the working group attempted to target meetings, enforcement activities, and home visits on the involved groups or gangs (McGarrell & Chermak, 2003).  The impact of the IVRP strategy was estimated using a dummy variable with April 1999 selected as the commencement of the postintervention period (McGarrell et al., 2006). The ARIMA analyses of the Indianapolis homicide time series estimated that the IVRP intervention was associated with a statistically significant 34% reduction in monthly numbers of homicides. The ARIMA models analyzing the other cities' homicide time series did not report any statistically significant associations between the timing of IVRP and subsequent decreases in monthly homicide numbers. In a subsequent analysis of Indianapolis homicide time series data,  used ARIMA models to analyze the impact of IVRP on gang and nongang homicides. The analyses found a statistically significant 38% reduction in gang homicides following the implementation of IVRP and did not find a statistically significant reduction in the nongang homicides during the postintervention time period.
Since IVRP was explicitly designed to reduce gang violence, the authors concluded that these results support the position that the intervention was indeed having the desired effects on violent gang offending.  (Braga, McDevitt, & Pierce, 2006  . The LPD had attempted to develop informants in the past but most these efforts had been unsuccessful.

Operation Peacekeeper in Stockton, CA
Through PSN, the LPD increased its efforts to develop intelligence about the structure of the city's Asian gangs and particularly the relationship between Asian gang violence and ongoing illegal gambling that was being run by local Asian businesses. In Lowell, Cambodian and Laotian gangs were comprised of youth whose street activities were influenced by "elders" of the gang .
Elders were generally long-time gang members in their 30s and 40s that no longer engaged in illegal activities on the street or participated in street-level violence with rival youth. Rather, these older gang members were heavily involved in running illegal gambling dens and informal casinos that were operated out of cafes, video stores, and warehouses located in the poor Asian neighborhoods of Lowell. The elders used young street gang members to protect their business interests and to collect any unpaid gambling debts. Illegal gaming was a very lucrative business that was much more important to the elders than any ongoing beefs the youth in their gang had with other youth . In contrast to acquiring information on individuals responsible for gun crimes in Asian communities, it was much easier to detect the presence of gambling operations through surveillance or a simple visit to the suspected business establishment.
The importance of illegal gaming to influential members of Asian street gangs provided a potentially potent lever to law enforcement in preventing violence. The authorities in Lowell believed that they could systematically prevent street violence among gangs by targeting the gambling interests of older members. When a street gang was violent, the LPD targeted the gambling businesses run by the older members of the gang. The enforcement activities ranged from serving a search warrant on the business that houses the illegal enterprise and making arrests to simply placing a patrol car in front of the suspected gambling location to deter gamblers from entering.
The LPD coupled these tactics with the delivery of a clear message, "when the gang kids associated with you act violently, we will shut down your gambling business. When violence erupts, no one makes money" (Braga et al., 2006, p. 40). Between October 2002 and June 2003, the height of the focused attention on Asian gangs, the LPD conducted some 30 search warrants on illegal gambling dens that resulted in more than 100 gambling-related arrests .

Cincinnati Initiative to Reduce Violence in Cincinnati, OH
In response to a disturbing increase in homicides between 2001 and 2006, Cincinnati's political leadership partnered with law enforcement officials, academics, medical professionals, street advocates, and community and business leaders, to form the Cincinnati Initiative to Reduce Violence (CIRV; Engel et al., 2010). Strategy development was assisted by Proctor and Gamble Co. in the hopes of strengthening institutionalization and sustainability (Engel, Tillyer, & Corsaro, 2013). Problem analyses suggested that violent street groups of active criminal offenders generated the bulk of homicides and shootings in Cincinnati; thus, members of criminally active street groups in Cincinnati were the target population for the pulling levers focused deterrence strategy. As described by Engel et al. (2010), Cincinnati implemented a group VRS that was modeled after the pulling levers focused deterrence strategy implemented in Boston and included law enforcement consequences for violence, along with social service opportunities and community engagement. In face-toface offender notification meetings, police, community activists, political figures, civil rights activists, ex-offenders, parents of murdered children, social service providers, medical personnel, and business, civic, and religious leaders told members of violent groups that the violence must stop, that there would be law enforcement consequences for the entire group if it did not, and that the community would support these consequences (Engel et al., 2010).
The working group partners also told violent group members that there was social service help for all who wanted it. and reduced funding for the overall initiative the following year (Engel et al., 2013).
Preliminary evaluations of the impact of the CIRV interventions yielded promising results (Engel et al., 2008(Engel et al., , 2009(Engel et al., , 2010, but a more complete assessment of the CIRV was completed by Engel et al. pooled time series regression models with controls for trends and seasonal variations were used to estimate the effects of the CIRV over time (Engel et al., 2013). Second, fixed-effects maximum likelihood regression models with a pooled cross-section time series were used to parse out the effects of social service provisions from the larger intervention (Engel et al., 2013).
Additionally, violent firearm offenses were significantly lower by 22% at both 24 months and 42 months postintervention. For comparative purposes, violent non-firearm offenses experienced a statistically nonsignificant 5% reduction both 24 months and 42 months following the intervention (Engel et al., 2013).
To tease out the effects of social services from the overall initiative, Engel and colleagues (2013) investigated whether the number of social service provisions offered to targeted individuals was related to GMI homicides and/or violent firearm offenses.
Regression results indicate that social services were not significantly related to instantaneous changes in GMI homicides or violent firearm offenses (Engel et al., 2013). Furthermore, lagged effects of social service provisions at 1-, 2-, and 6-month intervals were examined but not consistent relationship with outcome measures was observed (Engel et al., 2013).

Operation Ceasefire in Newark, NJ
In a peer-reviewed evaluation, researchers from the Violence Institute of New Jersey at the University of Medicine and Dentistry of New Jersey (UMDNJ) evaluated the Operation Ceasefire gang violence strategy in Newark, New Jersey (Boyle et al., 2010). The Newark Ceasefire strategy focused on preventing gun violence by individual gang members in a targeted "Ceasefire Zone." According to Boyle et al. (2010), the Newark strategy blended the law enforcement actions developed by the Boston Ceasefire pulling levers strategy (Kennedy et al., 1996) with the public health violence prevention activities developed by CeaseFire Chicago (Skogan, Hartnett, Bump, & Dubois, 2008). Shooting teams of detectives from In part, the goal was to determine which parts of the Boston Gun Project might be replicable in Los Angeles. In designing the replication, RAND drew a clear distinction between the process governing the design and implementation of the strategy (datadriven policy development; problem solving, working groups) and the elements and design (pulling levers, collective accountability, retailing the message) of the Boston model. Processes, in theory, can be sustained and adaptive, and as such can be utilized to address dynamic problems. By singling out process as an important component, the RAND team hoped to make clear that process can affect program effectiveness independently of the program elements or the merits of the actual design (see Tita et al., 2003).
The Los Angeles replication was unique in several important ways. First, the implementation was not citywide, but only within a single neighborhood (Boyle Heights) within a single Los Angeles Police Department Division (Hollenbeck). The project site, Boyle Heights, had a population that was relatively homogenous. Well over 80% of the residents were Latinos of Mexican origin. The same was true for the gangs, many of which were formed prior to the Second World War. These gangs were clearly "traditional" gangs, with memberships exceeding a hundred members or more. The gangs were strongly territorial, contain age-graded substructures, and are intergenerational in nature (Tita et al., 2003).
Unlike other cities where gang and group-involved violence was a rather recent phenomenon, Los Angeles represented an attempt to reduce gun violence in a "chronic gang city" with a long history of gang violence, and equally long history of gang reduction strategies. The research team had to first convince members of the local criminal justice and at large community that the approach we were espousing differed in important ways from these previous efforts to combat gangs. And in fact it does-the RAND project was not about "doing something about gangs," but rather "doing something about gun violence" in a community where gang members committed an overwhelming proportion of gun violence.
The independent analysis of homicide files confirmed the perception held by police and community alike that gangs were highly overrepresented in homicidal acts. From 1995 to 1998, 50% of all homicides had a clear gang motivation. Another 25% of the homicides could be coded as "gang related" because they involved a gang member as a victim or offender, but were motivated for reasons other than gang rivalries.
Given the social organization of violence in Boyle Heights, the multidisciplinary working group fully embraced the pulling levers focused deterrence strategy developed in Boston. A high-profile gang shooting that resulted in a double homicide in Boyle Heights triggered the implementation of the Operation Ceasefire intervention in October 2000. The processes of retailing the message were formally adopted, though it was mostly accomplished through personal contact rather than in a group setting. Police, probation, community advocates, street gang workers, a local hospital and local clergy were all passing along the message of collective accountability for gangs continuing to commit gang violence. Unfortunately, Tita et al. (2003) reported that the Los Angeles pulling levers intervention was not fully implemented as planned. The implementation of the Ceasefire program in the Boyle Heights was negatively affected by the well-known Ramparts LAPD police corruption scandal and a lack of ownership of the intervention by the participating agencies.
Despite the implementation difficulties, the RAND Corporation evaluated the Operation Ceasefire pulling levers strategy to reduce gun violence among gangs in the Boyle Heights area of Los Angeles (Tita et al., 2003). In their evaluation, RAND researchers examined the effects of the pulling levers gang VRS on violent crime (homicides, attempted homicides, robberies, assaults, and kidnapings), "gang crime" (violent crime and terrorist threats, firearm discharge, vandalism, and graffiti committed by gang members), and gun crime (any of the above crimes that involved use of a firearm).
The RAND evaluation analyzed changes in their key outcome variables for three time periods across three comparison areas (Tita et al., 2004). The three time periods were the 6 months prior to the in the deterrence period would suggest that the intervention may have had some long-term effects in changing behavior, or that shortterm application of some resources can produce a long-term deterrence effect (although we recognize our measure of deterrence is confounded by the continuation of some suppression activities; Tita et al., 2004, pp. 24-25).
The evaluation used a variety of approaches to detect the effects of the Ceasefire intervention across the study time period (Tita et al., 2004). The RAND researchers used Bayesian analyses of the countbased distributions of the outcome variables across these time periods in Boyle Heights relative to the remainder of Hollenbeck comparison and in the five targeted reporting districts relative to the remainder of Boyle Heights comparison. In their analyses of targeted Census block groups relative to matched comparison Census block groups, the RAND researchers used Probit and step-wise linear regression models to define the matched comparison Census block groups. They then used a slightly more rigorous version of the "difference in differences" approach that assumed the level of crime followed a Poisson distribution and considered serial time trends to evaluate the effects of the Ceasefire intervention across these time periods (Tita et al., 2004). BRAGA ET AL.

| 51 of 65
Their statistical analyses revealed that gang crime in Boyle Heights decreased significantly compared with other regions of Hollenbeck during the suppression period of the intervention, and violent, gang, and gun crime all decreased significantly in the deterrence period. The analyses suggested that the significant reduction in gang crime may have begun in the suppression period.
Violent crime, however, did not decrease significantly in the suppression period. In the five targeted police reporting districts, violent crime decreased significantly in comparison with the rest of Boyle Heights in the suppression and the deterrence periods, and gang crime decreased significantly in the suppression period. Neither gang crime in the deterrence period nor gun crime in the deterrence or suppression periods decreased significantly in comparison with the remainder of Boyle Heights. The RAND evaluation also reported that, in the Census block groups overlapping the targeted reporting districts, violent crime decreased significantly compared with the matched blocks (Tita et al., 2004). Their analyses also suggested that some of this significant reduction may have persisted into the deterrence period.
In addition to their analyses of the main effects of the intervention, RAND researchers examined the effects of the intervention on neighboring areas and gangs. Their analyses suggested a strong diffusion of violence prevention benefits emanating from the targeted areas and targeted gangs (Tita et al., 2004). In the 6 months after the intervention, the researchers reported in the six targeted Census block groups that violent crime had decreased by 34%, gang crime decreased by 28%, and gun crime decreased by 26%. In the 11 Census block groups immediately surrounding the targeted block groups, violent crime had decreased by 33%, gang crime decreased by 44%, and gun crime decreased by 28%. The RAND research team also examined gang crime by gangs not targeted by the Ceasefire intervention that were "socially tied" through conflicts and alliances to the target gangs. After the Ceasefire intervention was implemented, gang crimes committed by the targeted gangs and the nontargeted, socially tied gangs decreased by a matching 26%.

PSN in Chicago, IL
The PSN was implemented in two adjacent police districts in Chicago's West Side where rates of murder and gun violence were more than four times higher than the city average in 2002. As described by Papachristos et al. (2007), the PSN team reasoned that the best way to address Chicago's homicide and gun violence problem was to craft intervention strategies focused on the population with a very high risk of being a victim or offender of gun violence in the targeted neighborhoods. Two principles guided the design and development of PSN interventions: (a) that enforcement efforts be highly specified and targeted to those most at risk of being a victim and offender of gun violence, and (b) that serious efforts should be directed toward changing the normative side of gun violence, that is, the reasons young men use guns and their attitudes toward the law and law enforcers. Following these principles the PSN team devised several law enforcement, community outreach, and offender notification forums and follow-up re-entry programs. The

PSN interventions were implemented in May 2002.
A quasi-experimental design was used to evaluate the impact of the various PSN programs on neighborhood-level homicide rates in Chicago (Papachristos et al., 2007). As described, two adjacent police districts were nonrandomly selected from the city's 25 police districts as PSN treatment districts and, via propensity score matching procedures, two other police districts selected as nearequivalent controls. Monthly and quarterly counts of homicide incidents between January 1999 and December 2004 were identified as the key outcome variables (Papachristos et al., 2007;Meares, Papachristos, & Fagan, 2009); however, the evaluation also analyzed monthly and quarterly counts of gun homicide incidents, gang homicide incidents, and aggravated assault incidents in the treatment districts relative to the control districts.
The research team analyzed the overall effects of the PSN treatment as well as the four interventions that comprised the PSN treatment: (a) increased federal prosecutions for convicted felons carrying or using guns, (b) the length of sentences associated with federal prosecutions, (c) supply-side firearm policing activities (gun recoveries by ATF-CPD gun teams), and (d) social marketing of deterrence and social norms messages through justice-style offender notification meetings. In these offender notification meetings, randomly selected gun-and gang-involved recently released former prison inmates returning to the treatment districts were informed of their vulnerability as felons to federal firearms laws, with stiff mandatory minimum sentences; offered social services; and addressed by community members and ex-offenders. Using individual growth curve regression models, the research team found that the PSN treatment was associated with a statistically significant 37% reduction in the number of homicides in the treatment district relative to the control districts. The overall PSN treatment was also associated with statistically significant decreases in gun homicide incidents and aggravated assault incidents, and a nonstatistically significant decrease in gang homicide incidents.
The PSN intervention that generated the largest, statistically significant effect on decreased homicide in the treatment districts relative to control districts was the offender notification forums. In short, the greater the proportion of offenders who attended the forums, the greater the decline in treatment district levels of homicide. Increased federal prosecutions and the number of guns recovered by the gun teams were associated with modest but statistically significant declines in homicides in the treatment districts relative to the control districts. Getting more guns off the street and prosecuting more offenders federally for gun crimes were associated with small but meaningful homicide decreases. The length of sentences associated with federal prosecutions was not associated with the observed homicide decreases.
In a supplemental unpublished analysis, Fagan et al. (2008) analyzed recidivism rates of individuals who participated in the PSN notification forums. Using survival analyses, the authors found that those who attended a PSN forum were 30% less likely to be rearrested relative to a comparison group of similar recently released individuals from the same neighborhood. The program diminished recidivism levels for both gang and nongang members and seemed to be particularly effective for individuals who had only one prior felony conviction.
In an alternative assessment, Wallace et al. (2016) focused on the offending behaviors of individuals after they attended a PSN offender notification forum. Two types of survival analyses were utilized to investigate participant recidivism: Cox proportional hazard models and competing risk hazard models. Forum participants had a recidivism hazard of approximately 43% less the hazard of nonparticipants from the intervention neighborhood and from nontreated control neighborhoods (Wallace et al., 2016). Additionally, forum participants had 30% lower hazard for committing a new offense compared to control groups. Lastly, results suggested that attending a PSN forum was associated with lower hazards for committing certain types of offenses (weapons and murder) relative to comparison groups (Wallace et al., 2016).

Drug Market Intervention (DMI) in Nashville, TN
In peer-reviewed publication, Corsaro, Brunson, and McGarrell (2010)  The DMI seeks to shut down overt drug markets entirely (Kennedy, 2009). Enforcement powers are used strategically and sparingly, employing arrest and prosecution only against violent offenders and when nonviolent offenders have resisted all efforts to get them to desist and to provide them with help. Through the use of Regression results indicated that drug and narcotics offenses declined by 56% during the postintervention period (26 months), and a 38% reduction occurred in the adjoining area and only a 3% reduction in the remainder of the city (Corsaro et al., 2010). There were no statistically significant changes in UCR Type I offenses between pre and postintervention periods in the target neighborhood (4% reduction), adjoining area (20% reduction), or remainder of the city (3% increase; Corsaro et al., 2010). Calls for service were significantly lower by 13% in the target neighborhood postintervention, whereas nonsignificant reductions of 4% and 2% occurred in the adjoining area and remainder of the city, respectively (Corsaro et al., 2010).
Qualitative results suggested that most interviewees perceived crime as less of a problem in their neighborhood at the time of the interview compared to previous years (Corsaro et al., 2010).
Specifically, drug market activity was noted by most residents as reducing the most, a reduction that newer residents attributed to police efforts and longer-tenured residents attributed to the removal of public housing units from the neighborhood (Corsaro et al., 2010). Hierarchical generalized linear growth curve regression models with a dummy variable to represent the implementation of the DMI strategy were used to analyze trends in the treatment and comparison areas. The analyses reported that the DMI intervention was associated with a statistically significant 22% reduction in nonviolent offenses and a nonstatistically significant reduction in violent offenses in the Delancey Heights target neighborhood. The evaluation did not find any significant reductions in either violent offenses or nonviolent offenses in the remainder of Rockford.  also presented qualitative data from interviews with 34 adult residents from the Delancey Heights neighborhood.

DMI in Rockford, IL
The authors reported that the majority of the residents interviewed noted considerable crime and disorder improvements in their neighborhood after the DMI was implemented.

Operation Ceasefire in Rochester, NY
In response to high levels of homicide, Rochester sought a datadriven response to rampant violence. Many homicides were identified as involving groups that were also involved in selling drugs and other criminal activity (Delaney, 2006). This led to three attempts to implement a "pulling levers" Ceasefire model. After word of early success in Boston began to spread, the U.S. Attorney's Office of Western New York took the lead in Rochester's first attempt to implement a Ceasefire model in 1998. Delaney (2006) noted two primary reasons that led to the demise of this effort: (a) enforcement actions were never taken against groups that continued to engage in violence and (b) the intervention's deterrence message was actively disseminated to juveniles who were not involved in gangs or serious violence which undermined the credibility of the message. The second attempt to implement a Ceasefire model came in [2002][2003] and was resurrected by a team participating in the federal Strategic Approaches to Community Safety Initiatives (SACSI) program. Rather than groups, this second iteration targeted individual high-rate offenders but similar to its predecessor, enforcement actions failed to be taken against continuing offenders.
Despite previous failings, a third attempt was made and Rochester Police Department was finally able to lead a successful implementation of the Ceasefire model with the first call-in during October 2003 marking the official beginning of the postintervention period (Delaney, 2006). Unlike the two preceding attempts, the third iteration of Ceasefire was successfully able to carry out enforcement actions and used actions taken against the "Thurston Zoo" gang as an example to other criminally active groups. The intervention's message was delivered to identified groups at "call-in" meetings, where attendees were informed of the increased attention from criminal justice agencies that will ensue if they continue to be committing homicide (Delaney, 2006).
An interrupted time-series model was used to estimate the impact of the intervention on crime (Delaney, 2006). In addition to investigating the impact of the intervention on overall homicide, gun assaults, and gun robberies, the effects of the intervention were also examined for a specific high-risk demographic for violent victimization: black males ages 15 through 30. Whether the intervention had a delayed effect was also assessed using 1-month increments for a period of 4 months postintervention.
Controlling for trends and seasonal variation, regression results suggested at 1-, 3-and 4-month lags that the intervention was associated with statistically significant 25% and 27% reductions in homicide and gun robbery, respectively, involving black male victims ages 15 to 30; however, no significant reduction in gun assault victimization was found for that high-risk population (Delaney, 2006). No significant relationship effect was found for overall homicide, gun assault, and gun robbery.
Importantly, Delaney (2006)  reducing violence associated with those targeted markets (Kennedy, 2009). Previous assessments of the High Point DMI suggested improved perceptions of crime among residents (Frabutt, Gathings, Jackson, and Buford, 2005) and more challenges for narcotic officers attempts to conduct undercover drug buys (Kennedy & Wong, 2009); however, previous evaluations did not estimate the intervention's effect on violent crime with empirical rigor (Corsaro et al., 2012).
Rolling implementation of the High Point DMI took place from 2004 through 2007 (Kennedy & Wong, 2009). For each site, collaborative investigations and surveillance of key offenders lasted between 1 and 3 months. Four call-in meetings were held with offenders from four different neighborhoods over the course of the intervention. Key offenders with prior felony convictions were arrested while nonviolent offenders with no previous felony convictions were selected to participate in the interventions (Corsaro et al., 2012). In total, 83 dealers were identified across the four intervention locations; 20 were arrested and 63 were selected to participate in a call-in (Corsaro et al., 2012). At the call-in, the message communicated to participants consisted of deterrence, social services and opportunity, and changing norms.
Evaluation of the High Point DMI used annual violent crime data (homicide, rape and sexual offenses, assaults, and robberies) from 1998 through (Corsaro et al., 2012). Corsaro et al. (2012) evaluated the DMI in two stages. First, count-based difference-in-difference panel regression models were used to estimate the impact of the intervention on census blocks in treatment areas relative to comparison units matched via propensity score analyses. Second, group-based trajectory analyses (GBTA) were completed to estimate group-specific treatment effects (Corsaro et al., 2012).
Regression results suggested that violent crime in targeted neighborhoods decreased significantly (p < .001) by 18% in the postintervention period compared to the preintervention period (Corsaro et al., 2012). To assess whether displacement occurred, violent crime in the 59 nearest neighboring census blocks to the intervention sites were compared to the remainder of the city that was not treated. Although violent crime in neighborhoods surrounding intervention sites increased approximately 14%, the change was not significantly distinct relative levels of violence in the remainder of the nontreated city (Corsaro et al., 2012). Additionally, results indicated violent crime in targeted neighborhoods decreased significantly by 13% postintervention relative to their matched comparison sites (Corsaro et al., 2012).
For GBTA, three-groups were determined to be appropriate for categorizing census blocks. Using results from previous propensity score analyses, trajectory groups in the treated area were compared to trajectory groups in matched comparison areas (Corsaro et al., 2012). Chronic High Trajectory census blocks in the target area experienced a statistically significant 17% decrease in violent crime relative to matched comparison census blocks under the same trajectory classification. Violent crime in treated census blocks within the Moderate Stable Trajectory group decreased 11% compared to matched comparison groups within the same trajectory; however, this divergence was not statistically significant (Corsaro et al., 2012).
In contrast, violent crime the treated census blocks in the Negligible Trajectory group increased significantly relative to matched comparison census block and trajectory, an increase that equated to one additional incident per year (Corsaro et al., 2012).
In subsequent analyses, the effectiveness of the High Point DMI was examined by specific intervention sites and across different outcomes (Corsaro, 2013;. Corsaro (2013) used monthly incident data from January 1998 through August 2009 to analyze the effects of the DMI on violent crime, property crime, and drug and disorder crime. Autoregressive Poisson regression models were used to estimate the unique effects of the intervention in each of the four target neighborhoods. One intervention site (West End) experienced significant declines across all three categories of offenses whereas one intervention site (East Central) experienced no significant declines in any of the offense categories. Two intervention sites (Daniel Brooks and Southside) experienced significant (p < .05) 18% reductions in property crime but only marginally significant or nonsignificant changes in remaining offense types. Further analysis suggested that declines in property crime were likely attributable to similar declines citywide (Corsaro, 2013).  evaluated the High Point DMI after it was implemented in a fifth intervention site (Washington). Synthetic control weights were imported into a difference-in-difference negative binomial regression model to estimate the effectiveness of the intervention across four outcomes over a 12-month follow-up period . Overall, the intervention was associated with significant reductions in calls for service (16%) and violent crime (34%) but was not significantly related to drug crime or general crime reports . In terms of impacts on specific intervention sites, all four outcome measures decreased in one treatment location (West End) and three of four outcomes decreased in another treatment location (Washington). The remaining three intervention sites generally experienced declines in each of the outcome measures but not at a statistically significant level . Additionally,  found little evidence of displacement associated with the intervention. The authors concluded that evidence from a synthetic control model suggests the intervention produced long-term reductions of crime and its effects may be larger than suggested by previous evaluations .

DMI in Peoria, IL
Drawing on initiatives in High Point, NC, Nashville, TN, and Rockford, IL, the Peoria Police Department implemented a "pulling levers" strategy to address crime associated with an open-air drug market (Corsaro & Bruson, 2013). The intervention occurred in a single neighborhood that was identified as being associated with disproportionately high crime rates.
In March 2009, police began conducting surveillance and gathering intelligence on 29 suspected drug dealers (Corsaro & Brunson, 2013). This investigation led the arrest of 23 of the 29 dealers in October 2009, who were then subjected to enhanced prosecution. The six remaining suspected drug dealers were selected to participate in a call-in meeting. At the November 2009 call-in, actions taken against the 23 dealers recently arrested were used as examples to those in attendance (Corsaro & Brunson, 2013).
Attendees were delivered a stern message that further drug dealing would not be tolerated and increased enforcement and sanctions will be imposed on anyone who reoffends. In addition to the deterrence message, participants met with social service providers and local community leaders who encouraged reintegration into the community (Corsaro & Brunson, 2013).
A mixed methods approach was used to evaluate the Peoria "pulling levers" drug market intervention. First, an ARIMA approach was used to model the effects of the intervention on the time series (Corsaro & Brunson, 2013). Four outcome measures were examined in this study: violent crime, property crime, drug and disorder crime, and total calls for service. Monthly counts of these variables were analyzed from January 2006 through December 2010 (Corsaro & Brunson, 2013). Second, surveys were used to measure local residents' perceptions of the intervention and whether they perceived any changes in drug offending in the 6 months prior to completing the survey (Corsaro & Brunson, 2013).
Results from the interrupted time-series model indicated no statistically significant (p > .05) relationship between the intervention and any of the four outcome variables. The intervention was positively related to violent and property crime, and was inversely BRAGA ET AL.
| 55 of 65 related to drug/disorder crime and total calls for service (Corsaro & Brunson, 2013).
Although local media covered the November 2009 call-in, survey results indicated that only a minority of residents (31%) were even aware the intervention took place in their neighborhood (Corsaro & Brunson, 2013). In terms of the perceived effectiveness of the intervention, the majority (66%) of those familiar with the intervention thought it had no impact. As for perceptions of crime overall in the past 6 months, 32% thought crime was less of a problem, 36% thought crime was about the same, and 32% thought crime was more of a problem (Corsaro & Brunson, 2013). Specifically, 49% of respondents believed drug sales were about the same over the past 6 months, whereas 28% though drug sales were less of a problem and 23% believed drug sales were more of a problem (Corsaro & Brunson, 2013).
Operation Ceasefire II in Boston, MA  . After the interagency working group was firmly established, the Lucerne Street Doggz gang became the target of the first enforcement action of the second iteration of Boston's Operation Ceasefire .
A nonrandomized quasi-experimental design was used to compare trends in serious violence among gangs targeted by the intervention to comparison gangs matched via propensity score analyses. Because of general deterrent effect sought by "pulling levers" focused deterrence strategies, which violates the "stable unit treatment value assumption," gangs that were socially connected to those targeted by the intervention were excluded from the analysis . Of the 19 treated gangs, 16 were successfully matched with an isolated comparison group. Hierarchical negative binomial growth curve regression models controlling for trends and seasonal variation were used to estimate the effects of the intervention on quarterly counts of total gang-involved shootings, victim gang-involved shootings, and suspect gang-involved shootings . Trends in these offenses were analyzed from January 2006 through December 2010 using police incident data.
Standardized mean difference effect size statistics indicated that the intervention was associated with large effects for total ganginvolved shootings (d = −0.77) and suspect gang-involved shootings (−0.87) relative to matched comparison gangs, but had a modest and nonsignificant effect on victim gang-involved shootings (d = −0.48).
Growth curve regression results suggested gangs targeted by the Ceasefire intervention experienced statistically significant (p < .05) reductions in total gang-involved shootings (31%), suspect ganginvolved shootings (35%), and victim gang-involved shootings (27%; . Notably, seasonality was present for each of the three outcomes: gang-involved shootings were significantly high in the spring (April through June) and summer (July through September) quarters compared to the winter quarter (January through March; .
Because the number of gangs subjected to Ceasefire was staggered over the course of the intervention,  performed supplementary analysis examining the timing of the treatment on offending patterns of gangs specifically targeted. In other words, they investigated whether the intervention produced structural breakpoints on quarterly total gang-involved shootings.
Negative binomial regression results revealed that "13 of the 16 matched treatment gangs experienced their largest statistically significant reduction in total shootings in the same quarter as or the quarter immediately following the full implementation of Ceasefire" (Braga et al., 2014, p. 134).
In an alternative assessment, Braga et al. (2013) focused on whether Operation Ceasefire II produced spillover effects on offending behavior of gangs vicariously associated with those directly targeted by the intervention. Propensity score analyses were used to match treated gangs, vicarious gangs, and untreated gangs. Treated gangs and vicarious gangs experienced significant (p < .05) reductions of 36 and 27%, respectively, in total ganginvolved shootings. Compared to matched untreated gangs, vicarious gangs experienced statistically significant (p < .05) reductions in quarterly counts of total gang-involved shootings (24%) and suspect gang-involved shootings (27%). Victim gang-involved shootings were 19% less for vicarious gangs relative to their matched untreated comparison gangs, but the divergence was only marginally significant (p < . 10;Braga et al., 2013). The authors conclude these results are evidence of a spillover effect and consistent with the general deterrence aim of "pulling levers" focused deterrence strategies (Braga et al., 2013).

Community Initiative to Reduce Violence (CIRV) in Glasgow, Scotland
Faced with high levels of violence and a culture of weapon carrying among its youth, Glasgow implemented the CIRV. Modeled closely after Cincinnati's Initiative to Reduce Violence, Williams et al. (2014) described Glasgow's CIRV as a holistic focused deterrence public health approach aimed at reducing physical violence and weapon possession driven by gangs. While firearms were the focus in Cincinnati, cutting instruments and blunt objects were the weapons of interest in Glasgow (Williams et al., 2014).
The CIRV was a multiagency program headed by the Strathclyde Police. This program was active from October 24, 2008 to April 1, 2011 in two police divisions (BD and BA) that together represented Glasgow (Williams et al., 2014). Representatives of the CIRV invited gang-involved youth to self-referral sessions held at the Glasgow Sherriff Court where they met with a street worker who informed the attendee of the program and asked for the individual to sign a "no violence, no weapon" pledge (Williams et al., 2014). Additionally, a "needs analysis" was completed for the participant who was then connected to appropriate social services. Similar to other focused deterrence strategies, participants were informed the gang would be held collectively responsible and that one violation of the pledge by any member would result in the entire gang being temporarily excluded from the services provided by the CIRV (Williams et al., 2014). year (N = 234) treatment cohorts 21% and 31%, respectively (Williams et al., 2014). The 1-year intervention cohort (N = 204) experienced a 65% decline in weapon carrying postintervention, and the 2-year intervention cohort (N = 286) experienced an 84% decline (Williams et al., 2014). Both of these reductions were significantly larger than declines by their control group counterparts (35% and 40%, respectively). Importantly, Williams et al. (2014) noted that other violence reduction initiatives, such as the Gang Task Force, were present throughout the region that may have influenced results.

Group VRS in Chicago, IL
With a dual emphasis on focused deterrence and legitimacy, Chicago's Group VRS was designed to reduce gun violence citywide driven by gangs and gang factions (Papachristos & Kirk, 2015).
Chicago's VRS predominantly drew on the deterrence doctrine exhibited by Boston's Operation Ceasefire and other "pulling levers" interventions. Although not entirely absent from "pulling levers" strategies in Boston and elsewhere, Chicago's VRS emphasized legitimacy and procedural justice throughout the intervention. Program coordinators deliberately chose places of significance to the community to host offender notification meetings and deliver their message. The enforcement message delivered to attendees at call-ins centered on the increased enforcement and sanctions that would ensue if gun violence continues. Messages of changing norms and social services were also communicated, as community representatives plead with attendees to stop their violent behavior and opportunities were presented to participants for connecting with social services (Papachristos & Kirk, 2015).
Chicago's VRS was first put into action in August 2010 and continued through 2013. During that time, 18 call-ins were held and were attended by 438 individuals representing 149 gang factions (Papachristos & Kirk, 2015).
In recent years, the composition of gangs in Chicago has undergone a transformation. Historically, gangs in Chicago have followed a hierarchical structure (Venkatesh and Levitt 2000), but more recently gangs have been "splintering" into a large number of factions that are only loosely affiliated with a large gang (Papachristos & Kirk, 2015). Another difference in the recent landscape of gangs in Chicago compared to gangs historically is that intra-gang violence is more prevalent than inter-gang violence (Papachristos & Kirk, 2015). Papachristos and Kirk (2015) evaluated Chicago's VRS using a quasi-experimental design with matched comparison groups. Specifically, propensity score analyses were used to compare shooting trends among gang factions targeted by the intervention to shooting trends among matched comparison gangs (Papachristos & Kirk, 2015). Of the 149 gang factions that were represented at call-ins, 148 were matched with at least one comparison group via propensity score analyses (Papachristos & Kirk, 2015). While the study period spans January 2006 through March 2014, outcome variables were evaluated for the 12 months after each gang faction was represented at a call-in. The outcome of interest, the frequency of gang factioninvolved shootings, was measured in three ways: total shooting involvement, shooting victimization, and shooting offending (Papachristos & Kirk, 2015). In addition to the collective 12-month followup evaluation period, Cox proportional hazards models were used to examine the length of time after a call-in that a gang faction was involved in a shooting.
Overall, gang factions experienced a marginally significant (p = .10, one-tailed test) reduction of 23% in total shooting involvement compared to match comparison factions in the 12 months after a call-in. Shooting victimization (fatal and nonfatal) among treated gang factions was significantly (p < .05) lower by 32% lower relative to matched comparison groups. No significant relationship was found between call-in attendance and shooting offending by gang factions; however, Papachristos and Kirk (2015) noted the conclusiveness of this null finding was limited due to the relatively small sample size of known offending parties. Lastly, results from survival analyses indicated that treated gang factions had a significantly lower hazard of shooting involvement relative to their matched comparison (Papachristos & Kirk, 2015). In other words, more time elapsed for gang factions that attended a call-in before being involved with a shooting compared to factions that did not attend the call-in. BRAGA ET AL.

| 57 of 65
Group VRS in New Orleans, LA Funded by the New Orleans Community Support Foundation, the New Orleans Group Violence Reduction Strategy (GVRS) was implemented in response to serious violence driven disproportionately by gangs and groups throughout the city (Corsaro & Engel, 2015). With a history of scandals involving politicians and the police department, and a history plagued by chronic violence, Corsaro and Engel (2015) noted that New Orleans provided a unique context to test the effectiveness of focused deterrence strategies. The GVRS was coupled with the CURE Violence public health model to form one larger homicide reduction initiative called "NOLA for Life" (Skogan et al., 2008).
During the problem analysis of gangs in New Orleans, officers stressed the changing structure of gangs. Although gangs in the past tended to be hierarchical, intergenerational, and structured, recent years has seen gang membership and affiliation become looser and more fluid (Corsaro & Engel, 2015). Officers posited that displacement spurred by Hurricane Katrina may have been responsible for the disruption of traditional gang structure in New Orleans (Corsaro & Engel, 2015). Because of changes in the nature of gang structure, the GVRS targeted both gangs and the less restrictive classification of "groups." From October 2012 through March 2014, five offender notification meetings were held with a total of 158 individuals representing 54 gangs or groups (Corsaro & Engel, 2015). Attendees were informed of the enhanced enforcement and sanctions that would be imposed on the next group involved in a homicide or shooting. In addition to the enforcement message, social services were offered to attendees. The GVRS communication strategy also consisted of home visits, which were conducted with six individuals (Corsaro & Engel, 2015).
Assessment of the GVRS was completed in two stages. The first stage of analysis followed a quasi-experimental design with nonequivalent control groups. Corsaro and Engel (2015) used differencein-difference Poisson regression models to analyze changes in homicide trends in New Orleans following the implementation of the GVRS relative to 14 cities with persistently high homicide rates drawn from a pool identified by McCall, Land, and Parker (2011). In addition to the effects of the actual intervention, a placebo intervention period was also tested for comparing changes in homicide rates in New Orleans and comparable cities. Separately, group-based trajectory analyses were conducted and identified six cities with high-trajectory homicide trends for the 3 years immediately before GVRS was implemented (Corsaro & Engel, 2015).
Difference-in-difference regression models were used to compare annual homicide trends in New Orleans to annual homicide rates in the six high trajectory cities for the first year that GVRS was implemented (2013), when GVRS was partially implemented (2012), and placebo year when no treatment was in place (2011).
The second stage of analysis used a quasi-experimental timeseries design. Conditional negative binomial Poisson regression models controlling for trends and seasonal variations were used to estimate the longitudinal impact of the intervention. Complementary analysis examined the effects of the GVRS by race and age (Corsaro & Engel, 2015).
Comparing annual homicide rates in New Orleans to homicide rates in 14 cities with high homicide rates revealed a statistically significant decline in homicides that was unique to New Orleans for both four in which the GVRS was fully implemented (2013) and partially implemented (2012; Corsaro & Engel, 2015). Similarly, New Orleans experienced a statistically significant reduction in homicide rates when compared to the six immediate high trajectory cities in both 2013 and 2012 (Corsaro & Engel, 2015). For both comparisons, differences in homicide trends in New Orleans was distinct relative to nonequivalent comparison cities.
The second stage of the evaluation used monthly crime incident data from January 2010 through March 2014 to examine the effects of the GVRS on overall homicides, firearm-related homicides, firearm assaults, and gang-member-involved (GMI) homicides, as well as overall property and overall violent crime for comparative purposes (Corsaro & Engel, 2015). Evidence suggested that the GVRS was related to a significant (p < .05) 17% decrease in monthly total homicide rates postintervention. Additionally, results indicated that the GVRS was associated with a significant (p < .01) 32% reduction in GMI homicide, while rates of non-GMI homicides did not change significantly postintervention. Firearm-related homicides and nonfatal firearm assaults both declined 16% significantly (p < .05) following implementation of the GVRS (Corsaro & Engel, 2015). Corsaro and Engel (2015) completed supplemental analysis to investigate whether the intervention produced differential effects on the most "at-risk" populations. Pre and postintervention homicide victimization trends were examined for four groups: black male victims ages 20-29, black male victims ages 30 and over, all other victims ages [20][21][22][23][24][25][26][27][28][29], and all other victims ages 30 and over. This analysis found that that GVRS was significantly related to only one of the four groups: homicide victimization of black males ages 20-29 was 27% lower in the postintervention period (Corsaro & Engel, 2015).
Lastly, to isolate the effects of the GVRS focused deterrence strategy from the CURE Violence public health program that was taking place simultaneously, Corsaro and Engel (2015) analyzed changes in total homicides and GMI homicides citywide while excluding the area where CURE Violence was active (Central City).
Although Central City experienced declines postintervention for both total homicides and GMI homicides, those declines were nonsignificant whereas the declines in the remainder of the city were statistically significant (p < .05).

No Violence Alliance in Kansas City, MO
After a change in political leadership in 2012, Kansas City's BJAfunded Smart Policing Initiative (SPI) shifted its strategy of reducing violence from a foot patrol experiment to a focused deterrence "pulling levers" intervention that targeted group-related violence (Novak, Fox, Carr, McHale, and White 2015). Specifically, the Kansas City No Violence Alliance (KC NoVA) sought to reduce violence citywide by focusing on chronically violent, group-involved offenders (Fox et al., 2015). This initiative was led by an interagency Governing Board that included the county prosecutor, mayor, chief of police, probation, parole, FBI, ATF, U.S. District Attorney, and the Chancellor of the University of Missouri-Kansas City (Fox et al., 2015).
There were notable challenges encountered during the implementation of the KC NoVA, including a lack of directive to participating personnel, no clear decision-making structure, and ineffective communication and coordination (Fox et al., 2015). These inefficiencies emerged throughout the first year of the intervention (2013), but were eventually corrected by the end of the year; consequently, full implementation was not achieved until the beginning of 2014 (Fox et al., 2015).
The intervention's message was disseminated to the targeted audience through home visits, police stations, probation and parole offices, and call-in meetings. Targeted subjects were offered opportunities to utilize social services but were also informed of the increased enforcement they would encounter if they continue with their violent ways (Fox et al., 2015). In 2014, four call-ins were conducted and 149 group-involved individuals attended (Novak et al., 2015). Additionally, enforcement actions were taken against six groups (Novak et al., 2015).
Using police incident data, Fox et al. (2015) analyzed monthly counts of homicide and aggravated assault with a firearm from January 2010 through December 2014 to evaluate the effects of the KC NoVA. January 2014 was treated as the beginning of the postintervention period, and the intervention became increasingly implemented throughout the year. Interrupted time series models were used to compare violent crime rates before and after the intervention. The evaluation assessed crime outcomes at 1-, 3-, 6-, and 12-months after the intervention.
Results indicated that the KC NoVA was associated with significant and immediate reductions for both rates of homicide and aggravated assault with a firearm, but those effects diminished over time. The KC NoVA was associated with a statistically significant (p < .05) 40% reduction in homicide 1-month postintervention, a significant 34% reduction after 3 months, and a significant 29% reduction after 6 months; however, the effects of the intervention were no longer significant 12-months postintervention (Fox et al., 2015). Similarly, the KC NoVA was associated with significant reductions in gun-involved aggravated assaults of 19% and 14% at one and 3-month follow-up periods, respectively. However, no significant relationship was observed at 6 and 12 months after the intervention, with the direction of the relationship becoming positive at 12 months (Fox et al., 2015). In short, the longer the intervention was in place, the less influential it became; Fox et al. (2015) posited this finding may be suggestive of a decay effect.

Project Longevity in New Haven, CT
After experiencing rising trends in homicide and gun violence, New Haven became the pilot site for a statewide effort to curb gun violence (Sierra-Arevalo et al., 2015). New Haven's focused deterrence strategy, Project Longevity, sought to reduce fatal and nonfatal shootings citywide by targeting gangs or groups involved in gun violence. Given that much of the violence stemmed from reciprocal conflict across groups, local law enforcement conducted a group audit and an incident review with an emphasis on ongoing intergroup violence (Sierra-Arevalo et al., 2015).
Completion of the group audit led to the identification of 52 street unique groups that consisted of 440 identified street group members. However, only 22 of the 52 groups identified were involved in gun violence and, thus, targeted by the intervention.
Intervention staff prioritized the two most violent street groups and invited both groups to the first call-in, which was held in November 2012. At the call-in, law enforcement, social service providers, and community members joined together to deliver to attendees a deterrence message consisting of consequences that will ensue if their offending continues, a moral plea to cease their offending, and an extension of services available to those seeking assistance. In Third, the classification was for group-involved shootings was expanded to include possible group-involved shootings.
Regression results suggested that Project Longevity was associated with a statistically significant 37% reduction in total shootings and homicides citywide. Additionally, the intervention related to a statistically significant 73% decrease in group-member-involved homicides and shootings, which equates to approximately five fewer of such incidents per month. Supplemental analyses indicated that controlling for other parameters produces a slight reduction in the impact of the intervention but a statistically significant reduction in group-involved still persists (Sierra-Arevalo et al., 2015).

DMI in Guntersville, AL
After traditional law enforcement tactics proved ineffective, Guntersville PD implemented a DMI in response to persistent drug and crime problems that predominately centered on an overt drug market operating at a specific community park. An interagency working group consisting of criminal justice entities, social services, BRAGA ET AL. | 59 of 65 and community representatives was formed to guide the intervention and overcome a history of strained relationships between local residents and law enforcement that included two highly publicized lawsuits pertaining to excessive use of force (Saunders et al., 2016).
The intervention was implanted in the neighborhood of Lakeview, with an emphasis on the community's park, where crack cocaine, marijuana, prescription pills, and methamphetamines were being sold. A call-in was held in December 2011 and all six offenders selected to participate in the program attended the meeting. At the call-in, attendees were informed of the details of the intervention, heard from concerned members of the community, and encouraged to utilize social services that were made available.
A quasi-experimental design that incorporated synthetic control methods was used to evaluate the intervention. Negative binomial regression models controlling for trends were used to estimate the impact of the intervention on total crime, violent crime, property crime, and drug crime. Despite implementing the intervention as intended, regression results suggested there was no statistically significant relationship between the DMI and any of the four crime outcomes examined .

DMI in Montgomery County, MD
A city council member brought the DMI strategy to the attention of the police chief, who embraced the idea and took the lead implementing the strategy . The DMI in Montgomery County was very narrow in its geographic scope by targeting a one-block apartment complex called Damascus Gardens where all units were designated as Section 8 housing. Primarily, drugs being sold in the apartment complex were crack cocaine, heroin, prescription pills, and marijuana . Local law enforcement, an assistant state's attorney, and county's Department of Health and Human Services were all central actors in implementing the intervention (Saunders et al., 2016). Further, law enforcement placed a confidential informant in the apartment complex who played a critical role in identifying eight A-listers and nine B-listers to include in the intervention.
All eight B-listers who were selected to participate attended the call-in. Attendees were told to stay "out of trouble with the law" for the 12 months following the call-in or else the charged with the offenses police had already gathered (Saunders et al., 2016, p. 24).
While social services were offered to attendees, only minimal attention was given to providing social services to clients (Saunders et al., 2016). Notably, team members noticed a considerable shift in the community's attitudes toward drug dealing following the call-in, from complacent to optimistic that change can occur. Following the intervention, four law enforcement officers were dedicated to Damascus Gardens and the surrounding. They were able to police proactively in the targeted areas as they were relieved from responding to calls for service.
In terms of the intervention team's assessment, they believed the DMI successful shut down the overt drug market.  assessed the intervention quantitatively using a quasi-experimental design that incorporated synthetic control methods to generate a matched comparison area. Negative binomial regression models controlling for trends were used to estimate the effects of the DMI at Damascus Gardens on total crime, violent crime, property crime, and drug crime. Regression results indicated that there were no statistically significant reductions in any of the outcome measures associated with the DMI .

DMI in Roanoke, VA
Successes associated with the DMI in High Point, North Carolina inspired Roanoke's police chief to lead the formation of an interagency task for to implement a DMI in Roanoke (Saunders et al., 2016). In addition to reducing crime, the task force saw the DMI strategy as an opportunity to improve the community's perception of police.
The Roanoke DMI completed two call-ins: the first in December 2011 in the Hurt Park neighborhood and the second in January 2013 in the Melrose-Rugby neighborhood. The original target of Roanoke's DMI was the Hurt Park neighborhood, where the drugs most commonly sold were crack cocaine and marijuana . Although Hurt Park had consistently high crime rates, it also had a strong community organizing presence who supported the DMI.
Intelligence gathering led to 10 offenders being designated as Alisters and five as B-listers (Saunders et al., 2016). At the call-in meeting, attendees were told to refrain from criminal activity for the ensuing 6 months or else they would face charges for their previously observed crimes. Further, law enforcement, social services, and community leaders were all given a chance to speak at the call-in meetings and express their desire for attendees to change their criminal behavior (Saunders et al., 2016). For the second call-in, 15 Alisters were identified and five B-listers participated in the intervention.
A quasi-experimental design with a matched comparison generated through synthetic control methods was used to evaluate the impact of this intervention. Negative binomial regression models controlling for trends were used to estimate the impact of the DMI on crime. For Hurt Park, regression results suggested that the DMI was associated with a statistically significant 30% reduction in total crime 3-month postintervention, as well as a 19% reduction at 6 months, 28% reduction at 9 months, and 23% reduction at 12 months. Further, Hurt Park experienced a significant 45% decline in property rime at 6 months, 57% decline at 9 months, and 50% decline at 12-month postintervention. There was also a significant decrease in violent crime of 24% at 3 months and 29% at 9 months following the intervention. No significant relationship was found between the DMI and drug crime in Hurt Park. For the Melrose-Rugby neighborhood, violent crime was the only outcome significantly related to the DMI with reductions of 15% at 3-month and 34% at 6month postintervention. There was no significant relationship discovered between the DMI in Melrose-Rugby and total crime, property crime, or drug crime.

DMI in Seattle, WA
There were two areas targeted by the DMI in Seattle: 23rd Street Corridor in the fall of 2009 and the International District in 2013 . The first intervention site was selected because drug dealing concentrated predominately in two areas along the 23rd Street Corridor, which also had a history of gang violence.
Undercover drug buys and video surveillance led to the identification of 20 DMI candidates. One challenge that emerged for the first call-in pertained to the inability of members of Seattle PD on the intervention team to discuss the project with patrol and other personnel, which Saunders and colleagues speculated may have inhibit law enforcement's commitment to the program. This DMI also had to overcome negative community relations that were left behind by a Department of Justice "Weed and Seed" program that previously operated in the 23rd Street Corridor. The community did not embrace the previous program and it left some members of the community with disenfranchised and antipolice sentiments.
Seattle implemented a DMI in the International District in 2013.
A total of 12 individuals were identified as DMI candidates through investigative strategies. There were a number of challenges faced during the implementation stage: confusing between the DMI and a similar program called Law Enforcement Assisted Diversion, language barriers between the task force and the predominately Asian speaking community, and most targeted offenders lived outside of the target area resulting in less buy-in from the candidates and the community. Notably, a third DMI was held in early 2011 and focused on an area in Southeast Seattle; however, during the implementation of this iteration there was a change in command at Seattle PD couple with a change in the department's priorities.
Community mobilization efforts before the call-ins helped garner community support for the initiatives. In addition to law enforcement's message of deterrence, community members offered support and disapproval of continued drug dealing and social services were offered to attendees. After the call-in, uniformed patrols increased and calls for service were prioritized in the targeted areas. Six months after the call-in in the 23rd Street Corridor, drug dealing began increasing and SPD took enforcement actions against 17 drug dealers in the target area. The intervention team continued efforts to maintain positive relationships with the target communities after the call-ins by hosting community cleanups and barbecues.
To estimate the impact of the intervention, Saunders et al. (2015) used a quasi-experimental design with a matched comparison area created through synthetic control techniques. A series of negative binomial regression models were used to investigate the effects of the DMI across four outcome measures: total crime, violent crime, property crime, and drug crime.

Regression results indicated that the DMI in the International
District was related to a statistically significant 15% reduction in total crime at 3-and 6-month postintervention. Three months after the intervention, property crime was significantly 8% lower, violent crime was significantly 53% lower, and drug crime was significantly 29% lower . In contrast to the International District, which saw significant reductions across each of the four outcome measures, the intervention in the 23rd Street Corridor was not significantly associated with reductions in any of four crime types.

DMI in Ocala, FL
Initiated by the Chief of Police of the Ocala Police Department (OPD), the intervention team consisted of the OPD, the city's drug task force, an assistant state's attorney, a community activist, and social service providers. Initially, the state's attorney office resisted involvement in the DMI because of a perception that it was "soft on crime"; however, efforts to convince the state's attorney's office, led by OPD's Chief, of the DMI's merit were successful and resulted in their full commitment to the intervention. Two sites were selected as target areas for the DMI: the "Second Chance" neighborhood in November 2009 and the First Avenue housing project in the October 2010 . Problem analysis and investigative efforts led to the identification of 13 candidates and 15 candidates for the DMIs in "Second Chance" and First Avenue target areas, respectively. Those who were not chronic or violent offenders were invited to the call-in, where they were presented the evidence already collected against them, offered an opportunity to avoid being arrested by ceasing to dealing drugs, and were offered social services.
Prior to the call-ins, the DMI team attempted to garner support and build positive relationships with the community by hosting barbecues, special events, and town hall meetings. At the town hall meetings, the sergeant helped put to rest community concerns that the DMI was targeting young black men. Following the call-ins, police prioritized calls for service pertaining to drug complaints and increased their presence in each of the target neighborhoods. The DMI team continued to host events for the community and conduct cleanups, and social services follow-up with call-in attendees.
A quasi-experimental design with matched comparisons via synthetic control methods was used to evaluate the impact of the DMI. Negative binomial regression models were used to assess whether the intervention was associated with reductions in total crime, violent crime, property crime, and drug crime. Across all regression models, no significant relationship was found between the DMI and any of the four outcome measures. Despite the null impact on the outcome measures, the DMI team felt the intervention strongly contributed to the development of positive police-community relations.

APPENDIX D: STUDIES NOT FITTING SELECTION CRITERIA
During our systematic search of the literature, we identified a number of focused deterrence interventions that did not fit our inclusion criteria. Focused deterrence interventions that did not meet our eligibility standards due, broadly, for one of two reasons: (a) BRAGA ET AL. | 61 of 65 full program implementation was not achieved; and (b) key components of the traditional "pulling levers" focused deterrence approach were absent in the intervention. In this section, we detail studies of focused deterrence strategies that fell short of being eligible for our meta-analysis.

Selected Studies from the RAND Drug Market Intervention Cohort
The Bureau of Justice Statistics funded a cohort of seven sites to test whether the effects of the drug market intervention model used in High Point, North Carolina in different settings . Sites included in this DMI cohort were: Guntersville, AL; Jacksonville, FL; Gary, IN; Montgomery County, MD; New Orleans, LA; and Roanoke, VA. An assessment of program implementation fidelity completed by  identified two of these sites (Gary, IN and Jacksonville, FL) that failed to complete the first phase of the implementation. To have a more comprehensive outcome assessment, the Gary and Jacksonville DMIs were replaced in the cohort evaluation with two sites from a previous DMI cohort (Seattle, WA and Ocala, FL).
We elected to include only sites from this DMI cohort evaluation that completed each stage of the implementation process with at least medium or high fidelity, as rated by . Sites that achieved this standard, and therefore were included in our metaanalysis, were: Guntersville, Alabama; Montgomery County, Maryland; Roanoke, Virginia; Seattle, Washington; and Ocala, Florida. The remaining sites in the DMI cohort that were excluded from our metaanalysis for incomplete or poor program implementation included Flint, MI; Gary, IN; and Jacksonville, FL.
Notably, of the sites from the RAND DMI cohort that were excluded from our meta-analysis, Flint was the only one that conducted a call-in with identified offenders .
Unlike most focused deterrence strategies that originate from police departments or public officials, Flint's DMI spawned from the efforts of a local community organization (the Flint Area Congregation Together) who brought the idea to the mayor and city council (Saunders et al., 2016). The Flint DMI was a multiagency initiative that consisted of representatives from 21 organizations ranging from law enforcement and government departments to social services and community organizations. Originally called Flint Ceasefire, the name "Flint Lifelines" was later adopted.
The initial announcement of the DMI was met with skepticism by the media and community due to failures of previous crime reduction strategies. Feeling pressure to implement the DMI quickly, the initiative was launched and a call-in was scheduled prior to the task force receiving any formal training and before the exact composition and details of the intervention were determined. However, after several delays and attending the DMI training session held by Michigan State University, the intervention team realized they needed to take additional measures before hosting a call-in.
Ultimately, the first call-in was held in July 2011 nearly 18 months after the original call-in was scheduled. This considerable delay between the announcement of the program and actually enactment further fueled the media's and community's skepticism of the intervention.
The DMI was implement in the city's Second Ward, where vacant housing units were used to sell cocaine and heroin . In addition to having one of the most active drug markets in Flint, the target area was also selected for having a disproportionately high concentration of Part I index crimes (Saunders et al., 2016). Despite challenges in the intelligence gathering process, the intervention team was able to identify 20 offenders to use as examples of increased punishment and seven offenders to participate in a call-in, six of whom attended; later, in 2013, two additional callins were held (Saunders et al., 2016). Saunders and colleagues (2015) completed an outcome evaluation that utilized synthetic control models in a quasi-experimental design to compare crime trends in the targeted neighborhood to trends in the matched comparison area. Negative binomial regression models controlling for crime trends were used to estimate the impact of the intervention on the time-series. Results indicated that the intervention was not significantly related to any of the outcome measures assessed (total crime, violent crime, property crime, and drug crime).
There were three major shortcomings in the implementation of the Flint DMI that led to its exclusion from the present meta-analysis.
First, the problem analysis stage, specifically intelligence gathering, was hampered due the limited number of officers devoted to the intervention, lack of feedback from the community, and unreliable undercover informants. Second, due to strained and limited resources, no enforcement actions were carried out after the callin and crime problems in the target area could not be prioritized (Saunders et al., 2016). Third, quick bail and a refusal by judges to give arrested offenders harsher sentences undermined the deterrent message of the intervention.

Intervention Components
Project Hawaii Opportunity with Probation Enforcement (HOPE) in Honolulu, Hawaii HOPE intervention was a community supervision program aimed at substance-abusing probationers (Hawken & Kleiman, 2009). The program relied on a mandate to abstain from illicit drugs, backed by swift and certain sanctions for drug test failures, and preceded by a clear and direct warning. Probationers were sentenced to drug treatment only if they continued to test positive for drug use, or if they requested a treatment referral. The deterrence-based HOPE intervention differs significantly from typical drug court operations as it economizes on treatment and court resources. As Hawken and Kleiman (2009) suggest, HOPE does not mandate formal treatment for every probationer, and does not require regularly scheduled meetings with a judge; probationers appear before a judge only when they have violated a rule. HOPE is often linked to the DMI approaches as a related application of focused deterrence (see, e.g., Boyum, Caulkins, & Kleiman, 2011) as well as gang and group-based pulling levers focused deterrence based on the common strategy of certain punishment for offenders (Durlauf & Nagin, 2011).
The HOPE evaluation used a randomized controlled trial among general-population substance-abusing probationers where probationers assigned to treatment conditions were compared to probationers assigned to probation-as-usual control conditions (Hawken & Kleiman, 2009). In their unpublished report to the U.S. National Institute of Justice, Hawken and Kleiman (2009) state that that HOPE relies on a mandate to abstain from illicit drugs, backed by swift and certain sanctions and preceded by a clear and direct warning. Unlike most diversion programs and drug courts, it does not attempt to impose drug treatment on every participant. Under HOPE, probationers are sentenced to drug treatment only if they continue to test positive for drug use, or if they request a treatment referral. According to Hawken and Kleiman (2009), HOPE should be considered to be distinct from drug courts in economizing on treatment and court resources (probationers appear before a judge only when a violation is detected). HOPE's stated goals are reductions in drug use, new crimes, and incarceration.
The randomized controlled trial used an intent-to-treat design where all offenders randomly allocated to the treatment condition were included in the HOPE group whether they formally entered the program or not. Of the eligible probationers, two thirds were assigned to the HOPE treatment (n = 330) and one-third were assigned to the control group (n = 163). Ninety-three percent of the probationers assigned for treatment appeared for their initial HOPE warning hearing and participated in the intervention. The experiment commenced in October 2007 and the intervention period lasted for 1 year.
Based on their analyses of the experimental data, Hawken and Kleiman (2009) concluded that HOPE was very effective in changing the behaviors of substance-abusing probationers. Only 21% of HOPE probationers experienced new arrests as compared to 47% of control probationers (p < 01). HOPE probationers outperformed control probationers on a number of other performance measures such as missed probation appointments (treatment = 9%, control = 23%), positive urine drug test results (treatment = 13%, control = 46%), revocation rates (treatment = 7%, control = 15%), and the number of days sentenced to incarceration (treatment = 138 days, control = 267 days).
Based on our selection criteria, HOPE was not included in our final review. However, as stated earlier, several scholars believed that HOPE does fit within the general framework of pulling levers focused deterrence strategies. We agree that it is broadly similar to another evaluation included in our systematic review that is focused on a corrections population-Chicago's PSN intervention (Papachristos et al., 2007). The key elements of Chicago PSN strategy are administered by the Illinois Department of Correction and the U.S.
Attorney's Office (the call-in session is given to returning parolees to selected neighborhoods). The contribution of the Chicago Police Department is limited to increasing their gun policing efforts in the selected neighborhoods. The CPD does not select the returning parolees for the intervention nor do they run the communications strategy. Their only role is to increase gun recoveries and arrest those who commit violent crimes in these neighborhoods.
Moreover, probation has a central role in all of the gang/groupbased focused deterrence interventions included in our review.
Monitoring offenders in the community to ensure they are abiding by probation conditions, changing conditions, and revoking probation are key levers that are pulled in the application of focused deterrence strategies to gangs and criminally active groups. In interagency working group settings, all involved agencies govern the shape and content of the pulling levers interventions. While the police convence the working group meeting, they share governance with the other criminal justice agencies, social service providers, and community members in the group. Probation is involved as a key decision maker in the process.
Most applications of pulling levers focused deterrence strategies have therapeutic elements (e.g., Braga et al., 2001;Papachristos et al., 2007). Indeed, the working group has social service providers, street outreach workers, and community members as core members. suggested a positive relationship between being exposed to the notification of firearms statutes and subsequent involvement in illegal gun activity. Standard logistic regression results indicated this effect was statistically significant; however, although the direction of the relationship was maintained, rare-events logistic regression model did not produce statistically significant results indicating the relationship is sensitive to model specification. Barnes et al. (2010) also performed a survival analysis to assess the effectiveness of the program but were limited by relatively few cases of failure. Results from the Cox regression analysis suggested that the treatment group were quicker to commit a gun-related offense at a marginally significant rate (p < .10) than the comparison group.
The authors urged caution when interpreting the findings given the sensitivity of results due to model specification. They hypothesized the increase in gun-related offenses for the treatment group may have been the result of heightened awareness of firearms statutes among law enforcement entities potentially leading to a systematic change in reporting practices. Also, the authors note the collaboration between SCDPPPS and ATF may have increased reporting of firearms violations because of the federal assistance in prosecuting gun crimes.
We excluded this study from out meta-analysis for a variety of reasons. This evaluation focused exclusively on the impact of the notifications component of the larger CeaseFire intervention. How the notifications were carried out was not in accordance with procedures used in traditional "pulling levers" strategies. First, notably absent from this portion of the intervention were representatives from the community and social service providers. Relatedly, the authors note the vast range of agencies involved in the comprehensive CeaseFire initiative; however, this evaluation exclusively examines the impact of the notifications component that was conducted by SCDPPPS and appears to lack the interagency collaboration commonly found in the traditional focused deterrence model. Another reason this study was not deemed eligible was due to how notifications were carried out.
These notifications were not focused at any specific individuals, groups, or areas. Instead, the notification was presented to a broad population (e.g., all new probationers and parolees). Lastly, this notification program appears to lack enforcement action that is critical to focused deterrence strategies. Although law enforcement agencies were part of the larger intervention, their participation in the program evaluated in this study was noticeably absent.

PSN in the Eastern District of Missouri
In 2002, PSN was implemented in the Eastern District of Missouri.
The primary focus of this iteration of PSN was the high rates of homicide and gun violence in the city of St. Louis (Decker et al., 2007). Coordinated by the U.S. Attorney's Office, a task force was formed that included local, state, and federal law enforcement agencies; local and federal prosecutors; probation and parole offices; juvenile court; a hospital; city neighborhood services; street outreach workers; media partners; the regional justice information system; and researchers from the University of Missouri, St. Louis. This task force utilized a variety of strategies to address high levels of gun crime, including targeted enforcement, intensive gun case prosecution review, a most violent offender program, and notification meetings for high-risk probationers (Decker et al., 2007). Decker et al. (2007) utilized a quasi-experimental design to estimate the impact of the PSN intervention on neighborhood crime rates. Crime rates in 14 treatment neighborhoods were compared to two different comparison groups. The first comparison group consisted of control neighborhoods that were selected based on similarity of sociodemographic characteristics and crime rates. In order to detect displacement and diffusion, a second comparison group consisting of neighborhoods contiguous to target neighborhoods was used.
Results from the outcome assessment were supportive of strong program fidelity. Specifically, the number of federal prosecutions and offenders convicted for gun crimes at the state and federal levels both increased significantly. Less clear, however, is the intervention's impact on crime rates. Violent gun crime declined significantly in targeted neighborhoods, but it also declined significantly citywide.
Furthermore, the decrease in gun crime arrests in the treatment neighborhoods was not significantly different from arrest rates observed in control and contiguous neighborhoods (Decker et al., 2007).
Although PSN initiatives typically adhere to principles of "pulling levers" focused deterrence, we conclude that this particular intervention employs a collection of tactics that build on certain elements of focused deterrence but fails to implement a comprehensive focused deterrence strategy (see Kennedy, 2006).
For example, the gun case prosecution review component of this intervention was largely a reform of court and prosecution practices rather than an application of deterrence. Additionally, the targeted enforcement strategies and most violent offenders program as described in the evaluation appear strictly reactive rather than proactive, and omit the communication component of the traditional "pulling levers" approach. Lastly, the probation notification portion of this PSN intervention appears to be a standalone program without any firm criteria offered about why probationers were selected. It is also unclear whether this notification process was accompanied by any enhanced attention and enforcement.