Developmental evaluation and the ‘Stronger Economies Together’ initiative in the United States☆

https://doi.org/10.1016/j.evalprogplan.2013.11.004Get rights and content

Highlights

  • •

    We conducted a developmental evaluation of a national rural development initiative in the U.S.

  • •

    Ongoing evaluator/client communication facilitated transfer of research findings.

  • •

    Developmental evaluators must be flexible, nimble, creative, and adaptive.

  • •

    The client implemented actionable recommendations by the evaluators immediately.

Abstract

This article describes a developmental evaluation and explains its impact on the Stronger Economies Together (SET) initiative of the U.S. Department of Agriculture in collaboration with the Nation's four Regional Rural Development Centers and Land-Grant universities. Through a dynamic process, this evaluation of the early phases of an initiative led to continuous program alterations based on feedback. The relationship of the evaluation team to the initiative's coordinating team enabled seamless transfer of observations, suggestions, and recommendations to decision makers. The multidisciplinary character of the evaluation team provided a diverse set of perspectives with a depth of subject matter and knowledge from relevant fields. One lesson is that developmental evaluators must be flexible, nimble, creative, and adaptive. When expected data are imperfect or nonexistent, the team must collect alternate information and make recommendations to improve data collection. As the initiative proceeded and modifications came about, the evaluation team had to recognize the changes in the program and focus on different questions. This experience with developmental evaluation provides insights into how interdisciplinary teams may need to change course and conduct a developmental evaluation when a formative evaluation was originally envisioned.

Introduction

This article presents an evaluation effort designed to help improve ‘Stronger Economies Together’ (SET), a national initiative to enhance the capacity of rural regions in the United States to prepare economic development plans. The purpose of this article is to describe a formative-turned-developmental evaluation research project and explain its impact on SET.

In recent decades there has been a series of attempts to enact a comprehensive, well-funded rural development policy in the U.S. For a variety of reasons, these efforts have failed; predictably, some argue (Honadle, 1993, Honadle, 2001, Honadle, 2008, Honadle, 2011). In 2002 and 2008, comprehensive agricultural legislation established titles that encouraged the formation of locally initiated regions with the promise of grant funding and technical assistance. The 2008 Rural Collaborative Investment Program, the most recently defeated bill, would have provided resources for self-identified rural regions to support the development and implementation of strategies they devised to meet their local needs. In the wake of this latest setback to advocates, the U.S. Department of Agriculture Rural Development (USDA RD)1 community and economic development staff created the Stronger Economies Together (SET) initiative in 2009 in partnership with the nation's four Regional Rural Development Centers (RRDCs) and the land-grant universities in the states.2

SET is a novel approach for the USDA to catalyze economic prosperity in rural America. At the time of our evaluation SET provided approximately 35 h of training, customized data (an approximately $15,000 in-kind contribution),3 and technical assistance to participating regions. The initiative started in 2010 with eight pilot states (Arizona, Kentucky, Louisiana, Missouri, New Mexico, Ohio, Pennsylvania, and West Virginia). With multiple SET regions per state, there were twenty-two multi-county regions in the initial pilot phase (referred to as Phase I). In Phase II, eleven new states (Alabama, Arkansas, Colorado, Indiana, Michigan, Mississippi, New Hampshire, New York, Texas, Vermont, and Washington) were added and sixteen new regions were formed. In July 2012, six new states (Georgia, Nevada, North Carolina, Oklahoma, South Dakota, and Virginia) joined the program and eleven new regions were being formed at the time this manuscript was submitted.4

In the spring of 2011, the authors of this article were selected as an interdisciplinary research team by the Southern Rural Development Center (SRDC), located at Mississippi State University, and the United States Department of Agriculture Rural Development (USDA RD) to conduct an external (independent third party) evaluation of SET. In response to the request for proposals, our team submitted a plan for a formative evaluation based on a variety of rigorous methods for a robust external evaluation. Our evaluation project plan included a variety of methods, including analysis of applicant data, observation of training, review of curricular materials, and social network analysis using participant data. As the project unfolded, the initial (formative) evaluation plan became irrelevant and unnecessary because of the client's urgent needs for immediate, ongoing feedback from our team to support adaptive program development. In this article, we describe the shift from a formative to developmental evaluation and the impact of our work on the program.

Section snippets

Developmental evaluation

In addition to the “science, technology, and craft of evaluation” (Morell, 2010, pp. 1–2), it is important to systematically understand how to contend with “surprise, with situations in which programs and evaluations do not behave as expected.” This article describes how a proposed formative evaluation in response to a request for proposals (RFP) for an external evaluation was transformed into a developmental evaluation approach that was very much from an insider's perspective as an integral

Findings from and impacts of the evaluation on the SET program

The evaluation team provided feedback to the SET national coordinating team from the beginning of its involvement. Part of this feedback occurred in quarterly evaluative reports to the client as part of the contract. The SET national coordinating team gave serious consideration to all recommendations in the evaluation team's quarterly and ad hoc reports. Some were implemented; some will not be implemented (based on logistical considerations, resource constraints, and so on). In response to each

Discussion

Because this was a developmental evaluation project, we provided feedback based on our observations and interpretations on an ongoing, real-time basis rather than saving what we found for the next quarterly report. We now detail specific examples of how our recommendations went from actionable items to implemented recommendations within some of the key areas unique to developmental evaluation: (1) evaluation team; (2) organizing and interpreting data; (3) flexibility; and (4) formative

Conclusion

This paper aims to advance the understanding of developmental evaluation by showing how findings from an external evaluation team are being turned into actionable agenda items for consideration by a national team. Based on our experiences, developmental evaluation, adapted to the program's context, proved useful for meeting the client's needs. It also allowed us to develop a closer relationship with the program team than might otherwise have been expected, allowing an easy flow of information

Acknowledgements

The authors gratefully acknowledge Dr. Lionel (Bo) Beaulieu (Purdue University), formerly Director of the Southern Rural Development Center at Mississippi State University, and Dr. David W. Sears, U.S. Department of Agriculture Rural Development, and their staffs for funding the project and for their collegial support throughout the evaluation. They provided valuable comments on an earlier draft of this manuscript. We would also like to thank Zachary Woolard, former graduate research assistant

Beth Walter Honadle (Ph.D., Syracuse University) is Director of the Program Monitoring and Research Division in the Office of Policy Development and Research at the U.S. Department of Housing and Urban Development in Washington, DC. She was Principal Investigator on the External Evaluation of the Stronger Economies Together Initiative, which was conducted while she was a Professor at the University of Cincinnati. Previously she has been a professor at the University of Minnesota and Bowling

References (29)

  • J.A. Gamble

    A developmental evaluation primer

    (2008)
  • D. Hix et al.

    Formative evaluation: Ensuring usability in user interfaces

    (1992)
  • B.W. Honadle

    Rural development policy: Breaking the cargo cult mentality

    Economic Development Quarterly

    (1993)
  • B.W. Honadle

    Rural development policy in the United States: Beyond the cargo cult mentality

    Journal of Regional Analysis and Policy

    (2001)

    • Cited by (0)

    Beth Walter Honadle (Ph.D., Syracuse University) is Director of the Program Monitoring and Research Division in the Office of Policy Development and Research at the U.S. Department of Housing and Urban Development in Washington, DC. She was Principal Investigator on the External Evaluation of the Stronger Economies Together Initiative, which was conducted while she was a Professor at the University of Cincinnati. Previously she has been a professor at the University of Minnesota and Bowling Green State University and she was a researcher and national program leader at the U.S. Department of Agriculture. Her research interests include policy analysis, program evaluation, community and economic development, and intergovernmental relations. The views expressed in this article are the authors’ and are not official positions of the U.S. Department of Housing and Urban Development.

    Marisa A. Zapata (Ph.D., Regional Planning, University of Illinois Urbana-Champaign) was an Assistant Professor in the School of Planning at the University of Cincinnati at the time of this project. Her research interests include collaborative planning, social equity, and plan-making. She has professional experience in public planning, policy and health, domestically and internationally. In Fall 2013 she joined Portland State University as an Assistant Professor in Planning.

    Christopher Auffrey (Ph.D., Urban, Technological and Environmental Planning, University of Michigan) is an Associate Professor in the School of Planning at the University of Cincinnati. He is a co-investigator on the external evaluation of the Stronger Economies Together Initiative. His research interests include fiscal impact analysis and healthy urban planning. Professor Auffrey has professional experience with urban revitalization projects in Los Angeles and an environmental consulting firm in Washington, DC.

    Rainer vom Hofe (Ph.D., Regional Science, Cornell University) is an Associate Professor in the School of Planning at the University of Cincinnati. His research activities focus mainly on urban and regional economic analysis and spatial statistical modeling. Professor vom Hofe has professional experience with fiscal impact analysis and economic impact studies for local governments as well as private entities in the larger Cincinnati MSA. He is a co-investigator on the external evaluation of the Stronger Economies Together Initiative.

    Johanna W. Looye (Ph.D., Urban and Regional Theory, Cornell University) is Associate Professor, School of Planning, University of Cincinnati. She is co-investigator on the External Evaluation of the Stronger Economies Together Initiative. She has considerable experience in field research, including work for the U.S. Economic Development Administration, for the City of Cincinnati, work conducted while a Fulbright Scholar in Brazil, work funded by the North-South Center, by the ILO in Brazil, and by the Ford Foundation in Brazil. She is currently PI for a U.S. Department of Education FIPSE grant entitled “Urban Sustainability and the Built Environment: A US-Brazil Consortium.”

    ☆

    Note: This article does not reflect official positions of the U.S. Department of Housing and Urban Development.

    View full text
    Published by Elsevier Ltd.