Interfacing theories of program with theories of evaluation for advancing evaluation practice: Reductionism, systems thinking, and pragmatic synthesis
Introduction
Program evaluation theories are frameworks driving questions, methodologies, and development of new knowledge, based on assumptions believed to be true. They guide the evaluation activities and processes, allow feedback to the program, and inform practice about achievements or missing/weak links preventing successful outcomes. Program evaluation theories have two levels: theories of program and theories of evaluation. The theories of program provide assumptions about how to conceptualize a program for the purposes of analysis, understanding, and evaluation (Chen, 1990; 2015) while the theories of evaluation provide assumptions about how to design and conduct a sound and useful evaluation (Alkin, 2013; Shadish, Cook, & Leviton, 1991). These two types of theories are related, but are often discussed separately, thus forfeiting opportunities for understanding their relationships and emerging synergies.
This article attempts to contribute toward filling this gap by discussing these two types of theories and their relationships under three theoretical perspectives: reductionism, systems thinking, and pragmatic synthesis. Reductionism and systems thinking have been used as corner stones to build program evaluation as a discipline. However, these two transdisciplinary perspectives, originated from matured sciences, cannot meet all evaluation needs. Pragmatic synthesis, an original, gross-root perspective, is developed to address unmet evaluation needs. An examination of these three perspectives’ theories of program and theories of evaluation can better guide designing fruitful evaluations, improve the quality of evaluation practice, inform potential areas for developing cutting-edge evaluation approaches, and contribute to advancing program evaluation toward a mature applied science.
Section snippets
Reductionism
Reductionism is the philosophical belief that a system, complex or not, can be broken down into simpler fundamental components which are easier to analyze, understand, or explain (Kemeny and Oppenheim, 1956, Nagel, 1961). This belief is rooted in the scientific philosophy of parsimony. Reductionism stresses that the aim of science is to identify, isolate, and assess causal relationships between variables. It has been successfully applied in physical and social sciences. A major attraction of
Systems thinking
Systems thinking offers a view at the other end of the spectrum from reductionism: it indicates that a system cannot be understood by reducing it to smaller parts, but by embracing the complexity that emerges from interconnecting parts. The perspective is often introduced with the phrase: The whole is more than the sum of its parts. Systems thinking is very broad and of long standing, being developed as a linkage of multiple disciplines including biology, psychology, physics, computer science,
Pragmatic synthesis
In spite of their contributions, reductionism and systems thinking represent the extreme ends of a theoretical spectrum; many real-world programs may actually fall in the middle. Evaluators who work on these moderate-complexity programs have no choice but to select one of the two perspectives for guiding their work, often insufficiently or inappropriately addressing the stakeholders’ interest and need. It becomes evident that an alternative theoretical perspective must be developed to serve
Implications for future directions
The above discussions of the theoretical perspectives and approaches raise the following crucial issues that need to be addressed for further advancing program evaluation in the future.
Conclusions
This article discussed theories of program, theories of evaluation, and their relationships under three theoretical perspectives: Reductionism, systems thinking, and pragmatic synthesis. Each theoretical perspective has its own theories of program and theories of evaluation to guide and support evaluation practice. An understanding of these perspectives, their relative strengths and challenges, and related issues are helping evaluators to choose appropriate evaluation designs and better service
Acknowledgements
The author thanks Amy DeGroff, Nicoletta Stame, Nannette Tuner, Liliana Morosanu, Apollo Nkwake, and Nathan Morrow for valuable comments on earlier drafts of this article.
References (58)
- et al.
Systems thinking
Evaluation and Program Planning
(2008) - et al.
Using the exhibited generalization approach to evaluate a carbon monoxide alarm ordinance
Evaluation and Program Planning
(2014) The bottom-up approach to integrative validity: a new perspective for program evaluation
Evaluation and Program Planning
(2010)- et al.
A review of the recent contribution of systems thinking to operational research and management science
European Journal of Operational Research
(2010) Evaluation roots: a wider perspective of theorists’ views and influences
(2013)- et al.
Intervention mapping: a process for developing theory and evidence-based health education programs
Health Education & Behavior
(1998) - et al.
Systems thinking, knowledge and action: towards better models and methods
Evidence & Policy: A Journal of Research, Debate and Practice
(2010) - et al.
Greater than the sum: systems thinking in tobacco control [Report document or other monograph]
(2007) - et al.
Experimental and quasi-experimental designs for research
(1963) - et al.
Framework for design and evaluation of complex interventions to improve health
BMJ
(2000)
Assessing program outcomes from the bottom-up approach: an innovative perspective to outcome evaluation
The multi-goal, theory-driven approach to evaluation: a model linking basic and applied social science
Social Forces
Formal theory versus stakeholder theory new insights from a tobacco-focused prevention program evaluation
American Journal of Evaluation
Theory-driven evaluations
Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness
Practical program evaluation: theory-driven evaluation and the integrated evaluation perspective
Quasi-Experimentation: design and analysis issues for field settings
A system review of theory-driven evaluation practice from 1990 to 2009
American Journal of Evaluation
Developing and evaluating complex interventions: the new Medical Research Council guidance
BMJ
Program theory-driven evaluation science: strategies and applications
Lessons from a concurrent evaluation of eight antibullying programs used in Sweden
American Journal of Evaluation
Purposeful program theory
External validity: we need to do more
Annals of Behavioral Medicine
Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition
The American Journal of Public Health
The public health approach to the prevention of child maltreatment
The girlfriends project evaluating a promising community-based intervention from a bottom-up perspective
American Journal of Evaluation
Crossing the quality chasm: a new health system for teh 21th century
Logic model development guide
Cited by (30)
Living your best life: The mindful pursuit of student-athlete thriving
2023, Asian Journal of Sport and Exercise PsychologyOpioid risk screening: Program evaluation from the community pharmacists’ perspective
2022, Journal of the American Pharmacists AssociationCitation Excerpt :This study focuses on evaluation of preventative opioid risk screening from the pharmacists’ perspective. The evaluation of pharmacist perceptions of a pharmacist-delivered opioid risk assessment may offer value in identifying influential factors leading to the success of opioid risk screening in the community pharmacy setting.18,19 The purpose of this study is to evaluate how community pharmacists perceive the use of an opioid risk screening for patients receiving opioid prescriptions and the associated implications for improved patient-centered care.
Parent support programs for parents of children with autism spectrum disorders: What the research tells us
2020, Annales Medico-PsychologiquesThe need for theory evaluation in global citizenship programmes: The case of the GCSA programme
2018, Evaluation and Program PlanningCitation Excerpt :Articulating this understanding enables a shared awareness among programme stakeholders. A theory evaluation gives an important indication of whether the theory behind the programme is sound and whether the programme could be expected to achieve the desired results if implemented appropriately (Chen, 2016; Donaldson, 2007; Rossi et al., 2004; Weiss, 1998). Conducting a robust theory and design evaluation is a necessary step in evaluation practice because it provides a good foundation for future process and implementation, outcome and impact as well as cost and efficiency evaluations (Rossi et al., 2004).
Case study with a participatory approach: Rethinking pragmatics of stakeholder engagement for implementation research
2021, Annals of Family MedicineCitation Excerpt :Then again, researchers who adopt a transformative posture (as proposed by Reilly) will work closely with community or organizational partners in the co-construction of the implementation using a participatory case study approach. Finally, a “middle ground” approach20 may be to adopt a pragmatic posture (as proposed by Merriam), where researchers use a case study with a participatory approach to conduct an implementation analysis of a health care innovation while consulting community or organizational stakeholders in certain phases of the research. Adopting this epistemological posture, we will present the example of the PriCARE program5,6 in the next section.