PROTOCOL: Development evaluations in India 2000–2018: A country impact evaluation map

Journal title or Report Series

1 | BACKGROUND 1.1 | The problem, condition, or issue Development evaluation refers to evaluation of development programs, which are defined as any social or economic programs aimed at community development. These are generally funded by the government, aid agencies, nongovernmental organizations, foundations, or firms under corporate social responsibility. Invariably, most of them require that their programs be evaluated to understand if the intervention succeeded or failed (Bamberger, 2000).  (Chandrasekar, 2015). PC of India which was later dissolved and new organization named National Institution for Transforming India (NITI Aayog) was created on January 1st, 2015. The NITI Aayog is responsible for preparing the policy framework, designing the public programs and also monitors and evaluates the outcomes achieved by different programs (NITI Aayog, 2018).
There has been and increasing trend of impact evaluations in low and middle income countries over the years. The largest share (34.4%) of studies is conducted in Sub-Saharan Africa followed by Latin America and Caribbean with 24.7% of the total studies, while South Asia ranks third at 17%. Nearly half of South Asian studies are conducted in India (Sabet & Brown, 2018). Though the numbers of impact evaluations is growing there is no central repository for easy access or to identify sectors with evident gaps.
Also, in a diverse country like India implementation and flawless evaluation of all the public development programs is a great challenge. Timely evaluations of such programs are often pushed by the policy makers and the program implementers to increase the accountability and for improvement of the program (Malhotra, 2014).
Despite, effective measures taken by the union and state governments there are evident gaps present in evaluation of these public development programs. Identification of these gaps will help in making the programs more effective and will eventually support in improving the development indicators of the country.
Evaluations of public programs also contributes toward cost cutting and cost-effectiveness as it gives information about what works and what does not. This is essential because of the rise in public expenditure and economy of India. Appropriate monitoring and evaluation of a program helps to identify the challenges in implementation of a particular program and provides an evidence base for decisions pertaining to public resource allocation; leading to more effective public policy development (Malhotra, 2014).
In an effort to align the "Strategy for New India" with India's commitment to the United Nations' Sustainable Development Goals (SDGs), NITI Ayog with extensive consultations and inputs published "Strategy for New India @ 75." It has identified 41 different areas that require either a sharper focus on implementing the flagship schemes already in place or a new design and initiative to achieve India's true potential.

| Scope of the evaluation map
The proposed country evidence and gap map will include all pub-

| Existing EGMs and/or relevant systematic reviews
To our knowledge this will be the first country evaluation map for development impact evaluations for India. Though similar maps are ongoing in Uganda and Kenya.

| OBJECTIVES
The primary objective of the India country evaluation map is to make the existing impact evaluations accessible to the users, and to support evidence-informed decision-making across government, development partners, and civil society. Similar to EGMs, country evaluation maps systematically search for relevant systematic reviews and impact evaluations based on a prespecified, published search protocol, and provide graphical representation of available evidence. The scope of country evaluation map, however, is larger than typical EGM as the former will include all policy domains that will affect the welfare of the people of the country, and include only Impact evaluations.

| Evaluation map framework
The framework has been developed through the following process: Stage 1: Initial framework is constructed through consultation of strategy and policy documents as identified from NITI Ayog strategic goals for a new India by 2022.
Stage 2: Piloting of the framework with 10-20 included studies.
The framework was finalized based on pilot coding of 10-20 studies.
Any subsequent changes to the framework will be recorded as deviations from protocol.

| Population
The relevant population for this map is both the people (including refugees) and institutions of India. Population subgroups will be: rural/urban, poor, people with disabilities, children, youth, aged, ethnic minorities, refugees, conflict-affected persons.

| Intervention
The map will cover all development interventions. The categorization of the interventions follows the categories in NITI Ayog's vision 2022. These are shown in Table 1.

| Outcomes
The outcomes are based on the SDGs ( Table 2). The Government of India recently committed to developing a roadmap to achieve the SDGs, so the country evidence map will be a useful tool in that work (Table 3).

Types of study designs
Only Impact evaluations will be included. Definition of Impact evaluation as described by Scriven's (1967): Impact evaluations: Impact evaluations are defined as intervention evaluations or field experiments that use quantitative approaches applied to experimental or observational data to measure the effect of an intervention relative to a counterfactual representing what would have happened to the same group in absence of that intervention. Impact evaluations may also test different intervention designs.
We will include studies that assess the effects of interventions using experimental and quasiexperimental study designs that allow for causal inference. Specifically, we will include: Health and well-being SDG 3 Education SDG 4 Gender SDG 5 Water and sanitation SDG 6 Energy, industry and infrastructure provision SDGs 7 and 9 Urban development SDG 11 Environmental sustainability SDGs 12, 13, 14 and 15 Peace and justice SDG 16 Global partnerships SDG 17 Inequality SDG 10 Abbreviation: SDG, sustainable development goal. Before-versus-after studies with no comparison group will not be included. Impact evaluation designs will be coded as randomized control trials and other impact evaluations.
Since the inclusion criteria are for impact evaluations of development interventions in India, systematic reviews are not included in this map.

Types of settings
The map will include evaluations studies undertaken in India regardless of setting.

Status of studies
We will include both completed and ongoing impact evaluations; to capture the latter, we will include prospective study records in trial registries or protocols when available.

Search methods for identification of studies
Electronic searches. Relevant impact evaluations will be identified through the main impact evaluation databases. The searches will include studies published from inception to the present.
Where the database allows a data range then the search is for 2000.
If that is not possible then records pre-2000 are removed by manual screening.
T A B L E 3 Summary of PICOS The study must be an impact evaluation of a socio-economic development intervention (so clinical trials are excluded)

Organization Searchable by Overlap with IE databases
Other sources. We will search Google Scholar using the very general search string "India Impact evaluation" (not an exact string), applying the ≥2,000 date range to identify additional gray literature. We will also search other sources as will be identified through expert consultations and as submitted as a result of dissemination of the Evaluation Map.

| Screening and selection of studies
The screening and coding tools are devised by A. S., who is leading the study who has prepared and managed the search for several Campbell maps. The entire process is guided by Howard white who is leading and known development expert. The tool will be piloted by the team members. All the team members have previous experience of working on systematic reviews and are well versed with screening and coding methodology.
Titles and abstracts will be screened against the inclusion criteria and relevant records will be downloaded into the review management software EPPI reviewer. The initial screening of records will be conducted by several reviewers screening the records from different databases. At this stage we will be overinclusive to ensure relevant studies are not omitted because sufficient information is not reported in title or abstract. Two reviewers will then independently review abstracts that have been judged to be potentially relevant at the first stage in more detail to determine which papers should be retrieved and reviewed at full text.
Two reviewers will then independently assess full text studies for inclusion, with any disagreements determined by a third reviewer.
3.6 | Data extraction, coding, and management Coding will be carried out in EPPI reviewer by two independent reviewers with third resolving the conflict.
The studies shall be coded on the basis of the following information:

Outcomes
Economic development (including poverty and employment) SDGS 1 and 8 Sustainable agriculture SDG 2 Nutrition SDG 2 Health and well-being SDG 3 Education SDG 4 Gender SDG 5 Water and sanitation SDG 6 Energy, industry and infrastructure provision SDGs 7 and 9 Urban development SDG 11 Environmental sustainability SDGs 12, 13, 14 and 15 Peace and justice SDG 16 Global partnerships SDG 17 Inequality SDG 10

| Planned analyses
The EGM will be supplemented by an EGM report that-based on tables and/or graphs-will descriptively summarize the number of studies included in the EGM and their distribution across different coding categories such as study type, geography, target populations, interventions, and outcomes. Each table/graph will be accompanied by brief narrative descriptions. The report will also discuss the potential use of the EGM and highlight its boundaries and limitations.
The report accompanying the map will provide tables representing the number of studies by • The types of evaluations

• Sectors and subsectors
• SDG domains and subdomains • Year • Source • Nationality of authors

• Geographic distributions
Cross tabulation of some of these variables will also be presented.

| Presentation
The primary dimensions of the map will be the types of evaluation (in rows) and various SDG indicators and subcategories in the column.
Filters will be provided for secondary dimensions such as rural, urban, people with disabilities, children, youth, aged, ethnic minorities, refugees, and conflict-affected persons.
This framework for the map will be through a consultation process.
The initial framework was constructed through consultation of strategy and policy documents. The framework and later the draft and final reports will be discussed with the Advisory Group for their inputs and feedback.

ROLES AND RESPONSIBILITIES
• Content: H. W. who has over 10 years of experience in working with international development.
• EGM methods: H. W. and A. S., who have coauthored four EGMs together. H. W. has assisted the development of Campbell collaboration guidelines and standards.
• Information retrieval: A. S. has drafted and designed the search strategy for all the Campbell EGM's.
• Coding and data extraction: All the authors have worked in conducting systematic reviews and will be responsible for the activity. Training and supervision on Eppi-reviewer will be provided by A. S.

SOURCES OF SUPPORT
This EGM has seed support from the Campbell Collaboration. Additional funding is being sought.

DECLARATIONS OF INTEREST
None.

PRELIMINARY TIMEFRAME
Protocol: February 2020 Draft map and report: December 2020

FUNDING
This map does not have any funding.

PLANS FOR UPDATING THE EGM
Currently no funding is available for this purpose. It will be sought once the map is published.