Setting research priorities for maternal, newborn, child health and nutrition in India by engaging experts from 256 indigenous institutions contributing over 4000 research ideas: a CHNRI exercise by ICMR and INCLEN

Background Health research in low– and middle– income countries (LMICs) is often driven by donor priorities rather than by the needs of the countries where the research takes place. This lack of alignment of donor’s priorities with local research need may be one of the reasons why countries fail to achieve set goals for population health and nutrition. India has a high burden of morbidity and mortality in women, children and infants. In order to look forward toward the Sustainable Development Goals, the Indian Council of Medical Research (ICMR) and the INCLEN Trust International (INCLEN) employed the Child Health and Nutrition Research Initiative’s (CHNRI) research priority setting method for maternal, neonatal, child health and nutrition with the timeline of 2016–2025. The exercise was the largest to–date use of the CHNRI methodology, both in terms of participants and ideas generated and also expanded on the methodology. Methods CHNRI is a crowdsourcing–based exercise that involves using the collective intelligence of a group of stakeholders, usually researchers, to generate and score research options against a set of criteria. This paper reports on a large umbrella CHNRI that was divided into four theme–specific CHNRIs (maternal, newborn, child health and nutrition). A National Steering Group oversaw the exercise and four theme–specific Research Sub–Committees technically supported finalizing the scoring criteria and refinement of research ideas for the respective thematic areas. The exercise engaged participants from 256 institutions across India – 4003 research ideas were generated from 498 experts which were consolidated into 373 research options (maternal health: 122; newborn health: 56; child health: 101; nutrition: 94); 893 experts scored these against five criteria (answerability, relevance, equity, innovation and out–of–box thinking, investment on research). Relative weights to the criteria were assigned by 79 members from the Larger Reference Group. Given India’s diversity, priorities were identified at national and three regional levels: (i) the Empowered Action Group (EAG) and North–Eastern States; (ii) States and Union territories in Northern India (including West Bengal); and (iii) States and Union territories in Southern and Western parts of India. Conclusions The exercise leveraged the inherent flexibility of the CHNRI method in multiple ways. It expanded on the CHNRI methodology enabling analyses for identification of research priorities at national and regional levels. However, prioritization of research options are only valuable if they are put to use, and we hope that donors will take advantage of this prioritized list of research options.

India is the second most populous country in the world with many pressing health problems that, in fact, hugely determine the global health statistics. Maternal, neonatal, child health and nutrition (MNCHN) together contribute to the largest burden of disease in India. Public health research decisions in India have traditionally been guided by a small group of experts who are located mostly in the metropolis and are constrained by individual and organizational preferences. In 2011, in response to the seemingly unachievable Millennium Development Goals 4 and 5 (MDG4, MDG5), National Health Mission goals, and the upcoming Sustainable Development Goals 2030, the Indian Council for Medical Research (ICMR; the apex institution for medical research in India) and the INCLEN Trust International (INCLEN; which was the CHNRI Secretariat since 2010) came together to undertake this nationwide research priority setting exercise for MNCHN using the CHNRI methodology. Newborns, children (0-18 years), and reproductive age women (15-49 years, including pregnant women and lactating mothers) were identified to be the target population for prioritization along the life-course continuum. India has large population diversity along with regional-and state-level heterogeneity in governance, program performance, sociocultural milieu and economics. Hence, it was decided that research priorities would be identified at national and sub-national (regional) levels with a 10-year reference time period (2016-2025) and through inclusion of a large number of stakeholders for representativeness.

METHODS
The ICMR-INCLEN National Research Priority Setting (RPS) exercise was completed between 2012 and 2016. The exercise was coordinated by the RPS project management team at the Executive Office of IN-CLEN, New Delhi. The team had experts in the four core MNCHN disciplines (pediatrics, obstetrics and gynecology, community medicine, and public health nutrition) and was multilingual and hence, able to communicate and engage participants from across the country.
States and union territories were grouped into three regions in order to enable sub-national priorities. The three regions were: (i) Empowered Action Group (EAG) States (Rajasthan, Madhya Pradesh, Chattisgarh, Odisha, Jharkhand, Bihar, Uttar Pradesh and Uttarakhand) and North-Eastern (NE) States (Sikkim, Assam, Meghalaya, Tripura, Mizoram, Manipur, Nagaland, Arunachal Pradesh); (The Government of India has identified eight states with poor health and development indicators as EAG states for focused action. EAG and NE states share similarities in MNCHN contexts and program performance); (ii) Northern states and Union territories (Jammu & Kashmir, Punjab, Himachal Pradesh, Haryana, Chandigarh, Delhi, and West Bengal); and (iii) States and Union Territories in Southern and Western part of the country (Kerala, Tamil Nadu, Karnataka, Andhra Pradesh and Telangana, Maharashtra, Gujarat, Goa, Puducherry).
Four key structures were created to accomplish the task, outlined as follows. An RSC was constituted for each of the four themes. The RSCs' membership included technical experts (subject experts, basic scientists and public health specialists), social scientists, program specialists (health, and woman and child development), and donor agency representatives. Technical experts were identified through a literature search for active research contribution to respective MNCHN domains ( Table 2). The RSCs participated in the crowdsourcing processes along with the nationwide network. They also helped in the iterative refinement and consolidation of the research options (ROs) and in finalizing the scoring criteria and their definitions. Respective RSCs presented the study findings to the second meeting of the NSG for review.

The Nationwide Network for crowd sourcing
A network was established with experts identified from institutions and departments across the country. Faculty/researchers from departments that were directly or indirectly engaged in work pertaining to MNCHN (eg, obstetrics & gynecology, pediatrics, neonatology, community medicine, biochemistry, physiology, pathology, microbiology, midwifery, public health nutrition and home sciences, social sciences, statistics and demography, and agriculture) were contacted through their respective institutional heads. The effort was to secure similar proportion of faculty members/researchers with more than 10 years of research or teaching experience (ie, 'senior' faculty) and those who are junior/middle level with 5-10 years of experience. National and zonal office-bearers of major professional associations in MNCHN (the Indian Academy of Pediatrics, the National Neonatology Forum, the Federation of Obstetrics and Gynecological Societies of India, the Indian Association of Preventive and Social Medicine, the Indian Public Health Association, the Nutrition Society of India, and the Indian Dietetic Association) were also contacted for participation. Central and state-level policy-makers and program managers were also invited to participate in the exercise. These were from departments of health and of women and child development. Experts were also identified through snow-balling and invitations in personal capacity.
The members in the nationwide network consented to be allocated into one of the four themes according to their expertise and publication history to achieve equitable regional and disciplinary representation in each theme. In this manner, for the first round of crowd sourcing, 1423 experts (including the 112 in the RSCs) were identified, of whom 1178 could be contacted. Of these, 12 declined to participate. Of the remaining 1166 experts (

The Larger Reference Group (LRG)
Beyond 75% of CHNRI exercises published have not employed a LRG (mostly due to trouble composing the group). Of those that could, most have been conducted at a national level [7]. To incorporate broader societal perspectives and values within the exercise, we employed a LRG which was composed of policy decision makers (n = 24; Central and State politicians and bureaucrats from key Ministries, eg, Health and Family Welfare, Woman and Child Development, Human Resource Development), senior researchers (n = 17), MNCHN program managers from central and state governments (n = 24) and representatives from research funding organizations (n = 19). The LRG attributed relative weights to the scoring criteria which helped to generate criteria-weighed priority ranks for the ROs. Figure 1 shows the schematic flow of activities with timelines.

Review of literature and identification of areas of concern
Extensive review of literature on burden of MNCHN related conditions was done in 2012-13 with focus on Indian data. We searched published literature (indexed and non-indexed), Government of India' s policy documents and reports, program reviews and grey literature for the period of 1990 to 2012/2013. PubMed, CINAHL and Embase databases were searched. Based on the compendium of literature (available at: www.inclentrust.org), a draft list of "areas of concern" (AOCs) was prepared for each of the RPS themes and presented to the first meeting of the NSG for review. The AOCs accounted for conditions that collectively contributed to at least 75% of the mortality and morbidity burden in the respective theme.
The NSG suggested that maternal health encompassed three components viz., morbidity, mortality and stillbirths (most stillbirths occur in-utero and are thus are a maternal health concern). Similarly, it divided the nutrition theme into maternal and childhood nutrition components. The NSG advised to include three additional AOCs in each of the themes: "social determinants," "impact and improvement of existing composite public health packages," and "novel & innovative public health interventions." The final approved list of AOCs under the four themes is presented in Table 4.
The NSG also suggested that all research ideas (RIs) be segregated into the four domains of research: (i) description (burden of disease, epidemiology, etiology and risk factors, biomarkers, pathophysiological descriptions); (ii) discovery (identification of novel pathways, discovery of novel clinical and public health interventions/package, technology inventions, discoveries and innovations); (iii) delivery (health policy and systems research, including program evaluation and implementation research); and (iv) development (improving the existing intervention, ie, design, deliverability, affordability and sustainability).

First r o u n d o F c r o w d s o u r c i n g
Solicitation of research ideas (RIs) from the Nationwide Network: An online software was designed by INCLEN for submission of RIs by the network. The software had seven separate electronic forms: maternal health (n = 3: mortality, morbidity & stillbirth); newborn health (n = 1); child health (n = 1); and nutrition (n = 2: maternal & child nutrition) themes]. The experts in the nationwide network and RSCs were sent an initial email and then contacted over the phone: (i) to sensitize them about the method of the research priority setting exercise; (ii) to provide them the context and scope of the exercise; and, (iii) the provide them with the purpose of the first round of crowdsourcing. Each participant was provided with an individualized log-in username and password for the dedicated software. The participant could log in to only one of the seven electronic forms as pre-assigned to him/her. After logging-in, s/he was asked to enter personal details (name, area(s) of work, employment status (working/retired), institution, state/union territory, alternative email ID). S/he was then taken through a self-orientation power-point tutorial. The list of AOCs was then displayed on his/her computer screen and the participant was instructed to select any two AOCs to contribute RIs in the four domains of research (description, discovery, delivery and development). The expert was not limited in the number of RIs s/he could submit under each domain. The electronic forms allowed for completion over multiple sessions. An offline version of the form was prepared and shared with participants who had difficulty in accessing the internet. A total of 3497 RIs were obtained across the MNCHN themes from 498 experts (42.3% participation).

Refinement of the research ideas:
The RPS project management team at INCLEN along with the RSCs closely examined each RI and rephrased, split, and combined the RIs (as required) keeping the core idea  Low birth weight 3 Micro-nutrient deficiencies (iron/folic acid/zinc/iodine/Vitamin A) 4 Childhood overweight and obesity 5 Nutrition deficiency associated congenital malformations 6 Fetal and child nutrition and origin of adult chronic non-communicable diseases [eg, cardiovascular diseases, metabolic syndrome, obesity etc.] 7 Socio-cultural and economic determinants: time constraint with mothers entering into the work force 8 Socio-cultural and economic determinants: care and feeding practices 9 Socio-cultural and economic determinants: competing use of resources for goods and services other than nutrition/food 10 Socio-cultural and economic determinants: globalization & market forces influencing food habits 11 Socio-cultural and economic determinants: status of girl child and women in the community 12 Socio-cultural and economic determinants: use of pesticides & fertilizers 13 Socio-cultural and economic determinants: potable water, hygiene and sanitation 14 Socio-cultural and economic determinants: others (Please specify ____) 15 "Existing" composite public health packages with potential impact on Child nutrition [eg, ICDS, Mid-day Meal Program] 16 "Novel & Innovative" composite public health packages with potential impact on child nutrition [eg, Food fortification program, promotion of kitchen gardens/organic farming, deworming, convergent-innovation coalition to address issues of anemia, under-nutrition, obesity] Nutrition Theme: Maternal Nutrition Component 1 Anemia among women of reproductive age group 2 Iodine deficiency disorders among women 3 Vitamin D deficiency among women 4 Maternal overweight & obesity and other non-communicable diseases 5 Socio-cultural and economic determinants: time constraint with mothers entering into the work force 6 Socio-cultural and economic determinants: care and feeding practices 7 Socio-cultural and economic determinants: competing use of resources for goods and services other than nutrition/food 8 Socio-cultural and economic determinants: globalization & market forces influencing food habits 9 Socio-cultural and economic determinants: women' s status in the community, family structures and norms 10 Socio-cultural and economic determinants: others (Please specify ______) 11 "Existing" composite public health packages with potential impact on maternal nutrition 12 "Novel & Innovative" composite public health packages with potential impact on maternal nutrition intact and without discarding any RI. The original RI list was maintained as a separate file for ready reference at any time. The process was intuitive, consultative and iterative (completed through brainstorming by teams over several sittings). As far as possible, the RIs were refined in a way that described the population, intervention, control, and outcome (PICO). This process led to a compendium of 4003 RIs from the original 3497 RIs. ( Table 5).

Development of research options (ROs):
The 4003 RIs were consolidated onto 373 ROs. These were crystallized through iterative refinement to avoid duplication and redundancy. Each RO represented a portfolio of inter-related RIs that addressed a central research concept. Thus, the ROs addressed multiple AOCs and several of these pertained to cross-cutting issues across domains, components and themes. The ROs were finally categorized into four themes (maternal health: 122, newborn health: 56, child health: 101, nutrition: 94) ( Table 6).

second r o u n d o F c r o w d s o u r c i n g
Finalization of criteria for scoring: Previously published CHNRI exercises were reviewed extensively to retrieve scoring criteria used in past exercises. Two rounds of consultation were held with RSC members, international CHNRI experts, and experts from the World Health Organization who had been close-  ly associated with previous CHNRI exercises. Five succinctly worded criteria (answerability, relevance, equity, innovation and out-of-the-box thinking, and investment on research) were finalized. These criteria were believed to be consistently applicable across domains, themes and ROs (Box 2). The context and scope of the present exercise, nature of the ROs and the large number of scorers from various disciplines across India that were to score the research options were the key considerations while deciding on the scoring criteria to be used. The scorers were expected to evaluate the ROs against the criteria by choosing one of the following responses: 'Yes' if the research option favorably met the criterion query, 'No' if it did not, and 'Not my expertise' if the scorer felt that s/he was not sufficiently informed to adjudge the research option against the particular criterion. While other CHNRI exercises employed sub-questions under each criterion, we chose to forego sub-questions as we were advised that sub-questions usually had high agreement [12] and also because our exercise had a large number of ROs to be scored and we were 3. Innovation and out-of-box thinking to resolve complex, and refractory challenges. Does the new research have the potential for transformative change in the health system/ health care?
4. Equity. Is it likely that the research product will address the differences in health and nutrition that are systematically associated with social, cultural and economic hierarchies, ethnicity, gender, environment and geographic disadvantages, thereby reducing inequities?

Investment on research.
Is it likely that the potential impact and benefits of the new knowledge on health/ nutrition will outweigh the consideration of investments on research?
aiming to maximize retention of participants by minimizing scorer fatigue, especially in order to preserve the validity of our planned regional analyses.

Scoring of the research options by the Nationwide Network:
The scoring exercise was done using a user-friendly online interface (www.surveymonkey.com) that allowed for having individualized scorer accounts that could be accessed through an invitation email from the INCLEN RPS project management team. The software could readily archive access details (email and IP addresses) and responses selected by the scorer. Once the scorer logged in, s/he underwent a comprehensive orientation of the context and method of the exercise, and the scoring criteria and process. Thereafter, ROs appeared in a random sequence, one at a time, on the scorer' s computer/smart phone screen. The scorer was requested to score all the ROs for the assigned theme. As the number of ROs to be scored was high and could have led to high scorer burden and attrition, each scorer was randomly allocated a combination of two of the five criteria for scoring. Five such criteria combinations (survey questionnaires) had been prepared for scoring: (i) Answerability and Innovation; (ii) Answerability and Equity; (iii) Relevance and Innovation; (iv) Relevance and Investment on Research; and (v) Equity and Investment on Research. The nationwide network was stratified at two levels: first, according to their participation status in the first round of crowd sourcing ('participated', 'could not participate', or 'newly invited' experts); and, second, according to their region. Subsequently, the experts within each region were equally distributed across the five survey questionnaires within the theme through consecutive allocation (the expert with serial number 1 got Survey Questionnaire 1; the next in line got Survey Questionnaire 2 and so on; the questionnaire allocation cycle was restarted with every 6th expert).
It was mandatory for the scorer to evaluate the RO on the screen against both of the assigned criteria before moving on to the next RO (ie, skip logic was disabled). However, the scorer could review and edit his previous responses once s/he had moved forward. Completion over multiple sessions was allowed to avoid effects of scorer fatigue and overcome time constraints. The RPS project management team at IN-CLEN remained vigorously engaged with the nationwide network through email and telephone for immediate troubleshooting and timely reminders, and used continuous real-time data monitoring to check progress. Scorers who requested hard copies of the questionnaires instead of the online process were provided with the same for recording the responses. In the second round of crowdsourcing, 893 scorers participated (58.1% participation rate) ( Table 7).

Assignment of relative criteria weights by the LRG
The LRG members were given an in-depth explanation of the CHNRI exercise. They were then requested to assign relative weights to the scoring criteria by distributing a hypothetical amount of Indian Rupees (INR) 100 across the five criteria, giving the maximum amount to the criteria they felt to be the most important and the minimum to the least important. The relative weight for each criterion was computed by calculating the arithmetic mean of the average amount received by the respective criterion in each LR6 constituency (

Data management and analysis
The "Yes" and "No" responses were scored as "1" and "0" respectively. The "Not my expertise" responses were excluded from the calculations. Relative ranking and Research Priority Scores (RPS) were calculated as follows [13]:

EAG -Empowered Action Group, UT -Union Territories
Average scores received against each of the five criteria were calculated for each RO.
1. The criteria weights (as assigned by the LRG) were applied to the mean score received by each criterion.
2. Research Priority Scores (RPS) were calculated by adding together each criterion' s weighted scores for each RO.
The ROs were arranged in descending order of their RPS to get national and regional rankings. Work location of the scorer as entered by him/ her at the time of the scoring determined the regional ranking.
Average Expert Agreement (AEA) [14] was also calculated for each RO. The AEA is a proportion of scorers who scored the most common score for a particular RO divided by the total number of scorers who scored that RO.
The second meeting of the NSG reviewed the ranked list of national and regional research priorities. The group further suggested to identify ROs relevant to three more themes: (i) adolescence; (ii) issues cutting across four MNCHN themes for greater impact on health and health systems; and, (iii) areas requiring biotechnology methods from the compendium of 373 ROs, and generate ranked lists according to their RPS for each of these.
The results from all exercises are reported in-depth separately in manuscripts prepared for submission to the Journal of Global Health. The overall discussions by the National Steering Group on the results and way forward for the exercise has been accepted for publication in the Indian Journal of Medical Research.

DISCUSSION
The COHRED Working Group on Priority Setting highlighted that engagement of a wide spectrum of stakeholders is essential to identify priorities that reflect research needs, available technical and financial capacity, and societal values and ethics [15]. Stakeholder engagement, and data and capacity constraints frequently impeded the process for setting priorities, more so in the LMICs [16]. The current exercise, through systematic inclusion of diverse range of national stakeholders in a LMIC setting, identified priorities for maternal, neonatal and child health and nutrition at national and sub-national (regional) levels. The exercise leveraged the inherent flexibility of the systematic CHNRI method and built further methodological robustness. CHNRI exercises hitherto had taken a conservative approach in considering active contribution to research/policy as a selection pre-requisite for scorers. In contrast, we expanded the stakeholder base to include diverse range of doers and users (techno-managerial) of research in the field of MNCHN. This helped in including a variety of viewpoints in the scoring process and possibly, led to prioritization of ROs that was important to both.
Having Indian nationals as the exclusive contributors and scorers to this exercise makes it unique from previous exercises. In this way, this CHNRI exercise is truly a representation of, and driven by, India' s health and nutrition community. Moreover, the exercise is the first to conduct subnational-level analysis which, in a country as large and diverse as India, is imperative to truly explore research priorities and enable the country to tailor interventions regionally. With effective use of technology and building on IN-CLEN' s network for multi-centric studies, 498 experts from across India contributed research ideas and 893 experts were involved in the scoring process. About 75 (60-96) experts were involved per region per theme to score the ROs. The large number of scorers ("sample size") should have led to saturation and stable estimate of priority ranks at national and sub-national (regional) levels [8]. The improved response rates between first and second rounds of crowdsourcing should have reduced bias [17]. Gender distribution of scorers is a reflection of skewed gender participation in program management, research and academia for the themes considered in this exercise. The scorer profiles have been discussed in details in the respective thematic papers prepared for submission to JoGH.
To minimize scorer fatigue, we asked the participants to score against predefined pairs of criteria allocated randomly to them instead of all five criteria. The AEA for each evaluated research option represents the proportion of scorers that gave the most frequent (modal) response [14]. For the top 10 ROs at national level across the themes, the AEA for both individual and aggregate of the five criteria was fairly high (maternal health: 0.887-0.929; newborn health: 0.871-0.902; child health: 0.899-0.923; nutrition: 0.869-0.923) indicating consistency among the scorers. This also indicates minimal bias due to partial criteria scoring adopted in the current exercise and appears to be a pragmatic approach for better participant compliance without affecting the validity of the priority setting scoring. There were four distinct constituencies among the LRG; the LRG is to be viewed as a strength since different constituencies are likely to have differences in their collective perspective about research priorities [18]. It was interesting to observe that "Relevance" was accorded the highest weight by all the LRG sub-groups highlighting that priorities should be suited to the context.
In view of the disease burden and significance of the health systems in the implementation and delivery of services, the NSG suggested developing ranked priority lists for adolescent health, cross cutting themes and biotechnology related ROs from the 373 ROs spread across different themes. These lists will, at best, be an indicative priority list because the ROs were picked up from different thematic groups, scored by dissimilar set of experts with differences in their professional expertise. Although the overall AEA was high across themes, the validity of RO scores to determine their relative ranking shall remain unknown for these additional lists.
The exercise was the largest to-date use of the CHNRI methodology in terms of research ideas collected, processed and scored, and the number of participants and spectrum of stakeholder constituencies engaged. It expanded on the CHNRI methodology and thus, contributes to further evolution of the CHNRI method as a robust, inclusive, participatory, transparent and objective technique for identification of research priorities. It has been opined that prioritization processes will have an impact only if funders have a buy-in. It is also anticipated that there is an imminent challenge to develop tools to detect and evaluate the impact of CHNRI exercises on funder decision making and priorities [19]. A recent article in Lancet affixes with the research funders and research regulators, the primary responsibility of addressing the sources of avoidable waste once research priorities are set [20]. We hope that ICMR-INCLEN collaborative effort helps in rational distribution of health and nutrition research budget by the Government of India and donor agencies funding research in India and in similar LMIC contexts, and also inform any midcourse correction of currently funded research portfolio as needed. Sub-national (regional) prioritization should further help in matching the exercise' s findings to other LMIC contexts. This exercise can serve as a guidance for other LMICs, especially those with diversity among their populations, in setting research priorities nationally.