Crowdsourcing priorities: a new participatory ex-ante framework for crop improvement

Demand-led approaches to crop breeding involve ranking priorities across different disciplines and stakeholder categories, but the implications of decisions made during varietal development are frequently understood only years later. Breeding teams must work a priori to rank crop improvement priorities and product concepts considering the context of the current, and ideally future, environmental, production and market conditions that a variety will be entering upon release. We propose PEEP (Participatory Ex-antE framework for Plant breeding), a new ex-ante framework, as a methodological tool for priority setting in plant breeding. PEEP leverages two elements: the usage of a heterodox methodological approach and the strong emphasis on the participation of knowledge-rich stakeholders. PEEP ranks crop improvement impacts based on a heterogenous set of environmental, social, and economic benefits and it employs a recursive and tailored multi-stakeholder approach to relate crop improvement impacts and product concepts. PEEP builds on the need to engage technical as well as practical knowledge and utilizes a tailored engagement strategy for each knowledge-rich stakeholder involved. The outcome is an assessment that ranks crop improvement impacts and breeding product concepts according to designed set of criteria. PEEP is scalable, gender inclusive, and crop agnostic. The results of PEEP are ex-ante recommendations for breeding teams in National Agriculture Research centers (NARs) and CGIAR centers alike. This methods manuscript describes the theoretical foundations of PEEP and its four phases of implementation.


Introduction
Crop breeding is a unique field in which the implications and impacts of decisions made during varietal development will be understood only years later when the resulting variety is released to farmers.Breeders make decisions a priori that consider both current and, ideally future, environmental, production and market conditions into which a variety will be released.Significant shifts in breeding paradigms, under the banner of "modernization, " now position demand or market-led approaches to be non-negotiable (Tarjem et al., 2022).Yet this reorientation and need to respond to complex diversity are at odds with the limited resources most public sector crop breeding for development programs possess.This creates a need for research prioritization within crop breeding programs (Pemsl et al., 2022).A growing body of literature documenting how social differences drive trait and varietal preferences complicates the picture (Fisher and Carr, 2015;Weltzien et al., 2019), asking breeders to understand the Occelli et al. 10.3389/fsufs.2023.1265109Frontiers in Sustainable Food Systems 02 frontiersin.orgpriorities that women and men assign to genetically determined traits (Orr et al., 2018).Moreover, demand-led approaches necessitate ranking priorities with an interdisciplinary team, merging needs from different disciplines and experts, such as plant breeders, gender specialists, rural development experts, agricultural economists, and value chain stakeholders (Pemsl et al., 2022).Current demand-led paradigms that focus on triangulating on-farm genetic gains, market responsiveness, and social impacts need priority setting frameworks that link crop improvement priorities, preferred traits and expected impacts.In this methods manuscript, we describe a participatory ex-ante framework for plant breeding priority setting, focusing on crop improvement priorities1 and product concepts.The framework, called PEEP (Participatory Ex-antE framework for Plant breeding), is developed to relate crop improvement priorities, expected impacts and hypothetical new varieties.The framework centers on the question: when targeting a specific impact, which hypothetical new variety (expressed in the form of a product concept)2 should be prioritized by breeding programs?Complementing existing participatory breeding approaches, PEEP explores "why" a breeding priority is important and most impactful, in addition to "what" crop improvement priority is top-ranked.PEEP leverages two tenets to answer this question: a heterodox methodological approach and the engagement of knowledge-rich stakeholders.In contrast to ex-ante frameworks built on economic surplus and optimization modeling, PEEP ranks priorities based on a heterogenous set of environmental, social, and economic impacts determined by stakeholders.Furthermore, PEEP employs an iterative and tailored multi-actor approach.Building on the principle of engaging technical as well as practical knowledge, PEEP involves an array of knowledge-rich stakeholders.The outcome is an assessment that ranks crop improvement priorities and breeding product concepts according to a designed set of criteria, co-created with stakeholders and breeding programs.PEEP produces ex-ante recommendations for breeding teams in National Agriculture Research Centers (NARs) and other crop improvement research centers, including CGIAR, using an analysis which is scalable, gender inclusive, and crop agnostic (Mills, 1997).Complementarily, PEEP could also function as a monitoring tool, to align the research agenda of breeding programs to both existing high-level objectives (e.g., 2030 Sustainable Development Goals) and local stakeholders' needs.
Below we describe the theoretical foundations of the framework and its four phases.In section 2, we locate the work in the literature and highlight the novelty of the framework.In section 3 the four phases of the PEEP framework are described.A brief discussion (section 4) on advantages and limitations of PEEP concludes the manuscript.A pilot with the Institute of Environment and Agricultural Research (INERA) in Burkina Faso is underway to test the practicability and reflect on the experience of PEEP, and those results will be published separately.

Situating PEEP in ex-ante priority frameworks
Crop breeding is a discipline grounded in foresight of future needs of growers, processors and consumers.Setting research and development priorities ex-ante is therefore a necessity to succeed in meeting these demands.We examined the literature around ex-ante research priority setting approaches to situate our work and identify gaps and opportunities for methodological development.To date, different approaches have been developed to support priority setting in international agricultural research programs (Wiebe et al., 2021;Alston et al., 2022).Applications span from CGIAR-level (case studies can be found in Raitzer and Kelley, 2008), to national research prioritization (e.g., EMBRAPA in Brazil as described by Avila et al., 2002).Recently, the use of international agricultural research prioritization exercises has been more sporadic, with less data intensive procedures preferred (Thornton et al., 2018).These alternative methods range from simple qualitative scoring exercises to highly complex simulation models estimating the functional relationship between inputs (research investments) and agricultural outputs while accounting for the underlying uncertainty (see Braunschweig (2000), for an overview of different priority setting methods).
We summarize existing ex-ante priority setting frameworks (Supplementary Table A1) and characterize them along three dimensions of scale of analysis, translation of benefits into dollar values, and sex disaggregation.These three dimensions represent junctions at which ex ante frameworks that are both locally relevant and gender responsive distinguish themselves from more traditional approaches.Most existing frameworks focus on benefits in economic terms, and rarely using sex-disaggregated data (Supplementary Table A1).Countrylevel assessments are preferred, even though frameworks allowing a flexible scale of analysis (i.e., national and regional) are common.
Guided by an interest to develop a methodology that could be more participatory and engaging for respondents, we further looked at frameworks for their level of participation and engagement with stakeholders.Participation describes the degree at which studies involve a variety of different stakeholders, beyond scientists.Engagement exemplifies the frequency at which stakeholders are involved and if feedback and validation mechanisms are put in place.Studies in the early 1990s guided by economic surplus theory and cost benefit analysis are less participatory and engaging (for a review see Braunschweig, 2000).Recent modeling approaches with heterogeneous agents are also not participatory or engaging (e.g., Endresen et al., 2011;Groot et al., 2012 in Supplementary Table A1).Participatory varietal selection (PVS)-like approaches where alternative research options are validated by non-academic stakeholders (mostly farmers) are more participatory, but the engagement of stakeholders occurs only as an a-posteriori consultation with no feedback loop, where priorities from stakeholders are then validated jointly with scientists (e.g., Randolph et al., 2001;Pemsl 10.3389/fsufs.2023.1265109Frontiers in Sustainable Food Systems 03 frontiersin.orget al., 2022 in Supplementary Table A1).Crowdsourcing plant breeding methods (e.g., Steinke and van Etten, 2017 in Supplementary Table A1) engage stakeholders in an iterative manner with easy to implement approaches but they include primarily farmers.A rare example of a highly participatory and engaging framework was Blundo-Canto et al. (2020) in Supplementary Table A1, but also highly complex and abstract.
Methodologies currently available for ex-ante research priority setting encompass simple interactive scoring exercises (e.g., participatory ranking scenarios) to complex simulation models estimating the functional relationship between inputs and outputs (e.g., agent-based models; Supplementary Table A1).Allocating research efficiency and selecting the most promising research activities are issues directly tied to the scarcity of resources for plant breeding in development.Therefore, most existing frameworks place an emphasis on economic efficiency and on costs and benefits that can be expressed in monetary values (Braunschweig, 2000).The economic surplus analysis and the cost-benefit analysis are the tools most frequently utilized.Despite being easy to interpret, these two techniques present a few methodological disadvantages.First, they rarely include non-quantifiable and non-marketable outcomes (e.g., the shadow price of gender-related benefits).Second, in these tools, agricultural researchers provide most of the input, and active participation of stakeholders is quite limited.Third, these tools offer a static representation of the commodity market and thus tend to underestimate longitudinal non-linear dynamics which can affect the breeding process (Petsakos et al., 2018).For example, although some implementations allow for an explicit representation of dynamics in production and consumption (HarvestChoice, 1995), economic surplus models able to analyze well-structured foresight scenarios (like those proposed by the Intergovernmental Panel on Climate Change (IPCC) Davis et al., 1987), are frequently complex and data intensive.Lastly, this economic surplus paradigm has raised concerns because externalities, distributional effects, and longer-term impacts all tend to be neglected with a narrow focus on breeding costs and benefits (Dahlberg, 1988).

What's new? Novelty of PEEP
To develop PEEP, we took inspiration from the systematic, quantitative ex-ante priority assessment undertaken by the CGIAR Research Program on Roots, Tubers and Bananas (RTB) in the period 2012-2014(Pemsl et al., 2022)).PEEP preserves the systematic and quantitative nature of this assessment, while integrating attention to gender and adapting for utility for national level and crop-specific breeding projects.PEEP complements the literature on methods for ex-ante priority assessment in crop breeding through three methodological advancements.First, PEEP is adaptable at the national level, building on the existing work (Mills and Karanja, 1997;Randolph et al., 2001;Singh et al., 2020;Pemsl et al., 2022).Unlike these studies, PEEP leverages the national focus to better consider the relevance of gendered preferences and access to resources, which are context dependent.Second, PEEP is designed to be consultative in all phases, resembling the frameworks of Pemsl et al. (2022) and Steinke and Van Etten (2017).PEEP includes an array of stakeholders beyond scientists, so that crop improvement priorities are set and validated by social and natural scientists as well as practitioners.Here we define practitioners as local actors directly engaged in agricultural production, processing, and marketing and representatives from, e.g., seed companies, agricultural cooperatives, agricultural women's groups, and national or regional policy makers.PEEP is purposively combining multiple participatory tools to ensure that each category of stakeholders is involved through an approach which maximizes participation and engagement.Third, PEEP considers gender dynamics3 by seeking gender equity in the evaluation of alternative crop improvement priorities, and product concepts.To the best of our knowledge, no existing ex-ante framework thus far deliberately includes gender as a lens of analysis in prioritization.
Each of these three methodological innovations of PEEP has challenges.For example, PEEP requires a heterogenous group of stakeholders with deep contextual knowledge and local impact pathways.This may sometimes lead to conflict between national and local priorities.Establishing a process to reconcile these conflicts is a key component of the PEEP framework.Furthermore, it is particularly challenging to capture the views of the most vulnerable, but often least accessible, populations.PEEP includes nationally representative stakeholders who have an overview of the relative importance of different opportunities and barriers faced by these populations, but a risk remains that the needs of the most vulnerable respondents may still be missed.Finally, integrating a diversity of stakeholders and gender analysis requires additional expertise and creates a further level of complexity.Trade-offs hinge on who might lose or benefit from new varieties.
We leverage the use of heterodox methodological approaches and a strong emphasis on the engagement of knowledge-rich stakeholders to circumvent these challenges.PEEP ranks priorities based on a heterogenous set of benefits without quantifying impacts in terms of economic return (or net present value).Furthermore, in contrast with other scientific domains (e.g., health and medicine) where priority setting and ex-ante analysis are well-established practices and where engagement is actuated through the Delphi method (Linstone and Turoff, 1975), we utilize a less common iterative and tailored multistakeholder approach.Built on the principle of engaging technical as well as practice knowledge, PEEP adapts a diverse tool of engagement to each category of knowledge-rich stakeholders.This avoids multiple, highly resourced, and time intensive, rounds of consultation, especially where participants are not familiar with repetitive researchoriented routines.
How is PEEP any different from existing participatory plant breeding efforts?PEEP is interested in the "why" more than in the "what." Linking crop improvement impacts to product concepts, PEEP generates ex-ante impact pathways.These support breeders in justifying the impact of a future variety when the variety is still simply a product concept.Having clarity on which product concept targets the set of crop improvement priorities considered relevant by a heterogenous group of stakeholders assists breeders in (i) justifying the impact of new varieties under development; (ii) ensuring that product 10.3389/fsufs.2023.1265109 Frontiers in Sustainable Food Systems 04 frontiersin.orgconcepts target priorities and impacts are equitable and fair (iii) justifying the investment made by national and international funders; and, finally, assists breeders in (iv) better marketing of new varieties.

Description of the proposed method
PEEP has four sequential phases (Figure 1), detailed in this section.

Phase 1: creation of core implementing team
In the first phase, the breeding program implementing PEEP must form a core implementing team (hereafter core team).The core team oversees the process development, analyzes the results, and compiles the final assessment report.A set of competencies underlie the selection of individuals for the core team (Box 1) to ensure that the right balance of skills and experience are represented on the team are driving this process.Once assembled, the core team decides the geographical scope of the application (national vs. regional) as well as the target crop.PEEP can accommodate evaluations at the country level or regional level on any crop product concept to be evaluated.

Phase 2: selection of knowledge-rich stakeholders to form PEEP stakeholders' groups
Assembling a team of stakeholders to engage in the PEEP framework is the second -and possibly most critical -phase of the PEEP methodology.According to their degree of familiarity with the formal (academic) research process, stakeholders are assigned to two groups: (i) group R (Research), and (ii) group P (Practice).The Research group involves any member of a cross-functional breeding team who has technical knowledge of the target crop (see Box 2).Complementarily, the Practice group includes all stakeholders involved into the targeted crop value chain and possess practical knowledge (see Box 2).Engaging the R and P groups with transparency is important, clearly informing each group member about time and resources required to attend the process.For group R, incentives to participate in PEEP might be: (i) contribution to better define national or regional breeding objectives, (ii) the possibility to prioritize gender and climate in the national breeding agenda, (iii) better allocation of resources to crop breeding programs, (iv) possibility to conduct an evaluation which attracts international donors and funding agencies, and (v) networking and round table opportunity for new projects.For group P, incentives might be: (i) steering the breeding work to account for their needs and preferences, (ii) contribute to the development of varieties which are better suited for production and selling purposes, (iii) build social networks with researchers, (iv) tighter links with the local research community, which yield learning opportunities and higher engagement.
Selecting members for the Research group should include consideration of their expertise on the chosen crop, whether they are young researchers or senior leaders, as well as if they are regional or national collaborators working at in-country international centers.
For members of group P, choosing representatives at the national or regional level may be less straightforward.The core team needs to effectively sample a sub-population of producers, traders, processors, consumers, formal and informal agricultural cooperatives, and women's groups.Both at the national and regional level, NARs or CGIAR centers should utilize an informed stratified sampling strategy.If a list of crop producers, processors or traders for each area is available, we highly encourage the core team to use this information to calculate sampling weights 3.3.Phase 3: survey and choice experiment to identify crop improvement priorities

Survey
At the start of phase 3, relevant research priorities are elicited from group R through a large-scale expert survey conducted either online or in person.PEEP provides a generic structured questionnaire as guidance, but the core team adapts and tailors the survey to the crop of reference and specific national context so that it is most useful.Overall, the questions lead respondents to list and explain constraints related to breeding, economic, and gender, as well as climate issues.The questionnaire contains two sections: the first section includes open-ended questions common to all respondents, while the second section includes closed questions, tailored to each respondent's competencies.In the second section, respondents are asked to rate the importance of different research priorities using a five-point scale (from "not important" to "very important").Examples of research priorities are breeding for drought tolerance, improving seed storage, improving traditional processing techniques, developing new products for industrial application, reducing men's and women's health risks of on-farm insecticides use and many more.The first part of phase 3 closely resembles the second step of the RTB ex-ante framework described in Pemsl et al. (2022).
Research priorities are divided into thematic subsets.Each member of the group R will respond solely to the subset of priorities matching their domain of competence.This is done to avoid missing data or including biases in the ranking of research priorities.Data on the personal attributes of the respondents are collected in a third section of the survey.

Competencies for agricultural economists
Fundamental knowledge of agricultural marketing and economics Independently conduct of assists researchers in performing economic analysis on a variety of issues related to the agriculture sector Translates data into written reports/economic analysis, connecting results of analysis to actionable information Understands how language and culture shape meaning in socioeconomic data collection

Understands basic principles of sampling and controlled comparison in data collection
Has knowledge and experience of different ways to deliver products and services effectively to rural population, to define realistic goals and measurable impacts Has the skills to independently collect or supervise the collection of reliable social-disaggregated data Has knowledge and direct experience of ex-post impact assessments, and possibly ex-ante Has previously engaged actively with participatory varietal selection, or participatory plant breeding Knowledge of plant product concepts and market segments is a plus, but it is not required 10.3389/fsufs.2023.1265109 Frontiers in Sustainable Food Systems 06 frontiersin.org The following thematic subsets with their appropriate respondent categories are exemplified below:  2022), research priorities in the survey need to conform with the following criteria: (i) the research creates a global public good in the form of a new, adoptable product concept addressing a key constraint or targeting a PEEP opportunity for the crop of reference; (ii) impact would materialize within the 25-year assessment period; and (iii) the research scope is within the NAR or CGIAR center capacity and its mandate, prioritizing the needs of (smallholder) farmers and other vulnerable groups in the country.Furthermore, listed research priorities must be addressable in the next 5-10 years, given the technical and institutional capacity of the breeding programs involved.This helps the group to understand what research can be done, in addition to what is priority, and it gives a sense of possibility that can be acted on. 4Results from the survey are the first research output of PEEP and represent per se an interesting overview of how research priorities are listed and ranked among disciplines.
Overall scores within each thematic domain enable selection of research priorities.The first three 5 top-ranked research priorities in each thematic domain are selected to be included in the choice experiment with group P (see below), for a total of 12 possible research priorities.The domain of crop improvement is excluded because 4 This is analogous to research on traits.If a study presets the list of traits that is asking about, and this is based on traits that they can currently breed for, any priority information is immediately actionable.The utilization of open ended questions would deliver an array of information more detailed on trait preferences, but qualitative data are harder to act on immediately.This is a tension to be aware of.
5 We advise that the list is as complete as possible.However, as the number of combinations in the experiment grow exponentially with the objects to be evaluated (and there are computational limits to consider, to keep the framework as agile as possible), we highly recommend NARs/CGIAR centers having a maximum of 12 research priorities (3 top-ranked priorities for 4 different thematic domains, excluding crop improvement).
BOX 2 List of categories in group P and R

Group R (research)
Any member of a cross-functional breeding team, including, e. priorities in those areas are used to develop the product concepts. 6The selection of product concepts should consider the scope of the NAR or CGIAR center research activities to ensure a good match of assessed options with the program portfolio.In addition, the core team reviews the final list of research priorities to summarize if and how they are considerate of gendered preferences.
Once the list of crop improvement priorities is ranked by the group R, the core team will develop a list of crop improvement impacts which are directly linked and derived from the priorities selected.For example, if the research priority is "Development of new cowpea varieties which are drought resistant, " the corresponding impact will be "Help you deal better with drought while cultivating cowpea." This step ensures that priorities are intelligible for actors of group P and are seen as actionable.

Choice experiment
Once the list of rank crop improvement impacts is assembled, the group P is formally engaged into the PEEP framework.Group P is involved through a crowdsourcing method which closely mimics gamified choice experiments.Gamified choice experiments follow a strict and replicable guideline and provide quantitative results, which can be compared with previous evaluations and the results of alternative methods for priority setting (for an application in the domain of priority setting see the paper of Steinke and Van Etten, 2017).
The choices presented to respondents consist of a set of three product concepts, starting from available end-users' preferences on crop traits and existing product profiles.Here we draw on recent approaches of concept testing to explore motivations behind the choice of one variety over another (Rutsaert et al., 2022).Concept testing entails showing a new product idea through a description or visual material, with the goal to obtain feedback and eventual interest in purchasing the variety.Product concepts are brief narrative descriptions, easy to interpret and present.They describe the morphological characteristics of the seed and plant variety, the main grower requirement that the variety addresses and conclude with an additional list of standardized information (e.g., yield potential, fertilizer needs, maturity, grain usage).Beyond containing the narrative for the product concept, the script includes suggestions on posture, voice tone and other non-verbal instructions to ensure a clear understanding of the product concepts by the stakeholders (Rutsaert et al., 2022).Product concepts are developed by the core team based on existing product profiles, and crop improvement priorities from the survey with group R.
Each product concept is then presented along with the research impacts derived from the survey with group R (see Figure 2).Respondents will be asked "Which of these three new varieties will help you the most to achieve impact A? Which will help you the least?. " The three new varieties, presented in the form of product concepts, will be assessed against all 12 impacts (Figure 3 provides an example).Respondents will have the opportunity to declare that the impact is linked with none of the three product concepts proposed (exit strategy).
6 Examples of research priorities in the domain of crop improvement are "breeding for high yield," "breeding for biotic stresses," "improving soil fertility," etc.
Enumerators will also present research impacts in a random order to each respondent, to ensure that not always the same impacts are presented at last when survey fatigue is at the highest peak (Figure 3).
Once the choice experiment is concluded, the core team utilizes well-established ranking models, especially the Plackett-Luce model, to analyze the data.Calculating the log-worth of each research impact, the core team is able identify which is the product concept with the highest probability of being associated with each impact by the category of group P interviewed.

Phase 4: validation and feedback
The main result of phase 3 is the prioritization of impacts from crop improvement impacts and how they map to a set of given product concepts.This helps breeders to align their prioritization of product concepts to meet impacts.During this phase, breeders, geneticists, agronomists, pathologists, and other members of the breeding team are responsible for integrating crop improvement impacts.Concretely this means answering two main questions: are existing product concepts able to address the impacts?If not, what information is missing?
A tension might emerge from this validation: on the one hand, the breeding team needs to work with product concepts that are realistic given the technical and institutional capabilities of the NAR or the CGIAR center.On the other hand, the team needs to ensure that previous efforts do not get lost in the name of "breedability" of alternatives.
We encourage the ex-ante team to repeat the analysis at every new breeding cycle.If the team perceives that a change in priority might have occurred earlier-due to a new agricultural policy, a climate event, or an expected shock like the Covid-19 pandemic-the ex-ante exercise can be repeated prior to the closing of the breeding cycle.

Discussion
We started this methodological work out of necessity.When working with NARS breeding programs to provide support on how to systematically, and most importantly, inclusively set crop breeding priorities, we were unable to identify methods to do so.When we scoured the literature on methods to set crop breeding priorities ex-ante, holistically and inclusively we identified a gap that this methodological work has attempted to fill.We outline in this paper PEEP, a method that provides an answer to a key question for breeding teams: when intending to achieve a specific impact, which product concept should be prioritized for development?To our knowledge there is no other method or approach that seeks to directly engage farmers, and other end users and beneficiaries of crop varieties, to link impacts that are important to them, with options for breeding programs to work on (Brown et al., 2020).In this sense PEEP creates the opportunity for a broader range of stakeholders to "have a say" in crop breeding priority setting.This consultative, if not participatory, underpinning to the work is timely and an important contribution to the field at a time when demand-led breeding is taking center stage.Just as participatory plant breeding can be described as highly client-oriented breeding (Witcombe et al., 2005), we see PEEP as highly client-oriented crop breeding priority Creating meaningful choices of crop varieties can be understood to be a form of empowerment for marginalized farmers (Polar et al., 2021).Using this framing, we place importance on engaging a broad range of socially heterogenous farmers, with emphasis on engaging marginalized women and young farmers in design and implementation of PEEP.This allows the framework to conform with the minimum standard of sex-disaggregation in data collection needed to conduct gender analysis (Doss and Kieran, 2014) and can be adjusted to add layers of socioeconomic data to further enhance analysis.In doing so, PEEP allows breeding programs to observe if targeted impacts are prioritized equally by different actors involved, or if a socially distinct sub-group (e.g., illiterate women or men widows or widowers, single parents) would give different weights to alternative impacts.
Departing from cost-benefit and investment considerations, PEEP leverages methodological approaches typical of heterodox economic disciplines.This entails relaxing assumptions around economic efficiency in favor of non-quantifiable and non-marketable outcomes, produced by rankings elicited from technical and non-technical actors endowed with different types of knowledge and power.In doing so, PEEP contributes to expand the field of priority setting beyond ex-ante investments allocations to include considerations on externalities, distributional effects, and longer-term impacts.
In building a flexible and versatile ex-ante priority setting framework, we aim to demonstrate that national and international research organizations would benefit from systematic and integrated priority assessment cycles that are repeated and constantly adjusted over time, with deliberate learning incorporated into each loop.This will strengthen frameworks and processes, contribute to institutional memory and capacity building, and increase relevance of priority setting for decision making while reducing its costs.
These insights are crucial for the public sector breeding for development, that distinguishes itself by explicitly focusing on social  inclusion outcomes, such as gender equality, poverty alleviation and food security as laid out in the sustainable development goals.We expect the framework to guide public crop breeding institutions, such as national agricultural research centers and CGIAR centers.
Complementarily, the framework has the potential to be appealing and benefit private breeding programs too.Understanding clients' breeding priorities and expected impacts help private programs to create better products, with higher adoption, profits and return on investment (Ragot et al., 2018).
The framework has limitations.First, it does not account explicitly for trade-offs among crop improvement priorities and impacts.Tradeoffs hinge on who might lose or benefit from the breeding process.For example, reduction in yield loss might matter to a farmer but market share by women purchasing a lower-yielding but lower-labor or higher-nutrient crop might matter more to a seed company if it increases sales among women.In aggregating results from the choice experiment, trade-offs and win-lose become less visible to the breeding team.Second, the framework does not plan for a validation step between the survey and the choice experiment: the group P can choose the best and worst combination between impacts and concepts, but, in the current version of the framework, they can neither expand nor modify the set of impacts on which to perform the choice.Piloting the framework will help to mitigate these pitfalls, while possibly highlighting others.

Conclusion
There is an increasing need for systematic priority setting to guide resource allocation in international public agricultural research.Effective research prioritization in crop breeding requires an ex-ante evaluation of program activities.The PEEP priority setting ex-ante framework proposes an interdisciplinary, multi-stakeholder and gender-intentional approach to rank crop improvement priorities, impacts and product concepts to perform an ex-ante breeding assessment.While methods and tools within PEEP are not new, taking them to scale and incorporating multiple objectives by analyzing gender, and national/regional benefit allocation represents a substantial advancement over previous efforts.We also experiment with ambitious targeting and stakeholder engagement processes, which help to ground truth the selection of research options, resulting in a high level of stakeholder awareness, and yielding potentially important lessons learned.The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
FIGURE 1Map of the four sequential phases of PEEP.

BOX 1
Set of competencies for the core team Competencies common to all core team members Values diversity of perspectives and experience Seeks representation of social science in research teams and fosters interdisciplinary dialogue Builds a supportive culture within the working group Values academic as well as practical knowledge and fosters the exchange of knowledge among experts and stakeholders Bridges research and development practice Is open to continuous improvement as a method for improving the research process and its effectiveness Competencies for breeders, geneticists, pathologists, entomologists, and other members of the breeding team Demonstrates scientific rigor in the breeding subject of competence Contributes to breeding scheme design Contributes actively to developing or improving crop product concepts for the chosen crop in country Engaged with participatory varietal selection, or participatory plant breeding more broadly Engaged in the breeding team of the organization and contributes to breeding targets set by the organization Competencies for gender specialists Proven foundational gender analysis competencies Conducts high quality gender research Interprets and communicates the implications of gender relations as well as gender differences to a multi-disciplinary team at different stages in the research cycle to help the team identify constraints, opportunities, priorities, research outcomes, impacts that need to consider gender Demonstrate ability to propose and lead a scientific research project addressing social and gender issues Produces research on social and gender issues suitable for publication Applies advanced social science concepts and knows how to deepen analysis beyond simple sex-disaggregated comparisons to define implications or outcomes of gender inequality Makes skillful use of advanced social research design, data collection and analysis techniques to conduct research on strategic gender issues Leads and champions greater understanding of the relevance of gender to agricultural research Competencies for climate specialists Works with climate predictions and meteorological data Engaged in the use of meteorological data for agriculture Proposes and conducts research on climate and agriculture , entomologists, and other members of the breeding team* Agricultural economists (with expertise in local markets and prices, formal and informal seed systems)* Gender specialists* Extensionists, including some with focus on gender Nutrition specialists (if available) Climate experts (from National Meteorological Office)* Food scientists / Food processing experts Mechanization experts State or regional policymakers with mandate on agriculture National policymakers with mandate on agriculture Non-profit organizations involved in agriculture, development, and gender Donors -in country and regional missions Group P (practice) Producers (social heterogeneity represented, with equal voice to marginalized producersagricultural cooperatives and organizations Representatives from informal and formal agricultural networks Representatives from women's agricultural groups* Agro-input dealers Regulatory bodies for certifying seeds and GM production Consumers' organizations -including, among others, representatives from women's entrepreneurs' groups and representatives from women's advocate groups *Non-negotiable members of each group to ensure representative and actionable results. is therefore closer to the Blundo-Canto et al. (2020) ex-ante prioritization framework.

FIGURE 3
FIGURE 3Example of the choice experiment.
FIGURE 2Stylized representation of the choice experiment.
Data on the personal attributes of the respondents are collected in the third section of the survey.Following the strategy in Pemsl et al. (