Tracing the Impact of Proposals from Participatory Processes: Methodological Challenges and Substantive Lessons

Our understanding of participatory processes is increasing rapidly. However, one area that has received sparse attention is the impact of the proposals from participatory processes on the policy and practice of public administrations. Which proposals are converted into actual policy and practice; which are modified or simply ignored? The field lacks a systematic understanding of the fate of proposals. This paper reflects on the methodological strategy adopted by the Cherry-picking project to analyze the fate of proposals from participatory processes in Spanish municipalities. The innovative project studied the impact of 611 proposals from 39 participatory processes across 25 municipalities. The paper not only describes and discusses the methodological challenges faced by the project, but also presents preliminary findings and a review of the substantive lessons learned through the design and fieldwork process.

Do proposals that emerge from participatory processes translate into actual policies?While there is increasing academic interest in the use of participatory processesinstitutions specifically designed to increase and deepen civic participation in political decision-making (Smith 2009; Warren 2009)by public authorities, little systematic attention has been given to this fundamental question.Previous research has covered many different aspects of participatory engagement, for example: who participates and their experience of the process; how participation is organized; and the broader context that supports or inhibits participation.But there is a paucity of research devoted to the policy effects of participation.When such effects are considered, the existing literature tends to focus on whether the outcomes of such processes have broader societal impacts (Boulding & Wampler, 2009; Olken, 2010; Touchton & Wampler, 2014).This is of crucial normative concern, but tends to overlook the intermediate step that has become a 'black box' in studies of participatory processes, namely: What happens to the proposals that emerge from these processes?Why are some implemented and others not?Is it a question of chance or are there systematic factors that explain the fate of some types of proposals?The suspicion that a degree of 'cherry-picking' among proposals takes place is present in the academic literature and amongst practitioners (Smith, 2009, p. 93), but we have no clear indication of the determining factors.
A systematic analysis of the factors that explain adoption and implementation by public authorities of proposals that emerge from participatory processes has not been undertaken.One of the reasons is because for meaningful generalizations it requires a large-N quantitative strategy that analyses the factors that affect the fate of proposals from across a variety of different participatory processes.This kind of dataset does not currently exist, and arguably goes against the methodological tendencies within research on participatory governance.While research on deliberative and participatory practices has become methodologically more pluralistic (Font et al., 2012), the in-depth case studies approach that was characteristic of the early years in the field (Abers, 1998; Fung, 2001) remains somewhat dominant.It is widely recognized that while case studies have been fundamental in the development of this area of study, the typical choice of exemplary cases (e.g.Participatory Budgeting in Porto Alegre; the British Columbia Citizens' Assembly) has introduced significant bias in our understanding of these practices (Font et al., 2012; Smith et al., 2015).There has been a broadening of methodological ambition, from comparisons based in a medium number of cases, typically of the same type of participatory process (Baiocchi, Heller & Silva, 2011; Fournier et al., 2011), to the application of experimental designs (Grönlund et al., 2010) and fuzzy-set qualitative comparative analysis (Ryan & Smith, 2012).Large-N studies remain relatively rare, even more so where the aim is to assess the fate of proposals. 1is essay reports and reflects on the innovative methodological approach taken to develop the first large-N study of the fate of proposals from participatory processes undertaken by the Cherry-picking project team. 2 The team has collected data on 611 proposals (the unit of analysis) that emerged from 39 participatory processes organized by Spanish local authorities.Since most of the country's participatory activity has developed at the local level, we limited our research to this level of governance to reduce the range of contextual variability.The goal is not to explain the net policy impact of participation, but to track the fate of participatory proposals.Thus, we do not claim that policy adoption is the result of participatory inputs, but that proposals from participatory processes have different policy effects, and that these effects are not randomly distributed, but the result of a set of proposal and context related factors.
Our starting point was two already-existing datasets that included several hundred participatory processes enacted in three Spanish regions.From all these participatory processes, we selected a diverse sample and, for each institution, a sample of proposals.We then tracked the fate of each proposal to discover whether they had been implemented and if so, whether in the process of implementation original proposals had been modified.In other words, what is the evidence of cherry-picking and under what conditions does it take place?An innovative research project like this is not only valuable for the data it produces and the analysis that follows, but also represents an important opportunity to reflect on the challenges associated with collecting data on proposals and lessons that are learned about how the participatory policy process operates.It is this methodological learning from the data collection process that is the main focus of this paper.The details of the institutional black-box within which some proposals from participatory processes evolve into real policies while others disappear without trace has been seldom analysed, and never at this scale.The Cherrypicking project therefore offers an opportunity to reflect on the nature of the methodological challenges we faced in extracting meaningful information on variables that potentially shape the fate of proposals.As evidence that this methodological strategy has been effective, we describe briefly some preliminary results of the project.
The article is structured in two main parts.The first is devoted to explaining the methodological challenges and choices adopted.We describe the methodological design of the research project, the selection criteria for sampling both participatory processes and proposals (our unit of analysis), as well as the fieldwork strategy and some significant variables.The second section reviews different outcomes of the project.We first present general characteristics of the data and assess their quality as regards final sampling composition and reliability.Then we present a few preliminary findings and review substantive lessons learned through the design and fieldwork process.

General research design and scope of research
Types of proposals are likely to vary across different participatory processes and political contexts.As such the methodological strategy is based on a diverse selection of participatory processes.Given that no country holds official records across all participatory processes (Smith et al., 2015), it is impossible to know the complete population of participatory processes: as a result, those selected will not be a perfectly representative picture of all participatory activities, but can at least cover a broad range of institutionalised forms of participation (Font et al., 2014).
Previous research points to two basic factors that are likely to shape the fate of proposals resulting from a participatory process: contextual and policy-related factors.3Among the contextual factors, some relate to the characteristics of the local context; others to the design of the participatory process from where the proposals arise.As such the project faced its first challenge: to ensure variation at three levels (local context, participatory process and proposals).This variation is critical, since much of the existing literature displays variation at only one of the levels, examining sets of proposals emerging from a small set of fairly homogeneous participatory processes (Barrett et al., 2012; Fournier et al., 2011;  Olken, 2010).
Simultaneously, we wanted a controlled amount of contextual variation, since extremely diverse levels of socioeconomic development and very large differences in political and administrative rules and routines could create a scenario where alternative explanations would be impossible to control.Balancing these two concernsensuring diversity of institutions, but with a degree of control of contextual variationour decision has been to limit our selection to a single national context and a single layer of governance with a constant legal and administrative environment.For reasons of prior knowledge, the availability of existing datasets and access for interviews, our population of institutions is participatory processes established by Spanish local governments.We introduced contextual variation through the selection of diverse regions: Andalucía, Catalonia and Madrid. 4 also selected a specific time frame, from one local election (2007) to the next (2011), combining enough time for at least initial implementation of proposals (a minimum of three years from the generation of the proposal to our fieldwork) with the possibility that memories and administrative records are recent enough to be tracked.

Population and selection criteriawhich participatory processes to include?
The population for our research is the participatory processes developed by municipalities in Andalucía, Catalonia and Madrid during the period 2007-2011 that end up with specific proposals. 5Our final units of analysis, however, are not the institutions themselves, but the proposals resulting from those processes.Since it is a reasonable working hypothesis of the research that different proposals emerging from the same participatory process are treated differently by local government, we needed to follow the evolution of each (or a sample of each) proposal.
To construct the sampling frame for participatory processes, we drew on two existing datasets on the activities of subnational governments in Spain that had been created for an earlier research project (Font, della Porta, & Sintomer, 2014).The first dataset is comparative in nature, with data on processes in the three regions collected by web content mining (N = 292).The second dataset was collected in Andalusia only with a double survey strategy: an on-line questionnaire addressed to municipalities (CASI) and a follow-up (CATI) for those municipalities that had not answered our first online approach (N = 517).The two datasets generated a different picture of participatory processes in Andalusia, the main difference being that data mining over-represented processes developed in large cities (Galais et al., 2012).In recognition of this difference and to guarantee the presence of smaller municipalities (up to 20,000 inhabitants), cases have been selected from both datasets. 6fore selecting the cases of participatory processes and in order to adjust these two datasets to the scope of our research, we undertook the following data cleaning operations: 1. Elimination of non-eligible cases that were out of the temporal or territorial scope of our research. 72. Elimination of cases lacking relevant information (for instance the name or a minimal description of the process).3. Elimination of cases that would not end in proposals. 84. Since the Andalusian CASI/CATI database was included to cover the experience of smaller municipalities, in this dataset we only considered municipalities with less than 20,000 inhabitants.
As a result of these operations we had two datasets, with 214 cases of participatory processes (the comparative web-mining datafile) and 187 cases (the CASI-CATI survey datafile) that served as the starting point of the sample selection process.
Our aim was to achieve a sample of 40 participatory processes from which the fate of policy processes could be assessed.This number was chosen to ensure representation of a diversity of processes as well as a significant number of proposals that could then be subject to statistical analysis.With this aim in mind, we adopted a stratified sampling design, thus ensuring representation of potentially important independent variables: (1) region where the process has taken place, together with municipality size in Andalusia to include processes 6 For almost all of the contextual variables used in this project both initial datasets contained exactly the same information.For a few, equivalences were created (Font, della Porta, and  Sintomer, 2014, Appendix). 7108 processes were out of the temporal scope of this research (in most cases, developed before 2007) and 28 processes were out of scope because they had been developed by supra-local administrations.
8 For the web-mining comparative dataset we have checked the information about proposals available on the internet.Of the 236 processes, 214 have proposals (certainly or expected).With the CASI/CATI dataset we relied on responses given by the municipal officer and considered that a participatory process will almost certainly have proposals if the process is linked to formulation or decision stages of the policy process.These operations allowed the exclusion of many processes that almost certainly had no proposals.We kept the remaining cases, but they were excluded from the final selection if the fieldwork showed that they had no proposals (see table 6 below).
from smaller municipalities; (2) extent of experience of organizing participatory practices; and (3) participatory process design.The sample of participatory processes has been randomly selected from each of these strata.
1. Region / Municipality Size (in Andalusia): 10 participatory processes selected from the web-content mining dataset for each of the three regions, plus 10 additional cases from Andalusian municipalities under 20,000 inhabitants (CASI/CATI dataset).

Extent of Experience in Participatory Practice:
To ensure a reflection of diversity in experience in organizing local participatory processes, we used the number of participatory processes organized by the municipality as the next stratum.In each region we aimed to include two municipalities with three or more processes (as examples of more experienced localities), selecting three processes for each one.This allows us to compare also how different processes have performed in the same municipalities.In Catalonia there were only two municipalities with three or more processes, so in this region we considered experienced municipalities as those that have developed two or more participatory processes. 9The remaining four processes in each region have been selected from less experienced municipalities (having developed less than three processes or less than two in Catalonia).

Process Design:
We have distinguished four broad types of participatory process that capture the diversity of participatory design, in each region selecting at least two processes of each type.10 -Participatory budgeting -Strategic planning 11 -Policy councils and other permanent mechanisms -Other temporary processes Whenever choice was possible after applying the stratification criteria, 12 the final selection of participatory processes was achieved through random selection.The combination of these criteria resulted in the theoretic sample distribution shown in Table 1.In order to reach the highest possible response rate among the initially selected participatory processes we adopted a strict substitution policy.Initially sampled processes were substituted by similar ones only in two cases: either when the fieldwork showed that, contrary to our initial data and expectations, they were not eligible (processes that did not produce proposals, out of temporal scope, etc.) or when it became clear that there was not enough cooperation to collect most of the information we were interested in (refusals).In order to avoid introducing bias toward the best-documented processes, we have not substituted those cases where there was substantial missing information.In other words, lack of relevant information on policy proposal implementation has not been a reason for substitution.
The final sample is constituted by 39 participatory processes since for one of the selected municipalities with more than three processes in Andalusia, there were no records on one process.13

Documenting and selecting proposalswhich proposals to follow?
The next step was to trace any documents that listed proposals derived from each of the selected participatory processes.For some cases, this required cooperation with local officials as relevant documentation was not publicly available online.
In itself this was a challenging exercise as there is no standard information source for proposals (Table 2).In some cases, this step was fairly straightforward as there was a clearly identifiable document that represented the final outcome of the participatory process and provided a full list of the proposals.However, in other cases we uncovered more than one document, as a result, for example, of the use of different participatory procedures within the process or the same procedure applied to different groups of participants.In these ambiguous situations, we selected the information source that appeared to be closest to a final listing.In one case, we had to resort to interviews with relevant local actors to reconstruct the final list.Finally, when the process ended without producing any definitive list of specific proposals and it was not possible to reconstruct such a list during the fieldwork, we abandoned the process and selected a substitute.Certain processes had more than two hundred proposals (Table 3): following their fate would place impossible demands on the research team as each proposal requires at least two local interviews to verify data.A balance was struck between diversity of proposals and available resources by limiting the number of proposals for which information would be collected to 20 for each participatory process.
The selection of proposals was made through systematic random sampling. 14hen the total number of proposals emerging from a single process was less than 20, all were selected.The research team has applied a logic of triangulation of the data sources (Denzin,  1978) in order to guarantee the quality and reliability of the information gathered.
To this end, the initial information gathering process involved accessing a variety of sources, including official documents on the participatory process, publicly 14 Systematic sampling offered the advantage of respecting to a greater extent the structure of the listings of proposals, assuring a better representation for the different groups of proposals established as a consequence of the order followed in the documentation of the process (e.g. by thematic areas).For those cases where the proposals were recorded in different independent documents we determined the number of proposals to be selected from each document by way of proportional allocation.
available or not, media reports, personal blogs of participants and even audio recordings of interviews with participants in the initiative.
To answer our research objectives, we needed information about our dependent variable-to what extent a proposal has been adopted (see below for more details)-and quite diverse data that capture the potential explanatory factors, as well as some additional control variables.Since there has been relatively little research undertaken in this area and there are numerous possible impacts on the fate of proposals, information on over 100 variables was collected across the three different levels of analysis: municipality, participatory process and policy proposal, the same three levels we have used to select our final sample.Most of the information on the first two levels was already available in the previously existing datasets or in other publically available sources (e.g.municipal budget information, official electoral statistics).In other cases, particularly with regard to policy proposal level variables, interviews with municipal officers and other stakeholders were necessary.
The data collection was designed as a sequential process with the aim of collecting as much information from secondary sources as possible, before proceeding with the most costly step of face-to-face interviewing.Once a participatory process was selected, the first step was to get as much information as possible from the municipal web pages and other online sources.The main goals of this first step were to become familiar with the participatory process and to understand how much we needed the municipality's cooperation to be able to include the case.Ideally, and this has been the situation for most of the participatory processes included in our sample, we wanted to have the list of proposals at the end of this step.
The second step involved making an initial contact with the municipality, seeking to obtain as many official documents and records on the process and its proposals.This second step served to narrow our information requirements for the face-toface interviews and to identify the appropriate informants.
The interviews started with local officers and continued with other informants from civil society or the local political world.Usually, the first contact was made with the municipal officer who had been in charge of the participatory process.This person then typically provided access to the officers who had oversight of the relevant areas of policy: those affected by the proposals that had been selected.On occasion it was the same officer who had information on the fate of most of the proposals.Subsequent interviews with local governing and opposition politicians, civil society groups and external experts enabled completion and checking of information provided by the local officers.Table 4 shows the profiles and areas of expertise of the key contacts and the informants interviewed during the fieldwork process.
In total, 162 semi-structured interviews were held with 181 informants (a small number were held in groups).Three or more interviews were conducted in two thirds of the participatory processes in the study (26 out of 39) with a mean of 4.6 informants and 4.2 interviews per process.The fieldwork team consisted of three doctoral students and fieldwork lasted approximately six months. 15

Fieldwork instrumentshow to record observations?
The codebook 16 includes the coding procedures for the quantitative information collected by the research team for both the dependent and independent variables.The codebook includes about 100 variables across the three levels of analysis: 15 In almost all cases, interviewees were obliging, sometimes impressively so when the same person was required to provide information on 20 different proposals.Most data collected about the proposals stem from the information provided in the interviews with local officers and other key actors within the municipality.Relying so heavily on information provided by politicians and local officials again risks introducing bias: a more positive picture of the policy impacts of the proposals from participatory processes.This risk certainly exists, although was mitigated by balancing the accounts of officials with those of opposition politicians, civil society activists and external experts. 16The final version of the codebook is available at https://cherrypickingproject.wordpress.com/project/codebook/.
 Polity factors: such as size of municipality, participatory tradition, and ideology of the party in government.
 Process design factors: such as type of actors involved in the process, type of facilitation, resources, and participatory techniques employed.
 Policy related factors: such as the policy area of the proposal, boundaries of political competence for the issue, degree of support the proposal has within government and civil society.
The first draft of this codebook was built from the operationalization of research hypotheses established from a review of the theoretical and empirical evidence on the literature on the adoption of proposals arising from participatory processes.17 The forms designed to ease the process of data gathering and recording included open fields for each variable to enable comments, including doubts about data quality.The information in these fields was particularly useful for the development of a reliability index for the variables at the conclusion of the fieldwork (see below).
In addition, the fieldwork team produced a fieldwork journal for each participatory process. 18These documents detail the different steps followed in the information retrieval process for each participatory process, the links to relevant webpages and documents, a contact registry (date, names, function, contact mode and type of information retrieved), problems encountered and any operational decisions taken in the field.These fieldwork notes, together with the qualitative information registered in the data collection forms and the interview recordings allowed the research team to reconstruct information at the final coding stage where we needed to adjust or complete coding schemes in light of changes to the data collection protocol.
The variety of sources accessed and used to retrieve the information as well as the varied quality of records, willingness to cooperate and other case specific factors meant that there were important differences in both the depth and quality in the information collected for each proposal.Some of the information collected was based on official records; at other times on more subjective personal assessments.
To account for these differences, the data includes a set of variables assessing the reliability of the information recorded for the main variables in the codebook according to the quality of the information source (written source or oral report; number of sources; and/or mastery of the key informant) and the degree of agreement or disagreement among different sources.The categorization of reliability is a four-point scale from 'No reason to have doubts' to 'Maximum uncertainty'.A synthetic index has then been created from these specific variablelevel reliability data to assess the overall quality of results for each policy proposal and the cases with maximum uncertainty have been removed from analyses, resulting in 571 usable cases (see table 8

The main variables
There are three main implementation related variables we were interested to measure: implementation, changes between proposal and implementation and public explanations of these changes. 19The existence of changes and public explanations were less problematic and were captured through a dichotomous variable (yes/no).
Measurement of implementation was more problematic: how should we measure when a proposal had been implemented?We initially used a policy stages heuristic popular in textbooks: register how far the proposal had travelled, from purely symbolic approval, to appearing in a policy document, to the proposal being adopted in the practices of the authority.However, the fieldwork exposed how challenging it is to accurately capture the extent of implementation.To begin with, we had to include one additional variable in order to be able to distinguish the highest level of implementation attained and the outcome when the fieldwork was carried out.This distinction was necessary to accommodate a variety of situations that do not fit with the deterministic view of policy implementation as a linear and progressive process.There were, for example, proposals that required continuous implementation.They had been implemented for a couple of years (full implementation in the linear policy cycle variable) and then abandoned in the following years (not implemented at the point when fieldwork was carried variable) (n=29).Other proposals had been approved and included in the programme of work of a particular department, forgotten for a long time, but were in the process of being retrieved (n=14).Finally, there were others that had been so significantly modified that it was arguable that there was any resemblance to the original proposal (n=41).The description of this extremely rich and diverse reality may be the subject of a more extensive treatment at some point, but to develop a manageable quantitative analysis we simplified this complexity into three broad categories: no implementation, partial implementation (including this wide range of intermediate and complex outcomes just described) or full implementation, with each of these categories capturing a quite diverse range of impacts.It is thus necessary to keep in mind that this pragmatic resolution of diverse impacts into the simplified categories that statistical analysis requires, hides a much more complex and messy reality of policy implementation.
Most of the independent municipality or process level variables were reasonably factual and involved less problematic measurements.This was also the case with some of the proposal level variables (for example, number of departments involved in implementation).However, many were often more difficult to measure, generating two crucial problems.First, there was a lack of available information.For example, our goal was to collect exact cost estimates for each of the proposals.However, for many of them (especially those not implemented, but also for some that were implemented), these figures were not available.To solve this problem we created a proxy estimate with four main categories, for which we could capture information for 85 per cent of the cases (see table 5).Second, for some variables we had to rely on the subjective assessments made by a very diverse set of interviewees.This was the case, for example, with the challenging character of the proposals, an idea which was central in the initial hypotheses but which was difficult to capture precisely.We could only make a comparable assessment through the comparison of different perspectives.
Interviewees were asked whether the proposal represented a significant change or continuation from existing policy and practice of the local authority.Comparisons across interviewees generated a plural subjective proxy for the challenging character of the proposal.

Data: general characteristics and quality indicators
Between January and March 2014, the sample was selected and a pre-test of the study was conducted for one of the municipalities included in the sample.The fieldwork and the coding extended from April to December 2014.After the fieldwork stage, a full review of the data gathered was conducted in order to refine and clean up potential errors as well as ensuring that criteria for the coding of cases had been used similarly in all cases.
The main output of this data collection process is an innovative quantitative dataset that accounts for 611 proposals.Table 6 shows the final sample composition taking into account the main variables used as strata in the sample selection.Table 7 shows the total number of participatory processes included in our sample at any one moment of the research process.From the 56 processes considered, a little less than one third were excluded from the sample either due to their ineligibility for inclusion in the research (the process was not completed, it was out of the time frame of the study, or it did not have proposals) or due to lack of cooperation from municipality officials. 20This means that we reached an excellent response rate of 81.3%. 21In sum, 80% of the participatory processes included in the final sample were in the initial selection; this has contributed to minimum deviations of the final obtained sample as compared with the designed sample.Beyond the basic quality of the sample, coding of the reliability of data sources allows for an assessment of the quality of the data gathered for each of the units of analysis (proposals from participatory processes).Table 8 shows the distribution of the 611 proposals included in the data file according to the degree of reliability of the source and the number of interviews that, on average, were conducted to retrieve the information of proposals in each category. 22In three out of four proposals, reliability of information is high because it is derived either from written documentation in the process, or from interviews with different informants with knowledge and understanding of the process where there was no contradiction or doubt about their evidence.The proposals with a maximum degree of uncertainty are few (5.6 percent) and widely dispersed across the sample of participatory processes.In these cases there were either a lower number of interviews or the interviews generated contradictory assessments.Thus, after filtering the 40 proposals with low or unknown reliability, the dataset is still constituted by an impressive 571 proposals produced by 39 participatory processes developed for the 25 municipalities.

Preliminary results and lessons
Analysis of our dataset indicates that the degree of implementation of proposals is more optimistic than the results of previous research suggest (Lowndes et al.,  2001; Mazeaud et al., 2012).More than a third of the proposals are fully implemented without major changes, almost another third fit into the category of partial implementation and the rest of the proposals were not implemented, at least at time of fieldwork (Figure 2).Results are less positive with regard to accountability.When proposals are not implemented and when they are subject to significant changes, the most common outcome is that public explanations are not provided.

Figure 2 Main results: dependent variables
In spite of this relatively high level of implementation, our results show that there is plenty of room for cherry-picking.First, most participatory processes end up with a considerably high number of proposals implemented in some form (Table 9).Second, and most important, the processes where all the proposals have been fully implemented are few and all but one of them are processes that produced a very short list of proposals.The reverse is also true: no process in our sample complete rejection of its proposals (100 percent lack of implementation) and only three had no proposals that were fully implemented.In most cases, proposals emerging from the same process experience different fates, with some fully implemented, others modified and some abandoned.
The initial hypothesis that contextual level variables would be important in understanding these different fates gains credibility when we observe the distribution of the dependent variable by type of participatory process.For example, the type of participatory mechanism appears to be related to the degree of implementation of a proposal (χ 2 (6, N=604) = 52.11,p< 0.000): the odds that a proposal emerging from a participatory budget or other permanent mechanisms (e.g.citizen councils) is fully implemented double those of proposals coming out from a case of strategic planning or other temporary processes.Inversely, the average degree of rejection in strategic planning reaches 35 percent, but remains at a more moderate 24 percent in participatory budgeting processes, as well as among the other temporary processes category.If proposals from the same process have very different fates, it is either a random occurrence or we need to consider also proposal level variables to understand the pattern.23Finally, there are at least two significant challenges faced during the fieldwork that resulted in interesting insights about the nature of participatory policy processes.The first relates to sampling and the second to identifying the final list of proposals of a participatory process.
Life is full of trade-offs.Researchers interested in analysing participatory processes face several when selecting cases for study.Unfortunately, there is a high correlation between how well participatory processes are documented and the broader participatory qualities realized by these processes.This is also likely to correlate with how cooperative officials are towards intrusive external researchers.If we want to make life easier, we would always choose high quality processes, where we find easy access to well written reports and to friendly interviewees. 24On the other hand, from a rigorous social scientific perspective, the problems with generalizing findings if we choose the path of least resistance are clear.
Even though we were fully aware of the problem of selecting only good practice examples of participatory processes from the beginning of our study, there are at least two potential sources of bias that remain in our dataset.The first is the reliance on web content mining for one of the original datasets (the comparative dataset for Andalusia, Madrid and Catalonia).Those participatory processes that are deemed a failure or are poorly resourced are less likely to find their way onto web pages.Similarly for the CASI and CATI dataset, those authorities that have organized poor participation exercises may have reason not to respond to the surveys or to report them.As a result, some of the worst participatory processes would be missing from our initial sampling frame.That said, the datasets that were produced by these processes are arguably closer to the reality of actual practice than any other existing dataset of this type.
When it came to data collection, the extreme variability in how well documented processes are in practice became more and more evident as the fieldwork progressed.In all cases (even those where detailed documentation was available online), we needed to undertake interviews to collect relevant process and proposal level data.Most of the municipalities contacted have been very helpful, but attaining agreement to cooperate has been often a very costly and timeconsuming process, involving negotiating numerous gatekeepers and long delays.
For the majority of the participatory processes coded as a refusal to participate (only nine in four municipalities), there was no explicit refusal, just a repeated lack of response to our requests for relevant information. 25We were strict in keeping the processes that had been selected randomly within the study, but the trade-off is a real one: how many phone calls and weeks of waiting for official approval for interviews is enough before a case needs to be substituted?A strict policy of substitution of cases becomes a small organizational nightmare, but the common policy within the study of democratic innovations of analysing only friendly and well managed cases has obvious implications for our understanding of this field of practice.
Second, in relation to the outcomes of participatory processes, before the fieldwork we had assumed that most participatory processes would end up with some kind of list of proposals.Some processes might aim at generating public debate with no final conclusions, but we assumed that at least those processes that were explicitly organized to impact policy formulation or decision (see footnote 9) would result in some kind of proposals.They could be detailed and specific, representing a coherent policy package, or be general and vague, more like a heterogeneous wish list.Either way, a list.However, this was not always what we found.In a few of the cases, the participatory process had taken place but proposals did not exist, were not clarified, officially registered or adequately filed.
In other situations, the problem was not the lack of a list of proposals, but having too many lists.This was often the case when a particular participatory process was an amalgam of different technologies developed through different stages, each of which generated a list of proposals.Sometimes these different steps had been integrated into a final document, but in others this was not the case and the outcomes of the process were the reports of each of the consultation stages.When these had been integrated, this was often through a process that was not accountable to participants, opening up the debate as to which of these documents should be considered the final list of proposals (the final product) of the participatory process.

Conclusion
The aim of the Cherry-picking project has been to build a large dataset in order to better understand the fate of proposals that emerge from participatory processes; an issue that has lacked previous systematic consideration.This is not an easy task, but one that is necessary if we are to move the study of democratic innovations away from the tendency to focus on unrepresentative case studies and towards an appreciation of the impact of more 'everyday' participatory processes.
Since this is a relatively novel endeavour, our aim in this paper has been to describe and analyse the methodological challenges that the research design and data collection entailed and to draw some substantive lessons from the presentation of preliminary results.
Moving from research design to the implementation of data collection raised several concerns and uncertainties.We were not aware of attempts to use a similar approach and this lack of previous experience meant many open questions to be addressed.One possible outcome was a failure to gather information about a significant number of cases and proposals.In practice, our fears were not realized: the data collection process has been successful both in terms of the quantity and diversity of participatory processes and proposals assessed, as well as the overall quality of the data obtained.
The strategy adopted allows us to move beyond the case study approach that has dominated our field.The rich and reliable dataset we have generated will enable innovative analysis of the diverse contextual and proposal level factors that facilitate the adoption of proposals as final policies.The question of the fate of proposals can now be approached using a plural dataset that covers high and low quality participatory processes across diverse policy fields and types of participatory processes.At the same time, having rich complementary qualitative material allows us to not only track the data collection process and to assess its quality, but also to incorporate these richer qualitative materials (39 case studies of participatory processes) into the analysis.
Including diverse political contexts is an important characteristic of the dataset, but the full external validity beyond the Spanish context can be assessed only when the project design is replicated in other countries.The need for future replication in other national political contexts is clearly critical, another rationale for offering a detailed description and evaluation of our research design and data collection protocols.
The results show that the degree of implementation of proposals in participatory processes is extremely diverse, with full implementation ranging from 100 percent of proposals (mostly in processes that generated only a small number of proposals) to 0 percent.Even if the overall level of implementation is relatively high, the diverse fate of proposals within the same process is a clear sign that the content of proposals will be crucial in understanding outcomes.If the extent of implementation offers some 'good news', the accountability element indicates the limits of these processes, at least in the Spanish case, with a lack of explanation from public officials when proposals are not implemented or when they undergo substantial changes.
Applying the research design has taught us substantive lessons that are important for the field of study.First, even our extensive attempts to catalogue the full population of participatory processes will not capture everything.What we have achieved in our dataset is certainly more representative than earlier studies, but researchers working across large geographical areas and timescales will always generate samples that are to some extent unrepresentative.It is possible that some of the worst participatory processes are absent from our sample, thus influencing our descriptive statistics.Being clear about how samples are generated and how cases are selected is critical to enable a full understanding of the strengths, potential biases and weaknesses of the data collected.
Second, the diversity of proposals included in our dataset provides evidence of the challenges involved in following the fate of proposals.Initially this is as simple as being able to locate the actual list of proposals that have emerged from the process and to track their development.Conceptually and practically, the difficulties were significant, from adopting a clear criterion of what can be considered the final list of proposals of a participatory process to identifying the extent to which proposals have been adopted.Interesting questions for the participation community emerged, both in terms of what counts as implementation and how to gather evidence of impact.Our hope is that these lessons from how to collect data on the fate of proposals and the knowledge this has generated on the nature of the participatory process itself provide insights that go far beyond our analysis, and benefit other research projects in the field that will follow.