The role of quantitative cross-case analysis in understanding tropical smallholder farmers’ adaptive capacity to climate shocks

Climate shocks are predicted to increase in magnitude and frequency as the climate changes, notably impacting poor and vulnerable communities across the Tropics. The urgency to better understand and improve communities’ resilience is reflected in international agreements such as the Paris Agreement and the multiplication of adaptation research and action programs. In turn, the need for collecting and communicating evidence on the climate resilience of communities has increasingly drawn questions concerning how to assess resilience. While empirical case studies are often used to delve into the context-specific nature of resilience, synthesizing results is essential to produce generalizable findings at the scale at which policies are designed. Yet datasets, methods and modalities that enable cross-case analyses that draw from individual local studies are still rare in climate resilience literature. We use empirical case studies on the impacts of El Niño on smallholder households from five countries to test the application of quantitative data aggregation for policy recommendation. We standardized data into an aggregated dataset to explore how key demographic factors affected the impact of climate shocks, modeled as crop loss. We find that while cross-study results partially align with the findings from the individual projects and with theory, several challenges associated with quantitative aggregation remain when examining complex, contextual and multi-dimensional concepts such as resilience. We conclude that future exercises synthesizing cross-site empirical evidence in climate resilience could accelerate research to policy impact by using mixed methods, focusing on specific landscapes or regional scales, and facilitating research through the use of shared frameworks and learning exercises.


Introduction
Events such as El Niño are predicted to increase in magnitude and frequency as the climate changes (Yeh et al 2009). Understanding the distribution of impacts of these events within rural communities, and the resilience of affected households to these impacts, is critical to developing effective strategies to support adaptation to changing conditions across El Niñoaffected areas of the Tropics (Whitfield et al 2019). Yet evidence on actual social impacts of shocks, and related adaptation practices, is weak; extensive empirical datasets are rare, partly due to the difficulty of obtaining primary data following shocks that are difficult to predict (Holland et al 2017). No large-scale study currently shows the types of adaptation and attributes of resilience of local communities, especially in the aftermath of climate shocks (Vincent 2007, Tompkins et al 2018.
The timely need for such evidence is reflected in the international community's commitment to measuring global progress on adaptation under the 'global stocktake' of Article 7 of the Paris Agreement (United Nations 2015). This Article outlines the need to establish a global goal for adaptation, along with a framework for collating, reviewing and assessing adaptation evidence and progress globally in order to review the adequacy and effectiveness of support provided and needed for it (Kato and Ellis 2016, Magnan and Ribera 2016, Winkler, Mantlana and Letete 2017. Other Articles from the Paris Agreement set further reporting and communicating requirements for countries to develop strategies through National Adaptation Plans and their Nationally Determined Contributions. Despite this policy need, ways to aggregate and synthesize vast amounts of data are still to be determined (Huang 2018). To date there is no clear consensus on the conceptual framing, methods and modalities for assessing resilience for communicating and reporting on adaptation (Kato and Ellis 2016;Craft andFisher 2018, Tompkins et al 2018). In fact, few frameworks defining indicators of resilience have been systematically tested through application in the field, or on empirical datasets spanning multiple contexts (Ifejika Speranza et al 2014).
While empirical case studies are often used to explore the context-specific nature of resilience (Yin 1981, Misselhorn 2005, Seawright and Gerring 2008, synthesizing evidence beyond the local level is necessary to ensure the generalizability and representativeness of individual cases. Cross-case analyses and meta-analyses are used to build a body of knowledge from individual cases to determine their empirical generalizability and theoretical predictions (Tsang 2014). Although they can be used to generate new evidence without the prohibitive costs of new empirical studies, such exercises are often undervalued. In fact, cross-case analysis is necessary to support learning within a field by coherently synthesizing and validating findings from independent cases (Khan and VanWynsberghe 2008).
There are several well-known approaches and techniques depending on the aims and nature of the exercise and available data, although no gold standard exists. However, the current methodological fragmentation between cases means there has been limited systematic use of the numerous individual studies which have been conducted, despite its potentially high relevance for international policy priority-setting (Larsson 1993). Cross-case analysis is methodologically challenging, yet necessary to link lessons from local cases to coherent national and global adaptation plans (Cruzes et al 2015).
To date, most cross-case analysis exercises focus on qualitative interpretation of cases to build and confirm theories (Hoon 2013). For example, analyzing cases over a range of geographies can help explore how findings from climate resilience studies in specific locations are generalizable-or not. This enables the underlying factors affecting resilience and adaptive capacity to be established, by characterizing and reducing the variance associated with individual contexts (Epstein 1983). Yet few academic studies have quantitatively assessed the factors affecting resilience and adaptive capacity across multiple empirical case studies (Brown andWestaway 2011, Cinner et al 2015). This limits the evidence base upon which to discuss the legitimacy and robustness of studies used to advise global climate policy and practice.
Despite the breadth of data on factors influencing climate resilience being empirically produced, there is a lack of guidance and lessons on the use and validity of quantitative statistical analysis of aggregated empirical datasets. Several development programs measuring resilience indicators exist, although these rarely publish data openly (Brooks et al 2014, DFID 2014, Douxchamps et al 2017. Guidance is needed to increase the reliability and broad-scale validity of generalized methods to measure and assess resilience from local datasets (Burgass et al 2017).
This paper aims to test the usefulness of using quantitative cross-case analysis across local empirical datasets, and assess the validity of specific methods for measuring and assessing resilience across multiple case studies. The analysis uses five studies on the impacts of the 2015-16 El Niño event on smallholder farming households in five countries across the Tropics, to examine socio-demographic factors contributing to resilience to climate shocks in these households.
We focus on smallholder farmers as a demographically important social group that is highly vulnerable to climate shocks and changes. Smallholder communities are the backbone of agricultural production across the Tropics and produce 80% of the food consumed (IFAD 2013). Two thirds of the global rural population live in agricultural smallholder systems, particularly in the Tropics and sub-tropics. Most smallholders rely heavily on subsistence crop production and nearby natural resources for their livelihoods (Muyanga and Jayne 2014), making them a particularly vulnerable group to extreme climatic events that directly threaten their food security and well-being (Morton 2007, Godfray et al 2010, Harvey et al 2014. Additionally, empirical studies defining and assessing the resilience of smallholder farmers to climatic changes have risen sharply in recent years (Misselhorn 2005, Nightingale 2009 This study is part of the synthesis phase of the 'Understanding the Impacts of the Current El Niño' programme by the UK Natural Environment Research Council and Department for International Development, which originally funded 15 individual studies. We perform a cross-case analysis of five independent survey-based studies in Ethiopia, Ghana, Malawi, Tanzania and Papua New Guinea and ask: within the datasets collected, are there common socio-economic adaptive capacity factors affecting the resilience of smallholders to climate shocks across these geographies? Are these factors consistent across datasets, so that large-scale conclusions can be drawn regarding the climate resilience of vulnerable groups? What are the opportunities and challenges of using information aggregated across different datasets to draw general conclusions, and what are the implications for future research?

Conceptual framework
Methodologies for cross-scale analysis abound, although most focus on qualitative reviews of the findings, which can then allow for coding, categorization and a posteriori quantification. Where spatial analyses benefit from pre-alignment according to georeferenced systems, empirical surveys are often characterized by their flexible nature involving multiple forms of data collection (Cruzes et al 2015). To ensure replicability, we thus used a conceptual framework to identify the relevant information across cases.
There are a number of frameworks defining the closely linked concepts of adaptation and resilience (Béné et al 2012, Brown 2016). The Intergovernmental Panel on Climate Change (IPCC 2012) defines resilience as the ability to anticipate, absorb, accommodate or recover from the effects of an event or a shock. Resilience is often interpreted as the opposite of vulnerability, especially in programmatic goals aiming to improve resilience to reduce vulnerability (Gallopín 2006). Additionally, there is a trend towards viewing resilience as an attribute rather than an outcome, acknowledging its evolving and changing nature (Béné et al 2012, Eakin et al 2014, Cinner et al 2018, Whitfield et al 2019. Adaptive capacity thus refers to the factors or conditions that affect overall resilience by mitigating the impacts of a shock on systems, or units within a dedicated system, such as households within smallholder farming communities. This framing then leads development interventions to focus on strengthening adaptive capacity through programmatic responses to climatic extremes, thereby increasing climate resilience among vulnerable groups (Folke 2006).
We used a framework which categorises adaptive capacity into four factors: flexibility and diversity, capacity to organize, learning and knowledge, and access to assets (figure 1). We used these factors as a basis for a conceptual framework allowing us to investigate how social variables influence smallholder adaptive capacity during climatic events, and to identify particularly vulnerable groups of people. We mapped the results from the individual projects onto this a priori framework in a participatory exercise with all project teams, in order to establish its validity across country contexts.

Methods
This cross-case analysis uses data collected by five projects focused on smallholder farming households across different countries in the Tropics. All projects also shared similar research goals of understanding socio-economic and ecological impacts of El Niño through new primary data collection within similar time-frames (table 1).

Data preparation
Doing cross-case analysis requires in-depth familiarity with each case, or a pre-defined strategy for systematically synthesizing cases, along with the information they contain (Seawright and Gerring 2008). Thus contextual understanding of the datasets is essential to maintain the meaningfulness of original variables when comparing across projects (Cooper et al 2009). We collaborated with project teams prior, during and after the analysis to validate our approach, results, and inferences. First, we held a two-day workshop with all project teams to discuss the impacts of El Niño in each system and validate the proposed conceptual framework (figure 1). Each project described the main impacts recorded at different stages of the El Niño event. Participants then mapped their smallholder system onto the proposed framework, identifying common socio-economic and ecological factors that affected the resilience of smallholder farming households to the impacts of the El Niño event.
These participatory exercises highlighted that the most common impact felt by smallholder farming households was on crop yields. Given smallholder farmers primarily rely on subsistence agricultural production for their livelihood and well-being globally In terms of the households included in the analysis, there is no universal definition of smallholder farmers. However, most projects converged on identifying them as households with a small farm size, held and worked primarily by household members, except for some cash-crops, where production is primarily directed towards subsistence or local and national markets (Morton 2007). Therefore, in this study we removed households cultivating more than 20 hectares in order to respect smallholder definitions across the five contexts. Across projects, crop loss was specified in relative terms, as a reduction in yield because of the climate shock.
Four socio-demographic, household-level adaptive capacity factors were perceived to have improved or worsened household resilience to El Niño events by all projects; household size, age of household head, education of household head, and household access to assets. These factors were agreed to be of interconnected importance across the four categories of adaptive capacity (Adger et al 2004, Marshall et al 2010. Larger household size can provide more flexibility and livelihood diversity amongst household members, yet can become a burden on food reserves and expenditures for households without agency and assets (Orthner et al 2004). Age of household head reflects the maturity of the household composition, including its livelihood strategies and cohesion, thus households with older heads were felt to have higher capacity to self-organize (Cinner et al 2018). Education can provide greater flexibility to prevent and manage climate shocks, by providing knowledge, skills, and social capital which improve adaptive capacity (Lutz et al 2014). Access to assets generally allows households to adapt better during times of change by increasing baseline productivity and providing access to emergency capital, while maintaining living standards (Adato et al 2006).
We selected these impact and demographic variables as our explanatory variables, as they are commonly used in empirical studies of resilience and are often available in national demographic surveys, hence improving the possibility of replicating similar syntheses in further studies (Below et al 2012, Arouri et al 2015. Other key adaptive capacity factors were identified yet were not used in this paper due to methodological issues in cross-study standardization, including a lack of availability of data on some factors across projects. These include livelihood diversity (Harvey et al 2014), social capital and agency (Pretty andWard 2001, Jones andClark 2013), and institutional and political context (Cinner et al 2011).

Variable transformation
Following the workshop, project teams completed a metadata survey describing their datasets in terms of methods, variable types, time-frames and locations (supplementary table SM 2.1). As each project gathered the data using different units and the range of values varied between contexts, we standardized each variable into five groups of equal range, referred to as quintiles (supplementary material 2.2). This allowed for cross-study comparison of variables measured in different ways such as the impact of the El Niño on crop yield (table 2). The range quintiles were generated by creating five groups of equal range, rather than using other probability distributions, to allow for comparison between continuous variables and those collected on a five-point Likert scale which do not fall according to an equal distribution. To avoid biasing the absolute amounts of crop loss depending on the size of the landholding, when absolute crop loss was recorded it was divided by area of land owned (ACRES, BREAD, CET projects). In the other two projects, relative crop loss was captured using a Likert scale (ECOLIMITS, PNG projects).
In line with our goal of analyzing the crop yield impacts of El Niño with respect to each socio-demographic factor, independent of their external context, we first standardized data by project to account for the heterogeneity of each context. Project datasets were The ACRES project aimed to assess the impacts of the 2015-16 El Niño on cropping choices, yields and post-harvest losses of Conservation Agriculture (CA) and non-CA farmers in southern Malawi.
• Below normal rainfall (50%-100%) with one district (Machinga) suffering drought immediately following crop planting, which prevented seeds from germinating, and the other (Balaka), suffering heavy rains late in the season, which resulted in crop loss due to waterlogging • Household ability to adjust and the degree of crop loss experienced was mediated by individual health and access to farming resources and land.
Building Resilience in Ethiopia's Awassa region to Drought (BREAD)

University of Aberdeen
Ethiopia, three districts in the Awassa region

December 2016
The BREAD project aimed to quantify the impact of food insecurity resulting from El Niño exacerbated droughts in sub-Saharan Africa, both on local agriculture and on farmer livelihoods within the Awassa region of Ethiopia, while assessing potential interventions to improve resilience of these food systems (mainly soil management).
• Below average rains (50%-100%) in 2015 severely impacted crop productivity • Many forced to rely on unreliable wage work, mainly physical labor, or trading, however, these options were limited to those in good health and wealthy households respectively Socio-ecological response and resilience to El Niño shocks: the case of coffee and cocoa agroforestry landscapes in Africa (ECOLIMITS)

University of Oxford
Ghana, six villages in Assin North, Central Region July-August 2016 (Demographic info collected in 2015) ECOLIMITS aimed to understand the resilience of agroforestry systems to climate shocks and long-term climate change by examining the social and ecological impact of the 2015-16 El Niño on these landscapes and their farmers in Ghana.
• Wet season rains were delayed followed by higher than normal rainfall, with the proceeding dry season characterized by exacerbated drought and increased temperatures • Resulting impact on cash-crops varied widely by farm, with some reporting increased harvest Coping with El Niño in Tanzania: differentiated local impacts and household-level responses (CET) The PNG project aimed to improve understanding of the social and ecological impacts of the 2015-16 El Niño event, exploring how natural ecosystems in Papua New Guinea support people at times of need both directly and indirectly through ecosystem services.

University of Edinburgh
• Impacts of the El Niño varied by elevation with unusually high temperatures, bush fires, and drought in lower elevation villages; reduced rainfall during the dry season at mid-elevations, and periodic frosts at higher elevation • Drought and frost resulted in crop losses, both direct and indirectly through ecological processes . This was done with project team members who were familiar with the study sites. For example, the data for PNG was standardized by village first, as each village varied significantly in elevation, affecting crop impacts. Quintiles were then assigned within each subdivided dataset before they were aggregated to maintain a contextualized assessment of what each quintile meant. Extreme outliers were removed from the datasets as range-based quintiles are highly susceptible to skewing by outliers. Outlier identification was conducted visually and in consultation with project teams, to ensure that they were probably not true values, but instead likely to be caused by issues such as over-exaggeration on the part of the respondent or data entry errors. Transformations and standardizations were confirmed with each project team before beginning the aggregated analysis to ensure the variables and quintile thresholds were meaningful within the project contexts.

Analysis
We first ran regressions on each of the original individual datasets, each for one time point only. The response variable was crop loss, which was either collected as a Likert item by the project team, or which we made relative to land ownership to capture relative rather than absolute impact (e.g.: kilograms loss per hectare; supplementary material 3). All analyses were run in RStudio version 1.0.153. While location was accounted for in the aggregated analysis through subsetting, we treated this variable as a fixed effect in the individual regressions. We compared these results with Pearson's chi-square tests on crop-loss and demographic quintiles for each individual project dataset, with the expected values representing the count distribution of impact categories across sociodemographic quintiles, if the two variables are independent (supplementary material 4).
Sample sizes varied between projects (ECOLI-MITS: 94, ACRES: 180, PNG: 187, BREAD: 288, CET: 399 households). To avoid skewing aggregated results towards the project with the largest sample size (CET), we randomly subsampled this project's data to the modal sample size of 187 observations. Using a regression on the aggregated dataset, to test all variable sub-levels independently while controlling for other variables, was not possible given the sample size and the ordinal nature of all variables. Data limitations meant robust parametric observations could not be obtained. We therefore used Chi-squared tests on the aggregated dataset to see whether there were significant differences between the quintile levels of each explanatory variable (p-value=0.01). (table 3). Household size and education had no significant effect on crop loss in any country. Age of household head had a marginally significant positive effect, increasing probability of crop loss in one project, while access to assets was negatively correlated with crop loss in three of the five projects. Village location was also a significant predictor in two projects.

Predictors at the cross-project level
Once variables were transformed into quintiles to allow standardization between countries, only two variables, access to assets and age, remained significant predictors of crop loss at the project level, and only for one project (the BREAD project; supplementary material 4). When testing differences between quintiles of predictor variables in the aggregated dataset, however, significant differences in socio-demographic status were found between households experiencing different levels of crop loss, for all four of the predictor variables, at p-value <0.01 (supplementary material 5).
The degree of impact of the El Niño on crops varied significantly by household size, with very large households experiencing higher crop impact than expected probabilistically ( figure 2(a)). There were also fewer observations of very high crop impact than expected in medium-sized households. Amongst age groups, significantly fewer than expected young  households experienced no crop loss ( figure 2(b)). Amongst the oldest respondents, however, more observations of no crop loss than expected were observed. Households with little education recorded high crop loss significantly more than expected, whereas those with a mid-level education (equating to senior primary to junior secondary) were found to experience no crop loss more than expected ( figure 2(c)). Degree of impact of the El Niño on crops varied significantly by level of access to assets, with significantly more observations of no crop loss than expected for households with high levels of assets ( figure 2(d)).

Discussion
The association between socio-economic adaptive capacity factors and crop impacts Our cross-study results partially align with the findings from the individual projects, and with our a priori hypotheses. When looking at the predictors of crop loss impact project-by-project, we found that age of household head, access to assets, and village location affected impact in some of the projects, while education and household size were not associated with crop loss impact. Aggregating the datasets led to all four demographic variables becoming significant predictors of the level of crop loss felt by smallholder households during the 2015-16 El Niño climate shock. This may be a result of the larger sample size of the aggregated analysis, demonstrating the potential value of aggregating datasets, as the importance of these predictors may not be evident in projects with smaller sample size. The directions of the associations were consistent between the individual and aggregated analyses with the exception of age, which was only found to be significant in the BREAD project. Age, education, and access to assets also aligned with our a priori predictions about how demographic variables would affect degree of crop loss. Household size was not a significant predictor of crop loss in individual project analyses, although it was only marginally non-significant in the CET project. Overall, very large households suffered higher impacts than medium-sized households. This aligns with the assumption that large households with low access to assets and agency suffer the burden of having more people to provide for, while being unable to take advantage of the flexibility and diversity that being larger might facilitate. In fact, results from the ACRES's project team point to health and availability of labor as the largest constraints to adapting to climate shock (Jew et al in review). Therefore, we hypothesize that dependency ratio may be a better predictor of resilience than household size, perhaps explaining the lack of significance of household size in most of the individual analyses and the counterintuitive results in our aggregated analysis.
Age is not significant in the individual analyses, except in BREAD, yet the aggregated dataset suggests that older, more established households experienced less crop impact than the younger age group. The general message of the aggregated analysis of education is that the most educated had less than expected crop losses, and less educated, higher than expected losses. Both results are in line with theoretical assumptions, noting the difficulty of aggregating a variable like education, which varies widely in both content and quality across countries.
Access to assets significantly mediated crop loss in three individual projects, such that households with high access to assets experienced lower crop loss. The BREAD project team confirmed that access to farming resources was a major predictor of a household's ability to mitigate crop loss.

Aggregation of resilience data
Despite a loss of predictive power from transforming project variables into quintiles, aggregation gave us the power to detect patterns statistically that were not perceptible in the individual studies. However, when aggregating data on a complex construct such as resilience, and across highly different contexts, the usefulness of large-scale post-hoc data aggregation remains questionable.
First, not all key variables can easily be appropriately aggregated. This study highlights the challenges of synthesizing and aggregating datasets for quantitative cross-project analysis. It involved projects from the same research programme, designed under a common theme and collecting datasets over a similar time periods. Even so, the transformation requirements meant that most of the variables for which information was collected could not be used. If not all key predictors of resilience can easily be transformed to be considered in aggregated analyses, the socio-ecological systems, the resilience of which is being assessed, are not being appropriately represented. For example, an important adaptive strategy noted in PNG was reliance on social agency and tribal links, an important cultural aspect that was not necessarily considered in other projects. Similarly, the importance of beliefs and worldviews is highlighted in the ECOLI-MITS project (Hirons et al 2018). Discussion with project teams around the conceptual framework (figure 1) suggested these factors were recognized as important but generally treated qualitatively, and data were not collected at the household-level across all projects. Several adaptive capacity factors such as security, learning, and capacity to organize, can be difficult to capture quantitatively.
Second, while aggregation confirmed that all four demographic variables were significant predictors of crop loss, detailed policy recommendations at the site level cannot be based on a synthetic analysis. Rather, aggregated results must be re-contextualized into the case study systems to identify the mechanism for impacts and provide a robust basis for regional and national policies. In fact, despite being relatively the most significant predictor across our analyses, access to assets had different meanings across countries; including land, agricultural resources, household assets, and overall perceptions of wealth.
We conclude that while the aggregation of empirical datasets across contexts does not mean that the generalized trends are invalid, the usefulness of aggregated data in supporting policy making is limited. Instead, these types of syntheses are more useful for academic researchers, by providing evidence towards the support or refutation of general hypotheses about how different socio-demographic variables linked to adaptive capacity affect resilience. Similar conclusions were reached during the Voices of the Poor research (Narayan et al 2000), which emphasized that the multidimensionality of poverty varies by social group, time, location, and country, but nonetheless came up with some common factors.

Recommendations for future syntheses
A posteriori cross-case and synthetic analyses of empirical studies remain methodologically challenging exercises, yet with further tests of methods and modalities, lessons could emerge on how best to align local studies with the evidence needs for national-level policy making. Based on our results, we propose three practical steps for scientists and practitioners involved in the planning, design, implementation and analysis of future large-scale, cross-site empirical exercises to generate bottom-up evidence about climate resilience.
First, adaptive capacity factors are set within their specific socio-ecological settings. We therefore suggest that similar exercises might have more robust results if they are done at the landscape or regional rather than global scales (Vincent 2007). Apart from studies based on national statistics, most empirically-based quantitative exercises and indices have been at national scales within bounded ecological landscapes (Holland et al 2017). Defining culturally and ecologically homogenous systems for aggregation can help manage the tension between maintaining an aggregated indicator's sensitivity to contextual differences, while allowing comparison across sites. In our case, the bounds were set as smallholder farmers in tropical landscapes which were affected by the 2015-16 El Niño event, but the social, ecological and cultural contexts were perhaps too diverse. For example, the role of livestock as an asset varied substantially between the pastoralist and settled agriculture sites.
Second, the use of mixed methods, including qualitative data and expert assessments, is necessary to provide triangulated evidence upon which to base policies. Using exclusively quantitative indicators, whether aggregated or not, limits our understanding of the complexity of the socio-ecological systems in which resilience is grounded. In our analysis, we sought post-hoc understanding of the context through participatory development of a conceptual framework and checking back with teams on the validity and meaning of our results. This helped with interpretation of the results but not with choice of indicator, because we were still constrained by the limited consistency between datasets.
Last, studies and research programs on climate resilience should attempt to better foresee their potential for providing cross-site insights. In our case, the differences in datasets between studies produced methodological challenges which constrained our ability to draw conclusions, requiring us to lose data by coarsening the datasets for comparability. The reasons why data were collected in a given format were context-specific, and so it is unlikely that even if cross-site comparison was an a priori aim, it would have made sense to collect data in a standardized way. Nonetheless, there may have been opportunities for alignment if this had been considered an important goal in the design phase. Better alignment of key variables of interest could further help harmonize climate resilience insights across different types of study types, for example by adding climate-specific questions that are often currently missing in national censuses. This issue applies in general to other fields linked to a lack of methods for integrating data from different evidence bases.
Starting with a coherent high-level conceptual framework (such as figure 1), to derive a set of generally meaningful variables which can be collected in a similar way, is feasible without losing the richness of the individual context, if the data collection exercise remains geographically and institutionally bound. For example, the Poverty Environment Network has successfully utilized a consistent methodology to compare the environmental income of rural communities across 24 countries by developing a standardized questionnaire with adaptable modules (Angelsen et al 2014). Alternatively, more coherently aligned datasets allow methods to be used that can more precisely identify patterns of vulnerability, for example cluster analysis and layering of spatial and climate data sources (Sietz et al 2012).
While academic exercises are often conducted in isolation, development research and practice should consider a priori framing to better align research with other studies and with policy-makers' needs. Within national exercises linked to the Paris Agreements such as National Adaptation Plans and Nationally Determined Contributions, more systematic framing, monitoring and evaluation of empirical adaptation data are not only possible, but necessary to derive the insights required to anticipate and adapt to climate shocks. Despite the time and resources needed, post-hoc assessments of the data can also help make better sense of the evidence, while helping build participatory platforms for knowledge exchange. As an increasing body of evidence emerges to shape global and national policies on climate adaptation, cross-case analysis must play a growing role in corroborating evidence between cases and validating their generalizability to plan for uncertain futures. through the the Understanding the Impacts of the Current El Niño Event programme (Project Code: NE/P004091/1). All project teams are grateful for the efforts of our enumerators and partners, and the kindness and co-operation of participants in our study areas.

Data availability
The data that support the findings of this study are available from the corresponding author upon reasonable request, and upon confirmation by all participating studies included in this article. The individual project data is not available for ethical reasons and original studies must be independently contacted for access.