Whose resilience matters? Like-for-like comparison of objective and subjective evaluations of resilience

As resilience continues its rise to top of the international policy agenda, development funders and practitioners are under mounting pressure to ensure that investments in resilience-building are effective and targeted at those most in need. It is here that robust resilience measurement can make valuable contributions: identifying hotspots; understanding drivers; and inferring impact. To date, resilience measurement has been dominated by objectively-oriented approaches. These rely on external definitions of resilience (often informed by outside ‘experts’, literature reviews or resilience practitioners) and measured through observation or external verification. More recently, the potential for subjective approaches has been proposed. These take a contrasting approach, soliciting people’s judgements of what resilience means to them, and getting them to self-evaluate their own resilience. While both approaches have their strength and weaknesses, little is known about how objective and subjective modes of resilience measurement compare. To shed light on this relationship, we provide like-for-like comparisons of these two approaches using a regionally representative household survey of 2308 households in Northern Uganda. In so doing, we introduce a new measurement approach named the Subjective self-Evaluated Resilience Score (SERS). Outcomes from SERS are directly compared with an objectively-evaluated approach, the Resilience Index Measurement Analysis (RIMA), widely used by resilience practitioners. Findings from the survey suggest a moderate correlation between objectivelyand subjectivelyevaluated resilience modules. More importantly, both approaches share similar associations with many key socio-economic drivers of resilience. However, there are notable differences between the two. In some case, the approaches differ entirely regarding contributions of important traits, including coping strategies, levels of education and exposure to prior shocks. Our results highlight the need for resilience evaluators to consider a diversity of knowledge sources and seek greater use of evidence in indicator selection. We also investigate the properties of the SERS module itself. We find that characterisations of resilience that mimic various commonly-used frameworks produce similar resilience outcomes, suggesting that debates over the exact composition of resilience-characteristics may matter little. In addition, shorter SERS modules match the performance of the full set of SERS questions, allowing for quicker administration and reduced survey burden. Lastly, we call for evaluators to consider the strengths and weaknesses of subjective and objective measurement approaches, including options for combining


Introduction
Resilience has rapidly risen to the fore of the international development agenda (Roberts, Andrei, Huq, & Flint, 2015). As political support for resilience-building grows, funders and practitioners need better ways of evaluating the effectiveness of their resilience-building interventions (COSA, 2017). A wide range of measurement approaches have recently sprouted seeking to address this need (Schipper & Langston, 2015;Brooks, Faget, & Hiejkoop, 2019). To date, the vast majority of these rely on objective forms of measurement (Bahadur & Pichon, 2017). Broadly speaking, objective approaches can be described as those reliant on judgements and observations external to those being measured. Here objectivity can relate to two aspects: how resilience is defined, i.e. who decides what resilience is and the characteristics that make a household resilient?; and how it is measured, i.e. is resilience measured by means of external observation or self-assessed judgements? (Jones, 2019).
Objective approaches to resilience measurement have many advantages. Most use fixed and transparent definitions of resilience (Clare, Sagynbekova, Singer, Bene, & Rahmanberdi, 2018;Beauchamp, 2019); allow for different groups of people to be compared through standardised metrics (COSA, 2017); and rely on proxy indicators, many of which are routinely collected by governments and development agencies (Schipper & Langston, 2015). Yet they are not without their weaknesses (Levine, 2014). For one, agreeing on a common set of resilience indicators has so far proven a considerable challenge -despite numerous syntheses and technical reviews (Schipper & Langston, 2015;Bahadur & Pichon, 2017;FSIN, 2014).
In addition, while household resilience is partly driven by the availability of physical assets and infrastructure, much of it relates to 'soft' elements. These intangible processes -such as community cohesion or social capital -are difficult to see or measure (Adger, 1999). As a result, objectively-evaluated tools often use large lists of proxy indicators to try and account for them (Bahadur & Pichon, 2017). Doing so requires significant amounts of socioeconomic data -much of which difficult to collect and unavailable across the Global South (COSA, 2017). More worryingly, a preference to opt for more easily quantifiable capacities not only risks skewing measurement outcomes but shifts consensus narratives on resilience determinants (Clare et al., 2018).
Most importantly, objective approaches do not currently take into account the wealth of knowledge that people have of their own resilience and contextual information that can help inform it (Marshall & Marshall, 2010). This includes important internal factors such as mental states, aspirations and psychological resilience. Each of which is fundamental in shaping how households react in the face of external threats (Cox & Perry, 2011). With these factors in mind, alternative methods of resilience measurement have recently been sought (Maxwell, Constas, Frankenberger, Klaus, & Mock, 2015).
One promising approach comes in the form of subjective tools for assessment (Marshall, 2010;Jones & Tanner, 2017;Jones & Samman, 2016;Claire, Graber, Jones, & Conway, 2017;Seara, Clay, & Colburn, 2016;Nguyen & James, 2013;Béné et al., 2016a;Sutton & Tobin, 2012). Subjective approaches start from the premise that people have a legitimate understanding of the risks they face (Jones, 2019). As with objective measures, subjective approaches can relate either to how resilience is defined or how it's evaluated. They use people's own judgement of what constitutes resilience and self-evaluations of their ability to deal with risk. Crucially they place few, if any, constrains on what a respondent should consider in assessing their own resilience: measurement is largely directly by the respondent themselves 1 .
Subjective tools have been trialled in a number of different contexts (Jones & Samman, 2016;Marshall, 2010;Seara et al., 2016;Nguyen & James, 2013;Béné et al., 2016a), and may provide a useful complement to traditional objective approaches (Maxwell et al., 2015;Clare et al., 2017). Yet, in practice, we know very little about the relationship between objective and subjective forms of resilience measurement.
In this paper, we address this gap in knowledge by comparing objective and subjective measures of resilience using a regionally-representative survey in Northern Uganda. To do so, we introduce a new subjective approach, termed the Subjective self-Evaluated Resilience Score (SERS). SERS asks respondents to self-evaluate their own household via a series of nine capacityrelated questions. We use this module to make like-for-like comparisons between SERS and an objective measure, the Resilience Index Measurement Analysis (RIMA).
Data from the survey are used to examine three research questions. Firstly, given that each household in the survey is assigned both SERS and RIMA modules, we look at how subjective and objective-measures of resilience compare. Secondly, we take advantage of the wide range of livelihood information collected during the survey to examine whether SERS is associated with the same socio-economic drivers and indicators as RIMA. Our last research question looks at the properties of our subjective module in more detail. Specifically, we are interested in knowing whether different variants of SERS produce similar resilience outcomes.
In answering these three queries we stress that neither SERS nor RIMA are 'true' measures a household's resilience. Given that resilience is not directly observable (DFID, 2015), they offer two different ways of inferring resilience outcomes. Yet, considerable value can still be taken in examining how the properties of the two compare. This is especially relevant in testing assumptions that underlie selection of characteristics and indicators for resilience measurement.
We structure our paper as follows. We start by providing background information on resilience and ways of measuring it. We also clarity distinctions between subjective and objective forms of measurement. In the following section we detail our methods, including the properties of the survey as well as the SERS and RIMA modules used within it. We then present results, followed by a discussion structured around the paper's three primary research areas. We finish with a brief description of limitations and ways forward for the resilience measurement community of practice.

Background and context
The notion of resilience has a long history spanning multiple academic disciplines (Alexander, 2013). In recent decades, the term has gained prominence across the sustainability sciences in describing how socio-ecological systems respond to shocks and stresses. The rise in popularity has coincided with the adoption of resilience as a unifying framework in bridging humanitarian and development practices. Indeed, resilience is now central to a number of international policy commitments, including the UN's Agenda 2030 and Paris Agreement on climate change (United Nations, 2015a, 2015b. While its prevalence has helped to raise awareness for risk reduction, it has also contributed to considerable debate and confusion around the term's actual meaning. Historical applications of resilience (mainly those stemming from engineering and ecology) have long been associated with the ability of a system to return to a normalised state after disturbance or change (Holling, 1973;Walker, Ludwig, Holling, & Peterman, 1981). This clearly has many parallels for social systems. However, social scientists were quick to highlight the importance of unique processes such as adaptation and transformation in allowing societies to respond to threats like climate change or environmental degradation (Pelling, 2010;Miller, 2010). In many ways, these can be seen as at odds with notions of resilience as bouncing-back, further contributing to conceptual ambiguity (Olsson, Jerneck, Thoren, Persson, & O'Byrne, 2015).
Discrepancies like these have considerable implications for measurement efforts. Given that resilience cannot be directly observed, most measurement approaches choose to break resilience down in its constituent characteristics (DFID, 2016, Bahadur & Pichon, 2017. Whichever mix of characteristics the evaluator chooses to assign is therefore likely to play a large role in dictating 1 Subjectively-oriented questions should allow for respondents to internally consider and validate their own understanding of resilience, with outcomes feeding into a quantitative measure. measurement outcomes: and partly responsible for the vast number of different resilience tools that have emerged in recent years (Schipper & Langston, 2015). To make matters harder, resiliencerelated characteristics are seldom observable in themselves (Brooks et al., 2019). In the case of objectively-evaluated frameworks, tools often address this challenge by resorting to large lists of proxy-indicators tied to socio-economic traits or other development outcomes (see HSSAI, 2015;Bahadur & Pichon, 2017 for a review of different approaches).
With the above in mind, we focus our analysis on a narrow definition and application of resilience. Specifically, we hone in on a particular unit of analysis: the household. This is due to the centrality of the household unit in dictating responses to external stimuli whether at the individual, family or community-levels (Toole, Klocker, & Head, 2016). Indeed, many of the assets, capabilities and functions commonly assumed to support resilience in social systems derive from, and are dictated by, household-level dynamics (Frankenberger & McCaston, 1998). A focus on households also allows for distinctions to be drawn between psychological resilience -associated with the ability of an individual's psyche to deal with shock or trauma -and the resilience of the individual or household overall. This point is particularly relevant when assigning modules on subjectively-measured resilience (Windle, Bennett, & Noyes, 2011).
Before any effort at measuring resilience can start, one question has to be clarified: resilience to what? Resilience can be defined in relation to a specific hazard or a related set of hazards (Brooks et al., 2019). Here, the literature is replete with examples, including flood resilience (O'Sullivan et al. 2012); drought resilience (Keil, Zeller, Wida, Sanim, & Birner, 2008); or climate resilience (Tyler & Moench, 2012). Conceptually, the idea is to examine the ability of a given system (in our case a household) to cope with and respond to a particular hazard. However, hazards rarely occur in isolation. Households often have to contend with exposure to multiple overlapping risks (O'Brien & Leichenko, 2000;Kelman, 2010;Zobel & Khansa, 2011). As such, resilience is increasingly referred to in relation to broader systemic risk or outcome-related traitssuch as disaster resilience (Cutter, Burton, & Emrich, 2010), food systems resilience (Pingali et al., 2005) or economic resilience (Rose, 2004). Here, resilience is thought of as the ability of a system to maintain wellbeing outcomes in the face of diverse multi-hazard environments, many of which may interact in threatening a household's basic functions: 'Climate change, globalization, poverty, earthquakes, injustice, tropical cyclones, lack of livelihood opportunities, inequity, landslides, overexploitation of natural resources, epidemics, and lack of water supply-amongst many other ongoing challenges-often converge to most affect those who have the fewest options and resources for dealing with those challenges. Consequently, those with the fewest options and resources tend to be most vulnerable across all forms of threats, demonstrating multiple exposure to multiple threats simultaneously ' Kelman et al. (2010:23) In the context of this study our focus is on Karamoja, Northern Uganda. More so than any environment, Karamoja is one facing a wide range of overlapping threats: a confluence of colonial subjugation, regional and tribal isolation, and a harsh natural environment (Levine, 2010). Accordingly, we concentrate our analysis on a broader multi-risk conceptualisation of resilience. Yet, we recognise the importance of single-hazard approaches, and seek to compare our main results with these where-ever relevant (see Robustness checks below).
Another important point of clarity relates to the distinction between objectivity and subjectivity. For the purposes of this paper we make use of the objectivity-subjectivity continuum proposed by Jones (2018). The continuum refers to objectivity and subjectivity in resilience measurement according to two key tenants. The first is how resilience is defined. Objective definitions of resilience can be classed as those externally derived. In practice, this means that resilience is not determined by the people or system being assessed. Rather, the characteristics (or indicators) used to evaluate resilience are drawn from the wider academic literature or through use of extensive expert consultation (Schipper & Langston, 2015). Objective characterisations also tend to be standardised and fixed in their depiction of resilience and its properties (Jones, 2019). On the other side of the continuum, subjective measurement tools draw primarily on the judgement of those being measured themselves. This means that individuals (or a collection of individuals) are responsible for defining what resilience means to them and properties that make up a resilience person or system. These inputs are then used to guide the measurement approach that follows.
The second tenant of the objectivity-subjectivity continuum relates to measurement. Objective approaches to resilience rely on external observations and verification, i.e. little to no room for the judgement and perspectives of those being measured. For example, use of satellite imagery to evaluate the extent of damage to a property, or an assessment of household assets through a household survey can both be seen as objective measures. They involve little, if any, subjective judgement on the part the respondent. On the other hand, subjective assessments make use of people's perceptions in the measurement process itself. They typically involve asking respondents to self-evaluate themselves, drawing on their own internal judgement of their household's ability to deal with risk. The same approach is often used in evaluating subjective wellbeing, where people are asked to self-assess levels of life satisfaction or happiness (OECD, 2013;Dolan & Metcalfe, 2012).
The advantage of portraying this relationship as a continuum is that it highlights that many aspects of measurement fall somewhere in between the two ends of the spectrum. Indeed, few measurement approaches are entirely subjective or objective in nature: choice of objective indicators is often informed by bottom-up community consultations and piloting; subjective-evaluations are often worded and grouped according to objectively-defined definitions of resilience (Jones, 2019). As such, we are careful to make distinctions between the two categories of definition and measurement when referring to the properties of the objective and subjective measures used in our survey.

A survey comparing objective and subjective-measures of resilience
In order to shed light on the relationship between objective and subjectively-evaluated resilience, we carry out a representative survey of 2380 households in the Karamoja region of Northern Uganda. We assign separate RIMA and SERS modules to each household allowing like-for-like comparisons of both approaches. Below we provide further detail on how RIMA and SERS scores are computed, as well as the survey methods used to inform our analysis. a) RIMA: an objectively-evaluated resilience module The body of objective measures for resilience measurement is large and ever-growing (Schipper & Langston, 2015;Bahadur & Pichon, 2017). Amongst them, one of the most commonly applied quantitative measures is the Resilience Index Measurement Analysis (RIMA). RIMA is developed by the United Nations Food and Agriculture Organisation (FAO) and has undergone a number of iterations since its development by Alinovi, Mane, and Romano (2008) as an econometric model for estimating household-level resilience to food security. Its latest iteration, RIMA-II, comprises a multi-dimensional objective index devised through Structure Equation Modelling (SEM) and designed to be flexible in relating to outcomes aside from food security (FAO, 2016).
In terms of how RIMA-II conceptualises resilience, the approach acknowledges frameworks supported by the Technical Working Group on Resilience Measurement (FSIN, 2014) and unpacks resilience into four ''pillars": (1) Access to basic services: a household's access to enabling institutional and public services environments. It includes indicators like health facilities; education; credits; water; toilet; (2) Assets: income and non-income related assets that enable a household to make a living. It includes both productive (land; livestock; and other income generating activities); and non-productive assets (like households; and other durable goods). (3) Social safety nets: the network upon which a household can rely when and if faced with a shock. It includes both formal and informal transfers; the social network of solidarity; and reliance. (4) Adaptive capacity: a ''household's ability to adapt to the changing environment in which it operates" (FAO, 2016, p. 14). It includes factors such as education; number of income sources; and reliability of income.
Each pillar is considered a latent variable and is in turn made up of range of proxy socio-economic indicators gathered using household survey data (see Table 1).
In its most commonly used format, RIMA-II is estimated via a two-step procedure. First, a factor analysis is performed with indicators for the four RIMA pillars. Second, a Resilience Capacity Index is devised using the output of the factor analysis by means of a Multiple Indicators Multiple Causes (MIMIC) model -a type of structural equation model (SEM). The MIMIC model is comprised of both the SEM (where observed variables are considered causes of resilience as latent variables) and the measurement model (where the observed variables are considered indicators of resilience). The latter requires a reference unit: a variable assumed to be affected by a household's resilience, commonly associated with wellbeing-related metrics. Given the mandate of FAO, the chosen outcome is typically food security -often equated as combination of monthly per capita food expenditure and dietary diversity. The process allows for a single unit of resilience to be created for a household along a scale of 0 (lowest resilience) to 1 (highest resilience) using a min-max normalisation procedure. An annotated diagram of processes employed in devising the RIMA-II score is represented in Supplementary Fig. 1. For full details of the procedure see FAO (2016) and D'Errico, Grazioli, and Pietrelli (2017).
For the purposes of this paper, we also introduce an alternative specification of the RIMA-II model. This hybrid model -which we refer to henceforth as 'RIMA' -removes the score's tie to a food security outcome. The main advantage of this new model is that it better reflects resilience to broader livelihood outcomes (i.e. overall resilience), rather than their ability to solely maintain food security outcomes (the focus of the original RIMA-II model). As such, this hybrid RIMA is better suited for comparison with the SERS model which is similarly focused on a multi-hazard view of resilience. We therefore consider it our preferred specification for the paper's main analyses. However, we also recognize the wellestablished use and track-record of the RIMA-II method and run parallel analyses comparing SERS with the original RIMA-II approach too (see Section 4a Testing assumptions).
Our hybrid RIMA measure follows the same initial steps as the original RIMA-II and is based on the conceptual premises outlined in the RIMA-II's guidelines (see FAO, 2016;D'Errico et al., 2017). It uses the same four pillars of resilience, and includes all of the indicators used in RIMA-II. The only difference is that it does not include the measurement part of the structural equation model. Instead it adopts a basic structural equation model to estimate the latent variable that is defined as the co-ordinated result of the four key pillars of resilience. As per the original RIMA-II approach, RIMA scores are normalised on a scale of 0-1.

b) SERS: a subjectively-evaluated resilience module
For the subjective module of our survey we use the Subjectively self-Evaluated Resilience Score (henceforth referred to as SERS). Similar to RIMA, SERS considers resilience to be made up of a range of resilience-related capacities. The module is adapted from a hazard-specific variant proposed by Jones, Samman, and Vinck (2018) and features a total of nine resilience-related capacities and capitals chosen on the basis of an extensive review of available literature (see Table 2). Each resilience-related capacity is then adapted to self-elicited questions, with respondents asked to rate their levels of agreement ranging from strongly agree to strongly disagree. Pilot exercises of the module were also carried out in a nationally representative survey of Kenya (early 2017) and regional surveys in Hpa An, Myanmar (Jones, 2017).
Subjectivity is are at the core of the SERS approach. However, it is important to note that by prescribing a set of resilience-related characteristics, SERS falls under the category of objectivelydefined (i.e. the characteristics are selected from a review of the wider resilience literature, rather than those being measured themselves). The distinction highlights the non-binary nature of objectivity and subjectivity, and that most approaches will have elements of both. It also draws attention to the advantage of thinking of the relationship between the two along an objectivitysubjectivity continuum as described in Jones (2018). Respondents are asked to score their level of agreement with each capacity using a Likert scale with 5 response items (Strongly disagree = 1, Strong agree = 5). While numerical conversion of Likert scale responses of this type is typical across the social sciences, it is important to recognise that assumptions of cardinal comparability are disputed (Kristoffersen, 2017).
Each characteristic can either be compared individually or aggregated together to form a single collective score comprising multiple capacities. Together this aggregate score constitutes the household's resilience outcome, acting as a rough marker of overall resilience. A Cronbach's Alpha score of 0.79 suggests high internal consistency across the nine resilience-capacities. To ensure computational ease and transparency, we numerically convert answers for each of the resilience-related capacity questions and calculate an equally-weighted average mean score. As with the RIMA output, subjectively-evaluated resilience scores are normalised on a scale of 0-1 using min-max normalisation (higher scores indication higher resilience). While this score is neither exhaustive nor holistic in measuring a respondent's subjectively-evaluated resilience, it does provide a useful starting guide. As a robustness check, we also devise a score using an alternative weighting procedure derived from a Principal Component Analysis (PCA). As overall results appear to be almost identical between the simple averaging and PCA, we present results from the equal-weighted score in this paper (see Section 4a Testing assumptions, and Supplementary Tables 2 and 5). SERS is designed to be flexible with regards to different definitions and characterisations of resilience. As such, it allows evaluators to pick and choose any number of combinations of different resilience-related capacities. In this paper, our main model uses all 9 resilience-related capacities based on the approach used by Jones (2018). However, we also look at different variants of SERS based on widely used frameworks in the resilience literature. The first is a subset of three resilience-capacities comprising Absorptive, Adaptive and Transformative capacities named the SERS-AAT variant. This combination of capacities has been widely applied within the resilience literature (Pelling, 2010;Béné, Wood, Newsham, & Davies, 2012) as well as shaping resiliencebuilding programmes -see Oxfam's Framework for resilient development (Jeans, Castillo, & Thomas, 2017). On one hand it builds on early socio-ecological concepts of resilience, highlight the ability of system to maintain core functions in response to threats (Holling, 1973;Walker et al., 1981). One the other, it acknowledges the importance of adaption, and in some cases transformation, to allow systems to deal with evolving risk (Kates, Travis, & Wilbanks, 2012).
A second variant uses another subset of three capacities made up of anticipatory, absorptive and adaptive capacities named the SERS-3A variant. This mimics the 3A's framework first proposed by Bahadur et al. (2015) and used by the BRACED programme as well as range of other assessments (Wilson & Yarron, 2016;Bottazzi et al., 2018). This framework is similar to the AAT model, adding particular weight to the importance of anticipatory mechanisms, such as early warning systems and disaster risk reduction activities (in place of transformation) (Thomalla & Larsen, 2010).
Lastly, as noted in Section 2, we focus our SERS module on a multi-hazard view of resilience. As such, SERS questions make no reference to individual hazards. Instead we refer generically to 'threats', 'challenges' or 'disasters'. Doing so is also important in minimising the likelihood of priming effects amongst respondents (OECD, 2013). However, we recognise that specifying resilience to single hazards can also be of benefit to practitioners -particularly in cases where interventions targeted a single hazard. Accordingly, we also include a hazard-specific SERS module mimicking the example used by Jones et al. (2018) focused on drought riskthe primary threat facing livelihoods in Karamoja (see Supplementary Table 1 for wording).
Inherently the questions used in the SERS module cannot cover all aspects of resilience, nor do they seek to. Rather, they give a useful indication of a subset of capacities that are known to strongly influence a household's resilience. In addition, while each capacity is considered distinct in its own right, it is important to note that some degree of overlap is inherent, limiting the extent to which the unique contributions of each can be isolated. For example, close ties exist between adaptive and transformative capacities, as both relate to processes of structural change (Few, Morchain, Spear, Mensah, & Bendapudi, 2017). Yet, these are often referred to separately within the resilience literature (Pelling, 2010;Kates et al., 2010). As outlined above, different subsets of the SERS module can also be constructed to account for preferences in conceptualising resilience.
Finally, we note that attempts to quantify people's own assessments of resilience may not be altogether obvious. Subjective insights on resilience have been gathered extensively through qualitative means, recognising the richness and nuance that these methods provide (Ayeb-Karlsson, van der Geest, Ahmed, Huq, & Warner, 2016;Maxwell et al., 2015). SERS is by no means an attempt to replace the importance of qualitative contributions to our understanding of resilience through interviews, focus groups and immersive research methods. Rather, it seeks to complement it. It is a way of translating bottom-up subjective judgements into a quantifiable metric that can be readily compared, and potentially combined, with traditional objective approaches. In doing so, it answers recent calls for plurality of research methods in understanding resilience: 'Resilience measurement requires multiple method assessment approaches that capture perceptions, opinions, judgments and the nature of social interactions as well as the observable or easily measureable characteristics of social ecological systems'. Maxwell et al. (2015:4) c) Data To test the relationship between objective and subjectively evaluated resilience we make use of a household survey conducted in Karamoja, Uganda in 2016 by FAO. The purpose of the survey was twofold: to understand the resilience capacities of communities in Karamoja; and to determine baseline values for an impact  The survey is composed of a total of 2380 households. The sampling strategy is stratified according to the five strata: (1) target households, which are those reached by the JRS in 12 parishes of the Moroto and Napak districts; (2) direct spillover households, which are those located in the remaining parishes of the Moroto and Napak districts and are not involved in the JRS; (3) indirect spillover households, which are those located in the two districts where the JRS does not operate (Kotido and Nakapiripirit) but where other UN projects are ongoing; (4) a 'different ethnicity' group, which includes households located in two districts (Abim and Amudat) populated with ethnic groups that are different from the Karamojong (the principle ethnic group in the region); (5) and the pure control group, comprised of households located in the Kaabong district, which have the same ethnic group and socioeconomic conditions, mostly pastoralism, as the target group, but which are not involved in the JRS.
The household questionnaire is comprised of a range of thematic sections and piloted in Moroto in November 2016. Specifically, it collects detailed information on household characteristics, including food and non-food consumption, shocks, coping strategies, and so on. Much of the socio-economic data was then used to compile the RIMA indexes, and compared with subjective measures of household resilience. Results from the survey are shown below.

Results
The first research question that we explore is whether our objectively-evaluated (RIMA) and subjectively-evaluated (SERS) modules are correlated. Fig. 1 shows a series of different associations and traits comparing RIMA and SERS outcomes. The first thing to note is that the distribution of scores is far narrower for RIMA (s = 0.13, x = 0.31) than for SERS (s = 0.21, x = 0.49) (Fig. 1a).
Secondly, the association between RIMA and SERS scores is positive. Fig. 1b shows a count plot of the full range of resilience scores, while Fig. 1c We are also interested in comparing associations between our resilience modules and key socio-economic drivers of household resilience. For example, much of the literature cites the accumulation of asset wealth as a strong determinant of a household's ability to deal with disturbance (Tyler & Moench, 2012;Cutter et al., 2008). Interestingly, both RIMA and SERS modules demonstrate positive relationships (as shown in Fig. 2a), with higher wealth accumulation corresponding to higher levels of resilience. Similar positive associations are apparent for diversity of incomes sources as well as food security (represented by the CSI index 3 ), both traditionally considered as core drivers of resilience at the household level (Adger, 1999;Jabeen, Johnson, & Allen, 2010). Not all assumed associations overlap however. For example, while the highest level of education for household heads shows a marked positive association under RIMA, no such association is apparent for the SERS module 4 .
While descriptive and univariate analysis is useful in uncovering broad associations, it is also important to account for the effects of any confounding factors before drawing firm conclusions. To do so we run a series of OLS regression models with SERS and RIMA as our two dependent variables. A range of socio-economic traitseach considered to have a degree of association with resilience within the resilience literature -are gathered from the remainder of the survey modules and serve as independent variables within the models. Both models include area fixed effects with standard errors clustered at the sub-county level. In comparing a range of different setups, we also look at outcomes from regressions models using simple Ordinary Least Squares (OLS) with area fixed effects removed, area-fixed effects with robust standard errors and multi-level models with households nested within sub-countries and districts -see Robustness Checks.
The primary OLS set up with area-fixed effects is presented in Eq. (1). Here the outcome SERS_9C hc relates to the 9C variant of the SERS model indexing households h in sub-county c. SHOCK hc is a vector of dummy variables for a series of self-reported shocks, these include drought, flood, crop disease and illness. DRIVER hc is a vector of socio-economic variables commonly associated with drivers of household resilience. a c is shown as a sub-county fixed effect (expressed as dummies) with the error term represented by e hc .
Outputs from the above are compared directly with Eq.
(2). This model shares an identical structure to that of Eq. (1), simply replacing RIMA, through RIMA hc , as the outcome variable of interest.

RIMA hc
In effect, outputs from Eq. (2) are somewhat uninteresting in isolation: results simply inform us of the assumptions and weights assigned to various indicators that feed into the RIMA model. Instead, real utility comes from side-by-side comparisons of Eqs.
Given that the SERS approach does not factor any of the shocks or input variables when asking people to self-evaluate, a comparison of the two models serves as a quasi-independent check of the RIMA set-up and indicators used within. In theory, if both scores are wholly reflective of the same underlying property (and are void of bias), then we would expect similar trends and effects from the variables of interest. Though in practice it may be difficult to argue that RIMA and SERS are capturing an identical latent construct (i.e. overall household resilience), they undeniably overlap and should be expected to broadly reflect the same associations with relevant drivers of resilience. Fig. 3 presents side-by-side comparisons of outputs from the RIMA and SERS models. Variables above the dashed horizontal line share the same sign of association for both models (i.e. a positive or a negative association on resilience), those below have opposing signs. For ease of viewing and interpretation, variables are ordered in relation to the highest and lowest coefficients in the RIMA model (essentially showing us the magnitude of relative weightings used in the RIMA set up). Given the potential for non-linear relationships we also include quadratic terms for each of the socio-economic drivers (see Supplementary Table 7). Again many associations are matched between the two approaches -though there are some differences between linear and non-linear relationships (notably wealth and CSI).
2 For more information on the types of activities supported under the JRS and its evaluation see http://www.fao.org/3/ca0345en/CA0345EN.pdf. A follow-up survey is scheduled to take place by 2020 resulting in a panel dataset. 3 CSI is an indicator of household food security that uses a series of questions about how households manage to cope with a shortfall in food for consumption results in a simple numeric score. In its simplest form, monitoring changes in the CSI score indicates whether household food security status is declining or improving (see Maxwell, Watkins, Wheeler, & Collins, 2003).
Another way to compare our two resilience modules is to see whether the individual characteristics of RIMA's resilience model are associated with people's self-assessed scores. RIMA breaks resi-lience down into four core characteristics: adaptive capacity; assets; social safety nets; and access to basic services. To examine the extent to which these characteristics are reflected in people's  self-assessments we run an OLS model (Eq. (3)) with SERS as the outcome variable, SERS hc . As with prior models, we include areafixed effects and cluster standard errors at the sub-county level. Under this specification each of RIMA's four characteristics 5 are represented by AST hc (assets), ABS hc (access to basic services), SSN hc (social safety nets) and AC hc (adaptive capacity) -see FAO (2016) and D'Errico et al. (2017) for a full list of indicators associated with each pillar.

SERS hc
Outputs from Eq. (3) are compared directly with a parallel model that places RIMA scores, RIMA hc, as the outcome variable. Indeed, given that the indicators used in compiling RIMA's four pillars are largely independent of the SERS set-up it can loosely be considered as an independent check.
Again, Eq. (4) on its own is not particularly informative. It demonstrates the weightings assigned to each of four characteristics of resilience as specified within the RIMA model (and hence why the confidence intervals are far smaller for RIMA compared with SERS outcomes). However, when comparing the Eqs. (3) and (4) together we see that all four characteristics are positively associated with SERS (Fig. 4). Assets have the largest marginal effect with social safety nets the lowest. It is also worth noting that the effect sizes for anticipatory capacity and assets are far higher for the RIMA model compared with SERS. While this may imply that RIMA is overweighting, it's important to consider that selfassessments have a far wider range of potential influencing factors when compared with RIMA's four characteristics.
The paper's final research query seeks to compare outcomes of different versions of the SERS approach. As is clear from Fig. 5a, strong overlaps exist between the 9C and 3A variants of the SERS model, with both closely tracking a 1:1 ratio. A similar association is also apparent when comparing 9C and AAT variants in Fig. 5b. While values are more scattered, comparison of the 3A model of overall resilience with the 3A hazard-specific variant also demonstrates a high degree of overlap.   Fig. 6a shows that correlation coefficients between SERS variants are high. By way of an interesting point of comparison, the relationship between the original RIMA-II specification (the variant of RIMA that is tagged specifically to a food security outcome) and our hybrid version of RIMA (which has no outcome variable and is therefore a better reflection of overall resilience) is notably weaker with an R 2 of 0.48.
We can also look at correlations between the various resiliencerelated capacities that contribute to SERS. Fig. 6b highlights a wide range of associations, with the strongest tie appearing to be between Adaptive and Absorptive capacities. To get a more detailed understanding of the links between resilience-related capacities we also run a Principal Component Analysis with all nine capacities in the full SERS module. Supplementary Table 2 confirms that Absorptive and Adaptive capacities are the two strong contributors to the SERS scores, followed by Anticipatory and Transformative capacities. Interestingly, Political capacity appears to have consistently low correlations with other resilience-related capacities, with slight loadings in the first principal component. As such, its use is dropped as part of the SERS-PCA variant, weighted on the basis of the first principal component across all nine-capacities (see Supplementary Table 2).
While different subjective variants appear to be highly correlated, what about similarities between underlying socio-economic drivers? As with the analysis above we perform a series of OLS regressions that allow for side by side comparisons between the three main variants of the SERS subjectively-evaluated resilience approach. Specifications for these models are identical to Model 1 except that the outcome variable is replaced by the 3A and AAT variants of overall resilience respectively. As apparent from Fig. 6, associations between subjective self-evaluations of resilience and various drivers and shocks remain notably similar across the three variants (see also Supplementary Table 6). Indeed, there are only a small number of variables that do not exhibit the same sign and  level of statistical significance. Moreover, the two with opposing signs -whether the household has experienced a flood in the past 12 months and relationship status of the bread-winning couplehave such wide confidence intervals that little can be inferred about the differences.

a) Testing assumptions and robustness
In order to test the validity of the paper's findings we perform a number of robustness checks. As mentioned earlier, alongside our preferred model setups we also run alternative specifications to examine the consistently of our results. Supplementary Table 3 shows comparisons of RIMA and SERS when run using OLS with no area-fixed effects (1 and 5), OLS with area fixed effects (2 and 6), OLS with area fixed effects and clustered standard errors (3 and 7, and our preferred model setup) and a multi-level model that nests households within sub-counties and district (4 and 8). All OLS models feature robust standard errors unless otherwise stated. Though there are some apparent differences in the size of standard errors (notably in relation to the effect of crop diseases), all four models appear to be largely consistent.
Another important step taken in our analysis is the use of a modified version of the RIMA approach. The hybrid RIMA model is considered a more suitable comparison with SERS given that it better reflects overall resilience and is no longer tied to a foodsecurity outcome. Yet, we recognise that the original RIMA tool is well established within the literature and often used as a proxy for wider resilience outcomes too (not just food security). Therefore, we run the same model set-ups as before, using the original RIMA-II instead of hybrid version. Results are presented in Supplementary Table 4 with a similar range of regression runs as specified in SM Table 3. Some differences are noted between the two setups, though most are unsurprisingly in relation to food-related variables such diversity of food intake, annual food consumption and crop diversification.
A key factor in evaluating survey modules made up of multiple question is weightings. While we use an equally weighted average for most of the results presented in the paper, we also test our results with an alternative version (labelled SERS-PCA) that weights resilience-capacities according to the first principal component (with 8 questions retained). Accordingly, Supplementary Table 5 re-runs comparisons between RIMA and SERS-PCA. Again, we find few differences.
Lastly, we note that a weakness of many subjective measures is a tendency for respondents to agree with all questions, or provide consistently similar answers throughout a survey -known as acquiescence bias (OECD, 2013). To account for this, we remove any household from the sample that provides the same answer across each of the resilience-related capacity questions. When Model 2 is rerun under this set-up we see no qualitative differences (see Supplementary Table 6).

Discussions and conclusion
Results from our Karamoja survey point to a number of interesting findings and discussion points. Below we reflect on our three main research areas, and consider limitations and ways forward for future efforts to define, measure and promote resilience.
a. Comparing objective-and subjectively-evaluated resilience One of our main findings is that a positive linear relationship exists between the objective (RIMA) and subjectively-evaluated (SERS) modules used in our Karamoja survey. This relationship is clearly highlighted in Fig. 1c-d, with mean SERS scores consistently rising with higher aggregated RIMA values. The fact that two largely independent approaches appear to point in a similar direction will give some confidence to resilience evaluators. More importantly, the association suggests that, in the case of Northern Uganda, households that are assumed to be resilient (at least from the perspective of FAO's criteria) generally perceive themselves to be resilient as well.
Another point is abundantly clear: the relationship between RIMA and SERS is far from strong. An R 2 of 0.25 suggests that any correlation is moderate at best, and that the two measures should not be used interchangeably. The implications for resilience measurement depend on how the scores are used and compared. From one perspective, comparing raw scores paints a picture of a noisy relationship between subjective and objective-measures (Fig. 1b); from another, aggregated information shows neat and clear trends (Fig. 1c-d).
Differences in the distribution of both scores are also marked, with standard deviations (s) of 0.21 and 0.13 for SERS and RIMA respectively. Indeed, it suggests that, from the perspective of people's own judgements, levels of resilience are far more varied than assumed under RIMA. Part of this may reflect the fact that subjective-evaluations place no limits on the wide range of factors that an individual might consider in evaluating their household's resilience -as opposed to RIMA that is constrained to a handful of objective indicators. It may also reflect a diversity of subjective interpretations and judgements on resilience.
With this in mind, it is worth reinstating that neither RIMA nor SERS are direct measures of a household's resilience. They are two different ways of inferring resilience. Understanding which of the two most closely approximates a household's 'true' resilience requires tracking changes in wellbeing outcomes over time -itself subjective to different interpretations and inferences. As such, the value of both measures should be considered equivalently, based on the strengths of methodological assumptions and objectives of the evaluator. Our findings underscore the need for development actors to be mindful of the diversity of knowledge sources for resilience. Care should also be taking in assuming that aggregated resilience outcomes are homogenous across communities and households. Most importantly, as evaluators seek to refine and choose methods for evaluating development practices, it is imperative that the merits and limitations of different methods are made fully transparent.

b. Comparing associations with drivers of household resilience
The second dimension of our study looks at whether objective and subjectively-evaluated approaches share similar associations with key socio-economic drivers of resilience. Like-for-like comparisons in Figs. 2 and 3 show that most of the traits in our model (16/22) share the same sign of influence for both RIMA and SERS. Common significant drivers include: asset-wealth; diversification of income sources; livelihood type; distances to a hospital and livestock market; and access to agricultural inputs and access to credit. Most of these have a rich history of association with resilience amongst the wider resilience literature (Tyler & Moench, 2012;Cutter et al., 2008;Adger, 1999;Jabeen et al., 2010).
Many of the associations make logical and conceptual sense. For example, the importance of wealth and financial capitals is well documented as a driver of resilience, allowing households to accumulate and use assets during times of hardship (Tyler & Moench, 2012;Cutter et al., 2008). The same is true for income diversity, where development practitioners have long promoted diversity as a means of spreading risk (Jabeen et al., 2010). A negative association with distances to hospitals and markets is also reassuring, though it's important to note that both are largely found in urban areas and may be confounded by other unobserved variables.
Interestingly households reliant on farming as their primary source of livelihood appear to have lower scores than agro-pastoralists. This negative relationship may point to the benefits accrued by agro-pastoralists in being able to more easily reallocate assets and livestock in search of more favorable climes during times of drought (or other hardships) (Opiyo, Wasonga, Nyangito, Schilling, & Munang, 2015). The finding is particularly relevant in light of ongoing political and academic debates over tradeoffs between pastoral and settled livelihoods in Karamoja. Indeed, many development actors have historically portrayed nomadic pastoralism as a particularly vulnerable and unviable source of livelihood in the region (Levine, 2010).
Strong overlaps in association suggest that SERS is picking up on many of the same socio-economic drivers and indicators used in deriving the RIMA model. The fact that all four pillars of the RIMA model are positively associated with SERS outcomes further underscores this point (Fig. 4). Common associations are all the more significant as the two modules are largely independent of one another. None of the objective indicators that make up the RIMA model are used in SERS. Aside from potential priming effects, there is nothing to systematically encourage respondents to respond similarly across the two modules -indeed, given the size and complexity of the RIMA survey module this would be a considerable undertaking.
However, our objective and subjectively-evaluated module do not agree on associations amongst all drivers. Indeed, just as much can be learned from disagreements between the two. For a start, effect sizes differ markedly. For example, wealth has the strongest positive association with RIMA (a 0.048 rise for every one standard deviation increase in the wealth index). Yet, its effect on SERS is less than half (a 0.021 rise). The implication here is that while both RIMA and SERS recognise wealth as an important component of resilience, its association is considerably lower when considering people's own self-assessments. Similar patterns are true for other drivers, including diversity of income sources and years of schooling for female household members -both of which have significant positive associations, though with far weaker effects on SERS outcomes.
Interestingly, the converse is also true. A number of traits have far stronger effect sizes for SERS than for RIMA. Most notable are large differences for access to agricultural inputs and livelihood practices, with SERS ß values twice those for RIMA. Some drivers are significantly linked with one approach and not the other. For example, food consumption has a significant positive association with RIMA, with a negligible and insignificant role for SERS outcomes. Again, the implication being that the influence on socioeconomic drivers of resilience differs from the perspective of expert elicitation (i.e. RIMA) compared with people's own subjective judgements (SERS).
Perhaps the most important finding relates to instances where drivers have opposing signs of influence (those below the grey dotted line in Fig. 3). Here we observe traits that have fundamentally different associations between RIMA and SERS approaches. The largest such difference comes in the form of the Coping Strategies Index, considered a proxy for food insecurity. CSI has a large positive link with RIMA (households that are more food secure have higher RIMA scores). Yet, its association with SERS is not statistically significant. If anything, the sign of influence is slightly negative. Supplementary Table 7 also suggests the relationships may be non-linear, with opposing signs for quadratic terms. We would, however, caution against the blanket conclusion that food security plays a negative (or no) role in people's subjective judgements. Indeed, annual food consumption, diversity of food intake and access to agricultural inputs all have strong positive associations with resilience (far higher than for RIMA in fact). Rather, it highlights that food security is multi-faceted, and that different elements are likely to interact with resilience in different ways. Our findings may also suggest the need for greater evidence and clarity in the heavy use of CSI as a proxy or food security in weighting objective models.
Another interesting disparity relates to the role of education. Higher education levels of the household head have strong positive associations with RIMA. This is reflected in the wider resilience literature where higher education is linked with individual-level behaviours supportive of resilience and heightened awareness of future risk (Pissello et al., 2017). Yet, surprisingly, we find that education has a slight negative association with SERS. This may again reflect differences in judgement between subjective and objective measures. However, we believe that geography and context may also be playing a strong role here.
Karamoja is an arid landscape frequently affected by drought. Nomadic livelihoods therefore have some advantages as they are able to relocate during times of hardship (Opiyo et al., 2015;Levine, 2010) -this is underscored by the fact that pastoralists have higher SERS scores than farmers. While formal levels of education may benefit households in the area, it is likely to provide little added benefit when compared with local informal and indigenous knowledge gained in coping with persistent drought (particularly once controlling for income or asset wealth). Interestingly, a similar lack of association between subjectively-evaluated resilience and formal education is observed across a number of other subjective assessments by Jones and Samman (2016); Béné et al. (2016); and Claire et al. (2018). Together this suggest that a greater understanding of the links between education and household resilience is needed before strong conclusions can be drawn. More can also be done to distinguish between the roles of formal and informal education in supporting resilience, including how they are reflected in resilience metrics.
Lastly, we consider the role of past shocks. It is commonly assumed that exposure to shock will reduce a household's resilience capacities, as over time repeated shocks wear away at a household's ability to deal with future risk (Silbert & Useche, 2012;Kahn, 2005;Ibarraran et al., 2009). However, findings from the Karamoja survey suggest that this relationship may be more nuanced. Households that have not experienced a shock in the last 12 months are associated with lower SERS scores than those that have experienced a shock. This trend runs true across a number of shock types, including droughts, floods and illness (not so for crop diseases).
While the relationships may seem somewhat counterintuitive, there are conceptual and practical grounds to consider it. Households that have experienced a recent shock may be in a better position to not only gain insights into relevant coping strategies, but better anticipate and adapt to future risks using the experiences gained in recovery (Berkes & Turner, 2006). Experiential grounding, with knowledge built up through experience of past shocks to inform more effective future strategies is well documented (Tschakert & Dietrich, 2010). Though we strongly suspect that any such advantages would only accrue in the context of smaller shocks or stresses; it is harder to see how acute threats would be advantageous. Indeed, this may partially explain why crop disease shows an opposing trend, as Karamoja has a long history of devastating locust outbreaks and disease-related threats (Gartrell, 1985) -though this trait is difficult to verify without follow-up research and information.
We also consider that changes in risk perceptions may be a factor, with the wider literature mixed on this issue. Wachinger, Renn, Begg, and Kuhlicke (2013) highlight examples such as Ruin, Gaillard, and Lutoff (2007) Scolobig, De Marchi, and Borga (2012) -point to prior experience as leading to beliefs that future events are unlikely to affect people and thus lower risk perception. Given the consistency of negative associations for SERS across a range of shocks, we see sufficient grounds to challenge many objective approaches (including RIMA) in being explicit in their justification of assumed links between prior shocks and resilience.
Above all, our findings point to the importance of recognising different sources of knowledge on resilience. The fact that associations with many socio-economic drivers overlap between SERS and RIMA is certainly encouraging. Again, it suggests that both modules are picking up on similar underlying properties. However, the extent (and in some cases the sign) of some associations clearly differ between objective and subjectively-evaluated modules. In such instances, it is important for evaluators to rigorously examine the evidence base for key assumptions. It also calls for a plurality of knowledge sources to be considered. This is particularly relevant for picking indicators that feed into objective measures.
c. Different variants of the SERS subjective approach point in the same direction Our final research question looks at how different variants of the SERS module compare. Fig. 5 shows that all three variants (3A, AAT and 9C) of the SERS model are highly correlated. The same is also true in comparing versions of SERS that focus on hazardspecific and overall resilience (see Fig. 6). We believe that these traits have important implications for how resilience is characterised. To date, resilience can (and has) been chopped up in myriad ways (as alluded to in Section 2). In fact, considerable time is spent arguing over the right mix of capacities constituting a resilient social system: whether adaptation is needed to recognise evolving risks ; whether transformation features, even if systems have radically altered (Pelling, 2010;Bene et al., 2012); and whether a whole host of other capacities and capitals, such a learning or anticipating, play distinct roles (Tschakert & Dietrich, 2010;Bahadur et al., 2015). An assumption therefore prevails that an evaluative tool's choice of resilience-capacities will have considerable implications for measured outcomes.
Yet, by comparing two popular resilience frameworks (3A and AAT), as well as a much larger set of 9 resilience-related capacities drawn across the wider literature (9C), we find very similar results -both with regards to correlations as well as associated socioeconomic drivers (Fig. 7). Interestingly, the fact that Absorptive, Adaptive, Anticipatory and Transformative have the highest loadings in the first principal component of the PCA suggests that the resilience literature may be pointing in the right direction (and that differences between the mix of these four are of little importance). Of course, the limitations of subjectively-evaluated measures, and the potential for the influence of known biases, have to be considered. However, these findings present a challenge to lengthy debates over the exact composition of resilience-related capacities (Bahadur & Pichon, 2017). It may also lessen the burden placed on choosing the 'right' mix of resilience-related capacities used in measurement approaches -whether subjective or objectively-oriented.
Our findings also suggest that outcomes from shorter SERS modules largely mimic those in full range of resilience-related capacities. This is of critical importance in efforts to save survey space and reduce the time needed in interviewing households. As pressure grows on resilience evaluators to design tools that are ever cheaper and quicker to administer (Tiwari, Skoufias, & Sherpa, 2013), we believe that short subjectively-evaluated modules offer some promise.

d. Limitations and routes forward
In reflecting on the implications of our findings, we highlight a number of important considerations. Firstly, we reiterate that neither RIMA nor SERS should be interpreted as 'true' measures of resilience. They are both indirect attempts at approximating a household's resilience, with no means of validating either without continually tracking wellbeing over time. Instead, our findings highlight the importance of diversity in sources of knowledge for resilience. Having said that, we are reassured to see that two largely independent ways of measuring resilience show moderate correlations, and that our subjective module is associated with many of the same socio-economic drivers. To be clear, this largely infers that SERS outcomes closely match the indicators used in the RIMA model (as the socio-economic drivers chosen here largely match those used in formulating RIMA). The fact that other drivers differ entirely in terms of sign and significance also provides helpful impetus: challenging evaluators to re-examine long-held assumptions, and encouraging evidence-based selection of indicators.
As with any one-off survey, findings should be interpreted with some caution. Given the subjective nature of the SERS module, a number of cognitive biases may be at play. For example, questions within the subjective module follow the same sequencing and may have resulted in cognitive fatigue or acquiescence bias. This is especially relevant given the strong overlap in wording of questions and response-items. In addition, both the overall and hazard-specific SERS modules were placed closely together in the survey. Respondents may therefore be primed in their responses. More needs to be done to explore the role of cognitive influences on subjective measures of resilience before firm conclusions can be drawn. Practices such as reverse coding of response-items and randomisation of question and module order may be important next steps, and are planned in future iterations of this research. More importantly, our findings point to the importance of encouraging resilience evaluators to be transparent about the merits and limitations of different approaches. For example, objectively-oriented measures (like RIMA) have the advantage of clear and comprehensive lists of standardised indicators. Yet, they struggle to account for factors that are not directly visible or tangible (Levine, 2014). Though subjective tools are by no means a silver bullet, they prove an alternative solution by giving individuals the chance to factor 'softer' aspects such as social capital, entitlement and power into their internal judgements of resilience (Maxwell et al., 2015;Jones & Tanner, 2017).
While some of these tradeoffs are relatively well known, others require further insights and careful research. One important question-mark for resilience measurement is how to deal with context-specificity (Zhou, Wan, & Jia, 2010). Objectivelyevaluated approaches tend to have fixed indicators and weights, meaning that two households are measured in exactly the same way. While this brings advantages of direct comparison, it does not account for the fact that the supporting traits of household resilience in one country might be completely different to those in another (Pelling, 2010). Subjective measures don't rely on proxies, and as long as people view resilience in similar ways, should provide a valid way of comparing resilience across differing contexts. Sadly, the assumption of uniform views on resilience (just like happiness) is a large one, meaning that cross-cultural comparisons should be treated with caution (Ungar, 2008;Selin, 2003). Some methodological practices, such as anchoring vignettes, do however offer hope in this regard (King & Wand, 2007).
We also see considerable potential for combining subjective and objective approaches. Building on the strengths of each approach, it is certainly possible to design measures that mix elements of both: whether matching subjective definitions with objective evaluations, or through combining the use of objective indicators with self-assessments. Above all, we encourage evaluators to build on these findings, and capitalise on the advantages that both objective and subjective measures offer in promoting more diverse and comprehensive approaches to resilience measurement.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.