Resilience isn't the same for all: Comparing subjective and objective approaches to resilience measurement

Robust resilience measurement can improve our understanding of how people and societies respond to climate risk. It also allows for the effectiveness of resilience‐building interventions to be tracked over time. To date, the majority of measurement tools use objective methods of evaluation. Broadly speaking, these relate to approaches that solicit little, if any, judgment on behalf of the subject in question. More recently, subjective methods of evaluation have been proposed. These take a contrasting epistemological view, relying on people's self‐assessments of their own capacity to deal with climate risk. Subjective methods offer some promise in complementing objective methods, including: factoring in people's own knowledge of resilience and what contributes to it; more nuanced contextualization; and the potential to reduce survey length and fatigue. Yet, considerable confusion exists in understanding subjectivity and objectivity. Little is also known about the merits and limitations of different approaches to measurement. Here, I clarify the conceptual and practical relationships between objective and subjective forms of measuring resilience, aiming to provide practical guidance in distinguishing between them. In reviewing existing toolkits, I propose a subjectivity–objectivity continuum that groups measurement approaches according to two core tenets: (a) how resilience is defined and (b) how resilience is evaluated. I then use the continuum to explore the strengths and weaknesses of different types of toolkits, allowing comparison across each. I also emphasize that there is no one‐size fits all approach to resilience measurement. As such, evaluators should carefully consider: their epistemology of resilience; core objectives for measurement; as well as resource and data constraints, before choosing which methods to adopt.

Robust resilience measurement can improve our understanding of how people and societies respond to climate risk. It also allows for the effectiveness of resiliencebuilding interventions to be tracked over time. To date, the majority of measurement tools use objective methods of evaluation. Broadly speaking, these relate to approaches that solicit little, if any, judgment on behalf of the subject in question. More recently, subjective methods of evaluation have been proposed. These take a contrasting epistemological view, relying on people's self-assessments of their own capacity to deal with climate risk. Subjective methods offer some promise in complementing objective methods, including: factoring in people's own knowledge of resilience and what contributes to it; more nuanced contextualization; and the potential to reduce survey length and fatigue. Yet, considerable confusion exists in understanding subjectivity and objectivity. Little is also known about the merits and limitations of different approaches to measurement. Here, I clarify the conceptual and practical relationships between objective and subjective forms of measuring resilience, aiming to provide practical guidance in distinguishing between them. In reviewing existing toolkits, I propose a subjectivity-objectivity continuum that groups measurement approaches according to two core tenets: (a) how resilience is defined and (b) how resilience is evaluated. I then use the continuum to explore the strengths and weaknesses of different types of toolkits, allowing comparison across each. I also emphasize that there is no one-size fits all approach to resilience measurement. As such, evaluators should carefully consider: their epistemology of resilience; core objectives for measurement; as well as resource and data constraints, before choosing which methods to adopt. Ensuring that people and communities are resilient to climate variability and change is a key development priority. It is enshrined in flagship global accords such as the United Nation's Paris Agreement (UN, 2015a) and Agenda 2030 (UN, 2015b). This rise in policy interest has inevitably led to calls for identifying robust ways of measuring resilience across scales. The rationale is that accurate measurement can support more effective and targeted resilience-building interventions on the ground. Accordingly, myriad different Indeed, in some cases, authors argue that a complete transformation of a system may constitute, and be a necessary component of, a resilience process (Aldunce, Beilin, Howden, & Handmer, 2015;Kates, Travis, & Wilbanks, 2012). Evidence of the evolution of resilience over time can be seen in the changing nature of resilience within successive Intergovernmental Panel on Climate Change (IPCC) Assessment Reports (Table 1). While the Third Assessment describes a simple yet clearly defined concept that relates to maintaining the same system properties, the Fourth and Fifth Assessments have a much denser and all-encompassing definition. Intriguingly, these feature references to the capacity to adapt (AR4) as well as transform (AR5)-in apparent contradiction to the earlier definitions in the Third Assessment. By way of comparison, the IPCC's definition for adaptive capacity has seen little change over the same time period.
Needless to say, the definitional inconsistencies described are a considerable challenge to resilience measurement (see Nelson, 2011 andMcEvoy, Fünfgeld, &Bosomworth, 2013 for comprehensive descriptions). Despite this, a wide range of frameworks and toolkits have emerged in recent years aimed at both research and policy-making communities alike (Schipper & Langston, 2015).

| SUBJECTIVITY AND OBJECTIVITY IN RELATION TO RESILIENCE MEASUREMENT
Armed with a clearer sense of the concept of resilience and its evolution, we can delve into the distinctions between objectivity and subjectivity for resilience measurement. Here, I refer to measurement as processes taken to directly or indirectly measure a person (or wider system's) level of resilience. This can be done for a range of purposes including improved understanding of the properties of resilience and factors that cause it, or situational analyses used to determine the extent of a person or community's resilience. I also include efforts aimed at Monitoring and Evaluation (M&E), a subset of measurement that primarily seeks to evaluate the impact of projects and interventions. These typically make use of comparison between baseline data (measurements prior to an intervention) and endline-evaluations (measurements taken after the intervention).

| Objective modes of resilience measurement
Broadly speaking, objective methods can be thought of as independent of judgments arising from the subjects being evaluated (Cohen, Manion, & Morrison, 2002). In the context of resilience, objectivity can relate to a wide range of steps involved in the measurement process-from choices in definitions and frameworks, to how data is collected and used in quantifying resilience. For example, most measurement toolkits rely on frameworks for resilience that are based on expert-elicitation or the wider academic literature (Schipper & Langston, 2015). Such approaches are largely objective in sense that the resilience is externally defined; those being measured have little or no say in determining what constitutes resilience.
Objectivity also extends to the process of direct measurement. For example, many measurement toolkits include household or livelihood assets as one of the many proxies that feed into a resilience index (FAO, 2016;Frankenberger, Mueller, Spangler, & Alexander, 2013;Mayunga, 2007). The assumption being that greater levels of asset-wealth or diversity are associated with higher household resilience (Adger, 2000;Osbahr, Twyman, Adger, & Thomas, 2008). As such, enumerators are often asked to directly observe the subject's household-such as reporting on the type of building material used or counting household assets. This can be seen as objective in that measurement involves external observation on the part of the enumerator, independent of the subject's judgment. Resilience "Amount of change a system can undergo without changing state." "The ability of a social or ecological system to absorb disturbances while retaining the same basic structure and ways of functioning, the capacity for self-organisation, and the capacity to adapt to stress and change." "The capacity of social, economic, and environmental systems to cope with a hazardous event or trend or disturbance, responding or reorganizing in ways that maintain their essential function, identity, and structure, while also maintaining the capacity for adaptation, learning, and transformation." Adaptive capacity "The ability of a system to adjust to climate change (including climate variability and extremes) to moderate potential damages, to take advantage of opportunities, or to cope with the consequences." "The ability of a system to adjust to climate change (including climate variability and extremes) to moderate potential damages, to take advantage of opportunities, or to cope with the consequences." Distinctions are a somewhat blurred when it comes to the use of survey questions -the workhorse of most resilience measurement toolkits. Given that questions are posed directly to the subject, households surveys generally constitute self-assessments. Yet, the degree to which they are subjective or objective depends on the nature of the question(s) being asked. Those that solicit the subject's perceptions, preferences and judgments can viably be classed as subjective. Yet, for the most part, existing resilience toolkits rely on survey questions that are void of opinion. For example, use of a survey question such as "Has the head of household completed primary-level education" requires little subjective judgment on the part of the respondent. Moreover, provided the question is understood similarly by all, it should result in similar answers no matter which adult in the household is asked. Admittedly, however, some room for interpretation and judgment is always present-an issue I return to later.

| Subjective modes of measurement
While objective approaches to resilience measurement remain the norm, subjective modes have increasingly been advocated (Béné, Al-Hassan, et al., 2016;Claire et al., 2017;Jones & Samman, 2016;Jones & Tanner, 2017;Marshall, 2010;Maxwell et al., 2015;Nguyen & James, 2013;Seara, Clay, & Colburn, 2016;Sutton & Tobin, 2012). Subjective approaches take a contrasting epistemological view to objective methods. They challenge the notion that experts are best placed to evaluate other people's lives and have a better understanding of the factors that contribute to people's own resilience. Rather than relying on external judgment, subjective approaches consider the individuals in question to understand their own circumstances (Nguyen & James, 2013). At its simplest, subjective resilience relates to an individual's cognitive and affective self-assessment of the capabilities and capacities of their household, community or any other social system in responding to risk (Jones & Tanner, 2017).
Subjective assessments rely heavily on the measurement of perceptions, judgments and preferences (Maxwell et al., 2015), drawing on the conceptual and methodological advances made in related fields such as measurements of risk perception , psychological resilience (Connor & Davidson, 2003) and subjective well-being (Diener, 2006;Dolan & Metcalfe, 2012). These might include self-assessments of what resilience is, what factors might contribute to it, as well as whether or not people feel able to respond to current or future risks. If care is taken to design suitable methodologies and survey questions, a household's subjective resilience can, in theory, be readily quantified (Béné, Al-Hassan, et al., 2016;Maxwell et al., 2015).

| AN OBJECTIVITY-SUBJECTIVITY CONTINUUM FOR RESILIENCE MEASUREMENT
When it comes to resilience measurement, it is crucial to recognize that subjectivity and objectivity are neither binary nor mutually exclusive. Measurement approaches that are typically considered as objective will invariably have elements that are subjective in nature (and vice versa). Though nuances exist, at its simplest, subjectivity and objectivity can be thought of in relation to two core tenets:

How is resilience defined?
Objective approaches use external definitions of resilience (typically by the evaluator); subjective approaches allow the subject(s) in question to define resilience.

How is resilience evaluated?
Objective approaches are reliant on external observation; subjective approaches make use of a subject's judgments and self-evaluation of their own resilience.
In thinking through how the two tenets are related I propose that the relationship between subjectivity and objectivity is best thought of as a continuum. With that in mind, the objectivity-subjectivity continuum described in Figure 1 aims to aid researchers in identifying how particular modes of resilience evaluation draw on aspects of objectivity and subjectivity. It also helps to classify different types of resilience measurement toolkits, allowing strengths and weaknesses of different subjective and objective elements of measurement approaches to be readily identified.

| Subjectivity and objectivity in how resilience is defined
As alluded to, definitional ambiguities make the process of measuring resilience a considerable challenge. Researchers seeking to quantify resilience require clear specifications in how to define it (i.e., what is resilience?). This also extends to how resilience is characterized-the various characteristics that make up a resilience person or community (i.e., what does resilience look like?). These can include a range of different capacities including absorptive capacity, adaptive capacity, transformational capacity, and many others (Pelling, 2010;Schipper & Langston, 2015). Accordingly, in order to prevent confusion in references to both terms, I include characterizations of resilience within the concept of how resilience is defined.
The choice of whether to subjectively or objectively define resilience is principally an issue of epistemology. One option is to have a standardized definition of resilience that is externally determined and fixed-often through expert elicitation (usually by NGO staff or academics) or based on existing literature (i.e., the same framework of resilience is applied to everyone) (Schipper & Langston, 2015). Here resilience can be thought of as objectively defined and has little input from those being assessed, falling toward the right-hand side of the continuum.
Subjective definitions challenge this assumption. They operate on the basis that the respondent in question is better able to identify the factors that support their own resilience, and can be found along the left-hand side of the continuum. The distinction is important given that stakeholders have different understandings of how a system's resilience is derived. Herrera (2017) demonstrates this poignantly using the case study of food systems in Guatemala. Here, different stakeholders are asked to elaborate on aspects they considered important to contributing to local resilience: "While academics and delegates from the Non-Governmental Organization (NGO) participating in the study focused on enhancing virtuous cycles within the system, the central government delegates proposed solutions outsides the system's boundaries. All of these solutions, however, ignored the bounded rationality of the farmers and the premises of their decision-making process. Including only a few stakeholders in the process risks leaving many important aspects out of the scope of the analysis and therefore undermining its results." (Herrera, 2017, p. 14).
While there are approaches that sit at distant edges of the objectivity-subjectivity spectrum, in practice, most combine elements of both. For example, the approach adopted by the widely used Resilience Index Measurement and Analysis (RIMA) approach (D'Errico & Giuseppe, 2014) sits firmly within the objective camp when it comes to its definition of household resilience. RIMA has a standardized definition of resilience comprised of five separate dimensions (called "Pillars of resilience") and hundreds of individual indicators and proxies (FAO, 2016). Statistical analysis of household survey data is then used to weight these dimensions and compute an overall scoring. While surveyed individuals are neither asked to define what resilience means to them, nor what factors contribute to their own resilience, RIMA's approach can be seen to have some subjective elements. For example, the choice of each of the dimensions of resilience is based partly on extensive community-level engagements-people's perceptions and judgments of the factors that contribute to their own resilience-that then feeds up into the design of the overall framework (FAO, 2016). 1 At the other end of the continuum, subjectively oriented approaches such as Community Vulnerability and Capacity Assessment (CVCA) and Tracking Adaptation and Measuring Development (TAMD) make a strong point of using communities to self-identify the characteristics of their own resilience (Brooks et al., 2013;CARE, 2009). These are in turn used to formulate indicator-based scores and form the basis of their respective indexes. Yet even these methods can be seen to have some degree of objectivity: community-defined standardized characteristics are inherently external to the individual or household being evaluated and require some degree of aggregation (what constitutes resilience for one household within a community may not be the same as for all other households within it).
Things are further blurred when considering differences between the processes used to define resilience and the specific indicators used to measure them. To illustrate this, it is possible to conceive of an approach that has an externally determined characterization of resilience, and then asks households within a community to identify their own local indicators matching these predefined capacities. This mix of objective and subjective elements would naturally sit toward the center of the continuum and demonstrates not only the nonbinary nature of clarification, but the complexities associated with classifying different types of approaches.

| Subjectivity and objectivity in how resilience is measured
The second tenet that dictates the nature of subjectivity and objectivity relates to the mode of evaluation (the y-axis in Figure 1). Once resilience has been defined, evaluators must decide how to quantifying it. As there is no way of measuring resilience directly, objective evaluations often rely on proxy indicators of socioeconomic data. For example, the Livelihood Change Over Time (LCOT) approach (Vaitla, Tesfay, Rounseville, & Maxwell, 2012) uses "household food insecurity and access" as one of its characteristics of resilience. This is measured through use of a separate index made up nine individual survey questions related to the household's food intake (Coates et al., 2007). Each of these can be largely thought of as objective: they are externally verified and require little in the way of subjective judgment-neither on the part of the respondent nor on the surveyor.
This contrasts markedly with more subjective evaluations. Here, instead of using external observation, respondents are asked to self-evaluate their own levels of resilience using their judgment. The properties of interest are typically people's perceptions, preferences, and self-ratings of the status of their household or self (Maxwell et al., 2015). This is most commonly done using surveys that feature Likert scale response items (see Box 1 in Section 7) and draw heavily on similar tools in the assessment of subjective well-being, psychological resilience, and risk perception. 2 Again, it is difficult to conceive of approaches that fall strictly within objective or subjective categories. For example, many questions included in traditional household surveys can be thought of as objective in nature. For one, the LCOT assessment framework asks respondents the following question: 'In the past four weeks, did you worry that your household would not have enough food?'. In practice, however, any such answers will require the individual to internalize the question, interpret it according to their own understanding of the key concepts, and evaluate themselves accordingly. For example, the notion of "worrying" can be thought of as partly subjective, with the respondent prompted to define it as they see fit. Questions may also be affected by biases and heuristics that commonly affect survey responses such as priming, recall bias, and social desirability (Dolan & Metcalfe, 2012). Even relatively clear-cut objective indicators such as "a household's distance to markets (in kilometers)," an indicator used in the RIMA toolkit, still require some subjective judgments (FAO, 2016). For example, deciding on what constitutes a "market" requires a degree of internal judgment (either on the part of the evaluator or subject).
The points highlighted above underline the need for approaches in measuring resilience to be considered along a spectrum of objectivity and subjectivity, depending on which aspect of the approach one is focusing on. The objectivity-subjectivity continuum aims to assist in disentangling the processes that contribute to measurement outcomes.

| CLASSIFYING RESILIENCE MEASUREMENT APPROACHES
In order to apply the continuum in practice, I compile a list of 17 prominent evaluation methods (see Table 2). The list details toolkits extracted from a range of recent reviews that include both objective and subjective measures, namely Bours, McGinn, and Pringle (2014)   Purpose codes: UR = Understanding Resilience; PE = Project Evaluation; SA = Situational Analysis. c CVCA, CDRI, and TAMD are traditionally associated with qualitative assessments at the community-level; however, each approach can be readily adopted in a quantitative manner and at household-level scales. While TAMD focused primarily on adaptation, it makes repeated reference to resilience capacities and is easily translatable to a resilience context. resilience to climate variability and change at local levels, particularly those associated with individual-and household-level dynamics. Limiting the review to such geographic scales allows for far greater comparability and nuance; community-and national-level assessments often feature characteristics and indicators that differ markedly from those of localized evaluations (Adger, Arnell, & Tompkins, 2005;Vincent, 2007) and reviews of their core features appear more prominently within the academic literature (Ostadtaghizadeh, Ardalan, Paton, Jabbari, & Khankeh, 2015;Prior & Hagmann, 2014). Accordingly, each listed framework is screened for suitability against the review's primarily criteria, specifically: a main focus on disaster, climate or social resilience; an application at the individual or household level 3 ; and the ability to generate a quantifiable metric of overall resilience. It is important to note that this list is far from exhaustive. It aims simply to represent a body of widely applied methods, allowing for contrasting approaches to be identified and explored in detail. Using Table 2, I highlight the key differences in conceptual grounding and methods applied in the sections below (note that herein, references to the toolkits in Table 2 will relate to their abbreviated form, e.g., RIMA or MM07).
In gathering information on the 17 measurement toolkits it is also possible to group and place each directly onto the objectivity-subjectivity continuum. Doing so not only allows us to observe how different clusters of toolkits compare, but where gaps in our current application of measurement approaches exist. Figure 2 reveals the outcome of such an exercise. Assemblages of the various toolkits and the strengths and weakness of each quadrant are discussed below.
6 | CLASSIFYING TOOLKITS AND MAPPING THEIR EVOLUTION 6.1 | A brief history of objective methods of resilience measurement While Table 2 and Figure 2 reveal a plurality of frameworks, several common features are evident. For a start, it is clear that almost all early measurement tools are objective in nature, particularly when it comes to defining resilience. Indeed, each framework shortlisted in Table 2 (with the exception perhaps of CVCA and TAMD) has some form of predefined characterization of resilience used as a basis for evaluation. What this breakdown looks like, however, varies considerably. For example, several tools choose to adopt strict definitions of resilience guided predominantly by academic literature and theory. This is the case for World Bank's framework (WB15) that defines resilience as a household's change in welfare from pre-to postshock states over time (Alfani et al., 2015). The Weather and Climate-resilience Indexes have similarly strict definitions based largely on economic theory and the desire for a narrowly defined scope of assessment (Kimetrica, 2015).
FIGURE 2 Common resilience measurement frameworks along the objectivity-subjectivity continuum Notes: Cantril65 is a well-being framework used as a point of comparison discussed in text. Placement of the frameworks is meant to allow differences to be readily compared and carried out entirely on the basis of the author's own judgment in assessing toolkit handbooks Others choose more iterative processes for characterizing resilience. These may involve frameworks that are guided by the literature and technical experts, but also include some degree of input from communities of interest-often through the use of stakeholder consultations with local partners, focus group sessions, and key informant interviews. For example, the DRLA/ UEH Evaluation Resilience Framework and RIMA both adopt such strategies, choosing to combine external framings with some degree of localized input. Indeed, it is this latter category that is most prominent among the toolkits listed in Table 2. In many ways, this combines aspects of subjectivity into these assessments-a feature that I return to later.
Most objective tools opt to standardize: the same fixed framework is used for all households and individuals assessed. Once a characterization of resilience has been set, indicators are then assigned to each component of resilience allowing an overall score to be produced. For household-and individual-level assessments, these indicators will typically relate to key household assets or livelihood outcomes measured via a set list of externally variable questions and observations (MEL-CoP, 2016).

| The emergence of subjective methods of resilience measurement
Subjective tools have a much shorter history within the field of resilience measurement. While the early climate literature is replete with qualitative assessments that make use of subjective and perception-based methods (Buikstra et al., 2010;Gaillard, 2010;Miller et al., 2010;Twigg, 2009), few do so quantitatively. Perhaps the clearest first example of a standardized quantitative subjective approach are the methods devised by Marshall and Marshall (2007; hereafter MM07) for assessing "social resilience" within commercial fisheries. Using MMO7's approach, individual perceptions are measured in accordance with predefined sentiments such as "If there are any more changes I will not survive much longer," and measured using Likert scales.
Intriguingly, in the years that preceded MM07 few subjective toolkits emerged. Perception-based methods have, however, seen a revival within the social resilience literature in recent years with a suite of approaches developed in quick succession. Most notably, Nguyen and James (2013) devise a subjective model of household flood resilience that resolves around 10 attitude-based survey statements and the use of PCA. This is followed by a range of assessments by Lockwood et al. (2015), Jones and Samman (2016), Seara et al. (2016) and Béné, Al-Hassan, et al. (2016). Interest in subjective approaches looks likely to continue as more studies emerge, alongside guidelines for the use of subjective methods (Béné, Al-Hassan, et al., 2016;Claire et al., 2017;Jones & Tanner, 2017;Maxwell et al., 2015).
As evident from Box 1, many of the approaches listed in this review borrow heavily from related fields such as psychological resilience and risk perception. Indeed, in some cases it is difficult to make clear distinctions between the questions and methods of tools listed in Table 2 and Box 1 when compared with the common approaches in these neighboring fields-such as the Connor Davidson Resilience Scale (Connor & Davidson, 2003) or the Brief Resilience Scale (Smith et al., 2008) used in the assessment of psychological resilience. One main distinction is that the majority of tools used in the latter category place the individual psyche as the unit of assessment: how mentally equipped is an individual to bounce back from trauma or devastation? Those applied in the context of resilience on the other hand are frequently interested not just in the psychological components of resilience but the ability of individuals to draw on wider socioenvironmental networks: does an individual or household believe that they have the capabilities and resources needed to deal with climate variability or change? Nevertheless, the differences are inherently subtle and mean that drawing distinctions is a particular challenge. Approaches such as those used by Jones (2018) attempt to partially address this by focusing their subjective-evaluations at the household level (see Box 1).

| STRENGTHS AND LIMITATIONS OF COMMON MEASUREMENT APPROACHES
Deciding on whether to use subjective or objective methods is a process of neither right nor wrong. Rather, each type of approach will offer the evaluator certain benefits and drawbacks that need to be weighed up. Indeed, as underscored by the continuum, elements of both are likely to feature in any assessment of resilience. Thus, in attempting to guide evaluators in the choice of relevant toolkits I group key advantages and limitations into the four quadrants along the objectivity-subjectivity continuum. Figure 3 summarizes these main attributes graphically and these attributes are elaborated in the following sections.

| Objective characterization and objective evaluation
As evident from Figure 2, approaches that fall into the lower right-hand quadrant of the objectivity-subjectivity continuum (i.e., objective definition and objective evaluation) constitute the majority of existing frameworks. Reasons for this are numerous. Principal among them is that objective definitions allow for the same properties of resilience to be evaluated across households. When coupled with large sample sizes, evaluators can be somewhat confident that the same capacities are being measured: the same fixed definition and characterization of resilience will be measured in household A as for household B (though I return to whether they are capturing the same underlying properties in the sections that follow). Standardization of this sort also permits easy comparison across households and social groups. Moreover, it allows evaluators to hone in on specific capacities or indicators of interest. This is particularly relevant in cases where development actors may be concerned with the provision of targeted services, such as early warning systems focused on anticipatory capacity or social safety nets focused on absorptive capacity. Fixed framings of resilience of this sort enable evaluators to determine whether or not their interventions have an impact.
Toolkits in the lower right-hand quadrant of the continuum are, however, not without their weaknesses. For one, approaches that rely on objective evaluations require considerable amounts of socioeconomic data to be collected. Indeed, most frameworks in this category focus on the measurement of resilience-related capacities. These are inherently intangible, with no one indicator able to adequately capture the processes that make up any given capacity (Jones, Ludi, & Levine, 2010). As such, distilling a household's "coping capacity" to climate extremes down to a single indicator is not only challenging but in most cases misleading. Most objective approaches for evaluation therefore assign a large number of proxy indicators to each capacity, hoping to capture a range of different elements that relate to parts of the overall capacity (Constas et al., 2016). For example, the CCAFS15 framework measures "adaptive capacity" by identifying four "indicator dimensions," each evaluated using multiple individual indicators assigned to them. Inevitably, this not only requires judgment calls as to which mix of indicators is appropriate (both in terms of theoretical relevance and availability of data) but makes the processes of data collection lengthy and cumbersome (MEL-CoP, 2016). Resilience measurement questionnaires of this sort, such as RIMA, can therefore take multiple hours to carry out for each household interviewed given the number of questions and proxy indicators included within (FAO, 2016).
More importantly, by fixing the definition and indicators of resilience, evaluators risk undermining the validity one of their principle aims: interhousehold comparability. It is well known that the resilience of individuals and households is context specific and dependent on a number of socioeconomic and environmental factors (Adger et al., 2005). With that in mind, the factors that contribute to the resilience of a local trader in Nairobi, Kenya may differ considerably from those of a fisher in Mombasa. While the former may be heavily dependent on local market price volatility and the security of her available stock, the latter may rely more on the health of adjacent fisheries and ability of his boat to withstand turbulent seas during extreme weather events. Clearly, using the same set of indicators to measure the resilience of both individuals would be problematic. Thus, a key question for toolkits with objective characterizations of resilience is do the chosen characteristics and indicators of resilience truly reflect the resilience of each individual in question? Care must be taken with any cross-cultural comparison. Indeed, these types of approaches may be best suited to evaluate individuals and households that share similar contextswhether in relation to livelihoods, climate, or geography. Approaches in the upper right-hand quadrant of the continuum constitute the majority of existing toolkits associated with subjective resilience. They often involve fixed evaluator-defined definitions of resilience, and allow respondents to self-evaluate their resilience capabilities accordingly. Box 1 provides examples of self-evaluated response options from a range of different toolkits that fall under this category of assessment.
Though approaches such as JS16 and B16 have predefined characterizations of resilience (and can therefore be considered somewhat objective) they offer one way of addressing the issue of cross-cultural comparability. Instead of assigning fixed indicators to each characteristic of resilience, these toolkits allow the individual in question to self-evaluate themselves accordingly. In theory, given that the local trader or fisher are far more familiar with their individual circumstances and capabilities, and are being asked to weigh up the factors that support their own resilience, this method should generate scores that can be compared. Much of this relies on the assumptions that people have broadly similar understandings of what resilience constitutes and are in a good position to self-evaluate their own risk-profiles (Jones & Tanner, 2017). More importantly, the fact that the individual is factoring in their own knowledge of what makes them resilient and the risks that they face around them may imply that subjective methods are a more holistic and accurate reflection of the respondent's "true" resilience-particularly when compared to standardized indicators-based approaches that cannot take localized factors into account.
If the above assumptions hold true, then subjective evaluations may be of considerable relevance to project evaluators seeking to measure the impact resilience-related interventions over time. They offer a more bottom-up and participatory way of assessing resilience that places greater value on people's own understanding and judgment. Another key advantage relates ii. I am more likely to adapt to change compared to other fishers.
iii. If there are any more changes I will not survive much longer.

L15 (subset of 3 from 85 statements)
i. I am willing to try new things ii. In times of change I am good at adapting and facing up to challenges iii. I am still able to manage my property well during the tough times All statements employ a 1-5 scale, where 1 = strongly disagree and 5 = strongly agree.

JS16 (subset of 2 from 3 statements)
i. If an extreme flood occurred in the near future, how likely is it that your household could recover fully within 6 months?
ii. If extreme flooding were to become more frequent in the future, how likely is it that your household could change its source of income and/or livelihood, if needed? ii. If threats to my household became more frequent and intense, we would still find a way to get by All statements employ a 1-5 scale, where 1 = strongly disagree and 5 = strongly agree.

B16
i. With respect to [EVENT], how well do you consider you managed to recover?
ii. With respect to [EVENT], how well do you consider you did, compared to the rest of the community?
Response items for (i) are as follows: Not at all and I don't think I will be able to recover; Not yet fully recovered and it will be difficult/long; Not yet but hope very soon; Have fully recovered but it was long and painful; Have fully recovered and it was not too difficult; Have fully recovered and I am better off now. Response items for (ii) as follows: Did worse than most of the others; As bad as some people but better than others; Like most of the others; Did better than most of the others; Did better than anyone else. to survey length and duration. Many traditional toolkits described above require several hours of household surveying, owing to the need for large numbers of proxy indicators to match the multi-dimensional capacities of resilience. Contrastingly, a subjective evaluation can be carried out with just a handful of questions-through admittedly with potentially less detail and complexity (Jones & Tanner, 2017). This flexibility will not only help to reduce the burden of "survey fatigue," but also opens possibilities for greater innovation in survey delivering (Claire et al., 2017). More importantly, subjective evaluations can play an important role in holding NGOs and governments to account in relation to commitments and interventions; in theory, effective investment into resilience-building activities should be felt by those that are receiving support and having to deal with climate risk. 4 In practice, these assumptions should be considered carefully when interpreting the results of any subjective evaluation. For one, they depend heavily on people's interpretation of key definitions. "Resilience" is used in everyday language and means different things to different people. It also has varied meanings across languages and cultures (Crane, 2010). For these reasons, Jones and Tanner (2017) suggest that subjective measurements of this sort may not be suited to single-item questions, as is commonly found in the analysis of subjective well-being (OECD, 2013). Rather, subjective evaluation may benefit from breaking resilience down into questions that relate to easily communicable and translatable processes, such as: the ability to prepare for an upcoming extreme event; the ability to rebound quickly; and the ability to adapt to emerging future threats. This also avoids the need to use the word "resilience" in survey questions. However, by breaking resilience down into a set number of predefined components they also risk missing important contextual factors, undermining the point of subjective evaluations in general.
A further issue that needs to be considered is interpersonal and cross-cultural differences in responding to standardized questions. For example, two people might consider the circumstances of the same household (say a shared neighbor or their own community) and rate them very differently-from an objective point of view, they should derive the same scoring. One way of addressing both of these challenges may through be the use of anchoring vignettes (Hopkins & King, 2010). These are hypothetical narratives provided to people at the start of a survey. For example, an anchor might describe a household that has recently been affected by a drought and has had to sell off a number of livelihood assets as a result. People are then asked to rate the circumstances of the hypothetical person before rating themselves according to the same response scale. This allows for benchmarking of the responses and a way of explaining theoretical definitions of complicated concepts (King & Wand, 2006). It is worth noting that these methods have yet to be applied in the context of resilience measurement but offer promise in addressing some of its limitations.

| Subjective characterization and objective evaluation
Approaches that occupy the lower left quadrant of the continuum are those that seek respondents to define or characterize resilience themselves before subsequently evaluating them objectively. Here individuals (or members of the community) are asked to identify what makes them resilient. Objective indicators to track levels of resilience are then agreed upon and assigned using a similar consultative process. For example, a community might decide that the factors that contribute most strongly toward a household's resilience are levels of education and proximity to health care facilities. It can be said that while the indicators are subjectively chosen, the process of evaluating them are inherently objective: they involve external verification and little if any subjective judgment on the part the respondent. Note that it would certainly be possible for a measurement tool to allow individual households (or people) themselves to define resilience and objectively evaluate them accordingly. This would place the toolkit further toward the left-hand side of the continuum. In practice, few measurement toolkits have adopted this approach to date. Indeed, the TAMD approach is the sole example from among the shortlisted toolkits in Table 2.
Toolkits in the lower left quadrant of the continuum are particularly useful in trying to assess households in contexts where the factors that support resilience might be unknown or difficult to identify externally. They may even be applied in situations where characteristics of resilience change over time-a household could be asked to re-identify suitable indicators at each round of a panel survey for example. The advantage of such techniques is similar to those of all other subjective approaches: they provide a way of contextualizing resilience measurement and offer a more holistic conceptualization of resilience that factors in people's own knowledge of their capabilities and capacities.
Furthermore, objective evaluation helps to ensure that cognitive biases in self-reporting are largely (though not wholly) accounted for. The drawback is that the set of indicators chosen by one household (or community) may be wildly different from those of another. Evaluators that favor standardized approaches may therefore be reluctant to use these methods for cross-cultural comparison-though an argument could easily be made that self-selection of indicators means that comparison is in fact more robust. Getting households and communities to define resilience can also be very time consuming and requires excellent facilitation as the exercise is repeated constantly. This is perhaps the principal reason why so few examples of this type of approach exist-especially when applied at the household or individual level.

| Subjective characterization and subjective evaluation
The last group of toolkits relates to those in the upper left quadrant. Just like the processes listed above, households or communities are asked to subjectively identify the factors that constitute resilience from their own perspective. However, in this case, households are then tasked with self-evaluating themselves according to their own definitions. This could apply to one of two models depending on whether resilience is further broken down into individual factors. A household could, for example, identify that their ability to adapt to climate change is largely dependent on two qualities: their ability to make arrangements for their own financial security; and their ability to learn new skills in the labor market. A questionnaire could then be devised asking the respondent to answer the extent to which they agree or disagree with the following: "I believe that my household has made adequate plans for its financial security," or "Members of my household are able to learn new skills outside of the industry" (Marshall & Marshall, 2007). In some ways, this approach is similar to the methods used by the CVCA toolkit. CVCA relies heavily on community consultations to define resilience before applying mixed methods approaches for its evaluation. 5 A much simpler method would be to pose a single question asking people to subjectively measure their own resilience, such as "to what extent is your household able to adapt to the impacts of climate change." Here the respondent not only has to internally define what resilience means to them, but also has to consider the factors that contribute to it and self-evaluate themselves accordingly (Claire et al., 2017). There are few existing measures that adopt this single-item approach. Perhaps the closest is that used by Béné, Al-Hassan, et al. (2016) in deriving levels of subjective resilience, that ask "With respect to [a specified prior event], if it was to happen again in the near future how do you consider you would be able to recover?." The overall approach is very similar to commonly used approaches in the measurement of subjective well-being such as the Cantril ladder-inserted into Figure 2 for illustrative purposes (Diener, 2000;Kahneman & Krueger, 2006). It asks people to imagine a ladder with steps numbered from zero at the bottom (representing the worst possible life) to 10 at the top (best possible life), before stating: "which step of the ladder would you say you personally feel you stand at this time?" (Cantril, 1965). The ladder and other related methods have been used in a wide variety of contexts and have been shown to have validly in reflecting core components of an individual's quality of life (Diener, 2012;Helliwell & Barrington-Leigh, 2010). Indeed, measures of subjective well-being are beginning to have strong influence in guiding policy at national and international levels (Dolan, Layard, & Metcalfe, 2011;Layard, 2005).
There are certainly grounds to argue that a similar single-item approach to resilience measurement could be robust. After all, "happiness" or "life satisfaction" is as nebulous and diverse as the concept of "resilience." Claire et al. (2017) offer another such option that may hold promise. They ask respondents to assess how they expect to fare if they were to experience a range of self-selected shock events using a single-item question. However, many of the same limitations apply to single-item approaches as they do for measurements of subjective (OECD, 2013). For one, a single-item question would not allow an evaluator to isolate specific characteristics of interest-resilience is multi-dimensional after all (Nagoda, 2015). It would also be difficult to disentangle resilience to particular hazards unless explicitly stated in the question (as Claire et al., 2017 do with follow-up questions).
Above all, subjective approaches (of any sort) face the challenge of preventing survey responses from being affected by known cognitive biases and heuristics. Insights from psychology and behavioral economics show that effects such as the Peak-End rule (Tiberius, 2006), impact and retrospective bias (Durayappah, 2011) as well as hedonic adaptation (OECD, 2013) can each have a strong influence on how people subjectively assess themselves. Priming-a psychological process in which exposure to a stimulus can trigger a concept in memory that is then given increased weight in subsequent judgment tasks-is another key factor to consider (Lavrakas, 2008). While these issues are common to all social science surveys, with a wide body of existing literature, evaluators should take great care in deciding: how questions on subjective resilience are framed; the placement and order of questions on a survey; and the contextual environment within which respondents are being asked to answer questions.

| KNOWLEDGE GAPS AND AVENUES FOR FUTURE RESEARCH
Given the relative infancy of scholarly research on resilience measurement several key knowledge gaps remain.

| Differences in measurement approaches and the importance of transparency
Little is known about how different toolkits measure the same respondent: few like-for-like comparisons have assessed whether one approach generates results that are comparable to that of others. This not only applies to comparisons across subjective and objective categories of toolkits but even among toolkits that adopt the same general approach. Understanding the implications of using different characterizations of resilience and sets of indicators lists is important when considering the wide range of methods applied across available toolkits. Indeed, the diversity of definitions of resilience within the academic literature allows evaluators to justify almost any combination in the design of a toolkit (Olsson et al., 2015).
It is also important to recognize that different choices of measurement will lead to differences in measured outcomes: with the same households rated with higher or lower levels of resilience. For example, a framework that includes transformation in its definition of resilience may deliver wildly different results from one that does not. The situation is even more apparent with measurement approaches that employ subjective methods of evaluation (as each person may have their own individual interpretation). This has considerable repercussions for program M&E, and requires evaluators to be more transparent in the choices and assumptions made in choosing measurement tools-whether subjective, objective, or a blend of the two.

| Direct comparisons of objective and subjective modes of measurement
Establishing whether factors correlated with objective resilience match those for subjective evaluations is of keen interest. Early insights from studies by Jones and Samman (2016), Béné, Al-Hassan, et al. (2016) and Béné, Frankenberger, et al. (2016) show that many socioeconomic indicators are poor predictors of subjective resilience-posing a potential challenge to traditional objective assumptions. While more research drawing on a range of contexts is required for firm conclusions to be drawn, some reassurance can be gained from the well-being literature where subjective and objective evaluations show only weak relationships in many contexts and can correlate with different socioeconomic factors (Cummins, Eckersley, Pallant, Van Vugt, & Misajon, 2003). Clarifying the relationship between the results of subjective and objective evaluations will be a key to improving our understanding of the factors that support resilience. It may also serve to produce more holistic and robust methods of evaluation for development practitioners and funders.

| Understanding and accounting for cognitive effects
As subjective toolkits have only recently risen to prominence, and feature less extensively within the available literature, several avenues for future research exist. Understanding how cognitive biases and priming affect subjective evaluations of resilience is a useful start-drawing on the existing literature on survey methods and applications (Durayappah, 2011;Lavrakas, 2008;Tiberius, 2006). Doing so can allow evaluators to design more effective surveys and support the validity of subjective forms of measurement. It is also important to grasp the links between psychological resilience and subjective resilience. A further area relates to understanding the implications of structuring subjective questions differently. Present toolkits range from asking respondents to reflect on their resilience in relation to past climatic events (as per the "resilience index" in B16) or hypothetical future events (NJ13 and JS16). Toolkits also differ in the extent to which they explicitly measure a households' resilience to a specific climate hazard, or seek to evaluate their resilience to a range of unspecified climate-related hazards (as adopted by MM07 and Jones, 2018). Establishing the implications of these choices and appropriate occasions for their use is imperative, as results from either assessment are likely to differ considerably (Jones, 2018). Here again, much can be drawn from related literatures on subjective well-being and risk perception (Dolan et al., 2011;OECD, 2013).

| Recognizing context in resilience measurement
Clear gaps exist in understanding if and how subjective modes of measurement differ across contexts. For example, though subjective measurement tools have been applied in both developing (Béné, Frankenberger, et al., 2016;Jones & Samman, 2016;Nguyen & James, 2013;Waters & Adger, 2017) and developed (Lockwood et al., 2015;Marshall, 2010;Seara et al., 2016) country contexts, more can be done to understand how differences in environmental, sociocultural and cognitive factors shape people's responses. More importantly, while a wide body of literature exists exploring normative and societal influences on people's response to risk (Paton, 2003), as well as the importance of heuristics in decision-making and human behaviors (Kahneman & Tversky, 1979), few resilience measurement toolkits have accounted for these in their approaches and warrants further exploration.

| Exploring the potential to inform national and international policy processes
As resilience continues to rise up the political agenda, subjective and objective modes of measurement have an important role to play in a number of avenues. Firstly, as a way of mapping resilience. Communities and nations must have a thorough understanding of how resilience takes shape within their territories, and where hotspots of high and low resilience exist before being able to tailor suitable resilience-building interventions. It is here where identifying how objective and subjective modes of evaluation offer the most value in different contexts will be key. Importantly, measurement also provides an invaluable tool for assessing impact, and holding governments and other actors to account for public pledges-such as those made under the Paris Agreement and SDGs. For example, Target 1.5 under Goal 1 of the SDGs aims to "build the resilience of the poor and those in vulnerable situations" by 2030 (UN, 2015a). Yet, the indicators selected 6 thus far leave a lot to be desired, and are entirely objective in nature (at least from an evaluation sense). Opportunities exist in supporting more holistic assessments that take advantage of the benefits of subjective and objective modes of measurement, in tracking process against important international and national political commitments.

| Promoting innovation in resilience measurement
Lastly, there are plenty of opportunities for innovation in resilience measurement. This is particularly the case for tools that do not rely entirely on household survey data. For example, evaluators may wish to assess resilience on the basis of individual or household behaviors (rather than stated preferences or capacity assessments). This may help to overcome some of the cognitive biases that affect survey responses, as well as social desirability bias and Hawthorne effects (where respondents' awareness of being observed affects their responses). Interesting insights are also being learned from innovative uses of Big-Data and mobile technologies, such as using call data records to track the movement of individuals during cyclone events by organisations like Flowminder (Lu et al., 2016). Using these methods to validate other forms of resilience measurement, and finding way of integrating subjective elements into these new approaches may offer useful ways of complementing traditional forms of measurement.

| CONCLUSION
Interest in resilience measurement continues to grow, thanks in large part to a growing community of measurement practitioners, such as the FSIN network and the Rockefeller Foundation's Resilience Measurement Community of Practice (MEL-CoP). The subject is also receiving increasing amounts of financial and technical support from traditional development funders eager to track the effectiveness of their resilience-building investments. While these advancements are inherently positive, ambiguity and confusion persist regarding the merits and limitations of different measurement approaches. More recently, the advantages of subjective approaches have been trumpeted with a growing number of studies focusing of perception-based methods.
In this paper, I have provided greater clarity in understanding the distinctions between objective and subjective ways of assessing resilience through the development of a subjectivity-objectivity continuum. The continuum highlights two core tenets, consisting of (a) how resilience is defined (whether by the evaluator or the subject being observed) and (b) how resilience is measured (whether by external observation or internal judgment). I have also showcased the assumptions and weaknesses of toolkits associated with the four quadrants of the subjectivity-objectivity continuum.
In highlighting the use of different approaches, it is important to emphasize that there is no one-size fits all approach to resilience measurement. Evaluators should ultimately consider a number of factors before choosing which toolkit to adopt, including: their epistemology of resilience; core objectives for measurement exercises; and resource and data constraints. Most importantly, it is hoped that clearer insights into the relationship between subjectivity and objectivity can guide more holistic conceptualizations of the nature of resilience.
Note that a number of toolkits have multiple scales of application, including some that are predominantly associated with community and national assessments. Toolkits were included in the list if they featured identifiable characteristics at local levels and where methods were readily applicable at the individual or household scales. 4 This assumption may apply more to investments in tackling current risk profiles (particularly covariate risk) as the impact of those addressing longer-term or idiosyncratic shocks may not be felt by recipients for a long-time to come. 5 Strictly speaking, CVCA is typically used in the context of qualitative analysis. However, its methods can be readily applied to quantification and is largely used for illustrative purposes in this example. 6 See https://sustainabledevelopment.un.org/sdg1

RELATED WIREs ARTICLES
A review and classification of analytical methods for climate change adaptation The dynamics of vulnerability: why adapting to climate variability will not always prepare us for climate change