Measuring Outcomes in a Community Resilience Program : A New Metric for Evaluating Results at the Household Level

Authors David P. Eisenman

Community Disaster Resilience Project (LACCDR).We describe how the index was constructed and then examine how resilience varies across communities in the LACCDR project at baseline, prior to implementation of the LACCDR program.The overall goal of this paper is to inform future evaluations of disaster resilience programs both in the U.S. and globally that aim to improve community resilience and measure outcomes at the neighborhood and household levels.
The Los Angeles County Community Disaster Resilience project is a metropolitan public health agency led effort to enhance disaster resilience at the community level through the operationalization of four theoretical levers of resilience: education, engagement, self-sufficiency, and partnership.As defined by Chandra et al., education ensures ongoing information about preparedness, risks and resources before, during, and after a disaster.Engagement involves including community members and promoting participatory decision making in planning, response and recovery activities.Self-sufficiency refers to enabling and supporting individuals and communities to assume responsibility for their preparedness.Organizational partnership involves increasing and enhancing the linkages and collaborations between government and non-governmental organizations (NGOs) and between NGOs in the community.In brief, sixteen communities were randomly assigned to receive an experimental community resilience program or a comparison program.Public health nurses trained community coalitions to utilize a resilience toolkit in the experimental arm.The comparison communities received training in traditional disaster preparedness topics, such as household supplies and emergency communication plans.The project's goals, development and methods are described in previous publications.
We needed new strategies and metrics to measure project outcomes in LACCDR.We used the Public Health Response to Emergent Threats Survey (PHRETS), a population-level survey periodically fielded in Los Angeles County to guide disaster planning, to be the vehicle for evaluating LACCDR outcomes at the household and neighborhood level.The PHRETS baseline instrument was rewritten to measure several domains related to community resilience.Each of these survey domains were selected a priori as outcome indicators aligned to the theoretical levers of community resilience of Chandra and colleagues.These domains included: household preparedness for disaster; participation in community resilience building activities; self-efficacy for helping in a disaster; perceived collective efficacy of the community in a disaster; perceived benefits of disaster planning with neighbors; social networks available in a disaster; and, civic engagement.Survey questions were drawn from published studies and national surveys, and adapted to fit this project's evaluation.
We collected baseline survey data from adult residents (≥18) of the sixteen LACCDR communities between June 3 and August 7, 2013.The address-based sample was selected to be representative of the sex, age and race/ethnicity distribution of the 2010 census tracts in each of the communities.The survey was administered by landline and mobile telephone in English, Spanish and Korean.The study protocol was reviewed and approved by the institutional review board of the Los Angeles County Department of Public Health.

Data Analysis
Prior to analysis, the data were weighted using raking ratio estimation.2010 census blocks were used to determine the population benchmarks for age, sex and race/ethnicity.We imputed missing values for the weighting variables using the modal responses for each community prior to raking.Raked weights were trimmed at the 2nd and 98th percentiles to decrease variance.Survey questions were scaled so that the responses ranged from zero to one.This scaling method provided an equal weight for each question in the analysis.Higher values represented agreement with community resilience questions, so that binary questions were scored 0 for "no" and 1 for "yes."Likert-scale questions were scaled so that the responses ranged evenly from 0 for responses that reflected strongest disagreement, to 1 for responses that reflected strongest agreement.We reverse coded all negatively keyed questions.
Principal component analysis extraction was used to help determine how many factors existed across the 18 community resilience variables of interest.We determined the number of factors using the Scree test and examining how many eigenvalues were greater than 1.Factor analysis with Varimax rotation followed.We retained questions and separated factors using a 0.40 factor loading cutoff.A Cronbach alpha value was generated for each of the factors in order to assess the reliability of the community resilience domains and the overall index.Summary variables were generated for each of the identified factors by calculating the averages across each of its measures and multiplying this value by 100% for ease in interpretability.The calculations followed the model of the National Health Security Preparedness Index.An example of this calculation follows: Factor 5 was comprised of three measures: x ="Planning with my neighbors will help my household after a major disaster." x ="I am confident I can be of help to my neighbors or community in the event of a disaster, such as an earthquake." x ="People in my neighborhood know how to work together to prepare and respond to a disaster." Where for each observation i, x , x , x = {0="Strongly Disagree"; 0.333="Disagree"; 0.667="Agree"; 1="Strongly Agree"} The summary factor score was calculated by taking the average of these two measures, and multiplying the result by 100: The result was a number between 0 and 100 for each observation i.
Summary variables for each of the factors were examined across each of the sixteen communities.An overall community resilience index was then calculated by taking the average of the composite scores for each of the factors.
If an observation was missing a value or had a response of "Don't know" or "Refused," then a score was not calculated for that observation.This method of handling missing responses, listwise deletion, has been used in another study similar to ours and is the default method for treating missing data in SAS version 9.4 (SAS Institute, Inc., Cary, North Carolina).We also conducted a post-hoc missing data analysis to evaluate whether there were differences in the demographic characteristics of missing and non-missing observations.Data from 4700 respondents were included in this analysis (35% response rate).Despite recruiting respondents from a representative address-based sample, the low response rate led to underrepresentation of males, younger respondents (ages 18-29 and 30-44), Hispanics, and Asians.Weighting of sample respondents according to the age, sex and race/ethnicity distribution of the 2010 census tracts helped enhanced the representativeness of the sample.Table 1 presents the socio-demographic characteristics of this sample population after weighting.The majority of respondents were female (52%), between 30 and 44 years old (31%), white (40%), possessed a college degree or higher (34%), earned a household income between $10,000 and $30,000, were married or living with partner (57%), spoke mostly English in the home (57%), and reported no health limitations (79%).

Results
Factor analysis demonstrated five separate factors, which together accounted for 46.71% of the explained variance (Table 2).Based on the items meeting the 0.4 factor loading threshold, the factors were characterized as (1) community engagement, (2) emergency supplies, (3) communication with neighbors, (4) civic engagement, and (5) collective efficacy.
Community engagement (Cronbach α=0.68) included measures of having "attended a community meeting where preparing for emergencies or disaster was discussed," "worked or volunteered to help your neighborhood or community prepare for or respond to a disaster or emergency," "worked or volunteered with a group of organization that focused on community safety, such as Neighborhood Watch," "attended a training to help others in your community in a disaster or emergency, like first aid or CPR," "attended a training in Perceived collective efficacy (Cronbach α=0.34) measured agreement with the following statements: "planning with my neighbors now won't help my household after an earthquake or other major disaster," "I am confident I can be of help to my neighbors or community in the event of a disaster, such as an earthquake," and "people in my neighborhood know how to work together to prepare and respond to a disaster."We built and tested a LACCDR community resilience index that allows us to quantify and understand baseline resilience at the neighborhood level.This index will allow us measure the impact of the LACCDR initiative and can also guide other public health agencies in their evaluation efforts.
Exploratory factor analysis demonstrated that community resilience can be divided into five distinct domains.These factors overlap with the Chandra and colleague levers used to guide the development of the survey, supporting the structure of their framework and grounding our community resilience in a theoretical framework.
The community engagement factor aligns to the Chandra's engagement lever and education lever.The emergency supplies factor aligns with the self-sufficiency lever as they both focus on personal preparedness.The communication with neighbors and collective efficacy factors map to the partnerships lever, which calls for The five community resilience domains in the LACCDR index have similarities and differences with measures used in other community disaster resilience tools.Leykin et al.'s Conjoint Community Resilience Assessment Measure, which was developed to examine resilience among Israelis, is based on leadership, collective efficacy, preparedness, place attachment and social trust.Like our results, collective efficacy describes perceptions of a community's ability to work together by including such items as "I can depend on people in my town to come to my assistance in a crisis" and "I believe in the ability of my community to overcome an emergency situation."Social trust and preparedness also overlap with some of our measures in collective efficacy and emergency supplies.Place attachment, on the other hand, introduces a resilience concept that describes one's psychological connection with the environment.While important to community resilience, place attachment questions were not included in our survey as they were less amenable to change through the LACCDR public health intervention.
Pfefferbaum et al.'s Communities Advancing Resilience Toolkit (CART) survey, which was not published at the launch of LACCDR, was developed as a community assessment tool that provides a "snapshot of strengths and challenges", not as an outcome measure that can be fielded in a pre-post manner.It categorizes community resilience into four domains: connection and caring, resources, transformative potential, and disaster management.
As CART focuses on the respondent's perceptions of his or her community's disaster resilience to disaster, it has measures overlapping with our collective efficacy domain, but does not capture participatory behaviors such helping the neighborhood prepare for disaster, attending relevant trainings, and talking with neighbors about self-sufficiency.
The LACCDR index has several benefits as an outcome measure.First, by including a variety of diverse measures, it captures the multi-dimensionality of community resilience.While some of the factors did not have the high measures of internal consistency, together they contributed to a more reliable community resilience index (Cronbach α=0.61).Second, we outline straightforward calculations that can be easily replicated.This will not only help us compare our baseline results to future survey data, but other community resilience programs can use it as a guide for evaluating their outcomes.Finally, we developed an index that is flexible.The overall resilience index is comprised of separate factors that can be assigned separate weights.Researchers may want adjust weights to accentuate factors associated with evidence-based outcomes or to evaluate programs that place greater emphasis on certain domains over others.
The LACCDR community resilience index used a targeted population and weighted sample of respondents, so that it is possible to get a good representation of the level of community resilience in each neighborhood.Understanding the different characteristics of each community lends itself to interpreting baseline resilience trends.For instance, Acton and Agua Dulce, geographically isolated towns in the high desert region of the county, possessed the highest scores for community supplies and collective efficacy.One possible explanation for these findings is that residents may be especially aware of their vulnerability in a disaster, which is supported by our experience in these communities.Stockpiling household resources, such as water, can be particularly useful if people are stranded in a harsh climate during an emergency.Furthermore, the community's isolation from other towns may foster a culture of preparedness among neighbors.
In addition to geography, socio-economic status may also be contributing to the high community supplies and collective efficacy scores in Acton and Agua Dulce.These communities possesses the highest median household income of all the participating communities.Past emergency preparedness research supports the positive relationship between income level and possessing household supplies and emergency plans.Culver City and La Crescenta, two other high-income areas, similarly display this trend.With the three wealthiest study populations also possessing the highest total community resilience scores, it appears as though the baseline results may be influenced by socio-economic status.
Watts possessed the highest scores for community engagement and communication with neighbors and also had a high score for civic engagement.Ever since the 1965 Watts Rebellion, when riots broke out over racial tensions surrounding police brutality, community members have been actively involved in civic organizations, including community empowerment programs and churches.
The existing infrastructure of faith-and community-based organizations may provide opportunities to build disaster resilience within this community.
San Gabriel was the neighborhood with the lowest total resilience score.This community is home to many Asian immigrants speaking a variety of languages.In this survey sample, 39% of San Gabriel residents indicated that they did not speak English or Spanish in the home.Language barriers may therefore be contributing to lower community engagement and communication with neighbors.

Limitations
Some limitations to this analysis should be acknowledged.The study sample was selected from sixteen Los Angeles County communities and may not be representative of the county as a whole.The low response rate may also reduced the representativeness of the target population due to systematic non-response from the survey sample.However, we attempted to address this bias by weighting the respondents according to the age, sex and race/ethnicity distribution of the 2010 census tracts in each of the communities.Applying population weights has been show to enhance the representativeness of the sample and reduce threats to validity caused by low response rates.
In addition, missing response analysis revealed demographic differences among respondents with missing values, especially for the items in the collective efficacy factor.Missing responses occurred more frequently among those who are female, 60 years of age or older, white, possess a college degree or higher, and predominantly speak English at home.Underrepresentation of these groups may have biased the results.This analysis illustrates how to assess disaster resilience at the community level.Using the LACCDR community resilience index, we describe baseline neighborhood measures that can be used to track changes over time.These factor scores supplement qualitative evaluations that have been conducted throughout the course of the intervention.Once follow-up data is collected, we can use this index to investigate the impact of the Los Angeles County Community Disaster Resilience initiative.

Conclusion Corresponding Author
http://currents.plos.org/disasters/article/measuring-outcomes-in-a-community-resilience-program-a-new-metric-for-evaluating-results-at-the-household-level/ David P Eisenman, MD MSHS, Professor of Medicine and Public Health, David Geffen School of Medicine at UCLA and UCLA Fielding School of Public Health, Director -UCLA Center for Public Health and Disasters.

Table 1 .
Sample characteristics of respondents after weighting, Public Health Response to Emergent Threats Survey 2013 psychological first aid," and "looked for information about getting prepared for a disaster, for example on the internet or by calling the Department of Public Health."Emergency supplies (Cronbach α=0.54) described having the following household resources: a 3-day supply of water per person, a 3-day supply of non-perishable foods per person, a 7-day supply of prescription medication per person, a household plan to reunite, and additional emergency supplies useful in a disaster.Cronbach α=0.67) included belonging to a "community organization (e.g.school, church or other faith, community or volunteer organization) that you can depend upon in a disaster" and participating in "meetings or activities with voluntary organizations or associations like school groups, churches or temples, community centers, ethnic associations or other social or civic organizations" in the past 12 months. http://currents.plos.org/disasters/article/measuring-outcomes-in-a-community-resilience-program-a-new-metric-for-evaluating-results-at-the-household-level/

Table 3 .
Community resilience scores for each neighborhood, Public Health Response to Emergent Threats Survey 2013