Slum upgrading in Brazil: lessons from evaluation processes

BALBIM, Renato; KRAUSE, Cleandro. Slum upgrading in Brazil: lessons from evaluation processes. Revista C&Trópico, v. 43, special edition, p. 185-202, 2019. DOI: 10.33148/ CeTrópico_v.43n. esp(2019)p.177-195 Slum population has had a relative decrease during the 21st century in Brazil. Beginning in the 1990’s, the Brazilian public policy towards slums has been notable, both at national and local levels. Nevertheless, priority has been given to improvement works, evaluation of their results has been a neglected aspect, and a knowledge gap between investments and their impacts has broadened. This paper aims at summarizing the contributions of researchers at the Institute for Applied Economic Research (IPEA), who conducted slum-upgrading evaluations from 2009 until 2014. They comprise methodologies and procedures for a meta-evaluation of intervention projects as a means to improving programs; a logical model for the evaluation of the intervention in one of the largest favelas in the country; an evaluation guideline proposal for a nationwide precarious settlements urbanization program; and relevant findings of the focus group technique as a qualitative research method. The paper seeks to publicize the results achieved in an effort to advise governments and improve their policies; to foster discussion on tailored evaluation methodologies; to contribute critically with international development agencies; and to reinforce the exercise of citizenship through evaluation practices, transparency, and accountability.


Introduction
One in eight people, or an estimated one billion globally, live in slums (UN-HABITAT, 2016). Slums are comprised of populations suffering from at least one housing deprivation: lack of access to improved water supply and sewage, insufficient living area, housing durability, and tenure security (UNITED NATIONS, 2015). The Millennium Development Goals (MDG) characterize living conditions and housing stock in order to evaluate them independently of local slum denominations. The 2014 MDG Report found that 29.7 percent of the urban population in developing countries suffer from at least one housing deprivation (UNITED NATIONS, 2015).
While the UN Report indicates that absolute numbers continue to grow, slum dwellers population share in developing countries actually decreased from 39.4 percent in 2000 to 29.7 percent in 2014 (PURWANTO et al., 2017). The Brazilian MDG Report validates this improvement showing a relative slum population decrease in Brazil from 47.6 percent in 2001 to 36.6 percent in 2012 (Brazil, 2014), above average for developing countries.
Similar to other developing countries, slums in Brazil have several configurations, including favelas, irregular and clandestine land subdivisions, derelict housing complexes, and other local forms and nomenclatures. The first favelas census in Rio de Janeiro, formerly the Federal District, was conducted in the early 1950s, prior to the construction of the "subnormality" concept and information collection (CARDOSO, 2016). Subsequently, each decade reveals a demographic census with updated surveys. Since the 1991 Census, the term "subnormal agglomerates" has been used to designate settlements lacking public services with irregular land tenure and with disorderly and dense occupation (Brazil, 2014).
In 2010, 11.4 million people, or 6 percent of the country's population, lived in subnormal census tracts. About half of them resided in the Southeast, where the largest metropolitan areas are situated: Sao Paulo and Rio de Janeiro. Thus, the role of cities in the country's urban network is a spatial factor in determining subnormal agglomerates location. The metropolises higher in the urban hierarchy concentrate economic production, jobs, and the majority of the subnormal agglomerates as well (Brazil, 2011).
Slum upgrading or "reurbanization of favelas", as it is referred to in Brazil, originated in the 1980's with metropolitan cities initiatives -Projeto-Rio (1979) pioneering the Favela-Bairro program in Rio de Janeiro (1992), PREZEIS in Recife (1987), Santo Andre (1989) 3 , Belo Horizonte (1983) 4 , Sao Paulo (1980) and Diadema (1983. These cases have combined the provisions of essential public services, accessibility, and housing improvement with the extensive land regularization goal requiring lengthy legal proceedings. However, in Brazilian slums, lacking legal titles does not restrain public urban works. Indeed, priority is given to upgrades and improvements. This practice runs contrary to those advocated and supported by international agencies such as the World Bank in other developing countries under the name of "bankable slums" (JONES, 2012).
This strategy was reviewed by the Bank in "Thirty Years of World Bank Shelter Lending Report. What Have We Learned?" (2006, p. 28-31). In this case, the World Bank disputes the DeSoto finding that titling is necessary to financially "unlock" urban land capital citing the ability of several countries to create alternative tenure security instruments. Based on all the difficulties and delays in procurement procedures and the need of a financial system to convert the securities into capital, the Bank evaluation questions the gains associated with the land regularization investment: [...] depending on the existing constraints, there are a variety of tenure instruments that can be employed to convey property rights or freedoms. In addition, because many of these instruments do not require prior physical planning, infrastructure servicing, and surveying of the settlements, they are often an advantageous strategy from the perspective of widespread coverage at reduced costs (WORLD BANK, 2006, p. 31). Following the closure of the National Housing Bank (BNH) in 1986, urban policies began to reorganize first at the local level and subsequently at the national level in the late 1990s under the strategy developed at Brazilian municipalities that use different tenure instruments to convey land rights and urban upgrading.
3 About the awarded upgrading slums programs developed and implemented by Santo Andre's city, see: Denaldi, 2004. 4 "The land use planning and legislation now recognize the existence of favelas and provide for their consolidation through the adoption of urban patterns that are different from those applied to the formal city. Instruments are also applied to recognize land tenure, such as the CDRU (Concession of Real Right of Use) and the Collective Usucaption. In 1983, the municipal government of Belo Horizonte instituted PROFAVELA (Regularization of Favelas Municipal Program); in 1983 and 1987, the municipal government of Recife established the ZEIS (Zones of Special Social Interest) and PREZEIS (ZEIS Regularization Plan), respectively." (Denaldi, 2003:71) At the local level, the slums upgrading actions of the 1980's and 1990's were supported by the World Bank (i.e. the 'Rio Grande' project and the Environmental Sanitation of the Guarapiranga Reservoir in Sao Paulo, both in 1992). In 1993, the Inter-American Development Bank (IDB) financed the Favela-Bairro in Rio de Janeiro and, in 1994, the Programa Singapura in Sao Paulo. At the end of the 1990s, the convergence between the Brazilian programs and the guidelines of the international agencies was greater with the "Habitar Brasil BID, " (HBB) Federal program.
This pioneer program, HBB, invested in slum upgrading at the national scale. Combined with resources from the IDB and federal budget, HBB integrated implementation monitoring and evaluation processes including results and program evaluation into its principles and guidelines. International agencies program evaluations, however, are closer related to financial application effectiveness than to policy specific impacts and results (WORLD BANK, 2006;IDB, 2012;2014). As discussed in more detail below, HBB's evaluation guidelines comprised its interventions individually like "case studies", while conversely, the program evaluation was restricted to measuring only its financial effectiveness, something similar to a monitoring process.
It is important to emphasize that policy evaluation is not common practice in the Brazilian public sector. In policies with more developed evaluation processes, such as health and education, the results are used to improve services and accountability. Unlike slum upgrading processes, the evaluation in both cases tends to be simpler since the service is provided directly to the people subject to and part of the service assessment quality examination.
Aside from sectorial particularities and Brazilian public policies' challenges, evaluation processes in general have made progress in the first decade of the new millennium (NOGUEIRA; CAVALCANTE, 2009). However, the traditional urban policies' evaluation -e.g. housing, mobility and sanitation -does not differ from the inconsistency of planning and investments marking the public sector and therefore must be considered under such constraints.
However, since HBB began its activities in 2001, Brazil has made significant improvements in slum urbanization policies. Specifically, the significant and increasing financial and technical investments made under Priority Investments Policy (PAC) from 2007 to 2014, enabled considerable scale gain, development, and in some cases, consolidation of adapted methodologies and guidelines. The two phases of the PAC Slum Upgrading totalized 33.5 billion Reais (17 U$ billion) invested in 3,528 precarious settlements throughout the country. Ultimately, the PAC facilitated the disassociation of Brazilian slums upgrading programs from the international agencies' goals.
Public agents prioritized the production and improvement of housing over evaluating programs. This resulted in a knowledge gap between financial investments and induced impacts. Opportunities to combine public investments with community efforts toward improved living and housing conditions have also been missed.
From 2009 to 2014, researchers at the Institute for Applied Economic Research (IPEA) addressed theoretical and practical efforts for the development and application of innovative and tailored methods of slum upgrading policies, programs, project evaluations, and meta-evaluations in Brazil. In an effort to collaborate toward structuring a methodologically qualified and committed debate for improving public policies and social participation, this article evaluates specific innovations organized into four sections, each devoted to a specific set of methodologies and procedures, followed by concluding remarks. HBB meta-evaluation was the first developed and applied methodology followed by the PAC intervention evaluation logical model of Complexo do Alemão (one of the largest favelas in Rio de Janeiro). Additionally, methodological considerations were provided for the Precarious Settlements Urbanization Program of the Ministry of Cities evaluation guidelines proposal. Finally, relevant aspects that emerged during focus group sessions conducted at Complexo do Alemão were used to corroborate with concluding remarks.
This report serves multiple purposes: to publicize the results achieved in an effort to advise governments and improve public policies; to foster discussion on tailored evaluation methodologies; to contribute to similar efforts by banks and international development agencies; and to reinforce the exercise of citizenship through evaluation practices, transparency, and accountability.

Meta-evaluation: How to enhance a program by evaluating intervention results
It is important to distinguish program evaluation from individual intervention outcome evaluation. When stressing the need for tailored evaluation methods, we refer to the need to adequate or even overcome programs which were developed beyond the beneficiaries' reality; programs which had responded to external goals and excluded fundamental local conditions for their own improvement. It is in this way that the shortcomings of the HBB Program pioneering evaluation method should be resolved.
The HBB evaluations demonstrate how international development agencies prioritize case studies analysis, without concern for structuring adapted policies and programs that are capable of overcoming structural problems. Some IDB synthesis document analysis assessing program performance clarify that "evaluations" are only required to projects and interventions, in other words, cases. Evaluation at the level of policies corresponds to the Bank's goal: budget execution, "Bank's social housing operations effectiveness evaluation", and risk analysis (IDB, 2016;2014;MAGALHAES;VILLAROSA, 2012;MAGALHAES, 2016). In this case, the program in its entirety did not have its own evaluation methodology, but instead, the program presented only a methodological model to assess each isolated intervention.
Therefore, a methodology was developed to evaluate the program from the set of the interventions evaluations, plus the assessment of each applied guideline and also the evaluation process itself and the set of all these documents together, thus establishing a meta-evaluation. This all happened in 2010 when the Brazilian Ministry of Cities (MCidades) National Housing Secretariat expressed intent to better understand HBB efficiency and effectiveness. Subsequently, and because of the impossibility of doing so based on the available instruments provided by the program evaluation guidelines, IPEA developed a detailed meta-evaluation methodology of ex-post evaluations of slum upgrading projects in Brazil. In late 2010, municipalities and states that were implementing HBB's projects submitted 27 evaluation reports to MCidades. These documents were prepared in compliance with the guideline established in 2004 called "Pilot Projects of Investment/ Intervention on Favelas: Indicators Matrix for Post-Occupancy Evaluation". This matrix was developed by Brazilian stakeholders with the support of IDB and MCidades to achieve Bank guidelines.
The reports were analyzed by IPEA researchers initially to elaborate a statistical synthesis of results, allowing the Brazilian federal government to quantitatively evaluate the program. The goal was to develop an HBB program meta-analysis considering IDB guidelines, which only requested project evaluation and program monitoring. However, the analysis indicated that this objective was unattainable due to format diversity and adopted methodological procedures. This diversity was predictable considering MCidades accepted the inclusion of new indicators by evaluators at the local government level.
The initial research limitation resulted in a stimulus to a broader recommendations proposal evaluating favela urbanization, with meta-evaluation, a form of quality control applied to each evaluation and to the whole evaluation process (BALBIM et al., 2012).
At the time of this research, PAC's slum upgrading projects were already underway. Upon completion, they would be evaluated according to the current HBB standard. Consequently, procedure evaluations and revisions were urgent.
Meta-evaluation is a complex methodological procedure that seeks to analyze evaluation process components, including the evaluator and specific context assessment. According to Scriven (1991), who pioneered the concept, meta-evaluation represents both an ethical and a scientific commitment, since people's well-being is a factor for evaluation. As an ethical commitment, the qualitative nature of meta-evaluation requests value judgments, whose criteria must be predefined and disclosed.
For the purpose of this research, education evaluation criteria were adopted. This choice was made because education policies have been evaluated for a long time and in several countries, including Brazil, which makes the use of methodological advances in a more critical and tailored manner possible. In the United States, the Joint Committee on Standards for Educational Evaluation (JCSEE) has embraced evaluation criteria since 1981, and its broad acceptance is attributable to specificity and clarity of the methods and structures that they have proposed over the years (BUSTELO, 2002). We used the JCSEE's criteria based on the landmark book by Worthen et al. (1997) which includes 28 evaluation criteria, organized according to categories of utility, feasibility, propriety and accuracy, adapting them to the requirements and recommendations of the HBB / PAC evaluation standard.
Research led to synthesizing the results which could describe and evaluate HBB achievements. An instrument was developed broadening the evaluation function understanding in the policy cycle 5 , improving the standard and the HBB / PAC evaluation process.
The first research stage focused on the evaluation process quality. There was significant diversity in report form and contents; particularly variances in information sources and data collection procedures among reports. This is an interesting finding given the HBB had standard required information. Additionally, several reports resembled accountability documents rather than evaluations, which should disclose value judgments. This explains why only seven reports (26%) were considered valid at this research stage in leading into the analysis sequence.
The second stage was devoted to indicator matrix analysis included in the HBB evaluation standard, comprised of three axes: "housing and urban insertion" 6 , "social inclusion" and "resident satisfaction" supporting the "main final dimensions", defined as "effectiveness and sustainability of projects in post-occupancy". Each axis was divided into "dimensions", then subdivided into "indicators", with corresponding "descriptors" (BRAZIL, 2004). Due to their generic nature, the descriptors allowed for subdivision into measurable items, creating a more detailed matrix level. This diverts from existing IPEA researcher knowledge and public policy evaluation literature that the indicators (parameters or values) should correspond to the most detailed level.
The HBB evaluation matrix standard measurable items list is comprised of 192 items, associated with 100 descriptors. However, as local evaluating teams included new indicators, the total measurable items, including those in the evaluation reports, increased to 263, an excessive number that did not allow for concise and comparable assessments.
Measurable items were initially evaluated for adequacy, the ability to effectively measure what they describe, portraying the degree or magnitude in which some expected transformation of the reality occurs, i.e., being accurate, including the source eventually determined by the evaluation standard. Measurable items were also evaluated for their feasibility, the relationship between cost and benefit of assessing them in a given context, referring to the existence of a source of information, ease of access and its frequency. The assessment of adequacy and feasibility is indicative of the easiness and difficulties that the selected reports may have faced in order to follow the guidelines of the evaluation standard.
Of the total measurable items presented initially in the matrix, approximately one quarter were considered adequate and about one half, feasible. The former, smaller numbers indicate that no guidelines existed in the standard regarding data collection sources.
The meta-evaluation research resulted in general recommendations that were incorporated into a slum upgrading evaluation policy further presented.

Reverse engineering, or how to evaluate what was not planned: a slum upgrading logical model
In an effort to establish evaluative models of slum upgrading policies in Brazil, an evaluation matrix was tailored to what may have been the most emblematic PAC's intervention, the "Complexo do Alemão" favela in Rio de Janeiro. It is worth mentioning that this research was carried out through direct and frequent contacts with the PAC's managers and formulators, as well as with the local Caixa's (Brazil's bank for urban development) technical teams, in addition to the state and municipal governments.
According to the information presented, the Urban Intervention in the Complexo do Alemão (IUCA) was intended to address both the lack of infrastructure and the socioeconomic causes that contribute to it being a segregated neighborhood. A total of US$ 420 million was expended; of which 96% was attributed to physical works, including the construction of a cable car with a 3.4 km line and six stations. The remaining four percent of the investment was applied in land regularization actions (over 27 thousand households) and social work for the impacted residents.
The initial research posed a fundamental question for the evaluation: what issue(s) motivated the intervention, and what goal(s) should it achieve? Local teams reported anecdotal facts on the decision-making process that would have resulted in Complexo do Alemão, and other slums, to be included into PAC interventions. These cases suggest certain arbitrariness, or at least the strong incidence of political conjuncture in the decision-making.
It was current the version according to which the president of the republic and the governor of the state would have flown over the Rio de Janeiro slums in a helicopter and then decided which ones would be selected to make part of PAC. The fact is that three very large favelas were chosen -besides Alemão, also Manguinhos and Rocinha -, remembering that previous favela urbanization programs, such as Favela-Bairro, had given priority attention to smaller intervention areas.
The absence of an integrated project and lack of knowledge about intervention fundamentals led researchers to propose a logical model. Researchers applied reverse engineering, a technique that seeks to rescue the project theory, allowing for investigation to begin with the problem that led to the elaboration of the intervention proposal and planning. Stakeholders' perspectives (political, technical and social) were considered to contemplate the various aspects contributing to shaping the central problem. This procedure allowed for validation of proposed alternatives and impact summarization.
The methodology developed to evaluate the Complexo do Alemão intervention innovates not only by applying the logical model to a specific project (versus a program), but importantly, constructs and details a justification for decisions and resource allocation. This enables indicator proposition associated to a "theoretical" project and accomplishes an effective evaluation. This method is fully adapted to the public policy needs and shortages in Brazil as well as in other developing countries, including policy formulations that do not expose the expected "bets" involved in their causal relations.
The logical model built for IUCA relates resources (financial, material and human inputs), actions and products (outputs), which together contribute to the intermediate and final results. Intermediate results are associated to problem causes (they linguistically "transform" causes), and the final result is associated to the central problem at which the intervention aims. These linear connection chains simplify complex realities and are also useful to demonstrate program or project expectations while seeking desired results.
The logical model is equally as important to evaluate as the foundation of a reference, contract, plan or a project that establishes stakeholder expectation consensus. Therefore, the logical model minimizes divergences of evaluated program or project evaluation design, result interpretation, and improvement recommendations. The logical model also includes "intervention impacts" (indirect achievements not to be confused with direct results), and the "context factors identification", which may facilitate or hinder results achievement.
Once the IUCA logical model was developed and validated, the monitoring and evaluation indicators were then defined. Indicators relate to items in the logical model: products, intermediate results and final result. Simple product indicators compare the executed value with the programmed value. To verify, quantities are compared between those included in the project with those verified in work performed.
Result indicators verification occur immediately following the intervention or after a certain time. They compare a baseline and a post situation as originally designed by formulators. This transformation is attributable, totally or in part, to the products created.
The The IUCA logical model's final result statement called for "Complexo do Alemão to be integrated into the city including community public space appropriations and dwellers that exercise full citizenship" (BALBIM et al., 2013). The evaluation design proposed 12 indicators to gauge the multi-faceted outcome for verification in the territory of Complexo do Alemão and in its relationship with the city. For example, the performance measured by the Social Development Index (IDS), periodically calculated by the Rio de Janeiro municipal government; the income and education level of the head of the household, and the existence of connection to the general sewage network; the Complexo do Alemão non-residents' frequency to its new equipment including the cable car (demonstrating the reciprocity of integration); and the emergence of territory-based candidates for elected city council positions.
The research resulted in a Complexo do Alemão video documentary and book (RODRIGUES, 2016) combining texts by researchers, residents, public servants, and others. IPEA researchers involved in formulating the IUCA logical model, however, did not apply the evaluation matrix. Other public actors involved in the intervention, whom could apply the methodology or at least enforce it, didn't use the evaluation matrix either. This probably reveals the relative negligence of governments in carrying out evaluation processes that risk obtaining likely unexpected or negative outcomes.

A program guideline proposal
The Complexo do Alemão experience and HBB meta-evaluation required advancement of a new PAC interventions evaluation standard proposal. The challenge was to produce a feasible methodology for a diverse set of projects that allowed for case comparisons and improvement of the policy at the national level (BALBIM et al., 2013).
Reverse engineering was applied to a proposed project design and financeable items listed in the MCidades contracting norms. The result was a logical model of a complete urbanization intervention, a tailored, unique and complete radiography of the most important national slum-upgrading program. Such a logical model is fundamental for public managers to recognize the totality of eligible actions; providing a clear notion of the dimensions of the program and the logical connections of its actions.
A key finding was that a product designed, executed and delivered to the beneficiary at the end of the intervention does not necessarily result in the transformation of a reality. Overcoming a problem is a result of interactions between delivered products and intervention context, or fundamental conditions and particular characteristics defining interactions.
Considering the challenge of measuring changes, which are greater than the simple measurement of the delivery of certain products, researchers created immediate result indicators reflecting intervention effectiveness, in other words, its transforming character expressed in the outcomes. These indicators combine results satisfaction rates with product delivery rates.
Some intervention contexts can be derived from public information gathered at the national level (e.g. the demographic census or the annual household sample research); however, others are specific to each intervention. In the Brazilian upgrading slum policy case, the variety of data sources throughout the country is great, and the same applies for the difficulty of accessing and producing project-specific data.
Researchers developed specific indicators that measure only post work intervention data. By dismissing different sources and metrics, researchers ensured only data from public policy-effective standardized products were included. This overcomes barriers of absent baseline which can make evaluation challenging.
The effectiveness indicators perform associations between: • the extent to which the whole set of intervention beneficiaries: (a) positions itself on adequacy, through knowledge about product delivery, and/or (b) attributes value by assessing present product condition, compares preexisting and post intervention conditions; and • the product achievement rate delivered to the entire intervention area. • Interestingly, if 100% of the product target was implemented, and the beneficiary evaluation is positive for less than 50%, that reveals a less than expected improvement indicator. The indicator's principle assumes both dimensions to be evaluated together. • Detailed explanatory proposal sheets for each indicator presented, additional metadata, data sources, and measurements contribute to the standardization and nation-wide assessment comparisons.
As a result of the efforts to collaborate toward an evaluation model redesign and to improve the evaluation processes, a lesson learned proposal was created and is comprised of the following general and critical recommendations: • Abundant indicator variety with detailed theme diversity existing in a slum upgrading process can make comparisons difficult and consequently do not contribute to program improvement. On the other hand, strong standardization makes it impossible for local teams to create more personalized evaluations. • Evaluation processes should be complemented with a presentation of the accomplishments responsible to the evaluated improvements and a summary of the political and social context where the evaluation process was carried out. Although the primary output of an evaluation process is a list of ex-post indicators, it is critical to create a record of the evaluation context (i.e. economic, programmatic, political, communitarian and other context situations). • A high quantity of measures is crucial to prevent process evaluation bias, including ensuring qualified evaluators. • Information sources should be checked prior to the elaboration of evaluation standards in order to assure their feasibility. • Ensure that evaluation norms follow a thematic structure that facilitates comprehension of eligible actions specific to precarious settlements.
• Evaluation is dependent on strategic, comprehensive, integrated and updated projects. • Alternative information sources should be standardized and valued for the evaluation process -for example, investment composition tables (detailed budgets) and as-built designs. • Review social work projects considering only results and impacts associated with actions.
Utilizing these principles, a self-applicable evaluation matrix was proposed to MCidades for application to all national slum upgrading interventions. This proposal provides a graphical representation synthesis resulting in a synthetic indicator. Finally, an alternative was presented to produce final result indicators, combining graphical readings of intermediate results indicators with problem analysis and context factors 7 .

Slum focal groups' contributions to urbanization projects
During the Complexo do Alemão research, focus groups emerged as an enriching element of information gathering. Implemented with the initial purpose of understanding resident perceptions of an ongoing PAC intervention, the information collected facilitated the intervention evaluation matrix.
Researchers recognize the importance of focal groups in surpassing first level analysis limited to space configuration (e.g. the products distribution), that then leads to second level analysis marked by the perception or the conceived space (LEFEBVRE, 1991). This allows for full transcendence to what Lefebvre considered the lived space (1991), derived from inhabitants' product perceptions and interactions between actions and objects. Thus, the focus group is a tool to approach the analytical arsenal and the thickness of geographical space (SANTOS, 1996), and the elements of thirdspace (SOJA, 2002). To a certain extent, this runs in opposition to the usual and specialized knowledge that gives primacy to technical data to the detriment of sensitive data, therefore, electing the former as the only credible and useable data for urban planning.
Researchers applied the focal group technique because it allows for high quantities of information collection within limited fieldwork time. A focus group is an interview technique directed to a researcher-selected group with certain identity characteristics with the goal of obtaining qualitative information (MORGAN, 1988). The focus of interest is conversational compared to an interview that alternates questions between researcher and participant. Focus groups are based on participant interaction and information gathering following researcher-guided dialogue and debate. Participant reflection allows for presentation of concepts, impressions and attitudes as well as facts, practices, products or services.
This technique allows for cognitive aspects (opinions, influences and ideas) as well as interactions (conflicts, leadership and alliances) to emerge along with individual or group unique experiences. Consensus building is not required; conflicts should not be avoided. Focus groups do not intend to exhaust themes and the information obtained cannot be inferred as valid for the entire universe of research.
In the case of Complexo do Alemão, focus groups sought to verify PAC intervention knowledge from beneficiaries (or those affected), specifically on the following: family relocation process, intervention information dissemination, community mobilization and participatory processes, environmental education, and patrimonial (facilities) and professional training courses.
The focus groups were comprised of workers, residents (in two different categories: relocated and non-relocated), and businesses that were relocated due to street enlargement. Additionally, younger residents were gathered in specific focus groups. Simultaneously, incursions were made at key locations in the Complexo do Alemão, to gain territory spatial recognition and observe daily practices. The researchers' understanding of territory was critical to focus group success.
The focus group findings are: • Respondents feel stigmatized for living in Complexo do Alemão, a territory that has been constituted over the years as a locus for violence and drug trafficking. The press is the main producer of this image. However, residents insist that the majority of the population are honest and hardworking. Furthermore, place-based stigma is a factor that hinders residents' ability to secure work. • A majority of residents became aware of PAC intervention as construction approached. Few understood the urban project and the necessary yet invasive equipment required. Resident statements reveal that there was limited transparency throughout the planning and implementation process. There was a scarcity of information provided on access and mobility restrictions. • The family relocation process and proposed alternatives were described in considerable detail. Those reallocated in apartments displayed insecurity regarding property ownership and lack of clarity on housing reforms. Economic evaluations of previous properties and improvement payments were controversial, including acknowledgment of a reinvigorated real estate market and consequent property pricing increases, thus making it difficult to purchase a comparable house to that planned for demolition. Given this, a majority of new apartments are considered a better home than the previous ones.
Infrastructure and transportation equipment presence guaranteed economic and social means as one step to ensure mobility. Importantly, focus groups revealed that symbolic issues contribute to the effectiveness of proposed possibilities. The fine grain and deep resident understanding of socio-spatial relations within and between the Complexo do Alemão and the formal city revealed a disconnect between executed works and understood priorities formerly comprised of different expectations.
Therefore, participatory planning is critical for urban interventions. Additionally, focal groups are vital to information acquisition, particularly provided the goal is a profound transformation that overcomes interdicts and gains social mobility and social integration.

Final considerations
The evaluation methodologies presented in this paper are the result of research initiatives seeking to summarize actual evaluation processes and urban development policy contexts in Brazil. The reported initiatives take into account local technical and social demands for evaluation method designs, which do not necessarily comply with the main international agency objectives concerning resource application effectiveness. They seek deep knowledge of the intended and achieved outcomes and of the possible transformations of the local conditions.
At the local level, intervention evaluation can be used as a publicity instrument and method of social control to signal post-intervention correction and improvement factors or to help maintain obtained outcomes over time, which can make the intervention sustainable. Society benefits from both individual evaluations and evaluation observation findings of programs and their socio-spatial application differences that may be associated with nationwide socio-spatial inequalities.
Finally, the evaluation process provides a diagnosis for the construction of future policies, programs and political agendas. Explanation of actions, structures, forms of organization, prerequisites and other program details, along with expected causal relationships between actions taken and potential outcomes, can be obtained from the evaluation process. Thus, gathering the necessary information to make decisions to improve public policies.
A limitation may occur in a tradeoff between particularity (of individual cases) and generalization (allowing the comparability of interventions). Thus, the presentation of an evaluation methodology and stakeholder validation requires an understanding of two fundamental points for agreement between the parties for evaluation integrity.
First, there is instrument bias related to program objective and the associated institution. That is, for whom and for what the evaluation ultimately serves. Second, evaluations are not well-regarded instruments by authorities when revealing government failures. According to Murray Edelman, […] the demands of authorities for "loyalty" are always presented as though the alternative were disloyalty […]. This definition of the issue masks the alternative officials find most threatening: an independent stance that encourages skeptical examination of issues and governmental performance (EDELMAN, 1977: 145).
Both of these premises explain for the dominant governmental position in Brazil and in other countries regarding the evaluation of public policies generally and, particularly, of slum upgrading. It is important to emphasize that the continuity of actions and policies during the last decade in Brazil was the main factor in research efforts' successes, ultimately guaranteeing the innovation, depth, and comprehensiveness. The recent degradation of the political and fiscal situation in Brazil, specifically of its federal government, has posed difficulties to the subsistence of public policy evaluation processes. Moreover, the risks Brazilian democracy now faces may finally cease them.