ReviewGood practices for estimating area and assessing accuracy of land change
Introduction
Land change maps quantify a wide range of processes including wildfire (Schroeder, Wulder, Healey, & Moisen, 2011), forest harvest (Olofsson et al., 2011), forest disturbance (Huang et al., 2010), land use pressure (Drummond & Loveland, 2010) and urban expansion (Jeon, Olofsson, & Woodcock, 2013). Map users and producers are acutely interested in communicating and understanding the quality of these maps. Accordingly, guidance on how to assess accuracy of these maps in a consistent and transparent manner is a necessity. The use of remote sensing products depicting change for scientific, management, or policy support activities all require quantitative accuracy statements to buttress the confidence in the information generated and in any subsequent reporting or inferences made. Area estimation, whether of change in land cover/use or of status of land cover/use at a single date, is a natural value-added use of land change maps in many local, national and global land accounting applications. For example, the amount of land area allocated for a specific use is a key country reporting requirement to the United Nations (UN) Food and Agriculture Organization (FAO) statistics and the global forest resources' assessment (FAO, 2010) as well as for countries reporting under the Kyoto protocol and the evolving activities for the UN Collaborative Programme on Reducing Emissions from Deforestation and Forest Degradation — UN-REDD (Grassi et al., 2008, UN-REDD, 2008). Estimates of forest extent or deforestation are often derived via remote sensing (cf. Achard et al., 2002, DeFries et al., 2002, Hansen et al., 2010), and area estimation also plays a prominent role in ongoing efforts to establish scientifically valid protocols for forest change monitoring in the context of specific accounting applications to policy approaches for reducing greenhouse gas emissions from forests (DeFries et al., 2007, GOFC-GOLD, 2011).
A key strength of remote sensing is that it enables spatially exhaustive, wall-to-wall coverage of the area of interest. However, as might be expected with any mapping process, the results are rarely perfect. Placing spatially and categorically continuous conditions into discrete classes may result in confusion at the categorical transitions. Error can also result from the change mapping process, the data used, and analyst biases (Foody, 2010). Change detection and mapping approaches using remotely sensed data are increasingly robust, with improvements aimed at the mitigation of these sources of error. However, any map made from remotely sensed data can be assumed to contain some error, with the areas calculated from the map (e.g., pixel counting) also potentially subject to bias. An accuracy assessment identifies the errors of the classification, and the sample data can be used for estimating both accuracy and area along with the uncertainty of these estimates. While the notion of accuracy assessment is well-established within the remote sensing community (Foody, 2002, Strahler et al., 2006), studies of land change routinely fail to assess the accuracy of the final change maps and few published studies of land change make full use of the information obtained from accuracy assessments (Olofsson, Foody, Stehman, & Woodcock, 2013).
In this article, we synthesize the current status of key steps and methods that are needed to complete an accuracy assessment of a land change map and to estimate area of land change. This article addresses the fundamental protocols required to produce scientifically rigorous and transparent estimates of accuracy and area. The set of good practice recommendations provides guidelines to assist both scientists and practitioners in the design and implementation of accuracy assessment and area estimation methods applied to land change assessments using remote sensing. The accuracy and area estimation objectives are linked via a map of change. A change map provides a spatially explicit depiction of change and this spatial information can be readily aggregated to calculate the total mapped area or the proportion of mapped area of change for the region of interest (ROI). Accuracy assessment addresses questions related to how well locations of mapped change correspond to actual areas of change. A fundamental premise of the recommended good practices methodology is that the change map will be subject to an accuracy assessment based on a sample of higher quality change information (i.e., the reference classification). The higher quality reference classification is compared to the map classification on a location-specific basis to quantify accuracy of the change map and to estimate area. Although it is possible to estimate area of change without producing a change map (Achard et al., 2002, FAO, 2010, Hansen et al., 2010), we will assume that a map of change exists (although there will not necessarily be a map for each date). The focus for this document is change between two dates.
Before any detailed planning of the response and sampling designs is undertaken, a basic visual assessment should be conducted to identify obvious errors and concerns in the remotely sensed product. This assessment provides an evaluation of the map's suitability for the intended application and should detect if a map is so unsuitable for use that there is no value in proceeding to a more detailed assessment. The visual assessment should also highlight errors that are easy to remove enabling the map to be refined prior to initiating a detailed assessment or confirm that no obvious concerns exist and the map is ready for further rigorous evaluation.
We separate the accuracy assessment methodology into three major components, the response design, sampling design, and analysis (Stehman & Czaplewski, 1998). The response design encompasses all aspects of the protocol that lead to determining whether the map and reference classifications are in agreement. Because it is often impractical to apply the response design to the entire ROI, a subset of the area is sampled. The sampling design is the protocol for selecting that subset of the ROI. The analysis includes protocols for defining how to quantify accuracy along with the formulas and inference framework for estimating accuracy and area and quantifying uncertainty of these estimates. A separate section of this guidance document is devoted to each of these three major components of accuracy assessment methodology. These sections are followed by an example of the recommended workflow.
The good practice recommendations are intended to represent a synthesis of the current science of accuracy assessment and area estimation. We fully anticipate that improved methods will be developed over time. As the designation of “best practice” implies a singular approach, we prefer the use of “good practice” to indicate that “best” is relative and will vary, with one hard-coded approach not always appropriate. In communicating good practices, desirable features and selection criteria can be followed to ensure that the protocol applied satisfies – as thoroughly as possible – the accuracy and area estimation recommendations. The good practice recommendations do not preclude the existence of other acceptable practices, but instead represent protocols that, if implemented correctly, would ensure scientific credibility of the results. Furthermore, the recommendations presented herein allow flexibility to choose specific details of the different components of the methodology. For example, while the general recommendation for the sampling design is to implement a probability sampling protocol, there are numerous sampling designs that meet this criterion (Stehman, 2009). Similarly, the response design protocol allows flexibility to use a variety of different sources for determining the reference classification and multiple options exist for defining agreement between the map and reference classifications. The good practices recommendations represent an ideal to strive for, but it is likely that most projects will not satisfy every recommendation. Documenting and justifying deviations from good practices are expected features of many accuracy assessment and area estimation studies. For the most part, the good practice recommendations consist of methods for which there is considerable experience of practical use in the remote sensing community.
These good practice recommendations for area estimation and accuracy assessment of land change build on earlier guidelines for single-date land-cover maps described by Strahler et al. (2006). Strahler et al. (2006) presented general guiding principles of good practices with less emphasis on details of methodology. In the intervening years since Strahler et al. (2006), additional theory and practical application related to accuracy assessment and area estimation have been accumulated, and this current document avails upon these developments to delve more deeply into methodological details. We do not attempt to provide an exhaustive description of methods given the range of issues and the highly application-specific nature of the topic. Instead, our purpose is to focus upon the main issues needed to establish a common basis of good practice methodology that will be generally applicable and result in transparent methods and rigorous estimates of accuracy and area. A list of recommendations for all components of the process (sampling design, response design, and analysis) is presented in the Summary section (Section 6).
Estimating area and accuracy of change maps introduces additional methodological challenges that were not within the scope addressed by Strahler et al. (2006). In particular, the area estimation objective was not addressed at all by Strahler et al. (2006). Accuracy assessment of change highlights many unique challenges, including the dynamic nature of the reference data, and aspects of the change features including type, severity, persistence, and area. Another challenge is that change is usually a rare feature over a given landscape. The accuracy of a map and the area estimates derived with its aid are a function of the land-cover mosaic under study, the underlying imagery and the methods applied. Accuracy and area estimates for the same region will, for example, vary if using a per-pixel or object-based classification or if the spatial resolution of the imagery is altered (cf. Baker et al., 2013, Duro et al., 2012, Johnson, 2013).
Our recommendations also focus on methods for providing rigorous estimates of land (area) change and its uncertainties. A primary use of such estimates is in analysis and accounting frameworks such as national inventories. In evolving frameworks compensating for successful climate change mitigation actions in the forest sector (such as REDD +, DeFries et al., 2007), the consideration of uncertainties are likely linked with financial incentives and are subject to critical international political negotiations on reporting and verification (Sanz-Sanchez, Herold, & Penman, 2013). Understanding and management of uncertainties in area change is essential, particularly because data and capacity gaps in forest monitoring are large in many developing countries (Romijn, Herold, Kooistra, Murdiyarso, & Verchot, 2012). Accuracy assessments should also focus on identifying and addressing error sources, and prioritize on capacity development needs to provide continuous improvements and reduce uncertainties in the estimates over time. This also includes assessing the value of data streams from evolving monitoring technologies (de Sy et al., 2012, Pratihast et al., 2013) where the ultimate impact on lower uncertainties need to be proven in operational contexts. Thus, the methods of good practice presented here are generic for providing rigorous estimates, and having agreed-upon tools to do so will provide the saliency and legitimacy for using them in quantifying improvements in monitoring systems, and for dealing with uncertainties in financial compensation schemes (e.g., for climate change mitigation actions).
This article synthesizes key steps and methods needed to complete an accuracy assessment of a change map and to estimate area and accuracy of the map classes. It addresses the protocols required to produce scientifically rigorous and transparent estimates of accuracy and area.
Section snippets
Sampling design
The sampling design is the protocol for selecting the subset of spatial units (e.g., pixels or polygons) that will form the basis of the accuracy assessment. Choosing a sampling design requires a consideration of the specific objectives of the accuracy assessment and a prioritized list of desirable design criteria. The most critical recommendation is that the sampling design should be a probability sampling design. An essential element of probability sampling is that randomization is
Response design
For the accuracy assessment objective, the response design encompasses all steps of the protocol that lead to a decision regarding agreement of the reference and map classifications. For area estimation, the response design provides the best available classification of change for each spatial unit sampled. The four major features of the response design are the spatial unit, the source or sources of information used to determine the reference classification, the labeling protocol for the
Analysis
The analysis protocol specifies the measures to be used to express accuracy and class area as well as the procedures to estimate the selected measures from the sample data. In the context of studies of land change, there are two key objectives of the analysis: 1) accuracy assessment of the change classification, and 2) estimation of area of change. The confusion or error matrix (hereafter noted as the error matrix) plays a central role in meeting both the accuracy assessment and area estimation
Example of good practices: estimating area and assessing accuracy of forest change
The following hypothetical example illustrates the workflow of assessing accuracy of a forest change map and estimating area. Consider a change map for 2000 to 2010 consisting of two change classes and two stable classes: deforestation, forest gain, stable forest and stable non-forest. The map was produced by supervised classification of data from Landsat ETM + with the objective of estimating the gross rates of forest loss and gain. The first step in the assessment was to visually inspect the
Summary
Conducting an accuracy assessment of a land change map serves multiple purposes. In addition to the obvious purpose of quantifying the accuracy of the map, the reference sample serves as the basis of estimates of area of each class where area is defined by the reference classification. The accuracy assessment sample data also contribute to estimates of uncertainty of the area estimates. Without an accuracy assessment, there is no way to communicate map quality in a quantitative and meaningful
Acknowledgments
This research was funded by the USGS Award Support for SilvaCarbon and NASA through its support for the Carbon Monitoring System to Boston University, and NASA Grant Number NNX13AP48G to State University of New York. We acknowledge the European Space Agency (ESA) and NASA for their support to GOFC-GOLD and the CEOS working group of calibration and validation. We thank the anonymous reviewers for the comments that helped improve the manuscript.
References (89)
- et al.
A fuzzy set-based accuracy assessment of soft classification
Pattern Recognition Letters
(1999) - et al.
Correspondence analysis for detecting land cover change
Remote Sensing of Environment
(2006) - et al.
Detecting trends in forest disturbance and recovery using yearly Landsat time series: 2. TimeSync — Tools for calibration and validation
Remote Sensing of Environment
(2010) - et al.
Synergies of multiple remote sensing data sources for REDD + monitoring
Current Opinion in Environmental Sustainability
(2012) - et al.
Earth observations for estimating greenhouse gas emissions from deforestation in developing countries
Environmental Science and Policy
(2007) - et al.
A comparison of pixel-based and object-based image analysis with selected machine learning algorithms for the classification of agricultural landscapes using SPOT-5 HRG imagery
Remote Sensing of Environment
(2012) Status of land cover classification accuracy assessment
Remote Sensing of Environment
(2002)Assessing the accuracy of land cover change with imperfect ground reference data
Remote Sensing of Environment
(2010)- et al.
Characterizing the state and processes of change in a dynamic forest environment using hierarchical spatio-temporal segmentation
Remote Sensing of Environment
(2011) - et al.
An automated approach for reconstructing recent forest disturbance history using dense Landsat time series stacks
Remote Sensing of Environment
(2010)
Accuracy comparison of various remote sensing data sources in the retrieval of forest stand attributes
Forest Ecology and Management
Detecting trends in forest disturbance and recovery using yearly Landsat time series: 1. LandTrendr — Temporal segmentation algorithms
Remote Sensing of Environment
Estimation of 3D vegetation structure from waveform and discrete return airborne laser scanning data
Remote Sensing of Environment
Comparative assessment of the measures of thematic classification accuracy
Remote Sensing of Environment
Satellite image-based maps: Scientific inference or pretty pictures?
Remote Sensing of Environment
Making better use of accuracy data in land change studies: Estimating accuracy and area and quantifying uncertainty using stratified estimation
Remote Sensing of Environment
Sources of error in accuracy assessment of thematic land-cover maps in the Brazilian Amazon
Remote Sensing of Environment
An effective assessment protocol for continuous geospatial datasets of forest characteristics using USFS Forest Inventory and Analysis (FIA) data
Remote Sensing of Environment
Mapping wildfire and clearcut harvest disturbances in boreal forests with Landsat time series data
Remote Sensing of Environment
Selecting and interpreting measures of thematic classification accuracy
Remote Sensing of Environment
Practical implications of design-based sampling inference for thematic map accuracy assessment
Remote Sensing of Environment
Comparing estimators of gross change derived from complete coverage mapping versus statistical sampling of remotely sensed data
Remote Sensing of Environment
Estimating area from an accuracy assessment error matrix
Remote Sensing of Environment
Design and analysis for thematic map accuracy assessment: Fundamental principles
Remote Sensing of Environment
Statistical sampling to characterize recent United States land-cover change
Remote Sensing of Environment
Pixels, blocks of pixels, and polygons: Choosing a spatial unit for thematic accuracy assessment
Remote Sensing of Environment
Accuracy assessment of NLCD 2006 land cover and impervious surface
Remote Sensing of Environment
Opening the archive: How free data has enabled the science and monitoring promise of Landsat
Remote Sensing of Environment
Multi-temporal analysis of high spatial resolution imagery for disturbance monitoring
Remote Sensing of Environment
Validation of a large area land cover product using purpose-acquired airborne video
Remote Sensing of Environment
An accuracy assessment of forest disturbance mapping in the western Great Lakes
Remote Sensing of Environment
Determination of deforestation rates of the world's humid tropical forests
Science
In search of classification that supports the dynamics of science: The FAO Land Cover Classification System and proposed modifications
Environment and Planning B: Planning and Design
Does spatial resolution matter? A multi-scale comparison of object-based and pixel-based methods for detecting change associated with gas well drilling operations
International Journal of Remote Sensing
Using map category marginal frequencies to improve estimates of thematic map accuracy
Photogrammetric Engineering and Remote Sensing
Sampling techniques
Using semantics to clarify the conceptual confusion between land cover and land use: The example of ‘forest’
Journal of Land Use Science
Assessing the accuracy of remotely sensed data: Principles and practices
Carbon emissions from tropical deforestation and regrowth based on satellite observations for the 1980s and 90s
Proceedings of the National Academy of Sciences
Land-use pressure and a transition to forest-cover loss in the eastern United States
BioScience
Supporting large-area, sample-based forest inventories with very high spatial resolution satellite imagery
Progress in Physical Geography
Global forest resources assessment 2010
On the compensation for chance agreement in image classification accuracy assessment
Photogrammetric Engineering and Remote Sensing
Approaches for the production and evaluation of fuzzy land cover classifications from remotely sensed data
International Journal of Remote Sensing
Cited by (1862)
Mapping invasive iceplant extent in southern coastal California using high-resolution aerial imagery
2024, Ecological InformaticsSatellite-based woody canopy cover for Africa: Uncovering bias and recovering best estimates across years
2024, Science of Remote SensingModeling urban redevelopment: A novel approach using time-series remote sensing data and machine learning
2024, Geography and SustainabilityTesting temporal transferability of remote sensing models for large area monitoring
2024, Science of Remote SensingThe apparent effect of orbital drift on time series of MODIS MOD10A1 albedo on the Greenland ice sheet
2024, Science of Remote Sensing