Validation of the 2008 MODIS-MCD45 global burned area product using stratified random sampling
Introduction
Fire disturbance is one of the relevant Essential Climate Variables (ECV) as identified by the Global Climate Observing System (GCOS) program (GCOS, 2004). Fire affects atmospheric emissions of gases and aerosols and influences carbon budgets as it impacts carbon stocks and vegetation succession patterns.
Validation of global products is defined by The Committee on Earth Observing Satellites' Land Product Validation Subgroup (CEOS-LPVS) as “the process of assessing, by independent means, the quality of the data products derived from the system outputs” (CEOS-WGCV, 2012). The validation step is critical for end users because the results quantitatively assess the performance of a dataset, informing the end user of the limitations and advantages of the final data product. This is essential to facilitate the proper use of the data by the user community. CEOS-LPVS distinguishes four validation stages with the level of effort and rigor increasing at each stage. Stage 3 requires an accuracy assessment “characterized in a statistically robust way over multiple locations” (http://lpvs.gsfc.nasa.gov). The assessment carried out in this research is targeted to Stage 3.
Several global and regional burned area (BA) products have been made available to the international community in the last years. They are the basis for using BA information in many global atmospheric and dynamic vegetation models (Mouillot et al., 2014). Most of these products only include validation at stage 1 or 2. For instance, GlobCarbon (Plummer et al., 2007) and L3JRC (Tansey et al., 2008) were validated with independent data derived from 72 Landsat scenes globally distributed mostly from the year 2000. MODIS-MCD45A1 (Roy, Boschetti, Justice, & Ju, 2008) was validated using 11 Landsat scenes distributed across southern Africa (Roy & Boschetti, 2009). A regional Latin America BA product (from the AQL project) used 19 Landsat scenes and 9 China–Brazil Earth Resources Satellite (CBERS) scenes (Chuvieco et al., 2008) as the basis for validation. Global Fire Emissions Database Version 3 (GFED3), at a coarser spatial resolution (0.5°), was not formally validated; however, it provides uncertainty information modeled from validation results of the 500 m MODIS MCD64A1 BA product, which was used to produce the BA estimates (Giglio et al., 2009, Giglio et al., 2010). None of these validation efforts have used a probability sampling approach at the global scale.
The objective of this research was to develop a methodology to validate global BA products. To illustrate the application of the methodology, we present a case study in which the MODIS-MCD45 burned area product is validated. The MODIS-MCD45 BA product is based on a prognostic model of estimated versus actual reflectance for different MODIS spectral bands (Roy, Jin, Lewis, & Justice, 2005). It is derived in monthly files with pixel values referring to the burning date (Julian days, 1–365), as is common in BA products. Burned pixels between the reference image acquisition dates were coded as “burned”. The rest of the area was coded as “unburned” or “no-data”, the latter applied to unobserved pixels (i.e., pixels obscured by clouds or having corrupted data or missing values). The objectives of the case study include estimating accuracy globally and estimating accuracy for several important biomes.
This research was conducted in the context of the fire_cci project, which is part of the European space Agency's (ESA) Climate Change Initiative (CCI). The fire_cci project will generate long-term series of burned area information globally, and will provide validation and comparisons of BA products. The validation criteria used in this assessment were chosen to address the requirements defined by the end-users of the fire_cci product, as determined from a survey conducted within the project (Mouillot et al., 2014). This survey was mainly addressed to atmospheric chemistry and carbon modelers, but earth observation scientists were included as well. Two user recommendations for desired outcomes of BA product validation were: 1) High accuracy, understood as spatially explicit agreement between the BA product and reference data; and 2) Unbiased BA estimates, implying that the total BA derived from the map closely agrees with total burned area derived on the basis of the reference classification. Validation methods presented in this paper were generated specifically to address these recommendations, both in terms of accuracy and bias, and will be used to validate the fire_cci product when it becomes available. Validation of MODIS-MCD45 was based on cross tabulation analysis (error matrix) using data for 2008 collected from portions of 102 Landsat frames selected by a probability sampling protocol.
Section snippets
Sampling design
The probability sampling design employed a spatial stratification to ensure sufficient sampling in each of the major Olson biomes (Olson et al., 2001), with special focus on regions with high BA, the category of interest for BA products. Because of the time and cost to process each Landsat image needed for the sample, the sampling units were defined based on Landsat World Reference System II (WRS-II). That is, the sampling units were the Thiessen scene areas (TSAs) constructed by Cohen, Yang,
Results
Error matrices were computed for each TSA and the probability distributions of the derived accuracy measures are shown in Fig. 4. From visual inspection, the impact of the low prevalence of BA is evident on the estimated probability distributions of the accuracy measures. That is, because the proportion of BA in many TSAs is very close to 0, overall accuracy (OA) is nearly 1 in most TSAs because the dominant proportion of unburned area is classified correctly. The distributions of bias (B) and
Validation results
The strongly skewed distributions of the per TSA accuracy metrics measured (Fig. 4) are likely caused by a high positive spatial autocorrelation of fire events and consequent effect on the spatial distribution of errors (Griffith, 2009). The vast majority of TSAs have only a small amount of burned area and this area is likely to occur in small irregular patches which are difficult to detect by a classification algorithm. Therefore, as illustrated by the histograms, there is a high probability
Conclusions
This paper presents a statistically rigorous validation method using a probability sampling design applicable to different global BA products. Six accuracy measures, Ce, Oe, DC, OA, B and relB, have been used to address end-user specified desirable accuracy reporting requirements. We have illustrated the application of the sampling design and analysis protocols by assessing the MCD45 BA product, derived from MODIS data, which is one of the most extensively used BA products in atmospheric and
References (34)
- et al.
Mapping burned areas from Landsat TM/ETM + data with a two-phase algorithm: Balancing omission and commission errors
Remote Sensing of Environment
(2011) - et al.
Analysis of the conflict between omission and commission in low spatial resolution dichotomic thematic products: The Pareto Boundary
Remote Sensing of Environment
(2004) - et al.
Detecting trends in forest disturbance and recovery using yearly Landsat time series: 2. TimeSync—Tools for calibration and validation
Remote Sensing of Environment
(2010) Stratified sampling of satellite images with a systematic grid of points
Journal of Photogrammetry and Remote Sensing
(2005)- et al.
An active-fire based burned area mapping algorithm for the MODIS sensor
Remote Sensing of Environment
(2009) - et al.
Detecting trends in forest disturbance and recovery using yearly Landsat time series: 1. LandTrendr—Temporal segmentation algorithms
Remote Sensing of Environment
(2010) - et al.
Accuracy assessment using sub-pixel fractional error matrices of global land cover products derived from satellite data
Remote Sensing of Environment
(2004) - et al.
Ten years of global burned area products from spaceborne remote sensing—A review: Analysis of user needs and recommendations for future developments
International Journal of Applied Earth Observation and Geoinformation
(2014) - et al.
The collection 5 MODIS burned area product—Global evaluation by comparison with the MODIS active fire product
Remote Sensing of Environment
(2008) - et al.
Prototyping a global algorithm for systematic fire-affected area mapping using MODIS time series data
Remote Sensing of Environment
(2005)
International global burned area satellite product validation protocol
Working group on calibration and validation—Land product validation subgroup
Mapping probability of fire occurrence in San Jacinto Mountains, California, USA
Environmental Management
Global burned land estimation in Latin America using MODIS composite data
Ecological Applications
ESA CCI ECV fire disturbance—Product validation plan (v3.1)
Sampling Techniques
Assessing the accuracy of remotely sensed data: Principles and applications
Cited by (110)
Coarse-resolution burned area datasets severely underestimate fire-related forest loss
2024, Science of the Total EnvironmentMadagascar's burned area from Sentinel-2 imagery (2016–2022): Four times higher than from lower resolution sensors
2024, Science of the Total EnvironmentNeed and vision for global medium-resolution Landsat and Sentinel-2 data products
2024, Remote Sensing of EnvironmentBurned area detection and mapping using time series Sentinel-2 multispectral images
2023, Remote Sensing of EnvironmentTotal-variation regularized U-Net for wildfire burned area mapping based on Sentinel-1 C-Band SAR backscattering data
2023, ISPRS Journal of Photogrammetry and Remote SensingAssessment and characterization of sources of error impacting the accuracy of global burned area products
2022, Remote Sensing of Environment