Elsevier

Remote Sensing of Environment

Volume 144, 25 March 2014, Pages 187-196
Remote Sensing of Environment

Validation of the 2008 MODIS-MCD45 global burned area product using stratified random sampling

https://doi.org/10.1016/j.rse.2014.01.008Get rights and content

Highlights

  • A method was developed to validate MODIS-MCD45 at global scale.

  • Probability sampling was used to infer accuracy for the globe and the main biomes.

  • Accuracy measures chosen were guided by user requirements.

  • Accuracy was higher where burned area was more prevalent.

  • Thiessen areas provide a convenient sampling frame for global validation design.

Abstract

The 2008 global burned area product MODIS-MCD45 was validated and accuracy measures were estimated globally and for several terrestrial biomes. Stratified random sampling was used to select 102 non-overlapping Thiessen scene areas (TSA) and reference fire perimeters were determined from two multi-temporal Landsat TM/ETM + images for each sampled TSA. Error matrices and six accuracy measures were chosen to satisfy criteria specified by end-users of burned area products. Globally, MODIS-MCD45 had estimated commission and omission error rates of 46% and 72% respectively, and a Dice coefficient of 0.37. Burned area extent tended to be underestimated as the MODIS product detected an estimated 48% of the burned area as determined from the reference data. The two biomes with highest accuracy were Boreal Forest and Tropical & Subtropical savanna, two of the most fire-prone biomes. In general, accuracy slightly improved in those areas where burned area was more prevalent.

Introduction

Fire disturbance is one of the relevant Essential Climate Variables (ECV) as identified by the Global Climate Observing System (GCOS) program (GCOS, 2004). Fire affects atmospheric emissions of gases and aerosols and influences carbon budgets as it impacts carbon stocks and vegetation succession patterns.

Validation of global products is defined by The Committee on Earth Observing Satellites' Land Product Validation Subgroup (CEOS-LPVS) as “the process of assessing, by independent means, the quality of the data products derived from the system outputs” (CEOS-WGCV, 2012). The validation step is critical for end users because the results quantitatively assess the performance of a dataset, informing the end user of the limitations and advantages of the final data product. This is essential to facilitate the proper use of the data by the user community. CEOS-LPVS distinguishes four validation stages with the level of effort and rigor increasing at each stage. Stage 3 requires an accuracy assessment “characterized in a statistically robust way over multiple locations” (http://lpvs.gsfc.nasa.gov). The assessment carried out in this research is targeted to Stage 3.

Several global and regional burned area (BA) products have been made available to the international community in the last years. They are the basis for using BA information in many global atmospheric and dynamic vegetation models (Mouillot et al., 2014). Most of these products only include validation at stage 1 or 2. For instance, GlobCarbon (Plummer et al., 2007) and L3JRC (Tansey et al., 2008) were validated with independent data derived from 72 Landsat scenes globally distributed mostly from the year 2000. MODIS-MCD45A1 (Roy, Boschetti, Justice, & Ju, 2008) was validated using 11 Landsat scenes distributed across southern Africa (Roy & Boschetti, 2009). A regional Latin America BA product (from the AQL project) used 19 Landsat scenes and 9 China–Brazil Earth Resources Satellite (CBERS) scenes (Chuvieco et al., 2008) as the basis for validation. Global Fire Emissions Database Version 3 (GFED3), at a coarser spatial resolution (0.5°), was not formally validated; however, it provides uncertainty information modeled from validation results of the 500 m MODIS MCD64A1 BA product, which was used to produce the BA estimates (Giglio et al., 2009, Giglio et al., 2010). None of these validation efforts have used a probability sampling approach at the global scale.

The objective of this research was to develop a methodology to validate global BA products. To illustrate the application of the methodology, we present a case study in which the MODIS-MCD45 burned area product is validated. The MODIS-MCD45 BA product is based on a prognostic model of estimated versus actual reflectance for different MODIS spectral bands (Roy, Jin, Lewis, & Justice, 2005). It is derived in monthly files with pixel values referring to the burning date (Julian days, 1–365), as is common in BA products. Burned pixels between the reference image acquisition dates were coded as “burned”. The rest of the area was coded as “unburned” or “no-data”, the latter applied to unobserved pixels (i.e., pixels obscured by clouds or having corrupted data or missing values). The objectives of the case study include estimating accuracy globally and estimating accuracy for several important biomes.

This research was conducted in the context of the fire_cci project, which is part of the European space Agency's (ESA) Climate Change Initiative (CCI). The fire_cci project will generate long-term series of burned area information globally, and will provide validation and comparisons of BA products. The validation criteria used in this assessment were chosen to address the requirements defined by the end-users of the fire_cci product, as determined from a survey conducted within the project (Mouillot et al., 2014). This survey was mainly addressed to atmospheric chemistry and carbon modelers, but earth observation scientists were included as well. Two user recommendations for desired outcomes of BA product validation were: 1) High accuracy, understood as spatially explicit agreement between the BA product and reference data; and 2) Unbiased BA estimates, implying that the total BA derived from the map closely agrees with total burned area derived on the basis of the reference classification. Validation methods presented in this paper were generated specifically to address these recommendations, both in terms of accuracy and bias, and will be used to validate the fire_cci product when it becomes available. Validation of MODIS-MCD45 was based on cross tabulation analysis (error matrix) using data for 2008 collected from portions of 102 Landsat frames selected by a probability sampling protocol.

Section snippets

Sampling design

The probability sampling design employed a spatial stratification to ensure sufficient sampling in each of the major Olson biomes (Olson et al., 2001), with special focus on regions with high BA, the category of interest for BA products. Because of the time and cost to process each Landsat image needed for the sample, the sampling units were defined based on Landsat World Reference System II (WRS-II). That is, the sampling units were the Thiessen scene areas (TSAs) constructed by Cohen, Yang,

Results

Error matrices were computed for each TSA and the probability distributions of the derived accuracy measures are shown in Fig. 4. From visual inspection, the impact of the low prevalence of BA is evident on the estimated probability distributions of the accuracy measures. That is, because the proportion of BA in many TSAs is very close to 0, overall accuracy (OA) is nearly 1 in most TSAs because the dominant proportion of unburned area is classified correctly. The distributions of bias (B) and

Validation results

The strongly skewed distributions of the per TSA accuracy metrics measured (Fig. 4) are likely caused by a high positive spatial autocorrelation of fire events and consequent effect on the spatial distribution of errors (Griffith, 2009). The vast majority of TSAs have only a small amount of burned area and this area is likely to occur in small irregular patches which are difficult to detect by a classification algorithm. Therefore, as illustrated by the histograms, there is a high probability

Conclusions

This paper presents a statistically rigorous validation method using a probability sampling design applicable to different global BA products. Six accuracy measures, Ce, Oe, DC, OA, B and relB, have been used to address end-user specified desirable accuracy reporting requirements. We have illustrated the application of the sampling design and analysis protocols by assessing the MCD45 BA product, derived from MODIS data, which is one of the most extensively used BA products in atmospheric and

References (34)

  • L. Boschetti et al.

    International global burned area satellite product validation protocol

  • CEOS-WGCV

    Working group on calibration and validation—Land product validation subgroup

    (2012)
  • Y.H. Chou et al.

    Mapping probability of fire occurrence in San Jacinto Mountains, California, USA

    Environmental Management

    (1993)
  • E. Chuvieco et al.

    Global burned land estimation in Latin America using MODIS composite data

    Ecological Applications

    (2008)
  • E. Chuvieco et al.

    ESA CCI ECV fire disturbance—Product validation plan (v3.1)

    (2011)
  • W.G. Cochran

    Sampling Techniques

    (1977)
  • R.G. Congalton et al.

    Assessing the accuracy of remotely sensed data: Principles and applications

    (1999)
  • Cited by (110)

    View all citing articles on Scopus
    View full text