Highlighting continued uncertainty in global land cover maps for the user community

In the last 10 years a number of new global datasets have been created and new, more sophisticated algorithms have been designed to classify land cover. GlobCover and MODIS v.5 are the most recent global land cover products available, where GlobCover (300 m) has the finest spatial resolution of other comparable products such as MODIS v.5 (500 m) and GLC-2000 (1 km). This letter shows that the thematic accuracy in the cropland domain has decreased when comparing these two latest products. This disagreement is also evident spatially when examining maps of cropland and forest disagreement between GLC-2000, MODIS and GlobCover. The analysis highlights the continued uncertainty surrounding these products, with a combined forest and cropland disagreement of 893 Mha (GlobCover versus MODIS v.5). This letter suggests that data sharing efforts and the provision of more in situ data for training, calibration and validation are very important conditions for improving future global land cover products.


Introduction
Global maps of land cover derived from satellite-based earth observations have existed for almost two decades and represent one of the most important sources of baseline terrestrial information for a wide variety of applications, e.g. as inputs to global models of land use and land use change (Foley et al 2005, Verburg et al 2010, climate modelling (Pielke 2005), assessment of available land for biofuels (Cai et al 2011), food security (Liu et al 2008) and as the basis for crop distribution modelling (You et al 2009). Applications in other areas such as biodiversity and population are presented in , who acknowledges that we are still a considerable way from producing global land cover products that are of a high enough quality for many applications.
One avenue for research has involved the comparison of global land cover datasets, either against one another or with higher resolution regional products. Many of the studies found similar results, i.e. good overall agreement but disagreement in either the individual land classes or in the spatial distribution of the land cover (McCallum et al 2006, Fritz and See 2005, See and Fritz 2006, Neumann et al 2007, Herold et al 2008, Fritz et al 2010, Seebach et al 2011. This general trend continues to be the case when comparing the  (Friedl et al 2010) and GlobCover (Bicheron et al 2008, Bontemps et al 2011. The overall accuracy of these maps is reported as 68.5% (±5%) for GLC-2000 (Mayaux et al 2006), 74.8% (±1.3%) for MODIS and 67.1% for GlobCover 2005 (Bicheron et al 2008), which shows similar results when taking the 95% confidence bands into account. However, this paper will show that the overall spatial disagreement in both the forest and particularly the cropland domain continues to be very high. The purpose of this paper is therefore to make the user community aware of the continued uncertainties in these products, which could potentially impact the outcomes of any assessment or modelling exercises undertaken (see e.g. Quaife et al 2008, Feddema et al 2005, Havlík et al 2011. The paper concludes with a discussion of what is needed to improve the accuracy of land cover information.

Global land cover and agricultural datasets
The GLC-2000 was developed by the JRC to provide baseline information for the year 2000 and has the coarsest resolution at 1 km (Bartholomé and Belward 2005). It was first developed regionally using experts in the field and then integrated into a single global product. In contrast, Boston University's MODIS land cover product was developed using a top-down approach where a classification algorithm was applied to create a global product at a resolution of 500 m (MODIS v.5) for the year 2005 (Friedl et al 2010). At 300 m, GlobCover is the finest resolution product available for the year 2005-2006 (Bicheron et al 2008). This new product is intended to update and complement other existing comparable global products but the higher spatial resolution was also expected to provide improvements in thematic accuracy as the overall number of mixed classes found in a pixel decreased. A supervised and unsupervised classification algorithm was used to classify pixels into similar spectral and temporal classes. An automated labelling procedure using the Land Cover Classification System (Di Gregorio and Jansen 2000) and a global reference dataset (including the GLC-2000) was then used to create the final product. Compared to the GLC-2000, the more automated nature of the classification algorithms used to produce MODIS and GlobCover means that they can be easily repeated and produce updated products on a more regular basis. Further details of these datasets can be found in Fritz et al (2010). The focus of the disagreement between land cover products in this paper is in the cropland and forest classes. Table 1 provides the legend classes that fall in the cropland and forest domains in each land cover class while the full legend definitions for these classes are given in supplementary tables 1 and 2 (available at stacks.iop.org/ERL/ 6/044005/mmedia).

Aggregation to a common spatial resolution
In order to compare the three main global land cover products, the difference in their spatial resolutions was first reconciled. In the Plate-Carrée projection with a Geographic Lat/Lon representation, the spatial resolution of the GLC-2000 is 1/112 • × 1/112 • or approximately 1 km × 1 km at the equator; MODIS v.5 is 1/240 • × 1/240 • ; while GlobCover is 1/360 • × 1/360 • or approximately 300 m × 300 m at the equator. A common grid of 0.125 • × 0.125 • was chosen in which all land cover products could then be aggregated. This equates to an aggregation of 14 pixels for GLC-2000, 30 pixels for MODIS v.5 and 45 pixels for GlobCover. The percentage of forest and cropland of each aggregated grid cell was then determined using the minimum and maximum percentages from the class definitions (as provided in supplementary tables 1 and 2, available at stacks.iop.org/ERL/6/044005/mmedia). For example, if 98 out of 196 pixels for the GLC-2000 were of forest type (where 80%-100% in that pixel would be forest according to the definition of this land cover type), then the new aggregated grid cell would contain 50% forest for the maximum forest class, i.e. 100% divided by 2, and 40% forest for the minimum forest class, i.e. 80% divided by 2. A similar approach was used in Fritz et al (2010) to aggregate the pixels.

Creation of cropland and forest disagreement maps
The disagreement in cropland and forest cover between a given pair of land cover maps is derived using a concept termed in this paper as the Minimum Measurable Disagreement (MMD). This is a modified version of the approach used in Fritz and See (2008), which was originally based on the concepts proposed in Ahlqvist (2005) but is applied to the aggregated grid cell. Each aggregated grid cell in each of the land cover datasets will have a minimum and maximum cropland or forest cover. To compare a pair of land cover datasets and calculate the disagreement at each pixel, the range of cropland/forest cover is compared by examining the amount of definitional overlap. The calculation of disagreement is illustrated in supplementary figure 1 (available at stacks.iop.org/ERL/6/044005/mmedia). If there is any overlap in the definitions, then the disagreement or MMD is 0. Where there is no overlap, the MMD is calculated. For example, if the aggregated pixel for GlobCover has 0-40% cropland and MODIS has 60-100%, then the MMD or disagreement is 20%. The MMD takes the most conservative approach to assessing disagreement. This process was applied to three pairs of land cover comparisons: GLC-2000 andMODIS v.5, GLC-2000 andGlobCover, andMODIS v.5 and GlobCover to create three maps of disagreement for forest and three for cropland. The three cropland disagreement maps were then further summed to create a single map of cropland disagreement; this was repeated to create a single forest disagreement map. Finally, a combined cropland/forest disagreement map was created by summing together the per cent disagreement within all the disagreement maps in the cropland as well as in the forest domain. The following thresholds were then applied to the map: (i) 0 to less than 5%-no disagreement (i) from 5% to 40%-'disagreement' and (ii) greater than 40%-'high disagreement'.

Quantification of the thematic accuracy and the disagreement
The accuracy of the cropland and forest classes was calculated using the validation data and the confusion matrices published  (2010) and Bicheron et al (2008). The validation data for individual cropland and forest classes were first aggregated using table 1 and then the overall accuracy of these classes was calculated. The total area of disagreement between each pair of land cover products was calculated separately for cropland and forest as well as the areas of commission and omission for each pair. To provide a reference figure for comparison across pairs of land cover products, the average cropland and forest areas across all products was calculated and used as a denominator to quantify the amount of disagreement.

Results
The total areas under cropland based on GLC-2000, GlobCover and MODIS v.5 1-9) has an accuracy of 81%, for MODIS v.5 (classes 1-5) it is 80%, and for GlobCover 2005 (classes 40-120) it drops to 60%. Table 2 provides the overall differences in Mha and as a percentage of the average cropland and forest cover from the GLC-2000, MODIS v.5 andGlobCover 2005. The disagreement between the GLC-2000 and MODIS v.5 is 731 Mha and 326 Mha respectively for forest and cropland. Comparing the two newer products (i.e. GlobCover and MODIS v.5), the overall forest disagreement decreases to 387 Mha but the cropland disagreement increases to 506 MHa or 36% of the average area of the three land cover products.
The differences between the different land cover products highlighted in table 2 are even more significant when viewed spatially. Figure 1(a) provides a map of global disagreement between the two most recent and highest resolution products: MODIS v.5 and GlobCover 2005 for the cropland domain. Areas of high disagreement are visible across North America, Russia and across the tropical world.
Large cropland disagreements are evident in China, North America and many countries in Africa. A more detailed example is provided in figure 1(b), which highlights an example in Ethiopia. Cropland disagreement with more cropland in MODIS is displayed in yellow and red shades, and disagreement with more cropland in GlobCover is displayed in blue shades ( figure 1(b-A)). Individual land cover products are shown in figure 1(b-B) (GlobCover) and figure 1(b-C) (MODIS), which clearly demonstrates that they differ both thematically and spatially regarding the distribution of identical or similar land cover types. For example, large cultivated and managed areas on the GlobCover map are labelled forest/woody savannah or savannah on the MODIS map, and likewise, mosaic forest or shrubland on the GlobCover map is labelled as cropland on the MODIS map.
This phenomenon of large differences in both cropland and forest cover occurs in many regions of the world. The full set of maps showing the disagreement between each pair of land cover products as well as the combined disagreement for cropland and forest can be found on geo-wiki.org (Fritz et al 2009). A version showing disagreements in the urban domain can be found on urban.geo-wiki.org (Fritz et al 2011).

Discussion and conclusions
This paper has shown that there are critical differences between the land cover products as expressed by the spatial disagreement (particularly for cropland).
For example, 360 Mha are identified as cropland in GlobCover but as noncropland in MODIS, which is a discrepancy that equates to approximately 20% of the global cropland area. The thematic accuracy of GlobCover for the aggregated cropland and forest classes was also shown to be worse than that of the GLC-2000 and MODIS v.5, despite being a newer product that has already been downloaded more than 50 000 times by users (GEO 2011). These disagreements can be caused by differences in classification methodology, differences in training and ground reference data, the type of satellite sensors used and georeferencing errors (McCallum et al 2006, Fritz andSee 2005). A small portion of the differences might also be attributed to differences in the date of the satellite data acquisition used in creating the land cover maps. Therefore, as a minimum guideline, these maps cannot be used for land cover change detection since the error in the original map is higher than the change detected (e.g. GLC-2000 versus GlobCover). Due to the large disagreements between these land cover products, we recommend that the user community does not, by default, use the latest product with the highest resolution, but carefully examines the sensitivity of these products within a specific application. In the situation where the maps are used for national and regional applications, we would recommend examining the disagreement of the products in the areas of interest and also to compare them with high resolution ground data or aerial photography. One way to do this would be to use geo-wiki.org (Fritz et al 2009), a global land cover validation tool, which can be used to visualize the global land cover products and the disagreement directly on top of Google Earth. By exploring the discrepancies at the level of an individual country in combination with local knowledge, the user can gain insight into which product is better in a specific region and which product is better suited for a particular application.
More research efforts should be directed towards finding ways to improve global land cover. There are promising developments on the horizon such as open access to the Landsat archive (Woodcock et al 2008, Roy et al 2010, the development of a new 30 m global land cover product being developed by the USA and China (US Department of the Interior 2010), and the launch of the Sentinel satellites over the next decade with a finer temporal and spatial resolution (ESA 2011). However, validation of future products will remain a crucial issue. This will require greater involvement of experts on the ground and the collection of a larger quantity of in situ data. The task of improving validation data for land cover datasets is now being increasingly discussed by the CEOS Cal Val land cover validation subgroup, which advocates the collection of more 'authorized' validation data. Visualization tools such as those provided by Google Earth represent a promising platform for validation and for the collection of increasing amounts of citizen volunteered information on land cover through crowd-sourcing for validation of land cover products. Crowd-sourced data in the form of geo-tagged photos and information collected through Web 2.0 applications like geo-wiki and smart phones could also be harnessed as a rich source of training and calibration data for global land cover algorithms. Moreover, serious gaming has a potential role to play in developing applications that engage a wider community in data collection as part of contributing to environmental solutions.
The benefits from these more promising solutions will only be reaped in the medium to long term. Bottom up initiatives such as geo-wiki.org may provide, at the very least, a short-term solution (Macauley and Sedjo 2010), particularly in the development of an integrated global land cover product. Such an integrated product would use existing regional and national land cover products where available, the best global product in situations where higher quality national maps are not available and crowd-sourced data provided by citizens. The design of appropriate land use policies with global dimensions requires reliable and accurate land cover data. Although the results of an integrated product will need to be assessed against fitness for purpose, the integrated assessment community in particular is eager to work with the best available products now. In developing countries in particular, good baseline data and monitoring techniques for ecosystem services are needed immediately (Andelman 2011). Waiting for future developments, however promising, is not an option if we are to tackle the burning issues surrounding sustainable provision of ecosystem services and providing sufficient food for a growing global population.