Early-season crop mapping using improved artificial immune network (IAIN) and Sentinel data

Methodology overview

Figure 3 depicts the flowchart of this study. First, 15-day compositions of Sentinel-1 and Sentinel-2 data were generated. As winter wheat was harvested in June and cotton, spring maize and summer maize were harvested in September, we used images from April to May for early-season winter wheat identification and images from April to August for early-season cotton, spring maize and summer maize identification. We firstly used Jeffries–Matusita (JM) distance and the Gini importance score from the random forest (RF) algorithm to evaluate the crop separability and contribution of each feature on crop classification, and generated image time series of the optimal bands. Then, we proposed an improved artificial immune network (IAIN) which could deal with the missing values in the image time series. Finally, we varied time series length, used IAIN to identify crop types and evaluated the effect of time series length on crop classification accuracies. Based on these results, we determined how early the accurate distribution maps of winter wheat, cotton, spring maize and summer maize can be acquired.

Feature importance estimation

We used both Gini importance scores from the RF algorithm and JM distances to select the optimal features for early-season crop type classification (Breiman, 2001; Bruzzone, Roli & Serpico, 1995). The JM distance ranges from 0 to 2, and larger JM distance indicates higher level of separability between the two classes. We selected JM distance in this study to estimate pair-wise crop separability for each feature because previous studies proved that JM distance have high potential to measure crop separability (Medjahed et al., 2016), and the JM distance was implemented in Matlab 2012B in this study. In addition, Gini index generated from RF was used to measure the importance of the features when considering all crops. The RF model combines multiple classification trees, and when training the RF model, each tree is constructed using two-thirds of the training records, the remaining records are used for a test classification with an error referred to the “out-of-bag error”. There are two free parameters of RF; the number of trees (ntree) and the number of features to split the nodes (mtry) were defined as 1,000 and the square root of the total number of input features in this study. During the training procedure of RF, every time a split of a node is made on a variable, the Gini impurity criterion for the descendent nodes is less than that of the parent node. The importance score is the sum of the Gini decreases for each individual variable over all trees in the classification forest. The Gini importance score were implemented using the Randomforest package in R (Breiman et al., 2013). Finally, the optimal features for identifying crop types in Hengshui were selected by considering both RF Gini index and JM distance.

The improved artificial immune networks classifier

In this study, we IAIN algorithms for crop type identification. The IAIN methods are inspired by the human immune system and have shown high potential to solve pattern recognition problems (Gong, Im & Mountrakis, 2011). There are two procedures in IAIN: training and classification. The training procedure is to use training samples as “antigens,” then generate “antibodies” from the “antigens.” The classification procedure uses “antibodies” to classify images by identifying new crop “antigens.” Normally, “antibodies” contain three attributes: the crop type label of the antibody, the center vector, and recognition radius. An antibody can identify the antigens within its recognizing radius. “Antigens” contain two attributes: the crop type label and center vector.

After normalizing all features to a range of 0–1, the normal training procedures involve five steps: pre-selection, cloning, mutation, adaptive calculation of new antibodies and antibody reorganization (Zhong & Zhang, 2012). Pre-selection (Step 1) requires selection of an antigen which best represents the unrecognized antigens; Step 2 (cloning) is to clone a large number pre-selected antigens; Step 3 (mutation) is to mutate the cloned antigens randomly and generate “possible antibodies.” Afterward, the most complex step (Step 5, adaptively calculating new artificial antibodies) uses all “possible antibodies” to identify all the unrecognized antigens. The antibody that can identify the most antigens is then defined as a new antibody. Antigens that can be recognized by the new antibody are labeled as recognized antigens, and the other antigens remain as unrecognized antigens for generating the next antibody. Finally, when all antigens have been identified by the antibodies, the training procedure concludes. During the training procedure, a similarity measurement is used to estimate the distance between the antibody/antigen center vectors. This study used the Euclidean distance for this purpose as it is easily calculated (Lhermitte et al., 2011).

To deal with image time series containing missing values, we IAIN classifier in the classification procedure by adaptively selecting classification features from “antibodies” (Fig. 4). For example, if the “antibodies” have n features but feature “F4” of pixel x is missing, the IAIN first detects the indexes of the available features of pixel x, then generates “sub-antibodies” only containing the available features of pixel x. Then the Euclidean distances between pixel x and the center vectors of all “sub-antibodies” are calculated. Finally, the pixels are labeled according to the crop type label of the antibody with the lowest distance.

Early-season crop identification evaluation

To evaluate the earliest time phase that we could accurately identify the major crops in the study region, we set the April 16–30th as the start point, and extended the time series length by 15-day. The April 16–30th was set as the start point of this study because the Sentinel-2 data of April 1–15th composition contain too few cloud-free pixels. Next, we acquired several image series with different lengths, and then used the IAIN to classify these image time series. The classification accuracies were then assessed using the validation samples, and the influence of time series length on crop identification accuracies were evaluated based on these classification accuracies. Commonly, the classification accuracies increase with the time series length, and then become saturated. We compared the time phases that classification accuracies saturated and the harvest time of each crop, and finally deduced how long we could generate crop type distribution maps before the harvests.

Non-vegetation masks

Although the time series signals of different crops have unique characteristics, there can be some noise caused by miscellaneous factors such as residential regions. We generated a sparse vegetation mask which could remove alkaline land, residential regions, water surfaces and other land with low vegetation cover. The mask used the maximum NIR TOA value during the growing season as the NIR band shows good potential for discriminating between vegetation and non-vegetation (Zhu & Woodcock, 2014). Figure 5 shows the maximum NIR TOA values of different land cover types during the growing season. The threshold was defined by subtracting the mean value of vegetation land cover (cropland and natural vegetation) from the standard deviation. In this study, the non-vegetation threshold was defined as 0.3.

Optimal feature selection

As the winter wheat was harvested in early June, we used the April and May features for early-season winter wheat identification. When considering all crop types, the Gini score for the May NIR band was 75.3 (Fig. 6), which was the highest among all features during April and May. Then, pair-wise JM distances and Gini importances were calculated for each feature (Tables S1 and S2 in the supporting files). Figures 7 and 8 show that all four optical bands and the NDVI of April 16–30 and May 16–31 periods had high JM distances (generally higher than 1.5) and Gini scores (higher than 30) when separating winter wheat from cotton, spring maize and summer maize. Therefore, the optimal features and NDVI were used for winter wheat classification (Table 3). For the April 1–May 15 periods, we also included VH and VV data of May 1–15 as Sentinel-2 optical images were not acquired in this time phase.

We used the image time series of April–August for early identification of the summer crops (cotton, spring maize and summer maize). Five NIR features (from the April 16–30, May 16–31, June 1–15, June 16–30 and August 16–31 compositions) and three NDVI features (April 16–30, June 1–15, June 16–30 compositions) were selected among the top 10 features with high Gini scores (Fig. 9). Figures 10 and 11 show that NDVI and NIR data of April and May could separate cotton from woods, orchards and winter wheat, as these crops green up earlier than cotton. Then, spring maize was separable from cotton using June features, and summer maize was separable from cotton using June and July 1–15 composition features. Figures 12 and 13 show that woods, orchards and winter wheat were separable from spring maize in April and May, and features of the June 16–30 and July 1–15 compositions could discriminate spring maize and summer maize. For summer maize identification, as summer maize is mainly sown in late June and early July, its feature separability and importance during June–August were estimated. Woods, orchards and cotton crops were separable from summer maize in June (Figs. 14 and 15). However, summer maize and seedling nurseries had lower separability as their JM distances were lower than 1.0 during June and August. Only the NDVI of the August 1–15 composition had relatively high JM distance (0.91). Generally, as NDVI and NIR contributed the most to separating the summer crops, we used NDVI and NIR time series identify summer crops in early season, VH and VV data of May 1–15 were included because Sentinel-2 data were not acquired (Table 4).

Feature contribution results showed that features from April and May could discriminate summer crops (such as cotton and spring maize) from natural vegetation (such as woods) in this study, as natural vegetation greens up earlier than crops. June was the fast growing season of cotton and spring maize; therefore, features from this period can be used to discriminate between them as they green up at different speeds (Wardlow, Egbert & Kastens, 2007). Among the features used in this study, NDVI and NIR had the highest separability and contributed most to crop classification. This is consistent with Zhu & Woodcock (2014), who observed that NIR could best discriminate between agricultural land and forest. Although texture features are effective for discriminating between cropland and woods in very high-resolution images, such as Gaofen-1 data at 2 m resolution (Turker & Ozdarici, 2011), these features were not considered in this study because the spatial resolution of the Sentinel images used was 10 m. Hao, Wang & Niu (2015a) reported that cotton and spring maize in Xinjiang were often confused, but this was not the case in this study. Basically, the optimal time phases selected for early season crop type classification varied in different years and study regions, as the growing season of crops have inter-annual and spatial phenological variation (Hao et al., 2016; Zhang, Tan & Yu, 2014). Therefore, when training to identify crop types at early season in a new study region, the contribution of the time phases for crop classification should be reevaluated.

Classification accuracies of IAIN for “missing value” pixels

Although the 15-day composition method was used to improve the quality of the image time series and reduce the number of missing values, there are still some pixels containing “NaN” values. Meanwhile, most existing classifiers, such as maximum likelihood classification, support vector machine and RF, cannot deal with missing values, if the input data contain “NaN” value, the these classifiers will return “not classified” as output. In this study, we proposed IAIN classifier to deal with the “missing value” problem.

To assess the potential of the IAIN of dealing with “missing value” pixels, we firstly used April–June and April–August image time series to identify winter wheat and summer crops, and used validation samples with “missing value” to verify the classification accuracies. We acquired 4,293 samples during April–June for winter wheat validation, and 8,065 samples during April–August for summer crops validation. Results (Tables 5 and 6) show that IAIN could identify pixels with “missing value,” the OAs of winter wheat and summer crops were 99% and 98.55%, respectively. Only a few winter wheat, cotton and summer maize samples were mislabeled as “Non-crop.” While, as the number of spring maize was small (only 101 with “missing value”), the producer’s accuracy (PA) and user’s accuracy (UA) of spring maize classification maybe not 100% in reality.

Some subset images of winter wheat and summer crop classification maps are shown in Figs. 16 and 17. The red patterns in Fig. 16 and green/blue pattern in Fig. 17 were “missing value” patterns. And the classification results showed that the “missing value” patterns were generally correctly classified. However, there are still possible shortages when using IAIN classifier to deal with missing value. Firstly, the high separable features should be acquired for crop type classification, crops cannot be identified if only low separable features were used as inputs for the IAIN classifier. In addition, the similarity measurement used in this study is Euclidean distance, it is sensitive to the noise in the time series although it is easy calculating (Lhermitte et al., 2011). Thus, other similarity measurements could be considered to further improve the robustness of the IAIN classifier.

Early-season winter wheat, cotton, spring maize and summer maize mapping

Figure 18 shows that the VH and VV polarization data from the April 1–15 composition could not identify winter wheat, as both the PA and UA for winter wheat were low. Both PA and UA increased when the optical-band and NDVI data from the April 16–30 composition were used. For the April 1–May 15 and April 1–May 31 data, both PA and UA were higher than 95% (96.12% and 96.19% for May 15, and 99.28% and 95.42% for May 31, respectively). This indicated that winter wheat could be identified at May 15, which is approximately 20 days before the winter wheat harvest (early June). As both winter wheat and woody plants had high vegetation fractions, the misclassification of winter wheat is mainly due to confusion with orchards. This is consistent with the low separability of orchards, woods, seedling nurseries and winter wheat crops (Fig. 6). Figure 19 shows the spatial distribution of winter wheat generated from the April 1–May 31 data. Winter wheat was the dominant crop type in the study area and was present in each county. The winter wheat map was generated at 10 m resolution and shows the field boundaries clearly.

Figures 20 and 21 show that images obtained during April 1–May 30 could not identify cotton and spring maize, as the PAs and UAs were lower than 60% in most cases. This was mainly because the crops were in their early stage, and the early signatures captured by satellites cannot discriminate between cotton and spring maize. When the June 1–15 image composition was used, the PAs and UAs of cotton and spring maize began to increase, which was consistent with the high JM distance between the two crops. Then, both the PAs and UAs of cotton and spring maize became greater than 95% when the length of the time series was increased to April 1–July 15 (PA and UA were 95.42% and 99.30% for cotton, and 97.92% and 98.95% for spring maize, respectively, at July 15). Use of the longer image time series did not increase the classification accuracies of these two crops. Spring maize was harvested in late August and cotton was harvested during September and October. Hence, an accurate distribution map was generated from April 1–July 15 data, which was more than one month before the harvests of these crops.

Figure 22 shows that PAs and UAs were higher than 80% using images composited from June 1–15 and June 16–30 data. The relatively high accuracies obtained for early-stage summer maize was consistent with its high separability (Figs. 15 and 20) caused by its low vegetation fraction in the early season. With image time series lengths of June 1–August 15, the PA and UA were higher than 95% (95.81% and 99.88%, respectively, for summer maize at August 15), and use of a longer image time series did not improve the classification accuracy. The summer maize was harvested in late September and early October. Hence, an accurate early summer maize map could be generated from August 15 data—one month earlier than the harvest.

Figure 23 shows the spatial distribution of cotton, spring maize and summer maize generated from April 1–August 15 data. Summer maize was the dominant crop in the study area as it was sown after the harvest of winter wheat, while cotton was mainly planted in the southwest of the study area. Although some pixels had missing values due to cloud, the IAIN classifier used in this study could deal with these problems and generate crop maps of high quality. In addition, we only conducted pixel-based classification as the object-segment may introduce some uncertainty for heterogeneous cropland. The results clearly showed the field boundaries, which proves the potential of using Sentinel images for mapping fragmented crop fields.

We then estimated how early we could acquire accurate crop distribution maps. Our result shows that: (1) the classification accuracies of winter wheat is saturated with April 1–May 15 image time series, which is 20 days before harvest time (early June). (2) The classification accuracies of cotton and spring maize were saturated using the April 1–July 15 image time series, which was 4–6 weeks earlier than harvest time. (3) The classification accuracies of summer maize were saturated using the April 1–August 15 time series, which is one month earlier than the summer maize harvest. This study evaluated the potential for early season identification of multiple crops (winter wheat, cotton, spring maize and summer maize) at finer (10 m) spatial resolution than similar existing studies (Hao et al., 2015c; Skakun et al., 2017; Song et al., 2017; Vaudour, Noirot-Cosson & Membrive, 2015). It provides valuable information for crop growth monitoring and management in regions with fragmented crop fields.