Ramie Plant Counting Based on UAV Remote Sensing Technology and Deep Learning

ABSTRACT Plants number is an essential field phenotypic trait that affects the growth status and final quality of crop. In recent years, the integration of remote sensing technology and deep learning technology has provided a solution to the problem of crop plant counting in field. However, most of the previous studies have selected fixed crops (such as rice, wheat) for research, and few studies have reported the limitations in the application of this technology. In addition, as far as we know, there has been no report on the problem of ramie germplasm resources counting. In this study, in combination with DA (Data Augmentation) and three object detection algorithms, ramie germplasm resources were adopted to explore the accuracy of counting plant number under the condition of dense plant growth. The following functions were tested: (1) the influence of DA on the effect of plant counting; (2) the influence of ground sampling distance (GSD) on the effect of plant counting; (3) the influence of object detection algorithms on ramie detection object. The results showed that after the training sample was expanded by DA, the Precision of ramie plant counting model was increased by 6.630%. FCOS (Fully Convolutional One-Stage Object Detection) could perform better in small object and small sample data (Recall = 0.892, Precision = 0.819，RMSE = 0.089). It was necessary to ensure the consistency of GSD between training samples and verification samples for improving the accuracy of ramie plants counting. The ramie plant counting model has sufficient and stable ability to count ramie plants in the field, which can supplement the traditional manual counting method.


Introduction
Ramie is one of the important characteristic crops in China, which has abundant value in the fields of economy (Xiong et al. 1998), ecology (She, Jie, andXing et al. 2011), medicine (Tian et al. 2015) and culture (Cui et al. 2021). Plant number is one of the essential phenotypic traits of ramie, which can not only help producers to make correct management decisions by reflecting field density information and growth status (Wrather, Phipps, and Stevens et al. 2008), but also effectively evaluate and predict the final yield of ramie (Hu and Peng 1998), so as to assist breeders to quickly screen ramie varieties with high yield potential. Ramie plant counting by human visual detection is time-consuming, laborious and destructive. To solve this problem, many scholars have investigated the potential of plant counting using machine learning technology with images obtained from different crop phenotype platform (Fernandez et al. 2018;Madec et al. 2019). In these studies, ground fixed or mobile platforms can collect high-resolution and high-quality images, but were also inevitably limited by coverage area, topographic relief and other factors (Bronick and Lal 2005).
The development of unmanned aerial vehicle (UAV) remote sensing technology provides a new way to obtain crop information in a large area. UAV has the advantages of high efficiency, nondestructive, easy operation, and can carry multiple low-cost portable sensors to obtain high-definition crop canopy images (Jiang et al. 2022;Sun et al. 2022), so as to detect plant number. For example, He, Hao, and Xin (2020) improved wheat ear detection accuracy using the modified You Only Look Once v4 algorithm (YOLOv4) based on UAV remote sensing images, and the f1 score reached 96.71%. Velumani et al. (2021) used Faster Region-based Convolutional Neural Network (Faster-RCNN) to detect maize at the third-to-fifth leaf stage based on high-resolution remote sensing images, with the rRMSE reaching 0.10.
Current studies on plant counting using UAV remote sensing images have been focused on few crops, such as rice (Sun et al. 2022), wheat , cotton (Feng et al. 2020), etc., and crop varieties are relatively simple. The difference of image features among crops and crop varieties, and of different object detection network structure, will result in different detection effect. When conducting plant counting studies of a new crop, it is still a challenge to extract characteristic traits from original image data to develop efficient and stable models (David et al. 2020), because the performance of the model largely depends on the number and diversity of training sample (Alom et al. 2019;Ghosal et al. 2019), which needs to consider the variability of collection times, observation conditions, genotype differences and other aspects (Consuelo et al. 2021). Therefore, the limitations of plant counting based on UAV remote sensing technology will be discussed in this study.
A key question in this study is whether there are limitations in the sample number, sample diversity, image resolution and model algorithm when using UAV remote sensing images and Deep Learning Technology (DL) to construct plant counting model, and whether detection for dense crop object can be achieved with fewer marks by using Data Argumentation (DA) and DL. The objectives include: (1) evaluating the contribution of DA to ramie plant counting models; (2) evaluating the accuracy of ramie counting models with different algorithms and ground sampling distance (GSDs); (3) verifying the performance of this model in multi-genotype ramie.

Study sites
This study was conducted in Changsha, Hunan (28°11 ′01.981″ N, 113°04′ 10.159″ E) from 2019 to 2020. The area consisted of two experiments (Figure 1). With abundant precipitation, sufficient light and heat conditions, the study site is one of the main ramie production areas. Meanwhile, the flat terrain provides favorable conditions for UAV-based remote sensing monitoring of ramie.
Experiment site 1: The area contains two different varieties of ramie, with each variety grown in 12 plots (a total of 24 plots). Each plot area was about 12 m 2 , with 4 rows × 8 columns. The planting density was 92,500 plants per hectare, and the row spacing was 0.2 m. Experiment site 2: The area used for germplasm-resource experiment consisted of 154 plots, with each plot area of 2 m × 1.8 m, 2 lines × 4 stumps. The stump spacing was 0.4 m, the row spacing was 0.6 m, and the gutter width was 0.5 m. There was no repeated plot in the experimental area, namely one variety of ramie was planted in one experimental plot. There were 154 germplasm resources in total, and the materials had diversity in morphological characteristics and agricultural traits.
In order to complete the reconstruction of the model later and improve the accuracy of longitude, latitude and elevation position, multiple ground control points (GCPs) were uniformly arranged in space and height in the test site.

Experimental plan
Experiment 1: This experiment was designed to investigate whether there were limitations with respect to sample number during counting plant number based on UAV images and DL, and to evaluate the effect of DA on improving the accuracy of plant counting model constructed by small samples. Remote sensing images of ramie canopy at site 1 were collected by UAV during ramie maturity period (May 21, 2020). Forty-nine images were captured and divided into training samples and validation samples in a ratio of 1:4. DA was used to expand the training samples, which were used to establish ramie plant counting model. Experiment 2: This experiment was used to evaluate the effects of GSD and object detection algorithm on the ramie counting accuracy. During the ramie maturity period in the year of 2019 and 2020, UAV was used to collect ramie canopy images in site 2. The images were divided into three datasets: (1) V1 contained images from 2020 and was mainly used to compare the detection effects of different algorithms. (2) V2 included images from 2019 to 2020, which was used to evaluate the performance of ramie plant counting model in different years. (3) V3 included native high-resolution images acquired in 2020 and images with lower resolution transformed by Down-sampling method. V3 was used to evaluate the impact of different GSDs on the accuracy of ramie counting model (Table 1).

Data acquisition
Data collection was completed by Inspire 2 UAV platform (Shenzhen, China) with a high-resolution RGB camera. The UAV was flying at about 1.1 m/s speed at an altitude of 15 m. The camera moved above the crop vertically to maximize crop canopy information. The exposure mode of the camera was automatically selected. The exposure was 1/200s, ISO sensitivity was 100, and aperture was 1.53. Field images were acquired at the local time of 12:00-14:00 under clear and sunny weather conditions, which could ensure the quality of images.

Methods for ramie plant counting
The research steps are shown in Figure 2. LabelImg was used to label individual plants. Then, DA was used to expand the training samples, and Down-sampling was used to change the GSD of images. Finally, the object detection algorithm was used to construct ramie plant counting model.

Manual labeling of individual plants
The individual plants were labeled with an offline tool, LabelImg. Bounding boxes around each ramie plant were interactively drawn and saved in the Pascal VOC format as TXT files. The file contains the position information of each labeled plant, and the actual plant number.  Time  Image type  Image number  V1  2020  High-resolution images  177  V2  1  2020  High-resolution images  177  2  2019  High-resolution images  60  V3  1  2020  High-resolution images  177  2  2020  Medium-resolution images  177  3 2020 Low-resolution images 177 Figure 2. The research steps.

Data augmentation
Mosaic, a new DA approach was employed to perform data transformation of the sample database in advance. Through this DA method, four images were integrated by random scaling, random clipping and random arrangement.

Methods for image down-sampling
To create the synthetic lower resolution datasets, Gaussian filter were applied to downsample the native high-resolution datasets (Figure 3). Gaussian filter process can provide low-resolution images similar to the native low-resolution UAV images. In this study, a Gaussian filter with a sigma = 0.6 was used to create Medium-resolution images, and a Gaussian filter with a sigma = 0.8 were used to create Low-resolution images.

Training environment of object detection model
The machine adopted in this study was ThinkPad P15 GEN1 (CPU, Xeon W-10885 M with 8-core and 16-thread; GPU, Nvidia RTX5000, with 16 G video memory and 64 G running memory). During the application of YOLOv5, CSPDarknet was used as backbone, and YOLOv5s as pre-training weight. SGD was used to optimize function. During the application of Faster-RCNN, Resnet101 was used as backbone and FPN was used to optimize function. During the application of FCOS, Resnet101 was used as backbone, and FPN was used to optimize function. All models were trained 20 times.

Evaluation metrics
If Root mean square error (RMSE) and Coefficient of determination (R 2 ) were employed to quantify the performance of ramie plant counting. RMSE ¼ ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi 1 n Figure 3. The Ramie Canopy image before and after down-sampling.
Where, n represents the number of test images, Ri represents the reference counts for images, Di represents the detected counts for images, and¯Ri represents the mean reference count.

Data augmentation provides better model performance
In Experiment 1, the focus was placed on the effect of DA on the plants counting accuracy of ramie. It can be found that the performance of ramie plant counting model was optimized to a certain extent after the training samples expanding with DA. The number of original training samples was 214, which was increased to 1522 after DA. Table 2 presents the effect of DA on the counting accuracy of verified samples. After the application of DA, the Precision of this model increased by 6.630% from 0.920 before DA to 0.981 after DA. After DA, the Recall of this model decreased by 0.514% from 0.972 to 0.967. The R 2 of the regression model based on actual and predicted plant numbers was low before the application of DA, which was 0.03. After DA, R 2 was 0.728. The result indicated that DA was an effective method to deal with insufficient data in object detection task. The accuracy of object detection can also be improved by training the expanded samples.

FCOS is more suitable for small object detection
In Experiment 2, dataset V1 was divided into training samples and validation samples according to the ratio of 4:1, and then three object detection algorithms were respectively used to investigate the effect of object detection algorithm on the plants counting accuracy of ramie (Table 3). The results show that three models can detect ramie plants accurately, with the Recall of 0.892 ~ 0.897, the Precision of 0.796 ~ 0.819, and the RMSE of 0.089 ~ 0.127. FCOS had better detection performance, with the Precision being 0.819, the Recall 0.892 and RMSE 0.089. Figure 4 shows the classification results of validation samples based on confusion matrix. The actual number of ramie plants was 1409. During the application of Faster-RCNN, the number of correctly detected ramie plants was 1264, the number of incorrectly detected ramie plants was 324 and 145 were not detected. During the application of FCOS, the number of detected ramie plants was 1257, the number of incorrectly detected plants was 278, and 152 were not detected. During the application of YOLOv5, the number of correctly detected ramie plants was 1261, the number of incorrectly detected ramie plants was 315 and 148 were not detected. It can be seen that Faster-RCNN and YOLOv5 incorrectly detected more object than FCOS and had higher RMSE, which indicates that they were

Analysis of model generalization ability
In Experiment 2, in order to evaluate the applicability of ramie plant counting model, V1 was used as training sample, and V2 was used as validation sample. Table 4 shows that ramie plant counting model has good generalization ability and robustness. When data from different years were used, the Recall was 0.963, the Precision was 0.805, the RMSE was only 0.196, and 1582 plants were correctly detected. This result indicates that ramie plant counting in the field based on UAV remote sensing and deep learning is feasible and has practical value in plant detection of different varieties and different years. Table 5 analyzed the effect of GSD on the plants counting accuracy of ramie. As can be seen from this table, the ramie plant counting model based on FCOS was very sensitive to GSD. When the model was trained and applied on the high-resolution images, the Recall, and the Precision were 0.892 and 0.819, respectively. When the model was trained on high-resolution images and applied to mediumresolution images, the Recall and the Precision of the model were 0.890 and 0.822, respectively, which had no significant difference from the results of training and application on high-resolution images. When the model was trained on the medium-resolution images and applied to the highresolution images, the performance of ramie plant counting model was significantly lower than that of  trained and applied on the high-resolution images, with the Recall and the Precision reduced by 1.7% and 0.7%, respectively. When the model was trained on low-resolution images and applied to lowresolution images, the performance of ramie plant counting model was improved compared with the results of training and application on high-resolution image, the Recall and the Precision were increased by 0.3% and 0.1%, respectively. This indicates that ramie plant counting model was sensitive to GSD, and it is necessary to maintain the consistency of GSD between training samples and validation samples, because FCOS model tends to detect objects of similar size to training samples. In addition, it is found that higher image resolution does not mean more accurate ramie plant counting. When trained and applied on medium-resolution images, the model had better detection effect (Recall = 0.894, Precision = 0.822, RMSE = 0.088), indicating that lower resolution images contained enough information to accurately detect ramie plants, and the relatively lower resolution made the information concerned by the model more effective. Figure 5 shows the classification results of validation samples with different GSDs. It can be seen from the figure that the actual number of ramie plants was 1409. When the model was trained and applied on the medium-resolution image, 1260 ramie plants were correctly detected, 273 were incorrectly detected, and 149 were not detected. The results of incorrectly detection were 5 and 4 fewer than those of high-resolution image and the lowresolution image. The results demonstrated that the GSD could exert an impact on the final evaluation effect (R 2 , RMSE).

The effect of GSD on ramie plant counting
Where, Tn represents the training samples is native high-resolution images, T0.6 represents the training samples is medium-resolution images, T0.8 represents the training samples is low-resolution images, Vn represents the validation samples is native high-resolution images, V0.6 represents the validation samples is medium-resolution, V0.8 represents the validation samples is low-resolution images.

Limitations of crop counting
Under the condition of small samples, the proposed method with DA to expand the dataset is effective. By training samples converted with DA, the accuracy of ramie plant counting model is improved. The advantage of applying DA lies in reducing the workload of sample labeling, increasing the number of training samples and improving the generalization ability of this model. Currently, the complete largescale labeled database has not been established in the vast majority of crops. The proposed scheme has an important reference value for the investigation based on deep learning.
Lacking sample diversity is an important factor hindering the establishment and popularization of crop phenotypic analysis models. It is difficult to obtain samples containing different growth environments, genotypes, and other variable characteristics in a single experiment. In this study, a plant counting model based on 154 ramie germplasm resources was established, and the results demonstrated that the model had good performance with the Recall of 0.897, the Precision of 0.819.

Counting problems in the agricultural field and the evaluation of detection effects
In recent years, an increasing number of scholars have further improved the accuracy and calculation speed of plant counting with the assistance of the integrated application of image processing technology and deep learning. The method based on object detection can be employed to perform the object counting through collecting detected objects, and it can provide more detailed information, including the number, location and size of individual crops. In this study, the application effects of different object detection algorithms (FCOS, YOLOv5 and Faster RCNN) were compared. For ramie, one seed can produce multiple tillers, and ramie grows extremely dense, which significantly increases the difficulty of object detection. The result shows that FCOS has certain advantages in small object detection task.
Due to the fact that crop detection is accompanied by the whole crop growth stage, a real-time and efficient platform is required to obtain crop phenotypic information in the field. The crop canopy information can be rapidly obtained by UAV remote sensing technology, followed by the obtainment of the crop density information in the field. This study demonstrates that the plant number calculated from UAV-RGB images had a high consistency with that obtained by manual visual observation. This method is much more advanced than traditional manual counting. Firstly, UAV can obtain crop phenotypic information in a nondestructive, rapid and high-throughput manner, reduce the cost of manual counting, and save time and effort. Secondly, this method overcomes environmental factors and improves the accuracy of counting.

Influence of GSD on monitoring effect
In this study, Gaussian filtering was used to downsample the native images to synthesize lowresolution images. The result shows that there was no significant difference in model effect under different GSDs. When the model was trained and applied on medium-resolution images, it has better detection effect (Recall = 0.894, Precision = 0.822, RMSE = 0.088), which was inconsistent with the results of previous studies. Some studies believed the model constructed based on high-resolution images had more stable performance and can significantly improve the accuracy of crop object capture (Jin et al. 2017). This may be because the low-resolution ramie images contain enough valid information to accurately detect ramie plants. In addition, the ramie plant counting model based on FCOS was sensitive to image GSD. Therefore, the optimal aerial height should be selected to obtain high resolution images under the premise of not affecting the image quality. Meanwhile, the acquisition throughput of UAV remote sensing system should be controlled as much as possible to improve flight efficiency while providing high resolution.

Conclusion
In this research, we proposed a method to detect the number of ramie plants in UAV imagery, which solved the problems of small sample, complex crop background, small, dense detection objects. The results showed that data augmentation can improve the effect of plant counting and is an effective method to overcome the problem of insufficient sample. After DA, the detection precision was improved by 6.630%. We compared the performance of different object detection algorithms, and FCOS been proved have advantages in small objects with the Recall of 0.892, the Precision of 0.819, and the RMSE of 0.089. Moreover, ramie plant counting model was sensitive to image GSD, so it was necessary to keep the GSD consistency between training samples and validation samples. UAV remote sensing combined with deep learning technology provided a new way for crop density monitoring.
The proposed method is simple and inexpensive, which could be used for a wide range of applications, including detection for other crop types.