Real-time Weed Identification Using Machine Learning and Image Processing in Oil Palm Plantations

The effectiveness and efficiency of the operation of oil palm plantations are considered to be the most crucial factor to develop the productivity and profitability of the palm oil business. One of the major obstacles for the plants to optimally produce crops based on their capacity is caused by the presence of noxious weeds in the plantation area. However, weed control via chemical processes may potentially harm the surrounding environment if it is not properly managed. Therefore, an automatic system to assist the farmers to identify and control the weeds is required to minimize harmful impacts on the environment. Machine Learning (ML) and Artificial Intelligence (AI)-based systems provide powerful tools to perform such tasks. In this work, we aim for an ML-based system design to perform an automatic weed recognition task. The methodology can provide an effort for environmental sustainability in oil palm plantations. The weed identification involves the description, the local names, and tolerance class of the weeds as well as suggestions to control them. The flow of this work consists of weed and herbicide data acquisition, data labeling, model configurations, and data training. Further, the proposed system can be adopted as an android-based application in mobile devices that can deploy the trained model to predict weed category in both real-time and non-real-time tasks.


Introduction
The expansion of Indonesia's oil palm plantation sector becomes a pivotal part of the national economic development. According to the letter of Agricultural Ministerial Decree number 833/KPTS/SR. 020/M/12/2019 the total area of Indonesia's oil palm plantations is up to 16.38 million hectares across 25 different provinces [1]. A survey by the Indonesian government identifies that the under capacitated human resources in oil palm plantation operations is considered to be one of the main factors that may not meet the targeted demand of the oil palm production and supply, especially oil palm smallholder-owned plantations. The oil palm plantation industry is the backbone of the economy of around 16.2 million people in Indonesia who are directly and indirectly involved in that sector. The primary focus of the sector is to fulfill requirements for maintaining environmental sustainability to ensure the quantity and the quality of palm oil production. Improving the capacity and the capability of the oil palm farmers is one of the main goals according to Presidential Instruction  The principles of Agricultural Precision need to be implemented to improve operational effectivity and the efficiency of palm oil plantations for optimizing the productivity and the profitability of the palm oil industry [3]. Agricultural Precision is also known as Site-specific Management (SSM). The utilization of this methodology focuses on the management that is oriented toward the specified characterization of the plantation area in which different parts of the area are treated following their conditional state.
One of the major obstacles in optimal oil palm production is noxious weed invasion in plantations. Weeds are invasive plants that grow together with cultivated plants. The weeds compete with the cultivated plants for critical elements such as water, sunlight, and nutrients and hence decreasing the growth and the productivity of the plants. Therefore, weeds need to be identified and classified to provide correct information about weed control methods [4]. Recently, the most effective weed control relies on herbicide usage with various modes of action because the method is capable of killing 90% to 99% of the targeted weeds [5]. However, oil palm farmers may not have proper knowledge on recognizing weeds and planning an appropriate weed control strategy. In this regard, a supporting system to help farmers to identify weeds and to arrange a correct weed control scheme to minimize the environmental impacts.
A crucial step in weed control management is weed identification as a basis for choosing the type of herbicide and weed treatment. The system can be considered as part of Agricultural Precision to realize oil palm sustainability. This can be implemented by developing an automatic system via ML and AI-based computer vision that can accurately predict and classify targeted objects. The weed classification task includes not only the local name of weed species but also the tolerance class of weeds and advising an herbicide-based weed control. The weed control information consists of the trade names of the herbicide for the identified weeds, active chemical labels, and a proper herbicide application. To do so, in this work, we propose a system that adopts an AI-based computer vision to recognize weeds in oil palm plantations and integrates the system to recommend appropriate planning for herbicide-based weed management by including the aforementioned information about the herbicides. The system can be used for both real-time and offline purposes.

Weed control in oil palm plantations
Weed control is the main component of the oil palm plantation because the tropical climate condition, rainfall, and light intensity are not only optimal for the oil palm cultivations but also the noxious weed growth [6]. It is well-known that weeds that invade oil palm plantations can deteriorate the quality and decrease the quantity of the Fresh Fruit Bunch (FFB) of the palm oil plants, hinder the growth of the plants, increase the probability of invasion by pests and diseases [7,8]. The weed growth also interrupts the fertilization and crops that can increase the operational time and cost for the farmers [9].
There have been studies about several weed species that affect oil palm production such as Asystasia gangetica, Clidemia hirta, Dicranopteris linearis, and Imperata cylindrica. These weeds tend to grow to cover plantation lands and their aggressivity to compete for water and nutrients with oil palm plants. The quantification of crop failures attributed to the above species of weeds has been studied by Hakim et al. [9]. For example, Mikania micrantha, a species of weed that grows in between rows of oil palm trees, causes a drop in FFB up to 20% of the potential yields for four-year cultivation. On the other hand, weed species A. gangetica that covers spaces between plants can cause a crop failure estimated at 1.86 tons/hectare/year for more than two years. Other weeds such as I. cylindrica, D. linearis, and S. palutris give rise to a decrease of FFB for more than 20%. In addition to the crop failure, the uncontrolled weeds cause the operational cost to soar up to 17 -27% of the total cost to account for weed management before and during the production phase [10].
An adequate weed control effort relies heavily on the farmers' knowledge to identify different types of weeds. The lack of such knowledge becomes an issue in the majority of oil palm smallholder farmers as demonstrated by Edwina et al. and Yesmawati and Wibawa, including the proper usage of herbicide [11,12]. The effectiveness of herbicide-based control to kill weeds is more than 90% [5,13]. Nevertheless, other factors such as dosage, concentration, mixture, manual instruction, and targeted 3 weeds need to take into consideration by farmers before using herbicide [14]. There are about 250 active chemical substances in commercial herbicides sold with thousands of trade names in Indonesia. In reality, there is not a particular herbicide that can control all known noxious weeds [15].

Computer Vision in Palm Oil Plantation Sector
Computer vision is one of the artificial intelligence that can automatically extract patterns in images by providing training images for the algorithm to recognize and predict a new image [16]. One of the most popular AI techniques is Convolutional Neural Networks (CNN) that finds a wide variety of applications in computer vision, signal processing, weather prediction, and more [17][18][19][20]. CNN can learn complex and inherent patterns within an image by expanding them into arrays represented by pixel intensity (0 -255) for every point in the image. Next, CNN performs a deconvolution process, extracts thousands of relevant selected features, and then merges patterns that can be identified as a basis for recognizing a new image. With recent advances in libraries and Application Program Interface (API) for computer vision tasks, the utilization of machine learning for object identification and classification is available for various research fields, including agricultural precision [21][22][23]. We used Tensorflow for this work, which is one of the most popular open-source frameworks for ML.

Methodology
In accordance with the objective of this study, we propose a methodology to build a system design that can identify weeds via an AI model. The main flowchart of our method is shown in Figure 1. The weed data acquisition is obtained from a literature review regarding different types of noxious weeds growing in Indonesian oil palm plantations. These weeds are further categorized into four categories based on the dangerous level of the growth of palm oil trees as shown in Table 1. Meanwhile, the herbicide data is obtained from the official website for pesticide information of Directorate of Fertilizers and Pesticides, Directorate General of Agricultural Infrastructure and Facilities, Ministry of Agriculture [24]. Today there are around 300 active chemical substances in herbicides officially regulated in Indonesia. The herbicide information is arranged with labels that include targeted weeds, chemical compounds, trade names, and herbicide applications.

Data Labeling
The raw data from the previous step data that consists of images and texts proceed into the data labeling step. Each weed image is identified by experts by providing information about the species name as well as the local name of the plants. Further, the labeled data is arranged to be training data for the AI model.

Training Process
The configuration of the training data and the initiation is implemented using Python programming with Tensorflow library (see Figure 2). Despite the availability of high-performing AI models, the storage capacity of mobile devices and the model's prediction speed need to be wisely considered for efficient deployment in our proposed system. AI models with a huge number of hyperparameters may require a large size of memory on mobile devices and may have a very slow processing time of the prediction stage [25]. Fortunately, several models such as MobileNets and EfficientNets have been contributed to the current development of publicly available lightweight CNN models with relatively high performances in a mobile setting. Comparing the performances of these models (in terms of model accuracy, model efficiency, memory, and speed) is one of the pivotal steps in developing our AI-based design.
For the first training step, the used training data set is generally 30% of the total images obtained from the previous step. Images included for the training process are selected randomly and proportionally for each weed category. The number of data is increased gradually in a such way that the model can at least achieve an accuracy of 90% to classify weed images. The addition of data into 5 the set will be halted once the accuracy is around 98% to avoid model overfitting because the testing accuracy can decrease. If the trained model has achieved the desired accuracy, it will be saved for the prediction step.

Prediction Process and Generating Application
The development of information systems via mobile or web applications aims to provide two essential features: weed and herbicide data analysis via an AI-based recognition model and data management [26]. We have built such similar systems in our several previous agricultural-related research to support the above two features [27][28][29][30]. Tensorflow framework provides a service to generate an android-based application using different available templates. The interface used in this work is based on the template from Tensorflow. The file of the application with APK extension is downloaded and installed in mobile devices that support the android operation system. The training and prediction processes can be run via a lightweight computing cloud.
The image prediction step consists of two schemes. The first one is to predict data uploaded into the system (non-real-time) and the second scheme deals with real-time image prediction. The prediction utilizes a transfer learning method that uses the trained model from the training process to classify new images from users.

Non-real-time Image Prediction
The non-real-time prediction is started by uploading the weed images into the access of drive of devices. The allowed image formats are given as: A flowchart to describe this scheme is shown in Figure 3. The uploaded images are processed using the saved AI model. After that, the system will read the images to extract patterns for the recognition process and the results will be saved into a protobuf file. The system proceeds to label the results of the model based on the label map. This consists of the species name, the local name, the dangerous level, herbicide-based control advice. The final output of the system can be downloaded as a PDF file and saved on the device.  Figure 3. Flowchart of the non-real-time prediction

Real-time Image Prediction
The process for the real-time prediction starts by opening the application and running the camera to take weed pictures (see Figure 4). The system will read the obtained image and proceed to the prediction process that is similar to the non-real-time prediction. The results will be saved in a protobuf file. The output will be labeled using the same information as the first scheme. The reported results will be saved in PDF into the mobile device.

Testing the application
The last step in this proposed methodology is to test the accuracy in both schemes. The sample images used for the testing need to be from real weeds obtained in oil palm plantations. The prediction precision is the parameter that needs to be examined that compares the results from the system and the annotation made by the weed experts.

Conclusion
In this work, we propose a methodology to develop an automatic system to help palm oil farmers to identify weeds, which are invasive plants that have detrimental effects on the oil palm plantation sector. The system adopts an AI-based model in the Tensorflow framework that can be easily deployed into mobile devices that run an Android operation system. The system is extended to be connected with a database obtained from data acquisition that contains information about weed control and herbicides usage. To further improve the proposed system, the farmers need to collect as many local weed images as they can find in the plantation areas so that the system can be adapted for sitespecific purposes in their area since the characteristics of palm oil plantations, including the weeds species, might be different for different locations. Further, the performance and the efficiency of various lightweight CNN models deployed in the mobile system need to be compared to determine. In addition, the methodology may include useful information such as fertilization and yields so that it can be combined with weed management to estimate quantitatively the operational cost.