A Deep Learning-based Mobile Application for Segmenting Tuta Absoluta’s Damage on Tomato Plants

-With the advances in technology, computer vision applications using deep learning methods like Convolutional Neural Networks (CNNs) have been extensively applied in agriculture. Deploying these CNN models on mobile phones is beneficial in making them accessible to everyone, especially farmers and agricultural extension officers. This paper aims to automate the detection of damages caused by a devastating tomato pest known as Tuta Absoluta. To accomplish this objective, a CNN segmentation model trained on a tomato leaf image dataset is deployed on a smartphone application for early and real-time diagnosis of the pest and effective management at early tomato growth stages. The application can precisely detect and segment the shapes of Tuta Absoluta-infected areas on tomato leaves with a minimum confidence of 70% in 5 seconds only.

INTRODUCTION Agriculture is an important economic sector of the Tanzanian economy, contributing about 29.1% of Gross Domestic Product (GDP) and 67% of total employment [1]. Tanzanian farmers grow a variety of crops for food and economic purposes. As one of the most widely grown crops in the world [2], the tomato plant is also grown in different parts of Tanzania. In 2017, 247,135 tons of tomatoes were harvested in 54,520 hectares, which is equivalent to 64% of all fruits and vegetables in the country [1]. Tomato (Solanum Lycopersicum L.) is considered a high-value crop and income resource for smallholder farmers in Sub-Saharan Africa [3]. Nevertheless, the invasive pest known as Tuta Absoluta is a major threat to tomato production [4]. It can greatly damage tomato yield to the extent that growers may give up production due to the costs and losses it causes. Yield losses can reach percentages as high as 80-100% if no control measures are taken [5]. The pest is native to South America but has spread quickly, not only across the Mediterranean basin but also across Europe, Middle East, Asia, and Africa, where it was first recorded in Algeria in 2008 [6][7][8]. The first case of Tuta Absoluta in Tanzania was recorded in August 2014 in Arumeru District, Arusha, and it has since spread to the other regions of the country [9]. The number of extension officers who are key facilitators in providing farmers with proper information on plant diseases and pest control is very limited to meet the farmers' demands in Tanzania [10]. Although farmers and extension officers struggle with different methods to control the pest, there has not yet been an effective mechanism to exploit the Tuta Absoluta's infestation on tomato leaves at early stages before causing great yield losses to farmers.
The application of Artificial Intelligence (AI) plays an important role in precision agriculture due to its flexibility, high performance, accuracy, and cost efficiency [11][12][13]. Computer vision techniques, such as deep Convolutional Neural Networks (CNNs) have shown promise to transform the agricultural field in plant disease diagnosis. Several researchers developed CNN models using image datasets for pest and disease diagnosis in plants like tomato, banana, apple, cassava, cherry, alfalfa, wheat, and grapevine [14][15][16][17][18][19][20][21]. Deploying these CNN models on mobile phones would be beneficial in making them accessible to everyone, especially farmers and agricultural extension officers. Despite the widespread use of smartphones, CNN models are currently being deployed only on a few smartphone applications. The use of mobile applications to diagnose plant diseases has received little attention in the literature. For instance, Petrellis [22] developed a windows smartphone application using image processing techniques for identifying vineyard diseases by extracting features from grape leaf images. It can be used as a standalone application or combined with a remote server, and it is also extensible to different plant diseases. Accuracy higher than 90% was achieved in recognizing the vineyard diseases. As an extension of their previous works [22][23][24], the authors in [25] proposed a low complexity image processing and a classification method implemented on smartphones for disease diagnosis in grape, peach, and citrus plants with an extensible set of diseases. Classification algorithms such as decision tree, random forest, Naïve Bayes, and neural networks were used in their experiments. The experimental results show that the developed mobile application achieved accuracy between 80 to 98% for disease recognition. Moreover, authors in [26] developed an image processing algorithm deployed in smartphones for detecting diseases in the wheat plant using a dataset of 3637 leaf images. The smartphone application was used to capture wheat leaf images that were then diagnosed online by comparison with stored previous algorithm results with accuracy higher than 80%.
Similarly, the authors in [27] developed a CNN model based on Single Shot Multibox (SSD) architecture then deployed it on a mobile application for real-time detection of diseases in cassava plants. The model performed better with F1-scores of 79% and 54% on test data and real-world images respectively. Authors in [28] proposed two deep CNNs (Resnet152 and Inceptionv3) using a dataset of 3000 leaf images to detect Fusarium wilt race 1 and black Sigatoka banana diseases. The Inceptionv3 model was deployed on an Android mobile phone and achieved an accuracy of 99% in detecting the two diseases in real environment. Authors in [29] proposed an automated irrigation and plant-leaf disease detection system using an Android mobile application. The mobile application was used to take photos of the suspected plant leaves, which were then sent to a cloud server, where the image was processed by matching it to the plant leaf images with diseases stored in the cloud database. In this work, the classification was based on Artificial Neural Networks (ANNs). In a recent study, researchers reported the performance of a deep learning object detection model for diagnosing plant diseases and pests [30]. The model was trained on 2756 images of cassava leaves exhibiting pest and disease symptoms. It was subsequently deployed as a mobile application in Android smartphones, which was then tested in the field and proved to be 74-88% accurate in diagnosing cassava disease and pest symptoms. This diagnosis accuracy was higher than that of agricultural extension officers (40-58%) and farmers (18-31%). Authors in [31] trained 5 CNN models using the PlantVillage dataset [32] with 18,160 tomato leaf images to classify 10 labels. The CNN model ResNet50, which had better prediction accuracy, was subsequently deployed in a mobile application to classify and identify tomato plant diseases successfully.
Early and real-time detection of Tuta Absoluta's damages on tomato plants can play a vital role in managing the pest and enhancing farmers' decisions. This study presents a mobile application called TutaSegmenter [33] deployed with a deep CNN model to detect and segment the effects of a tomato leaf miner (Tuta Absoluta) on tomato plants. This intervention approach for early pest detection and effective management at primary tomato growth stages will help farmers avoid massive economic losses.

II. METHODS
We used TensorFlow framework to deploy the CNN semantic segmentation model in a smartphone to detect Tuta Absoluta's damage on tomato plants. We employed transfer learning based on the U-Net architecture which has shown best performance on the International Symposium on Biomedical Imaging (ISBI) dataset [34] for semantic segmentation.
Authors in [35] introduced this U-shaped CNN architecture, which has performed exceedingly well in the biomedical image segmentation and later in many other fields, outperforming earlier segmentation methods even after being trained with only a few images [36]. We fine-tuned the model parameters to our dataset, which comprised of 1212 tomato leaf images damaged with Tuta Absoluta at early growth stages. The tomato leaf image dataset was built with images collected from experimental fields in Arusha and Morogoro regions in Tanzania. Examples of leaf images in this dataset are shown in Figure 1.  The experiments were conducted on a computer preinstalled with Windows 10 equipped with one Intel® Core™ i7-8550U 3.6GHz CPU, Intel® Iris® Plus Graphics, 512GB SSD storage, and 16GB memory. Google Collaboratory with Tesla P100-PCIE GPU and 27GB memory was utilized. We implemented our proposed network using Python v3.7.10 and the Keras [37]  the affected areas on tomato plants. Figure 2 shows the conceptual framework of this work.

A. Data Preprocessing
The dataset of healthy and Tuta Absoluta-infested tomato leaf images was collected from the fields in Tanzania using Canon EOS Kiss X7 and Samsung SM-G570F cameras. The complete details of this dataset have been reported in [39] and are freely available to the research community at the open access repository in [40]. For this study, 1212 images of infested tomato plants were extracted from the dataset to develop the semantic segmentation model. Each sample plant contained an average of 6 leaves which is equivalent to 7272 leaflets for the selected images in our dataset. LabelMe [41], an open-source graphical annotation tool (see Figure 3), was then used to produce ground truth masks by manually drawing and labeling irregular polygons following the shape of the infested area. Annotations were then saved in VOC [42] format with their corresponding images. Then, the dataset was split into training and test sets in 80:20 ratio as shown in Table I.

B. The CNN Model
The approach presented in this paper is an extension of the method described in [39] where a Tuta Absoluta semantic segmentation model based on U-Net architecture was proposed. In this work, a U-Net architecture was trained for 200 epochs using an annotated dataset of 1212 images resized to 512×512 pixels. The learning rate was set to 0.01 using the Adam [42] optimization function and the sigmoid activation function. The cross-entropy loss, Intersection over Union (IoU), and dice coefficient metrics were used to evaluate the model's performance. Then, an optimized Tuta Absoluta semantic segmentation model was deployed in an Android smartphone.

C. Model Deployment
This refers to integrating a machine learning model into an existing production environment to make practical decisions. We deployed the proposed model into a mobile phone so that farmers and extension officers can use their smartphones to automatically detect affected areas on tomato plants. To deliver reliable, user-friendly, and efficient software in a short time, agile software development methodology was used in developing the Tuta Absoluta segmentation mobile application based on users' feedback. We used the TensorFlow Lite (TFLite) to convert our suggested model into a lighter format because CNN models are complex and heavy, requiring a lot of memory and storage space to run. Converting the models into TFLite format reduces their file size, increases execution speed, and introduces optimizations that do not affect accuracy. This enables us to execute CNN models efficiently and run inference on mobile devices with limited computing and memory resources. The CNN model was converted to a mobile-compatible format using the python programming language. The application's interface and functionality were then defined using Extensible Markup Language (XML) and Kotlin programming language respectively. The mobile application software was implemented in Android Studio Integrated Development Environment (IDE) and tested in a Samsung SM-A025F Galaxy A02s smartphone. Smartphone applications are becoming increasingly popular and necessary [44]. We developed an Android application since Android holds the bigger market share in Africa of over 84% [45], where this research focuses.
Moreover, to achieve its purpose, the Tuta Absoluta segmentation mobile application was designed to capture and upload images, provide general information about tomatoes and the Tuta Absoluta pest, run inference using the CNN segmentation model, and display the segmentation results. Since the application operates completely offline, it only takes 5s to run the inference on a captured tomato leaf image and accurately segment the Tuta mines. The Tuta Absoluta segmentation mobile application is simple and easy to use, even without any guidance.

D. Software Design
In this paper, the design of the mobile application software is visualized using Unified Modelling Language (UML) diagrams. This includes the use case, activity and sequence diagrams. A use case model outlines how different categories of users interact with the system, their expectations, and the actions that the system must take to meet these objectives. Figure 4 depicts the use case diagram for the Tuta Absoluta segmentation mobile application. Each of the use cases shown is described in Table II. Moreover, the activity diagram describes the dynamic aspect of the application software. It graphically represents a series of actions or flow of control in a system with support for iteration and concurrency. The activity diagram for Tuta Absoluta segmentation mobile application is shown in Figure  5. Also, the sequence diagram depicts how objects interact with each other for a particular scenario. It details the way operations in the application software are carried out. Figure 6 shows the sequence diagram for the Tuta Absoluta segmentation mobile application.  The activity diagram for Tuta Absoluta segmentation application. The user(s) can access their mobile phone's camera to take a photo of a tomato plant. Then the system will automatically run inference on the photo using the CNN model in the background to segment Tuta mines.

Farmer or extension officer
Upload an image The user(s) can upload a tomato plant image from their mobile phone's gallery. Then the system will automatically run inference on the uploaded photo using the CNN model in the background to segment tuta mines.  III. RESULTS AND DISCUSSION The model's performance was evaluated using the crossentropy loss, IoU and dice coefficient evaluation metrics. The experimental results show that the U-Net segmentation model achieved 78.60% and 82.86% of IoU and dice coefficient, respectively. Figure 7 shows the U-Net training loss curve over 200 epochs. The losses dropped rapidly during early training iterations before stabilizing around 60 epochs, indicating that the model fits well on the characteristics of our dataset at both the early and late stages of the training process.  Figure 8 illustrates the confusion matrix for our segmenter. Since our task is binary segmentation, we map 0 for the background class and 1 for Tuta, the object of interest. As mentioned above, the U-Net model was converted to TFLite format and then embedded into a mobile application in Android Studio. Deploying on mobile devices would help democratize model access while also protecting user privacy that runs inference offline. Confusion matrix for the Tuta Absoluta segmentation model. We developed a simple and user-friendly mobile application named TutaSegmenter [33] to allow smooth interaction between the farmers/extension officers and the application. The application can be downloaded, free of charge, from the Google Playstore. The farmer/extension officer clicks on the application icon which welcomes them with a colorful splash screen that lasts only for 2s before landing on the scrollable home page. The home screen has two clickable cards with quick facts about the tomato plant and Tuta Absoluta, and a clickable floating button for Tuta segmentation, as illustrated in Figure 9. The user can read general information about tomatoes, such as their scientific name, production statistics, and cropping information such as the amount of water, soil, and fertilizer they require by clicking on the tomato plant card, as shown in Figure 10(a). By selecting the Tuta Absoluta card from the main page, the user can view general information about the pest, including common and scientific names, physiology, and lifecycle (see Figure 10(b)). This will assist the farmer in learning and comprehending the pest to adopt effective pest management and tomato plant care procedures. Additionally, on every page of the mobile application, we placed a clickable floating button for Tuta segmentation so that the farmer can quickly navigate to the disease diagnostics page to detect and segment Tuta mines in tomato leaf images. The disease diagnostics page allows the user to capture or upload images, then shows the results (predicted masks and masks overlay) displayed in a horizontally scrollable section along with the original image. The user can also view the input image size and detection details (labels found and the execution time) in the bottom sheet layout (see Figure 11(c)). If the image is devoid of Tuta mines, the application will display a text "No labels found" as shown in Figure 11 IV. CONCLUSION AND FUTURE WORK This paper presents a cost-effective mobile application for real-time detection and segmentation of Tuta Absoluta's damage on tomato plants at their early growth stages. We utilized a CNN model trained on a 1212 tomato leaf images dataset and optimized it for mobile deployment. We evaluated the performance of the deployed model on a mobile application that detects Tuta mines in tomato leaf images with minimum detection confidence of 70%. The application was able to precisely detect and segment the shapes of Tuta Absolutainfected areas on tomato leaves in just 5s. Since the deployed model operates offline, there are no running costs once installed. The application provides early and real-time diagnosis of the devastating pest, and effective management at early tomato growth stages to avoid huge economic losses. Providing these services in the absence of a mobile network, makes the application ideal for use in remote areas, which is beneficial to smallholder farmers especially in developing countries. This study proves that the use of computer vision techniques is vital in revolutionizing disease and pest management in agriculture.
In the future, we expect to continuously improve the diagnostic model in Tuta Absoluta segmentation application and add more knowledge as it becomes available to provide farmers with an ever-improving link to expert knowledge. Also, we aim to make the application capable of determining the extent of damage and suggest appropriate actions to be taken to control the pest based on the severity status. The application will be translated to Swahili, a widely spoken language in the Eastern, Central and Southern parts of Africa.