Unmanned aerial vehicles and deep learning for assessment of anthropogenic marine debris on beaches on an island in a semi-enclosed sea in Japan

The increasing prevalence of marine debris is a global problem, and urgent action for amelioration is needed. Identifying hotspots where marine debris accumulates will enable effective control; however, knowledge on the location of accumulation hotspots remains incomplete. In particular, marine debris accumulation on beaches is a concern. Surveys of beaches require intensive human effort, and survey methods are not standardized. If marine debris monitoring is conducted using a standardized method, data from different regions can be compared. With an unmanned aerial vehicle (UAV) and deep learning computational methods, monitoring a wide area at a low cost in a standardized way may be possible. In this study, we aimed to identify marine debris on beaches through deep learning using high-resolution UAV images by conducting a survey on Narugashima Island in the Seto Inland Sea of Japan. The flight altitude relative to the ground was set to 5 m, and images of a 0.81-ha area were obtained. Flight was conducted twice: before and after the beach cleaning. The combination of UAVs equipped with a zoom lens and operation at a low altitude allows for the acquisition of high resolution images of 1.1 mm/pixel. The training dataset (2970 images) was annotated by using VoTT, categorizing them into two classes: ‘anthropogenic marine debris’ and ‘natural objects.’ Using RetinaNet, marine debris was identified with an average sensitivity of 51% and a precision of 76%. In addition, the abundance and area of marine debris coverage were estimated. In this study, it was revealed that the combination of UAVs and deep learning enables the effective identification of marine debris. The effects of cleanup activities by citizens were able to be quantified. This method can widely be used to evaluate the effectiveness of citizen efforts toward beach cleaning and low-cost long-term monitoring.


Introduction
The rapid increase in anthropogenic marine debris has been recognized as a global environmental problem (Ostle et al 2019). The majority of marine debris consists of plastic (Barnes et al 2009), and it is estimated that 4.4-12.7 million metric tons of plastic are discharged into the ocean annually (Jambeck et al 2015). Previous studies have indicated that in the western Pacific Ocean, particularly in the Kuroshio Current area, more than 3.5 million pieces of debris per square kilometer at the ocean surface may be present (Yamashita and Tanimura 2007). The problem with marine debris is not only the amount of inflow but also its persistence. Plastics are chemically resistant to degradation and remain in the marine environment for hundreds to Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI. thousands of years (Derraik 2002). Therefore, they continue to accumulate over a long time and have a variety of effects, such as an impact on tourism due to aesthetic degradation and ingestion by marine organisms. In addition, plastic fragmentation and degradation driven by environmental factors such as ultraviolet (UV) light result in small particles that are known as microplastics (Andrady 2011, Cole et al 2011. Marine debris affects the human economy, particularly, shipping, fishing, and tourism (McIlgorm et al 2011). For example, in tourist destinations, aesthetic degradation by marine litter decreases revenue from tourism, and local authorities need to bear the cost of cleanup (Iñiguez et al 2016). The dynamics of marine debris widely vary, and heavier items may accumulate on the seafloor. There have been attempts to collect marine debris, but these are costly and sometimes dangerous. For instance, in a project to remove debris from the seabed in South Korea, about 460 tons of trash were recovered at a cost of $3.8 million, and fatal incidents were involved (Cho 2011).
Marine ecosystems are also affected by marine debris. Ingestion and entanglement are two impacts of marine debris on marine organisms (Derraik 2002). Marine organisms from a wide range of taxonomic groups ingest marine debris, sometimes leading to death (Gregory 2009). Entanglement in discarded fishing gear-related debris is known as 'ghost fishing' and is recognized as a serious threat. At least 690 species are threatened by marine debris, of which at least 17% are classified as near-threatened or higher on the IUCN Red List (Gall and Thompson 2015). To mitigate the impact of marine debris, a quantitative assessment of the distribution of marine debris is the first step. Once the location and amount of marine debris present are understood, the removal of marine debris can be implemented.
The accumulation of marine debris has been confirmed over a wide area from the Arctic to the Antarctic and in the deep sea (Woodall et al 2014). However, the amount of plastic debris observed in the ocean is much lower than that previously expected (Cózar et al 2014). Various studies have been conducted to determine where the 'missing plastic' exists (Law and Thompson 2014). This 'missing plastic' may be drifting in currents (Eriksen et al 2014) or water columns (Dai et al 2018) or is accumulating on the seafloor (Chiba et al 2018). However, the importance of focusing on beaches has been highlighted (Brennan et al 2018, Olivelli et al 2020. About half of the world's population lives near the coast, making coastal areas hotspots for microplastic pollution (Cole et al 2011). Shorelines are also strongly influenced by terrestrial factors such as stormwater and road distribution (Willis et al 2017), and backshore areas with vegetation can be a sink for large marine debris (Olivelli et al 2020). In addition, plastic products on beaches are exposed to UV light that causes photo-oxidative degradation (Singh andSharma 2008 Sathish et al 2019). On beaches in the daylight, the thermal degradation of plastic progress faster than in seawater (Andrady 2011).
In recent years, unmanned aerial vehicles (UAVs) have been used in a wide range of research fields (Jiménez López and Mulero-Pázmány 2019). They are also used to survey marine debris on beaches (Kako et al 2012, Bao et al 2018, Deidun et al 2018, Lo et al 2020, Merlino et al 2020. UAVs have the advantage of cost effectiveness and the ability to obtain high-resolution images. Studies have conducted marine debris surveys using satellite images, but the high cost and low resolution of the data have been issues. For example, WorldView-4 satellite imagery, with a resolution of about 0.3 m, cannot detect the majority of marine debris on beaches (Topouzelis et al 2019). UAVs also facilitate surveys in difficult to access areas such as islands (Lavers and Bond 2017). In addition, UAVs can be used to collect substantial data using a standardized method and compare the distribution of marine debris from different locations. However, the analysis of image data taken by UAVs remains labor-intensive (Jiménez López and Mulero-Pázmány 2019). Deep learning can improve the efficiency of data processing and analysis. However, research on the automatic identification of marine debris from UAV images is a new field, and there are few previous studies on this topic In this study, we aimed to identify anthropogenic marine debris on a beach via high-resolution UAV images using deep learning. The challenges we hope to address in this study are (1) whether UAV images are suitable for automatic identification by deep learning and (2) whether identifying anthropogenic marine debris from the variety of objects present on the beach is possible. If these can be accomplished, it will be possible to survey a wide area with a standardized protocol, which could significantly improve the world's future marine debris management.

Study area
The Seto Inland Sea is the largest semi-enclosed sea in Japan, with about 31 million residents, more than 700 islands, and a total coastline of about 7,230 km. The climate is mild throughout the year, and the landscape attracts many tourists. The estimated total annual inflow of marine debris into the Seto Inland Sea is 4,500 tons, of which about 70% comes from land (Fujieda et al 2010). Training and test images were taken at a site located on Narugashima Island (34.288351N, 134.952303E) in Hyogo Prefecture (figure 1). Narugashima Island is located in the southeastern part of Awajishima Island. The island has an area of about 25-ha and a total length of about 3 km. There are beaches on the east and west sides, facing Osaka Bay and Yura Bay, respectively. The shorelines of this island are strongly affected by tides, waves, and winds, and a large quantity of marine debris is therefore accumulated. This area belongs to Setonaikai National Park, so local volunteers engage in cleanup activities once a year. Preliminary investigations showed many plastic products drifting on the west rather than the east side of the island (figure 2). Both beaches were set as UAV photography areas to collect the training images.

UAV survey protocols
The DJI Mavic 2 zoom was used in this study. This UAV is equipped with a 24-48 mm optical zoom lens and a camera with a 1/2.3 CMOS and a 12.4 M pixel sensor. The field of view (FOV) is 83°for the 24-mm zoom and 48°for the 48-mm zoom. It is possible to obtain higher-resolution images with the zoom lens. The flight route was mapped using the DJI GS Pro application, which allows the user to set the flight area and speed in advance and check the shooting status in real time. A 0.81-ha flight plan was created in advance, and shooting was conducted on November 27 and December 4, 2020, and January 25, 2021. Since the flight time of one battery is only 26 min, the battery was continuously replaced. The operators walked on the beach alongside the UAV and used a Jackery portable battery 700 to recharge the depleted batteries for reuse. The flight altitude was set to 5 m, calculated at the point where the UAV took off; therefore, the exact altitude may slightly vary for each image. The shooting interval was set for 50% horizontal and 50% vertical image overlap. The camera's ISO, aperture, and shutter speed were set to AUTO mode, and images with a resolution of 4000×3000 pixels were recorded. In this study, the images obtained from the survey of Narugashima Island on November 27, 2020, were used as the training images (table 1). In addition, to improve the model, training images that contained only a PET bottle were obtained from the Kamigamo test site of the Kyoto University Field Science Center on December 16, 2020. Since the cleanup activity was performed on this island on December 5, test images were acquired before and after the day to evaluate the effectiveness of the cleanup activity. Finally, 4,421 test images from the west beach were obtained from Narugashima Island's survey on December 4, 2020, and January 25, 2021.

The deep learning algorithm Annotations
We annotated images using VoTT, an open-source labeling software developed by Microsoft. We annotated a total of 2,970 images by classifying them into either 'anthropogenic marine debris' or 'natural objects' that mainly consisted of large driftwood. Marine debris larger than 100 pixels was annotated. The classified images were exported in Pascal VOC format and used for training. In this study, 80% of the images were set as training data and 20% were used as validation data.

Data augmentation
Deep learning requires a large amount of training data to avoid overfitting. Therefore, we augmented the training datasets using the 'imgaug' python library. For each image, image scaling, random erasing, and contrast changes were applied with a probability of 50%. For random erasing, a rectangular region of 10% of the image size was created and filled with random RGB values in order to detect partially hidden objects during identification. To mask the part of the image, 5-10 of these rectangular areas were randomly generated. Finally, contrast changes were randomly varied between 50% and 200% to correspond to sunlight exposure.

RetinaNet
In this study, we trained the model using RetinaNet (Lin et al 2017), an object detection model derived from region-based convolutional neural networks (Girshick et al 2016). RetinaNet consists of a backbone network and two subnetworks. The depth was set to 50 because the F-score is the highest in parameter tuning (supplementary information 1 (available online at stacks.iop.org/ERC/4/015003/mmedia)). We set the depth to 50, the epochs to 60, and the batch size to 6 for training. A threshold of more than 0.5 for the score was used for our model to detect marine debris. In addition, non-maximum suppression (NMS) was applied after object detection (Girshick et al 2014). NMS is a method that integrates multiple detections of the same object into a single result. In this experiment, the object with the highest probability was used as the standard, and other objects with more than 50% overlap with the standard were removed.

Orthomosaic images
The same marine debris may have been identified multiple times because overlapping images were taken. Therefore, orthorectified images were used for identification using the Agisoft PhotoScan software. This software uses the Structure from Motion algorithm to create orthomosaic images from overlapping images. In this study, we used 200 images to create each orthomosaic image from a total of 4,421 images. Some areas were not orthorectified and were displayed in white due to low image overlap. Therefore, we processed the data to avoid detection when areas had more than 80% white pixels.

Accuracy assessment
We segmented the orthorectified images created for the test set with the same size as training images. After identification, an observer randomly selected 100 images to calculate the true positive (TP), false negative (FN), and false positive (FP) rates. We defined FNs when marine debris larger than 100 pixels was not identified. The model was evaluated using sensitivity, precision, and the F-score. For these indices, the following equations were used: sensitivity=TP/(TP+FN); precision=TP/(TP+FP); and F-score=2TP / (2TP+FP+FN). Changes in the marine debris were quantified by comparing the number of pieces and the area. The comparison sites were selected from the north, central, and south parts of each beach. Our model marks the detected marine debris with a blue frame. The number of blue frames surrounding the marine debris was defined as counts of marine debris, and we compared the number of marine debris per orthorectified image for each comparison site. The area of the marine debris was calculated using the following method: First, the area containing the blue frame was calculated. Next, the area of the marine debris within the blue frame was calculated for 100 randomly selected samples, which was used as the average marine debris percentage. Finally, the total area of the blueframed areas in the comparison site was calculated and multiplied by the average marine debris percentage to obtain the estimated area of marine debris. The actual length per pixel in the images was calculated by measuring the size of the PET bottle. ImageJ was used to calculate the area of the blue boxes and marine debris.

Results
The deep learning model successfully identified marine debris in the high-resolution images (figure 3). For data from Narugashima Island in December 4, the sensitivity was 59% and the precision was 90% (table 2). On the other hand, for the data taken in January 25, the sensitivity was 43% and the precision was 62%. The average of both results was 51% for sensitivity and 76% for precision. The main FPs were waves and rocks (figure 4), and when the marine debris contacted driftwood, it was not detected (figure 5). As a result of the identification by our deep learning model, the numbers of marine debris at the site on December 4 were 53, 297, and 118 pieces in the north, central, and south parts of the beach, respectively (table 3). The mean number of marine debris (±standard error (SE)) was 156 pieces (±72.95 pieces). On January 25, the numbers of marine debris at the comparison site were 80, 30, and 93 pieces in the north, central, and south parts of the beach, respectively. The mean number of marine debris (±SE) was 68 pieces (±19.20 pieces). On December 4, the marine debris density was 0.12, 0.66, and 0.26 pieces/m 2 in the north, center, and south, respectively. The average marine debris density (± SE) was 0.35 pieces/m 2 (±0.16 pieces/m 2 ), and the marine debris density on January 25 was 0.18, 0.06, and 0.21 pieces/m 2 in the north, center, and south, respectively. On this observation date, the average marine debris density on this observation date (±SE) was 0.15 pieces/m 2 (±0.04 pieces/m 2 ).
The areas of marine debris decreased more on January 25 than on December 4. By calculating the actual length of the pixels, the area of the debris could be calculated from the images. The actual length of one pixel (±SE) was 1.06 mm (±0.04 mm). Based on this length, the areas of the blue frames on December 4 were 2.77, 12.81, and 3.94 m 2 in the north, central, and south part of the beach, respectively. The average blue-framed area (± SE) was 6.51 m 2 (±3.17 m 2 ). On January 25, the areas of the blue frames were 3.95, 4.26, and 6.12 m 2 in the north, central, and south beach, respectively. The average blue-framed area (± SE) was 4.78 m 2 (±0.68 m 2 ). The mean marine debris percentage in the blue-framed area (± SE) was 62.6% (±1.77%). Based on this percentage, the areas of marine debris on December 4 were 1.74, 8.02, and 2.47 m 2 in the north, center, and south, respectively. The average area (±SE) was 4.08 m 2 (±1.99 m 2 ). The areas of marine debris on January 25 were 2.48, 2.67, and 3.83 m 2 in the north, center, and south, respectively. The average area (± SE) was 2.99 m 2 (±0.42 m 2 ).
The area coverages could also be estimated from the images. The area coverages of marine debris were 0.41%, 1.79%, and 0.54% in the north, central, and south parts of the beach in December 4. The average area (± SE) was 0.92% (±21.16%). Similarly, the area of marine debris was 0.57%, 0.56%, and 0.86% at the three parts of the beach, respectively, in January 25. The average area (± SE) was 0.66% (±3.88%). The number and area coverages of marine debris were lower in January compared to December. In addition, it was demonstrated that deep learning could identify marine debris from UAV images and that changes in the debris could be quantified.

Discussion
In this study, high-resolution images of marine debris were acquired by combining a UAV equipped with a zoom lens and low altitude flight operation. In addition, the study showed that UAV images can be used for the object detection of marine debris. Although UAVs have also been used in marine debris surveys, high-resolution image acquisition was a challenge. For example, the resolution in the previous study was 4.4 mm/pixel when the DJI Phantom 4 was used at 10 m and 5.0 mm/pixel when it was used at 17 m (Fallati et al 2019; Kako et al 2020). In this study, a resolution of 1.1 mm/pixel was obtained using the DJI Mavic 2 zoom from an altitude of 5 m. The resolution of the present study was high compared to previous studies, and it was achieved using the combined approach of a UAV that was equipped with the zoom lens and operated from a low altitude.
The advantage of UAVs is that the vehicles can perform survey at a low cost, so periodic monitoring can reveal the spatio-temporal distribution of marine debris. In this study, we acquired 4,421 images and performed an intensive survey at a low altitude to obtain high-resolution images from the target area. Our survey over an area of 0.81-ha was implemented by replacing the battery 12 times. To the best of our knowledge, this is the first study in which a UAV equipped with a zoom camera was operated at a 5 m altitude to identify marine debris.
Our model achieves a sensitivity of 0.59 because of the high-resolution images from the UAV. Moreover, our model could identify anthropogenic marine debris from the wide variety of objects on a beach. Training data containing multiple types of debris enabled the high performance of our model. In addition, we annotated two classes, anthropogenic marine debris and natural objects, allowing the model to distinguish between human and natural-origin debris. The types and composition of these categories differ, and annotating typical natural objects improved the performance of the model.  Our technique evinced a decrease in marine debris through cleanup activities. These activities have been performed worldwide on beaches. However, continuously monitoring a large area after cleanup activities is labor-intensive. In our study, we found that combining UAVs and deep learning makes it possible to compare  the amount of debris before and after a cleanup activity and quantify the effectiveness of the cleanup effort. Moreover, we clarified that the distribution of marine debris varied depending on the site. Ocean currents and wind direction may affect marine debris distribution. In the future, continuous surveys using this technique might reveal the factors behind marine debris accumulation.
Despite using the same model, the identification accuracy differed, as sensitivity decreased when the test images were taken on January 25. The reduction in the amount of marine debris on the beach may affect the model's sensitivity, and more marine debris could be detected by changing the threshold of our model. In addition, detecting entangled debris in vegetation was challenging. Furthermore, the orientation of the marine debris may affect the results. Debris entangled in vegetation can be scattered at various angles, and the training images did not contain these varied angles. Therefore, improving the training images will improve the model performance.
Our model underestimated the amount of marine debris owing to a high false negatives. Previous studies have shown that identifying marine debris using UAVs also underestimates the amount of debris compared to a human ground survey (Martin et al 2018). The shadows of debris can also affect identification, and it is recommended that UAVs be operated around noon to reduce shadows (Martin et al 2018;Lo et al 2020). In our study, the images were consecutively taken between 10:00 and 15:00. Therefore, the high false negatives may be attributed to the Sunlight conditions in each image.
Some limitations still exist in this study. First, detecting small anthropogenic marine debris such as fragments of plastic products is difficult. The criteria for the annotation also affect the low detection of small debris. Marine debris was annotated if the size was larger than 100 pixels because it was difficult to annotate all of the debris. Further consideration of criteria for the annotation will be needed to detect the smaller size of debris. Second, the results of our study show lower sensitivity, indicating that the model cannot detect all marine debris because of the excesses of false negatives. A variety of anthropogenic marine debris was annotated as one class, such as PET bottles, plastic bags, and cans. Therefore, separating the class into more subdivided categories may improve the sensitivity. Although high-resolution images acquired operated UAV at a low altitude, this operation required multiple batteries; for example, batteries were replaced 12 times in this study. In addition, operations over several hours also affect the light conditions of the image; thus, this difference may have affected the performance. Additionally, object detection was used in this study, but it may be necessary to consider other approaches, such as semantic segmentation for more accurate estimation. Finally, our results show microplastic detection is quite challenging; however, monitoring these distributions is essential to understanding plastic pollution in the marine environment. Citizen science can also be beneficial to survey microplastic pollution with other protocols.
Although beaches attract attention as places that often accumulate marine debris, few studies have identified marine debris on beaches using UAVs combined with deep learning (Fallati et al 2019). Some studies used aircraft to survey the distribution of marine debris (Moy et al 2018); however, the cost is prohibitive. The advantage of UAVs is that they can acquire higher-resolution images at a lower cost than aircraft. In addition, UAVs enable safe surveys in dangerous areas where marine debris accumulates, such as areas containing medical waste. In these areas, remote sensing surveys of beaches using UAVs can be conducted. Using UAVs is also beneficial from an ecological perspective (Harris et al 2015). Beaches are essential for the breeding of some species of marine wildlife. For example, the human use of beaches affects areas where sea turtles lay their eggs (Antworth et al 2006) and the behavior of seabirds (Burger and Niles, 2013). Thus, minimizing disturbance to marine wildlife by conducting surveys without directly entering the beach is possible.
The present study, showed that deep learning could identify marine debris obtained from high-resolution UAVs images. Marine debris is expected to increase in the future, and the monitoring is an urgent global issue. Using UAVs and deep learning will allow for uniform surveys to be conducted worldwide at a low cost. To help solve the marine debris problem, it is necessary to clarify the current situation at a global scale. Citizen science in obtaining marine debris has continued to grow (Sandahl and Tøttrup 2020), and our model can be used to evaluate cleanup activities. The worldwide use of methods such as those described in this study, which combines field surveys and information science technology, will help enable the efficient management of marine debris.