Rapid quantification of biofouling with an inexpensive, underwater camera and image analysis

To reduce the transport of potentially invasive species on ships’ submerged surfaces, rapid—and accurate—estimates of biofouling are needed so shipowners and regulators can effectively assess and manage biofouling. This pilot study developed a model approach for that task. First, photographic images were collected in situ with a submersible, inexpensive pocket camera. These images were used to develop image processing algorithms and train machine learning models to classify images containing natural assemblages of fouling organisms. All of the algorithms and models were implemented in a widely available software package (MATLAB © ). Initially, an unsupervised clustering model was used, and three types of fouling were delineated. Using a supervised classification approach, however, seven types of fouling could be identified. In this manner, fouling was successfully quantified over time on experimental panels immersed in seawater. This work provides a model for the easy, quick, and cost-effective classification of biofouling.


Introduction
In considering ships as vectors of aquatic nuisance species (ANS), the ballast water and sediments can be considered a sub-vector, while biofouling of the wetted surface areas is another sub-vector. By far, ballast water (and sediments) have historically captured most of the scientific and political attention. More than a decade was devoted to the development and application of the International Convention for the Control and Management of Ballast Water and Sediments (BWM Convention) by the International Maritime Organization (IMO). The BWM Convention was adopted in 2004 (IMO 2004). Given the complicated nature of the subject matter (which marries biology and technology), by the time the BWM Convention entered into force in 2017 (IMO 2018), it was accompanied by 14 sets of guidelines to provide direction on topics such as sampling organisms in ballast water (e.g., IMO 2008). Biofouling of ships' wetted surface areas, however, is increasingly viewed as a key factor, and it may represent a bigger driver of invasive species introductions and translocations than ballast water (Hewitt and Campbell 2010).
Measures to manage biofouling on submerged structures-specifically ship hulls and wetted surfaces-require regular cleaning, the application of fouling-control coating systems, or both (e.g., Visscher 1923;Hewitt et al. 2009;Callow and Callow 2011). Despite these efforts, biofouling persists. The consequences of such biofouling include not only the transport of potentially invasive species (e.g., Carlton 1987;Hay 1990;Godwin 2003;Coutts and Taylor 2004;Davidson et al. 2016), but also higher fuel consumption (e.g., Schultz 2007;Schultz et al. 2011) and the concomitant evolution of greenhouse gasses (Townsin 2003).
New guidelines and policies are in place or under development to reduce biofouling loads and the risk of transporting potentially invasive species. At an international level, for example, the 2011 Guidelines for the Control and Management of Ships' Biofouling to Minimize the Transfer of Invasive Aquatic Organisms (IMO Biofouling Guidelines, IMO 2011) have been promulgated, and they are in use in a growing number of jurisdictions, including Denmark (Bohn et al. 2016), New Zealand (Ministry of Primary Industries, MPI 2014), and California, USA (CCR 2017). Further, in 2020, the IMO began to review the global implementation of the Biofouling Guidelines (IMO 2018).
Although the IMO Biofouling Guidelines provide direction on choosing appropriate anti-fouling systems, and they stipulate the use of a Biofouling Management Plan and a Biofouling Record Book, the Guidelines do not define standard, maximum, acceptable levels of fouling. In contrast, the Craft Risk Management Standard (CRMS) for New Zealand (NZ), which became effective in May 2018, requires vessels to arrive in NZ with a "clean hull" (MPI 2014). This term is defined by thresholds limiting the percent cover of specific living organisms (e.g., ≤ 1% cover of either tubeworms, bryozoans, or barnacles) (MPI 2014). At present, the NZ requirements may be met by following the IMO Biofouling Guidelines or other best practices, so a ship's biofouling coverage does not need to be quantified, as long as the Guidelines are followed.
The growing regulatory interest and the increasingly stringent requirements for biofouling management are stimulating efforts to develop reliable methods to rapidly quantify biofouling on active ship hulls for optional use now, as well as to meet future regulatory mandates (e.g., First et al. 2014;Zabin et al. 2018;Scianni and Georgiades 2019). At present, quantifying biofouling is typically a painstaking, manual task (e.g., Butler et al. 2010). Further, the types of fouling are often coarsely categorized, with little taxonomic information (e.g., U.S. Navy 2006). Consequently, it is clear that an accurate, reliable, rapid method to quantify biofouling will be useful now and may be required in the future.
To that end, this pilot study developed an automated method for quickly quantifying and classifying biofouling accumulation on wetted surfaces using digital images collected with a submersible pocket camera. The development of new methods to rapidly and inexpensively assess biofouling loads and identify potentially undesirable species will become increasingly important. Such methods are needed not only for helping both industry and government to achieve future performance goals but also to meet regulatory objectives regarding biofouling.

Materials and methods
Photographic images collected in situ were used to develop and train algorithms to classify images containing natural assemblages of marine fouling organisms. To increase the accessibility of this approach, the analysis was performed using the commercial software package MATLAB© (Mathworks, Natick, MA). Notably, this work was informed by the great amount of research in the field of remote sensing, particularly as it is applied to measure aquatic biological parameters, including submerged benthic habitats (e.g., Louchard et al. 2003), seagrasses (e.g., Dierssen et al. 2009;Hill et al. 2014), and coral reefs (e.g., Hochberg et al. 2003;Mumby et al. 2004).

Photographic images of biofouling panels
Color, digital images of fouled experimental panels were collected at the U.S. Naval Research Laboratory Key West, FL, USA (NRLKW, 24.6°N; 81.8°W). All images were collected in situ using a waterproof, hand-held digital camera (Panasonic Lumix DMC-TS5; Panasonic North America, Newark, NJ), which captured three color channels (red, green, blue [RGB]) at a resolution of 4608 × 3456 pixels. The camera was mounted to a jig to standardize the size of the target panel within the camera's field of view and provide a consistent distance between the camera and the target panel ( Figure 1). A clean reference coupon was included with each image to correct for differences in exposure, contrast, and illumination across images. Illumination was provided by a diffuser placed over the camera's flash to reduce specular hotspots. In this manner, the lighting-as well as the shutter speed and exposure-were consistent as images were collected. Using this setup, the images were sufficiently uniform in terms of contrast and brightness such that no standardization or other image modification was necessary prior to the classification analyses.
The experimental panels (10.2 × 10.2 × 0.5 cm) were cut from cold-rolled steel. To protect surfaces from corrosion, panels were coated with a coating typically applied to ship hulls, Mare Island Epoxy (applied following military specification MIL-DTL-24441). By design, this coating had no fouling-release properties and contained no fouling-inhibiting chemicals.
A digital library required for training the algorithm was constructed from images of five, heavily fouled panels that had been submerged off the seawall or in a seawater trough with slowly flowing ambient seawater at NRLKW for approximately 25 months. Additionally, six newly immersed, initially clean panels were submerged in the seawater trough for four months (November 2016-March 2017) and photographed weekly. The flow-through seawater troughs were situated in a semi-enclosed room and exposed to indirect sunlight through open bay doors and skylights. The fouling on these initially clean (test) panels was assessed using the algorithm that was developed using images of the heavily fouled panels.

Image analysis
Two approaches were used to attempt to classify and quantify biofouling loads on the test panels. First, an unsupervised clustering approach was conducted on the heavily fouled panels to determine the number and taxonomic fidelity of the classes that could be identified without prior knowledge. Next, given the relatively coarse resolution of the unsupervised clustering analyses, a supervised classification was performed using a digital library of seven fouling organisms created manually from five of the heavily fouled panels. The library of identified organisms was used in a supervised classification to analyze the biofouling accumulation over time on the six newly immersed panels. All routines for image processing, classification, and analysis were performed using MATLAB.  (M05), which were used (along with other heavily fouled panels) to verify the seven classes using supervised classification. These classes accounted for the complete range of coloration across the panels. Note that five of the seven classes are found on these panels: 1) clean, unfouled surface, 2) turf algae, 3) encrusting tunicates, 4) Cnidarian polyps, and 5) other fouling (taxonomy unknown).

Unsupervised clustering
Photographic images from five of the heavily fouled panels were cropped to remove all non-panel pixels and reshaped into an (m × n × 3) matrix, where m, n represent the pixel dimensions and 3 denotes the three channels in the color image. The k-means clustering method (Bishop 2006), as implemented in MATLAB, was applied to all of the color points in the images, generating from 2 to 8 classes of optically distinct pixels in different iterations. The classes were then mapped onto each original pixel, generating a pseudo-color image defining the distribution of classes across the panels. Finally, the classified images were visually compared to the original RGB image and matched to the appropriate organism.
There are many clustering algorithms that can be used for this purpose, such as k-means clustering, Connectivity-based clustering, Expectation-Maximization (EM) clustering using Gaussian mixture models (GMM) (Bishop 2006), and more recent subspace clustering algorithms designed for high-dimensional data (Kriegel et al. 2012). Among these clustering algorithms, the k-means algorithm is the simplest, most efficient, yet effective approach. Thus, it was chosen for this study, with the goal to develop a fast algorithm for rapid biofouling type classification. Note that determining an optimal value for "k" in the k-means algorithm for a given data set is still an open problem (Bishop 2006). Our solution for this challenge was to run the algorithm with values from 2 to 8 for "k" then use visual inspection to explain the clustering results. Values larger than 8 produced meaningless results, so additional iterations were not needed.

Supervised classification
An image library of seven biofouling types was constructed from photographs of the five heavily fouled panels (see Figure 2 for an example). Individual organisms or fouling types (in the case of biofilms and filamentous Other fouling composed of unidentified or mixed taxa, including spirorbid polychaetes, solitary tunicates (Clavelina picata), and small bryozoans algae) were identified and located on the images visually. Representative patches of the identified organisms were selected to create a library defining the optical characteristics of each fouling type. A two-component classifier was trained using the library patches as ground truth to recognize each of the seven biofouling types (Table 1). A small window was used to scan the image, and each window was classified to one of the types by the trained classifier. The sparse coding classifier was trained using a supervised classification experiment. Sparse coding classification is a recently developed method that has been successfully applied to many applications, including detecting scars in seagrass beds (Oguslu et al. 2014(Oguslu et al. , 2018. It was chosen for biofouling classification because a sparse coding classifier is simple to implement and is effective for image classification. It first transformed image patches to new representations that were more robust in the classification because raw pixel values may change with object orientation and other parameters (e.g., Oguslu et al. 2014).
Sparse coding is a two-step classifier. In brief, sparse coding reconstructed an image patch using a set of basis functions (to allow interpolation), and the resulting reconstructed coefficients were used as a new representation for the image patch. Because only a few of the reconstructed coefficients are usually non-zero during the reconstruction optimization, the reconstructed representation is termed "sparse". The basis functions were either learned from image data through an optimization procedure or just directly were sampled from training image patches. Experimental results showed that randomly sampled basis functions could achieve the same level performance as learned ones, making the sparse coding step especially easy to be implemented (Oguslu et al. 2012(Oguslu et al. , 2014(Oguslu et al. , 2015. After the sparse coding step, each image patch was converted to a sparse vector (its reconstruction coefficients) that was classified by the supportvector machine (SVM) model (Fan et al. 2008). The SVM model was trained by providing it with a set of sparse vectors together with their known type labels. The learning process was guided by an optimization routine that minimized sums-of-square-errors under the constraints that the resulting model should perform well on new, unseen data. Afterwards, this robust classification model was used to assign the new representations to one of the seven biofouling types.
At each location during the scan, an image patch of the window size was extracted, the patch was then converted to a sparse vector, and the classifier was used to bin the patch into one of the seven biofouling types. The effect of patch size (window size) was investigated by trial-and-error, and patches of 9 × 9 pixels were determined in our investigation. Smaller patch sizes did not capture enough inter-taxon variation to reliably classify organisms, and larger sizes did not noticeably improve the classification. From each fouling type, 2000 randomly selected patches were used to train the classifier. The accuracy of the resulting classification maps was validated by visually comparing them to the original photographic images.

Determination of the accuracy of classification approaches
To assess the accuracies of the classification approaches, randomly chosen points on images were categorized by visual inspection, and the results were compared to the output of the classification schemes. For each image, 2 unique sets of points were randomly generated, and two analysts classified the fouling organism (or clean panel) on their unique sets of points. In this manner, a total of 200 randomly generated points was viewed for each image representative of the unsupervised or supervised classification approaches. For the unsupervised classification, 2 images were assessed, for a total of 400 data points. For the supervised classification, 1 image was assessed, for a total of 200 data points. Afterwards, the visual rankings were compared to the automated classification in an error matrix format. Accuracy was calculated as the relative agreement between the visual and automated classification.

Unsupervised clustering
Three classes of fouling were delineated according to the fouling burden: (1) "clean" pixels that represented unfouled surfaces, (2) "lightly fouled" pixels that contained a mixture of the clean pixels and some fouling, and (3) "heavily fouled" pixels in which no residual signature from the clean panel was evident (examples of this classification are shown in Figures 3  and 4). The optimum cluster size was determined to be 3; additional clusters (up to 8) did not produce consistent delineation of additional classes (data not shown). Once each pixel was classified, the percent of each panel covered by each of the classes was determined.
In all panels, the bulk of the surface was lightly fouled or heavily fouled. In Figure 3, using three clusters to analyze fouling, 26% of the panel was clean, 34% was lightly fouled, and 40% was heavily fouled. In Figure 4, 22%  of the panel was clean, 45% was lightly fouled, and 33% was heavily fouled. Thus, this procedure allowed the percentage of each panel covered by each of the 3 classes to be easily determined. It was unable, however, to distinguish among the various fouling taxa (e.g., barnacles vs. algae). For example, in Figure 4, the black tunicate on the right side of the photographic image was indistinguishable from the mass of bryozoans in the center lefthand side of the image: in Figure 4B (and in some areas of 4C), both are designated by green color. A more robust classification scheme was needed.

Supervised classification
Visual inspection of the heavily fouled panels generated seven classes ranging from a clean, unfouled surface to multicellular organisms (Table 1). A minimum of five distinct patches for each biofouling type were used to create each class in the library. The accuracy of the resulting classification maps was verified by visual comparison against the original photographic images ( Figure 2) and by using randomly chosen points as above (Figures 3-5). It was unnecessary to standardize the images against a clean reference panel prior to classification (data not shown). Eliminating this unnecessary step will further streamline and simplify the approach developed here.
The biofouling communities on the newly immersed panels were analyzed to create a time-series of biofouling, and, in general, the six panels followed similar trajectories with respect to biofouling accumulation during the incubation period (see examples in Figures 6 and 7, which were typical of the newly immersed panels). Due to the relatively short immersion time, the season (winter, typically a period of low biofouling accumulation [NRLKW, unpublished data]), and the oligotrophic location (NRLKW, unpublished data), the fouling load was much lower than that in the panels used in the unsupervised clustering (Figures 4 and 5). Throughout the four-month incubation, most of the panels' surfaces were free of visible fouling or were covered with microalgal biofilm (Table 2). In this case, the fouling accumulation on the time series panels represented only a subset of the classes in the library. The most abundant fouling taxon throughout this examination consisted of microalgal biofilms that covered > 90% of the panel surfaces in November and December 2016.
When data from all six panels were analyzed, initially, the biofilm consisted almost exclusively of microalgal biofilm that disappeared around 30 December 2016 ( Figure 8A, B). The panels remained nearly free of fouling through January 2017, and then they began to accumulate new biofilm layers and macrofouling organisms, primarily solitary tunicates, spirorbid polychaetes, and filamentous macroalgae ( Figure 8C). A few solitary tunicates, spirorbid polychaetes, and nascent macroalgal filaments-all grouped into the "other fouling" class-became more common in February 2017. Biofilm cover dropped to about 10% in January and February 2017 and increased again in March 2017 ( Figure 8B).

Accuracy of classification approaches
Accuracy was judged by comparing visual classifications of random points to the automated identification of these same points (Figures 3-5; the results are summarized in the confusion matrices shown in Tables 3-5). Unsupervised classification with two classes had the highest accuracy: 89% agreement between the analysts and the algorithm rankings (n = 397 points for 2 images, each with 2 sets of points; note that 3 points fell upon the panel   Table 1; the most abundant features in the supervised classification (bottom panels) are: microalgal biofilm (cyan areas), turf algae (green areas) and clean, unfouled surface (white areas).  Table 1; the most abundant features in the supervised classification (bottom panels) are: clean, unfouled surface (white areas), microalgal biofilm (cyan areas), and other fouling composed of unidentified or mixed taxa (yellow areas). fasteners and were excluded) (Table 3). However, the class sizes were uneven (n = 86 and 311 for Classes 1 and 2, respectively). Because of this disparity, a high rates of false negatives (47%) for Class 1 did not drive down the overall accuracy of the classification. For unsupervised classification with three classes, accuracy was lower (68%) ( Table 4). This was mostly due to disagreements between moderate and heavy fouling; both methods were generally in agreement regarding unfouled locations (94%; n = 31 of 397 points). For supervised classification, overall agreement was 58% (Table 5). However, amid this lower overall agreement, certain categories showed high agreement. For example, both visual and automated were in agreement (75%) on classifying tunicates ( n = 49 of 65 points). The effort needed to classify the images differed between the manual and automated applications. The automated classifications-either using the unsupervised or supervised approach-occurred instantaneously once the algorithms had been trained. Scoring the organisms (or clean portions of the panels) in the photographs manually took approximately 30 minutes per 100 points. Further, classifying organisms by eye required a notable understanding of biology and robust knowledge of the local, benthic community.

Discussion
This work demonstrated that images of biofouling accumulation on submersed panels collected with a simple pocket camera can be used to (1) identify and quantify several classes of marine biofouling types and (2) assess surfaces ranging from clean to heavily fouled. The method was also sensitive enough to detect temporal changes on individual panels deployed over time. As this was a proof-of-concept study, the conditions were ideal, i.e., images were collected in a carefully controlled environment with consistent  Table 1 were combined as "all other organisms". The legend for the panels is shown in panel C. Note the differences in scale among the y-axes. Table 3. Confusion matrix demonstrating the accuracy of the actual visual (Actual) determinations vs. predictions from unsupervised classification (Predicted) with 2 classes. Classes 1 (clean) and 2 (fouled) are shown as column and row headers in yellow-shaded cells. Category values are shaded from white (no data) to blue (most data). Orange-shaded cells are the column and row sums (here, n = 397, shown in bold font). For emphasis, instances of agreement are shown in bold font and along the diagonal. The sensitivity and specificity of each class is also shown. lighting, and the surfaces were flat. A next step is to determine the suitability of this approach in water with a high concentration of suspended solids or organisms and with irregular surfaces, e.g., in tubes mimicking bow thrusters. Images used here were collected under ideal conditions, including standard camera position and angle. In-water imaging will require approaches to determine the scale and perspective of the images (Gormley et al. 2018) Although it required the construction of a reference library, the supervised classification procedure represented a demonstrable improvement over the unsupervised clustering approach with respect to the types of organisms delineated. Seven classes were identified and assigned to taxonomic groups in the supervised classification approach, as opposed to the three levels of fouling accumulation (with no taxonomic information) achieved with the unsupervised approach. Advances in the supervised approach, for example, using a wider variety of fouling organisms, will likely allow more detailed clustering (identification) of fouling organisms. We note that the set of fouling organisms identified in these panels is not comprehensive, and it may not represent typical marine fouling communities, and some ubiquitous fouling organism (e.g., barnacles) were not present on these panels.
Our goal was to perform a pilot study using digital images from an inexpensive, underwater camera and machine learning algorithms for rapid, automatic biofouling type identification. Based on our previous research experience and results from the literature, we chose the k-means algorithm for unsupervised clustering and the sparse coding method for supervised classification. We did not attempt to customize our algorithms for biofouling identification. Indeed, in doing so, there might be a superior unsupervised clustering method or a supervised classification algorithm that can provide better biofouling identification, following on the "No Free Lunch Theorem" (Wolpert and Macready 1997). Indeed, as the field of image analysis changes rapidly, especially with advances in artificial intelligence, other object-recognition approaches can and should be applied (He et al. 2016). Regardless, our experimental results showed that automatic biofouling type identification is feasible, and in future work, we plan to increase the algorithms' performances using customized methods.
In some cases, the unsupervised clustering method may be sufficient despite its inability to identify types of organisms. For example, measuring the composition and prevalence of fouling organisms to determine when a ship's hull will need to be cleaned may not require the community to be delineated more finely than determining if the biofouling is, for example, lightly vs. heavily fouled. If the goal is to track or identify potentially invasive species, then the unsupervised classification can potentially be sufficient by yielding a bulk measurement of the biomass of organisms transported to new locations, whereas supervised classification methods are likely to provide more fruitful results if organisms of concern must be monitored. Our analysis comparing visual to automated classification found high agreement in several cases, especially when identifying unfouled regions. With additional classes added, agreement between analysis and the automated classification declined. Nevertheless, some organisms (e.g., tunicates) were identified accurately by the automated approach. Thus, specific organism types with well-defined characteristics may be identified with high accuracy and distinguished among other fouling groups. Combining the digital, white-light imaging with imaging fluorometry to map distributions of chlorophyll and other photopigments (e.g., Eggert et al. 2006) may improve classification accuracy.
Importantly, this work was done using an inexpensive, waterproof camera and standard classification routines available in many software packages. Thus, the technology required to implement this approach is highly developed. Combined, these elements represent a noteworthy step forward in efforts to quickly and easily classify biofouling without the intervention of human specialists. For example, the required images could be captured by a diver or a remotely operated vehicle (ROV). Both are routinely used for hull surveys, and using the automated software routine, the fouling on a ship's hull could be quickly assessed. Finally, the supervised approach also permits the incorporation of new classes into the analysis scheme as they become identified, potentially increasing the number of classes, and, therefore, the taxonomic specificity of the analysis.
A hand-held camera could also be used to collect data from hard-toreach "niche" areas, such as propellers, sea chests, and lateral thruster tunnels. These areas represent only a small portion of the total wetted surface of ships, on the order of 10% (Moser et al. 2016(Moser et al. , 2017. Despite their disproportionally small area, given hydrodynamic conditions and operational profiles, niche areas can harbor elevated densities of organisms relative to more exposed regions along the hull (e.g., Coutts and Dodgshun 2007;Davidson et al. 2009;Sylvester and MacIsaac 2010). Thus, niche areas can have outsized influence on the transfer of ANS. Therefore, it is important to develop methods that both reliably and accurately assess fouling in these areas (as well as the hull).
As the IMO BWM Convention (IMO 2004) has entered into force in 2017, the scientific and policy discussions are turning from ballast water to biofouling as a vector of ANS. Among all nations, however, only NZ has issued a biofouling standard, and it may be met by using best practices. Regardless, the IMO guidelines for biofouling on ships (IMO 2011) and recreational craft (IMO 2012) are in effect. More importantly, the efficacy of the 2011 guidelines is being reviewed (IMO 2018). This decision by the IMO elevates the issue of biofouling. Further, across administrations and regions, similar gaps and challenges exist -there is no standardized procedure for testing the efficacy of cleaning technologies, nor does a procedure exist for assessing ships' compliance with any potential biofouling standards . Thus, there is a pressing need for a means to rapidly, reliably assess biofouling.
Although commercial shipping represents a global and highly visible economic sector, it is typically a low-margin industry that experiences frequent boom-and-bust cycles, limiting the resources that can be applied to the problem of biofouling detection. Likewise, many regulators operate with limited resources. Thus, an economic means to quantify fouling would be useful. This work demonstrates the potential to do so using an inexpensive underwater camera and widely available software.