SealNet: A fully-automated pack-ice seal detection pipeline for sub-meter satellite imagery
Introduction
The Southern Ocean (SO) harbors major seasonal hotspots for primary productivity (Arrigo and Dijken, 2003). The cold, nutrient rich waters of the SO play a fundamental role regulating global climate, both by absorbing large amounts of heat and sinking fixed carbon (Frölicher et al., 2015; Morrison et al., 2016). Food webs in the SO are trophically shallow (Clarke, 1985), but they more than compensate in terms of biomass, sustaining massive concentrations of phytoplankton consumers (Bester et al., 2002a; Nowacek et al., 2011). Among these consumers, a small crustacean, Antarctic krill (Euphasia superba), is especially important; krill is the main food item for a wide range of upper tier consumers, from fish and penguins to seals and whales, and serves as a fundamental link between predators and primary producers. Due to krill's role in the SO food web, assessing and tracking Antarctic krill stocks is central to Antarctic ecology. This is especially true now that climate change (Flores et al., 2012; Klein et al., 2018), ocean acidification (Kawaguchi et al., 2013) and krill fisheries (Forcada et al., 2012) threaten to shift the abundance and distribution of this key Antarctic species. Challenging our efforts to track Antarctic krill, however, is its small size and patchy distribution (Voronina, 1998). One way to circumvent those difficulties is to use krill predator abundances as a proxy for krill distribution (Huang et al., 2011). Antarctic pack-ice seals (crabeater seals [Lobodon carcinophaga], Weddell seals [Leptonychotes weddelli], leopard seals [Hydrurga leptonyx] and Ross seals [Omnatophoca rossii], within the Phocidae family), as a group, represent a promising vehicle to gauge krill stocks for they are not only key krill consumers (Botta et al., 2018; Forcada et al., 2012; Hückstädt et al., 2012; Siniff and Stone, 1985) but they are also large enough to be individually spotted with high spatial resolution satellite imagery.
The potential of pack-ice seals as indicators of environmental health in the SO has not gone unnoticed; polar ecologists have channeled sizeable efforts into estimating pack-ice seal population sizes, the most notable of these attempts being the Antarctic pack-ice seal (APIS) project (Anonymous, 1997), a joint effort of six countries to estimate Antarctic seal populations using aerial surveys (Ackley et al., 2006). Conducting such large-scale aerial survey programs in Antarctica is extremely expensive, necessarily requiring extensive collaboration among Antarctic national programs. Fortunately, very high spatial resolution (VHR) satellite imagery may soon be a viable alternative for aerial surveys, providing greater spatial coverage and, due to its dramatically lower cost, increased repeatability. The use of VHR satellite imagery for wildlife survey has exploded in recent years, and includes demonstration projects for southern elephant seals (McMahon et al., 2014), polar bears (Stapleton et al., 2014) and African ungulates (Xue et al., 2017; Yang et al., 2014), as well as seabird species whose presence and abundance can be estimated indirectly using the guano stain at the colony (LaRue et al., 2014; Lynch et al., 2012). Pack-ice seals, while large enough to be seen in VHR imagery, are particularly hard to detect since their preferred haul out environment (pack ice; Bengtson and Stewart, 1992; Lake et al., 1997) changes on short (hourly) and long (seasonal) time scales and the information content of each individual seal in an image is exceptionally low (Fig. 1).
Though it is possible to find seal-sized objects in VHR imagery manually, this laborious approach is only feasible at local scales (e.g., LaRue et al., 2011), introduces observer biases (Dickinson et al., 2010), and is not easily scaled to allow annotation of every high spatial resolution image captured within the range of pack-ice seals. Thus, repeatable, large scale wildlife surveys require automated detection systems (Conn et al., 2014). Traditional pixel or object-based methods for remote sensing scene understanding (RSISU) (e.g. Koju et al., 2018; McNabb et al., 2016), perhaps due to their reliance on hand-crafted features and spectral signatures, struggle at the increased granularity posed by high spatial resolution satellite imagery. As is the case for many fields such as computer vision (Voulodimos et al., 2018) and natural language processing (Do et al., 2019), deep learning, in the specific flavor of Convolutional Neural Networks (CNNs), are now the state-of-the art for RSISU (Gu et al., 2019), and is likely our best candidate for automated seal detection in high spatial resolution imagery. CNNs work by learning a series of convolution kernels – analogous to image processing kernels – as they learn to map inputs in the training data to their corresponding labels. CNNs have now been successfully employed in many ecological settings such as identifying whales (Borowicz et al., 2019; Polzounov et al., 2016), finding mammals in the African Savanna with UAV imagery (Kellenberger et al., 2018) and classifying animals in camera trap pictures (Norouzzadeh et al., 2018).
In this work, we explore the viability of CNNs to locate pack-ice seals in Antarctica and the scalability of this approach, with the ultimate goal of facilitating continental-scale population counts for pack-ice seals and other large bodied animals. Like many other wildlife detection sampling schemes (Kellenberger et al., 2018; Xue et al., 2017), however, the vast majority of the VHR imagery contains no true positives (i.e. seals), creating the potential for significant false positives even if the false positive rate is low. We propose a seal detection pipeline that i) determines whether a portion of the image is occupied by seals; ii) counts seals in that portion of the image and; iii) locates the centroid of each identified seal. All of the above is performed in a single pass with our proposed CNN architecture, SealNet.1 In our validation and test sets, this approach is superior to pure regression or semantic segmentation approaches.
Section snippets
Selecting imagery
For this pipeline, we use Worldview 3 (WV03) imagery provided by DigitalGlobe, Inc., which has the highest available resolution for commercial imagery with a 0.3 m resolution at nadir in panchromatic imagery and 1.5 m with 16 multispectral bands (Red, Green, Blue, Red Edge, Coastal, Yellow and 2 near-infrared bands). Only the panchromatic band was used for this work because individual seals are difficult to spot at lower resolutions and because the color information is not highly informative
Validation
SealNet, with added branches for counting and occupancy, attained 0.887 precision and 0.845 recall at our validation set, outperformed base U-Net (precision = 0.250, recall = 0.993), but was slightly outperformed by U-Net + count (precision = 0.897, recall = 0.853) (Fig. 6a). Adding a counting branch to U-Net, when compared with heatmap thresholding approach, improved precision at our validation set more than threefold, at the cost of a small decrement in recall. Adding an occupancy branch to
CNN performance
Even with a relatively small training set (Table 1), weakly-supervised training samples and a test set with only 1168 seals distributed over 150,000 non-overlapping patches, our pipeline often produces reasonable predictions, including unmistakable seals missed by our double-observer count (Fig. 7). In contrast with typical usages of deep learning for RSISU, which rely on bounding box based approaches (e.g. YOLO [Redmon et al., 2015]), we explore instance-based approaches, in the form of U-Net
Author contribution
BG selected scenes for training and testing and did the manual annotation of imagery for training. BG and HL designed the testing double-observer approach, did the manual annotation of imagery for testing, and interpreted results. BG led the SealNet development and coding. BS helped with code development and computational scaling. All authors contributed to the manuscript.
Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
We thank the Institute for Advanced Computational Science and the National Science Foundation EarthCube program (Award 1740595) for funding this work. Geospatial support for this work provided by the Polar Geospatial Center under NSF-OPP awards 1043681 and 1559691. The development of our CNN detection pipeline would not be possible without the advice of Felipe Codevilla Moraes, Hieu Le, Dimitris Samaras and the Stony Brook Computer Vision lab. We thank the Polar Geospatial Center for curating
References (68)
- et al.
Coexistence of mesopredators in an intact polar ocean ecosystem: the basis for defining a Ross Sea marine protected area
Biol. Conserv.
(2012) - et al.
Distribution, density, and abundance of pack-ice seals in the Amundsen and Ross Seas, Antarctica
Deep Sea Res. Part 2 Top. Stud. Ocean.
(2011) - et al.
Isotopic niche overlap and partition among three Antarctic seals from the Western Antarctic Peninsula
Deep Sea Res. Part II Top. Stud. Oceanogr.
(2018) - et al.
Responses of Antarctic pack-ice seals to environmental change and increasing krill fishing
Biol. Conserv.
(2012) - et al.
Detecting mammals in UAV images: best practices to address a substantially imbalanced dataset with deep learning
Remote Sens. Environ.
(2018) - et al.
Uncommon or cryptic? Challenges in estimating leopard seal abundance by conventional but state-of-the-art methods
Deep. Res. Part I Oceanogr. Res. Pap.
(2008) Comparative abundance and distribution of major filter-feeders in the Antarctic Pelagic Zone
J. Mar. Syst.
(1998)- et al.
The International Antarctic Pack Ice Seals (APIS) Program. Multi-disciplinary Research into the Ecology and Behavior of Antarctic Pack Ice Seals. Summary Update
(2006) - et al.
Improving Object Counting With Heatmap Regulation
(2018) Antarctic pack ice seals: an international research program co-ordinated by the SCAR group of specialists on seals
Phytoplankton dynamics within 37 Antarctic coastal polynya systems
J. Geophys. Res.
Diving and haulout behavior of crabeater seals in the Weddell Sea, Antarctica, during March 1986
Polar Biol.
Habitat partitioning among Antarctic pack ice seals
Population densities of pack ice seals in the Lazarev Sea, Antarctica
Antarct. Sci.
Population densities of pack ice seals in the Lazarev Sea, Antarctica
Antarct. Sci.
Aerial-trained deep learning networks for surveying cetaceans from satellite imagery
PLoS One
Detection errors in wildlife abundance estimates from Unmanned Aerial Systems (UAS) surveys: synthesis, solutions, and challenges
Methods Ecol. Evol.
Energy flow in the Southern Ocean food web
Count-ception: counting by fully convolutional redundant counting
Estimating multispecies abundance using automated detection systems: ice-associated seals in the Bering Sea
Methods Ecol. Evol.
Citizen science as an ecological research tool: challenges and benefits
Annu. Rev. Ecol. Evol. Syst.
Deep learning for aspect-based sentiment analysis: a comparative review
Expert Syst. Appl.
Continental estimates and population trends of Antarctic ice seals
Impact of climate change on Antarctic krill
Mar. Ecol. Prog. Ser.
Dominance of the Southern Ocean in anthropogenic carbon and heat uptake in CMIP5 models
J. Clim.
A survey on deep learning-driven remote sensing image scene understanding: scene classification, scene retrieval and scene-guided object detection
Appl. Sci.
Distribution, density and abundance of Antarctic ice seals off Queen Maud Land and the eastern Weddell Sea
Polar Biol.
Relative changes in krill abundance inferred from antarctic fur seal
PLoS One
Diet of a specialist in a changing environment: the crabeater seal along the western Antarctic Peninsula
Mar. Ecol. Prog. Ser.
Risk maps for Antarctic krill under projected Southern Ocean acidification
Nat. Clim. Chang.
Impacts of rising sea temperature on krill increase risks for predators in the Scotia Sea
PLoS One
A two-scale approach for estimating forest aboveground biomass with optical remote sensing images in a subtropical forest of Nepal
J. For. Res.
Influence of time of day and month on Weddell seal haul-out patterns at the Vestfold Hills, Antarctica
Polar Biol.
Cited by (29)
Seals from outer space - Population census of southern elephant seals using VHR satellite imagery
2022, Remote Sensing Applications: Society and EnvironmentCitation Excerpt :The relative error seems to be an error that cannot be excluded in any way and indicates that with the currently available VHR image resolution, it is not possible to obtain a solution to identify SES females with the use of very popular deep learning models for which precision and recall would be 1. Currently, the selection of objects that are admitted to the training stage of any machine learning model is usually made by two independent observers (i.e., Gonçalves et al., 2020; Duporge et al., 2021), and it is customary for them to require consensus on a given object. Any ambiguity in interpretation that is not identified by both observers is removed (Duporge et al., 2021), or the observers seek consensus by analysing the questionable results (Gonçalves et al., 2020).
Using Web images to train a deep neural network to detect sparsely distributed wildlife in large volumes of remotely sensed imagery: A case study of polar bears on sea ice
2022, Ecological InformaticsCitation Excerpt :A key advantage of CNNs is that they independently determine the distinguishing visual characteristics of target features with sophisticated abstraction, offering greater plasticity than classical automated image analysis approaches that rely on a predefined set of simplistic, hard-coded variables (Ball et al., 2017). Deep learning approaches are increasingly being used in ecology (Christin et al., 2019), including for automated detection of a variety of wildlife in aerial imagery, such as marine mammals (Gonçalves et al., 2020; Gray et al., 2019a; Maire et al., 2015), terrestrial mammals (Corcoran et al., 2019; Eikelboom et al., 2019; Kellenberger et al., 2018), birds (Borowicz et al., 2018; Boudaoud et al., 2019; Hong et al., 2019) and turtles (Gray et al., 2019b). To achieve high accuracy and plasticity, CNNs must first be trained with a large and varied assortment of examples of target features, typically involving hundreds or thousands of images (Shahinfar et al., 2020).
Understanding the synergies of deep learning and data fusion of multispectral and panchromatic high resolution commercial satellite imagery for automated ice-wedge polygon detection
2020, ISPRS Journal of Photogrammetry and Remote SensingCitation Excerpt :Owing to its success in everyday image understanding, DLCNNs have now rapidly been adapted in VHSR remote sensing image analysis while superseding contemporary paradigms like GEOBIA (Zhang et al., 2016; Zhang et al., 2018; Ma et al., 2019). The potentials of adapting cutting-edge DLCNNs from computer vision applications to remote sensing image understanding problems have been successfully demonstrated in automated mapping of shrubs (Guirado et al., 2017), whales (Guirado et al., 2019), seals (Gonçalves et al., 2020), and also IWPs (Zhang et al., 2018; Abolt et al., 2019) from VHSR imagery. DL approaches are being adapted to finely classify Arctic permafrost features from high-resolution remote sensing imagery, imagery-derived digital elevation models (DEMs), and LiDAR-based DEMs.