SealNet: A fully-automated pack-ice seal detection pipeline for sub-meter satellite imagery

https://doi.org/10.1016/j.rse.2019.111617Get rights and content

Highlights

  • First automated system for surveying seals using satellite imagery

  • Finds 30% of seals while only generating less than 2 false-positives per correct detection

  • Over 10× faster than an experienced human observer using a single GPU

Abstract

Antarctic pack-ice seals, a group of four species of true seals (Phocidae), play a pivotal role in the Southern Ocean foodweb as wide-ranging predators of Antarctic krill (Euphausia superba). Due to their circumpolar distribution and the remoteness and vastness of their habitat, little is known about their population sizes. Estimating pack-ice seal population sizes and trends is key to understanding how the Southern Ocean ecosystem will react to threats such as climate change driven sea ice loss and krill fishing. We present a functional pack-ice seal detection pipeline using Worldview-3 imagery and a Convolutional Neural Network that counts and locates seal centroids. We propose a new CNN architecture that detects objects by combining semantic segmentation heatmaps with binary classification and counting by regression. Our pipeline locates over 30% of seals, when compared to consensus counts from human experts, and reduces the time required for seal detection by 95% (assuming just a single GPU). While larger training sets and continued algorithm development will no doubt improve classification accuracy, our pipeline, which can be easily adapted for other large-bodied animals visible in sub-meter satellite imagery, demonstrates the potential for machine learning to vastly expand our capacity for regular pack-ice seal surveys and, in doing so, will contribute to ongoing international efforts to monitor pack-ice seals.

Introduction

The Southern Ocean (SO) harbors major seasonal hotspots for primary productivity (Arrigo and Dijken, 2003). The cold, nutrient rich waters of the SO play a fundamental role regulating global climate, both by absorbing large amounts of heat and sinking fixed carbon (Frölicher et al., 2015; Morrison et al., 2016). Food webs in the SO are trophically shallow (Clarke, 1985), but they more than compensate in terms of biomass, sustaining massive concentrations of phytoplankton consumers (Bester et al., 2002a; Nowacek et al., 2011). Among these consumers, a small crustacean, Antarctic krill (Euphasia superba), is especially important; krill is the main food item for a wide range of upper tier consumers, from fish and penguins to seals and whales, and serves as a fundamental link between predators and primary producers. Due to krill's role in the SO food web, assessing and tracking Antarctic krill stocks is central to Antarctic ecology. This is especially true now that climate change (Flores et al., 2012; Klein et al., 2018), ocean acidification (Kawaguchi et al., 2013) and krill fisheries (Forcada et al., 2012) threaten to shift the abundance and distribution of this key Antarctic species. Challenging our efforts to track Antarctic krill, however, is its small size and patchy distribution (Voronina, 1998). One way to circumvent those difficulties is to use krill predator abundances as a proxy for krill distribution (Huang et al., 2011). Antarctic pack-ice seals (crabeater seals [Lobodon carcinophaga], Weddell seals [Leptonychotes weddelli], leopard seals [Hydrurga leptonyx] and Ross seals [Omnatophoca rossii], within the Phocidae family), as a group, represent a promising vehicle to gauge krill stocks for they are not only key krill consumers (Botta et al., 2018; Forcada et al., 2012; Hückstädt et al., 2012; Siniff and Stone, 1985) but they are also large enough to be individually spotted with high spatial resolution satellite imagery.

The potential of pack-ice seals as indicators of environmental health in the SO has not gone unnoticed; polar ecologists have channeled sizeable efforts into estimating pack-ice seal population sizes, the most notable of these attempts being the Antarctic pack-ice seal (APIS) project (Anonymous, 1997), a joint effort of six countries to estimate Antarctic seal populations using aerial surveys (Ackley et al., 2006). Conducting such large-scale aerial survey programs in Antarctica is extremely expensive, necessarily requiring extensive collaboration among Antarctic national programs. Fortunately, very high spatial resolution (VHR) satellite imagery may soon be a viable alternative for aerial surveys, providing greater spatial coverage and, due to its dramatically lower cost, increased repeatability. The use of VHR satellite imagery for wildlife survey has exploded in recent years, and includes demonstration projects for southern elephant seals (McMahon et al., 2014), polar bears (Stapleton et al., 2014) and African ungulates (Xue et al., 2017; Yang et al., 2014), as well as seabird species whose presence and abundance can be estimated indirectly using the guano stain at the colony (LaRue et al., 2014; Lynch et al., 2012). Pack-ice seals, while large enough to be seen in VHR imagery, are particularly hard to detect since their preferred haul out environment (pack ice; Bengtson and Stewart, 1992; Lake et al., 1997) changes on short (hourly) and long (seasonal) time scales and the information content of each individual seal in an image is exceptionally low (Fig. 1).

Though it is possible to find seal-sized objects in VHR imagery manually, this laborious approach is only feasible at local scales (e.g., LaRue et al., 2011), introduces observer biases (Dickinson et al., 2010), and is not easily scaled to allow annotation of every high spatial resolution image captured within the range of pack-ice seals. Thus, repeatable, large scale wildlife surveys require automated detection systems (Conn et al., 2014). Traditional pixel or object-based methods for remote sensing scene understanding (RSISU) (e.g. Koju et al., 2018; McNabb et al., 2016), perhaps due to their reliance on hand-crafted features and spectral signatures, struggle at the increased granularity posed by high spatial resolution satellite imagery. As is the case for many fields such as computer vision (Voulodimos et al., 2018) and natural language processing (Do et al., 2019), deep learning, in the specific flavor of Convolutional Neural Networks (CNNs), are now the state-of-the art for RSISU (Gu et al., 2019), and is likely our best candidate for automated seal detection in high spatial resolution imagery. CNNs work by learning a series of convolution kernels – analogous to image processing kernels – as they learn to map inputs in the training data to their corresponding labels. CNNs have now been successfully employed in many ecological settings such as identifying whales (Borowicz et al., 2019; Polzounov et al., 2016), finding mammals in the African Savanna with UAV imagery (Kellenberger et al., 2018) and classifying animals in camera trap pictures (Norouzzadeh et al., 2018).

In this work, we explore the viability of CNNs to locate pack-ice seals in Antarctica and the scalability of this approach, with the ultimate goal of facilitating continental-scale population counts for pack-ice seals and other large bodied animals. Like many other wildlife detection sampling schemes (Kellenberger et al., 2018; Xue et al., 2017), however, the vast majority of the VHR imagery contains no true positives (i.e. seals), creating the potential for significant false positives even if the false positive rate is low. We propose a seal detection pipeline that i) determines whether a portion of the image is occupied by seals; ii) counts seals in that portion of the image and; iii) locates the centroid of each identified seal. All of the above is performed in a single pass with our proposed CNN architecture, SealNet.1 In our validation and test sets, this approach is superior to pure regression or semantic segmentation approaches.

Section snippets

Selecting imagery

For this pipeline, we use Worldview 3 (WV03) imagery provided by DigitalGlobe, Inc., which has the highest available resolution for commercial imagery with a 0.3 m resolution at nadir in panchromatic imagery and 1.5 m with 16 multispectral bands (Red, Green, Blue, Red Edge, Coastal, Yellow and 2 near-infrared bands). Only the panchromatic band was used for this work because individual seals are difficult to spot at lower resolutions and because the color information is not highly informative

Validation

SealNet, with added branches for counting and occupancy, attained 0.887 precision and 0.845 recall at our validation set, outperformed base U-Net (precision = 0.250, recall = 0.993), but was slightly outperformed by U-Net + count (precision = 0.897, recall = 0.853) (Fig. 6a). Adding a counting branch to U-Net, when compared with heatmap thresholding approach, improved precision at our validation set more than threefold, at the cost of a small decrement in recall. Adding an occupancy branch to

CNN performance

Even with a relatively small training set (Table 1), weakly-supervised training samples and a test set with only 1168 seals distributed over 150,000 non-overlapping patches, our pipeline often produces reasonable predictions, including unmistakable seals missed by our double-observer count (Fig. 7). In contrast with typical usages of deep learning for RSISU, which rely on bounding box based approaches (e.g. YOLO [Redmon et al., 2015]), we explore instance-based approaches, in the form of U-Net

Author contribution

BG selected scenes for training and testing and did the manual annotation of imagery for training. BG and HL designed the testing double-observer approach, did the manual annotation of imagery for testing, and interpreted results. BG led the SealNet development and coding. BS helped with code development and computational scaling. All authors contributed to the manuscript.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

We thank the Institute for Advanced Computational Science and the National Science Foundation EarthCube program (Award 1740595) for funding this work. Geospatial support for this work provided by the Polar Geospatial Center under NSF-OPP awards 1043681 and 1559691. The development of our CNN detection pipeline would not be possible without the advice of Felipe Codevilla Moraes, Hieu Le, Dimitris Samaras and the Stony Brook Computer Vision lab. We thank the Polar Geospatial Center for curating

References (68)

  • K.R. Arrigo et al.

    Phytoplankton dynamics within 37 Antarctic coastal polynya systems

    J. Geophys. Res.

    (2003)
  • J.L. Bengtson et al.

    Diving and haulout behavior of crabeater seals in the Weddell Sea, Antarctica, during March 1986

    Polar Biol.

    (1992)
  • J.L. Bengtson et al.

    Habitat partitioning among Antarctic pack ice seals

  • M.N. Bester et al.

    Population densities of pack ice seals in the Lazarev Sea, Antarctica

    Antarct. Sci.

    (2002)
  • M.N. Bester et al.

    Population densities of pack ice seals in the Lazarev Sea, Antarctica

    Antarct. Sci.

    (2002)
  • A. Borowicz et al.

    Aerial-trained deep learning networks for surveying cetaceans from satellite imagery

    PLoS One

    (2019)
  • I.V. Brack et al.

    Detection errors in wildlife abundance estimates from Unmanned Aerial Systems (UAS) surveys: synthesis, solutions, and challenges

    Methods Ecol. Evol.

    (2018)
  • E. Clarke

    Energy flow in the Southern Ocean food web

  • J.P. Cohen et al.

    Count-ception: counting by fully convolutional redundant counting

  • P.B. Conn et al.

    Estimating multispecies abundance using automated detection systems: ice-associated seals in the Bering Sea

    Methods Ecol. Evol.

    (2014)
  • J.L. Dickinson et al.

    Citizen science as an ecological research tool: challenges and benefits

    Annu. Rev. Ecol. Evol. Syst.

    (2010)
  • H.H. Do et al.

    Deep learning for aspect-based sentiment analysis: a comparative review

    Expert Syst. Appl.

    (2019)
  • A.W. Erickson et al.

    Continental estimates and population trends of Antarctic ice seals

  • H. Flores et al.

    Impact of climate change on Antarctic krill

    Mar. Ecol. Prog. Ser.

    (2012)
  • T.L. Frölicher et al.

    Dominance of the Southern Ocean in anthropogenic carbon and heat uptake in CMIP5 models

    J. Clim.

    (2015)
  • Y. Gu et al.

    A survey on deep learning-driven remote sensing image scene understanding: scene classification, scene retrieval and scene-guided object detection

    Appl. Sci.

    (2019)
  • E. Gurarie et al.

    Distribution, density and abundance of Antarctic ice seals off Queen Maud Land and the eastern Weddell Sea

    Polar Biol.

    (2016)
  • T. Huang et al.

    Relative changes in krill abundance inferred from antarctic fur seal

    PLoS One

    (2011)
  • Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q., 2017. Densely connected convolutional networks, in:...
  • L. Hückstädt et al.

    Diet of a specialist in a changing environment: the crabeater seal along the western Antarctic Peninsula

    Mar. Ecol. Prog. Ser.

    (2012)
  • S. Kawaguchi et al.

    Risk maps for Antarctic krill under projected Southern Ocean acidification

    Nat. Clim. Chang.

    (2013)
  • E.S. Klein et al.

    Impacts of rising sea temperature on krill increase risks for predators in the Scotia Sea

    PLoS One

    (2018)
  • U.A. Koju et al.

    A two-scale approach for estimating forest aboveground biomass with optical remote sensing images in a subtropical forest of Nepal

    J. For. Res.

    (2018)
  • S.E. Lake et al.

    Influence of time of day and month on Weddell seal haul-out patterns at the Vestfold Hills, Antarctica

    Polar Biol.

    (1997)
  • Cited by (29)

    • Seals from outer space - Population census of southern elephant seals using VHR satellite imagery

      2022, Remote Sensing Applications: Society and Environment
      Citation Excerpt :

      The relative error seems to be an error that cannot be excluded in any way and indicates that with the currently available VHR image resolution, it is not possible to obtain a solution to identify SES females with the use of very popular deep learning models for which precision and recall would be 1. Currently, the selection of objects that are admitted to the training stage of any machine learning model is usually made by two independent observers (i.e., Gonçalves et al., 2020; Duporge et al., 2021), and it is customary for them to require consensus on a given object. Any ambiguity in interpretation that is not identified by both observers is removed (Duporge et al., 2021), or the observers seek consensus by analysing the questionable results (Gonçalves et al., 2020).

    • Using Web images to train a deep neural network to detect sparsely distributed wildlife in large volumes of remotely sensed imagery: A case study of polar bears on sea ice

      2022, Ecological Informatics
      Citation Excerpt :

      A key advantage of CNNs is that they independently determine the distinguishing visual characteristics of target features with sophisticated abstraction, offering greater plasticity than classical automated image analysis approaches that rely on a predefined set of simplistic, hard-coded variables (Ball et al., 2017). Deep learning approaches are increasingly being used in ecology (Christin et al., 2019), including for automated detection of a variety of wildlife in aerial imagery, such as marine mammals (Gonçalves et al., 2020; Gray et al., 2019a; Maire et al., 2015), terrestrial mammals (Corcoran et al., 2019; Eikelboom et al., 2019; Kellenberger et al., 2018), birds (Borowicz et al., 2018; Boudaoud et al., 2019; Hong et al., 2019) and turtles (Gray et al., 2019b). To achieve high accuracy and plasticity, CNNs must first be trained with a large and varied assortment of examples of target features, typically involving hundreds or thousands of images (Shahinfar et al., 2020).

    • Understanding the synergies of deep learning and data fusion of multispectral and panchromatic high resolution commercial satellite imagery for automated ice-wedge polygon detection

      2020, ISPRS Journal of Photogrammetry and Remote Sensing
      Citation Excerpt :

      Owing to its success in everyday image understanding, DLCNNs have now rapidly been adapted in VHSR remote sensing image analysis while superseding contemporary paradigms like GEOBIA (Zhang et al., 2016; Zhang et al., 2018; Ma et al., 2019). The potentials of adapting cutting-edge DLCNNs from computer vision applications to remote sensing image understanding problems have been successfully demonstrated in automated mapping of shrubs (Guirado et al., 2017), whales (Guirado et al., 2019), seals (Gonçalves et al., 2020), and also IWPs (Zhang et al., 2018; Abolt et al., 2019) from VHSR imagery. DL approaches are being adapted to finely classify Arctic permafrost features from high-resolution remote sensing imagery, imagery-derived digital elevation models (DEMs), and LiDAR-based DEMs.

    View all citing articles on Scopus
    View full text