Elsevier

Medical Image Analysis

Volume 55, July 2019, Pages 228-240
Medical Image Analysis

Attentive neural cell instance segmentation

https://doi.org/10.1016/j.media.2019.05.004Get rights and content

Highlights

  • The model could segment neural cell instances accurately.

  • The model is computationally efficient.

  • The model could be applied to other biomedical instance segmentation tasks.

Abstract

Neural cell instance segmentation, which aims at joint detection and segmentation of every neural cell in a microscopic image, is essential to many neuroscience applications. The challenge of this task involves cell adhesion, cell distortion, unclear cell contours, low-contrast cell protrusion structures, and background impurities. Consequently, current instance segmentation methods generally fall short of precision. In this paper, we propose an attentive instance segmentation method that accurately predicts the bounding box of each cell as well as its segmentation mask simultaneously. In particular, our method builds on a joint network that combines a single shot multi-box detector (SSD) and a U-net. Furthermore, we employ the attention mechanism in both detection and segmentation modules to focus the model on the useful features. The proposed method is validated on a dataset of neural cell microscopic images. Experimental results demonstrate that our approach can accurately detect and segment neural cell instances at a fast speed, comparing favorably with the state-of-the-art methods. Our code is released on GitHub. The link is https://github.com/yijingru/ANCIS-Pytorch.

Introduction

The cellular mechanism involved in the lineage path from a single neural stem cell remains mysterious in neuroscience. With the aid of real-time microscopy imaging system (Ravin et al., 2008), the specification of neurons, astrocytes, and oligodendrocytes from a neural stem cell could be recorded as a time-lapse video. Neural cell instance segmentation, which aims to detect and segment every cell in a microscopic image simultaneously, lays the foundation for many important neuroscience applications, such as exploring fate specification in neural stem cells. An accurate and fast instance segmentation tool can be applied to large microscopic video datasets and is thus crucial to the analysis of neural cell behavior. Nevertheless, there are many difficulties in neural cell instance segmentation. First, neural cells vary in shape and size. Second, many neural cells have tiny and slender structures, such as filopodia and lamellipodia, which are essential to neural cell behavior analysis but quite difficult to segment. Third, neural cells tend to adhere to each other due to cell interaction. Last but not least, neural cells may have obscure contours, and microscopic images usually have low contrast and background impurities. These challenges are illustrated in Fig. 1. The aim of the current study is to provide a solution to segmentation of fine processes associated with filopodia and lamellipodia, which would help pinpoint the physical interaction points between neural stem cells during this critical period of development when cell type is being defined.

Detection and segmentation of cells in microscopic images have been extensively studied. However, most existing methods exclusively focus on either cell detection or segmentation. Although a few of them attempt to address both the two tasks, they generally treat the detection and segmentation separately with multiple stages. To name a few, Althoff et al. (2005) detect the neural stem cells with a multi-scale Laplacian of Gaussian (LoG) filter, and then segment each cell via dynamic programming. Peng et al. (2009) localize the stem cells using multi-scale blob and curvilinear structure detectors, and then delineate each cell with multi-level sets. Wu et al. (2015) detect cells via greedy search and then obtain the boundary of each cell using active contour. The above-mentioned unsupervised methods are sensitive to intensity variations, and many efforts would be required to adjust their parameters for each dataset.

Instance segmentation is a task combining both object detection and segmentation. Recently, this topic has received much attention along with the development of supervised deep neural network (DNN) techniques. DNN methods have achieved remarkable performance in many computer vision problems, such as object detection (Girshick, Donahue, Darrell, Malik, 2014, Girshick, 2015, Ren, He, Girshick, Sun, 2015, Redmon, Divvala, Girshick, Farhadi, 2016, Liu, Anguelov, Erhan, Szegedy, Reed, Fu, Berg, 2016, Fu, Liu, Ranga, Tyagi, Berg) and semantic segmentation (Long, Shelhamer, Darrell, 2015, Noh, Hong, Han, 2015, Yang, Huang, Axel, Metaxas, 2018). Based on these single-task networks, several DNN models were proposed for instance segmentation, such as MNC (Dai et al., 2016), FCIS (Li et al., 2017), Mask R-CNN (He et al., 2017), and MaskLab (Chen et al., 2018). Existing instance segmentation methods are mainly based on two-stage object detector, at the core of which is a region proposal network (RPN) (Ren et al., 2015). To perform segmentation, they generally adopt the deep feature maps that contain high-level semantics, while ignoring the shallow ones which are rich in low-level shape and texture information of the objects. Consequently, these methods are insufficient for capturing details, such as the tiny and slender structures of neural cells.

To overcome the drawbacks mentioned above, we propose an attentive instance segmentation model that is able to accurately capture the neural cell instances at a fast speed. The overview of our approach is shown in Fig. 2. In particular, our method employs a single shot multi-box detector (SSD) (Liu et al., 2016) to detect neural cells in the input image. To improve detection accuracy and speed, we propose two strategies. First, we employ a feature fusion module, which consolidates shallow (fine) and deep (coarse) feature maps to facilitate the detection of small cells. Second, we incorporate an attention module in SSD to help it focus on useful image regions while suppressing the irrelevant background information. With the bounding box predictions from SSD, we then crop the cell instance regions accordingly from multi-scale feature maps and pass them into the mask prediction module. To perform the cell segmentation, we build a U-net (Ronneberger et al., 2015) that shares the backbone layers with SSD. The U-net propagates semantics from the deep layers to the shallow ones through a skip connection. To highlight useful regions and suppress the noisy information, we design and incorporate an attention module to the skip connection.

The proposed model is fast and accurate. It is capable of capturing the tiny and slender structures of neural cells. This paper makes several contributions. (1) Different from the state-of-the-art instance segmentation networks, the proposed model builds upon a one-stage object detector SSD and inherits its fast speed. (2) Existing CNN-based instance segmentation networks rely on ROI pooling or ROI aligning strategy, which samples a cropped region to a small fixed size from a particular deep feature map. This strategy loses details of neural cells. In contrast, our segmentation network combines multi-scale feature maps and therefore is able to capture the tiny and slender structures of neural cells. (3) We design two different attention units to respectively improve the accuracy of neural cell detection and segmentation.

This paper is an extension of our preliminary works (Yi, Wu, Hoeppner, Metaxas, 2018a, Yi, Wu, Jiang, Hoeppner, Metaxas, 2018b) with several modifications. First, we introduced two kinds of attention mechanisms to enhance the accuracy of both detection and segmentation. Second, we added ablation studies, more comparison experiments and mathematical formulations.

The rest of this paper is organized as follows. Section 2 reviews relevant works of detection, segmentation, and instance segmentation. Section 3 describes the proposed attentive neural cell instance segmentation approach. Section 4 presents the experimental results, including the ablation study, comparison experiments. Finally, Section 5 concludes this paper.

Section snippets

Related work

In this section, we first summarize some current cell detection and segmentation methods. We also review the state-of-the-art deep learning based detection and segmentation methods. Then, we review the development of instance segmentation techniques. Finally, we briefly introduce the attention models.

Methods

An overview of the proposed network is given in Fig. 2. The proposed model is a unified and end-to-end trainable network that simultaneously performs cell detection and segmentation. The input image is resized to 512 × 512 before being fed into the network. The convolutional layers, from conv2 to conv4, are residual networks (He et al., 2016). The specific settings of the convolutional layers are listed in Table 1. Below we introduce our neural cell detection and segmentation modules in details.

Data description

We sample 644 neural cell images from a collection of time-lapse microscopic videos of rat CNS stem cells (Ravin et al., 2008), where the image size is 640 × 512. The neural cell data is captured using the Nomarski DIC optics with a 40x oil NA 1.3 lens and the ORCA ER CCD camera which captures 16-bit grayscale with 2 × 2 binning at final 640 × 512 px at 1.4 um/px. Images are updated every 2 min. We randomly select 386 images for training, 129 for validation, and 129 for testing. Our annotations

Conclusion

This paper proposes an attentive neural cell instance segmentation method. The employment of the one-stage object detector makes the proposed model accurate and fast in detection. Besides, the proposed attention modules enhance the model ability to learn the cell objectness information as well as capture the slender and tiny structures of the cells. These properties indicate a great potential of our method in physical interaction study between neural stem cells.

Future endeavors will be devoted

Conflict of interest

This manuscript is an extension of our previous works:

  • 1.

    Pixel-wise neural cell instance segmentation. ISBI, 2018.

  • 2.

    Instance segmentation of neural cells. ECCV Workshop, 2018.

The manuscript inherits the main instance segmentation network that jointly employs SSD and U-net. The manuscript additionally proposes two kinds of attention mechanism to improve the accuracy of both detection and segmentation. Moreover, the manuscript adds more ablations studies, comparison experiments, mathematical

References (62)

  • Y.Y. Boykov et al.

    Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images

    Proc. IEEE Int. Conf. Comput. Vis.

    (2001)
  • L. Breiman

    Random forests

    Mach. Learn.

    (2001)
  • L.-C. Chen et al.

    MaskLab: instance segmentation by refining object detection with semantic and direction features

    Proc. IEEE Comput. Vis. Pattern Recognit.

    (2018)
  • L.-C. Chen et al.

    Attention to scale: scale-aware semantic image segmentation

    Proc. IEEE Comput. Vis. Pattern Recognit.

    (2016)
  • C. Cortes et al.

    Support-vector networks

    Mach. Learn.

    (1995)
  • J. Dai et al.

    Instance-aware semantic segmentation via multi-task network cascades

    Proc. IEEE Comput. Vis. Pattern Recognit.

    (2016)
  • N. Dalal et al.

    Histograms of oriented gradients for human detection

    Proc. IEEE Comput. Vis. Pattern Recognit.

    (2005)
  • J. Deng et al.

    ImageNet: a large-scale hierarchical image database

    Proc. IEEE Comput. Vis. Pattern Recognit.

    (2009)
  • M. Everingham et al.

    The PASCAL visual object classes (VOC) challenge

    Int. J. Comput. Vis.

    (2010)
  • Fu, C.-Y., Liu, W., Ranga, A., Tyagi, A., Berg, A. C., 2017. DSSD: deconvolutional single shot detector....
  • E. Garcia et al.

    Automatic lymphocyte detection on gastric cancer IHCimages using deep learning

    Proc. IEEE Int. Symp. on Computer-Based Medi. Sys.

    (2017)
  • R.B. Girshick

    Fast R-CNN

    Proc. IEEE Int. Conf. Comput. Vis.

    (2015)
  • R.B. Girshick et al.

    Rich feature hierarchies for accurate object detection and semantic segmentation

    Proc. IEEE Comput. Vis. Pattern Recognit.

    (2014)
  • K. He et al.

    Mask R-CNN

    Proc. IEEE Int. Conf. Comput. Vis.

    (2017)
  • K. He et al.

    Deep residual learning for image recognition

    Proc. IEEE Comput. Vis. Pattern Recognit.

    (2016)
  • H. Hu et al.

    Relation networks for object detection

    Proc. IEEE Comput. Vis. Pattern Recognit.

    (2018)
  • S. Ioffe et al.

    Batch normalization: accelerating deep network training by reducing internal covariate shift

    Proc. Int. Conf. Mach. Learn.

    (2015)
  • P. Kainz et al.

    You should use regression to detect cells

    Proc. Med. Image Comput. Comput. Assist. Interv., Part 3

    (2015)
  • M. Kass et al.

    Snakes: active contour models

    Int. J. Comput. Vis.

    (1988)
  • C.F. Koyuncu et al.

    Smart markers for watershed-based cell segmentation

    PloS one

    (2012)
  • Y. Li et al.

    Fully convolutional instance-aware semantic segmentation

    Proc. IEEE Comput. Vis. Pattern Recognit.

    (2017)
  • Cited by (85)

    • Deep learning in mesoscale brain image analysis: A review

      2023, Computers in Biology and Medicine
    View all citing articles on Scopus
    View full text