Attentive neural cell instance segmentation
Introduction
The cellular mechanism involved in the lineage path from a single neural stem cell remains mysterious in neuroscience. With the aid of real-time microscopy imaging system (Ravin et al., 2008), the specification of neurons, astrocytes, and oligodendrocytes from a neural stem cell could be recorded as a time-lapse video. Neural cell instance segmentation, which aims to detect and segment every cell in a microscopic image simultaneously, lays the foundation for many important neuroscience applications, such as exploring fate specification in neural stem cells. An accurate and fast instance segmentation tool can be applied to large microscopic video datasets and is thus crucial to the analysis of neural cell behavior. Nevertheless, there are many difficulties in neural cell instance segmentation. First, neural cells vary in shape and size. Second, many neural cells have tiny and slender structures, such as filopodia and lamellipodia, which are essential to neural cell behavior analysis but quite difficult to segment. Third, neural cells tend to adhere to each other due to cell interaction. Last but not least, neural cells may have obscure contours, and microscopic images usually have low contrast and background impurities. These challenges are illustrated in Fig. 1. The aim of the current study is to provide a solution to segmentation of fine processes associated with filopodia and lamellipodia, which would help pinpoint the physical interaction points between neural stem cells during this critical period of development when cell type is being defined.
Detection and segmentation of cells in microscopic images have been extensively studied. However, most existing methods exclusively focus on either cell detection or segmentation. Although a few of them attempt to address both the two tasks, they generally treat the detection and segmentation separately with multiple stages. To name a few, Althoff et al. (2005) detect the neural stem cells with a multi-scale Laplacian of Gaussian (LoG) filter, and then segment each cell via dynamic programming. Peng et al. (2009) localize the stem cells using multi-scale blob and curvilinear structure detectors, and then delineate each cell with multi-level sets. Wu et al. (2015) detect cells via greedy search and then obtain the boundary of each cell using active contour. The above-mentioned unsupervised methods are sensitive to intensity variations, and many efforts would be required to adjust their parameters for each dataset.
Instance segmentation is a task combining both object detection and segmentation. Recently, this topic has received much attention along with the development of supervised deep neural network (DNN) techniques. DNN methods have achieved remarkable performance in many computer vision problems, such as object detection (Girshick, Donahue, Darrell, Malik, 2014, Girshick, 2015, Ren, He, Girshick, Sun, 2015, Redmon, Divvala, Girshick, Farhadi, 2016, Liu, Anguelov, Erhan, Szegedy, Reed, Fu, Berg, 2016, Fu, Liu, Ranga, Tyagi, Berg) and semantic segmentation (Long, Shelhamer, Darrell, 2015, Noh, Hong, Han, 2015, Yang, Huang, Axel, Metaxas, 2018). Based on these single-task networks, several DNN models were proposed for instance segmentation, such as MNC (Dai et al., 2016), FCIS (Li et al., 2017), Mask R-CNN (He et al., 2017), and MaskLab (Chen et al., 2018). Existing instance segmentation methods are mainly based on two-stage object detector, at the core of which is a region proposal network (RPN) (Ren et al., 2015). To perform segmentation, they generally adopt the deep feature maps that contain high-level semantics, while ignoring the shallow ones which are rich in low-level shape and texture information of the objects. Consequently, these methods are insufficient for capturing details, such as the tiny and slender structures of neural cells.
To overcome the drawbacks mentioned above, we propose an attentive instance segmentation model that is able to accurately capture the neural cell instances at a fast speed. The overview of our approach is shown in Fig. 2. In particular, our method employs a single shot multi-box detector (SSD) (Liu et al., 2016) to detect neural cells in the input image. To improve detection accuracy and speed, we propose two strategies. First, we employ a feature fusion module, which consolidates shallow (fine) and deep (coarse) feature maps to facilitate the detection of small cells. Second, we incorporate an attention module in SSD to help it focus on useful image regions while suppressing the irrelevant background information. With the bounding box predictions from SSD, we then crop the cell instance regions accordingly from multi-scale feature maps and pass them into the mask prediction module. To perform the cell segmentation, we build a U-net (Ronneberger et al., 2015) that shares the backbone layers with SSD. The U-net propagates semantics from the deep layers to the shallow ones through a skip connection. To highlight useful regions and suppress the noisy information, we design and incorporate an attention module to the skip connection.
The proposed model is fast and accurate. It is capable of capturing the tiny and slender structures of neural cells. This paper makes several contributions. (1) Different from the state-of-the-art instance segmentation networks, the proposed model builds upon a one-stage object detector SSD and inherits its fast speed. (2) Existing CNN-based instance segmentation networks rely on ROI pooling or ROI aligning strategy, which samples a cropped region to a small fixed size from a particular deep feature map. This strategy loses details of neural cells. In contrast, our segmentation network combines multi-scale feature maps and therefore is able to capture the tiny and slender structures of neural cells. (3) We design two different attention units to respectively improve the accuracy of neural cell detection and segmentation.
This paper is an extension of our preliminary works (Yi, Wu, Hoeppner, Metaxas, 2018a, Yi, Wu, Jiang, Hoeppner, Metaxas, 2018b) with several modifications. First, we introduced two kinds of attention mechanisms to enhance the accuracy of both detection and segmentation. Second, we added ablation studies, more comparison experiments and mathematical formulations.
The rest of this paper is organized as follows. Section 2 reviews relevant works of detection, segmentation, and instance segmentation. Section 3 describes the proposed attentive neural cell instance segmentation approach. Section 4 presents the experimental results, including the ablation study, comparison experiments. Finally, Section 5 concludes this paper.
Section snippets
Related work
In this section, we first summarize some current cell detection and segmentation methods. We also review the state-of-the-art deep learning based detection and segmentation methods. Then, we review the development of instance segmentation techniques. Finally, we briefly introduce the attention models.
Methods
An overview of the proposed network is given in Fig. 2. The proposed model is a unified and end-to-end trainable network that simultaneously performs cell detection and segmentation. The input image is resized to 512 × 512 before being fed into the network. The convolutional layers, from conv2 to conv4, are residual networks (He et al., 2016). The specific settings of the convolutional layers are listed in Table 1. Below we introduce our neural cell detection and segmentation modules in details.
Data description
We sample 644 neural cell images from a collection of time-lapse microscopic videos of rat CNS stem cells (Ravin et al., 2008), where the image size is 640 × 512. The neural cell data is captured using the Nomarski DIC optics with a 40x oil NA 1.3 lens and the ORCA ER CCD camera which captures 16-bit grayscale with 2 × 2 binning at final 640 × 512 px at 1.4 um/px. Images are updated every 2 min. We randomly select 386 images for training, 129 for validation, and 129 for testing. Our annotations
Conclusion
This paper proposes an attentive neural cell instance segmentation method. The employment of the one-stage object detector makes the proposed model accurate and fast in detection. Besides, the proposed attention modules enhance the model ability to learn the cell objectness information as well as capture the slender and tiny structures of the cells. These properties indicate a great potential of our method in physical interaction study between neural stem cells.
Future endeavors will be devoted
Conflict of interest
This manuscript is an extension of our previous works:
- 1.
Pixel-wise neural cell instance segmentation. ISBI, 2018.
- 2.
Instance segmentation of neural cells. ECCV Workshop, 2018.
The manuscript inherits the main instance segmentation network that jointly employs SSD and U-net. The manuscript additionally proposes two kinds of attention mechanism to improve the accuracy of both detection and segmentation. Moreover, the manuscript adds more ablations studies, comparison experiments, mathematical
References (62)
- et al.
Dcan: deep contour-aware networks for object instance segmentation from histology images
Med. Image Anal.
(2017) - et al.
A decision-theoretic generalization of on-line learning and an application to boosting
J. Comput. Syst. Sci.
(1997) - et al.
Deepmitosis: mitosis detection via deep detection, verification and segmentation networks
Med. Image Anal.
(2018) - et al.
Fully automatic detection and segmentation of abdominal aortic thrombus in post-operative cta images using deep convolutional neural networks
Med. Image Anal.
(2018) - et al.
Potency and fate specification in CNS stem cell populations in vitro
Cell Stem Cell
(2008) - et al.
A survey of thresholding techniques
Comput. Vis. Graph. Image Process.
(1988) - et al.
Improved automatic detection and segmentation of cell nuclei in histopathology images
IEEE Trans. Biomed. Eng.
(2010) - et al.
Combined segmentation and tracking of neural stem-cells
Proc. Scand. Conf. Image Anal.
(2005) - et al.
Cell segmentation and tracking in phase contrast images using graph cut with asymmetric boundary costs
Proc. IEEE Int. Symp. Biomed. Imaging
(2015) - et al.
Finding dots: segmentation as popping out regions from boundaries
Proc. IEEE Comput. Vis. Pattern Recognit.
(2010)
Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images
Proc. IEEE Int. Conf. Comput. Vis.
Random forests
Mach. Learn.
MaskLab: instance segmentation by refining object detection with semantic and direction features
Proc. IEEE Comput. Vis. Pattern Recognit.
Attention to scale: scale-aware semantic image segmentation
Proc. IEEE Comput. Vis. Pattern Recognit.
Support-vector networks
Mach. Learn.
Instance-aware semantic segmentation via multi-task network cascades
Proc. IEEE Comput. Vis. Pattern Recognit.
Histograms of oriented gradients for human detection
Proc. IEEE Comput. Vis. Pattern Recognit.
ImageNet: a large-scale hierarchical image database
Proc. IEEE Comput. Vis. Pattern Recognit.
The PASCAL visual object classes (VOC) challenge
Int. J. Comput. Vis.
Automatic lymphocyte detection on gastric cancer IHCimages using deep learning
Proc. IEEE Int. Symp. on Computer-Based Medi. Sys.
Fast R-CNN
Proc. IEEE Int. Conf. Comput. Vis.
Rich feature hierarchies for accurate object detection and semantic segmentation
Proc. IEEE Comput. Vis. Pattern Recognit.
Mask R-CNN
Proc. IEEE Int. Conf. Comput. Vis.
Deep residual learning for image recognition
Proc. IEEE Comput. Vis. Pattern Recognit.
Relation networks for object detection
Proc. IEEE Comput. Vis. Pattern Recognit.
Batch normalization: accelerating deep network training by reducing internal covariate shift
Proc. Int. Conf. Mach. Learn.
You should use regression to detect cells
Proc. Med. Image Comput. Comput. Assist. Interv., Part 3
Snakes: active contour models
Int. J. Comput. Vis.
Smart markers for watershed-based cell segmentation
PloS one
Fully convolutional instance-aware semantic segmentation
Proc. IEEE Comput. Vis. Pattern Recognit.
Cited by (85)
Label-free microscopic cell images adaptive enhancement via weighted fusion of bright, dark, and weak structure features
2024, Biomedical Signal Processing and ControlDeep learning in mesoscale brain image analysis: A review
2023, Computers in Biology and MedicineNuclei instance segmentation using a transformer-based graph convolutional network and contextual information augmentation
2023, Computers in Biology and MedicineSaliency and ballness driven deep learning framework for cell segmentation in bright field microscopic images
2023, Engineering Applications of Artificial IntelligenceReal-time and accurate deep learning-based multi-organ nucleus segmentation in histology images
2024, Journal of Real-Time Image ProcessingImage segmentation based on U-Net++ network method to identify Bacillus Subtilis cells in micro-droplets
2024, Multimedia Tools and Applications