Attention-aware concentrated network for saliency prediction

Abstract

This paper presents a biologically-inspired saliency prediction method to imitate two main characteristics of the human perception process: focalization and orienting. The proposed network, named ACNet is composed of two modules. The first one is an essential concentrated module (CM), which assists the network to “see” images with appropriate receptive fields by perceiving rich multi-scale multi-receptive-field contexts of high-level features. The second is a parallel attention module (PAM), which explicitly guides the network to learn “what” and “where” is salient by simultaneously capturing global and local information with channel-wise and spatial attention mechanisms. These two modules compose the core component of the proposed method, named ACBlock, which is cascaded to progressively refine the inference of saliency estimation in a manner similar to that humans zoom in their lens to focus on the saliency. Experimental results on seven public datasets demonstrate that the proposed ACNet outperforms the state-of-the-art models without any prior knowledge or post-processing.

Introduction

Saliency prediction aims to identify the spatial locations of human’s eye fixation points over an image, and it has received increasing attention in recent years. According to visual cognition science [1], humans selectively attend to distinctive salient regions of an image for capturing visual structure better and ignore the others. Numerous saliency models are proposed during the past few years and have been used in various applications and other vision tasks such as salient object detection [2], object segmentation [3], [4], image cropping [5] and diagnosis of mental illness [6].

According to biological evidence, traditional methods [7], [8], [9], [10] applied hand-crafted features based on low-level cues (e.g., contrast, texture, color) for saliency prediction. However, these features sometimes fail to simulate the complex activation of the human visual system, particularly in complex scenarios.

Recently, methods based on convolutional neural networks (CNNs) have emerged for saliency prediction and reported significant improvements. Owing to the complex semantic representations learned by CNNs, the learning based features have been demonstrated superior to that of the hand-crafted features on saliency prediction. Despite being studied actively, how to devise an effective yet efficient deep neural network model for saliency prediction remains an open challenge.

In this paper, a biologically-inspired architecture named attention-aware concentrated network (ACNet) is proposed for saliency prediction. In general, the operation of visual attention is thought as two-stage process [11]. In the first stage, attention is distributed uniformly over the external visual scenario and concurrently processes information. Subsequently, attention is concentrated on a specific area of the visual scenario. Such a cognitive process reveals two characteristics of the human perception process: focalization and orienting. In order to imitate these characteristics, a concentrated module is designed along with a parallel attention module, together forming the proposed method. The two aspects of our motivation are described as follows.

Firstly, the model is motivated by the focalization of the human perception process, which is the act of concentrating on a discrete aspect of information while ignoring the other perceivable information. While encountering complex scenarios, in which the salient objects have different sizes, locations and similar structure as background, humans can adaptively zoom in the lens to concentrate on salient regions and the rest of the area becomes invisible. To imitate such a zoom-in process, a module that can extract features with various receptive fields is needed. Therefore, we design a concentrated module (CM) to perceive various local contexts and explicitly handle the problem of extracting multi-scale saliency features. As is shown in Fig. 1(b), features from CM response with multi-receptive-field, which is suitable for a variety of cases, e.g., long or short distance(2, 3 row), small or big object size(1, 3 row). However, some features from CM maybe also harmful for saliency prediction. As shown in first and third columns of Fig. 1(b), features with large receptive fields may respond to redundant details of background, while the features with small receptive fields may lose some salient information. Therefore, these features need to be sifted by another module, which is described in the next paragraph.

Secondly, motivated by the orienting of visual attention, which acts to selectively attend to an object or location over others by moving eyes to point in that direction [12], we design a parallel attention module (PAM) that inherits the feature-enhancing ability and extract global and local information simultaneously to generate context-aware attention map. With such parallel attention module, the network can explicitly learn “what” and “where” should be emphasized. As is shown in Fig. 1(c), the first column represents spatial attention maps generated by the PAM. As we can see, clustered background that may generate distractions is compressed while salient regions are highlighted. As for channel-wise attention shown in Fig. 1(d), different channels that response to various objects of the input images are assigned weights to reflect the relative importance, i.e., to emphasize the salient object. Similar to humans who are attracted by salient objects and move eye fixations to discrete areas, the proposed network is explicitly guided to choose salient objects (by channel-wise attention) and focus on salient regions (though spatial attention). In addition, this module serves as a feature filter for CM to adaptively select appropriate scale and receptive-field for generating saliency regions.

The CM and PAM construct the ACBlock, which is able to progressively refine the inference of saliency estimation by explicitly guiding the network to “see” images with appropriate receptive field and learn “what” and “where” is salient.

To summarize, the contributions of this paper are as follows:

• It provides a deeper insight into Saliency Prediction through the imitation of the focalization and orienting of the human perception process and an end-to-end framework that refines saliency estimation in a progressive manner.

• We design a concentrated module (CM) which perceives various local contexts though a series of group convolutional filters with different receptive fields, thereby explicitly enlarging the receptive field of saliency model.

• We present a parallel attention module (PAM) for discriminative saliency representations with integration of both global and local information.

• We perform extensive experiments on seven widely used challenging datasets, in which the proposed method yields consistent improvements over a number of strong baselines.