Affiliations: [a] School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui, China
| [b] School of Electronic Engineering and Information Science, University of Science and Technology of China
| [c] Insight Centre for Data Analytics, University of Galway, Galway, Ireland
| [d] Faculty of Engineering, IBB University, IBB, Yemen
| [e] School of Computer Science and Engineering, Centeral South University, Changsha, China
| [f] School of Computer Science, Shenyang Aerospace University, Shenyang, China
Abstract: Visual attention has emerged as a prominent approach for improving the effectiveness of image captioning, as it enables the decoder network to focus selectively on the most salient regions in the image content, thereby facilitating the generation of precise and informative captions. Although visual attention achieves the improvement, the small numerical values of its input have a negative impact on its softmax, decreasing its effectiveness. To address this limitation, we propose a refined visual attention (RVA) framework that internally reweights visual attention by leveraging the language context of previously generated words. We first feed the language context into a fully connected layer to obtain appropriate dimensions for the visual features. Then, we use a sigmoid function to obtain a probability distribution to reweight the softmax’s input by applying the multiplication process. Experiments conducted on the MS COCO dataset demonstrate that RVA outperforms traditional visual attention and other existing image captioning methods, highlighting its effectiveness in enhancing the accuracy and informativeness of image captions.
