Coordinated and specific autoencoder for cross-modal retrieval

Menghan Xu; Bo Sun; Chong Wang; Fangxiang Feng

doi:10.1117/12.2652351

10 November 2022 Coordinated and specific autoencoder for cross-modal retrieval

Menghan Xu, Bo Sun, Chong Wang, Fangxiang Feng

Proceedings Volume 12331, International Conference on Mechanisms and Robotics (ICMAR 2022); 123313M (2022) https://doi.org/10.1117/12.2652351
Event: International Conference on Mechanisms and Robotics (ICMAR 2022), 2022, Zhuhai, China

Abstract

This paper considers the problem of cross-modal retrieval, e.g. using a text query to search for images and vice-versa. Existing approaches usually learn a common subspace where the shared parts of different modalities can be directly compared. However, no previous works explicitly show that the learned space contains only the common information but without the modality-specific information. And the division between these two types of information would benefits the task of cross-modal retrieval. In this paper, we present a COordinated and Specific autoEncoder (a.k.a. COSE) that can distinguish the common part from modality-specific part of different modalities. The proposed model COSE consists of two subnetworks, each with two representation layers. The common representation layer learns the common patterns shared within different modalities. And the modality-specific representation layer learns the modality-specific patterns owned by individual modalities. We evaluate our model on three publicly real-world datasets with the task of cross-modal retrieval. The extensive experiments demonstrate the effectiveness of our COSE.

Citation Download Citation

Menghan Xu, Bo Sun, Chong Wang, and Fangxiang Feng "Coordinated and specific autoencoder for cross-modal retrieval", Proc. SPIE 12331, International Conference on Mechanisms and Robotics (ICMAR 2022), 123313M (10 November 2022); https://doi.org/10.1117/12.2652351

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available