Skip to main content
SpringerLink
Log in

VEMO: A Versatile Elastic Multi-modal Model for Search-Oriented Multi-task Learning

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14608))

Included in the following conference series:

  • 285 Accesses

Abstract

Cross-modal search is one fundamental task in multi-modal learning, but there is hardly any work that aims to solve multiple cross-modal search tasks at once. In this work, we propose a novel Versatile Elastic Multi-mOdal (VEMO) model for search-oriented multi-task learning. VEMO is versatile because we integrate cross-modal semantic search, named entity recognition, and scene text spotting into a unified framework, where the latter two can be further adapted to entity- and character-based image search tasks. VEMO is also elastic because we can freely assemble sub-modules of our flexible network architecture for corresponding tasks. Moreover, to give more choices on the effect-efficiency trade-off when performing cross-modal semantic search, we place multiple encoder exits. Experimental results show the effectiveness of our VEMO with only 37.6% network parameters compared to those needed for uni-task training. Further evaluations on entity- and character-based image search tasks also validate the superiority of search-oriented multi-task learning.

N. Fei and H. Jiang—Contribute equally.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: a joint video and image encoder for end-to-end retrieval. In: International Conference on Computer Vision (ICCV), pp. 1728–1738 (2021)

    Google Scholar 

  2. Biten, A.F., Gómez, L., Rusiñol, M., Karatzas, D.: Good news, everyone! Context driven entity-aware captioning for news images. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12466–12475 (2019)

    Google Scholar 

  3. Chen, S., Zhang, Y., Yang, Q.: Multi-task learning in natural language processing: an overview. arXiv preprint arXiv:2109.09138 (2021)

  4. Chen, Y.-C., et al.: UNITER: UNiversal Image-TExt Representation learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12375, pp. 104–120. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58577-8_7

    Chapter  Google Scholar 

  5. Ch’ng, C., Chan, C.S., Liu, C.: Total-text: toward orientation robustness in scene text detection. Int. J. Doc. Anal. Recogn. 23(1), 31–52 (2020)

    Article  Google Scholar 

  6. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT), pp. 4171–4186 (2018)

    Google Scholar 

  7. Dosovitskiy, A., et al.: An image is worth 16 \(\times \) 16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR) (2021)

    Google Scholar 

  8. Feng, W., He, W., Yin, F., Zhang, X.Y., Liu, C.L.: TextDragon: an end-to-end framework for arbitrary shaped text spotting. In: International Conference on Computer Vision (ICCV), pp. 9076–9085 (2019)

    Google Scholar 

  9. He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729–9738 (2020)

    Google Scholar 

  10. Huang, Z., Zeng, Z., Liu, B., Fu, D., Fu, J.: Pixel-BERT: aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849 (2020)

  11. Jia, C., et al.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International Conference on Machine Learning (ICML), pp. 4904–4916 (2021)

    Google Scholar 

  12. Karatzas, D., et al.: ICDAR 2015 competition on robust reading. In: International Conference on Document Analysis and Recognition (ICDAR), pp. 1156–1160 (2015)

    Google Scholar 

  13. Kendall, A., Gal, Y., Cipolla, R.: Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7482–7491 (2018)

    Google Scholar 

  14. Kuhn, H.W.: The Hungarian method for the assignment problem. Nav. Res. Logist. Q. 2(1–2), 83–97 (1955)

    Article  MathSciNet  Google Scholar 

  15. Li, J., Li, D., Xiong, C., Hoi, S.C.H.: BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning (ICML), pp. 12888–12900 (2022)

    Google Scholar 

  16. Li, J., Selvaraju, R.R., Gotmare, A., Joty, S.R., Xiong, C., Hoi, S.C.: Align before fuse: vision and language representation learning with momentum distillation. In: Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 9694–9705 (2021)

    Google Scholar 

  17. Li, X., Feng, J., Meng, Y., Han, Q., Wu, F., Li, J.: A unified MRC framework for named entity recognition. In: Annual Meeting of the Association for Computational Linguistics (ACL), pp. 5849–5859 (2020)

    Google Scholar 

  18. Li, X., et al.: Oscar: object-semantics aligned pre-training for vision-language tasks. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12375, pp. 121–137. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58577-8_8

    Chapter  Google Scholar 

  19. Liao, L., He, X., Zhao, B., Ngo, C.W., Chua, T.S.: Interpretable multimodal retrieval for fashion products. In: ACM International Conference on Multimedia (ACM-MM), pp. 1571–1579 (2018)

    Google Scholar 

  20. Lin, T., Goyal, P., Girshick, R.B., He, K., Dollár, P.: Focal loss for dense object detection. In: International Conference on Computer Vision (ICCV), pp. 2999–3007 (2017)

    Google Scholar 

  21. Lin, T.Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  22. Liu, B., Liu, X., Jin, X., Stone, P., Liu, Q.: Conflict-averse gradient descent for multi-task learning. In: Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 18878–18890 (2021)

    Google Scholar 

  23. Liu, S., Johns, E., Davison, A.J.: End-to-end multi-task learning with attention. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1871–1880 (2019)

    Google Scholar 

  24. Liu, Y., Chen, H., Shen, C., He, T., Jin, L., Wang, L.: ABCNet: real-time scene text spotting with Adaptive Bezier-Curve Network. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9806–9815 (2020)

    Google Scholar 

  25. Liu, Y., Jin, L., Zhang, S., Luo, C., Zhang, S.: Curved scene text detection via transverse and longitudinal sequence connection. Pattern Recogn. 90, 337–345 (2019)

    Article  Google Scholar 

  26. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (ICLR) (2019)

    Google Scholar 

  27. Lu, H., Fei, N., Huo, Y., Gao, Y., Lu, Z., Wen, J.: COTS: collaborative two-stream vision-language pre-training model for cross-modal retrieval. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15671–15680 (2022)

    Google Scholar 

  28. Lyu, P., Liao, M., Yao, C., Wu, W., Bai, X.: Mask TextSpotter: an end-to-end trainable neural network for spotting text with arbitrary shapes. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 71–88. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_5

    Chapter  Google Scholar 

  29. Navon, A., ET AL.: Multi-task learning as a bargaining game. In: International Conference on Machine Learning (ICML), pp. 16428–16446 (2022)

    Google Scholar 

  30. Nayak, P.: MUM: a new AI milestone for understanding information. Google Blog (2021). https://www.blog.google/products/search/introducing-MUM/

  31. Nayef, N., et al.: ICDAR2017 robust reading challenge on multi-lingual scene text detection and script identification - RRC-MLT. In: International Conference on Document Analysis and Recognition (ICDAR), pp. 1454–1459 (2017)

    Google Scholar 

  32. Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: collecting region-to-phrase correspondences for richer image-to-sentence models. In: International Conference on Computer Vision (ICCV), pp. 2641–2649 (2015)

    Google Scholar 

  33. Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning (ICML), pp. 8748–8763 (2021)

    Google Scholar 

  34. Rafailidis, D., Manolopoulou, S., Daras, P.: A unified framework for multimodal retrieval. Pattern Recogn. 46(12), 3358–3370 (2013)

    Article  Google Scholar 

  35. Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 140:1–140:67 (2020)

    Google Scholar 

  36. Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I.D., Savarese, S.: Generalized intersection over union: a metric and a loss for bounding box regression. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 658–666 (2019)

    Google Scholar 

  37. Sang, E.F.T.K., Meulder, F.D.: Introduction to the CoNLL-2003 shared task: language-independent named entity recognition. In: Conference on Natural Language Learning (CoNLL), pp. 142–147 (2003)

    Google Scholar 

  38. Sun, S., Chen, Y.C., Li, L., Wang, S., Fang, Y., Liu, J.: LightningDOT: pre-training visual-semantic embeddings for real-time image-text retrieval. In: Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT), pp. 982–997 (2021)

    Google Scholar 

  39. Tay, Y., Zhao, Z., Bahri, D., Metzler, D., Juan, D.: HyperGrid transformers: towards a single model for multiple tasks. In: International Conference on Learning Representations (ICLR) (2021)

    Google Scholar 

  40. Vandenhende, S., Georgoulis, S., Van Gansbeke, W., Proesmans, M., Dai, D., Van Gool, L.: Multi-task learning for dense prediction tasks: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 44(7), 3614–3633 (2022)

    Google Scholar 

  41. Wang, M., Li, H., Tao, D., Lu, K., Wu, X.: Multimodal graph-based reranking for web image search. IEEE Trans. Image Process. 21(11), 4649–4661 (2012)

    Article  MathSciNet  Google Scholar 

  42. Wang, P., et al.: PGNET: real-time arbitrarily-shaped text spotting with point gathering network. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 2782–2790 (2021)

    Google Scholar 

  43. Wang, W., Bao, H., Dong, L., Wei, F.: VLMo: unified vision-language pre-training with mixture-of-modality-experts. arXiv preprint arXiv:2111.02358 (2021)

  44. Wang, X., et al.: Improving named entity recognition by external context retrieving and cooperative learning. In: Joint Conference of Annual Meeting of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL-IJCNLP), pp. 1800–1812 (2021)

    Google Scholar 

  45. Wen, K., Xia, J., Huang, Y., Li, L., Xu, J., Shao, J.: COOKIE: contrastive cross-modal knowledge sharing pre-training for vision-language representation. In: International Conference on Computer Vision (ICCV), pp. 2208–2217 (2021)

    Google Scholar 

  46. Yamada, I., Asai, A., Shindo, H., Takeda, H., Matsumoto, Y.: LUKE: deep contextualized entity representations with entity-aware self-attention. In: Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6442–6454 (2020)

    Google Scholar 

  47. Zeng, B.: Go beyond the search box: Introducing multisearch. Google Blog (2022). https://blog.google/products/search/multisearch/

  48. Zhang, P., et al.: VinVL: revisiting visual representations in vision-language models. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5579–5588 (2021)

    Google Scholar 

  49. Zhang, X., Su, Y., Tripathi, S., Tu, Z.: Text spotting transformers. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9509–9518 (2022)

    Google Scholar 

  50. Zhang, Y., Yang, Q.: A survey on multi-task learning. IEEE Trans. Knowl. Data Eng. 34(12), 5586–5609 (2022)

    Article  Google Scholar 

  51. Zhu, E., Li, J.: Boundary smoothing for named entity recognition. In: Annual Meeting of the Association for Computational Linguistics (ACL), pp. 7096–7108 (2022)

    Google Scholar 

  52. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable DETR: deformable transformers for end-to-end object detection. In: International Conference on Learning Representations (ICLR) (2021)

    Google Scholar 

Download references

Acknowledgement

This work was supported by National Natural Science Foundation of China (62376274).

Author information

Authors and Affiliations

  1. School of Information, Renmin University of China, Beijing, China

    Nanyi Fei

  2. Huawei Poisson Lab, Hangzhou, Zhejiang, China

    Hao Jiang, Tuo Fan & Zhao Cao

  3. Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China

    Haoyu Lu, Jinqiang Long, Yanqi Dai & Zhiwu Lu

Authors
  1. Nanyi Fei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hao Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haoyu Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jinqiang Long
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yanqi Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tuo Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Zhao Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Zhiwu Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhiwu Lu .

Editor information

Editors and Affiliations

  1. Georgetown University, Washington, WA, USA

    Nazli Goharian

  2. University of Pisa, Pisa, Italy

    Nicola Tonellotto

  3. King's College London, London, UK

    Yulan He

  4. University College London, London, UK

    Aldo Lipani

  5. University of Glasgow, Glasgow, UK

    Graham McDonald

  6. University of Glasgow, Glasgow, UK

    Craig Macdonald

  7. University of Glasgow, Glasgow, UK

    Iadh Ounis

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fei, N. et al. (2024). VEMO: A Versatile Elastic Multi-modal Model for Search-Oriented Multi-task Learning. In: Goharian, N., et al. Advances in Information Retrieval. ECIR 2024. Lecture Notes in Computer Science, vol 14608. Springer, Cham. https://doi.org/10.1007/978-3-031-56027-9_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56027-9_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56026-2

  • Online ISBN: 978-3-031-56027-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation