Skip to main content
SpringerLink
Log in

Semantic consistent feature construction and multi-granularity feature learning for visible-infrared person re-identification

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In the real-world 24/7 surveillance systems, the images collected during the day and night are visible light images and infrared images, respectively. Infrared images lack color and texture information. In this case, it is more practical to use cross-modality person re-identification (re-ID) to process visible-infrared images. In fact, the cross-modality semantic alignment and specific discriminative feature extraction of different modalities are important for the improvement of modal performance. Therefore, a Semantic Consistent Feature Construction and Multi-granularity Feature learning (SCC–MGL) method is proposed for visible-infrared person re-ID in this paper. The SCC–MGL consists of a Semantic Consistent Feature Construction (SCC) module and a Multi-Granularity Information Enhancement (MGIE) module. In SCC, the features of different modalities are guided by analyzing the relation between feature maps channels and pedestrian’s body parts to form consistent semantic information on the corresponding channels, which reduces the impact caused by the misalignment of semantic information. In MGIE, a local modality difference elimination strategy is proposed to remove the modality difference. Meanwhile, the local feature discrimination is improved by reasonably constraining multi-granularity features. The effectiveness and superiority of proposed method are validated by experimental results from SYSU-MM01 and RegDB datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability.

Not applicable.

Code availability.

Not applicable.

References

  1. Wang, S., Liu, R., Li, H., Qi, G., Yu, Z.: Occluded person re-identification via defending against attacks from obstacles. IEEE Trans. Inf. Forensics Secur. 18, 147–161 (2023)

    Article  Google Scholar 

  2. Li, H., Chen, Y., Tao, D., Yu, Z., Qi, G.: Attribute-aligned domain-invariant feature learning for unsupervised domain adaptation person re-identification. IEEE Trans. Inf. Forensics Secur. 16, 1480–1494 (2021)

    Article  Google Scholar 

  3. Li, H., Dong, N., Yu, Z., Tao, D., Qi, G.: Triple adversarial learning and multi-view imaginative reasoning for unsupervised domain adaptation person re-identification. IEEE Trans. Circuits Syst. Video Technol. 32(5), 2814–2830 (2022)

    Article  Google Scholar 

  4. Wang, S., Huang, B., Li, H., Qi, G., Tao, D., Yu, Z.: Key point-aware occlusion suppression and semantic alignment for occluded person re-identification. Inf. Sci. 606, 669–687 (2022)

    Article  Google Scholar 

  5. Li, S., Li, F., Wang, K., Qi, G., Li, H.: Mutual prediction learning and mixed viewpoints for unsupervised-domain adaptation person re-identification on blockchain. Simul. Model. Pract. Theory 119, 102568 (2022)

    Article  Google Scholar 

  6. Li, L., Xie, M., Li, F., Zhang, Y., Li, H., Tan, T.: Unsupervised domain adaptive person re-identification guided by low-rank priori(in chinese). J. Chongqing Univ. 44, 57–70 (2021)

    Google Scholar 

  7. Zhang, Y., Wang, Y., Li, H., Li, S.: Cross-compatible embedding and semantic consistent feature construction for sketch re-identification. In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 3347–3355 (2022)

  8. He, S., Luo, H., Wang, P., Wang, F., Li, H., Jiang, W.: Transreid: transformer-based object re-identification. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14993–15002 (2021)

  9. Dai, Y., Liu, J., Sun, Y., Tong, Z., Zhang, C., Duan, L.-Y.: Idm: an intermediate domain module for domain adaptive person re-id. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11844–11854 (2021)

  10. Li, H., Xu, K., Li, J., Yu, Z.: Dual-stream reciprocal disentanglement learning for domain adaptation person re-identification. Knowl. Based Syst. 251, 109315 (2022)

    Article  Google Scholar 

  11. Wang, Y., Qi, G., Li, S., Chai, Y., Li, H.: Body part-level domain alignment for domain-adaptive person re-identification with transformer framework. IEEE Trans. Inf. Forensics Secur. 17, 3321–3334 (2022)

    Article  Google Scholar 

  12. Li, H., Kuang, Z., Yu, Z., Luo, J.: Structure alignment of attributes and visual features for cross-dataset person re-identification. Pattern Recognit. 106, 107414 (2020)

    Article  Google Scholar 

  13. Li, H., Xu, J., Yu, Z., Luo, J.: Jointly learning commonality and specificity dictionaries for person re-identification. IEEE Trans. Image Process. 29, 7345–7358 (2020)

    Article  MathSciNet  Google Scholar 

  14. Li, H., Yan, S., Yu, Z., Tao, D.: Attribute-identity embedding and self-supervised learning for scalable person re-identification. IEEE Trans. Circuits Syst. Video Technol. 30(10), 3472–3485 (2020)

    Article  Google Scholar 

  15. Wang, G., Yuan, Y., Chen, X., Li, J., Zhou, X.: Learning discriminative features with multiple granularities for person re-identification. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 274–282 (2018)

  16. Zhuo, J., Chen, Z., Lai, J., Wang, G.: Occluded person re-identification. In: 2018 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6 (2018)

  17. Wang, G., Wang, G., Zhang, X., Lai, J., Lin, L.: Weakly supervised person re-identification: cost-effective learning with a new benchmark. CoRR, vol. abs/1904.03845 (2019)

  18. Wang, G., Lai, J., Huang, P., Xie, X.: Spatial-temporal person re-identification. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8933–8940 (2019)

  19. Zhu, Z., Luo, Y., Chen, S., Qi, G., Mazur, N., Zhong, C., Li, Q.: Camera style transformation with preserved self-similarity and domain-dissimilarity in unsupervised person re-identification. J. Vis. Commun. Image Represent. 80, 103303 (2021)

    Article  Google Scholar 

  20. Li, Y., Chen, S., Qi, G., Zhu, Z., Haner, M., Cai, R.: A gan-based self-training framework for unsupervised domain adaptive person re-identification. J. Imaging 7(4), 62 (2021)

    Article  Google Scholar 

  21. Xie, J., Ge, Y., Zhang, J., Huang, S., Chen, F., Wang, H.: Low-resolution assisted three-stream network for person re-identification. Vis. Comput. 2515–2525 (2022)

  22. Jia, Z., Li, Y., Tan, Z., Wang, W., Wang, Z., Yin, G.: Domain-invariant feature extraction and fusion for cross-domain person re-identification. Vis. Comput. 1205–1216 (2023)

  23. Zhong, C., Jiang, X., Qi, G.: Video-based person re-identification based on distributed cloud computing. J. Artif. Intell. Technol. 1(2), 110–120 (2021)

    Article  Google Scholar 

  24. Zhong, C., Qi, G., Mazur, N., Banerjee, S., Malaviya, D., Hu, G.: A domain adaptive person re-identification based on dual attention mechanism and camstyle transfer. Algorithms 14(12), 361 (2021)

    Article  Google Scholar 

  25. Liang, W., Wang, G., Lai, J., Xie, X.: Homogeneous-to-heterogeneous: unsupervised learning for rgb-infrared person re-identification. IEEE Trans. Image Process. 30, 6392–6407 (2021)

    Article  MathSciNet  Google Scholar 

  26. Wang, G., Lai, J.-H., Liang, W., Wang, G.: Smoothing adversarial domain attack and p-memory reconsolidation for cross-domain person re-identification. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10565–10574 (2020)

  27. Wen, X., Feng, X., Li, P., Chen, W.: Cross-modality collaborative learning identified pedestrian. Vis. Comput. (2022)

  28. Wang, G., Zhang, T., Cheng, J., Liu, S., Yang, Y., Hou, Z.: Rgb-infrared cross-modality person re-identification via joint pixel and feature alignment. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3623–3632 (2019)

  29. Wang, Z., Wang, Z., Zheng, Y., Chuang, Y.-Y., Satoh, S.: Learning to reduce dual-level discrepancy for infrared-visible person re-identification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), pp. 618–626 (2019)

  30. Wang, G.-A., Zhang, T., Yang, Y., Cheng, J., Chang, J., Liang, X., Hou, Z.-G.: Cross-modality paired-images generation for rgb-infrared person re-identification. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 12144–12151 (2020)

  31. Zhang, Z.Y., Jiang, S., Huang, C.Z.T., Li, Y., Da, X., Yi, R.: Rgb-ir cross-modality person reid based on teacher-student gan model. Pattern Recognit. Lett. 150, 155–161 (2021)

    Article  Google Scholar 

  32. Dai, P.Y., Ji, R.R., Wang, H.B., Wu, Q., Huang, Y.Y.: Cross-modality person re-identification with generative adversarial training. In: International Joint Conference on Artificial Intelligence(IJCAI), p. 6 (2018)

  33. Fan, X., Jiang, W., Luo, H., Mao, W.J.: Modality-transfer generative adversarial network and dual-level unified latent representation for visible thermal person re-identification. Vis. Comput. 1–16 (2020)

  34. Li, K.F., Wang, X.L., Liu, Y., Zhang, B.J., Zhang, M.H.: Cross-modality disentanglement and shared feedback learning for infrared-visible person re-identification. Knowl. Based Syst. 252, 109337 (2022)

    Article  Google Scholar 

  35. Kansal, K., Subramanyam, A.V., Wang, Z., Satoh, S.: Sdl: spectrum-disentangled representation learning for visible-infrared person re-identification. IEEE Trans. Circuits Syst. Video Technol. 30, 3422–3432 (2020)

    Article  Google Scholar 

  36. Choi, S., Lee, S., Kim, Y., Kim, T., Kim, C.: Hi-cmd: hierarchical cross-modality disentanglement for visible-infrared person re-identification. In: The IEEE/CVF Conference on Computer Vision and Pattern recognition(CVPR), pp. 10257–10266 (2020)

  37. Pu, N., Chen, W., Liu, Y., Bakker, E.M., Lew, M.S.: Dual gaussian-based variational subspace disentanglement for visible-infrared person re-identification. In: the 28th ACM International Conference on Multimedia, pp. 2149–2158 (2020)

  38. Zhu, X.K., Zheng, M.H., Chen, X.P., Zhang, X.Y., Yuan, C.H., Zhang, F.: Information disentanglement based cross-modal representation learning for visible-infrared person re-identification. Multimed. Tools Appl. 1–27 (2022)

  39. Zhang, L.Y., Du, G.D., Liu, F., Tu, H.W., Shu, X.B.: Global-local multiple granularity learning for cross-modality visible-infrared person re-identification. IEEE Trans. Neural Netw. Learn. Syst. 1–11 (2021)

  40. Liu, H.J., Tan, X.H., Zhou, X.C.: Parameter sharing exploration and hetero-center triplet loss for visible-thermal person re-identification. IEEE Trans. Multimed. 23, 4414–4425 (2020)

    Article  Google Scholar 

  41. Ling, Y.G., Luo, Z.M., Lin, Y.J., Li, S.Z.: A multi-constraint similarity learning with adaptive weighting for visible-thermal person re-identification. In: International Joint Conference on Artificial Intelligence(IJCAI), pp. 845–851 (2021)

  42. Wang, H.Z., Zhao, J.Q., Zhou, Y., Yao, R., Chen, Y., Chen, S.L.: Amc-net: attentive modality-consistent network for visible-infrared person re-identification. Neurocomputing 463, 226–236 (2021)

    Article  Google Scholar 

  43. Ye, M., Shen, J.B., David, J.C., Shao, L., Luo, J.B.: Dynamic dual-attentive aggregation learning for visible-infrared person re-identification. In: European Conference on Computer Vision(ECCV), pp. 229–247 (2020)

  44. Ye, M., Shen, J.B., Lin, G.J., Xiang, T., Shao, L., Hoi, S.: Deep learning for person re-identification: a survey and outlook. IEEE Trans. Pattern Anal. Mach. Intell. 44, 2872–2893 (2021)

    Article  Google Scholar 

  45. Radenović, F., Tolias, G., Chum, O.: Fine-tuning cnn image retrieval with no human annotation. IEEE Trans. Pattern Anal. Mach. Intell. 41(7), 1655–1668 (2018)

    Article  Google Scholar 

  46. Deep feature learning with relative distance comparison for person re-identification. Pattern Recognit. 48(10), 2993–3003 (2015)

  47. Park, H., Lee, S., Lee, J., Ham, B.: Learning by aligning: visible-infrared person re-identification using cross-modal correspondences. In: 2021 IEEE International Conference on Computer Vision (ICCV), pp. 12026–12035 (2021)

  48. Wei, Z., Yang, X., Wang, N., Gao, X.: Flexible body partition-based adversarial learning for visible infrared person re-identification. IEEE Trans. Neural Netw. Learn. Syst. 33(9), 4676–4687 (2021)

    Article  Google Scholar 

  49. Ding, C.X., Wang, K., Wang, P.F., Tao, D.C.: Multi-task learning with coarse priors for robust part-aware person re-identification. IEEE Trans. Pattern Anal. Mach. Intell. 44, 1474–1488 (2022)

    Article  Google Scholar 

  50. Zhao, C., Lv, X., Dou, S., Zhang, S., Wu, J., Wang, L.: Incremental generative occlusion adversarial suppression network for person reid. IEEE Trans. Image Process. 30, 4212–4224 (2021)

    Article  Google Scholar 

  51. Lin, M., Chen, Q., Yan, S.: Network in network (2013). arXiv preprint arXiv:1312.4400

  52. Liang, W., Wang, G., Lai, J., Zhu, J.-Y.: M2m-gan : many-to-many generative adversarial transfer learning for person re-identification (2018)

  53. Luo, H., Gu, Y., Liao, X., Lai, S., Jiang, W.: Bag of tricks and a strong baseline for deep person re-identification. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1487–1495 (2019)

  54. Sun, Y.F., Zheng, L., Yang, Y., Tian, Q., Wang, S.J.: Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline). In: European Conference on Computer Vision (ECCV), pp. 480–496 (2018)

  55. Wu, A., Zheng, W.-S., Yu, H.-X., Gong, S., Lai, J.: Rgb-infrared cross-modality person re-identification. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5390–5399 (2017)

  56. Wu, A., Zheng, W.-S., Gong, S., Lai, J.: Rgb-ir person re-identification by cross-modality similarity preservation. Int. J. Comput. Vis. 128, 1765–1785 (2021)

    Article  MathSciNet  Google Scholar 

  57. Nguyen, D.T., Hong, H.G., Kim, K.W., Park, K.R.: Person recognition system based on a combination of body images from visible light and thermal cameras. Sensors 17, 605 (2017)

    Article  Google Scholar 

  58. Kang, J.K., Lee, M.B., Yoon, H.S., Park, K.R.: As-rig: adaptive selection of reconstructed input by generator or interpolation for person re-identification in cross-modality visible and thermal images. IEEE Access 9, 12055–12066 (2021)

    Article  Google Scholar 

  59. Sun, Y.F., Zheng, L., Yang, Y., Tian, Q., Wang, S.J.: Visible thermal person re-identification via dual-constrained top-ranking. In: International Joint Conference on Artificial Intelligence(IJCAI), p. 2 (2018)

  60. Ye, M., Lan, X.Y., Wang, Z., Yuen, P.: Bi-directional center-constrained top-ranking for visible thermal person re-identification. Ann. Math. Stat. 15, 407–419 (2019)

    Google Scholar 

  61. Wang, X.G., Doretto, G., Sebastian, T., Rittscher, J., Tu, P.: Shape and appearance context modeling. In: 2007 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1–8 (2007)

  62. Zheng, L., Shen, L.Y., Tian, L., Wang, S.J., Wang, J.D., Tian, Q.: Scalable person re-identification: a benchmark. In: 2015 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1116–1124 (2015)

  63. Ye, M., Ruan, W.J., Du, B., Shou, M.Z.: Channel augmented joint learning for visible-infrared recognition. In: 2021 IEEE/CVF International Conference on Computer Vision, pp. 13567–13576 (2021)

  64. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 400–407 (1951)

  65. Fan, X., Jiang, W., Luo, H., Fei, M.: Spherereid: deep hypersphere manifold embedding for person re-identification. J. Vis. Commun. Image Represent. 60, 51–58 (2019)

    Article  Google Scholar 

  66. Wei, Z., Yang, X., Wang, N., Gao, X.: Rbdf: reciprocal bidirectional framework for visible infrared person reidentification. IEEE Trans. Cybern. 52(10), 10988–10998

  67. Wang, X.J., Chen, C.Q., Zhu, Y., Chen, S.G.: Feature fusion and center aggregation for visible-infrared person re-identification. IEEE Access 10, 30949–30958 (2022)

    Article  Google Scholar 

  68. Huang, Z.P., Liu, J.W., Li, L., Zheng, K.C., Zha, Z.J.: Modality-adaptive mixup and invariant decomposition for rgb-infrared person re-identification. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1034–1042 (2022)

  69. Gao, Y.J., Liang, T.F., Jin, Y., Gu, X.Y., Liu, W., Li, Y.D., Lang, C.Y.: Mso: Multi-feature space joint optimization network for rgb-infrared person re-identification. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 5257–5265 (2021)

  70. Liu, H.J., Chai, Y.X., Tan, X.H., Li, D., Zhou, X.C.: Strong but simple baseline with dual-granularity triplet loss for visible-thermal person re-identification. IEEE Signal Process. Lett. 28, 653–657 (2021)

    Article  Google Scholar 

  71. Wei, Z., Yang, X., Wang, N., Gao, X.: Syncretic modality collaborative learning for visible infrared person re-identification. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 225–234 (2021)

  72. Chen, C., Ye, M., Qi, M., Wu, J., Jiang, J., Lin, C.-W.: Structure-aware positional transformer for visible-infrared person re-identification. IEEE Trans. Image Process. 31, 2352–2364 (2022)

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China No. U2034209.

Author information

Authors and Affiliations

  1. School of Automation, Chongqing University, Chongqing, 400044, China

    Yiming Wang, Kaixiong Xu, Yi Chai & Yutao Jiang

  2. State Key Laboratory of Power Transmission Equipment and System Security and New Technology, Chongqing University, Chongqing, 400044, China

    Yiming Wang, Kaixiong Xu, Yi Chai & Yutao Jiang

  3. Computer Information Systems Department, State University of New York at Buffalo State, Buffalo, NY, 14222, USA

    Guanqiu Qi

Authors
  1. Yiming Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kaixiong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi Chai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yutao Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guanqiu Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization was contributed by YW, KX, YC. Data curation was contributed by YW, KX, YJ. Formal analysis was contributed by YW, KX. Investigation was contributed by YJ, GQ. Methodology was contributed by YW, KX, YC. Validation was contributed by YW, GQ. Writing—original draft preparation, was contributed by Yiming Wang, Yutao Jiang. Writing—review and editing, was contributed by KX, YC, GQ. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Yi Chai.

Ethics declarations

Conflict of interest

The authors have no financial or proprietary interests in any material discussed in this article.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Xu, K., Chai, Y. et al. Semantic consistent feature construction and multi-granularity feature learning for visible-infrared person re-identification. Vis Comput 40, 2363–2379 (2024). https://doi.org/10.1007/s00371-023-02923-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-02923-w

Keywords

Search

Navigation