Skip to main content
SpringerLink
Log in

Semantic and geometric information propagation for oriented object detection in aerial images

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Unlike the natural scenes, aerial objects are often in arbitrary orientations and surrounded by cluttered backgrounds, causing the heterogeneous object features contaminated with backgrounds and interleaved clustered objects. Though segmentation denoising has been proposed to enhance arbitrary-oriented object detection in previous studies, the typical semantic segmentation brings ambiguous background information and ignores the significant geometric information propagation. In this paper, we explore the non-monotonic impacts of segmentation on object detection. Based on this, we propose a novel aerial object detector named PCG-Net to enhance the semantic and geometric information, as well as, alleviate the noise interference through segmentation guidance. PCG-Net includes a pseudo-siamese multi-scale relation module (PSM), a complementary segmentation map (CSM) and a global context module with adaptive distinction(GCMA). PSM captures the critical geometric and semantic features through parallel horizontal and oriented box-wise segmentation branches with interaction. As the guidance of each branch in PSM, CSM synergizes binary segmentation map and foreground semantic segmentation map so as to refine the semantic information and restrain the ambiguous background simultaneously. Based on the segmentation features from PSM, GCMA rebuilds and complements the global contextual information with pixel-level discrimination. Through exploiting the rich semantic, geometric and contextual information, the proposed framework enhances the feature representation powerfully in aerial images and suppresses noise interference effectively. Experiment results on three challenging aerial datasets including DOTA, HRSC2016 and DIOR-R, demonstrate the effectiveness of our approach.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availibility Statement

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Chen H, Zhang X, Wang L, Xing L, Pedrycz W (2022) Resourceconstrained self-organized optimization for near-real-time offloading satellite earth observation big data. Knowl-Based Syst 253:109496

    Google Scholar 

  2. Ding J, Xue N, Xia G-S, Bai X, Yang W, Yang MY, Belongie S, Luo J, Datcu M, Pelillo M et al (2021) Object detection in aerial images: A large-scale benchmark and challenges. IEEE Trans Pattern Anal Mach Intell 44(11):7778–7796

    Google Scholar 

  3. Li K, Wan G, Cheng G, Meng L, Han J (2020) Object detection in optical remote sensing images: A survey and a new benchmark. ISPRS J Photogramm Remote Sens 159:296–307

    ADS  Google Scholar 

  4. Han J, Ding J, Xue N, Xia G–S (2021) Redet: A rotation-equivariant detector for aerial object detection. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 2786–2795

  5. Yang X, Yang J, Yan J, Zhang Y, Zhang T, Guo Z, Sun X, Fu K (2019) Scrdet: Towards more robust detection for small, cluttered and rotated objects. In: Proceedings of the IEEE/CVF International conference on computer vision, pp 8232–8241

  6. Wu X, Hong D, Chanussot J, Xu Y, Tao R, Wang Y (2019) Fourier-based rotation-invariant feature boosting: An efficient framework for geospatial object detection. IEEE Geosci Remote Sens Lett 17(2):302–306

    ADS  Google Scholar 

  7. Shi Q, Zhu Y, Fang C, Wang N, Lin J (2022) Raod: refined oriented detector with augmented feature in remote sensing images object detection. Appl Intell 52(13):15278–15294

    Google Scholar 

  8. Wen G, Cao P, Wang H, Chen H, Liu X, Xu J, Zaiane O (2022) Msssd: multi–scale single shot detector for ship detection in remote sensing images. Appl Intell 1–19

  9. Li Y, Ouyang S, Zhang Y (2022) Combining deep learning and ontology reasoning for remote sensing image semantic segmentation. Knowl- Based Syst 243:108469

    Google Scholar 

  10. Ma L, Luo X, Hong H, Zhang Y,Wang L,Wu J (2022) Scribble-attention hierarchical network for weakly supervised salient object detection in optical remote sensing images. Appl Intell 1–19

  11. Wang J, Yang W, Li H-C, Zhang H, Xia G-S (2020) Learning center probability map for detecting objects in aerial images. IEEE Trans Geosci Remote Sens 59(5):4307–4323

    ADS  Google Scholar 

  12. Zhang C, Lam K-M, Wang Q (2023) Cof-net: A progressive coarse-tofine framework for object detection in remote-sensing imagery. IEEE Trans Geosci Remote Sens 61:1–17

    Google Scholar 

  13. Ming Q, Miao L, Zhou Z, Dong Y (2021) Cfc-net: A critical feature capturing network for arbitrary-oriented object detection in remote-sensing images. IEEE Trans Geosci Remote Sens 60:1–14

    Google Scholar 

  14. Zhou Y, Wang S, Zhao J, Zhu H, Yao R (2022) Fine-grained feature enhancement for object detection in remote sensing images. IEEE Geosci Remote Sens Lett 19:1–5

    Google Scholar 

  15. Wang P, Sun X, Diao W, Fu K (2019) Fmssd: Feature-merged single-shot detection for multiscale objects in large-scale remote sensing imagery. IEEE Trans Geosci Remote Sens 58(5):3377–3390

    ADS  Google Scholar 

  16. Dong X, Qin Y, Fu R, Gao Y, Liu S, Ye Y, Li B (2022) Multiscale deformable attention and multilevel features aggregation for remote sensing object detection. IEEE Geosci Remote Sens Lett 19:1–5

    Google Scholar 

  17. Gao T, Niu Q, Zhang J, Chen T, Mei S, Jubair A (2023) Global to Local: A Scale-Aware Network for Remote Sensing Object Detection. IEEE Trans Geosci Remote Sens 61:1–14

    Google Scholar 

  18. Wang J, Ding J, Guo H, Cheng W, Pan T, Yang W (2019) Mask obb: A semantic attention-based mask oriented bounding box representation for multi-category object detection in aerial images. Remote Sens 11(24):2930

    ADS  Google Scholar 

  19. Yang F, Li W, Hu H, Li W, Wang P (2020) Multi-scale feature integrated attention-based rotation network for object detection in vhr aerial images. Sensors 20(6):1686

    ADS  PubMed  PubMed Central  Google Scholar 

  20. Liu S, Zhang L, Lu H, He Y (2021) Center-boundary dual attention for oriented object detection in remote sensing images. IEEE Trans Geosci Remote Sens 60:1–14

    CAS  Google Scholar 

  21. Xu C, Li C, Cui Z, Zhang T, Yang J (2020) Hierarchical semantic propagation for object detection in remote sensing imagery. IEEE Trans Geosci Remote Sens 58(6):4353–4364

    ADS  Google Scholar 

  22. Yang X, Yan J, Liao W, Yang X, Tang J, He T (2022) Scrdet++: Detecting small, cluttered and rotated objects via instance-level feature denoising and rotation loss smoothing. IEEE Trans Pattern Anal Mach Intell 45(2):2384–2399

    Google Scholar 

  23. He Z, Ren Z, Yang X, Yang Y, Zhang W (2022) Mead: a maskguided anchor-free detector for oriented aerial object detection. Applied Intell 52:4382–4397

    Google Scholar 

  24. Wu Y, Zhang K, Wang J, Wang Y, Wang Q, Li Q (2022) Cdd-net: A context-driven detection network for multiclass object detection. IEEE Geosci Remote Sens Lett 19:1–5

    Google Scholar 

  25. Kim H-K, Yoo K-Y, Park JH, Jung H-Y (2019) Traffic light recognition based on binary semantic segmentation network. Sensors 19(7):1700

    ADS  PubMed  PubMed Central  Google Scholar 

  26. Hao S, Zhou Y, Guo Y (2022) A brief survey on semantic segmentation with deep learning. Neurocomputing 18:83–113

    Google Scholar 

  27. Yang X, Yan J, Feng Z, He T (2021) R3det: Refined single-stage detector with feature refinement for rotating object. Proceedings of the AAAI Conference on Artificial Intelligence 35:3163–3171

  28. Cheng G, Wang J, Li K, Xie X, Lang C, Yao Y, Han J (2022) Anchorfree oriented proposal generator for object detection. IEEE Trans Geosci Remote Sens 60:1–11

    Google Scholar 

  29. Qian W, Yang X, Peng S, Yan J, Guo Y (2021) Learning modulated loss for rotated object detection. Proceedings of the AAAI Conference on Artificial Intelligence 35:2458–2466

    Google Scholar 

  30. Xu Y, Fu M, Wang Q, Wang Y, Chen K, Xia G-S, Bai X (2020) Gliding vertex on the horizontal bounding box for multi-oriented object detection. IEEE Trans Pattern Anal Mach Intell 43(4):1452–1459

    Google Scholar 

  31. Qian W, Yang X, Peng S, Zhang X, Yan J (2022) Rsdet++: Point-based modulated loss for more accurate rotated object detection. IEEE Trans Circ Syst Video Technol 32(11):7869–7879

    Google Scholar 

  32. Cheng G, Yuan X, Yao X, Yan K, Zeng Q, Xie X, Han J (2023) Towards large-scale small object detection: Survey and benchmarks. IEEE Trans Pattern Anal Mach Intell 45(11):13467–13488

    PubMed  Google Scholar 

  33. Ding J, Xue N, Long Y, Xia G–S, Lu Q (2019) Learning roi transformer for oriented object detection in aerial images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2849–2858

  34. Han J, Ding J, Li J, Xia G-S (2021) Align deep features for oriented object detection. IEEE Trans Geosci Remote Sens 60:1–11

    Google Scholar 

  35. Yang X, Yan J (2020) Arbitrary–oriented object detection with circular smooth label. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII 16, pp 677–694

  36. Yang X, Hou L, Zhou Y, Wang W, Yan J (2021) Dense label encoding for boundary discontinuity free rotation detection. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 15819–15829

  37. Zhang Y, Guo W, Wu C, Li W, Tao R (2023) FANet: an arbitrary direction remote sensing object detection network based on feature fusion and angle classification. IEEE Transactions on Geoscience and Remote Sensing 61:1–11

    Google Scholar 

  38. Li W, Chen Y, Hu K, Zhu J (2022) Oriented reppoints for aerial object detection. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 1829–1838

  39. Zhao P, Qu Z, Bu Y, Tan W, Guan Q (2021) Polardet: A fast, more precise detector for rotated target in aerial images. Int J Remote Sens 42(15):5831–5861

    Google Scholar 

  40. Xie X, Cheng G, Wang J, Yao X, Han J (2021) Oriented r–cnn for object detection. In: Proceedings of the IEEE/CVF International conference on computer vision, pp. 3520–3529

  41. Zhang G, Lu S, Zhang W (2019) Cad-net: A context-aware detection network for objects in remote sensing imagery. IEEE Trans Geosci Remote Sens 57(12):10015–10024

    ADS  Google Scholar 

  42. Yu Y, Yang X, Li J, Gao X (2022) Object detection for aerial images with feature enhancement and soft label assignment. IEEE Trans Geosci Remote Sens 60:1–16

    Google Scholar 

  43. Zhang T, Zhang X, Zhu P, Chen P, Tang X, Li C, Jiao L (2021) Foreground refinement network for rotated object detection in remote sensing images. IEEE Trans Geosci Remote Sens 60:1–13

    Google Scholar 

  44. Zhang J, Xie C, Xu X, Shi Z, Pan B (2020) A contextual bidirectional enhancement method for remote sensing image object detection. IEEE J Sel Top Appl Earth Obs Remote Sens 13:4518–4531

    ADS  Google Scholar 

  45. Feng X, Han J, Yao X, Cheng G (2020) Tcanet: Triple context-aware network for weakly supervised object detection in remote sensing images. IEEE Trans Geosci Remote Sens 59(8):6946–6955

    ADS  Google Scholar 

  46. Chen X, Wang C, Li Z, Liu M, Li Q, Qi H, Ma D, Li Z, Wang Y (2023) Coupled global-local object detection for large vhr aerial images. Knowl-Based Syst 260:110097

    Google Scholar 

  47. Xiao J, Yao Y, Zhou J, Guo H, Yu Q, Wang Y-F (2023) FDLR-Net: A feature decoupling and localization refinement network for object detection in remote sensing images. Expert Syst Appl 225:120068

    Google Scholar 

  48. Cui Z, Sun H-M, Yin R-N, Jia R-S (2022) SDA-Net: a detector for small, densely distributed, and arbitrary-directional ships in remote sensing images. Appl Intell 52(11):12516–12532

    Google Scholar 

  49. Wang J, Yu J, He Z (2022) ARFP: A novel adaptive recursive feature pyramid for object detection in aerial images. Appl Intell 52(11):12844–12859

    Google Scholar 

  50. Xiong S, Wu X, Chen H, Qing L, Chen T, He X (2021) Bi-directional skip connection feature pyramid network and sub-pixel convolution for high-quality object detection. Neurocomputing 440:185–196

    Google Scholar 

  51. Zhong Y, Cheng X, Chen T, Zhang J, Zhou Z, Huang G (2022) Prpn: Progressive region prediction network for natural scene text detection. Knowl-Based Syst 236:107767

    Google Scholar 

  52. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778

  53. Lin T–Y, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 2117– 2125

  54. Cao, Y., Xu, J., Lin, S., Wei, F., Hu, H. (2019) Gcnet: Non–local networks meet squeeze–excitation networks and beyond. In: Proceedings of the IEEE/CVF International Conference on Computer VisionWorkshops, pp 1971–1980

  55. Xia G–S, Bai X, Ding J, Zhu Z, Belongie S, Luo J, Datcu M, Pelillo M, Zhang L (2018) Dota: A large–scale dataset for object detection in aerial images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3974–3983

  56. Liu Z, Yuan L, Weng L, Yang Y (2017) A high resolution optical satellite image dataset for ship recognition and some new baselines. In: Proceedings of the 6th International conference in pattern recognition applications and methods, pp 324–331

  57. Li K, Wan G, Cheng G, Meng L, Han J (2020) Object detection in optical remote sensing images: A survey and a new benchmark. ISPRS J Photogramm Remote Sens 159:296–307

    ADS  Google Scholar 

  58. Ren S, He K, Girshick R, Sun J (2017) Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39:1137–1149

    PubMed  Google Scholar 

  59. Hou L, Lu K, Xue J, Li Y (2022) Shape-adaptive selection and measurement for oriented object detection. Proceedings of the AAAI Conference on Artificial Intelligence 36:923–932

    Google Scholar 

  60. Lin T–Y, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2980–2988

  61. Gong L, Huang X, Chao Y, Chen J, Lei B (2023) An enhanced SSD with feature cross-reinforcement for small-object detection. Appl Intell 53:19449–19465

    Google Scholar 

  62. Zhou J, Feng K, Li W, Han J, Pan F (2022) Ts4net: Two-stage sample selective strategy for rotating object detection. Neurocomputing 501:753–764

    Google Scholar 

  63. Li C, Xu C, Cui Z,Wang D, Zhang T, Yang J (2019) Feature–attentioned object detection in remote sensing imagery. In: IEEE International Conference on Image Processing, pp. 3886–3890. IEEE

  64. Qian X, Wu B, Cheng G, Yao X, Wang W, Han J (2023) Building a bridge of bounding box regression between oriented and horizontal object detection in remote sensing images. IEEE Trans Geosci Remote Sens 61:1–9

    Google Scholar 

  65. Ye T, Qin W, Li Y, Wang S, Zhang J, Zhao Z (2022) Dense and small object detection in UAV-vision based on a global-local feature enhanced network. IEEE Trans Instrum Meas 71:1–13

    Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China [grant numbers 62271336, 62211530110]; Natural Science Foundation of Sichuan Province [grant number 2022NSFSC0922]; and Research Fund of Guangxi Key Laboratory of Machine Vision and Intelligent Control [grant number 2022B05].

Author information

Authors and Affiliations

  1. College of Electronics and Information Engineering, Sichuan University, Chengdu, 610064, China

    Tong Niu, Xiaohai He, Honggang Chen, Linbo Qing & Qizhi Teng

  2. Guangxi Key Laboratory of Machine Vision and Intelligent Control, WuZhou University, Wuzhou, 543002, China

    Tong Niu & Honggang Chen

Authors
  1. Tong Niu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaohai He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Honggang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Linbo Qing
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qizhi Teng
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Tong Niu: Conceptualization, Methodology, Software, Writing-Original draft preparation. Xiaohai He: Conceptualization, Supervision, Writing- Reviewing and Editing. Honggang Chen: Conceptualization, Supervision, Writing- Reviewing and Editing. Linbo Qing: Supervision, Writing- Reviewing and Editing. Qizhi Teng: Conceptualization, Supervision, Writing- Reviewing and Editing.

Corresponding author

Correspondence to Qizhi Teng.

Ethics declarations

The authors of the submitted manuscript declare that does not involve any ethical issues.

Competing Interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Niu, T., He, X., Chen, H. et al. Semantic and geometric information propagation for oriented object detection in aerial images. Appl Intell 54, 2154–2171 (2024). https://doi.org/10.1007/s10489-023-05227-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05227-7

Keywords

Search

Navigation