Skip to main content
SpringerLink
Log in

PnP-UGCSuperGlue: deep learning drone image matching algorithm for visual localization

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

In response to the significant positioning errors that arise in visual localization algorithms for unmanned aerial vehicles (UAVs) when relying on drone image matching in areas devoid of satellite signals, we propose a deep learning-based algorithm named PnP-UGCSuperGlue. This algorithm employs a convolutional neural network (CNN) that is enhanced with a graph encoding module. The resulting enriched features contain vital information that refines the feature map and improves the overall accuracy of the visual localization process. The PnP-UGCSuperGlue framework initiates with the semantic feature extraction from both the real-time drone image and the geo-referenced image. This extraction process is facilitated by a CNN-based feature extractor. In the subsequent phase, a graph encoding module is integrated to aggregate the extracted features. This integration significantly enhances the quality of the generated feature keypoints and descriptors. Following this, a graph matching network is applied to leverage the generated descriptors, thereby facilitating a more precise feature point matching and filtering process. Ultimately, the perspective-n-point (PnP) method is utilized to calculate the rotation matrix and translation vector. This calculation is based on the results of the feature matching phase, as well as the camera intrinsic parameters and distortion coefficients. The proposed algorithm’s efficacy is validated through experimental evaluation, which demonstrates a mean absolute error of 0.0005 during the drone’s hovering state and 0.0083 during movement. These values indicate a significant reduction of 0.0010 and 0.0028, respectively, compared to the USuperGlue network.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Availability of data and access

The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.

References

  1. Shendryk Y, Sofonia J, Garrard R, Rist Y, Skocaj D, Thorburn P (2020) Fine-scale prediction of biomass and leaf nitrogen content in sugarcane using UAV LiDAR and multispectral imaging. Int J Appl Earth Observ Geoinf 92:102177. https://doi.org/10.1016/j.jag.2020.102177

    Article  Google Scholar 

  2. Budiyono A, Higashino SI (2023) A review of the latest innovations in UAV technology. J Instrument Autom Syst 10(1):7–16. https://doi.org/10.5281/zenodo.8062292

    Article  Google Scholar 

  3. ZHANG J (2021) Review of the light-weighted and small UAV system for aerial photography and remote sensing. National Remote Sens Bull 25(3):708-724. https://doi.org/10.11834/jrs.20210052

  4. Gupta A, Fernando X (2022) Simultaneous localization and mapping (slam) and data fusion in unmanned aerial vehicles: recent advances and challenges. Drones 6(4):85. https://doi.org/10.3390/drones6040085

    Article  Google Scholar 

  5. Zhou T, Hasheminasab SM, Habib A (2021) Tightly-coupled camera/LiDAR integration for point cloud generation from GNSS/INS-assisted UAV mapping systems. ISPRS J Photogram Remote Sens 180:336–356. https://doi.org/10.1016/j.isprsjprs.2021.08.020

    Article  Google Scholar 

  6. Asadzadeh S, de Oliveira WJ, de Souza Filho CR (2022) UAV-based remote sensing for the petroleum industry and environmental monitoring: state-of-the-art and perspectives. J Petrol Sci Eng 208:109633. https://doi.org/10.1016/j.petrol.2021.109633

    Article  Google Scholar 

  7. Pan M, Chen C, Yin X, Huang Z (2021) UAV-aided emergency environmental monitoring in infrastructure-less areas: LoRa mesh networking approach. IEEE Internet Things J 9(4):2918–2932. https://doi.org/10.1016/10.1109/JIOT.2021.3095494

    Article  Google Scholar 

  8. Dong J, Ren X, Han S, Luo S (2022) UAV vision aided INS/odometer integration for land vehicle autonomous navigation. IEEE Trans Veh Technol 71(5):4825–4840. https://doi.org/10.1109/TVT.2022.3151729

    Article  Google Scholar 

  9. Chaturvedi SK, Sekhar R, Banerjee S, Kamal H (2019) Comparative review study of military and civilian unmanned aerial vehicles (UAVs). INCAS Bull 11(3):183-198. https://doi.org/10.13111/2066-8201.2019.11.3.16

  10. Deng L, He Y, Liu Q (2019) Research on application of fire uumanned aerial vehicles in emergency rescue. 2019 9th International Conference on Fire Science and Fire Protection Engineering (ICFSFPE), pp 1–5. https://doi.org/10.1109/ICFSFPE48751.2019.9055875

  11. Lippiello V, Cacace J (2021) Robust visual localization of a uav over a pipe-rack based on the lie group se (3). IEEE Robot Autom Lett 7(1):295–302. https://doi.org/10.1109/LRA.2021.3125039

    Article  Google Scholar 

  12. Guo Y, Zhou Y, Yang F (2023) AGCosPlace: a UAV visual positioning algorithm based on transformer. Drones 7(8):498. https://doi.org/10.3390/drones7080498

    Article  Google Scholar 

  13. Jiang X, Ma J, Xiao G, Shao Z, Guo X (2021) A review of multimodal image matching: methods and applications. Inf Fusion 73:22–71. https://doi.org/10.1016/j.inffus.2021.02.012

    Article  Google Scholar 

  14. Ma J, Jiang X, Fan A, Jiang J, Yan J (2021) Image matching from handcrafted to deep features: a survey. Int J Comput Vis 129:23–79. https://doi.org/10.1007/s11263-020-01359-2

    Article  MathSciNet  Google Scholar 

  15. Luo C, Yang W, Huang P, Zhou J (2019) Overview of image matching based on ORB algorithm. Int J Phys Conf Ser 1237(3):032020. https://doi.org/10.1088/1742-6596/1237/3/032020

    Article  Google Scholar 

  16. Wang S, Guo Z, Liu Y (2021) An image matching method based on sift feature extraction and FLANN search algorithm improvement. J Phys Conf Ser 2037(1):012122. https://doi.org/10.1088/1742-6596/2037/1/012122

    Article  Google Scholar 

  17. Zeng Q, Adu J, Liu J, Yang J, Xu Y, Gong M (2020) Real-time adaptive visible and infrared image registration based on morphological gradient and C_SIFT. J Real Time Image Process 17:1103–1115

    Article  Google Scholar 

  18. Sedaghat A, Ebadi H (2015) Remote sensing image matching based on adaptive binning SIFT descriptor. IEEE Trans Geosci Remote Sens 53(10):5283–5293. https://doi.org/10.1109/TGRS.2015.2420659

    Article  Google Scholar 

  19. Jiang Z, Liu X, Wang Q (2019) Visible and infrared image registration algorithm based on saliency and ORB. Laser Infrared 49(02):251–256

    Google Scholar 

  20. Ye Y, Bruzzone L, Shan J, Bovolo F, Zhu Q (2019) Fast and robust matching for multimodal remote sensing image registration. IEEE Trans Geosci Remote Sens 57(11):9059–9070. https://doi.org/10.1109/TGRS.2019.2924684

    Article  Google Scholar 

  21. Zeng C, Wang J, Shi P (2013) A stereo image matching method to improve the DSM accuracy inside building boundaries. Can J Remote Sens 39(4):308–317. https://doi.org/10.5589/m13-039

    Article  Google Scholar 

  22. Cao SY, Shen HL, Chen SJ, Li C (2020) Boosting structure consistency for multispectral and multimodal image registration. IEEE Trans Image Process 29:5147–5162.

    Article  Google Scholar 

  23. Uss ML, Vozel B, Lukin VV, Chehdi K (2016) Multimodal remote sensing image registration with accuracy estimation at local and global scales. IEEE Trans Geosci Remote Sens 54(11):6587–6605.

    Article  Google Scholar 

  24. DeTone D, Malisiewicz T, Rabinovich A (2018) Superpoint: Self-supervised interest point detection and description. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 224–236. https://doi.org/10.48550/arXiv.1712.07629

  25. Sarlin PE, DeTone D, Malisiewicz T, Rabinovich A (2020) Superglue: Learning feature matching with graph neural networks. In: Proceedings of the IEEE/CVF Conference on Computer vision and Pattern Recognition, pp 4938–4947.

  26. Lan C, Lu W, Yu J, Xu Q (2021) Deep Learning algorithm for feature matching of cross modality remote sensing images. Acta Geodaetica et Cartographica Sinica 50(2):189-202.

  27. Guo Y, Zhou Y, Yang F (2023) UAV scale enhanced cross-modality graph matching net-USCMGM-net. Multimedia Tools Appl, pp 1–20. https://doi.org/10.1007/s11042-023-16103-4

  28. Zhou Y, Guo Y, Lin KP, Yang F, Li L (2023) USuperGlue: an unsupervised UAV image matching network based on local self-attention. Soft Comput, pp 1–21.

  29. Zheng Z, Wei Y, Yang Y (2020) University-1652: A multi-view multi-source benchmark for drone-based geo-localization. In Proceedings of the 28th ACM International Conference on Multimedia, pp 1395–1403. https://doi.org/10.1145/3394171.3413896

  30. Wang Z, Liu S, Chen G, Dong W (2021) Robust visual positioning of the UAV for the under bridge inspection with a ground guided vehicle. IEEE Trans Instrument Measure 71:1–10. https://doi.org/10.1109/TIM.2021.3135544

    Article  Google Scholar 

  31. Sun J, Shen Z, Wang Y, Bao H, Zhou X (2021) LoFTR: Detector-free local feature matching with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 8922-8931. https://doi.org/https://doi.org/10.48550/arXiv.2104.00680

  32. Zhuang S, Zhao Z, Cao L, Wang D, Fu C, Du K (2023) A robust and fast method to the perspective-n-point problem for camera pose estimation. IEEE Sens J 23(11):11892–11906.

    Article  Google Scholar 

  33. Charroud A, El Moutaouakil K, Palade V, Yahyaouy A (2024) Enhanced autoencoder-based lidar localization in self-driving vehicles. Appl Soft Comput 152:111225. https://doi.org/10.1016/j.asoc.2023.111225

    Article  Google Scholar 

  34. Charroud A, Yahyaouy A, El Moutaouakil K, Onyekpe U (2022) Localisation and mapping of self-driving vehicles based on fuzzy K-means clustering: a non-semantic approach. In: 2022 International Conference on Intelligent Systems and Computer Vision (ISCV). IEEE, pp 1–8.

  35. Dai M, Hu J, Zhuang J, Zheng E (2021) A transformer-based feature segmentation and region alignment method for uav-view geo-localization. IEEE Trans Circuits Syst Video Technol 32(7):4376–4389. https://doi.org/10.1109/TCSVT.2021.3135013

    Article  Google Scholar 

  36. Dai M, Zheng E, Feng Z, Qi L, Zhuang J, Yang W (2023) Vision-based UAV self-positioning in low-altitude urban environments. IEEE Trans Image Process, pp 1–13. https://doi.org/10.1109/TIP.2023.3346279

  37. Berton G, Masone C, Caputo B (2022) Rethinking visual geo-localization for large-scale applications. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 4878–4888. https://doi.org/10.48550/arXiv.2204.02287

  38. Couturier A, Akhloufi MA (2021) A review on absolute visual localization for UAV. Robot Autonom Syst 135:103666. https://doi.org/10.1016/j.robot.2020.103666

    Article  Google Scholar 

  39. Wang Z, Shi D, Qiu C, Jin S, Li T, Shi Y, Liu Z, Qiao Z (2024) Sequence matching for Image-Based UAV-to-Satellite Geolocalization. IEEE Trans Geosci Remote Sens 62:1–15. https://doi.org/10.1109/TGRS.2024.3359605

    Article  Google Scholar 

  40. Pan S, Wang X (2021) A survey on perspective-n-point problem. In: 2021 40th Chinese Control Conference (CCC). IEEE, pp 2396–2401. https://doi.org/10.23919/CCC52363.2021.9549863

Download references

Acknowledgements

This work was supported by the Special Foundation for Beijing Tianjin Hebei Basic Research Cooperation (J210008, H2021202008), and the Inner Mongolia Discipline Inspection and Supervision Big Data Laboratory (IMDBD202105).

Author information

Authors and Affiliations

  1. School of Electronic and Information Engineering, Hebei University of Technology, Tianjin, 300401, China

    Ya Guo, Fan Yang, Yazhong Si, Yipu Yang, Wei Zhang & Yatong Zhou

  2. Chinese Electronics Technology Group 54th Institute, Shijiazhuang, 050081, China

    Xiaolong Zhang

  3. Hebei Province Unmanned System Intelligent Measurement and Control Information Technology Key Laboratory, Shijiazhuang, 050081, China

    Xiaolong Zhang

Authors
  1. Ya Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yazhong Si
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yipu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaolong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yatong Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization was presented by YG; methodology was provided by YG; software was developed by YG and FY; validation was approved by YG, YZ, and FY; formal analysis was performed by YG; data curation was conducted by YG and XZ; writing—original draft preparation was revised by YG; writing—review and editing were prepared by YG, YS, YY and WZ; visualization was provided YG; supervision was conducted by YZ and FY; project administration was approved by YZ; funding acquisition was analyzed by YZ. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Yatong Zhou.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Ethical and informed consent for data used

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, Y., Yang, F., Si, Y. et al. PnP-UGCSuperGlue: deep learning drone image matching algorithm for visual localization. J Supercomput (2024). https://doi.org/10.1007/s11227-024-06128-3

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11227-024-06128-3

Keywords

Search

Navigation