Abstract
Image alignment is a highly researched topic in computer vision, which aligns a pair of images due to image changes. Despite the numerous studies conducted on this topic, large object transformation and huge illumination changes between a pair of images are still commonly encountered in real-world scenes, making the task of image alignment very challenging. In this paper, a novel image alignment algorithm is proposed. By inputting a pair of images that need to be aligned into the correlated siamese neural network, a series of blocks are extracted in feature layers from the reference image, and those blocks are correlated in the feature layers of the target image. Finally, the homography parameters between images are then regressed from the correlate layers. Compared with the classical image alignment algorithms, supervised deep homography, and unsupervised deep homography, the experimental results of our method demonstrate a superior performance on the image alignment tasks involving illumination changes, camera translation, and rotation.
Similar content being viewed by others
Availability of data and materials
The data used in this research involve sensitive information; however, we are willing to consider requests for access to a limited portion of the data to support transparency and reproducibility of our findings.
References
Zhao, Q., Ma, Y.K., Zhu, C., Yao, C.F., Feng, B.L., Dai, F.: Image stitching via deep homography estimation. Neurocomputing 450, 219–229 (1995)
Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74, 59–73 (2007)
Ban, Y., Wang, Y., Liu, S., Yang, B., Liu, M., Yin, L., Zheng, W.: 2D/3D multimode medical image alignment based on spatial histograms. Appl. Sci. 12, 8261 (2022)
Xu, M., Savolainen, T., Anderson, J., Zubko, N., Schuh, H.: Impacts of the image alignment over frequency for vlbi global observing system. Astron. Astrophys. 663 (2022)
Suzuki, K., Inoue, T., Nagata, T., Kasai, M., Nonomura, T., Matsuda, Y.: Markerless image alignment method for pressure-sensitive paint image. Sensors 22, 453 (2022)
Zeng, X., Xu, M.: Gum-Net: Unsupervised geometric matching for fast and accurate 3D subtomogram image alignment and averaging. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn. 2020, 4072–4083 (2020)
López-Rubio, E., Molina-Cabello, M.A., Castro, F.M., Luque-Baena, R.M., Marín-Jiménez, M.J., Guil, N.: Anomalous object detection by active search with PTZ cameras. Expert Syst. Appl. 181, 115150 (2021)
Xu, Z., Li, J., Meng, Y., Zhang, X.: CAP-YOLO: channel attention based pruning YOLO for coal mine real-time intelligent monitoring. Sensors 22 (2022)
Xu, X., Chen, X., Wu, B., Wang, Z., Zhen, J.: Exploiting high-fidelity kinematic information from port surveillance videos via a YOLO-based framework. Ocean Coast. Manag. 222, 106117 (2022)
Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004)
Bay, H., Ess, A., Tuytelaars, T., Gool, L.V.: Speeded-up robust features (SURF). In: Computer Vision and Image Understanding, vol. 110, No. 3, pp. 346–359. Similarity Matching in Computer Vision and Multimedia (2008)
Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: IEEE International Conference on Computer Vision, pp. 2564–2571 (2011)
Smith, S., Brady, M.: SUSAN-a new approach to low level image processing. Int. J. Comput. Vis. 23, 45–78 (1997)
Dai, W., Kan, H., Tan, R., Yang, B., Guan, Q., Zhu, N., Xiao, W., Dong, Z.: Multisource forest point cloud registration with semantic-guided keypoints and robust ransac mechanisms. Int. J. Appl. Earth Observ. Geoinf. 115, 103105 (2022)
Shi, G., Xu, X., Dai, Y.: SIFT feature point matching based on improved RANSAC algorithm. In: 5th International Conference on Intelligent Human-Machine Systems and Cybernetics, vol. 1, pp. 474–477 (2013)
Li, B., Wu, F., Liu, S., Tang, J., Li, G., Zhong, M., Guan, X.: CA-Unet++: An improved structure for medical CT scanning based on the Unet++ architecture. Int. J. Intell. Syst. 37 (2022)
Dhanya, V.G., Subeesh, A., Kushwaha, N.L., Vishwakarma, D.K., Kumar, T.N., Ritika, G., Singh, A.N.: Deep learning based computer vision approaches for smart agricultural applications. Artif. Intell. Agric. 6, 211–229 (2022)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)
DeTone, D., Malisiewicz, T., Rabinovich, A.: Deep image homography estimation. abs/1606.03798 (2016)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. abs/1409.1556 (2014)
Nan, S.N.L.: Feature extraction and segmentation processing of images based on convolutional neural networks. Optical Memory Neural Netw. 30, 67–73 (2021)
Le, H., Liu, F., Zhang, S., Agarwala, A.: Deep homography estimation for dynamic scenes. In: The IEEE Conference on Computer Vision and Pattern Recognition (2020)
Chen, Z., Fang, X.-N., Zhang, S.-H.: Local homography estimation on user-specified textureless regions. J. Comput. Sci. Technol. 37, 615–625 (2022)
Nguyen, T., Chen, S., Shivakumar, S., Taylor, C., Kondepogu, V.: Unsupervised deep homography: a fast and robust homography estimation model. IEEE Robot. Autom. Lett. 3(3), 2346–2353 (2018)
Zhang, J., Wang, C., Liu, S., Jia, L., Wang, J., Zhou, J., Sun, J.: Content-aware unsupervised deep homography estimation. In: European Conference on Computer Vision. Springer (2020)
Gonzalez, R., Faisal, Z.: Digital Image Processing, 2nd Edn (2019)
Lin, Z., Jia, J., Huang, F., Gao, W.: Feature correlation-steered capsule network for object detection. Neural Netw. 147, 25–41 (2022)
Yang, L., Kong, C., Chang, X., Zhao, S., Cao, Y., Zhang, S.: Correlation filters with adaptive convolution response fusion for object tracking. Knowl. Based Syst. 228, 107314 (2021)
Valmadre, J., Bertinetto, L., Henriques, J., Vedaldi, A., Torr, P.H.S.: End-to-end representation learning for correlation filter based tracking. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5000–5008 (2017)
Li, B., Yan, J., Wu, W., Zhu, Z., Hu, X.: High performance visual tracking with siamese region proposal network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Cen, M., Jung, C.: Fully convolutional siamese fusion networks for object tracking. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 3718–3722 (2018)
Funding
This research did not receive any funding support.
Author information
Authors and Affiliations
Contributions
ZH Validation, Data curation, Formal analysis, Investigation, Writing—original draft. XZ Methodology, Investigation, Formal analysis, Visualization, Writing—original draft, Writing—review and editing. SW Investigation, Formal analysis, Writing—review. GX Data curation, Investigation. HW Data curation, Investigation. LZ Conceptualization, Investigation, Formal analysis. CY Investigation, Formal analysis. All authors reviewed the manuscript.
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare that they have no competing interests.
Ethical approval
Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Hu, Z., Zheng, X., Wang, S. et al. All-day Image Alignment for PTZ Surveillance Based on Correlated Siamese Neural Network. SIViP 18, 615–624 (2024). https://doi.org/10.1007/s11760-023-02720-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-023-02720-x