Abstract
Skeleton-based human action recognition has achieved a great interest in recent years, as skeleton data has been demonstrated to be robust to illumination changes, body scales, dynamic camera views, and complex background. Nevertheless, efficient modeling of relationships between non-adjacent nodes in 3D skeleton spacetime is still an open problem. In this work, we propose a Joint Spatiotemporal Collaborative Relationship (JSCR) network which model utilizes the query and key vectors of each node to construct the relationship feature matrix between nodes, and realizes the feature capture and reorganization of nodes in the space-time dimension. In our JSCR model, Spatial Collaborative Relation Module (SCRM) is used to model the relationship between non-adjacent nodes within a frame, and Temporal Collaborative Relation Module (TCRM) is used to understand the interaction between non-adjacent nodes across multiple frames. The two are combined in a two-stream network which outperforms state-of-the-art models using the same input data on both NTU-RGB + D and Kinetics-400.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aggarwal, J.K.: Human activity analysis: a review. ACM Comput. Surv. 43(3), 1–43 (2011)
Poppe, R.: A survey on vision-based human action recognition. Image Vis. Comput. 28(6), 976–990 (2010)
Yong, D., Wei, W., Liang, W.: Hierarchical recurrent neural network for skeleton based action recognition. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1110–1118. CVPR, Boston (2015)
Shahroudy, A., Liu, J., Ng, T. T., et al.: Ntu rgb+ d: a large scale dataset for 3d human activity analysis. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1010–1019. CVPR, Las Vegas (2016)
Kiwon, Y., Jean, H., Debaleena, C., et al.: Two-person interaction detection using body-pose features and multiple instance learning. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 28–35. CVPRW, Providence (2012)
Gunnar, J.: Visual perception of biological motion and a model for its analysis. Percept. Psychophys. 14, 201–211 (1973)
Zhengyou, Z.: Microsoft kinect sensor and its effect. IEEE Multimedia 19(2), 4–10 (2012)
Zhe, C., Tomas, S., Shih-En, W., et al.: Realtime multi-person 2d pose estimation using part affinity fields. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1302–1310. CVPR, Honolulu (2017)
Shotton, J., Fitzgibbon, A., Cook, M., et al.: Real-time human pose recognition in parts from single depth images. In: CVPR 2011, pp. 1297–1304. IEEE, Colorado Springs (2011)
Sijie, S., Cuiling, L., Junliang, X., et al.: Skeleton-indexed deep multi-modal feature learning for high performance human action recognition. In: 2018 IEEE International Conference on Multimedia and Expo, pp. 1–6. ICME, San Diego (2018)
Jinmiao, Ca., Nianjuan, J., Xiaoguang, H., et al.: Jolo-gcn: mining joint-centered light-weight information for skeleton-based action recognition. In: 2021 IEEE Winter Conference on Applications of Computer Vision, pp. 2734–2743. WACV, Waikoloa (2021)
Yuxin, C., Ziqi, Z., Chunfeng, Y., et al.: Channel-wise topology refinement graph convolution for skeleton-based action recognition. In: 2021 IEEE/CVF International Conference on Computer Vision, pp. 13359–13368. ICCV, Montreal (2021)
Pranay, G., Anirudh, T., Aditya, A., et al.: Quo vadis, skeleton action recognition? Int. J. Comput. Vision 129(7), 2097–2112 (2021)
Sijie, Y., Yuanjun, X., Dahua, L.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, Washington, DC (2018)
Maosen, L., Siheng, C., Xu, C., et al.: Actional-structural graph convolutional networks for skeleton-based action recognition. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3590–3598. CVPR, Long Beach (2019)
Ziyu, L., Hongwen, Z., Zhenghao C., et al.: Disentangling and unifying graph convolutions for skeleton-based action recognition. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 140–149. CVPR, Seattle (2020)
Bin, L., Xi, L., Zhongfei, Z., et al.: Spatio-temporal graph routing for skeleton-based action recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 8561–8568. AAAI, Washington, DC (2019)
Lei, S., Yifan, Z., Jian, C., et al.: Twostream adaptive graph convolutional networks for skeletonbased action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12026–12035. CVPR, Long Beach (2019)
Acknowledgement
This work was supported by Natural Science Foundation of Shandong Province (NSFSP) under grant ZR2020MF137.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Lu, H., Wang, T. (2023). Joint Spatiotemporal Collaborative Relationship Network for Skeleton-Based Action Recognition. In: Huang, DS., Premaratne, P., Jin, B., Qu, B., Jo, KH., Hussain, A. (eds) Advanced Intelligent Computing Technology and Applications. ICIC 2023. Lecture Notes in Computer Science, vol 14086. Springer, Singapore. https://doi.org/10.1007/978-981-99-4755-3_67
Download citation
DOI: https://doi.org/10.1007/978-981-99-4755-3_67
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-4754-6
Online ISBN: 978-981-99-4755-3
eBook Packages: Computer ScienceComputer Science (R0)