Periodic-Aware Network for Fine-Grained Action Recognition

Abstract

Recently, skeleton-based action recognition has gained increasing attention and achieved remarkable results in coarse-grained action recognition. Despite the positive results shown in these attempts, they are less effective in scenarios that require a detailed comparison between fine-grained classes, e.g. different moves during a vault. In such scenarios, existing methods make it hard to distinguish subtle differences between actions with different numbers of repetitions. In this article, to solve the above problem, we introduce periodicity into fine-grained action classification and propose a novel network architecture named periodic-aware network (PAN) to distinguish fine-grained actions with different numbers of repetitions. Firstly, a periodicity feature extraction module (PFEM) is proposed to capture periodicity information and extract periodicity features of different levels. Then, a periodicity fusion module (PFM) is proposed to fuse periodicity features and spatiotemporal features. We apply multiple periodicity fusion modules to fuse different levels of features. Finally, the results are obtained by classifying the fusion features. Extensive experiments on two fine-grained skeleton-based action recognition datasets, namely FineGym and Diving48, show that our proposed method outperforms previous skeleton-based action recognition methods.

S. Luo and J. Xiao—These authors contributed equally to this work and should be considered co-first authors. This work was supported by the Guangdong Basic and Applied Basic Research Foundation No. 2021A1515011867, National Natural Science Foundation of China (NSFC) (61976123); Taishan Young Scholars Program of Shandong Province; and Key Development Program for Basic Research of Shandong Province (ZR2020ZD44).