Part-Wise Adaptive Topology Graph Convolutional Network for Skeleton-Based Action Recognition

Abstract

Human action recognition is a computer vision challenge that involves identifying and classifying human movements and activities. The behavior of humans comprises movements of multiple body parts, and Graph Convolutional Networks (GCNs) have emerged as a promising approach for this task. However, most contemporary GCN methods perform graph convolution on the entire skeleton graph without considering that the human body consists of distinct body parts. To address these shortcomings, we propose a novel method that optimizes the representation of the skeleton graph by designing temporal and spatial convolutional blocks while introducing the Part-wise Adaptive Topology Graph Convolution (PAT-GC) technique. PAT-GC adaptively learns the segmentation of different body parts and dynamically integrates the spatial relevance between them. Furthermore, we utilize hierarchical modeling to divide the skeleton graph, capturing a more comprehensive representation of the human body. We evaluate our approach on three publicly available large datasets: NTU RGB + D 60, NTU RGB + D 120, and Kinetics Skeleton 400. Our experimental results demonstrate that our approach achieves state-of-the-art performance, thus validating the efficiency of our proposed technique for human action recognition.

1. Introduction

Human action recognition is a significant task with numerous applications, including video comprehension, robotic vision, autonomous driving, and virtual reality. Skeleton-based action recognition has recently received more attention due to the development of low-cost motion sensors and human pose estimation algorithms. The goal of this task is to use skeleton graph sequences to identify corresponding actions. In comparison to video data [1,2,3,4], skeleton data has a number of advantages for action recognition. In particular, skeleton data is a more reliable data source for this task because it is more robust to occlusion, provides a more compact data representation, and is less sensitive to lighting conditions. Furthermore, since skeleton data doesn’t include any private visual information about the person being observed, it offers increased privacy. The use of skeleton-based data for action recognition has grown in popularity as a result of these benefits.

Unlike RGB or thermal video, skeleton data only includes the 2D or 3D positions of human joints. This lessens some redundant information while also reducing the amount of information compared to raw video data. The joints in the human skeleton can be thought of as nodes in a sparse graph, and the natural connections between them as edges. These connections between joints are typically described using the adjacency matrix. Given that graph convolution is a logical choice for handling graph-structured data, graph convolutional networks have found extensive use in processing the skeleton of the human body’s graph structure. Relationships within human skeletons have been described using predetermined topology in more recent methods. Predetermined topology, however, has difficulty simulating relationships between joints that are not physically connected, such as brain-controlled motor coordination between the arms and legs that are not physically connected.

Some methods add more manually crafted edges to the physically connected graph structure in order to extract complex information. To improve network performance, attention mechanisms and multiscale operators are also used. Some methods divide the adjacency matrix into subgraphs connected to the arms, legs, or torso in order to extract fine-grained features from the human skeleton. Body part representation can highlight the significance of each component and how they relate to one another in both space and time. However, the partitioning of the adjacency matrix determines how effective these methods are, and various segmentation methods may produce various outcomes. It is still unclear how to accurately describe the human skeleton and use the data it contains. In our opinion, a practical segmentation approach should be data-driven in addition to relying on predetermined topology.

Understanding and analyzing human motion requires the examination of human movement patterns and the relationship between body parts. The interaction between different body parts is a critical aspect of various types of movements, such as walking, running, jumping, and throwing. The human body is organized as an assembly of body parts linked through joints, with all body parts controlled by the brain. While the precise mechanism through which the brain generates and controls movements is unclear, specific effects can be observed. As depicted in Figure 1a, joints within the same part of the body tend to exhibit similar movement patterns. For instance, the shoulder joint and elbow joint both move similarly as they are both situated in the upper arm and connected to the same muscles. The knee and hip joints move similarly as they are situated in the lower body and connected to the same muscles. In Figure 1b, coordination involves both symmetrical and asymmetrical coordination of limbs. A crucial dynamic effect in human walking is that limb movements are nearly synchronized. The limbs move in a rhythmic pattern, with the arms and legs swinging in opposite directions almost simultaneously. This synchronized movement helps maintain balance and enables the body to move forward more efficiently. In Figure 1c, each person’s movement patterns are unique. Due to differences in age, gender, health, and level of physical fitness, these patterns can vary greatly between people. Action recognition may be inaccurate as a result of predetermined topology.

In this paper, we focus on optimizing the description of the human skeleton. Specifically, we describe the skeleton data using both static and dynamic relationships. For static relations, we also use predetermined topology. Motivated by the approach presented in [5] to divide the human skeleton into different parts, we propose a hierarchical approach to dividing the skeleton topology into fine and coarse scales. At the fine scale, the skeleton is subdivided into five parts: trunk (including the head and torso), left arm, right arm, left leg, and right leg. This allows the model to concentrate on the movement patterns of specific body parts and capture their role in the overall motion. The joints are combined into larger groups based on similar movements or relationships between body parts at the coarse scale. For instance, the same side of the arm and thigh can be combined, or the left and right arms can be combined. This larger grouping enables the model to capture more complex movement patterns and relationships between different body parts during motion, such as hand-foot coordination.

The static topology provides a set of fixed connections between joints defined before training the model. While static topology can be effective for certain graphs, it may not always capture all the relevant relationships and dependencies within the data. This is due to the fact that manually created topologies are constrained by prior knowledge of the graph and may not accurately represent more intricate patterns and relationships. Therefore, we treat the relationships between joints as data-driven and aim to capture the latent dependencies between joints. Specifically, we propose a part-wise adaptive topology graph convolution (PAT-GC), which models adaptive topology and extracts discriminative features. The adaptive topology obtained through PAT-GC can be seen as a refinement of the static topology. PAT-GC enables more fine-grained adjustments to the connections between joints and can capture the local structure of the graph in each layer and learn more meaningful representations.

We propose the part-wise adaptive topology graph convolutional network (PAT-GCN), which stacks multiple spatio-temporal blocks to extract features from human skeleton stream data. The spatio-temporal block includes two parts: spatial modeling and temporal modeling. Spatial modeling is achieved using PAT-GC as the basic module and utilizing a hierarchical structure to aggregate motion information. Temporal modeling is accomplished using the multi-scale temporal convolution module, consisting of several branches with different temporal graph convolutions.

To validate the effectiveness of our proposed PAT-GCN, we conducted extensive experiments on three public datasets: NTU RGB + D 60, NTU RGB + D 120, and Kinetics Skeleton 400. The results demonstrate that our method achieves state-of-the-art performance. To show the importance of each component, we also conducted ablation experiments. In summary, the main contributions of our work are as follows:

• We propose a hierarchical approach to partition the skeleton topology into multiple parts at two different scales. This method enables the exploration of movement patterns for various body parts, as well as their interrelationships during motion.
• We propose a part-wise adaptive topology graph convolution design that leverages data-driven methods to obtain an adaptive topology and extract discriminative features.
• The extensive experimental results highlight the benefits of the part-wise adaptive topology graph convolution. Our proposed PAT-GCN outperforms state-of-the-art methods on three different, skeleton-based action recognition benchmarks.

The remainder of this paper is organized as follows: In Section 2, we review related studies, including graph convolutional networks, GCN-based skeleton action recognition, and part-based skeleton action recognition. In Section 3, we elaborate on our approach. Section 4 presents and analyzes experimental details and results. Finally, in Section 5, we summarize our work and provide an outlook for the future.

2. Related Work

2.1. Graph Convolutional Network

Convolutional Neural Networks (CNNs) have been remarkably successful at processing Euclidean data, such as images. Traditional convolution, on the other hand, is restricted to regular grid data and is ineffective for handling general graphs. Graph convolution has been created to overcome this limitation and enable the extraction of local patterns from data using graph structures. Graph Convolutional Networks (GCNs) were created with the specific purpose of directly extracting features from non-Euclidean data. GCN-based methods can be divided into spectral methods [6,7,8] and spatial methods [9,10,11,12]. Spectral methods define convolution in the spectral domain using a set of learned filters in the graph Fourier domain. The main idea is to leverage the eigenvalues and eigenvectors of the graph Laplacian matrix, which captures the structural properties of the input graph, to define a set of filters that can be applied to the graph signal. The convolution filters, which are defined in the spectral domain, are not localized in the vertex domain, which is a significant disadvantage of spectral methods. On the other hand, spatial methods typically update each node layer by layer by choosing neighbors, merging the features from the neighbors that were chosen, and then applying the activation function to the merged features. In this work, we use spatial methods and aggregate features based on skeleton graphs with pre-designed structures.

2.2. Gcn-Based Skeleton Action Recognition

Early studies on skeleton action recognition [13,14,15,16] generally utilized hand-crafted features to record human body motion. However, these approaches primarily focused on utilizing the relative 3D rotations and translations between joints, resulting in complex feature design and sub-optimal performance. In recent years, deep learning methods have made significant progress in the field of skeleton action recognition.

GCN-based models are built on a series of skeleton graphs, enabling the extraction of as much discriminative information as possible in the spatial and temporal domain. As a result, GCN-based approaches have shown significant improvement in skeleton action recognition. ST-GCN [17], which defines a spatial and temporal graph convolutional network, is one notable example of GCN-based techniques. It takes into account both the natural human body structure and the temporal motion correlation in the spatio-temporal domain. Another well-known model is 2s-AGCN [18], which takes the original 3D joint coordinates as joint stream data and the 3D coordinate difference between two adjacent joints as bone stream data. To learn the non-local dependencies in the spatial dimension, this model constructs an adaptive graph to give adaptive attention to each joint. MS-G3D [19] introduces the concept of cross-time-space aggregation, which combines spatial domain graph convolution and temporal convolution and uses a multi-scale aggregation scheme to differentiate the importance of nodes in different neighborhoods for effective remote modeling. The G3D module can use dense cross-time-space edges as skip connections to directly propagate information on the time-space graph. A mobile graph convolutional network called Shift-GCN [20], which combines both spatial and temporal shift graph convolution, was created to recognize skeleton actions. In comparison to standard graph convolution, the non-local spatial shift method performs noticeably better and uses less processing power. While maintaining recognition effectiveness, adaptive temporal shift graph convolution can adapt the receptive field while also requiring less computation and parameter input. In this study, we extract the skeleton feature representation using a GCN-based network.

2.3. Part-Based Skeleton Action Recognition

The contribution of joint points to recognition in various motions varies for the motion categories. Some methods re-weight the joint data to produce better representations. They might be motivated for a variety of reasons, such as to solve the sample imbalance problem [21] or discovering the discriminative joints [22]. The network is able to better capture temporal and spatial information [23] by using the attention mechanism, which also helps the network focus attention on the important areas [24]. The definition and connection relationships of joints can be viewed as prior information for skeleton data. An accurate prior can speed up convergence, increase accuracy, and strengthen the model’s robustness. The part-based approach modifies the previously known information. Similar to the method previously mentioned, it is also a form of re-weighting mechanism that modifies the joints and the adjacency matrix at the neighborhood level.

In order to learn part representations, Du et al. [25] suggest dividing the human skeleton into five parts and feeding each part into a different recurrent neural network (RNN). These part representations are then combined into higher-level features for action recognition using a hierarchical RNN. To extract spatial features from various skeleton parts and learn temporal features from stacked frames, Si et al. [26] propose a novel model made up of a spatial reasoning network (SRN) and a temporal stack learning network (TSLN). To concentrate on the most important body parts for each action sequence, PA-ResGCN [27] makes use of multiple input branches and a part-wise attention module. The Multi-View Interactional Graph Network (MV-IGNet) [28] is an idea put forth by Wang et al. that makes use of various views of the graph to produce supplementary features. These views can be attained by taking into account various joint subsets or by representing various connections with various edge types. Fixed or single-scale segmentation, however, has a limited ability to scale and be flexible. Our proposed model allows to capture more complex motion patterns by including adaptive topology.

The representation of skeleton data can be improved by multiscale technology [29]. In earlier works, joints were clustered to produce a coarser pose. MSR-GCN [30] abstracts a human pose by combining joints that are close to one another and replacing the group with a pseudo joint. DMS-GCN [31] includes a different number of joints at each level. By averaging the locations of the joints in the fine-level joint group, the coarse-level joints are calculated. Our approach does not produce any pseudo joints and coarse poses. Instead, in order to find more informative joints and connections, we learn the communication relationships between joints at coarse and fine levels. The joints are shared by regions of importance, and the hierarchical representation provides a more flexible representation.

3. Methods

In this section, we first formulate the part-wise graph convolution. Then we present the part-wise adaptive topology graph convolution (PAT-GC) and describe the goal of PAT-GC. Finally, we introduce the whole structure of our PAT-GCN.

3.1. Part-Wise Graph Convolution

Formally, a human skeleton can be represented as a graph with joints as vertices and bones as edges. The graph is defined as $\mathcal{G}=(\mathcal{V},\mathcal{E})$ where $\mathcal{V}=\left\{v_1,v_2,\dots ,v_N\right\}$ is the set of N joints and $\mathcal{E}$ is the set of edges. The graph connectivity can be represented by the adjacency matrix $\mathbf{A}\in {\mathbb{R}}^{N\times N}$. Normally, the element value takes 1 or 0 indicating whether the positions of $v_i$ and $v_j$ are adjacent. The neighborhood of $v_i$ is denoted as $\mathcal{N}\left(v_i\right)=\left\{v_j\mid a_{ij}\ne 0;v_i,v_j\in \mathcal{V}\right\}$. Given a skeleton sequence, we first compute a set of features $\mathcal{X}=\left\{x_{t,n}\mid 1\le t\le T,1\le n\le N;n,t\in \mathbb{Z}\right\}$ that can be represented as a feature tensor $\mathbf{X}\in {\mathbb{R}}^{C\times T\times N}$, where $x_{t,n}={\mathbf{X}}_{t,n}$ denotes the C dimensional features of the node $v_n$ at time t.

The normal graph convolution attempts to optimize the weight $\mathbf{W}$ for extracting output features Y. The graph convolution is denote as

$${\mathbf{y}}_i=\sum _{v_j\in \mathcal{N}\left(v_i\right)}{a}_{ij}\mathbf{W}{\mathbf{x}}_j.$$

(1)

In this paper, we use the spatial graph convolution to capture the spatial correlations between human joints within each frame. The spatial graph convolution can be defined as:

$$\mathbf{Y}=\rho \left({\mathbf{D}}^{-\frac{1}{2}}\left(\mathbf{A}+\mathbf{I}\right){\mathbf{D}}^{-\frac{1}{2}}\mathbf{W}\mathbf{X}\right),$$

(2)

where $\mathbf{D}$ is the degree matrix of $\mathbf{A}$. $\mathbf{X}$ denotes the input and $\mathbf{Y}$ denotes the output features respectively. $\mathbf{W}$ are the graph convolution weights. $\rho (·)$ is the activation function.

The human skeleton graph can be divided into multi parts, where each sub-graph is a part of the body. Formally, the whole skeleton graph can be described by each part as:

$$\mathcal{G}=\bigcup _{k\in \{1,\dots ,K\}}{\mathcal{P}}_k\mid {\mathcal{P}}_k=\left({\mathcal{V}}_k,{\mathcal{E}}_k\right),$$

(3)

where ${\mathcal{P}}_k$ is the k-th sub-graph, which indicates that the human skeleton is divided into K parts. Similar to the definition of the overall graph, ${\mathcal{V}}_k$ and ${\mathcal{E}}_k$ represent the set of joints and the set of bones respectively. In the case of the single frame, the graph convolution on a sub-graph of the single part can be defined as follows:

$$\mathbf{Y}\left(v_i\right)=\sum _{v_j\in S\left(v_i\right)}\mathbf{W}\left(v_j\right)\mathbf{X}\left(v_j\right);v_i,v_j\in P_k,$$

(4)

where $\mathbf{Y}\left(v_i\right)$ represents the output feature map for $v_i$. $S\left(v_i\right)$ is the sampling region for $v_i$. The output of graph convolution is the weighted average of the input feature maps of the nodes within the sampling region. The size of the sampling region is dependent on the size of the convolution kernel.

Here, we present two different division strategies for obtaining spatial features from the skeleton topology, as illustrated in Figure 2. The first strategy involves a fine-scale division that separates the skeleton into five distinct parts: the left arm, right arm, left leg, right leg, and trunk. This approach enables the detection of more subtle differences in motion between various body parts.

The second strategy explores relationships between various body parts while moving by combining the five parts into a coarse-scale division. For instance, one combination might explore movements on one side of the body by involving the arm and leg on the same side, while another might examine coordination between the hands and feet by involving the arm and leg on different sides. It is possible to comprehend the relationships between various movements better by using these body part combinations. Each separated part can be denoted as $P_k$.

3.2. Adaptive Topology Graph Convolution

The skeleton graph’s graph convolution is computed using a predetermined graph, which may not be the best option. It may be difficult to accurately capture each person’s distinctive features and movement traits using a predetermined topology. A predetermined topology might also be incapable of adjusting to evolving movement patterns. As a result, we introduce part-wise adaptive topology graph convolution (PAT-GC) and extend $P_k$ into $\widehat{P_k}$. This extension is accomplished using data-driven methods that allow for more flexible modeling of the relationship between joints.

We present a part-wise adaptive topology graph convolution (PAT-GC) block for extracting spatial features, as illustrated in Figure 2. The PAT-GC block contains a toal of three paths, the top and middle paths are used for feature transformation, and the bottom path is used to generate an adaptive topology.

3.2.1. Feature Transformation

In the feature transformation stage, we utilize an attention-based module to enhance the vanilla graph convolution. Specifically, we utilize a $1\times 1$ convolution to convert input features into high-level representations. Additionally, we incorporate a channel attention module comprising global pooling, convolution, and activation functions to recalibrate the features. The inputs are converted to $1\times 1\times C$ by global pooling. After ReLU, the number of channels is made consistent with the number of channels of the skeleton feature by a $1\times 1$ convolution. Then sigmoid activation function is used to transform the dot product into a range between 0 and 1, representing the attention weights.

3.2.2. Adaptive Topology

The adaptive topology strategy comprises static and dynamic components. The static topology graph convolution utilizes the predetermined topology as mentioned before, whereas the dynamic topology graph convolution is based on the input features. The dynamic adjacency matrix is generated in a data-driven manner and optimized alongside the other parameters during the training phase.

For topology generation, we use bilinear transformation to extract the feature correlations between vertices. This process is formulated as:

$$\tilde{\mathbf{D}}=\psi \left(\mathbf{X}\right)\varphi {\left(\mathbf{X}\right)}^T,$$

(5)

where $\tilde{\mathbf{D}}\in {\mathbb{R}}^{N\times N}$ is the correlation matrix. $\psi \left(\mathbf{x}\right)$ and $\varphi \left(\mathbf{x}\right)$ is the transformed feature after convolution and temporal pooling. Here, we choose the same $\psi$ and $\varphi$ with reduction rate r to extract compact representations. After activation function $\sigma$, we obtain the correlation $\mathbf{D}\in {\mathbb{R}}^{N\times N}$, which represents relationships of vertices under a certain type of motion feature. Finally, by utilizing the dynamic topology $\mathbf{D}$ and the predetermined static topology $\mathbf{A}$, we obtain the adaptive topology $\tilde{\mathbf{A}}$:

$$\tilde{\mathbf{A}}=\mathbf{A}+\alpha ·\mathbf{D},$$

(6)

where $\alpha$ is a trainable scalar to adjust the intensity of dynamic topology.

3.3. Network Architecture

Using the proposed adaptive topology graph convolution and part aggregation strategy, we design a powerful graph convolutional network called PAT-GCN for skeleton action recognition. The network architecture is illustrated in Figure 3. The proposed network leverages the relationships between different body parts by utilizing GCN architecture to perform action recognition. The backbone of the network comprises ten basic spatial-temporal blocks, followed by a global average pooling layer, and a softmax classifier to obtain the prediction. Formally, given an input sequence of 2D or 3D joint locations $\mathcal{X}=\left\{x_{t,n}\in {\mathbb{R}}^{d\in \{2,3\}}\mid 1\le t\le T,1\le n\le N;t,n\in \mathbb{Z}\right\}$, the output of the backbone can be defined as:

$${\mathcal{Y}}^{\left(backbone\right)}=\left\{y_{t,n}^{\left(backbone\right)}\in {\mathbb{R}}^{C^{\left(backbone\right)}}\mid 1\le t\le T,1\le n\le N;t,n\in \mathbb{Z}\right\},$$

(7)

where $y_{t,n}^{\left(backbone\right)}$ is the output of joint n at frame t,and $C^{\left(backbone\right)}$ represents the channel dimension of the output features. The number of channels in the first basic spatial-temporal block is 64. After the fifth and eighth blocks, the channel dimension is doubled, and the temporal dimension is halved using strided temporal convolution. Each block contains a spatial modeling module, a temporal modeling module, and two residual connections.

Spatial Modeling. Figure 3 illustrates that we use a hierarchical structure to stack the PAT-GC blocks. Specifically, we utilize the residual structure to organize the PAT-GC module, separately processing the segmented parts of two different scales. Using multiple PAT-GCs for the same region allows for obtaining multiple adaptive variants. Therefore, there are $p\times k$ PAT-GC blocks in the spatial modeling, where p is the number of parts and k is the number of adaptive variants for each part. Each block is responsible for processing a specific topology. After batch normalization and ReLU activation, the aggregated features from each block are transformed into the inputs of temporal modeling.

Temporal Modeling. To model actions of different durations, we utilize a multi-scale temporal modeling module that can capture both short-term and long-term temporal dependencies. Our module is inspired by the work of Liu et al. [19], who employed multiple branches with different temporal resolutions to disentangle appearance and motion features. However, unlike their method, we use fewer branches to avoid decreasing inference speed. As shown in Figure 3, each branch consists of a $1\times 1$ convolution layer that reduces the number of channels in the input feature map. Additionally, four branches contain a temporal layer that operates on the temporal dimension, which can be either a dilated convolution with a specific dilation rate or a max pooling operation. The dilation rates are set to 1, 2, and 4, respectively. By using various dilation rates, we effectively enlarge the receptive field and capture multi-scale temporal patterns. The output feature maps of all branches are concatenated along the channel dimension to form the final output of the proposed ST block.

Multi-stream Ensemble. Following privous works [20,32,33,34], we generate four streams for each skeleton sequence, including joint, bone, joint motion, and bone motion. The raw 3D positions of each joint are contained in the joint stream specifically. The displacement between two adjacent joints in the predetermined human skeleton structure produces the bone stream. The source of the joint motion stream is the displacement of joint data between two adjacent frames, and the bone motion stream is produced similarly by the displacement of bone data. The final prediction in this study is calculated by adding the softmax scores from each stream.

4. Experiments

We evaluate our action recognition method on three widely used skeleton action recognition datasets, including the NTU RGB + D 60, NTU RGB + D 120, and Kinetics Skeleton 400. The experimental details and results are presented in the following section.

4.1. Datasets

NTU RGB + D 60 [2] is a widely known dataset for skeleton action recognition consisting of 56,880 skeleton sequences of 60 action classes. The sequences were recorded from 40 different subjects and three distinct camera perspectives using Microsoft Kinect v2 cameras. Each skeleton sequence includes the 3D spatial coordinates of 25 joints. The dataset offers two evaluation benchmarks: Cross-Subject (X-sub) and Cross-View (X-view). In X-sub, the training set comprises 40,320 sequences from 20 subjects, while the testing set contains 16,560 sequences from the remaining 20 subjects. In X-view, the training set consists of 37,920 sequences from the front and two side views, and the testing set contains 18,960 sequences from the left and right 45-degree views. Both evaluation benchmarks contain all skeleton sequences, which are divided for cross-validation according to subjects or views. We train on both benchmarks from scratch and verify the recognition accuracy independently.

NTU RGB + D 120 [35] is currently the largest dataset available for 3D joint annotation-based human action recognition, including 113,945 action samples across 120 action classes performed by 106 volunteers with three camera views. The dataset includes 32 different setups, each representing a specific location and background. The authors of this dataset recommend two benchmarks to evaluate its performance: Cross-Subject (X-sub) and Cross-Setup(X-setup). In X-sub, the training data (63,026 videos) comes from 53 subjects, and the testing data (50,922 videos) comes from the other 53 subjects. In X-setup, the training data (54,471 videos) from samples with even setup IDs, and testing data (59,477 videos) from samples with odd setup IDs. NTU RGB + D 120 extends NTU RGB + D 60 with additional action categories and performers. We evaluate the model using a strategy similar to that of NTU RGB + D 60.

Kinetics Skeleton 400 [36] is a large-scale dataset of human action video clips sourced from YouTube, comprising 306,245 video clips in 400 classes. Each class contains at least 400 video clips. The dataset includes only raw video clip samples without skeleton sequences and selects two people for multi-person clips based on the average joint confidence. The original video clip is approximately 10 s long. We employ subject-invariant validation strategy to evaluate the performance of our model on the released data (Kinetics-Skeleton). The dataset is divided into three parts, one for training with 250–1000 videos per class, one for validation with 50 videos per class, and one for testing with 100 videos per class. Researchers utilize the publicly available OpenPose toolbox to estimate the positions of 18 joints in each frame of the clip.

4.2. Implementation Details

The experiments were conducted using the Pytorch1.6 deep learning framework on four GTX1080Ti GPUs. Stochastic gradient descent (SGD) was employed with a momentum of 0.9 and weight decay of 0.0001. The model was trained for 50 epochs on NTU-RGB + D 60 and 65 epochs on NTU-RGB + D 120, respectively. The initial learning rate was set to 0.1, decreasing by a factor of 0.1 at epoch 35. For these two datasets, the batch size was set to 64, and input preprocessing was conducted in accordance with previous research [19]. For Kinetics Skleton 400, the batch size was set to 128, with a learning rate of 0.1 that decreased by a factor of 0.1 at epoch 45. The model was trained for a total of 80 epochs. To improve training stability, the learning rate gradually increased during the first 5 epochs in all experiments.

4.3. Ablation Study

We analyze the individual components and their configurations to demonstrate the effectiveness. The reported performance is conducted on the X-Sub subset of the NTU RGB + D 60 using only the joint data.

Configuration Exploration of PAT-GCN. We explore various configurations of PAT-GCN, including the number of adaptive variants of each part, whether or not to use the division configuration, and the choice of spatial modeling structure. As shown in

Table 1. Comparing the model accuracy of PAT-GCN with different settings. # represents the number of variants.

Design	#Variant	Hierarchical	Acc (%)
PAT-GCN	1	✔	89.9
PAT-GCN	2	✔	90.4
PAT-GCN	4	✔	90.3
w/o D	2	✔	89.6
w/o A	2	✔	89.3
w/o fine-scale	2	✔	90.0
w/o coarse-scale	2	✔	89.5
1 PAT-GC	2	×	90.1
2 PAT-GC	2	×	90.4

, we observe that the accuracy improvement by increasing the number of adaptive variants is limited. Next, we verify the influence of static topology A and dynamic topology D by removing them from the PAT-GC respectively. Without dynamic topology, the accuracy decreased by 0.8% indicating the importance of dynamic topology. Similarly, the performance of PAT-GCN without A decreases by 1.1%, confirming that it is difficult to learn a specific topology for each part without any constraints. In addition, we explore the effect of the spatial modeling structure, as shown in Figure 4. The experimental results show that fine-scale parts can benefit from the stacking of PAT-GC blocks, while using a hierarchical structure can reduce the number of parameters and achieve comparable accuracy.

Effectiveness of PAT-GC. To validate the efficacy of PAT-GC block to capture discriminative spatial features, we employ GC blocks in ST-GCN [17] to construct spatial modeling as the baseline for controlled experiments. Unless stated, the other results are obtained using the optimal settings in Table 1. We modify each component of the PAT-GC block and their parameters individually to obtain the accuracy in different situations. The experimental results are shown in

Table 2. Comparing the model accuracy with different reduction ratios and activation function.

Methods	r	$\mathit{\sigma }$	Acc (%)
Baseline	-	-	88.1
A	4	Tanh	90.2
B	8	Tanh	90.4
C	16	Tanh	90.0
D	8	Sigmoid	89.9
E	8	ReLU	90.2

. First, we observe that the accuracy is improved when we replace the GC blocks in the baseline with our proposed PAT-GC blocks. We observe that the best result outperforms the baseline by 2.3%. Besides, we explore different configurations of PAT-GC, including the reduction ratio r and the kind of activation function $\sigma$ of PAT-GC. As shown in Table 2, we observe that PAT-GC under all configurations outperform the baseline, confirming the robustness of PAT-GC. (1) Comparing models A, B and C, we find models with $r=8$ (model B) achieve better result than other models. (2) Comparing modes B, D and E, Tanh function performs better than Sigmoid and ReLU. Hence, we choose model B as our final model.

4.4. Comparisons with the State-of-the-Art Methods

The majority of state-of-the-art methods adopt multi-stream fusion frameworks to exploit complementary information. To make a fair comparison, we adopt the same strategy as in [20,33,37]. Specifically, we fuse the results of four modalities data, namely joint, bone, joint motion, and bone motion.

We compare our network with the state-of-the-art methods on NTU RGB + D 60, NTU RGB + D 120, and Kinetics Skeleton 400 in

Table 3. Comparison of the Top-1 accuracy (%) with the state-of-the-art methods on the NTU RGB-D 60 skeleton dataset.

Methods	NTU RGB + D 60
	X-Sub (%)	X-View (%)
ST-GCN (2018) [38]	81.5	88.3
AS-GCN (2019) [39]	86.8	94.2
2s-AGCN (2019) [18]	88.5	95.1
DGNN (2019) [34]	89.9	96.1
SGN (2020) [40]	89.0	94.5
Shift-GCN (2020) [20]	90.7	96.5
Dynamic GCN (2020) [37]	91.5	96.0
MS-G3D (2020) [19]	91.5	96.2
MST-GCN (2021) [33]	91.5	96.6
4s DualHead-Net (2021) [32]	92.0	96.6
Ta-CNN (2022) [41]	90.4	94.8
Ta-CNN+ (2022) [41]	90.7	95.1
EfficientGCN-B4 (2022) [42]	91.7	95.7
PAT-GCN (Joint Only)	90.4	95.1
PAT-GCN (Joint+Bone)	92.2	96.4
PAT-GCN	92.7	97.1

,

Table 4. Comparison of the Top-1 accuracy (%) with the state-of-the-art methods on the NTU RGB-D 120 skeleton dataset.

Methods	Param. (M)	NTU RGB + D 120
		X-Sub (%)	X-Set (%)
2s-AGCN (2019) [18]	6.9	82.9	84.9
SGN (2020) [40]	1.8	79.2	81.5
Shift-GCN (2020) [20]	2.7	85.9	87.6
MS-G3D (2020) [19]	6.4	86.9	88.4
Dynamic GCN (2020) [37]	14.4	87.3	88.6
MST-GCN (2021) [33]	12.0	87.5	88.8
4s DualHead-Net (2021) [32]	12.0	88.2	89.3
Ta-CNN+ (2022) [41]	4.4	85.7	87.3
EfficientGCN-B4 (2022) [42]	1.1	88.4	89.1
PAT-GCN	5.4	89.2	90.6

and

Table 5. Comparison of the Top-1 accuracy (%) and Top-5 accuracy (%) with the state-of-the-art methods on the Kinetics Skeleton 400 dataset.

Methods	Kinetics Skeleton 400
	Top-1 (%)	Top-5 (%)
ST-GCN (2018) [38]	30.7	52.8
AS-GCN (2019) [39]	34.8	56.5
2s-AGCN (2019) [18]	36.1	58.7
DGNN (2019) [34]	36.9	59.6
MS-G3D (2020) [19]	38.0	60.9
Dynamic GCN (2020) [37]	37.9	61.3
MST-GCN (2021) [33]	38.1	60.8
4s DualHead-Net (2021) [32]	38.4	61.3
PAT-GCN	39.2	62.4

respectively. On all three datasets, our method outpeforms all existing methods on almost all evaluation benchmarks. Specifically, as shown in Table 3, our method achieves state-of-the-art performance on the X-sub and X-view of the NTU RGB + D 60, which outpeforms the state-of-the-art DualHead-Net by 0.7% and 0.5%. We show some qualitative results about the diffcult actions in Figure 5. These actions are easily confused with other categories, such as “writing” and “type on a keyboard”. Our method achieves better performance. However, it can be improved in the temporal modeling. DualHead-Net with more complex temporal modeling achieves better performance on “put on a shoe” and “take off a shoe”, which are confused in terms of temporal relationship. The inference speed of our method is 32.9 sequences/(second ∗ GPU). For NTU RGB + D 120 dataset, as shown in Table 4, our method outperforms the EfficientGCN-B4 by 0.8% on the X-sub and outperforms the DualHead-Net by 1.3% on the X-set respectively. In addition, for the Kinetics Skeleton 400 dataset, as shown in Table 5, our four-stream model outperforms the DualHead-Net by 0.7 in terms of the top-1 accuracy. The results demonstrate the effectiveness of the proposed model.

5. Conclusions

This paper presents a novel part-wise adaptive topology graph convolution (PAT-GC) for skeleton action recognition. PAT-GC processes multiple parts instead of the entire skeleton graph. We divide the topology representation of the skeleton into two types, static and dynamic, corresponding to a predetermined adjacency matrix and a trainable matrix, respectively. Static topology provides effective prior information, and dynamic topology guides the adjaceny matrix to focus on more informative connections. We also design a hierarchical structure as spatial modeling to integrate features of two-scale parts. Our method can extract more discriminative features by considering the part-wise information of skeleton data. Finally, we combine the proposed spatial and multi-scale temporal modules to construct the PAT-GCN for skeleton action recognition. Extensive experiments on three large datasets demonstrate that our model outperforms state-of-the-art methods. In the future, we will extend the strategy of temporal modeling and multi-stream ensemble by customizing the structure of each stream to produce more efficient models.