Spatio-Temporal Information Fusion and Filtration for Human Action Recognition

Abstract

Human action recognition (HAR) as the most representative human-centred computer vision task is critical in human resource management (HRM), especially in human resource recruitment, performance appraisal, and employee training. Currently, prevailing approaches to human action recognition primarily emphasize either temporal or spatial features while overlooking the intricate interplay between these two dimensions. This oversight leads to less precise and robust action classification within complex human resource recruitment environments. In this paper, we propose a novel human action recognition methodology for human resource recruitment environments, which aims at symmetrically harnessing temporal and spatial information to enhance the performance of human action recognition. Specifically, we compute Depth Motion Maps (DMM) and Depth Temporal Maps (DTM) from depth video sequences as space and time descriptors, respectively. Subsequently, a novel feature fusion technique named Center Boundary Collaborative Canonical Correlation Analysis (CBCCCA) is designed to enhance the fusion of space and time features by collaboratively learning the center and boundary information of feature class space. We then introduce a spatio-temporal information filtration module to remove redundant information introduced by spatio-temporal fusion and retain discriminative details. Finally, a Support Vector Machine (SVM) is employed for human action recognition. Extensive experiments demonstrate that the proposed method has the ability to significantly improve human action recognition performance.

1. Introduction

In the realm of modern human resource management (HRM), precise human action recognition (HAR) has emerged as a fundamental element across several domains. These domains encompass human resource recruitment (HRR) [1], employee performance assessment (EPA) [2], and employee training (ET) [3]. In the context of HRM, it is important to explore effective and efficient HAR methods for recruitment selection, employee performance assessment, and providing personalized training.

Currently, HAR is categorized into three modalities based on data types: RGB sequence-based HAR (RGB-HAR) [4,5], skeleton sequence-based HAR (Skeleton-HAR) [6,7], and depth sequence-based HAR (Depth-HAR) [8,9,10]. RGB-HAR, although widely used in the early stages, is sensitive to variations in lighting conditions and environmental factors. Changes in illumination, the presence of shadows, and background interference can compromise its recognition accuracy, particularly in complex indoor settings. Skeleton-HAR relies on skeletal coordinate data obtained from depth sensors, encompassing the positions of key joint points such as the head, hands, shoulders, and hips. Acquiring skeletal sequences can be labor-intensive, requiring specialized pose estimation algorithms. In contrast, Depth-HAR involves the automatic identification and categorization of human action through the modeling of depth video sequences. These sequences are directly captured using a depth camera, with each frame of the depth image recording the object’s distance from the camera. Depth video sequences offer not only ease of acquisition but also resilience against variations in lighting conditions and environmental changes. Therefore, Depth-HAR is more appropriate for various HRM scenarios.

Depth video sequences record both temporal and spatial information. Spatial information records the coordinates, shapes, sizes, and positional relationships between body parts, which can clearly describe the static posture of human actions. Temporal information reflects the movement trajectory over time, which summarizes the dynamic movement of human actions. At present, mainstream approaches [10,11,12,13] to human action recognition tend to focus on capturing either spatial or temporal information in isolation, while ignoring the intricate interactions between these two dimensions. Symmetrically capturing space and time information and understanding the interactions between them are crucial for HAR tasks in complex HRM environments [1,2,3]. The temporal and spatial dimensions of movement are intertwined and together shape the characteristics and patterns of movement. For example, the action of a person lifting their hand includes not only the trajectory of the arm (temporal feature) but also the position and angle of the hand (spatial feature). This synthesis makes the interplay between temporal and spatial features a key factor in accurately recognizing and understanding human movements.

In this paper, we propose an efficient and effective HAR method tailored to HRM scenarios. Our approach symmetrically leverages both temporal and spatial information to enhance recognition accuracy. To begin, we extract DMM and DTM features from depth video sequences to characterize the temporal and spatial aspects of human actions, respectively. Subsequently, we design a novel feature fusion technique, CBCCCA, aimed at enhancing the fusion performance of space and time features by reconciling marginal and central information in the sample space. We then introduce a spatio-temporal information filtration module to eliminate redundant information introduced through spatio-temporal fusion, along with irrelevant information that does not contribute to action classification. Finally, we employ an SVM for human action recognition. Extensive and comprehensive experiments conducted on three representative human action datasets demonstrate that our method significantly improves the accuracy of action classification, thereby laying a solid foundation for the successful application of HAR technology within HRM scenarios.

Our main contributions are summarized as follows:

• We propose a novel Depth-HAR method tailored to HRM scenarios. To the best of our knowledge, the proposed method is the first HAR scheme that performs information fusion and filtration for symmetrically extracted space and time features.
• According to our technical contributions, we propose a new feature fusion algorithm called CBCCCA, which improves the feature fusion performance by collaboratively reconciling marginal and central information in the sample space.
• We construct a spatio-temporal information filtration module based on the information bottleneck theory, which can effectively remove redundant and irrelevant information to improve the discriminability of features.
• Our method achieves cross-subject accuracy of 97.1% on the MSR Action3D dataset [14], cross-subject accuracy of 95.3% on the UTD-MHAD dataset [15], and cross-subject accuracy of 96.2% on the DHA dataset [16], outperforming state-of-the-art methods.

The remainder of the paper is organized as follows. Section 2 presents related work. Section 3 introduces the proposed Depth-HAR method. Section 4 presents the experimental results and analysis. In Section 5, relevant issues with our approach are discussed. Finally, the paper presents the conclusions in Section 6.

2. Related Work

In this section, we first review Depth-HAR methods. Then, popular feature fusion techniques are briefly discussed.

2.1. Depth-Video-Based Human Action Recognition

In earlier works, Motion Energy Images (MEI) [9] and Motion History Images (MHI) [9] were introduced as temporal templates for human motion. MEI outlines the two-dimensional space contour of the action but does not contain time information. MHI collects both time information and space outlines through brightness attenuation. However, due to overlapping human actions occurring at the same spatial locations, the space and time information in MHI is incomplete. In [10], DMM is proposed as a representational method for human movements. DMM projects depth frames from three viewpoints and computes motion differences for each viewpoint, representing human action by accumulating these motion differences. In [17], DMM is employed to extract Local Binary Pattern (LBP) characteristics as descriptors. However, DMM cannot capture time information, which makes it difficult to distinguish actions with similar spatial trajectories but different time sequences. To address this problem, Hierarchical Pyramid Depth Motion Maps (HP-DMM) are investigated in [12]. HP-DMM extends DMM and combines time information. In [11], Three Motion History Images (MHIs) and Three Static History Images (SHIs) are introduced to integrate motion and static time information. In [18], Kamel et al. demonstrated that, with the aid of the Moving Joints Descriptor (MJD), effective recognition results can be achieved using only front-view Depth Motion Images (DMI) while maintaining lower computational complexity.

In addition to the above methods, deep learning algorithms are frequently applied in HAR research. Adrián et al. [19] proposed a 3D Fully Convolutional Neural Network (3D FCNN) that directly encodes spatiotemporal information from raw depth sequences for action classification. In [20], a Bidirectional Recurrent Neural Network (BRNN) network was introduced, which projects three-dimensional depth images onto three different two-dimensional planes and inputs them into separate BRNN modules for action classification. Keceli et al. [21] combined depth sequences with 2D CNN and 3D CNN models to generate a three-dimensional volume representation using the spatio-temporal information from depth sequences, which is then fed into a 3D CNN for human action recognition.

2.2. Feature Fusion Techniques

Feature fusion allows for the better use of features with different characteristics and the joint modeling of these different features. The fusion of multiple different features provides a more comprehensive representation of the target sample. Even if the differences between some features in different categories of samples are small, other features can still be distinguished, so the whole algorithm is more robust. The key to successful fusion is to effectively utilize information from multiple perspectives. Canonical Correlation Analysis (CCA) et al. [22] is a multivariate statistical analysis method that uses the correlation between pairs of composite variables to reflect the overall correlation between two sets of data. Several attempts at correlation have been proposed. Rasiwasia et al. [23] proposed cluster canonical correlation analysis (cluster-CCA) for the joint dimensionality reduction of two sets of data points. Only correlations between samples from the same set or class are considered. Kan et al. [24] proposed a Multi-View Discriminant Analysis (MvDA) method, which seeks a discriminative common space by jointly learning multi-view-specific linear transformations for robust object recognition from multiple views in an unpaired manner. Kan et al. [25] extended MvDA to generate a deep network. In addition to this, some researchers have combined the idea of manifolds. Sun et al. [26] proposed local methods such as Local Linear Embedding (LLE) and Local Preserving Projection (LPP) to discover low-dimensional manifolds embedded in the original high-dimensional space. Shen et al. [27] proposed a unified multiset canonical correlation analysis framework based on graph embedding for dimensionality reduction (GbMCC-DR), which combines different discriminative methods with graph regularisation CCA in combination. Mungoli et al. [28] proposed Adaptive Feature Fusion (AFF) to bolster the generalization capacity of deep learning models through adaptively regulating the fusion procedure of feature representation. Hou et al. [29] proposed a novel local-aware spatiotemporal attention network with multi-stage feature fusion based on compact bilinear pooling for human action recognition.

3. Methodology

In this section, we first describe how to symmetrically extract space and time features of depth sequences. Then, a Center Boundary Collaborative Canonical Correlation Analysis (CBCCCA) is designed to efficiently fuse the space and time features. Next, a spatio-temporal information filtration module is introduced to filter the redundant and irrelevant information from the fused space and time features to improve the discriminative performance of the fused space and time features. Finally, the filtered spatio-temporal features are fed into an SVM in order to recognize human actions. Figure 1 shows the overall flowchart of our method.

3.1. Symmetric Extraction of Space and Time Features

Spatial feature maps: As depicted in Figure 2a, Depth Motion Images (DMI) [18] characterize the spatial aspects of actions by identifying the most prominent appearance features from the same spatial location in all depth frames. This process results in a unified representation that encapsulates the specific spatial appearance of each action, offering distinct features for each action. The DMI is calculated as follows:

$$\mathit{DMI}(i,j)=\underset{t=1,\dots ,T}{MAX}\left(\mathit{I}(i,j,t)\right)$$

(1)

where $\mathit{I}(i,j,t)$ represents the pixel value at the tth depth frame at pixel position $(i,j)$. The variable t ranges from frame 1 to T, where T signifies the total number of frames. The pixel value in the $\mathit{DMI}$ is determined as the maximum value at the same pixel position, signifying the most salient appearance feature of the action within the depth sequence.

Temporal feature maps: The creation of Depth Temporal Maps (DTM) [13] is illustrated in Figure 2b. Each depth frame undergoes projection onto three views, and the resultant three 2D projection images are subsequently projected onto two axes. This projection involves computing the row and column sums for each 2D projection image, resulting in two 1D projection lists for each 2D projection image. With three projection views and two projection axes, a total of six 1D projection lists are obtained.

$${\mathit{list}}_{v,w}^k=\sum _{i=1}^W{\mathit{image}}_v^k(i,j)$$

(2)

$${\mathit{list}}_{v,h}^k=\sum _{j=1}^H{\mathit{image}}_v^k(i,j)$$

(3)

$${\mathit{DTM}}_{v,a}\left(k\right)={\mathit{list}}_{v,a}^k$$

(4)

where $a\in \{w,h\}$ is the projection axes, and $v\in \{f,s,t\}$ corresponds to the Cartesian views. ${\mathit{list}}_{v,a}^k$ signifies the projected list of the kth video frame on view v and axis a. ${\mathit{image}}_v^k$ represents the projection image on projection view v. W and H denote the width and height of the projection images, respectively. ${\mathit{DTM}}_{v,a}\left(k\right)$ refers to the kth row of the ${\mathit{DTM}}_{v,a}$.

Finally, we use the Histogram of Oriented Gradient (HOG) algorithm to characterize the local shape of each $\mathit{DMM}$ and $\mathit{DTM}$, forming space and time features denoted as ${\mathit{X}}_s$ and ${\mathit{X}}_t$, respectively.

3.2. Center Boundary Collaborative Canonical Correlation Analysis (CBCCCA)

In the feature extraction phase, we symmetrically compute space and time features. To further capture the complex interaction between space and time dimensions, we design a novel feature fusion algorithm called CBCCCA in this paper.

In traditional Canonical Correlation Analysis (CCA) [22], the features to be fused are projected into a common semantic space. CCA is designed to learn projection matrices that maximize the correlation between the projected samples. In this paper, we introduce a novel feature fusion algorithm CBCCCA. CBCCCA enhances the fusion of spatial features ${\mathit{X}}_s$∈${\mathbb{R}}^{D_s\times N}$ and temporal features ${\mathit{X}}_t$∈${\mathbb{R}}^{D_t\times N}$ by collaboratively learning center and boundary information in class feature spaces; it can be formalized as follows:

$$\begin{array}{cc}\hfill & \underset{{\mathit{W}}_s,{\mathit{W}}_t}{max}Trace\left({\mathit{W}}_s^{\mathrm{T}}({\mathit{C}}_{st}+\alpha {\mathit{B}}_{st}){\mathit{W}}_t\right)\hfill \\ \hfill & subjectto{\mathit{W}}_s^{\mathrm{T}}({\mathit{C}}_{ss}+\alpha {\mathit{B}}_{ss}){\mathit{W}}_s=\mathit{I};{\mathit{W}}_t^{\mathrm{T}}({\mathit{C}}_{tt}+\alpha {\mathit{B}}_{tt}){\mathit{W}}_t=\mathit{I}\hfill \end{array}$$

(5)

where ${\mathit{C}}_{st}$ is the cross-correlation matrix between two center information matrices from space and time features, and ${\mathit{C}}_{ss}$ and ${\mathit{C}}_{tt}$ are the auto-correlation matrices of center information matrices. ${\mathit{B}}_{st}$ is the cross-correlation matrix between two boundary information matrices from space and time features, and ${\mathit{B}}_{ss}$ and ${\mathit{B}}_{tt}$ are the auto-correlation matrices of boundary information matrices. ${\mathit{W}}_s$∈${\mathbb{R}}^{K\times D_s}$ and ${\mathit{W}}_t$∈${\mathbb{R}}^{K\times D_t}$ are the projection matrices for ${\mathit{X}}_s$ and ${\mathit{X}}_t$, and K is the number of projected dimensions. $\mathit{I}$ represents the identity matrix. $\alpha$ is a tuning parameter that collaboratively reconciles the marginal and central information. The sensitivity regarding $\alpha$ also indicates the level of noise in the training samples. A higher sensitivity value of $\alpha$ corresponds to a greater level of noise. We adopt the same solution in [22]; ${\mathit{W}}_s$ in Equation (5) can be solved from the generalized eigenvalue decomposition as follows:

$${\mathit{M}}_{sd}{\mathit{M}}_{dd}^{-1}{\mathit{M}}_{sd}^T{\mathit{W}}_s={\lambda }^2{\mathit{M}}_{ss}{\mathit{W}}_s$$

(6)

$${\mathit{M}}_{st}={\mathit{C}}_{st}+\alpha {\mathit{B}}_{st}$$

(7)

$${\mathit{M}}_{ss}={\mathit{C}}_{ss}+\alpha {\mathit{B}}_{ss}$$

(8)

$${\mathit{M}}_{tt}={\mathit{C}}_{tt}+\alpha {\mathit{B}}_{tt}$$

(9)

where $\lambda$ is the eigenvalue in the generalized eigenvalue decomposition. After computing ${\mathit{W}}_s$, solving for $\frac{{\mathit{M}}_{tt}^{-1}{\mathit{M}}_{st}^T{\mathit{W}}_s}{\lambda }$ yields ${\mathit{W}}_d$. Based on the projection matrices, we project the spatial features ${\mathit{X}}_s$ and temporal features ${\mathit{X}}_t$ into the common semantic space and concatenate them to obtain the fused spatio-temporal features $\mathit{X}$.

The key to CBCCCA is knowing how to determine the center information matrices and boundary information matrices. In this paper, we adopt high-confidence center matrices as the center information matrices and use the original space and time feature matrices as the boundary information matrices. In CBCCCA, the collaborative reconciliation of centre and margin information mitigates the effect of outlier samples on spatio-temporal fusion. In the presence of outlier samples within the sample space, the location of the sample distribution centers is affected by these outlier sample points (called pseudo-centres). Obviously, the presence of pseudo-centres may adversely affect the feature fusion effect. To address this issue, we compute the high-confidence centers of each sample space. Taking the spatial feature ${\mathit{X}}_s$ as an example, we first use the maximum likelihood method to estimate the parameters for each feature dimension of each class in ${\mathit{X}}_s$ as described below:

$$\begin{array}{c}\mathrm{In}L\left(\mu ,{\sigma }^2\right)={\displaystyle \sum _{i=1}^n}\mathrm{In}f\left(x_i\mid \left(\mu ,{\sigma }^2\right)\right)\hfill \\ =-\frac{n}{2}\mathrm{In}\left(2\pi \right)-\frac{n}{2}\mathrm{In}\left({\sigma }^2\right)-\frac{1}{2{\sigma }^2}{\displaystyle \sum _{i=1}^n}{\left(x_i-\mu \right)}^2\hfill \end{array}$$

(10)

where $x_i$ is the sample instance for each feature dimension of each class in ${\mathit{X}}_s$ and n is the number of sample instances. After obtaining $\widehat{\mu }$ and ${\widehat{\sigma }}^2$, i.e., the estimates of the mean $\mu$ and variance ${\sigma }^2$, we partition the feature dimension into three intervals using $\widehat{\mu }-\widehat{\sigma }$ and $\widehat{\mu }+\widehat{\sigma }$ as watersheds. The central interval [$\widehat{\mu }-\widehat{\sigma }$, $\widehat{\mu }+\widehat{\sigma }$] is considered a high-confidence interval, due to the fact that outlier sample points are typically in the minority and tend to lie outside of this interval. Subsequently, we compute the mean of the values in [$\widehat{\mu }-\widehat{\sigma }$, $\widehat{\mu }+\widehat{\sigma }$] at all dimensional locations and form them into a high-confidence center for the sample features. Finally, all samples in ${\mathit{X}}_s$ are replaced with the corresponding high-confidence centers to obtain the high-confidence center matrix.

3.3. The Spatio-Temporal Information Filtration Module

Multi-feature fusion strategy is the mainstream approach to improve the performance of recognition tasks. However, feature fusion operations inevitably generate a certain amount of redundant information. In this paper, we design a spatio-temporal information filtration module. On the one hand, the spatio-temporal information filtration module is used to remove the redundant information introduced by feature fusion. On the other hand, the spatio-temporal information filtration module is also utilized to filter useless information that is not relevant to the HAR task, such as body type information.

The spatio-temporal information filtration module adopts the idea of information bottleneck theory [30]. The spatio-temporal information filtration module wants to obtain a filtered spatio-temporal feature $\mathit{M}=\{m_1,m_2,\dots ,m_N\}$, which is desired to have as large a correlation as possible with the label $\mathit{Y}=\{y_1,y_2,\dots ,y_N\}$ but as small a correlation as possible with the initial fusion feature $\mathit{X}=\{x_1,x_2,\dots ,x_N\}$. Keeping the correlation with $\mathit{Y}$ as large as possible ensures that $\mathit{M}$ has the ability to maximally retain useful information for action recognition. Keeping the correlation with $\mathit{X}$ as small as possible ensures that $\mathit{M}$ can maximally remove redundant information resulting from the fusion of spatio-temporal information. According to the information bottleneck theory [30], the spatio-temporal information filtration module can be formulated as follows:

$$\mathsf{\Omega }=I(\mathit{M},\mathit{Y})-\beta I(\mathit{M},\mathit{X})$$

(11)

where $\beta$ is Lagrange Multiplier. $I(\mathit{M},\mathit{Y})$ is the mutual information of $\mathit{M}$ and $\mathit{Y}$ and is denoted as:

$$I(\mathit{M},\mathit{Y})=\int p(y,m)log\frac{p(y\mid m)}{p\left(y\right)}\mathrm{d}y\mathrm{d}m$$

(12)

where $p(y,m)$ denotes the joint probability of m and y, $p\left(y\right)$ denotes the probability of y, and $p(y\mid m)$ is the probability of y under condition m. Due to the uncertainty of m and the different dimensions of m and y, $p(y\mid m)$ cannot be computed directly. We design an encoder $f(·)$ and a decoder $g(·)$ for $\mathit{X}$ such that $f\left(x\right)=m^{\prime }$, $g\left(m^{\prime }\right)=m^{″}$. Both the encoder and decoder consist of multilayer perceptrons. We compute $p(y\mid m^{″})$ as an approximation to $p(y\mid m)$ (where $p(y\mid m^{″})\le p(y\mid m)$). Thus, $I(\mathit{M},\mathit{Y})$ can be written as:

$$\begin{array}{c}I(\mathit{M},\mathit{Y})\ge \int p(y,m)log\frac{p(y\mid m^{″})}{p\left(y\right)}\mathrm{d}y\mathrm{d}m\hfill \\ =\int p(y,m)logp(y\mid m^{″})\mathrm{d}y\mathrm{d}m-\int p(y,m)logp\left(y\right)\mathrm{d}y\mathrm{d}m\hfill \end{array}$$

(13)

Since our method is a type of supervised learning, the label y is known. Thus, $p\left(y\right)=1$ and $\int p(y,m)logp\left(y\right)\mathrm{d}y\mathrm{d}m=0$. Meanwhile, the structure of the spatio-temporal information filtration module can be viewed as $\mathit{X}\to \mathit{M}\to \mathit{Y}$. According to the Markov chain theory [31], $p(y,m)=p\left(x\right)p(y\mid x)p(m\mid x)$. Therefore, Equation (13) can be written as:

$$I(\mathit{M},\mathit{Y})\ge \int p\left(x\right)p(y\mid x)p(m\mid x)logp(y\mid m^{″})\mathrm{d}x\mathrm{d}y\mathrm{d}m$$

(14)

As in Equation (12), $I(\mathit{M},\mathit{X})$ can be written as:

$$\begin{array}{c}I(\mathit{M},\mathit{X})=\int p(x,m)log\frac{p(m\mid x)}{p\left(m\right)}\mathrm{d}x\mathrm{d}m\hfill \\ =\int p(x,m)logp(m\mid x)\mathrm{d}x\mathrm{d}m-\int p(x,m)logp\left(m\right)\mathrm{d}x\mathrm{d}m\hfill \end{array}$$

(15)

Due to the uncertainty of m, $p\left(m\right)$ is not computable. We compute $p\left(m^{\prime }\right)=p\left(f\left(x\right)\right)$ as an approximation of $p\left(m\right)$. According to $KL\left[p\left(m\right)\right|\left|p\left(m^{\prime }\right)\right]\ge 0$, we obtain the following:

$$\int p(x,m)logp\left(m\right)\mathrm{d}x\mathrm{d}m\ge \int p(x,m)logp\left(m^{\prime }\right)\mathrm{d}x\mathrm{d}m$$

(16)

Thus, Equation (15) can be expressed as:

$$\begin{array}{c}I(\mathit{M},\mathit{X})\le \int p(x,m)logp(m\mid x)\mathrm{d}x\mathrm{d}m-\int p(x,m)logp\left(m^{\prime }\right)\mathrm{d}x\mathrm{d}m\hfill \\ =\int p(x,m)log\frac{p(m\mid x)}{p\left(m^{\prime }\right)}\mathrm{d}x\mathrm{d}m\hfill \\ =\int p\left(x\right)p(m\mid x)log\frac{p(m\mid x)}{p\left(m^{\prime }\right)}\mathrm{d}x\mathrm{d}m\hfill \end{array}$$

(17)

The association of Equation (14) with Equation (17) yields:

$$\begin{array}{c}\mathsf{\Omega }\ge \int p\left(x\right)p(y\mid x)p(m\mid x)logp(y\mid m^{″})\mathrm{d}x\mathrm{d}y\mathrm{d}m\hfill \\ -\int p\left(x\right)p(m\mid x)log\frac{p(m\mid x)}{p\left(m^{\prime }\right)}\mathrm{d}x\mathrm{d}m\hfill \end{array}$$

(18)

Let $L^{\prime }=-(\int p\left(x\right)p(y\mid x)p(m\mid x)logp(y\mid m^{″})\mathrm{d}x\mathrm{d}y\mathrm{d}m-\int p\left(x\right)p(m\mid x)log\frac{p(m\mid x)}{p\left(m^{\prime }\right)}\mathrm{d}x\mathrm{d}m$). According to [32], $L^{\prime }$ can be approximated as follows:

$$\begin{array}{c}{L}^{\prime }\approx L=\frac{1}{N}{\displaystyle \sum _{n=1}^N}\left[-logp\left(y\mid m^{″}\right)\right]+\beta \mathrm{KL}\left[p\left(m\mid x\right)\left|\right|p\left(m^{\prime }\right)\right]\hfill \\ =\frac{1}{N}{\displaystyle \sum _{n=1}^N}\left[-logp\left(y\mid g\left(f\right(x\left)\right)\right)\right]+\beta \mathrm{KL}\left[p\left(m\mid x\right)\left|\right|p\left(f\left(x\right)\right)\right]\hfill \end{array}$$

(19)

where $\mathrm{KL}\left[p\left(m\mid x\right)\left|\right|p\left(f\left(x\right)\right)\right]$ is the Kullback–Leibler Divergence of $p\left(m\mid x\right)$ with $p\left(f\right(x\left)\right)$. According to Equation (18), L has a lower bound. We use L as a loss function to train the encoder and decoder. The initial fusion features $\mathit{X}$ are fed into the encoder, and then we can obtain the filtered spatio-temporal features $\mathit{M}$.

Finally, we send the filtered spatio-temporal features $\mathit{M}$ to an SVM for action recognition.

4. Experiments

This section provides an overview of the datasets used in this paper. To evaluate the effectiveness of our proposed method, we select three widely recognized HAR datasets: the MSR Action3D [14], the UTD-MHAD [15], and the DHA [16]. Then, we thoroughly compare the proposed method with state-of-the-art methods. Finally, a series of ablation studies are implemented to further validate the contribution of each component of the proposed method.

4.1. Datasets

In order to simulate the HAR task in HRM scenarios, three widely used indoor human action datasets are selected for comparison experiments in this paper.

The MSR Action3D dataset [14] is one of the most popular action recognition datasets and contains 567 sequences formed by 10 subjects performing 20 different human actions. We use the same cross-subject testing configuration as in [14], using odd subjects for training and even subjects for testing.

The UTD-MHAD dataset [15] yields 861 samples of depth sequences containing 27 actions from eight subjects. Each subject performs each action four times. Our experimental setup is consistent with the configuration introduced in [15], i.e., half of the subjects are assigned to the training set and the remaining half constitute the test set. Figure 3 shows a depth video sequence of bowling from the UTD-MHAD dataset as an example of action samples.

The DHA dataset [16] has 483 depth sample sequences representing 23 different movements performed by 21 subjects. Each subject carried out each of the 23 actions once. We followed the experimental setup in [16], which contains all 23 actions. The actions of the odd-numbered subjects were used in the training set, and the actions of the even-numbered subjects were used in the test set.

4.2. Comparison with the State-of-the-Art Methods

To validate the effectiveness of our method, comparative experiments with state-of-the-art methods are conducted on three widely used human action datasets.

4.2.1. MSR Action3D Dataset

In the last decade, there are many Depth-HAR methods that validate their performance on the MSR Action3D dataset, including ROP, DMM-HOG, BSC, HP-DMM-CNN, P4Transformer, and PointMapNet. We summarize the results of these methods in

Table 1. Recognition results of different methods on the MSR Action3D dataset.

Methods	Accuracy
Baseline [14]	74.7%
ROP [37]	86.50%
DMM-HOG [10]	88.7%
Actionlet ensemble [38]	88.2%
HON4D [39]	88.9%
DSTIP [40]	89.30%
Tran et al. [41]	91.9%
BSC [42]	90.36%
Rang-Sample Feature [43]	95.62%
Kamel et al. [12]	94.51%
HP-DMM-CNN [12]	92.31%
GMHI + GSHI + CRC [11]	94.5%
Ahmad et al. [44]	87.88%
DDPDI [45]	96.15%
Azad et al. [46]	95.24%
PointLSTM-late [33]	95.38%
P4Transformer [34]	90.94%
PointMapNet [35]	91.91%
CBBMC [13]	96.3%
SequentialPointNet [36]	92.6%
Proposed	97.1%

. As shown in Table 1, our method achieves a recognition accuracy of 97.1% in the cross-subject setting, which is the highest compared to other methods. In the Baseline method [14], a straightforward yet efficient projection-based sampling scheme is proposed for extracting a set of 3D points from depth maps. Compared with the baseline method, we improve the extraction of space and time features and design a fusion and filtering scheme for space and time features to enhance the recognition performance. It is worth noting that, like our methods, DMM-HOG [10], and HP-DMM-CNN [12] also convert depth videos into feature maps and then perform HAR. However, these methods only focus on the extraction of spatial or temporal features individually and do not capture the complex interaction of temporal and spatial information. Recently, point cloud processing techniques have become more advanced. Point cloud sequence network methods are an extension of static point cloud techniques in 3D space to 4D time-space; they directly consume spatially disordered and temporally ordered point cloud sequences to identify human actions. However, point cloud sequence network methods, like PointLSTM-late [33], P4Transformer [34], PointMapNet [35], and SequentialPointNet [36], require a large number of samples and high computational cost for model training, which hinders the application of HAR techniques in HRM scenarios.

4.2.2. UTD-MHAD Dataset

The UTD-MHAD dataset is another widely used human action dataset. We compare the proposed method with a series of Depth-HAR methods, including Baseline [15], 3DHOT-MBC [47], HP-DMM-HOG [12], DTMMN [48], PointMapNet [35], and CBBMC [13]. The results of the comparison experiments are tabulated in

Table 2. Recognition results of different methods on the UTD-MHAD dataset.

Method	Accuracy
Baseline [15]	66.1%
3DHOT-MBC [47]	84.4%
HP-DMM-HOG [12]	73.72%
DMI [18]	82.79%
Yang et al. [49]	88.37%
DTMMN [48]	93.0%
PointMapNet [35]	91.6%
CBBMC [13]	94.4%
Proposed	95.3%

. We can see from the table that our method has a huge lead over other methods. In the Baseline method [15], the DMM for the three projected views are computed and form a feature vector. Then, Principal Component Analysis (PCA) is used to reduce the dimension of the spliced features. However, characterizing human action using only the DMM cannot capture the complex interaction of temporal and spatial information. Similar to our method, CBBMC [13] symmetrically extracts space and time features and effectively fuses them together but does not perform information filtration on the fused space and time features. The information filtration operation used in our method removes redundant and irrelevant information and improves the recognition performance.

4.2.3. DHA Dataset

On the DHA dataset, we compare the proposed method with a series of Depth-HAR methods, including D-STV/ASM [16], DMM-LBP-DF [17], D-DMHI-PHOG [50], DMMS-FV [51], and CBBMC [13]. As shown in

Table 3. Recognition results of different methods on the DHA dataset.

Method	Accuracy
Baseline [16]	86.8%
DMM-LBP-DF [17]	91.3%
SDM-BSM [52]	89.5%
D-DMHI-PHOG [50]	92.4%
DMMS-FV [51]	95.44%
CBBMC [13]	95.6%
Proposed	96.2%

, our method obtains the best performance by a large margin, which further validates the effectiveness of the proposed method. In the Baseline method [16], the depth video is represented as overlapping spatio-temporal volumes. Each spatio-temporal volume is divided into a number of cubes and represented as descriptors. Then, a spatio-temporal feature vector is constructed by connecting the descriptors of all the cubes in the volume. The proposed method significantly improves the discriminability of spatio-temporal information compared to the Baseline method. The key to the success of our method lies in the effective fusion of symmetrically extracted spatio-temporal features. In this paper, the sample class center and boundary information are learned collaboratively, leading to the efficient fusion of spatio-temporal features while fully exploiting the complex interactions between them. In addition, the creative design of the spatio-temporal information filtration module further purifies the fused spatio-temporal information, which significantly improves the classification effect of the spatio-temporal features.

4.3. Ablation Study

In this section, comprehensive ablation experiments are conducted to affirm the contribution of various components in our methodology.

4.3.1. Effectiveness of Symmetric Extraction of Space and Time Information

In the feature extraction phase, we symmetrically extract the space and time features of human actions. To validate the contribution of symmetrically extracting space and time information, we conduct experiments using only spatial or temporal features. It is essential to note that in this ablation study, the feature fusion algorithm is not utilized in any of the tests to ensure fairness, while the information filtration module is used. As shown in Figure 4, when using only spatial features, our method achieves a recognition rate of 92.1%. Using only temporal features results in a recognition rate of 87.8%. The symmetric use of both space and time features leads to a 3.5% increase in recognition accuracy, reaching 95.6%. The experimental results substantiate that symmetrically extracting space and time features significantly enhances the accuracy of HAR.

4.3.2. Fusion Performance Analysis of CBCCCA

In this paper, we present CBCCCA, a feature fusion algorithm designed to integrate space and time features. CBCCCA alleviates the effect of noisy samples on spatio-temporal fusion by reconciling central and marginal information in the sample space. In this subsection, we perform a sensitivity analysis on the parameters of CBCCCA. Subsequently, we qualitatively analyze the fusion performance of CBCCCA. Finally, we evaluate CBCCCA against other feature fusion algorithms to validate its effectiveness.

Parameter sensitivity analysis: CBCCCA contains the parameter $\alpha$ that coordinates the marginal and central information in the sample space. The values of $\alpha$ are in the range of {0.01, 0.1, 1, 10, and 100}. The performance of CBCCCA under different values of $\alpha$ varies, as shown in Figure 5. The best performance is achieved when $\alpha$ is set to 0.1. This optimal performance stems from the role of $\alpha$ in reconciling marginal and central information. Too small a value fails to describe the class-centred information, which is critical to mitigating the effects of outlier samples. Conversely, too large a value weakens the decision margin information required for effective categorisation. Therefore, we fix the value of $\alpha$ to 0.1.

Qualitative analysis of fusion performance: In order to qualitatively analyze the temporal and spatial fusion effects, we show a visualisation of the semantic space of CBCCCA in Figure 6a. As a reference, simple fusion features obtained by direct splicing of space and time features are also visualized in Figure 6b. In this subsection, t-SNE [53] is used to reduce the high-dimensional semantic features to 2D features for visualization. Comparison experiments are performed on the MSR Action3D dataset, and the first eight classes of samples are visualized. From the figure, we can see that the fusion features produced by CBCCCA have smaller intra-class distances and larger inter-class distances in the semantic space than the simple fusion features. Therefore, we can obtain a conclusion that the fusion features obtained through CBCCCA are more discriminative than simple fusion features, thus indicating that CBCCCA has excellent feature fusion performance.

Comparison with other fusion technologies: To further evaluate the fusion performance of CBCCCA, we compare CBCCCA with several classical fusion algorithms. The fusion algorithms involved include CCA [22], MeanCCA [23], and JFSSL [54]. The comparison experiments are performed on the MSR Actions3D and shown in

Table 4. Comparison of different fusion technologies on the MSR Action3D dataset.

Method	Accuracy
CCA [22]	93.1%
MeanCCA [23]	94.6%
JFSSL [54]	95.7%
CBCCCA	97.1%

. It should be noted that all tests in this experiment keep the same settings except for the fusion algorithms. Impressively, the proposed CBCCCA reaches the top accuracy of 97.1%. Compared with other fusion algorithms, CBCCCA collaboratively exploits the marginal and central information of the sample space and decreases the impact of outlier samples on spatio-temporal fusion.

4.3.3. Effectiveness of the Spatio-Temporal Information Filtration Module

In this paper, the spatio-temporal information filtration module is designed to perform information purification on the fused spatio-temporal features, which removes redundant information while reducing useless information that is not relevant to HAR. We analyze the recognition results of the two HAR frameworks. Framework A represents the complete method proposed in this paper. Framework B is the HAR method without using the spatio-temporal information filtration module. As shown in

Table 5. Recognition results with and without spatio-temporal information filtration module.

Method	Accuracy
Framework B	96.0%
Framework A	97.1%

, the spatio-temporal information filtration module improves the recognition accuracy by 1.1%, which indicates that the spatio-temporal information filtration module can effectively contribute to the HAR performance.

4.3.4. Computational Efficiency Analysis

In order to analyze the real-time running efficiency of the proposed method, we show the running time of each step. The experimental results are shown in

Table 6. Running time per sample video on the MSR Action3D dataset.

Step	Time
Feature extraction	0.024 s
Feature fusion	0.002 s
Feature Filtration	0.007 s
Feature classification	0.056 s
Total	0.089 s

. It should be mentioned that the running time of all the modules is the model forward inference time. The experiment is conducted on a machine with an Intel(R) Xeon(R) W-3175X CPU.

From the table, we can see that the proposed method consumes only 0.089 s to classify a depth video much more than the real-time running requirement. Excellent running efficiency makes it possible to deploy the proposed approach in real time in HRM scenarios.

5. Discussion

In this section, we first compare the proposed method with other spatio-temporal feature extraction techniques. Then, we discuss the application of our method in HRM scenarios. Finally, the limitations and potential challenges of our method are analyzed.

Spatio-temporal feature extraction techniques in gesture recognition: Knowing how to extract and utilize the space and time information of human actions is the core problem of this paper. In the experimental section, we compare our proposed method with popular Depth-HAR methods on three widely used human action datasets. In addition to the Depth-HAR methods, various advanced spatio-temporal feature extraction techniques have been proposed in recognizing specific action tasks, such as gesture recognition [55,56,57,58,59]. In [55], seven types of spatio-temporal features such as 2D distances from face to hands and areas of face and hands intersection are combined into a single vector, which is then sent to the BiLSTM network for gesture recognition. Jiang et al. [56] use a 3DCNN network to extract spatio-temporal features from depth videos. In [57], a novel neural ensembler is introduced, incorporating various models (I3D, TimeSformer, and SPOTER) and diverse data sources (RGB, Masked RGB, Optical Flow, and Skeleton) for training the individual sources within the ensemble. These methods have been remarkably successful in recognizing specific actions and provide useful inspiration for subsequent improvements of our method.

Application in HRM scenarios: In this paper, we design a new Depth-HAR scheme for HRM scenarios. In HRM scenarios such as human resource recruitment (HRR), employee performance assessment (EPA), and employee training (ET), accurate and efficient HAR facilitates the development of each field. In HRR, HAR technology revolutionizes the interview process by analyzing candidates’ non-verbal cues, offering a comprehensive understanding of communication skills, emotional expression, and adaptability. This enhances the objectivity of candidate assessment, providing a scientific and objective approach to recruitment. In EPA, HAR methods offer crucial insights by monitoring workplace actions. This reduces subjective biases in evaluations, providing fair assessments and clear development directions for employees, fostering continuous improvement. In ET, HAR optimizes training effectiveness through personalized plans and real-time feedback. Tailored training content meets individual needs, while real-time feedback corrects unfavorable learning habits promptly, advancing talent development strategies. These applications showcase the versatile potential of HAR in HRM, introducing innovative and scientific approaches to several domain.

Limitations and potential challenges: Extensive experiments have verified that our method not only has outstanding HAR accuracy but also has excellent running efficiency. In this subsection, we try to discuss the limitations and potential shortcomings of our approach with respect to real-time deployment. The diversity of observational perspectives is a common challenge for all current HAR research. Due to differences in observation perspectives, sample data collected for the same action have large intra-class differences. In fact, we execute a series of operations in order to mitigate this issue. The depth video used in this paper is a type of 3D data that exerts a measure of observation perspective robustness. In addition, we perform the three-view projection for the depth video to simulate three different observation perspectives. However, it is worth mentioning that our approach does not completely avoid the impact of the diversity of observation perspectives on HAR, especially in the case of complex actions. In a practical deployment, cooperative sampling with multiple shooting positions would be an effective solution to this problem. However, it may degrade the real-time running performance of the proposed method.

6. Conclusions

HAR technology has shown promising versatile potential in numerous HRM scenarios. In this paper, we develop a novel HAR algorithm for HRM scenarios. To enhance the robustness and accuracy of the proposed algorithm, we symmetrically propose space and time features of human actions and design a feature fusion algorithm, CBCCCA, to capture the complex interactions between temporal and spatial dimensions. In addition, a creative spatio-temporal information filtration module is introduced to remove redundant and irrelevant information. We simulate human actions in HRM scenarios using three public human indoor action datasets and conduct extensive experiments. The experimental results verify that the proposed method has excellent recognition performance and real-time running efficiency. We can conclude that the proposed method is an excellent HAR scheme in HRM scenarios and introduces an innovative and scientific approach to HRM fields.

For future work, on the one hand, we will continue to explore more advanced HAR techniques, and on the other hand, we intend to further promote the development of human resource management based on HAR techniques.