Novel Cross-View Human Action Model Recognition Based on the Powerful View-Invariant Features Technique

Abstract

One of the most important research topics nowadays is human action recognition, which is of significant interest to the computer vision and machine learning communities. Some of the factors that hamper it include changes in postures and shapes and the memory space and time required to gather, store, label, and process the pictures. During our research, we noted a considerable complexity to recognize human actions from different viewpoints, and this can be explained by the position and orientation of the viewer related to the position of the subject. We attempted to address this issue in this paper by learning different special view-invariant facets that are robust to view variations. Moreover, we focused on providing a solution to this challenge by exploring view-specific as well as view-shared facets utilizing a novel deep model called the sample-affinity matrix (SAM). These models can accurately determine the similarities among samples of videos in diverse angles of the camera and enable us to precisely fine-tune transfer between various views and learn more detailed shared facets found in cross-view action identification. Additionally, we proposed a novel view-invariant facets algorithm that enabled us to better comprehend the internal processes of our project. Using a series of experiments applied on INRIA Xmas Motion Acquisition Sequences (IXMAS) and the Northwestern–UCLA Multi-view Action 3D (NUMA) datasets, we were able to show that our technique performs much better than state-of-the-art techniques.

1. Introduction

Human action data are all-encompassing, which is why such data are of significant interest to the computer vision [1,2] as well as machine learning communities [3,4]. There are various views through which we can study human action data. One example is a group of dynamic actions captured by different views of the camera, as shown in Figure 1. Classifying this kind of data is quite difficult in a cross-view situation because the raw data are captured at varying locations by different cameras and, thus, could look completely different. A perfect example is represented in Figure 1a, where the action captured from a top view appears different from the one captured from a side view. This implies that features obtained from a single camera view cannot provide enough discriminative aspects to classify actions in another camera view. Many studies concentrate on ways of developing view-invariant pictures for action recognition [5,6], where all the actions captured on video are treated as frame time series. Many approaches have utilized a self-similarity matrix (SSM) descriptor [5,7] to replay actions and have proven to be robust in cross-view outlines. Information shared among camera views are each kept and transferred to all the views [7,8]. It is assumed that the shared features contribute equally with samples from different views. Through our research, we found that this assumption is not true because the discriminative parameters of one of the views could be very far away from the parameters of other views. Therefore, this may result in a misunderstanding by classifiers, as they do not control the sharing of data between action categories, which would result in an incorrect model result.

As a response to the assumption of the equal contribution of share features, in this paper, we put forward original networks that can learn view-invariant features for cross-view action categorization, and we have introduced a novel sample-affinity matrix (SAM) that can accurately determine similarities of video samples. By encouraging incoherence between shared and private features, we learned discriminative view-invariant information. Our approaches retain two types of features—strong private features as well as shared features across views acquired by a single autoencoder. SAM focuses on the resemblance between samples, but SSM concentrates on the video frames. The vanity between these facets was achieved by strengthening the coherence between the mapping matrices. In addition, in a layer-wise fashion, we piled several layers of features in order to learn them. After a set of experiments was carried out on three multi-view sets of data, we found out that our method performs much better than state-of-the-art methods. The following pieces of information are covered in the next sections of this paper. We first analyze works related to our topic and follow this with an analysis of view-invariant features. Then, we present a detailed description of the structure of our method together with the algorithm used. Finally, through a set of experiments, we show that our method performs much better than state-of-the-art methods.

2. Research Problem Definition

It is difficult for various view-invariant methods to find similarity among various frames captured from different RGB camera (standard CMOS sensor camera) views at the same time. However, with an RGB-D camera (depth sensor added), this is not an issue because the camera provides a powerful feature that allows for easy extraction of the required feature from the plan (Figure 2). The only challenge is generating artificial views that have all the facets necessary for understanding the action. In this paper, we also introduce a novel algorithm based on sample affine matrix (SAM) and various powerful autoencoders that allows for extraction of shared and unshared facets needed for identifying human action.

3. Related Research

A multi-view study, as cited in [4], establishes the similarities between two different views. There are several other published methods used to serve the same purpose. These methods have been published with the intention of concentrating their interest on expressive as well as discriminative facets from low-level observations [9,10,11,12,13,14]. Similarly, [15] entirely used the intrinsic characteristics extracted from the views by combining color and depth information, which resulted in the improvement of their perspective-invariant feature transform (PIFT) for RGB-D images. However, [15] focused on the combined used of the RGB and depth component and gave less attention to the features (shared and global) to be extracted from each viewpoint. Zhang et al. [16] obtained a dictionary that can be used to convert 2D video to a view-invariant sparse representation, as well as a classifier to recognize actions with an arbitrary view by using an end-to-end framework to learn view invariance jointly. The authors of [16] also introduced a 3D trajectory which can describe the action better; however, it does not emphasize the percentage of contribution of each view to that trajectory. Kerola et al. [17] took a temporal sequence of graphs as an illustration of a graph representation action and used that to create a feature descriptor by applying a spectral graph wavelet transform. The authors of [17] also emphasized two well-known types of view-invariant graphs—key point-based and skeleton-based graphs. Rahmani et al. [18] devised a histogram of oriented principal components (HOPC) that is robust to noise. Unlike several articles, [18] obtained cloud points by directly dealing with point clouds for cross-view action recognition from unknown and unseen views. Also, [19] proposed a different approach when extracted functionality is needed. Hsu et al. [19] used the Euclidean distance between the spatiotemporal characteristic vectors represented in a spatiotemporal matrix (STM). Daumé and Kumar [20] presented a cotraining technique that trains various learning algorithms for every view and determines a certain correlation between two pairs of information among various views. Zhang et al. [21] and He et al. [22] introduced another approach called canonical correlation analysis (CCA) that helps in monitoring a common space among various views. Moreover, in [23], a challenge was encountered when studying an incomplete view owing to an assumption that multiple views are created from a shared space. Kumar et al. [24] provided another approach called generalized multi-view analysis (GMA). According to Liu et al. [11], which found that matrix factorization was applicable in the clustering of multiple views, [10] was introduced alongside the collective matrix factorization (CMF) method, which learns relationships between feature matrices. Similarly, Ding et al. [14] came up with a low-rank controlled matrix factorization model that provides a solution to the challenge faced in the multi-view learning. Various studies have tried to provide solutions to the challenges associated with view-invariant action recognition. One of these challenges is related to the generation of action labels in a scenario where multiple views are involved. Özuysal et al. [25] introduced an approach that offers a structured categorization of the 3D histogram of oriented gradients (HOG) and local separation with an intention to represent successive images. Dexter and colleagues [5,26] showed an SSM-based approach that extracts view-invariant descriptors in a log-polar block of the matrix by determining a frame-wise resemblance matrix in a video. However, a multitask learning technique can be used to enhance the SSM power of the representation [7]. This technique comprises shared facets among different views as examined in [6,8,27,28], where more explicit approaches exist. Gao et al. [6] used an MRM-Lasso method to keep latent adjustment through various views. This was achieved by examining a lowly ranked matrix comprising weights of specific patterns. However, Jiang et al. [8] and Jiang and Zheng [29] introduced handy dictionary pairs that support the sparse common feature space. Compared to other methods [20,22,23,30], our approach enables us to keep multiple layers of learners and to examine view-invariant facets more effectively. In addition, it enables us to record complicated movements that exist in certain views. Our approach uses private facets and supports the inconsistencies among shared as well as private facets. Compared to other methods of sharing knowledge [6,8,27,31,32], our approach achieves the sharing of information among different views as per sample similarities. Since samples of data can appear the same in some views, our method provides a solution that enables us to distinguish different classes. While [30] calculated between and within classes of Laplacian matrices, SAM Z approach takes directly within and between classes. Moreover, the space between two views of one sample is determined using SAM Z. Such a distance cannot be encoded by [30].

4. Sharp Study of View-Invariant Features

4.1. Sample-Affinity Matrix (SAM)

SAM is used here to determine the similarities among views (a pair of videos). Assume we have as input two training videos with N views: {X^t,y^t}^t_N=1, where the data of the view frame at the interval of time t, X^t has of N actions videos X^t = [x₁^t, ···, x^t_N] ∈ R^d×N with their associate labels y^t = [y^t₁, ···, y^t_N]. SAM Z ∈ R^{V N×V N} shows as a block diagonal matrix:

$$\mathit{Z}=\mathrm{diag}(Z_1,\dots ,Z_N),Z_i=\left(\begin{array}{cccc}0& Z_i^{12}& \dots & Z_i^{1V}\\ Z_i^{21}& 0& \dots & Z_i^{2V}\\ ⋮& ⋮& \ddots & ⋮\\ Z_i^{V1}& Z_i^{V2}& \dots & 0\end{array}\right)$$

(1)

NB: dial(.) generates a diagonal matrix, and $Z_i^{mn}$ shows the space between two view frames in the ith sample obtained by $Z_i^{mn}=\exp (\left|\left|Xn–Xm\right|\right|/2c$) parameterized by c. In different views within one class, the appearance of variations is characterized by the block Z_i in Z. This illustrates clearly how an action might be seen differently from different viewpoints and allows us to share information between views that results in the construction of robust cross-view features. Furthermore, the presence of 0 on the off-diagonal blocks in our SAM Z limits the transfer of information between classes in the same view, with the direct impact of encouraging the distance between the features from various classes having the same view frame. It also gives us the possibility of differentiating multiple action categories if they seem similar in some view frames.

4.2. Preliminary on Autoencoders

Based on popular deep learning approaches [33,34,35], we have built an autoencoder (AE) that links the raw input X to hidden unit H using a powerful “encoder” e1(.): H = e1(X) and uses the “decoder” e2(.): O = e2(H) to connect the concealed units to outputs. The objective of studying AE is to strengthen matching or related input and output pairs where the restoration failure is decreased once decoding is over:

$$\min {{\displaystyle \sum }}_{i=1}^N\parallel \mathrm{X}i-\mathrm{e}2\left(\mathrm{e}1\left(\mathrm{X}i\right)\right)\parallel {}^2$$

(2)

where N is the number of training samples. As the process of reconstruction keeps track of incoming information, neurons in the latent layer represent the inputs. On the other hand, the two-phase coding and decoding in autoencoders [33] is emphasized on the marginalized stacked denoising autoencoder (mSDA), which is used to reap the ruined information by use of one mapping W:

$$\min {{\displaystyle \sum }}_{i=1}^N\parallel \mathrm{Y}i-\tilde{\mathrm{a}}i \parallel {}^2$$

(3)

where $\tilde{\mathrm{a}}$ represents the corrupted version of our Yi and computes by assigning 0 to each feature with a given probability p. Regarding the mSDA method, let us understand that for achieving a better result, it needs to pass n times over the training set with a different corruption each time. This causes a nonconformity regularization [36]. In the objective to achieve a robust transformation matrix W, n is set as n → ∞ so that mSDA will effectively use an infinite number of noisy data copies. Furthermore, mSDA is solved in closed form and is also stackable.

4.3. Single-Layer Feature Learning

As mentioned so far, our proposed method is based on mSDA. We aimed to get the shared facets among private features, especially those that belong to a single view frame and multiple view frames for cross-view action recognition categorization. Furthermore, in order to construct more robust features that are aware of the very large motion difference in diverse view frames, we introduced SAM Z to learn shared facets with the objective of equating information among view frames.

By the help of the objective function below, we have learned private and shared features:

$$\begin{gathered} \underset{W,\left\{G^v\right\}}{\min }\Delta ,\Delta ={\left|\left|W\tilde{X}-XZ\right|\right|}_F^2+{{\displaystyle \sum }}_v[\propto {\left|\left|G^v{\tilde{X}}^v-X^v\right|\right|}_F^2+\beta \left|\left|W^T{G}^v\right|\right|{}_F^2+ \\ \delta T_r\left(P^v{X}^{v}L{X}^{vT}{P}^{vT}\right)] \end{gathered}$$

(4)

where W and {G^v}^V_v=1 are, respectively, the mapping matrix used to learn shared features and a collection of mapping matrices used to learn private features of each view frame, and p^v can be expressed as p^v = (W;G^v). Equation (4) Consists of four expressions: $\mathsf{\psi }$ = ${\left|\left|W\tilde{X}-XZ\right|\right|}_F^2$ is used to learn shared features (SF) among view frames. SF view is particularly used in the reconstruction process of the action data taken from a single view frame with the help of the data extracted from other view frames. To ascertain certain unshared facets that seem similar to shared features, the second terms is φv = ${\left|\left|G^v{\tilde{X}}^v-X^v\right|\right|}_F^2$. Nevertheless, $r_{1v}$ = $\left|\left|W^T{G}^v\right|\right|{}_F^2$, the third term, minimizes redundancies between the mapping matrices, and $r_{2v}$ = Tr (Pv Xv LXvT PvT), as the fourth term, emphasizes the similarity of the private and shared features belonging to the same class and view frame. The parameters α, β, and γ are discussed below. However, we can introduce them as elements used to balance the components discussed beforehand. It is important to note that data obtained from all view frames in cross-view action recognition are availed in training so that our model can learn private and shared features. However, with the testing phase, data collected from some view frames are not present.

4.3.1. Shared Features

The action recognition ability found in humans is easily understood from a single view, but does how the action appear if observed from multiple views? This is possible, since we regularly observe the same action from several views. This problem is one of the reasons why we tried to restore the action data view from one point with the help of action data from multiple views. Considering $\mathsf{\psi }$, the disparity between the data of all the V source views and the data of the $v\mathrm{th}$ target view can be expressed as:

$$\mathsf{\psi }={{\displaystyle \sum }}_{i=1}^N{{\displaystyle \sum }}_{v=1}^V{\left|\left|W{\tilde{X}}_i^v-{{\displaystyle \sum }}_u{X}_i^u{Z}_i^{uv}\right|\right|}^2={\left|\left|W{\tilde{X}}_{ }^{}-X^{ }Z\right|\right|}_F^2$$

(5)

where $Z_i^{uv}$ represents the value measuring the endowment of the u-th view action in the remodeling of the sample $X_i^v$ of the $v\mathrm{th}$ view frame. Also, W is a single linear mapping for the corrupted input ${\tilde{X}}_i^v$ of all the views frames with W ∈ ${ℝ}^{d\ast d}$. From the sample-affinity matrix, which encodes all the values {$Z_i^{uv}$}, also called weights, we have Z ∈ ${ℝ}^{VN\ast VN}$. It is good to mention that the corrupted version of X matrices, which is $\tilde{X}$ ∈ ${ℝ}^{d\ast VN}$ [33], performs a drop out regularization on the model [33,36]. Furthermore, to accurately regulate the information transfer among view frames and learn more easily discriminative shared features, SAM Z is used. As an alternative of using equal values (weights) [8], all the samples are used, so that we remodel the i-th training sample taking from the $v\mathrm{th}$ view. Let us note, by the help of Figure 3, a greater similarity will be found between a sample of side view frame (S1) and side view t (which is the target view) compared to the one found between S1 and top view frame (S2). These result in the increase of weight for S1 with the objective to learn more expressive features for t.

4.3.2. Private Features

Selective information can still be found in each view despite the information shared across view frames. By improving the robustness of that information, in [33], we have used robust feature learning, and by using a matrix $G^v$ ∈ ${ℝ}^{d\ast d}$, we have “learned view specific private features” for the samples in $v\mathrm{th}$ view frame:

$${\varnothing }_v={{\displaystyle \sum }}_{i=1}^N{\left|\left|G^v{\tilde{X}}_i^v-X^v\right|\right|}^2={\left|\left|G^v{\tilde{X}}_i^v-X^v\right|\right|}_F^2$$

(6)

where the feature matrix $X^v$ has for corrupted version ${\tilde{X}}^v$ of the $v\mathrm{th}$ view. Using associate inputs of different view frames, we obtained the V mapping matrices ${\left\{G^v\right\}}_{v=1}^V$. We should take into consideration that Equation (6) can keep some redundant shared information from $v\mathrm{th}$ view, but by promoting the inconsistency among the view-specific mapping matrix $G^v$ and view-shared mapping matrix W, we reduce some redundancy in our project:

$$r_{1v}={\left|\left|W^T{G}^v\right|\right|}_F^2$$

(7)

4.3.3. Label Information

We should keep in mind that action data collected from different viewpoints may come with a considerable proportion of posture and motion variations. Thus, features (private and shared) collected by applying Equations (5) and (7) could appear not sufficiently selective for classifying action with considerable variation. Facing this issue, our approach leads to the same view and the same class to enforce similarity between private and shared features. To normalize view-specific mapping matrix $G^v$ and view-shared mapping matrix W, we have defined a “within class” and “within view frame” variance as:

$$\begin{gathered} r_{2v}={{\displaystyle \sum }}_{i=1}^N{{\displaystyle \sum }}_{j=1}^N\left[{\left|\left|G_{ }^v{X}_i^v-G_{ }^v{X}_j^v\right|\right|}^2+{\left|\left|W{X}_i^v-W{X}_j^v\right|\right|}^2\right] \\ =T_r(G^v{X}^{v}L{X}^{vT}{G}^{vT})+T_r\left(W{X}^{v}L{X}^{vT}{W}^T\right) \\ =T_r\left(P^v{X}^{v}L{X}^{vT}{P}^{vT}\right) \end{gathered}$$

(8)

where L ∈ ${ℝ}^{N\ast N}$ and L = D − A is label view the Laplacian matrix with degree matrix D(i,j) = ${{\displaystyle \sum }}_{i=1}^N{\mathrm{a}}_{\left(i,j\right)}$ and the adjacent matrix A. Note that the (i,j)-th element: a(i,j) in A is 1 if $y_i=y_j\mathrm{or}0,\mathrm{if}{y}_i\ne y_j$. As we have implicitly required facets from various view frames to be similar in Equation (5), we do not need it here. Furthermore, by using facets from different view frames of a given sample, we can better illustrate expected facets of a given sample. This can be possible due to the mapping function (based on mapping matrix W) applied on the shared features (SF) so that the SF is mapped to a new space. Base on label information results obtained from Equation (8), in a supervised approach (SA) and by considering γ = 0, we can derive an unsupervised equation. In the following lines, a supervised approach is renaming.

4.4. Learning Process

Using a coordinate convergent algorithm, we are able to optimize parameters W and the V mapping matrices ${\left\{G^v\right\}}_{v=1}^V$. Our process resolves the optimization problem in Equation (5). Furthermore, after determining the derivative of Δ_w.r.t to the parameter and allocating 0 to it, one parameter matrix within every step is updated by setting the others. First of all, we update W by fixing the derivative, $\frac{d\Delta }{dW}=0$ as:

$$W={\left[{{\displaystyle \sum }}_v\left(\beta X^{v}L{G}^{vT}+\gamma G^v{G}^{vT}+I\right)\right]}^{-1}(XZ{\tilde{X}}^T)[\tilde{X}{\tilde{X}}^T+I{]}^{-1}$$

(9)

Keep in mind that in updating W, matrices ${\left\{G^v\right\}}_{v=1}^V$ are fixed; also by performing m → $\infty$ times the corruption, we obtain after computation F₁ and F₂ for $\tilde{X}{\tilde{X}}^T$ and XZ${\tilde{X}}^T$, respectively. As referred to in [33], the weak law of large numbers applied on $\tilde{X}{\tilde{X}}^T$ and XZ${\tilde{X}}^T$ gave us the computation values: F₁ = E_p ($\tilde{X}{\tilde{X}}^T$) and F₂ = E_p (XZ${\tilde{X}}^T$), where p is the corruption probability.

Similar to the previous update schema, by fixing the derivative $\frac{d\Delta }{d{G}^v}=0$, parameter $G^v$ is updated with ${\left\{G^u\right\}}_{u=1,u\ne 1}^V$ and W sets with default value. Furthermore, $X^v{\tilde{X}}^{vT}\mathrm{and}{\tilde{X}}^v{\tilde{X}}^{vT}$ computation values are obtained by applying E_p (expectation with corruption p):

$$G^v={\left(\beta X^{v}L{X}^{vT}+\gamma W^{ }{W}^T+I\right)}^{-1}(\mu X^v{\tilde{X}}^{vT})[\mu {\tilde{X}}^v{\tilde{X}}^{vT}+I{]}^{-1}$$

(10)

As a resolution approach of Equation (1)’s problem, let us subdivide it into V + 1 subproblems, where by considering one variable, each one is a convex problem. Thus, an optimal solution to each subproblem is surely found by using the learning algorithm which will consequently converge to a local solution.

Every mentioned method has some advantages and disadvantages where need to be taken into account when designing solution as shown in table (

Table 1. Advantages and disadvantages of used approaches.

Approach	Advantages	Disadvantages
Multi-view learning approach [9,10,11,12,13,14]	Focuses on expressive and discriminative features	Does not focus much on private features
Cotraining method [20]	Trains various learning algorithms for every view and finds explicit correlation of two pairs of information among various views	Cannot handle more than two views simultaneously
[21,22]	Maintains common distance between views, and utilizes the two projection matrices on a common feature space in order to map multimodal information	Has little interest in private features
[25] method	It achieves a structured categorization of the 3D histogram of oriented gradients (HOG).	It does not keep enough layers of learners.
[30] method	Calculates between-class as well as within-class Laplacian matrices	Does not measure the space between two views of the same sample.
Our approach	It can keep several layers of learners so as to study view-invariant features in a more effective manner	Because of the large amount of computation involved, the approach can process fewer views
	It equipoises the sharing of information among views, as per sample similarities	Requires the use of various computer resources.
	Measures the distance between two views using SAM Z

). We are also defining our own approach as the alternative.

5. The Design of the Proposed Approach—Deep Architecture

From the papers [33,37], we find a deep model developed by piling multiple layers facets discussed earlier in single layer feature learning where we utilized the nonlinear feature mapping function $\mathsf{\theta }(.)$ on the output of every layer, so that for $C_g^v=\mathsf{\theta }\left(X^v{G}^v\right)$ and $C_w=\mathsf{\theta }\left(XW\right)$, the outcome is a series of matrices of latent features. We utilized a “layer wise” training approach to train the networks ${\left\{G_k^v\right\}}_{v=1, k=1}^{V,K}$ and ${\left\{W_k\right\}}_{k=1}^K$ that have K layers. It is imperative to note that the input of the (k + 1)th layer is the output of the nth layer $C_{kg}^v\mathrm{and}{C}_{kw }.$ This gives the input of our matrix ${\left\{G_{k+1}^v\right\}}_{v=1 }^V$ and $W_{k+1}$. Moreover, since k = 0 (implying layer 1 because there are $K$ layers), X and $X^v$ have for raw features $C_{0g}^v\mathrm{and}{C}_{0w}$ in that order.

5.1. Flow Chart of our Project

Figure 3 shows the steps of our project. These steps are described below:

• Take three videos from different perspectives at the same time: Here, we try to capture images of a person from varying angles.
• We then obtain key features from the captured pictures by utilizing Equations (5)–(7): The pictures obviously have various features in common because they belong to one subject and they were taken at the same time. These features are called shared features, while the unique features that every picture has are called private features. We submit these two types of features to the next component as input.
• Applying a novel invariant feature algorithm: This step is a learning point pertinent for the process.
• Create the target views: In this step, we solve the sample-affinity matrix Z for every arrow. We also solve the W mapping matrix and create the target view having all the relevant features that will help in understanding the action.
• Allocate a label and an explanation of the action taking place.

5.2. Novel View-Invariant Features Algorithm

While in this paper we are targeting to present a powerful view-invariant feature, considering two subjects acting in the same environment as shown in figure (Figure 4a), our algorithm selects the zone of interest, extracts share, and private features as shown in figure (Figure 4b). It uses SAM to determine the similarities among views (a pair of videos). Additionally, figure (Figure 4c) shows a Clear mapping among view as described by our computed W mapping matrix.

Using our supervising technique, we determined the similarity between our target view and source views (Figure 5). A view shared mapping matrix W can be used to accomplish this process. The matrix is incorporated into Equation (5) to calculate weight value (Z) and get the shared feature. The arrow with the largest weight is the one situated between our target view and source view.

Here (Figure 6), we derived a supervised approach from our previous one. W is the mapping matrix updated through several iterations (as show in our algorithm), while the shared and private features are determined. We aimed through our approach to determine or generate a target view (obtained from several viewpoints after applying W matrix) accurate enough so that the weight (Z) of the arrow will be same as the one obtained after applying Equation (5):

Input: ${\left\{\left(X_i^v,y_i\right)\right\}}_{\mathit{v}=\mathbf{1},\mathit{i}=i}^{V,N}$

Output: ${\left\{{\mathit{G}}_{\mathit{k}}^{\mathit{v}}\right\}}_{\mathit{v}=\mathbf{1},\mathit{k}=\mathbf{1}}^{\mathit{V},\mathit{K}}$, ${\left\{{\mathit{W}}_{\mathit{k}}\right\}}_{\mathit{k}=\mathbf{1}}^{\mathit{K}}$

i$\leftarrow$ 0

While Layer i ≤ k do

Input ${\mathit{C}}_{\mathit{w}\mathit{i}}$ for learning Wk.

Input ${\mathit{C}}_{\mathit{g}\mathit{i}}^{\mathit{v}}$ for learning ${\mathit{G}}_{\mathit{k}}^{\mathit{v}}$

Do

Update Wk applying (9);

Update ${\left\{{\mathit{G}}_{\mathit{k}}^{\mathit{v}}\right\}}_{\mathit{v}=\mathbf{1}}^{\mathit{V}}$ applying (10);

While converge

Compute ${\mathit{C}}_{\mathit{w}\mathit{k}} \mathrm{by}:{\mathit{C}}_{\mathit{w}\mathit{k}}=\mathsf{\vartheta }({\mathit{C}}_{\mathit{w}\mathit{i}}{\mathit{W}}_{\mathit{i}})$.

Compute ${\{{\mathit{G}}_{\mathit{g}\mathit{k}}^{\mathit{v}}\}}_{\mathit{v}=\mathbf{1}}^{\mathit{V}}$ by: ${\mathit{C}}_{\mathit{i}\mathit{g}}^{\mathit{v}}=\mathsf{\vartheta }({\mathit{C}}_{\mathit{g}\mathit{i}}^{\mathit{v}}{\mathit{G}}_{\mathit{i}}^{\mathit{v}})$.

i $\leftarrow \mathit{i}+\mathbf{1}$

end while

In opposite of [3,6,8,27,28] where features extracted from views are used individually, we further process those features, and we derive a target view which contends the combined relevant features.

As specified in (Figure 4 and Figure 6). Our approach achieves state of the art result because we apply the recognition algorithm on the target view. Thus, it is done at this level as we have a single viewpoint.

6. Experiment

We evaluated our method using three multi-view datasets and the Daily and Sports Activities (DSA) dataset [3], multi-view IXMAS dataset [39], and the Northwestern–UCLA Multi-view Action 3D (NUMA) dataset [40]). It is important to note that the datasets we used were also used in many studies, such as [3,6,8,27,28]. Many-to-one and one-to-one were treated here as two cross-view categorization situations. The first was trained on one view and examined another view, but the second was trained on V-1 views and examined the remaining views. The corresponding ${\mathrm{X}}^{\mathrm{v}}\leftarrow 0$ in Equation (5) was used for the vth view meant for testing. In addition, we employed the intersection kernel support vector machine (IKSVM) as our classifier with parameters C = 1. γ = 1, β = 1, α = 1, p = 0, and K = 1 were default parameters with the default number of layers being 1. We also took into consideration NUMA and IXMAS. These are datasets for multiple camera view video. We acquired the first from three Kinect sensors in five scenarios comprising 10 human actions, while the second was taken from one top-view camera, and four side-view cameras. In addition, we employed a k-means clustering approach to create video words and quantize the descriptors. This led to the likelihood of using a histogram of the feature to represent a given video. As a further contribution, we were curious to see how our model would react if the recording was not done continuously for the performing action; that is, will it be possible for our model to keep the same accuracy if we remove some frame of the video? As described in Section 4.1, for an interval of time t, we have N actions. If we pause the recording during a small period (p << t), we can still see a flow described by feature points of the target view, as shown in Figure 7.

It is also worth noting that V feature vectors constituted a representation of an action taken from V camera angles.

6.1. Using the IXMAS Dataset

Just like in [33], we obtained a histogram of intense trajectory and slanting optical flow and a dictionary for every feature was obtained by making use of k-means. In addition, a bag-of-words model was used to turn every video into a feature and encode each of the features. A “leave one action class out” training scheme was used to obtain a reasonable concurrence, just like in [8] and [28]. A single action class was used every time we needed to test. Moreover, we removed all videos from the feature steps of learning in order to examine the ability of our method to transfer information, and we introduced a fifth component, which was the result of our method when we performed several p pauses (p << t) (

Table 2. One-to-one cross-view recognition outcome for different controlled methods on the IXMAS dataset. The values enclosed in brackets are the recognition accuracies for [8,29,41,42] and our supervised method (with and without the p pause).

	TEST VIEW 0	TEST VIEW 1	TEST VIEW 2	TEST VIEW 3	TEST VIEW 4
TRAINING VIEW 0	(71, 99.4, 99.1, 100, 97.6)	(82, 96.4, 99.3, 100, 86.8)	NA	(76, 97.3, 100, 100, 94.2)	(72, 90.0, 96.4, 100, 88.1)
TRAINING VIEW 1	(80, 85.8, 99.7, 100, 94.3)	(77, T81.5, 98.3, 100, 88.2)	(73, 93.3, 97.0, 100, 92.1)	(72, 83.9, 98.9, 100, 96.3)	NA
TRAINING VIEW 2	(75, 98.2, 90.0, 100, 94.8)	(75, 97.6, 99.7, 100, 91.3)	(73, 99.7, 98.2, 99.4, 98.1)	NA	(76, 90.0, 96.4, 100, 89.5)
TRAINING VIEW 3	(72, 98.8, 100, 100, 94.2)	NA	(74, 99.7, 97.0, 99.7, 96.3)	(70, 92.7, 89.7, 100, 87.9)	(66, 90.6, 100, 99.7, 89.7)
TRAINING VIEW 4	NA	(79, 98.8, 98.5, 100, 93.2)	(79, 99.1, 99.7, 99.7, 93.5)	(68, 99.4, 99.7, 100, 97.2)	(76, 92.7, 99.7, 100, 84.9)
Average	(74, 95.5, 97.2, 100, 95.2)	(77, 93.6, 98.3, 100, 89.4)	(76, 98.0, 98.7, 99.7, 95)	(73, 93.3, 97.0, 100, 93.9)	(72, 92.4, 98.9, 99.9, 88)

).

6.1.1. Many-to-One Cross-View Action Recognition

A single view was used in our experiment as a test view, while all other views were used for training. The experiment enabled us to evaluate our method of learning shared and private features.

The renowned methods in [5,25,28,29] were compared with our method and it was found that approximately 99.9% was achieved by our method, as shown in

Table 3. One-to-one cross-view recognition results of various unsupervised approaches on the IXMAS dataset. The results in brackets are the recognition accuracies of [8,27,28,29,43], and our unsupervised approach, respectively.

	Test View 0	Test View 1	Test View 2	Test View 3	Test View 4
Training view 0	(79.6, 92.1, 99.4, 82.4, 72.1, 100)	(76.6, 89.7, 97.6, 79.4, 86.1, 99.7)	NA	(79.8, 94.9, 91.2, 85.8, 77.3, 100)	(72.8, 89.1, 100, 71.5, 62.7, 99.7)
Training view 1	(82.0, 83.0, 87.3, 57.1, 48.8, 99.7)	(68.3, 70.6, 87.8, 48.5, 40.9, 100)	(74.0, 89.7, 92.1, 78.8, 70.3, 100)	(71.1, 83.7, 90.0, 51.2, 49.4, 100)	NA
Training view 2	(73.0, 97.0, 87.6, 82.4, 82.4, 100)	(74.1, 94.2, 98.2, 80.9, 79.7, 100)	(74.0, 96.7, 99.4, 82.7, 70.9, 100)	NA	(66.9, 83.9, 95.4, 44.2, 37.9, 100)
Training view 3	(81.2, 97.3, 97.8, 95.5, 90.6, 100)	NA	(75.8, 96.4, 91.2, 77.6, 79.7, 99.7)	(78.0, 89.7, 78.4, 86.1, 79.1, 99.4)	(70.4, 81.2, 88.4, 40.9, 30.6, 99.7)
Training view 4	NA	(79.9, 96.7, 99.1, 92.7, 94.8, 99.7)	(76.8, 97.9, 90.9, 84.2, 69.1, 99.7)	(76.8, 97.6, 88.7, 83.9, 98.9)	(74.8, 84.9, 95.5, 44.2, 39.1, 99.4)
Average	(79.0, 94.4, 93.0, 79.4, 74.5, 99.9)	(74.7, 87.8, 95.6, 75.4, 75.4, 99.9)	(75.2, 95.1, 93.4, 80.8, 72.5, 99.9)	(76.4, 91.2, 87.1, 76.8, 72.4, 99.9)	(71.2, 84.8, 95.1, 50.2, 42.6, 99.7)

. Our method Seb1 achieved a better performance compared to the other approaches, which illustrates the benefit of using our private and shared feature approach in our paper. To determine the resemblance among video samples across camera views, our method implements the sample-affinity matrix with the direct consequence of accurate characterization of the similitude across views by the learned shared features. Furthermore, the learned private features become more edifying for categorization, as the redundancy between private and shared features is reduced.

6.1.2. One-to-One View Action Recognition

In this experiment, we trained our model with information from a single camera view and we performed the test based on the data extracted from another view. The private features were discarded here and only learned shared features were utilized since private features of one view do not suffice.

By doing a comparison between our approach (Seb1) and the reported recognition results in

Table 4. Many-to-one cross-view action recognition results on the IXMAS dataset, where each column corresponds to a test view.

Methods	Test View 1	Test View 2	Test View 3	Test View 4	Test View 5
Yan et al. [7]	91.2	87.7	82.1	81.5	79.1
Liu et al. [28]	86.1	81.1	80.1	83.6	82.8
Zheng and Jiang [29]	97.0	99.7	97.2	98.0	97.3
Zheng and Jiang [29]-2	99.7	99.7	98.8	99.4	99.1
Zheng et al. [8]	98.5	99.1	99.1	100	90.3
Liu and Shah [44]	76.7	73.3	72.0	73.0	N/A
Weinland et al. [25]	86.7	89.9	86.4	87.6	66.4
Our supervised method	100	99.7	99.5	100	100

of [8,29,44], we can say that our method performed best in 18 out of 20 combinations, which is considerably better than all the other methods. Again, our approach reached 99.8% in 16 instances, showing the potency of the learned shared features. Due to the importance of discriminative information obtained from the label information and the learned shared features, our approach is resistant enough to viewpoint variation and demonstrates high performance that is cross-view invariant.

6.2. Use of NUMA 3D Dataset

6.2.1. Many-to-One Cross-View Action Recognition

It is good to keep in mind that the features used here are similar to those of the IXMAS dataset. As mentioned in [40], many-to-one cross-view recognition accuracy in three cross-view scenarios can be expressed as cross-camera view, cross-subject, and cross-environment. Following [40], our approach is compared with [31,40,45,46,47].

Table 5. Cross-view, cross-environment, and cross-subject action recognition outcomes on the NUMA dataset.

Methods	Cross-Subject	Crossview	Cross-Environ
Sadanad and Corso [45]	24.6	17.6	N/A
Li et al. [31]	54.2	45.2	28.6
Li and Zickler [38]	50.7	47.8	27.4
Felzenszwalb et al. [47]	74.8	46.1	68.8
Wan et al., LowR + [40]	81.6	73.3	79.3
Wang et al. [40]	78.9	65.3	71.9
Maji et al. [46]	54.9	24.5	48.5
Our supervised method	83.2	77.3	89.8

shows that our approach is better than [40] Low-resolution visual features (LowR) by 10.4% and 3.9% in cross-environment and cross-view scenario, respectively. Furthermore, in the cross-subject scenario, it reaches an incredible performance with LowR added to [40]. A close look at the different comparison shows that the most important gain of performance of Seb1 in cross-environment, cross-subject, and cross-view scenarios are, respectively, 62.3% (over [48]), 30.4% (over [48]), and 32.0% (over [31]). This shows an incredible gain of performance obtained by the Seb1 approach due to the use of SAM to determine similarities of samples in different views and shared and private facts for modeling cross-view data.

6.2.2. Parameter Analysis

Let us evaluate the sensibility of our method when we applied α, β, and γ parameters, as shown in Figure 8. We can observe the mean performance of many-to-one cross-view action recognition given values of 0.001, 0.01, 0.1, 1, 10, and 100 of parameters α, β, and γ. We concluded that our approach is insensible to that variation (parameter values). Despite the 2% observed as the large performance gap when parameters α, β, and γ were set, we came again to the conclusion that our method is robust.

7. Conclusions

In this paper, we proposed an approach that can label human actions from cross views. Our research focuses on two new methods using unshared and shared facets to precisely classify human action with varying appearances and viewpoints. We have introduced a sample-affinity matrix that is used to determine similarities across views. This matrix has also been used in the monitoring of shared features as well as in controlling the transfer of information so that the contribution of every sample can be measured accurately. Our methods can keep several layers of learners to study view-invariant features more efficiently. It equipoises the sharing of information among views as per sample similarities and can measure the distance between two views using SAM Z. We also found that our method is robust to some variation in the recording time, as shown in Table 2. However, we noticed during our experiments that our approach required the use of various computer resources. We will further improve our model so that it uses fewer computational resources. To also help in measuring the contribution of every sample precisely, we carried out a series of experiments in NUMA and IXMAS, where we found out that our methods performed well when it came to the categorization of cross-view actions. We have also seen the potentials of our approach, and we now intend to handle image extract flow and space-time instead of taking it during a time t in order to handle activities. We also intend to adjust our model and algorithm in a way based on our previous projects [49] so that they enable us to capture many activities in a more accurate manner.