Robust Latent Common Subspace Learning for Transferable Feature Representation

Abstract

This paper proposes a novel robust latent common subspace learning (RLCSL) method by integrating low-rank and sparse constraints into a joint learning framework. Specifically, we transform the data from source and target domains into a latent common subspace to perform the data reconstruction, i.e., the transformed source data is used to reconstruct the transformed target data. We impose joint low-rank and sparse constraints on the reconstruction coefficient matrix which can achieve following objectives: (1) the data from different domains can be interlaced by using the low-rank constraint; (2) the data from different domains but with the same label can be aligned together by using the sparse constraint. In this way, the new feature representation in the latent common subspace is discriminative and transferable. To learn a suitable classifier, we also integrate the classifier learning and feature representation learning into a unified objective and thus the high-level semantics label (data label) is fully used to guide the learning process of these two tasks. Experiments are conducted on diverse data sets for image, object, and document classifications, and encouraging experimental results show that the proposed method outperforms some state-of-the-arts methods.

1. Introduction

Collecting massive labeled data is an expensive and time-consuming process in realistic scenarios [1]. Meanwhile, visual classification models often are required to be well trained for accurate prediction by sufficient labeled data. In this case, there is an urgent need to use labeled and relevant data from various data sets for facilitating the training process [2]. However, in some complex applications, the data from different data sets have different distributions. Thus, the key point of the problem is how to recover the knowledge gained from existing or well-constructed data sets for a novel task. Transfer learning is such a technique that attempts to learn an appropriated model for target application by recovering the knowledge from the source domain [3,4] In other words, transfer learning attempts to transfer the knowledge from the source domain where the data are labeled to a different but related target domain for obtaining a better model [5,6].

A major problem in transfer learning is how to decrease the different probability distributions of data in source and target domains. Intuitively, finding a better data representation that not only reduces the discrepancy distributions among different domains as best as we can but also simultaneously preserves essential properties (such as local geometric properties and intrinsic discriminable structure of data) of different domains is a feasible solution [7] which makes the model trained from the labeled source domain use for target domain directly. Many conventional subspace learning methods aim to find a good data representation (low dimensional subspace) that can preserve some specific properties of original data and thus they can be used for transfer learning [8,9,10,11]. Transfer subspace learning methods use the idea of subspace learning to find an appropriate data representation, i.e., latent common subspace, for the data of source and target domains in which the different distributions of the source and target data are also reduced as much as possible [12,13,14]. Nonetheless, these proposed transfer subspace learning methods only focus on low-level features (visual features) of data from the source and target domains, which are independent of the subsequent tasks such as visual classification. Thus, the high-level semantic information (label information) is not fully exploited to guide classifier learning [15,16].

It is well known that in transfer learning a good feature representation should be transferable [2,3,5]. However, to learn a discriminative classifier, we need to learn a discriminative feature representation. Therefore, we assume that we can learn a discriminative and transferable feature representation from the original feature representation. To this end, this paper attempts to find an appropriate data feature representation for reducing the difference in probability distributions between different domains. Therefore, this paper proposes a novel transfer learning method called robust latent common subspace learning (RLCSL) by finding a proper data feature representation that can not only reduce the distribution discrepancy but also improve the discriminative ability of the new feature representation. In this way, the new feature representation is beneficial for the subsequent learning task, i.e., classifier learning. Previous transfer subspace learning methods merely focus on the transferable feature representation learning. However, the proposed method uses data reconstruction to find a latent common subspace where the data from source and target domains have similar probability distributions by using the low-rank constraint. Moreover, the data from different domains but with the same label can be aligned together by using the sparse constraint. Current locality reconstruction methods only work on the scenario that the data come from the same domain, i.e., the data have a similar distribution. For example, locally linear embedding (LLE) [17] and non-negative matrix factorization (NMF) [18] cannot guarantee a neighborhood-to-neighborhood reconstruction. To this end, the proposed method imposes low-rank and sparse constraints on the reconstruction coefficients which can make the reconstruction coefficient matrix obtain a block-wise structure as shown in Figure 1. Therefore, the neighborhood-to-neighborhood reconstruction can be obtained. Specifically, we introduce the low-rank and sparse constraints to constrain the reconstruction coefficient matrix to achieve this goal so that the reconstruction process can automatically select the neighbor data sharing the same label but from different domains to complete the reconstruction as much as possible [15,19]. To enhance the robustness of our algorithm, we also introduce a sparse matrix to simulate the noise for reducing the effect of noisy data during the data reconstruction. To learn a suitable classifier parameter, we integrate the classifier learning and feature representation learning into a unified optimization objective. The whole framework of our method is demonstrated in Figure 2. We apply the proposed RLCSL method to the task of transfer learning. Massive experiments are conducted on the image data, object data, and document data and encouraging experiment results show the outstanding performance of our method.

The main contributions of this paper are summarized as follows.

(1) We impose the low-rank constraint on the reconstruction coefficient matrix to make data from different domains interlace well for obtaining a transferable feature representation. Moreover, by imposing a sparse constraint on reconstruction coefficient matrix, the data from different domains but with the same label can be aligned together for learning a discriminative feature representation.

(2) The classifier learning and feature representation learning are integrated into a unified optimization objective for achieving the best of them.

(3) Massive experiments demonstrate that the proposed method outperforms the state-of-the-art transfer learning methods.

The remainder of this paper is organized as follows. Section 2 reviews some related works. Section 3 introduces the basic idea of robust latent common subspace learning and some related discussions. Massive experiments are conducted in Section 4. Section 5 gives the limitations of the proposed method. Finally, we present the conclusion of the paper in Section 6.

2. Related Works

Recently, transfer learning has been widely studied in the field of machine learning and computer vision. In this section, we review some related works on transfer learning.

Two outstanding surveys of transfer learning can be found in [20,21]. The properties of domains and tasks are commonly used as a common way to classify the type of transfer learning. For example, inductive transfer learning aims to use the data in the same domain for various tasks [22,23,24,25]. Inductive learning can be also used as a supervised multi-task learning model if the data from both source and target domains have class labels [26]. Self-taught learning is a special case of inductive learning if only the data from the target domain have class labels [25]. Inductive transfer learning can be referred to as domain adaptation, where the source domain is transformed by manipulating the distribution or feature representation of the source domain [2,3,4,5,7]. A significant amount of research work is being devoted to solving the domain adaptation problem by attempting to resolve the divergence between the source and target domains [27,28,29,30,31]. The idea behind all these methods is that they learn one or more subspaces to mitigate the domain shift [2]. When label information is applicable for source and target domains, these methods can not only pass down the discriminative power but also align source and target data well.

Recently, iterative modification of the classifier is a basic way to perform transfer learning [32,33,34]. In these methods, an iterative strategy is used to adapt the knowledge from the source domain to the target domain gradually. However, these methods severely depend on the quality of the model obtained by subsequent iterations. The subspace learning method is commonly used in transfer learning by discovering the common subspace for the source and target data. The objective of subspace learning is to find a subspace in which the desired data property is preserved. For example, locality preserving projection (LPP) [35], neighborhood preserving embedding (NPE) [36], and isometric projection (ISOP) [37] are proposed to find a subspace where the intrinsic geometry structure of data is preserved. Linear discriminant analysis (LDA) [10], local discriminant embedding (LDE) [38], and locality Fisher discriminant analysis (LFDA) [39] are proposed to improve the algorithmic discriminant ability by using the label information. Recently, low-rank constraint-based subspace learning methods exploit the low-rankness to find the subspace structure of data [2,4,40,41]. Compared to conventional subspace learning methods which assume a specific noise such as Gaussian noise, low-rank constraint subspace learning methods can effectively deal with different types of noise with large magnitudes. Since our paper is based on subspace learning, we will introduce many subspace learning-based transfer learning methods. For example, Si et al. applied many traditional subspace learning methods to solve the problem of transfer learning by learning a subspace to reduce the divergence of the distribution [42]. Shao et al. proposed a generalized transfer subspace learning via low-rank constraint (LTSL) [2]. In LTSL, a unified transformation is utilized to transform both source and target data into a common subspace. The use of the low-rank constraint is done to guarantee good data alignment. Robust visual domain adaptation with low-rank construction (RDALR) is proposed to strictly transform the source domain data into the target domain [4]. The low-rank constraint imposed on the reconstruction coefficient matrix is to ensure that the source domain and target domain have a similar distribution. Kan et al. proposed a domain adaptation method, called targetized source domain (TSD) bridged by common subspace, for face recognition [3]. The idea behind TSD is to convert the source domain images to the target domain while preserving its supervisory information. The sparse reconstruction is used in TSD which is more flexible as a non-parameter measurement. Transfer component analysis (TCA) is also a transfer subspace learning method that learns a latent common subspace to reduce the difference between the margin distributions of the source and target domains [7]. Although the above methods are closely related to our method, i.e., reducing the differences in the marginal distributions of the source and target domains by learning a common subspace, they significantly differ from our proposed method in the following aspects. First, these methods learn a latent common subspace in which the distribution divergence between the source domain and target domain can be reduced. However, our proposed method learns a latent common subspace in which the distribution discrepancy can be reduced and the discriminative ability of new feature representation can be also largely improved. Second, these methods adopt a two steps strategy to address the problem of transfer learning. However, our proposed method unifies these two steps into an optimization objective to seek the best of them.

3. Robust Latent Common Subspace Learning

3.1. Notation

Denote ${\sigma }_i\left(Z\right)$ as the i-th singular value of matrix Z, we define ${\parallel Z\parallel }_{*}={\sum }_i{\sigma }_i\left(Z\right)$ and ${\parallel Z\parallel }_{2,1}={\sum }_{i=1}^{dim}\sqrt{{\sum }_{j=1}^{n_t}{Z}_{ij}^2}$ as the unclear norm and ${\ell }_{2,1}$-norm of matrix Z, respectively. Denote F as the binary label matrix, we define it as follows: for each sample $x_i$, $y_i\in {\Re }^c$ is the corresponding label. Suppose that $x_i$ is from the k-th class ($k=1,2,\dots ,c$), then only the k-th entry of $y_i$ is one and all the other entries are zero. Denote $E\in {\Re }^{dim\times n_t}$ as the noise matrix, we define ${\parallel E\parallel }_1={\sum }_{ij}\left|E_{ij}\right|$ as ${\ell }_1$-norm of matrix E. A description of many variables used in this paper is shown in

Table 1. Description of different variables.

Variable	Description
$X_t\in {\Re }^{m\times n_t}$	target domain data matrix
$X_s\in {\Re }^{m\times n_s}$	source domain data matrix
$F\in {\Re }^{n_s\times c}$	binary label matrix
$Z\in {\Re }^{n_s\times n_t}$	reconstruction coefficient matrix
$E\in {\Re }^{dim\times n_t}$	noise matrix
$P\in {\Re }^{dim\times c}$	transformation matrix for label space
$Q\in {\Re }^{m\times dim}$	transformation matrix for latent common subspace
m	dimension of data
$n_s$	number of source domain data
$n_t$	number of target domain data
$dim$	dimension of latent common subspace

.

3.2. Objective Function

A classifier with better classification performance in visual object recognition tasks always requires finding a good feature representation. Especially when the data from different domains have different distributions [15], which is a difficult task to learn a good feature representation. To this end, we propose to learn a latent common space in which the following properties should be preserved.

(1) The distribution discrepancy between source and target domains is reduced as soon as possible.

(2) The neighborhood-to-neighborhood reconstruction should be emphasized to keep enough discriminant information for classifier learning [3].

(3) The noisy information should be filtered as much as possible.

To achieve the above purposes, we use transformation matrix $Q\in {\Re }^{m\times dim}$ to transform the data from different domains into a latent common subspace. Similar to works in [2,3,4], we also suppose the dimensionalities of data from source and target domains are the same for the convenient statement. In real applications, we can use two different transformation matrices to transform data from different domains into the latent common subspace. For each sample $x_i$, $Q^T{x}_i$ is the corresponding representation in the latent common subspace. To satisfy the first property, we assume that the transformed target data can be linearly reconstructed by the transformed source data in the latent common subspace. A naive least-square criterion-based reconstruction is not suitable for the neighborhood-to-neighborhood reconstruction which may over-fit the data and ignore structure information in both source and target domains. LTSL [2] and RDALR [4] impose the low-rank constraint on the reconstruction coefficient matrix so as to reduce the distribution discrepancy between source and target domains. Although the low-rank constraint can make the distance between the means of transformed source domain data and transformed target domain data close in the latent common subspace, the neighborhood to neighborhood reconstruction may be lost owing to the global low-rank constraint. To this end, we impose the joint low-rank and sparse constraints on the reconstruction coefficient matrix such that the first two properties are satisfied simultaneously. Thus, we have the following objective function

$$\underset{Z,Q}{min}{\lambda }_{1}rank\left(Z\right)+{\lambda }_2{\parallel Z\parallel }_{2,1}s.t.Q^T{X}_t=Q^T{X}_{s}Z$$

(1)

where ${\lambda }_1$ and ${\lambda }_2$ are the non-negative regularization parameters that aim to balance the corresponding terms. It is well known that if the data points are neighborhood, then they have the same label with a high probability. By using the constraint ${\parallel Z\parallel }_{2,1}$, each transformed data point from the target domain tends to select nearest neighbors from the transformed source domain for performing sparse reconstruction. This guarantees that the neighborhood-to-neighborhood reconstruction can be preserved to provide enough discriminative ability for the new feature representation, i.e., $Q^T{X}_s$ or $Q^T{X}_s$. In other words, each transformed data point from the target domain select the transformed data points from the source domain with the same label to perform the reconstruction. Thus, the transformed data points from different domains but with the same label can be aligned together. Moreover, the use of low-rank constraints can ensure that the data from both domains can be interlaced with each other so that the distribution discrepancy between source and target domains is also reduced [2].

In order to fulfill the third property, we introduce a sparse matrix to fit the noise so that the effect of noisy data points can be reduced. As displayed in Figure 2, the noisy data points do not participate in the reconstruction process, which is formulated as

$$\begin{array}{c}\hfill \underset{Z,Q}{min}{\lambda }_{1}rank\left(Z\right)+{\lambda }_2{\parallel Z\parallel }_{2,1}+{\lambda }_3{\parallel E\parallel }_1\end{array}$$

(2)

$$s.t.Q^T{X}_t=Q^T{X}_{s}Z+E$$

where ${\lambda }_3$ is the non-negative regularization parameter.

We introduce a linear function $P\in {\Re }^{dim\times c}$ to forecast the mapping relationship between the latent common subspace and the label space, that is,

$$\begin{array}{c}\hfill F=X^{T}QP\end{array}$$

(3)

The least-squares loss function is utilized to learn P. Thus, we propose the following objective function

$$\underset{Z,P,Q}{min}{\parallel F-X_s^{T}QP\parallel }_F^2+{\lambda }_1{\parallel Z\parallel }_{*}+{\lambda }_2{\parallel Z\parallel }_{2,1}+{\lambda }_3{\parallel E\parallel }_1+{\lambda }_4{\parallel P\parallel }_F^2$$

(4)

$$s.t.Q^T{X}_t=Q^T{X}_{s}Z+E$$

where ${\lambda }_4$ is the non-negative trade-off parameter. In order to make sure that the problem is solvable, we impose the orthogonal constraint on the transformation matrix Q, and thus reformulate Formula (4) as the following problem:

$$\underset{Z,P,Q}{min}{\parallel F-X_s^{T}QP\parallel }_F^2+{\lambda }_{1}rank\left(Z\right)+{\lambda }_2{\parallel Z\parallel }_{2,1}+{\lambda }_3{\parallel E\parallel }_1+{\lambda }_4{\parallel P\parallel }_F^2$$

(5)

$$s.t.Q^T{X}_t=Q^T{X}_{s}Z+E,Q^{T}Q=I$$

From the above formulation, it is clear that we actually learn two transformation matrices Q and P in which Q is used to transform the data from the original feature space into the latent common subspace and P is used to learn the classifier parameter. The second (the low-rank constraint) and third (the sparse constraint) terms can effectively guarantee that the first and second properties are fulfilled. The fourth term ensures that the learning process is robust. The fifth term can be used to prevent the over-fitting problem by minimizing the Frobenius norm of matrix P.

The optimization problem in Formula (5) is intractable to be resolved, due to the fact that the rank minimization problem is not convex. Following [2], we could replace this rank minimization by its surrogate, nuclear norm minimization, and reformulate it as

$$\underset{Z,P,Q}{min}{\parallel F-X_s^{T}QP\parallel }_F^2+{\lambda }_1{\parallel Z\parallel }_{*}+{\lambda }_2{\parallel Z\parallel }_{2,1}+{\lambda }_3{\parallel E\parallel }_1+{\lambda }_4{\parallel P\parallel }_F^2$$

(6)

$$s.t.Q^T{X}_t=Q^T{X}_{s}Z+E,Q^{T}Q=I$$

where ${\parallel .\parallel }_{*}$ represents the nuclear norm of a matrix. By learning matrix Q, we can obtain a discriminative and transferable feature representation which is benefit from that we impose the joint sparse and low-rank constraints on reconstruction coefficient matrix Z.

3.3. Optimization Algorithm

Problem (6) could be addressed by using the popular alternating direction method (ADM) [2,4,40,41]. However, ADM requires introducing two extra variables for solving (6) and the time-consuming matrix inversions are required in each iteration. To this end, we introduce a linearized alteration direction method with adaptation penalty (LADMAP) [43] to solve (6). Moreover, we introduce two extra variables W and H to make the objective function separable (please note that with the setting $QP=W\in {\Re }^{m\times c}$, we can obtain the classifier W for classifying the target data directly).

$$\underset{Z,P,Q}{min}{\parallel F-X_s^{T}QP\parallel }_F^2+{\lambda }_1{\parallel Z\parallel }_{*}+{\lambda }_2{\parallel Z\parallel }_{2,1}+{\lambda }_3{\parallel E\parallel }_1+{\lambda }_4{\parallel P\parallel }_F^2$$

(7)

$$s.t.Q^T{X}_t=Q^T{X}_{s}Z+E,Q^{T}Q=I,QP=W,Z=H$$

The augmented Lagrangian function of problem (7) is

$$\begin{array}{c}\hfill \hslash (W,Q,P,Z,H,E,Y_1,Y_2,Y_3,Y_4)={\parallel F-X_s^{T}W\parallel }_F^2\\ \hfill +{\lambda }_1{\parallel Z\parallel }_{*}+{\lambda }_2{\parallel H\parallel }_{2,1}+{\lambda }_3{\parallel E\parallel }_1+{\lambda }_4{\parallel F\parallel }_F^2+Tr(Y_1^T(Q^T{X}_t\\ \hfill -Q^T{X}_{s}Z-E\left)\right)+Tr\left(Y_2^T(Q^{T}Q-I)\right)+Tr\left(Y_3^T(QP-W)\right)\\ \hfill +Tr\left(Y_4^T(Z-H)\right)+\frac{\mu }{2}({\parallel Q^T{X}_t-Q^T{X}_{s}Z-E\parallel }_F^2\\ \hfill +\left({\parallel Q^{T}Q-I\parallel }_F^2\right)+\left({\parallel QP-W\parallel }_F^2\right)+{\parallel Z-H\parallel }_F^2)\end{array}$$

(8)

where $Y_1$, $Y_2$, $Y_3$, and $Y_4$ are Lagrange multipliers and $\mu$ is a penalty parameter. The LADMAP is utilized to alternately update the variables W, Q, P, Z, H, and E, through minimizing ℏ with other fixed variables. Therefore, we obtain six update steps corresponding to all variables, and all steps have closed form solution.

Step 1. Update Q: The solution of Q can be obtained by solving (9).

$$\begin{array}{c}\hfill \hslash \left(Q\right)=arg\underset{Q}{min}Tr\left(Y_{1,k}^T(Q^T{X}_t-Q^T{X}_s{Z}_k-E_k)\right)\\ \hfill +Tr\left(Y_{2,k}^T(Q^{T}Q-I)\right)+Tr\left(Y_{3,k}^T(Q{P}_k-W_k)\right)\end{array}$$

(9)

$$s.t.Q^{T}Q=I$$

By setting the derivation $\frac{\partial \hslash \left(Q\right)}{\partial Q}=0$, we obtain

$$\begin{array}{c}\hfill \frac{\partial \hslash \left(Q\right)}{\partial Q}=X_t{Y}_{1,k}^T-X_s{Z}_k{Y}_{1,k}^T+2Q{Y}_{2,k}+Y_{3,k}{P}_{1,k}^T=0\\ \hfill \Rightarrow Q=(X_s{Z}_k{Y}_{1,k}^T-X_t{Y}_{1,k}^T-Y_{3,k}{P}_k^T)Y_{2,k}\end{array}$$

(10)

Step 2. Update P: The solution of P can be obtained by solving (11).

$$\begin{array}{c}\hfill \hslash \left(P\right)=arg\underset{P}{min}{\lambda }_4{\parallel P\parallel }_F^2+\frac{\mu }{2}{\parallel Q_{k+1}P-W_k+\frac{Y_{3,k}}{{\mu }_k}\parallel }_F^2\end{array}$$

(11)

By setting the derivation $\frac{\partial \hslash \left(P\right)}{\partial P}=0$, we obtain

$$\begin{array}{c}\hfill P={(2{\lambda }_{4}I+\mu Q_{k+1}^T{Q}_{k+1})}^{-1}\left(\mu Q_{k+1}^T(W_k-\frac{Y_{3,k}}{{\mu }_k})\right)\end{array}$$

(12)

where I is the identity matrix with an appropriate size.

Step 3. Update W: The solution of W can be obtained by solving (13).

$$\begin{array}{c}\hfill \hslash \left(W\right)=arg\underset{W}{min}{\parallel F-X_s^{T}W\parallel }_F^2+\frac{{\mu }_k}{2}{\parallel Q_{k+1}{P}_{k+1}-W+\frac{Y_{3,k}}{{\mu }_k}\parallel }_F^2\end{array}$$

(13)

By setting the derivation $\frac{\partial \hslash \left(W\right)}{\partial W}=0$, we obtain

$$\begin{array}{c}\hfill W={(2X_s{X}_s^T+\mu I)}^{-1}(\mu (Q_{k+1}{P}_{k+1}+\frac{Y_{3,k}}{{\mu }_k})+2X_{s}F)\end{array}$$

(14)

Step 4. Update Z: The solution of Z can be obtained by solving (15).

$$\begin{array}{c}\hfill \hslash \left(Z\right)=arg\underset{Z}{min}{\lambda }_1{\parallel Z\parallel }_{*}+\psi (Q_{k+1},X_t,X_s,Z_k,E_k,H_k,Y_{1,k},Y_{4,k})\end{array}$$

(15)

where $\psi (Q_{k+1},X_t,X_s,Z_k,E_k,H_k,Y_{1,k},Y_{4,k})=\frac{{\mu }_k}{2}({\parallel Q_{k+1}^T{X}_t-Q_{k+1}^T{X}_s{Z}_k-E_k+\frac{Y_{1,k}}{{\mu }_k}\parallel }_F^2+{\parallel Z_k-H_k+\frac{Y_{4,k}}{{\mu }_k}\parallel }_F^2)$. Here, the quadratic term $\psi$ can be represented by its first order approximation at the previous iterate and then an additional approximation term is appended, i.e.,

$$\begin{array}{c}\hfill \hslash \left(Z\right)=arg\underset{Z}{min}{\lambda }_1{\parallel Z\parallel }_{*}+<{\nabla }_Z\psi (Q_{k+1},X_s,X_t,Z_k,E_k,H_k,\\ \hfill Y_{1,k},Y_{4,k}),Z-Z_k>+\frac{\theta {\mu }_k}{2}{\parallel Z-Z_k\parallel }_F^2\\ \hfill =arg\underset{Z}{min}{\lambda }_1{\parallel Z\parallel }_{*}+\frac{\theta {\mu }_k}{2}\parallel Z-Z_k\\ \hfill +\frac{{\nabla }_Z\psi (Q_{k+1},X_s,X_t,Z_k,E_k,H_k,Y_{1,k},Y_{4,k})}{\theta {\mu }_k}{\parallel }_F^2\\ \hfill =arg\underset{Z}{min}{\lambda }_1{\parallel Z\parallel }_{*}+\frac{\theta {\mu }_k}{2}\\ \hfill \parallel Z-Z_k+\frac{X_s^T{Q}_{k+1}{J}_0+2X_s^T{Q}_{k+1}{Q}_{k+1}^T{X}_s{Z}_k+Z_k-J_1}{2\theta }{\parallel }_F^2\end{array}$$

(16)

where $J_0=Q_{k+1}^T{X}_t-E_k+\frac{Y_{1,k}}{{\mu }_k}$ and $J_1=H_k-\frac{Y_{4,k}}{{\mu }_k}$. ${\nabla }_Z\psi (Q_{k+1},X_s,X_t,Z_k,E_k,H_k,Y_{1,k},Y_{4,k})$ is the partial differential of $\psi$ with respect to Z and $\theta ={\parallel X_s\parallel }_F^2$.

Problem (16) can be solved through singular value thresholding (SVT) [2].

$$\begin{array}{c}\hfill Z=j_{\frac{{\lambda }_1}{\theta {\mu }_k}}[Z_k-\frac{X_s^T{Q}_{k+1}{J}_0+2X_s^T{Q}_{k+1}{Q}_{k+1}^T{X}_s{Z}_k+Z_k-J_1}{2\theta }]\end{array}$$

(17)

where $j_{\frac{{\lambda }_1}{\theta {\mu }_k}}\left(X\right)=U{S}_{\frac{{\lambda }_1}{\theta {\mu }_k}}{V}^T$ is the thresholding operator with respect to $\frac{{\lambda }_1}{\theta {\mu }_k}$, where $S_{\frac{{\lambda }_1}{\theta {\mu }_k}}\left(X\right)=sign\left(X_{ij}\right)max(0,|X_{ij}-\frac{{\lambda }_1}{\theta {\mu }_k}\left|\right)$ is the soft-thresholding operator and $X=US{V}^T$ is the singular value decomposition of X.

Step 5. Update H:H can be updated through solving the optimization problem (18) with the closed form solution.

$$\begin{array}{c}\hfill \hslash \left(H\right)=arg\underset{H}{min}{\lambda }_2{\parallel H\parallel }_{2,1}+\frac{{\mu }_k}{2}{\parallel Z_{k+1}-H+\frac{Y_{4,k}}{{\mu }_k}\parallel }_F^2\end{array}$$

(18)

which can be solved by the following Lemma 1.

Lemma 1

([41]). Let A be a given matrix and if the optimal solution to

$$\underset{B}{min}\alpha {\parallel B\parallel }_{2,1}+\frac{1}{2}{\parallel A-B\parallel }_F^2$$

(19)

is $B^{*}$, then the ith column of $B^{*}$ is

$$B^{i*}=\left\{\begin{array}{cc}\frac{{\parallel B^i\parallel }_2-\alpha }{{\parallel B^i\parallel }_2}{A}^i\hfill & if {\parallel A^i\parallel }_2>0\hfill \\ 0\hfill & otherwise\hfill \end{array}\right.$$

(20)

where $B^i$ and $A^i$ are the ith columns of matrix B and A, respectively.

Step 6. Update E: E can be updated by solving the optimization problem (21) with the closed form solution (22).

$$\begin{array}{c}\hfill \hslash \left(E\right)=arg\underset{E}{min}{\lambda }_3{\parallel E\parallel }_1+\frac{{\mu }_k}{2}{\parallel Q_{k+1}^T{X}_t-Q_{k+1}^T{X}_s{Z}_{k+1}-E+\frac{Y_{1,k}}{{\mu }_k}\parallel }_F^2\end{array}$$

(21)

$$\begin{array}{c}\hfill \Rightarrow E_{k+1}={\xi }_{\frac{{\lambda }_3}{{\mu }_k}}(Q_{k+1}^T{X}_t-Q_{k+1}^T{X}_s{Z}_{k+1}+\frac{Y_{1,k}}{{\mu }_k})\end{array}$$

(22)

where ${\xi }_{\frac{{\lambda }_3}{{\mu }_k}}=shrink(Q_{k+1}^T{X}_t-Q_{k+1}^T{X}_s{Z}_{k+1}+\frac{Y_{1,k}}{{\mu }_k},\frac{\lambda 2}{{\mu }_k})$ and $shink(x,a)=signmax\left(\right|x|-a,0)$.

Step 7. Update $Y_1$, $Y_2$, $Y_3$, $Y_4$ and $\mu$: We update the Lagrange multipliers and penalty parameter as follows ($\rho \ge 1$).

$$\left\{\begin{array}{c}{Y}_{1,k+1}=Y_{1,k}+{\mu }_k(Q_{k+1}^T{X}_t-Q_{k+1}^T{X}_s{Z}_{k+1}-E_{k+1})\hfill \\ Y_{2,k+1}=Y_{2,k}+{\mu }_k(Q_{k+1}^T{Q}_{k+1}-I)\hfill \\ Y_{3,k+1}=Y_{3,k}+{\mu }_k(Q_{k+1}{P}_{k+1}-W_{k+1})\hfill \\ Y_{4,k+1}=Y_{4,k}+{\mu }_k(Z_{k+1}-H_{k+1})\hfill \\ {\mu }_{k+1}=min({\mu }_{max},\rho {\mu }_k)\hfill \end{array}\right.$$

(23)

The complete algorithm is outlined in Algorithm 1.

3.4. Classification

Once the optimal solution W is obtained, we first obtain $F_s=X_{s}W$ and $F_t=X_{t}W$, respectively. Then $F_s$ and $F_t$ are respectively used as training set and test set and the nearest neighbor classification (NN) [2] is utilized as the baseline classifier to classify $F_t$.

Algorithm 1: Solving RLCSL by LADMAP.

Input: Data set matrix $X_t$ and $X_s$; Source domain data label indicator matrix F; Parameters ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$, ${\lambda }_4$ and the latent common subspace $dim$; Initialization:$Z_0=H_0=\mathbf{O}$; $E_0=\mathbf{O}$; $P_0=\mathbf{O}$; $W_0=\mathbf{O}$; $Q_0=arg{max}_{Q}Tr\left(Q^T{X}_s{X}_s^{T}Q\right), s.t.Q^{T}Q=I$; $Y_{1,0}=\mathbf{O}$; $Y_{2,0}=\mathbf{O}$; $Y_{3,0}=\mathbf{O}$; $Y_{4,0}=\mathbf{O}$; ${\mu }_0=0.1$; ${\mu }_{max}={10}^7$; ${\rho }_0=1.01$; $\theta ={\parallel X_s\parallel }_F^2$; $k=0$; $\sigma \le {10}^{-6}$ while not converged do  1. Fix the others and update Q by solving (9)  $Q\leftarrow$ orthogonal Q.  2. Fix the others and update P by solving (11).  3. Fix the others and update W by solving (13).  4. Fix the others and update Z by solving (15).  5. Fix the others and update H by solving (18).  6. Fix the others and update E by solving (21).  7. Update the multipliers as follows  $\left\{\begin{array}{c}{Y}_{1,k+1}\leftarrow Y_{1,k}+{\mu }_k(Q_{k+1}{X}_t-Q_k{X}_s{Z}_{k+1}-E_{k+1})\hfill \\ Y_{2,k+1}\leftarrow Y_{2,k}+{\mu }_k(Q_{k+1}^T{Q}_{k+1}-I)\hfill \\ Y_{3,k+1}\leftarrow Y_{3,k}+{\mu }_k(Q_{k+1}{P}_{k+1}-W_{k+1})\hfill \\ Y_{4,k+1}\leftarrow Y_{4,k}+{\mu }_k(Z_{k+1}-H_{k+1})\hfill \end{array}\right.$  8. Update the parameter $\mu$ follows  ${\mu }_{k+1}=min({\mu }_{max},\rho {\mu }_k)$  9. Check the convergence conditions  $\Omega \le ϵ$  where $\Omega =max(\parallel Q_k^T{X}_t-Q_k^T{X}_{s}Z-E{\parallel }_{\infty },\parallel Q_k{P}_k-W_k{\parallel }_{\infty },\parallel Z_k-H_k{\parallel }_{\infty })$  10. Update k: $k\leftarrow k+1$. end while Output: W.

3.5. Computation Complexity, Memory Requirement, and Convergence

(1) Computation Complexity: For simplicity, we assume that both $X_s$ and $X_t$ are of $m\times n$ matrices. From the subsequent experiments, we know that the dimensionality of the latent common subspace is very small, i.e., $dim\ll m$. The main computation burdens of Algorithm 1 are:

(1) Matrices multiplication and inverse in steps (1), (2), and (3).

(2) SVD computation of an $n\times n$ matrix in step (4).

We discuss each part in detail. The main computation cost of steps (1), (2), and (3) are respectively $\mathcal{O}(m{n}^2+mndim+mcdim)$ (c is the number of classes), $\mathcal{O}\left(di{m}^{2}m\right)$, and $\mathcal{O}(m^{2}n+m^3+mnc+m^{2}c)$. The SVD computation in step (4) takes $\mathcal{O}\left(n^3\right)$ and thus the main computation cost of step (4) is $\mathcal{O}(n^3+nmdim+n^{2}m)$. When the number of samples, i.e., n is large, its computation overhead becomes prohibitively high. Fortunately, Liu et al. [41] provide a better way to deal with the problem in step (4).

For any optimal solution $Z^{*}$ to the following problem,

$$\begin{array}{c}\hfill \underset{Z}{min}\lambda 1{\parallel Z\parallel }_{*}+\lambda 2{\parallel Z\parallel }_{2,1}s.t.X_t=X_{s}Z\end{array}$$

(24)

We have

$$\begin{array}{c}\hfill Z^{*}\in span\left(X_s^T\right)\end{array}$$

(25)

which indicates that the optimal $Z^{*}$ lies the space spanned by $X_s^T$. So, we can compute the orthogonal basis of $X_s^T$ in advance and a compact $Z^{*}$ can be obtained by: $Z^{*}=K^{*}{\tilde{Z}}^{*}$, where $K^{*}$ is the orthogonal columns of $X_s^T$. In this way, we have rewritten the original problem in (24) as

$$\begin{array}{c}\hfill \underset{Z}{min}\lambda 1{\parallel \tilde{Z}\parallel }_{*}+\lambda 2{\parallel \tilde{Z}\parallel }_{2,1}s.t.X_t=M\tilde{Z}\end{array}$$

(26)

where $M=X_s{K}^{*}$. After solving for $\tilde{Z}$ the solution for (24) can be recovered by $K^{*}{\tilde{Z}}^{*}$. M is of full column rank if we give an appropriate dictionary $X_s$ [2]. Because the number of rows of $\tilde{Z}$ is at most $r_s$ and thus the computation cost of SVT in one iteration in (26) is $\mathcal{O}\left(n{r}_s^2\right)$. Combining the above results, the total computation complexity of Algorithm 1 is about $\mathcal{O}\left(N(2m{n}^2+2mndim+mcdim+di{m}^{2}m+m^{2}n+m^3+mnc+m^{2}c+n{r}_s^2)\right)$, in which N is the number of iterations.

(2) Memory Requirement: For the memory requirement, we give the memory requirements of main variables in

Table 2. Memory requirement (byte (B)) of different variables.

Variable	Type	Memory Requirement
$X_t\in {\Re }^{m\times n_t}$	Double	8$m{n}_t$B
$X_s\in {\Re }^{m\times n_s}$	Double	8$m{n}_s$B
$F\in {\Re }^{n_s\times c}$	Double	8$n_{s}c$B
$Z\in {\Re }^{n_s\times n_t}$	Double	8$n_s{n}_t$B
$H\in {\Re }^{n_s\times n_t}$	Double	8$n_s{n}_t$B
$E\in {\Re }^{dim\times n_t}$	Double	8$dim{n}_t$B
$P\in {\Re }^{dim\times c}$	Double	8$dimc$B
$W\in {\Re }^{m\times c}$	Double	8$mc$B
$Q\in {\Re }^{m\times dim}$	Double	8$dimm$B
$Y_{1,k}\in {\Re }^{dim\times n_s}$	Double	8$dim{n}_s$B
$Y_{2,k}\in {\Re }^{dim\times dim}$	Double	8$dimdim$B
$Y_{3,k}\in {\Re }^{m\times c}$	Double	8$mc$B
$Y_{4,k}\in {\Re }^{n_s\times n_t}$	Double	8$n_s{n}_t$B

. From Table 2, we can see that the main memory requirements are about $8(m{n}_t+m{n}_s+n_{s}c+3\left(n_s{n}_t\right)+dim{n}_t+dimc+2\left(mc\right)+dimm+dim{n}_s+di{m}^2)$. We also give the quantitative assessment for the case of $MSRC\to VOC$ by using MATLAB function of [user,sys]=memory and the result indicates that MemAvailableAllArrays: 3.244 $\times {10}^9$.

(3) Convergence: The convergence properties of the inexact ALM have been well investigated in [41] for the case that the number of variables is at most two. Nevertheless, there are six variables for problem (6). Moreover, the objective function in (6) is un-smooth which makes that convergence cannot be guaranteed. Based on the theoretical results in [2,41,44], two conditions are sufficient for Algorithm 1 to converge which are as follows

(1) The dictionary $X_s$ is of full column rank.

(2) The optimality gap in each iteration step is monotonically decreasing, i.e.,

$$\begin{array}{c}\hfill \tau ={\parallel (Z_k,P_k,Q_k,H_k)-arg\underset{Z,P,Q,H}{min}\hslash \parallel }_F^2\end{array}$$

(27)

where $Z_k$, $P_k$, $Q_k$, and $H_k$ are the solutions generated in the k-th iteration. We previously showed that the first condition is easy to obey. The second condition is hard to be proved directly, and thus in the section of the experiment we show that this condition can hold in real-world applications. Figure 3 displays the convergence curves of our method on two different cases. It can be seen that the object value decreases monotonically with the increase in the number of iterations. This demonstrates that the proposed optimization algorithm is efficient and has a fast convergence behavior, say within 20 iterations.

3.6. Connections to Existing Works

As discussed in Section 2, our proposed method significantly differs from previous transfer learning methods such as LTSL [2], RDALR [4], and TSD [3] in the following aspects.

(1) A great deal of work has been proposed for visual object classification, e.g., scene classification and image tagging [15,45] by transforming the data from different domains into the latent common subspace to reduce the distribution difference. Although these methods can push the data from different domains together, the data from different domains but with the same label cannot align together. Therefore, our proposed method uses matrix Q to transform the data from the original feature space into the latent common subspace for reducing the distribution discrepancy. More importantly, we impose the joint low-rank and sparse constraints on the reconstruction coefficient matrix to guarantee that: (a) the data from different domains can be closely interlaced; (b) the data from different domains but with the same label can be aligned together to seek the neighborhood to neighborhood reconstruction.

(2) Conventional transfer learning methods [8,22,46,47,48] mainly learn a transformation to reduce the distribution but ignore the classifier learning. In the proposed method, the new feature representation learning and classifier learning are integrated into the unified optimization framework. Thus, we can learn a suitable feature representation and use it as the input for learning the discriminative classifier parameter.

To our knowledge, LTSL [2] and RDALR [4] are most closely related to our RLCSL. For the clearness of comparison, the difference between these methods are as follows:

(1) Difference from LTSL [2] (lost local structure information): In LTSL, a unified transformation is used to transform the data from both the source and target domains into a common subspace where the discrepancy of the source and the target domains is decreased with a low-rank constraint, and the classification is performed in the common subspace. RLCSL first uses a transformation matrix to transform the data from both the source and target domains into the latent common subspace for eliminating or reducing the distribution discrepancy and simultaneously the classifier is learned from the latent common subspace by another transformation matrix. In this way, the common subspace learned by RLCSL is more discriminative to fit the labels than LTSL. In addition, in our RLCSL, the sparse constraint can effectively improve the discriminative ability of the new feature representation.

(2) Difference from RDALR [4]: In RDALR, only the source data are transformed into an intermediate subspace, which has too little freedom to pull the source domain close enough to the target domain. Moreover, when the source domain data are transformed into the target domain, the data of different subjects may overlap each other and thus it is difficult to separate them. In contrast, our method transforms both of them into a latent common subspace, which can make them close enough to each other. Additionally, a specific linear classifier is learned to ensure that they are separated as much as possible.

It is necessary to note that LTSL and RDALR only use the low-rank constraint to address the transfer learning problem which cannot guarantee that the data from the same subject in one domain may choose the data points of the same subject from another domain. By using the sparse constraint, RDALR tends to select the data points from different domains but the same subject for reconstruction. This is useful to find the discriminant structure of the data from different domains [3].

4. Experiments

In this section, we conducted massive experiments on three different data sets, i.e., object recognition, image classification, and text classification, to evaluate the classification performance of RLCSL.

4.1. Data Set Preparation

Text Data set: Reuters-21,578 [22,49] is a benchmark text corpora that is widely used for testing the performance of transfer learning. Reuters-21,578 is a complex text data set with many top and subcategories. The three largest top categories are orgs, people, and place, each of which is comprised of many subcategories.

Object Data set: Office is the visual domain benchmark data, including common object categories from three different domains, i.e., Amazon, DSLR, and Webcam. In this data set, each domain contains 31 object categories, such as laptop, keyboard, monitor, bike, etc., and the total number of images is 4652. In the Amazon domain, each category has 90 images on average while in DSLR or Webcam each category has 30 images on average. Caltech-256 is also a standard data set for object recognition, which has 30,607 images from 256 categories. In our experiments, the public Office + Caltech data sets released by Gong [50] are adopted. SURF features are extracted and quantized into an 800-bin histogram with codebooks computed with K-means on a subset of images from Amazon. Then the histograms are standardized by z-score. In sum, we have four domains: A (Amazon), D (DSLR), W (Webcam), and C (Caltech-256). In addition to the SURF feature, we also selected the convolutional neural network (CNN) feature to test the performance of different methods. For the CNN feature, eight layers with five convolutional layers and three fully connected layers of CNN were trained on the ImageNet in [51]. Our experiments used the output of the 6th layer with 4096 dimensionalities. More details of the architecture and training protocol can be found in [51].

Image Data set: MSRC and VOC2007 [52] are used in our experiments. The MSRC data set is released by Microsoft Research Cambridge, containing 4323 images labeled by 18 classes. The VOC2007 data set consists of 5011 images annotated with 20 concepts. These two data sets share six common semantic classes: airplane, bicycle, bird, car, cow, and sheep.

Table 3. Detailed information of different data sets (note the number in parentheses is the dimensionality).

Data Set	Subset	Abbr.	#Images	Features	# Classes
Office-Caltech256	Amazon Caltech DSLR Webcam	A C D W	958 1123 157 295	SURF(800) DeCAF7(4096)	10
Reuters-21,578	orgs people place	or pe pl	1237 1208 1016	Pixel(4771)	2
MSRC-VOC2007	MSRC VOC2007	M V	1269 1530	DSIFT(240)	6

shows the detailed introduction of these data sets.

4.2. Comparison Methods

In order to evaluate the validity of the proposed RLCSL method with different configurations of these data sets, we compared RLCSL with some competitive state-of-the-art methods including geodesic flow kernel (GFK) + NN [50], transfer component analysis (TCA) + NN [7], transfer subspace learning (TSL) + NN [42], low-rank transfer subspace learning (LDA) + NN (LTSL) [2], robust visual domain adaptation with low-rank reconstruction (RDALR) + NN [4], transfer feature learning with joint distribution adaptation (JDA) [53], scatter component analysis (SCA) [54], discriminative transfer subspace learning via low-rank and sparse representation (DTSL) [55], joint feature selection and subspace learning (FSSL) [56], transfer joint matching (TJM) for unsupervised domain adaptation [52], and 1-nearest neighbor classification (NN) and principle component analysis (PCA) + NN. In the experiment, TSL adopts Bregman divergence instead of maximum mean discrepancy (MMD) as the distance for comparing distributions. NN is chosen as the baseline classifier because there is no need for parameter tuning. Please note that partial experiments results are quoted from [54].

4.3. Experiments on the Office, Caltech-256 Data Sets

By randomly selecting 2 different domains as the source domain and target domain respectively, we constructed 12 different cross-domain object data sets, e.g., A→D, A→W, A→C, ⋯, C→W. The experimental results of single source domain and single target domain on these 12 cross-domain object data sets are shown in

Table 4. Classification accuracies (%) of different methods on the Office and Caltech-256 data sets.

Dataset	NN	PCA	GFK	TSL	TCA	RDALR	LTSL	SCA	DTSL	JDA	RLCSL
C→A	23.70	36.95	41.02	44.47	37.89	38.20	25.26	43.74	51.25	44.78	50.52
C→W	25.76	32.54	40.68	34.24	26.78	38.64	19.32	33.56	38.64	41.69	42.71
C→D	25.48	38.22	38.85	43.31	39.49	41.40	21.02	39.49	47.13	44.59	47.13
A→C	26.00	34.73	40.25	37.58	34.73	37.76	16.92	38.29	43.37	39.36	43.37
A→W	29.83	35.59	38.98	33.90	28.47	37.63	14.58	33.90	36.61	37.97	38.98
A→D	25.48	27.39	36.31	26.11	34.39	33.12	21.02	34.21	38.85	39.49	42.67
W→C	19.86	26.36	30.72	29.83	26.36	29.30	34.64	30.63	29.83	31.17	30.45
W→A	22.96	31.00	29.75	30.27	31.00	30.06	39.56	30.48	34.13	32.78	35.69
W→D	59.24	77.07	80.89	87.26	83.44	87.26	72.61	92.36	82.80	89.17	87.26
D→C	26.27	29.65	30.28	28.50	30.28	31.70	35.08	32.32	30.11	31.70	31.70
D→A	28.50	32.05	32.05	27.56	30.90	32.15	39.67	33.72	32.05	32.15	32.67
D→W	63.39	75.93	75.59	85.42	73.22	86.10	74.92	88.81	72.20	86.10	77.29
Average	31.37	39.79	42.95	42.37	39.75	43.61	34.55	44.30	44.74	45.91	46.70

.

In order to evaluate the classification performance of the comparison methods and RLCSL better, we conducted the experiments of multiple sources domains vs. single target domain on the Office and Caltech 256 data sets. We randomly choose two subsets as the source domain while a single data set as the target domain. Thus, we also can construct 12 different cross-domain object data sets, e.g., AC→D, AC→W, ⋯, DW→C. The experimental results are shown in

Table 5. Classification accuracies (%) of multiple source domains vs single target domain on the Office and Caltech-256 data sets.

Dataset	NN	PCA	GFK	TSL	RDALR	LTSL	RLCSL
A,C→D	33.76	40.13	45.86	46.50	35.67	34.39	52.23
A,C→W	31.19	37.97	39.32	33.56	28.47	27.46	42.37
A,D→C	28.50	37.22	39.89	41.67	36.33	21.73	45.41
A,D→W	49.15	55.25	66.78	54.24	66.78	26.78	62.71
A,W→C	27.60	35.62	37.40	42.03	36.60	26.98	45.06
A,W→D	64.33	73.25	81.53	63.06	77.07	41.40	73.24
C,D→A	24.32	34.55	37.27	45.20	39.56	26.30	53.86
C,D→W	34.92	48.14	65.76	50.85	60.34	29.83	59.66
C,W→A	24.43	35.70	39.25	45.20	41.02	30.06	52.40
C,W→D	47.13	66.24	78.98	52.23	73.89	38.22	68.78
D,W→A	29.23	35.80	38.10	34.24	32.99	37.89	38.41
D,W→C	25.47	28.58	30.45	31.26	29.92	33.57	33.57
Average	35.00	44.04	50.05	45.00	46.55	31.22	52.30

. From Table 4 and Table 5, the experimental results show that RLCSL is able to obtain good classification accuracies. Figure 4 provides the classification accuracy of different methods with the CNN feature. As shown in Figure 4, we can find that all methods obtain better classification results and our method is still the best competitor. The reason of supporting the excellent classification performance of RLCSL is twofold: the joint low-rank and sparse constraints can reduce the distribution discrepancy, plus the transferable feature representation learning and classification learning are integrated into the unified optimization objective. Thus, the proposed method can benefit from the high-level feature for improving classification accuracy. Although LTSL and RDALR use the low-rank constraint to learn the transferable feature representation, they separately learn the transferable feature representation and classifier. Therefore, the high-level feature, i.e., the semantic label is not directly related to the learning process in LTSL and RDALR. This also means that RLCSL can benefit from the high-level feature but LTSL and RDALR may not.

4.4. Experiments on the Reuters-21,578 Data Set

For the Reuters-21,578 data set, we can generate six different cross-domain text data sets orgs → people (or → pe), people → orgs (pe → or), orgs → place (or → pl), place → orgs (pl → or), people → place (pe → pl), and place → people (pl → pe) by utilizing the three largest top categories. For the fairness of comparison, we directly adopted the preprocessed version of Reuters-21,578 provided by Long [49] (http://ise.thss.tsinghua.edu.cn/~mlong/, accessed on 31 December 2021).

Table 6. Classification accuracies (%) of different methods on the Reuters-21,578 data set.

Dataset	NN	PCA	GFK	TSL	RDALR	LTSL	RLCSL
or→pe	72.85	70.53	75.00	72.52	72.85	/	76.15
pe→or	72.03	71.22	75.91	74.54	72.03	/	81.08
or→pl	67.50	64.43	69.80	69.51	67.50	/	73.72
pl→or	61.12	63.29	68.21	65.35	61.12	/	69.98
pe→pl	52.65	58.22	60.63	58.22	52.65	/	64.81
pl→pe	53.39	58.22	59.05	57.66	53.39	/	60.63
Average	63.26	64.32	68.10	66.30	63.25	/	71.06

shows the classification performance of different methods on the Reuters-21,578 data set. The results in Table 6 show the superiority of RLCSL with respect to classification performance.

4.5. Experiments on the MSRC and VOC2007 Data Sets

Following [52], for these two data sets, we constructed MSRC vs. VOC2007 (M→V) by choosing a total of 1269 images in MSRC as the source domain, and a total of 1530 images in VOC2007 as the target domain. Besides, we switched the data set with another data set: VOC2007 vs. MSRC (V→M). All the source and target domain images are uniformly rescaled to $16\times 16$ pixels in length, and extract 128-dimensional dense SIFT (DSIFT) features utilizing the VLFeat open-source package. Then K-means clustering is utilized to obtain a 240-dimensional codebook. As such, the training and test data are constructed to share the same label set and feature space. The classification results are shown in

Table 7. Classification accuracies (%) of different methods on the MSRC and VOC2007 data sets.

Dataset	NN	PCA	GFK	TSL	RDALR	SCA	DTSL	FSSL	TJM	RLCSL
M→V	28.63	28.82	28.76	30.92	28.95	32.75	34.71	29.74	32.75	35.36
V→M	48.94	49.09	48.86	47.44	48.94	48.94	53.82	37.93	49.41	50.11
Average	38.78	38.95	38.81	39.18	38.94	40.85	44.27	33.83	41.08	42.73

in which our proposed method obtains the best classification result. Figure 5 gives receiver operating characteristic (ROC) curve on this data set from which we can see that class 2 and class 4 achieve the best classification performance on cases of M→V and V→M, respectively.

The training time of different methods is shown in

Table 8. Time cost(s) of different methods on different data sets.

Dataset	LTSL	GFK	RDALR	TSL	DTSL	RLCSL
M→V	413.61	6.95	51.97	459.59	355.84	375.42
A→D	32.27	9.74	7.40	59.60	51.60	10.38
C→W	37.23	10.71	5.37	61.44	62.08	17.80
A→D(CNN)	301.13	290.12	101.23	424.23	436.65	118.37
C→W(CNN)	293.46	302.34	104.37	461.28	472.59	109.18
Average	215.54	123.97	54.06	293.22	275.75	126.23

. The results in Table 8 show that the proposed method RLCSL obtains the third ranking in training time. The main reason may be that RLCSL takes a large amount of time to solve the joint low-rank and sparse optimization problem.

4.6. Visualization Analysis of Matrix Z

Figure 6 gives the visualization of matrix Z. It is obvious that the learned matrix Z is sparse and low-rank which means that although we used matrix H to replace matrix Z in our optimization algorithm, the algorithm finally satisfies constraint $Z=H$ after algorithm convergence. We eventually obtain the sparse and low-rank reconstruction coefficient matrix Z that is very useful to align the data from different domains, which further confirms the motivation of our method from the view of optimization. In addition, we can see that the block-wise structure in matrix Z is somewhat obscure. The reason may be that the distribution discrepancy of data from different domains is not completely eliminated. However, the block-wise structure in matrix Z is very obvious in locations of many classes. This indicates that the data from different domains but with the same label can be aligned together and then the distribution discrepancy of different domains is reduced and representations of $Q^T{X}_s$ and $Q^T{X}_t$ are discriminative. In other words, the latent common subspace, i.e., Q can be used as an intermediate that can not only reduce the distribution discrepancy but also improve the discriminative ability of new data representation $Q^T{X}_s$ and $Q^T{X}_t$. From this point, the latent common subspace is suitable for transfer learning and classifier learning.

4.7. Parameter Sensitivity

There are four parameters ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$, and ${\lambda }_4$ in our objective function. Specifically, ${\lambda }_1$ and ${\lambda }_2$ are used to balance the importance of the low-rank and sparse constraint terms. The goal of ${\lambda }_3$ is to control the influence of noise and ${\lambda }_4$ is used to avoid the problem of over-fitting. Theoretically, large values of ${\lambda }_1$ or ${\lambda }_2$ can make the reconstruction coefficient matrix Z more important in the proposed method. In our method, we always set ${\lambda }_1=1$ for obtaining better experimental results. However, Figure 7 shows that both ${\lambda }_1$ and ${\lambda }_4$ have slight effects on the classification performance on the case of $C\to D$. We given different combinations of these two parameters from a reasonable discrete set $\{{10}^{-4},{10}^{-3},{10}^{-2},{10}^{-1},{10}^1,{10}^2\}$ in Figure 7, which indicates that the classification performance of our proposed method is roughly consistent over a wide range of values of these two parameters. For parameters ${\lambda }_2$ and ${\lambda }_3$, we can see from Figure 7 that the classification accuracy of our proposed method is very robust to different settings provided the parameters are set in a feasible range. For dimensionality $dim$, we can see from Figure 7 that the proposed method is robust to the value of $dim$. From Figure 7, we can also find that it is an easy job to pick up a suitable parameters combination for the proposed method.

4.8. Ablation Studies

In order to study the proposed RLCSL deeply, we conducted the ablation studies to test the indispensability of each component of RLCSL by comparing RLCSL with four variants of RLCSL. The objective of the first variant $RLCS{L}_{v_1}$ is as follows

$$\underset{Z,P,Q}{min}{\parallel F-X_s^{T}QP\parallel }_F^2+{\lambda }_2{\parallel Z\parallel }_{2,1}+{\lambda }_3{\parallel E\parallel }_1+{\lambda }_4{\parallel P\parallel }_F^2$$

(28)

$$s.t.Q^T{X}_t=Q^T{X}_{s}Z+E,Q^{T}Q=I$$

The goal of $RLCS{L}_{v_1}$ is to test the effectiveness of the low-rank constraint. The objective of the second variant $RLCS{L}_{v_2}$ is as follows

$$\underset{Z,P,Q}{min}{\parallel F-X_s^{T}QP\parallel }_F^2+{\lambda }_3{\parallel E\parallel }_1+{\lambda }_2{\parallel Z\parallel }_F^2+{\lambda }_4{\parallel P\parallel }_F^2$$

(29)

$$s.t.Q^T{X}_t=Q^T{X}_{s}Z+E,Q^{T}Q=I$$

whose goal is to verify the effectiveness of the joint low-rank and sparse constraints. The objective of the third variant $RLCS{L}_{v_3}$ is as follows

$$\underset{Z,P,Q}{min}{\parallel F-X_s^{T}QP\parallel }_F^2+{\lambda }_1{\parallel Z\parallel }_{*}+{\lambda }_2{\parallel Z\parallel }_{2,1}+{\lambda }_4{\parallel P\parallel }_F^2$$

(30)

$$s.t.Q^T{X}_t=Q^T{X}_{s}Z,Q^{T}Q=I$$

whose goal is to verify the effectiveness of the sparse term (the main goal is to reduce the negative affect of noisy term). The objective of the fourth variant $RLCS{L}_{v_4}$ is as follows

$$\underset{Z,P,Q}{min}{\parallel F-X_s^{T}QP\parallel }_F^2+{\lambda }_1{\parallel Z\parallel }_{*}+{\lambda }_3{\parallel E\parallel }_1+{\lambda }_4{\parallel P\parallel }_F^2$$

(31)

$$s.t.Q^T{X}_t=Q^T{X}_{s}Z+E,Q^{T}Q=I$$

whose goal is to test the effectiveness of sparse constraint.

The experimental results are shown in Figure 8. From the results in Figure 8, we can see that the low-rank constraint is more important than other terms on aligning data from the different domains ($RLCS{L}_{v_4}$ is the second-best competitor). The experimental results also show that the sparse constraint, i.e., ${\ell }_{21}$ sparse constraint, also plays an important role in improving the classification accuracy ($RLCS{L}_{v_1}$ is the third competitor). The sparse noisy term can reduce the effect of noisy term and thus it can improve the classification accuracy ($RLCS{L}_{v_3}$ is the fourth competitor). When we removed the joint low-rank and sparse constraints the classification performance is relatively poor ($RLCS{L}_{v_2}$ obtains the lower classification accuracy). Therefore, it is necessary to integrate all components into RLCSL for ensuring better classification performance.

5. Limitations

The limitations of the proposed method are as follows: (1) the dimension of latent common subspace needs to be set in advance although the algorithm is robust to the variation of its value; (2) the mathematical programming formulation is nonconvex. Although the convergence curve indicates that the proposed optimization algorithm has weak convergence properties, there is no strict theory to guarantee that the convergence behavior is always satisfied.

6. Conclusions

This paper proposes a novel robust latent common subspace learning method which utilizes the joint low-rank and sparse constraints to constrain the reconstruction coefficient matrix for obtaining a transferable and discriminative feature representation. Simultaneously, the latent common subspace is used as an intermediate that reduces the semantic gap between the low-level data representation and the high-level semantics. By integrating the classifier learning and latent common subspace learning into a unified framework, we learn a discriminative classifier parameter. The main difference between the proposed method and most related methods is shown in

Table 9. Comparison between most related works.

Comparison Perspective	JDA	TCA	RDALR	LTSL	DTSL	RLCSL
Data Reconstruction		√	√	√	√	√
Low-Rank Constraint			√	√	√	√
Sparse Constraint					√	√
Low-Rank and Sparse Constraints					√	√
Subspace Learning	√	√	√	√	√	√
Convex Optimization	√	√
Classifier Learning					√	√

. The encouraging experimental results show the effectiveness of the proposed method. In the future, we are planning to extend our method to the semi-supervised transfer learning scenario and apply our proposed method into medical data analysis [57,58] for computer-aided medical diagnosis [59,60].