Robust Fisher-Regularized Twin Extreme Learning Machine with Capped L₁-Norm for Classification

Abstract

Twin extreme learning machine (TELM) is a classical and high-efficiency classifier. However, it neglects the statistical knowledge hidden inside the data. In this paper, in order to make full use of statistical information from sample data, we first come up with a Fisher-regularized twin extreme learning machine (FTELM) by applying Fisher regularization into TELM learning framework. This strategy not only inherits the advantages of TELM, but also minimizes the within-class divergence of samples. Further, in an effort to further boost the anti-noise ability of FTELM method, we propose a new capped ${\mathit{L}}_1$-norm FTELM (C$L_1$-FTELM) by introducing capped ${\mathit{L}}_1$-norm in FTELM to dwindle the influence of abnormal points, and C$L_1$-FTELM improves the robust performance of our FTELM. Then, for the proposed FTELM method, we utilize an efficient successive overrelaxation algorithm to solve the corresponding optimization problem. For the proposed C$L_1$-FTELM, an iterative method is designed to solve the corresponding optimization based on re-weighted technique. Meanwhile, the convergence and local optimality of C$L_1$-FTELM are proved theoretically. Finally, numerical experiments on manual and UCI datasets show that the proposed methods achieve better classification effects than the state-of-the-art methods in most cases, which demonstrates the effectiveness and stability of the proposed methods.

1. Introduction

Extreme learning machine [1,2], as a remarkable single hidden layer feed-forward neural networks (SLFNs) [3] training method, has been widely studied and applied in many fields such as efficient modeling [4], fashion retailing forecasting [5], fingerprint matching [6], metagenomic taxonomic classification [7], online sequential learning [8], and feature selection [9]. The  weights of the input layer and hidden layer offsets are randomly generated. The output weight of the network is calculated effectively by minimizing the training error and the norm of the output weight. In addition, many researchers have tried to extend the extreme learning machine model to the support vector machine (SVM) learning framework to solve the classification problem [10]. Frenay et al. [11] found that the transformation performed by the first layer of ELM can be viewed as a kernel that can be plugged into SVM. Due to solving the support vector machine (SVM) type of optimization method that can be utilized to resolve the ELM model, an extreme learning machine based on the optimization method (OPTELM) was proposed in [12]. For binary classification problems, traditional ELM needs to compute all the sample points of training data at the same time in the training stage, which is time-consuming. The singe hyperplane was trained to perform the classification task in the traditional ELM, which enormously restricts its application prospect and the direction of evolution. Jayadeva et al. [13] proposed twin SVM (TWSVM), which is a famous non-parallel hyper-plane classification algorithm for binary classification. Inspired by TWSVM, Wan et al. [14] proposed the twin extreme learning machine (TELM). Compared with ELM, TELM trains two non-parallel hyperplanes for classification tasks by solving two smaller quadratic programming problems (QPPs). Compared with TWSVM, TELM’s optimization problem has fewer constraints, so the training speed is faster and the application prospect is broader. In recent years, researchers have made many improvements to TELM, such as sparse twin extreme learning machine [15], robust twin extreme learning machine [16], time efficient varient of twin extreme learning machine [17], and a generalized adaptive robust distance metric driven smooth regularization learning framework [18], etc.

Although the above ELM-based algorithm has a good classification effect, the statistical knowledge from the data itself is ignored. However, the knowledge of mathematical statistics from the data is very important to construct an efficient classifier. Fisher discriminant analysis (FDA) is an effective discriminant tool by minimizing the intra-class divergence while keeping the inter-class divergence of the data constant. From the above discussion, it can be known that it is necessary to reconstruct a new classification model by combining the characteristics of ELM model and FDA. In recent years, Ma et al. [19] have successfully combined them and proposed a Fisher-regularized extreme learning machine (Fisher-ELM), which not only has the advantages of efficient solution of ELM but also fully considers the statistical knowledge of the data.

Although the above models have good classification performance, most of them consider the $L_2$-norm. When the data contains noise or outliers, they can not deal with noise and outliers well, which degrades the classification performance of the model. In recent years, researchers have tried to introduce the ${\mathit{L}}_1$-norm into various models [20,21,22,23] to reduce the impact of outliers. This studies have shown that the ${\mathit{L}}_1$-norm was able to reduce the effect of outliers to some extent. However, it was still unsatisfactory when the data contains a large number of outliers. Recently, researchers have introduced the idea of truncation into the $L_1$-norm, constructed a new capped ${\mathit{L}}_1$-norm, and applied it to various models [24,25,26]. Many studies [27,28] show that the capped $L_1$-norm not only inherits the advantages of the $L_1$-norm, but also is bounded. So it is more robust and it approaches the $L_0$-norm to some degree. For instance, by applying the capped ${\mathit{L}}_1$-norm to the twin SVM, Wang et al. [29] proposed a new robust twin support vector machine (C$L_1$-TWSVM). Based on twin support vector machine with privileged information [30] (TWSVMPI), a new robust TWSVMPI [31] is proposed by replacing the $L_2$-norm with capped $L_1$-norm. The new model further improves the anti-noise ability of the pattern.

In order to utilize the advantanges of the twin extreme learning machine and FDA, we first put forward to a novel classifier named Fisher-regularized twin extreme learning machine (FTELM). Also considering the instability of the $L_2$-norm for the outliers, we introduce the capped $L_1$-norm into the FTELM model and propose a more robust capped $L_1$-norm FTELM (C$L_1$-FTELM) model.

The main contributions of this paper are as follows:

(1) Based on twin extreme learning machine and Fisher-regularization extreme learning machine (FELM), a new Fisher-regularized twin extreme learning machine (FTELM) is proposed. FTELM minimizes intra-class divergence while fixing the inter-class divergence of samples. FTELM takes full account of the statistical information of the sample data, and the training speed is faster than FELM.

(2) Considering the instability of $L_2$-norm and Hinge loss used by FTELM, we introduce capped $L_1$-norm instead of them and propose a new capped $L_1$-norm FTELM model. C$L_1$-FTELM uses the capped $L_1$-norm to reduce the influence of noise points, and at the same time utilizes Fisher regularization to consider the statistical knowledge of the data.

(3) Two algorithms are designed by utilizing the successive overrelaxation (SOR) [32] technique and the re-weighted technique [27] to solve the optimization problems of the proposed FTELM and C$L_1$-FTELM, respectively.

(4) Two theorems about convergence and local optimality of C$L_1$-FTELM are proved.

The organizational structure of this paper is as follows. In Section 2, we briefly review related work. In Section 3, we describe the FTELM model in detail. The robust capped $L_1$-norm FTELM learning framework along with related theoretical proofs are described in detail in Section 4. In Section 5, we describes numerical experiments on artificial and benchmark datasets. We summarize this paper in Section 6.

2. Related Work

In this section, we first define some concepts of symbols needed for this paper, and then we briefly review Fisher regularization, Fisher-ELM, TELM and successive overrelaxation algorithm.

2.1. The Concept of Symbols

$\mathbf{e}$ is a vector whose components are all ones, an identity matrix is represented by $\mathbf{I}$, and a matrix(vector) of zeros is represented by $\mathbf{0}$. Then, ${∥·∥}_2$ is the ${\mathit{L}}_2$ norm, and ${∥·∥}_{\mathit{F}}$ stands for the Frobenius norm.

A binary classification problem in Euclidean space (${\mathit{R}}^d)$ can be formulated in the following form:

$$T=\left\{x_i,y_i\right\}\in \left(\mathcal{X},\mathcal{Y}\right),\left(i=1,\dots ,m\right)$$

(1)

where $x_i\in \mathcal{X}\subset {\mathit{R}}^{\mathit{d}}$ is expressed as an input sample in a d-dimensional Euclidean space. Similarly, $y_i\in \mathcal{Y}=\left\{-1,+1\right\}$ is represented as an output label corresponding to an input instance $x_i$. In addition, $m_1$ and $m_2$ represent the number of sample data of the positive class and negative class, respectively, and $m=m_1+m_2$.

2.2. Fisher Regularization

Fisher regularization has the following form:

$${∥f∥}_F^2={\mathbf{f}}^T\mathbf{N}\mathbf{f}=\sum _{i\in {\mathcal{I}}_{+}}^{}{\left(f\left(x_i\right)-{\overline{f}}_{+}\right)}^2+\sum _{i\in {\mathcal{I}}_{-}}^{}{\left(f\left(x_i\right)-{\overline{f}}_{-}\right)}^2$$

(2)

where $\mathbf{f}={\left[f\left(x_1\right),f\left(x_2\right),\dots ,f\left(x_m\right)\right]}^T$, $\mathbf{N}=\mathbf{I}-\mathbf{G}$, $\mathbf{I}\in {\mathit{R}}^{m\times m}$ is the identity matrix and $\mathbf{G}$ is the matrix with the elements:

$${\mathbf{G}}_{ij}=\left\{\begin{array}{c}\frac{1}{m_1},fori,j\in {\mathcal{I}}_{+}\hfill \\ \frac{1}{m_2},fori,j\in {\mathcal{I}}_{-}\hfill \\ 0,otherwise\hfill \end{array}\right.$$

(3)

where ${\mathcal{I}}_{\pm }$ are the index sets of positive and negative training data, $m_1=\left|{\mathcal{I}}_{+}\right|$, $m_2=\left|{\mathcal{I}}_{-}\right|$. The average value of $\mathbf{f}\left(\mathbf{x}\right)$ over the positive sample set is expressed as ${\overline{f}}_{+}$, the average value of $\mathbf{f}\left(\mathbf{x}\right)$ over the negative sample set is expressed as ${\overline{f}}_{-}$. From Equation (2), we can see that the meaning of the Fisher regularization is the intra-class divergence of the data.

The proof of Formula (2) is as follows:

$$\begin{array}{cc}\hfill & \sum _{i\in {\mathcal{I}}_{+}}^{}{\left(f\left(x_i\right)-{\overline{f}}_{+}\right)}^2=\sum _{i\in {\mathcal{I}}_{+}}^{}\left(f^2\left(x_i\right)-2·f\left(x_i\right)·{\overline{f}}_{+}+{\overline{f}}_{+}^2\right)=\sum _{i\in \mathcal{I}+}^{}{f}^2\left(x_i\right)-m_1·{\overline{f}}_{+}^2\hfill \\ \hfill & ={\mathbf{f}}_{+}^{\mathbf{T}}·{\mathbf{f}}_{+}-\frac{1}{m_1}·\left({\mathbf{f}}_{+}^{\mathbf{T}}·\mathbf{e}·{\mathbf{e}}^{\mathbf{T}}·{\mathbf{f}}_{+}^{\mathbf{T}}\right)={\mathbf{f}}_{+}^{\mathbf{T}}·{\mathbf{I}}_{+}{\mathbf{f}}_{+}-{\mathbf{f}}_{+}^{\mathbf{T}}·{\mathbf{M}}_{+}·{\mathbf{f}}_{+}^{\mathbf{T}}\hfill \\ \hfill & ={\mathbf{f}}_{+}^{\mathbf{T}}·({\mathbf{I}}_{+}-{\mathbf{M}}_{+})·{\mathbf{f}}_{+}={\mathbf{f}}_{+}^{\mathbf{T}}·\left({\mathbf{N}}_{\mathbf{1}}\right)·{\mathbf{f}}_{+}\hfill \end{array}$$

(4)

where $\mathbf{e}={\left[1,\dots ,1\right]}^T$ is a vector of $m_1$ dimensions, ${\mathbf{f}}_{+}=\left(f\left(x_1\right),f\left(x_2\right),\dots ,f\left(x_i\right),\dots ,f\left(x_{m_1}\right)\right), i\in {\mathcal{I}}_{+}$, ${\mathbf{I}}_{+}\in {\mathit{R}}^{m_1\times m_1}$ is the identity matrix. ${\mathbf{M}}_{+}\in {\mathit{R}}^{m_1\times m_1}$, and all the entries in the matrix ${\mathbf{M}}_{+}$ are $\frac{1}{m_1}$.

Similarly, it can be obtained:

$$\begin{array}{c}\hfill \sum _{j\in {\mathcal{I}}_{-}}^{}{\left(f\left(x_i\right)-{\overline{f}}_{-}\right)}^2={\mathbf{f}}_{-}^{\mathbf{T}}·({\mathbf{I}}_{-}-{\mathbf{M}}_{-})·{\mathbf{f}}_{-}={\mathbf{f}}_{-}^{\mathbf{T}}·\left({\mathbf{N}}_{\mathbf{2}}\right)·{\mathbf{f}}_{-}\end{array}$$

(5)

where ${\mathbf{f}}_{-}=\left(f\left(x_1\right),f\left(x_2\right),\dots ,f\left(x_i\right),\dots ,f\left(x_{m_2}\right)\right),i\in {\mathcal{I}}_{-}$, ${\mathbf{I}}_{-}\in {\mathit{R}}^{m_2\times m_2}$ is the identity matrix. ${\mathbf{M}}_{-}\in {\mathit{R}}^{m_2\times m_2}$, and all the entries in the matrix ${\mathbf{M}}_{-}$ are $\frac{1}{m_2}$.

Combining Equations (4) and (5), we can get another form of Equation (2):

$$\begin{array}{cc}\hfill & {\mathbf{f}}_{+}^{\mathbf{T}}·({\mathbf{I}}_{+}-{\mathbf{M}}_{+})·{\mathbf{f}}_{+}+{\mathbf{f}}_{-}^{\mathbf{T}}·({\mathbf{I}}_{-}-{\mathbf{M}}_{-})·{\mathbf{f}}_{-}\hfill \\ \hfill & ={\left({\mathbf{f}}_{+},{\mathbf{f}}_{-}\right)}^T·\left[\mathbf{I}-\left[\begin{array}{cc}{\mathbf{M}}_{+}& {\mathbf{0}}_{\mathbf{1}}\\ {\mathbf{0}}_{\mathbf{2}}& {\mathbf{M}}_{-}\end{array}\right]\right]·\left({\mathbf{f}}_{+},{\mathbf{f}}_{-}\right)\hfill \\ \hfill & ={\mathbf{f}}^{\mathbf{T}}·\left(\mathbf{I}-\mathbf{G}\right)·\mathbf{f}={\mathbf{f}}^{\mathbf{T}}·\mathbf{N}·\mathbf{f}\hfill \end{array}$$

(6)

where ${\mathbf{0}}_{\mathbf{1}}\in {\mathbf{0}}^{m_1\times m_2}$, ${\mathbf{0}}_{\mathbf{2}}\in {\mathbf{0}}^{m_2\times m_1}$, $G=\left[\begin{array}{cc}{\mathbf{M}}_{+}& {\mathbf{0}}_{\mathbf{1}}\\ {\mathbf{0}}_{\mathbf{2}}& {\mathbf{M}}_{-}\end{array}\right]$.

2.3. Fisher-Regularized Extreme Learning Machine

The primal problem of Fisher-regularized extreme learning machine (FELM) is as follows:

$$\begin{array}{cc}\hfill & \underset{\alpha ,\xi }{min}\frac{1}{2}{\beta }^T·\beta +C_1·{\mathbf{e}}^{\mathbf{T}}·\xi +\frac{1}{2}{C}_2·{\alpha }^T·{\mathbf{K}}_{ELM}·\mathbf{N}·{\mathbf{K}}_{ELM}.\alpha \hfill \\ \hfill & s.t.\mathbf{Y}·\mathbf{H}·\beta \ge \mathbf{e}-\xi \hfill \\ \hfill & \xi \ge \mathbf{0}\hfill \end{array}$$

(7)

According to the representer theorem $\beta ={\sum }_{i=1}^m{\alpha }_i\mathbf{h}\left(x_i\right)={\mathbf{H}}^{{}^T}\alpha$, problem (7) can be written as problem (8):

$$\begin{array}{cc}\hfill & \underset{\alpha ,\xi }{min}\frac{1}{2}{\alpha }^T·{\mathbf{K}}_{ELM}·\alpha +C_1·{\mathbf{e}}^{\mathbf{T}}·\xi +\frac{1}{2}{C}_2·{\alpha }^T·{\mathbf{K}}_{ELM}·\mathbf{N}·{\mathbf{K}}_{ELM}.\alpha \hfill \\ \hfill & s.t.\mathbf{Y}·{\mathbf{K}}_{\mathbf{ELM}}·\alpha \ge \mathbf{e}-\xi \hfill \\ \hfill & \xi \ge \mathbf{0}\hfill \end{array}$$

(8)

where ${\mathbf{K}}_{ELM}\in {\mathit{R}}^{m\times m}$ is a Gram matrix with elements $k_{ELM}\left(x_i,x_j\right)$, $k_{ELM}\left(x_i,x\right)=\mathbf{h}{\left(x\right)}^T·\mathbf{h}\left(x_i\right)$, $\mathbf{h}\left(x\right)$ denotes the output of some hidden node, $\mathbf{Y}\in {\mathit{R}}^{m\times m}$ is a diagonal matrix with elements $y_i$, $C_1$, $C_2$ are the regularization parameters, and $\xi$ is a nonnegative slack vector.

According to the optimization theory, the dual form of the problem (8) can be obtained as follows:

$$\begin{array}{cc}\hfill & \underset{\theta }{min}\frac{1}{2}{\theta }^T·\mathbf{Q}·\theta -{\mathbf{e}}^T·\theta \hfill \\ \hfill & s.t.\mathbf{0}\le \theta \le C_1·\mathbf{e}\hfill \end{array}$$

(9)

where $\mathbf{Q}=\mathbf{Y}·{\left({\left(\mathbf{I}+C_2·\mathbf{N}·{\mathbf{K}}_{ELM}\right)}^{-1}\right)}^T·{\mathbf{K}}_{ELM}·\mathbf{Y}$.

The decision function of Fisher-regularized extreme learning machine is:

$$\mathit{f}\left(x\right)=sign\left(\sum _{i=1}^m{\alpha }_i·k_{ELM}\left(x_i,x\right)\right)$$

(10)

2.4. Twin Extreme Learning Machine

Similar to the form of TWSVM [13], the primal problem of TELM [14] can be expressed in the following:

$$\begin{array}{c}\hfill \mathbf{Primal}{\mathbf{TELM}}_{\mathbf{1}}:\underset{{\beta }_1}{min}\frac{1}{2}{∥{\mathbf{H}}_1·{\beta }_1∥}_2^2+C_1·{\mathbf{e}}_{\mathbf{2}}^{\mathbf{T}}·\xi \\ \hfill s.t.-{\mathbf{H}}_{\mathbf{2}}·{\beta }_1\ge {\mathbf{e}}_{\mathbf{2}}-\xi \\ \hfill \xi \ge \mathbf{0}\end{array}$$

(11)

$$\begin{array}{c}\hfill \mathbf{Primal}{\mathbf{TELM}}_{\mathbf{2}}:\underset{{\beta }_2}{min}\frac{1}{2}{∥{\mathbf{H}}_2·{\beta }_2∥}_2^2+C_2·{\mathbf{e}}_{\mathbf{1}}^{\mathbf{T}}·\eta \\ \hfill s.t.{\mathbf{H}}_{\mathbf{1}}·{\beta }_2\ge {\mathbf{e}}_{\mathbf{1}}-\eta \\ \hfill \eta \ge \mathbf{0}\end{array}$$

(12)

where ${\mathbf{H}}_1$ and ${\mathbf{H}}_2$ respresent the outputs of the hidden layer for positive and negative samples, $\xi$ and $\eta$ represent the slack vectors, $\mathbf{0}$ is a zero vector, $C_1$, $C_2\ge 0$ are penalty parameters, ${\mathbf{e}}_1\in {\mathit{R}}^{m_1}$ and ${\mathbf{e}}_2\in {\mathit{R}}^{m_2}$ are vectors of ones.

By introducing Lagrange multipliers $\alpha$ and $\vartheta$, the dual problem of (11) and (12) can be written as follows:

$$\begin{array}{c}\hfill \mathbf{Dual}{\mathbf{TELM}}_{\mathbf{1}}:\underset{\alpha }{min}\frac{1}{2}{\alpha }^T·{\mathbf{H}}_2{\left({\mathbf{H}}_1^T·{\mathbf{H}}_{\mathbf{1}}\right)}^{-1}·{\mathbf{H}}_2^T·\alpha -{\mathbf{e}}_{\mathbf{2}}^{\mathbf{T}}·\alpha \\ \hfill s.t.\mathbf{0}\le \alpha \le C_1·{\mathbf{e}}_2\end{array}$$

(13)

$$\begin{array}{c}\hfill \mathbf{Dual}{\mathbf{TELM}}_{\mathbf{2}}:\underset{\vartheta }{min}\frac{1}{2}{\vartheta }^T·{\mathbf{H}}_1{\left({\mathbf{H}}_2^T·{\mathbf{H}}_{\mathbf{2}}\right)}^{-1}·{\mathbf{H}}_1^T·\vartheta -{\mathbf{e}}_{\mathbf{1}}^{\mathbf{T}}·\vartheta \\ \hfill s.t.\mathbf{0}\le \vartheta \le C_2·{\mathbf{e}}_1\end{array}$$

(14)

The solution of (13) and (14) are as follows:

$$\begin{array}{c}\hfill {\beta }_1=-{\left({\mathbf{H}}_{\mathbf{1}}^{\mathbf{T}}·{\mathbf{H}}_{\mathbf{1}}+{△}_1\mathbf{I}\right)}^{-1}·{\mathbf{H}}_2^T·\alpha \end{array}$$

(15)

$$\begin{array}{c}\hfill {\beta }_2=-{\left({\mathbf{H}}_{\mathbf{2}}^{\mathbf{T}}·{\mathbf{H}}_{\mathbf{2}}+{△}_2\mathbf{I}\right)}^{-1}·{\mathbf{H}}_1^T·\vartheta \end{array}$$

(16)

where ${△}_1$ and ${△}_2$ are two small positive constants and $\mathbf{I}$ is an identity matrix. The decision function of twin extreme learning machine is:

$$f\left(x\right)=arg\underset{k=1,2}{min}{d}_k\left(x\right)=arg\underset{k=1,2}{min}\left|{\beta }_k^T·\mathbf{h}\left(\mathbf{x}\right)\right|$$

(17)

2.5. Successive Overrelaxation Algorithm

The successive overrelaxation algorithm [32] mainly aims at the following optimization problems:

$$\begin{array}{cc}\hfill & \underset{\mu }{min}\frac{1}{2}{∥{\mathbf{H}}^T\mu ∥}_2^2-{\mathbf{e}}^T\mu \hfill \\ \hfill & s.t.\mu \in S=\left\{\mu |\mathbf{0}\le \mu \le C\mathbf{e}\right\}\hfill \end{array}$$

(18)

Let $\mathbf{H}{\mathbf{H}}^T=\mathbf{L}+\mathbf{E}+{\mathbf{L}}^T$, the strictly lower triangular matrix of the matrix $\mathbf{H}{\mathbf{H}}^T$ is $\mathbf{L}$, and the diagonal elements of the matrix $\mathbf{H}{\mathbf{H}}^T$ form the diagonal matrix $\mathbf{E}$.

The gradient projection optimality condition is the necessary and sufficient optimality condition for Equation (18):

$$\mu ={\left(\mu -\pi {\mathbf{E}}^{-1}\left(\mathbf{H}{\mathbf{H}}^T\mu -\mathbf{e}\right)\right)}_{\#},\pi \ge 0$$

where the 2-norm projection onto the feasible region of Equation (18) is denoted by ${\left(·\right)}_{\#}$, that is:

$${\left({\left(\mu \right)}_{\#}\right)}_i=\left\{\begin{array}{c}0,if{\mu }_i\le 0,i=1,2,\dots ,m\hfill \\ {\mu }_i,if0<{\mu }_i<C,i=1,2,\dots ,m\hfill \\ C,if{\mu }_i\ge C,i=1,2,\dots ,m\hfill \end{array}\right.$$

(19)

The matrix $\mathbf{H}{\mathbf{H}}^T$ is expressed in the following form:

$$\begin{array}{cc}\hfill & \mathbf{H}{\mathbf{H}}^T={\pi }^{-1}\mathbf{E}\left(\mathbf{B}+\mathbf{C}\right)\hfill \\ & s.t.\mathbf{B}-\mathbf{C}ispositivedefinite\hfill \end{array}$$

(20)

Here:

$$\mathbf{B}=\left(\mathbf{I}+\pi {\mathbf{E}}^{-1}\mathbf{L}\right),\mathbf{C}=\left(\left(\pi -1\right)\mathbf{I}+\pi {\mathbf{E}}^{-1}{\mathbf{L}}^T\right),0<\pi <2$$

(21)

According to the [33], the matrix splitting algorithm is as follows:

$${\mu }^{i+1}={\left({\mu }^{i+1}-\mathbf{B}{\mu }^{i+1}-\mathbf{C}{\mu }^i+\pi {\mathbf{E}}^{-1}\mathbf{e}\right)}_{\#}$$

(22)

Substituting Equation (21) into Equation (22), it can be obtained:

$${\mu }^{i+1}={\left({\mu }^i-\pi {\mathbf{E}}^{-1}\left(\mathbf{H}{\mathbf{H}}^T{\mu }^i-\mathbf{e}+\mathbf{L}\left({\mu }^{i+1}-{\mu }^i\right)\right)\right)}_{\#}$$

(23)

3. Fisher-Regularized Twin Extreme Learning Machine

3.1. Model Formulation

As mentioned above, TELM solves two smaller QPPs, which can get the solution quickly. However, it ignores the prior statistical knowledge from data. FELM minimizes the within-class scatter while controlling the between-class scatter of samples, but FELM needs to solve a large-scale quadratic programming problems which is time-consuming. In this paper, by combining the advantages of FELM and TELM, we first propose the Fisher-regularized twin extreme learning machine (FTELM) by introducing the Fisher regularization into the TELM feature space. FTELM only needs to solve two smaller quadratic programming problems and meanwhile utilizes the prior statistical knowledge from data. The pair of FTELM primal problems is as follows:

$$\begin{array}{c}\hfill \mathbf{Primal}{\mathbf{FTELM}}_{\mathbf{1}}:\underset{{\beta }_1,\xi }{min}\frac{1}{2}{∥{\mathbf{H}}_1·{\beta }_1∥}^2+C_1·{\mathbf{e}}_2^T·\xi +\frac{C_2}{2}·{\mathbf{f}}_1{\left(x\right)}^T·{\mathbf{N}}_1·{\mathbf{f}}_1\left(x\right)\\ \hfill s.t.-{\mathbf{H}}_2·{\beta }_1+\xi \ge {\mathbf{e}}_2\\ \hfill \xi \ge \mathbf{0}\end{array}$$

(24)

$$\begin{array}{c}\hfill \mathbf{Primal}{\mathbf{FTELM}}_{\mathbf{2}}:\underset{{\beta }_2,\eta }{min}\frac{1}{2}{∥{\mathbf{H}}_2·{\beta }_2∥}^2+C_3·{\mathbf{e}}_1^T·\eta +\frac{C_4}{2}·{\mathbf{f}}_2{\left(x\right)}^T·{\mathbf{N}}_2·{\mathbf{f}}_2\left(x\right)\\ \hfill s.t.{\mathbf{H}}_1·{\beta }_2+\eta \ge {\mathbf{e}}_1\\ \hfill \eta \ge \mathbf{0}\end{array}$$

(25)

From the Equations (4) and (5), we can know that ${\mathbf{N}}_1={\mathbf{I}}_{+}-{\mathbf{M}}_{+}$ and ${\mathbf{N}}_2={\mathbf{I}}_{-}-{\mathbf{M}}_{-}$, $C_1$, $C_2$, $C_3$, $C_4>0$ are regularization parameters, $\xi$ and $\eta$ are the error vectors, and all the elements in vectors ${\mathbf{e}}_1\in R^{m_1}$ and ${\mathbf{e}}_2\in R^{m_2}$ are one. FTELM first inherits the advantage of the classical twin extreme learning machine, which computes two non-parallel hyperplanes to solve the classification problem. Secondly, FTELM takes full account of the statistical information of the samples and further improves the classification accuracy of the classifier. The optimization objective function in (24) of FTELM mainly has three terms: minimizing the distance from the positive class sample points to the positive class hyperplane, minimizing empirical loss, and minimizing the intra-class divergence from the samples. The constraint condition in (24) of the optimization objective function is that the distance between the negative class sample points and the positive class hyperplane is greater than or equal to one. In a word, FTELM makes the positive class sample points closer to the positive class hyperplane, and the negative class sample points far away from the positive class hyperplane. At the same time, the positive class sample points are more concentrated in the center of the positive class sample points. There is a similar explanation for the model (25).

According to the representer theorem $\beta ={\sum }_{i=1}^m{\alpha }_i\mathbf{h}\left(x_i\right)={\mathbf{H}}^{{}^T}\alpha$, then ${\beta }_1={\mathbf{H}}_1^T·{\alpha }_1$ and ${\beta }_2={\mathbf{H}}_2^T·{\alpha }_2$. We know that $\mathbf{f}=\mathbf{H}·\beta$. Therefore, the problem (24) and (25) can be written in the following forms:

$$\begin{array}{cc}\hfill & \underset{{\alpha }_1,\xi }{min}\frac{1}{2}{\alpha }_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{K}}_{ELM1}·{\alpha }_1+C_1·{\mathbf{e}}_2^T·\xi +\frac{C_2}{2}{\alpha }_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\alpha }_1\hfill \\ \hfill & s.t.-{\mathbf{H}}_{\mathbf{2}}·{\mathbf{H}}_1^T·{\alpha }_1+\xi \ge {\mathbf{e}}_{\mathbf{2}}\hfill \\ \hfill & \xi \ge \mathbf{0}\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill & \underset{{\alpha }_2,\eta }{min}\frac{1}{2}{\alpha }_2^T·{\mathbf{K}}_{ELM2}·{\mathbf{K}}_{ELM2}·{\alpha }_2+C_3·{\mathbf{e}}_1^T·\eta +\frac{C_4}{2}{\alpha }_2^T·{\mathbf{K}}_{ELM2}·{\mathbf{N}}_2·{\mathbf{K}}_{ELM2}·{\alpha }_2\hfill \\ \hfill & s.t.{\mathbf{H}}_{\mathbf{1}}·{\mathbf{H}}_2^T·{\alpha }_2+\eta \ge {\mathbf{e}}_{\mathbf{1}}\hfill \\ \hfill & \eta \ge \mathbf{0}\hfill \end{array}$$

(27)

where ${\mathbf{K}}_{ELM1}={\mathbf{H}}_1·{\mathbf{H}}_{\mathbf{1}}^{\mathbf{T}}$ and ${\mathbf{K}}_{ELM2}={\mathbf{H}}_2·{\mathbf{H}}_{\mathbf{2}}^{\mathbf{T}}$ are Gram matrices.

3.2. Model Solution

Introducing Lagrange multipliers $\theta ={\left({\theta }_1,\dots ,{\theta }_{m_2}\right)}^T$ and $\vartheta ={\left({\vartheta }_1,\dots ,{\vartheta }_{m_2}\right)}^T$, the Lagrange function of (26) can be written as follows:

$$\begin{array}{cc}\hfill & \mathcal{L}\left({\alpha }_1,\xi ,\theta ,\vartheta \right)=\frac{1}{2}{\alpha }_1^T·{\mathbf{K}}_{ELM1}·\left({\mathbf{I}}_1+C_2·{\mathbf{N}}_1\right)·{\mathbf{K}}_{ELM1}·{\alpha }_1+C_1·{\mathbf{e}}_2^T·\xi \hfill \\ \hfill & -{\theta }^T·\left(-{\mathbf{H}}_2·{\mathbf{H}}_1^T·{\alpha }_1+\xi -{\mathbf{e}}_2\right)-{\vartheta }^T·\xi \hfill \end{array}$$

(28)

According to the KKT conditions, we get:

$$\frac{\partial \mathcal{L}}{\partial {\alpha }_1}={\mathbf{K}}_{ELM1}·\left({\mathbf{I}}_1+C_2·{\mathbf{N}}_1\right)·{\mathbf{K}}_{ELM1}·{\alpha }_1+{\mathbf{H}}_1·{\mathbf{H}}_{\mathbf{2}}^{\mathbf{T}}·\theta =\mathbf{0}$$

(29)

$$\frac{\partial \mathcal{L}}{\partial \xi }=C_1·{\mathbf{e}}_2-\theta -\vartheta =\mathbf{0}$$

(30)

$${\theta }^T·\left(-{\mathbf{H}}_2·{\mathbf{H}}_1^T·{\alpha }_1+\xi -{\mathbf{e}}_2\right)=\mathbf{0}$$

(31)

$${\vartheta }^T·\xi =\mathbf{0}$$

(32)

$$\theta \ge \mathbf{0}$$

(33)

$$\vartheta \ge \mathbf{0}$$

(34)

From (29) and (30), we can get:

$${\alpha }_1^{*}=-{\left({\mathbf{K}}_{ELM1}·\left({\mathbf{I}}_1+C_2·{\mathbf{N}}_1\right)·{\mathbf{K}}_{ELM1}\right)}^{-1}·{\mathbf{H}}_1·{\mathbf{H}}_2^T·\theta$$

(35)

$$\mathbf{0}\le \theta \le C_1·{\mathbf{e}}_2$$

(36)

By substituting (29)–(34) into (28), the dual optimization problem for (26) can be written in the following form:

$$\begin{array}{c}\hfill \mathbf{Dual}{\mathbf{FTELM}}_{\mathbf{1}}:\underset{\theta }{min}\frac{1}{2}{\theta }^T·{\mathbf{Q}}_{\mathbf{1}}·\theta -{\mathbf{e}}_{\mathbf{2}}^{\mathbf{T}}·\theta \\ \hfill s.t.\mathbf{0}\le \theta \le C_1·{\mathbf{e}}_2\end{array}$$

(37)

Here ${\mathbf{Q}}_1={\mathbf{H}}_2·{\mathbf{H}}_1^T·{\left({\mathbf{K}}_{ELM1}·\left({\mathbf{I}}_1+C_2·{\mathbf{N}}_1\right)·{\mathbf{K}}_{ELM1}\right)}^{-1}·{\mathbf{H}}_1·{\mathbf{H}}_2^T$.

Similarly, we can obtain the dual of (27) as:

$$\begin{array}{c}\hfill \mathbf{Dual}{\mathbf{FTELM}}_{\mathbf{2}}:\underset{\lambda }{min}\frac{1}{2}{\lambda }^T·{\mathbf{Q}}_{\mathbf{2}}·\lambda -{\mathbf{e}}_{\mathbf{1}}^{\mathbf{T}}·\lambda \\ \hfill s.t.\mathbf{0}\le \lambda \le C_3·{\mathbf{e}}_1\end{array}$$

(38)

Here $\lambda ={\left({\lambda }_1,\dots ,{\lambda }_{m_1}\right)}^T$ is the vector of Lagrange multipliers and we can get: ${\mathbf{Q}}_2={\mathbf{H}}_1·{\mathbf{H}}_2^T·{\left({\mathbf{K}}_{ELM2}·\left({\mathbf{I}}_2+C_4·{\mathbf{N}}_2\right)·{\mathbf{K}}_{ELM2}\right)}^{-1}·{\mathbf{H}}_2·{\mathbf{H}}_1^T$.

We use the successive overrelaxation (SOR) [32] technique to solve the convex quadratic optimization problems of (37) and (38) (The SOR-FTELM algorithm is summarized as Algorithm 1). We can get $\theta$ and $\lambda$. Therefore, we can obtain the solution for problems of (24) and (25) in the following:

$${\beta }_1=-{\mathbf{H}}_1^T·{\left({\mathbf{K}}_{ELM1}·\left({\mathbf{I}}_1+C_2·{\mathbf{N}}_1\right)·{\mathbf{K}}_{ELM1}+{\delta }_1·{\mathbf{I}}_{\mathbf{1}}\right)}^{-1}·{\mathbf{H}}_1·{\mathbf{H}}_2^T·\theta$$

(39)

$${\beta }_2={\mathbf{H}}_2^T·{\left({\mathbf{K}}_{ELM2}·\left({\mathbf{I}}_2+C_4·{\mathbf{N}}_2\right)·{\mathbf{K}}_{ELM2}+{\delta }_2·{\mathbf{I}}_{\mathbf{2}}\right)}^{-1}·{\mathbf{H}}_2·{\mathbf{H}}_1^T·\lambda$$

(40)

The decision function of FTELM is:

$$f\left(x\right)=arg\underset{k=1,2}{min}\left|{\beta }_k^T·\mathbf{h}\left(\mathbf{x}\right)\right|$$

(41)

Algorithm 1 The procedure of SOR-FTELM.

Input:        Training set $T={\left\{x_i,y_i\right\}}_{i=1}^m$, where $x_i\in {\mathit{R}}^d$, $y_i=\pm 1$, the number of hidden node number $\mathit{L}$, tolerance $\epsilon$, regularization parameters $C_1$, $C_2$, $C_3$, $C_4$. Output:     ${\beta }_1$, ${\beta }_2$, and the decision function of FTELM. 1: Compute the graph matrix ${\mathbf{N}}_1$, ${\mathbf{N}}_2$ by Equations (4) and (5). 2: Choose an activation function such as $G\left(x\right)=\frac{1}{1+e^{-x}}$ and compute the hidden layer output matrix ${\mathbf{H}}_1$, ${\mathbf{H}}_2$ by $\mathbf{h}\left(x_i\right)=G\left({\sum }_{j=1}^d{\omega }_{ji}{x}_j+b_i\right)$ and compute ${\mathbf{K}}_{ELM1}$=${\mathbf{H}}_1{\mathbf{H}}_1^T$ and ${\mathbf{K}}_{ELM2}$=${\mathbf{H}}_2{\mathbf{H}}_2^T$. 3: Choose $t\in \left(0,2\right)$, start with any ${\theta }^0\in {\mathit{R}}^{m_2}$,Having ${\theta }^i$, compute ${\theta }^{i+1}$ as follows: $${\theta }^{i+1}={\left({\theta }^i-t{\mathbf{E}}_1\left({\mathbf{Q}}_1{\theta }^i-{\mathbf{e}}_2+{\mathbf{L}}_1\left({\theta }^{i+1}-{\theta }^i\right)\right)\right)}_{\#}$$    until $\left|{\theta }^{i+1}-{\theta }^i\right|\le \epsilon$, where ${\mathbf{e}}_2$ is a vector of ones of appropriate dimensions. ${\mathbf{L}}_1\in {\mathit{R}}^{{\mathit{m}}_{\mathit{2}}\times {\mathit{m}}_{\mathit{2}}}$ is the strictly lower triangular matrix, where $l_{ij}=q_{ij},i>j$. ${\mathbf{E}}_1\in {\mathit{R}}^{{\mathit{m}}_{\mathit{2}}\times {\mathit{m}}_{\mathit{2}}}$ is the diagonal matrix, where $e_{ij}=q_{ij}$, $i>j$.    Then, given any ${\lambda }^0\in {\mathit{R}}^{m_1}$, Having ${\lambda }^i$, compute ${\lambda }^{i+1}$ as follows $${\lambda }^{i+1}={\left({\lambda }^i-t{\mathbf{E}}_2\left({\mathbf{Q}}_2{\lambda }^i-{\mathbf{e}}_1+{\mathbf{L}}_2\left({\lambda }^{i+1}-{\lambda }^i\right)\right)\right)}_{\#}$$ 4: Compute the output weights ${\beta }_1,{\beta }_2$ using Equations (39) and (40). 5: Construct the following decision functions: $$f\left(x\right)=arg\underset{k=1,2}{min}\left|{\beta }_k^T·\mathbf{h}\left(\mathbf{x}\right)\right|$$

4. Capped ${\mathit{L}}_{\mathbf{1}}$-Norm Fisher-Regularized Twin Extreme Learning Machine

4.1. Model Formulation

The Fisher-regularized twin extreme learning machine proposed in the previous section not only inherits the advantages of the twin extreme learning machine but also makes full use of the statistical information of the samples. However, due to the use of the squared ${\mathit{L}}_2$-norm distance and hinge loss function, the Fisher-regularized twin extreme learning machine is not robust enough when noisy points are present, which often enlarges the impact of abnormal values. In order to reduce the influence of outliers and improve the robustness of the FTELM, we propose a capped ${\mathit{L}}_1$-norm Fisher twin extreme machine (C$L_1$-FTELM) by replacing the ${\mathit{L}}_2$-norm and hinge loss in the FTELM with capped ${\mathit{L}}_1$-norm. The primal C$L_1$-FTELM is in the following:

Primal C$L_1$-FTELM${}_1$:

$$\begin{array}{c}\hfill \underset{{\alpha }_1,\xi }{min}\sum _{i=1}^{m_1}\underset{}{min}\left({∥{\mathbf{h}}^T\left(x_i\right)·{{\mathbf{H}}_{\mathbf{1}}}^T·{\alpha }_1∥}_1,{\epsilon }_1\right)+C_1·\sum _{j=1}^{m_2}\underset{}{min}\left({∥{\xi }_j∥}_1,{\epsilon }_2\right)\\ \hfill +\frac{C_2}{2}·{\alpha }_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\alpha }_1\\ \hfill s.t.-{\mathbf{H}}_2·{\mathbf{H}}_1^T·{\alpha }_1+\xi \ge {\mathbf{e}}_2\end{array}$$

(42)

Primal C$L_1$-FTELM${}_2$:

$$\begin{array}{c}\hfill \underset{{\alpha }_2,\eta }{min}\sum _{j=1}^{m_2}\underset{}{min}\left({∥{\mathbf{h}}^T\left(x_j\right)·{{\mathbf{H}}_{\mathbf{2}}}^T·{\alpha }_2∥}_1,{\epsilon }_3\right)+C_3·\sum _{i=1}^{m_1}\underset{}{min}\left({∥{\eta }_i∥}_1,{\epsilon }_4\right)\\ \hfill +\frac{C_4}{2}·{\alpha }_2^T·{\mathbf{K}}_{ELM2}·{\mathbf{N}}_2·{\mathbf{K}}_{ELM2}·{\alpha }_2\\ \hfill s.t.{\mathbf{H}}_1·{\mathbf{H}}_2^T·{\alpha }_2+\eta \ge {\mathbf{e}}_1\end{array}$$

(43)

where $C_1$, $C_2$, $C_3$, $C_4$ > 0 are regularization parameters, ${\epsilon }_1$, ${\epsilon }_2$, ${\epsilon }_3$, ${\epsilon }_4$ are thresholding parameters.

C$L_1$-FTELM uses the capped $L_1$-norm to reduce the influence of noise points, and at the same time utilizes Fisher regularization to consider the statistical knowledge of the data. Based on FTELM, C$L_1$-FTELM changes the $L_2$-norm metric and Hinge loss function of the original model to the capped $L_1$-norm. The capped $L_1$-norm is bounded and can constrain the impact of noise within a certain range. Therefore, the anti-noise ability of the model can be improved. The optimization objective function in (42) of C$L_1$-FTELM also contains three terms: minimizing the distance between the positive class sample points and the positive class hyperplane by using capped $L_1$-norm metric, minimizing empirical loss by using capped $L_1$-norm loss function, and minimizing the within-class scatter of the samples. The constraints in (42) of C$L_1$-FTELM are as follows: the distance between the negative class sample points and the positive class hyperplane is greater than or equal to one. In summary, C$L_1$-FTELM inherits the advantages of FTELM, while further improving the noise immunity of the model by replacing the metric and loss function with the capped $L_1$-norm. However, the C$L_1$-FTELM is a non-convex and non-smooth problem. Here, we use the reweighting technique [27] to solve the problem corresponding to the C$L_1$-FTELM model, which is shown below:

C$L_1$-FTELM${}_1$:

$$\begin{array}{c}\hfill \underset{{\alpha }_1,\xi }{min}\frac{1}{2}{\alpha }_1^T·{\mathbf{K}}_{ELM1}·\mathbf{F}·{\mathbf{K}}_{ELM1}·{\alpha }_1+\frac{C_1}{2}·{\xi }^T·\mathbf{D}·\xi \\ \hfill +\frac{C_2}{2}·{\alpha }_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\alpha }_1\\ \hfill s.t.-{\mathbf{H}}_2·{\mathbf{H}}_1^T·{\alpha }_1+\xi \ge {\mathbf{e}}_2\end{array}$$

(44)

where $\mathbf{F}$ and $\mathbf{D}$ are two diagonal matrices with i-th and j-th diagonal elements as:

$$f_i=\left\{\begin{array}{c}\frac{1}{\left|{\mathbf{h}}^T\left(x_i\right)·{\mathbf{H}}_1^T·{\alpha }_1\right|},\left|{\mathbf{h}}^T\left(x_i\right)·{\mathbf{H}}_1^T·{\alpha }_1\right|\le {\epsilon }_1,i\in \left(1,\dots ,m_1\right)\hfill \\ \\ {\sigma }_1,otherwise\hfill \end{array}\right.$$

(45)

$$d_j=\left\{\begin{array}{c}\frac{1}{\left|{\xi }_j\right|},\left|{\xi }_j\right|\le {\epsilon }_2,j\in \left(1,\dots ,m_2\right)\hfill \\ \\ {\sigma }_2,otherwise\hfill \end{array}\right.$$

(46)

Here ${\sigma }_1$, ${\sigma }_2$ are two small constants.

C$L_1$-FTELM${}_2$:

$$\begin{array}{c}\hfill \underset{{\alpha }_2,\eta }{min}\frac{1}{2}{\alpha }_2^T·{\mathbf{K}}_{ELM2}·\mathbf{R}·{\mathbf{K}}_{ELM2}·{\alpha }_2+\frac{C_3}{2}·{\eta }^T·\mathbf{S}·\eta \\ \hfill +\frac{C_4}{2}·{\alpha }_2^T·{\mathbf{K}}_{ELM2}·{\mathbf{N}}_2·{\mathbf{K}}_{ELM2}·{\alpha }_2\\ \hfill s.t.{\mathbf{H}}_1·{\mathbf{H}}_2^T·{\alpha }_2+\eta \ge {\mathbf{e}}_1\end{array}$$

(47)

where $\mathbf{R}$ and $\mathbf{S}$ are two diagonal matrices with j-th and i-th diagonal elements as:

$$r_j=\left\{\begin{array}{c}\frac{1}{\left|{\mathbf{h}}^T\left(x_j\right)·{\mathbf{H}}_2^T·{\alpha }_2\right|},\left|{\mathbf{h}}^T\left(x_j\right)·{\mathbf{H}}_2^T·{\alpha }_2\right|\le {\epsilon }_3,j\in \left(1,\dots ,m_2\right)\hfill \\ \\ {\sigma }_3,otherwise\hfill \end{array}\right.$$

(48)

$$s_i=\left\{\begin{array}{c}\frac{1}{\left|{\eta }_i\right|},\left|{\eta }_i\right|\le {\epsilon }_4,i\in \left(1,\dots ,m_1\right)\hfill \\ \\ {\sigma }_4,otherwise\hfill \end{array}\right.$$

(49)

Here ${\sigma }_3$, ${\sigma }_4$ are two small constants.

4.2. Model Solution

Introducing Lagrange multipliers $\alpha$, the Lagrange function of (44) can be written as follows:

$$\begin{array}{cc}\hfill & \mathcal{L}\left({\alpha }_1,\xi ,\alpha \right)=\frac{1}{2}{\alpha }_1^T·{\mathbf{K}}_{ELM1}·\left(\mathbf{F}+C_2·{\mathbf{N}}_1\right)·{\mathbf{K}}_{ELM1}·{\alpha }_1+\frac{C_1}{2}·{\xi }^T·\mathbf{D}·\xi \hfill \\ \hfill & -{\alpha }^T·\left(-{\mathbf{H}}_2·{\mathbf{H}}_1^T·{\alpha }_1+\xi -{\mathbf{e}}_2\right)\hfill \end{array}$$

(50)

According to the KKT conditions, we can get the following formula:

$$\frac{\partial \mathcal{L}}{\partial {\alpha }_1}={\mathbf{K}}_{ELM1}·\left(\mathbf{F}+C_2·{\mathbf{N}}_1\right)·{\mathbf{K}}_{ELM1}·{\alpha }_1+{\mathbf{H}}_1·{\mathbf{H}}_2^T·\alpha =0$$

(51)

$$\frac{\partial \mathcal{L}}{\partial \xi }=C_1·\mathbf{D}·\xi -\alpha =0$$

(52)

$${\alpha }^T·\left(-{\mathbf{H}}_2·{\mathbf{H}}_1^T·{\alpha }_1+\xi -{\mathbf{e}}_2\right)=0$$

(53)

$$\alpha \ge 0$$

(54)

From Equations (51) and (52), we can get:

$$\begin{array}{cc}\hfill & {\alpha }_1=-{\left({\mathbf{K}}_{ELM1}·\left(\mathbf{F}+C_2·{\mathbf{N}}_1\right)·{\mathbf{K}}_{ELM1}\right)}^{-1}·{\mathbf{H}}_1·{\mathbf{H}}_2^T·\alpha \hfill \\ \hfill & \xi =\frac{1}{C_1}·{\mathbf{D}}^{-1}·\alpha \hfill \end{array}$$

Similarly, we can get:

$$\begin{array}{cc}\hfill & {\alpha }_2={\left({\mathbf{K}}_{ELM2}·\left(\mathbf{R}+C_4·{\mathbf{N}}_2\right)·{\mathbf{K}}_{ELM2}\right)}^{-1}·{\mathbf{H}}_2·{\mathbf{H}}_1^T·\lambda \hfill \\ \hfill & \eta =\frac{1}{C_3}·{\mathbf{S}}^{-1}·\lambda \hfill \end{array}$$

Thus, we can get the dual problem of (44) as follows:

Dual C$L_1$-FTELM${}_1$

$$\underset{\alpha \ge \mathbf{0}}{min}\frac{1}{2}{\alpha }^T·\left(\left({\mathbf{H}}_2{\mathbf{H}}_1^T\right){\mathbf{Q}}_1^{-1}\left({\mathbf{H}}_1{\mathbf{H}}_2^T\right)+\frac{1}{C_1}{\mathbf{D}}^{-1}\right)·\alpha -{\mathbf{e}}_2^T·\alpha$$

(55)

where ${\mathbf{Q}}_1={\mathbf{K}}_{ELM1}·\left(\mathbf{F}+C_2·{\mathbf{N}}_1\right)·{\mathbf{K}}_{ELM1}$.

In the same way, we can obtain the dual problem of the Equation (47) as follows:

Dual C$L_1$-FTELM${}_2$

$$\underset{\lambda \ge \mathbf{0}}{min}\frac{1}{2}{\lambda }^T·\left(\left({\mathbf{H}}_1{\mathbf{H}}_2^T\right){\mathbf{Q}}_2^{-1}\left({\mathbf{H}}_2{\mathbf{H}}_1^T\right)+\frac{1}{C_3}{\mathbf{S}}^{-1}\right)·\lambda -{\mathbf{e}}_1^T·\lambda$$

(56)

where ${\mathbf{Q}}_2={\mathbf{K}}_{ELM2}·\left(\mathbf{R}+C_4·{\mathbf{N}}_2\right)·{\mathbf{K}}_{ELM2}$.

After solving (55) and (56), $\alpha$ and $\lambda$ are derived, and then ${\alpha }_1$ and ${\alpha }_2$ are obtained. So, the decision function of C$L_1$-FTELM is as follows:

$$y=arg\underset{k=1,2}{min}\left|{\alpha }_k^T·{\mathbf{H}}_k·\mathbf{h}\left(\mathbf{x}\right)\right|=arg\underset{k=1,2}{min}\sum _{i=1}^{m_k}{\alpha }_{k_i}·k_{EL{M}_k}\left(x,x_i\right)$$

(57)

Based on the above discussion, our algorithm will be presented in Algorithm 2. Next, we give the convergence analysis of Algorithm 2.

Algorithm 2 The procedure of C$L_1$-FTELM.

Input:        Training set $T={\left\{x_i,y_i\right\}}_{i=1}^m$, where $x_i\in {\mathit{R}}^d$, $y_i=\pm 1$, the number of hidden node number $\mathit{L}$, regularization parameters $C_1$, $C_2$, $C_3$, $C_4>0$, ${\epsilon }_1$, ${\epsilon }_2$, ${\epsilon }_3$, ${\epsilon }_4>0$, ${\rho }_1$, ${\rho }_2$, ${\sigma }_1$, ${\sigma }_2$, ${\sigma }_3$, ${\sigma }_4$. Output:        ${\alpha }_1^{*}$, ${\alpha }_2^{*}$ and the decision function of C$L_1$-FTELM. 1: Initialize ${\mathbf{F}}_0\in {\mathit{R}}^{m_1\times m_1}$, ${\mathbf{D}}_0\in {\mathit{R}}^{m_2\times m_2}$, ${\mathbf{R}}_0\in {\mathit{R}}^{m_2\times m_2}$, ${\mathbf{S}}_0\in {\mathit{R}}^{m_1\times m_1}$. 2: Compute the graph matrix ${\mathbf{N}}_1$, ${\mathbf{N}}_2$ by Equations (4) and (5). 3: Choose an activation function such as $G\left(x\right)=\frac{1}{1+e^{-x}}$ and compute the hidden layer output matrix ${\mathbf{H}}_1$, ${\mathbf{H}}_2$ by $\mathbf{h}\left(x_i\right)=G\left({\sum }_{j=1}^d{\omega }_{ji}{x}_j+b_i\right)$ and compute ${\mathbf{K}}_{ELM1}$ = ${\mathbf{H}}_1{\mathbf{H}}_1^T$ and ${\mathbf{K}}_{ELM2}$ = ${\mathbf{H}}_2{\mathbf{H}}_2^T$. 4: Set t = 0. 5: While       Solving (55) and (56), the  ${\alpha }^t$ and ${\lambda }^t$ can be obtained.       Then get the solution ${\alpha }_1^t$, ${\alpha }_2^t$, ${\xi }^t$, and ${\eta }^t$ by $${\alpha }_1^t=-{\left({\mathbf{K}}_{ELM1}·\left({\mathbf{F}}_t+C_2·{\mathbf{N}}_1\right)·{\mathbf{K}}_{ELM1}\right)}^{-1}·{\mathbf{H}}_1·{\mathbf{H}}_2^T·{\alpha }^t,{\xi }^t=\frac{1}{C_1}·{\mathbf{D}}_t^{-1}·{\alpha }^t$$ $${\alpha }_2^t={\left({\mathbf{K}}_{ELM2}·\left({\mathbf{R}}_t+C_4·{\mathbf{N}}_2\right)·{\mathbf{K}}_{ELM2}\right)}^{-1}·{\mathbf{H}}_2·{\mathbf{H}}_1^T·{\lambda }^t,{\eta }^t=\frac{1}{C_3}·{\mathbf{S}}_t^{-1}·{\lambda }^t$$       Update the matrices $F_{t+1}$, $D_{t+1}$, $R_{t+1}$, and $S_{t+1}$ by (45), (46), (48) and (49), respectively.       Compute the objective function values $J_1^{t+1}$ and $J_2^{t+1}$, by         $$\begin{array}{c}\hfill J_1^{t+1}=\frac{1}{2}{\left({\alpha }_1^t\right)}^T·{\mathbf{K}}_{ELM1}·{\mathbf{F}}_{t+1}·{\mathbf{K}}_{ELM1}·{\alpha }_1^t+\frac{C_1}{2}·{\left({\xi }^t\right)}^T·{\mathbf{D}}_{t+1}·{\xi }^t+\\ \hfill \frac{C_2}{2}·{\left({\alpha }_1^t\right)}^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\alpha }_1^t\end{array}$$ (58)        $$\begin{array}{c}\hfill J_2^{t+1}=\frac{1}{2}{\left({\alpha }_2^t\right)}^T·{\mathbf{K}}_{ELM2}·{\mathbf{R}}_{t+1}·{\mathbf{K}}_{ELM2}·{\alpha }_2^t+\frac{C_3}{2}·{\left({\eta }^t\right)}^T·{\mathbf{S}}_{t+1}·{\eta }^t+\\ \hfill \frac{C_4}{2}·{\left({\alpha }_2^t\right)}^T·{\mathbf{K}}_{ELM2}·{\mathbf{N}}_2·{\mathbf{K}}_{ELM2}·{\alpha }_2^t\end{array}$$ (59)       if $\left|J_1^{t+1}-J_1^t\right|\le {\rho }_1$ and $\left|J_2^{t+1}-J_2^t\right|\le {\rho }_2$.             break       else             t = t+1 6: end while 7: Stop the iteration process and get the solution of ${\alpha }_1^{*}$, ${\alpha }_2^{*}$.

4.3. Convergence Analysis

Before we prove the convergence of the iterative algorithm, we first review two lemmas [34].

Lemma 1.

For any non-zeros vectors x, y$\in {\mathit{R}}^n$, if $f\left(x\right)={∥x∥}_1-\frac{{∥x∥}_1^2}{2{∥y∥}_1}$, then the following inequalities $f\left(x\right)\le f\left(y\right)$ hold.

Lemma 2.

For any non-zeros vectors x, y, p, q$\in {\mathit{R}}^n$, if $f\left(x,p\right)={∥x∥}_1-\frac{{∥x∥}_1^2}{2{∥y∥}_1}+C\left({∥p∥}_1-\frac{{∥p∥}_1^2}{2{∥q∥}_1}\right)$, $C\in {\mathit{R}}^{+}$, then the following inequalities $f\left(x,p\right)\le f\left(y,q\right)$ hold.

The proof of two lemmas is detailed in [34].

Theorem 1.

Algorithm 2 monotonically decreases the objectives of problems (42) and (43) in each iteration until it converges.

Proof. 

Here, we only use problem (42) as an example to prove Theorem 1.

$$\begin{array}{cc}\hfill & J\left({\alpha }_1,\xi \right)=\underset{{\alpha }_1,\xi }{min}\sum _{i=1}^{m_1}\underset{}{min}\left({∥{\mathbf{h}}^T\left(x_i\right)·{\mathbf{H}}_{\mathbf{1}}^T·{\alpha }_1∥}_1,{\epsilon }_1\right)+C_1·\sum _{j=1}^{m_2}\underset{}{min}\left({∥{\xi }_j∥}_1,{\epsilon }_2\right)\hfill \\ \hfill & +\frac{C_2}{2}·{\alpha }_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\alpha }_1\hfill \end{array}$$

(60)

when ${∥{\mathbf{h}}^T\left(x_i\right)·{\mathbf{H}}_1^T·{\alpha }_1∥}_1<{\epsilon }_1$ and ${∥{\xi }_j∥}_1<{\epsilon }_2$,we have:

$$\begin{array}{cc}\hfill & J\left({\alpha }_1,\xi \right)=\underset{{\alpha }_1,\xi }{min}\sum _{i=1}^{m_1}{∥{\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1∥}_1+C_1\sum _{j=1}^{m_2}{∥{\xi }_j∥}_1\hfill \\ \hfill & +\frac{C_2}{2}·{\alpha }_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\alpha }_1\hfill \end{array}$$

(61)

We take the derivative of Equation (61) with respect to ${\alpha }_1$ and $\xi$ separately and then obtain that:

$$\left\{\begin{array}{c}\sum _{i=1}^{m_1}\frac{{\mathbf{H}}_1\mathbf{h}\left(x_i\right){\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1}{\left|{\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1\right|}+C_2·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_{\mathbf{1}}·{\mathbf{K}}_{ELM1}·{\alpha }_1=0\hfill \\ \\ C_1·\sum _{j=1}^{m_2}\frac{{\xi }_j}{\left|{\xi }_j\right|}=0\hfill \end{array}\right.$$

(62)

by the above Equation (62), we can get:

$$\begin{array}{cc}\hfill & \sum _{i=1}^{m_1}\frac{{\mathbf{H}}_1\mathbf{h}\left(x_i\right){\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1}{\left|{\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1\right|}+C_1·\sum _{j=1}^{m_2}\frac{{\xi }_j}{\left|{\xi }_j\right|}\hfill \\ \hfill & +C_2·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_{\mathbf{1}}·{\mathbf{K}}_{ELM1}·{\alpha }_1=0\hfill \end{array}$$

(63)

We define $f_i=\frac{1}{\left|{\mathbf{h}}^T\left(x_i\right)·{\mathbf{H}}_1^T·{\alpha }_1\right|}$ and $d_j=\frac{1}{\left|{\xi }_j\right|}$ as the diagonal entries of $\mathbf{F}$ and $\mathbf{D}$, respectively. Thus we can rewrite Equation (63) as follows:

$${\mathbf{H}}_1·{\mathbf{H}}_1^T·\mathbf{F}·{\mathbf{H}}_1^T·{\mathbf{H}}_1·{\alpha }_1+C_1·\mathbf{D}·\xi +C_2·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_{\mathbf{1}}·{\mathbf{K}}_{ELM1}·{\alpha }_1=0$$

(64)

Obviously, Equation (64) is the optimal solution to the following problem:

$$\begin{array}{c}\hfill \underset{{\alpha }_1,\xi }{min}\frac{1}{2}{\alpha }_1^T·{\mathbf{K}}_{ELM1}·\mathbf{F}·{\mathbf{K}}_{ELM1}·{\alpha }_1+\frac{C_1}{2}·{\xi }^T·\mathbf{D}·\xi \\ \hfill +\frac{C_2}{2}·{\alpha }_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\alpha }_1\end{array}$$

(65)

Now, assume that ${\overline{\alpha }}_1$ and $\overline{\xi }$ denote the updated ${\alpha }_1$ and $\xi$ of Algorithm 2, respectively. Thus we can get:

$$\begin{array}{cc}\hfill & \frac{1}{2}{\overline{\alpha }}_1^T·{\mathbf{K}}_{ELM1}·\mathbf{F}·{\mathbf{K}}_{ELM1}·{\overline{\alpha }}_1+\frac{C_1}{2}·{\overline{\xi }}^T·\mathbf{D}·\overline{\xi }\hfill \\ \hfill & +\frac{C_2}{2}·{\overline{\alpha }}_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\overline{\alpha }}_1\hfill \\ \hfill & \le \frac{1}{2}{\alpha }_1^T·{\mathbf{K}}_{ELM1}·\mathbf{F}·{\mathbf{K}}_{ELM1}·{\alpha }_1+\frac{C_1}{2}·{\xi }^T·\mathbf{D}·\xi \hfill \\ \hfill & +\frac{C_2}{2}·{\alpha }_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\alpha }_1\hfill \end{array}$$

(66)

we have rewritten Equation (66) as follows

$$\begin{array}{cc}\hfill & \sum _{i=1}^{m_1}\frac{{\left({\mathbf{K}}_{ELM1}{\overline{\alpha }}_1\right)}^T\left({\mathbf{K}}_{ELM1}{\overline{\alpha }}_1\right)}{2\left|{\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1\right|}+\sum _{j=1}^{m_2}\frac{C_1{\left({\overline{\xi }}_j\right)}^2}{2\left|{\xi }_j\right|}\hfill \\ \hfill & +\frac{C_2}{2}{\overline{\alpha }}_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\overline{\alpha }}_1\hfill \\ \hfill & \le \sum _{i=1}^{m_1}\frac{{\left({\mathbf{K}}_{ELM1}{\alpha }_1\right)}^T\left({\mathbf{K}}_{ELM1}{\alpha }_1\right)}{2\left|{\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1\right|}+\sum _{j=1}^{m_2}\frac{C_1{\left({\xi }_j\right)}^2}{2\left|{\xi }_j\right|}\hfill \\ \hfill & +\frac{C_2}{2}{\alpha }_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\alpha }_1\hfill \end{array}$$

(67)

Here, we let $x={\mathbf{K}}_{ELM1}{\overline{\alpha }}_1$, $y={\mathbf{K}}_{ELM1}{\alpha }_1$, $C=C_1$, $p={\overline{\xi }}_j$, $q={\xi }_j$. Based on Lemma 2, we have

$$\begin{array}{c}\hfill \left|{\mathbf{K}}_{ELM1}·{\overline{\alpha }}_1\right|-\frac{{\left|{\mathbf{K}}_{ELM1}·{\overline{\alpha }}_1\right|}^2}{2\left|{\mathbf{K}}_{ELM1}·{\alpha }_1\right|}+C_1·\left(\left|{\overline{\xi }}_j\right|-\frac{{\left|{\overline{\xi }}_j\right|}^2}{2\left|{\xi }_j\right|}\right)\\ \hfill \le \left|{\mathbf{K}}_{ELM1}·{\alpha }_1\right|-\frac{{\left|{\mathbf{K}}_{ELM1}·{\alpha }_1\right|}^2}{2\left|{\mathbf{K}}_{ELM1}·{\alpha }_1\right|}+C_1·\left(\left|{\xi }_j\right|-\frac{{\left|{\xi }_j\right|}^2}{2\left|{\xi }_j\right|}\right)\end{array}$$

(68)

then we can get

$$\begin{array}{c}\hfill \sum _{i=1}^{m_1}\left(\left|{\mathbf{h}}^T\left(x_i\right)·{\mathbf{H}}_1^T·{\overline{\alpha }}_1\right|-\frac{{\left|{\mathbf{K}}_{ELM1}·{\overline{\alpha }}_1\right|}^2}{2\left|{\mathbf{K}}_{ELM1}·{\alpha }_1\right|}\right)+C_1\sum _{j=1}^{m_2}\left(\left|{\overline{\xi }}_j\right|-\frac{{\left|{\overline{\xi }}_j\right|}^2}{2\left|{\xi }_j\right|}\right)\\ \hfill \le \sum _{i=1}^{m_1}\left(\left|{\mathbf{h}}^T\left(x_i\right)·{\mathbf{H}}_1^T·{\alpha }_1\right|-\frac{{\left|{\mathbf{K}}_{ELM1}·{\alpha }_1\right|}^2}{2\left|{\mathbf{K}}_{ELM1}·{\alpha }_1\right|}\right)+C_1\sum _{j=1}^{m_2}\left(\left|{\xi }_j\right|-\frac{{\left|{\xi }_j\right|}^2}{2\left|{\xi }_j\right|}\right)\end{array}$$

(69)

combining (67) and (69), we can get the following inequalities

$$\begin{array}{cc}\hfill & \sum _{i=1}^{m_1}\left(\left|{\mathbf{h}}^T\left(x_i\right)·{\mathbf{H}}_1^T·{\overline{\alpha }}_1\right|\right)+C_1\sum _{j=1}^{m_2}\left(\left|{\overline{\xi }}_j\right|\right)\hfill \\ \hfill & +\frac{C_2}{2}{\overline{\alpha }}_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\overline{\alpha }}_1\hfill \\ \hfill & \le \sum _{i=1}^{m_1}\left(\left|{\mathbf{h}}^T\left(x_i\right)·{\mathbf{H}}_1^T·{\alpha }_1\right|\right)+C_1\sum _{j=1}^{m_2}\left(\left|{\xi }_j\right|\right)\hfill \\ \hfill & +\frac{C_2}{2}{\alpha }_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\alpha }_1\hfill \end{array}$$

(70)

further, we can get

$$\begin{array}{cc}\hfill & \sum _{i=1}^{m_1}min\left(\left|{\mathbf{h}}^T\left(x_i\right)·{\mathbf{H}}_1^T·{\overline{\alpha }}_1\right|\right)+C_1\sum _{j=1}^{m_2}min\left(\left|{\overline{\xi }}_j\right|\right)\hfill \\ \hfill & +\frac{C_2}{2}{\overline{\alpha }}_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\overline{\alpha }}_1\hfill \\ \hfill & \le \sum _{i=1}^{m_1}min\left(\left|{\mathbf{h}}^T\left(x_i\right)·{\mathbf{H}}_1^T·{\alpha }_1\right|\right)+C_1\sum _{j=1}^{m_2}min\left(\left|{\xi }_j\right|\right)\hfill \\ \hfill & +\frac{C_2}{2}{\alpha }_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\alpha }_1\hfill \end{array}$$

(71)

Therefore, we have $J\left({\overline{\alpha }}_1,\overline{\xi }\right)\le J\left({\alpha }_1,\xi \right)$. Similarly, when ${∥{\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1∥}_1\le {\epsilon }_1$ and $∥{\xi }_j∥\ge {\epsilon }_2$, or ${∥{\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1∥}_1\ge {\epsilon }_1$ and $∥{\xi }_j∥\le {\epsilon }_2$, or ${∥{\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1∥}_1\ge {\epsilon }_1$ and $∥{\xi }_j∥\ge {\epsilon }_2$, we can obviously get $J\left({\overline{\alpha }}_1,\overline{\xi }\right)\le J\left({\alpha }_1,\xi \right)$. Thus, the inequality $J\left({\overline{\alpha }}_1,\overline{\xi }\right)\le J\left({\alpha }_1,\xi \right)$ holds. The three terms in Equation (60) are equal to or greater than 0. Meaning that Algorithm 2 decreases objective of problem (42) until convergence. □

Theorem 2.

Algorithm 2 will converge to a local optimum to the problem in (42).

Proof. 

Here, we only use (42) as an example to prove Theorem 2.

When ${∥{\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1∥}_1\le {\epsilon }_1$ and ${∥{\xi }_j∥}_1\le {\epsilon }_2$, we write out the formula of (42) Lagrange function:

$$\begin{array}{cc}\hfill & {\mathcal{L}}_1\left({\alpha }_1,\xi ,\lambda \right)=\sum _{i=1}^{m_1}\left({∥{\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1∥}_1\right)+C_1\sum _{j=1}^{m_2}\left({∥{\xi }_j∥}_1\right)\hfill \\ \hfill & +\frac{C_2}{2}{\alpha }_1^T·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\alpha }_1-{\lambda }^T\sum _{j=1}^{m_2}\left({\mathbf{h}}^T\left(x_j\right){\mathbf{H}}_1^T{\alpha }_1+{\xi }_j-1\right)\hfill \end{array}$$

(72)

Then, we take the derivative of $\mathcal{L}\left({\alpha }_1,\xi ,\lambda \right)$ with respect to ${\alpha }_1$

$$\begin{array}{cc}\hfill & \frac{\partial {\mathcal{L}}_1}{\partial {\alpha }_1}=\sum _{i=1}^{m_1}\frac{{\mathbf{H}}_1\mathbf{h}\left(x_i\right){\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1}{\left|{\mathbf{h}}^T\left(x_i\right){\mathbf{H}}_1^T{\alpha }_1\right|}+C_2·{\mathbf{K}}_{ELM1}·{\mathbf{N}}_1·{\mathbf{K}}_{ELM1}·{\alpha }_1\hfill \\ \hfill & +{\mathbf{H}}_1·{\mathbf{H}}_2^T·\lambda ={\mathbf{K}}_{ELM1}·\left(\mathbf{F}+C_2·{\mathbf{N}}_1\right)·{\mathbf{K}}_{ELM1}·{\alpha }_1+{\mathbf{H}}_1·{\mathbf{H}}_2^T·\lambda =\mathbf{0}\hfill \end{array}$$

(73)

Similarly, we get the Lagrangian function of problem (44):

$$\begin{array}{cc}\hfill & {\mathcal{L}}_2\left({\alpha }_1,\xi ,\alpha \right)=\frac{1}{2}{\alpha }_1^T·{\mathbf{K}}_{ELM1}·\left(\mathbf{F}+C_2·{\mathbf{N}}_1\right)·{\mathbf{K}}_{ELM1}·{\alpha }_1+\frac{C_1}{2}·{\xi }^T·\mathbf{D}·\xi \hfill \\ \hfill & -{\lambda }^T·\left(-{\mathbf{H}}_2·{\mathbf{H}}_1^T·{\alpha }_1+\xi -{\mathbf{e}}_2\right)\hfill \end{array}$$

(74)

Taking the derivative of ${\mathcal{L}}_2\left({\alpha }_1,\xi ,\alpha \right)$ with respect to ${\alpha }_1$:

$$\begin{array}{c}\hfill \frac{\partial {\mathcal{L}}_2}{\partial {\alpha }_1}={\mathbf{K}}_{ELM1}·\left(\mathbf{F}+C_2·{\mathbf{N}}_1\right)·{\mathbf{K}}_{ELM1}·{\alpha }_1+{\mathbf{H}}_1·{\mathbf{H}}_2^T·\lambda =\mathbf{0}\end{array}$$

(75)

The other three cases are similar. From the discussion above, we may safely draw that Equations (73) and (75) are equivalent, so we can use problem (44) instead of problem (42) to solve C$L_1$-FTELM, which further illustrates that Algorithm 2 can converge to a local optimal solution. □

5. Experiments

Description of the four comparison algorithms:

OPTELM: The optimization function of the model consists of minimizing the $L_2$-norm of the weight vector and minimizing empirical loss. It neither consider the establishment of two non-parallel hyperplanes to deal with classification tasks, nor consider the statistical information of samples. At the same time, since it uses $L_2$-norm metric and Hinge loss, it has weak anti-noise ability.

TELM: The optimization function of the model consists of minimizing the distance from the sample points to the hyperplane as well as minimizing empirical loss. TELM does not fully consider the statistical information of the sample. At the same time, its metric uses the $L_2$-norm metric and the loss function uses the Hinge loss. When there is noise in the data set, the influence of noise data will be amplified and the accuracy of classification will be reduced.

FELM: The optimization function of the model includes minimizing the $L_2$-norm of the weight vector, minimizing empirical loss, and minimizing the within-class scatter of the number sample data. Although FELM takes into account the statistics of the sample, it has to deal with a much larger optimization problem than the twin extreme learning machines, which is time-consuming. At the same time, FELM still continues the metric and loss used by OPTELM, so its anti-noise ability is weak.

C$L_1$-TWSVM: C$L_1$-TWSVM is formed on the basis of twin support vector machines by changing the model’s metric and loss to capped $L_1$-norm. Although C$L_1$-TWSVM has the ability to resist noise, it does not fully take into account the statistics of the data. Meanwhile, C$L_1$-TWSVM not only needs to solve the weight vector of the hyperplane, but also needs to solve the bias of the hyperplane, so it is time-consuming.

We systematically compare our algorithm above advanced algorithms (OPTELM [12], TELM [14], FELM [19], and C$L_1$-TWSVM [29]) on artificial synthetic datasets and UCI real datasets to verify the effectiveness of our FTELM and C$L_1$-FTELM. In Section 5.1, we describe the relevant experimental setting in detail. We describe their performance in different cases in Section 5.2 and Section 5.3, respectively. In Section 5.4, we use the one-versus-rest multi-classification method to perform data classification tasks in four image datasets: Yale “http://www.cad.zju.edu.cn/home/dengcai/Data/FaceData.html (accessed on 15 February 2023)”, ORL “http://www.cad.zju.edu.cn/home/dengcai/Data/FaceData.html (accessed on 15 February 2023)”, USPS “http://www.cad.zju.edu.cn/home/dengcai/Data/MLData.html (accessed on 15 February 2023)” handwritten digit dataset and MNIST “http://www.cad.zju.edu.cn/home/dengcai/Data/MLData.html (accessed on 15 February 2023)” dataset.

5.1. Experimental Setting

All experiments were implemented in MATLAB R2020a installed in a personal computer (PC) with an AMD Radeon Graphics processor (3.2 GHz), and 16 GB random-access memory (RAM). For C$L_1$-TWSVM, and C$L_1$-FTELM, we take the maximum number of iterations to be 100 and the iteration stopping threshold to be 0.001. The activation functions used in a total of five models (OPTELM, TELM, FELM, FTELM, and C$L_1$-FTELM) are $G\left(x\right)=\frac{1}{1+e^{-x}}$. The Gaussian kernel function $K(x,z)=e^{-\frac{{∥x-z∥}^2}{2{\sigma }^2}}$ was used for C$L_1$-TWSVM. The parameters selected by all the above algorithms are as follows: ${\epsilon }_1$, ${\epsilon }_2$, ${\epsilon }_3$, ${\epsilon }_4$ were selected from $\left\{{10}^i|-6,-5,-4\right\}$, $C_1,C_2,C_3,C_4$ were selected from $\left\{{10}^i|-5,-4,\dots ,4,5\right\}$, $\sigma$ was chosen from $\left\{2^i|-3,-2,\dots ,2,3\right\}$, and the hidden layer node number L was chosen from $\left\{50,100,200,500,1000,2000,5000,10,000\right\}$. The optimal parameters used by the model are selected by 10-fold cross-validation and grid search. Normalization was performed for both artificial and UCI datasets. For image datasets, we randomly select 20% of the data as the test set to get the classification accuracy of the algorithm. All experimental processes are repeated 10 times and the average of the 10 test results is used as the performance measure, and the evaluation criterion selected in this paper is classification accuracy (ACC).

5.2. Experiments on Artificial Datasets

We first do experiments on the Banana, Circle, Two spirals, and XOR datasets which are generated by trigonometric function(sine, cosine), two circle lines, two spirals lines, and two intersecting lines, respectively. The two-dimensional distributions of the four synthetic datasets are shown in Figure 1. Dark blue ‘+’ represents class 1, and cyan ‘∘’ represents class 2. Figure 2 illustrates the experimental results of four twin algorithms namely TELM, FTELM, C$L_1$-TWSVM, and C$L_1$-FTELM for four datasets with 0%, 20%, and 25% noise in terms of accuracy. From Figure 2a, we can observe that the classification accuracy of our FTELM and C$L_1$-FTELM in Banana and Two spirals datasets is higher than the other two methods. In the Circle and XOR datasets, the classification accuracy of the four methods is similar. The experimental results show that fully considering the statistical information of the data can effectively improve the classification accuracy of the classifier, which shows that our C$L_1$-FTELM method is effective. From Figure 2b,c, we can see that the overall effect of FTELM is better than TELM. This shows the importance of fully considering the statistical information of the sample. At the same time, we can see that C$L_1$-FTELM has the best effect, followed by C$L_1$-TWSVM. It shows that the capped $L_1$-norm can control the influence of noise on the model in a certain range, and further shows the effectiveness of using the capped $L_1$-norm. In summary, Figure 2 illustrates the effectiveness of considering sample statistics information and changing the distance metric and loss of the model into capped $L_1$-norm at the same time.

To further show the robustness of C$L_1$-FTELM, we add noise with different ratios to the Circle dataset. Figure 3 shows the accuracy of TELM, FTELM, C$L_1$-TWSVM, and C$L_1$-FTELM algorithms on the Circle dataset in different noises ratios. The ratio is set in the range of $\left\{0.1,0.15,0.2,0.25\right\}$. We plot the accuracy results of ten experiments with different noise ratios in a box-shaped plot. By observing the median of the four subgraphs, we can find that the median of C$L_1$-FTELM algorithm is much higher than the other three algorithms. And C$L_1$-FTELM method in four different noise ratios results is relatively concentrated. In other words, the variance of ten experimental results obtained by the C$L_1$-FTELM algorithm is smaller and the mean value is larger. The above results show that our C$L_1$-FTELM has better stability and better classification effect in environments containing noise. This shows the effectiveness and noise resistance of the distance metric and loss functions of the model using the capped $L_1$-norm.

5.3. Experiments on UCI Datasets

In this section, we conduct the numerical simulation on UCI datasets.

Table 1. Characteristics of UCI datasets.

Datasets	Instances	Attributes	Datasets	Instances	Attributes
Australian	690	14	Vote	432	16
German	1000	24	Ionosphere	351	35
Breast cancer	699	9	Pima	768	8
WDBC	569	30	QSAR	1055	41
Wholesale	440	7	Spam	4601	57

describes the features of the UCI datasets used in detail. We also added two algorithms (OPTELM, FELM) to verify the classification performance of FTELM and C$L_1$-FTELM in ten UCI data sets.

All experimental results obtained based on the optimal parameters are shown in

Table 2. Experimental results on UCI datasets, The best results are marked in bold.

Datasets	OPTELM	TELM	FELM	FTELM	C${\mathit{L}}_1$-TWSVM	C${\mathit{L}}_1$-FTELM
	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)
	Times (s)	Times (s)	Times (s)	Times (s)	Times (s)	Times (s)
Australian	85.31$\pm$0.34	85.60$\pm$0.44	85.46$\pm$0.19	86.79$\pm$0.33	85.82$\pm$0.28	87.13$\pm$0.52
	0.682	0.593	1.698	0.456	1.676	2.533
German	76.26$\pm$0.52	76.40$\pm$0.16	76.50$\pm$0.42	76.56$\pm$0.47	76.70$\pm$0.25	77.15$\pm$1.18
	1.182	0.979	4.555	0.474	5.318	7.006
Breast cancer	95.70$\pm$0.24	96.35$\pm$0.15	96.45$\pm$0.09	97.07$\pm$0.15	96.39$\pm$0.13	97.32$\pm$0.53
	0.601	0.668	1.646	0.505	4.011	3.902
WDBC	96.71$\pm$0.27	97.13$\pm$0.48	97.55$\pm$0.17	98.55$\pm$0.26	97.09$\pm$0.25	97.86$\pm$0.21
	0.416	0.605	1.144	0.578	3.618	4.551
Wholesale	87.35$\pm$0.93	89.86$\pm$0.84	90.26$\pm$0.12	90.56$\pm$0.33	89.89$\pm$0.30	90.70$\pm$0.56
	0.278	2.091	0.665	0.359	1.246	1.377
Vote	95.31$\pm$0.16	95.56$\pm$0.30	96.04$\pm$0.24	96.12$\pm$0.31	95.21$\pm$0.54	96.43$\pm$0.35
	0.256	0.502	0.651	0.445	1.077	0.992
Ionosphere	90.59$\pm$0.84	91.38$\pm$0.52	92.32$\pm$0.32	92.74$\pm$0.83	92.56$\pm$0.54	93.32$\pm$1.21
	0.184	0.476	0.421	0.268	1.128	2.237
Pima	76.83$\pm$0.73	77.51$\pm$0.08	77.79$\pm$0.10	78.24$\pm$0.49	77.49$\pm$0.37	78.82$\pm$0.98
	0.858	0.795	2.099	0.932	1.743	4.708
QSAR	83.91$\pm$0.66	86.56$\pm$0.19	87.12$\pm$0.18	87.35$\pm$0.23	85.72$\pm$0.59	87.50$\pm$0.56
	1.442	0.979	2.489	2.864	2.665	14.288
Spam	85.57$\pm$0.65	91.38$\pm$0.52	89.67$\pm$0.21	91.94$\pm$1.23	90.56$\pm$1.23	92.27$\pm$0.54
	125.498	64.314	488.251	108.232	158.145	170.261

. Here, the average running time according to the optimal parameters is denoted by Times(s), and the average classification plus or minus standard deviation is denoted by $ACC\pm$. From Table 2, we can see that FTELM performs better than OPTELM, TELM, and FELM on all ten datasets. This indicates that adding Fisher regularization term on the basis of TELM framework can significantly improve the accuracy of model classification. In addition, the average training time of FTELM algorithm on most data sets is smaller than that of FELM algorithm, which indicates that FTELM has inherited the advantages of TELM’s short training time. In addition, we also can draw our C$L_1$-FTELM in most data sets has achieved the highest classification accuracy besides the WDBC data set. Through the analysis of the above results, we can conclude that the Fisher regularization and capped ${\mathit{L}}_1$-norm added to the TELM learning framework can effectively improve the performance of the classifier. It is shown that the proposed FTELM and C$L_1$-FTELM are efficient algorithms.

In order to more significantly verify the robustness of C$L_1$-FTELM to outliers, we added 20% and 25% Gaussian noise to 10 data sets, respectively. All experimental results are presented in

Table 3. Experimental results on UCI datasets with 20% noise, The best results are marked in bold.

Datasets	OPTELM	TELM	FELM	FTELM	C${\mathit{L}}_1$-TWSVM	C${\mathit{L}}_1$-FTELM
	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)
	Times (s)	Times (s)	Times (s)	Times (s)	Times (s)	Times (s)
Australian	79.68$\pm$1.75	80.37$\pm$0.56	79.06$\pm$1.36	80.44$\pm$1.34	81.98$\pm$0.87	82.78$\pm$0.57
	0.621	0.728	1.756	0.224	1.708	3.224
German	69.67$\pm$0.97	73.57$\pm$1.85	71.99$\pm$1.35	72.76$\pm$0.88	73.86$\pm$1.35	74.32$\pm$1.12
	1.318	0.981	4.102	0.398	5.673	6.764
Breast cancer	70.60$\pm$0.45	76.97$\pm$0.42	70.32$\pm$0.37	77.81$\pm$0.56	79.84$\pm$0.37	80.14$\pm$0.91
	0.803	0.706	1.552	0.315	4.572	5.034
WDBC	82.98$\pm$0.15	84.38$\pm$1.01	83.29$\pm$0.68	89.43$\pm$1.15	89.98$\pm$0.30	93.77$\pm$0.32
	0.419	0.204	0.992	0.376	3.899	4.861
Wholesale	73.40$\pm$0.93	73.77$\pm$0.69	73.74$\pm$0.76	74.77$\pm$0.56	78.74$\pm$0.91	79.47$\pm$2.58
	0.275	0.543	0.659	0.404	0.849	1.420
Vote	93.48$\pm$0.62	94.36$\pm$0.60	94.24$\pm$0.82	94.10$\pm$0.94	93.90$\pm$0.44	94.29$\pm$0.61
	0.277	0.619	0.549	0.114	1.048	1.398
Ionosphere	80.79$\pm$2.88	82.71$\pm$2.09	81.00$\pm$3.11	86.06$\pm$1.67	85.76$\pm$1.58	87.74$\pm$1.08
	0.159	0.021	0.456	0.737	0.391	2.081
Pima	65.79$\pm$0.23	67.07$\pm$0.56	66.12$\pm$0.12	66.30$\pm$1.34	70.25$\pm$1.57	71.42$\pm$0.94
	0.873	0.649	2.051	1.492	1.758	3.968
QSAR	68.32$\pm$2.48	68.80$\pm$0.95	68.54$\pm$2.50	72.28$\pm$2.18	71.09$\pm$2.02	72.31$\pm$1.98
	1.534	3.089	4.578	0.892	1.828	9.151
Spam	83.16$\pm$0.57	87.38$\pm$2.31	85.66$\pm$0.65	87.98$\pm$0.87	85.77$\pm$2.21	86.75$\pm$0.45
	128.798	60.565	432.257	106.267	147.365	160.231

and

Table 4. Experimental results on UCI datasets with 25% noise, The best results are marked in bold.

Datasets	OPTELM	TELM	FELM	FTELM	C${\mathit{L}}_1$-TWSVM	C${\mathit{L}}_1$-FTELM
	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)
	Times (s)	Times (s)	Times (s)	Times (s)	Times (s)	Times (s)
Australian	73.68$\pm$2.20	75.41$\pm$1.52	74.25$\pm$2.01	76.40$\pm$1.19	80.56$\pm$1.07	81.63$\pm$0.71
	0.585	0.673	1.627	0.206	2.205	2.261
German	69.72$\pm$0.13	72.87$\pm$0.82	71.41$\pm$0.88	73.15$\pm$0.87	73.13$\pm$1.16	73.25$\pm$0.76
	1.565	0.871	3.855	0.342	5.233	6.798
Breast cancer	67.59$\pm$0.18	70.43$\pm$0.79	67.23$\pm$0.24	71.65$\pm$0.58	70.93$\pm$0.52	72.71$\pm$0.49
	0.654	0.513	1.438	0.309	4.476	5.124
WDBC	79.61$\pm$0.78	81.66$\pm$0.84	79.83$\pm$0.72	87.96$\pm$1.13	88.50$\pm$0.74	92.43$\pm$0.76
	0.417	0.197	0.887	0.334	3.675	4.861
Wholesale	71.79$\pm$1.03	71.63$\pm$0.89	69.63$\pm$0.38	71.60$\pm$1.02	75.53$\pm$1.02	75.74$\pm$3.48
	0.570	2.021	0.623	0.338	1.147	1.387
Vote	92.62$\pm$0.88	92.95$\pm$0.50	93.12$\pm$0.80	93.21$\pm$0.80	93.21$\pm$0.68	93.50$\pm$1.00
	0.252	0.503	0.514	0.121	1.213	1.390
Ionosphere	78.15$\pm$2.94	78.79$\pm$3.01	76.62$\pm$3.67	83.59$\pm$1.49	82.94$\pm$2.90	85.03$\pm$2.28
	0.229	0.058	0.313	0.737	0.576	1.987
Pima	65.67$\pm$0.12	65.45$\pm$1.55	65.89$\pm$0.12	65.79$\pm$0.14	69.01$\pm$1.55	68.51$\pm$2.75
	0.803	0.761	2.182	0.471	5.532	4.012
QSAR	67.49$\pm$3.08	67.81$\pm$1.63	70.30$\pm$2.33	71.53$\pm$3.00	70.49$\pm$2.13	69.72$\pm$2.14
	2.067	2.730	4.251	0.849	1.783	12.564
Spam	71.77$\pm$1.05	75.35$\pm$0.72	70.89$\pm$1.23	76.43$\pm$1.16	83.56$\pm$0.26	84.75$\pm$0.78
	99.541	61.254	462.221	116.267	142.365	165.214

. From Table 3 and Table 4, we find that the classification accuracy of all six algorithms decreases after adding noise. However, the classification accuracy of our algorithm C$L_1$-FTELM is the highest of the eight datasets, which further reveals the effectiveness of our method using capped $L_1$-norm instead of Hinge loss and $L_2$-norm distance metric. Compared with the other five algorithms, our C$L_1$-FTELM algorithm is more time-consuming. This is due to that C$L_1$-FTELM requires a lot of time in the process of training to iterative calculation, eliminating outliers, and computing graph matrices. In addition, we used different noise factor values (0.1, 0.15, 0.2, 0.25, 0.3) on the Cancer, German, Ionosphere, and WDBC for the six algorithms. The experimental results are given in Figure 4. It can be seen from Figure 4a that when the Breast Cancer dataset contains 10% noise, the effects of our FTELM and C$L_1$-FTELM are comparable. This shows that it is important to consider the statistical information of the sample. As the ratio of noise increases, the classification accuracy of all methods decreases, but our C$L_1$-FTELM still has the highest accuracy. This illustrates the effectiveness of our using the capped $L_1$-norm. Figure 4b shows that with the increase of noise ratio, the decline trend of accuracy of C$L_1$-TWSVM and C$L_1$-FTELM is similar, but C$L_1$-FTELM is still the most stable among the six methods when facing the influence of noise. From both Figure 4c,d, we can clearly observe that the anti-noise effect of our C$L_1$-FTELM is the best. This illustrates the effectiveness of using the Fisher regularization term as well as the capped $L_1$-norm.

We also conduct experiments on four data sets (Breast cancer, QSAR, WDBC, and Vote) to verify the convergence of the proposed Algorithm 2. As shown in Figure 5, we plot the objective function value of each iteration. It can be seen that the objective function value converges to a fixed value rapidly with the increase in the number of iterations. This shows that our algorithm can make the objective function value can converge to a local optimal value within a limited number of iterations. The effectiveness and convergence of the Algorithm 2 are demonstrated.

5.4. Experiments on Image Datasets

The image datasets include Yale, ORL, USPS, and MNIST. Figure 6 illustrates examples of four high-dimensional image datasets. The number of samples and characteristics of the four image datasets are shown in

Table 5. Characteristics of image datasets.

Datasets	Instances	Attributes	Datasets	Instances	Attributes
Yale	165	1024	ORL	400	1024
USPS	9298	256	MNIST	70,000	784

. These four image datasets are used to investigate the performance of our FTELM and C$L_1$-FTELM for multi-classification. Specifically, for the MNIST dataset, we only select the first 2000 samples to participate in the experiment.

Table 6. Experimental results on images and handwritten digits datasets. The best results are marked in bold.

Datasets	OPTELM	TELM	FELM	FTELM	C${\mathit{L}}_1$-TWSVM	C${\mathit{L}}_1$-FTELM
	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)	ACC$\pm$S (%)
	Times (s)	Times (s)	Times (s)	Times (s)	Times (s)	Times (s)
Yale	89.39$\pm$2.85	91.44$\pm$1.58	90.54$\pm$2.01	92.23$\pm$1.29	91.54$\pm$1.07	93.12$\pm$1.71
	0.126	0.101	0.262	0.136	0.135	0.492
ORL	87.72$\pm$1.53	90.87$\pm$0.52	90.41$\pm$0.78	92.45$\pm$0.67	92.32$\pm$1.16	93.25$\pm$0.46
	1.169	0.483	3.064	0.529	1.338	2.695
USPS	98.76$\pm$0.18	98.83$\pm$0.69	98.23$\pm$0.24	99.65$\pm$0.68	99.23$\pm$0.42	99.89$\pm$0.89
	118.729	17.536	134.438	6.795	358.368	355.762
MNIST	89.61$\pm$0.58	90.66$\pm$0.74	89.83$\pm$0.75	91.26$\pm$1.13	90.88$\pm$0.14	91.53$\pm$0.56
	8.723	1.237	41.656	0.868	14.258	14.973

shows the specific experimental results. As can be seen from the results of the experiment, our C$L_1$-FTELM and C$L_1$-TWSVM have similar training times. This is because this paper uses an iterative algorithm to solve non-convex optimization problem of C$L_1$-FTELM, which is time-consuming. Simultaneously, the C$L_1$-FTELM at Yale, ORL, USPS, and MNIST four datasets classification accuracy is highest among the six algorithms. In addition, the classification accuracy of our FTELM algorithm on the four image datasets is the second highest after our C$L_1$-FTELM. The above results fully show the effectiveness of our two algorithms in dealing with multi-classification tasks.

6. Conclusions

In this paper, we have proposed FTELM and C$L_1$-FTELM. FTELM not only inherits the advantages of TELM but also takes full account of the statistical information of samples, so as to further improve the classification performance of the classifier. Specifically, when there is no noise in the data or the ratio of noise is very small, our FTELM algorithm can deal with the classification problem very well, not only time-saving but also with high classification accuracy. C$L_1$-FTELM further improves the anti-noise ability of the model by replacing the $L_2$-norm and hinge loss in FTELM with capped $L_1$-norm. It not only utilizes the distribution information of the data but also improves the anti-noise ability of the model. Furthermore, we have designed two algorithms to solve the problems of FTELM and C$L_1$-FTELM. In addition, we present two theorems to prove the convergence of our C$L_1$-FTELM. However, in terms of computational cost, FTELM is better than C$L_1$-FTELM to some extent. Therefore, in future work, we will propose some new tricks to accelerate the computation of the C$L_1$-FTELM. In addition, trying to extend FTELM and C$L_1$-FTELM from supervised learning to semi-supervised learning framework is also a future research focus.