MCMS-STM: An Extension of Support Tensor Machine for Multiclass Multiscale Object Recognition in Remote Sensing Images

Abstract

The support tensor machine (STM) extended from support vector machine (SVM) can maintain the inherent information of remote sensing image (RSI) represented as tensor and obtain effective recognition results using a few training samples. However, the conventional STM is binary and fails to handle multiclass classification directly. In addition, the existing STMs cannot process objects with different sizes represented as multiscale tensors and have to resize object slices to a fixed size, causing excessive background interferences or loss of object’s scale information. Therefore, the multiclass multiscale support tensor machine (MCMS-STM) is proposed to recognize effectively multiclass objects with different sizes in RSIs. To achieve multiclass classification, by embedding one-versus-rest and one-versus-one mechanisms, multiple hyperplanes described by rank-R tensors are built simultaneously instead of single hyperplane described by rank-1 tensor in STM to separate input with different classes. To handle multiscale objects, multiple slices of different sizes are extracted to cover the object with an unknown class and expressed as multiscale tensors. Then, M-dimensional hyperplanes are established to project the input of multiscale tensors into class space. To ensure an efficient training of MCMS-STM, a decomposition algorithm is presented to break the complex dual problem of MCMS-STM into a series of analytic sub-optimizations. Using publicly available RSIs, the experimental results demonstrate that the MCMS-STM achieves 89.5% and 91.4% accuracy for classifying airplanes and ships with different classes and sizes, which outperforms typical SVM and STM methods.

1. Introduction

The diverse types and sizes of objects bring a challenge to object recognition in remote sensing images (RSIs). Recently, mainly owing to its powerful feature abstraction ability, various deep learning technologies have achieved impressive success in different object recognition tasks, such as Fast R-CNN [1], Faster R-CNN [2], local attention based CNN [3], ${\text{ℛ}}^2$ CNN [4], SSD [5] and YOLO [6]. In addition to object recognition, deep learning-based methods are widely used for a wide variety of classification tasks based on remote sensing data acquired by different sensors, such as graph convolution neural networks [7] based hyperspectral image classification [8], and multimodal deep learning-based multisource image classification [9]. Despite the recent advances, deep learning-based methods rely heavily on massive available labeled samples. In comparison, the machine learning-based object recognition method can obtain effective results using a small number of samples.

For the object recognition in RSIs, the general procedure consists of two steps, i.e., extracting an object slice using object detection method and recognizing the type of object contained in the slice using a trained classifier. By virtue of the object detection methods, e.g., region proposal-based methods [10,11,12] and saliency-based methods [13,14,15,16], the positions of objects in RSIs can be obtained. Then, the slice of the object can be extracted for subsequent classification. It is worth noting that most existing classifiers, including deep learning methods [1,2,3,4,5] and machine learning methods [17,18,19,20], can only deal with the input of the fixed size. Thereby, two image resizing operations are always used to extract slices containing objects with the fixed size, as shown in Figure 1, one of which extracts the slices by the appropriate size to ensure that the object can be exactly covered, and then resizes the slice to the same size using the image interpolation method [10,11,12]. Another way is to extract slices according to the maximum size of objects to cover each size of object [13,14,15,16]. After obtaining the slices with the fixed size, a proper classifier should be designed to predict the corresponding class label. Among the various machine learning based classifiers, the support vector machine (SVM) as well as its tensor extension, i.e., support tensor machine (STM), are the most worth considering due to a sound theoretical foundation and excellent generality capability. Besides the standard SVM (i.e., $C-\mathrm{SVM}$) [17], various variants of SVM were established to improve the performance of SVM from different perspectives, including $\nu -\mathrm{SVM}$ [18], least-square SVM [19], etc. Since the standard SVM can be considered as a bi-category classifier and thus cannot be used to address multiclass classifications directly, the One-Versus-One (OVO) [20], Error Correcting Output Codes (ECOC) [21], and One-Versus-Rest (OVR) [22] strategies were employed in order to use various binary SVMs to achieve multiclass classifications indirectly. Different from the classic SVM that relies on auxiliary strategies to achieve multiclass classification, the multiclass SVM was developed to classify multiclass objects by solving parameters of multiple classification hyperplanes simultaneously [22]. Besides, based on the multiclass classification mechanism of [22], some other multiclass-oriented SVM variants were developed, e.g., the graph embedded multiclass SVM [23] and the least-squares twin multiclass SVM [24].

Since the object slice can be represented naturally as a tensor in RSI, the tensor-based classifier, i.e., STM, was developed to exploit the structural information embedded in RSI better. According to the supervised tensor learning framework [25] established in 2005, the $C-\mathrm{SVM}$ and $\nu -\mathrm{SVM}$ were extended to $C-\mathrm{STM}$ and $\nu -\mathrm{STM}$ by applying multilinear operator of tensor space, respectively. Besides, a large number of variants of STMs were established to deal with different classification tasks, such as linear support higher-order tensor machine (SHTM) [26], multi-kernel STM [27], higher rank support tensor machine [28], and support tucker machine [29]. Moreover, the support multimode tensor machine (SMTM) [30] was built to exploit the multimode product of tensor to obtain the results of the multiclass classification for different perspectives, while its multiclass classification strategy is similar to conventional ECOC strategy.

For classifying objects with multiple classes and different sizes, most existing STM methods need to exploit OVO or OVR strategies to train a series of binary classifiers to perform multiclass classification indirectly. The OVO and OVR strategies inherently assume that multiclass classifications can be solved by multiple independent bi-category classifications, ignoring the correlation between multiple classes [23]. Although there exist a few SVM methods that can handle multiclass classification directly, e.g., multiclass SVM [22] and its variants [23,24], their multiclass classification mechanism that is similar to OVR may generate excessive constraints to limit the classification hyperplane, leading to the increase of complexity for training classifier [31]. On the other hand, since the existing STMs only accept tensors with a fixed size as input, it needs to resize the object slices of different classes in RSIs into a fixed size. Considering that objects with different classes usually present different sizes, the typical two resizing operations will bring adverse effect on object recognition, of which the first type of resizing operation will lead to the loss of objects’ scale information, and the second type of resizing operation is easy to cause the slice to contain more background interferences. The examples of two types of resizing operations are given in Figure 1. It is seen that using the second type of resizing operation will lose the scale information of the ships and thus cannot identify ships according to their size feature, and using the first type of resizing operation will generate slices with a large size so that the slice containing ship with class 1 covers excessive background interferences. To deal with the slices to be classified present different sizes, the deep learning method, i.e., the scale free convolution neural network, is built to utilize the global average pooling to map feature maps to unify size [32]. However, for representative machine learning methods, e.g., STM, there is no related work that can process the input slices with different sizes. In comparison to image resizing, the objects with different sizes should be contained by slices with proper sizes to reduce the impact of background interferences and maintain inherent scale information of the contained object, and these slices can be naturally represented as tensors with different dimensions, denoted as multiscale tensors in this paper, while the existing STMs can only process tensor with the same size and cannot process the slices with different sizes represented as multiscale tensors.

Motived by the abovementioned issues, the multiclass multiscale support tensor machine (MCMS-STM) is proposed in this paper. To deal with mutliclass classification, by integrating OVR and OVO strategies to optimization problems, a new multiclass classification mechanism is constructed to use multiple hyperplanes defined by rank-R tensors instead of a single hyperplane defined by rank-1 tensor of STM, where each hyperplane is needed to separate samples with specific classes. Furthermore, to classify objects with different sizes, according to positions of objects obtained from detection results, it is necessary to extract the objects slices with proper sizes rather than the fixed size to reduce the impact of background interferences and maintain inherent scale information of the contained object. These slices with different sizes can be naturally represented as tensors with different dimensions, denoted as multiscale tensors in this paper. Note that the existing STM methods can only process tensor with the same size and cannot process the slices with different sizes represented as multiscale tensors. To deal with input of multiscale tensors, instead of the fixed-dimensional hyperplane used in STM, the M-dimensional hyperplanes are built to separate input of multiscale tensors, and the resulting projecting value is used to predict the class label of input to achieve cross-scale object recognition. In addition, to train the OVO version of MCMS-STM efficiently, a decomposition algorithm is proposed to split the dual problem of the MCMS-STM into a series of sub-optimizations to accelerate the training.

The remainder of this paper is organized as follows. Section 2 consists of some preliminaries, such as the basic definitions and notions, the classical SVM and STM methods. In Section 3, the OVR version and OVO version of MCMS-STM and the corresponding solving methods are presented. Then, the decomposition algorithm is constructed to accelerate the training of the OVO version of MCMS-STM, and the relationship between the multiclass classification mechanism used in MCMS-STM and the existing methods is discussed. In Section 4, the experiments are conducted on publicly RSIs to analyze the parameter setting and the impact of image resizing operation and evaluate the performance of the MCMS-STM. Our conclusion is given in Section 5.

2. Preliminaries and Related Work

Before presenting the MCMS-STM, the notations, abbreviations, the basic tensor algebra used throughout this paper, and the traditional SVM and STM are introduced briefly as follows.

2.1. Notations, Abbreviations, and Tensor Operation

Tensor is the expansion of vector and matrix to the higher dimension, and tensor algebra, also known as multilinear algebra, is the extension of linear algebra to multiway data. To distinguish between tensors, matrices, and vectors, according to the convention in [33], the used symbols throughout this paper are summarized in

Table 1. The symbols and their corresponding description used in this paper.

Symbol	Description
lowercase letters (e.g., x,y)	scalar
lowercase boldface letters (e.g., x,y)	vector
uppercase boldface letter (e.g., M)	matrix
calligraphy letter (e.g., $\mathcal{X}$)	tensor

.

Then, we summarize all the notations and abbreviations used throughout this paper in

Table 2. The description of notations and abbreviations for the proposed MCMS-STM.

Notations and Abbreviations	Description
M	the number of classes, as well as the number of scales.
R	the CP rank of projection tensor.
L	the number of orders for projection tensor.
$w_{m,m^{\prime }}^{\left(r,l\right)}$	$\mathrm{the} \mathrm{projection} \mathrm{vector} \mathrm{of} \mathrm{mode}-l \mathrm{used} \mathrm{to} \mathrm{separate} \mathrm{class} m \mathrm{from} \mathrm{class} m^{\prime }$
$w_m^{\left(r,l\right)}$	the projection vector of mode-l used to separate class m from others
${\xi }_i^m$	slacking variable
C	regularization parameter
$b_{y_i,m}$	the bias used to separate class yi from class m
$b_{y_i}$	the bias used to separate class yi from others
$a_i^m$	dual variable
SVM	support vector machine
STM	support tensor machine
OVO	one-versus-one
OVR	one-versus-rest
CP	CANDECOMP/PARAFAC

.

The used basic tensor operations throughout this paper are given as follows.

Definition 1.

Frobenius norm and inner product of tensors: The Frobenius norm${\Vert \cdot \Vert }_F$of a tensor$\mathcal{A}\in {ℝ}^{I_1,\dots ,I_M}$is calculated as

$${\Vert \mathcal{A}\Vert }_F=\sqrt{{\displaystyle \sum _{i_1=1,\dots ,i_M=1}^{i_1=I_1,\dots ,i_M=I_M}\mathcal{A}\left(i_1,\dots ,i_M\right)\times \mathcal{A}\left(i_1,\dots ,i_M\right)}}$$

(1)

The inner product between tensors $\mathcal{A}\in {ℝ}^{I_1,\dots ,I_M}$, $\text{ℬ}\in {ℝ}^{I_1,\dots ,I_M}$ is defined as the sum of the products of their corresponding entries, i.e.,

$$\langle \mathcal{A},\mathcal{B}\rangle ={\displaystyle \sum _{i_1=1,\dots ,i_M=1}^{i_1=I_1,\dots ,i_M=I_M}\mathcal{A}\left(i_1,\dots ,i_M\right)\times \mathcal{B}\left(i_1,\dots ,i_M\right)}$$

(2)

Definition 2.

Mode-k product of tensor: Given tensor${\mathcal{A}\in ℝ}^{I_1,\dots ,I_M}$and matrix$\mathit{B}\in {ℝ}^{I_k,I_k^{’}}$, the mode-k product between$\mathcal{A}$and B is denoted as$\mathcal{A}{\times }_k\mathit{B}$, whose results are tensor${\mathcal{C}\in ℝ}^{I_1\times \dots \times I_k^{’}\times \dots \times I_M}$, as calculated by Equation (3).

$$\mathrm{C}\left(I_1,\dots ,I_{k-1},q,I_{k+1},\dots ,I_M\right)=\mathcal{A}{\times }_k\mathit{B}={\displaystyle \sum _{p=1}^{I_k}\mathcal{A}\left(I_1,\dots ,I_{k-1},p,I_{k+1},\dots ,I_M\right)}\times \mathit{B}\left(p,q\right)$$

(3)

Definition 3.

Outer product: The outer product of M vectors (i.e.,${\left\{{\mathit{a}}_m\in {ℝ}^{I_m}\right\}}_{m=1}^M$) is denoted as$\mathcal{A}={\mathit{a}}_1\circ \dots \circ {\mathit{a}}_M$, and the corresponding results are represented as an M-order tensor$\mathcal{A}\in {ℝ}^{I_1,\dots ,I_M}$, whose entry with coordinate$\left(i_1,\dots ,i_M\right)$is calculated by

$$\mathcal{A}\left(i_1,\dots ,i_M\right)={\mathit{a}}_1\left(i_1\right)\times ,\dots ,\times {\mathit{a}}_M\left(i_M\right)$$

(4)

2.2. Classical Binary and Multiclass Support Vector Machine

Consider bi-category classification task, given training set containing N samples, i.e., $\left\{x_i,y_i\right\}|\begin{array}{l}N\\ i=1\end{array}$, where $x_i\in {ℝ}^{I_1}$ and $y_i\in \left\{1,-1\right\}$ denote the input vector of ith sample and the corresponding label, respectively. SVM aims at learning the parameters of classification hyperplane with the largest classification margin from the training set, which can be drawn using the following optimization problem.

$$\begin{array}{l}\underset{w,b}{min}\frac{1}{2}{w}^{T}w+C{\displaystyle \sum _{i=1}^N{\xi }_i}\\ s.t.y_i\left(x_i{}^{T}w+b\right)\ge 1-{\xi }_i 1\le i\le N\\ \hspace{1em}{\xi }_i\ge 01\le i\le N\end{array}$$

(5)

where w, b, ${\xi }_i$ and C denote the normal vector for classification hyperplane, the bias, the slacking variable, and the regularization parameter, respectively. To obtain the optimal w, the dual problem can be drawn as follows. The detailed derivation can refer to [17].

$$\begin{array}{l}\underset{w,b}{min}\frac{1}{2}aQa-a^{T}e\\ s.t.{\displaystyle \sum _i{y}_i{a}_i}=0\\ \hspace{1em}0\le a_i\le C\hspace{1em}1\le i\le N,\end{array}$$

(6)

where $a$ denotes the Lagrangian multiplier. Q is a positive semidefinite matrix, where $\mathit{Q}\left(i,j\right)=y_i{y}_j{x}_i^T{x}_j$. After solving the optimal $\mathbf{\alpha }$, the $w$ can be calculated by Equation (7).

$$w={\displaystyle \sum _i{a}_i{y}_i{x}_i}.$$

(7)

Then, the label of test sample x can be predicted by Equation (8).

$$y\left(x\right)=sign\left(x^{T}w+b\right),$$

(8)

where $sign(\cdot )$ denotes the sign function.

The standard SVM can only deal with the bi-category classification problem, while most practical applications can be regarded as multiclass classification problems. Facing this situation, the multiclass SVM, i.e., an extension of SVM in multiclass classification problems, is established to identify samples with different classes.

Given the training set ${\left\{x_i,y_i\right\}|}_{i=1}^N$, where $x_i$ and $y_i\in \left\{1,\dots ,M\right\}$ denote the input vector of the ith sample and the corresponding class label. The multiclass SVM aims at learning M classification hyperplanes simultaneously by solving the following optimization problem.

$$\begin{array}{l}\underset{{w_m|}_{m=1}^M,b}{min}\frac{1}{2}{\displaystyle \sum _{m=1}^M{w}_m{}^T{w}_m}+C\times {\displaystyle \sum _{i=1}^N{\displaystyle \sum _{m\ne y_i}{\xi }_i^m}}\\ s.t.x_i{}^T{w}_{y_i}+b_{y_i}\ge x_i{}^T{w}_m+b_m+2-{\xi }_i^m 1\le i\le N m\ne y_i\\ \hspace{1em}{\xi }_i^m\ge 0 1\le i\le N m\ne y_i,\end{array}$$

(9)

where ${w_m|}_{m=1}^M$, b_m and ${\xi }_i^m$ denote the normal orientation of M classification hyperplanes, the mth bias, the slacking variable, respectively. The exact solving method can refer to [22].

According to the resulting ${w_m|}_{m=1}^M$ and b_m, the class label of test sample x can be predicted using the following decision function.

$$y\left(x\right)=\underset{m}{max}\left\{x^T{w}_m+b_m|1\le m\le M\right\}$$

(10)

2.3. Support Tensor Machine

To better use the structural information of input data represented as a tensor, the STM is extended from SVM to directly separate input in tensor space.

The given training set containing N samples $\left\{{\mathcal{X}}_i,y_i\right\}|\begin{array}{l}N\\ i=1\end{array}$, where ${\mathcal{X}}_i\in {ℝ}^{I_1,\mathrm{\dots },I_M}$ denotes the input tensor and $y_i\in \left\{1,-1\right\}$ denotes the corresponding label. Compared to SVM that separates input in vector space, the STM utilizes the mode-k product to process input in tensor space. The corresponding optimization problem of STM is given in Equation (11).

$$\begin{array}{l}\underset{w_i|\begin{array}{l}M\\ i=1\end{array}}{min}\frac{1}{2}{\Vert w_1\circ ,\dots ,\circ w_m\Vert }_F^2+C{\displaystyle \sum _{i=1}^N{x}_i}\\ s.t.y_i\left({\mathcal{X}}_i{\displaystyle \prod _{j=1}^M{\times }_j{w}_j}+b\right)\ge 1-{\xi }_i 1\le i\le N\\ \hspace{1em}{\xi }_i\ge 0 1\le i\le N\end{array}$$

(11)

where $w_j\in {ℝ}^{I_j}$ denotes the projection vector along with the jth mode of the input tensor. The $\mathcal{W}=w_1\circ \dots \circ w_m\in {ℝ}^{I_1,\dots ,I_M}$ denotes the projection tensor used to indicate the orientation of the normal orientation for the classification hyperplane. Since the parameters of STM $w_1\circ \dots \circ w_m\in R^{I_1\times ,\dots ,\times I_M}$ is the M-order tensor, it is difficult to be solved directly. In this way, the alternating optimization scheme is introduced to solve the above optimization, i.e., optimize each $w_{i_k}$ in turn by fixing the rest ${w_i|}_{i\ne i_k}$ in the kth iteration, where $i_k=mod\left(k,m\right)+1$ denotes the index of order to be optimized. The optimization problem in kth iteration is equivalent to that in SVM (see Equation (5)) it can be solved by using SVM solver. After the alternating optimization procedure is terminated, the optimal parameters ${w_j|}_j$ and b can be utilized to predict the label of the test sample $\mathcal{X}$, as shown below.

$$y\left(\mathcal{X}\right)=sign\left(\mathcal{X}{\displaystyle \prod _{j=1}^l{\times }_j{w}_j}+b\right)$$

(12)

3. Multiclass Multiscale Support Tensor Machine

As discussed above, the standard STM fails to deal with multiclass classification, and it requires resizing preprocessing to generate slices of the fixed size to comply with its input requirements, causing loss of scale information or increase of background interferences. To address multiclass multiscale object recognition in RSIs, the MCMS-STM is proposed by solving simultaneously hyperplanes of multiple dimensions defined by multiscale rank-R tensors to directly classify objects with different classes and sizes, as illustrated in Figure 2.

3.1. Construction of Multiclass Multiscale Support Tensor Machine

Starting from the standard STM, the classifier is extended gradually from two aspects, i.e., multiclass and multiscale, to form the proposed MCMS-STM.

3.1.1. Extend STM to Deal with Multiclass Classification

Consider the training set ${\left\{{\mathcal{X}}_i,y_i\right\}}_{i=1}^N$ set with N samples of M classes, where ${\mathcal{X}}_i$ and $y_i\in \left\{1,2,\dots ,M\right\}$ denote the L-order tensor representation of i-th image slice and the corresponding class label, respectively. Note that the standard STM can process input represented as a tensor, while it cannot process multiclass classifications directly. In contrast, the multiclass SVM can deal with multiclass classifications, while it cannot process input represented as a tensor. To achieve multiclass classification in tensor space, a straightforward idea is to construct the following optimization problem to learn parameters of multiple hyperplanes in tensor space simultaneously by integrating the merits of STM (see Equation (11)) and multiclass SVM (see Equation (9)).

$$\begin{array}{l}\underset{{w_m^l|}_{m=1,l=1}^{M,L},b}{min}\frac{1}{2}{\displaystyle \sum _{m=1}^M{\Vert w_m^1\circ \dots \circ w_m^L\Vert }_F^2}+C\times {\displaystyle \sum _{i=1}^N{\displaystyle \sum _{m\ne y_i}{\xi }_i^m}}\\ s.t.{\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_{y_i}^l}+b_{y_i}\ge {\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_m^l}+b_m+2-{\xi }_i^m\\ \hspace{1em}{\xi }_i^m\ge 0 1\le i\le N m\ne y_i,\end{array}$$

(13)

where $w_m^1\circ \dots \circ w_m^L\in {ℝ}^{I_1,I_2,\dots ,I_L}$ denotes the mth projection tensor used to determine the normal orientation of mth hyperplane. Compared with the optimization problem of multiclass SVM (see Equation (9)), the optimization problem in Equation (13) allows the tensor ${\mathcal{X}}_i$ as input by constructing projection tensor ${\mathcal{W}}_m$ with the same dimensions as ${\mathcal{X}}_i$, while more detailed modifies are essential to improve the performance of multiclass classification.

According to the definition of tensor rank, e.g., CANDECOMP/PARAFAC (CP) rank [34], the tensor ${\mathcal{W}}_m$ belongs to the rank-1 tensor. Referring to the study in [28,29] that the single rank-1 tensor cannot be used to describe the classification hyperplane accurately, it is considered to utilize multiple rank-1 tensors to improve the effect of classification, as shown in Equation (14).

$$\begin{array}{l}\underset{{w_m^{\left(r,l\right)}|}_{m,g,l},b}{min}\frac{1}{2}{\displaystyle \sum _{m=1}^M{\displaystyle \sum _{r=1}^R{\Vert w_m^{\left(r,1\right)}\circ \dots \circ w_m^{\left(r,L\right)}\Vert }_F^2}}+C{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{m=1}^M{\xi }_i^m}}\\ s.t.{\displaystyle \sum _r{\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_{y_i}^{\left(r,l\right)}}}+b_{y_i}\ge {\displaystyle \sum _r{\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_m^{\left(r,l\right)}}}+b_m+2-{\xi }_i^m\\ \hspace{1em}{\xi }_i^m\ge 0 m\ne y_i 1\le i\le N,\end{array}$$

(14)

where R denotes the number of rank-1 projecting tensors. In Equation (14), the hyperplane is defined by the sum of R rank-1 projection tensors (i.e., the rank-R projecting tensor ${\mathcal{W}}_m$). The illustration of hyperplanes defined by rank-1 projecting tensor and hyperplanes defined by rank-R tensor is shown in Figure 3. It is seen that the linear combination of rank-1 projecting tensors can be used to obtain the hyperplane with more orientations, which indicates that the rank-R projection tensor is likely to obtain a more effective hyperplane.

Note that the multiclass classification mechanism of Equation (14) is equivalent to that of multiclass SVM (see Equation (9)). In other words, the multiclass classification mechanism of Equation (14) can be described as an OVR strategy that solves M hyperplanes simultaneously, of which mth hyperplane defined by ${\mathcal{W}}_m$ is used to separate class m from the other. Considering the related research that the OVO strategy based SVM always achieves better results than OVR strategy based SVM, it is taken into consideration to embed OVO strategy to tensor space, of which the hyperplane defined by projection tensor ${\mathcal{W}}_{m,m^{\prime }}$ is used to separate samples from class m and samples from class $m^{\prime }$. In this way, the optimization problem in Equation (14) can be changed to the optimization problem in Equation (15).

$$\begin{array}{l}\underset{w_{m,m^{\prime }}^{\left(r,l\right)},b}{min}\frac{1}{2}{\displaystyle \sum _{m\ne m^{\prime }}{\displaystyle \sum _{r=1}^R{\Vert w_{m,m^{’}}^{\left(r,1\right)}\circ \dots \circ w_{m,m^{’}}^{\left(r,L\right)}\Vert }_F^2}}+C{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{m=1}^M{\xi }_i^m}}\\ s.t.{\displaystyle \sum _r{\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_{y_i,m}^{\left(r,l\right)}}}+b_{y_i,m}\ge {\displaystyle \sum _r{\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m,y_i}^{\left(r,l\right)}}}+b_{m,y_i}+2-{\xi }_i^m m\ne y_i 1\le i\le N\\ \hspace{1em}{\xi }_i^m\ge 0 m\ne y_i 1\le i\le N,\end{array}$$

(15)

where ${\mathrm{W}}_{m,m^{\prime }}={\displaystyle \sum _{r=1}^R{w}_{m,m^{\prime }}^{\left(r,1\right)}\circ \dots \circ w_{m,m^{\prime }}^{\left(r,L\right)}}$ and $b_{m,m^{\prime }}$ denote the normal orientation of the hyperplane used to separate class m from class $m^{’}$ and the corresponding bias. Since OVR and OVO strategies have their own advantages, these two classification mechanisms are used as different versions of MCMS-STM to cope with different classification tasks.

3.1.2. Classify the Multiscale Objects without Image Resizing Preprocessing

For the objects to be classified presenting different sizes, the convention manner is to adjust slices of different sizes into the fixed size, whereas this operation is easy to lead to the loss of scale information or increase of background interferences.

To maintain the scale information of objects and reduce the impact of background interferences, it is needed to extract slices with proper size to cover the object with a specific class, as shown in Figure 4. Assuming that each class of object presents a specific size, it is necessary to extract M sizes of slices of different sizes. Even for the same object, M slices of different sizes still need to be extracted because the category of the test sample is unknown (see Figure 4b). Therefore, different from the conventional manner that each object is described as a fixed-size slice, for each object, it is needed to extract M sizes of slices represented M-scale tensors ${{\mathcal{X}}_i^{\left(m\right)}|}_{m=1}^M$. In this way, a multiscale sample set ${\left\{{{\mathcal{X}}_i^{\left(m\right)}|}_{m=1}^M,y_i\right\}}_{i=1}^N$ is constructed, where ${\mathcal{X}}_i^{\left(m\right)}\in {ℝ}^{I_1^m,\dots ,I_L^m}$ denotes the slice of mth size containing ith object. To separate input represented as M-scale tensors ${{\mathcal{X}}_i^{\left(m\right)}|}_{m=1}^M$ in tensor space, it is also required to construct projection tensors with M scales, i.e., $\left\{{\mathcal{W}}_{m,m^{\prime }}\in {ℝ}^{I_1^m,\dots ,I_L^m}|1\le m\le M,m\ne m^{\prime }\right\}$, where the projection tensor with specific dimension is used to separate input slice with specific scale. In this way, the classification model in Equation (15) can be further extended to identify the multiscale objects without image resizing operation, and the corresponding optimization problem is shown in Equation (16).

$$\begin{array}{l}\underset{w_{m,m^{\prime }}^{\left(r,l\right)},b}{min}\frac{1}{2}{\displaystyle \sum _{m\ne m^{\prime }}{\displaystyle \sum _{r=1}^R{\Vert w_{m,m^{\prime }}^{\left(r,1\right)}\circ \dots \circ w_{m,m^{\prime }}^{\left(r,L\right)}\Vert }_F^2}}+C{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{m=1}^M{\xi }_i^m}}\\ s.t.{\displaystyle \sum _r{\mathcal{X}}_i^{\left(y_i\right)}{\displaystyle \prod _{l=1}^L{\times }_l{w}_{y_i,m}^{\left(r,l\right)}}}+b_{y_i,m}\ge {\displaystyle \sum _r{\mathcal{X}}_i^{\left(m\right)}{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m,y_i}^{\left(r,l\right)}}}+b_{m,y_i}+2-{\xi }_i^m m\ne y_i 1\le i\le N\\ \hspace{1em}{\xi }_i^m\ge 0 m\ne y_i 1\le i\le N,\end{array}$$

(16)

Equation (16) is the final optimization problem of OVO version of MCMS-STM. For OVR version of MCMS-STM, the final optimization problem is given directly as follows.

$$\begin{array}{l}\underset{w_{m,m^{\prime }}^{\left(r,l\right)},b}{min}\frac{1}{2}{\displaystyle \sum _{m=1}^M{\displaystyle \sum _{r=1}^R{\Vert w_m^{\left(r,1\right)}\circ \dots \circ w_m^{\left(r,L\right)}\Vert }_F^2}}+C{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{m=1}^M{\xi }_i^m}}\\ s.t.{\displaystyle \sum _r{\mathcal{X}}_i^{\left(y_i\right)}{\displaystyle \prod _{l=1}^L{\times }_l{w}_{y_i}^{\left(r,l\right)}}}+b_{y_i}\ge {\displaystyle \sum _r{\mathcal{X}}_i^{\left(m\right)}{\displaystyle \prod _{l=1}^L{\times }_l{w}_m^{\left(r,l\right)}}}+b_m+2-x_i^m m\ne y_i 1\le i\le N\\ \hspace{1em}{\xi }_i^m\ge 0 m\ne y_i 1\le i\le N,\end{array}$$

(17)

Note that the OVR version of MCMS-STM only uses M hyperplanes defined by ${\mathcal{W}}_m={\displaystyle \sum _{r=1}^R{w}_m^{\left(r,1\right)}\circ \dots \circ w_m^{\left(r,L\right)}}$ to separate samples.

On the one hand, the proposed MCMS-STM can use multiscale tensors to capture the category and scale information of objects and utilize grouped rank-R tensors to improve the effectiveness of multiclass classification problems. On the other hand, it can be found that the optimization problem of MCMS-STM is more complex than the binary STM, which needs to establish a specific solving method to obtain the optimal parameters.

3.2. Solving of the Optimization Problem for MCMS-STM

Since the procedures of training OVO and OVR versions of MCMS-STM are similar, we only give the solving methods for the OVO version of the MCMS-STM in this section and put the solving methods for the OVR version of the MCMS-STM in the Appendix A.

To solve the optimization problem in Equation (16) effectively, an alternating optimization scheme [25] is adopted for model training, i.e., optimize $\left\{{\mathcal{w}}_{m,m^{\prime }}^{\left(r,l_{iter}\right)}|1\le r\le R,1\le m\le M,m\ne m^{\prime }\right\}$ by fixing the other normal vectors in the kth iteration, where $l_{iter}=mod(k,L)+1$. First, all the projection vectors $\left\{w_{m,m^{\prime }}^{\left(r,l\right)}|1\le r\le R,1\le l\le L,1\le m\le M,m\ne m^{\prime }\right\}$ are initialized using uniformly distributed random numbers within 0 to 1. Then, we can get the Lagrangian function for alternating optimization in the kth iteration as follows.

$$\begin{array}{l}\mathcal{L}\left({w_{m,m^{\prime }}^{\left(r,l_{iter}\right)}|}_{m,m^{\prime },r},{b_{m,m^{\prime }}|}_{m,m^{\prime }},{a_i^m|}_{i,m},{b_i^m|}_{i,m}\right)=\frac{1}{2}{\displaystyle \sum _{m\ne m^{\prime }}{\displaystyle \sum _{r=1}^R{\Vert w_{m,m^{\prime }}^{\left(r,1\right)}\circ \dots \circ w_{m,m^{\prime }}^{\left(r,L\right)}\Vert }_F^2}}+\\ C\times {\displaystyle \sum _{i,m}{\xi }_i^m}-{\displaystyle \sum _i{\displaystyle \sum _m{a}_i^m({\displaystyle \sum _r{\mathcal{X}}_i^{\left(y_i\right)}{\displaystyle \prod _{l=1}^L{\times }_l{w}_{y_i,m}^{\left(r,l\right)}}}+b_{y_i,m}}}-{\displaystyle \sum _r{\mathcal{X}}_i^{\left(m\right)}{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m,y_i}^{\left(r,l\right)}}}-b_{m,y_i}-2+{\xi }_i^m)-{\displaystyle \sum _{i,m}{\beta }_i^m{\xi }_i^m}\\ s.t.{\alpha }_i^m\ge 0 1\le i\le N m\ne y_i\\ \hspace{1em}{\beta }_i^m\ge 0 1\le i\le N m\ne y_i\\ \hspace{1em}{\alpha }_i^{y_i}=0 1\le i\le N\\ \hspace{1em}{\beta }_i^{y_i}=0 1\le i\le N,\end{array}$$

(18)

where ${\alpha }_i^m$ and ${\beta }_i^m$ denote the dual variables, and ${\alpha }_i^{y_i}$ and ${\beta }_i^{y_i}$ denote the dummy dual variables. Let the partial derivatives of $\mathcal{L}\left({w_{m,m^{\prime }}^{\left(r,l_{iter}\right)}|}_{m,m^{\prime },r},{b_{m,m^{\prime }}|}_{m,m^{\prime }},{{\alpha }_i^m|}_{i,m},{{\beta }_i^m|}_{i,m}\right)$ with respect to the original variables be zeros. We have

$$\begin{array}{l}\frac{\partial \mathcal{L}}{\partial w_{m,m^{\prime }}^{\left(r,l_{iter}\right)}}=0\Rightarrow \\ w_{m,m^{\prime }}^{\left(r,l_{iter}\right)}=\frac{1}{{\displaystyle \prod _{l\ne l_{iter}}^L{w}_{m,m^{\prime }}^{\left(r,l\right)}{}^T{w}_{m,m^{\prime }}^{\left(r,l\right)}}}\times ({\displaystyle \sum _{\left\{i|y_i=m\right\}}{\alpha }_i^{m^{\prime }}{\mathcal{X}}_i^{\left(m\right)}{\displaystyle \prod _{l\ne l_{iter}}^L{\times }_l{w}_{m,m^{\prime }}^{\left(r,l\right)}}}-{\displaystyle \sum _{\left\{i|y_i=m^{\prime }\right\}}{a}_i^m{X}_i^{\left(m\right)}{\displaystyle \prod _{l\ne l_{iter}}^L{\times }_l{w}_{m,m^{\prime }}^{\left(r,l\right)}}})\end{array}$$

(19)

$$\begin{array}{l}\frac{\partial \text{ℒ}}{\partial b_{m,m^{\prime }}}=0\Rightarrow {\displaystyle \sum _{\left\{i|y_i=m\right\}}{\alpha }_i^{m^{\prime }}}-{\displaystyle \sum _{\left\{i|y_i=m^{\prime }\right\}}{\alpha }_i^m}=0\\ \frac{\partial \text{ℒ}}{\partial {\xi }_i^m}=0\Rightarrow C-{\beta }_i^m-{\alpha }_i^m=0.\end{array}$$

(20)

Substituting Equation (19) and Equation (20) into Equation (18) yields the following Lagrangian dual problem.

$$\begin{array}{l}\underset{{{\alpha }_i^m|}_{i,m}}{min}\frac{1}{2}({\displaystyle \sum _{m\ne m^{\prime }}{\displaystyle \sum _{r=1}^R{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{j=1}^N(c_j^{y_i}{\mu }_{\left(r,m\right)}^{y_i}\langle {\overline{\mathit{a}}}_{i,r,m},{\overline{\mathit{a}}}_{j,r,m}\rangle -2c_i^m{c}_j^{m^{\prime }}{\mu }_{\left(r,m^{\prime }\right)}^m\langle {\overline{\mathit{a}}}_{i,r,m^{\prime }},{\overline{\mathit{a}}}_{j,r}^m\rangle +c_i^{m^{\prime }}{c}_j^{m^{\prime }}{\mu }_{\left(r,m^{\prime }\right)}^m\langle {\overline{\mathit{a}}}_{i,r}^m,{\overline{\mathit{a}}}_{j,r}^m\rangle )}}}})-2{\displaystyle \sum _{i,m}{\alpha }_i^m}\\ s.t. {\displaystyle \sum _{\left\{i|y_i=m\right\}}{\alpha }_i^{m^{\prime }}}-{\displaystyle \sum _{\left\{i|y_i=m^{\prime }\right\}}{\alpha }_i^m}=0 m\ne m^{\prime }\\ \hspace{1em}\hspace{1em}0\le {\alpha }_i^m\le \mathrm{C} 1\le i\le N m\ne y_i\\ \hspace{1em}\hspace{1em}{\alpha }_i^{y_i}=0 1\le i\le N,\end{array}$$

(21)

where

$$\begin{array}{l}{c}_j^m=\{\begin{array}{l}1 if y_j=m\\ 0 otherwise\end{array}\\ {\mu }_{\left(r,m^{\prime }\right)}^m=\frac{1}{{\displaystyle \prod _{l\ne l_{iter}}^L{w}_{m,m^{\prime }}^{\left(r,l\right)}{}^T{w}_{m,m^{\prime }}^{\left(r,l\right)}}}\\ {\overline{\mathit{a}}}_{i,r,m}=a_i^m{\mathcal{X}}_i^{\left(y_i\right)}{\displaystyle \prod _{l\ne l_{iter}}^L{\times }_l{w}_{y_i,m}^{\left(r,l\right)}}\\ {\overline{\mathit{a}}}_{i,r}^m=a_i^m{\mathcal{X}}_i^{\left(m\right)}{\displaystyle \prod _{l\ne l_{iter}}^L{\times }_l{w}_{m,y_i}^{\left(r,l\right)}}.\end{array}$$

(22)

For convenience, rewrite the dual problem in the matrix form as follows.

$$\begin{array}{l}\underset{a}{min}\frac{1}{2}{\mathit{a}}^T\mathit{Q}\mathit{a}-2{\mathit{a}}^T{\mathit{e}}_{N\times M}\\ s.t.{\displaystyle \sum _{\left\{i|y_i=m\right\}}{\alpha }_i^{m^{\prime }}}-{\displaystyle \sum _{\left\{i|y_i=m^{\prime }\right\}}{\alpha }_i^m}=0 m\ne m^{\prime }\\ \hspace{1em}0\le {\alpha }_i^m\le C 1\le i\le N m\ne y_i\\ \hspace{1em}{\alpha }_i^{y_i}=0 1\le i\le N,\end{array}$$

(23)

where $\mathit{Q}={\displaystyle \sum _{m=1}^M{\displaystyle \sum _{r=1}^R\left[{\mathit{K}}_1^{\left(r\right)}-{\mathit{K}}_2^{\left(r,m\right)}-{\mathit{K}}_2^{\left(r,m\right)T}+{\mathit{K}}_3^{\left(r,m\right)}\right]}}$, and

$$\begin{array}{l}{\mathit{e}}_{N\times M}=\left[\underset{N\times M}{\underbrace{1,1,\dots ,1}}\right]\\ {\mathit{K}}_1^{\left(r\right)}\left(\left(i-1\right)\times M+m_1,\left(j-1\right)\times M+m_2\right)=\\ c_j^{y_i}\times {\mu }_{\left(r,m_1\right)}^{y_i}\times \tilde{\delta }\left(y_i,m_2\right)\delta \left(m_1,m_2\right)\times \\ \langle {\mathcal{X}}_i^{\left(y_i\right)}{\displaystyle \prod _{l\ne l_{iter}}^L{\times }_l{w}_{y_i,m_1}^{\left(r,l\right)}},{\mathcal{X}}_j^{\left(y_j\right)}{\displaystyle \prod _{l\ne l_{iter}}^L{\times }_l{w}_{y_j,m_2}^{\left(r,l\right)}}\rangle \end{array}$$

(24)

$$\begin{array}{l}{\mathit{K}}_2^{\left(r,m\right)}\left(\left(i-1\right)\times M+m_1,\left(j-1\right)\times M+m_2\right)=\\ c_i^{m_2}{c}_j^m\tilde{\delta }\left(m,m_2\right){\mu }_{\left(r,m\right)}^{y_i}\times \\ \langle {\mathcal{X}}_i^{\left(y_i\right)}{\displaystyle \prod _{l\ne l_{iter}}^L{\times }_l{w}_{y_i,m}^{\left(r,l\right)}},{\mathcal{X}}_j^{\left(m_2\right)}{\displaystyle \prod _{l\ne l_{iter}}^L{\times }_l{w}_{m_2,m}^{\left(r,l\right)}}\rangle \\ {\mathit{K}}_3^{\left(r,m\right)}\left(\left(i-1\right)\times M+m_1,\left(j-1\right)\times M+m_2\right)=\\ c_i^m{c}_j^m\delta \left(m_1,m_2\right)\times \tilde{\delta }\left(m,m_2\right){\mu }_{\left(r,m\right)}^{m_1}\times \\ \langle {\mathcal{X}}_i^{\left(m_1\right)}{\displaystyle \prod _{l\ne l_{iter}}^L{\times }_l{w}_{m_1,m}^{\left(r,l\right)}},{\mathcal{X}}_j^{\left(m_2\right)}{\displaystyle \prod _{l\ne l_{iter}}^L{\times }_l{w}_{m_2,m}^{\left(r,l\right)}}\rangle \\ a=\left[a_1^1,\dots ,a_1^M,a_2^1,\dots ,a_N^M\right],\end{array}$$

(25)

where

$$\delta \left(m_1,m_2\right)=\{\begin{array}{l}1 if m_1=m_2\\ 0 otherwise.\end{array}$$

(26)

$$\tilde{\delta }\left(m_1,m_2\right)=\{\begin{array}{l}0 if m_1=m_2\\ 1 otherwise.\end{array}$$

(27)

It is found from Equation (23) that the dual problem of MCMS-STM in kth iteration is the quadratic optimization in terms of $\mathit{a}$, which can be solved by classical quadratic programing method (e.g., interior-point method [35]). According to the resulting $\mathit{a}$, the $w_{m,m^{’}}^{\left(r,l_{\mathit{iter}}\right)}$ can be updated using Equation (19).

Note that the values of the objective function (see Equation (16)) in each iteration form a bounded and nonincreasing sequence. Therefore, there is a finite limit for that sequence. To stop iterating at the proper time, the termination criteria are constructed in Equation (28).

$${\displaystyle \sum _{m\ne m^{\prime }}{\displaystyle \sum _{r=1}^R{\displaystyle \sum _{l=1}^L{\Vert \left(w_{m,m^{\prime }}^{\left(r,l\right)}-w_{m,m^{\prime }}^{\left(r,l\right)’}\right)\Vert }_F^2}}}\le \delta ||k\ge T_{iter},$$

(28)

where $w_{m,m^{\prime }}^{\left(r,l\right)}$, $\delta$ and $T_{iter}$ denote the updated $w_{m,m^{\prime }}^{\left(r,l\right)}$ in last iteration, a pre-established positive number and the threshold for the number of iterations. When the termination criteria are met, output current projection vectors $w_{m,m^{’}}^{\left(r,l\right)}$. To calculate bias b, according to the K.K.T. conditions [36], including complementary slackness, dual feasibility, and the equivalent constraints in Equation (19) and Equation (20), we obtain

$$\begin{array}{l}{a}_i^m({\displaystyle \sum _r{\mathcal{X}}_i^{\left(y_i\right)}{\displaystyle \prod _{l=1}^L{\times }_l{w}_{y_i,m}^{\left(r,l\right)}}}+b_{y_i,m}-{\displaystyle \sum _r{\mathcal{X}}_i^{\left(m\right)}{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m,y_i}^{\left(r,l\right)}}}-b_{m,y_i}-2+{\xi }_i^m)=0\\ {\beta }_i^m{\xi }_i^m=0\\ {\beta }_i^m\ge 0\\ {\alpha }_i^m\ge 0\\ C-b_i^m-a_i^m=0.\end{array}$$

(29)

Combined with Equation (29), the equations and inequalities in terms of b can be drawn in following:

$$\begin{array}{l}0<{\alpha }_i^m<C\Rightarrow p_i^m=0\\ {\alpha }_i^m=C\Rightarrow p_i^m\le 0\\ {\alpha }_i^m=0,y_i\ne m\Rightarrow p_i^m\ge 0,\end{array}$$

(30)

where $p_i^m={\displaystyle \sum _r{\mathcal{X}}_i^{\left(y_i\right)}{\displaystyle \prod _{l=1}^L{\times }_l{w}_{y_i,m}^{\left(r,l\right)}}}+b_{y_i,m}-{\displaystyle \sum _r{\mathcal{X}}_i^{\left(m\right)}{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m,y_i}^{\left(r,l\right)}}}-b_{m,y_i}-2$.

When the optimal solution is reached, the above equations and inequalities hold. Therefore, b can be calculated by following linear programming.

$$\begin{array}{l}\underset{b,u}{min}{\displaystyle \sum _{m^{\prime },m,j}\left|u_j^{\left(m^{\prime },m\right)}\right|}\\ {\displaystyle \sum _{y_i=m^{\prime }}{p}_i^m}+u_1^{\left(m^{\prime },m\right)}=0\\ {\displaystyle \sum _{y_i=m^{\prime }}{p}_i^m}+u_2^{\left(m^{\prime },m\right)}\le 0 \\ {\displaystyle \sum _{y_i=m^{\prime }}{p}_i^m}+u_3^{\left(m^{\prime },m\right)}\ge 0\\ 1\le m^{\prime }\le M m\ne m^{\prime } {\alpha }_i^m=0,\end{array}$$

(31)

where $u_1^{\left(m^{\prime },m\right)}$, $u_2^{\left(m^{\prime },m\right)}$ and $u_3^{\left(m^{\prime },m\right)}$ denote the slacking variables used to maintain the feasibility of linear programming. Using the classical simplex method [37], the Equation (31) can be solved to obtain the optimal b. At present, the training procedure of MCMS-STM has been finished.

After training MCMS-STM, for any test sample ${{\mathcal{X}}^m|}_{m=1}^M$, the following decision function is used to predict the corresponding label.

$$arg \underset{m}{max}\#\{{\displaystyle \sum _r{\mathcal{X}}_i^{\left(m\right)}{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m,m^{\prime }}^{\left(r,l\right)}}}+b_{m,m^{\prime }}>{\displaystyle \sum _r{\mathcal{X}}_i^{\left(m^{\prime }\right)}{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m^{\prime },m}^{\left(r,l\right)}}}+b_{m^{\prime },m}|m^{\prime }\ne m\},$$

(32)

where operator $\#\{\cdot \}$ denotes the number of elements in the set. Note that there may be multiple candidate classes that maximize the Equation (32). For this situation, just like the operation in [20], we simply select the candidate class with the smallest label as the predicted class.

3.3. Acceleration of Training of OVO Version of MCMS-STM

In this section, combining the existing decomposition algorithm [38], a MCMS-STM-oriented decomposition algorithm is presented to break the complex quadratic programming (QP) for dual problem of MCMS-STM (see Equation (23)) into a series of simple analytic QP problems to train the OVO version of MCMS-STM efficiently.

To solve the QP in Equation (23) efficiently, based on the idea of a general decomposition algorithm, the dual variables $a$ are split into working set and non-working set, where variables in the working set are updated, and those in the non-working set are fixed in each iteration. To select the proper variables as the working set, a graph-based constraints model is illustrated in Figure 5 to demonstrate the constraints in Equation (23) clearly.

As shown in Figure 5, there are $M\times M$ vertices in the graph, where each vertex indicates a set of dual variables, i.e., ${\mathit{a}}_1^1=\left\{{\alpha }_i^{m^{\prime }}|1\le i\le N,y_i=1,m^{\prime }=1\right\}$. The constraints in Equation (23) can be described visually via the graph-based constraints model, where the value of dual variables in each vertex should be between 0 and C, and the sum of the values for dual variables contained in the vertices (i,j) is equal to that in vertex (i,j), i.e., the vertex with the same color in Figure 5 should have the same sum. Since updating the variables contained in the vertices of the same color does not affect the feasibility of other variables, it is taken into consideration to select variables from the vertices of the same color as the working set.

First, the zeros values are assigned to all the dual variables as the initial feasible solution of the dual problem in Equation (23). Subsequently, select a working set from the vertices with the same color ${\mathit{a}}_i^j$ and ${\mathit{a}}_i^j$. For variables in ${\mathit{a}}_i^j$ and ${\mathit{a}}_i^j$, the constraints can be described as follows.

$$s.t.{\displaystyle \sum _{\left\{k|y_k=i\right\}}{\alpha }_k^j}-{\displaystyle \sum _{\left\{k|y_k=j\right\}}{\alpha }_k^i}=0\Rightarrow {\displaystyle \sum _{\left\{y_k=i|k\right\}}{y}_k^{\left(i,j\right)}{\alpha }_k^j}+{\displaystyle \sum _{\left\{y_k=j|k\right\}}{y}_k^{\left(j,i\right)}{\alpha }_k^i}=0,$$

(33)

where $y_k^{\left(j,i\right)}=1$ and $y_k^{\left(j,i\right)}=-1$. It is noticed that the equivalent constraints in Equation (33) are equal to those in C-SVM (see Equation (6)). Therefore, similar to the variables selection strategy in C-SVM [38], the two variables that maximize the following objective function are selected as a working set.

$$\begin{array}{l}\underset{{\alpha }_{i_1}^{j_1},{\alpha }_{i_2}^{j_2}}{min}{d}_1\frac{\partial f\left(\mathit{a}\right)}{\partial {\alpha }_{i_1}^{j_1}}+d_2\frac{\partial f\left(\mathit{a}\right)}{\partial {\alpha }_{i_2}^{j_2}}\\ s.t.\left\{d_1,d_2,{\alpha }_{i_1}^{j_1},{\alpha }_{i_2}^{j_2}\right\}\in S_1\left|\right|S_2\left|\right|S_3\left|\right|S_4\\ S_1=\left\{d_1=1,d_2=-1,{\alpha }_{i_1}^{j_1},{\alpha }_{i_2}^{j_2}|{\alpha }_{i_1}^{j_1}\in {\mathit{a}}_i^j,{\alpha }_{i_2}^{j_2}\in {\mathit{a}}_i^j,{\alpha }_{i_1}^{j_1}<C,{\alpha }_{i_2}^{j_2}>0\right\}\\ S_2=\left\{d_1=-1,d_2=1,{\alpha }_{i_1}^{j_1},{\alpha }_{i_2}^{j_2}|{\alpha }_{i_1}^{j_1}\in {\mathit{a}}_j^i,{\alpha }_{i_2}^{j_2}\in {\mathit{a}}_j^i,{\alpha }_{i_1}^{j_1}>0,{\alpha }_{i_2}^{j_2}<C\right\}\\ S_3=\left\{d_1=1,d_2=1,{\alpha }_{i_1}^{j_1},{\alpha }_{i_2}^{j_2}|{\alpha }_{i_1}^{j_1}\in {\mathit{a}}_i^j,{\alpha }_{i_2}^{j_2}\in {\mathit{a}}_j^i,{\alpha }_{i_1}^{j_1}<C,{\alpha }_{i_2}^{j_2}<C\right\}\\ S_4=\left\{d_1=-1,d_2=-1,{\alpha }_{i_1}^{j_1},{\alpha }_{i_2}^{j_2}|{\alpha }_{i_1}^{j_1}\in {\mathit{a}}_i^j,{\alpha }_{i_2}^{j_2}\in {\mathit{a}}_j^i,{\alpha }_{i_1}^{j_1}>0,{\alpha }_{i_2}^{j_2}>0\right\},\end{array}$$

(34)

where $f\left(\mathit{a}\right)=\frac{1}{2}{\mathit{a}}^T\mathit{Q}\mathit{a}-2{\mathit{a}}^T{\mathit{e}}_{N\times M}$, and d₁, d₁ denote the updating direction of variables in working set. The objective function in Equation (34) means selecting variables with the steepest descent direction as the working set. The constraints in Equation (34) ensure the feasibility of variables ${\alpha }_{i_1}^{j_1},{\alpha }_{i_2}^{j_2}$ in the working set after updating. If $\underset{{\alpha }_{i_1}^{j_1},{\alpha }_{i_2}^{j_2}}{min}{d}_1\frac{\partial f\left(\mathit{a}\right)}{\partial {\alpha }_{i_1}^{j_1}}+d_2\frac{\partial f\left(\mathit{a}\right)}{\partial {\alpha }_{i_2}^{j_2}}\ge 0$, it indicates that there are no proper working variables in $a_j^i$ and $a_j^i$ to decrease the $f\left(a\right)$, otherwise, select the $\left\{{\alpha }_{i_1}^{j_1},{\alpha }_{i_2}^{j_2}\right\}$ and $\left[d_1,d_2\right]$ as the working set and the updating direction, respectively. Then, solve the following optimization problem to determine the optimal step size for updating.

$$\begin{array}{l}\underset{d_{step}}{min}\frac{1}{2}{\left({\mathit{a}}_B+d_{step}\times \left[d_1,d_2\right]\right)}^T{\mathit{Q}}_{BB}\left({\mathit{a}}_B+d_{step}\times \left[d_1,d_2\right]\right)+\\ \left({\mathit{a}}_B+d_{step}\times \left[d_1,d_2\right]\right){\mathit{Q}}_{BN}{\mathit{a}}_N+\frac{1}{2}{\mathit{a}}_N{\mathit{Q}}_{NN}{\mathit{a}}_N-2{\left({\mathit{a}}_B+d_{step}\times \left[d_1,d_2\right]\right)}^T{\mathit{e}}_2-2{\mathit{a}}_N{}^T{\mathit{e}}_{N\times M-2}\\ s.t.0\le {\mathit{a}}_B+d_{step}\times \left[d_1,d_2\right]\le C,\end{array}$$

(35)

where $d_{step}$, ${\mathit{a}}_B$ and ${\mathit{a}}_N$ denote the step-size used for updating variables, the variables in working set, and the variables in non-working set, respectively. The $\left[\begin{array}{l}{Q}_{BB},Q_{BN}\\ Q_{NB},Q_{NN}\end{array}\right]$ is the permutation of the Q according to the selected working set. To obtain the optimal $d_{step}$, let the derivative of the objective function be zero. We have

$$d_{step}^{\ast }={\left({\left[d_1,d_2\right]}^T{\mathit{Q}}_{BB}\left[d_1,d_2\right]\right)}^{-1}\times {\left[d_1,d_2\right]}^T\left(2e_2-\left[Q_{\mathit{B}B},{\mathit{Q}}_{BN}\right]\mathit{a}\right),$$

(36)

where $d_{step}^{\ast }$ denotes the optimal step size. To avoid the updated ${\mathit{a}}_B$ lie outside the range from 0 to C, the final step size is determined by

$$\begin{array}{l}{d}_{step}=d_{step}^{\ast }-max\left\{d_{temp}^1,d_{temp}^2\right\}\\ d_{temp}^1=\{\begin{array}{l}\left({\alpha }_{i_1}^{j_1}+d_{step}^{\ast }\times d_1\right)-Cif\left({\alpha }_{i_1}^{j_1}+d_{step}^{\ast }\times d_1\right)>C\\ 0otherwise\end{array}\\ d_{temp}^2=\{\begin{array}{l}-\left({\alpha }_{i_2}^{j_2}+d_{step}^{\ast }\times d_2\right)if\left({\alpha }_{i_2}^{j_2}+d_{step}^{\ast }\times d_2\right)<0\\ 0otherwise,\end{array}\end{array}$$

(37)

where $d_{step}$ denotes the final step size. Use ${\mathit{a}}_B={\mathit{a}}_B+d_{step}\times \left[d_1,d_2\right]$ to update the variables in working set. Then, use Equation (34) to re-select the working set for updating in the next iteration. This process is performed alternatively until the following terminal criteria are met, i.e., $\underset{{\alpha }_{i_1}^{j_1},{\alpha }_{i_2}^{j_2}}{min}{d}_1\frac{\partial f\left(\mathit{a}\right)}{\partial {\alpha }_{i_1}^{j_1}}+d_2\frac{\partial f\left(\mathit{a}\right)}{\partial {\alpha }_{i_2}^{j_2}}\ge 0$ for any vertex $\left(i,j\right),i\ne j$.

After the terminal criteria are satisfied, the current $a$ is optimal. The decomposition algorithm for solving the dual problem of the OVO version of the MCMS-STM is summarized in Algorithm 1.

Algorithm 1. Decomposition algorithm of solving the dual problem of OVO version of the MCMS-STM.

Input: the Q of the dual problem of MCMS-STM and the labels $y_{i}1\le i\le N$.

Output: optimal a

for vertex in $\forall \left(i,j\right),i\ne j$

Step 1: Select working set using Equation (34).

If $\underset{{\alpha }_{i_1}^{j_1},{\alpha }_{i_2}^{j_2}}{min}{d}_1\frac{\partial f\left(\mathit{a}\right)}{\partial {\alpha }_{i_1}^{j_1}}+d_2\frac{\partial f\left(\mathit{a}\right)}{\partial {\alpha }_{i_2}^{j_2}}\ge 0$, continue; otherwise, perform Step 2.

Step 2: Calculate the step size using Equations (36) and (37), and update the variables in working set.

Step 3: If the terminal criteria are met, output the $a$ as optimal solution, and return.

end

4. Discussion of the Multiclass Classification Mechanism Used in MCMS-STM

To analyze the multiclass classification mechanism used in MCMS-STM compared with the existing OVO and OVR strategy, a detailed discussion is carried out as follows.

4.1. Discussion of OVR Version of MCMS-STM Compared with OVR Strategy Based STM

Considering the hard margin (i.e., the $C=\infty$) based $C-\mathrm{STM}$ [25] with OVR strategy, it is required to construct M classifiers, where the mth classifier is used to separate the class m from the other M-1 classes, as shown in the following optimization problem.

$$\begin{array}{l}\underset{{w_m^{\left(l\right)}|}_l,b}{min}\frac{1}{2}{\Vert w_m^{\left(1\right)}\circ \dots \circ w_m^{\left(L\right)}\Vert }_F^2\\ s.t.{\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_m^{\left(l\right)}}+b_m\ge 1 y_i=m\\ \hspace{1em}\hspace{1em}{\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_m^{\left(l\right)}}+b_m\le -1 y_i\ne m\end{array},$$

(38)

where $w_m^{\left(1\right)}\circ \mathrm{\dots }\circ w_m^{\left(L\right)}$ denotes the normal orientation for the mth hyperplane.

Then, let $R=1$ and $C=\infty$ for OVR version of MCMS-STM (see Equation (17)), the corresponding optimization problem can be converted to Equation (39).

$$\begin{array}{l}\underset{{w_m^{\left(l\right)}|}_{m,l},b}{min}\frac{1}{2}{\displaystyle \sum _{m=1}^M{\Vert w_m^{\left(1\right)}\circ \dots \circ w_m^{\left(L\right)}\Vert }_F^2}\\ s.t.{\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_{y_i}^{\left(l\right)}}+b_{y_i}\ge {\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_m^{\left(l\right)}}+b_m+2\\ \hspace{1em}m\ne y_i 1\le i\le N\end{array}$$

(39)

By adding the two inequalities in Equation (38), it is easy to find that the feasible solution of Equation (38) is also the feasible solution of Equation (39). Thus, the solution of Equation (38) is the feasible solution of Equation (39), but not necessarily the optimal one. That means the MCMS-STM may obtain the optimal solution with a larger classification margin compared with $C-\mathrm{STM}$ using OVR strategy, i.e., the better generalization ability.

4.2. Discussion of OVO Version of MCMS-STM Compared with OVO Strategy Based STM

When using OVO strategy [20] based $C-\mathrm{STM}$ to process M class classification problems, $\frac{M\left(M-1\right)}{2}$ classifiers are constructed, where each one is used to separate specific two classes. For separating classes m and m^′, the corresponding optimization problem of $C-\mathrm{STM}$ is shown below.

$$\begin{array}{l}\underset{{w_{m^{\prime }m}^{\left(l\right)}|}_l,b}{min}\frac{1}{2}{\Vert w_{m^{\prime },m}^{\left(1\right)}\circ \dots \circ w_{m^{\prime },m}^{\left(L\right)}\Vert }_F^2\\ s.t.{\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m^{\prime },m}^{\left(l\right)}}+b_{m^{\prime },m}\ge 1 y_i=m^{\prime }\\ \hspace{1em}{\mathcal{X}}_j{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m^{\prime },m}^{\left(l\right)}}+b_{m^{\prime },m}\le -1 y_j=m\end{array},$$

(40)

where $w_m^{\left(1\right)}\circ \mathrm{\dots }\circ w_m^{\left(L\right)}$ $b_{m{m}^{\prime }}$ denote the normal orientation of the hyperplane and bias of the hyperplane, respectively. For OVO version of MCMS-STM (see Equation (16)), let the $R=1$ and $C=\infty$. The corresponding optimization problem is shown below.

$$\begin{array}{l}\underset{{w_{m,m^{\prime }}^{\left(l\right)}|}_{m,m^{\prime },l},b}{min}\frac{1}{2}{\displaystyle \sum _{m=1}^M{\displaystyle \sum _{m^{\prime }¹m}{\Vert w_{m,m^{\prime }}^{\left(1\right)}\circ \mathrm{\dots }\circ w_{m,m^{\prime }}^{\left(L\right)}\Vert }_F^2}}\\ s.t.{\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m^{\prime },m}^{\left(l\right)}}+b_{m^{\prime },m}\ge {\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m,m^{\prime }}^{\left(l\right)}}+b_{m,m^{\prime }}+2\\ y_i=m^{\prime } m^{\prime }\ne m 1\le i\le N\end{array}$$

(41)

For any feasible solution of $w_{m\prime m^{\prime }}^{\left(1\right)}$ and $b_{m^{\prime },m}$ in Equation (40), let $w_{m\prime m^{\prime }}^{\left(1\right)}=-w_{m^{\prime },m}^{\left(1\right)}$ and ${\mathrm{b}}_{m,m^{\prime }}=-b_{m^{\prime },m}$, we can obtain $w_{m\prime m^{\prime }}^{\left(1\right)}$, $b_{m^{\prime },m}$, $w_{m\prime m^{\prime }}^{\left(1\right)}$ and $b_{m^{\prime },m}$ as the feasible solution of Equation (41). On the contrary, for the constraints in Equation (41), we obtain:

$$\begin{array}{l}{\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m^{\prime },m}^{\left(l\right)}}+b_{m^{\prime },m}\ge {\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m,m^{\prime }}^{\left(l\right)}}+b_{m,m^{\prime }}+2 y_i=m^{\prime }\\ {\mathcal{X}}_j{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m,m^{\prime }}^{\left(l\right)}}+b_{m,m^{\prime }}\ge {\mathcal{X}}_j{\displaystyle \prod _{l=1}^L{\times }_l{w}_{m^{\prime },m}^{\left(l\right)}}+b_{m^{\prime },m}+2 y_j=m\\ \Rightarrow \\ {\mathcal{X}}_i{\displaystyle \prod _{l=1}^L{\times }_l\left(w_{m^{\prime },m}^{\left(l\right)}-w_{m,m^{\prime }}^{\left(l\right)}\right)}+b_{m^{\prime },m}-b_{m,m^{\prime }}-1\ge 1 y_i=m\\ {\mathcal{X}}_j{\displaystyle \prod _{l=1}^L{\times }_l\left(w_{m^{\prime },m}^{\left(l\right)}-w_{m,m^{\prime }}^{\left(l\right)}\right)}+b_{m^{\prime },m}-b_{m,m^{\prime }}-1\le -1 y_j=m^{\prime }\end{array}$$

(42)

According to Equation (42), the constraints in Equation (41) can be converted to the same form as the constraints in Equation (40). Note that the feasible solutions of different methods can be transformed into each other using simple operations. Therefore, the MCMS-STM has the OVO interpretation.

Through the above analysis, the MCMS-STM present both the OVR and OVO interpretations under the specific parameter setting. More importantly, compared with the OVR and OVO strategies that need to learn multiple classification hyperplanes separately, a remarkable advantage is that the MCMS-STM can learn the multiple classification hyperplanes simultaneously to mine the correlation between multiple classes.

5. Experiments and Analysis

To demonstrate the superiority of the MCMS-STM for multiclass multiscale object recognition, two datasets containing image slices of multiclass multiscale objects are used to evaluate the performance of the proposed MCMS-STM, and the detailed information of datasets is introduced as follows.

(1)

Dataset 1: To verify the performance of MCMS-STM for multiclass multiscale airplane classification, the RSIs containing 218 airplanes with five types are collected from Google Earth service with a spatial resolution of 0.5 m and R, G, and B spectral bands. Then, using two image resizing operations, these 218 airplanes are cut separately from RSIs to build two slice sets. For slices set 1 generated from image resizing operation 1, the slices are cut according to the type of contained objects and then resized to a fixed size using a bilinear interpolation method. For slices set 2 generated from image resizing operation 2, these slices are cut at a size large enough (i.e., $120\times 120$) so that all the types of objects are contained completely in the corresponding slice. These slices contain various backgrounds, and the contained multiclass airplanes present different orientations and sizes. Some representative slices from two slices sets of dataset 1 are displayed in Figure 6.

(2)

Dataset 2: The HRSC-2016 [39] dataset contains 1070 harbor RSIs with R, G, and B spectral bands collected from Google Earth service. To evaluate the performance of MCMS-STM for multiscale object recognition, 342 ships with five types are sliced from HRSC-2016 whose spatial resolution is equal to 1.07 m. Similar to dataset 1, these slices are cut by two image resizing operations to form two slices set. For slices set 1 generated from image resizing operation 1, the slices are cut according to the type of contained objects and then resized to a fixed size using a bilinear interpolation method. For slices set 2 generated from image resizing operation 2, these slices are cut at a size large enough (i.e., $770\times 170$) to contain airplanes with different types. Some image slices with different types of ships in two slices sets of dataset 2 are shown in Figure 7.

The experiments are composed of four parts. In Section 5.1, the impact of parameter setting is analyzed. In Section 5.2, the affection of image resizing preprocessing for the recognition results is examined. In Section 5.3, the efficiency of the proposed decomposition algorithm is verified compared with the typical interior-point method and active-set method. In Section 5.4, the recognization accuracy of MCMS-STM is evaluated compared with typical SVM and STM methods using dataset 1 and dataset 2. In Section 5.5, the performance of MCMS-STM is further evaluated compared with typical deep learning methods. All the simulations are running on the computer with Windows 10 operating system and Intel i7-7700CPU at 3.6 G Hz.

The code of MCMS-STM can be found at https://github.com/supergt3/MCMS-STM (accessed on 23 November 2021).

5.1. Analysis of the Impact of Parameter Setting on Classification Performance

The main parameters in MCMS-STM include the R and C, which denote the CP rank of the projection tensor and the regularization parameter, respectively. To demonstrate the impact of parameter setting, the N-fold cross-validation method is adopted to partition the slices set 1 of dataset 1 into N subsets, and test the recognization accuracy using the one subset according to the trained classifier using the remaining N-1 subsets. This process is then repeated N times, with each of the N subsets used exactly once as the test samples. Through adjusting the values of R from $\left\{1,2,3,\dots ,10\right\}$ and C from $\left\{{10}^0,{10}^1,{10}^2\right\}$, the obtained classification accuracies of different versions of MCMS-STM are shown in Figure 8.

From Figure 8, it is observed that the recognization accuracy of MCMS-STM is affected by different parameter settings. In detail, it is seen that the increase of C may have a positive or negative effect on recognization accuracy for different values of R. Therefore, it is difficult to set an effective C in advance. In addition, for the case of using the same C, MCMS-STM with R < 3 obtains smaller recognization accuracy than that with $R\ge 3$, because the projection tensor with a small R is difficult to define an accurate classification hyperplane. Through observing Figure 8, it is found that the best recognization accuracy is obtained by OVO version of the MCMS-STM. The reason is probably that OVO version of the MCMS-STM may obtain a larger classification margin compared with OVR version of the MCMS-STM.

5.2. Analysis of the Impact of Image Resizing on Classification Performance

One of the significant advantages of the MCMS-STM method is that it can use multiscale projection tensors to effectively classify objects with different sizes, avoiding the loss of objects’ scale information caused by image resizing. To highlight the superiority of MCMS-STM, two slices sets obtained by different image resizing methods are utilized to examine the recognization accuracy of the MCMS-STM with multiscale projection tensors and single-scale projection tensors. The experiments are implemented in three control groups. In detail, for control group 1, the MCMS-STM with single-scale projection tensors and slices set 1 are utilized to obtain the classification results. For control group 2, the MCMS-STM with single-scale projection tensors and slices set 2 are utilized to obtain the classification results. For control group 3, the MCMS-STM with multiscale projection tensors and the slices set 1 are utilized to obtain the classification results. To use multiscale projection tensors, it is needed to decide the sizes of multiscale tensors. In the experiments, the sizes of slices for five types of ships are set to $770\times 170$, $360\times 50$, $400\times 70$, $550\times 140$ and $290\times 70$, respectively, according to the average sizes of objects for the different types. The obtained classification results for MCMS-STM under different conditions are plotted in Figure 9.

The accuracy in Figure 9 denotes the largest accuracy with parameters C and R selected from $C=\left\{{10}^0,{10}^1,{10}^2\right\}$ and $R=\left\{2,4,6,8,10\right\}$ under different versions of MCMS-STM. For MCMS-STM, the OVO version indicates the larger classification margin, but at the same time, it will also cause misclassification because the decision function (see Equation (32)) may generate multiple results. Therefore, for dataset 2, it is observed that the recognization accuracy of OVR version of MCMS-STM is higher than that of OVO version of MCMS-STM. In addition, note that the resizing operation used to generate slices set 1 will lead to the loss of object’s scale information, and the resizing operation used to generate slices set 2 will bring excessive background interferences, i.e., the two resizing operations have their own disadvantages. In comparison, the multiscale projection tensors used in MCMS-STM can maintain the scale information of objects while avoiding excessive background interferences. Therefore, it is observed that the recognization accuracy of MCMS-STM without resizing operations is better than using MCMS-STM with resizing operations. The above analysis concludes that the MCMS-STM with multiscale projection tensors is effective for multiscale object recognition tasks.

5.3. Evaluation of the Performance of the Decomposition Algorithm for Training MCMS-STM

To verify the efficiency of the proposed decomposition algorithm for training OVO version of MCMS-STM, the experiments are conducted on dataset 1 to evaluate the time consumption of training MCMS-STM and classical multiclass SVM using different optimization solving algorithms, including the proposed decomposition algorithm, interior-point method [35] and active-set method [40], under different numbers of training samples, as shown in Figure 10.

The time consumption in Figure 10 represents the performing time to solve the dual problem of the MCMS-STM or multiclass SVM once. From Figure 10, when using the same optimization algorithm, i.e., interior-point algorithm or active-set algorithm, it can be seen that the time consumption of solving dual problem of MCMS-STM is similar to that of solving dual problem of multiclass SVM. In addition, it is seen that the time consumption of the interior-point algorithm is less than that of the active-set algorithm under different sizes of training sample sets. Remarkably, it can be seen that the proposed decomposition algorithm significantly reduces the time consumption compared with the interior-point method and active-set method for training MCMS-STM, especially for the training set with a larger size. It indicates the efficiency of the proposed decomposition algorithm for training the MCMS-STM, especially for the training set with a large size.

5.4. Evaluation of the Performance of MCMS-STM Compared with Existing SVM and STM Methods

In this section, the experiments are conducted on dataset 1 and dataset 2 to evaluate the performance of MCMS-STM for multiclass multiscale airplane classification and ship classification compared with typical SVM and STM methods. To avoid the classification results from being affected by different feature extraction algorithms, the image slices represented as tensors are used as input directly for all the classification methods. In detail, according to the size of different objects, the dimensions of multiscale tensors for airplanes with five types are set to for $100\times 100\times 3$, $80\times 80\times 3$, $120\times 120\times 3$, $70\times 70\times 3$ and $80\times 80\times 3$, respectively, and those for ships with five types are set to $770\times 170\times 3$, $360\times 50\times 3$, $400\times 70\times 3$, $550\times 140\times 3$ and $290\times 70\times 3$, respectively. The first order, second order and third order for multiscale tensors denote the horizontal spatial order, vertical spatial order, and spectral order, respectively. For the comparison methods, two slice sets from dataset 1 and dataset 2 obtained by different slice cropping methods are utilized to obtain the classification results under two types of image resizing operations, respectively. In addition, note that the SVM and STM methods can only deal with bi-category classification problems. Therefore, the OVO and OVR strategies are introduced to perform multiclass classifications for binary SVM and STM methods indirectly. Considering that the SVM method cannot process tensors directly, the vectorization operation is utilized to convert image slice to vector to comply with its input requirement. Then, all the methods select parameters from $C=\left\{{10}^0,{10}^1,{10}^2\right\}$, $\nu =\left\{0.1,0.2,0.3,0.4\right\}$ and $R=\left\{2,4,6,8,10\right\}$ that corresponds to the best classification results to obtain the final recognization accuracy. Using the N-fold cross-validation, the accuracies of MCMS-STM for different classification tasks under different conditions are displayed in

Table 3. The accuracies of different methods for multiscale airplanes recognization using 5-fold cross-validation.

Method	Parameter Setting	Accuracy
OVR version of MCMS-STM	C = 1 R = 8	84.9%
OVO version of MCMS-STM	C = 1 R = 6	85.3%
$C-SVM$ (OVO) 1	C = 10	82.2%
$C-SVM$ (OVR) 1	C = 100	73.8%
$\nu -SVM$ (OVO) 1	$\nu =0.1$	80.7%
$\nu -SVM$ (OVR) 1	$\nu =0.2$	76.2%
Multi-class SVM1	C = 100	82.6%
$C-STM$ (OVO) 1	C = 10	71.4%
$C-STM$ (OVR) 1	C = 1	66.5%
$C-SVM$ (OVO)2	C = 10	79.4%
$C-SVM$ (OVR) 2	C = 100	77.1%
$\nu -SVM$ (OVO) 2	$\nu =0.1$	80.3%
$\nu -SVM$ (OVR) 2	$\nu =0.1$	75.2%
Multi-class SVM2	C = 10	81.7%
$C-STM$ (OVO) 2	C = 1	75.2%
$C-STM$ (OVR) 2	C = 1	70.6%

,

Table 4. The accuracies of different methods for multiscale airplanes recognization using 10-fold cross-validation.

$\mathrm{Method}C-STM$	Parameter Setting	Accuracy
OVR version of MCMS-STM	C = 10 R = 8	88.1%
OVO version of MCMS-STM	C = 10 R = 8	89.5%
$C-SVM$ (OVO) 1	C = 10	84.0%
$C-SVM$ (OVR) 1	C = 1	76.6%
$\nu -SVM$ (OVO) 1	$\nu =0.1$	85.3%
$\nu -SVM$ (OVR) 1	$\nu =0.2$	77.3%
Multi-class SVM1	C = 1	85.3%
$C-STM$ (OVO) 1	C = 100	79.4%
$C-STM$ (OVR) 1	C = 1	74.8%
$C-SVM$ (OVO)2	C = 100	87.6%
$C-SVM$ (OVR) 2	C = 1	78.0%
$\nu -SVM$ (OVO) 2	$\nu =0.1$	88.6%
$\nu -SVM$ (OVR) 2	$\nu =0.2$	79.4%
Multi-class SVM2	C = 100	88.1%
$C-STM$ (OVO) 2	C = 1	77.5%
$C-STM$ (OVR) 2	C = 1	71.2%

,

Table 5. The accuracies of different methods for multiscale ships recognization using 5-fold cross-validation.

Method	Parameter Setting	Accuracy
OVR version of MCMS-STM	C = 100 R = 6	90.2%
OVO version of MCMS-STM	C = 1 R = 6	87.4%
$C-SVM$ (OVO) 1	C = 1	86.1%
$C-SVM$ (OVR) 1	C = 1	80.7%
$\nu -SVM$ (OVO) 1	$\nu =0.1$	86.1%
$\nu -SVM$ (OVR) 1	$\nu =0.1$	79.0%
Multi-class SVM1	C = 1	85.3%
$C-STM$ (OVO) 1	C = 10	76.9%
$C-STM$ (OVR) 1	C = 1	73.1%
$C-SVM$ (OVO)2	C = 10	83.3%
$C-SVM$ (OVR) 2	C = 100	79.2%
$\nu -SVM$ (OVO) 2	$\nu =0.1$	84.5%
$\nu -SVM$ (OVR) 2	$\nu =0.1$	79.5%
Multi-class SVM2	C = 100	84.2%
$C-STM$ (OVO) 2	C = 1	77.0%
$C-STM$ (OVR) 2	C = 1	70.6%

, and

Table 6. The accuracies of different methods for multiscale ships recognization using 10-fold cross-validation.

Method	Parameter Setting	Accuracy
OVR version of MCMS-STM	C = 10 R = 8	87.7%
OVO version of MCMS-STM	C = 1 R = 8	91.4%
$C-SVM$ (OVO) 1	C = 1	87.4%
$C-SVM$ (OVR) 1	C = 100	83.6%
$\nu -SVM$ (OVO) 1	$\nu =0.1$	87.0%
$\nu -SVM$ (OVR) 1	$\nu =0.1$	82.4%
Multi-class SVM1	C = 1	89.9%
$C-STM$ (OVO) 1	C = 1	78.2%
$C-STM$ (OVR) 1	C = 1	74.4%
$C-SVM$ (OVO)2	C = 100	85.6%
$C-SVM$ (OVR) 2	C = 1	81.0%
$\nu -SVM$ (OVO) 2	$\nu =0.1$	86.2%
$\nu -SVM$ (OVR) 2	$\nu =0.2$	80.5%
Multi-class SVM2	C = 100	86.2%
$C-STM$ (OVO) 2	C = 1	76.4%
$C-STM$ (OVR) 2	C = 10	69.6%

, respectively.

In these tables, the bolding accuracy indicates the best results, and the notation $C-SVM$ (OVO) ₁ denotes using OVO strategy based $C-SVM$ under slice set 1. Overall, note that the classification results for 10-fold cross-validation are better than those for 5-fold cross-validation, mainly because the 10-fold cross-validation indicates that more samples are used for training. From these tables, it is observed that applying OVO strategy based comparison methods present better classification results than applying OVR strategy based comparison methods because the OVO is a competitive multiclass classification strategy compared with OVR strategy. For N-fold cross-validation, by comparing the classification accuracies between N = 5 and N = 10, it can be found that there is a minor difference between different N for STM methods and a significant difference between different N for SVM methods. The reason is that the SVM method is embedded with much more parameters, and thus it is easy to improve performance as the number of samples increases. In addition, it can be seen that the recognization accuracy of multiclass SVM is better than most comparison methods under OVR or OVO strategy. Since the two resizing operations have their own advantages and disadvantages, i.e., the first type of resizing operation can reduce the impact of background interferences, while it loses the scale information of object, and the image the second type of resizing operation can maintain the scale information of objects, while it brings much more interferences for objects with small size, it is observed that using the slices set 1 obtains the largest accuracy for ship recognization under 10-fold cross-validation (see Table 6), and using the slices set 2 obtains the largest accuracy for airplane recognization under 10-fold cross-validation (see Table 4). That means two image resizing operations present comparable results. Moreover, it is seen that the optimal R that corresponds to the largest accuracy is located at the range from 4 to 8, as the projection with small R is difficult to describe the effective classification hyperplane, and the projection tensor with large R is easy to over-fit the training samples. It is worth observing that the MCMS-STM gets the best classification results among all the methods for different classification tasks. In detail, MCMS-STM gets 89.5% and 91.4% recognization accuracy for airplane recognization and ship recognization, respectively, while the largest accuracies of comparison methods for airplane recognization and ship recognization are equal to 88.6% and 89.9%, respectively. Therefore, it is concluded that the MCMS-STM is more effective for multiclass multiscale object recognition using remote sensing images compared with the comparison methods.

In addition, to verify whether the MCMS-STM makes improvements from a statistical view, the statistical test is applied to analyze the performance of the MCMS-STM against the SVM and STM methods.

For convenience, the most commonly applied test method, i.e., right-tailed significance t-test [41], is used to test the null hypothesis H₀ that the difference of accuracy between MCMS-STM and the competitor is equal to zero. On the other hand, the alternate hypothesis H₁ is that the difference of accuracy between MCMS-STM and the competitor is greater than zero.

To obtain effective results, half of samples in dataset 1 are used to train a classifier, and the rest of the samples in dataset 1 are used to evaluate the recognization accuracy of trained classifier for five classes of airplanes. The same processing is performed for dataset 2. In this way, ten paired observations are obtained for the sign test. According to obtained ten paired observations, use paired-sample right-tailed significance t-test [41], and the resulting p-value for MCMS-STM against the other methods is shown in

Table 7. The accuracies of different methods using the second type of normalization operation for multiscale airplanes classification using 10-fold cross-validation.

Method	Positive	Negative	Ties	p-Value
$C-SVM$ (OVO)	9	0	1	<0.0001
$C-SVM$ (OVR)	8	1	1	0.0045
$\nu -SVM$ (OVO)	9	0	1	<0.0001
$\nu -SVM$ (OVR)	8	2	0	0.0049
Multi-class SVM	8	0	2	0.0097
$C-STM$ (OVO)	9	0	1	<0.0001
$C-STM$ (OVR)	10	0	0	<0.0001

.

Where positive, negative, ties, and p-value denote the number of times that the MCMS-STM outperforms the comparison method, the number of times that the comparison method outperforms the MCMS-STM, the number of times that both the MCMS-STM and comparison method obtain the same results, and the probability of the observation under H₀. From Table 7, it is found that all the p-values are less than 0.05. Therefore, the null hypothesis can be rejected at the 0.05 significance level for all the comparison methods. It indicates that the proposed MCMS-STM outperforms the existing SVM and STM methods from the view of statistics.

5.5. Evaluation of the Performance of MCMS-STM Compared with Deep Learning Methods

Compared with deep learning methods relying on a large number of training samples, the proposed MCMS-STM can recognize multiclass multiscale objects effectively using a small number of training samples. To verify this advantage, the experiments are conducted on datasets 1 and 2 to compare the recognition accuracies of the MCMS-STM with two deep learning methods, i.e., GhostNet [42] and ResNeXt networks [43], of which the GhostNet utilizes the Ghost module to improve the feature representation power of the conventional convolutional layer and the ResNeXt improved from deep residual networks can exploit the aggregated residual transformations to mine the effective features to improve the recognization results. When training the GhostNet and ResNeXt, the learning rate and the batch size are set to 0.001 and 128, respectively. Since these two deep learning methods can only deal with slices with a fixed size, the slice sets 1 from two datasets are used to examine their performances. For MCMS-STM, the multiscale projection tensors are exploited, and the detailed sizes of projection tensors can be found in Section 5.4. To compare the recognization results of the proposed OVO version of MCMS-STM with the deep learning methods under different sizes of training sets, different proportions of samples in the dataset are selected as training samples and the rest as test samples, and the obtained recognization results are shown in

Table 8. The accuracies of the MCMS-STM and deep learning methods for dataset 1 under different numbers of training samples.

	p = 0.1	p = 0.2	p = 0.3	p = 0.4	p = 0.5
GhostNet	47.09%	48.50%	57.82%	61.42%	71.03%
ResNeXt	40.86%	46.06%	52.08%	54.84%	60.19%
MCMS-STM	61.73%	63.79%	73.20%	79.39%	81.65%

and

Table 9. The accuracies of the MCMS-STM and deep learning methods for dataset 2 under different numbers of training samples.

	p = 0.1	p = 0.2	p = 0.3	p = 0.4	p = 0.5
GhostNet	35.41%	35.03%	46.43%	50.66%	56.31%
ResNeXt	30.37%	34.97%	44.64%	47.74%	54.23%
MCMS-STM	62.21%	70.07%	70.29%	74.76%	79.53%

.

Where the notation p in Table 8 and Table 9 denotes the proportion of the training samples occupying the data set. From the above tables, it is observed that the recognization accuracy is generally improved with the increase of training set size because more training samples can ensure sufficient training of the classifier. In addition, it is seen that the GhostNet obtains better results than ResNeXt under the same p. It indicates that the GhostNet can extract more effective features using its Ghost module. Since the deep learning methods rely on a large number of training samples, they obtain worse results under a small p. Remarkably, it is worth noting that the proposed MCMS-STM obtains significantly better results compared to deep learning methods, especially when the training samples are small. This funding implies the superiority of the proposed MCMS-STM for the small sample case compared with the deep learning methods.

6. Conclusions

To classify the multiclass multiscale objects in RSIs effectively, the MCMS-STM is proposed incorporating multiple hyperplanes defined by multiscale projection tensors to map the input of object slices with different sizes to multiclass class space, getting rid of the conventional image resizing operation. The main contributions of this paper can be summarized in the three folds below.

(1)

To achieve multiclass classifications for objects in RSIs, the MCMS-STM is proposed to learn multiple hyperplanes defined by rank-R projection tensors simultaneously to map input represented as tensor into class space. This new multiclass classification mechanism makes it easy to construct the corresponding decomposition algorithm to accelerate the training of the MCMS-STM and enables the classifier to present OVO and OVR interpretations, ensuring the MCMS-STM can deal with different classifications tasks effectively.

(2)

To identify multiscale objects in RSIs, instead of the conventional image resizing operation, according to the object position obtained from detection results, multiple slices of different sizes are extracted to describe the contained object with unknown class, and multidimensional classification hyperplanes are established to separate input of multiple slices with different sizes to achieve cross-scale object recognition. This multiscale classification mechanism can avoid the loss of scale information and reduce the impact of background interferences caused by conventional image resizing preprocessing.

(3)

To accelerate the training of OVO version of MCMS-STM, combining graph-based analysis, a decomposition algorithm is proposed to break the dual problem of OVO version of MCMS-STM as a series of small sub-optimizations, reducing the time consumption caused by the large Q. It ensures the MCMS-STM can be trained efficiently for more samples and classes.

The future work will focus on extending the MCMS-STM to adapt to the objects with different orientations and constructing a universal decomposition algorithm to accelerate the training of different versions of MCMS-STM.