Saliency Preprocessing Locality-Constrained Linear Coding for Remote Sensing Scene Classification

Abstract

Locality-constrained Linear Coding (LLC) shows superior image classification performance due to its underlying properties of local smooth sparsity and good construction. It encodes the visual features in remote sensing images and realizes the process of modeling human visual perception of an image through a computer. However, it ignores the consideration of saliency preprocessing in the human visual system. Saliency detection preprocessing can effectively enhance a computer’s perception of remote sensing images. To better implement the task of remote sensing image scene classification, this paper proposes a new approach by combining saliency detection preprocessing and LLC. This saliency detection preprocessing approach is realized using spatial pyramid Gaussian kernel density estimation. Experiments show that the proposed method achieved a better performance for remote sensing scene classification tasks.

1. Introduction

Over recent decades, an overwhelming amount of high-resolution (HR) remote sensing images have become available. Since remote sensing images have abundant structural patterns and spatial information that are difficult to be fully applied directly, we need to correctly classify them by senses before further processing. Therefore, remote sensing scene classification is a central issue in remote sensing applications [1,2].

Various methods have been proposed to classify remote sensing scenes over the years. Bag-of-Features (BoF) [3,4] is a classical method in whole-image categorization tasks. This method first forms a histogram based on a remote sensing image’s local features, and then uses the histogram to represent the remote sensing image. However, this method lacks consideration of the spatial layout information of features in remote sensing images. There are some improvements based on the BoF method, such as those shown in References [5,6], and the Spatial Pyramid Matching (SPM) method [7], which is a successful method. The SPM method divides a remote sensing image into different scale spatial sub-regions. Then, histograms of local features from each sub-region are computed. Usually, $2^l\times 2^l$ sub-regions (where $l=0,1,2$) are used. SPM has shown a better performance than BoF in most image classification tasks; however, the traditional SPM approach also has limitations. It requires nonlinear classifiers to complete classification. To improve SPM, a new coding algorithm, named Locality-constrained Linear Coding (LLC), was proposed [8]. The LLC method is widely applied in image classification tasks. It considers both the locality constraints and global sparsity when coding remote sensing image. LLC shows a state-of-the-art classification accuracy. In this paper, the proposed remote sensing scene classification approach is based on LLC. The proposed approach will improve the codebook technology in traditional LLC by combing saliency detection technology, which is good for remote sensing scene classification.

LLC has achieved state-of-the-art performances in several image classification tasks; however, it is still based on modeling the cognitive mechanism of human vision to complete the image classification task. Physiological and psychophysical evidence indicates that the human’s visual system has evolved a specialized focus processing treatment called “attentive” mode, which is directed to particular locations in the visual field [9]. Based on the “attentive” mode, researchers have proposed several visual saliency detection methods [10]. LLC does not take into account visual saliency detection. Thus, we can improve LLC by combining it with visual saliency detection.

Saliency detection was introduced into the field of computer vision in the late 1990s. According to the computational view, saliency detection can be grouped into different algorithms. One category is the center-surround thought. This type of algorithm assumes that a local window exists, which can divide an image into a center containing an object and a surround [10,11,12,13]. Another category of saliency detection algorithms is based on frequency domain methods [14,15,16]. The other category is relying on information theory concepts [17,18,19]. In Reference [13], Fast and Efficient Saliency Detection (FESD) was proposed, which belongs to the first saliency detection algorithm category. In the FESD method, a saliency map is built by computing the kernel density estimation, which has a faster computing performance compared with several other density estimations [20,21]; however, most remote sensing images, in which saliency maps are obtained using FESD, are not always the best. This is because remote sensing images usually have a wide field of view and the size of the kernel used for FESD, chosen by experience, is not necessarily optimal. However, we found that FESD can be improved using a simple technique. The technique is named spatial pyramid Gaussian kernel density estimation (SPGKDE), which is suitable for saliency detection of remote sensing images. In our research, SPGKDE is used for the preprocessing of remote sensing images. Then, by combining LLC encoding, the classification accuracy of remote sensing images can be improved.

The main contribution of this paper is to propose a new kind of remote sensing scene classification method by combing SPGKDE saliency detection preprocessing and LLC to improve remote sensing scene classification accuracy. This paper is structured as follows. Section 2 describes the remote sensing image scene classification proposed method in detail. In particular, this section outlines SPGKDE saliency detection preprocessing, and explains how and why remote sensing scene classification accuracy can be improved by SPGKDE preprocessing. Section 3 shows a comparison of the experimental results based on traditional LLC classification and the proposed method, followed by discussions. Section 4 concludes the paper.

2. Methodology

As mentioned above, this paper proposes a saliency preprocessing locality-constrained linear coding method for remote sensing scene classification. Figure 1 shows the realization of this method. The core technologies are SPGKDE and saliency preprocessing LLC. More specifically, descriptions of these two techniques are as follows.

2.1. Spatial Pyramid Gaussian Kernel Density Estimation Saliency Detection Preprocessing

FESD builds a saliency map of a remote sensing image using Gaussian kernel density estimation [13]. This method also implicitly considers sparse sampling and center bias. Since the human eye is more focused on the center of an image and is accustomed to taking a photo with the subject in the center, this method works well for most images. However, this method is not universal for remote sensing images. In some remote sensing images, the scene occupies the entire image. Center bias can lead to the loss of some salient information, which is detrimental to the classification of remote sensing images. We therefore need to improve FESD and propose SPGKDE. Figure 2 shows the weakness of the FESD method, which is the loss of salient information.

In this paper, a spatial pyramid Gaussian kernel density estimation (SPGKDE) saliency detection based on FESD is proposed. It requires minor changes but offers a great improvement on FESD. It is proposed for obtaining a remote sensing image’s saliency map more effectively, which is used for saliency preprocessing locality-constrained linear coding.

The spatial pyramid method (SPM [7]) is a simple and practical method in computer vision. This method divides a remote sensing image, from coarse to fine, by level. Then, local features in each level are aggregates later. Usually, $2^l\times 2^l$ sub-regions (where $l=0,1,2$) are used. The proposed method, SPGKDE, is based on this thought. Figure 3 shows the realization of SPGKDE.

Assume that there is a remote sensing image $I$ and $R_i^l$ is one of its sub-regions. For each sub-region image $R_i^l$, each pixel exists as $x=(\overline{x},f)$, where f is a feature vector extracted from $R_i^l$, and $\overline{x}$ denotes the coordinate of pixel x in $R_i^l$. For each sub-region $R_i^l$, we can get its corresponding saliency map $S_i^l$ using the FESD method proposed in Reference [13].

$$S_i^l(x)=\mathfrak{A}\ast {[P_r^n(1|f,\overline{x})]}^{\alpha }$$

(1)

where ∗ is convolution operator, $\mathfrak{A}$ is a circular averaging filter, $P_r^n(1|f,\overline{x})$ is a calculated probability for characterizing the pixels in the saliency areas, and $\alpha \ge 1$ is a factor that affects high probability areas.

Then, the saliency map S of remote sensing image I can be calculated as follows:

$$S=d_0{S}_1^0+d_1{\displaystyle \sum _{i=1}^4{S}_i^1+}{d}_2{\displaystyle \sum _{i=1}^{16}{S}_i^2=}{\displaystyle \sum _{l=0}^2{\displaystyle \sum _{i=1}^{4^l}{d}_l{S}_i^l}}$$

(2)

where $d_l=(l=0,1,2)$ means a weight, and it can be defined as $d_l=\frac{1}{2^l}$, and $S_i^l$ is acquired by Equation (1). When the level l is increasing, the weight d_l is decaying to prevent the collection of too much useless salient information.

Finally, a preprocessed image is obtained by:

$$I^{\prime }=I+\xi \cdot S$$

(3)

where I’ is the preprocessed image, and ξ ∈ (0,1] is used in order to avoid adding invalid details caused by the saliency map.

SPGKDE has a stronger nature of saliency detection than FESD because of its re-aggregation of salient information from different image space scales. After saliency detection preprocessing, the salient information can increase the inter-class variations between different remote sensing scenes. This is beneficial for improving the accuracy of classification tasks. In this paper, the Gaussian kernel is fixed as a 9 × 9 size, and the proportional control coefficient ξ is fixed at 0.5.

2.2. Saliency Preprocessing Locality-Constrained Linear Coding

Different kinds of coding algorithms have been proposed in the past few decades [5,8], most of which usually consist of feature extraction and feature coding. Experimental results have shown that, with a certain visual codebook, utilizing different coding schemes will directly affect the remote sensing scene classification accuracies [22,23]. Meanwhile, sparsity is less essential than locality under certain assumptions, as pointed out in Reference [24]. Therefore, Locality-constrained Linear Coding (LLC) was proposed [8].

LLC is widely applied in image classification tasks. It adds local restrictions to achieve global sparsity. Based on a given visual codebook, LLC provides analytical solutions. This coding method also has a fast coding speed. In this paper, LLC is selected as the basic method for remote sensing image feature coding.

Let F be a set of L-dimensional local descriptors extracted from the remote sensing image, i.e., $F=[f_1,f_2,...,f_P]\in R^{L\times P}$. Given a codebook $B=[b_1,b_2,...,b_Q]\in R^{L\times Q}$ with Q entries, LLC obeys the following criteria:

$$\underset{C}{{\displaystyle \min }}{\displaystyle \sum _{i=1}^P{\Vert f_i-B{c}_i\Vert }^2+\lambda {\Vert d_i\odot c_i\Vert }^2},s.t.1^{\mathrm{T}}{c}_i=1,\forall i$$

(4)

where $d_i\in R^Q$ is the locality adaptor, and ⊙ is the element-wise multiplication. Usually, we have:

$$d_i=\exp (\frac{dist(f_i,B)}{\sigma })$$

(5)

where $dist(f_i,B)={[dist(f_i,b_1),...,dist(f_i,b_q)]}^{\mathrm{T}}$, and $dist(f_i,b_i)$ represent the distance between $f_i$ and b_i. $\sigma$ is the weight used for adjusting the locality adaptor decay speed. Further, we can get the following solutions:

$${\tilde{c}}_i=(C_i+\lambda dia{g}^2(d_i))1$$

(6)

$$c_i={\tilde{c}}_i/1^{\mathrm{T}}{\tilde{c}}_i$$

(7)

$$C_i=(B^{\mathrm{T}}-1f_i^{\mathrm{T}}){(B^{\mathrm{T}}-1f_i^{\mathrm{T}})}^{\mathrm{T}}$$

(8)

Unlike most coding methods, LLC can provide analytical solutions. It is of great benefit to computation. It also can be seen from Equation (8) that the quality of the given codebook $B$ directly affects the coding results in LLC, and, further, indirectly affects the classification accuracy of remote sensing image scene categories. To better complete the task of remote sensing scene classification, we make full use of the advantages of the saliency detection information. Thus, we propose saliency preprocessing LLC.

For each remote sensing image $I$, we can obtain its preprocessed image I’ by the method proposed in Section 2.1. We use image $I$ and I’ to get the corresponding codebook $B$ and $B^{\prime }$. Then, the given codebook $B$ in Equation (8) can be replaced by $B_{final}$, which uses the following formula:

$$B_{final}=\frac{1}{2}(B+B^{\prime })$$

(9)

where codebook $B$ represents the original remote sensing images features codebook, and $B^{\prime }$ comes from the computations of corresponding saliency preprocessed remote sensing images. Figure 4 shows the differences between the traditional LLC codebook and the proposed method codebook. Obviously, the features used to generate the new codebook $B_{final}$ are more prominent than those of codebook $B$ constructed from traditional LLC features, and this new codebook $B_{final}$ will improve the performance of remote sensing scene classification.

Saliency preprocessing LLC only solves the problem of feature coding. In this paper, Scale Invariant Feature Transform (SIFT) [25] and Local Binary Pattern (LBP) [26] are used to extract image features and a support vector machine (SVM) [27,28] is used as the training classifier. Of course, to prove the effectiveness of the proposed method, this paper uses a public 19-class remote sensing scene dataset to conduct experiments [29,30]. The dataset is proposed by Dengxin Dai and Wen Yang, and named as WHU-RS dataset. The images in this dataset are a fixed size of 600 × 600 pixels. All the images are collected from Google Earth. In the early days, the dataset contained 12 categories of physical scenes in the satellite imagery [29]; later, it was expanded to 19 classes [30]. The 19 classes of the dataset are airport, beach, bridge, river, forest, farmland, meadow, mountain, pond, parking, port, park, viaduct, desert, football field, railway station, residential area, industrial area and commercial area. The dataset has higher intra-class variations and smaller inter-class dissimilarities. The experimental results and discussions are given in the next section.

3. Experiments and Discussion

In this section, we will focus on experiments based on a WHU-RS dataset, demonstrate the advantage of SPGKDE preprocessing for remote sensing images, and then show the performance of both the traditional LLC and the proposed method. Thereafter, the results will be briefly discussed.

3.1. SPGKDE Preprocessing

Figure 5 shows several sample images of the WHU-RS dataset and the corresponding saliency detection results computed by SPGKDE (proposed in this paper).

As shown in Figure 5, most of the images can obtain useful and expected saliency maps via the proposed SPGKDE saliency detection, and after detection preprocessing, key parts of the images can be made more prominent. The preprocessing technique can increase inter-class variations and reduce inter-class dissimilarities. Though, inevitably, there are some unsatisfactory saliency maps for images that may lead to classification confusion; for instance, the pond scene image in Figure 5, where the features of the pond are vague, whereas the bridge becomes rather prominent after saliency detection preprocessing and would be wrongly classified into the park scene class, though this phenomenon is very rare. In fact, a very high percentage of images can correctly receive saliency detection preprocessing.

3.2. Performance of Traditional LLC and Proposed Method

In this paper, two kinds of low-level feature are used to form a codebook for LLC separately; namely, Scale Invariant Feature Transform (SIFT) and Local Binary Pattern (LBP).

3.2.1. Performance Based on SIFT Feature

The SIFT vector has a dimension of 128. Half of the dataset is used as the training set and the other half is used for the test. The accuracy of classification of traditional LLC and the proposed method based on SIFT is shown in

Table 1. Classification accuracies based on Scale Invariant Feature Transform (SIFT).

	Methods	Traditional BoF	Traditional LLC	Proposed Method
Descriptors
SIFT		72.87%	73.27%	79.01%

, where the BoF performance is also shown as a baseline method.

Figure 6 shows the confusion matrices generated by the traditional LLC and the proposed method based on SIFT. It is benefit for observing more differences in detail between the two methods. We can observe that more scene image categories, such as airport, bridge and commercial area were correctly classified by the proposed method. Although the classification accuracies of meadow, parking and residential area categories were reduced, especially the classification accuracy of meadow, which dropped sharply by 12%, the method improved the classification accuracy of the entire public dataset by about 6%.

To investigate the impact of saliency detection preprocessing, different proportions of training and test are used for the experiments. The training proportion is ranged from 20% to 80%. The experimental results are shown in Figure 7. The classification accuracies are clearly improved with our proposed method.

3.2.2. Performance Based on LBP Feature

LBP vector is a 256-dimension feature vector. Half of the dataset is used as the training set and the other half is used for the test. The accuracy of classification of the traditional LLC and the proposed method based on LBP is shown in

Table 2. Classification accuracies based on Local Binary Pattern (LBP).

	Methods	Traditional BoF	Traditional LLC	Proposed Method
Descriptors
LBP		68.71%	72.87%	78.22%

, where the BoF performance is shown as a baseline method as well.

To observe more classification details of the proposed method, we gave the confusion matrices of traditional LLC and proposed method by Figure 8. The confusion matrices are generated by the traditional LLC and the proposed method based on LBP. We can see that the classification accuracies of the four categories is reduced, namely football field, industrial, meadow and mountain, accounting for 21% of all categories, whereas the classification accuracy of the entire public dataset is improved by approximately 5.4%.

To investigate the impact of saliency detection preprocessing, different proportions of training and test are used for the experiments. The training proportion ranged from 20% to 80%. The experimental results are shown in Figure 9. As can be seen, the classification accuracies are improved with our proposed method.

Through the above experiments, we can see that the proposed method can improve the scene classification accuracies of remote sensing images, regardless of the features extracted by the SIFT or LBP methods. However, the classification accuracies of the meadow category are reduced using the proposed method. To determine why the classification of the meadow scene images is more difficult to identify using the proposed method, we reviewed the entire process of processing meadow category images. We found that almost all the meadow-category images in the WHU-RS dataset did not have an obvious saliency region, as shown in Figure 10. The most striking feature of the meadow category images was the color. When we used SPGKDE to preprocess meadow category images, not only did we get little color information, but there was also additional untrue saliency region information. Furthermore, weakening the color information and forcibly mining saliency region information, can lead to generating the wrong codebook for LLC, which is detrimental for classification based on LLC. This may explain why the proposed method reduced the classification accuracies of the meadow category. Fortunately, there are saliency regions in most remote sensing images and this happens only for meadow category images.

From the above experimental results, we have good reason to believe that the proposed method can improve remote sensing scene classification performance.

4. Conclusions

In this paper, an improvement saliency detection method named SPGKDE is proposed based on the existing saliency detection FESD for remote sensing image preprocessing. The new saliency detection method re-aggregates salient information from different image space scales and is obviously more applicable to remote sensing images. SPGKDE has a wider field of view than FESD.

Thus, a new kind of remote sensing scene classification method that combines SPGKDE saliency detection preprocessing and LLC is proposed. The method is easy to operate. Visual saliency detection plays an important role in remote sensing image analysis and the traditional LLC classification technology ignores this technology, resulting in a limited accuracy for image classification. This paper proposes integrating an improved saliency detection method—SPGKDE— into the LLC classification. This method can increase inter-class variations and reduce inter-class dissimilarities. In fact, the preprocessing method improves the codebook technology in traditional LLC. It is a core technology that directly determines the classification result. This method achieves a better simulation of the human visual system than traditional LLC. In this paper, both SIFT and LBP features were used for experiments. The experiments show that the proposed method is useful and can improve remote sensing scene classification accuracy.