A New Method of Fuzzy Support Vector Machine Algorithm for Intrusion Detection

Abstract

Since SVM is sensitive to noises and outliers of system call sequence data. A new fuzzy support vector machine algorithm based on SVDD is presented in this paper. In our algorithm, the noises and outliers are identified by a hypersphere with minimum volume while containing the maximum of the samples. The definition of fuzzy membership is considered by not only the relation between a sample and hyperplane, but also relation between samples. For each sample inside the hypersphere, the fuzzy membership function is a linear function of the distance between the sample and the hyperplane. The greater the distance, the greater the weight coefficient. For each sample outside the hypersphere, the membership function is an exponential function of the distance between the sample and the hyperplane. The greater the distance, the smaller the weight coefficient. Compared with the traditional fuzzy membership definition based on the relation between a sample and its cluster center, our method effectively distinguishes the noises or outlies from support vectors and assigns them appropriate weight coefficients even though they are distributed on the boundary between the positive and the negative classes. The experiments show that the fuzzy support vector proposed in this paper is more robust than the support vector machine and fuzzy support vector machines based on the distance of a sample and its cluster center.

1. Introduction

Intrusion detection systems (IDS) are essential to information security. IDS can be divided into signature-based IDS and anomaly-based IDS [1]. Both are based on pattern detection. Signature-based IDS matches the system behavior against the known attack and lacks the ability to detect zero-day attack. Anomaly-based methods construct normal behavior based on prior knowledge and judge the deviation between the current behavior and the normal behavior [2]. The advantage of the anomaly-based method is the ability to detect new attacks.

A system call requested by an application is a function built into the operation system kernel. A system call sequence is a detailed account of the system calls occurring on a host. The behavior of an application can be described in terms of the sequence of system calls. It is easy to get the system call sequence in real-time. Therefore, the data of a system call sequence is often used as audit data for analysis and classification of malicious processes.

The current methods of anomaly detection are based on traditional statistics, which is the study of the asymptotic theory. That is, the limit property can be reached when the number of samples approaches infinity. In intrusion detection systems, the observation samples are limited or even a small number. This cannot satisfy the preconditions of a detection method based on traditional statistics. As a result, the false alarm rate and missing rate are high.

The algorithms of a system call in anomaly-based IDS need a lot of data. This is because these algorithms are based on traditional statistics, which is the study of the asymptotic theory. That is, the limit property can be reached when the number of samples approaches infinity. Unfortunately, the anomaly data in the intrusion detection system is very limited. Therefore, we classify system calls using an SVM-based algorithm, which is based on statistical learning theory. Statistical learning theory makes the SVM-based classifier only depend on a small part of the support vectors (SVs). This is very helpful for the training of classifiers with insufficient data. SVM-based algorithm, like the other algorithm, also has an inherent shortcoming, that is, it is sensitive to noises and outliers. In order to distinguish noises near the boundary from SV, fuzzy support vector machine is proposed to solve the problem. Because there are no uniform guiding principles, the existing FSVM-based algorithms are inconsistent with reality when classifying system call sequences. Therefore, the purpose of our paper is to construct a fuzzy support vector machine that is suitable for classifying system call sequences.

Support vector machine [3,4,5,6], as a machine learning method based on statistical learning theory, derives from the idea of the dual form to solve the large dimensional problems, makes the classifier only depend on a small part of the support vectors, implements the structural risk minimization principle in statistical learning theory, and solves the problems of nonlinearity and local minima. A system call sequence can be converted into a vector in a high dimensional space by the frequency of short system call sequences of a certain length. Therefore, abnormal detection can be carried out based on SVM.

There are always noises and outliers in solving practical engineering applications due to statistical methods, human error and other factors. These noises and outliers cannot satisfy the precondition that all samples are independent and identically distributed. The noises and outliers near the boundary play the same role as SVs in constructing the optimal classification hyperplane. To solve this problem, researchers proposed fuzzy support vector machine (FSVM), that is, different weights are assigned to different samples, so that different samples contribute differently to the optimal classification hyperplane. In order to eliminate the influence of noises and outliers, the small weights are given to these samples. The design of the membership function is the key of the whole fuzzy algorithm and is no uniform criterion to be followed. At present, there are many ways to construct a membership function. Most of these methods are based on the distance between a sample and its cluster center. The closer the sample is to the cluster center, the greater the weight coefficient is. The noises and SVs distributed on the boundary are far from the cluster center. They are all given less weight coefficients. This is quite different from the objective situation. SVs should be given greater weight coefficients.

To solve the above problem, our paper proposed a new fuzzy support machine method based on support vector data description (SVDD). The contributions of our work are as follows:

• Our paper proposed a new fuzzy membership function which can effectively distinguish the noises and SVs distributed on the boundary based on SVDD. SVs are given larger weight coefficients while noises are given smaller coefficients. In this way, our method avoids imperfection of FSVM based on the distance between a sample and its cluster center. Such a fuzzy membership function structure method is more in line with reality.
• The method proposed in our paper uses the hyperplane, which passes through the cluster center and takes the line of two cluster centers as the normal vector to replace each cluster center. This is in accordance with the geometric principle of SVM. In other words, two hyperplanes with maximum space are used to separate the training samples. Therefore, using the hyperplane in class to replace the cluster center can better approximate the actual situation.
• Our method is more efficient, especially for anomaly-based IDS with high real-time requirements. In our method, the noises and outliers are identified by a sphere with minimum volume while containing the maximum of the samples. Some uncontributed vectors are eliminated by pre-extracting the boundary vector set containing the support vectors. This reduces the number of training samples and speeds up the training. It is important that IDS speed up detection as much as possible by reducing computation and storage.

The remainder of the paper is organized as follows. In Section 2, the previous work based on system calls are reviewed, while Section 3 describes the proposed method. In Section 4, experiments and evaluations are presented. Section 5 gives some conclusions.

2. Previous Work

Anomaly detection can be studied by system call sequence from different angles. These methods focus primarily on data processing, data representation and other feature selections derived from system call sequences. In this section, some methods of anomaly-based IDSs based on system call will be discussed.

As the original data of system call traces is large, preprocessing and feature selection methods contribute to obtaining typical features and avoiding the influence of irrelevant and redundant features on detection rate and processing cost [7,8]. Methods commonly used for natural language processing are used to preprocess system call traces. The n-gram method is used to construct the system call databases of normal behavior by a sliding window with a single length or multiple lengths [9,10]. Aron Laszka et al. [11,12] investigated and claimed that the optimal n-gram is 6-gram in UNM dataset and 7-gram in ADFA-LD dataset. Suaad et al. [13] continued to prove that 6-gram and 10-gram have the advantage of time efficiency and detection rate respectively in a dataset collected from a virtual machine. Feature selection methods reduce redundancy and irrelevance by selecting interesting features. These methods facilitate the reduction of computation time and storage requirements, understanding data out noise and avoiding the over-fitting problem, increasing the accuracy rate. Feature selection can be divided into th wrapper approach [14], filter approach [15] and hybrid approach [16] according to the correlation of algorithms. This depends on the feedback represented by the accuracy rate; the wrapper approach implements the selection of best features. The filter approach evaluates the attributes of a learning algorithm by using the statistical learning data. The wrapper approach gets better classification performance than filter approach at the expense of expensive computation. The filter approach is better suited to handle high dimensional data than the wrapper approach. A hybrid approach was produced by combining the advantages and disadvantages of the filter approach and wrapper approach.

The enumerating sequences-based [17,18,19] methods are simple and efficient to implement by removing system call parameters. During database construction stage, normal behaviors are represented by short system sequences. During the monitoring stage, the short sequences of testing data are obtained and tested. The enumerating sequences-based methods need constructing, updating and maintainance of the normal database for each individual program [20,21]. The Murmurhash [22,23,24] is utilized with the Bloom-filter-based method to ensure that it has the advantage over STIDE in terms of memory occupation, searching speed and privacy preservation. Although the Bloom-filter-based method shows simplicity and effectiveness, it has the limitation of false positives.

The system call sequence can be represented as a vector. Qing et al. extracted a minimized set of rules to define a normal behavior and detected anomaly behavior based on a rough set [25,26]. Pandit predefined workflow and added a knowledge base of workflow [27,28,29]. A search engine is then applied to discover the hidden knowledge [30]. The drawback of a rule-based approach [31,32,33], since these rules are derived from small-scale datasets, is that the rules are constantly updated. Matej et al. [34] presented a new tool data collection system for Windows PC. Different from previously distributed data collection systems, this system uses less resources based on host and client structures. The system shows good performance in the real test environment. However, it is just a preliminary stage and is going to be a lot of work. IDS plays an important role in the network. Qiuhua et al. [35] proved a classification algorithm based on data clustering and data reduction by mini batch K-Means algorithm in the training stage and sorting cluster in the detection stage. Experiments indicated that the computational complexity was reduced significantly and the accuracy maintained high. However, the implementation of this classification method is complex.

In recent years, neural networks have made remarkable achievements in computer vision [36,37,38] and natural language processing [39,40,41]. Researchers have also tried to use neural networks to process system call sequences [42,43,44,45,46]. AnRAD [47] performs probabilistic inference by self-structuring confabulation network. Their network continuously refines their knowledge base and is capable of fast incremental learning. Sheraz et al. [48] implemented intrusion detection in a real environment based on a convolutional neural network. In order to accelerate the training process, multiple GPUs must be deployed on a physical host. The challenge of solutions based on a neural network is pricey and space consuming due to the increasing amount of data.

Ambusaidi et al. [49] proved that their method contributes more critical features for the least square support vector machine to achieve a better detection rate and lower computation cost. Gideon [50] applied a semantic structure to system calls. This approach facilitates the representation of software behavior and obtains excellent results in UNM dataset and KD98 dataset. Wael et al. [51] presented a heterogeneous detector which consisted of sequence time-delay embedding, hidden Markov model [52,53,54] and a one-class support machine. In addition to satisfactory results, the heterogeneous detector also exhibits the reliability. Michael et al. [55] detected features of the system in the hypervisor. The experiments demonstrated that their detection accuracy achieves 90% whilst the method has the detecting ability of DoS attacks. The algorithms based on frequency can be realized at a lower computation cost by reducing the dimension of the frequency vectors [56]. SVM is sensitive to noises and outliers [57,58,59,60]. FSVM is based on fuzzy theory to reduce the influence of noises or outliers on the classification hyperplane [61,62,63,64,65,66]. Lin et al. [67] proposed a method based on the relation between samples and their cluster center. Zhang et al. [68] proposed a new FSVM method after considering the imperfection of a distance-based algorithm. However, the above methods also reduce the effect of SVs on the hyperplane when reducing the influence of noises or outliers on the hyperplane.

3. Methodology Based on SVDD

Due to human error, random error and other factors in the data collection process, the sample set contains a small number of noise samples. Noise samples have a great influence on the construction of the optimal classification hyperplane, which makes it deviate from the optimal position, reduces the normalization ability of the classifier, and affects the classification effect. FSVM assigns different weight coefficients to different samples. The purpose is to make each sample have a different effect on the optimal classification hyperplane.

Figure 1 shows the overview of our algorithm. The algorithm consists of three steps. The first step is to obtain the positive and negative minimum hyperspheres based on the training data and SVDD. By finding the minimum volume hypersphere containing a sample set, the target samples are included in the hypersphere as much as possible, and the non-target samples are excluded from the hypersphere. The second step is to calculate the distance between the samples and the hyperplanes. The sample position is determined by distance and radius difference. In the third step, different samples inside and outside the hyperspheres are represented by different functions. Our algorithm is described in detail below.

3.1. Analysis

The definition of fuzzy membership function is the key of FSVM algorithm. There are many definitions of fuzzy membership functions, but there are no general guidelines. Traditional fuzzy membership functions based on the distance between the sample and its cluster center are not effective to distinguish noises or outliers from SVs. Sometimes the distance is not the only criterion for judging whether it is normal. As shown in Figure 2, point A on the left, in Figure 2a, has a high probability of being a valid sample. Point A on the right, in Figure 2b, has a high probability of being a noise or outlier. It is not enough to just rely on linear functions of distance. The relative position relation between samples should be considered. That is, fuzzy membership function must consider the affinity between samples.

As shown in Figure 3, the distance between B and the cluster center is not the same as the distance between C and the cluster center. Since these two points have the same distance to the classification hyperplane, they contribute the same to the hyperplane. Compared with point B and point C, point D is further away from the cluster center, but closer to the classification hyperplane. Point D is the point that contributes the most to the hyperplane. So we have to take that into account when we design membership functions.

The optimal classification hyperplane and its nearby support vectors are far away from the cluster center. The closer the sample is to the cluster center, the greater the weight coefficient. The noises and SVs distributed on the boundary are far from the cluster center. They are all given less weight coefficients. This is quite different from the objective situation. SVs should be given greater weight coefficients.

The optimal hyperplane of standard SVM is determined by SVs. The geometric principle of SVM is to use two hyperplanes with maximum spacing to separate the training samples as far as possible in the original space or feature space. Therefore, using the hyperplane in class to replace the center can better approximate the actual situation.

The solution of the optimal classification hyperplane of SVM is usually converted to solving quadratic programming problem. However, the solving complexity of the quadratic programming problem will increase significantly with the sample increase. When the sample size is large, the traditional fuzzy support vector machine needs large memory to store and calculate the kernel function matrix. Therefore, by selecting the boundary vector containing SVs in advance, the number of training samples and the number of quadratic programming solutions can be reduced. This has practical significance for improving training speed and accuracy.

3.2. Support Vector Data Description

The task of one-class classification is to distinguish the target sample from non-target samples. The boundary has to be constructed by a hypersphere around the target samples. By finding the minimum volume hypersphere containing the sample set, the target samples are included in the hypersphere as much as possible, and the non-target samples are excluded from the hypersphere.

For ease of description, $\mathsf{\phi }$:${\mathit{R}}^{\mathit{n}}$ → H represents the mapping of the input space to the high-dimensional space. Assume T = {${\mathit{x}}_{\mathit{i}}$, i=1, 2, ⋯, l} contains l data objects, and the hypersphere is described by center a and radius R. The minimum hypersphere can be obtained by solving the following quadratic programming (1).

$$\begin{gathered} \underset{\mathit{R},\mathsf{\alpha },\mathsf{\xi }}{\min }{\mathit{R}}^2+C{\displaystyle \sum }_{i=1}^l{\xi }_i \\ s.t. ||\phi \left({\mathit{x}}_i\right)-{\mathit{a}||}^2\le {\mathit{R}}^2+{\xi }_i \\ {\xi }_i\ge 0, i=1,2,\dots ,l \end{gathered}$$

(1)

where ||$\cdot$|| is the Euclidean distance, ${\mathsf{\xi }}_{\mathit{i}}$ is slack variables, and C is the trade-off between the volume of the hypersphere and the errors. Lagrangian multipliers ${\mathsf{\alpha }}_{\mathit{i}}$ and ${\mathsf{\beta }}_{\mathit{i}}$ are introduced to construct a Lagrangian function. The Lagrangian function is constructed as shown in Equation (2).

$$L\left(R,\mathit{a},\xi ,\mathit{\alpha },\mathit{\beta }\right)={\mathrm{R}}^2+C{\displaystyle \sum }_{i=1}^l{\xi }_i-{\displaystyle \sum }_{i=1}^l{\alpha }_i\left[{\mathrm{R}}^2+{\xi }_i-||\phi \left({\mathit{x}}_i\right)-{\mathit{a}||}^2\right]-{\displaystyle \sum }_{i=1}^l{\beta }_i{\xi }_i$$

(2)

The center a and radius R can be obtained from the solution to the dual problem and the solution to KKT conditions. a and R are calculated according to Formulas (3) and (4) respectively.

$$\mathit{a}={\displaystyle \sum }_{i=1}^l{\alpha }_i^{*}\phi \left(x_i\right)$$

(3)

$${\mathit{R}}^2=||\phi \left({\mathit{x}}_j\right)-{\mathit{a}||}^2=K\left({\mathit{x}}_j,{\mathit{x}}_j\right)-2{\mathsf{\Sigma }}_{i=1}^l{\alpha }_i^{*}K\left({\mathit{x}}_i,{\mathit{x}}_j\right)+{\mathsf{\Sigma }}_{i=1}^l{\mathsf{\Sigma }}_{j=1}^l{\alpha }_i^{*}{\alpha }_j^{*}K\left({\mathit{x}}_i,{\mathit{x}}_j\right)$$

(4)

According to (5), if the Euclidean distance between the point $\mathsf{\phi }\left(\mathit{x}\right)$ and center a is less than radius R then it is normal. Conversely, it is the noise point when the Euclidean distance is greater than radius R.

$$||\phi \left(\mathit{x}\right)-{\mathit{a}||}^2=K\left(\mathit{x},\mathit{x}\right)-2{\mathsf{\Sigma }}_{i=1}^l{\alpha }_i^{*}K\left({\mathit{x}}_i,\mathit{x}\right)+{\mathsf{\Sigma }}_{i=1}^l{\mathsf{\Sigma }}_{j=1}^l{\alpha }_i^{*}{\alpha }_j^{*}K\left({\mathit{x}}_i,{\mathit{x}}_j\right)\le \mathit{R}$$

(5)

3.3. Design Fuzzy Membership Function

According to the basic principle of the support vector machine, the optimal classification hyperplane is determined by the support vectors. If each class of the two classification problems is considered as a convex set, then these support vectors lie on the relative boundary of the two convex sets far away from the center of the two classes.

Given T = {(${\mathit{x}}_{\mathit{i}}$,${\mathit{y}}_{\mathit{i}}$), i = 1, 2, ⋯, l} contains l data objects. If l⁺ and ${\mathit{l}}^{-}$ respectively represent the number of positive samples ${\mathit{x}}_{\mathit{i}}^{+}$, i = 1, 2, ⋯, l⁺ and the number of negative samples ${\mathit{x}}_{\mathit{i}}^{-}$, i = 1, 2, ⋯, ${\mathit{l}}^{-}$, then l⁺ + ${\mathit{l}}^{-}$= l.

If a⁺ and ${\mathit{a}}^{-}$ respectively represent the minimum hypersphere center of positive samples and the minimum hypersphere center of negative samples, R⁺ and ${\mathit{R}}^{-}$ respectively represent the minimum hypersphere radius of positive samples and minimum hypersphere radius of negative samples, then a⁺ and ${\mathit{a}}^{-}$ are always located in the geometric center, if the normal vector of the hyperplane is established by the maximum sum of the distance [69] from two centers to the hyperplane. As shown in Figure 4, in order to maximize the sum of the distances, it should satisfy d=||a⁺$-$A||+||${\mathit{a}}^{-}-$B||$\le$||a⁺$-$O||+||${\mathit{a}}^{-}-$O|| = ||a⁺$-{\mathit{a}}^{-}$||. That is, the distance is maximized d=||a⁺$-{\mathit{a}}^{-}$|| when hyperplane and vector a⁺$-{\mathit{a}}^{-}$ are perpendicular to each other.

For the given data, the relative positions of the two hyperspheres can be intersected, tangent and separated. The two separate cases and tangent case can be classified as one case. Although in the case of intersection, the radial basis kernel function can always choose parameters to make the two hyperspheres separated, this will cause an overfitting phenomenon. Therefore, our paper summarizes the above three cases as two cases of separation and intersection.

As shown in the Figure 5 and Figure 6, the normal vector to the hyperplane is w = a⁺$-{\mathit{a}}^{-}$ according to the principle of maximum distance. These two hyperplanes that go through a⁺ and ${\mathit{a}}^{-}$ with a normal vector of a normal vector of w are H⁺: ${\mathit{w}}^{\mathit{T}}$(x$-$a⁺) = 0 and H^-: ${\mathit{w}}^{\mathit{T}}$(x$-{\mathit{a}}^{-}$) = 0.

The optimal classification hyperplane is only determined by the SVs. Therefore, samples can be screened in advance, and those samples that may become support vectors can be selected to be trained as new training samples, which will simplify the computation of quadratic programming and improve the training speed. The optimal classification hyperplane lies between positive and negative hypersphere centers. Then, positive and negative samples between H⁺ and H⁻ can be selected as the new training sample set. As shown in the above two figures, the shaded parts inside the hyperspheres are the normal sample, in which the positive class is represented by orange slashes, while the negative class is represented by blue slashes. The samples with “+” and “*” in the new training sample set located outside the hypersphere represent noises or outliers respectively.

If ${\mathit{l}}_{\mathrm{new}}^{+}$ and ${\mathit{l}}_{\mathrm{new}}^{-}$ are the number of positive and negative samples in the new sample set, then the distance between the samples and the hyperplane in each class is calculated according to (6).

$$\left\{\begin{array}{c}D\left({\mathit{x}}_i^{+}\right)=\frac{\left|{\mathit{>w}}^T\left(\mathit{x}-{\mathit{a}}^{+}\right)\right|}{||\mathit{w}||}, i=1,2,\cdots ,l_{new}^{+}\\ D\left({\mathit{x}}_i^{-}\right)=\frac{\left|{\mathit{w}}^T\left(\mathit{x}-{\mathit{a}}^{-}\right)\right|}{||\mathit{w}||}, i=1,2,\cdots ,l_{new}^{-}\end{array}\right.$$

(6)

where ${\mathit{w}}^{\mathit{T}}$(x$-$a⁺) = 0 and H⁻: ${\mathit{w}}^{\mathit{T}}$(x$-{\mathit{a}}^{-}$) = 0 are the two hyperplanes that go through a⁺ and ${\mathit{a}}^{-}$; w = a⁺$-{\mathit{a}}^{-}$ is the normal vector to the hyperplane.

The distance between the samples and the center of the hypersphere is calculated according to (7).

$$\left\{\begin{array}{c}d\left({\mathit{x}}_i^{+}\right)=||{\mathit{x}}_i^{+}-{\mathit{a}}^{+}||, i=1,2,\cdots ,l_{new}^{+}\\ d\left({\mathit{x}}_i^{-}\right)=||{\mathit{x}}_i^{-}-{\mathit{a}}^{-}||, i=1,2,\cdots ,l_{new}^{-}\end{array}\right.$$

(7)

Therefore, membership functions of both positive and negative sample points are constructed according to (8) and (9).

$$s_i^{+}=\left\{\begin{array}{lll}0.6*\frac{D\left({\mathit{x}}_i^{+}\right)}{d\left({\mathit{x}}_i^{+}\right)}*\frac{d\left({\mathit{x}}_i^{+}\right)}{R^{+}}+0.4,& d\left({\mathit{x}}_i^{+}\right)\le R^{+},& i=1,2,\cdots ,l_{new}^{+}\\ 0.4*{\left[\frac{1}{\frac{d\left({\mathit{x}}_i^{+}\right)}{R^{+}}+\frac{D\left({\mathit{x}}_i^{+}\right)}{d\left({\mathit{x}}_i^{+}\right)}}\right]}^P,& d\left({\mathit{x}}_i^{-}\right)>R^{-},& i=1,2,\cdots ,l_{new}^{+}\end{array}\right.$$

(8)

$$s_i^{-}=\left\{\begin{array}{lll}0.6*\frac{D\left({\mathit{x}}_i^{-}\right)}{d\left({\mathit{x}}_i^{-}\right)}*\frac{d\left({\mathit{x}}_i^{-}\right)}{R^{-}}+0.4,& d\left({\mathit{x}}_i^{-}\right)\le R^{-},& i=1,2,\cdots ,l_{new}^{-}\\ 0.4*{\left[\frac{1}{\frac{d\left({\mathit{x}}_i^{-}\right)}{R^{-}}+\frac{D\left({\mathit{x}}_i^{-}\right)}{d\left({\mathit{x}}_i^{-}\right)}}\right]}^P,& d\left({\mathit{x}}_i^{-}\right)>R^{-},& i=1,2,\cdots ,l_{new}^{-}\end{array}\right.$$

(9)

The minimum value of the membership function inside the hypersphere is 0.4, and the maximum value of the membership function outside the hypersphere is 0.4. The membership value of the sample increases with the value of the distance between the sample and hyperplane. Given $\mathit{p}\ge$2, the bigger $p$, the faster ${\mathit{s}}_{\mathit{i}}^{+}$ and ${\mathit{s}}_{\mathit{i}}^{-}$ decay. Similarly, for nonlinear cases, mapping function $\mathsf{\phi }(\mathit{x})$ is introduced by kernel function K(${\mathit{x}}_{\mathit{i}}$,${\mathit{x}}_{\mathit{j}}$) to map data to a high-dimensional space. The normal vector is w = a⁺$-{\mathit{a}}^{-}$ by a rule for the maximum sum of distance from two hyperspheres’ centers to the separation hyperplane. The hyperplanes of the two classes with w as the normal vector and going through a⁺ and ${\mathit{a}}^{-}$ respectively are H⁺: ${\mathit{w}}^{\mathit{T}}$($\mathsf{\phi }(\mathit{x}) -$ a⁺) = 0 and H⁻: ${\mathit{w}}^{\mathit{T}}$($\mathsf{\phi }(\mathit{x})-{\mathit{a}}^{-}$) = 0. If ${\mathit{l}}_{\mathrm{new}}^{+}$ and ${\mathit{l}}_{\mathrm{new}}^{-}$ are the number of positive and negative samples in the new sample set, then the distance between the samples and the hyperplane in each class is calculated according to (10).

$$\left\{\begin{array}{c}D\left(\phi \left({\mathit{x}}_i^{+}\right)\right)=\frac{\left|{\mathit{w}}^T\left(\phi \left(\mathit{x}\right)-{\mathit{a}}^{+}\right)\right|}{||\mathit{w}||},\hspace{1em}\phi \left(\mathit{x}\right)=\phi \left({\mathit{x}}_i^{+}\right), i=1,2,\cdots ,l_{new}^{+}\\ D\left(\phi \left({\mathit{x}}_i^{-}\right)\right)=\frac{\left|{\mathit{w}}^T\left(\phi \left(\mathit{x}\right)-{\mathit{a}}^{-}\right)\right|}{||\mathit{w}||},\hspace{1em}\phi \left(\mathit{x}\right)=\phi \left({\mathit{x}}_i^{-}\right), i=1,2,\cdots ,l_{new}^{-}\end{array}\right.$$

(10)

The distance between the sample and its center of the hypersphere in each class is calculated according to (11).

$$\left\{\begin{array}{c}d\left(\phi \left({\mathit{x}}_i^{+}\right)\right)=||\phi \left({\mathit{x}}_i^{+}\right)-{\mathit{a}}^{+}||, i=1,2,\cdots ,l_{new}^{+}\\ d\left(\phi \left({\mathit{x}}_i^{-}\right)\right)=||\phi \left({\mathit{x}}_i^{-}\right)-{\mathit{a}}^{-}||, i=1,2,\cdots ,l_{new}^{-}\end{array}\right.$$

(11)

Therefore, membership functions of both positive and negative sample points are constructed according to (12) and (13).

$$s_i^{+}=\left\{\begin{array}{ll}0.6*\frac{D\left(\phi \left({\mathit{x}}_i^{+}\right)\right)}{d\left(\phi \left({\mathit{x}}_i^{+}\right)\right)}*\frac{d\left(\phi \left({\mathit{x}}_i^{+}\right)\right)}{R^{+}}+0.4,& d\left(\phi \left({\mathit{x}}_i^{+}\right)\right)\le R^{+}, i=1,2,\cdots ,l_{new}^{+}\\ 0.4*{\left[\frac{1}{\frac{d\left(\phi \left({\mathit{x}}_i^{+}\right)\right)}{R^{+}}+\frac{D\left(\phi \left({\mathit{x}}_i^{+}\right)\right)}{d\left(\phi \left({\mathit{x}}_i^{+}\right)\right)}}\right]}^P,& d\left(\phi \left({\mathit{x}}_i^{+}\right)\right)>R^{-}, i=1,2,\cdots ,l_{new}^{+}\end{array}\right.$$

(12)

$$s_i^{-}=\left\{\begin{array}{ll}0.6*\frac{D\left(\phi \left({\mathit{x}}_i^{-}\right)\right)}{d\left(\phi \left({\mathit{x}}_i^{-}\right)\right)}*\frac{d\left(\phi \left({\mathit{x}}_i^{-}\right)\right)}{R^{-}}+0.4,& d\left(\phi \left({\mathit{x}}_i^{-}\right)\right)\le R^{-}, i=1,2,\cdots ,l_{new}^{-}\\ 0.4*{\left[\frac{1}{\frac{d\left(\phi \left({\mathit{x}}_i^{-}\right)\right)}{R^{-}}+\frac{D\left(\phi \left({\mathit{x}}_i^{-}\right)\right)}{d\left(\phi \left({\mathit{x}}_i^{-}\right)\right)}}\right]}^P,& d\left(\phi \left({\mathit{x}}_i^{-}\right)\right)>R^{-}, i=1,2,\cdots ,l_{new}^{-}\end{array}\right.$$

(13)

4. Experimental Evaluation

This section presents the evaluation of our method in terms of detection performance, overhead and impaction of parameters. In order to test the performance of our algorithm, in addition to comparing the SVM-based algorithms SVM, FSVM1 [66], FSVM2 [65], and FSVM3 [68], the proposed algorithm is also compared with other algorithms in this section.

4.1. The Experimental Data

In order to facilitate an experimental performance comparison with similar studies, the system call datasets UNM_sendmail and UNM_live_lpr published by the New Mexico university are used in this section [70]. The UNM_sendmail data set consists of 346 normal traces and 25 abnormal traces. The abnormal data contains sunsendmailcp (sccp) intrusions and decode intrusions, in which the sccp intrusions enable the local user to obtain the root access by using special command options to make sendmail attach an e-mail message to a file, and the decode intrusions enable remote users to make changes to certain files on the local system. Data for UNM_live_lpr data set includes 15 months of activity and consists of 4298 normal tracks and 1003 abnormal tracks. The abnormal data contains lprcp symbolic link intrusions that take advantage of the vulnerability of the lpr program to control the files on the host computer and tamper with the contents of the files.

The trace consists of a sequence of system calls in chronological order. The meaning of trace varies from program to program. Each trace file lists pairs of numbers, the first number represents the process identity (PID) of the execution process, and the second number represents the specific system call (SC). The child processes forked by the parent process are traced individually. We take UNM_sendmail as an example to show the specific format of experimental data as follows:

$$\begin{array}{l}88404,88402,88405,884066,88405,\cdots ,88406\\ 8843115,884315,884399,8843120,884317,\cdots ,884312,\\ 65452,65455,\cdots ,65452,65452,65452\end{array}$$

The data consists of different data units. Each data unit consists of PID and SC. In the experiment, data is segmented according to the PID number and the SCs after the same PID are arranged together in chronological order. In the above data, the numbers 8840, 8843 and 6545 are PIDs and the number 4, 2, 5, 66, …, 2 are SCs. The lists of system calls issued by 8840, 8843 and 6545 are denoted as R₈₈₄₀ = (4, 2, 5, 66, 5, …, 6), R₈₈₄₃ = (115, 15, 99, 120, …, 17) and R₆₅₄₅ = (2, 55, …, 2) respectively.

The vector form of a system call sequence composed of the frequency of the short system call sequences. Given n is the number of traces, m is the total number of short system call sequences. Then the ith element in the vector is the frequency at which the short system sequence numbered i occurs in the system call sequence of the process. The vector corresponding to the jth trace, represented by the frequency of short system call sequence, is denoted by (f_1j,⋯, f_mj). That is to say, samples are represented as (X_j, y_j) j = 1,⋯, n, where X_j = (f_1j,⋯, f_mj), y_j $\in$ (+1, −1).

4.2. The Experimental Performance

The experimental data of UNM_live_lpr process were composed of 4298 normal tracks and 1003 abnormal tracks. The 1003 anomaly traces contain LPRCP attacks that control and tamper with host files. The experimental data for the UNM_sendmail process consisted of 346 normal traces and 25 abnormal traces, including 20 SCCPS that used E-mail to obtain root directory information and 5 decode attacks that remotely modified local files.

False alarm is to judge the normal behavior of the program as abnormal behavior. If ${\mathit{L}}_{\mathit{N}}$ normal traces participate in the evaluation, and ${\mathit{N}}_{\mathit{FAR}}$ normal traces are misjudged as abnormal traces, then the false alarm rate is equal to ${\mathit{N}}_{\mathit{FAR}}/{\mathit{L}}_{\mathit{N}}$. Detection rate refers to the proportion of detected abnormal traces in the total number of abnormal traces. If ${\mathit{L}}_{\mathit{AN}}$ abnormal traces participate in the test and ${\mathrm{N}}_{\mathrm{HR}}$ were detected, then the detection rate is ${\mathit{N}}_{\mathit{HR}}/{\mathit{L}}_{\mathit{AN}}$. The missing rate indicates the proportion of unrecognized abnormal traces in the total number of abnormal traces. If ${\mathit{L}}_{\mathit{AN}}$ abnormal traces participate in the evaluation and ${\mathit{N}}_{\mathit{M}}$ cannot be detected, then the missing rate is ${\mathit{N}}_{\mathit{M}}/{\mathit{L}}_{\mathit{AN}}$. DR =${\mathsf{\rho }}_{\mathit{HR}}*$(1−${\mathsf{\rho }}_{\mathit{FAR}}$) is used as the comprehensive detection formula, in which ${\mathsf{\rho }}_{\mathit{HR}}$ and ${\mathsf{\rho }}_{\mathit{FAR}}$ stands for detection rate and missing rate respectively.

Gauss kernel k (x, y) = exp ($-$|| x$-$y ||²/2$\sigma$²) is used in all algorithms during training. ${\mathsf{\sigma }}_1$ and ${\mathsf{\sigma }}_2$ can be selected according to the maximum error rate allowed in the target set. ${\mathit{C}}_1$ and ${\mathit{C}}_2$ can be adjusted according to the equality of upper bound of positive and negative error rate. That is, 1/${\mathit{l}}_1{\mathit{C}}_1$ = 1/${\mathit{l}}_2{\mathit{C}}_2$, 1/${\mathit{l}}_1$≤${\mathit{C}}_1$≤ 1, 1/${\mathit{l}}_2$≤${\mathit{C}}_2$≤ 1. Grid search method is adopted to select the optimal parameters. The search range of parameter C is {$2^{-24}$,$2^{-23}$,⋯,$2^{23}$, $2^{24}$}. The parameter $\mathsf{\sigma }$ and $\mathsf{\beta }$ both have a search range of {$2^{-24}$,$2^{-23}$,⋯,$2^{23}$, $2^{24}$}. The step length and short sequence length are set to 1 and 6 respectively.

Table 1. Performance comparison on data set UNM_live_lp.

Algorithm	Detection Rate (%)	Missing Rate (%)	False Alarm Rate (%)
SVM	79.88%	20.12%	8.64%
FSVM1	83.71%	16.29%	7.30%
FSVM2	85.56%	14.44%	8.17%
FSVM3	86.20%	13.80%	6.92%
Our algorithm	92.63%	7.37%	6.16%

summarizes the comparison results of detection performance between our method and the other eight methods on UNM_live_lp. The detection rate of SVM, FSVM1, FSVM2, FSVM3, and our algorithm are 61.53%,76.92%, 76.92%, 83.21%, and 84.61% respectively. The missing rates of SVM, FSVM1, FSVM2, FSVM3, and our algorithm are 38.47%, 23.08%, 23.08%, 16.23%, and 15.39% respectively. The false alarm rates of SVM, FSVM1, FSVM2, FSVM3, and our algorithm are 10.14%, 9.57%, 8.17%, 6.92%, and 4.51% respectively. In all the algorithms, our algorithm has the highest detection rate and the lowest false alarm rate and missing rate.

As shown in Table 1, SVM has the lowest detection rate, 79.88%, among the five algorithms. Due to the construction of a fuzzy membership function, the other four algorithms treat the different contributions of different samples in the construction of the objective function differently, which makes their average detection rate reach 87.03%. FSVM1 algorithm designs a fuzzy membership function based on the linear function of the distance between the sample and its cluster center. The larger the distance is, the smaller the coefficients is. However, this membership design method is sometimes unable to distinguish abnormal points effectively. The FSVM2 algorithm not only considers the distance in the FSVM1 algorithm, but also considers the position relation between samples. FSVM2 algorithm determines the membership function according to the position of the sample and hypersphere. When the samples are located inside the hypersphere, the membership function is defined by the linear function of the distance. The coefficient of the sample decreases with the increase of the distance. When the samples are located outside the hypersphere, these samples are regarded as abnormal samples. The membership function is represented by different functions. The FSVM3 algorithm, like the FSVM2 algorithm, takes account of the distance between the sample and its cluster center and the affinity between samples. The difference between the FSVM3 and FSVM2 is that the FSVM3 algorithm is based on the SVDD algorithm by introducing two different parameters to control the affinity between positive and negative samples.

The method treats all samples equally during the training process, which makes the contribution of samples to constructing the optimal classification hyperplane equal. As a result, when training samples contain abnormal samples, the classification hyperplane obtained is not the optimal hyperplane. Our algorithm abides by the maximum sum of distance from the hypersphere to hyperplane, replaces the cluster center with the hyperplane inside the class, and designs the membership function according to the distance from the sample to the hyperplane inside the hypersphere. Therefore, it can be seen from the table that with the continuous improvement of the membership function, the detection rate of the algorithm increases successively. The detection rate changed from 83.71% of FSVM1 algorithm to 85.56% of FSVM2 algorithm, then to 86.20% of WCS-FSVM algorithm, and finally reached 92.63% of our algorithm.

The formula of comprehensive detection rate is closer to reality. The comprehensive detection rates of SVM, FSVM1, FSVM2, and FSVM3 are 72.97%, 77.59%, 78.56%, and 80.23% respectively. As show in Figure 7, our algorithm has the highest comprehensive detection rate, 86.92%, among five algorithms.

At the same time, it is also evident from the data in

Table 2. Performance comparison on data set UNM_sendmail.

Algorithm	Detection Rate (%)	Missing Rate (%)	False Alarm Rate (%)
SVM	61.53%	38.47%	10.14%
FSVM1	76.92%	23.08%	9.57%
FSVM2	76.92%	23.08%	8.17%
FSVM3	83.21%	16.23%	6.92%
Our algorithm	84.61%	15.39%	4.51%

that the detection rate, false alarm rate and missing rate of UNM_sendmail data are all lower than that of UNM_live_lpr data. The reason for this is the amount of data. The UNM_sendmail process data consists of 346 normal tracks and 25 abnormal tracks, which is less experimental data than the 4298 normal tracks and 1003 abnormal tracks of the UNM_live_lpr process. Therefore, there is sufficient data for UNM_live_lpr process training, which is conducive to pre-extracting relative boundary vectors containing enough support vectors and ensuring sufficient data parameters. In the experiments of UNM_live_lpr process and UNM_sendmail process, SVM uses the same error penalty factor for all samples. It will have a negative impact on the classification results when the SVM algorithm classifies unbalanced samples. This is the reason that the SVM always has the lowest detection rate among all algorithms. FSVM1 algorithm, FSVM2 algorithm, FSVM3 algorithm, and our algorithm are equivalent to using a penalty factor for each sample to be treated differently. When these algorithms are used to classify the quantity imbalance samples, good classification results can be obtained. In other words, membership function can be used to describe error penalty on samples. This is consistent with the conclusion that fuzzy support vector machine is equivalent to a multi-penalty support vector machine.

The comprehensive detection rates of SVM algorithm, FSVM1 algorithm, FSVM2 algorithm, FSVM3 algorithm, and our algorithm on UNM sendmail process data were 55.29%, 69.58%, 70.63%, 77.45%, and 80.79%, respectively. As shown in Figure 8, among the five detection algorithms, SVM algorithm has the lowest comprehensive detection rate and the weakest detection performance. The algorithm proposed in this paper has the highest comprehensive detection rate and the strongest detection performance. The comprehensive detection performance of FSVM1 algorithm, FSVM2 algorithm and FSVM3 algorithm also varies with the fuzzy membership function, but the overall trend is consistent with the test results obtained from UNM_live_lpr process data, that is, the detection performance is improved with the improvement of fuzzy membership function.

As shown in

Table 3. Detection performance of different k on UNM_live_lpr data.

Our Method	k = 3	k = 4	k = 5	k = 6	k = 7	k = 8
Detection rate	78.30%	83.25%	86.31%	92.63%	88.26%	83.27%
Missing rate	21.70%	16.75%	13.69%	7.37%	11.74%	16.73%
False alarm rate	11.76%	11.32%	10.60%	6.16%	6.30%	7.56%

and

Table 4. Detection performance of different k on UNM_sendmail data.

Our Method	k = 3	k = 4	k = 5	k = 6	k = 7	k = 8
Detection rate	77.22%	82.45%	85.37%	84.61%	86.46%	81.37%
Missing rate	22.78%	17.55%	14.63%	15.39%	13.54%	18.63%
False alarm rate	14.67%	12.12%	11.68%	4.51%	6.85%	9.53%

, the length of the system call short sequence k has a direct impact on the detection performance. When k increases from 3 to 5, the detection rate, missing alarm rate and false alarm rate are increasing. When k is 6, our algorithm obtains the highest detection rate, the lowest false alarm rate and the lowest missing rate. When k is greater than 6, the detection rate of our algorithm starts to decrease, and the missing rate and false alarm rate start to increase. On UNM_sendmail data, the detection rate, false alarm rate, and missing alarm rate of PVFSVM algorithm have the same trend as on UNM_live_lpr data.

As shown in Figure 9 and Figure 10, if k = 6, the comprehensive detection rates of our algorithm on UNM_live_lpr and UNM_sendmail are 86.92% and 80.79% respectively, which are higher than the comprehensive detection rate corresponding to other k values on UNM_live_lpr and UNM_sendmail. When k increases from 3 to 6, the comprehensive detection rate also increases gradually. When k increases from 6 to 8, the comprehensive detection rate decreases gradually.

This is consistent with professor Forrest’s conclusion on the selection of short sequence length, and professor Wenke Lee’s conclusion from the perspective of information theory that the best system call short sequence length is 6 or 7. Compared with the value of k of 6, when the value of short sequence is too small, the timing relation between short sequence patterns is lost. When the short sequence becomes longer, the short sequence pattern loses local information, no matter what kind of information loss will make the comprehensive detection performance worse. Therefore, the short sequence length can only be chosen as 6, which makes the detection performance of our algorithm reach the optimal level.

Table 5. Comparison between different algorithms on UNM_live_lp.

Algorithm	Detection Rate (%)	Missing Rate (%)	False Alarm Rate (%)
N.bayes	65.11%	30.12%	4.77%
Uniqueness	39.1%	55.3%	2.30%
Hybrid Markov	45.6%	40.22%	14.26%
Bayes1-step Markov	62.6%	33.3%	4.1%
IPMA	45.05%	47.37%	7.58%
Sequence matching	36.7%	58.2%	5.91%
Compression	33.9%	50.1%	16%
Closeness	76.5%	19.2%	4.3%
Our algorithm	92.63%	7.37%	6.16%

summarizes the comparison results of detection performance between our method and the other eight methods on UNM_live_lp. The detection rate of N.bayes, Uniqueness, Hybrid Markov, Bayes1-step Markov, IPMA, Sequence matching, Compression, Closeness, and our algorithm are 65.11%, 39.1%, 45.6%, 62.6%, 45.05%, 36.7%, 33.9%, 76.5%, and 92.63% respectively. The missing rates of N.bayes, Uniqueness, Hybrid Markov, Bayes1-step Markov, IPMA, Sequence matching, Compression, Closeness, and our algorithm are 30.12%, 55.3%, 40.22%, 33.3%, 47.37%, 58.2%, 50.1%, 19.2%, and 7.37% respectively. The false alarm rates of N.bayes, Uniqueness, Hybrid Markov, Bayes1-step Markov, IPMA, Sequence matching, Compression, Closeness, and our algorithm are 4.77%, 2.30%, 14.26%, 4.1%, 7.58%, 5.91%, 16%, 4.3%, and 6.16% respectively.

These two methods, Compression method and Sequence Matching method, respectively consider the detection problem from the aspect of data reversibility and similarity matching degree, and fail to consider the sequence characteristics between system calls. Therefore, they are the two algorithms with the worst comprehensive detection performance. The IPMA method and Hybrid Markov method are complicated due to investigating the transition characteristics of system calls one by one. Although the ${\mathsf{\rho }}_{\mathit{FAR}}$ of IPMA method and Hybrid Markov method are 7.58% and 14.26%, the ${\mathsf{\rho }}_{\mathit{MR}}$ is 47.37% and 40.22%. The ${\mathsf{\rho }}_{\mathit{MR}}$ of Bayes 1-step Markov method is 33.3% and the ${\mathsf{\rho }}_{\mathit{FAR}}$ is 4.1%. The comprehensive test performance evaluation formula has a high value, so the test performance is good. Since this method looks the frequency of rare system calls, the ${\mathsf{\rho }}_{\mathit{MR}}$ of Uniqueness method is 55.3% and the ${\mathsf{\rho }}_{\mathit{FAR}}$ is 2.3%. However, this requires statistics for all the system calls that are used. N.bayes belongs to Naive Bayesian method in essence, and it has good noise tolerance and fast calculation, but the false alarm rate is too high. Therefore, the comprehensive detection performance is good. The closeness method extracts user behavior patterns from the perspective of combination. It shows good detection performance under different closeness thresholds, but it ignores the characteristics of attack intensity. That is, the attacker will complete the attack task in the shortest possible time and try to behave normally the rest of the time. As show in Figure 11, our algorithm has the highest comprehensive detection rate 86.92% among five algorithms. In all the algorithms, our algorithm has the highest detection rate, and the lowest false alarm rate and missing rate. The formula of the comprehensive detection rate is closer to reality. The comprehensive detection rate of N.bayes, Uniqueness, Hybrid Markov, Bayes1-step Markov, IPMA, Sequence matching, Compression, Closeness, and our algorithm are 62%, 36.91%, 39.13%, 60.03%, 41.63%, 34.82%, 28.47%, 73.22%, and 86.92% respectively.

Table 6. Comparison between different algorithms on UNM_ sendmail.

Algorithm	Detection Rate (%)	Missing Rate (%)	False Alarm Rate (%)
N.bayes	63.88%	32.12%	4.00%
Uniqueness	37.4%	60.3%	2.30%
Hybrid Markov	45.3%	40.44%	14.26%
Bayes1-step Markov	67.3%	30.8%	1.9%
IPMA	40.63%	42.37%	17%
Sequence matching	35.9%	60.2%	3.9%
Compression	29.7%	50.5%	19.8%
Closeness	75.2%	20.1%	4.7%
Our algorithm	84.61%	15.39%	4.51%

summarizes the comparison results of detection performance between our method and the other eight methods on UNM_sendmail. The detection rates of N.bayes, Uniqueness, Hybrid Markov, Bayes1-step Markov, IPMA, Sequence matching, Compression, Closeness, and our algorithm are 63.88%, 37.4%, 45.3%, 67.3%, 40.63%, 35.9%, 29.7%, 75.2%, and 84.61% respectively. The missing rates of N.bayes, Uniqueness, Hybrid Markov, Bayes1-step Markov, IPMA, Sequence matching, Compression, Closeness, and our algorithm are 32.12%, 60.3%, 40.44%, 30.8%, 42.37%, 60.2%, 50.5%, 20.1%, and 15.39% respectively. The false alarm rates of N.bayes, Uniqueness, Hybrid Markov, Bayes1-step Markov, IPMA, Sequence matching, Compression, Closeness, and our algorithm are 4%, 2.30%, 14.26%, 1.9%, 17%, 3.9%, 19.8%, 4.7%, and 4.51% respectively.

Similarly, the Compression method and Sequence Matching method pay attention to reversibility and similarity matching degree, respectively, and fail to consider the sequence characteristic between system calls. The IPMA method and Hybrid Markov method are complicated due to investigating transition characteristics of system calls one by one. Although the ${\mathsf{\rho }}_{\mathit{FAR}}$ of IPMA method and Hybrid Markov method is 17% and 14.26%, the ${\mathsf{\rho }}_{\mathit{MR}}$ is 42.37% and 40.44%. The ${\mathsf{\rho }}_{\mathit{MR}}$ of Bayes 1-step Markov method is 30.8%, and the ${\mathsf{\rho }}_{\mathit{FAR}}$ is 1.9%. The comprehensive test performance evaluation formula has a high value, so the test performance is good. Since this method looks at the frequency of rare system calls, the ${\mathsf{\rho }}_{\mathit{MR}}$ of Uniqueness method is 60.3%, and the ${\mathsf{\rho }}_{\mathit{FAR}}$ is 2.3%. However, this requires statistics for all the system calls that are used. N.bayes belongs to Naive Bayesian method in essence, and it has good noise tolerance and fast calculation, but the false alarm rate is too high. Therefore, the comprehensive detection performance is good. The closeness method extracts user behavior patterns from the perspective of combination. It shows good detection performance under specific closeness thresholds, but it ignores the characteristics of attack intensity. That is, the attacker will complete the attack task in the shortest possible time and try to behave normally the rest of the time. As shown in Figure 12, our algorithm has the highest comprehensive detection rate 86.92% among five algorithms. In all the algorithms, our algorithm has the highest detection rate, and the lowest false alarm rate and missing rate. The formula of comprehensive detection rate is closer to reality. The comprehensive detection rates of N.bayes, Uniqueness, Hybrid Markov, Bayes1-step Markov, IPMA, Sequence matching, Compression, Closeness, and our algorithm are 61.32%, 36.54%, 38.84%, 60.02%, 33.72%, 34.49%, 23.81%, 71.67%, and 80.79% respectively.

5. Conclusions

In order to solve the defects of FSVM. This paper presents a new FSVM algorithm based on SVDD. In our method, the noises and outliers are identified by a hypersphere with minimum volume while containing the maximum of the samples. The definition of fuzzy membership considered not only the position of samples inside the hypersphere, but also the distance between the hyperplane and samples. For the samples inside the hypersphere, the fuzzy membership function is a linear function of the distance. The greater the distance is, the greater the coefficient is. For the samples outside the hypersphere, the fuzzy membership function is an exponential function of the distance. The greater the distance is, the smaller the coefficient is. Compared with the FSVM based on the relation between the sample and its cluster center, our algorithm effectively distinguishes the noises or outliers from samples. The experiments show that our FSVM is more robust than the SVM and FSVM based on the distance between the sample and its cluster center. Our algorithm is suitable for the data of system call sequence and can contribute to accurate prediction.