Abnormal Monitoring Data Detection Based on Matrix Manipulation and the Cuckoo Search Algorithm

Abstract

Structural health monitoring is an effective method to evaluate the safety status of dams. Measurement error is an important factor which affects the accuracy of monitoring data modeling. Processing the abnormal monitoring data before data analysis is a necessary step to ensure the reliability of the analysis. In this paper, we proposed a method to process the abnormal dam displacement monitoring data on the basis of matrix manipulation and Cuckoo Search algorithm. We first generate a scatter plot of the monitoring data and exported the matrix of the image. The scatter plot of monitoring data includes isolate outliers, clusters of outliers, and clusters of normal points. The gray scales of isolated outliers are reduced using Gaussian blur. Then, the isolated outliers are eliminated using Ostu binarization. We then use the Cuckoo Search algorithm to distinguish the clusters of outliers and clusters of normal points to identify the process line. To evaluate the performance of the proposed data processing method, we also fitted the data processed by the proposed method and by the commonly used 3-$\sigma$ method using a regression model, respectively. Results indicate that the proposed method has a better performance in abnormal detection compared with the 3-$\sigma$ method.

1. Introduction

Due to environmental and ageing effects, structural behaviours and material properties of dams often have changes compared to initial or designed values after running for years, which may affect the heath status of dams [1,2]. In practical engineering, to evaluate the real time running status of dams, engineers commonly install a series of monitoring devices inside the dam and monitor dam’s structural parameters, such as displacement, seepage, and rotation. Once the deformation of the dam exceeds the safety value, the risk of the dam break problem would be very high when the reservoir is operated with high water level. Thus, analyzing the deformation status of the dam is important for evaluating the running status of the dam. Interpolating the displacement data of all monitoring point can provide the deformation field of the dam body. Therefore, displacement is the most crucial parameter [3,4]. Displacement monitoring data modeling is consider as one of the most effective methods to assess a dam’s health status.

Previous researchers developed a numbers of displacement prediction models using monitoring data [5,6,7]. In addition to traditional statistical models, machine learning algorithms such as the neural network method [8], support vector machine method [9,10], and extreme learning machine method [11] were applied to displacement monitoring data modeling in recent years. Most previous studies put emphasis on improving the accuracy of displacement prediction models. As research progressed, the precision of these models has been fairly high.

The reliability of monitoring data modeling not only relies on the performance of prediction model but also depends on the quality of monitoring data [12,13,14]. However, measurement errors of monitoring device are unavoidable in practical engineering due to technical problems such as false reading [15]. Therefore, detecting the abnormal data of displacement monitoring data is of great importance for improving the reliability of displacement prediction models.

Previous studies have proposed different methods to detect outliers in a dataset [14,16,17,18,19]. In early studies, criterion-based methods such as Pauta criterion, Chauvenet criterion, and Grubbs criterion were used to detect outliers [20,21,22]. Each criterion has different usage conditions. Grubbs criterion is applicable to a dataset with few data, whereas Pauta criterion is applicable for dataset with more data. In recent years, statistical theories such as 3$\sigma$ criterion have been commonly used to detect abnormal values in monitoring datasets. To increase the rate of outliers being detected, many studies have been conducted to enhance traditional methods [23,24,25]. For example, Zhao et al. improved the 3$\sigma$ criterion using the minimum covariance determinant [26]. Song et al. developed an detection method based on the multi-variable panel data model and K-means clustering method [27]. Zhang et al. provided a multi-source information fusion model for outlier detection [28].

These methods exhibited fairly good performance in identifying gross errors. One disadvantage of these methods is that they are mostly computationally complex. Moreover, the outlier detection often depends on time variation tendencies without considering the fluctuations of environmental factors, such as water level and external temperature [29]. In addition, the performance of outliers detection is affected by the setting of threshold, which relies on artificial selection, which may lead to missing judgment and misjudgment problems.

To overcome the shortcomings of these methods, we proposed a outlier detection method which combines matrix manipulation and the Cuckoo Search algorithm to deal with the abnormal dam displacement monitoring data. The principle of the proposed method is that the process line of monitoring data should be continuous while the outliers deviates from the process line [30]. We first generated the scatter plot of the original monitoring dataset, which includes clusters of isolated outliers, normal points, and clusters of outliers. The objective of the proposed method is to identify isolated outliers and clusters of outliers from the scatter plots. For the matrix manipulation method, Gaussian blur and Ostu binarization are used to detect isolated outliers [31,32,33]. We then applied the Cuckoo Search algorithm, which imitates the habit of brood parasitism, to distinguish clusters of normal points and clusters of outliers [34]. To ensure the efficiency of outliers detection, we implement the process of matrix detection and CS algorithm cyclic until the detection results converges. We also compared the abnormal detection performance of the proposed method with the commonly used 3$\sigma$ method.

This paper is organized as follows. Section 2 presents the principles of the proposed method, which combines matrix manipulation and the Cuckoo Search algorithm. Section 3 introduces the dataset. The displacement monitoring data of the dam at Jinping-I hydropower station is selected as the dataset. The detection results of the proposed method are presented in Section 4. Comparisons of the proposed model with 3$\sigma$ criterion are also discussed. Concluding remarks complete the paper in Section 5.

2. Data Processing Method Using Matrix Manipulation and the Cuckoo Search Algorithm

This section presents the mathematical details of the abnormal data processing method on the basis of matrix manipulation and Cuckoo Search algorithm. We first generate a scatter plot of the monitoring data. Once the scatter plot has been drawn, we then consider the scatter plot as an image and export the matrix of the image. Then, the matrix can be pre-processed using Gaussian blur and Ostu binarization. The gray scales of isolated outliers are reduced using Gaussian blur. Then, the isolated outliers are eliminated using Ostu binarization. We then use Cuckoo Search algorithm to distinguish the clusters of outliers and clusters of normal points, so as to identify the process line. Figure 1 shows the flowchart of the proposed abnormal data processing method.

2.1. Data Pre-Processing Using Gaussian Blur and Ostu Binarization

The Principe of the pre-processing method is to consider the plotted data sequence as an image (i.e., matrix), and identify the outliers in the plot using filters (Gaussian blur and Ostu binarization). Figure 2 shows the linear plot and scatter plot of an example data sequence. It can see from the figure that outliers in scatter plot is easier to be separated from the process line as compared with the linear plot. Thus, the first step of data pre-processing is to generate the scatter plot of the monitoring data sequence. Once the scatter plot has been drawn, we then consider the scatter plot as an image and export the matrix of the image. Then, noises can be reduced using various image-processing techniques, such as Gaussian blur and binary threshold. Scatter plot of monitoring data include isolate outliers, clusters of outliers, and clusters of normal points. At the pre-processing stage, most of the isolate outliers in the matrix can be detected and eliminated using Gaussian blur and Ostu binarization.

The Gaussian blur feature is obtained by smoothing an image using a Gaussian function to reduce the noise level. It can be considered as a nonuniform low-pass filter that preserves low spatial frequency and reduces image noise and negligible details in an image. From a mathematical perspective, the Gaussian blur process is the convolution of a matrix with a normal distribution. Convolving an image with a circular box blur will generate a more precise out-of-focus rendering effect. Since the Fourier transform of a Gaussian function is another Gaussian function, Gaussian blur is a low-pass filter for images. It is typically achieved by convolving an image with a Gaussian kernel. The Gaussian kernel filtering function $G\left(x,y\right)$ follows a two-dimension Gaussian distribution:

$$G\left(x,y\right)=\frac{1}{2\pi {\sigma }^2}{e}^{-\frac{x^2+y^2}{2{\sigma }^2}}$$

(1)

where $\sigma$ is the standard deviation of the Gaussian distribution. It controls the variance around a mean value of the Gaussian distribution, which determines the extent of the blurring effect around a pixel. With an increase in $\sigma$, the high-frequency information content reduces around the pixel. $x,y$ are the coordinates of neighbor pixels.

The Gaussian weighted matrix $W_i$ is:

$$W_i=\left[\begin{array}{ccc}\frac{G\left(-1,1\right)}{{\displaystyle \sum _{j=1}^n}{G}_j}& \frac{G\left(0,1\right)}{{\displaystyle \sum _{j=1}^n{G}_j}}& \frac{G\left(1,1\right)}{{\displaystyle \sum _{j=1}^n{G}_j}}\\ \frac{G\left(-1,0\right)}{{\displaystyle \sum _{j=1}^n{G}_j}}& \frac{G\left(0,0\right)}{{\displaystyle \sum _{j=1}^n{G}_j}}& \frac{G\left(0,1\right)}{{\displaystyle \sum _{j=1}^n{G}_j}}\\ \frac{G\left(-1,-1\right)}{{\displaystyle \sum _{j=1}^n{G}_j}}& \frac{G\left(-1,0\right)}{{\displaystyle \sum _{j=1}^n{G}_j}}& \frac{G\left(-1,1\right)}{{\displaystyle \sum _{j=1}^n{G}_j}}\end{array}\right]$$

(2)

where the central pixel is considered as the origin of the coordinate and n is the sum of surrounding pixels and the central pixel. The relation between the gray scale of the central pixel $Gra{y}_c^{\prime }$ and the gray scale of the surrounding pixels $Gra{y}_i$ can be written as:

$$Gra{y}_c^{\prime }=\sum _{i=1}^n{W}_i·Gra{y}_i$$

(3)

The Gaussian blur filter provides gradual smoothing and preserves the edges better than any other mean filter. We have used Gaussian blur to reduce the high-frequency components. The size of the Gaussian kernel depends on the noise level in the image. If the kernel size is too large, small features within the image may get suppressed, and the image may look blurred. If the kernel size is too small, eliminating the noises within the image will be compromised.

Ostu binarization is often used to separate intra-image pixels into two parts and determine the threshold of the two parts. This algorithm generate a binary image that helped in displaying the desired scatter areas. The binarization is performed on the mask using an elliptical structuring element to smooth the contour of the scatters, break narrow isthmuses that connected the scatters, remove the outlier pixels, and eliminate thin protrusions from the scatters. For a gray image, $G=\left\{\left.i\right|i=1,2,\dots ,255\right\}$ represents the possible set of gray scales, $P=\left\{\left.n\right|n=1,2,\dots ,N\right\}$ denotes the set of all pixels. The threshold of the gray scale $T_G$ can be expressed as:

$$T_G=\left\{\left.T_G\right|max\left({\sigma }^2\right)\right\}$$

(4)

where ${\sigma }^2$ denotes the variance between each part and can be written as:

$${\sigma }^2=q_1\left[1-q_1\right]{\left[{\mu }_1-{\mu }_2\right]}^2$$

(5)

with:

$$q_1=\sum _{i=0}^t{P}_i,{\mu }_1=\sum _{i=0}^{t}i{P}_i/q_1,{\mu }_2=\sum _{i=t+1}^{255}i{P}_i/\left(1-q_1\right)$$

(6)

where $N_i$ has a number of pixels with gray scale i, $P_i$ is the ratio of $N_i$ to N, $q_1$ is the ratio of number of pixels with gray scale lower than $T_G$ and N, ${\mu }_1$ the mean value of the gray scale of pixels in $P_1=\left\{\left.n\right|G_n\le T_G\right\}$, ${\mu }_2$ is the mean value of gray scale of pixels in $P_2=\left\{\left.n\right|G_n>T_G\right\}$, and $G_n$ is the gray scale of $n^{th}$ pixel.

2.2. Process Line Identification Using Cuckoo Search Algorithm

Using Gaussian blur and Ostu binarization, the scatter plot is processed to be a matrix with clusters of pixel. We define the set including all clusters of pixel as C whose expression can be written as:

$$C=\left\{C_1,C_2,\dots ,C_n\right\}\left(n=N\right)$$

(7)

The set of all possible combinations of the clusters of pixels $\xi$ can be expressed as:

$$\xi =\left\{\left.Com{b}_i\right|\forall C_i\in Com{b}_i,C_i\in C\right\}$$

(8)

The objective function of process line is:

$$\begin{array}{cc}\hfill F\left(\zeta \right)& =a_1\sum _{j=1}^s\left|x_j^l-x_j^f\right|+a_2\sum _{j=1}^s\left|y_j^{max}-y_j^{min}\right|\hfill \\ \hfill & -b\sum _{j=1}^{s-1}\left|y_{j+1}^l-y_j^f\right|\hfill \end{array}$$

(9)

where $a_1,a_2$ are gain coefficients, b is the loss coefficient, ($x_j^f$, $y_j^l$) denotes the coordinate of the first pixels, ($x_j^l$, $y_j^l$) shows the coordinate of the last pixel, and $y_j^{max}$ and $y_j^{min}$ are the maximum and minimum threshold of vertical coordinates. In addition, each $C_i$ in a $Com{b}_i$ obey:

$$x_{j-1}^b<x_j^b<x_{j+1}^b,j=2,3,\dots ,s-1$$

(10)

For arbitrary two pixel clusters $C_a$ and $C_b$ in $Com{b}_i$, $x_a^l<x_b^l$ when $x_b^f>x_a^f$. Then, we can consider the process line identification as an optimization question:

$$\begin{array}{cc}\hfill & maxF\left(\zeta \right)\hfill \\ \hfill & \begin{array}{cccc}\hfill s.t.& \hfill x_{j-1}^b& \hfill <x_j^b<x_{j+1}^b,& j=2,3,\dots ,s-1\\ \hfill & \hfill x_a^l& <x_b^l,\hfill & \end{array}.\hfill \end{array}$$

(11)

The Cuckoo Search algorithm is a stochastic optimization model which is developed based on the brood parasitism of cuckoo birds. Figure 3 shows the flowchart of the Cuckoo Search algorithm.

The algorithm follows three principal rules: (a) Each cuckoo lays only one egg each time and picks one nest to place the egg randomly; (b) The best host nests with the highest-quality eggs is kept in the following generation; (c) The number of available host nests is fixed. Details of the algorithm are as follows:

Step 1: Develop the objective function and determine inputs including the threshold, number of iterations, objective accuracy, etc.

Step 2: Establish the initial generation $x_1,x_2,\dots ,x_N$ randomly. Each cuckoo denotes a dataset of attribute values of continuous point, whose expression can be written as:

$$\begin{array}{cc}\hfill N& =\left[\left.\begin{array}{c}{x}_1\\ x_2\\ \cdots \\ x_N\end{array}\right|\begin{array}{c}F\left(x_1\right)\\ F\left(x_2\right)\\ \cdots \\ F\left(x_N\right)\end{array}\right]\hfill \\ \hfill & =\left[\left.\begin{array}{cccccc}{x}_1^e& x_1^b& y_1^{min}& y_1^{max}& y_1^e& y_1^b\\ x_2^e& x_2^b& y_2^{min}& y_2^{max}& y_2^e& y_2^b\\ \cdots & \cdots & \cdots & \cdots & \cdots & \cdots \\ x_N^e& x_N^b& y_N^{min}& y_N^{max}& y_N^e& y_N^b\end{array}\right|\begin{array}{c}F\left(x_1\right)\\ F\left(x_2\right)\\ \cdots \\ F\left(x_N\right)\end{array}\right]\hfill \end{array}$$

(12)

where N is the number of cuckoos. Each cuckoo represents a set of attribute values of continuous points. The best cuckoo $x_b^t$ and the objective function for each cuckoo are determined in this step.

Step 3: Implement the Levy flight. The expression of Levy flight is:

$$x_i^{t+1}=x_i^t+\alpha \oplus Levy\left(\lambda \right)\left(i=1,2,\cdots ,n\right)$$

(13)

where $\alpha$ denotes the step size, ⊕ denotes the entry-wise multiplications, $x_i^t$ and $x_i^{t+1}$ are the positions of $t^{th}$ and $t+1^{th}$ generations of cuckoos, respectively, and $Levy\left(\lambda \right)$ is a random searching vector which follows Levy distribution:

$$\begin{array}{cc}\hfill Levy\left(\lambda \right)& \sim \frac{\varphi u}{{\left|v\right|}^{1/\beta }}\varphi \hfill \\ \hfill & ={\left\{\frac{\Gamma \left(1+\beta \right)\times sin\left(\pi \times \beta /2\right)}{\Gamma \left\{\left[\left(1+\beta \right)/2\right]\times \beta \times 2^{\left(\beta -1\right)/2}\right\}}\right\}}^{1/\beta }\hfill \end{array}$$

(14)

where $\Gamma$ is the Gamma function, u and v are random numbers which follow Gaussian distributions, and $\beta$ is the parameter of Levy flight. The nests location for the next generation of cuckoos $x_i^{t+1}$ is given by:

$$x_i^{t+1}=x_i^t+alph{a}_0\frac{\varphi \times u}{{\left|v\right|}^{1/\beta }}\left(x_i^t-x_b^t\right)$$

(15)

where $x_b^t$ is the best nest in $t^{th}$ generation and ${\alpha }_0$ is the scaling factor.

Step 4: Eliminate the alien eggs with a probability of $P_a\in \left[0,1\right]$. The mathematical expressions can be written as:

$$x_i^{t+1}=\left\{\begin{array}{c}{x}_i^t+r·\left(x_{r_1}^t-x_{r_2}^t\right),ifr<P_a\hfill \\ x_i^t,otherwise\hfill \end{array}\right.$$

(16)

where r is a random number in the range of 0 to 1 and $x_{r_1}^t$ and $x_{r_2}^t$ are two random nest locations in the $t^{th}$ generation, respectively.

Step 5: Determine the objective function of renewal nest locations as well as the optimal cuckoo of $t+1^{th}$ generation $x_b^{t+1}$. Here, the smaller value between $x_b^{t+1}$ and $x_b^t$ is kept as the $t+1^{th}$ optimal cuckoo.

Step 6: Repeat Step 3 and 4 until the number of iteration other termination criteria reach the set values.

Step 7: To enhance the effect of detecting outliers, the procedure combination needs to be implemented cyclic until the result satisfies the requirement. The threshold of $R_y$ and $R_y^{\prime }$ are:

$$R_{ymax}=max\left\{y_1,y_2,\cdots ,y_m\right\},R_{ymin}=min\left\{y_1,y_2,\cdots ,y_m\right\}$$

(17)

$$R_{ymax}^{\prime }=max\left\{y_1,y_2,\cdots ,y_n\right\},R_{ymin}^{\prime }=min\left\{y_1,y_2,\cdots ,y_n\right\}$$

(18)

where m counts the number of the pixel clusters in raw data and n counts the number of pixel clusters in processed data. The processed data can be validated once $R_{ymax}=R_{ymax}^{\prime }$ and $R_{ymin}=R_{ymin}^{\prime }$.

Figure 4 shows the example of process line detection.

3. Dataset

In this study, we used the monitoring data of the dam at Jinping-I hydropower station as the dataset. The dam is located at Yalong River, Sichuan Province, China, and famous as being one of the highest concrete arch dam worldwide. The elevation of the dam’s crest is 1885 m and that of the dam’s foundation is 1580 m. The normal impounded water level is 1880 m. Figure 5a,b show the geological location and the photo of the dam, respectively.

As shown in Figure 6, the monitoring points were well distributed on the cross section of the dam. As the dataset, we used displacement monitoring data of six selected monitoring points which located on three different plumb lines. The selected monitoring points PL11-1, PL11-3, PL13-1, PL13-3, PL16-1, and PL16-3 are highlighted by red square in Figure 6. The data sequence was separated into two parts, where 80% of the dataset was used to test the detection ability (data during the period 1 July 2017 to 30 February 2018) and 20% of the dataset was used to evaluate the prediction performance (from 1 March 2018 to 30 May 2018).

4. Results

4.1. Optimal Settings of the Scatter Plot of the Original Data

For the data processing method based on matrix manipulation and Cuckoo Search algorithm, the first step is to generate a scatter plot of the original data. Then, the scatter plot is considered an image and the matrix of the image is exported. Attributes of scatter such as shape and size affect the performance of matrix manipulation including Gaussian blur and Ostu binarization. Thus, we first determine the optimal settings of attributes of scatters. Figure 7 shows the stack of patterns with different shapes including square, cross, and isscross. All these three patterns have nine pixels. It can be seen from the figure that when the patterns stacks together, the cross pattern could keep more information as compared with the square pattern and isscross pattern.

To obtain the optimal settings of the attributes of scatters, we constructed a scatter plot of a sample data sequence using four different shapes of scatters (circle, square, cross, and isscross) with the same size, and compared the filtering performance of Gaussian blur and Ostu binarization. Figure 8a–d exhibit the plots processed by Gaussian blur and Ostu binarization using a circle, square, cross, and isscross as scatters, respectively.

It can be seen from the figure that the data processing using cross as the scatter shape has the best performance in outlier detection. Using cross as the scatter shape, more outliers are eliminated, and the clusters of continuous points are identified. Comparing with the square and isscross, cross patterns have more dispersing distributed pixels. Using cross as the scatter shape, the gray scale of isolated outliers can be reduced more intensely by Gaussian blur, and thus, the outlier can be easier eliminated by the filter. In addition, using a circle and square as scatter shapes, the outliers detection performance is significantly worse than that of cross and isscross. The detection performance of scatter plots using a cross is slightly higher than using isscross. Therefore, we selected cross as the shape of scatter in the data preprocessing using Gaussian blur and Ostu binarization.

As the shape of scatters have been selected, the size of the scatters should be determined. To determine the optimal set of scatter size, we compare the outlier detection performance of scatter plot using cross scatter with four different sizes. As shown in Figure 9, the number of pixels of these four sizes are 5, 9, 13, and 17, respectively. Comparing Figure 9 with Figure 8a,b, using a cross with five pixels, the detection performance is similar to those using a circle and square as scatter shapes. This is because when the size of the cross is small, the pixels are centrally distributed, that is, the micro shape is similar to a circle and square. When the size of the cross is increased, the scatter is less centralized distributed. The cross with nine pixels has the best performance in outlier detection, that is, more outliers are detected and eliminated after the data processing. Therefore, a nine-pixels cross is the optimal size and shape.

4.2. Results of Abnormal Data Detection Based on the Proposed Method

According to the analysis in Section 4.1, a cross of nine pixels is selected as the scatter in the pre-processing procedure using Gaussian blur and Ostu binarization. For the process line identification using the Cuckoo Search algorithm, Levy flight $\beta$ is set to 2.0 and the discard probability $P_a$ is set to 0.2.

Using the sample data sequence as an example, Figure 10 shows the whole procedure of the proposed method, which combines matrix manipulation and the Cuckoo Search algorithm. As presented in Section 2, the processing procedure of the proposed method consists of three steps: (1) Gaussian blur, (2) Ostu binarization, and (3) process line identification using the Cuckoo Search algorithm.

Displacement monitoring datasets of the six selected monitoring points installed in the dam at Jinping-I hydropower station are used to validate the propose method. Here, in order to verify the processing ability of the proposed method, we added the numbers of outliers in the original dataset, so as to increase the detection difficulty. We then use the proposed method to detect and eliminate outliers in the data sequence.

Table 1. The total data number $N_t$ and number of outliers $N_o$ detected by the proposed method.

Monitoring Points	PL11-1	PL11-3	PL13-1	PL13-3	PL16-1	PL16-3
$N_t$	830	788	872	820	860	860
$N_o$	86	91	75	76	70	89

shows the total data number $N_t$ and number of outliers $N_d$ detected by the proposed method for each monitoring point.

4.3. Comparison of the Proposed Method with 3-$\sigma$ Method

To evaluate the efficiency of the proposed method, we process the same dataset using the 3-$\sigma$ method, which is a classical method in outlier detection, combining multidimensional regression and 3-$\sigma$ criterion.

The factors dominating the displacement of dam includes three components: the hydrostatic component ${\delta }_H$, the temperature component ${\delta }_T$, and the aging component ${\delta }_{\theta }$. The expressions of ${\delta }_H$, ${\delta }_T$, and ${\delta }_{\theta }$ are as follows:

$${\delta }_H=\sum _{i=0}^4{a}_i{H}^i$$

(19)

$${\delta }_T=\sum _{j=1}^4{b}_j{T}_j$$

(20)

$${\delta }_{\theta }=c_1\theta +c_{2}ln\theta$$

(21)

where H is the upstream water level, $T_j$ is the external temperature, and $\theta =\frac{t}{100}$ and t is time. The displacement $\delta$ can be written as:

$$\delta =a_0+\sum _{i=1}^4{a}_i{H}^i+\sum _{j=1}^4{b}_j{T}_j+c_1\theta +c_{2}ln\theta$$

(22)

where $a_0$, $a_i$, $b_j$, $c_1$, and $c_2$ are the coefficients of explanatory variables and can be solved using the least square regression method.

According to the principle of the 3-$\sigma$ method, the probability of the absolute error between modeling data and original data $\left|\epsilon \right|$ less than 3$\sigma$ is 99.7%. Here, $\sigma$ is the residual standard deviation whose expression can be written as:

$$\sigma =\sqrt{\frac{{\displaystyle \sum _{i=1}^n}{\left[Y\left(i\right)-\widehat{Y}\left(i\right)\right]}^2}{n-k-1}}$$

(23)

where n is the data number of the dataset, k is the degree of freedom of the model, and $Y\left(i\right)$ and $\widehat{Y}\left(i\right)$ denote the monitoring data sequence and modeling data sequence, respectively.

We suppose that outliers should have a large deviation from the modeling displacement. Thus, 3$\sigma$ can be used as the threshold for outlier detection. That is, the monitoring data are regarded as outliers once $\left|\epsilon \right|$ exceeds 3$\sigma$. The mathematical descriptions can be written as:

$$\left\{\begin{array}{c}\left|\epsilon \right|>3\sigma Y\left(i\right)isoutlier\hfill \\ \left|\epsilon \right|\le 3\sigma Y\left(i\right)isvalidvalue\hfill \end{array}\right.$$

(24)

where:

$$\left|\epsilon \right|=\left|Y\left(i\right)-\widehat{Y}\left(i\right)\right|$$

(25)

Figure 11 compares the outlier detection results obtained by the 3-$\sigma$ method and the proposed method. Black dots present the original data sequence without outlier detection, and the black dots without marks present outliers detected by the 3-$\sigma$ method and the proposed method. Red cross and yellow square denote outliers detected by 3-$\sigma$ method and the proposed method, respectively. For the 3-$\sigma$ method, only outliers located in the area between monitoring data and modeling data exceeds 3-$\sigma$ can be detected. Outliers located in the surrounding areas of the process line can not be eliminated. Compared with the 3-$\sigma$ method, the proposed method has a better performance. It can detect most outliers existing in all these six data sequences.

We defined the ratio of number of detected outliers $N_d$ to number of outliers $N_o$ as detection ratio $r_d$:

$$r_d=\frac{N_d}{N_o}$$

(26)

Table 2. $N_d$ and $r_d$ of the 3-$\sigma$ method and proposed method for each monitoring point.

Monitoring Points	The Proposed Method		3$\mathit{\sigma }$ Method
	${\mathit{N}}_{\mathit{d}}$	${\mathit{r}}_{\mathit{d}}$(%)	${\mathit{N}}_{\mathit{d}}$	${\mathit{r}}_{\mathit{d}}$(%)
PL11-1	75	87.20	31	36.04
PL11-3	84	92.31	24	26.37
PL13-1	72	96.00	29	38.66
PL13-3	72	94.73	28	36.84
PL16-1	70	100.00	36	51.42
PL16-3	88	98.87	31	34.83

exhibits the number of detected outliers detected $N_d$ and the detection ratio $r_d$ of each monitoring point for the proposed method and 3$\sigma$ method. The average of $r_d$ is 32.22% for the 3-$\sigma$ method and 9% for the proposed method. Using the 3-$\sigma$ method, $r_d$ ranges between 26.37% and 51.42% for all monitoring points. Using the proposed method, $r_d$ ranges between 87.20% and 100% for all monitoring points. In general, the proposed method has a significantly higher performance in outlier detection than the 3-$\sigma$ method.

4.4. Regression Model Development Using Processed Data

The regression models are developed using data processed by the 3-$\sigma$ method and proposed method, to verify the efficiency of outlier detection for monitoring data modeling. The principal expression of the regression model is:

$$\widehat{y}=a_0+a_1{x}_1+a_2{x}_2+\cdots +a_i{x}_i$$

(27)

where $\widehat{y}$ is the modeling data, $x_i$ is the explanatory variables, and $a_i$ is the coefficients of explanatory variables. $x_i$ consists of the three components introduced in Section 4.3: the hydrostatic component ${\delta }_H$, the temperature component ${\delta }_T$, and the aging component ${\delta }_{\theta }$. The coefficients of explanatory variables can be solved using the ordinary least square method.

Figure 12 exhibits the fitting results using data processed by the 3-$\sigma$ method and proposed method. The displacement modeled using data processed by both these two methods are fitted well with the monitoring data. In general, the prediction results obtained using both datasets are quite similar to the observed data.

The coefficient of determination $R^2$ and standard deviation RMSE are selected as indicators, to quantify the fitting performance using these two dataset and the predicting accuracy of the model. The equations of $R^2$ and RMSE are as follows:

$$R^2=\frac{{\displaystyle \sum _{i=1}^n}{\left({\widehat{y}}_i-{\overline{y}}_i\right)}^2}{{\displaystyle \sum _{i=1}^n}{\left(y_i-{\overline{y}}_i\right)}^2}$$

(28)

$$RMSE=\sqrt{\frac{{\displaystyle \sum _{i=1}^n}{\left(y_i-\widehat{y_i}\right)}^2}{n}}$$

(29)

where ${\overline{y}}_i$ is the average of the monitoring data, ${\widehat{y}}_i$ is the modeling data, $y_i$ is the monitoring data, and n is the total data.

Table 3. $R^2$ and RMSE of the regression models using the dataset processed by the 3-$\sigma$ method and proposed method.

Monitoring Points	${\mathit{R}}^2$		RMSE
	The Proposed Method	3-$\mathit{\sigma }$ Method	The Proposed Method	3-$\mathit{\sigma }$ Method
PL11-1	0.982	0.954	0.943	2.371
PL11-3	0.983	0.959	0.538	2.228
PL13-1	0.998	0.941	0.228	2.274
PL13-3	0.993	0.974	0.393	2.213
PL16-1	0.933	0.962	1.236	2.734
PL16-3	0.992	0.947	0.304	2.561

exhibits the $R^2$ and RMSE of these two dataset for each monitoring point. $R^2$ exceeded 0.9 for both datasets, which indicates that the regression model can be validated. $R^2$ ranges between 0.9474 and 0.9854 using the dataset processed by the 3-$\sigma$ method, ranges between 0.9933 and 0.9998 using dataset processed by the proposed method. It can be noted that the regression model has a better fitting performance using the dataset processed by the proposed method. The regression model developed using the dataset with fewer outliers performs better in prediction accuracy.

5. Conclusions

Displacement monitoring data analysis is an effective method to evaluate the running status of dams. Measurement error greatly affects the accuracy of monitoring data modeling. In order to process the abnormal dam displacement monitoring data, we proposed a data processing method by combining matrix manipulation and the Cuckoo Search algorithm.

In this paper, we first generate a scatter plot of the original monitoring data. Once the scatter plot has been drawn, we then consider the scatter plot as an image and export the matrix of the images. The matrix consists of isolated outliers, clusters of outliers, and clusters of normal data. At the pre-processing stage, the isolated outliers are detected and eliminated using Gaussian blur and Ostu binarization. Gaussian blur reduce the gray scales of isolated outliers, and Ostu binarization eliminate these isolated outliers from the matrix. Using the Cuckoo Search algorithm, we search the optimal series of clusters for determining the process line.

The proposed method is validated using the displacement monitoring data of the dam at Jinping-I hydropower station. By comparing the pre-processed results obtained by different sets of scatter plot, the scatter plots of nine-pixels cross is used in this study. The ratio of outlier detected $r_d$ using the proposed method is over 85% for each monitoring point, and it is significantly higher than that of the 3-$\sigma$ method. In addition, we regress the processed dataset and original dataset using a statistical model, respectively. The results indicate that the regression model fitted with data pre-processed by the proposed model has a better performance compared with the regression model using the original dataset and dataset pre-processed using the 3-$\sigma$ method.

The proposed method provides a novel solution for detecting outliers in time series data with continuous characteristics. Engineering application of the method in this article is to detect abnormal data in monitoring data of dam displacement. The proposed method is not applicable for datasets without continuous characteristics, i.e., time series data without varying patterns or time-invariance data. One future direction of this study is to increase the engineering applications of the proposed method. The application can be extended to other structures, such as bridges, slopes, etc. In addition, this study introduced the image processing method into abnormal data detection. For both the image processing part and detection part, we selected mutual and routine methods; the detection accuracy can be further improved if we use other high performance methods. Therefore, future studies will need to improve the abnormal detecting performance by introducing a high-ability image processing method and searching algorithm.