Analysis of Different Neural Networks and a New Architecture for Short-Term Load Forecasting

Abstract

Short-term load forecasting (STLF) has been widely studied because it plays a very important role in improving the economy and security of electric system operations. Many types of neural networks have been successfully used for STLF. In most of these methods, common neural networks were used, but without a systematic comparative analysis. In this paper, we first compare the most frequently used neural networks’ performance on the load dataset from the State Grid Sichuan Electric Power Company (China). Then, considering the current neural networks’ disadvantages, we propose a new architecture called a gate-recurrent neural network (RNN) based on an RNN for STLF. By evaluating all the methods on our dataset, the results demonstrate that the performance of different neural network methods are related to the data time scale, and our proposed method is more accurate on a much shorter time scale, particularly when the time scale is smaller than 20 min.

1. Introduction

Accurate short-term load forecasting (STLF) can play a significant role in power construction planning and power grid operation, and also has crucial implications for the sustainable development of power enterprises. STLF can predict future loads for minutes to weeks. Because of the nonlinearity, non-stationarity, and non-seasonality of STLF, it is very challenging to predict accurately. Inaccurate load forecasting may increase operating costs [1]. By contrast, with an accurate electric load forecasting method, fundamental operating functions, such as unit maintenance, reliability analysis, and unit commitment, can be operated more efficiently [2]. Thus, it is essential for power suppliers to build an effective model that can predict power loads, accomplish a balance between production and demand, reduce production costs, and implement pricing schemes for various demand responses. According to the length of the forecast period, power load forecasting is divided into four categories: long-term load forecasting, medium-term load forecasting, STLF, and ultra-STLF [3].

There have been many efforts to develop accurate STLF; many methods have been propsoed [4,5,6,7,8,9,10,11,12,13,14,15,16]. In much earlier works, researchers attempted to forecast load precisely using a mathematical statistics approach. The most representative is the regression analysis approach [4], which uses a set of functional linear regression models. In [5], the authors used the Kalman filter to develop a very short-term load predictor, and the Box–Jenkins autoregressive integrated moving average approach was proposed in [6]. In [17], the relationships between demand and driver variables were evaluated by using semi-parametric additive models. Additionally, in [18], the authors came up with a new SVD-based exponential smoothing formulation. Based on linear regression and patterns, an univariate models were proposed for 34 daily cycles of a load time series in [19]. In [20], by combining a Bayesian neural network with the hybrid Monte Carlo algorithm, the authors assumed a new model for STLF. Modified artificial bee colony algorithm, extreme learning machine, and the wavelet transform were combined to construct a novel STLF method in [21].

However, mathematical statistics approaches are based on linear analysis; thus, it is difficult for them to predict nonlinear and non-stationary problems. For better learning of nonlinear features, machine learning is a good approach. Two broad categories of methods exist: support vector regression (SVR) and artificial neural networks (ANNs). In the SVR approach, a generic strategy for STLF based on the SVR has been proposed [22,23,24,25]. In this method, there are two considerable improvements to SVR-based load forecasting methods: the procedure for generating model inputs and the subsequent model input was selected by feature selection algorithms. By combining SVR with other algorithms, many hybrid methods have been proposed. An SVR model combined with the differential empirical mode decomposition algorithm and autoregression was proposed in [26,27,28], and provides higher accuracy and interpretability, and better generation ability. To extend the SVR method, a chaotic genetic algorithm was presented to improve forecasting performance in [29]. In [30,31], the authors combined support vector machines (SVMs) with a genetic algorithm, and combined SVR with ant colony optimization to forecast the system load. In [32],an attempt based on double seasonal exponential smoothing was used for STLF. Besides these studies, in [7], in order to achieve better regression and forecasting performance, the proposed SVR-based STLF approach, a supervised machine learning approach with the preprocessing of input data is required. Simultaneously, STLF models were developed using fuzzy logic and an adaptive neuro-fuzzy inference system [33], with efficient load forecasting and has been the alternative approach for STLF in Turkey.

For ANNs, the backpropagation neural network (BPNN) was the first ANN method used for load forecasting. In [34], the authors presented a BPNN approach with a rough set for complicated STLF with dynamic and nonlinear factors to enhance the performance of predictions. By combining a Bayesian neural network and BPNN, Ningl et al. [35] proposed a Bayesian-backpropagation method to forecast the hour power load of weekdays and weekends. Based on the BPNN, the authors discussed the relationship between the daily load and weather factors in [36]. Because the BPNN is a type of feedforward ANN, it cannot learn the features of time sequential data, but power load data can be considered as sequential data. Recurrent neural networks (RNNs) have been introduced into STLF. In [37,38,39,40], the authors proposed using an RNN to capture a compact and robust representation. A multiscale bi-linear RNN was proposed for STLF [41]. Additionally, a long-short term memory (LSTM) network as a type of more complex RNN has been used in STLF [42,43,44]. However, the performance of LSTM is not very effective.

Although neural networks have been frequently used in short-term power load prediction, there barely has been a systematic comparison of the role of neural networks in this problem to determine how to solve the key problems and which approach exerted the least negative effect on the neural network. Therefore, we systematically compare the advantages and disadvantages of different types of commonly used neural network methods and then analyse the performance of neural networks for different time scales. Simultaneously, according to the difference in network performance, we design new RNN architecture that balance memory and the current scenario at any time by referring to highway neural networks [45] in the time dimension. Thus, our main contributions in this paper are as follows: first, we systematically analyze the performance of commonly used neural networks on our power load data. Second, according to the advantages and disadvantages of these networks, we propose a new neural network architecture, the gate-RNN, to forecast STLF.

2. Methods

To explore the performance of different types of neural networks applied to STLF, we use four types: three types of the most commonly used neural networks and an improved neural network that we call the gate-RNN.

2.1. BPNN

A BPNN is a type of feedforward neural network (FNN). The simple architecture of a BPNN is shown in Figure 1, where the neurons in the same layer of the BPNN are only connected with adjacent layers’ neurons; there are no connections among neurons in the same layer. The BPNN contains three types of layers: the input layer, hidden layers, and output layer. Input layer inputs data into the neural network. Output layer outputs the neural network’s computational results. The hidden layers are the layers between the input layer and output layer. The values of connections between different layers are weights denoted by $w^i$ where i denotes i-th layer. All the knowledge that the neural network has learned is stored in the weights.

The goal of training a BPNN is to determine a set of suitable W so that the network can obtain the correct output when test data is input into it by training on the train dataset. Suppose there is a training set $〈\left(x_1,y_1\right),(x_2,y_2),\dots ,(x_n,y_n)〉$, which contains n tuples. Each tuple contains input data $x_i$ and target label $y_i$. To train a BPNN, the first step is forward computing, which can be computed as

$$\left\{\begin{array}{c}{z}^{l+1}=W^l{a}^l,\hfill \\ a^{l+1}=f\left(z^{l+1}\right),\hfill \end{array}\right.$$

(1)

where $z^l$ is the l-th layer’s input vector, $a^l$ is the l-th layer’s output vector, and $f(·)$ is an active function. In this paper, we chose the rectified linear unit as our active function, which is defined as

$$f\left(x\right)=\left\{\begin{array}{ccc}x,\hfill & \hfill if& \hfill x>=0,\\ 0,\hfill & \hfill else.\end{array}\right.$$

(2)

After forward computing, the BP networks need to update the network through the losses that were calculated from the target labels so that the network can determine the suitable W, where W is a set of $\left\{W^1,\dots ,W^L\right\}$ and L is the number of layers. One of the most commonly used update methods is the gradient descent method with BP. The BP process starts by defining a loss function, Loss. The loss function measures the distance between the outputs of the BP network and the true targets. The mean-square error (MSE) is a common loss function for prediction. It is defined as Equation (3):

$$Loss=\frac{1}{2n_L}\sum _{i=1}^{n_L}{∥y_i^L\left(x\right)-a_i^L∥}^2,$$

(3)

where $y_i^L$ is the i-th output of the output layer, $a_i^L$ is the i-th target label of output layer, and $n_L$ is the total number of outputs in the output layer. $\frac{1}{2}$ is for better computing the derivative of loss. Because $a_i^L$ is computed by W, loss function $Loss$ is the function of weight W.

To use the gradient descent method to determine the optimal W, we need to compute $▽W$ which is defined as

$$▽W=\frac{\partial Loss}{\partial W},$$

(4)

and update W using

$$W=W-\alpha ▽W,$$

(5)

where $\alpha$ is the learning rate, which controls the learning step during each update. We define an error term in the i-th neuron l-th layer as ${\delta }_i^l=\frac{\partial L}{\partial z_i^l}$ for better computation of gradient $▽W$ in each layer.

In the output layer, we can compute the gradient of $w_{ij}^L$ directly by combining Equation (3) and using the chain rule:

$$\frac{\partial Loss}{\partial w_{}^L}=\frac{\partial Loss}{\partial z_{}^L}\frac{\partial z_{}^L}{\partial w_{}^L}={\delta }_{}^L{\left(a_{}^{L-1}\right)}^T.$$

(6)

Then, we compute the other layers’ error terms using back propagation:

$${\delta }_{}^l=\frac{\partial Loss}{\partial z_{}^l}=\frac{\partial Loss}{\partial z_{}^{l+1}}\frac{\partial z_{}^{l+1}}{\partial z_{}^l}=\dot{f}\left(z_{}^l\right)\left({\left(w_{}^l\right)}^T{\delta }_j^{l+1}\right).$$

(7)

Then, we compute the remaining layers’ weight gradients from back to front in matrix form as

$$▽W^l={\mathbf{\delta }}^{l+1}{\left(a^l\right)}^T.$$

(8)

Thus, the learning process of the BPNN can be presented as Algorithm 1.

Algorithm 1: The BP network update process with gradient decent.

2.2. RNN

STLF is a type of prediction based on the previous time. It is based on the history load information forecasting the next time load value. We can consider it as a sequential problem. Among all types of neural networks, the RNN is good at solving sequential problems. An RNN contains recurrent connections. A simple architecture of an RNN is shown in Figure 2. It contains an input layer, recurrent layer, and output layer. The difference between an RNN and BPNN is that an RNN has connections among the same recurrent layer’s neurons.

For a better understanding of the computation of an RNN, we can unroll the RNN on the time dimension as shown in Figure 3. In the unrolled RNN, the neurons in the hidden layer at each time step t can be considered as one layer of an FNN. If the time step is T, then the unrolled RNN has T hidden layers. Based on the unrolled network, we can train the RNN using backpropagation through time (BPTT) [46], specifically epoch-wise BPTT, using the following steps:

• Forward computing: the output of network at time t can be computed as the following equation:

  $$\left\{\begin{array}{c}s(t+1)=x\left(t\right)+wy\left(t\right),\hfill \\ y(t+1)=f\left(s\right(t+1\left)\right).\hfill \end{array}\right.$$

  (9)
• Define the loss as

  $$E_{(t^{\prime },t)}^{total}=\sum _{\tau =t^{\prime }+1}^{t}E\left(\tau \right),\tau \in \left[t_0+1,t\right],$$

  (10)

  where $E\left(t\right)=-\frac{1}{2}{\left[d\left(t^{\prime }\right)-y\left(t^{\prime }\right)\right]}^2$ is the error at time $t^{\prime }$. At each time step, $d\left(t^{\prime }\right)$ represents the target label and $y\left(t^{\prime }\right)$ is the output of the network at time $t^{\prime }$.
• Compute gradient $▽w$ of w as follows:

  $$▽w_{}=\frac{\partial E_{(t_0,t_1)}^{total}}{\partial w_{}}=\sum _{\tau =t_0+1}^t\frac{\partial E_{(t_0,t_1)}^{total}}{\partial w_{}\left(\tau \right)}.$$

  (11)
  Combine with the chain rule and define $e_{}\left(\tau \right)=\frac{\partial E\left(\tau \right)}{\partial y_{}\left(\tau \right)}$, which can be written as

  $$\frac{\partial E_{(t_0,t_1)}^{total}}{\partial w_{}\left(\tau \right)}=\delta \left(\tau \right){\left(y_{}(\tau -1)\right)}^T,$$

  (12)

  where

  $${\delta }_k\left(\tau \right)=\left\{\begin{array}{ccc}f\dot{(}{s}_{}\left(\tau \right))e_{}\left(\tau \right),\hfill & when\hfill & \tau =t,\hfill \\ f\dot{(}{s}_{}\left(\tau \right))(e_{}\left(\tau \right)+w^T\delta (\tau +1)),\hfill & when\hfill & t_0<\tau <t.\hfill \end{array}\right.$$

  (13)
  Hence, the gradients of weights can be computed as

  $$▽w_{}=\delta \left(\tau \right){\left(y(\tau -1)\right)}^T.$$

  (14)
• Then, update the weights in the RNN using Equation (5) until the network converges.

2.3. LSTM

LSTM is a type of RNN, but it has a more complicated architecture. A common LSTM network consists of many LSTM blocks, which are called memory cells. An LSTM cell stores the input for some period of time. The flow of information into and out of the cell was determined by the values in the cell and regulated by the three gates. A classical cell architecture is shown in Figure 4. The cell receives input $x_t$ at time t, output $h_{t-1}$ and state vector $C_{t-1}$ at time $t-1$. There are three gates in the cell to control the computation of the cell: the input gate, output gate and forget gate. The input gate selectively records new information into the cell state. The output gate determines which information is worth outputting. The forget gate selectively forgets some information and retains much more valuable information. The forward pass of an LSTM unit with a forget gate can be computed as following equations

$$\left\{\begin{array}{c}{f}_t={\sigma }_g(W_f{x}_t+U_f{h}_{t-1}),\hfill \\ i_t={\sigma }_g(W_i{x}_t+U_i{h}_{t-1}),\hfill \\ o_t={\sigma }_g(W_o{x}_t+U_o{h}_{t-1}),\hfill \\ c_t=f_t·c_{t-1}+i_t·{\sigma }_c(W_c{x}_t+U_c{h}_{t-1}),\hfill \\ h_t=o_t·{\sigma }_h\left(c_t\right),\hfill \end{array}\right.$$

(15)

where $x_t$ is the input vector to the cell. $W\in {\mathbb{R}}^{h\times d}$ and $U\in {\mathbb{R}}^{h\times h}$ are the weight matrices and bias vector parameters, where h and d refer to the number of input features and number of hidden units. $f_t,i_i$ and $O_t$ are the forget gate, input gate and output gate’s output, respectively. $h_t$ is the hidden state vector, which is also called the output vector. $c_t$ is the cell state vector. ${\sigma }_g,{\sigma }_c$ and ${\sigma }_h$ are activation functions, where ${\sigma }_g$ is the sigmoid function, ${\sigma }_c$ is the hyperbolic tangent function and ${\sigma }_h$ is the hyperbolic tangent.

To update the parameters of LSTM, a common way is using epoch-wise BPTT algoritm as the same with RNN, the detail can be found in [47].

2.4. Gate-RNN

As we can see, BPNN is an FNN. If BPNNs are used to predict load, they will weaken the relationships in the time dimension. Regarding RNN and LSTM, they are good at capturing temporal features. However, RNN can only memory history information through the weights among neurons, which is much is too simple to handle the input information and memory information. By contrast, during an experiment, we found that LSTM was too complicated to train stably. Thus, we propose an RNN cell that can control the computation of input information and memory information, converges well in the training process, and has excellent results in ultra STLF.

Figure 5 shows the cell architecture of our gate-RNN. At time t, the input vector of the cell is $x_t$ and the history information is output $h_{t-1}$ at time $t-1$. When input vector $x_t$ is input into the cell, it is divided into three branches: two branches for computing two gates’ values that are $1-G$ and G, and one for computing the cell’s state that is S. Gate G controls the effect of the cell state on the output and gate $1-G$ controls the effect of history information on the output. The output of the cell $h_t$ combines the output of both gates. We use the $“+”$ operator as our combination approach, and it adds the values of both gates at the same position.

The computation of the cell is as follows:

$$\left\{\begin{array}{c}{g}_t=W_g{x}_t,\hfill \\ s_t=W_s{x}_t,\hfill \\ {\tilde{h}}_t=W_h{h}_{t-1},\hfill \\ h_t=g_t·s_t+{\tilde{h}}_t·(1-g_t),\hfill \end{array}\right.$$

(16)

as for LSTM, $W\in {\mathbb{R}}^{h\times d}$ denotes the weight matrices parameters. Many gate-RNN cells are combined to obtain a layer and then a neural network. To update the gate-RNN, we compute the gradient of the weight at each time step. We also define an error term at the t-th time step (same as the l-th layer) as ${\delta }_t=\frac{\partial Loss}{\partial h_t}$ to better compute the gradient of $▽W$ in each layer. In the t-th time step, we can compute the gradients of $W_g$, $W_s$, $W_h$ directly by combining Equation (3) and using the chain rule, as follows:

$$\begin{array}{cc}\hfill ▽W_g=\frac{\partial Loss}{\partial W_g}& =\frac{\partial Loss}{\partial h_t}·\frac{\partial h_t}{\partial g_t}·\frac{\partial g_t}{\partial W_g}\hfill \\ \hfill & ={\delta }_t·{((s_t-{\tilde{h}}_t)·\left(x_t\right))}^T,\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill ▽W_s=\frac{\partial Loss}{\partial W_s}& =\frac{\partial Loss}{\partial h_t}·\frac{\partial h_t}{\partial s_t}·\frac{\partial s_t}{\partial W_s}\hfill \\ \hfill & ={\delta }_t·{(g_t·\left(x_t\right))}^T,\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill ▽W_h=\frac{\partial Loss}{\partial W_h}& =\frac{\partial Loss}{\partial h}·\frac{\partial h_t}{\partial {\tilde{h}}_t}·\frac{\partial {\tilde{h}}_t}{\partial W_h}\hfill \\ \hfill & ={\delta }_t·{((1-g_t)·\left(h_{t-1}\right))}^T.\hfill \end{array}$$

(19)

Then, we update the entire network using Algorithm 2.

Algorithm 2: The update process of gate-RNN.

Input  :        The input dataset $D=\left\{(x,y^L)\right\}$;        The learning rate $\alpha =0.0001$;    Output:        The weight of the gate-RNN network $\mathbf{W}$ after training; 1  initialize the model parameter $\mathbf{W}$ with uniform $\left(-\sqrt{6}/k,\sqrt{6}/k\right)$; 2  For each sample $(x,y^L)\in D$, set $a^1=x$; 3  Do forward computing by using Equation (16); 4  Compute loss by Equation (3); 5  Compute the last time step T error term by ${\delta }_g^T=\frac{\partial Loss}{\partial g^T},{\delta }_h^T=\frac{\partial Loss}{\partial h^T},{\delta }_s^T=\frac{\partial L}{\partial s^T}$; 6  Compute gradients of $W_s,W_h,W_g$, by using Equation (17)–(19); 7  Update W by Equation (5); 8  Repeat to line 2 until converge.

3. Experiments Results and Discussion

3.1. Experiment Data Processing and Experiment Settings

Our dataset contained an entire year’s electrical load of the 2016 Electric Power Company in Sichuan Province, China. The data values distribution is shown in Figure 6. The average, variance, standard deviation and coefficient of variation are 16,965.71, 11,692,942.24, 3419.49 and 4.96, respectively. To avoid overlap between the training data and test data, all the data were sorted by time, and the first three-quarters of the total data was chosen as our training data and the remainder as our test data. It means that we used nine months’ load values as the train date, the rest of the three months’ load values as test sets. Because, for training neural networks, it is suggested to use raw data, we don’t use any regularization or pay special attention to the special days in the whole years such as holidays and Chinese New Year day. The original data recorded electrical load with a 1 min interval. According to the data processing method in [7], the data were sampled for different time scales. For a better analysis of neural network performance, the data were sampled for 5-min, 20-min, 30-min, and 40-min time scales after dividing the training data and test data into different time scales. The data size of different time scales’ train dataset and test dataset are shown in

Table 1. The data size of different time scales.

Dataset	5 min	20 min	30 min	40 min
train	78,624	19,656	13,104	9828
test	26,496	6624	4416	3312

. It can be visualized as in Figure 7. As we can see, with the time scale become larger, the size of dataset become smaller. Because of sampling by different time scales, the data have been divided into different sizes. Regarding training the BPNN, the inputs of the neural networks were the last 10 samples and the output was the prediction value of the 11th moment load value. For the RNN series methods, the input was the previous nine load values and the current load value, and the output was the next time predicted load value. All the input values are the nearest ten values and ordered in time; this is the same way as mentioned in [35,43]. The complete structures of ANNs are shown in Figure 8. After predicting the eleventh load value, the second value to the eleventh value is chosen to predict the twelfth value. We iterate this procedure until finishing the prediction of the last value. The prediction mechanism of the neural network is shown in Figure 9.

In the experiment, the following criteria were used to evaluate all the mentioned methods: the root mean square error (RMSE), mean absolute error (MAE) and mean absolute prercentage error (MAPE), which are widely used in STLF [1,7,17]. They are calculated as follows:

$$RMSE=\sqrt{\frac{1}{N}\sum _{i=1}^N{(x_i-\dot{x_i})}^2},$$

(20)

$$MAE=\frac{1}{N}\sum _{i=1}^N\left|x_i-\dot{x_i}\right|,$$

(21)

$$MAPE=\frac{1}{N}\sum _{i=1}^N\left|\frac{x_i-\dot{x_i}}{x_i}\right|\times 100\%,$$

(22)

where $x_i$ is the actual load value, $\dot{x_i}$ is the predicted load value, N is the number of test samples. These criteria represent three types of deviation between the forecast and actual values: the smaller the criteria, the higher the forecasting accuracy. MAE is the basic metric for STLF, RMSE is sensitive to the regression point with large deviation, and MAPE considers both the error and the ratio which are between the predicted value and true value.

In order to balance performance and accuracy, we tried several trials to decide the size of all the neural networks in a 30-min time scale data set, the trails’ results on 30-min test set are shown in

Table 2. Results of different hidden layer sizes in 30-min time scale test set.

Number of Hidden Neurons	RMSE	MAPE	MAE
BP_32	585.22	2.38	439.96
BP_64	529.23	2.12	393.11
BP_128	581.59	2.21	408.03
GATE_2	524.75	2.02	374.16
GATE_4	530.09	2.00	372.40
GATE_8	520.05	1.83	337.12
LSTM_2	1323.30	5.58	1041.50
LSTM_4	1443.63	6.16	1161.02
LSTM_8	1800.25	8.17	1467.79
RNN_2	518.58	1.97	365.87
RNN_4	506.58	1.85	341.61
RNN_8	518.02	1.87	343.78

. The number after the methods is the number of neurons. We have tried a three-layer BPNN architecture, and the results show it can forecast the load values well when the number of hidden neurons is 64. More or less hidden neurons will cause a decrease in performance. It means that a three-layer BPNN with 64 hidden neurons is a good choice for this load forecast task. Therefore, we have not added more layers into BPNN. The same as with BPNN, from the results, we can see that one recurrent layer with four neurons is a good choice for this load forecast task. The specific configuration of the mentioned neuron networks for STLF was as follows: The BPNN contained three layers. Specifically, the first layer was the input layer, which contained 10 neurons, and each input was a sampled load value; the second layer was a hidden layer whose size was 64; and the last layer was the output layer, which only contained one neuron, which represented the predicted value. The RNN, LSTM, and gate-RNN input layers contained one neuron, which was input into a 4-neuron RNN, LSTM, or gate-RNN cell, and the output was the same as that of the BPNN. The RNN cell calculated the value of the last 10 moments to predict the value of the next moment. All the networks in this paper were implemented using the Keras framework. The loss function of these methods was Equation (22). The optimizer of the neural networks was RMSProp and its parameters were set as Keras default parameters except for the learning rate. During the training process, the learning rate of epoch 1 to 5 was $0.01$ and the remaining epoches learning rate was $5\times {10}^{-5}$. The total number of epoch was 55 and the value of weight decay is $5\times {10}^{-5}$.

3.2. Comparison of the Models with the Baseline Method

To evaluate the effectiveness of the neural network methods mentioned in the section above, several most commonly used traditional methods were chosen as the baseline methods. They are SVR [22], decision tree (DT) [9], autoregressive integrated moving average model (ARIMA) [48], and one random forest (RF) proposed in [49]. SVR is one of the most popular methods that is widely used to forecast short-term load. DT, ARIMA and RF introduced in [49] are useful methods. All these methods have been used in STLF in different time scales, 5-min, 20-min, 30-min and 40-min time scale [9,48,49]. The results of all the methods test on the test set of three months are shown in

Table 3. Results of eight methods on the 5-min time scale in a three months’ test dataset.

Methods	RMSE	MAPE (%)	MAE
SVR [22]	913.61	4.61	766.82
DT [9]	145.26	0.63	133.68
ARIMA [48]	131.42	0.53	131.43
RF [49]	159.95	0.68	122.67
BP	137.69	0.59	108.62
RNN	130.69	0.55	100.12
LSTM	667.49	2.75	507.67
gate-RNN	116.94	0.49	89.36

,

Table 4. Results of eight methods on the 20-min time scale in three months’ test dataset.

Methods	RMSE	MAPE (%)	MAE
SVR [22]	880.66	4.04	729.80
DT [9]	386.30	1.63	301.47
ARIMA [48]	402.06	1.47	269.82
RF [49]	310.2	1.69	427.38
BP	451.72	1.81	334.30
RNN	403.14	1.65	304.58
LSTM	828.09	3.37	626.32
gate-RNN	337.00	1.31	242.75

,

Table 5. Results of eight methods on the 30-min time scale in three months’ test dataset.

Methods	RMSE	MAPE (%)	MAE
SVR [22]	830.00	3.80	682.88
DT [9]	517.70	2.14	394.55
ARIMA [48]	670.24	2.41	442.21
RF [49]	646.02	2.41	447.30
BP	529.23	2.12	393.11
RNN	506.58	1.85	341.61
LSTM	1443.63	6.16	1161.02
gate-RNN	530.09	2.00	372.40

and

Table 6. Results of eight methods on the 40-min time scale in three months’ test dataset.

Methods	RMSE	MAPE (%)	MAE
SVR [22]	1155.91	5.65	979.10
DT [9]	701.78	3.02	553.725
ARIMA [48]	818.07	3.43	628.24
RF [49]	943.62	3.48	647.44
BP	715.68	2.85	527.99
RNN	762.60	2.91	537.40
LSTM	2056.35	8.49	1598.33
gate-RNN	792.15	3.23	601.51

. From these tables, we can see that among the eight methods, the neural network methods achieved better comprehensive performance compared with other traditional methods in all different time scales, except LSTM. In a smaller time scale, specifically in 5-min and 20-min, gate-RNN achieved better performance than any other methods. ARIMA achieved the best performance among all the traditional methods. On a larger time scale, 30-min and 40-min, BP achieved better comprehensive performance than any other methods. Meanwhile, DT achieved the best performance among all traditional methods. It has shown that the performance of methods is influenced by the time scale.

To clearly illustrate the forecast value, from the test set of three months, we chose 1000 time scales’ actual and predicted electric load values obtained by the eight methods to draw a figure on a 30-min time scale, which is shown in Figure 10. The orange line stands for real load values, the blue line stands for prediction values and the green line stands for the average value of real load values. Furthermore, 1000-time scales are nearly one month, from Figure 10, we can see the blue line is far away from the green line and close to the orange line. All of the methods’ prediction values are much closer to the real values; this means that the prediction values have no relations to the average value for some months and even one year. Specifically, it can be seen that the predicted values obtained by the LSTM method have the maximum error for the original data compared with the other seven methods; particularly, the other three types of neural network methods. This is because the structure of LSTM is much more complicated than the other three types of neural networks. To achieve equivalent performance, LSTM needs much more data than BP, RNN, and gate-RNN. It is difficult to train an LSTM for STLF using only one year of load data. Additionally, we plot the error distribution of each model in 30-min time scale, which is shown in Figure 11. RNN and gate-RNN are more accurate and effective, while the errors of other methods are scattered in a wider distribution space.

3.3. Performance Analysis of the Neural Networks

In this section, we analyze the performance of four neural network methods which include the training process of different neural networks and the performance of the neural networks. To better compare these methods, we used four time scales to process the data: 5 min, 20 min, 30 min, and 40 min.

3.3.1. Training Process Analysis for the Neural Networks

To assess whether the neural networks were trained and learned the distribution of data, a direct approach is to assess whether the train loss and test loss converged. From Figure 12, Figure 13, Figure 14 and Figure 15, we can see that the RNN and gate-RNN train loss and test loss converged well for all time scales; LSTM had large fluctuations and BP had slight fluctuations. For the 5-min time scale, even though LSTM had a large fluctuation, all the methods achieved a small train loss and test loss, and the trends of the train loss and test loss were the same. This means that all the methods learned the distribution of train data and achieved good performance on the test set. When the time scale was larger than 5 min, even though the trends of the train loss and test loss were the same, the convergence values of the train loss and test loss were different, and the losses increased as the time scale increased. This means that, as the time scale increased, the performance of the neural network became worse.

From Figure 12, Figure 13, Figure 14 and Figure 15, we also can find that our proposed method converged after training in approximately five epochs, which was faster than any of the other methods. BP converged in approximately ten epochs and RNN converged in approximately seven epochs. This means that our proposed method had significant power to forecast ultra short-term load. We believe that it benefits from the suitable gate mechanism. The RNN method has no gate mechanism; it cannot choose the memory information, but the LSTM method was too complicated to train on the data.

3.3.2. Examination of All the Neural Networks

After analyzing the train loss and test loss, we analyzed the criteria for each method for different time scales. The results for the four neural networks are shown in

Table 7. Results of the RMSE for different time scales.

Methods	5 min	20 min	30 min	40 min
BP	137.69	451.72	529.23	715.68
RNN	130.69	403.14	506.58	762.60
LSTM	667.49	828.09	1443.63	2056.35
gate-RNN	116.94	337.00	530.09	792.15

,

Table 8. Results of the MAPE for different time scales.

Methods	5 min	20 min	30 min	40 min
BP	0.59	1.81	2.12	2.85
RNN	0.55	1.65	1.85	2.91
LSTM	2.75	3.37	6.16	8.49
gate-RNN	0.49	1.31	2.00	3.23

and

Table 9. Results of the MAE for different time scales.

Methods	5 min	20 min	30 min	40 min
BP	108.62	334.3	393.11	527.99
RNN	100.12	304.58	341.61	537.40
LSTM	507.67	626.32	1161.02	1598.33
gate-RNN	89.36	242.75	372.40	601.51

. Table 7, Table 8 and Table 9 which present the RMSE, MAPE, and MAE, respectively, for four time scales. Table 7 shows that gate-RNN achieved the best performance for the 5-min and 20-min time scales; RNN achieved the best performance for the 30-min time scale; and BP achieved the best performance for the 40-min time scale. The results in Table 8 and Table 9 show the same performance as Table 7 where gate-RNN achieved the best performance for the 5-min and 20-min time scales; RNN achieved the best performance for the 30-min time scale; and BP achieved the best performance for the 40-min time scale. Overall, gate-RNN achieved good results among all the four methods. This proves that our proposed method is better at forecasting ultra short-term load again.

Further visualization results are shown in Figure 16. We can see it intuitively that the performance of all the methods became worse as the time scale increased. Overall, all the methods had a similar trend: they began at a relatively small value and ended with a large value. Figure 16 also shows that our method achieved the best results for the criteria for small time scales, but, as the time scales increased, our method’s performance became slightly worse. Overall, the performance of LSTM for all criteria for different time scales was worse, and the values for the criteria were far from those of the other neural network methods.

We believe that this scenario was caused by the following main points. First, we consider the reason that the data is time scale. The larger the time scale, the more likely that unpredictable states will affect the load value, such as sudden weather changes and holidays. Even though the neural networks could predict the trend of the load, the values of the load would have a large turbulence. Second, we consider the neural networks’ features. As we know, the complexity of the structures increases gradually in the following order: BP, RNN, gate-RNN, and LSTM. The more complex the architecture of a neural network, the more difficult it is to train it, and the more data are needed to train it. After we process the data into different time scales, the data size changes according to the time scale. For example, the number of total data items is $T^{total}$ recorded every 1 min, so, when our time scale is 5 min, the data items become $T^{5min}$, which is $\frac{1}{5}$ of the total items, that is, five times smaller than $T^{total}$. All of the time scales can be represented by $T^{total}=5\times T^{5min}=20\times T^{20min}=30\times T^{30min}=40\times T^{40min}$. Thus, when using these different time scales to process data to train the networks, the actual data size decreased with the time scale, and then the performance became worse with an increase in the time scale.

4. Conclusions

The aim of our work was to explore the performance of different types of neural networks in the STLF, and, based on the insufficiency of current neural networks, we proposed a new neural network architecture called the gate-RNN. Our proposed method is extremely suitable for handling ultra STLF. From the experimental results, we found that neural networks have the STLF ability. They can achieve better performance than the tranditional methods. However, the more complex the neural network’s architecture, the more data are needed to train the neural network. Considering that different neural networks have different characteristics, for our dataset, the gate-RNN achieved the highest score overall. For a large time scale, the BP achieved better performance than RNNs.

In future research, we plan to improve our work in two aspects. First, we will extend our data to multivariate data. Collecting more electrical load data and adding more relevant factors can influence load forecasting, for example, weather and some breaking news information. Second, based on the gate-RNN, to train the neural network more efficiently, we plan to design a loss function that can reflect the forecasting value online.