Transformers-Based Encoder Model for Forecasting Hourly Power Output of Transparent Photovoltaic Module Systems

Abstract

Solar power generation is usually affected by different meteorological factors, such as solar radiation, cloud cover, rainfall, and temperature. This variability has shown a negative impact on the large-scale integration of solar energy into energy supply systems. For successful integration of solar energy into the electrical grid, it is necessary to predict the accurate power generation by solar panels. In this work, solar power generation forecasting for two types of solar system (non-transparent and transparent panels) was configured by the smart artificial intelligence (AI) modelling. For deep learning models, the dataset obtained from the target value of electricity generation in kWh and other features, such as weather conditions, solar radiance, and insolation. In PV power generation values from non-transparent and transparent solar panels were collected from 1 January to 31 December 2021 with an hourly interval. To prove the efficiency of the proposed model, several deep learning approaches RNN models, such as LSTM, GRU, and transformers models, were implemented. Transformers model for forecasting power generation expressed the best model for non-transparent and transparent solar panels with lower error rates for MAE 0.05 and 0.04, and RMSE 0.24 and 0.21, respectively. The proposed model showed efficient performance and proved effective in forecasting time-series data.

1. Introduction

Maintaining a balance between efficiency and energy generation is one of the essentials for the reliable operation of solar power plants (SPP) [1]. The present solar generation facilities cannot guarantee the total electric power output at specified time. It is important that the forecast of the amount of energy generation and supply to the grid is highly relevant. Few research works have recently introduced the concept of energy meteorology (“energy-meteorology”) [2] as a new scientific discipline, the subject of which is the quantitative assessment of power generation based on renewable energy sources (RES) especially solar energy at hourly to minutes time intervals. In particular, several attentions have been paid to the hourly short-term forecast for solar energy generation forecasting. It is known that solar energy has a lot of advantages over other traditional energy sources, such as coal and natural gas; however, the power generation by photovoltaic arrays is highly variable [3,4]. Generally, the variability in solar power generation depends not only on solar radiation and temperature, but also on other meteorological factors, such as wind speed, hours of sunshine, humidity, cloud cover, and precipitation [5]. In addition, the Sun is an intermittent power source as it absorbs sunlight which is normally available in daytime. Such variability and intermittency of solar energy still challenge its large-scale integration into the energy system. So far, the energy from sun faces the unexpected changes by unwanted factors and negatively affecting the balance of the network, which resulting in increasing operating costs. Due to its irregular and unmanageable nature, an accurate solar power generation forecast is essential for the grid operator and companies supplying electrical energy from a photovoltaic system. Therefore, it is essential to develop an algorithm to maintain the stability of the power supply system. System stability can be achieved by creating methods to predict generated power and estimate future production. There are still several flaws and dependencies in the current PV power generation systems, such as temperature, humidity, and wind, which are typically caused by solar irradiation [6].

Solar energy generation forecasting methods are divided into four categories:

• Meteorological methods: Indirect methods based on numerical weather forecasting and processing of satellite images allow first to predict the intensity of solar radiation, then convert it into the output power of a photovoltaic system.
• Statistical methods: These methods use statistical approaches, such as the autoregressive moving average model (ARMA) [7], auto-regressive integrated moving average model (ARIMA) [8], and exponential smoothing (ES) [9]. These models can be used to directly predict the power output of photovoltaic arrays without the need first to predict solar radiation.
• Machine learning methods: These approaches use machine learning algorithms, such as k-nearest neighbors, neural networks (NN) [10], support vector machine (SVR) [11], and pattern sequence-based forecasting (PSF) [12], to predict output directly the power of photovoltaic batteries. As a rule, there are two approaches to applying machine learning methods: building one forecasting model or combining several forecasting models to form an ensemble of forecasting models.
• Hybrid methods: These methods combine models or different components from the previous three categories. Unlike ensembles that combine machine learning models, hybrid ones typically combine machine learning meteorological, and statistical models or their components. The use of machine learning methods, such as RNN [13] and SVR, and statistical methods, such as ARIMA and ES, are widespread for building models to predict solar plant capacity. However, most of these methods are based on one general prediction model for all meteorological conditions and their corresponding daily photovoltaic characteristics.

This work is focused on investigating the performance of existing modern deep learning methods for predicting solar power generation and create new methods for better forecasting results with improved performance. The time-series prediction model proposed in this study has a transformer structure. The original transformer architecture (Vanilla Transformer) [14] is an auto-encoder. The encoder receives a sequence with positional information as input. The decoder receives part of this sequence and the encoder output. However, the architecture of some transformer-based models consists only of an encoder. As a result, transformers maintain direct links to all previous temporal values, allowing information to reproduce over longer sequences. Another work related to transformer model has been studied [15] on the problem of forecasting influenza-like diseases. The author compared some results of various models working with time series. The base architecture is the same as the original transformers model. Adam was used as an optimizer in this work. For regularization, the authors added dropout and a dropout rate of 0.2 for each layer. In the other work [16], the authors applied the idea of creating a generative model for forecasting, where they applied generative adversarial neural networks. As a result, the error of a conventional transformer is about three times less than ARIMA (0.036). Herein, PV power generation forecasting for two solar panels (non-transparent and transparent) has been done by deep learning approaches in which some working AI models with time series are compared. Among various working models, the architecture original transformer shows the best AI model for both solar panels.

2. Experimental and Methods

2.1. Dataset

The original PV data was collected at New and Renewable Energy Materials Development center (NewREC) located in Buan-gun, Jeollabuk-do, Republic of Korea. In this work, the PV power generation data was extracted from two different solar modules, i.e., non-transparent and transparent modules, as shown in Figure 1 and details of modules summarized in

Table 1. Physical characteristics of PV modules.

Type	Non-Transparent	Transparent
Module output	400 W	405 W
Module size	2039 × 1001 × 40 mm	2039 × 1001 × 40 mm
Module weight	22.2 kg	22 kg

. The period of collected PV data is from January to December 2021 on an hourly basis and contains weather-related information along with module energy generation outputs.

The main reason for collecting PV power generation data from two different modules in this work is to evaluate their different energy generation capability. In the case of a transparent module, the energy output is slightly higher than that of the non-transparent module due to its capability to absorb solar radiation from both sides. Bifacial transparent solar modules allow photovoltaic cells to capture reflected sunlight on the back surface. The light passes right through and collides, then bounces back towards the panels with a highly reflective surface (white roof, sandy, or rocky light soil). In our case, the surface of installed solar panels is a cement, which can reflect solar irradiation with a higher reflection rate. Table 1 shows the physical characteristics of PV modules. As seen in Figure 2, the transparent PV panels present the higher PV power generation as compared to non-transparent PV panels during the same period.

The weather data is utilized as a feature during the training process for energy forecasting, as shown in Figure 3.

2.2. Attention Based Encoder

Various computational algorithms have been developed at the academic and private sectors to increase forecasting accuracy. Transformers, a new technique in neural networks develop non-recursive modeling algorithms that specialize in predicting future elements within sequences. Transformers are initially positioned as a neural network for processing and understanding natural language [17]. One of the simplest examples is Google’s BERT language model, which is developed in 2018 [18]. However, in the four years since their inception, they have gained popularity, and have appeared in many other aspects of Deep Learning.

The traditional models, such as RNN models, are hard to train due to vanishing/exploding gradient problems, wherein older inputs have almost no effect on the output of the current stage. Additionally, the data in these models process sequentially, resulting in a poor possibility of parallelization. Even though using more powerful GPUs will not increase the speed of the training process. As a result, the processing a large amount of data is difficult. Utilizing Transformers architecture has considerable advantages over traditional RNN models. The main advantages of Transformers over LSTM [19] are reduced complexity by eliminating recurrence and parallelization technique, which improves overall computation efficiency.

In this work, the Transformers model was implemented for time-series PV power generation forecasting, and, furthermore, compared results with traditional recurrent neural network models, such as LSTM and GRU [20].

The advantage of RNN over traditional Feed-Forward Networks (FFN) is saving the input of the current layer and feeding it back in the loop form to predict the layer’s output. RNN is widely applied to solve issues of Feed-Forward Networks where FFN cannot handle sequential data. RNN also accepts not only current input, but previously received input as well. It is due to RNN’s internal memory, which memorizes historical information. However, these advantages over FFN may result in some drawbacks [21]. The Vanishing Gradient problem is the main disadvantage of using RNNs, where gradients used to compute weights become closer to zero. The deeper the model, the closer gradient to zero, resulting in a much longer computation time. The occurrence of this problem is due to certain activation functions, such as tangent hyperbolic or sigmoid functions, which squishes large input space into small output space.

To resolve these problems, a Transformer-based network is implemented, in which the recurrence is not utilized for sequential calculation. Moreover, its computation is parallelized, resulting in the acceleration of the training process. The transformer has an architecture in which several encoders are primarily accumulated. An encoder is called a Multi-Head Attention. It consists of a layer and a Feed Forward Neural Network layer. As mentioned above, the Attention module has parallelized computation repeatedly, and each repeated process is called an ‘attention head’. Primarily, the attention module has its Query, Key, and Value parameters, which are passed through the scaled dot product attention independently. Afterward, every Head is concatenated and combined with a final weight matrix. This technique is called Multi-Head Attention and is powerful for encoding multivariate time series. (Refer to [4] for more details). Figure 4A,B visualizes the implemented model with the original encoder algorithm.

The mathematical formula can be defined as:

$$\mathrm{Multihead}\left(\mathrm{Q},\mathrm{K},\mathrm{V}\right)=\mathrm{Concat}\left({\mathrm{head}}_1,\dots ,{\mathrm{head}}_{\mathrm{h}}\right){\mathrm{W}}^{\mathrm{O}}$$

(1)

where $hea{d}_i=Attention\left(Q{W}_i^Q,K{W}_i^K,V{W}_i^V\right)$.

Learnable parameters of Multi-Head Attention layer:

$$W_{1..h}^Q\in R^{D\times d_k},W_{1..h}^K\in R^{D\times d_k},W_{1..h}^V\in R^{D\times d_v} and W^O\in R^{h\ast d_k\times d_{out}}$$

(2)

where D is the input dimensionality.

The overall workflow of the power generation forecasting system is described in Figure 5. In this work, the pre-processing stage required observation of the collected dataset to analyze all dependent variables. Dependent variables as features are wind, humidity, temperature, solar radiance, solar insolation, and time. In the second step, the data divided into two parts as collected two target variables (label) from non-transparent and transparent type solar modules. Afterward, thoroughly inspecting the raw dataset, the pre-processed dataset was split into training and test subsets in random shifted order. Furthermore, the training subset feeds into the model for the training process and accuracy results obtained on the test dataset.

3. Results and Discussions

3.1. Performance Evaluation

Transformer models have shown state-of-the-art performance on many time series forecasting problems [22,23]. To evaluate the effectiveness of the proposed method, experiments were conducted using one year of data (from 1 January to 31 December 2021) from NewREC located in Buan-gun, Republic of Korea. For each day, data were collected only during daylight hours, from 5 am to 7 pm. The original PV data is collected at one-hour intervals. Upon completion of the prediction process on the test dataset, the result is compared with the actual and predicted values with implemented Deep Learning models, including LSTM, GRU, and Transformer models.

To evaluate the accuracy of the implemented model on the test dataset, we utilized MAE as the loss function and RMSE and MSE as accuracy metrics, as these are the primarily used metrics in the time-series models. These metrics provide a good summary of the model’s capability to make a prediction on unseen data.

The loss function can be expressed as:

$$\mathrm{MAE}=\frac{{{\displaystyle \sum }}_{i=1}^n\left|y_i-\widehat{y_i}\right|}{n}$$

(3)

where $y_i$ is the actual value and $\widehat{y_i}$ is the predicted one. The mean absolute error MAE is the average of the absolute errors between the predicted and actual values, reflecting the actual predicted error. The lower the error value, the better the model.

Two coefficients used to assess the accuracy of predictive models are presented in formulas (4) and (5), respectively:

$$\mathrm{MSE}=\frac{1}{n}{\displaystyle \sum }_{i=1}^N{\left(y_i-\widehat{y_i}\right)}^2$$

(4)

where N is the number of observed samples, $y_i$ is the actual number, and $\widehat{y_i}$ is the predicted value by the model. The essence of the metric is to minimize the sum of squared deviations of the actual values from the predicted ones (Sum of Squared Errors).

$$\mathrm{RMSE}=\sqrt{\frac{{{\displaystyle \sum }}_{i=1}^N{\left(y_i-\widehat{y_i}\right)}^2}{N}}$$

(5)

RMSE error is simply the square root of the MSE error. Unlike MSE, it has the same units as the original values of the dataset. As a result, it also operates with smaller absolute values and has a faster computation time.

Table 2. Evaluation metric results for Non-Transparent and Transparent modules.

Non-Transparent Module				Transparent Module
Model/Metrics	MSE	MAE	RMSE	Model/Metrics	MSE	MAE	RMSE
LSTM	0.070	0.217	0.27	LSTM	0.057	0.19	0.23
GRU	0.657	0.210	0.26	GRU	0.052	0.18	0.22
Transformers	0.057	0.187	0.24	Transformers	0.04	0.17	0.21

presents the overall metrical results of the implemented models (traditional RNN models-LSTM, GRU, and Transformers) for non-transparent and transparent solar panels. Additionally, the accuracy results of traditional RNN models-LSTM and GRU, and our best implemented Transformer model summarized in terms of MSE, MAE, and RMSE values. It is seen that the implemented Transformer model outperforms the other two models in terms of the accuracy of forecasted values.

3.2. Visualization Results

Upon completion of the training process, the result is compared with the actual values of the test dataset. We visualized the test results of the implemented Transformer model in Figure 6 and Figure 7 for both Transparent and non-Transparent module types, respectively. Figures show the actual hourly generated electricity compared to the forecasting models’ predictions. We can observe that forecasts match the actual data, particularly in the winter season, when the system does not generate much power.

3.3. Feature Importance

In our dataset, we have six features to train our model. The followings are temperature, wind speed, wind direction, humidity, irradiance, and insolation. It is a well-known fact that not all dataset features will equally impact the performance of the trained model. Therefore, determining which features have the most predictive power is another critical step in model-building. A higher score on the feature means a more significant effect on the model`s ability to predict target variables.

Figure 8 demonstrates the average impact value of each feature of our dataset. The top three features with the highest impact are irradiance, insolation, and temperature. By analyzing this data, one can understand the relationship between the features and target variables. In support, the low impact features on the model can be deleted after checking the relationship with variables. As a result, the model would become lighter and increasing the speed the model’s prediction of the target variables.

To further analyze the impact score of each feature, each instance of the test dataset on the model’s performance using SHAP [24] values have been observed. Figure 9 demonstrates the feature importance analysis of all 708 instances of the test dataset. As mentioned above, the most critical features are irradiance, insolation, and temperature, and most of their instances positively affect the prediction ability of the model. However, it can be seen that some of the instances are negative numbers, which means the overall accuracy loss.

4. Conclusions

This research presents a method for predicting the PV power generation of photovoltaic stations in eastern part of Korea by the Transformers algorithm. The original dataset for deep learning models is obtained from the target value of electricity generation in kWh and other features, such as weather conditions, solar radiance, and insolation. All PV power generation data were collected from two operational PV panels, such as non-transparent and transparent solar panels. Deep learning models with time series algorithms were designed to forecast the hourly PV power generation of non-transparent and transparent solar panel systems. The study results show that the developed predictive model has a mean absolute error (MAE) of approximately 0.18 kWh and 0.17 kWh for both non-transparent and transparent type modules over the predicted interval. The root mean square error of the model (RMSE) is 0.24 kWh and 0.21 kWh, respectively. The developed model makes it possible to increase the efficiency of the operation of photovoltaic stations and reduce economic losses. In addition to a higher power generation capability, the transparent module is less prone to be heated up, making it possible to do without cooling devices and lowering the cost of building a solar system overall. Moreover, we can further improve the predictive model to reduce the prediction error by considering accumulated experience, more detailed analysis of the input data, and precisely tuning neural networking layers.