Convolutional Neural Network for High-Resolution Cloud Motion Prediction from Hemispheric Sky Images

Abstract

A novel high-resolution method for forecasting cloud motion from all-sky images using deep learning is presented. A convolutional neural network (CNN) was created and trained with more than two years of all-sky images, recorded by a hemispheric sky imager (HSI) at the Institute of Meteorology and Climatology (IMUK) of the Leibniz Universität Hannover, Hannover, Germany. Using the haze indexpostprocessing algorithm, cloud characteristics were found, and the deformation vector of each cloud was performed and used as ground truth. The CNN training process was built to predict cloud motion up to 10 min ahead, in a sequence of HSI images, tracking clouds frame by frame. The first two simulated minutes show a strong similarity between simulated and measured cloud motion, which allows photovoltaic (PV) companies to make accurate horizon time predictions and better marketing decisions for primary and secondary control reserves. This cloud motion algorithm principally targets global irradiance predictions as an application for electrical engineering and in PV output predictions. Comparisons between the results of the predicted region of interest of a cloud by the proposed method and real cloud position show a mean Sørensen–Dice similarity coefficient (SD) of 94 ± 2.6% (mean ± standard deviation) for the first minute, outperforming the persistence model (89 ± 3.8%). As the forecast time window increased the index decreased to 44.4 ± 12.3% for the CNN and 37.8 ± 16.4% for the persistence model for 10 min ahead forecast. In addition, up to 10 min global horizontal irradiance was also derived using a feed-forward artificial neural network technique for each CNN forecasted image. Therefore, the new algorithm presented here increases the SD approximately 15% compared to the reference persistence model.

1. Introduction

Short-time cloud motion prediction has a huge impact on the future behavior of the power generation output of solar photovoltaic (PV) power plants [1]. Clouds are a major modulator of the global horizontal irradiance (GHI) and a source of severe fluctuation when, for example, passing in front of the sun. Clouds can even increase the solar radiation at the surface by reflection and/or forward scattering [2,3,4]. To compensate for these ramp events, very short-term forecasting/forecasts can help power plant operators to accurately manage PV power plants. The analyses of clouds play an important role in both scientific and business enterprises, where these severe fluctuations in the energy output are incompatible with the established safety standards for the electricity distribution systems [5].

In this context, the introduction of hemispheric sky imager (HSI) systems as efficient ground surface equipment for cloud data assessment have already been proven by various authors [6]. However, even with good high-resolution cloud detections, cloud movement forecast is still a topic of research due to its high degree of complexity [7,8,9]. Figure 1 shows how quickly cloud changes can occur within three minutes.

Looking at cloud detection methods, we can mention threshold-based algorithm [10] and machine learning methods [11,12,13,14,15]. Threshold-based algorithms normally use a red/blue ratio of the three RGB (red, blue, and green) channels from the image pixels for cloud classification [16,17,18,19]. Cloud pixels are identified as high R and B values, while sky pixels have low R and high B values. However, this method has some weaknesses, primarily distinguishing or detecting clouds near the horizon and close to the sun [20]. In addition, different pattern recognition algorithms have been developed. Ren and Malek [21] proposed a cloud segmentation algorithm utilizing superpixel. This algorithm divides the image into blocks (or clusters) and the division is based mainly in the continuity of cloud contours, the texture and brightness of each pixel. A hybrid framework has been proposed to forecast hourly global solar radiation [22]. This approach includes two different methods: support vector machine and machine learning techniques. The results showed that it is possible to predict next-day hourly values of solar radiation by reducing the root mean absolute error ($rMAE$) by 15.2% compared to the reference persistence model.

Machine learning methods have been used successfully for cloud detection [23] and cloud coverage estimations [24]. Crisosto [25] developed a method using HSI images to predict cloud concentrations one minute in advance using artificial neural networks (ANNs). The results showed a 30% reduction of errors when compared to the persistence model under diverse cloud conditions. In addition, a similar method has been used as an important step for predicting GHI one hour in advance with one-minute intervals [26]. Other advanced and sophisticated techniques, like convolutional neural networks (CNNs), have been developed and applied in recent years to forecast solar irradiance [27] by offering significant advantages for large image datasets [28], evaluating the non-linearity and other more complex relationships [29].

The main objective of this work is to propose a preliminary pre-processing method required to target solar irradiance predictions that allows companies to make more accurate horizon time predictions. The CNN algorithm can be important for very short-term GHI forecasts, and subsequently better marketing decisions for primary and secondary control reserve (cloud position and the GHI up to 5 min in advance). In this paper, deformation cloud vectors 10 min ahead were determined under different cloud and all-weather conditions. We applied an ANN technique to estimate the respective GHI of the forecasted cloud clusters, for methodology validation. Section 2 briefly describes the data acquisition methods. The methodologies of this study are described in Section 3. The results are given in Section 4. The conclusions and future work are discussed in Section 5.

2. Data

The HSI equipment used was a digital compact charge-coupled device camera and a fish-eye objective with a field of view of 183° inside a weatherproof box, which provided hemispherical images of the entire sky [30]. Exposure time was 1000/s, and there was an acquisition time of 1 min between each image. In total, 150,000 pictures were produced between 2014 and 2016. From these 150,000 manually segmented images, 5000 were selected for testing (i.e., these pictures were independent of the training data). The system is installed on the roof of the Institute of Meteorology and Climatology (IMUK) of the Leibniz University Hannover in Germany (52.4° N, 9.7° E). Completely overcast images were not used in this analysis, since GHI values under 100 $\mathrm{W}/{\mathrm{m}}^2$ are usually not relevant for the production of solar energy and we were more interested in larger GHI ramp effects. In addition, the GHI data was obtained using a CMP11 pyranometer (Kipp & Zonen, Delft, The Netherlands) [31].

3. Methodology

3.1. Cloud Identification

The method used to identify and separate clouds and sky pixels is an improved sky index image-processing algorithm [32]. The haze index consists of identifying cloud pixels combining the red, blue, and green channels, as detailed by Schrempf [33], and serves as an improvement for hazed areas. Every pixel is then classified as cloudy or clear sky based on a threshold (see Figure 3). Equation (1) presents the haze index, which is applied only to hazed areas, based on thresholds of the sky index.

$$Haze Index=\frac{\frac{{\mathrm{count}}_{\mathrm{red}}+{\mathrm{count}}_{\mathrm{blue}}}{2}-{\mathrm{count}}_{\mathrm{green}}}{\frac{{\mathrm{count}}_{\mathrm{red}}+{\mathrm{count}}_{\mathrm{blue}}}{2}+{\mathrm{count}}_{\mathrm{green}}}$$

(1)

3.2. Semantic Segmentation (Acquiring Labelled Data)

Deep learning and specifically CNNs have drastically improved the way in which intelligent algorithms learn. With convolutional layers, pooling layers, and fully connected layers, CNNs allow computational models to represent data with multiple levels of abstraction [34]. With the automatic cloud–sky separation derived by the haze index algorithm, the automatic cloud segmentation is realized. Therefore, cloud clusters are labelled as ground truth for the automatic segmentation, and thus, for further cloud motion forecasts. Sky clusters were not taken into consideration. Figure 2 shows the process of acquiring the ground truth for the input parameters of the CNN. The first column shows the original image. The results of the haze index algorithm can be seen in the second column. The third column shows the two classes: cloud clusters in white and sky clusters in gray. We can see how the CNN learns to recognize different regions of interest (ROIs) for further simulations. The training process consists of learning how clouds can change frame by frame consecutively.

3.3. CNN Development and Training

Once the ROIs were identified by the haze index, they were used as ground truth parameters for training the CNN. The input parameters were the original HIS images and their corresponding cloud clusters (see Figure 2). The network accepts one HIS image (.jpg) and one cloud cluster image (.jpg) and learns exactly, frame by frame, where the clouds are.

The input layers were resized to 256 × 256, max pooling of 2 × 2 resulting in an output layer of 256 × 256. In the training phase, a pre-trained CNN for classification and detection (VGG-16) [34] was selected and extended to automatically learn the ROI changes in whole pictures frame by frame. The training process finished when the network learned as accurately as possible the ROI changes in that time. The binary cross entropy function was minimized during the training process, and the activation layers were very simple rectified linear units, or $\mathrm{ReLUs}$, defined as $\mathrm{ReLU}\left(\mathrm{z}\right)=\max \left(0,z\right)$ or variants of this function proposed by He et al. [35]. Adaptative moment estimation (Adam) [36] was chosen as a stochastic optimization method and the batch size was 16. We trained the model using 100 epochs

Once the trained network was ready (the network learned the cloud movements frame by frame in more than 145,000 images), the simulation phase began. The program first identified the location of the current ROI at $t$, and after saving this information, went back 5 min ($t-5\Delta t$). Then the program went forward, frame by frame, to forecast the new ROI at $t+\Delta t$ by applying probabilistic accuracies learned in the training phase. In other words, the trained network delivers frame by frame the best matching cloud location, and the output is the new (estimated) ROI (cloud location) for the next minute. Furthermore, the ROI estimated for $t+\Delta t$ will be the base ROI for $t+2\Delta t$, and so on.

3.4. An Artificial Neural Network Used for Validating Our Model

To validate the results of our algorithm, we applied an extra artificial neural network (ANN), as explained by Crisosto et al. [26]. This ANN only needs an HSI image to predict GHI one hour ahead in one-minute resolution. This input image is the output of our CNN method. Therefore, the new predicted ROIs of our algorithm were fed into the ANN as input parameters to derive their correspondent GHI values 10 min ahead.

3.5. Statistical Metrics

The Sørensen–Dice ($SD$) similarity coefficient [37] and overlap ($VO$) [38] were used for the method evaluation. $SD$ is defined as the division between twice the number of elements common to both sets and the sum of the number of elements in each set (Equation (2)). $VO$ is defined as a quotient of the intersection of both X and Y segmentations (Equation (3))

$$SD\left(X,Y\right)=\frac{2|X\cap Y|}{\left|X\right| + \left|Y\right|}$$

(2)

$$VO\left(X,Y\right)=\frac{|X\cap Y|}{|X\cup Y|}$$

(3)

where $\left|X\right| \mathrm{and} \left|Y\right|$ is the cardinality of two sets. The mathematical definition of the root mean square error (RMSE) and the coefficient of determination ($R^2$) are expressed as follows:

$$RMSE=\sqrt{\frac{1}{N}{\displaystyle \sum }_{i=1}^n\left(y_i-x_i\right)2}$$

(4)

$$R^2=\raisebox{1ex}{$\left({{\displaystyle \sum }}_{i=1}^N\left(y_i-\overline{y}\right)\left(x_i-\overline{x}\right)\right)$}\!\left/ \!\raisebox{-1ex}{${\left[\right({{\displaystyle \sum }}_{i=1}^N{\left(y_i-\overline{y}\right)}^2)\left({{\displaystyle \sum }}_{i=1}^N{\left(x_i-\overline{x}\right)}^2\right)]}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}$}\right.$$

(5)

where $y_i$ is the forecast value, $x_i$ is the measured value, and $N$ is the total number of samples. Additionally, $\overline{x}={{\displaystyle \sum }}_{i=1}^N{x}_i$ and $\overline{y}= {{\displaystyle \sum }}_{i=1}^N{y}_i$.

The new algorithm was compared with the scaled persistence model [39] defined as the ROI configuration vector ($ROI\text{\_}CV$) where the next minute would be identical to the current minute ($ROI\text{\_}C{V}_{t+1}=ROI\text{\_}C{V}_t$). This model is the reference model for short-term solar forecasting [40]. For the irradiance evaluation, only the movement of the sun was taken into consideration.

4. Results

For a better visual representation of the results, the following example cases were selected to show the effectiveness and efficiency of the algorithm: 15 April 2014 at 15:54, 11 July 2014 at 13:09, and 1 June 2014 at 17:42. After that, 5000 cloud images with different cloud positions were simulated and the results are presented.

4.1. Analysis of the Example Cases

Figure 3 shows the one-minute ahead simulation for the three days, including the observed (target) ROI to be simulated (column 1), the simulated ROI by our model (column 2), and the simulated ROI using the persistence model (column 3). In all cases, we can see that our CNN model performs better than the persistence for the first minute. The $SD$ values were 92.4 ± 2.9% (mean ± standard deviation), 92.7 ± 2.6%, and 88 ± 3.4%, respective to each case, while the $SD$ for the persistence was 85.8 ± 4.9%, 85.8 ± 5.6%, and 84 ± 7.1%, respectively.

The statistical comparison with forecasts up to 10 min ahead can be seen in

Table 1. Statistical indicators of the artificial neural network (ANN) model for four different time periods: the mean Sørensen–Dice similarity ($SD$) coefficient and the overlap ($VO$) for for 1-, 2-, 5-, and 10-min forecasts. CNN: convolutional neural network.

Day	Model	Statistical Parameters
		$\mathit{SD} \mathbf{\left(}\mathbf{\%}\mathbf{\right)}$				$\mathit{VO}\mathbf{\left(}\mathbf{\%}\mathbf{\right)}$
		1-min	2-min	5-min	10-min	1-min	2-min	5-min	10-min
24 March 2014 at 09:48	CNN Persistence	93 89	83 79	71 68	51 49	91 86	80 79	69 60	47 42
1 June 2014 at 17:37	CNN Persistence	92 88	82 78	69 64	57 52	90 84	81 75	62 57	49 43
24 May 2014 at 14:02	CNN Persistence	94 87	87 82	73 62	62 48	92 87	84 79	69 59	51 48
9 May 2015 at 15:26	CNN Persistence	87 83	79 71	62 58	51 49	85 80	78 70	57 55	46 40

. As expected, the performance of the algorithm decreases substantially as the simulated time progresses. In addition, our model outperforms the persistence model for the full 10-min period. For larger timescale forecasts (for example, 5 and 10 min), these results were already expected; however, for very short forecasts (for example, 1 and 2 min), our CNN shows improvements in forecasting cloud movement.

4.2. Analysis of the Simulation for All Simulated Datasets

Table 2. Comparison between the statistical indicators of the proposed CNN model and the persistence model: the mean Sørensen–Dice similarity coefficient ($SD$) and overlap ($VO$) of the 5000 simulated cases for 1-, 2-, 5-, and 10-min forecasts.

Model	Mean Statistical Parameters
	$\mathit{SD} \mathbf{\left(}\mathbf{\%}\mathbf{\right)}$				$\mathit{VO} \mathbf{\left(}\mathbf{\%}\mathbf{\right)}$
	1-min	2-min	5-min	10-min	1-min	2-min	5-min	10-min
CNN	94	83	60	49	92	80	58	43
Persistence	89	78	55	44	86	69	45	37

presents the statistical indicators of the cloud ROI changes in the 5000 tested images. Table 1 and Table 2 show the quality of the results decreasing in forecasts for longer time scales, with $SD$ values up to 44.4 ± 12.3% and $VO$ of 37.7 ± 15.3% obtained for 10-min ahead forecasts.

4.3. Application of the Presented CNN Algorithm

To validate our algorithm, we applied an ANN as described in Section 3.4. For each predicted image from our CNN, the GHI value was predicted at the same time. Figure 4 shows a comparison between the target images and simulated images one-minute ahead generated as output by the CNN and their correspondent measured and simulated GHI.

Figure 5 shows the results for two examples of 10-min ahead simulations utilizing our method as an irradiance simulations tool.

Table 3. Presentation of the statistical indicator for the comparison between our model and the persistence model. $RMSE$ and the $R^2$ of all 100 compared cases.

Model	Statistical Parameters
	$\mathit{R}\mathit{M}\mathit{S}\mathit{E} \mathbf{\left(}\mathbf{W}\mathbf{/}{\mathbf{m}}^{\mathbf{2}}\mathbf{\right)}\mathbf{\left(}\mathit{\%}\mathbf{\right)}$				${\mathit{R}}^{\mathbf{2}}$
	1-min	2-min	5-min	10-min	1-min	2-min	5-min	10-min
CNN	32	54	101	148	0.81	0.65	0.53	0.42
Persistence	45	72	125	187	0.76	0.61	0.48	0.39

shows a comparison between 100 observed images with their corresponding simulated values. Figure 6 shows the distribution of the relative deviations as a boxplot for different time horizons.

5. Conclusions

In this study, a novel method to forecast cloud motion up to 10 min ahead was presented. A convolutional neural network (CNN) was trained using hemispherical sky images as inputs, and a statistical approach for forecasting future cloud motion was performed.

According to the simulation results, the method presented here is capable of predicting cloud changes for the first minute with very high confidence using CNN, with a coefficient of determination ($R^2$) of 0.97 and a Sørensen–Dice similarity coefficient ($SD$) of 94 ± 2.6%. For the same simulated datasets, the persistence model reached an $R^2$ of 0.92 and a $SD$ of 89 ± 3.8%. The method was also tested for different forecast time scales, however, unsatisfactory results (an $R^2$ of 0.40 and a $SD$ of 49 ± 11.8%) were obtained for 10-min simulations by our model, although they were better than the results of the persistence model. In addition, the global horizontal irradiance (GHI) output results predicted by the CNN showed a forecast accuracy for the decreased amount of energy one-minute ahead, achieving a $RMSE$ of 32 ($\mathrm{W}/{\mathrm{m}}^2$) and a $R^2$ of 0.81. The persistence model achieved a $RMSE$ of 45 ($\mathrm{W}/{\mathrm{m}}^2$) and a $R^2$ of 0.76. However, for the GHI prediction for the next 10-min ahead, the $RMSE$ was 148 ($\mathrm{W}/{\mathrm{m}}^2$) and the $R^2$ was 0.42 for our model and the $RMSE$ was 187 ($\mathrm{W}/{\mathrm{m}}^2$) and the $R^2$ was 0.39 for the persistence model.

The research presented here can be used as a first step for PV companies to understand cloud movement and to implement an end-to-end forecasting system (as a pipeline) within a fully automated server with the goal of forecasting global horizontal irradiance minutes ahead. This fully automated pipeline implementation will help to allow PV companies to make accurate horizon time predictions and better marketing decisions for primary and secondary control reserves (i.e., up to 5 min in advance).

Future research is needed to better understand cloud movement through wind speed and wind direction, and also to understand how to improve forecast results for periods longer than 1 min or when the sky is totally covered. Different methodologies and maybe different analyses of data should be considered.

Despite the good results, the existence of other models offers new ways to process big data. For example, long short-term memory networks (LSTMs) appear to be an alternative. Since the architecture of these networks is more complex, LSTMs are suitable for processing long data sequences while avoiding vanishing or exploding gradients, currently problems that CNNs still have. As an outlook for further projects, the utilization of LSTM and hybrid models should be taken into consideration.