OpΕnergy: An Intelligent System for Monitoring EU Energy Strategy Using EU Open Data

Abstract

In this paper, the basic structure of an ICT platform of energy indicators, Openergy, is analytically presented, leveraging energy open data to help address the energy crisis more democratically. More specifically, its applicability as a dynamic tool for the management of climate, environmental, and socioeconomic information is described, and its efficiency in helping uncover insights for optimal data-driven decisions is depicted. Openergy uses data from the official portal for European data and the Eurostat site. Its database consists of data related to six energy categories, EU 2020 energy targets, energy balance, electricity production, transport fuels, heat production, and gas emissions, and each one includes its own indicators for EU countries. The platform includes visualizations of these data as well as time series modeling and forecasting, and the results are depicted at Openergy platform. The time series modeling provides forecasts with confidence intervals of each indicator until 2020 in each energy category. Empirical validation, RMSE, and MAE values showed that in almost all cases and estimations, the predicted values are in very good agreement with the observed values.

1. Introduction

Stable, adequate, and secure energy is vital for the sustainable development of humanity. Sustainability presupposes economic development by ensuring climate change mitigation and adaptation, environmental protection, and social cohesion, e.g., [1]. Towards this direction, energy resources that are strongly related to economic development need to be sustainably managed. The 7th Sustainable Development Goal (SDG) of the 2030 Agenda for Sustainable Development supports ensuring access to modern, sustainable, reliable, and affordable energy for all [2]. The EU demonstrates global leadership by considering energy strategies and initiatives, especially under the threat of climate change. The important pillars of EU’s energy policy are briefly (i) decarbonize towards a low-carbon economy, (ii) promote clean energy technologies, (iii) improve energy efficiency, (iv) cut emissions, (v) reduce dependence on energy import, (vi) ensure the functioning of a fully integrated internal energy market, and (vii) diversify Europe’s sources of energy. All these pillars need motives, objectives, and priorities in order to succeed in a “metamorphosis” of the energy sector.

It goes without saying that energy consumption and the corresponding pressures on the environment and climate are accompanied by many unsustainable practices that humanity adopted in the past, which are still present in the energy system, e.g., [3,4,5]. Promoting and realistically developing strategies and policies that aim to achieve an integrated energy market, security of energy supply, and a sustainable energy sector is crucial and embedded in the core of the EU’s energy policy [6]. Themes such as distributed generation energy systems and energy communities are acquiring an increasingly important role, both in scientific literature and in the construction of scenarios and policies [7]. In addition, the transformation of the EU’s energy sector should include ambitious national and EU measures and alternatives for minimizing the consumption of fossil fuel related to industrial, transport, agricultural, commercial services, tourism, and other economic sectors. Penetration of renewables (hydro, solar, wind, tide), natural gas as a transition energy option, circular economy, industrial symbiosis, and Waste-to-Bioenergy [8], the cogeneration of heat and power, are included in the set of proposals in the European Green Deal [9].

Structural transformations in the energy sector require reliable information related to all the aforementioned initiatives both at the EU and national levels. The use of Information and Communication Technology (ICT) science constitutes an important tool for assessing the progress achieved toward the transformation of the EUs’ energy system and, in general, sustainable development. Measuring and reliably predicting energy indicators such as energy consumption demand, sectoral analysis of energy demand, penetration of alternative energy sources (e.g., biofuels or renewables), energy efficiency (e.g., heating and cooling efficiency), energy intensity, or eco-efficiency of the energy sector is a major challenge for energy managers and decision-makers in the EU.

Open data principles have become an important part of the data revolution and have been identified worldwide as a key engine for achieving the 17 UN Sustainable Development Goals (SDGs) [10].

Open data in the Open Definition published by the Open Knowledge Foundation is defined as: “Open data is data that can be freely used, re-used and redistributed by anyone—subject only, at most, to the requirement to attribute and share alike.” Additionally, it must be available under an open license and provided in a modifiable form that is machine-readable [11,12].

Open data result in open government, where government acts as an open system and interacts with its environment. The data should not only be published but also actively sought to learn how governance can be improved [13].

Additionally, open data support public oversight of governments and help reduce corruption by enabling greater transparency in various fields, such as health, education, climate, energy, intelligent transport systems, and smart cities. Open data give the opportunity to monitor government activities, such as tracking public budget expenditures or policy targets and impacts [14,15].

Within the European Union, more and more countries are developing policies to release their data as Open Government Data. There are open data platforms that provide access to open data from international, EU, national, regional, local, and geo data portals, such as the official European Data Portal: https://data.europa.eu/en (accessed on 22 August 2022) or the official national data portals, such as for Greece: https://www.data.gov.gr (accessed on 22 August 2022).

There are also monitoring tools that help citizens track different aspects of openness; for example, there are tools that assess the state of open data around the world, such as the Open Data Index: http://global.survey.okfn.org (accessed on 22 August 2022), the Open Data Barometer: https://opendatabarometer.org/ (accessed on 22 August 2022) and the Open Data Monitor: https://opendatamonitor.eu (accessed on 22 August 2022).

Moreover, there are tools such as Open Budgets [16,17,18], which help citizens and other stakeholders obtain an overview of public spending, or the Red Flags platform [19], which identifies red flags during the implementation process for projects of the National Strategic Reference Framework.

Additionally, in [20], some papers related to open data and energy analytics are commented on. Specifically, [21] addresses the barriers to energy data acquisition as there is a lack of open data in the energy sector. In [22,23], there is described the need for energy mapping tools to support urban planners, energy agencies, and public administrations in planning the heating and cooling supply at different scales. The first describes an approach to collect and analyze market data related to space heating and domestic hot water systems and their performance in Europe, and the latter compares country-level and municipal-level building stock data for gross floor area and energy demand for space heating and domestic hot water. In [24], the potential of data in supporting energy analyses is also shown. They focus on the use of different energy-consuming activities from social media to understand and improve the energy consumption behavior of individuals. In [25], the authors present an application to industrial energy efficiency where they apply a clustering modeling approach for the short-term forecasting of energy demand in industrial facilities.

However, openness in energy research lags behind other fields, according to [26], where there is a list of projects that tried to make data and models more transparent. They conclude that openness in energy research, sharing code and data, is the initial step to conducting research and monitoring and engaging decision-makers with models and the assumptions embedded within them.

OpEnergy is an application developed by Open Knowledge Foundation Greece (https://okfn.gr/ (accessed on 22 August 2022)) in the context of the first ‘EU Datathon’ that took place in Brussels in November 2017 to monitor and promote the re-use of energy data from the EU Open Data Portal. It was organized by the Publications Office of the European Union together with the Estonian Presidency of the Council of the European Union, and the main challenges were to create innovative applications for citizens to address specific societal issues in line with three key priorities for the European Union: jobs, growth and investment; the digital single market; and democratic change.

This work aims to (i) present the basic structure and components of an innovative methodology to develop an ICT platform of energy indicators, (ii) depict its applicability and utilization as a dynamic tool for the management of climate, environmental, social, and economic information, and (iii) depict its efficiency in order to predict important aspects towards the transformation of the EU energy sector. In this context, the rationale behind the platform’s development is presented, and some indicative results for the EU are briefly discussed.

Tackling climate change and energy crisis best practices requires sharing knowledge across geographic and economic boundaries, which is why we believe energy open data is essential. The information should be available in such a form that it facilitates decision-making by decision-makers and is also understandable by citizens.

Openergy is an application that achieves two innovative goals:

• A good practice of open data use in the energy sector;
• Analysis and forecast of open energy data in order to help citizens and governments track their progress in achieving their EU energy targets.

2. Materials and Methods

2.1. Data

In this study, data from two EU portals were used. The first one is the official portal that provides access to open data from international, EU, national, regional, local, and geo data portals which is accessible at https://data.europa.eu/en (accessed on 22 August 2022). From this portal, the data that were used were data related to the energy balance, EU 2020 energy target, country energy profiles, energy production, transport fuels, heat production, and gas emissions.

The second one is the portal of Eurostat (https://ec.europa.eu/eurostat) (accessed on 22 August 2022), the official statistical office of the European Union, from which the data used were data about the targets of each country, e.g., for primary or final energy consumption or for greenhouse gas emissions. All data were transformed into the JSON format to be used in our architecture, as described below.

2.2. System Architecture

The Openergy architecture shown in Figure 1 consists of five modules, the database that stored data from the two EU portals described above, a REST API, an OpenCPU API, the R libraries for time series modeling and forecasting, and the website http://openergy.okfn.gr (accessed on 26 August 2022) that visualizes the results.

The database consists of data related to six main energy categories distributed on six pages at the Openergy tool, EU 2020 energy targets, energy balance, electricity production, transport fuels, heat production, and gas emissions. Each includes its own indicators for EU and EU countries.

The REST API was developed in Laravel and was used for HTTP requests to manage the data from the database, enabling the exchange of data between two applications. This API allows access to data and interacts with the OpenCPU server, which communicates with the R libraries responsible for time series modeling and predictions (Figure 2).

Access to the six main energy categories data is provided by the REST API. For example, when the user selects the energy balance page, he will access indicators represented graphically with maps, bar or pie charts, time series/line charts that describe the production, imports, exports, energy branch consumption, and final energy consumption by fuel or sector for each EU country. There are also market indicators about natural gas, such as entities bringing gas into a selected country, the main entities of cumulative market share, the largest retailer of market share, etc., and electricity, such as producers, main producers, main retailers, etc.

The country profiles page includes 27 indicators with their corresponding time series charts and their predicted values until 2020. These indicators are related to natural gas, renewables, electricity, energy consumption, transformation/distribution losses, and combined heat and power that form the general energy profile of a selected country.

Forecasts performed for each indicator in every energy category delivered by Rest API as a product of communication by OpenCPU with the R libraries. These forecasts include the predicted values with confidence intervals of the selected indicator until 2020.

2.3. Time Series Modeling and Forecasting

Forecasts of the Openergy tool are implemented through the OpEnergyAnalytics package [27], and they are not published by the European Commission. The OpEnergyAnalytics package is mostly based on TimeSeries.OBeu package [28] for modeling and predicting time series data, which was developed in the context of the Open Budgets project (http://openbudgets.eu/) (accessed on 26 August 2022) funded from the European Union’s H2020 EU research and innovation program.

In this package, first, stationary testing is performed through statistical hypothesis tests of stationarity that are designed for determining whether differencing is required, such as Augmented Dickey-Fuller (ADF) and Kwiatkowski-Phillips-Schmidt-Shin (KPSS) tests. Stationarity is an important concept of time series analysis which practically means that the statistical properties of a time series do not change over time.

Autoregressive integrated moving average (ARIMA) models are appropriate to forecast time series data [29,30,31] and, according to [32,33,34], outperform some other methods. ARIMA model is a statistical model used for forecasting time series data based on historical data.

An ARIMA model consists of three components. The Autoregression (AR) component refers to the model that shows that the time series is regressed with its previous values, which order is denoted as p. The Integration (I) component refers to the order of differencing d of the time series raw data to become stationary. The Moving Average (MA) component indicates that the time series is regressed with the residuals of the past observations, in which order is denoted as q. If non-stationary data occur, the next step is to differencing the time series to conclude in stationary data. The equation of an ARIMA model for the predicted value of y in period t has the following form:

y_t = θ₀ + φ₁·y_t−1 + φ₂·y_t−2 + … + φ_p−y_t−p + ε_t−θ₁·ε_t−1−θ₂·ε_t−2− … −θ_q·ε_t−q,

(1)

where y_t and ε_t are the actual value and random error at time period t, respectively; φ_i (i = 1, 2, …, p) and θ_j (j = 1, 2, ..., q) are AR and MA model parameters respectively. p and q are integers and are often referred to as orders of the model. Box and Jenkins established that the MA parameters should be written with a negative sign rather than a positive sign. The general form of an ARIMA model is denoted as ARIMA (p, d, q) [35,36].

3. Results and Discussion

3.1. OpEnergy Presentation

3.1.1. EU 2020 Energy Targets

On the OpEnergy tool, the monitoring process involves six energy categories: EU 2020 Energy Targets (https://www.eea.europa.eu/highlights/eu-achieves-20-20-20 (accessed on 10 October 2022)); Energy Balance (https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Energy_balance_-_new_methodology (accessed on 10 October 2022)); Electricity Production (https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Electricity_and_heat_statistics (accessed on 10 October 2022)); Transport Fuels (https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Transport_statistics_introduced (accessed on 10 October 2022)); Heat Production and Gas Emissions (https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Greenhouse_gas_emission_statistics_-_air_emissions_accounts (accessed on 10 October 2022)), along with the Country Profiles of the EU countries.

The starting page is the EU 2020 Energy Targets page, where there is information related to the targets of EU countries and the EU as a whole entity. Specifically, there are seven energy indicators, Gross Final Energy Consumption (https://ec.europa.eu/eurostat/web/products-datasets/-/t2020_rd330 (accessed on 10 October 2022)), Gross Electricity Consumption, Heating and Cooling, Transport, Final Energy Consumption (https://ec.europa.eu/eurostat/web/products-datasets/-/sdg_07_11 (accessed on 10 October 2022)), Greenhouse Gas Emissions (https://data.europa.eu/data/datasets/dat-170-en?locale=en (accessed on 10 October 2022)), and Primary Energy Consumption (https://ec.europa.eu/eurostat/web/products-datasets/-/sdg_07_10 (accessed on 10 October 2022)) that are visualized on an EU heat map, a bar chart, and a line chart. Additionally, there is a sidebar that includes the EU countries. In this way, there is the capability of performing comparisons among EU countries by selecting at least two countries to visualize. In case of multiple countries selection, the bar chart and the line chart will be adapted to the user selection.

The heat map shows the magnitude of the selected indicator for a selected country on the EU map as color and can be used for comparisons when the EU is selected from the sidebar (Figure 3).

The bar chart shows the value in 2015, the predicted value, and the target of the selected indicator for a selected EU country (Figure 4).

The line chart is a time series chart of a selected indicator for one or more EU countries. In this chart, there is the capability to show the predictions along with the confidence intervals until 2020 (Figure 5).

At the bottom of this page, there is a semi-automated constructed report, where the user can access some key points of the charts and a summary of whether a country or EU as a whole is expected to achieve its 2020 target according to the desired energy indicator (Figure 6).

3.1.2. Country Profiles

The Country Profiles page consists of 27 indicators that altogether formulate the energy profile of an EU country. These indicators are related to natural gas, renewables, electricity, energy consumption, transformation/distribution losses, and combined heat and power that form the general energy profile of a selected country. On this page, there is a dynamic table with the indicators, their 2015 value, and the sparkline from 1990 to 2015. The sparkline shows trends for the 1990-2015 period, such as increases or decreases and maximum and minimum values of a selected indicator. By clicking on a row, a time series plot is shown with the forecast with the corresponding confidence intervals of the selected indicator until 2020. (Figure 7).

3.1.3. Energy Balance, Electricity Production, Transport Fuels, Heat Production, and Gas Emissions

The Energy Balance page allows us to see the importance of the various fuels and their contribution to the economy. Specifically, the user can select information related to the production, imports, exports, energy branch consumption, the final energy consumption by fuel/product and by sector, which are visualized on an EU heat map, bar chart or pie chart, an area chart and a tree map chart using a slide bar to select a year. The heat map is similar to the EU Energy Targets page, with the difference that there are different indicators in this category. The bar chart, which can be transformed into a pie chart, shows the percentages of subcategories of an indicator which is represented as the total (Figure 8).

The area chart shows some market indicators related to natural gas and electricity. This chart shows how a market indicator of one or more EU countries changes over time (Figure 9).

Similar to the EU Energy Targets page, there is a time series chart of a selected country for a selected indicator with the forecasts of the indicator’s values along with its confidence intervals.

The treemap is used to display data that is nested in a hierarchical structure and can capture relative sizes of indicator subcategories, showing factors that are large contributors to each category. Color can identify factors that are under- or over-performing compared to other factors from the same category (Figure 10).

The functionality of the Electricity Production, Transport Fuels, Heat Production, and Gas Emissions pages are similar to the Energy Balance, excluding the chart of the market indicator.

3.1.4. Empirical Validation

Empirical validation is the comparison of model predictions with observations from the real system, together with an assessment of whether the model is adequate for its purpose. Figure 11 presents the annual variation comparisons for EU-27 countries between the final energy consumption (observed) and the prediction by OpEnergy for the years 2016, 2017, 2018, and 2019. Figure 11 clearly depicts that the real-life energy consumption, in comparison to the predicted values (from the ARIMA model) for all the years under consideration, shows that the results of the model are in very good agreement with the predicted concentrations on an annual basis. Germany is by far the country with the highest final energy consumption, with an average of 202.40 Mteo for the time period 2016–2019. The corresponding average prediction for the same time period is 211.45 Mteo, which gives approximately a 4% prediction error. In general, the precision of the model predictions is more than acceptable, considering the purpose of the OpEnergy model. Germany is followed by France (observed average 2016–2019: 141.54 Mteo, predicted average 2016–2019: 142.7 Mteo, accuracy 99.18%), Italy (observed average 2016–2019: 113.15 Mteo, predicted average 2016–2019: 112,75 Mteo, accuracy 99.64%), Spain (observed average 2016–2019: 80.3 Mteo, predicted average 2016–2019: 80.72 Mteo, accuracy 99.47%), and Poland (observed average 2016–2019: 68.39 Mteo, predicted average 2016–2019:62.05 Mteo, accuracy 91%). These countries are the top-5 energy consumers and also the top-5 in GHG emissions in EU-27.

In general, the OpEnergy model is in very good agreement with the observed values for the categories under consideration, i.e., (i) Final energy consumption (mtoe), (ii) GHG emissions, (iii) Gross electricity consumption, (iv) Gross final energy consumption, (v) Heating and cooling, (vi) Primary energy consumption, and (vii) Transport. Considering the amount of data under consideration and for reasons of economies of space,

Table 1. Root mean square error and mean absolute error estimated for the OpEnergy model for five significant energy-intensive categories (EU-27, 2016-2019).

Category of Prediction	Measure	2016	2017	2018	2019
Final Energy Consumption	RMSE	1.7	2.2	2.7	2.7
	MAE	0.9	1.2	1.3	1.3
Greenhouse Gas Emissions	RMSE	5.4	7.9	9.0	8.0
	MAE	3.2	4.2	5.1	4.5
Gross Electricity Consumption	RMSE	1.1	2.2	2.9	4.2
	MAE	1.1	1.8	2.5	3.6
Heating and Cooling	RMSE	2.9	3.0	4.6	5.1
	MAE	1.8	2.1	3.1	3.5
Transport	RMSE	2.2	2.1	2.6	3.7
	MAE	1.3	1.4	1.8	2.4

is constructed. This is clearly depicted in Table 1, where the Root Mean Square Error (RMSE) and the Mean Absolute Error (MEA) are estimated for the OpEnergy model for five significant energy-intensive categories (EU-27, 2016-2019). These measures are maybe the most frequently used to highlight the differences between values predicted by a model and the values observed in the real system. The main conclusion is that in almost all cases and estimations, real values are in very good agreement with the predicted values.

4. Conclusions

The EU, especially in this era of questioning energy sources and security, needs to demonstrate intense leadership at the global level, considering the transformation of the energy system via efficient and sustainable strategies and initiatives, especially under the threat of carbon emissions and climate change. Smart energy systems need smart tools that will support the transformation of the energy system in the main energy consumption categories [37]. As the main conclusion of this work, the Openergy platform, through its described architecture and modules, can be considered a useful tool that manages energy information and indicators via the umbrella of open data use in the energy sector. The material presented demonstrates that the platform efficiently adopts open databases, analyses them, and provides both reliable and valuable predictions in order to assist citizens and governments track their progress in achieving their EU energy targets.

The COVID-19 pandemic has shown that research data when shared and opened in times of global crisis, help both in transparency and in the search for optimal solutions to address crises that threaten the security and sustainability of society. Correspondingly, open data is also necessary for dealing with the climate and energy crisis, which is an extremely complex phenomenon and the solutions require the efforts of many sectors and actors. However, for knowledge to reach people and be effective open data needs data visualization and data analysis applications such as Openergy, which will help in climate and energy justice. Energy open data is not a future goal but a present necessity.

Based on the above presentation and analysis, Openergy is essentially a data analysis tool that collects and analyzes open data related to the energy system and can be very useful (i) in improving energy-intensive processes and sectors, (ii) help uncover insights to make optimal data-driven decisions, and (iii) optimize modeling and monitoring at the European and national levels. The Openergy application highlights a path on how open data, as a horizontal policy of the European Union, can be of crucial importance in order to create smart energy data management systems. Such online tools can be the basis for putting forward strategies and policies in the EU related to sustainable and renewable energy sources, enhancing transparency and control while improving informed government decisions.

Predicting the value of crucial energy indicators by using the ARIMA proceeds proved reliable. Apart from the accuracy that is apparent in the analysis presented, the process is characterized by simplicity as well as the flexibility it offers, adequately fulfilling the forecasting needs of the current study. This is particularly helpful because it allows handy estimation of the levels and annual trends of energy consumption (primary, final, and gross), CO₂ and GHG emissions, electricity consumption, heating and cooling, and the transport sector. The model adopted in the Openergy analysis is successfully validated on an independent set of observations and real-life values. Considering the different characteristics of EU countries, the model may not be expected to perform similarly.

The Openergy platform could be used with more data and more prediction algorithms (e.g., neural networks), semantically describe the datasets, and link them with other sources following the linked open data principals (e.g., climatic data, hazard data). The expectation of the Openergy platform would be to become an application that combines data from other sources and other domains that would visualize it and provide predictions with various algorithms in a form that is understandable to the public and as a monitoring tool for decision-makers. Although generally, there are gaps and uncertainties in data analysis, the tool presented is simple, objective, and quantitative and can be useful for the user.