European Energy Regulatory, Socioeconomic, and Organizational Aspects: An Analysis of Barriers Related to Data-Driven Services across Electricity Sectors

Abstract

Data-driven services offer a major shift away from traditional monitoring and control approaches that have been applied exclusively over the transmission and distribution networks. These services assist the electricity value chain stakeholders to enhance their data reach and improve their internal intelligence on electricity-related optimization functions. However, the penetration of data-driven services within the energy sector poses challenges across the regulatory, socioeconomic, and organizational (RSEO) domains that are specific to such business models. The present review examines the existence and importance of various obstacles across these domains regarding innovative energy services, new business models, data exchanges, and other actors’ synergies across the electricity data value chain. This research is centered around the European landscape, with a particular focus on the five demonstration countries (Greece, Spain, Austria, Finland, and Croatia) of the SYNERGY consortium. A state-of-the-art analysis on the regulatory, socioeconomic, and organizational aspects related to innovative energy services (IESs) revealed a plethora of such potential obstacles that could affect, in various degrees, the realization of such services, both at a prototyping and a market replication level. More specifically, 13 barriers were identified in the regulatory domain, 19 barriers were identified in the socioeconomic domain, and 16 barriers were identified in the organizational domain. Then, a comprehensive, survey-based data gathering exercise was designed, formulated, and conducted at a national level as well as at a stakeholder type level. To ensure that our analysis encompassed business-wide perspectives and was validated from the whole electricity data value chain, we utilized a trilevel analysis (i.e., partner, stakeholder type, demo country) to formulate qualitative interviews with business experts from each stakeholder type (namely TSOs, DSOs, aggregators/ESCOs, facility managers/urban planners, and RES Operators). By combining the quantitative data with the qualitative interviews, further recommendations on identifying and facilitating ways to overcome the identified barriers are provided. For the regulatory domain, it is recommended to treat nationally missing regulations by conforming to the provisions of the relevant EU directives, as well as to provide a flexibility-related regulation. For the socioeconomic domain, recommendations were made to increase consumer awareness and thus alleviate the three more impactful barriers identified in this domain. All organizational barriers can be alleviated by taking complex big-data-related issues away from the hands of the organizations and offering them data-as-a-service mechanisms that safeguard data confidentiality and increase data quality.

1. Introduction

Under the traditional top-down business model, power system optimization relies on centralized decisions based on data silos preserved by the electricity sector stakeholders. The European energy sector is in the process of a major shift from a traditional centralized structure to a more inclusive system that incorporates digitalization, includes a wider variety of stakeholders, and requires new organizational processes. This major shift of the energy sector towards decarbonization and digitization has brought a plethora of reasons that have made data acquisition/analytics a “game-changer” for the stakeholders across the energy value chain. A study performed by the key players of the information and communications technology (ICT) industry, such as computer software and hardware corporations, claims that the big data analytics market in the energy sector is expected to grow at a compound annual growth rate (CAGR) of 11.28% in 2021–2026 [1]. Advanced computing power, declining costs of information and communications technology, and the rollout of smart meters have dramatically increased the availability of data and the subsequent opportunities to extract intrinsic value. Furthermore, the scarcity of fossil fuels, the volatility of oil and gas prices, and the overarching challenge of climate change mitigation have driven major advances in renewable energy source (RES) technologies which in turn have led to RESs reaching grid parity conditions in terms of their total expenditure (TOTEX) [2] compared with the fossil alternatives.

RES grid and market integration, however, poses significant challenges for grid and system planning and operation [3]. Additionally, small-scale, nondispatchable renewables from prosumers, electric vehicle (EV) market growth, local and centralized energy storage, and implicit/explicit demand response channel the complexity of the electricity system’s energy balance through a dynamic market-signal-based consumption market. This complexity, along with the rising number of active actors which are required to fill these emerging service sectors (e.g., aggregators, energy service companies, EV charging infrastructure providers, facility managers), creates new needs for data exchanges and data analytics. Acknowledging the new data-driven energy sector environment, organizations from the whole spectrum of the electricity value chain (commercial, public, or research and academia) aim at benefiting from the use of innovative energy services (IESs) that rely on data analytics, either by investing in in-house development or by purchasing market-available solutions.

The successful adoption of these changes is reliant on considering the relevant regulatory, socioeconomic, and organizational (RSEO) factors needed for the successful integration of new services. In order to mitigate energy and price imbalances, European countries are in the process of interconnecting their power systems and coupling their energy markets, as well as standardizing related equipment and processes. Consideration of the regulatory elements is necessary to align any new initiatives with current legislation. This, in turn, drives the development and gradual adoption of a common regulatory basis for the member countries, aiming to enhance environmental protection, promote emerging technologies, increase efficiency and stability of the interconnected power and energy system, increase the efficiency of the building stock, and eliminate energy poverty [4]. At the same time, this regulatory transition will allow enough freedom for energy markets to grow and relevant stakeholders to profit within the boundaries of a healthy and prosperous commercial environment. In order to ensure that this transition does not overstress member countries and to safeguard the success of this systemic effort, various factors must be taken into consideration, such as societal norms, economic background, and prevailing mentality at an interorganizational level in the energy sector of each country. Respecting the socioeconomic factors, such as societal attitudes, motivations, and the distribution of wealth, ensures that the context in which the innovation is deployed is taken into account. Understanding a company’s processes and capabilities is important as the organizations are ultimately responsible for implementing the proposed changes. Consequently, a thorough country-based investigation of the potential regulatory and socioeconomic barriers as well as a stakeholder-type-based investigation of the organizational barriers to the adoption of innovative energy services (IESs) is necessary.

In the last decade, several studies conducted across the EU have assessed the regulatory practices in the electricity sector. Indicatively, Gianfreda and Vantaggiato in [5] investigated regulatory dynamics and competencies across 34 national regulatory authorities of European and non-European countries using questionnaires that explored the regulators’ ex ante powers and ex post enforcement powers. The survey revealed a certain degree of difference in regulators’ powers among EU regulators, such as between smaller and bigger markets, or between the Scandinavian regulators. The study also showed many similarities in the responses of some countries in couples such as Lithuania–Estonia, Turkey–Romania, Italy–Algeria, and Bulgaria–Albania. Regarding the transmission sector, the research showed a great deal of diversity of replies compared to the generation, distribution, and retail sectors. Research from [6] focused on the regulatory challenges and opportunities for collective RES prosumers. This study provided a starting point to distill policy implications for improving legal frameworks relevant to this sector across European stakeholders. On the digital transformation side, Duch-Brown and Rossetti in [7] investigated a sample of over 200 digital platforms running energy-related activities across the EU regional markets, offering empirical evidence and reflection to provide energy policy to take advantage of the potential of new technologies.

In addition to the above, there have been several recent studies exploring several barriers hindering the access to energy services resulting from social, cultural, and economic factors, usually focusing on specific energy sectors. The authors of [1] provide a review of available evidence emerging from the recent economic literature by framing it in a comprehensive framework for access to modern energy services and overall policy context. The study focuses on the household level and is conducted for both access to electricity and to modern and efficient cookstoves, providing an in-depth discussion of the factors that contribute to energy poverty from a socioeconomic perspective. On the other hand, [8] discusses barriers to the deployment of renewable energy related to social, economic, technological, and regulatory barriers. Data for the study were collected through online questionnaires that were distributed to 223 energy professionals working in companies around the world. The findings of this research show that technological and regulatory barriers have a very significant impact on the deployment of renewable energy while social barriers have a positive impact. On the other hand, the economic barriers do not directly impact the deployment of renewable energy but are interrelated with social, technological, and regulatory barriers, thus indirectly affecting the deployment of renewable energy.

Socioeconomic and organizational behaviors were studied in [9] with the aim to reveal the prominent barriers to the adoption of new energy services. Forty-five in-depth semistructured interviews were conducted with energy professionals in Sweden, and the analysis of the results showed that three main barriers dominate the perceptions from the socioeconomic viewpoint: lack of customer interest, lack of customer knowledge, and shortage of financial resources. Internal barriers that in effect reflect the interorganizational environment were mainly attributed to the lack of internal will to change, lack of competence, and lack of strategic vision from top management.

Looking into a more specific area in the energy domain, Guntram et al. [10] utilized the Delphi method to identify the main obstacles to the adoption of local energy markets and local flexibility markets (LEMs and LFMs), with most market participants referring to standardization issues as the main barrier. Using the Delphi method, 14 market actors from the PARITY project [11] were recruited and formed a group of experts aiming at reaching a consensus about this topic. The most common barriers that were identified were the lack of previous experience of energy companies, the use of blockchain technology for verifying transactions, the initial investment that companies and prosumers must make, the lack of regulations in most countries, the complexity of the systems and contracts to be understood, and personal data and privacy concerns.

From the above, it is evident that associating the regulatory, socioeconomic, and organizational domains in the process of identifying barriers in the adoption of new data-driven services is less reported. As such, this paper proposes a data-driven strategy that enables the development of holistic data-driven tools which facilitate the generation of valuable insights as well as the maximization of energy efficiency and performance in a digitized solution environment. Some of the future trends that affect data analytics are the integration of electricity value chain areas, larger shares of varying generation from renewables, time-varying electricity tariffs, higher flexibility from EVs [12], stationary storage [13], and automated demand response [14]. Additionally, the electricity sector will be increasingly coupled with other energy sectors such as mobility, heat, and gas, leading to multienergy systems. Hence, data analytic approaches that are applicable to multiple areas are expected to play a crucial role in the transition and integration of these future trends. Additionally, with the rising adoption of residential batteries, rooftop solar panels, electric heat pumps, electric vehicles, smart metering infrastructure, and smart home appliances, both the availability of data and the need for new solutions will grow substantially [15]. In order to deal with the extreme quantity of data, the smart grid requires the adoption of advanced data analytics, big data management, and powerful monitoring techniques. Machine learning algorithms, analytic models, and big data solutions help companies to manage and effectively use their resources, control energy flows, regulate grids, optimize work, and avoid costly mistakes. The use of real-time and predictive analytics and data science solutions requires significant investment and readiness to face the challenges, learn, and introduce new complex operations.

This survey, therefore, presents a thorough analysis of the regulatory, socioeconomic, and organizational obstacles to innovation regarding innovative energy services, data exchanges and synergies, and new business models in Europe. Regarding the regulatory aspect of this work, this document provides a state-of the-art analysis on the current European policies in force (regulations, legislation, rights, and guidelines) which relate to all aspects of data-driven services in the energy sector. In the socioeconomic part of the survey, building on previous experience and relevant literature review, an aggregation of a wide range of obstacles is presented with the aim to investigate in further detail which of them possibly apply to the energy data value chain. Special attention is provided to the interorganizational obstacles which might be hindering the implementation of data-driven services.

Apart from the necessary literature and regulatory review, which constitutes the backbone of this survey, the most important added value provided is the conclusions drawn from the interaction with the various data value chain stakeholders related to their assessment of the regulatory, socioeconomic, and organizational aspects associated with data-driven services in the electricity sector. Through utilizing appropriately formulated questionnaire-based surveys, this review proceeds with highlighting the relevant regulatory, socioeconomic, and organizational barriers to data-driven services at the country (namely Greece, Spain, Austria, Finland, and Croatia) and organizational levels. The results of the research associated with stakeholders are provided in a qualitative and quantitative manner, covering geographical and topological categorization as well as conclusions referring to specific (types of) organizations.

The overall structure of this paper is organized into five sections. Section 2 provides a thorough mapping of the current policies and directives as well as strategies at the EU level, which are relevant to innovative data-driven services. Additionally, an analysis on the identification of social, economic, and organizational barriers to data-driven services is presented, facilitating a comprehensive targeting of the considerations to be further analyzed through interactions with stakeholders. Section 3 introduces the methodological approach that was followed throughout this survey analysis, involving the information-acquiring strategy through literature review, survey conduction, analysis of results, and designing of interviews with stakeholders. Section 4 is dedicated to reviewing the RSEO barriers and presents all aspects associated with the interaction between data value chain stakeholders, providing qualitative and quantitative insight stemming from the results of this interaction. Section 5 provides the conclusions emerging from the analysis performed in all three domains.

1.1. Literature Review

The integrated electricity ecosystem involves different active actors throughout the energy value chain. Different actors that interact with each other assume vast amounts of data with different characteristics that need to be processed, analyzed, and exchanged. Under this reality, various issues arise related to how data should be managed, protected, or exchanged. The first step in investigating the RSEO barriers was to conduct an extensive literature review into the potential obstacles to innovation to identify specific relevant barriers within these domains. The search strategy and information sources used to develop our work on regulatory, socioeconomic, and organizational obstacles were developed to ensure a holistic approach that embraces academic excellence in conjunction with research and industrial expertise.

1.1.1. Regulatory Domain

A thorough investigation was performed on the current policies and directives at a European level that are in principle associated with the various objectives and means of realization of data-driven technologies. The regulatory landscaping exercise conducted comprised an approach that ensures that all appropriate regulations are identified and the regulatory gaps are highlighted; EU directives and relevant international documentation were gathered using open searches in EC relevant repositories. For the regulatory analysis, the different regulations and directives that affect innovative data-driven services of the energy sector are categorized as horizontal, vertical, and hierarchical (Figure 1).

The horizontal regulations are regulations that affect the whole energy value chain and all associated directives, i.e., Regulation on the Governance of the Energy Union. The Regulation on the Governance of the Energy Union ensures that the objectives of the Energy Union, especially the EU’s 2030 energy and climate targets, will be achieved by setting out a political process defining how EU countries and the Commission work together, and how individual countries should cooperate, to achieve the Energy Union’s goals [16].

The vertical regulations are regulations that act as facilitators through the electricity data value chain (EDVC) which can be linked to technologies, applications, or concepts. One of the regulations included in this category is the Regulation on Risk Preparedness, which focuses on the internal electricity market that establishes regional operating centers in order to facilitate cross-border management of the electricity grid and cooperation of transmission system operators [17]. With respect to smart metering and all the data collected from all sources of the integrated electricity data value chain, they should be subject to different restrictions of processing and transmission according to the General Data Protection Regulation (GDPR) [18]. On the same note, smart metering legislation is possibly one of the most important enablers out of the vertically applied regulations of the energy transformation at a European and worldwide level [19]. The Electronic Identification, Authentication and Trust Services for Electronic Transactions in the Internal Market and Repealing Directive (eIDAS) aims to enhance trust in electronic transactions between businesses, citizens, and public authorities by providing a common legal framework for the cross-border recognition of electronic ID and consistent rules on trust services across the EU [20]. The Electricity Market Design Directive introduced a fair deal for consumers and focused on defining new rules for the functioning of the wholesale and retail energy markets, while promoting consumer empowerment to participate in energy markets through demand response in an effort (among others) to also fight energy poverty around the EU [21]. Finally, the guidelines contained in ethics in artificial intelligence are addressed to all artificial intelligence (AI) stakeholders designing, developing, deploying, implementing, using, or being affected by AI in the EU, including companies, researchers, public services, government agencies, institutions, civil society organizations, individuals, workers, and consumers [22]. Under this broad category, smart contracts and blockchain aspects that are of high relevance to data-driven services are additionally included [23].

The final category is that of the hierarchical regulations which are regulations that fall under the horizontal regulations and are facilitated by the vertical regulations. These are applied in different levels of the power system and the EDVC. First, the Renewable Energy Directive establishes an overall policy for the production and promotion of energy from renewable sources in the EU, aiming at the increase in renewable energy use in Europe [24]. The Energy Consumers Rights regulations and the energy transition goals of the EU imply that the energy consumer is at the middle of the chain and valorizes the effort of system decarbonization [24]. The Energy Performance in Buildings Directive aims at improving energy efficiency in buildings and encourages building renovation [25]. The Electricity Regulation aims to make the electricity market fit for flexibility, decarbonization, and innovation by providing for undistorted market signals, revising the rules for electricity trading, clarifying the responsibilities of the market participants, and defining principles for assessing capacity needs and for market-based capacity mechanisms [26]. Additionally, the concept of the Energy Communities Legislation aims at tackling energy poverty while promoting energy sustainability through RES production, storage, and self-consumption at the same time [27]. Finally, the Revised Energy Efficiency Directive sets a binding 30% EU energy efficiency target for 2030 through (among others) consumer awareness, behavioral change, and participation in demand response transactions [28].

1.1.2. Socioeconomic Domain

The barriers related to the socioeconomic domain were considered within the subdomain of data-driven innovations. This included areas such as innovative energy services (IESs), big data analytics (BDA), and data sharing. The list is divided into two categories, as depicted in Figure 2, consisting of the social barriers which are related to social aspects such as trust, democracy, and fairness and the economic barriers which are related to monetary aspects such as CAPEX and financial risk.

Hertel [29] proposed that the integration of energy-efficiency-related technologies should be considered as a social process. Of course, technical considerations are required when introducing new products and services; however, many of the potential barriers for integration fall under the social category and are related to motivations, attitudes, and social environment. Therefore, within innovative energy services, there needs to exist a system that ensures a fair distribution of the benefits associated with the system change and market mechanisms to all stakeholders in the transition [30,31]. Additionally, the societal perspective of IESs is often one in which the opportunities for investment and wealth are restricted solely to the wealthy [32]. Therefore, IESs should provide equal opportunity for investment, create new roles, and evenly distribute generated wealth amongst all affected stakeholders. Processes that lack democracy can lead to distrust, lack of acceptability, and skepticism from the community [33,34]. The availability of wealth and opportunities and the existence of democratic legitimacy are linked to people’s desire to be involved in IESs. However, if there is no true involvement for stakeholders, a lack of agency will develop from neglected stakeholders [5]. Furthermore, lack of meaningful involvement and consideration towards citizens can also result in the exclusion of societal groups due to issues such as lack of knowledge or access to products and services [35]. To avoid these negative consequences, all stakeholders involved in IESs will want to see a meaningful change and have legitimate agency in the change process [36,37]. As well as a real sense of agency, societies and stakeholders within an energy community need to have a real sense of empowerment [5,6,32]. Furthermore, past IES initiatives have been subject to offering a narrative of empowerment rather than real empowerment. This narrative can be very limited and weak in that public participation only serves to legitimize preordained decisions [38].

Many of the factors mentioned thus far, such as democratic legitimacy, sense of agency, and empowerment, all link to the important psychological motivator of trust. Successful implementation of any IES is reliant upon the existence of a high level of trust between project leaders and the community [39]. Trust is increasingly important within projects with a high level of complexity due to the potential lack of knowledge about the technical aspects which can occur within communities and individual stakeholders [40]. To increase trust within an IES project that utilizes data sharing, there needs to be clarity on how the data is being used, with links to real project initiatives and goals. A vast amount of private information is likely to be contained in big data systems, and there is a need to promote enhanced security and privacy through proper legislation and company practices [41]. Because the flow of information is an important factor for increasing trust, there needs to be consideration towards how information is communicated and also how it relates to valuing individual perspectives and refraining from adopting the approach of telling individuals the situation and expecting them to follow set rules and guidelines [42,43]. Consideration towards the various perspectives of stakeholders also includes considering the variety of individual barriers and obstacles each stakeholder may encounter. There needs to be an acceptance of diversity of motivations for public opposition to IESs [44,45]. Factors such as the diversity of interest require IESs to be implemented over a sustained period with consideration afforded to the dynamic nature of communities and local context. Consequently, people may change their perspectives or opinions, which can create a potential barrier of sustainability or fallibility of commitment [46,47,48].

Obstacles to IES integration can also arise with regard to the economic structure of a business. The overall cost of implementation (CAPEX) or the lack of financial support or business sponsorship to deal with CAPEX can also be a deterrent for some organizations [49]. Additionally, integrating renewable energy initiatives will require system flexibility [50]. Due to the difficulty of transitioning innovative renewable energy schemes into business as usual, there is a need for careful consideration on how to achieve the switch through curtailment schemes [51]. The motivation to switch to IESs would also require economic changes with regard to auditing, such as penalties for noncompliance and standardized regulations [52]. Another factor associated with the successful implementation of innovation is the ability to understand and manage the associated risk [53,54,55]. A lack of understanding of the benefits of innovation will result in a significant obstacle to acceptance when organizations weigh the risks of an innovative endeavor against the potential benefits. As such, there is also a need to place focus on strategies that can reduce the risk for organizations when integrating innovation [56].

1.1.3. Organizational Domain

Finally, an explicit list of organizational barriers was compiled, specifically aiming to identify interorganizational characteristics across the energy value chain that potentially constitute hindering factors against data-driven services in the energy sector. As depicted in Figure 3, the list is divided into two main categories: the first one consists of barriers that are related to the operational activities of an organization, and the second relates to technical barriers affecting an organization.

As well as the barriers related to socioeconomic factors that have been outlined thus far, there are multiple barriers that exist related to organizational factors. For example, the sense of agency is not only an issue at the societal level, but it also applies in a business context, falling under the operational category of organizational barriers. Lack of agency in a business can create a barrier to the adoption of IESs due to the misalignment of the owner and agent [57]. Under the same category fall the absorptive capacity of an organization, which can also hinder the integration of IESs [57]. Absorptive capacity refers to the ability of a firm to recognize the value of new external information, assimilate it, and apply it, and it is usually due to the lack of energy management personnel/systems as well as the lack of current use of renewable energy technology [57]. Similarly, organizations that neglect flexibility in their processes and adopt a state of inertia will encounter problems when integrating IESs [50]. The obstacle of inertia for blocking the diffusion of innovative energy services in large technology companies is due to the existence of a link between energy systems and the broader economic system [58]. This link to the economy means that transitioning from current systems would be a monumental shift beyond anything experienced by the energy system hitherto [59].

Another critical area with regard to business-related barriers is the issues that arise with big data analytics, falling under the technical category of organizational barriers. Big energy data is a relatively new concept and is likely to be a new endeavor for many companies. Although current staff within companies will likely possess analytical skills, these skills may not be sufficient for the data management and data processing required in big data analytics [49]. Coping with the switch to big data analytics may require staff to take courses and programs in data management, data science, energy science, and social sciences [60]. Big data is a major development with high potential which will require organizations to realign their work practices and business models to fit with the big data process to maximize the benefits from its potential value [61]. Each organization needs to integrate systems to allow for the valuable data to be filtered and used efficiently [60]. This can be achieved through proper data governance strategies based on standardized procedures to ensure the timeliness, integrity, accuracy, and consistency of data processing. However, accounting for the merging and processing of valuable data is a potentially complex technical issue since incompatibility between sources may hinder research objectives [60]. Current IT infrastructures in energy companies may struggle to cope with the required processing speed and vast quantity of data associated with big data analytics. Therefore, IT infrastructure needs to be improved in its capability of network transmission, data storage capacity, processing, data exchange/interaction, and data visualization [60]. Finally, the large amount of personal data involved in big data analytics means it can be vulnerable to cyber-attack. Therefore, security issues are a serious challenge for IESs and big data analytics [62].

Figure 4 represents the distribution of the different identified barriers across the regulatory, socioeconomic, and organizational domains with an indication of overlapping barriers among the domains. Barriers related to data security, protection, and sharing are overlapping among the three domains since such barriers are concerning on a regulatory level, a social level, and an organizational level, while energy efficiency and performance are concerning across the three domains due to the associated directives affecting organizations on an economic as well as on an operational level. The following figure can also be utilized when providing recommendations for alleviating these overlapping barriers. Specifically, best practices and policy recommendations can be horizontally applied across the three domains when such overlaps between barriers occur.

2. Materials and Methods

This section is dedicated to the presentation of the methodological approach followed throughout this survey analysis, as indicated in Figure 5. The methodology was designed appropriately to assess the RSEO factors with the aim to identify the potential barriers to innovation within these domains. All research directions evolved simultaneously with parallel yet similar activities.

The primary and fundamental action in all research topics was a thorough background literature review. Research performed in the regulatory framework provided a list of current European policies and regulations pertinent to data-oriented technologies. Similarly, in the socioeconomic and organizational domain, the initial step of the research was a state-of-the-art analysis through a thorough review of literature related to the socioeconomic and organizational aspects pertaining to the main pillars of data-driven innovation.

The comprehensive literature review exercise was used to structure and formulate questionnaires that were used to acquire feedback from the partners of the SYNERGY consortium representing various types of stakeholders in the electricity value chain, with a large geographical and operating regime diversity. The H2020 SYNERGY project aims to implement an innovative big-data platform that promotes collaboration between currently diversified and fragmented electricity actors. SYNERGY offers innovative energy services that provide a wealth of data, analytics, and applications that will rely on data sharing and exchange between the beneficiaries, utilizing smart energy contracts and blockchain technology that will ensure secure and transparent transactions across the electricity value chain. Thus, the ultimate aim of this survey was to enable understanding of the existing barriers that relate to the realization of the data-driven services, rate them accordingly, and consider them during design so as to facilitate their overcoming.

In order to have a representative sample, a variation of the stakeholders involved in the specific sector was gathered prior to the conduction of the survey. Having identified the relevant stakeholders, the method applied to achieve the best possible participation in the survey was an electronic distribution through a mailing list. Specifically, a distribution of the questionnaire via a link to Google forms was performed. In total, 46 questionnaires were collected, and an Excel database was created for the collected data. Questions that investigated the various barriers were administered to representatives from the SYNERGY consortium. The respondents included a variety of stakeholders from five European countries, including 21 responses from Greece, 12 responses from Spain, 6 responses from Austria, 8 responses from Finland, and 1 response from Croatia. The SYNERGY consortium provided to this survey a variety of stakeholders originating in countries that follow different regulatory regimes and vary on a cultural level, resulting in different socioeconomic and organizational characteristics. As such, this group of countries gives a good understanding of how the barriers of the three domains are distributed among countries on the European level. Regarding the specialties of the respondents, there was representation from a wide range of energy stakeholder types, including urban planners, retailers, aggregators, facility managers, RES operators, and network operators. These types of stakeholders include 11 different organizations: 1 of these organizations is an urban planner, 3 are retailers, 2 are aggregators, 2 are facility managers, 1 is a RES operator, and 2 are network operators. Regarding the individual responses, 4 of the responses represent urban planners, 13 represent retailers, 6 represent aggregators, 5 represent facility managers, 6 represent RES operators, and 14 represent network operators.

The results of the research associated with stakeholders are provided in a quantitative and qualitative manner, covering geographical and topological categorization as well as conclusions referring to specific (types of) organizations. Specifically, a quantitative analysis was performed on the regulatory questionnaire results at a national level as regulations are obligated to be followed at national, European, and international levels. This quantitative analysis enabled or hindered the realization of data analytic innovation inherent to energy-related services. Subsequently, similarly to the survey on regulatory aspects, a quantitative analysis was performed in the socioeconomic questionnaire, through which valuable conclusions were extracted on a country level since the social factors are likely to be consistent within a country and its culture. To ensure that our analysis is validated from the whole electricity data value chain, we utilized our stakeholder type analysis to formulate a quantitative survey on the organizational barriers with business experts from each stakeholder type as stakeholders with the same type of business are likely to have similar operations and similar barriers faced.

Apart from directly presentable outcomes stemming from the answers of the participants, the outcomes unveiled interesting discrepancies among the perspectives of the different types of stakeholders. The next step of the above processes is the extended information sourcing from qualitative analysis based on dedicated interviews with business leaders from the SYNERGY partners. Findings that encompass feedback from three interviews engaging with the TSO, aggregator, and facility manager stakeholder types of the electricity data value chain in Europe are reported in this review.

3. Results

3.1. Quantitative Analysis

3.1.1. Regulatory Domain

The first part of the questionnaire was dedicated to linking countries to the regulation categories of the previous sections. Questionnaire respondents were asked to check if these regulations are missing or not at the national level.

Table 1. Missing regulations at a national level.

	Country
Question Number	Regulation	Greece	Spain	Austria	Finland	Croatia
RE-Q1	Regulation on the Governance of the Energy Union		Missing
RE-Q2	Regulation on Risk Preparedness	Missing				Missing
RE-Q3	General Data Protection Regulation (GDPR)
RE-Q4	Smart Meters’ Legislation Identification
RE-Q5	Electronic, Authentication and Trust Services (eIDAS)	Missing	Missing	-		Missing
RE-Q6	Electricity Market Design Directive		Missing	-
RE-Q7	Ethics in artificial intelligence	Missing	Missing	Missing		Missing
RE-Q8	Renewable Energy Directive			Missing
RE-Q9	Energy Consumers Rights		Missing
RE-Q10	Energy Performance in Buildings Directive		-	-
RE-Q11	Electricity Regulation		Missing	Missing
RE-Q12	Energy Communities Legislation		Missing	-		Missing
RE-Q13	Energy Efficiency Directive		Missing	-

shows the existence or lack of the regulations at the national level based on the feedback given by the participants in 2020.

As indicated in Table 1, no regulations were missing at a national level in Finland. Greece and Austria had a total number of three missing regulations, and Croatia had four missing regulations. Spain had a total of eight missing regulations.

The representatives from each country were asked to evaluate the impact that each regulation would have on the energy data value chain. This has enabled the identification of important regulations that are missing at a national level.

Table 2. Average impact score of regulatory barriers for each country.

	Country
Question Number	Regulation	Greece	Spain	Austria	Finland	Croatia
RE-Q1	Regulation on the Governance of the Energy Union	3.12	3.08	3.00	3.00	3.00
RE-Q2	Regulation on Risk Preparedness	1.83	2.75	3.00	2.00	2.00
RE-Q3	General Data Protection Regulation (GDPR)	4.05	3.50	5.00	2.88	5.00
RE-Q4	Smart Meters’ Legislation Identification	4.72	3.33	5.00	3.75	5.00
RE-Q5	Electronic, Authentication and Trust Services (eIDAS)	2.92	2.25	-	3.75	3.00
RE-Q6	Electricity Market Design Directive	4.47	4.42	-	3.38	2.00
RE-Q7	Ethics in artificial intelligence	3.00	2.63	3.00	3.38	3.00
RE-Q8	Renewable Energy Directive	3.77	4.42	3.00	2.38	4.00
RE-Q9	Energy Consumers Rights	3.22	0.83	3.00	2.63	5.00
RE-Q10	Energy Performance in Buildings Directive	2.08	-	-	4.63	5.00
RE-Q11	Electricity Regulation	4.58	4.67	3.00	3.38	5.00
RE-Q12	Energy Communities Legislation	2.25	0.67	-	3.13	4.00
RE-Q13	Energy Efficiency Directive	2.62	3.25	-	3.63	4.00

displays the average impact rating for respondents within each country.

When considering the results of Table 1 and Table 2, we are able to highlight the regulations’ absence in relevance to data-driven services by disregarding the relation with the directives having a low importance rating, i.e., lower than 4. The mapping is an insight on the emphasis that each country should give in adopting each of the missing regulations. This includes the absence of “Electricity Market Design Directive” and “Electricity Regulation” in Spain and “Energy Communities Legislation” in Croatia. Extra care should be given in these extreme cases where the regulations are important yet are not in place. Under this prism, two scenarios should be investigated: the first is if there are regulations that are hindering the implementation of data-driven services in each country or if there are missing regulations that can be treated otherwise. Figure 6 displays the average impact scores rating the regulatory barriers when responses are aggregated across all questionnaires.

As shown in Figure 6, the most impactful barrier is “Smart Meters’ Legislation Identification”, which falls under the vertical category of barriers, followed by the “Electricity Regulation” and “General Data Protection Regulation (GDPR)”. The lowest rated barrier is that of “Regulation on Risk Preparedness” along with “Energy Communities Legislation”.

3.1.2. Socioeconomic Domain

The following section presents the quantitative analysis of the socioeconomic barrier questionnaire.

Table 3. Average impact score of socioeconomic barriers for each country.

	Country
Question Number	Socioeconomic Barrier	Greece	Spain	Austria	Finland	Croatia
SE-Q1	CAPEX	3.75	3.20	3.00	2.33	5.00
SE-Q2	No sponsorship for CAPEX	3.38	3.20	3.00	2.33	5.00
SE-Q3	Neglecting value of system flexibility	4.19	3.00	5.00	2.83	4.00
SE-Q4	Energy performance contracting risk			3.00	3.00	5.00
SE-Q5	No holistic regulatory framework	4.15	3.20	3.00	2.50	4.00
SE-Q6	Lack of equal opportunities in wealth	3.44	2.71	3.00	2.50	2.00
SE-Q7	Converting innovation into business as usual	3.06	3.71	5.00	2.83	3.00
SE-Q8	No true participation for all actors	3.30	4.00	3.00	3.50	4.00
SE-Q9	No belief in consumer empowerment	2.40	4.50	5.00	2.75
SE-Q10	Understanding financial risk vs. potential	3.50	3.20	3.00		3.00
SE-Q11	Lack of auditing procedures		2.33		2.67	3.00
SE-Q12	No sustained commitment	1.00	3.00	3.00	3.00	5.00
SE-Q13	Lack of trust	3.50	3.00	4.00	3.00	4.00
SE-Q14	Perceived lack of democratic legitimacy	3.75	4.00	3.00	2.50	4.00
SE-Q15	Lack of clarity in profit and loss	3.22	4.00	3.00	2.67	3.00
SE-Q16	No consideration for diversity of interests	3.28	3.60	3.00	3.00	4.00
SE-Q17	No consumer awareness of benefits	4.75	5.00	4.00	4.00	5.00
SE-Q18	Exclusion of societal groups	4.50	4.00	4.00	2.00	4.00
SE-Q19	Perception of data security vulnerability	3.29	2.67	5.00	2.50

and Figure 7 present the relevant socioeconomic barriers for each country and the impact of each barrier, respectively. The average impact score of each barrier among all countries was also calculated to provide information on the barriers at the national level. This analysis gives us the opportunity to compare responses provided by each participant to a broader national context.

Regarding the Greek questionnaires, the highest-rated barrier is “No consumer awareness of benefits”, followed by “Exclusion of societal groups”, both falling under the category of social barriers. The lowest rated barrier is that of “No sustained commitment”. “No consumer awareness of benefits” is also the highest-rated barrier in the Spanish questionnaires, followed by “No belief in consumer empowerment”, indicating that the barriers are mostly related to social aspects rather than economic. The lowest rated barrier is “Lack of auditing procedures”. According to the Austrian questionnaires, there are four important barriers rated with 5, the highest possible rate, namely, “Neglecting value of system flexibility”, “Converting innovation into business as usual”, “No belief in consumer empowerment”, and “Perception of data security vulnerability”, representing a combination of barriers related to both social and economic aspects. For the Finnish questionnaires, mostly social barriers are prominent, with “No consumer awareness of benefits” being again the highest-rated barrier, similarly to the Greek and Spanish questionnaires. “No true participation for all actors” follows with an impact score of 3.5. The lowest-rated barrier is “Exclusion of societal groups”, unlike the Austrian, Greek, and Croatian questionnaires that rated this barrier as the second most important barrier. Finally, for the Croatian questionnaires, there are five significant barriers rated with the highest score of 5, namely “CAPEX”, “No sponsorship for CAPEX”, “Energy performance contracting risk”, “No sustained commitment”, and “No consumer awareness of benefits”, indicating that the barriers for Croatia are mostly related to economic aspects. The lowest rated barrier is “Lack of equal opportunities in wealth”, similarly to the Austrian questionnaires. Figure 7 displays the average rating of the SE barriers when responses are aggregated across countries.

As shown in Figure 7, the most impactful barrier that should be given primary focus when considering SE barriers in general is “No consumer awareness of benefits”. “Neglecting value of system flexibility” and “Exclusion of societal groups” were also highly rated (between 3.75 and 4) and should also be given attention as potential SE barriers. “Lack of auditing procedures” was the lowest-rated barrier. All other barriers had a moderate rating between 3 and 3.52.

3.1.3. Organizational Domain

The following section presents the quantitative analysis of the organizational barrier questionnaire. Responses were obtained from 11 organizations comprising six different stakeholder groups (aggregator, facility managers/ESCOs, network operators, RES operators, retailers, and urban planners) in the electricity data value chain across the five different countries. A quantitative analysis of the organizational and stakeholder barrier questionnaire per stakeholder group was performed, with scores averaged from each organization that falls within a specified stakeholder group.

Table 4. Average impact score of organizational barriers for each stakeholder group.

	Country
Question Number	Organizational Barrier	Network Operators	Facility Managers	RES Operators	Retailers	Urban Planners
OR-Q1	Lack of agency	4.00	3.80		4.00	2.75
OR-Q2	Inability to recognize data value	3.35	4.20	2.33	2.29	2.25
OR-Q3	No energy management personnel/systems	3.30	3.60	1.00	2.29	2.25
OR-Q4	Inability to combine energy data	3.90	4.00	2.00	2.00	2.50
OR-Q5	No knowledge of renewable energy		3.33	2.00	1.50	2.00
OR-Q6	IT infrastructure	3.80	2.60	2.00	3.57	2.25
OR-Q7	Lack of data governance	4.20	4.00	3.00	3.86	2.00
OR-Q8	No compatibility for multisource data	4.45	4.00	3.33	3.00	2.25
OR-Q9	Data complexity	4.00	4.00	3.50	3.57	2.25
OR-Q10	Inertia	3.85	4.00	3.50	2.43	2.50
OR-Q11	Neglecting external data value	3.45	3.60	4.00	3.00	1.50
OR-Q12	Data interoperability	3.45	3.40	3.33	2.57	2.00
OR-Q13	Closed ICT systems	3.90	1.60	3.00	3.29	3.00
OR-Q14	Data sharing	3.60	2.00	4.00	3.29	2.25
OR-Q15	GDPR	4.67	1.70		3.00	3.00
OR-Q16	No knowledge of data technology	3.27	3.75	3.50	1.96	2.00

presents the most relevant organizational barriers for each stakeholder group.

As shown in Table 4, for network operators, the OR barriers of GDPR, no compatibility for multisource data, and lack of data governance were rated as the most impactful, indicating that focus should be placed on the technical barriers of the activities for this stakeholder group. For facility managers, the OR barrier of the inability to recognize data value was rated as the most impactful, placing the focus on operational barriers of the activities for this SH group. For RES operators, the OR barriers of neglecting the value of external data and data sharing were rated as the most impactful and should be the focus of the activities for this SH group. The OR barriers of lack of agency and lack of data governance were rated as the most impactful for retailers, and these should be the focus of the activities for this SH group. The OR barriers of closed ICT systems and issues related to GDPR were rated as the most impactful for urban planners, indicating that focus should be placed on technical barriers of the activities for this SH group. Again, for the aggregators, attention should be placed on technical barriers such as no compatibility for multisource data and issues related to GDPR.

According to Figure 8, when aggregating the impact scores of the organizational barriers, the most impactful barrier is “No compatibility for multisource data”, followed closely by “GDPR”. The least impactful barrier is “No knowledge of renewable energy”. The aggregated results indicate that the most prominent barriers are related to the technical aspects of incorporating data analytics into a business compared to the operational aspects.

3.2. Qualitative Analysis

The collaborative approach utilized in the SYNERGY project was extended in the proceedings of this review by formulating an appropriate process to increase the information sourcing, acquiring further feedback as well as validating the main conclusions stemming from the survey and questionnaire-based analyses. Our analysis on RSEO barriers provided in the previous section revealed several issues that are identified by the questionnaire respondents as potential obstacles in driving the energy transition via data-driven services. These barriers naturally differ among countries and organizations. Nevertheless, impactful obstacles do exist; the aim of the interviews reported herewith has been to:

• Identify appropriate partners from the SYNERGY consortium that represent the full electricity data value chain;
• Evaluate comprehensively their responses from the initial questionnaire-based engagement;
• Engage with appropriate business leaders from their organizations that could offer an expert view on regulatory and organizational barriers.
• Define, structure, and perform dedicated interviews that aimed at revealing the main pains of the electricity data value chain stakeholders as well as providing further input, separate from the input provided by the business contacts working in the SYNERGY project, regarding gains expected to be achieved by a data value intensive project applied in their domain.

Initial stakeholder groups were as such created, representing the TSOs, facility managers, aggregators, RES operators, and DSO industry, respectively. Appropriate planning was undertaken to help the interviewees prepare for the interview and acquire the envisaged feedback from the industry experts. Specifically, the interviews were structured in three parts. The first part contains a generic presentation that focuses on the benefits (i.e., data preprocessing services, data outreach increase, analytics, and services) that the organizations can enjoy by being part of the data-driven service ecosystem. The second part comprises generic questions aimed at identifying the perception of the interviewee regarding the value of data-driven services for their organization as well as identifying any blocking issues on data exchanges or additional considerations that data-driven services could undertake further from the initial objectives provided in the presentation. Additionally, the second part contains partner-specific questions which were formulated to elaborate on the main barriers identified through their questionnaire responses as well as discuss any additional aspects that could enhance the design of data-driven platforms to help them overcome their identified constraints. For this reason, the identified most impactful barriers were communicated in advance of the interview. Finally, the third part is an unstructured part that aims to provide the time and environment to the interviewee to bring up any open feedback, concerns, or considerations related to their view as an industry representative and overall data-driven services in the energy sector.

The main outcomes from the interviews are summarized as follows:

• According to barrier “Inability to recognize data value”, the value of data from various sources is currently not recognized, and customers are reluctant to provide their data if the value of such provision is not clear. In addition, it indicates that there is a lack of understanding and skills in combining multisource data and analyzing/deriving insightful value from such processes.
• The access and handling of large volumes of data could be potentially minimized by using data platforms, specifically when such platforms are compatible with smart city platforms that are currently under development. The internal effort required in organizations for performing such big data handling processes could potentially be deferred, while additional overheads could be minimized by using technology on secure, automated contracts. As such, “Inertia” and “Converting innovation to business as usual” are potential barriers to this transition.
• The various barriers identified regarding data sharing such as “GDPR”, “Data sharing”, “GDPR”, and “Perception of data security vulnerability” indicate that users are not usually positive in sharing their data if their use is not clearly anonymized and/or remunerated.
• Data sharing from utilities has additional complexities that require alignment with internal policies and national/EU regulations, such as “GDPR”, which contain various levels of interorganizational constraints reflecting the numerous data owners. Furthermore, different approval levels for data sharing might exist as European utilities are on the verge of digital transformation nowadays, and data from different systems and users might require different approvals from various departments, as well as potential nondisclosure agreements with legal departments for particular data use.
• Regulation is driving innovation in the flexibility of energy systems, and when existing frameworks do not promote or facilitate such innovation, it is hard for regulated entities and aggregators to create the necessary skills and processes required to facilitate such developments. As such, the barrier “No holistic regulatory framework” points out the lack of a holistic regulatory framework that fosters innovation providing whole system benefits, e.g., no mechanisms for trading and remunerating flexibility.
• Innovative data-driven business models need to take into account also the cost of accessing assets “CAPEX” and “No sponsorship for CAPEX” that are not “connected assets” and as such require further infrastructure to enable data-sharing-driven business cases. Data quality, accuracy, and multisource compatibility are blocking the identification of the value of data (“No compatibility for multisource data”). Data linking interorganizationally is not always exploited, as denoted by “Inability to combine energy data” and “Data interoperability”.
• Data security aspects particularly related to the advancements of communications technology require further understanding. The translation and secure use or compliant cofunctioning of legacy protocols require further understanding (“No knowledge of data technology”).

4. Discussion

The quantitative analysis has provided valuable descriptive data on relevant barriers for countries and SH groups. However, an essential next step of this research was to build on these quantitative data through a qualitative analysis via interviews. These highlights, along with the remaining points raised from our analysis, provide useful inputs for the data-driven energy platform design, as well as further focus points on identifying and facilitating ways to overcome them. A summary of these outcomes can be observed in Figure 9. The inputs from the regulatory, socioeconomic, and organizational analyses were utilized to design interviews that aimed to specifically address any discrepancies identified from the above analyses, elaborate on any other barriers that the SYNERGY partners may foresee, and provide the opportunity to discuss practical implementation aspects. An initial stakeholder group was created with the aim to include a diversified and complete cluster of representative experts from the European electricity data value chain from partner organizations participating in SYNERGY.

The overall regulatory analysis indicated that no significant barriers exist in implementing data-driven services in the countries of Greece, Spain, Finland, Austria, and Croatia, and where minor barriers do still exist, they are expected to be addressed by mid-2021. For the regulatory aspects where there is still an absence of related regulations, such as for blockchain, data-driven platforms can be developed to facilitate transparency and flexibility in complying with future regulations.

A flexibility-related regulation/legislation allows the national TSOs to rely on the flexibility offered by flexible asset owners in order to safeguard network resilience and power adequacy. This has been realized through the implementation of flexible power auctions. At the same time, although there is no blocking regulation for the establishment of a flexibility market for the distribution level, there is a lack of an appropriate regulatory framework that will enable the launch of a flexibility market at the LV and MV level and (lack of) relevant incentives towards the D&T network operators. Such a regulatory framework would be necessary to establish participation rules, treat operational aspects, implement compensation schemes, etc.

The integrated approach of the grid dictates coordination and communication among operators such as distribution and transmission system operators. This may also include active entities that may play an important role in the operational scheduling such as energy communities as suggested by the related directive. In any case, there is no blocking regulation for the establishment of a cooperation context between operators, but there is a lack of an appropriate regulatory framework. eIDAS that are missing can be treated through conforming to the provisions of the relevant EU directive that is expected to be transposed to the national regulatory framework.

The integrated approach of the grid and the unified energy value chain dictates coordination and communication among operators for addressing operational challenges such as network asset management and planning. The energy value chain includes important players such as energy communities that can be responsible for managing their own assets as suggested by the related directive that is in place. No specific restrictions or hindering factors are present in relation to network asset management, and to this end, specific services can be used that will enable readiness at the operators’ side once the relevant EU directive is transposed to the national context.

Retailers need to have access to the data of their customers either as a sole party or as an aggregated entity through other players in order to estimate/manage the given services to the network operators either through bilateral contracts or through the market. This needs to be implemented in full compliance with EU Directives (eIDAS, ethics in AI) until they are transposed to the national context.

The main targets in the energy domain of energy community self-consumption maximization and energy costs reduction could be realized according to the existing regulatory framework since they refer to the very essence of the energy communities. Currently, this should be implemented by combining real-life and simulation actions in a hybrid framework until all relevant regulations are in place in order to support transparent benefit sharing through the use of blockchain technology. Furthermore, GDPR and blockchain technology should be addressed as complementary to each other to safeguard data protection and transparency.

Regarding the most impactful barrier in the socioeconomic domain “Lack of consumer awareness for the benefits towards flexibility”, it is generally observed that consumers have a positive attitude towards smart appliances and shifting of consumption but are not willing to change their habits or experience a reduction in their comfort unless they achieve substantial financial benefits [63]. As such, the future electricity system should be able to provide opportunities to increase consumers’ engagement through greater awareness and participation. Increasing awareness and enhancing societal perception can be achieved through the exploitation of mainstream communication channels, the attraction of additional societal groups, conducting information and awareness-raising campaigns, and organizing workshops or focus groups.

Increasing consumer awareness will also alleviate the second barrier, which is “Neglecting the value of system flexibility”. The new system could empower consumers by giving them access to information and tools to understand their energy consumption and manage their bills. Several energy analytics solutions focus on engaging consumers in energy management strategies by mainly exploiting building energy reports, high-level analytics, comparative feedback, and benchmarks. These comparative feedback mechanisms and generalized guidance interfaces are used to achieve high customer engagement and provide appropriate feedback to consumers that can be easily understood and can be used to generate value.

Finally, the third barrier, “Exclusion of societal groups”, can be addressed by involving a variety of stakeholders and consumers, along with their associated communities, in integrated collaboration activities for cocreation of shared value and cultivation of innovative ecosystems directly addressing emerging societal needs. The future energy sector should be an equality-based environment that constitutes an intersection between policy, society, and economy that offers win–win outcomes and mutual benefits among all members of our society.

All organizational barriers point out that specific attention needs to be given to the development of appropriate mechanisms of data-driven services towards safeguarding data confidentiality, protection, and privacy; enhancing anonymization; always considering the trade-off against utility; offering easy-to-use tools and interfaces for mapping data into common data models and structures; providing added-value services for increasing data quality through curation and integrity enhancement; and tackling inexperience in data management, data analytics, and data sharing by taking complex big-data-related issues away from the hands of the organizations and offering them in a data-as-a-service manner in a user-friendly manner.

5. Conclusions

The study and analysis of the regulatory domain comprised an extensive literature review to identify relevant regulations at the European level and formulate a detailed survey which was circulated to all pilot partners who participate in the SYNERGY project. The aim of the survey was to identify whether appropriate regulations exist at a national level and their impact on data-driven services. Considering the national enforcement of the EU policies in the countries of SYNERGY’s consortium, Spain is found to be the one that currently presents the most regulatory gaps pertinent to SYNERGY innovation, compared to Greece, Austria, and Croatia. However, most missing regulations in Spain were expected to be in place by the end of 2020, while more specifically, regulation on energy communities was expected to be implemented by mid-2021. On the other hand, Finland is the only country that presents no regulatory gaps relative to the identified EU directives. Policies related to the introduction of new technologies such as Electronic Identification, Authentication and Trust Services; smart contracts and blockchain; or ethics in artificial intelligence are currently missing from almost all countries. On the contrary, legislation relative to smart metering has been applied to all countries. The overall regulatory analysis indicated, however, that no significant barriers exist in implementing data-driven services in the countries of Greece, Spain, Finland, Austria, and Croatia; where minor barriers do still exist, they are expected to be addressed in the upcoming years. For the regulatory aspects where there is still an absence of related regulation, such as for blockchain, data-driven platforms [7] can be proactively developed to facilitate transparency and flexibility in complying with future regulations.

During the quantitative analysis of the surveys, it was observed that a few discrepancies were identified attributed to the perception of the questionnaire respondents, either from a personal capacity in their business role and their comprehension of each regulation or the perceived impact that each regulation has on their company. One of the largest discrepancies was the respondents’ perception of GDPR. When it comes to asset management and network planning, asset-related data are usually used in combination with consumer data (i.e., smart meter data). It is, therefore, apparent that GDPR might be highly impactful for the DSO who manages the smart meters, i.e., third party data, but might be of much less importance to the TSO who is concerned with proprietary data whose utilization may be internally governed following already existing rules based on regulatory reporting standards. Perception differences in the way that respondents understand the existing impact or the potential impact that a directive might have could be seen in relation to green power purchase agreements (GPPAs) between RES operators and retailers. The difference in perception on whether the Energy Communities Legislation has significant impact could be attributed to the fact that this directive currently does not affect per se the establishment of GPPAs using energy coming from PV plants owned/operated by an RES operator; however, if the Energy Communities Legislation will not explicitly facilitate the participation of energy communities in such agreements, it might restrict the scalability of such agreements, as potentially large green energy production segments such as energy communities might be excluded from appropriate regulation concerning certification of GPPAs. These discrepancies provided input to the interviews performed as part of the methodology in the form of embedding the input from this perception analysis into specific questions that aimed at capturing any barriers foreseen by the participating data value chain actors. As the initial engagement through the interviews showed that no major barriers exist, or where there are complexities, measures such as internal governance procedures and experimental agreements can be put in place to overcome these barriers, these discrepancies were attributed to the personal perception of the survey respondents as part of their business role or the specific role of their organization in relation to data-driven practices.

The socioeconomic aspects related to data-driven services were studied by means of an initial comprehensive literature review targeting the identification of such barriers through prominent literature sources. This state-of-the-art analysis was complemented by the relevant survey towards the participants which offered results that relate to the national level. When aggregating the scores from all countries, three main SE barriers were demonstrated to be the most impactful above all others. The most impactful SE barrier was related to a lack of consumer awareness for the benefits towards flexibility, opportunity, cost saving, and revenue generation. The second and third highest rated SE barriers were neglecting the value of system flexibility and the exclusion of particular societal groups, respectively.

Similarly to the regulatory domain, multiple large discrepancies were identified between the responses due to multiple organizations contributing to the averaged impact scores in the SE barrier analysis. In addition to the analysis of the barriers per demo case, the variance of scores was calculated for the demo cases where multiple organizations responded to create an average score. The most conflicting barriers are “CAPEX”, “No sponsorship for CAPEX”, “Neglecting the value of system Flexibility”, “Lack of equal opportunities in wealth”, “Converting innovation into business as usual”, and “No consideration for diversity of interests”. Following up on these conflict barriers during the interviews allowed us to identify why there is a difference of opinion between each organization with regard to a specific use of data-driven services. The CAPEX-related conflict barriers are highly dependent on the type of data processed by each organization. As such, any innovative data-driven business models need to be customized for the type of data processes and take into account also the cost of accessing assets that are not “connected assets” and as such require further infrastructure to enable data-sharing-driven business cases. Regarding system flexibility and business-as-usual innovation, regulation is driving innovation in the flexibility of energy systems, and when existing frameworks do not promote nor facilitate such innovation, it is hard for regulated entities and aggregators to create the necessary skills and processes required to facilitate such developments.

On the organizational level, some barriers are commonly highlighted in the results across almost all stakeholder types. The analysis done per stakeholder provides information on the most impactful barriers which affect a particular SH group across multiple organizations. It would be beneficial to investigate the barriers which are deemed important from the perspective of the SHs which are integral to data-driven services. When assessing the aggregated scores from all stakeholder types, the analysis presented here suggests there are four main impactful OR barriers. These barriers were related to a lack of compatibility for multisource data, issues related to GDPR, issues related to the complexity of data, and a lack of data governance.

The analysis done per stakeholder provides information on the most impactful barriers which affect a particular SH group across multiple organizations. It is therefore beneficial to investigate the difference in perspective of the surveyed stakeholders exhibiting a conflict of opinion with regard to the impact of organizational barriers. Such conflicting barriers include “Inability to recognize data value”, “Lack of data governance”, “Data complexity” and “Closed ICT systems”. To this end, during the interviews, questions related to these conflicting barriers were included in order to potentially provide an understanding of the perspective of each key SH. The interviews indicated that perception of data governance, recognition, and complexity is highly dependent on the type of data processed by each stakeholder (such as customer data, public infrastructure data, national interconnections data, and market data). In order to be able to analyze and derive insightful value from combining multisource data, the value of data from various sources should be clearly stated and understood. The recognition of data value can be increased by ensuring data quality, accuracy, and multisource compatibility, as well as exploiting data linking of information coming from different domains.

Finally, research related to data analytics is becoming more specialized and fragmented since numerous new methods and fields of application are emerging. As such, this review can be extended to analyze the barriers in the different areas of application of data-driven solutions along the entire value chain, such as transition, distribution, generation, consumption, and trading as described in [3]. Data-driven services can also range over different applications such as forecasting and prediction, controlling and monitoring, and clustering. The identified barriers can thus be further examined based on their impact on these applications. A direct extension of this review would be to include more countries not only on a European level but also on an international level that will potentially introduce additional regulatory barriers or a correspondence between the European regulations and the corresponding regulations existing in other countries. Additionally, identifying different case studies and business scenarios related to data-driven services will allow us to extend this research by considering the impact of barriers on specific test cases, similarly to the work done in [64]. This will give us a better insight into why these barriers affect data-driven services in each case study and also recommend how to take into account and overcome particular organizational and regulatory constraints.