Comprehensive Assessment of Smart Grids: Is There a Universal Approach?

Abstract

A comprehensive assessment of smart grids is critical for their development. Existing scientific research testifies to the urgency and complexity of the problem of implementing smart grids effectively, both in terms of a single project performance and from the standpoint of creating a local, and later global, energy system. The multidimensionality of smart grids makes it challenging to assess the effectiveness of their implementation. Difficulties in evaluation arise because it is challenging to consider technical, technological, economic, and other relevant aspects of smart grids’ development within a single evaluation system. There are currently a significant number of smart grid assessment systems. However, it remains debatable how systematically and comprehensively they measure the efficiency of a smart grid. This, in turn, raises the question of whether there is a universal evaluation system that integrally considers all the crucial components of smart grids and is suitable for evaluating smart grid projects of different content. This article analyzes the most well-known approaches to comprehensive assessment of smart grids according to the completeness of their coverage of smart grids’ most critical components. This paper identifies the essential areas for assessing smart grids according to the most cited and authoritative research and regulatory documents of the European Union. As a result, seven groups of indicators were identified. These groups of indicators are the basis for comparative analysis of the comprehensive assessment approaches of smart grids. The results of the study should help determine the appropriateness of applying a particular evaluation system. In addition, the analysis of existing evaluation systems allows the shaping of a universal approach to evaluating smart grids comprehensively and systematically.

1. Introduction

Energy system modernization responds to today’s economic, environmental, social, and political challenges [1,2]. Simultaneously, the introduction of smart grids is a crucial tool for change in this direction. After all, they meet the environmental and political priorities and requirements of the energy system and contribute to many strategic objectives, such as improving countries’ economic security and their associations [3,4,5]. A necessary condition for the spread of smart grids is the achievement of economic efficiency. This will allow for a large-scale transition from a traditional energy system to a smart one. This, in turn, will facilitate the growth of sustainable smart cities [6,7]. The development of smart grids can also contribute to economic growth, energy consumption, and decarbonization [8,9,10,11,12]. These problems are currently unsolved. Solving these problems will be beneficial to human society. The development of smart grids should be considered as a tool to reduce energy efficiency gaps [13,14,15,16,17,18,19,20,21]. It is one of the strategically essential tasks that integrate smart grids and sustainable development goals [22,23,24,25,26,27,28,29]. Solving this problem requires a balanced policy [30,31,32,33,34] considering technical and technological capabilities and limitations and innovative development [35,36,37,38,39,40,41,42,43]. However, the COVID-19 pandemic could break the implementation of existing smart grids development plans. It could lead to a temporary recession and deglobalization [44].

Currently, there are a significant number of technologies used to create new, and to develop existing, power grids. A considerable amount of research is devoted to determining the effectiveness of a smart grid’s technologies or components [45,46,47]. It affects the approaches to evaluating their effectiveness. After all, the assessment is carried out in a separate direction, rather than comprehensively. This approach allows a detailed analysis of a narrow problem, but does not answer the feasibility of implementing a smart grid’s efficiency. Determining the efficiency of smart grids should be based on a comprehensive and systematic approach to their evaluation. This helps identify the direct and indirect effects of smart grid projects, clarifies the prospects for developing a smart grid, and decides on the feasibility of a particular smart grid project.

Thus, a comprehensive assessment of the smart grid is important because it helps to improve the quality of monitoring achievements in the development of smart grids, as well as more efficiently using investment resources, including budget funds. This is especially true in difficult economic conditions, and in particular, due to the COVID-19 pandemic. It is also critical for developing countries. At the same time, the involvement of such countries in smart grids development is crucial for constructing a global energy system that meets modern requirements. The value of this is underestimated, because when developing countries do not make efforts to implement smart grids, it affects not only the efficiency of their energy system. It offsets the achievement of progressive, economically developed, and socially responsible governments (countries) in pursuing energy policy goals, implementing a circular economy, and combating climate change.

Approaches to the systematic evaluation of the efficiency of smart grid development have been considered in the papers [48,49]. However, these studies did not aim at establishing which of the studied evaluation systems most fully considers all the critical areas of a smart grid’s operation. Thus, based on these studies, it is impossible to conclude which of the comprehensive assessment systems has the most systematic and comprehensive approach to evaluating smart grids.

Logical and structural analysis is the fundamental methodological tool used in this research. The study is informed by the legislative and regulatory documents of the European Union, reports of governmental and non-governmental organizations on implementing energy policy, and scholarly research on smart grids published in scientific journals with a high citation index. By means of this approach, it was possible to identify the most significant directions for developing smart grids. Sufficiently considering each direction of a smart grid is a criterion that allows the comparative analysis of a comprehensive assessment system. The implementation of such an analysis reveals the advantages and disadvantages of each comprehensive assessment system available. The comparative analysis of assessment systems allows ascertaining whether there is a universal assessment system for smart grids which gives the best results. This is the purpose of this research.

This article has the following structure. Section 2 covers the most critical directions of smart grids. Section 3 describes the existing comprehensive assessment systems of smart grids. Section 4 is devoted to the methodology of comparatively analyzing the existing comprehensive assessment systems for exhaustive coverage of the directions defined in Section 2. Section 5 expounds the results of a comparative analysis of the existing smart grid assessment systems.

2. Materials and Methods

This section describes step by step the method used for comparative analysis of comprehensive smart grid assessment systems.

The first step of the research is to identify the existing comprehensive assessment systems. An assessment system can be called comprehensive if it takes into account several areas of smart grids. The analysis of scientific publications is a helpful tool for identifying such systems. The second stage of the study is to diagnose critical areas of smart grids. Bibliometric analysis, research of the legal framework, and reports of leading specialized organizations are sources for determining these areas. The next stage is to distribute indicators of existing systems for assessing smart grids in certain areas (groups of indicators). Redistributing indicators of existing assessment systems create an analytical basis used to determine the coverage of areas of smart grids’ development and operation. Necessary conditions for the indicators’ redistribution are the ability to attribute indicators to a specific direction of the smart grid’s development or operation, self-sufficiency of indicators, and the absence of its duplication. Subgroups of indicators were identified to be included in the analysis of more direct and indirect effects. Comparison of the existing assessment systems by the extent of covering smart grid areas will be carried out on the basis of scoring for each subgroup of indicators with the calculation of aggregate group score and final evaluation for each of the existing smart grid assessment systems.

The quantitative analysis of indicators for assessing smart grids involves the use of the following approach:

$$\left\{\begin{array}{c}{\mathrm{B}}_{\mathrm{i}}\ge 3\Rightarrow {\mathrm{A}}_{\mathrm{ij}}=\max ,\\ 2\le {\mathrm{B}}_{\mathrm{i}}\Rightarrow {\mathrm{A}}_{\mathrm{ij}}=\mathrm{avg},\\ {\mathrm{B}}_{\mathrm{i}}\le 1\Rightarrow {\mathrm{A}}_{\mathrm{ij}}=\min ,\end{array}\right.$$

(1)

where

• B_i is the number of indicators in the subgroup;
• A_ij is the score of the i-th subgroup within the j-th group;
• max = 3 points; avg = 2 points; min = 1 point.

Points are assigned so that if the assessment system for a particular subgroup has no indicators or has one indicator or group of indicators that evaluate one process, phenomenon, or event, this system receives 1 point. If the assessment system in this subgroup considers two different processes, phenomena, or events, then two points are awarded. Accordingly, if more than two processes, phenomena, or events are evaluated, three points are awarded.

Aggregate evaluation of existing assessment systems within certain groups of indicators is as follows:

$${\mathrm{I}}_{\mathrm{gr}}={\sum }_{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{A}}_{\mathrm{i}},$$

(2)

where

• I_gr is group assessment;
• n is the number of subgroups within the group.

Checking the adequacy of the distribution of smart grids’ efficiency indicators is carried out to avoid duplication of indicators for identified groups:

$$\left\{\begin{array}{c}{\mathrm{L}}_{\mathrm{i}}\in {\mathrm{Mas}}_{\mathrm{i}},\\ {\mathrm{L}}_{\mathrm{i}}\notin {\mathrm{Mas}}_{\mathrm{g}},\\ \mathrm{i}\ne \mathrm{g},\hfill \end{array}\right.,$$

(3)

where

• L_i is the index of the i-th subgroup;
• i is the identifier of the subgroup to which the indicator L is assigned;
• g is the identifier of any subgroup to which the indicator L is not assigned;
• Mas is a set of indicators of the subgroup.

Quantitative characteristics of the coverage of the smart grids’ areas by assessment systems, taking into account the adjustment that balances the value of indicators in groups with different numbers of subgroups, is carried out by the formula:

$${\mathrm{I}}_{\mathrm{agg}}={\displaystyle \sum }_{\mathrm{j}=1}^{\mathrm{m}}{\mathrm{I}}_{\mathrm{gr}}\cdot \mathrm{q},$$

(4)

where

• I_agg is the final assessment of the coverage of the smart grids’ development and operation;
• m is the number of groups of indicators;
• q is the equilibrium coefficient.

This study assumes the equivalence of all areas of development and operation of a smart grid. However, provided that there are priority areas for the development of smart grids, the calculation of the final assessment should be carried out using a weighting factor. The need to introduce weights to determine the priority of some areas of smart grids’ development can be justified by the objectives of state energy policy, and technical, technological, or organizational and economic constraints.

Weights can be calculated by direct estimation. According to this approach, experts use a score to determine the importance of the smart grids’ development during the development of a specific smart grid project or the implementation of state (regional or local) energy policy.

At the first stage, the average score for each area of a smart grid is calculated by the formula:

$${\mathrm{S}}_{\mathrm{i}}=\frac{{\sum }_{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{S}}_{\mathrm{ij}}}{\mathrm{N}},$$

(5)

where

• N is the number of experts;
• S_ij is the sum of points for each indicator.

At the second stage, weights for each area of a smart grid are calculated according to the formula:

$${\mathsf{\mu }}_{\mathrm{i}}=\frac{{\mathrm{S}}_{\mathrm{i}}}{{\sum }_{\mathrm{i}=1}^{\mathrm{m}}{\mathrm{S}}_{\mathrm{i}}},$$

(6)

where μ_i is the weighting factor of the group of indicators.

The calculation of the final assessment of maximizing the coverage of direct and indirect effects in all areas of a smart grid by existing assessment systems, taking into account the calculated weighting factors, is carried out according to the formula:

$${\mathrm{I}}_{\mathrm{agg}}={\sum }_{\mathrm{j}=1}^{\mathrm{m}}{\mathrm{I}}_{\mathrm{gr}}\cdot \mathrm{q}\cdot {\mathsf{\mu }}_{\mathrm{j}},$$

(7)

where

• I_agg is the final assessment of the coverage of direct and indirect effects in all areas of a smart grid by the existing assessment system;
• I_gr is assessment of the coverage of direct and indirect effects a particular group of indicators;
• m is the number of groups of indicators;
• q is the equilibrium coefficient;
• μ_i is the weighting factor of the group of indicators.

The proposed methodological tools for comparative analysis of existing systems for evaluating smart grids are shown in Figure 1.

3. Results

3.1. Critical Areas for Evaluating the Smart Grids Efficiency

This subsection describes the sources that make up the information base for determining areas for assessing smart grids. It contains information on the analysis of the information base, including bibliometric research and study of regulatory documentation in the field of energy.

Three sources formed the basis for determining the areas of evaluation of smart grids in this study. The first source comprises regulations governing energy sector development [50,51,52,53,54,55,56]. Energy regulation, including the establishment of environmental restrictions and requirements, cannot be ignored as an information basis for determining smart grids’ directions. In particular, in combination with financial and economic instruments [38,57], environmental regulation transforms the energy sector. It contributes to the formation of a global competition model [58], characterized by weakening the competitiveness of countries whose economies are heavily dependent on traditional energy resources [59]. However, determining the directions of smart grids solely based on the regulatory framework is limited because it does not fully take into account the set of effects that, directly or indirectly, stem from the operation of a smart grid. Accordingly, the use of the regulatory framework as a source for forming the evaluation system is appropriate to identify critical areas or indicators of industry development, but no more.

Another source for constructing the smart grid assessment system is the accumulated experience of companies implementing projects in this direction and reports of governmental and non-governmental organizations on the results of energy policy implementation [60,61,62]. However, the disadvantage of this approach may be in the fact that relevant companies and organizations take a more excellent account of the aspects in which they specialize. Instead, the implementation of smart grid components that are not directly related to such companies and organizations’ activities can be taken into account only partially or wholly ignored. It is advisable to rely on scientific achievements of specialized researchers, whose works have been published in reputable scientific journals, to avoid the shortcomings of the first two sources. A bibliometric analysis of publishing activity in the field of smart power grids was carried out in the paper [63].

The bibliometric analysis in the area of smart grids evaluation showed a significant interest in the researched question. In total, 1956 publications from the Scopus database (2000–2020) were reviewed. Three combinations of words with the logical operator “and” (“smart” AND “grid” AND “evaluation”) were applied. The “title, abstract, keywords” field of search in the Scopus database was used. The visualization of the results was made using the VOSviewer program to graphically map the material. The Scopus screening tools showed that in 2010, the number of articles that focused on smart grid evaluation began to increase. Using the VOS viewer program, the main nine clusters of the scholarly papers were identified (Figure 2).

The first (most powerful) cluster (red cluster) is formed around the concept of “smart power grids” (occ.—1175, total link strength—10434, links—881). This cluster includes the following main keywords: “smart grid”, “smart meters”, “phasor measurement units”, “smart cities”, “automations”, “network security”.

The second (green) cluster covers keywords such as “electric power distribution”, “reliability”, “electric utilities”, “quality control”, “decision making”, and others, and is formed around the concept “electric power transmission networks” (occ.—1175, total link strength—9990, links—885).

The third (dark blue) cluster is tied to the concept “renewable energy resources” (occ.—138, total link strength—1533, links—450) and covers keywords “renewable energies”, “solar energy”, “wind power”, “energy storage”, “optimization”, “solar power generation”, “costs”, “energy utilization”, and “sustainable development”.

At the heart of the fourth (yellow) cluster is the concept “smart grids” (occ.—113, total link strength—1083, links—391). The following main keywords belong to this cluster: “distribution system”, “power electronic”, “energy resources”, and “micro grid”.

The fifth (purple) cluster is formed around the concept “forecasting” (occ.—79, total link strength—797, links—273). It is suggested to include the following keywords in this cluster: “machine learning”, “learning systems”, “forecasting methods”, and “demand response”.

The sixth cluster (blue) is formed around the concept “energy management” (occ.—79, total link strength—797, links—273). This cluster includes the following main keywords: “energy management system”, “electric vehicles”, “charging (batteries)”, and “vehicle-to-grid”.

Other clusters are formed around such concepts as “photovoltaic cells”, “commerce”, and “Monte Carlo methods”.

On the international level, the investigated area was represented by the following countries: China, the United States, Germany, Italy, United Kingdom, India, Brazil, Japan, Iran, and Canada.

Figure 3 shows the prevalence of technical publications, in particular, in the field of engineering, informatics, and energy. The share of publications related to economic sectors remains low, but there is a tendency for their number to increase.

For an all-embracing evaluation of smart grids, publishing activity in engineering, energy, and computer science must intensify, and the number of publications in these areas is traditionally large. However, the development of economic, humanitarian, and multidisciplinary research is also essential, as it allows enhancing approaches to a comprehensive evaluation of smart grids (i.e., taking into account the economic, environmental, and social components in the evaluation more comprehensively).

The distribution of scientific papers in the critical areas is given in Figure 4.

Until 2009, the number of publications on this topic in business, management and accounting, energy, environmental science, social sciences, multidisciplinary studies, economics, econometrics, and finance was insufficient. The period of active interest and a flurry of publishing activity on smart grids began in 2009. Since then, there has been a trend towards increasing the number of publications. Figure 4 also shows that more than 71.4% of all research materials were published in the period from 2016 to 2020.

The highest citation rate—3319—was achieved in 2020. In addition, the largest number of citations per publication was in 2020 (15 citations per 1 publication).

At the same time, the share of publications in the field of energy in the period from 2016 to 2020 accounted for more than 3/4 of the total. Among publications in business, management and accounting, environmental science, social sciences, multidisciplinary studies, economics, econometrics, and finance, the most significant number of publications during this period belongs to environmental science (45%) and social sciences (33%).

But none of the above sources for determining the evaluation areas of smart grids are self-sufficient and allow forming a complete list of areas. The bibliometric analysis of smart grids assessment allows the determination of a limited number of relevant research areas of smart grids. However, based on semantic analysis, individual clusters focus on specific “narrow” areas of evaluation, and individual clusters can be combined based on semantic analysis into a single direction of evaluation. As a result, the number of areas for assessing smart grids according to bibliometric analysis in smart grids assessment differs significantly from the number of clusters identified. To fully reveal the potential of scientific publications in the field of smart grids as an information base for determining the directions of their evaluation, it is advisable to consider the results of the study published in the paper [63]. As mentioned above, the regulatory framework identifies priority areas for the practical implementation of smart grids. Specific regulations determine key areas, but it is possible to find other important areas for estimating the effectiveness of smart grids in such acts. Content analysis allows identification of these areas and their importance. The importance of using the experience of companies implementing projects in this direction and reports of governmental and non-governmental organizations on the results of energy policy implementation is that they help structure and detail the areas of smart grids evaluation.

The analysis of these sources allowed identification of seven areas of assessment of smart grids, which systematically and comprehensively cover the smart grid’s direct and indirect effects.

The seven groups of indicators defined include:

1.

Stability of the smart grid (safety and reliability)—a group of indicators that characterizes the technical aspects of smart grids’ operation concerning the declared operational parameters, ability to respond to destructive influences of natural and artificial natures, and to restore the entire grid;

2.

Information efficiency (information technology and cybersecurity)—this characterizes the use of information and communication systems for collecting, storing, processing, and transmitting data from smart grids (including monitoring, control, and automation functions). This group includes indicators of monitoring, control, and informatization of clients; the energy internet, and the risks associated with its operation [64]; and the use of ERP systems and decision support;

3.

Economic efficiency—a set of effects that can be expressed in monetary terms to determine the project’s cost, income, and expenses at all or some stages of its implementation. It is proposed to measure economic efficiency by indicators of volume, structure, and sources of capital investments; asset management optimization systems; and the formed business model’s efficiency [65]. A smart grid’s cost-effectiveness is essential for the transition from a centralized to a decentralized grid [66,67].

4.

Technical efficiency—a complex of characteristics of the technical and technological state of the network. This group includes indicators of equipment automation and process productivity, and technical and technological solutions for integrating distributed power generation into the power system.

5.

Environmental friendliness—a group of generalized indicators aimed at assessing the efficiency of using the potential of renewable energy [68]. This group includes indicators that characterize the level of eco-destructive impact and decarbonization of smart grids, including renewable energy technologies and land-use efficiency.

6.

Communication efficiency (openness and interaction)—a group of indicators that characterize the level of smart grids’ transparency for stakeholders through the use of information policy of equal access, universal procedures and standards, and involving consumers in remote energy management. It is worth noting that this component is closely related to the grid’s security [69].

7.

The availability of electric transport infrastructure is a group of indicators that evaluate the dynamics and geospatial parameters of the electrification infrastructure and consider its growth within the integrated smart grid.

3.2. Existing Smart Grids Comprehensive Assessment Systems

This subsection identifies and characterizes the existing smart grid comprehensive assessment systems that are widely used and studied by scholars.

Evaluation is an essential aspect of the smart grids’ deployment. It allows determination of the effectiveness of both the implementation of a particular project and the energy sector’s modernization based on smart grids’ performance. Comprehensive assessment systems are advantageous in this context because they consider direct and indirect effects of existing smart grids’ operation or determine the expected effectiveness of future projects at the stage of their development or selection.

Comparing comprehensive assessment systems requires selecting the most common and representative ones. Selected comprehensive assessment systems can be attributed to complex systems (i.e., those that evaluate several smart grid areas). However, as mentioned earlier, there are many “narrow” approaches to assessing smart grids’ aspects. These aspects include the cost-effectiveness of smart grids [70,71], the efficiency and development potential of alternative energy [72], pre-investment evaluation of energy efficiency [73,74,75], or smart grids’ environmental component [76]. This study did not analyze approaches to a comprehensive assessment of smart grids which do not provide excellent detail of the assessment processes [77,78].

Thus, the list of researched approaches to evaluating smart grids includes:

1.

IBM Smart Grid Maturity Model (IBM);

2.

DOE Smart Grid Development Evaluation System (DOE);

3.

EPRI Smart Grid Construction Assessment Indicators (EPRI);

4.

EU Smart Grid Assessment Benefits Systems (EUA);

5.

“Two Type” grid index system (TTS);

6.

Grid development assessment index system (GDA);

7.

Smart grid pilot project evaluation indicator system (PPE);

8.

Evaluation Model of a Smart Grid Development Level Based on Differentiation of Development Demand (DDD).

Each of these comprehensive systems of evaluating smart grids’ efficiency contains a significant number of measurable indicators, which can be hierarchically organized into a three-tier system. First-level indicators are a set of indicators that characterize a particular direction of a smart grid’s development or operation. Second-level indicators—detail indicators—represent a specific direction of a smart grid’s development or operation. Indicators of the third level are the most detailed indicators, which characterize the course or result of a smart grid’s operation processes. The study uses indicators of the second level for comparative analysis of these comprehensive assessment systems.

Before comparing the comprehensive assessment systems on the criterion of covering smart grids’ areas, it is necessary to consider the features of each.

The IBM Smart Grid Maturity Model (IBM), proposed by IBM, American Productivity & Quality Center (APQC), and the Global Smart Grid Federation [26], provides, in addition to evaluation indicators, an algorithm for developing a smart power grid. According to the vision of creating a smart grid, the smart grid maturity model provides for five smart grid maturity levels and eight components that need to be considered to build an efficient grid.

Based on research and materials [48,79], a smart grid’s maturity levels can be interpreted according to the IBM model as follows:

1.

Research and initiation—studying the possibilities of transition from the power grid’s existing model to a smart one. At this stage, it is characteristic of forming a vision of future transformations that are not included in a specific strategy or program of action;

2.

Functional investment—investing in one or more components of a smart grid, which provides a partial implementation of its functions;

3.

Cross-functional integration is the implementation of a limited number of a smart grid’s functions, which leads to the start of interaction and integration of the directions of its operation;

4.

Broad optimization (enterprise) is achieving the ability to integrate information and ensure a high level of control over the enterprise by transforming infrastructure and processes that can create new economic or trade benefits;

5.

Next-wave innovation is the grid’s ability to take full advantage of new operational, environmental, social, and business opportunities as they arise, and to thrive.

In addition to the stages, the Smart Grid Maturity Model includes eight components that form the analytical basis for smart grid planning [79,80,81,82,83]:

1.

Strategic management and regulation;

2.

Organization and structure;

3.

Technology;

4.

Social and environmental component;

5.

Network operations;

6.

Resource management;

7.

Customer relations management;

8.

Integration of value chains.

DOE Smart Grid Development Evaluation System (DOE). This system of evaluating the development of smart grids, like the IBM model, combines two approaches: first, the vision of a smart grid, strategic directions of its development, and end goals; second, indicators for evaluating the efficiency of a smart grid.

The DOE Smart Grid Development Evaluation System contains the following six parameters [48,61,62,80,81,82,83,84,85,86,87,88]:

1.

Enabling informed participation by customers;

2.

Accommodating all generation and storage options;

3.

Enabling new products, services, and markets;

4.

Providing the power quality for the range of needs;

5.

Optimizing asset utilization and operating efficiently;

6.

Operating flexibly when disturbances, attacks, and natural disasters happen.

Based on the smart grid’s six characteristics, the US Department of Energy offers an evaluation system that includes many indicators. The number of indicators varies depending on the type of smart grid project. The indicators can be grouped to simplify the evaluation system, as was done in [48]. Besides, the DOE Smart Grid Development Evaluation System indicators can be classified into technical and cost indicators. Technical indicators characterize a smart grid’s properties, which qualitatively reflect the degree of development and progress in its creation. Cost indicators quantify the results to be achieved due to the development and operation of a smart grid.

EPRI Smart Grid Construction Assessment Indicators (EPRI). The American Electric Power Research Institute (EPRI) has developed a system for building a smart grid and evaluating projects based on a matrix to assess a smart grid’s development [89,90]. Compared to the smart grid’s characteristics proposed by the US Department of Energy (DOE), EPRI emphasizes the importance of the grid’s self-healing function. It means that when using EPRI, the focus shifts towards the grid’s technical parameters compared to DOE. According to paper [48], the EPRI Smart Grid Construction Assessment Indicators areas for evaluating are: enables informed participation by customers; accommodates all generation and storage options; enables new products, services, and markets; provides power quality for the range of needs; optimizes asset utilization and operating efficiency; addresses disturbances through automated prevention, containment, and restoration; and operates resiliently against all hazards.

The EU Smart Grid Assessment Benefits Systems (EUA). The critical parameters for assessing smart grids in the EU are the characteristics of the energy market, safety, electricity quality, and environmental impact. Attention to the importance of environmental protection and other growing challenges forces the European smart grid to focus on the relationship between renewable and distributed energy, reliability, electricity quality, and creating added value of network services for all customers [48]. The European smart grids model focuses on an uninterrupted supply of electricity to consumers in compliance with advanced safety requirements, using innovative products and services, intelligent monitoring, control, communications, and self-diagnostic technologies [91,92,93,94]. The European smart grid is designed on the principle of the maximum possible involvement of consumers in distributed energy generation, assigning them the consumer and the electricity supplier’s role.

The main areas of assessing the efficiency of smart grids in accordance with the EU Smart Grid Assessment Benefits Systems are [48,95]:

1.

Increasing the resilience of the energy system;

2.

Sufficiency of the grid capacity to distribute and transport electricity produced from all energy sources to the final consumer;

3.

Harmonization and standardization of network connection procedures that provide access to it of any users;

4.

Improving the level of security and quality of energy supply;

5.

Increasing efficiency and improving service in the course of electric network functioning and power supply realization;

6.

Applying market mechanisms and effective support of the pan-European electricity market [96];

7.

Coordinated planning and development of networks with the involvement of common European, regional, and local energy networks;

8.

Economic efficiency of implemented solutions;

9.

Providing new business models and developing innovative products and services.

Based on the indicators included in these groups, the benefits of creating a smart grid are assessed. This evaluation system can determine the feasibility of implementing and developing efficient smart grid technologies and can evaluate and rank smart grid projects. It helps to select the most effective of them [48].

The “Two Type” grid index system (TTS) is helpful to evaluate the efficiency of smart grids. This system is mainly applied in the People’s Republic of China (PRC). Information about applying this comprehensive assessment system outside the PRC is scarce. The “Two Type” grid index system focuses on the efficient use of resources. The purpose of this assessment system is to determine the parameters of a balanced energy network, adapted to the conditions (state) of socio-economic development and aimed at optimizing its interaction with the environment. In essence, the “Two Type” grid index system is analogous to the sustainable development concept in China’s energy sector.

This system contains generalized indicators that characterize the general situation of energy network development and specific indicators that reflect individual processes.

According to the data presented in the study [48], the “Two Type” grid index system includes:

• Measurable indexes on the planning stage, construction stage, and operation stage;
• Effective indexes including resource-saving and environment-friendly.

The use of the assessment system is based on applying a tree-like hierarchy of goals, where the values of specific indicators are first determined in quantitative terms. Then, a system of generalized indicators is formed. The evaluation model first analyzes the work performed in smart grid planning, development, construction, production processes, and technical transformations, and then forms the measured indicators. They are analyzed from the standpoint of efficiency.

The Grid Development Assessment index system (GDA) is used mainly in China, and focuses on evaluating the development of the smart grid in such areas as economic development, growth of system capacity, and volume (scale) of construction. The system offers a method and a pattern for quantifying smart grid efficiency. The formation of the Grid Development Assessment index system (GDA) as a system for assessing the grid efficiency was carried out before the concept of a smart grid and its components were defined. The evolution of the Grid Development Assessment index system has led to forming areas in which a smart grid is assessed. According to one study [48], social and environmental factors have been added to the primary factors traditionally used to assess the grid, particularly safety and economy, which is useful to define the relationships among the various components of the smart [97].

The structure of indicators which the Grid Development Assessment index system takes into account covers the following areas: security, economy, quality, coordination, and intelligence.

The smart grid Pilot Project Evaluation indicator system (PPE) is based on dividing smart grid projects into three groups [48]: intelligent substations (or subsystems), automated distribution systems, and analytical systems (systems for collecting information on energy use). According to this classification of projects in smart grids, this system provides a means for assessing their effectiveness using some indicators. Based on analyzing such aspects as technical level, economic and social benefits, and practicality, it is possible to coordinate, generalize, and unify pilot projects to deploy smart grids. The smart grid Pilot Project Evaluation indicator system in all three areas were characterized in the paper [48].

The Evaluation Model of a Smart Grid Development Level Based on Differentiation of Development Demand (DDD), whose main provisions were developed in [49], focuses on determining the smart grids’ development in some territory. It allows the selection of a need for future action to improve the smart grid’s efficiency and helps to address the development of a new smart grid. The Evaluation Model of a Smart Grid Development Level Based on Differentiation of Development Demand has two components, divided into groups of indicators by individual areas. These components are called layers. The first is the effect layer, and the second is the base layer. The layer of effects is aimed at the development of smart grids and contributes to their development tasks. The base layer focuses on the structure of a smart grid [49]. In general, the system of indicators in this evaluation system is divided into three levels, which allows a detailed analysis of each direction of smart grids’ development.

3.3. The Comparative Analysis of Smart Grids Comprehensive Assessment Systems

This subsection covers the distribution of indicators of existing comprehensive assessment systems into seven groups and results of the comparative analysis, namely summary and generalized comparative matrix and profiles of smart grid assessment systems.

For comparing smart grid assessment systems, it is necessary to redistribute the indicators of existing assessment systems into seven defined groups. These groups are:

1.

The stability of the grid;

2.

Information efficiency;

3.

Economic efficiency;

4.

Technical efficiency;

5.

Environmental friendliness;

6.

Communication efficiency;

7.

Availability of electric transport infrastructure.

Groups and subgroups of indicators are defined according to the approach described in the subsection Critical Areas for Evaluating the Smart Grids Efficiency. These groups and subgroups of indicators are shown in

Table 1. The system of indicators for smart grids’ evaluation.

Indicator Group	Indicator Subgroup	Symbol
The stability of the grid	System self-recovery	S1ki
	System reliability	S2ki
	System security	S3ki
Information efficiency	Customer monitoring, control, and informatization system	I1ki
	Energy internet and customer informatization	I2ki
	ERP systems and decision support	I3ki
Economic efficiency	Capital Investments	E1ki
	Optimization of asset management	E2ki
	Forming business model	E3ki
Technical efficiency	Automation	T1ki
	Distributed energy generation	T2ki
	Productivity	T3ki
Environmental Friendliness	Reducing harmful emissions	Ef1ki
	Land use	Ef2ki
	The use of alternative energy and distributed energy generation	Ef3ki
Communication Efficiency	Openness policy	C1ki
	Interaction with consumers	C2ki
Availability of electric transport infrastructure	Electric vehicles	El1ki

.

The information base for distributing indicators for smart grids assessment is a set of the second level indicators, obtained from all available comprehensive systems for evaluating smart grids, which are studied in this paper.

Distribution of indicators is a complex and most controversial issue in this study. The problem is that similar indicators of different evaluation systems sometimes have differences in their calculation and differ in content. Each assessment system has indicators that evaluate or measure the same process (phenomenon, event). For the study, such indicators should be considered as one. These factors have led to an aggregated approach in the comparative analysis of smart grid assessment systems. The method of calculating specific indicators in the study was ignored. Instead, much attention was paid to their essence. Some redundant indicators that detail a particular process, phenomenon, or event that other indicators have already characterized are excluded from the study. It is done to simplify understanding of the research results and avoid excessive amounts of irrelevant data to achieve the purpose of the study.

The results of distributing indicators of smart grid assessment systems by groups and subgroups are shown in the

Table 2. Distribution of indicators of existing comprehensive assessment systems in certain areas of smart grids.

Group of Indicators	Indicators	IBM	DOE	EPRI	EUA	TTS	GDA	PPE	DDD
The stability of the grid	System self-recovery	sSR		sRT	sTT, sRPL		sIS		sDGR, sSSR
	System reliability	sDO	sSRI	sRF, sIR, sRGO	sIR, sLV, sLTT		sISt	sSRI, sVD	sRS
	System security	sSS		sCO	sLV		sISS	sSI	sAR, sNA
Information efficiency	Customer monitoring, control, and informatization system	iOA, iGA, iRA	iDE		iMC, iAM	iDT, iOS		iIF	iCSG, iSE
	Energy internet and customer informatization	iAC, iDEF	iCS, iOAr	iPD, iSCA	iIC, iNMT		iBSG, iASG	iEI, iDNS, iIS, iCSt	iSSO, iNI, iCSF, iBN
	ERP systems and decision support	iTF, iADM, iAsC, iPS	iDP		iIF, iUSM		iRC	iOM	iERP, iLAS
Economic efficiency	Capital Investments	ePI, eCA, eMI	eVI			eCC	eCE	eCC, eLP
	Optimization of asset management	ePS, eAO, eWS			eDNP, eAM	eIM	eAMO, eRE, eEC	eMOC, eSM, eLL	ePE, eTP
	Forming business model	eTF, eLM, eASP, eFBM			eNMA
Technical efficiency	Automation	tAS, tDD, tCEM, tDM, tSA	tAS, tCE, tAM		tSE			tIE, tMA, tIC	tTM, tSS, tFS, tDDM
	Distributed energy generation	tES, tBM, tDRS	tDRS		tDRS, tMC, tDGV, tIS				tBM, tDRS, tSDG
	Productivity	tCBM, tECO, tSEG	tRPO, tMM, tPF, tDP	tNP, tAUL, tRL	tML, tRL, tICT, tOP	tNI, tNB, tNO	tNB, tITL, tQC	tFI, tNS, tGES, tPO	tML, tESL, tPLO
Environmental friendliness	Reducing harmful emissions				efCO	efCO, efE	efEC	efCO
	Land use					efL		efL	efL, efEA
	The use of alternative energy and distributed energy generation			efDE		efS, efEM		efS	efWP, efDE, efUN, efEP
Communication efficiency	Openness policy	cMI, cDF, cRR	cEP, cEPR		cUS, cMI		cSH		cDD, cIS, cIO
	Interaction with consumers	cCP, cSA		cNC, cNM	cTR, cASG, cAP				cSP, cQA, cESC
Availability of electric transport infrastructure	Electric vehicles		elV	elVs	elVc, elI				elVs, elC, elDC

.

The list of indicators is included in Table 2 by groups. The stability of the grid, information efficiency, and economic efficiency are shown in Appendix A. The list of indicators used in the study by the groups technical efficiency, environmental friendliness, communication efficiency, and availability of electric transport infrastructure is shown in Appendix B.

The proposed algorithm of comparative analysis of smart grid assessment systems allows the determination of those that most fully characterize the efficiency of a smart grid. At the same time, it will enable determination of the specialization of each assessment system, establishing which areas of the smart grid are evaluated most fully, and which indirectly.

The results of the comparative analysis are shown in the summary matrix of smart grid evaluation systems in Figure 5.

This matrix is formed based on the results of scoring according to the method described above (

Table 3. Quantitative evaluation of assessment systems by coverage of the smart grids’ areas.

Smart grids Assessment Systems	The Stability of the Grid	Information Efficiency	Economic Efficiency	Technical Efficiency	Environmental Friendliness	Communication Efficiency	Availability of Electric Transport Infrastructure	Final Score
IBM	6	9	9	9	3	9	3	48
DOE	4	7	4	8	3	6	6	38
EPRI	7	5	4	5	5	6	6	38
EUA	8	9	5	8	4	9	9	52
TTS	3	5	5	5	8	3	3	32
GDA	7	6	6	5	4	6	3	37
PPE	6	7	7	7	6	3	3	39
DDD	8	9	4	9	7	9	9	55

).

To answer the question posed in the paper of whether there is a universal system for evaluating smart grids, it is also advisable to look at the consolidated results of comparative analysis of assessment systems. These results are presented in Figure 6.

Based on the data received as a result of the comparative analysis, it is possible to construct a profile of each of the investigated assessment systems. The profile emphasizes the strengths and weaknesses of the assessment system. It facilitates decision making on the appropriateness of assessment systems used in evaluating a specific smart grid.

The IBM Smart Grid Maturity Model is one of the best in covering smart grids areas and detailing evaluation results (Figure 7). The following groups are primarily well evaluated: information efficiency, technical efficiency, communication efficiency, and economic efficiency. A significant advantage of this evaluation system is the systematic and algorithmic evaluation, which accompanies all development stages of the smart grid project. It is typical for grid maturity assessment systems [98].

The logic behind forming this assessment system is that the smart grid project goes through mandatory steps, without which it cannot be successfully implemented. Each stage is evaluated according to quantitative and qualitative indicators, which allows formation of a system vision of the network, identifying and mutually agreeing on a complex combination of effects that affect the efficiency of a smart grid. It also allows consideration of the IBM Smart Grid Maturity Model as a possible basis for studying the stakeholders’ interaction in the smart grids’ development [99].

Step-by-step evaluation involves calculating similar in-value indicators at different stages of the project. This makes it possible to track smart grid development dynamics, identify problems of its development, and allow for corrective action. However, the weaknesses of the IBM Smart Grid Maturity Model comprise the incomplete consideration of aspects of network operation, such as availability of electric transport infrastructure and environmental friendliness.

Such an asymmetry in the evaluation system reflects the vision of smart grid development, which is based on high standards of manufacturability and safety, and self-sufficiency.

The DOE Smart Grid Development Evaluation System has high and above average estimates of the coverage of smart grid areas (Figure 7). Technical parameters, security, and application of information and communication technologies in the smart grid are evaluated in detail by this evaluation system. The difference from the IBM system is the higher level of formalizing indicators of electric transport involvement and development of electric transport infrastructure. In addition, unlike the previous evaluation system, there is no step-by-step measurement of network performance.

According to the study results, EPRI Smart Grid Construction Assessment Indicators is one of the most balanced approaches to determining the efficiency of smart grids (Figure 7). This assessment system is adaptive. The set of evaluation indicators can vary significantly, depending on the specifics of a particular project. The EU Smart Grid Assessment Benefits Systems is a common approach that is also characterized by balanced assessment indicators.

A significant part of this system’s indicators is aimed at determining the level of technical efficiency, productivity, process automation, reliability, and security of the energy system (Figure 7). Economic efficiency is assessed at the primary level. The environmental component has significant detail, but does not cover the full range of environmental impacts of smart grid projects.

The “Two Type” grid index system is a system for determining the efficiency of a smart power grid, which, unlike most of the considered evaluation systems, is characterized by a wide range of indicators of the smart grid’s impact on the environment (Figure 7).

The advantages of the Grid Development Assessment index system include a high level of covering such areas of smart grids’ development as sustainability, information, and communication efficiency. Instead, the ecological component and the development of electric transport infrastructure are insufficiently taken into account (Figure 7).

The smart grid Pilot Project Evaluation indicator system is a balanced system focused on determining the level of applied technologies and network efficiency based on a set of technical parameters and a detailed assessment of economic efficiency. Instead, such aspects as communication efficiency, development of electrical infrastructure, and use of electric vehicles are weaknesses of this evaluation system (Figure 7). In addition, this technique has a differentiated approach to evaluating different types of smart grids and their components. The Evaluation Model of a Smart Grid Development Level Based on Differentiation of Development Demand combines a comprehensive approach and a detailed algorithm for calculating indicators for most smart grids’ areas and has a balanced and systematic approach to their evaluation (Figure 7). This system essentially takes into account the direct and derivative effects of the operation of a smart grid. However, this evaluation system has significant limitations. It focuses on assessing the level of territorial development of smart grids.

4. Discussion

The systematic assessment of smart grids requires considering all areas of smart grids and a complex system of indicators. It is crucial to maintain a balance between the number of indicators and the coverage of the most relevant processes, events, and phenomena resulting from the operation of a smart grid. The areas of smart grids identified in the study are not an exhaustive list, but allow assessment of the most significant results of the smart grid’s operation. The analysis showed that all the assessment systems studied could be successfully applied to the evaluation of smart grids. The resulting data of using these systems will be high quality and reliable. However, each of the studied systems has its characteristics, advantages, and disadvantages.

After all, the study does not consider that the smart grids’ assessment can be carried out for different purposes. The assessment of the existing smart grids, the development of smart grids, and the evaluation of projects before their implementation may require different methodological approaches and evaluation systems. It is of interest for further research. In addition, suppose that there are one or more priority areas of the smart grid that should be evaluated as fully as possible. In that case, it is advisable to choose the evaluation system that has the appropriate specialization. The generalized comparative matrix of smart grid assessment systems obtained from the research facilitates the correct choice of the evaluation system.

5. Conclusions

According to the obtained results of the comparative analysis of assessment systems, the Evaluation Model of a Smart Grid Development Level Based on Differentiation of Development Demand, the EU Smart Grid Assessment Benefits Systems, and the IBM Smart Grid Maturity Model most fully cover the areas of smart power grids. In contrast to the Evaluation Model of a Smart Grid Development Level Based on the Differentiation of Development Demand, the latter two systems are characterized by an imbalance in the assessment of a smart grid, although they are structured approaches with a high level of detail. However, the results obtained do not mean that only these systems should be used for evaluation.

The study results do not allow one to claim the existence of a universal assessment system that effectively evaluates all areas of a smart grid. In addition, according to the comparative analysis results, none of the assessment systems has a sufficient set of indicators that evaluate the possibilities of smart grids’ development and integration, which are implemented through creating a business model. The IBM Smart Grid Maturity Model can be considered a conditional exception in this context. It is the established business model that can act as a catalyst for developing the energy sector based on smart technologies. However, its formation requires the coordination of the stakeholders’ interests. The formation of a business model as a critical factor in developing a smart grid cannot be ignored in systems for evaluating the efficiency of smart grids. Accordingly, there is a need to improve approaches to evaluating smart grids.