A Comprehensive Survey on Cyber-Physical Smart Grid Testbed Architectures: Requirements and Challenges

Abstract

The integration of improved control techniques with advanced information technologies enables the rapid development of smart grids. The necessity of having an efficient, reliable, and flexible communication infrastructure is achieved by enabling real-time data exchange between numerous intelligent and traditional electrical grid elements. The performance and efficiency of the power grid are enhanced with the incorporation of communication networks, intelligent automation, advanced sensors, and information technologies. Although smart grid technologies bring about valuable economic, social, and environmental benefits, testing the combination of heterogeneous and co-existing Cyber-Physical-Smart Grids (CP-SGs) with conventional technologies presents many challenges. The examination for both hardware and software components of the Smart Grid (SG) system is essential prior to the deployment in real-time systems. This can take place by developing a prototype to mimic the real operational circumstances with adequate configurations and precision. Therefore, it is essential to summarize state-of-the-art technologies of industrial control system testbeds and evaluate new technologies and vulnerabilities with the motivation of stimulating discoveries and designs. In this paper, a comprehensive review of the advancement of CP-SGs with their corresponding testbeds including diverse testing paradigms has been performed. In particular, we broadly discuss CP-SG testbed architectures along with the associated functions and main vulnerabilities. The testbed requirements, constraints, and applications are also discussed. Finally, the trends and future research directions are highlighted and specified.

1. Introduction

In previous decades, the global consumption of electricity has jumped approximately three times from $7.323$ TWh in 1980 to $22.3$ TWh in 2017 [1]. Consequently, to adapt to the considerable consumption increase, the National Institute of Standards and Technology (NIST) rolled out nationwide endeavors to develop the next-generation electric power system, commonly referred to as the SG [2]. Comparing the conventional power grid with SG, the investments and improvements have improved the efficiency, reliability, and sustainability [3,4]. The critical infrastructure (CI) of SG has been considered to be at high risk for cyber-attacks [5,6,7]. Because of the SG idiosyncrasies, the disruptions of the SG due to cyber-attacks transcend the cyber-realm to affect the physical realm as well. Hence, the combination of both cyber-security and physical security leads to the term CP-SG security [3].

Essentially, cyber-physical systems are the leading influence of the upcoming smart grids as an integrated groundwork system comprising of computing techniques, communication network, and exogenous structures [8].

A typical end-to-end CP-SG architecture consists of three main layers: the SG application layer, the power or physical layer, and the communication layer [9,10,11]. The physical layer can be divided into two sub-layers including the physical structure that hosts the external system being controlled and the smart connection of sensors-actuators. The application layer provides all CP-SG applications and services that interface between the CP-SG and human operators. The network layer provides a communication environment for transmitting the data collected from the surrounding environment and thus actuates on a prescribed physical system, while the cyber-layer is responsible for producing the decisions and providing the communication structure for the CP-SG [11].

In large Industrial Control Systems (ICS), the cyber-layer is typically composed of a Supervisory Control and Data Acquisition (SCADA) system [10,12]. The integration of SCADA by CP-SG poses numerous challenges to the stability, reliability, and resilience of the power grid. Hence, the cyber, cognitive and human composite inter-dependencies, which may lead to system failures, malfunctions, and faulty events, and tend to aggravate security vulnerabilities, need a robust framework that can handle them [13].

The integration of Industrial Process Automation (IPA), which resulted in SCADA system implementation, has reduced the Operation Cost (OC) and is expected to double the investments in the market. As mentioned in [14] “the electrical SCADA market was USD 1.42 Billion in 2016 and is expected to grow at a Compound Annual Growth Rate (CAGR) of 7.48 percent from 2017 to 2022 to reach a market size of USD 3.29 Billion by 2022”. As such, designing a robust supervision system and providing it with effective data/information processing and appropriate evaluation of the system performance is done by identifying the critical vulnerabilities first.

ICS system operators are looking for appropriate, reliable, and secure techniques to implement different layers, for next-generation SG, such as monitoring, metering, operation, automation, protection, and markets [15,16,17,18]. However, because of the complication and diversity of CP-SGs, a high fault/fail ratio and security risks are presented and lead to major privacy, reliability, safety, and economic issues [19], which are essential matters in CP-SGs. Consequently, CP-SG examination is essential prior to the deployment in real-time systems [20,21,22]. Such examination should be performed for both hardware and software components of the SG system prior to the actual system construction. This can take place by developing a prototype to mimic the real operational circumstances with adequate configurations and precision.

The majority of testbeds are primarily developed for specific task assessment and validation. Some testbeds provide insights for specific research fields, but few give a full hardware and software assessment policies for research purposes. The closely connected network infrastructure of the SG necessitates a comprehensive testbed concept to have the ability to enable a variety of research needs. All SG fields need to be assessed independently and should be connected to attain a comprehensive consciousness of the continuing research. SG testbeds involve two parts including the physical/hardware elements and cyber/software packages. The physical part contains three main elements including the generator bases, transmission lines, and load models, while the cyber part comprises communication and information infrastructure. Implementing the testbed systems requires a high-cost and experienced workforce. Also, SG investigations necessitate electrical research and practical tests such as protection and energy management as well as rigorous safety conditions for testbeds. Furthermore, owing to the excessive financial and safety requirements of hardwired power systems, the majority of testbeds embrace simulation/emulation models. Quite a few of them are qualified for conducting experiments using real physical components such as the actual generators or distribution connections. While the current testbed models are limited, modern research is investing in the area of SG. Indeed, it is predictable that testbeds with a variety of functionalities will be developed in the foreseeable future [23]. The main objectives of this work can be summarized as follows:

• Provide a thorough investigation of CP-SG along with the vital role of testbeds in actual implementation.
• Explore the latest developments of testing techniques for CP-SG architecture including dedicated functionality paradigms and objectives.
• Summarize the techniques for evaluating CP-SG regarding the modelling techniques, tools involved, simulation methods employed, as well as vulnerabilities and threats.
• Evaluate the features and functionalities of the existing testbeds.
• Discuss the current and future trends that are needed to be considered while building new CP-SG testbeds or rehabilitating the existing ones.

The research questions and subsequent methodology adopted in this paper include:

• What is the current status of the CP-SG and CP-SG testbeds? This has been addressed by conducting a thorough investigation on CP-SG along with the vital role of testbeds in actual smart grid implementation.
• What are the latest testing techniques employed in CP-SG architecture including the functionality paradigms and objectives? This has been addressed by summarizing the essential components of the CP-SG testbeds and by categorizing them based on their respective layers.
• What are the gaps that need to be addressed to improve the quality of CP-SG testbeds? This has been discussed by summarizing the techniques used to evaluate CP-SG vulnerabilities and threats. We have also assessed the features and functionalities of the existing testbeds.
• What is the cutting-edge research in this field and future trend that are needed to be considered while building new CP-SG testbeds or rehabilitating the existing ones? This has been addressed by highlighting the important current and future trends implemented in CP-SG testbed deployment to provide valuable insights for CP-SG researchers to create their own CP-SG testing environments.

The remainder of this paper is organized as follows: Section 2 presents the CP-SG with emphasis on the incorporation of Information-Communication Technology (ICT) into the power system then the challenges of CP-SG followed by the vulnerabilities of CP-SG and ended with the needs of the testbed. Section 3 describes the CP-SG Testbed, which highlights the essential components of the CP-SG testbed, requirements, Classification, overview of the existing testbeds, challenges and limitations of testbeds, and recent advancements made in testbeds. The conclusion of this paper is provided in Section 4.

2. CP-SG Perspective

Conventional power grids have drawbacks including varied generator types, high cost and expensive assets, time-consuming demand response (the time to reduce the stress on the power grid and high electricity prices), and relatively high carbon emissions. On the other hand, the SG power network relies on communication and information technology in all its domains of generation, transmission, and consumption of energy, which provide real-time effective control, operational efficiency, grid resilience, and decreased carbon footprint [24]. SG structure, as illustrated in Figure 1, provides the vital infrastructure and communication channels allowing real-time bidirectional interaction between utility companies and consumers. The dissemination of the information via digital mediums such as Fiber Optics (FO), Power Line Communications (PLC), Digital Subscriber Line (DSL), etc. to provide transport of high bit-rate digital information. The advantages of this communication are observed in processes that allow auto metering and maintenance, efficient energy management, self-healing, reliability, and security [25,26,27].

The operation of power systems is undergoing a rise in complex cyber-infrastructures which is a crucial component of the future SG. As the spread of SG deployments in power systems continues, threats to its critical infrastructures have increased substantially [28]. CP-SG consists of a collection of interconnected physical and computing resources employed in order to achieve a specific mission [29]. In the critical infrastructure networks, CP-SG plays a significant role from power distribution to utility networks. The evolving concept of SG is a significant critical CP-SG infrastructure that depends on two-way communications between SG devices to enhance efficiency, reliability and reduce costs. However, compromised devices in the SG lead to numerous security challenges which can be costly to manage. As a result, new security concerns arise from the use of CP-SG [30,31]. Therefore, vulnerabilities of the widely automated distributed power system security have become a significant target for attackers which is getting the attention of government, energy industries and consumers [24].

A typical CP-SG incorporates cyber system, as represented in the control center, containing computation and communication operations and physical system managing the utility networks as a whole. According to the CP-SG model of Figure 1, sensors are used to measure physical quantities, after which they are converted into electrical signals. These electrical signals are sent to the control center to be sampled for computing. The CP-SG uses various algorithms including state estimation to analyse the sampled values and sends control commands to the physical system, as represented in the substations, through the actuators which convert the electrical signals into physical actions [32,33]. Power system engineers can control tasks based on data acquired from remote facilities thanks to the integrated ICT in SG. In CP-SG, communication between remote sites and control centers can be performed using public or private networks as a result of the wide geographical area. Therefore, the ICT systems support the on-line data acquisition to monitor and control the power system [34].

2.1. Incorporating ICT into Power Systems

ICT schemes have turned into a substantial portion of each facet of our everyday life and its assimilation into the power networks has been increased to suit the growing need in the electric power system [35]. As shown in Figure 2, the implementation of ICT consists of four essential categories for the operations of power system including acquisition, implementation, processing and communication of subsystems [35,36,37,38,39].

• Acquisition: Collects system data such as instantaneous power flow measurements, bus voltages and frequency calculations, the state of circuit breaker, the status of switching tools and then conveys all the data via the communication medium to the processing stage.
• Processing: Based on data collected from the acquisition stage, regulates the state of power system and notifies the system operator about the status of the real-time system. At this stage, the existence of state estimator tool is crucial in order to obtain reliable power network based on suitable decisions for implementation.
• Implementation: Carries out the required actions based on the system processing results. Some of the actions include activating the protecting relays and circuit breakers next to any failure detected on the power system.
• Communication: Coordinates all subsystems with the power network using wired or wireless communication channels. ICTs deployment in power system assists operators to be efficient and improve the power system reliability. Intelligent applications such as intelligent monitoring, protection and control, two-way real-time communication enhances the situational awareness of the network. Consequently, this helps in making correct decisions to operate the power system efficiently. Adequate awareness ultimately improves the reliability of the power system to its highest levels [40].

The frequency and duration of power supply interruptions have decreased drastically with the integration of ICT. In addition, ICT infrastructure implemented in power network provides efficient Energy Management System (EMS) as the increase of its size and complexity has contributed to a reduction of total operation costs [36]. The overall power system reliability and efficiency are enhanced as the increase of the tight coupling of the ICT and power system in the cyber power network. Therefore, these ICT factors should be considered adequately in the planning stage of the power network [35].

2.2. Challenges in CP-SGs

The development and use of CP-SG is confronted with several challenges related to its design, operation, scalability and cybersecurity.

1.

Design and Operation Challenges: The CP-SG consists of many applications that are designed for achieving various purposes such as improving small-signal stability and transient stability, enhance voltage profiles, reduce power losses, and minimizing the probability of cascading failures [41]. Such applications require fast data sampling, otherwise the expected performance might not be attained. Therefore, latency challenges in the communication networks prompted by time delay might result in link failure and packet loss. Hence, this factor has to be effectively considered in designing CP-SGs. On the other hand, some applications does not require such fast data sampling resulting in minor impacts on CP-SG performance [41,42].

2.

Cybersecurity Challenges: The SG network comprises several legacy systems combined with modern technologies and architectural methods within a common interaction communication system to confront the challenges of the forthcoming electric power network systems. To achieve this goal, cybersecurity architecture for future power grid communications is based on cybersecurity requirements, legacy installation dependency, and regulations and industry standards. However, major challenges [43] are facing the desired objective to obtain a secure power grid communication system which includes the following:

• Security Policy and Operation: Adequate operations of many components of the power grid and the connection between them determines the reliability of the power grid. Federated identity management is essential to consider which authenticating entities involved in power production from a remote organization [43]. Technical solutions are possible to such issues based on different security policies such as Security Assertion Markup (SAML) [44], Web Services Trust (WS-Trust) [45], and PKI [46].
• Security Services: Network operators are capable of identifying, controlling, and managing the security risks of the power grid with the assistance of security services. The typical security services in the future power grid are described by the operationalization of cybersecurity across people, process, policy, and technology foundation for each organization.
• Internetworking: Due to the lack of built-in security of power grid communication networks applications and devices, vulnerabilities are introduced across the various internet-connected networks. Commercial networks connected to the Internet might trigger the grid to threats in a form of multiple types of attacks which cause interruption of power [43].
• Efficiency and Scalability: To ensure efficiency and scalability, system availability should be seriously considered in cyber-physical networks such as SG. Key issues have to be considered to fulfill the availability objective of a system including the efficiency of the computation and communication resources, adequate error management functions for handling failures, proper redundancy built-in to avoid system collapses, and auxiliary security support functions to detect cyber-attacks [47].
• Differences between Enterprise Network and SG Network Security objectives: For SG, the most significant objective is ensuring the reliability of the system and protecting the equipment and power lines. On the other hand, enterprise networks are mainly concerned with data protection including data integrity, confidentiality, and availability. The differences in objectives between the two networks are challenging the cybersecurity as the enterprise networks are short in providing cybersecurity solutions at the control and automation levels [43,48].

2.3. Vulnerabilities of CP-SG

The heterogeneity and complexity of the CP-SG introduce significant difficulties to the privacy and security of the CP-SGs. The composite cyber-physical interaction poses vulnerabilities to both cyber and physical systems of SG [49].

1.

Cyber Vulnerabilities: a Cyber-vulnerability is identified as the weakness that can be used by an attacker to execute harmful activities on the CP-SG parties using a networked system. Cyber-vulnerabilities of the CP-SG can be usually targeted through CP-SG communication, CP-SG software, or CP-SG privacy.

• Communication Vulnerabilities: Local area networks in substations are Ethernet-based networks that are vulnerable to interception and Man-in-the-Middle (MitM) attacks. Such attacks enable attackers to impersonate components and injects false data as well as releasing confidential information [50,51,52]. The information infrastructure of the power network depends on limited internet protocol standards involving known vulnerabilities that might be used to launch attacks on the network. The connectivity of some communication protocols such as the Internet-connected general-purpose TCP/IP is supposed to connect to control centers. Though, due to network misconfiguration, the Internet-based networks are connected directly or indirectly to control centers which cause vulnerabilities to the network [49,53].
• Software vulnerabilities: Servers in control centers that are internet-connected to the local network can be vulnerable to malicious attacks impacting the desired operations. Historical and customer information could be exposed through web-related vulnerability using SQL injection that enables attackers to unauthorized access to database records [51,54,55]. Some devices such as the Expanded Smart Meters (SMs), which can be upgraded remotely, bring about critical vulnerability. Such features open doors to attackers to control switches causing blackouts [49]. In addition, software bugs can take advantage of such vulnerability by malicious attackers as the network components are accessible in every household [56,57].
• Privacy vulnerabilities: The two-way communications connecting the customer’s meter to the utility are providing a new type of vulnerabilities regarding customer’s privacy. Private information of customers such as daily habits and the presence or absence can be exposed by attackers seizing traffic from smart meters [49,58].

2.

Cyber-Physical Vulnerabilities: Cyber-physical vulnerabilities are identified as the weakness resulting from the integration of the Cyber part with the physical part of the CP-SGs. Cyber-Physical vulnerabilities of the CP-SG commonly exist through the network communication vulnerabilities or smart meters vulnerabilities.

• Network Communication vulnerabilities: Power system infrastructure depends on protocols such as Modbus and DNP3 in which each protocol is vulnerable. Modbus protocol, which is the standard communication in many ICS, is limited to basic security measures which make it vulnerable to a variety of attacks [49]. Attacks such as eavesdropping attacks resulting from lack of encryption make data integrity disputed [49,59,60,61]. Unlike the Modbus protocol, DNP3 has a simple integrity measure using a Cyclic Redundancy Check (CRC). Similar to Modbus, DNP3 protocol has no encryption or authentication mechanisms [62,63].
• Smart meter vulnerabilities: Interactions between the two-end communication of smart meters pose serious security concerns. Smart meters might have back-doors that could be taken advantage of the factory login account which gives full control to the user over the SG as Santamarta analyzed in [64]. Another major security weakness is that the communication is transmitted through telnet which sends unencrypted data in “cleartext”. As attackers take over the control of smart meters, power disruption occurs by malicious interactions with other devices or inject wrong data to make wrong decisions. Also, attackers could use the meter as “bot” to launch attacks against other systems within the network. In addition, the power bill can be changed to false data in order to reduce the power cost [64].

2.4. CP-SG and the Need for Testbeds

The two-way power and information flow enhanced the conventional power grid forming a smart grid that is equipped with intelligent features of self-healing, customer involvement, and adaptive protection and control [23,65]. The major driving force of the smart grid is the implementation of CP-SG as a foundational support system despite its deployment challenges. Adequate ways to implement future smart grid concepts are the focus of power systems operators to obtain security for different application layers such as operation, monitoring, metering, protection, automation, and markets [15,16,23].

The complex nature of smart grid structure requires the implementation of testbeds including different capabilities for extensive experimental verifications. Prototype implementations are required to achieve real-world application results on actual testbeds. Such testbed implementations and their fast verification will stimulate research results for the power systems industry. In addition, testbed provide educational platforms for researchers with multi-user experimental facilities and proof of concept of verifications for numerous smart grid domains [23,66]. Cybersecurity vulnerabilities and interpretability are major concerns to be tested using properly developed smart grid testbeds with extensive capabilities. However, most testbeds do not provide complete hardware and software platforms to test for all research applications simultaneously. As a result, the tightly coupled structure of the smart grid necessitates a comprehensive testbed structure in order to facilitate experiments at the same time [67,68].

3. CP-SG Testbed

Following the widely publicized cyber-attack events in the smart grid in recent decades, the security and resilience of the ICS have become a major concern to power operators and various governments alike. These attacks are a result of the enhanced integration of information and communication technologies into the control and operations of the power grid. As discussed in the previous section, to overcome these challenges without impacting the real-time environment negatively, testbeds are required for the exploration, development, evaluation, and validation of security controls and algorithms in the power system. The cyber-physical testbed must be able to faithfully model and simulate the power grid for the test and validation of various control, operation, and security algorithms. The testbeds are also important for vulnerability and impact analysis of cyber-attacks on smart power systems.

Testbed design and development have grown in the past few years to the point where replicating ICS networks through simulation and modeling is considered a viable option to explore and address cybersecurity challenges [69]. This is partly due to the potential negative impact of testing cyber-attacks on the live power systems and also the high cost of deploying and using real system hardware and software for testing purposes [70]. Hence, testbeds make it possible to use a model of an actual power grid rather than directly working on the real physical system.

Testbeds that are able to successfully integrate both cyber and physical components of the smart grid provide ideal environments to perform and evaluate research efforts geared towards making the power system more resilient. Unfortunately, the development process of the testbed is not well established because of the complexity involved in integrating the required cyber and physical resources while also incorporating simulation mechanisms needed to model power systems, cyber network dynamics, and security events [71]. Several design strategies will normally lend themselves to different research endeavors. Therefore, a full understanding of the testbed architecture and its development constraints are important to enhance future efforts in ICS research.

A generic physical architecture of a typical CP-SG testbed is illustrated in Figure 3. The testbed is essentially sectioned into three layers; the physical layer, the cyber layer, and the control layer. The physical layer consists of the physical components in the substations such as Remote Terminal Units (RTUs), Real-Time Digital Simulator (RTDS), and Intelligent electronic devices (IEDs). The cyber layer is what makes the power grid smart. It makes provision for real-time communication between components in the substation and the control center. It also enhances the automation of various processes in the smart grid. As shown in Figure 3, the control layer is essentially the control center where all measurements are analyzed and control actions are communicated to the substation for the actuators to carry out.

3.1. Essential Components of CP-SG Testbeds

The essential components of the CP-SG testbeds in the electric power domain can be categorized into communication, control, and power systems. The measurements and status data required for situational awareness of the substations are obtained in the power system model. Both the measurement and the commands required for the smooth running of the power system are transmitted between the substation and the control center via the communication system using various standard protocols and communication media. Various equipment used and deployed in the control center are all categorized under the control system.

The logical architecture of the testbed is illustrated in Figure 4. The power system component of the testbed can either be an actual power system or simulated as is the case in the RTDS [71]. As illustrated in Figure 4, the measurements and actuation commands are either sensed directly from a physical device (represented as A) or simulated and transmitted over the network (represented as B). The RTDS implementation is more widely used as it is more economical. Item C shows how information such as device status, measurements, and protection commands are transmitted through the substation. If regional control is simulated in the testbed, it is carried out as shown in item D where substations communicate with regional control and energy management functions via Wide Area Network (WAN). Finally, item F shows how inter-control center communication also via WAN is executed for system scheduling. A detailed description of these components is provided as follows:

• Power System: Owing to the fact that the control center software is time-dependent, power system simulation in the testbed must run in real time and operate for a long period of time with high precision. These are the factors that make RTDS [72] the most widely used application for power system simulation in real time [73,74,75]. The RTDS is not only capable of implementing a real-time electromagnetic transient simulation of the power system, but it also supports several data interfaces which makes it convenient for data exchange with other external devices in the testbed setup. PowerWorld simulation tool [76] is another commercially available application used for simulating the operation of large power systems [77,78,79]. It is a flexible and useful tool to simulate system states and perform contingency analysis. This tool is also very efficient for cost analysis, power flow analysis, and voltage control. The DIgSILENT “PowerFactory” software is another popular application used for power system simulations [80,81,82]. It is often used to perform non-real-time power system simulations. Although it does not have the inter-connectivity capability that the RTDS has, it provides means for more advanced system analysis and includes algorithms for state estimation and contingency analysis [71].
• Control System: The CP-SG testbeds typically use industry-standard software and protocols for all control functions to achieve more realistic cyber vulnerability research. The control system part of the testbed is usually made up of a multifunctional control center and substations [71,74]. EMS from different vendors are adopted in the testbeds to act as the master station at the control center. Either hardware-based or software-based RTUs are implemented to provide consistency with real events in the power system. The RTUs are responsible for aggregating data from the power system (e.g., RTDS model) and transmitting them to the control center. It also serves as the channel for the remote control and remote regulation signals produced by the control center.
• Communication System: The essential communication components of the testbed includes physical network architecture and substation network protocols. It is very important to create a WAN network within the testbed which mimics the inter-connectivity of the control center to the RTUs in the substations. The IEC 61850 protocol is typically deployed to communicate status and commands between the IEDs and RTUs within a substation. On the other hand, the communication between the control center and the RTU is usually performed with the DNP3 similar to real world SCADA systems. Modbus is another substation protocol that can be implemented in the testbed. It is the actual standard that is openly published, and is approximately used for 40% of the communication within industrial appliances [83]. In the CP-SG testbed, it can be used for communication between master stations and RTUs [77,78,84].

3.2. Testbed Requirements

A CP-SG testbed should provide an avenue for researchers to deploy, test, and verify complex scientific methods and algorithms by ensuring the fidelity, repeatability, measurement accuracy, and safe execution of experiments [85]. Furthermore, the testbed should also be able to support the execution of complex, large-scale, and disruptive experiments seamlessly [86]. The most important features and requirements of CP-SG testbed in the power domain are listed as follows:

1.

Fidelity: It is the degree of correlation between simulation results and real-world observation [87,88,89]. It simply connotes the ability of the testbed to accurately reproduce real power systems in terms of tools (hardware and software technologies), functionalities and operations [70]. Testbeds that involve physical simulation is usually considered to have the highest fidelity, while software-based testbeds have much lower fidelity [90].

2.

Repeatability: The results from CP-SG testbed must be repeatable and capable of producing consistent results when performed independently [22]. It refers to the ability of the testbed to produce similar outcomes when an identical design or setup is replicated. However, the important point that has to be stressed is what are the differences in the results that could occur during actual testing. To achieve this feature, a researcher must be able to set up the experiment platform in the initial state and trigger the necessary events in the right order and at the appropriate time frame [86,87,91].

3.

Flexibility: It is the ability of the CP-SG testbed to be easily redefined for alternative test cases or scenarios [92]. For instance, a testbed is said to be flexible if it was initially set up for vulnerability analysis, but can easily be restructured for security impact analysis [70].

4.

Safe Execution: One of the major reasons for the deployment of a testbed is to be able to run different tests and scenarios in a safe manner. Hence, the testbed must be able to support disruptive experiments with physical processes safely [84].

5.

Scalability: This refers to the ability of the testbed to increase the size of the setup [71,87]. This feature can be demonstrated by the ability to add components to the existing testbed setup without the need for extensive redesign or re-configuration [70]. This feature is especially important for testbeds used to validate new processes or algorithms.

6.

Cost-Effectiveness: The cost implication required to achieve the goals and objectives of a testbed must be within the financial budget affordable for research purposes [89,93]. Since the initial aim of deploying testbed is to reduce cost (in comparison to the cost of using the actual system) and still achieve the same design objectives and scenario as the real one, it is important that the overall cost of putting the testbed together is reduced to the barest minimum. One way of achieving this aim is through the design of a portable testbed system that is able to simulate several services and scenarios [94].

7.

Diversity: This is the ability of a testbed to incorporate a wide range of components and devices without affecting its scalability [70]. Such a testbed system is said to be effective since it can mirror a variety of CP-SG setups [87,95].

3.3. Classification of CP-SG Testbeds Based on Methodologies

While the legacy testbeds were predominantly software-based, most of the contemporary testbed deployment are integrating more real physical components to the implementation of testbed design. The main advantage of including industrial-grade physical devices in the design of testbed is high fidelity. The trade-off, however, is the cost of implementing a physical simulation testbed. To take full advantage of the benefit from both software-based and physical-based testbeds, most institutions and research organizations are now deploying hybrid testbeds that have improved fidelity and cost much less than physical-based testbeds. As shown in

Table 1. Classification of Testbed Based on Methodologies.

Classifications	Cost	Fidelity	Flexibility	Time	Methodology
Software-Based	Low	Low	High	Low	Modeling
Hybrid-Based	Medium	Medium	Medium	Medium	Modeling and Replication
Hardware-Based	High	High	Low	High	Replication

, hybrid testbeds save cost and have a higher fidelity compared to both software-based and Hardware-based testbeds. The remainder of this subsection will provide details regarding the advantages, disadvantages, and implementation methods of each testbed classification.

1.

Software-based Testbeds: This type of testbeds is considered to be economical because instead of using physical devices and information systems, modeling mechanisms are used for their deployment and implementations. Several software-based testbeds use modeling tools such as Matlab, Modelica, Ptolemy, and PowerWorld to model various ICS processes, while applications such as OPNET, OMNet++, and more recently NS3 are used to simulate wide area network models in the testbeds [96]. The major disadvantage of using virtual devices deployed in the tools mentioned above is low fidelity. Another disadvantage is their limitation in simulating certain cybersecurity scenarios due to the unavailability of software models of some devices.

2.

Physical-based Testbeds: The physical and network layers of physical-based testbeds are deployed using real hardware and industrial-grade software. This type of testbeds typically has very high fidelity as they closely mimic the real smart power grid. An example of such a testbed is the National SCADA Testbed (NSTB) built by Idaho National Laboratory. This testbed is made up of 61-mile 138 kV transmission lines, seven substations, and more than 3000 monitoring sites. NSTB is the first actual grid testing environment in the world with full replication of real hardware and software [97]. The major disadvantage of this type of testbed is the cost, as it requires a very high cost of implementation. Another limitation introduced by the implementation of a physical-based testbed is lack of enough flexibility and reconfiguration capability.

3.

Hybrid Testbeds: The idea behind the hybrid testbed is to take full advantage of the replication ability of physical-based testbeds and the flexibility and easy reconfiguration ability of software-based testbeds. In a nutshell, it integrates the methodology of model and replication. This is an effective method of deploying testbeds because it is cost-effective and provides improved replication of the real power grid components and processes. A typical example of hybrid-based testbed is the one deployed at the Washington State University [23]. Industrial grade physical components such as protective relays, RTUs, Phasor Measurement Units (PMUs), and several network switches are integrated with simulators such as RTDS and Network Simulator (NS3) [98].

3.4. Overview of Existing CP-SG Testbed Applications

The review of previous developmental efforts conducted by researchers over the years has demonstrated several research applications supported by CP-SG testbeds.

Table 2. Taxonomy of existing cyber-physical smart grid testbeds.

provides a comprehensive list of existing CP-SG testbeds from various research institutes. Moreover, it highlights the applications, hardware and software components as well as the communication protocols adopted in these testbeds. The most essential applications of the existing testbeds include the following:

1.

Control Validation: Testbeds are used to validate the correctness of the control logic in complex CP-SGs. Since one of the primary objectives of deployment of ICS in the power grid is the remote control of the grid processes and devices, it is important to have an environment that supports the testing and validation of the control logic implemented in such setup. For instance, the Florida International University (FIU) deployed a testbed mainly with the objective of conducting research studies that relate to various control logic implemented in the smart grid [99]. A Chinese-based company (NARI Technology) also developed a flexible hardware-in-the-loop CP-SG testbed which provides an environment for studies in the performance of the stability control system of the smart grid [82]. The testbed was designed to assess the impact of communication error on the stability control equipment of the power system.

2.

Vulnerability and Impact Analysis: The cyber-physical system uses multiple hardware, software, communication protocols, and media to achieve its objectives. Many of these technologies are deployed in environments that are not readily available to the general public. Moreover, they are very expensive to set up, hence creating a bottleneck in conducting vulnerability and impact analysis on the system. The deployment of a testbed creates an avenue for researchers to evaluate the physical impact of different types of cyberattacks on the power systems [100]. Vulnerability analysis activities such as vulnerability scanning, cryptography analysis, and software testing are also conducted on the CP-SG testbeds [71]. For instance, the testbed deployed in the University of Arkansas was designed mainly for research tasks on the detection of false data injection attacks and vulnerability analysis of the Distributed Energy Resources (DER) cybersecurity schemes [101]. The Institute for the Protection and Security of the Citizen in Italy also designed a CP-SG testbed for cyber vulnerability studies of the SCADA system in the power system [102]. The National SCADA testbed at the Idaho National Laboratory (INL) is also being used extensively for several research studies on vulnerability and impact analysis [103]. The Electric Power and Intelligent Control (EPIC) testbed at Missouri University of Science and Technology has been developed to uncover potential integrity vulnerabilities in electrical synchronous generators [104]. Also, this testbed is used to assess the impact of cyber threats against physical infrastructures and provides a repeatable assessment of the effect of cyber attacks [18,86]. In the same vein, an CP-SG testbed that creates an environment for testing the impact of various time delays cyber-attacks on SCADA systems was developed at the University of Binghamton [105]. The testbed has been used for various research projects to study the physical impact of such attacks on the ICS system [106,107].

3.

Performance Studies: Reliability is critical to the operations of any CP-SG due to its reliance on communication. Hence, there is a need for testbeds to have the capability of testing the performance of the CP-SG in the electric power domain under different operating conditions. CP-SG Health testbed [108] was designed to observe any mal-operation in the cyber, physical, and overall health of the smart power grid. The health of CP-SG is tested and calculated during Denial-of-Service (DoS) attacks. Real-Time Automation Controller (RTAC) was deployed in the testbed for storage of the control decisions required if the health of the power system is depleting, and the power system is simulated using RTDS. Ghada et al. [109,110] designed a cost-effective software-based testbed to assess the performance of IEC 61,650 under various cyber-attack scenarios on the sensors, communication network and embedded systems of the testbed. A Microgrid testbed platform that is made up of hardware-in-the-loop and network-simulator-in-the-loop was designed to study and test the effect of different communication channel delays in the performance of the smart power grid [111].

4.

Security Validation: A lot of CP-SG testbeds focus on different aspects of power system security concerns such as cybersecurity, communication security, and physical security. Cybersecurity compliance requirements are becoming increasingly common as a way of ensuring the security and protection of critical infrastructures [71]. Due to the smart grid’s high availability requirement and the heavy usage of proprietary systems, there is a constraint in the applicability of common vulnerability scanning methods [112]. Hence, there is a need for testbed environments that implement industrial software, communication protocols, and configurations which would help validate the effectiveness of traditional security assessment techniques while also providing a medium for testing new security algorithms. An example of such a testbed was developed at the University of Arizona [113]. The testbed is extensively used to validate the effectiveness and performance of various protection techniques used in the smart power grid [77]. The University of Illinois also developed an CP-SG testbed to support decision-making in the power grid cyber-infrastructure for cyber-security purposes [78]. Apart from this primary objective, the testbed was also intended to be integrated into other testbeds for the exploration of the performance and security of SCADA protocols and equipment in an inter-connected testbeds setup [84]. Some testbeds are also designed for protection device validation and tuning. An example of such a testbed is developed at the University of North Carolina, where it is used mainly for the validation of synchrophasor relays [114]. Some other testbeds are designed mainly for security research that involves intrusion detection in synchronous generators. One of such testbeds was developed at the Mississippi State University where it has been used for research studies that involve the deployment of Intrusion Detection System (IDS) for synchronous generator security monitoring [75]. Another testbed with similar functionality was developed at the Center for Development of Advanced Computing (C-DAC), where it is used to detect potential intrusions at the RTU of the synchronous generator [115]. The testbed developed at the University of Idaho (ISAAC) is another security-oriented testbed that emulates a realistic power utility and is used to test various integrated cybersecurity solutions [116,117,118]. This testbed was used for experimental evaluations, whereby the data of normal and attacked communications were collected for data-driven stochastic anomaly detection on smart grid communications [29].

5.

Multi-functional: To make the most of the various research possibilities presented by the testbed environment, some existing testbeds are set up to be multi-functional in nature. This type of testbeds is usually very robust, flexible, and easily scalable. They provide a platform on which a variety of tests and validations can be conducted on the same testbed unit. An example of such a testbed is the one developed at Iowa State University. This testbed was designed for multipurpose use, although its primary objective is to create an environment for testing and validating smart grid algorithms in real time [71,119,120,121]. Some of the capabilities of this testbed include cyber-attack detection in the smart grid, measurement of the impact of the attacks, and intensive vulnerability analysis [119]. Other research studies conducted on this testbed concentrate on mitigation research, data and model development, security validation, interoperability, and cyber forensics [71]. Extensive studies on the impact of cyber-attack on the Automation Generation Control (AGC) have also been conducted on this testbed [121]. Washington State University has also designed and deployed a state-of-the-art testbed which is intended to be flexible enough to accommodate diverse research studies [122,123,124]. The authors in [3,125] gave a detailed description of how this testbed was designed, assembled, and configured. They also validated the performance of the setup by conducting a cyber-attack impact assessment on the testbed to study the impact of cyber-attack on the smart power grid. The testbed has also been used to test the accuracy of synchrophasor devices such as PMUs and Phasor Data Concentrators (PDCs) [126]. In addition to the research efforts described above, the authors in [127,128] provided an in-depth description of various cybersecurity vulnerability and impact analysis conducted on the testbed. This robust testbed has also been used for the validation of distributed application as described in [129] where the authors validated a Distributed Remedial Action Schemes (DRAS) on the testbed. As it is very essential to deploy a standardized security assessment metric on the cyber-physical system, the authors in [130] were able to propose and validate Multi-Criteria Decision-Making (MCDM) technique on this testbed. The Sandia National Laboratories also developed a multi-functional testbed which has been used for vulnerability analysis [79], validation of new topologies, hardware, controls, communication, and security of microgrid [131,132]. In another research study, the testbed was used to compare the performance of a virtual testbed to an actual physical system [133]. The North China Electric Power University (NCEPU) designed a multi-functional CP-SG testbed for various research application that ranges from vulnerability analysis, cybersecurity, to integration of different renewable energy resources [74,134]. This testbed is very versatile because the physical layer of the testbed was realized with a source-grid co-simulation system (in which the energy sources and the power grid network were simulated separately). Various security-oriented research tasks have been carried out on this testbed with the main targets of the attack being the AGC modules and the measurement collected at the tie lines of the power system. Pacific Northwest National Laboratory (PNNL) designed and deployed a robust multi-functional CP-SG testbed called PRIME [135]. This realistic testbed environment is made up of industry-grade software coupled with hardware-in-the-loop to perform various verification and validation studies. It is also used for several Wide-Area Monitoring, Protection, and Control (WAMPAC) prototyping, impact analysis of diverse cyber-attack scenarios on the operation of the grid, and operator training.

6.

Education: The application of testbed in the education system plays a vital role. It allows students to work and gain experience with Industrial security systems. It is dangerous to direct research and training on valuable plants, as slight distraction can rapidly prompt harmful instances. Because of this justification, testbeds are very essential in education as well as research. Purchasing genuine industrial hardware for testbeds is very costly especially for the education field. Therefore, researchers have come across few testbeds with low cost for purpose of education. For instance, the LICSTER (A Low-Cost ICS Security Testbed for education and research) [136]. It is an open-source testbed that helps students and researchers to gain knowledge and experience related to industrial security. It costs 500 Euro to build the testbed. The educational testbed developed for a course on industrial communication networks at the Engineering Faculty, University of Catania [137]. A WoT Testbed for Research and Course Projects, building a WoT testbed is implemented in two main axes; the first axis is to configure and connect hardware components that simulate the set of environmental events (IoT layer), and the second axis is to build the application layer in terms of mini-projects on top of the IoT layer [138]. As illustrated in the paper [139] “Development of Smart Grid Testbed with Low-Cost Hardware and Software for Cybersecurity Research and Education”, the testbed is a useful resource for cybersecurity research and education on different aspects of SCADA systems such as protocol implementation, and PLC programming [139].

7.

Forensic: SCADA systems run 24/7 to control and monitor industrial and infrastructure processes. In case of potential security incidents, several challenges exist for conducting an effective forensic investigation [140]. In the light of the significance of SCADA for the resilience of critical infrastructures and the related targeted incidents against them (e.g. the development of Stuxnet), cybersecurity and digital forensics emerge as priority areas [141]. The paper [142] describes an architecture that supports the forensic analysis of SCADA systems and networks. The design is actualized in a prototype networked condition using the Modbus TCP protocol. The study of forensic attacks plays an important role in the SCADA system to reach the accountability requirement of the data security objectives. Probable vulnerabilities are detected by forensic attacks before they get manipulated by malicious entities. As stated by Chris et al. in [143], the first step in preparing for any forensic attack is to identify and exploit weaknesses. In [144], a four-stage approach is made by the authors to perform forensic attacks on SCADA systems. The main technique for cyber defense when a security breach occurred is digital forensics. It is a method of acquisition, examination, study, and recording of the evidence. For instance, the authors in paper [145] have implemented and improvised a forensic testbed by implementing a sandboxing technique in the context of real time-hardware-in-the-loop setup. The paper [146] presents a SCADA testbed recently built at the University of New Orleans for purpose of cybersecurity and forensic research, and education on diverse aspects of SCADA systems such as PLC programming, protocol analysis, and demonstration of cyberattacks.

8.

Safety Standards Development: Because of the susceptibility and security concerns of a conventional SCADA/DCS, the power system managers should consider building and implementing emergency plans to define the appropriate steps to be followed by their workers or suppliers in a situation where the facility is deliberated in a cyber-attack. The typical cyber-attack emergency plan ought to incorporate several elements including the logical network diagram, the network connection loss impact analysis, the zones of vulnerabilities identified on the logical network diagram, the access-control lists that identify which personnel should be allowed access to the industrial network, the step-by-step standard operating procedures of what activities to perform immediately following an incident, and finally, the access logs detailing time-stamped activities on the network IT. For instance, the paper [147] implemented “An integrated testbed for locally monitoring SCADA systems in smart grids”. Using the developed testbed, a recently proposed local monitoring approach was investigated [147]. The paper [148] has developed a “Testbed for Secure and Robust SCADA Systems” for checking vulnerabilities and validating security solutions.

3.5. Challenges Facing the Existing Approaches of Testbeds

One of the major factors limiting the effectiveness of the existing testbeds is fidelity. As in-depth knowledge regarding a plethora of parameters is simulated for CP-SG testbeds, managing the fidelity of such setup becomes a major challenge [149]. High fidelity of the cyber layer of the testbed can be achieved by the deployment of real components, including real software, network devices, and standard protocols [86]. As discussed in Section 3.2, researchers must be able to set up the experiment platform in the initial state and trigger the necessary events in the right order and at the right time to obtain consistent results. For this reason, it is important that the experimentation workflow of different cases simulated on the testbed must be automated in order to achieve repeatability and measurement accuracy. On the other hand, testbed with a fully simulated cyber layer such as those described in [109,150], provide precise repeatability and measurement accuracy, although at the expense of decreasing the setup’s fidelity [151]. The hybrid setup of real components with simulators in the cyber layer also poses its own limitation as the strong fidelity provided by the real components is weakened by the simulated cyber layer [86,152].

The major bottleneck of testbeds with full real component deployment both for the physical and cyber layer is its lack of flexibility and its very high-cost. Security experiments conducted on such setups can be highly disruptive and the use of advanced malware tools can have an unpredictable impact on the physical setup [153,154]. Re-configuring these testbeds to explore diverse research studies is often complicated and could even be financially infeasible. Hence, the fidelity provided by this setup is offset by its poor flexibility, high-cost implication, and high safety risk [86].

Another factor limiting the functionalities of the existing testbeds is insufficient diversity and heterogeneity of equipment. Most of these testbeds are assembled with devices from a single vendor or supplier. This factor makes it difficult to meet the requirement of vulnerability assessment of certain protocols and equipment [97].

3.6. Current Trends in CP-SG Testbed

In this section, we will highlight some of the important current trends implemented in CP-SG Testbed deployment.

• Hybrid Testbeds: As discussed in Section 3.5, fidelity is an essential factor that must be considered when designing and deploying a testbed. The result and dynamics of the components of a testbed are expected to be as close as possible to that of an actual power system. The fidelity of the legacy CP-SG testbeds was often compromised because the majority of them are predominantly software-based. While the physical-based testbeds have high fidelity, it is quite difficult to reconfigure such testbeds for different research endeavors and their deployment can be very expensive as well. Due to these factors, most of the contemporary testbeds are designed to incorporate both hardware and industry-grade software. This paradigm shift has tremendously improved the reliability and fidelity of the CP-SG testbeds in the power domain.
• Inter-connectivity of Testbeds: Another advancement in the implementation of CP-SG testbed is the inter-connectivity of testbeds from different universities and research institutes. This development affords researchers a medium and platform to perform intensive research tasks using robust infrastructure which gives room for an open and convenient collaboration across vast distances. A good example of such a setup is the Idaho Regional Optical Network (IRON) in Idaho state. IRON is a regional optical network used by researchers and educators to transfer big data between research universities, other educational entities, the national laboratory system, and the health sector. The connected institutions include Boise State University, Brigham Young University-Idaho, Idaho Hospital Association, Idaho National Laboratory, Center for Advanced Energy Studies (CAES), Idaho State University, State of Idaho – Department of Administration, University of Idaho, and Washington State University. The platform has allowed for productive collaborative research projects among the listed institutions.
• Software Defined Networking (SDN): Incorporation of SDN into the CP-SG testbed network is another paradigm being considered by several research institutions. SDN aids the creation of an open networking architecture which makes it possible to get a holistic perspective of the entire network and make global changes in the network without having to access individual device hardware [23]. The overall security and resilience of the CP-SG testbed network can be improved by the deployment of SDN technology. ISAAC testbed at the University of Idaho, for instance, has deployed several SDN switches in their testbed, with the aim of comparing the security and performance of a network with and without the presence of SDN technology. This would go a long way in helping to verify the feasibility of SDN technology in the existing smart grid infrastructure.
• Distributed Control: The vast inter-connectivity and complexity of the contemporary smart grid makes it almost impossible to have a single centralized control system for the whole grid. Due to this factor, distributed control is one of the current trends in smart grid technology [23]. The dynamic nature of the smart grid even makes it necessary to deploy distributed control in the grid. Since the CP-SG testbed is meant to represent a faithful replication of the actual smart grid, it is important for institutions and research organizations to start incorporating distributed control into their CP-SG testbeds so that diverse test scenarios can be conducted to evaluate the performance of distributed control on the smart grid.

4. Conclusions

In this paper, we illustrated the challenges faced by the conventional power grid which necessitated the development of the smart grid. We elaborated on the factors that make smart grid testbeds essential in analyzing power systems. We provided three major components that make up the CP-SG testbed. Then we presented a comprehensive overview of the existing CP-SG testbeds. We thoroughly investigated the infrastructures of these CP-SG testbeds from the perspectives of their architecture and corresponding functional analyses, the main vulnerabilities and threats, testbed requirements, constraints, and their applications. We also discussed current trends and future research directions to provide valuable insights for CP-SG researchers to create their own CP-SG testing environments.

In conclusion, the main points that can be derived from this comprehensive survey of CP-SG testbeds are:

• The major drawbacks of actual testing on the conventional power grid include time-consuming demand response, high computational costs, and expensive assets. As a result, testbeds are required to simulate an actual CP-SG to verify various concepts and extensive research purposes.
• Communication infrastructure is an essential component of a smart grid. It is used in generation, transmission, and distribution domains of the power grid. This vastly improves the operational efficiency of the power system. However, the introduction of communication into the power grid exposes the system to various types of cyber-attacks. CP-SG testbeds are used to perform vulnerability and impact analysis of cyber-attacks on the smart grid.
• Most of the existing testbeds are simulation-based due to the complexity involved in modeling the actual smart grid system. Simulation-based testbeds are known to be economical because software is deployed instead of using physical devices. However, the major disadvantage of this type of testbed is the limited capability to represent the real-time features of actual systems.
• Physical-based testbeds have a relatively high capability to represent the real-time features in actual systems, but the cost implication increases drastically as the scale of the system increases. Another disadvantage of this implementation is the lack of flexibility and reconfiguration capability.
• Control Validation, Security Validation, Performance Studies, Vulnerability, and Impact Analysis are some of the most common applications of existing CP-SG testbeds.
• Inter-connectivity of CP-SG testbeds from different research institutes provides researchers with a platform for extensive and expansive research tasks using robust infrastructure which promotes highly productive collaborations across vast distances.
• Some research institutions are already integrating SDN into their CP-SG testbed setup. This will aid and enhance the feasibility study of the deployment of SDN technology in existing smart grid infrastructure.
• Distributed control is an aspect of the contemporary smart grid that must be considered for integration into the existing CP-SG testbeds.