Review of Design Elements within Power Infrastructure Cyber–Physical Test Beds as Threat Analysis Environments

Abstract

Cyber–physical systems (CPSs) are an integral part of modern society; thus, enhancing these systems’ reliability and resilience is paramount. Cyber–physical testbeds (CPTs) are a safe way to test and explore the interplay between the cyber and physical domains and to cost-effectively enhance the reliability and resilience of CPSs. Here a review of CPT elements, broken down into physical components (simulators, emulators, and physical hardware), soft components (communication protocols, network timing protocols), and user interfaces (visualization-dashboard design considerations) is presented. Various methods used to validate CPS performance are reviewed and evaluated for potential applications in CPT performance validation. Last, initial simulated results for a CPT design, based on the IEEE 33 bus system, are presented, along with a brief discussion on how model-based testing and fault–injection-based testing (using scaling and ramp-type attacks) may be used to help validate CPT performance.

1. Introduction

Electricity, used as a medium for either data or power transfer, plays an essential roll in maintaining and advancing the quality of life for modern society. As its penetration in day-to-day life becomes ubiquitous, our dependency on electricity’s presence and vulnerability in its absence increases. Therefore, ensuring the reliability and resilience of the electric power grid is essential. Natural disasters are the most common threat to the modern-day electric grid, accounting for 62% and 90% of major power outages in 2016 and 2017, respectively, according to the Department of Energy, Office of Electricity (DOE-OE) Electric Disturbance Events OE-417 forms [1]. Cyberattacks also have the potential to cause widespread blackouts [2] and damage to power transformers (via remote control of breakers) [3] or generators [4,5]. Additionally, cyberattacks may be deployed en masse (with frequencies as high as 10,000 attacks per minute [6]) alongside a natural disaster. To address these challenges and ensure resilient and reliable power-grid operation, the interplay between the digital and physical realm must be understood and properly guarded.

The need for reliability negates the possibility of direct experiment on critical infrastructure, and the cost to produce a direct replica is often too high. To overcome this challenge, cyber–physical test beds (CPTs), with a primary aim to explore how the physical and digital world impact each other, are needed. Varying degrees of hardware-in-the-loop (HIL) connected with simulations or emulations are most-often employed as a cost-effective means to probe the cyber–physical nature of critical systems [7]. These test beds must strike the appropriate balance among what is simulated, emulated, and physically manifested as HIL while maintaining the flexibility to cost-effectively study the resilience posture of many types of system typologies and configurations.

Many highly varied types of CPTs have been investigated to aid in the development of manufacturing [8,9] unmanned aerial vehicles [10], cellular [11], electric vehicles [12], maritime systems [13], control systems [14], and more. The unifying connection between these systems is the electric power grid. Without power, cyber–physical systems will not function. Thus, the aim of this work is to focus primarily on CPTs for power systems. From this point forward, all references to CPTs will be considered within a power-system context.

This paper attempts to review various design elements that must be considered when constructing a CPT, as shown in Figure 1. Section 2 reviews the physical components that comprise CPTs: hardware, emulators, and simulators. Trade-offs between different physical components and examples of their implementations are discussed. Section 3 reviews soft components within CPT (communication and timing protocols and wide-area monitoring) within the context of test-bed scope and application to facilitate appropriate protocol selection. Section 4 presents a custom visualization and alert system, as well as various design considerations that went into its construction for a power-distribution CPT. Section 5 reviews various testing methodologies for CPSs and attempts to extrapolate these concepts for application to CPT performance validation. To the authors’ best knowledge, no discussion has appeared in the literature of universal test methods or the benchmarks researchers may use to compare one CPT with another. This section attempts to formulate these testing methods. Section 6 discusses an initial effort to design a CPT for power-distribution systems and provides an example of scaling and ramp attacks against a photovoltaic (PV) inverter, as well as how these results may be used in the model-based testing (MBT) and fault injection-based testing (FBT) described in Section 5. Section 7 contains concluding remarks.

2. Hardware Components for Constructing Cyber–Physical Test Beds

2.1. Advantages and Disadvantages of Physical Hardware, Emulators And Simulators

CPTs are composed of different combinations of hardware, emulators, and simulators.

Table 1. Generalized advantages and disadvantages of simulation, emulation, and hardware components for a cyber–physical system.

	Simulation	Emulation	Hardware
Cost	Low	Medium	High
Fidelity	Low	Medium	High
Scalability	High	Medium-high	Low
Interoperability	Low	Medium-high	High
Computational expense	Low	High	None

qualitatively lists generalized advantages and disadvantages of each approach. Ideally, a CPT may organize all three elements to minimize the disadvantages and maximize the advantages each brings to bear.

A purely physical–hardware-based CPT would provide the most ideal representation of real systems. One example of a purely hardware-based CPT is Idaho National Laboratory’s (INL’s) Critical Infrastructure Test Range Complex (CITRC) [15]. CITRIC boasts of containing its own fully functioning substation, which contains both distribution- and transmission-class voltages and is ideally located for testing new power-grid solutions under a wide range of weather conditions. The testing and maintenance costs of this system, however, are very high compared to a real-time simulation with HIL setup. Hydro Quebec also has a purely hardware-based distribution CPT [16]. This test bed operates at 25 kV and has solar, wind, and storage assets attached; it is fed by its own independent transformer from a distribution substation. While these purely hardware-based CPT systems are ideal for testing and validation of system components, they require large amounts of real estate and are not practical for most research institutions. Although simulation and emulation have less fidelity, they can help reduce cost and size constraints on a CPT.

To the authors’ best knowledge, a purely simulation- or emulation-based CPT was not found. A common strategy observed was to simulate the power-grid portion while emulating or using real hardware for the cybernetic component or specific distribution-energy resources (DERs) [7,17,18]. Real-time simulation platforms—e.g., RTDS, Opal-RT, dSPACE, and Typhone HIL—have power systems models readily available to easily scale the size of the power grid modeled in the CPT. Thus, real-time simulation provides a cost-effective means to make the CPT more flexible and scalable.

Another advantage of simulation and emulation is the ability to connect test beds separated by large geographic distances [19]. Although data latency issues present some limitations and must be addressed when considering a real-time simulation or emulation remote connection, the strategic expansion of test-bed assets may well be worth the tradeoff. One strategy is to separate a power-system model from the control-system interface, as outlined in [20], where one CPT specializes in power-system modeling, and the other in data visualization. Monti et al. reported on an intercontinental CPT connection over real-time simulation, using high-voltage direct-current (HVDC) partitioning in the real-time simulation and VILLAS framework [19]. The HVDC links require less information exchange compared to high-voltage alternating-current (HVAC) links to maintain simulation-timing integrity. The VILLAS framework also reduces the communication overhead by reverting to a peer-to-peer style of communication, rather than using a centralized communication authority.

2.2. Hardware-, Emulator-, And Simulator-Based Representaitons of Physical, Cybernetic, and Cyber–Physical Elements within CPTs

Figure 2 provides examples of simulated, emulated, and physical-hardware representations for the main components within a CPS: physical, cybernetic, and cyber–physical interfaces. The physical system represents hardware responsible for generating, conditioning (e.g., using capacitor banks), transporting, sensing (e.g., by means of current transformers), and interrupting power to the loads. The cybernetic system comprises digital control devices that are able to manipulate physical components to facilitate efficient operation or prevent damage to the system. The cyber–physical interface is generally where the conversion of digital information to physical changes on the system occurs or where physical measurements (typically analog) are converted to digital representations [21,22]. Each of the three components within a CPS are synchronized by time. A CPT attempts to represent these three areas via simulation, emulation, physical hardware, or by some combination thereof.

Real-time simulations are typically carried out on special platforms which produce calculations within fixed time steps. Due to their low cost in comparison to a purely hardware system, simulations are typically a good way to start building a CPT. Until actual hardware is connected, the simulation does not need to be in real-time, which allows for faster debugging and development. While OPAL-RT and RTDS are very popular commercial solutions for real-time power-grid simulaltions, others have attempted to adopt Raspberry Pi as a lower-cost alternative [23]. GNS3 and OPNET are network simulators and may be used to interface with physical-system simulators, as discussed in [24]. The main drawback of network simulators like GNS3 and OPNET is a lack of real-time functionality; thus, the authors in [24] opted to use network emulators running on a series of Raspberry Pis, along with control algorithms written in Python.

Emulating an entire physical power grid is challenging because emulators typically attempt to mimic single components or bulk-grid inertia [25]. Collecting enough emulators to comprise a sizable grid would be expensive. In [26], a fully reconfigurable emulated test bed was reported to allow for greater time-scale flexibility, compared to real-time simulations, and a wider range of voltage-class systems compared to actual hardware-based test beds. A LabView control-room interface was used to monitor and operate the power grid; however, no mention of cybernetworks was provided. Current-transformer (CT) and voltage-transformer (VT) measurements were simply fed directly from the emulation into NI-CompactRIO running the control-room interface. In [24], an OPAL-RT system was used to simulate the power grid while real-time Raspberry Pis, running NetEm, a Linux network emulator, were used to emulate network-control traffic. DeterLab and ISEAGE are other network-emulation tools that may be used to study network security for smart grids [27,28]. Control-room software, such as RTDMS, GE iFIX SCADA, and Modbus, could be run in an emulated environment; however, there is no disadvantage to directly running control-room software on physical machines [29,30,31].

Cyber–physical systems may also include servers for data storage, in addition to running supervisory control and data acquisition (SCADA) software [29,30,31]. Physical hardware that interfaces with measurement devices such as CTs, VTs, and phasor-measurement units (PMUs), is location of the cyber–physical interface. Including these devices can be more cost effective than attempting to emulate them and save on computation expense. Likewise, it is common to have microgrid components such as solar panels, batteries, and charge controllers because these are more affordable, and simulation or emulation resources may then be reserved for more-challenging tasks. Physical transmission or distribution lines, for example, are typically not practical for most institutions; thus, they require real-time simulation or emulation.

3. Soft Components for Cyber–Physical Testbeds

3.1. Common Communication Protocols for CPSs and CPTs

Communication protocols are a critical part of CPT and are used to link the various components: real-time simulations, real-time emulators, or hardware. The selection of communication protocols to be added is also an important aspect of CPT design to ensure it adequately reflects the operation of real power grids, provides a justifiable means to answer research questions, and fits within the test bed scope (e.g., distribution, transmission, microgrid, etc.). In this section, a brief description of popular communication protocols used in the power industry is presented.

Table 2. Description of various popular protocols used within cyber–physical test beds.

Protocols	Locations	Advantages	Vulnerabilities
DNP3 (IEEE 1815)	Control center (master unit) and outstation devices [32]	High reliability and flexibility	Unsolicited message attack, Data set injection, Passive network reconnaissance [32]
Modbus	Control center (master unit) and outstation devices [33], substation networks	Open access standard, easy implementation	Malware, spoofing, Man-in-the-Middle, DoS, Replay [33,34]
OPC	Control center and outstation devices	Operating system agnostic, open access standard	malware [35], Relay attacks
IEC 60870	Control center, substation networks	Follows the OSI model	Spoofing, sniffing, data modification, relay, non-repudiation [36]
IEC 61850	Substation networks	Highly flexible, focus on adaptable substation automation, substation hierarchy easily viewed	Unauthorized access, DoS, spoofing, Man-in-the-Middle, data interception [37]
IEEE C37.118	WAN, substation networks	Supports real-time data transfer	DoS, reconnaissance, authentication, man-in-the-middle, replay [38]

summarizes the protocols described in this section.

DNP3: Distributed Network Protocol 3 (DNP3) was originally designed for SCADA applications and made available to the public in 1993. DNP3 focused on sending multiple smaller-sized packets in a deterministic sequence to enhance communication reliability and error detection. DNP3 has been widely adopted by North American power utilities and has gained popularity within the water, oil, and gas industries [39,40]. For use over local area networks (LANs), DNP3 must be wrapped inside an internet protocol (IP) such as TCP/IP. DNP3 has adapted to support a wide range of communication modes, such as traditional client/server, peer-to-peer, multimaster, and hierarchical. The adaptivity and flexibility of DNP3 to industry demands, coupled with its high degree of reliability, has made it the dominant protocol of choice for power-distribution networks in North America today [39,40].

Modbus: Modbus was first developed in 1979 as a communication protocol between programmable logic controllers (PLCs). The standard became very popular due to its facile implementation and open access to the standard. Modbus is supported by a variety of different transmission protocols for asynchronous serial transmission, TCP/IP, and Modbus plus. This allows the protocol to be used across many different device types—human machine interfaces (HMIs), PLCs, relays, network gateways, and other input/output (I/O) devices—over a large area network [39]. With the adoption of TCP/IP into the standard, communication to many power system devices and SCADA applications became possible. The data packets used over Modbus were variable in size, depending on how large the data field was. This caused issues with data integrity because portions of very large packets may have become corrupt or disrupted during transmission. The biggest drawback of the Modbus protocol was a lack of security in data or command authentication, which made systems using Modbus vulnerable to, e.g., man-in-the-middle or spoofing cyberattacks.

OPC: The Open Platform Communications (OPC) was first introduced as an open standard in 1996 for automation control devices to interface with HMIs. The standard was updated in 2008 to a unified architecture (UA) version, which included many of the legacy features from previous versions, including accessing process data, transmitting events or alarms, transferring historical data, and leveraging eXtensible Markup Language (XML) to encode data access. OPC-UA also aimed to be operating-system agnostic and offered security features such as encryption and user authentication. Although popular within industrial processes, OPC-UA was not widely adopted within the power-system community [35]. Microgrids, on the other hand, have made OPC-UA a popular choice for communication of their automation controls [35,41].

IEC 60870: The International Electrical Commission (IEC) 60870 standard was first introduced in 1990 for remote control of power-system operations. The standard adheres to the open-systems interconnection (OSI) model and focuses on the physical, data link, and application layers. The standard originally suffered from a broad execution interpretability, which lead to a large variety of incompatible manifestations of the 60870 standard [40]. To solve this issue, the standard was updated in 2001 to better define how different devices should communicate. The updated standard also required devices on a network to have present instructions regarding packet structures to avoid sending this information within the packets themselves, which improved communication efficiency. Coupled with an update from 2000, the standard also supported TCP/IP communication between substations and control centers. Despite these updates, the standard still lacked clarity for specific use cases, again resulting in diverse implementations, and the TCP/IP implementation was operationally restrictive, limiting information types and configuration parameters.

IEC 61850: First published in 2003, IEC 61850 sought to introduce a standard focused on automation and flexibility for intelligent substations. The United States National Institute of Standards and Technology (NIST) identified this as one of five “foundational” standards for smart-grid interoperability and cybersecurity [42]. The standard introduces its own substation configuration language based of XML, a high-level programming language compatible with a wide variety of communication protocols, to facilitate system-wide component configuration. Substation communication is binned into one of three different categories: process (e.g., I/O devices and sensors), unit (e.g., protection and substation devices), and substation (the control computer or operators control HMI) levels. Within each of these communication levels, a series of protection and control functions are defined for various objects (also referred to as logic nodes (LNs)). Each LN corresponds to various substation device functions and can be grouped to logic devices that represent intelligent electrical devices (IEDs). The protocol also includes provisions for transmitting generic object-oriented substation events (GOOSE). Although previous protocols allowed for custom applications to configure and automate substation settings and operations, IEC 61850 includes specific instructions for how to do this, with definitions for over 100 LNs and more than 2000 data objects or data attributes. Additionally, users can access information hierarchies based on all LNs and objects to gain a sense of how substations are organized logically. The main drawback of IEC 61850 is its higher complexity compared to legacy protocols. IEC 61850 is described as having a steep learning curve and requiring significant effort to implement [39]. Because of these difficulties and the lack of manpower to support a significant upgrade, IEC 61850 has not been widely adopted in North America [43,44].

IEEE C37.118: Establised in 2005, this protocol was designed for real-time exchange of synchronized phasor-measurement data between power-system equipment [45]. Initial versions included both measurement and real-time data-transfer requirements. It provides an open-access method to facilitate the development and use of synchrophasors, allowing data transmission and accretion within a phasor–measurement system [45]. IEEE Standard C37.118-2005 was eventually split into two standards, one with measurement requirements and the other with the data-transfer requirements. This allowed for the use of IEEE C37.118 with other communication protocols. Further, this protocol was created with sufficient flexibility to account for future developments and enable a smooth transition of synchrophasor systems to new protocols as necessitated [45].

3.2. Timing and Data Synchronization

Modern smart grids commonly consist of interconnected hardware and software components in distributed substations, communicating with each other to achieve a common goal [46]. In order to function and make decisions properly, the correct timing of data measured throughout geographically distributed sensors in the system must be considered [47]. Therefore, time synchronization is one of the primary elements in smart grids that enables accurate monitoring and protection and optimal control [47,48]. Thus, timing is also critical for CPTs.

The requirement for time synchronization varies from one microsecond to hundreds of nanoseconds, depending on the device used, customer demands, and application of interest [48]. For example, traveling-wave fault detection requires synchronization on the order of hundreds of nanoseconds to precisely locate a fault [48]. In [49], a traveling-wave fault-detection CPT was designed using an OPAL-RT system with a field-programmable gate array (FPGA) to generate transient signals over fiberoptic cables with a 500 ns time step. This CPT allowed for testing the detection functionality for various fault-locator devices. A synchrophasor or phasor measurement unit (PMU), on the other hand, measures the magnitude and phase angle to determine the health of the electrical grid and only requires 30 observations per second [50]. Adikari et al. built a CPT to explore PMU-control interactions with the power grid by leveraging RTDS and various PMU HIL possibilities [51]. They generated several time-synchronized cyber–physical data sets of various cyberattacks in order to aid in intrusion-detection sensor development.

The time synchronization requirements for power grids are often satisfied using GPS- or protocol-based time synchronization [48]. In GPS-based time synchronization, a standard-reference atomic time signal into substations’ components is used. Protocol-based time synchronization uses network-based time-distribution protocols such as the Network Time Protocol (NTP).

Popular methods currently used for time distribution in smart grids described here are summarized in

Table 3. Description of various timing synchronization schemes used within cyber–physical testbeds.

Protocols	Applications	Advantages	Vulnerabilities
GNSS	Synchrophasor [48]	Time synchronization across large geographic areas	Spoofing [52], DoS [53]
IRIG (IEEE 1344)	Synchrophasors	Contains a clock, quality indicator	DoS, eavesdropping (if not encrypted) [54]
NTP	Substation, microgrid, control center, power electronics outstation devices, SCADA	Universally adopted	Malicious packet delays [55], ARP spoofing [55]
IEEE 1588	Control center, substation networks	High degree of accuracy	Time synchronization attacks [56]

:

• Global Navigation Satellite System (GNSS) is a system of satellites with global coverage, facilitating geospatial positioning and precise time [50]. GNSS is an American company. GLONASS is a similar system owned by the Russian state corporation Roscosmos. Time references provided by these GPS systems have accuracy to less than 100 nanoseconds, sufficient for most power-system applications [50].
• The American Inter Range Instrumentation Group (IRIG) contains several standards, including IRIG Standard 200-98, IRIG-B, and IRIG Standard 200-04. This method uses a continuous stream of binary data to distribute time information. IRIG-B is the most common standard; it facilitates geographically separated locations synchronizing to a single time source [50].
• Network Time Protocol (NTP) is designed to synchronize clocks of multiple computers over a packet network. In order to synchronize clocks over the network, the network delay between clocks must be known. Therefore, the accuracy of NTP depends on network traffic. The accuracy of this method on LANs is around 1 millisecond and is on the order of tens of milliseconds for wide area networks (WANs) [50].
• IEEE 1588 is designed for systems which require highly accurate time synchronization. Rather than using packet network, this approach uses “hardware time-stamping” to distribute time. The accuracy of this method lies under a microsecond [50] and is a popular standard to synchronize clocks on distributed systems.

Most often in CPTs, the timing component is handled by the real-time simulator, with little need for timing network protocols. In [51], for example, network protocol IEEE C37.118 was used to communicate between various PMU devices in studying wide-area measurement systems, but provided no mention of timing protocols used in the study, if there were any. Many PMU devices typically have internal GPS clocks that are able to time-stamp measurements [29]. Additionally, most CPT components are within close proximity to each other, which negates the need to account for data transmission over long distances. However, the SCADA Security Laboratory and Power and Energy Research Laboratory at Mississippi State University comprise two remote sites on campus, one of which contains a PMU and GPS substation control unit [14]. This would enable various studies involving attacks against network timing synchronization to explore potential impacts on various control schemes and physical-system typologies (simulated by RTDS and HITL).

The design goals of the CPT may also impact what communication and timing standards are pared. For example, an automated control scheme using peer-to-peer communication among various IDEs would benefit from IEC 61850, which allows for high-resolution, low-latency transmission of contextualized (e.g., providing the device of origin) data [57]. A more precise timing protocol, such as the IEEE 1588, may be required for those use cases. DNP3 was designed for SCADA communication [58] and can be used for power-grid automation [59]; however, it is not considered to be sufficiently flexible to handle all conceivable scenarios within the smart grid and, in particular, subsecond device controls [60]. However, DNP3 was found to be a much more resilient protocol to packet rendering, data corruption, jitter, and bandwith limitations than IEC 61850 [61]. A CPT that focuses on providing situational awareness and human-in-the-loop studies might more strongly consider DNP3, which supports a wide range of timing protocols. Modbus is most advantageous when dealing with serial communication [62]. Although, Modbus is capable of transmitting at faster rates than DNP3 [63,64] and is considered to be an important protocol for smart grids [62], it is less popular in North America and Europe [62]. Like DNP3, Modbus is used for system monitoring and supports a wide range of timing protocols [65].

3.3. Wide-Area Situational Awareness

Figure 3 represents a conceptual architecture for developing a real-time wide-area situational awareness (WASA) system. The conceptual architecture of WASA consists of three main components: heterogeneous database, performance metrics, and visualization dashboards.

The control center of a DER integrated distribution grid receives multidimensional grid measurements from DER client nodes, system logs from network sensors, firewall alerts from network sensors, and topology logs from other management systems. Therefore, a heterogeneous database system (HDS) is required to store these data sets for later use in other applications, such as resilience metrics, forensic analysis, and wide-area control (WAC). In addition, it can be used to facilitate event visualization through real-time processing of incoming data.

4. User interface for Cyber–Physical Testbeds

Event Visualization Dashboard

The current power grid consists of several distributed sensors that rely on various communication protocols, hardware, and software resources to provide multidimensional data sets with varying sampling rate to the control center. The significant increase in volume, velocity, and veracity of incoming grid measurements has led to big data challenges that make it difficult for system operators to efficiently monitor grid networks and take necessary corrective actions. Therefore, an event-visualization dashboard that can process physical measurements, communications network traffic, system topology, system logs, firewall rules, and geographical information is needed to facilitate real-time cyber–physical situational awareness. Figure 4a,b show the visualization system pioneered by INL, which focuses on creating a simple real-time actionable interface for dispatchers and cyberdefenders to use for their various roles. The goal of this display is to aggregate meaningful information together, facilitating rapid operational decisions and complementary context for the roles, as the root cause of events can include both cybernetic and physical elements.

To minimize the amount of visual clutter, a simple object that is able to densely pack all required information was needed. Inspiration for the design of the icon comes from the National Fire Protection Association’s hazard identification system, NFPA 704 [66]. This system uses a simple diamond that has been split into four sections. Each of these sections requires a different response. When viewed together the NFPA 704 system provides immediate information about response. The same logical goals were desired for the INL-developed resilience icon, shown in in Figure 4a.

The resilience icon is divided into three sections to represent a system’s physical (using traditional reliability metrics), cybernetic (also using traditional reliability, along with malware detection), and resilience condition. Each of these sections will have colors change based on the state of the system represented by the section. These colors take three forms: green for normal status, yellow to indicate a warning (i.e., that action may be required to prevent a system violation), and red, indicating a system violation has occurred.

The resilience icon also shares similar function to the developed operational trust indicator (OTI) developed for the CyberSAVe application [67]. The OTI system focuses on different metrics, but the idea is the same: a simple and straightforward icon that allows for immediate decisions indicated by the structure and colors of the icon.

The left-most section of the icon is concerned with the physical health of the system. This can include anything that is related to the physical behavior of any components within the power grid (e.g., faults, under voltages, generators nearing capacity limits). The right section of the icon is associated with the cybernetic health of the system, including erroneous connections, failed connections, failed login attempts, suspicious activity, or virus detection. The final (bottom) section displays the resilience indications and uses the adaptive-capacity metric discussed in [68,69,70,71]. In brief, the adaptive capacity of a device shows how much additional real and reactive power could be used to respond to and recover from a disturbance based on a components thermal limits. This metric easily aggregates the adaptive capacity of collections of grid assets. Colors may be assigned in accordance with NERC or IEEE standards with regards to thermal capacity. Furthermore, the icon has a mouse-over feature shown in Figure 4b, which allows for immediate messages to be presented without the delay associated with an actual drill down.

The icon can be associated with single components or aggregations. Figure 5 shows an example of the visualization for the IEEE 33 bus system with several of the busses grouped into aggregated system resources (ASRs). Each of the different ASR units can be selected to drill down into lower levels that display the ASR’s internal components, as shown in Figure 6, where each bus now possesses its own resilience icon. By displaying information relevant to predefined levels of specific aggregated-component resolutions, the user is easily able to locate relevant information without becoming overwhelmed. The interconnections between all of the different elements also represent different states, such as normally closed, closed, normally open, or opened (Figure 5). Thus, the whole state of the system can be visualized accurately to maintain a high degree of state awareness.

5. Cyber–Physical System Testing

Because CPTs are so diverse in nature, developing general standards that enable easy cross comparison is difficult. However, as CPTs are CPSs, it makes sense to examine widely adopted CPS-testing methods in order to determine appropriate testing methods for a particular CPT. With this in mind, Zhou et al. conducted a survey of CPS testing and test beds that identified six testing methods for CPSs: model based, search based, monitor based, fault-injection based, big data driven, and cloud based [72].

Table 4. Testing methods for cyber–physical testbeds adopted from techniques used to test cyber-physical systems.

Testing Method	Description	Drawback
Model based	Simulates testbed behavior to validate performance	Depends on model accuracy, may lack practicality on CPTs largely comprised of simulations
Search based	Discovers anomalous operating points and scope test bed limitations	Large effort to creating SBT algorithm, time consuming testing
Monitor based	Analyzes test bed properties (e.g., voltage) for conformity to expected results	Logical outputs may not always be intuitively known
Fault injection	Injects artificial failure to test for expected response	Test bed fault response may not always be intuitively known
Big data driven	Leverages big data techniques (e.g., statistics) to test for expected response	Big data collection not always available or practical
Cloud based	Leverages cloud computing to test for expected response	Big data collection and cloud connection not always available or practical

summarizes each of these methods.

MBT uses simulations of the same physical, cybernetic, or cyber–physical configurations to validate the CPT by comparing deviations of performance. This method was used in [26] to validate the custom-designed emulators of transmission power lines by comparing the emulation results to Simulink/MatLab models. This form of testing also has the advantage of not being limited to real-time; thus, it may be used to quickly generate results for physical or cybercomponents [73,74,75,76,77,78,79,80,81,82].

Search-based testing (SBT) is a process that leverages genetic algorithms, simulated annealing, or like algorithms to create operating points or scenarios to be tested [72]. Typically, researchers will test a CPT for proper functionality under expected circumstances. In works such as [80,83,84,85,86] SBT was applied in an attempt to discover testing scenarios that would cause abnormal behavior in the CPSs—thus revealing flaws in the design. These same techniques could be applied to CPTs in order to quantify their level of uncertainty or scope of reasonable operation. In combination with MBT, SBT could be an effective means for understanding the limitations of CPTs.

Monitor-based testing of CPSs is the process of conducting an analysis of the time-series data produced by a system [72]. This analysis can include transformations, statistical methods, or simple reporting of the time-based data to verify the result is reasonable [87]. For CPTs this may simply mean troubleshooting outputs from various components to ensure results are reasonable. This is most commonly performed by analyzing raw data as statistical or transform (e.g., Fast Fourier transform) methods may make intuitive analysis difficult. Similar to the monitor-based testing, FBT of CPSs deliberately induces an artificial failure and evaluates the system’s response, making system enhancements as necessary [72]. This method may be more challenging for CPTs because system response to faults is not always known and is often the point of a specific study. However, the number of reasonable responses to a given fault is limited, a condition which may be leveraged to assess the validity of a CPTs simulation or emulation result.

Big data-driven CPS testing uses big data analytical technique to aid in testing by leveraging or enhancing the CPS’s ability to process and store data [72]. Examples of big data-driven CPS testing include creating a big data system architecture, creating a framework for real-time, dynamic data processing, and creating prediction and diagnosis methods [88,89,90,91,92,93,94,95,96]. While big data techniques may not be useful for initial CPT development validation, they could find application in a well-established CPT that seeks to expand and must process large amounts of data. Likewise, cloud-based testing is not likely to be a useful technique for early developmental validation of a CPT, but may be used for well-established CPTs. Cloud-based testing involves feeding data from a CPS (or CPT) to the cloud, where it is then analyzed. This may include network-traffic testing, testing a sensor’s interaction with actuators, and security monitoring [97,98,99,100,101,102,103,104,105,106,107].

The six testing methods in [72] may be used to improve four identified areas of CPSs also identified by Zhou et al.—conformance to standards, robustness of the process, security of the system, and fragility of the system. Conformance of the process attempts to quantify “the degree of compliance between the implementation and the required standards” [72]. More simply stated, the degree of likeness between the intended result and the actual result. For the power grid, this may mean measuring the deviation of voltage or frequency of power delivered to the loads from adopted standards like IEEE or the American National Standards Institute. The robustness of the process refers to assessing the fault tolerance of a system. The security of the system assesses any physical- or cybersecurity issues within the CPSs. The fragility of the system refers to a CPS’s ability to continue operation within acceptable tolerances despite abnormal perturbations to operating conditions (this is also known as system resilience). CPTs are an effective way to assess each of these four areas. In order to develop and validate CPTs, however, MBT, SBT, and monitor-based testing may be effective tools to ensure accurate behavior. Fault-injection, big data, and cloud-based testing, on the other hand, may be limited to more-intuitive use cases for functionality validation in already established test beds.

6. Example of Cyber–Physical Analysis And Design

6.1. Simulation-Based Case Study

Figure 7 presents a modified IEEE 33-bus distributed system that was modeled as a radial network with the system rating voltage of 12.66 kV. This system consists of 33 buses and 32 connecting lines. Further, it was classified into 6 ASRs, which are grouped based on proximity, similar to a microgrid [70], and were fed by a synchronous generator. In this system, the total connected active power load and the reactive power load demands are 3.715 MW and 2.300 MVAr. The given system was modeled in ARTEMiS/SSN (eMEGASIM) in the MATLAB-Simulink environment and simulated at a time step of 50 microseconds in the electromagnetic transient (EMT) domain. In addition, circuit breakers, tie-line reclosers, fault indicators, and a 10-kW grid-connected PV array on Bus 25 were modeled. The modeled tie-lines, initially set to open, provided interconnections between multiple ASRs and also facilitated network reconfiguration during line contingencies, including line faults. The modeled PV array was operating as a constant power-factor mode or active-reactive power (P-Q) control mode while supplying active power of 10 kW.

6.2. Cyber-Attack Vectors

The increased dependency on information and communication technologies (ICTs) has made power systems increasingly vulnerable to various cyber–physical attacks [108]. These attacks range from reconnaissance attacks, the objective of which is to gain information on the system, to attacks that attempt to disrupt the system such as denial of service (DoS), replay, or data-insertion attacks [109,110]. DoS attacks are some of the most-common approaches to disrupt communication networks. DoS can be used by an adversary to affect the dynamic performance of power systems, leading to unstable behavior [111]. Replay attacks capture real messages to be replayed later so as to obfuscate the current state of the system [112,113]. False-data-injection attacks manipulate communication data to create confusion and trigger incorrect responses that disrupt the system while preventing detection [114]. Ramp and scaling attacks are examples of false-data-injection attacks. These attacks consist of making small or gradual modifications to true measurements to confuse the system and trigger control actions that are not appropriate for the actual state of the system. Ramp attacks are gradual modifications of true measurements while scaling attacks add or subtract a small percentage value to measurements. These types of attacks can be specifically tuned to cause disruption while evading detection by carefully choosing the scale of the modifications. Using a representative pool of cyberattacks to validate detection and mitigation mechanisms is essential for cyber–physical system testing.

As an illustration of FBT, ramp and scaling attacks against the PV-integrated distribution system (Figure 7) were considered. Further, it was assumed that the inverter of the PV array was compromised, and the attacker was able to modify the internal setting of the inverter by applying the following attack templates.

• Scaling attack: This attack involves modifying the measurement signal to a higher or lower value, depending on the scaling attack parameter, ${\lambda }_{scale}$, as shown in (1).
• Ramp attack: This attack vector involves adding a time-varying ramp signal to the input control signal based on a ramp signal parameter, ${\lambda }_{ramp}$, as shown in (2).

$$P_{scale}=P_i(1+{\lambda }_{scale})$$

(1)

$$P_{ramp}=P_i+{\lambda }_{ramp}\ast t$$

(2)

6.3. Results and Discussions

Figure 8 and Figure 9 show the injected disturbances in power flows at Bus 25 during pulse and ramping attacks on the 10-kW PV array. During the ramp attack, a time-varying ramp signal with the specified parameter (${\lambda }_{ramp}$ = ±200) is added to the DC link reference point (Vdcref) inside the three-phase three-level voltage source converter (VSC) of the PV array after 8 seconds. During the ramp-up attack (${\lambda }_{ramp}$ = +200), it can be observed that the power flow at Bus 25 increases to around 575.6 kW at 9.6 s. However, during the ramp-down attack (${\lambda }_{ramp}$ = −200), the system has a minor impact where the power flow at Bus 25 is gradually reduced to 564.7 kW at 8.2 s, and power flow at this bus is finally recovered at 8.8 s.

During the scaling attack, the Vdcref was modified by half its original value (${\lambda }_{scale}$ = ±0.5), and this attack was performed after 8 s. During the scale-up ((${\lambda }_{scale}$ = +0.5) on Vdcref, the initial power flow was increased to 590 kW at 8.05 s and exhibited a major oscillation with low frequency. During the scale-down ((${\lambda }_{scale}$ = −0.5), the power flow was reduced to 563.4 kW, and a minor oscillation was observed, with high-frequency components as compared to the previous scale-up attack. From these two experiments, it can be inferred that the ramp-up and scale-up attacks have more severe impact than do ramp-down and scale-down attacks. Further, it can be concluded that the impact of cyberattacks depends on the nature of attack, and the scaling attack injects more transient instability than a ramp attack because of its instantaneous change of the signal to extreme values. This result was expected and is an example of FBT validation, discussed in Section 5 as large instantaneous changes (scale attack) should produce more power-flow instability than gradual changes (ramp attack). Additionally, emulated or hardware-based test beds of the IEEE 33 bus system may use models like this to validate their performance (i.e., MBT, also discussed in Section 5).

6.4. Potential Mitigation Solutions for Data-Integrity Attacks

There exist several approaches to development of intrusion-detection systems (IDSs) to detect different classes of data-integrity attacks, which include pulse and scaling attacks. In general, these approaches can be classified into two broad categories: signature-based IDS and anomaly-based IDSs.

• Signature-based IDS relies on network traffic to detect different classes of data-integrity attacks based on the defined attack-signature database. Several IDS tools, including BRO (Zeek), Snort, Firestorm, and Spade can be applied in developing signature-based IDS in real-time in a cyber–physical test bed environment.
• Anomaly-based IDS detects intrusions based on deviations from the normal behavior of the distribution system. It includes different types, such as model-based IDS, machine-learning-based IDS, multi-agent-based IDS. These are discussed below.

  (a)

  Model-based IDS utilizes the current grid information, historical measurements, and other relevant information to develop a baseline model and detects attacks based on the statistical and temporal correlation analysis of incoming grid measurements.

  (b)

  Learning-based IDS applies machine learning, deep-learning, and data mining algorithms to identify different types of stealthy and sophisticated attacks using grid measurements. Further, it also distinguishes them from other events, including line faults, extreme weather events, etc. For example, decision tree algorithms can be utilized in detecting different data integrity attacks using synchrophasor measurements in real-time.

  (c)

  Multi-agent-based IDS consists of several distributed agents that utilize both cyber and physical measurements to develop anomaly detection algorithms through agent co-ordination and information sharing. Further, it can be utilized for developing attack-resilient protection and control schemes that can detect attacks at an early stage and initiate necessary mitigation strategies to restore the normal operation of the power grid.

7. Conclusions

The design tradeoffs between various elements in a CPT test bed can be broken down into three different categories: physical components, soft components, and user interfaces. Representations of CPTs physical, cybernetic, cyber–physical parts were reviewed within the context of balancing cost, computational expense, and fidelity. The scalability of simulated systems within CPTs enables them to be highly cost effective, but with a lower resolution than more computationally expensive system emulators. Physical hardware was considered to have no computational expense, but had the highest financial cost associated with operation and maintenance. Relevant communication protocols were described, as were timing considerations to be used based on the goals of the CPT. Wide-area test bed representations with data visualization aspects of CPTs were also explored. Methods for testing CPSs were leveraged as potential avenues for developing generalized testing methods to validate the performance of CPTs. An initial demonstration on an IEEE 33 bus system, together with examples for how MBT and FBT may be applied to validate the CPT performance, was also discussed. Lastly, detection strategies for these types of attacks were considered. The authors hope to inspire more discussion about CPT testing and validation to enable better comparison among different test beds. CPTs enable easy exploration for improving CPSs that impact everyday life. Thus, developing effective methods to ensure proper functionality and better defining the limitations of these CPTs is an important subject in need of further exploration.