Strategies for Software and Hardware Compatibility Testing in Industrial Controllers

Abstract

Mass customization, small batch sizes, high variability of product types and a changing product portfolio during the life cycle of an industrial plant are current trends in the industry. Due to an increasing decoupling of the development of software and hardware components in an industrial context, compatibility problems within industrial control systems arise more and more frequently. In this publication, a strategy concept for compatibility testing is derived and discussed by means of a literature review and applied research. This four-phase strategy concept identifies incompatibilities between software and hardware components in the industrial control environment and enables test engineers to detect problems at an early stage. By automating the compatibility test on an external I-PC, the test can be run both when new software is installed on the industrial controller and when the controller is restarted. Thus, changes to the components are constantly detected and incompatibilities are avoided. Furthermore, early incompatibility detection can ensure that a system remains permanently operational. Based on a discussion, additional strategies are identified to consolidate the robustness and applicability of the presented concept.

1. Introduction

Modern trends in manufacturing are characterized by mass customization, small batch sizes, high variability in product types, and a changing product portfolio during the life cycle of an industrial plant [1]. These trends imply more complex plants [2] that support changes in physical layout, including major engineering upgrades. The complexity of plants, including automation hardware and automation software, is increasing. As the percentage of system functionality realized by software increases, concepts to support automation engineers in dealing with this complexity are urgently needed [3].

Automated testing can help minimize the resources required for software development. However, changes to software necessitate the re-evaluation of functionality through testing. To reduce resource consumption, existing relevant tests can be re-run after ensuring their compatibility with the software after the changes [4]. If a software or its environment are changed, it is necessary to check, on the one hand, whether the desired function is fulfilled and, on the other hand, whether there are any unwanted changes or side effects [5].

The compatibility test as mentioned in this manuscript is based on the use of an industrial programmable logic controller (PLC). These controllers are a crucial technological foundation for automating industrial processes. Despite the advent of Industry 4.0 and the industrial internet, it is reasonable to anticipate that these controllers will remain essential for tomorrow’s production to a significant degree [6]. In order to use PLCs in the future, important paradigms of Industry 4.0 need to be followed. These paradigms (P1–P4) also build the basis for the compatibility test in this manuscript.

• P1: Introduction of Basic Web Technologies

PLCs need to incorporate web servers and HTML pages for browser-based configuration and diagnosis.

• P2: Global Networking of Process Data

Additional modules must enable bidirectional process data transmission between PLCs and supervisory systems using web technologies.

• P3: Introduction of Service Principles

The integration of service functions in PLCs using standardized protocols allows for service-based access to process data.

• P4: Virtualization of PLCs

PLCs must be able to be used as a virtualized representation in the cloud.

In the proposed concept, the target and actual state of the software and hardware components are an essential part of the compatibility test. A stringent test procedure, which can always be repeated in exactly the same way, forms the framework for the concept and is also presented in this paper. The development of a test script, which generates the result table from the compared target and actual states, was not part of this work. However, we have outlined what components such a test script would have to incorporate.

As a result of the compatibility test, the concept offers an overview of found incompatibilities and shows possible reactions. The concept was tested and evaluated on a module of the P2O-Lab [7] of the TU Dresden. The results met the concept requirements, allowed for the detection of incompatibilities, and were therefore published as a German preprint [8].

In the following sections of this manuscript, the used research methods are explored to introduce the reader to the state of the art. The two important concepts of Software-in-the-Loop and Hardware-in-the-loop are explained. In Section 3, the proposed concept is outlined. The proof-of-concept (Section 4) is explained thereafter and is followed by a discussion (Section 5) and a conclusion (Section 6), summarizing the main aspects of this manuscript. At the end of this work, future research directions (Section 7) are outlined.

2. Methods and State of the Art

The analysis of the requirements for the proposed concept was backed by a theoretical examination of the existing compatibility testing strategies in virtual commissioning (VC) and cyber security (CS). These strategies serve as a valuable foundation for developing a compatibility testing concept and are described in the following sections of this article (see Section 2.1, Section 2.2, Section 2.3 and Section 2.4).

The primary criterion for inclusion in compatibility testing, for both VC and CS, was the presence of a model to test against. Consequently, only two test strategies met this criterion and were classified as essential for compatibility testing.

In the following section, the most important literature findings, which were found to be most beneficial to the proposed concept, are highlighted. From virtual commissioning of the testing strategies, Software-in-the-Loop (SiL) and Hardware-in-the-loop (HiL) were found to be a good source of guidance for the development of the proposed concept. The findings from current developments in the literature are highlighted in Section 2.1 and Section 2.2. The other major literature topic which was used to influence the concept development was cyber security. The test strategy of anomaly-based detection (see Section 2.4), from the topic of cyber security, was found to be connected to an approach which could be used for the proposed concept.

2.1. Software-in-the-Loop

In the SiL approach, a virtual PLC is instantiated to test the automation code associated with the behavior models in the simulation layer [9].

This approach makes it possible to integrate software components with an environmental simulation [10]. In addition, this approach enables very fast testing of different scenarios and control algorithms and their flexibility.

The costs for implementing an SiL environment are around sixty times lower than those for an HiL environment. The SiL environment can be available to any developer, while separate equipment is required for HiL [10].

SiL tests are carried out by running the software on normal PC hardware, which makes it possible to identify the most important errors in the functional area. However, the compiler and the processor of a PC may behave differently than on the final automation platform [11].

2.2. Hardware-in-the-Loop

In the HiL approach, a real physical PLC is connected to a simulation layer that executes the system’s behavior models.

All VC processes are based on a virtual model that is connected to a PLC. In the case of an HiL simulation, the PLC is a real hardware controller [12]. Consequently, it is possible to carry out the VC with the PLC, which is then integrated into the production system. According to Mazza [11], this is particularly interesting for the following processes:

• The validation of PLC control strategies based on a virtualized environment with the ability to represent the expected dynamics of the real machine.
• The improvement or comparison of real-world measured data with simulated data (e.g., from virtual sensors).
• Supporting operators during real machine operation through simulated predictions or diagnostics fed by a ’digital twin’ with real data from the field.

2.3. Use of SiL and HiL for the Concept

To summarize, the following reasons can be found why the VC strategies SiL and HiL are very useful:

• Control strategies can be virtually validated without endangering human lives or machines.
• Costs can be reduced thanks to the possibility of debugging (error correction could occur too late during the design process).
• Operators can familiarize themselves with the control systems, including those under construction, thanks to the creation of virtual systems.
• Errors can be found within a few minutes with the help of ‘virtual time’ through simulation.

2.4. Anomaly-Based Detection

Anomaly-based detection uses statistical methods and artificial intelligence to detect unknown attacks [13]. Ourston et al. [14] presented an approach that uses hidden Markov models to detect complex cyber attacks. This method is able to address the problem of multi-stage attacks. Experimental results have shown that this method is more effective than classical machine learning techniques, such as decision trees and artificial neural networks.

Mukkamala et al. [15] developed a method for detecting attacks using K next neighbor algorithms (KNNs) and support vector machines (SVMs). KNNs and SVMs were used to create classifiers based on a list of features. Experimental results showed that KNNs and SVMs are able to detect anomalies and known intruders. Pan et al. [16] developed a hybrid method for detecting attacks by combining KNNs and decision tree algorithms. Experimental results showed that KNNs can detect DoS and probing attacks more effectively than detecting unauthorized access from a remote machine and authorized access to local superuser attacks.

Zhang et al. [17] developed a method based on random forests to detect network intrusions. This method was demonstrated on an intrusion detection data-set. The experimental results showed that the proposed method can achieve a high detection rate with a low false positive rate.

Gaddam et al. [18] developed an anomaly detection approach using cascading K-Means clustering and ID3 decision tree learning algorithms. This method was used to analyze a data-set of network anomalies. Experimental results showed that the detection accuracy is up to 96.24% with a false positive rate of 3%.

Liao and Vemuri [19] developed a classifier for intruder detection using the k-nearest neighbor (kNN) algorithm. This method was used to classify the behavior of programs as normal or intrusive. Experimental results showed that the kNN classifier can effectively detect attacks with a low false positive rate.

Sabhnani and Serpen [20] analyzed an intrusion detection data-set using a set of machine learning algorithms. The data-set includes four types of major attacks, including probing, DoS, user-to-root, and remote-to-local attacks. Simulation results showed that certain classification algorithms are more effective for a particular attack category.

Lee et al. [21] introduced an attack detection method based on cluster analysis to proactively detect DoS attacks. A hierarchical clustering algorithm was used to analyze a data-set for attack detection. Experimental results showed that this method is capable of detecting DoS attacks.

3. Proposed Concept

For the proposed concept, an SiL approach, based on [9], and an HiL approach, based on [12] strategies from VC, highlight the crucial role of a pre-established model in conducting effective tests. Additionally, from the field of CS, the strategy of anomaly-based detection, as described by Ourston et al. [14], emphasizes the use of a predefined model to detect deviations and potential attacks.

Based on this insight, it becomes evident that a model, referred to as the target state, is fundamental in compatibility testing. This target state encompasses the intended software and hardware configurations for compatibility testing.

In contrast to the target state, akin to the SiL or HiL strategies, there is the system to be tested, whether simulated or the physical PLC, to which the test is applied. In the proposed context of compatibility testing, the system under test is referred to as the actual state. It represents the current state of the hardware components connected to the PLC and the state of the software running on these devices. These two main points, the determination of a target and the actual state, form the foundation of the compatibility testing concept.

The model checking and anomaly-based detection strategies from VC and CS can be adapted for use in the context of software uploading and restarting by introducing an additional component external to the PLC. This additional component takes on the role of monitoring the PLC and automatically initiating a compatibility check whenever a software update is pending or the PLC undergoes a restart. In the proposed concept, as outlined in Section 3, this external component is an industrial PC (I-PC). The I-PC runs a test script responsible for managing the software upload to the PLC and monitoring the software restart process of the PLC.

The proposed concept consists of four phases (see Figure 1). Phase one conducts an automated self-test on the PLC connected to a test I-PC. This ensures basic PLC operation requirements, like CPU and I/O module presence, memory checks and power availability (see Section 3.1).

Phases two and three determine the actual and target state of software and hardware components in the system. First, the actual state (see Section 3.3) of the system is determined; then, the determination of the target state (see Section 3.2) follows. It is important to note that the proposed concept is not generally applicable. The proposed concept is based on the TIA Openness API as a foundation for the determination of the actual and target state. The Openness API of the totally integrated automation (TIA) portal from Siemens offers an application programming interface (API) for integrating third-party or custom software solutions with the Siemens ecosystem. The TIA portal also provides developers with the option to program and configure PLCs remotely.

In the fourth phase, the test compares the target and actual states to identify differences and categorizes incompatibilities in error detection tables (see Section 3.5).

3.1. PLC Self-Test

The PLC self-test includes the self-testing and diagnosis of the PLC under test. As required by DIN EN 61131-2, manufacturers of PLC systems must provide means for self-testing and diagnosing the operation of these systems. Furthermore, the self-test must allow a statement about the proper condition of a PLC system.

The PLC self-test according to DIN EN 61131-2 must provide diagnostic means to perform the following actions:

• Monitoring the application program (watch dogs);
• Checking the integrity (freedom from errors) of the memory;
• Checking the correctness of the data exchanged between the memory, processing unit and I/O modules;
• Checking the power supply of the system;
• Monitoring the state of the main processing unit.

The output of the PLC self-test is essential to determine the suitability of the PLC for compatibility testing. The self-test provides the basis for meeting the hardware requirements of the compatibility test. A system which is not in proper condition, i.e., which does not pass the self-test, cannot be used as part of the concept for testing the compatibility between software and hardware.

3.2. Export and Import of Target State

Before the target state can be imported to check compatibility with the new hardware, the files required for the import must first be obtained. Depending on the use case, the requirements for importing the target state differ.

3.2.1. Use Case A—Determination upon PLC Restart

For use case A, the target state is imported from stored data on the test PC, representing the last known actual state before the PLC restart. To ensure an automated sequence, the test PC retains the last actual state, making it available for compatibility tests after a PLC restart. A continuous ping between the test script and the PLC detects restarts, triggering automatic compatibility checks. See Figure 2 for the relevant components. Since the target state is already on the I-PC during a restart, it is simply loaded by the test script.

3.2.2. Use Case B—Determination upon PLC Update

To install new software on the PLC, the target state comes from user-provided update data. These data typically include a TIA project file, which is first opened using the TIA Openness API. The open-source software TiaExportBlocks [22] extracts variable tables in XML format from the TIA project and exports them to an XML file. The hardware topology is exported as an AML file [23] using the CAx export function of the TIA Openness API.

This process results in software data in XML format and hardware data in AML format (CAEX standard) [24].

The test script now has the target state for the software and hardware, which is essential for the compatibility check. The CAEX format in AML uses the PLCopen XML standard [25] for machine-readable hardware topology.

By determining the target state, the necessary hardware and software information is collected for comparison with the actual state in the next step. See Figure 3 for a visual representation of this process.

3.3. Determination of the Actual State

To determine the current state of the software and hardware components, the network state is first assessed using the DCP Scan. Then, the I-PC programmatically contacts the PLC via the Siemens TIA portal Openness API [26].

The DCP “Identify All” command is initially broadcasted over the PROFINET network connected to the test PC. PROFINET (Process Field Network) is an industrial Ethernet standard which is widely used in industrial automation applications for real-time communication between industrial devices such as PLCs, sensors, actuators and other automation components. This command, illustrated in Figure 4, helps identify PROFINET devices physically connected to the network and supporting the DCP protocol. These devices, once found, return hardware information such as the device name, IP address, firmware version and MAC address via the DCP “GET” command.

Subsequently, the PLC software is downloaded to the I-PC through the TIA Openness API. Upon successful download, the AML file for the hardware configuration is generated by selecting “Export CAx data” from the “Tools” tab.

Additionally, variables in the PLC code can be automatically exported to machine-readable XML files from an open project using the open-source software “TiaExportBlocks”.

3.4. Comparison of Target’s Actual State

The core element of the concept is the comparison of the target and actual state, which emerges from the previous chapters. The components and IDs of the target software are compared to the actual software from the PLC, and the imported expected hardware AML structure is compared to the determined DCP scan output. In doing so, the differences are identified and possible problems and inconsistencies are revealed.

When looking at the differences, incompatibilities between the hardware and software components can be identified and categorized in a table (see Section 3.5). The complete comparison process is automated by a custom test script on the I-PC. This automated test process can be performed when new software is uploaded or when the system is rebooted.

The comparison of the two states is implemented by a test script which runs on the I-PC. Since the implementation of this test script was not in the scope of this work, any types of computation requirements or PLC programming language limitations are yet to be determined.

3.5. Error Detection Table

Shown below is the error detection table (

Table 1. Error and inconsistency detection table.

Stage	Detectable Error	Reaction
1—PLC self-test	CPU, E/A-Modul n.a.	Repair hardware on site, install, replace (L)
	Error in the application program	Check code and update (R)
	Data exchange faulty	Check PLC (L)
	Memory integrity violated	Check memory (L)
	Comm. Interface n.a.	Check power supply (L)
2—Import and export of the target state	Error in the logic of the PLC code	Fix logic errors in code (R)
	H/W topology n.a.	Check file structures (R)
	SPS code n.a.	Check file structures (R)
	File/XML structure incorrect	Re-export and import AML (R)
3—Determination of the actual software state	PLC code n.a.	Check connection to PLC (L)
	PLC system n.a.	Check connection to PLC (L)
	PLC in wrong network	Check network configuration (R)
4—Determination of the actual hardware state	No H/W devices found	Check connections (L)
	DCP scan unsuccessful	Check if DCP protocol is supported (L)
	Data from device not retrievable	Check connections (L)
	H/W device in wrong network	Check network configuration (R)

) that is automatically created by the test script as a result of the compatibility check. In order to improve accessibility, the resulting error table can be accessed via the Human Machine Interface from which the compatibility test was administered. The different rows are divided by components of the test flow, as shown in Figure 1. Possible reactions to the incompatibilities found are also included in the table. To simplify the notations within Table 1, the following abbreviations are introduced:

• Hardware (H/W).
• Software (S/W).
• Not available (n.a.).
• Locally solvable problem (L).
• Remotely solvable problem (R).

3.6. Reactions to Incompatibilities

The reactions outlined in Section 3.5 offer a framework for test engineers to respond effectively to identified incompatibilities. It is important to note that these reactions are not automated by the system and require intervention from the test engineer.

The proposed concept for reacting to incompatibilities involves pinpointing which location in the system action is needed to resolve the issues. Additionally, the error detection table identifies whether a specific incompatibility can be fixed remotely (R) or if an on-site engineer intervention (L) is required.

4. Proof of Concept

In the following, the TIA portal and other software tools from Siemens will be used explicitly to evaluate the strategy concept using a reference system to conduct a proof of concept. The evaluation method to be used is a test of the concept on a real plant.

4.1. Proof of Concept Criteria

The criteria for the evaluation correspond to the following requirements for the compatibility test:

• The compatibility test must make a statement about the compatibility, i.e., the compatibility of the simultaneous operation of the hardware and software components connected to the PLC, and display this to the test engineer.
• The compatibility check must be able to run automatically during the PLC restart or during the loading of software onto the PLC.
• The result of the compatibility check must enable a statement to be made about the incompatibilities and errors found.
• At the end of the compatibility check, the test engineer should be shown appropriate reactions to the incompatibilities and errors found.
• The compatibility check should work with hardware and software from different manufacturers in the industry.

4.2. Reference System

The evaluation was carried out by testing the installation of new software on a hardware PLC on a reference system from the biopharmaceutical equipment supplier Sartorius. The concept presented for the compatibility check was applied and carried out. Two Sartorius industrial control systems were used as possible real-world application examples for the compatibility test. There were two different control system models which were used for the proof of concept.

One of the models contains a so-called software PLC. A virtual instance of a fully configurable and usable PLC is created on the I-PC. Communication with the field devices is then realized via an I/O module connected to the I-PC. Most engineering companies also use automation scripts in their systems, which make it easier to process updates.

The second model uses a regular hardware PLC. This is supplied by an external power supply and connected to the I-PC via PROFINET to enable the PLC to be programmed. The hardware PLC is physically connected to the field devices via digital and analog I/O modules.

The assumed real-world existing systems were developed by Sartorius with the TIA portal from Siemens and are therefore suitable for demonstrating the application of the concept. However, no Sartorius control system could be used to evaluate the concept on site. Therefore, the individual steps of the compatibility test were evaluated on a similar system in the form of a decentralized periphery ET200 from Siemens in the P2O-Lab [7] at TU Dresden. The industrial control systems in the P2O-Lab mostly correspond to real-world models with hardware PLCs.

4.3. Evaluation Procedure

In order to evaluate the proposed concept, the necessary steps as defined before were carried out in manual sequence. The following tasks were carried out:

• Powering up the PLC and conducting PLC self-test.
• Preparation of the I-PC and installation of the TIA portal.
• Performing a connectivity check between the I-PC and PLC.
• Determining the actual software status by downloading the PLC software and exporting it to XML using TIAExportBlocks; see Figure 5 and Figure 6.
• Determining the actual hardware status by exporting the AML file; see Figure 7.
• Determining the software target state by importing the update file and exporting it to XML.
• Determining the hardware target state by importing the update file and exporting it to AML.
• Comparing the target and actual status.

4.4. Evaluation Results

As the target and actual states were available in XML and AML at this point, the differences between the states could be identified by comparing them. It was noticed that a PROFINET HMI device specified in the software target state, which was to be addressed by the PLC, could neither be found in the loaded hardware configuration of the AML file nor in the DCP scan.

The new software update contained a change that would have led to an incompatibility of hardware and software if the data had been transferred to the PLC. Accordingly, variables were used in the PLC control code that referred to the non-existent HMI. As a result of the comparison of the target and actual state, the incompatibilities found were shown in the error detection table and made visible to the test engineer.

The test engineer was advised to check the device connections. This enabled him to determine whether it was just a connection error. The HMI device in question was actually found on the module. It was recognized that the PROFINET connection of the device was not properly connected and therefore the device could not be found. Once the error had been rectified locally by the test engineer’s appropriate response, the software update could be transferred to the connected PLC using the TIA portal.

comparison target and actual state
Detected Error	Reaction
H/W device HMI01 not present in actual state	check device connections (L)

5. Discussion

The compatibility check concept presented gives test engineers of industrial control systems an overview of incompatibilities before they occur during operation of the control system. Differences between the target and actual status are shown and possible reactions are identified.

By using an I-PC, the compatibility check is carried out automatically when the PLC is restarted or new software is installed. This means that errors and inconsistencies are detected at an early stage and can be rectified accordingly.

The concept facilitates an automated comparison of the hardware and software of a PLC’s target and actual states, allowing for the detection of inconsistencies or incompatibilities. This check can be applied when restarting the PLC or when updating the software. The error detection methods in Section 3.5 offer an overview of potential errors and reactions. These tables are not complete and cover more inconsistencies in practice than presented, but have been condensed to focus on the core concept.

The concept relies solely on Siemens software and the TIA portal. Future work could explore extending it to other major industrial control manufacturers, contributing to broader research in this domain.

Since the proposed concept does not include an own implementation of a test script, this could also be a useful extension to further verify the applicability of the presented concept. The script could be a starting point for a more in-depth analysis of how the concept could be integrated into other domains of industrial control systems, where different types of data-sets are available.

During the implementation of the proposed concept, the following technical challenges had to be faced:

• Communication Protocol Variability

Supporting multiple communication protocols used by PLCs, such as Modbus, PROFIBUS, PROFINET and EtherNet/IP.

• Version Complexity

Ensuring compatibility across different versions of PLC firmware and software, including updates and changes in communication protocols.

• Operating System Dependencies

Handling compatibility with different operating systems required by PLC programming environments and communication software.

• Hardware Interface Compatibility

Ensuring compatibility with diverse hardware interfaces and communication modules used by PLCs for connecting to devices.

• Data Format Standardization

Addressing the challenge of diverse data formats and structures exchanged between PLCs and the I-PC.

6. Conclusions

In this work, the need for automated compatibility testing is outlined and underpinned by a concept, based on a literature review. The findings of the literature review were presented in the chapter on the state of the art, existing test strategies from VC and CS were discussed, and the concept of compatibility from various literature sources was brought together and examined. The SiL and HiL strategies from the VC and anomaly-based detection from CS were considered to be particularly relevant and decisive for the compatibility testing presented in this study.

Based on this, the most important requirements for the concept were then derived and identified by means of a requirement analysis.The results of the requirement analysis are shown in the concept section of this work.

Furthermore, it was discussed how the strategies determined from the VC and CS can be transferred to the processes of restarting the PLC and installing new software on the PLC.

The proposed concept for carrying out an automatic compatibility check was developed using applied research into the early detection of incompatibilities. A four-phase concept was presented, which is characterized by the comparison of the target and actual states of the software and hardware components.

As a result of the compatibility check, the concept offers an overview of the incompatibilities found and shows possible reactions.

The functionality of the concept was implemented and evaluated on a module of the P2O Lab at TU Dresden. The results met the concept requirements and made it possible to identify incompatibilities.

7. Future Directions

There are numerous avenues for further research in the field of compatibility testing, building upon the theoretical basis established in this work. These possibilities include the following:

• Implementation Variations: Expanding on the presented concept by creating various implementations to assess its flexibility and adaptability to different scenarios. Other scenarios could also involve completely different industry domains.
• Multi System Compatibility: Trying to implement the concept with other PLC types from Siemens, such as LOGO or S7-1200.
• Multi Vendor Compatibility: evaluating and extending the concept to encompass software and hardware configurations from a range of manufacturers, providing a more comprehensive solution.
• Automated Test Script: developing a test script that automates the different phases of the concept, streamlining the compatibility testing process.
• Data Source Extension: expanding the concept to incorporate data from additional sources for determining target and actual states, enhancing its robustness and applicability.
• Fully Automated System: In the future, fully automated compatibility testing systems could significantly benefit test engineers and integrators of industrial control systems. This would enable the early detection of incompatibilities in various Industry 4.0 components, particularly in the face of new hardware and software developments and changes to existing PLC architectures.

These research directions show potential to advance the field of compatibility testing, making it a valuable asset in the ever-evolving landscape of industrial control systems.