Method for Systematic Assessment of Requirement Change Risk in Industrial Practice

Abstract

Requirement changes and cascading effects of change propagation are major sources of inefficiencies in product development and increase the risk of project failure. Risk management regarding these requirement changes yields the potential to handle such changes efficiently. Currently unlocked, a systematic approach is required for risk management to assess the risk of a requirement change with appropriate effort in industrial application. Within the paper at hand, a novel method for systematic assessment of requirement change risk is presented. It is developed in a multiple case study approach with three product development projects from different industrial branches. The change risk is assessed by combining change likelihood and change impact. Propagation effects are considered by analyzing requirement interrelations. To limit application effort, a tailorable approach towards assessment of change causes based on generalized influence factors and a pre-defined rule set for semi-automatized assessment of requirements interrelations is used. A software prototype is developed and implemented to enable evaluation and transfer to industrial application. The approach is evaluated using a combination of case study projects, stakeholder workshops, questionnaires and semi-structured interviews. Applying the method, the risks of requirement changes are assessed systematically, and subsequent risk management is enabled. The contribution at hand opens up the research space of risk management in handling requirement changes which is currently almost unexploited. At the same time, it enables more efficient product development.

1. Introduction

The development of technical systems is characterized by various uncertainties. Uncertainties are inevitable. They are induced internally as well as externally [1,2]. Internally, by insufficient in-depth understanding of the system in early product development phases as a result of complexity, for instance leading to incomplete requirement sets [3]. Externally, for instance, by volatile customer needs and regulations [2]. Both can lead to changes in the requirement set. Those changes occur in all development phases and can be caused by new requirements, changed requirements or deleted requirements (Figure 1).

Studies on the extent of changes indicate that more than half of all system requirements have a high risk of change throughout the project [6]. Each change may result in adaption effort and time-consuming iterations, especially if it leads to a subsequent engineering change. The impact of changes multiplies due to change propagation. Requirements as well as system elements are highly interconnected amongst each other [7]. Hence, changing one requirement often initiates a number of subsequent changes [8,9], each resulting in further development effort due to change propagation.

This is illustrated by an example from the development of an LED front headlamp for BMW [10]: The requirement for the maximum temperature of the LED substrate is a central influence factor regarding performance of the headlamp and was originally set to 120 °C. This was reduced to 90 °C in the course of development. In order to lower the temperature, more LEDs had to be installed, each with lower energy consumption. This, in turn, required to change the requirements on installation space. Overall, the initial requirement change caused change propagation towards requirements for energy consumption, number of LEDs and geometry.

Managing such changes in an efficient manner is one of the biggest challenges in development projects [6,11,12]. The aim of the contribution at hand is to open up a new action space for requirement change management in research and industrial practice, by providing the basis for systematic risk management.

Today, requirement changes are often managed in a reactive and unsystematic manner in industrial practice; changes are based on subjective judgement instead of objective data [13,14]. This is confirmed by various studies [6,15,16,17,18,19]: Inefficiencies by requirement changes are one of the most important causes for development projects to miss project objectives (quality, cost and time).

Despite those findings, there is little research on how to manage requirement changes efficiently [6,20]. Contributions mostly come from Requirements Change Management (RCM) [20] or Engineering Change Management (ECM) [21]. From a time perspective, approaches can be divided into three categories: Pre-change, in-change and post-change [21]. Established approaches focus on decision making (in-change) and change implementation (post-change) [20,22,23]. Especially in the context of risk management for requirement changes (pre-change) very few approaches exist [1,6,13]. Risk management may unlock a new scope of action (for instance, change prevention or reduction of change impact). Therefore, it yields large potential in the efficient management of requirement changes, as it does in other application fields. To enable risk management, the risk of a requirement changes need to be assessable as a prerequisite. Then, based on the specific risk characteristics, targeted risk management actions can be initiated.

So far, the risk of requirement changes is either not assessed at all or comes from an unsystematic expert estimation with no differentiation of risk factors (for instance, drivers of change likelihood and impact). Therefore, the selection of proper actions is vague and based on insufficient information. To exploit the potential of risk management in the efficient handling of requirement changes, a major challenge is to provide a support for systematic assessment of requirement change risks.

The second challenge comes from the wide-spread phenomena of insufficient management of requirement change in practice [24,25,26]. The challenge is to reach low application effort of risk assessment in order to increase usability in practice. Today, key areas of research in RCM und ECM are dependency analysis and impact analysis. Both are essential for the assessment of change propagation as well as the decision-making and change implementation. Furthermore, the dependency and impact analysis cause a major proportion of application effort and therefore play a critical role in industrial application. Interviews with practitioners also show that those activities are major barriers to systematic change management in general, due to a lack of methodical support, cross-disciplinary system understanding and problem awareness regarding change effects. As a result, the perceived cost-benefit ratio of systematic dependency and impact analysis is negative. The interviewees were eleven experts from the industrial partners of the OptiAMix project (Section 3). Three of them work as developers and eight interviewees are acting in leading positions in product development (for instance, chief engineers, senior development experts or project leaders).

Research on ECM has mechanical engineering background and focuses on the solution space [21]. ECM approaches support the management of changes to engineering elements, for instance, by analysis of physical dependencies between those elements (for instance [27]). Since the number of (physically) interconnected engineering elements are usually not as high as the number of interconnected system requirements, those approaches more often rely on expert evaluation [20,21]. Anyway, the reduction of application effort and therefore the development of (semi-) automated approaches gains relevance, especially to support dependency analysis (for instance [19,28,29]).

Research in RCM mostly comes from software development [20]. Software development projects face high numbers of system requirements and therefore, automation is a key challenge. Several approaches with a high degree of automation exist [30,31], but rely on structured and software-specific data like source code or Unified Modeling Language (UML) diagrams (for instance [32,33]) for dependency analysis. Therefore, the application in the development of (partly) mechanical systems is limited.

Approaches from both research fields have in common that they predominantly rely on Machine Learning (ML) or expert assessment. Both have unique characteristics that demand for a context specific selection. ML requires large, structured datasets, especially if the analysis consists of multiple parameters (for instance dependency types). Such datasets often do not exist due to a prevalent lack of systematic, document-based requirements engineering in technical development projects [26], which applies in particular to Small and Medium-sized Enterprises (SME). In case appropriate datasets exist, they usually come from requirement change databases of requirements engineering tools like IBM DOORS (for instance [34]). Those can be valuable sources, but the findings may lack of transferability to other development contexts (system, company or branch) and transparency in reasoning. The advantage of ML is that there is very little application effort and no expert knowledge needed after the training phase [35].

Expert assessment has the opposite advantages and disadvantages. Experts’ ability to abstract and express knowledge increases transferability and transparency. The most severe disadvantage is its high application effort and dependence on expert knowledge. The application effort increases with the amount of estimations needed. For dependency analysis, many approaches rely on pairwise comparison between requirements to determine the existence and type of dependencies [24,36,37]. This leads to an exponential growth in application effort in relation to the number of requirements. Thus, its application usability is limited to small requirement sets. Quality of expert assessment strongly relies on expert knowledge and experiences. Therefore, it is subjective and biased. This reduces comparability and validity of results, especially in interdisciplinary contexts when diverse knowledge is needed [34]. Although there are approved actions to reduce subjectivity like group assessments, those lead to even higher application effort.

Approaches that combine ML and expert assessment potentially dissolve the before mentioned disadvantages. However, a systematic literature indicates that no approaches exist in the context of risk management for requirement change [35]. They transform expert knowledge into structured, machine-readable data (for instance by ontologies), which can subsequently be joint with ML algorithms (for instance [31]).

Established requirements engineering software products like IBM DOORS or Siemens Polarion Requirements also do not provide sufficient support in risk management of requirement changes. Those tools enable to model requirement dependencies and track changes manually (traceability). However, they do not include functions for automated dependency detection besides inconsistency analysis. Therefore, those tools facilitate expert assessment, mainly by providing user interfaces, but do not reduce application effort. Additionally, those tools are merely used by large companies. SMEs more often rely on generic software like Microsoft Office for requirements management [26] and have no support in dependency analysis and risk assessment at all.

Overall, there is a lack of research in two areas: (1) the systematic assessment of requirement changes and (2) approaches to reduce application effort of risk assessment without relying on large datasets of past changes. On that account, the contribution at hand aims to answer the following two Research Questions (RQ):

• RQ 1: How to assess the risk of requirement changes systematically?
• RQ 2: How to reach an application effort for risk assessment, which is appropriate for industrial application?

Answering these questions helps to reduce negative effects of requirement changes in development projects and thereby the risk of project failure [6,38]. Due to the dynamic environment and inevitability of uncertainties in development as well as the increase in system complexity, the efficient management of requirement changes will continuously gain importance for companies to stay competitive and focus on the latest stakeholder needs. The paper at hand contributes a method for systematic assessment of requirement change risk and a software prototype that enables semi-automated application within the development context. Findings from status quo analysis with the industry partners clarifies the needs and leverage points for support in this field and evaluation proves usability and applicability of the method. Accordingly, the systematic assessment of requirement change risk provides the prerequisite for future research on risk management for requirement changes and its integration into development practices.

2. Materials and Methods

2.1. Requirement Change–Definition and Demarcation

Before developing a method to assess risk of requirement change systematically, it is necessary to develop a concept of what is defined as a requirement change and what factors need to be considered for risk assessment. In literature, there is no common definition of a requirement change, although this is essential to define the boundary of support. Some research sees requirement changes as part of Engineering Change (EC) (for instance [6,29]). Following the predominant understanding of EC, it is focused on the solution space solely [21]. Within the contribution at hand, EC is separated from requirement change and defined as follows:

“Engineering changes are changes and/or modification in fits, functions, materials, dimensions, etc. of a product and constituent components after the design is released”

([39]; S.481)

In contrast to EC, requirement change focuses on the problem space and in particular requirement artifacts. Those can be of any specification level [40], but need to be formalized. Formalized is understood as somehow processable and includes natural language description. Within the contribution at hand, the following definition is used:

“Requirement changes are defined as any change made to formal requirements.”

(adapted from [6,29])

This definition already includes another demarcation: In literature, requirement change is often used as a general term for all changes in the requirement set. Following the understanding of Forsberg (Figure 1), changes in the requirement set come from new requirements, changed requirements and deleted requirements. The ongoing refinement of a requirement by a set of subordinated requirements is included in new requirements. According to the definition of requirement changes, the contribution at hand focuses on changed requirements. Accordingly, new or deleted requirements are not primarily addressed.

2.2. Conceptual Model of Requirement Change Risk Factors

So far, no holistic model of risk factors exists due to the lack of attention for risk management of requirement changes. The existing approaches focus on either one or both of the two basic dimensions of risk: Likelihood of change and impact of change. This is adequate for rough expert estimations but needs to be detailed for a decisive risk assessment as a basis of risk management. Risk factors need to be unambiguous and independent to be analyzed and assessed systematically. Therefore, the risk dimensions likelihood and impact are broken down into its underlying risk factors.

To identify risk factors that influence the likelihood of change, the root causes of a change need to be analyzed. Root causes trigger changes and therefore constitute the change likelihood. Studies show that many requirement changes are triggered by the solution space [40,41,42]. Although EC and requirement change are differentiated, there still is a close dependency between them. On the one hand, requirement changes can affect the boundaries of solution space and thereby lead to EC. On the other hand, requirement changes can be triggered by the solution space, for instance, design improvements or increased understanding of the technical solution [20]. Requirement changes can also come from the evolution or change of stakeholder needs (for instance, customer preferences and priorities), which directly affect the problem space [43,44,45]. A systematic review on causes of requirement changes from Jayatilleke and Lai revealed that besides the interconnection of problem and solution space there are four additional categories of change causes [20]:

• External market (including stakeholders like customers/users, government bodies and competitors).
• Customer organization (changes impact the needs of the customer and as a result, impact the design and requirements).
• Project vision (better understanding of the problem space from a customer point-of-view and the emergence of new opportunities and challenges).
• Requirement specification (developer’s point-of-view and their improved understanding of the problem space and resolution of ambiguities related to requirements).
• Solution (related to the solution of the customer’s requirements and the techniques used to resolve this).

In addition to causes of an initial requirement change, change propagation can also lead to requirement changes. The importance of propagation effects as a subsequent cause for requirement changes is stressed, for instance, by a long lasting but clearly documented case study on the development of a complex sensor system in aerospace industry with 41,500 change requests over a period of eight years [8]. The study shows that in parts, up to 50% of requirement changes were triggered by other requirement changes. To capture those, the current understanding of requirement change causes is enriched by the two dimensions exogenous and endogenous. Exogenous change causes trigger an initial requirement change. Those are not related to requirement interplay and come from outside the requirement set. Within the exogenous change causes, the more common differentiation of external and internal causes is used to categorize causes from inside the organization (internal; for instance, incomplete system understanding) or outside the organization (external; for instance, volatile customer needs). Endogenous changes can only occur after an initial change was triggered and come from the propagation effects between requirements. The overall concept of requirement change causes is illustrated in Figure 2.

Looking at the impact of change, there is no in-depth analysis of risk factors published yet. Still, the two basic risk factors of change impact can be derived from approaches for change impact analysis (for instance [9,19,46]). On the one hand the incoming change request and on the other hand the change behavior of the affected requirement influences the change impact. The change request can be triggered by an exogenous change cause or by change propagation. The change behavior depends on a range of individual factors like the permissible range of requirement parameters or the experts’ decision on change implementation. It results in a local change impact and potentially in an outgoing change request (change propagation).

From a risk assessment point of view, change requests from exogenous change causes as well as change behavior are context-specific. In contrast, endogenous change requests resulting from change propagation can be estimated at least approximately by means of generic dependency analysis; for instance, by identifying causal requirement dependencies based on physical laws like weight, stiffness and material. For this reason, a distinction is made between context-specific risk factors and partly context-independent risk factors. The initial change request and the local impact of a change related to a single requirement are context-specific. Propagation effect is partly context-independent. This differentiation is useful because context-specific risk factors usually need to be assessed by experts or based on specific datasets, whilst context-independent risk factors might be assessable by generic sources (for instance, formulas or open access datasets). The differentiation of initial and subsequent changes from a change propagation perspective is illustrated in Figure 3.

The risk factors of change likelihood as well as change impact are merged into a conceptual model. This model combines all relevant risk factors and can be used as a starting point to develop methods, enable the assessment of one or more risk factors and to merge those into an overall risk assessment of requirement changes. Within the model, the two aforementioned dimensions of change risk are separated. Looking at the change likelihood, an analysis of exogenous change causes enables to assess the chance of an initial change request. For requirements not directly affected by this initial change request, the endogenous change likelihood can be estimated based on change propagation. This enables an assessment of change likelihood for directly as well as indirectly affected requirements. To assess the overall change impact, the local change impacts need to be added up for all requirements affected by change propagation. It starts with the initial change request as the trigger and the change behavior of the affected requirement. From then on, a chain effect might be activated, consisting of requirement change behavior sending out subsequent change requests to interconnected requirements, potentially leading to further changes (change propagation). The overall conceptual model of risk factors for requirement change is illustrated in Figure 4.

The concept covers risk factors for all kinds of changes in the requirement set: New, changed or deleted requirements (Figure 1). Still, the following approach to assess the change risk is focused on requirement changes solely. This is because the risk assessment and the subsequent risk management might have a different focus for new or deleted requirements. For example, to perform risk management on requirements added to the requirement set later on, activities that improve requirement elicitation process, and thereby lower the number of new requirements added, will play a major role. Risk management regarding changes made to formal requirements will most likely put a higher weight on change propagation instead of improving the elicitation process.

3. Research Approach

To answer the two research questions derived in Section 1, the Design Research Methodology (DRM) proposed by Blessing and Chakrabarti [47] was used. DRM was selected due to the iterative, flexible research approach and context specific guidance. To adequately incorporate industrial practice it is enriched by implications from Ulrich [48]. He focuses on applied research and encourages continuous incorporation of practitioners and an extensive analysis of the application context. For each research question, DRM was utilized separately, but not independently. The overall research approach and the related steps of DRM are illustrated in Figure 5.

Results of the initial step “Research Clarification” were generated by structured literature research following the approach of Machi and McEvoy [49]. Details on search terms can be found in Figure 5.

Within the second step “Descriptive Study I”, initial reasoning of the research field is extended by further theoretical and practical investigations [47]. For the theoretical perspective, a literature analysis was conducted for both RQs. In addition, interviews with industrial partners from various companies were conducted to get insights into what causes application effort, current state of tools and methods used and needs from industrial application. Companies included SMEs as well as larger companies from various industrial branches such as additive contract manufacturing, machinery and equipment engineering, automotive development and forming technology.

Based on this in-depth analysis, a method for systematic assessment of requirement change risk and a software prototype to support the application of the method were developed within the “Prescriptive Study”. The assessment included a priority score and a dependency score. The priority score combined change impact and exogenous change likelihood. It was calculated according to a Failure Mode and Effect Analysis (FMEA) logic, which is well known in industrial practice and enables to quantitatively merge different influence factors of change risk [43,50]. The dependency score was used to assess change propagation and endogenous change likelihood. The two-stage approach relies on a requirement structure matrix [51]. Within the first stage, the active and passive sum of each requirement was assessed [25,52] and used for initial categorization. For further differentiation between requirements of one category, the Euclidean distance was used [53].

In a multiple case study approach according to Yin [54], three industrial product development projects were accompanied and used as a data source. Case studies were provided by four industrial partners from the third-party funded research project “OptiAMix”. Within this project, a novel approach towards holistic product development, topology optimization and a digital process chain for additive manufacturing was developed. Part of this was a generic requirement change risk assessment which was tailored for, but not limited to, additively manufactured products. The three case studies cover a large range of both, enterprise characteristics and industrial branches to ensure broad applicability in industrial practice. Details on the three case studies are shown in

Table 1. Case study details.

No.	Title	Industrial Branch	Enterprise(s)	Exposure to Requirements Engineering
1	Grab	Forming	Large	Low
2	Control cam	Bakery technology/machinery	SMEs 1	Low
3	Rear wing mount	Automotive	Large	High

¹ This case study included two industrial partners in close cooperation: A company developing bakery machinery as well as an engineering service provider in the context of additive manufacturing.

below.

A multiple case study approach was chosen to ensure broad applicability in industrial practice by considering requirements from practice for both, method and software and from a range of industrial branches and enterprise characteristics. Besides the requirements from those case studies, they were used to test interim results in an industrial setting and generate data for the evaluation within “Descriptive Study II”.

Within the final step “Descriptive Study II”, data acquired during the case studies was analyzed with regard to the potential improvement by applying method and software in industrial practice. Evaluation was done by combining the analysis of historical data and data gathered during the product development projects carried out in context of the collaborative project “OptiAMix” (www.optiamix.de), funded by the German Federal Ministry of Education and Research (BMBF). Additionally, end-user workshops on correctness and completeness of functionality as well as unexploited leverages were conducted. For application and success evaluation, expert questionnaires about the status quo, the method and software prototype as well as future research potentials were used and elaborated by findings from semi-structured interviews (60–90 min) with developers and project leaders. Results of DS-II are presented in Section 5.

4. Results

Within this Section, the results of the “Prescriptive Study” of the DRM were presented. Within Section 4.1, the method for systematic assessment of requirement change risk (RQ1) is introduced. Afterwards the development of the software prototype (RQ2) is described.

4.1. Method for Systematic Assessment of Requirement Change Risk

To cover all risk factors, the method was based on two methodical approaches. Within Section 4.1.1, an approach to consider the change impact as well as the exogenous change likelihood was presented. Those risk factors were grouped, since both rely on context specific data. Therefore, from a methodical point of view they cannot be assessed based on the requirement data itself and require expert knowledge. In the following Section 4.1.2, change propagation was assessed to estimate endogenous effects of requirement changes. These effects can partly be derived from context-independent data like dependencies and interrelations of requirements (Section 2) and therefore allow a different methodical approach than context specific assessments.

4.1.1. Consideration of Change Impact and Exogenous Change Likelihood

To assess the change impact and exogenous change likelihood a literature study previously published in [55] indicated that, various boundary conditions such as the source of the requirements, the market situation of the enterprise or the product life cycle have to be considered. To assess change impact and exogenous change likelihood, the sources of requirements and their change behavior have to be considered [55]. Yet, this knowledge is implicit and requires formalization. Within the three case studies, the theoretical requirements were extended by a practical perspective. Semi-structured problem centered interviews (see [56]) were conducted with designers from all three case study enterprises. Results were clustered and the following practical requirements were derived (see [14] for details):

• Efficiency of application.
• User-friendly approach.
• Transparency of results.
• Integration into existing business and requirements engineering processes.
• Formalization of implicit knowledge.
• Increasing learning effect.

To assess change impact and exogenous change likelihood, various criteria can be applied [14,55]. Each requirement needs to be analyzed based on those criteria to estimate change impact and exogenous change likelihood. For an efficient application, requirements were categorized. Such generalization enables not assessing each requirement individually but to assess categories that include a set of similar requirements. It also enables to draw generic conclusions on change risk and thereby helps to build a database for later on analysis (Section 1). The assessment of each category may be industry-, enterprise- or even project-specific. The trade-off between a significant reduction of effort for application and loss in accuracy that comes with the degree of generalization was discussed in a project workshop with all industrial partners. This led to the need to determine the degree of generalization individually for each usage of the method. This was integrated by a flexible database which can be extended or reduced based on the individual project characteristics like budget, novelty or willingness to take risks.

First step of the method is to identify exogenous change triggers (influence factor) which may result in requirement changes. Examples are customer needs, regulations, pricing, technical specifications and production machinery. Those are used as categories for requirements with similar background and likelihood to be affected by a change request. Second step is to assess each category based on a set of criteria: Uncertainty (U), dynamics (D) and relevance (R) [14]. A detailed description of these criteria is shown in

Table 2. Description of criteria.

	Criterion	Description
U	Uncertainty	Assessment of predictability, measurability, completeness and individual assessment of the likelihood of a requirement change resulting from the influence factor.
D	Dynamics	Frequency and intensity of changes to the influence factor as well as timing and potential to discover changes.
R	Relevance	Criticality and interference of changes to the influence factor as well as feasibility of changes of such and the potential to handle these.

below. The assessment is supported by a list of sub-criteria and reflexive questions outlined in

Table 3. Reflexive questions.

	Criterion	Reflexive Questions
U	Uncertainty	Can we precisely specify or quantify the requirements related to this influence factor? Can we assume that the requirements related to this influence factor are fully known? (implicit vs. explicit requirements). How familiar are requirements related to this influence factor and how precisely can we assess the statements (needs/interests)? (new vs. existing customer; anonymous customer). Are the requirements related to this influence factor company-specific or standard requirements in the industry?
D	Dynamics	How often, in our estimation, will there be changes to requirements related to this influence factor? (Frequency/constancy of requirements). When do changes in requirements related to this influence factor usually become known? (immediately vs. delayed). When do changes of requirements related to this influence factor occur in the course of the project? Can we influence changes in the requirements related to this influence factor? How severe are the changes of requirements related to this influence factor?
R	Relevance	How critical to success are the requirements of this sphere of influence? Are the changes in the requirements of this sphere of influence feasible? Is it possible to limit the effects of changes in the requirements of this sphere of influence?

.

All three criteria were assessed on a scale from 1 (low) to 10 (high). In order to reduce subjectivity, a set of reflexive questions is given (Table 3). Additionally, the assessment was made as a group decision, including experts from all involved domains. Biases were reduced by conducting the assessment while considering different perspectives (for instance, coming from the domain specific background) [36]. In the case study projects, the assessment was primarily made by the development team (2 to 10 people). In case of difficulties to assess cross-domain interdependencies, further domain experts were consulted.

For an intuitive applicability, influence areas are prioritized based on the three criteria in an approach similar to FMEA [50,57]. The priority score is calculated according to the equation below:

$$Priority Score \left(PS\right)=Uncertainty \left(U\right)\ast Dynamics \left(D\right)\ast Relevance \left(R\right)$$

with the priority score, both change impact and exogenous change likelihood are assessed in an intuitive, industrially common approach similar to FMEA. The integral for the priority score is [1, 1000]. The priority score is then combined with the dependency score described in the following Section 4.1.2.

4.1.2. Consideration of Change Propagation Impact and Endogenous Change Likelihood

Despite change impact and exogenous change likelihood, endogenous factors such as endogenous change likelihood and the impact of change propagation have to be considered in assessing the risk of a requirement change. Based on the requirements dependency model proposed by Pohl [58], a matrix based-approach based on a modified DSM—the Requirements Structure Matrix (RSM) was used (for detail, see [59]). To enhance the practical applicability of the method especially in SMEs, an automatized assessment for requirement interrelations was applied (see [51]).

Requirement interrelations are currently not tracked due to a shortcoming in application of RE tools in SMEs and use of such information (Section 1). By applying the semi-automatized approach, the effort for assessing requirement interrelations can be reduced significantly, compared to a manual pairwise assessment. In order to assess requirement interrelations semi-automatically by applying artificial neural networks, machine learning or ontology-based Artificial Intelligence (AI) approaches, the required number of samples has to be high enough to ensure precision. When it comes to product development projects especially in SMEs, the number of samples is relatively low.

Therefore, instead of using a large number of samples and AI, the semi-automatic assessment of requirement interrelations is based on a formalized rule set [51]. To ensure broad applicability and transferability as well as comprehensiveness [60], the main feature list developed by Pahl and Beitz [61] was chosen as a basis for the rule set and tailored towards the project context. In a first step, requirements were elicited using the main feature list and documented in a requirements list. Within this list, the related main feature was assigned to each requirement. Interrelations among these main features and further context specific features (i.e., hygienic design in case of case study 2 “control cam”) were assessed by experts on the scale 0, 1 and 2. “0” represented no interrelation, “1” a weak interrelation and “2” a strong interrelation between requirements related to the main features. By considering generic main features rather than each specific requirement, the application effort was reduced from an exponential correlation of the number of requirements and assessment effort (pairwise assessment) towards a one-time effort with disproportionately low assessments to be made. Similar to the categorization by influence factors, the degree of generalization within the rule set can be determined individually to suit the project characteristics. Additionally, by generalization the dataset on interrelations can be transferred to similar development projects, further reducing the application effort beyond the initial use. The generalization—though potentially reducing accuracy—was judged positively by the case study partners as the potential benefit of reduced effort succeeded the potential loss of accuracy. This assessment was made with participants from all industrial partners. Practitioners agreed that application effort of individual assessment would be too high for industrial application, if the requirement set exceeds about 10 requirements (exponential growth). Therefore, generalization was considered as useful to reduce application effort and maintain sufficient accuracy of results. This estimation was later confirmed as part of the evaluation (Section 5). The rule set is shown in Figure 6 below.

Given the semi-automatized assessment of requirement interrelations, unconnected elements in the requirement list were transferred into a RSM [51]. The transfer process is shown in Figure 7 below. Requirements were structured according to correlated main features (structured requirements list). This list was used as an input for semi-automatized assessment of requirement interrelations. Results are presented to the user for check-up and adaptation if required.

Based on the RSM, interrelations between requirements can be analyzed. Graph-theoretical approaches such as active and passive sum are capable for the characterization of requirements [14,24,48,62]:

• By the Active Sum (AS), the impact of a single requirements on other requirements is described.
• By the Passive Sum (PS), the potential impact from other requirements on a single requirement is described.

In order to classify the change propagation impact, both AS_i and PS_i of a requirement r_i are considered for classification based on the average AS (ØAS) and PS (ØPS), see

Table 4. Classification of requirements according to Active Sum (AS) and Passive Sum (PS).

	AS
PS	ASi ≥ ØAS	ASi < ØAS
PSi ≥ ØPS	Class I	Class II
PSi < ØPS	Class I	Class III

below.

Three classes among requirements were differentiated: Within Class I, all requirements with an above average AS were clustered. These were considered as most important due to their driving nature for change propagation. Within the second class, Class II; all requirements with an above average PS, but below average AS were clustered. These requirements were likely to be influenced by other requirements (mostly from Class I) and therefore, potentially, have a high endogenous change likelihood. Class III contained requirements with both AS and PS below average. This means minor influence on other requirements. For increased precision in classification, a dependency score is calculated based on the classification described. Therefore, the following assignment of numerical values on the interval [0, 3] shown in

Table 5. Classification and assigned numerical values.

Class	Numerical Boundaries
	Lower	Upper
I	2	3
II	1	2
III	0	1

is defined:

For a more detailed differentiation within the numerical boundaries, the assignment of rational numbers was done using the Euclidean distance of a requirement r_i specified by the coordinate values (AS_i/PS_i) to the class boundaries (ØAS and ØPS). The resulting numerical value on the interval [0, 3] was defined as the dependency score.

By using algorithms based on the Google Page Rank algorithm (see [63]) [64], indirect influences can be considered in the assignment of the dependency score. Due to its converging nature and the efficient calculation, indirect requirement interrelations for large requirement sets can be computed efficiently. Using a modified page rank algorithm involving AS and PS was proposed (see [64]). Results can be converted into the numerical values for the dependency score.

Both, change impact and exogenous change likelihood as well as change propagation impact and endogenous change likelihood are merged into risk assessment of requirements and visualized in a risk portfolio, described in Section 4.1.3. The overall method for systematic assessment of requirement change risk is shown in Figure 8 below. Evaluation results (Descriptive Study II) are presented in Section 5.

4.1.3. Requirement Change Risk Portfolio

Change impact and exogenous change likelihood were incorporated in the priority score while change propagation impact and endogenous change likelihood were incorporated in the dependency score. With the case study partners, potential illustration forms were discussed in a workshop. Hereby, a portfolio representing all relevant aspects was rated highest by the industrial applicants. The rating was due to an intuitive understanding by product developers, project teams and project managers compared to for instance table-based approaches. The change risk portfolio is illustrated in Figure 9.

Within the risk of requirement change portfolio, classification boundaries on the priority score-axis (100 and 400) can be defined manually dependent on the risk affinity and project characteristics (for instance, budget, priority or degree of novelty). Presented values of 100 and 400 were proposed values used as default within the method.

4.2. Implementation of a Software Prototype

For answering RQ2, a software prototype was implemented using Mathworks Matlab. Containing all calculation methods and the rule base described in Section 4.1, the software prototype was used for support evaluation parallel to development (Section 4.3), tracking and documentation of requirement changes within the three case studies as well as application and success evaluation (Section 5).

The following main functions were implemented:

• Upload of requirement lists either with or without assigned main features and influence factors (Microsoft Excel data format .xlsx).
• Upload, editing and export of a rule base for requirement interrelations (Microsoft Excel or Mathworks Matlab proprietary format).
• Upload, editing and export of influence factors and their prioritization (Microsoft Excel or Mathworks Matlab proprietary format).
• Assignment of main features and influence factors to requirements.
• Adaptation of classification boundaries for priority score.
• Calculate requirement change risk for the requirement list.
• Illustration using the requirement change risk portfolio.
• Detailed requirement view to identify actively influencing requirements and passively influenced requirements related to a requirement r_i.
• Export of requirement lists enriched by change risk (Microsoft Excel data format .xlsx).
• Distinction of users (for traceability of editing).
• Change history with comment section (requirement changes).
• Editing history with comment section (adaptions of the rule base of prioritization).
• Determination and selection of an individual dataset (rule base and influence factors: Type of project, branch and customer).

The software prototype is illustrated in Figure 10 below.

4.3. Support Evaluation

Following the DRM (Section 3), the evaluation of support (method and software prototype) was divided into three aspects: Support, application and success evaluation. Support evaluation takes place parallel to the development of support (prescriptive study) and tests whether the support elements work as intended individually and in combination with each other. In this context, this was about the ability to assess the risk factors and the overall change risk. Application evaluation investigates whether the support can be used for the task it is intended for. Therefore, it was focused on whether the support is able to provide sufficient information about risk of requirement changes with an application effort appropriate for industrial application. Finally, the success evaluation identifies whether the support indeed contributes to more efficient management of requirement changes. An overview of the evaluation approach is illustrated in Figure 11.

For support evaluation the development of new elements of support (i.e., software functionality) was followed by internal tests on functionality in an iterative manner. Tests were performed based on a set of tests procedures created beforehand. After basic functionalities were developed and integrated, an initial software prototype was provided for external testing to project partners. First experiences were discussed in a project workshop with multiple attendees from all partners. Focus of this workshop is to ensure correctness and completeness of functionality, identify leverages to reduce application effort and raise undetected needs from practice. The results were integrated for the second prototype release (for instance, usage of risk portfolio and import/export via Microsoft Excel data format). After the second release, the prototype was tested regarding usability and applicability in the three case study projects. In preparation for this, all partners used the generic database and adapted it to their individual company and project characteristics. Findings from external testing were discussed in periodic meetings with the development teams and subsequently integrated into the prototype. After two more release and testing cycles, the prototype was considered as sufficiently useable and applicable in industrial practice.

5. Application and Success Evaluation

Then application and success evaluation was executed. This was done by expert evaluation in the form of questionnaires and semi-structured interviews on the one hand and performance measurements based on data from the case study projects on the other hand. The questionnaires were designed under consideration of guidelines by Porst [65] and handed out personally to six experts: Three end-users of the support within the case study projects as well as the three case study project leaders. All participants come from the field of mechanical engineering or industrial engineering and work in the development of additively manufactured products. The six questionnaires are analyzed, consisting of 49 questions with different question types in the following areas:

• Personal background.
• Requirement changes (general).
• Method for systematic assessment of requirement change risk.
• Software prototype.
• Future research potentials.

(detailed results see Appendix A)

Information for both application evaluation as well as success evaluation was gained by the questionnaires. Afterwards one interview per case study (60–90 min) was conducted with participants. In preparation of the interviews, the questionnaires were analyzed to identify open issues and discrepancies. Those were discussed and elaborated. Additionally, the interviews were used to gather more open-ended feedback.

No objective data on requirement changes was available in the partner companies beforehand (Section 1 and Section 4). Therefore, the case study projects are used to systematically document requirement changes and create a database for application evaluation. Based on a template, practitioners captured information about each change like change causes, change occurrence, change impact and management of the change throughout the case study project. This database was used to assess accuracy of support results.

Analysis of the questionnaires and interviews led to the overall result, that the developed method and software prototype target an existing demand for aid in managing requirement changes from a risk perspective and helps to reduce inefficiencies from those. Mainly, by identifying high risk requirements and sensitize developers to consider risk of requirement changes in general and for critical requirements in particular. Additionally, it helps to improve the identification of dependencies across departments/disciplines and communication within the development team.

An in-depth analysis for the case study “control cam” revealed that requirement changes of requirements assessed as high risk led to additional development effort of 176 work hours. Those changes came from better understanding of the problem space by the customer and could have been addressed by risk management actions. After deducting the application effort of the support, users estimated potential savings to about 130 work hours. Although this estimate was insecure, it was consistent with findings from the other case studies and indicates the potential leverage of risk assessment and subsequent risk management for requirement changes. Change documentation from case studies also showed, that besides requirement changes the addition of new requirements triggers iterations and thereby causes additional effort to adapt the requirement set and the current solution.

With regard to the first research question RQ1, the method for risk assessment was considered appropriate. The generalization by using the main feature list and an FMEA logic as well as the three defined priority categories are sufficient. The results of assessment are reasonable and accurate, but strongly depend on the underlying database and therefore the quality of results might vary. This leads to a mixed rating. Due to the ongoing improvement of this database in the course of usage (Section 4) this was expected for initial application. Presumably, the quality of results will increase with experts’ gaining experience and requirement change data was continuously integrated into the database.

The second research question RQ2 addressed the application effort. In comparison to manual pairwise analysis, generalization and automated calculation of risk factors led to a substantial reduction of application effort. Still, analysis showed mixed results. Attendees evaluated the application effort as sufficient for reoccurring application within the course of a project (for instance for each milestone meeting) and across projects in case the database can mostly be reused. For one-time application the effort was too high for the perceived benefit. The ability to create different databases for individually defined categories (for instance, product or customer segments) and reuse them in the course of time was seen as an advantage. A disadvantage of the method was that high effort was required to set up the database initially. In the development of the method, the functionality was implemented to individually assess the required accuracy of risk assessment (for instance, based on project priority and budget) and correspondingly invest effort to create the database. Interviews revealed that this promoted to invest low effort into creating the database. This might also has led to before mentioned issues in quality of results. As a result, application effort was seen as the biggest improvement potential: Especially the reduction of effort to create the initial database, but also the assignment of main features and influence areas to requirements. Since input from support evaluation was continuously integrated during the development of the software prototype, the functionality and usability is considered as appropriate. Still, for industrial application, intuitiveness of workflow and application guidance needs further improvements.

6. Discussion

Evaluation results may be limited due to the number of case studies (three), the number of users interviewed and the prototypical character of the three case studies. To overcome these deficits, both method and software tool were discussed with three more project managers at the companies, not involved so far, who were responsible for new product development projects of various volumes. The use and applicability was confirmed in these interviews. Additionally, four workshops with six to ten experts from various positions of a large engineering company in automotive industry, independent from the OptiAMix project, were conducted. The workshops were used to evaluate the current state of research results and indicate further research directions to foster industrial application. Assessment of results as well as future research was in line with evaluation results. Still, for an extensive success evaluation and as part of ongoing research, the method needs to be applied in several other companies and branches and results need to be benchmarked to other projects.

Since the case studies were in the context of business-to-business markets and centered around additive manufacturing, the evaluation results as well as the underlying database for risk assessment must be reassessed before application in other fields. For instance, there was a predominance of technical aspects in change causes over emotional ones. Although the introduced method captures all categories of change causes, the assessment of change likelihood might be less accurate when emotional aspects were more important and the target group is more diverse (for instance in the field of business-to-consumer markets). As a key to address the diversity of change causes [20,43,44,45], research on different types of uncertainty [2] is essential for validating clear limitations in terms of applicability.

The presented approach contributes to the demand for methodical support in requirement change management research [6,10,13] and practice [15,16,17,18,19]. Its holistic approach on risk assessment enables existing approaches from ECM [9,21,28,36] and RCM [6,10,20] to be integrated into the broader context of risk assessment and management. Additionally, it reveals the usability of domain independent methods for dependency analysis to reduce application effort [1,30,31,32,33].

Evaluation also indicated future research directions. Most importantly, the support needs to be linked to risk management activities more closely. As intended, risk assessment is the first step towards holistic risk management of requirement changes and does not provide sufficient support on its own. To unlock the potential of risk management to reduce risk of project failure, the selection of risk management actions needs to be supported as well as the integration into product development processes and activities. To promote this change and to foster the adaption of design practices, the application effort needs to be further reduced. One option is to fully automate dependency analysis. Furthermore, implementing “learning algorithms” from historical data and applications seems promising. Since there is a lack of such data within the companies, researchers need to find solutions to enables learning from public sources.

With regard to efficiency potentials not addressed so far, two aspects weighed most heavily. First, the scope should be enlarged to include the addition of new requirements. Second, the value presumably increases with the complexity of the system to be developed and the size of the project team. Looking at new requirements added later on in the project, they occurred frequently in the case study projects and caused high development effort. Similar to studies on requirement changes (Section 1), a high proportion of those came from insufficient requirement elicitation and specification. Therefore, risk management might also include actions to encourage the creation of complete and accurate requirement sets. Enlarging the areas of application, the field of complex technical systems seems promising. The case studies are in additive manufacturing and represent elements of subsystems of a complex system. The evaluation indicates that with a larger scope of the system and more interdisciplinary dependencies to be considered, experts’ ability to overlook and manage those decreases and at some point comes to a natural limit. To support risk assessment and management of requirement changes for complex systems requires more automation due to the increased number and variety of requirements but holds great potential. Methodically, the rule set needs to be adapted. Latest research provides a main feature list for interdisciplinary systems that can be used as a basis [66,67]. Future research might address those aspects and help to reduce the risk of project failure for a broader range of development projects. An overview of the evaluation results and future research directions is illustrated in Figure 12.

7. Conclusions

To answer the two research questions, a method for systematic assessment of requirement changes and a software prototype to enable semi-automated application of this method was developed in the contribution at hand. The method was based on a conceptual model of risk factors and uses a combination of expert assessment and categorization to detect and characterize requirements dependencies, exogenous change triggers and change impact. Those are merged to a dependency and priority score, in order to enable systematic risk assessment. To reduce application effort, generalization of influence factors and dependency analysis based on requirement categories is used. This is partly executed by algorithms and implemented in a software-prototype. For integration into development processes, it enables import and export of requirement data, capturing expert knowledge and automated risk calculation from a tailored, reusable dataset. Methodically, application effort is reduced by requirement categorization by influence factors and main features. To enable further improvement of analysis results, learning from historical data and data-based automation (for instance by AI), the software prototype also provides change history tracking and feature to capture expert interactions within the database.

Evaluation with three industry case studies, several interviews and questionnaires proves the overall need for risk management of requirement changes and that the method leads to feasible results. It helps to identify high risk requirements and their dependencies. This enables risk management and contributes to a more efficient way of developing technical products. Still, there is further research required to increase degree of automation and to provide support beyond risk assessment. This includes the integration of risk management into development processes as well as a risk specific selection of countermeasures. Besides this, evaluation indicates great application potential of the method in the development of highly complex, interdisciplinary systems.