A Reverse Engineering Role-Play to Teach Systems Engineering Methods

Abstract

Students engaged in systems engineering education typically lack experience and understanding of the multidisciplinary complexity of systems engineering projects. Consequently, students struggle to understand the value, rationale, and usefulness of established systems engineering methods, often perceiving them as banal or trivial. The paper presents a learning activity based on a three-stage reverse engineering role-play developed to increase students’ awareness of the importance of correctly using systems engineering methods. The activity was developed and integrated in the Systems Engineering course given at Blekinge Institute of Technology. Its effectiveness was analyzed through semistructured self-reflection reports along with two editions of the course. The results showed the development of students’ understanding of how to use systems engineering methods. In particular, the students realized the need to deliver detailed and easy-to-read models to the decision makers. This result was in line with the achievement of some of the intended learning outcomes of the course.

1. Introduction

The evolution of humanity has always been characterized by a continuous attempt to improve the quality of life and to create better living conditions. The search for new ways of satisfying fundamental needs has often led to the engineering of complex systems that, in order to be successful, needed to be socially acceptable and provide value. At present, the term systems engineering (SE) is commonly used to refer to the engineering effort of satisfying articulated sets of needs from different stakeholders, with the intent of developing solutions providing value to an overarching system [1]. The concept of SE was coined in response to the need to develop increasingly complex systems in the aerospace industry [2]. The application of methods and tools for SE has largely impacted, for instance, the development of aircraft or satellites (see References [3,4]). The increasing socioeconomic challenges related to globalization, population growth, economic interdependence, and sustainability are stressing the need for SE competencies to be applied beyond the aerospace context [1]. While the need for such competences is increasingly recognized in industry and research, those are still poorly addressed from a pedagogical perspective for what concerns the education of new systems engineers. As highlighted by the International Council of Systems Engineering, current SE programs focus on practice, with little emphasis on underlying theory, especially for what concerns human and social sciences [1].

SE projects require multidisciplinary skills and cross-functional design teams, including a wide set of disciplines, such as design, manufacturing, system analysis, knowledge management, and sustainability analysis (see, for instance, References [5,6,7]). Different SE methods to structure, formalize, and validate knowledge have been developed by both researchers and industrial practitioners since the 1980s. Those have been applied in a variety of industrial contexts to increase standardization and reduce the risk of misinterpretation and ambiguity [8]. However, while SE methods are easily perceived as beneficial in complex industrial contexts, research has shown that their benefits are not evident for students engaged in SE education. This is because students do not commonly have an understanding of the system complexity and perceive the methods as “open doors”. Students lack the background to understand what the methods offer and, therefore, risk perceiving them as trivialities [9].

This paper addresses this issue by proposing a learning activity developed to increase students’ awareness on the importance of knowledge formalization and communication in early design through established SE methods. The activity, named “Reverse Engineering Role-Play”, was inspired by the principle of active learning [10] and was developed in the frame of an educational initiative to promote the CDIO (Conceive—Design—Implement—Operate) framework [11]. The learning activity featured the active participation of students in a role-play in which they acted as development engineers in a reverse engineering analysis of a real product (stage 1), proposed a new design of the product (stage 2), and assumed the role of managers in decision making, evaluating, and choosing the design to be promoted for further development (stage 3). The knowledge created during the three stages was formalized through SE methods, and it was exchanged between different teams during the role-play.

The purpose of this research was to study the effectiveness of the proposed role-play in developing students’ critical thinking and self-confidence in choosing and applying systems engineering methods. Data about the effectiveness of the activity in contributing to the intended learning outcomes were collected through semistructured individual self-reflection reports. The results showed an increased students’ understanding of the desired quality and level of detail needed for SE methods to be effective in supporting design decision making. Such results were supported by the students’ proposal of concrete actions to be taken to improve the quality of their own SE models produced in stage 1.

The structure of the paper is as follows. Section 2 describes the research approach and the educational context, detailing the course structure and defining how the proposed learning activity fits with the intended learning outcomes. Section 3 presents the challenges in SE education and the current effort in reforming the overall engineering education in light of the CDIO framework. Section 4 details the reverse engineering role-play, describing activities, support material, methods used, and how the knowledge was shared. Section 5 reports the results of the data analysis and discusses them in relation to the intended learning outcomes. Section 6 draws the final conclusions.

2. Research Approach and Educational Context

The research presented investigated SE education literature in search for teaching initiatives following the guidelines promoted by the CDIO framework for engineering education. Research data concerning the use of the proposed learning activity were collected during two editions of the Systems Engineering course that took place in the Mechanical Engineering department at Blekinge Institute of Technology” in 2016 and 2017. In total, 46 students enrolled in Master level studies in Mechanical Engineering, Industrial Economics, and Product-Service Systems Innovation were involved. Data about students’ activities and performances were analyzed by means of retrospective analysis of individual semistructured self-reflection reports, filled in as the last activity of the role-play. The self-reflection reports were analyzed by categorizing the content into five categories, namely: The reason for a choice, the confidence in the decision, the wish for information, the wish for communication, and the individual learning.

2.1. Description of the Systems Engineering Course

The Systems Engineering course at Blekinge Institute of Technology is designed for university students at Master level. It addresses both SE theories and SE practical skills needed for practical application of SE methods and tools in practice. The course accounts for 7.5 ECTS points in the European Credit Transfer Systems. The European Credit Transfer Systems is based on learning achievements and students’ workload, with 60 ECTS points corresponding to a full academic year with a workload range from 1500 to 1800 hours. Every course accounts for a specific number of ECTS points, defining the expected amount of workload for the students, subsequently guiding the definition of the learning objectives and the duration of the course. The intended learning outcomes of the course are listed here:

• Understand what SE is and what characteristics sets it apart from other development approaches;
• Assess why and when an SE approach is desirable;
• Be able to undertake an SE program definition study and define an SE plan for a realistic project;
• Be able to develop an SE concept design as the basis for further detailed design;
• Apply SE tools (e.g., requirements development and management, robust design, design structure matrix) to realistic problems;
• Know how to proactively design for system lifecycle targets.

The course is structured in two internal modules which run in April and May. In the first module, time is spent in classical frontal lectures, where the basic concepts of SE are introduced. As a conclusion to the first module, a class activity is organized in two sessions. This aims to let students experience and test SE methods prior to their extensive application in the second module. In the second module, approximately 90 hours are spent on a course project dealing with an SE problem provided by a partner company. Additional project-related lectures and guest lectures by industrial representatives are eventually run based on the topic of the project. The reverse engineering role-play presented in the paper was run in between the two modules as a class activity to let the students experience the application of SE methods and learn and reflect on their characteristics, quality, and need for standardization. Figure 1 visualizes the high-level structure of the course.

3. Framing the Proposed Approach into the Systems Engineering Education Challenges and the CDIO Initiative

The academic discussion about the role of SE education goes back to the 1980s. Most of the early contributions focused on specific contents to grant an education that was in line with the professional needs of industries interested in employing the newly graduated students [12]. Asbjornsen and Hamann [12], in their work toward the definition of a unified SE education, listed a few requirements in relation to the SE process outlined by Asbjornsen [13]. According to those requirements, future systems engineers would need to understand and analyze a broad and multidisciplinary knowledge-base and, at the same time, would need the skills to work with deep quantitative knowledge in specialized areas. The skills of a systems engineer were also defined as strongly related to the ability to understand other disciplines and to explain one’s own. This is related to the ability to learn, share, and communicate new knowledge, ultimately lifting the need for systems engineers to be able to understand and analyze the human interrelations inside an organization (as also partially described by Senge [14] concerning practices in learning organizations). Asbjornsen and Hamann [12] also highlighted the need for “softer” skills, such as loyalty, individual responsibility, and global and environmental concerns. Their work, however, did not prescribe specific learning activities to address such multidisciplinary, neither indicated effective ways to teach SE to university students. A more recent work by Muller and Bonnema [9] focused on finding a teaching method fitting the limited students’ experience in the SE context. The authors investigated how to effectively teach SE and listed the differences between teaching SE compared to the other engineering disciplines. Specifically, five characteristics were described as critical:

• The broader scope, encompassing technical problems and many nontechnical issues at the same time;
• The ill-defined nature of the problems;
• The presence of unknowns, uncertainties, contradictions, and ambiguities in the problem definition;
• The presence of problems and solutions that are so heterogeneous that methods, techniques, and formalisms often have to be adapted to the situation;
• The absence of a single unique “best” answer, given the many-dimensional field of stakeholders and concerns.

The effectiveness of teaching engineering disciplines is a topic that goes beyond the focus of SE and concerns both the structure of the learning activities in the class and the learning styles of the students. About the latter, different models have been proposed in the literature (e.g., Myers–Briggs type indicator [15], Kolb learning style model [16], Felder–Silverman learning style [17]). Felder [18] listed a set of recommendations for engineering education to address the larger possible set of students learning styles, underlying, among others, the benefits of the following approaches:

• Teach theoretical material by first presenting phenomena and problem that related to the theory;
• Balance conceptual information with concrete information;
• Use physical analogies and demonstrations;
• Give occasionally experimental observations before presenting general principles;
• Provide class time to think about the material presented.

A parallel stream of literature focused on the analysis of the effect of active learning initiatives on students’ motivation and engagement in their own learning. In the early 1990s, a book by Bonwell and Eison examined the major contributions on active learning, concluding that its application improved students’ attitude as well as their thinking and writing ability [10]. Further, the review by Prince [19] highlighted the extensive empirical support that active learning has collected during the years and showed the induced benefits concerning the students’ improved capability to remember the topic of a lecture.

The CDIO international educational framework defines engineering education as a combination of conceive, design, implement, and operate activities [11]. Learning activities combining active learning, direct participation of students in concrete design tasks, and the participation to workshops and project-oriented group tasks constitute the backbone of the CDIO philosophy. Since its formulation, more than 150 academic institutions have joined the CDIO initiative [20] with the vision that “engineering graduates should be able to: Conceive—Design—Implement—Operate complex value-added engineering systems in a modern team-based engineering environment to create systems and products” [21].

The role-play is a form of active learning where students work through a given scenario adopting different personas and interacting in their assumed role [22]. Literature shows role-play to be an approach that is particularly effective for learning about complex systems and for maintaining students’ engagement while avoiding anxiety and increasing independent learning [22]. Research studies interviewing students several months after the closure of a course show that students tend to remember more information from the role-plays than from other lectures [23]. The role-play was also described as a valuable approach to enhance knowledge acquisition, particularly when it includes the observation and the acquisition of information from others [24]. The potential of role-plays in higher education was also explored by Rao and Stupans [25], proposing a typology of role-play learning designs enabling teachers to select the role-play approach that would best fit the intended learning outcomes.

The reverse engineering role-play presented in this paper is an instantiation of the effort to reform the learning activities of the SE course in the frame of the CDIO initiative. It uses active learning and self-reflection reports to increase students’ engagement and understanding of the topic and, at the same time, integrates Felder’s recommendations on addressing different students’ learning styles.

4. Description of the Learning Activity

The reverse engineering role-play was designed to address the intended learning outcomes number three and five of the course (see Section 2.1). In particular, the activity focused on providing a realistic project in which students can apply SE methods and subsequently reflect on their benefits, drawbacks, and challenges. The proposed learning activity aimed to foster students’ understanding of the importance of effective information communication by means of established methods in SE decision making. To achieve this goal, the role-play was organized as a two-day activity in which the students acted initially as members of an engineering team dealing with the reverse engineering of a product (stage 1), assuming later (stage 2) the role of members of a product innovation team in charge of the redesign of the available product. The two stages were followed by an individual activity, in which each student was asked to take the role of a manager to select, among the set of proposed innovations, the most relevant design to be further developed. This was done to replicate a cross-functional development scenario in which designers and decision makers are two different groups of stakeholders.

The role-play took place one month after the beginning of the course, when the students had already acquired knowledge about SE methods and tools (e.g., quality functional deployment, functional analysis, functional decomposition, requirement definition, N² diagrams) and had previously seen examples of reverse engineering activities. The following subsections describe in detail each stage of the reverse-engineering role-play.

4.1. Stage 1: Reverse Engineering and Knowledge Formalization

The activities of stage 1 were introduced with a 15-min presentation describing the reference products. The students were randomly divided into teams of 3 (and occasionally 4) members. Two products were proposed as subjects of reverse engineering, namely an electrically-powered coffee machine and an electrically-powered orange juicer (Figure 2). Each team received either the coffee machine or the orange juicer, the necessary instruments to disassemble and test the product, and the original packaging used for transportation. The choice of rather “simple” products, such as a coffee machine and an orange juicer, was driven by the necessity to give the students a product they could easily relate to and that was possible to analyze in a restrict timeframe. In total, 15 student teams took part in stage 1, six of which worked on the reverse engineering of the coffee machine and nine worked on the reverse engineering of the orange juicer.

The reverse engineering kicked off with the teams receiving the physical product, with the assignment to deliver its functional decomposition and its functional analysis, including the functional flow block diagram and an N² diagram [3]. Those models needed to be delivered following the shared SE standards (previously taught in the course lectures), and they needed to be uploaded on a common databased created for the activity. The activity started at 10:00 a.m., and no specific time limit was assigned. However, the teams were asked to gather again at 11:45 a.m. to verify the status of advancement of the work and were requested to upload the results on the database before the end of the day. Figure 3 illustrates the activities and the material produced in stage 1.

4.2. Stage 2: Propose Product Innovations

Stage 2 featured the same time setting and the same team composition of stage 1. It started with a brief introduction of the assignment of the day given by the teacher. Stage 2 took place on a different day than stage 1; this was decided to avoid stressing the delivery of the models during stage 1. In stage 2, the teams changed their role, not dealing with reverse engineering any longer but rather focusing on proposing product innovations. The peculiarity of stage 2 stemmed from teams switching the reference product of their work, that is, students’ teams that performed the reverse engineering of the coffee machine were asked to propose a new design of the orange juicer, while teams previously working on the orange juicer were asked to propose a new design of the coffee machine. To fill the knowledge gap given by the product switch, the teams were given access to the database with all the models produced during stage 1 by their classmates. This setting was proposed to replicate a typical situation in SE projects, in which different design teams take care of different stages of the development and need to rely on information that is produced and shared by other teams.

After accessing the database with SE models, the teams received a list of needs and problems about the coffee machine and the orange juicer. This was artificially developed by the teacher for the purpose of the activity and encompassed a broad set of stakeholders, including customers, logistics, production, and marketing. The list of needs and problems for both products is available in Appendix A and Appendix B, showing, in the first column, which stakeholder expressed a specific need and, in the second column, the formulation of the need or problem. In total, eight statements for the coffee machine and twelve for the orange juicer were provided. No further categorization or hierarchical analysis of the statements were provided to the students. The teams were asked to produce and document a new design based on the knowledge base. The requirements for the activity were to deliver a sketch of the new product, with related textual explanation, and to fill in a self-assessment report template (available in Appendix C). The structure of the self-assessment report was similar to the previous list of needs and problems, although it added three columns, in which the teams were asked to evaluate: How much the new design fulfilled the provided needs in relation to the actual product (column 3), how confident the team was in the assessment for each need/problem (column 4), and how experienced the designers were in using the coffee machine or the orange juicer (column 5). The report was complemented by a visual indication on how to self-assess the new design based on a scale from 1 to 9, using the original product as a baseline. Furthermore, a table mimicking the knowledge maturity scale proposed in SE literature by Johansson et al. [26] was provided to guide the assessment of the confidence of the teams in column 4. After completing the tasks, all the documentation was uploaded to the shared database to enable the evaluation of the decision maker. Figure 4 illustrates the activities and the material produced in day 2.

4.3. Stage 3: Decision Making and Self-Reflection

In the last part of the role-play, the students were asked to act as individual decision makers going through the available documentation and selecting the most promising design of coffee maker and orange juicer to be further developed. The students were asked to individually select one design for each product and justify their choice in written text. Furthermore, the students were also asked to self-reflect on the models that the teams produced during the activity, discussing if and how those were useful in the decision-making activity. The following questions where provided as a guideline for the students’ reflections:

• Was the information enough to be confident with your choice? If not, what kind of information would you have asked the design teams? And, eventually, in which form would you have such information communicated to you?
• Have you found one or more functional flow block diagrams more useful than others? If yes, why? If no, what would you expect to learn more from such diagrams? If you would do your diagram again, would you do it differently?
• Have you found one or more N² matrices more useful than others? If yes, why? If no, what would you expect to learn more from such diagrams? If you would do your N² matrix again, would you do it differently?
• Have you found one or more functional decompositions more useful than others? If yes, why? If no, what would you expect to learn more from such diagrams? If you would do your functional decomposition again, would you do it differently?

The reflections were formalized in a text document and submitted to the teacher as part of the course examination valuable for the final grade.

5. Data Analysis and Results

The documentation generated from stage 3 constituted the set of raw data that was analyzed to verify the effectiveness of the proposed learning activity. The reports were analyzed looking for answers to five questions, namely:

• What were the main reasons to choose a design?
• Was the information enough to be confident in your decision?
• What kind of information would you have asked the design teams?
• In which form would you have preferred to have such information communicated to you?
• If you would do your models again, what would you do differently?

Students were free to provide open answers, indicating one or more aspects that they would have liked to improve or that influenced their decision; thus, the analysis of the data accounted for possible multiple answers. The following subsections describe the findings for each question.

5.1. Reason to Choose a Design and Confidence in the Choice

All 46 students provided the reasons to choose a particular design. Figure 5 shows the distribution of answers to question 1.

The data show two dominant reasons to select a proposed design, namely the “personal feeling” of the decision maker and the perception that the new design is satisfying the needs of different stakeholders. Even if dominant in relation to the other response, those two reasons were mentioned only by 19 of the 46 students (corresponding to the 41% of the sample). Despite the satisfaction of the needs being central, only 12 of the 46 students admitted having considered the teams’ self-assessment report concerning needs satisfaction in their choice, while 10 out of 46 students were influenced in their choice by the accuracy of the report submitted. Such data are interesting to be analyzed in relation to the students’ confidence in making decisions. From the 40 students that explicitly discussed their confidence, 27 (corresponding to the 67.5%) stated that there was not enough information to be confident in their decision, although they were required to make it anyway. These findings are particularly relevant in the frame of the different roles that the students took in the role-play. This highlights the challenges that decision makers in an SE context face when asked to make decisions with product and systems information poorly defined and formalized. In other words, 27 out of 40 students realized that, to make a confident decision, the SE models produced by the class needed to have a higher level of detail and precision. Such data suggested the activity was effective in allowing students to experience the typical dynamics of decision making in an SE context.

5.2. Type of Information to Be Further Asked to the Design Team

The students were further asked to state what kind of additional information they would have asked the design team to increase their confidence in the decision. Forty-one students addressed this question in their report, and the distribution of answers is shown in Figure 6. Out of 41 students, 20 recognized the need for enhanced visual representation of the designs. Sixteen students (i.e., 39% of the respondents) also highlighted the need for decision makers to receive more accurate information about the product parts, and 10 students expressed the desire to have a description of the product working process. Eight students requested an explanation of the rationale for the design choices, and seven would have preferred a more detailed analysis of the stakeholders’ needs. Of particular interest in relation to the intended learning outcomes is the students’ awareness of the necessity of having more accurate information about the product parts and the product working process. Those two aspects are respectively addressed by the functional decomposition and the functional analysis, models previously experienced and produced by the students in stage 1. These data show that, despite experiencing the models in stage 1, many students failed to deliver appropriate SE models during stage 2. The recognition of such failure happened in stage 3, during the decision-making phase. From a pedagogical perspective, such a self-recognition process should be regarded as a positive result of the learning activity, in line with the desired intended learning outcomes.

Out of the 41 students addressing the need for more information, 28 detailed in which specific form they would have liked to receive it. No specific form emerged as dominant in the answers, with 9 students expressing the desire for an oral presentation, 7 wishing to access CAD models and physical prototypes, and few other students expressing the desire for more concise description, pictures, and diagrams. The absence of dominant results does not allow to draw any specific pedagogical reflection about this aspect.

5.3. What Would You Do Differently

The last question was the most relevant for what concerned the identification of the learning results from the reverse engineering role-play. In light of their experience as innovators in stage 2 and decision makers in stage 3, the students were asked to reflect on what they would do differently if they came back to stage 1 and redid the functional models and the functional decomposition again. The question was of primary importance to understand if the learning activity contributed to the objective of raising students’ awareness about the importance of using defined and standardized SE methods for knowledge communication in cross-functional teams. Figure 7 shows the distribution of the answers given by the 32 students that explicitly addressed this question in their reports.

The majority of the students (24 over 32, equal to 75%) stated that they would make a bigger effort in adding details to the SE models produced in stage 1. Eleven students (35% of the total) also raised the necessity for the new models to be more readable than the previous ones, also adding pictures to facilitate the communication (8 students). In light of such data, it is possible to state that the students developed a critical perspective toward their own work, reflecting on the necessity to use SE methods in a way that allows easier knowledge and information communication at different functional levels and for different uses. This is shown by the shared understanding that the level of detail of the models that the teams delivered during stage 1 did not reflect the level of detail that was later needed by the other teams when acting as product innovators and decision makers. Furthermore, the recognition of the need for more readable SE models can reflect the understanding of the multidisciplinary application of such methods in an SE context. From an overall perspective, the finding from the data analysis described a shared understanding about the necessity to use well-formalized, detailed, and effectively communicated methods during SE, suggesting that the learning activity was effective in avoiding the risk of the SE method being considered as a triviality by students lacking contextual SE knowledge (a concern also expressed by Muller and Bonnema [9]).

6. Concluding Remarks

The evolving needs of society and industry have constantly changed the way in which engineering education is performed. Engineering education continuously strives to educate students about the skills and abilities needed to efficiently and effectively create value in society. Among the different areas of improvement, this paper has focused on a specific aspect, that is, the capability of engineers to work in an SE context.

This paper presented a learning activity for SE education that addresses parts of the challenges raised in the educational literature described in Section 3. The paper focused on the necessity of conveying the relevance of SE methods to students that struggle to perceive the usefulness of such models. The analysis of the data collected from the students’ self-reflections showed an increased awareness of the students on the necessity of extensively reviewing their initial use of SE methods. The addition of details and the improvement of readability emerged as key actions to provide better-formulated information to decision makers, ultimately enabling more confident design decisions. This is interpreted as a positive result in relation to the satisfaction of the course-intended learning outcomes. Concerning the limitation of the research, it must be highlighted that the data presented were collected from a sample of 46 students, and this can limit the generalizability of the findings. Engineering literature describes several examples of experiments with university students featuring an even smaller number of participants (see References [27,28]), although the verification of the effectiveness of the described educational approach would benefit from a larger data sample, also including students from different institutions and cultures.

The reverse engineering activity was based on commercial products not typically developed through an SE process. This allowed the students to model the behavior of a product in a usage context that was familiar to them. The benefit of this was limiting the risk of including too many assumptions about the functional decomposition and the functional analysis of the products. The adoption of the new learning activity in the course was framed in the university effort to move toward the transformation objectives highlighted by the CDIO framework. In line with the active learning principle, the role-play was designed around an important learning outcome of the course (i.e., the capability of applying SE tools to realistic problems) and was designed to promote the engagement of the students in the course. In conclusion, the proposed learning activity was shown to promote students’ critical thinking and to increase students’ self-confidence and understanding of the usefulness of SE methods, providing the students with an answer to the question: “Why should I care about learning this?”

Future evolution of the proposed learning activity concerns the exploration of the possibility to run a larger-scale reverse engineering role-play involving SE students of multiple academic institutions, in order to replicate the challenges given by distributed and globally located design teams.