Learning MBSE Online: A Tale of Two Professional Cohorts

Abstract

Research has shown that creating an online learning community is vital in Model-Based Systems Engineering (MBSE) training programs and can be facilitated via the Community of Inquiry (CoI) framework. For professional learners, an online learning community is influenced by their organizational affiliations. The purpose of this research is to explore learning experiences in groups of professional learners with different and homogenous organizational affiliations in an asynchronous online MBSE module. Through the case study methodology, this research examines four sources of data from two cases: Case 1—learners from different organizations (n = 7); and Case 2—overwhelming majority of learners from the same organization (n = 19). Results showed that learners from the same organization reported higher social presence, which, in turn, corresponded to a higher cognitive presence and higher motivation for future MBSE learning. Based on our findings, we recommend that organizations seeking MBSE adoption coordinate with online course providers to create cohorts to participate in the same offerings to facilitate the process of learning community building. We also recommend MBSE course providers facilitate social interaction on multiple communication platforms and create orientation activities for learners from different organizations to promote social presence.

1. Introduction

To address the increasing demand for Model-Based Systems Engineering (MBSE) adoption in the workplace and its associated challenges, more professional development programs are being created. Numerous studies have examined what prevents professionals and organizations from adopting MBSE [1,2,3], with many of the identified obstacles being related to overcoming the fear of change and reshaping old habits [1]. Similarly, Vogelsang and colleagues [3] summarized two overarching forces that drive industry away from MBSE: inertia, which involves the personal, organizational, methodological habits of maintaining the traditional SE approach; and anxiety related to the stress of dealing with new technologies and tools. Apart from cultural resistance, clearly defining the scope and purpose of MBSE adoption prior to deployment is also challenging as companies of various sizes and goals may have different definitions of MBSE adoption and what it might mean within their own contexts [4]. Thus, responsive MBSE professional development programs address those challenges by promoting awareness of the benefits of MBSE; this awareness provides the transferable technical knowledge to promote MBSE adoption in learners’ organizations and motivate participants to keep engaging with advances in MBSE practice in the future.

Creating a learning community in MBSE professional development programs can help reach MBSE learning goals. A learning community refers to a group of learners with shared learning purposes who build relationships with each other while engaging in collaboration [5]. Research has shown that a higher sense of community can enhance learning while also speeding up learners’ knowledge application process and enhancing their satisfaction [6,7]. This fact is especially important as most MBSE professional development programs are organized online to accommodate learners’ needs [8]; creating learning communities online can be hard due to increased distance and the asynchronous nature of online communication [9].

The Community of Inquiry (CoI) framework is a useful tool to facilitate in the formation of online learning communities. The CoI framework is a creative procedure widely adopted by educators and course designers that has the goal of delivering meaningful learning experiences for online and blended courses [10]. CoI emphasizes meaningful learning experiences resulting from the interaction between three distinct kinds of presence: social, teaching, and cognitive presence [11]. Social presence is the degree that online learners can project themselves as a “real” person, identify themselves with the learning group, and develop meaningful personal relationships [10,12,13]. Teaching presence is relevant to the instructor—the expert—who designs and delivers the course. It involves the course design, such as the learning materials and organization of the course, as well as the course delivery, such as providing feedback for assignments and facilitating discussions. Cognitive presence refers to how learners inside a given online community can construct knowledge through interactions with peers and instructors [14]. The causal relationships among these three kinds of presence have been examined in various learning contexts. Social presence is positively correlated with learners’ satisfaction and learning outcomes, mainly in the context of online credit courses in higher education [15,16,17]. Researchers suggest group discussions, collaborative activities, real-time communication activities, and/or platforms that can enhance social presence and further improve student cooperation and learning experiences [18].

It is encouraging to know that building a community of inquiry within a virtual learning context is possible through enhancing social presence with specific instructional strategies. However, the discussion is mainly centered around credit courses in higher education in which the learners are mostly degree-seekers who are intrinsically motivated to be part of the learning community. What needs to be added to this discussion is to what extent we can apply this assumption in the context of professional development of online learning opportunities in which the learners (professionals) might be unable to communicate with their peers to form a learning community freely.

Research has shown that the culture and work environment of professional learners will affect their virtual learning environments, which, in turn, will influence learning outcomes [19]. More specifically, past literature indicates that differences in organizational cultures may create barriers to forming learning communities for numerous reasons [20]. For example, organizational membership may result in clustered or scattered geographical locations of learners and will influence community building due to potential misinterpretation of online communication [21]. Varying organizational cultures will also affect learners’ attitudes towards sharing and communicating work experiences, especially when it comes to proprietary information [22,23]. Trust issues may also play a factor as it is most easy to gain trust in a face-to-face setting or among people who already share common identities, such as their professional affiliations, as the elevated disclosure of personal information can facilitate trust-building [24,25,26].

Current literature about professional learners’ online educational experiences primarily focuses on aspects such as self-efficacy, learning styles, and familiarity with technologies, etc., or innovative educational interventions [27,28,29,30]. While there have been studies dedicated to professional development programs customized for certain companies [31], little attention has been given to programs that are open to learners from multiple organizations. Studying how professional learners’ organizational affiliations can impact their learning experiences is essential as it will generate insights on how to better support professional learners within various organizations. More particularly for MBSE education, it will help in overcoming cultural resistance to MBSE adoption.

This research aims to examine professional engineering learners’ experiences in an asynchronous online MBSE module by investigating their participation within the module (e.g., discussions and group project), learning outcomes, and perceptions of communities of inquiry within the module. To explore how professional learners’ organizational affiliations may influence their learning experiences, we use the case study methodology [32] to examine four sources of data from two cases: Case 1—learners from different organizations (n = 7); and Case 2—learners from the same organization, (n = 19). We examine data collected on learner interaction, learner motivation, the three constructs (social presence, cognitive presence, and teaching presence) of CoI framework, and MBSE awareness to understand learner experiences from multiple perspectives. The overarching research question is: How do learning experiences compare for a group of learners from the same company and a group of learners from different companies? From this overarching research question, we developed sub-research questions that correspond with the different data sources collected in this study. The sub-research questions are:

• How does social presence differ between these two groups of learners?
• How do different levels of social presence correspond to learning outcomes?

Based on the literature about online learning communities and MBSE adoption, we hypothesize that learners who have similar organizational affiliations will have the following traits: (1) higher social presence, (2) higher cognitive presence, and (3) higher motivation to engage in further MBSE learning.

2. Materials and Methods

2.1. Research Method

We used a case study methodology [32] to compare two offerings of an MBSE professional development online module. More specifically, we conducted a case comparison study to explore how learning experiences differ between professional learners who were from the same company and those who were from different companies when enrolled in an MBSE online module. The case study was used when the aim of the research was to examine and expand the experiences to develop tacit, instead of propositional, knowledge [33]. Our choice of research method allowed us to examine multiple sources of data regarding two offerings of an MBSE online module in a holistic and in-depth manner [32], especially when sample size was small. It also allowed us to triangulate these different data sources to test our hypothesis between the different case contexts and learner experiences.

2.2. Research Context & Case Description

Over the past few years, our team developed seven online modules with the aim of teaching MBSE to professional and graduate learners. As documented in past publications [34], the main goal of the modules was to apply social constructivism, the CoI framework, and experiential learning theories in the module design. In this study, we compared two cohorts of professional learners who took the first of our modules (titled Introduction to Systems Engineering and MBSE for Production Systems) as part of a non-credit professional development program. These two cohorts will be treated as separate cases in this study.

The goal of this first module was to introduce learners to basic topics of systems thinking, systems engineering, MBSE, and how these approaches could be applied in real-world scenarios. The overarching learning outcomes for the module were defined as: (1) “Summarize the basic elements and terminology of systems engineering”; (2) “Summarize the basic elements and terminology of MBSE”; and (3) “Apply systems engineering frameworks to understand complex engineering problems”. In terms of content organization, the module contained videos where instructors presented the technical content in a short format (<25 min); discussion prompts, where learners were encouraged to post about how concepts in the class align with their professional lives and incentivized to reply to others; case studies, where learners were expected to go through selected readings and videos analyzing three engineering projects from the perspective of systems engineering and MBSE; a group project, where learners had the opportunity to reflect on a system of their choice; and a quiz, where learners were expected to respond to technical questions about the module’s content.

Learners were given a total of 3 weeks to finish the module at their own pace. However, we provided learners with a suggested timeline to assist them in progressing through the module. The timeline and content summary of the required learning activities are shown in

Table 1. Suggested timeline for module completion and associated learning activities.

	Week 1	Week 2	Week 3
Videos	Motivation for systems engineering Systems concepts	Systems thinking Models Systems engineering	How to practice systems engineering
Discussion posts	Introductions Definition of system and complex system	Complex problem in the work environment and how DSRP framework could have helped	SE and MBSE at the learners’ organizations
Case studies	Read the three case studies Watch the videos about the case studies
Group project	Defining and identifying system of interest	Reflecting about the use of models in the system	Systems engineering analysis Systems engineering reflection
Quiz			Testing technical concepts

. We used CoI framework to guide our module design. More specifically, we implemented discussion prompts and a group project to promote social presence and encouraged learners to collaboratively construct meanings for the knowledge learned. Instructor presence was embedded in the lecture videos and instructor feedback to learners. We promoted cognitive presence by incorporating case studies as real-world examples of MBSE adoption in various industries, including aerospace, automobile, and electronics industries. The discussion prompts and group project also provided opportunities to connect the MBSE concepts with learners’ personal and professional experiences. Cognitive presence was also manifested in the module quiz. The quiz consisted of 10 questions that were mostly short-response or multiple-choice questions designed to address the granular learning objectives that we created for each of the topics in the module.

As the main assignment for the module, the group project was a thorough demonstration of our intention to intersect all three CoI presences to create meaningful learning experiences. Learners were required to analyze a system that they either knew about in real life or was present in one of the case studies. The analysis included five steps: (1) summary of the situation surrounding the system—which led to its success or failure; (2) system identification—defining the system, identifying external systems, structure, behavior, etc.; (3) usage of models—listing the models and reasons for their inclusions; (4) systems engineering analysis—stakeholder needs, involved engineering disciplines, motivation for SE; and (5) systems thinking reflection—where group members had to reflect on how they applied systems engineering frameworks and handled different engineering perspectives. Learners were required to communicate with their group colleagues on the discussion board when working on the group project. The final deliverable was a written report containing all the required steps of system analysis.

As stated previously, this study is a case comparison study, comparing the learning experiences between two groups of learners. Case 1 consisted of the cohort that took the module in 2021 and had a total of seven learners from two different companies. Due to the small cohort size, learners from Case 1 were all required to work as a single group—meaning that people from different companies were required to interact with each other to complete the activities in the module.

Case 2, on the other hand, consisted of a cohort of 19 learners who took the module in 2022. In Case 2, the overwhelming majority of learners (>90%) were from the same company. Due to the high number of learners, they were assigned to work in separate groups for the module activities, resulting in a total of five groups. Therefore, most learners were not required to interact with people from a different company in Case 2. Learners from Case 2 were also asked about their modes of communication during the module. Most learners from Case 2 reported that they used external modes of communication to discuss module-related activities, such as emails and virtual meeting tools, in addition to the discussion board.

Additionally, when asked to rate their familiarity on a scale of 1 to 6, both Case 1 learners and Case 2 learners reported higher levels of familiarity with SE (Case 1: M = 3.29, SD = 1.50; Case 2: M = 3.05, SD = 1.08) than MBSE (Case 1: M = 1.29, SD = 0.49; Case 2: M = 2.31, SD = 1.06).

We made minimal changes to the module between the two cases presented in this study. The changes were only made to the wording on the group assignment to ease understanding of the instructions. In both cases, the same instructors were responsible for delivering the module.

Table 2. Leaner demographics from 2021 and 2022 cohort.

	Case 1 (n = 7)	Case 2 (n = 19)
Gender
Man	57%	63%
Woman	29%	26%
Other/Prefer not to answer	14%	11%
Race/Ethnicity
Asian/Pacific Islander	14%	11%
Black or African American	14%	--
White	57%	68%
Other	14%	21%
Age range
20–30	29%	53%
30–40	29%	26%
40–50	14%	--
50–60	29%	21%
Time related to module
Allowed to use working hours for module	14%	11%
Allowed to use working hours for module, but also used non-working hours.	63%	11%
Expected to use non-working hours for module.	14%	79%

displays learner demographic information collected at the beginning of the module for both cases.

2.3. Data Collection & Pre-Processing

According to Yin [34], case study research design and data collection should focus on collecting multiple sources of evidence that allow for triangulation of the data sources and research questions. In other words, researchers should collect data from more than one source to establish converging lines of inquiry. In this case comparison study, we collected data from four aspects. Firstly, we sought to understand how learners perceived the learning tasks and content at the beginning of the module. Secondly, to understand the social interactions between learners, we collected the discussion posts that learners created throughout the module. Thirdly, we assessed the CoI presences as perceived by learners throughout the module. Finally, we looked at the awareness, and motivation for future learning of the learners at the end of the module. An overview of the different data collected and how different types of data triangulate to address sub-research questions is provided in

Table 3. Data sources used in this case study.

Sub-Research Questions	Constructs	Data Sources	Data Analysis
How does social presence differ between these two groups of learners?	Learner interaction	Discussion posts	Social network analysis
	Social presence	CoI survey [35]	Descriptive statistics
How does different levels of social presence correspond to learning outcomes?	Cognitive presence	CoI survey	Descriptive statistics
	Teaching presence	CoI survey	Descriptive statistics
	MBSE awareness	Adapted Nanotechnology Awareness Instrument–Awareness question [36]	Thematic analysis
	Pre-module motivation	PE–EVC scale [37]	Descriptive statistics
	Post-module motivation	Adapted Nanotechnology Awareness Instrument– Motivation scale	Descriptive statistics

. These data sources allow us to trace learners’ experience throughout the module and provide insights on different perspectives of learning experiences.

According to CoI framework, one method to understand the social presence is to investigate learners’ interactions within discussion board forums. In this study, we collected the discussion posts throughout the module in both cases to capture learner interaction. Learners had three discussion prompts to which they could respond throughout the module. In our suggested timeline offered at the start of the module (Table 1), we recommended learners create a post each week and check the discussion board regularly to interact with others. In addition, each learner group had their own group discussion space, where they were required to engage in discussions about how to complete the group assignment. Both the class discussion posts and the group discussion posts were collected for social network analysis to illustrate learners’ interactions (e.g., frequency and connections) within each discussion forum and how they evolved over time. We anonymized all student names before conducting the analysis and reporting the results.

To study learning outcomes, we collected data on students’ perceived cognitive and teaching presence, their learning motivation before and after the module, and their awareness of MBSE after completing the module. In total, we administered two surveys throughout the duration of the module: a pre-module survey that learners were reminded to fill out at the end of Week 1 and a post-module survey after completing the module. The surveys mainly consisted of Likert-type questions and one free-response, short-answer question (in the post-module survey). Participation in the surveys was not mandatory.

As the main goals for MBSE professional development programs were associated with learner motivation and attitudes towards MBSE adoption, we examined learners’ attitudes towards learning before and after the module. In the pre-module survey, we used the Professional Engineer Expectancy–Value–Cost (PE-EVC) Scale to measure learners’ perceived learning goal and their self-estimated ability to reach that goal [38]. This survey used the Expectancy–Value–Cost theory to evaluate learner motivation in three dimensions specifically for online learning, such as Massively Open Online Courses (MOOCs). While the module analyzed in this study is of a smaller scale than MOOCs, the survey still aims to measure learner motivation while balancing multiple factors and perceived costs that were likely to affect learning. Expectancy measured learners’ belief in their ability to master the content. An example item for expectancy was “I can understand the material in this module”. Value referred to how much importance learners assign to the learning tasks. For example, value items asked learners to rate their agreement with statements such as “I think this module is useful to me”. Lastly, cost evaluated the tangible costs learners have to invest to achieve their learning goal. It included statements such as “This module requires too much time”.

In the post-module survey, learning outcomes were assessed via multiple constructs, including MBSE awareness and motivation for future learning; both constructs were assessed through selected items from the Nanotechnology Awareness Instrument. Originally designed for the field of nanotechnology, the Nanotechnology Awareness Instrument has shown evidence of good validity when adapted to measure learner motivation in other new technical fields experiencing growing demand [39]. Similarly, we adapted the survey questions to gauge learner motivation for future MBSE learning. We asked about their interest levels in participating in additional MBSE learning through various means, such as reading more about MBSE, attending other professional development offerings (i.e., conferences and seminars), etc. We also adapted a free-response question from the Nanotechnology Awareness Instrument to measure learners’ awareness of MBSE as increased awareness will produce a positive feedback loop to motivation and, in the case of MBSE, promote MBSE adoption in the industry. In the adapted MBSE awareness question, we asked learners to provide an example of how MBSE could directly affect their jobs.

Lastly, we measured learners’ perceived level of teaching, social, and cognitive presence in the module—assessed with the Community of Inquiry survey. Since the module was designed using the CoI framework, measuring learners’ perceived level of these CoI presences can provide insights on the module’s effectiveness. The CoI survey is commonly used in literature to evaluate learning environment and instructional design for online or blended learning. It can also help researchers examine relationships between the different CoI elements and other data sources [40]. We used the CoI survey data to examine learners’ perceived sense of community (i.e., social presence) during our module and explore whether coming from one or various companies may affect their learning experiences (i.e., cognitive and teaching presence).

Before data analysis, minor data pre-processing was necessary. We removed duplicates in pre-module survey responses from Case 2. Although we provided a module timeline documenting due dates for the surveys, we included survey responses submitted within 3 weeks of the module’s end date because of the self-paced structure of our module and the fact that learners may need to complete the learning activities outside of working hours (as indicated in the pre-module survey response shown in Table 2). Lastly, responses with a completion rate of 50% or less were eliminated. After these pre-processing steps, Case 1 had a total of seven pre-module survey responses and five post-module survey responses, whereas Case 2 had 19 pre-module survey responses and eight post-module survey responses.

2.4. Data Analysis

In this study, we used SNA to examine learner–learner interaction within the weekly discussion forums related to the weekly learning content and the group discussion forums created for the leaners to collaborate on their group projects. Social network analysis (SNA) views “individuals or groups as ‘points’ and their relations to each other as ’lines’. It is concerned with the patterns formed through the points and lines and involves exploring these patterns, mathematically or visually” [41] (p. 1). It has been increasingly applied in higher education to explore student engagement in a variety of contexts (e.g., online learning discussion forums). Apart from representing the extent of social interaction mathematically and using visual network maps, SNA also provides useful information about the types of roles an individual takes on while engaging with others in the learning environment. An individual can either be a point within a cluster or a connector (also known as a bridge) linking two clusters together [42]. Identifying connectors within an SNA map is important as past research indicated that connectors are usually the first aspects to receive new information and may be responsible for spreading new ideas or behaviors. We conducted social network analysis and generated the associated graphs using the igraph package in R.

To analyze the Likert-type questions in the surveys, as our intention is to calculate a composite score of multiple items belonging to the same construct, we followed the suggestions of the original survey designers and best practices suggested for similar questions [37,43]. We assigned a numerical value to each scale. A score for each construct was calculated by summing up the scores of items within that construct and dividing the sum by the number of items included. As a result, each survey response has scores for expectancy, value, cost, motivation for future learning, perceived teaching presence, social presence, and cognitive presence. Except for expectancy, value, and cost scores, which have a possible range of 1 to 6, all of the other scores have a possible range of 1 to 5.

To analyze the free-response question included in post-module survey, we used thematic analysis to discover emerging themes using the inductive coding method [44]. We followed this approach as thematic analysis is a useful research method to analyze textual data sources, such as short-answer survey responses [45]. Additionally, it is a suitable method to approach the data in an exploratory and open-ended way, i.e., without pre-established coding frameworks or schemes. In thematic analysis, researchers go through the data to find emerging themes to create “subjective and cultural/contextual message of the data” [46] (p. 2). Two researchers were involved in this process to ensure the validity of the theme-identifying process. The researchers first reviewed the data and identified codes and emerging themes from the responses individually. They then compared and discussed their codes to create a code book. Using this code book, they repeated the coding process individually and compared each other’s codes until consensus was reached.

3. Results

In the following sections, we present results for the various data sources collected. To understand learning experiences, we operationalize this broad concept as the extent of learner interactions, motivation before and after the module, perceived CoI presences, and awareness of MBSE after completing the module. We will first present SNA results (both in descriptive statistics for discussion posts and SNA maps) to summarize learner interactions. We will then present survey results on the pre- and post-module surveys to measure motivation. Similarly, the results of the CoI survey are presented to reflect learners’ perception of their educational experiences in terms of social, cognitive, and teaching presences. Lastly, thematic analysis results provide emerging contextual meanings within learners’ responses when asked to demonstrate MBSE awareness.

3.1. Social Network Analysis

Table 4. Weekly discussion forum posts for Case 1 and Case 2.

		Week 1		Week 2		Week 3
		Case 1	Case 2	Case 1	Case 2	Case 1	Case 2
Number of learners who posted	n	5	18	4	9	3	4
Number of discussion posts per student	Median	2	3	2.5	2	1	2
	Mean	2	3.89	2.5	2	1.2	2
	Standard deviation	1	5.16	1.29	1.23	0.58	0.82

contains the number of discussion posts and replies for the weekly class discussions, while

Table 5. Group discussion forum posts for Case 1 and Case 2.

		Case 1	Case 2
		Single Group	Group 1	Group 2	Group 3	Group 4	Group 5
Number of learners who posted	n	6	3	3	4	2	4
Number of discussion posts per student	Median	3	6	4	2.5	4	7
	Mean	3.33	8	4.67	4	4	7
	Standard deviation	2.42	4.36	3.05	4.08	1.41	4.16

covers the group discussion forums. The descriptive statistics in the tables were calculated considering the total number of people who posted to that board. In other words, these calculations do not account for everyone enrolled in the modules but only for the learners who engaged in the discussions. Figure 1 contains the social network analysis graphics that visualize class-level learner interactions for both cases on a weekly basis. The mathematical and visual representations of learner interaction provided information on the level of learner interaction in both cases and how their interaction changed over time. The results showed that learner engagement in class-level discussions decreased significantly after week 1 for both cases. However, one learner from Case 2 who initiated the interaction with their peers remained active and engaged in the discussions while most of the Case 2 learners stopped participating. To reiterate, the weekly class-level discussion posts were not a mandatory part of the module but were encouraged. Moreover, based on the SNA maps in Figure 1 we were able to identify a connector in Case 2—Eric—who acted as a bridge linking clusters of learners together. In Case 1, no connector was identified in the class-level discussions. As for mandatory group-level discussions, the results also showed that Case 1 had a lower group-level interaction than groups from Case 2, despite having only one group with a larger number of learners.

3.2. Learning Motivation

As mentioned previously, we asked about learners’ perceived expectancy, value, and cost of participating in MBSE learning activities at the beginning of the modules. Due to how cost is defined, a higher cost score is likely to decrease motivation [37], whereas expectancy and value scores have positive relationships with motivation.

Table 6. Learners’ perceived expectancy, value, and cost of MBSE learning activities.

Construct	Case 1 (n = 7)	Case 2 (n = 19)
Expectancy	5.38 (0.89) 1	5.26 (0.95)
Value	5.17 (0.78)	4.77 (0.88)
Cost	2.93 (0.67)	1.99 (0.71)

¹ Standard deviations are shown in parentheses.

below contains the comparison between the two cohorts. Responses are on a scale of 1 (strongly disagree) to 6 (strongly agree), while standard deviations are presented in parentheses. Looking at score standard deviations, Case 2 learners did not report drastically different scores from each other despite their larger group size. While both cases report similar average expectancy and value scores, learners in Case 2 generally attached less cost to the learning tasks involved in the module, even though most of them reported that they were expected to use time outside of work to complete the module (Table 2).

Looking at the adapted motivation scale results in the post-module survey, a trend opposite to the pre-module survey data can be observed between the two cases. Learners in Case 2 reported higher motivation to engage in future MBSE learning (M = 4.33, SD = 0.69) than Case 1 learners (M = 3.72, SD = 1.08) after completing the module. Learners’ motivation responses in the adapted Nanotechnology Awareness Instrument are scored on a scale of 1 (strongly disagree) to 5 (strongly agree).

3.3. CoI Survey

The post-module survey we used the CoI survey [35] to investigate learners’ perceptions of the teaching, cognitive, and social presences they experienced while taking the module.

Table 7. Teaching, cognitive, and social presences perceived by learners.

Construct	Sub-Construct	Definition 1	Case 1 (n = 5)	Case 2 (n = 8)
Teaching presence	Design & Organization	Structure of the online learning experience	4.40 (0.65) 2	4.46 (0.73)
	Facilitation	Guiding and promoting learner interactions	3.80 (0.74)	4.21 (0.94)
	Direct Instruction	Learner feedback and guidance	3.33 (0.49)	3.75 (0.83)
Social presence	Affective Expression	Confidence to express feelings related to learning experiences	2.93 (1.38)	3.79 (0.99)
	Open Communication	Mutual and respectful communication	3.53 (1.43)	4.13 (1.25)
	Group Cohesion	Learning commitment at the group level	3.67 (0.78)	4.04 (0.97)
Cognitive presence	Triggering Event	An event that leads learners to the process of critical thinking	3.93 (0.92)	4.13 (0.62)
	Exploration	Exploring knowledge that will help make sense of the triggering event	3.47 (0.99)	4.25 (0.53)
	Integration	Integrating the explored concepts into coherent knowledge	3.87 (1.15)	4,46 (0.62)
	Resolution	Application of the newly constructed knowledge	3.80 (0.87)	4.33 (0.64)

¹ Definition based on Garrison et al. [10]. ² Standard deviations are shown in parentheses.

below shows the scores for sub-constructs (definition based on Garrison et al. [10]) measured via the CoI survey for both cases. Responses are on a scale of 1 (strongly disagree) to 5 (strongly agree), while standard deviations are presented in parentheses. Case 2 learners reported consistently higher levels across all presences when compared to learners from Case 1. However, in both cases learners reported the lowest level of social presence among the three pillars of the CoI framework.

3.4. MBSE Awareness

We also used a free-response, short-answer question in the post-module survey to gauge learners’ MBSE awareness by asking learners to list an example of how MBSE can affect their jobs. We conducted thematic analysis to reveal emerging themes. Learners answered this question from two perspectives: the benefits of MBSE and how MBSE will be used in their job. These two overarching themes are not mutually exclusive; in other words, there were learners who included both themes in their responses.

For the first emerging theme—MBSE benefits—learners mainly demonstrated their awareness of how MBSE can benefit the manufacturing and production industry. Under this over-arching theme, learners mentioned how MBSE can help improve information sharing throughout a product’s lifecycle, enhance collaboration within teams, reveal the relationship/interactions among parts of a system, transition to a digital enterprise, and engage learners in systems thinking.

The other emerging theme is related to how learners will use or benefit directly from the module content. Curiously, no learners in Case 1 answered the question from this aspect. Learners in Case 2 mentioned several direct uses for MBSE knowledge acquired. Most of them responded to this question from the angle of meeting a demand. This demand included fulfilling customers’ needs and staying updated with the company’s trend of adopting a MBSE approach. Fewer learners approached the question from a growth perspective that referenced their personal development and their intention of teaching others about MBSE.

4. Discussion

4.1. Levels of Social Presence

4.1.1. Participation in Discussion Decreased over Time

It is noticeable from Figure 1 that learners’ participation in the weekly discussion forums decreased over time for both cases as the number of nodes in the graph decreased on a weekly basis. At the beginning of the module, most learners participated; in Case 1, six learners (75%) participated in Week 1′s discussion, whereas in Case 2 18 learners (95%) participated in Week 1’s discussion. These high levels of participation might have been observed because learners were asked to introduce themselves to each other as a part of the Week 1 discussion activity. The decrease in class-level discussion forum participation was larger in Case 2 than in Case 1. For example, in week 2 only nine learners participated in the discussion for Case 2, which was a 50% decrease; this rate only dropped by 33% in Case 1. The same pattern continued in Week 3’s discussion with an additional 44% decrease in Case 2 and 25% decrease in Case 1. Despite differences in scale, the decrease in participation within our module aligns with what Gillani and Eynon [47] observed in their study of MOOCs: “students started off with high-volume participation in online discussions, and over time, their apparent commitment to these conversations tailed off” (p. 23). Curiously, despite the larger decrease in participation among learners from the cohort in Case 2, these learners still reported higher levels of social presence in the CoI survey. Past research on online learner interaction indicated that learners’ dissatisfaction with discussion tools can result in decreased interaction as learners progress through the course and accumulate more negative experiences with the platform [48]. Decreased participation over time might also be partially due to the fact that learners from Case 2 utilized other modes of communication within their company, such as Slack or email. As the learners in Case 2 progressed through the module, they might have moved to other more convenient platforms to engage in discussions, especially when completing the group assignment.

4.1.2. Class-Level Discussion and Group Project Forum

In both cases, we also identified a couple of learners who participated in the class-level discussion constantly throughout the whole module. For example, one active participant in Case 2—Eric—created 21 posts in Week 1′s discussion when the median was only three posts that week. His high level of engagement was indicated via the larger node size in Figure 1 (Week 1). It is essential to point out that Eric only received three replies from his peers in Week 1. Nevertheless, he remained active during Weeks 2 and 3, having the most connections with his peers. When analyzing learners’ participation in the group project forum, we noticed that not all learners participated except in Groups 3 and 5 in Case 2. The descriptive statistics show that learner participation in group project forum was higher than class-level discussion, which is unsurprising given the fact that the group project was a required component for learners to receive a grade. From Table 5, we also noticed that the groups in Case 2 had a higher level of participation than the single group from Case 1. This may be potentially explained by the fact that most of the learners in Case 2 are from the same company and, thus, felt more comfortable interacting with each other. The nature of class-level discussion prompts and the group project forum may have also contributed to the differences in participation. In the class-level discussion prompts, we asked more reflective questions about how systems engineering and MBSE are or can be used in their professional organizations. The necessity of discussing their organizations, work environments, and cultures may deter some learners from interacting as research has shown that different organizations or disciplines may have various attitudes towards sharing [20,49]. The same concern did not exist in the group project, where we asked learners to apply MBSE concepts to a real-world example (i.e., case study) without having to disclose many details about their professional organizations.

4.1.3. Connectors

Our social networking analysis of class-level discussion and the group project forum also showed that some learners were positioned in the center of the maps, acting as a connector bridging fellow learners together. The results suggested that these learners were more engaged and strived to interact with their peers more often. In the class-level discussions among the Case 2 learners shown in Figure 1, Eric acted as the main connector throughout the module. As past SNA literature revealed that connectors have higher ability to introduce new behaviors and attitudes into the social network [40], it is possible that Eric’s presence in Case 2’s class-level discussion and his continuous participation throughout the module duration had a positive influence on learners‘ comfort level to share and interact with others. As social presence is essentially concerned with how comfortable learners feel to construct knowledge together, having a connector may be one of the major contributors to the elevated social presence in Case 2. Similarly, several connectors can be identified in the group project forums from both cohorts, with most groups having one or two connectors. In Case 2, there were two groups which had no learners acting as connectors at all. However, this may be due to the smaller group sizes (e.g., Group 4 only had 2 learners). Among the connectors identified, only one connector was active on both the class and group discussions. The other connectors only participated actively in the group discussion board.

4.1.4. Higher Social Presence among Learners from the Same Company

The results from the CoI survey items indicate that the learners in Case 2 reported higher levels of social presence than learners in Case 1. This may be largely associated with the fact that most of the learners work for the same company. Literature has associated the degree of social presence with several factors, including privacy during online communication [50], the sense of comfort in the learning community [49], and decreased distance and time between learners [51]. These enhancers of online interaction can be amplified if the learners are from the same company. However, the higher social presence in Case 2 was not evident in the number of discussion posts created, as shown in Table 4 and Table 5. This means that a large part of the social presence experienced by learners in Case 2 was not captured in the discussion board. It is possible that these learners moved onto the other communication platforms. In contrast, as Case 1 learners were not overwhelmingly from the same company, communicating through the discussion board was probably the most convenient option for them.

4.2. Different Levels of Learning Outcomes Associated with Social Presence

4.2.1. Higher Social Presence Corresponds to Higher Teaching and Cognitive Presences

Apart from a perceived higher social presence, learners in Case 2 also consistently reported higher teaching and cognitive presences than learners in Case 1. The learners in Case 2 mainly reported more teaching presence on two sub-constructs: facilitation and direct instruction. Considering that module delivery and course design underwent minimal changes between the two cohorts and that the instructional team was the same in both cases, it is interesting to see the elevated teaching presence experienced by Case 2 learners. One potential explanation may be derived from the definitions of these sub-constructs. Based on the definition of sub-constructs in Table 7, these two aspects of teaching presence have a large overlap with social presence and interactions on the discussion board. The higher facilitation and direct instruction scores from Case 2 indicate that these learners felt more encouragement and scaffolding from the instructor to interact with each other. It is also worth noting that direct instruction scores were the lowest among teaching presence sub-constructs in both cases. This indicates that perhaps clearer and more frequent instructions and feedback regarding the discussion posts could potentially increase social presence even further.

By definition, a higher cognitive presence indicates that learners have a higher ability to construct knowledge [14]. Since Case 2 learners reported a higher overall cognitive presence score, it means that these learners perceived more positive learning outcomes. More specifically, the sub-construct scores showed that learners initially engaged in critical thinking at a similar level (as suggested via the small difference in the triggering event score between Case 1 and 2). However, the perceived learning outcomes resulting from critical thinking were different. Learners in Case 2 reported higher levels of exploration, integration, and resolution than learners in Case 1. This means that learners in Case 2 felt more capable with applying the knowledge acquired in the module, which led to a higher level of overall cognitive presence. Part of the higher cognitive presence can be attributed to the higher social presence perceived among Case 2 learners [52]. Additionally, Case 2 learners’ higher teaching presence may also be related to Case 2 learners’ elevated teaching presence. Although instructor–learner interaction was not captured through the SNA maps, instructors may interact with learners by providing direct feedback for the group project deliverables. It is possible that learners in Case 2 had more in-depth conversations and received more personalized feedback from instructors in the group project, which facilitated their learning.

4.2.2. Higher Social Presence Corresponds to Higher Motivation in Future Learning

At the beginning of the module, we noticed that learners from both cases had high levels of motivation, as suggested through the high expectancy and value scores and low cost score. Previous reports examining the expectancy, value, and cost scores of graduate learners in MOOCs using the same instrument reported similar values for the expectancy construct and higher scores for the value and cost constructs [37]. The lower scores observed for the value construct could be due to the different reasons learners have for enrolling on online modules: for MOOCs, learners may mostly choose to self-enroll, while in a professional development course such as our module they might be incentivized or required to enroll by their employers.

However, after engaging in the module Case 1 learners did not indicate a strong desire to continue in future MBSE learning. Case 2 learners, on the other hand, showed an opposite trend and indicated stronger enthusiasm, which may be attributed to higher levels of social presences perceived by these learners and their reasoning for learning MBSE. This result added more evidence to past research on online graduate courses, which suggests that the different circumstances of professional/adult learners might influence their motivation, such as the course’s relevance for one’s job or taking the course along with a friend [40]. Although professional/adult learners are more self-motivated, their motivation may still fluctuate through learning; thus, maintaining their initial levels of motivation is not guaranteed. Finally, the learners in Case 2 were able to develop a more personally relevant awareness of MBSE. One possible explanation for this fact is that the particular organizational culture and environment helped learners to maintain a higher level of motivation throughout the module. Past research on online courses suggests that goal-setting and creating strategic plans before learning can keep learners motivated [53].

4.2.3. Higher Social Presence Corresponds to More Personally Relatable MBSE Awareness

As previously mentioned, Case 2 learners perceived higher levels of learning outcomes. This result is reflected by their higher levels of cognitive and teaching presence, as well as their greater enthusiasm for future learning. Another important indicator for better learning outcome is learners’ awareness of MBSE and how it can directly connect to their daily job. Our thematic analysis revealed that the learners in Case 2 developed more personally relatable awareness of MBSE. As they were experiencing an external need to learn MBSE to fulfill stakeholder needs and meet customers’ requirements, they were able to build direct connections between the knowledge learned in the module and their work environment. It is possible that the company was transitioning to an MBSE approach and encouraging the culture shift within their organization, which, in turn, positively affected learners’ learning outcomes [1,3].

5. Conclusions

In conclusion, we found that learners from the same organization have higher social presence compared to a cohort of more isolated learners. Moreover, higher social presence corresponds with greater motivation to participate in future MBSE learning after taking the module. The elevated social presence is also associated with better learning outcomes, as shown in a higher cognitive and teaching presence in the CoI survey results, as well as more personally relatable MBSE awareness. This study provides additional empirical support to some of the prevalent perceptions about MBSE adoption and education. Based on our findings, we provide several actionable items to improve MBSE professional development programs.

For industry cultivating an MBSE workforce, our recommendations are:

• Work closely with MBSE course providers and instructors to create a cohort of learners with common learning goals and environments to promote online community building;
• Provide employees enrolled in professional development programs concrete and tangible reasons or incentives to engage in MBSE learning.

For MBSE professional development instructors, our recommendations are:

• When teaching learners who are mostly from the same company, instructors should find ways to facilitate social presence on multiple communication platforms;
• When teaching learners who are more heterogenous in their organizational affiliations, additional scaffoldings are needed to promote a sense of community and social interaction. Examples for scaffoldings are:
  • Include orientation activities prior to teaching and allow learners to get to know each other;
  • Create activities that are applicable to learners’ jobs and, yet, do not require them to share sensitive proprietary information about their organizations.

6. Limitations & Future Work

One limitation of our study was the small sample size; however, this case study as a research method is particularly beneficial for understanding learning situations when the researcher has little-to-no control over phenomena (e.g., the number of participants that enroll in a course) [32]. In addition, the case study allows for multiple sources of data to richly examine the research questions. Since surveys were not mandatory, there was a mismatch between the response number of pre- and post-module survey, especially in the 2022 cohort. This can potentially affect the results since it is likely that learners who responded to the post-survey were the individuals who stayed engaged and were more satisfied with their learning experiences. Future work should work to develop strategies to strengthen participation in the surveys. Designing and delivering genuine opportunities for professional learners to form a learning community where they can meaningfully interact around content is a real challenge. Future studies should continue to examine the efficacy of the approach and how conversations within the learning community serve to strengthen learners’ ability to apply MBSE concepts in their actual work. Since teaching presence in this study is not fully implemented, future research should also focus on implementing the full CoI framework and develop ways to created more effective MBSE professional development programs to speed up MBSE adoption.