Next Article in Journal
Exploring the Role of Fossil Fuels and Renewable Energy in Determining Environmental Sustainability: Evidence from OECD Countries
Next Article in Special Issue
Determinants of University Students’ Intention to Use Video Conferencing Tools during COVID-19 Pandemic: Case of Somalia
Previous Article in Journal
The Impact of COVID-19 Process on Sustainability in Education: Work Alienation of Physical Education and Sports Teachers
Previous Article in Special Issue
Usage Intention of e-Learning Systems in Ghanaian Tertiary Institutions: A Case Study of the University for Development Studies
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sustainability
Volume 15
Issue 3
10.3390/su15032049
sustainability-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Statistical Assessment on Student Engagement in Asynchronous Online Learning Using the k-Means Clustering Algorithm

by
Sohee Kim
1,
Sunghee Cho
1,
Joo Yeun Kim
1 and
Dae-Jin Kim
2,*
1
Center for Teaching and Learning, Kyung Hee University, Seoul 02447, Republic of Korea
2
Department of Architectural Engineering, Kyung Hee University, Yongin 17104, Republic of Korea
*
Author to whom correspondence should be addressed.
Sustainability 2023, 15(3), 2049; https://doi.org/10.3390/su15032049
Submission received: 18 December 2022 / Revised: 15 January 2023 / Accepted: 18 January 2023 / Published: 20 January 2023
(This article belongs to the Collection Education, Innovation and Training for Sustainable Development in the Context of COVID-19)
Browse Figures
Versions Notes

Abstract

:
In this study, statistical assessment was performed on student engagement in online learning using the k-means clustering algorithm, and their differences in attendance, assignment completion, discussion participation and perceived learning outcome were examined. In the clustering process, three features such as the behavioral, emotional and cognitive aspects of student engagement were considered. Data for this study were collected from undergraduate students who enrolled in an asynchronous online course provided by Kyung Hee University in Republic of Korea in the fall semester of 2021. The students enrolled in the asynchronous online course were classified into two clusters with low and high engagement perceptions. In addition, their differences in attendance, assignment completion, discussion participation, interactions and perceived learning outcome were analyzed. The results of this study indicate that quantitative indicators on students’ online behaviors are not sufficient evidence to measure the level of student engagement and the students enrolled in the asynchronous online course were classified into two groups with low and high engagement perceptions. It is recommended that online instructors consider various strategies to facilitate interaction for the students with low engagement perceptions.

1. Introduction

During the COVID-19 pandemic, the number of online courses offered in universities has dramatically increased all over the world [1,2]. Although transition from traditional offline classes to online education was smooth in most cases, it is still a challenging issue whether students can achieve meaningful learning outcomes within online learning environments. In particular, in the middle of the pandemic, students experienced various types of online classes, such as pre-recorded video lectures and real-time online classes using conventional video communications platforms as well as virtual reality platforms [3,4,5]. This has brought about a great change in the perception of students’ role in online learning environment and how they are engaged with online education. Most of researchers agree that student engagement is a critical factor for meaningful online learning. Martin and Bolliger [6] emphasized the importance of student engagement in online learning as it can increase student satisfaction, reduce the sense of isolation, and improve student performance in online courses. Related studies [7,8] also revealed that the effectiveness of online courses offered in the university is closely related to the active engagement of students. Weller [9] and Keith [10] suggested that online instructors should create multiple opportunities for learners to actively participate in their learning process. Considering all these results, student engagement can be regarded as an essential factor in online learning, which positively affects its effectiveness as well as learners’ psychological aspects such as student motivation and satisfaction.
Student engagement was originally understood as observable behavior and time taken to complete learning tasks. Then, the emotional aspect of students’ experience in the process of learning was gradually incorporated into the concept of student engagement [11,12]. Fredricks et al. [13] defined student engagement as a meta-construct that includes behavioral, emotional and cognitive engagement. This three-factor model is generally accepted, but there are still many studies focusing on its observable and behavioral aspect such as attendance, assignment completion and discussion participation. For example, Fall and Robert [14] investigated students’ behavioral engagement effects on academic achievement and learning completion rates. In particular, the course completion rate is a decisive indicator for successful learning in asynchronous online courses. Thus, several researchers [8,15] classified learners into clusters based on their behavioral data such as video viewing and task completion and analyzed the differences of learning outcomes between the clusters.
Nonetheless, it is necessary to investigate whether objective indicators such as attendance, task completion and discussion participation reflect the student’s subjective perception of engagement. Parker et al. [16] found that approximately 80% of the students, who took online courses during the pandemic, did not actively participate in the online course activity. Ober and Kochmanska [17] reported that many students experienced distraction and decreased concentration in the process of online learning. This means that the objective indicators may not be matched by students’ subjective perception of engagement. This task is also related to the issue of how to measure student engagement. Atapattu and Falkner [18] recently utilized the objective indicators as a means of examining students’ engagement patterns. Yoon et al. [19] focused on clarifying the relationship between these behaviors and student engagement without conceptualizing them. It is still hard to measure and evaluate the level of student engagement in the learning process even though many researchers explored this topic [20]. It is not clear that the evaluation of student engagement through students’ self-reporting is consistent with actual learning behavior although students’ subjective perception and emotional aspect can be identified. Furthermore, it is difficult to know the emotional aspect of engagement only from objective behaviors reported by students as well as whether these actions have a positive effect on meaningful learning. Fredricks et al. [13] recommended to consider measurement methods to explain different types of engagement to help students understand reasons for underachievement.
Student engagement can be affected by various factors involved in the online teaching–learning process. Cole et al. [21] pointed out the importance of interactions facilitating student engagement utilizing an expression of “climate”. The climate in online learning means the perceived relationships between the instructor and students. The interactions between them can create a positive classroom atmosphere, encouraging students’ participation [22]. In online learning, instructor–student and student–student interactions can reduce students’ feeling of isolation [7,23], and bring about positive learning outcomes [6]. As strategies for increasing the level of students’ behavioral engagement, instructors may provide feedback on student’s performance, give a question as well as its answer, exchange opinions in a discussion board and take time for ice-breaking. If a positive atmosphere through these activities is formed between the instructor and students, it may also affect the perception of engagement in the emotional and cognitive aspect.
Therefore, this study intends to inspect students’ engagement patterns by considering both their awareness of engagement and actual log behaviors recorded in learning management system (LMS). In order to clarify the characteristics of student engagement, we examined whether students exhibit common patterns of engagement during online learning process as well as whether these patterns are ultimately related to meaningful learning. Since student engagement is a variable characteristic depending on class climate rather than a student’s inherent characteristic, we investigated how instructor–student and student–student interaction can contribute to the quality of student engagement.
By utilizing various data collected from an asynchronous online course offered at Kyung Hee University in Republic of Korea in the fall semester of 2021, we (1) collected students’ log behaviors recorded on LMS as well as questionnaire data for measuring student engagement, interactions and perceived learning outcomes, (2) examined the correlation with behaviors data and engagement perceptions, (3) classified the enrolled students according to their perception of engagement, and (4) analyzed the difference in interactions and perceived learning outcomes between the identified clusters. An understanding of students’ different engagement patterns helps to inform the instructional strategies that can meet the individual needs of students in terms of behavioral, emotional, and cognitive engagement. In particular, the results of this study can provide useful interaction strategies that play an important role in creating a positive atmosphere in online learning environment.
Specifically, we address the following research questions in this study:
  • How are students’ log behaviors correlated with behavioral, emotional and cognitive engagement perceived by students?
  • How can students be clustered based on their engagement patterns in an online course?
  • Do differences exist between identified clusters regarding students’ log behaviors, instructor–student interaction, student–student interaction and perceived learning outcome?

2. Theoretical Background

2.1. Understanding of Student Engagement

Student engagement can be defined as an active and continuous effort made by students in the process of understanding contents [24,25]. Fredricks et al. [13] conducted in-depth research to conceptualize engagement in learning as complex meta-construction including behavior, emotion and cognition. Reeve [26] insisted that the student engagement consists of behavioral, emotional and cognitive engagement. The behavioral engagement includes participation in academic and social or extracurricular activities and is considered important in achieving excellent academic results [11,13]. It is also related to student’s behavior toward the task, ranging from simply working and following the rules to participating in student council. The emotional engagement is related to student’s attitudes and interests, and includes positive and negative reactions to teachers, classmates and schools [13]. The cognitive engagement is related to motivational goals and self-regulated learning process, including willingness to make efforts to understand complex ideas and master difficult skills. Lei et al. [27] reported that there is a positive correlation between the academic achievement and overall level of learning engagement, and that the impact of the three types of engagement on academic achievement is significant in the order of the behavioral, cognitive and emotional engagement.
In addition, to research identifying key factors related to student engagement, some other researchers examined group characteristics by classifying students into similar engagement patterns. Khalil and Ebner [8] investigated students’ engagements in MOOCs and classified them into categories based on their level of engagement according to reading and writing frequency, video watching and quiz attempts. After comparing these features within the same group, they found that “dropout” cluster has a low level of engagement for all variables. Moubayed et al. [15] categorized online behaviors into two groups such as interaction-related and effort-related groups and suggested a methodology for classifying students with common behavioral patterns using k-means clustering. According to this research, the interaction-related behaviors include reading content, reading and posting forums and reviewing quizzes, while the effort-related behaviors lateness and duration indicator for assignment submission. Although the studies discussed above provided some basic data for identifying unengaging students based on online behavioral cues, they still have limitations in suggesting how to encourage students to actively participate in online learning process. Indeed, student engagement is an important factor that affects students’ learning and understanding in online education, which can help students retain online content. Therefore, students familiar with face-to-face instruction need guidance on how to effectively engage with online courses [16,28].

2.2. Student Engagement and Interaction

The success or failure of learning is affected by how often students and instructors participate in interactions taking place in the learning process. In education, interaction is regarded as a process of communication, in which two or more individuals create their knowledge and influence each other. It is an important factor for meaningful learning in online classes and acts as a factor in forming a positive class atmosphere and encouraging students’ participation [22]. The interaction in online classes affects the behavioral and emotional aspects of students, and can improve academic achievement, learning motivation, and a sense of belonging to the class. It also has a significant relationship with online class satisfaction, flow on learning and success/failure of learning [29]. Kim and Kim [30] reported that the interactions with other students and presence of instructors have a positive effect on student engagement, leading to enhanced students’ satisfaction in asynchronous online courses. Bergdahl and Bond [31] and Mutalib et al. [32] suggested that the lack of instructor–learner interaction affects the level of student engagement.
Moore and Kearsley [33] suggested three types of interaction such as student–content interaction, instructor–student interaction and student–student interaction. The student–content interaction changes one’s own cognitive structure through the process of selecting, understanding and reorganizing learning contents. The instructor–student interaction refers to feedback and guidance provided by instructors on the students’ activities and outputs. The student–student interaction can enhance emotional stability by strengthening the psychological bond between them, and further enhance learning effect by exchanging ideas or learning materials.
The instructor–student interaction is the most significant variable that can predict learning effect, and students usually put the greatest value on interaction with the instructor. The student–student interaction is not directly related to learning, but helps to create a positive learning environment. The two types of interaction can reduce students’ feeling of isolation and produce positive learning outcomes [7,23]. Therefore, this study focuses on these two types of interactions among various types of interaction that can occur in the online learning environment.
Meanwhile, Cole et al. [21] pointed out the importance of interactions facilitating student engagement using an expression of “climate”. The climate in online learning indicates perceived relationships between the instructor and students. The instructor–students interaction can create a positive classroom atmosphere, leading to students’ participation [22]. This is because engagement arises from the interaction of participants with the context and responds to changes in learning environment [11]. For this reason, some studies on interaction perceived by students were conducted considering the psychological dimension of interaction [34] as well as its quantity, quality, and type.

2.3. Student Engagement and Perceived Learning Outcome

As discussed in Section 2.1, it can be stated based on theoretical and empirical studies that students’ active engagement in online learning is associated with successful learning performance. Carini et al. [35] reported that desirable learning outcomes such as critical thinking and high grades are closely related to the level of student engagement. Dixson [7] reported that the effectiveness of online learning at university is closely related to students’ active participation. Gray and DiLoreto [36] found that student engagement mediates the relationship between student–student interaction and perceived learning outcomes.
However, in the study of Kim and Kim [30], no significant relationship was found between student engagement and academic achievement represented by grades. It can be noted that successful learning performance entails not only behavioral changes but also internal changes of students. Alavi et al. [37] insisted that learning may not always be reflected in behavior or performance and need to consider the relative changes in students’ mental models rather than those in behavior. Based on the discussion above, perceived learning outcome in this study refers to change in perception of the student’s own knowledge and skill level compared afterward.

3. Methodology

3.1. Survey Data

Data for this study were collected from undergraduate students who enrolled in an asynchronous online course provided by Kyung Hee University in Korea in the fall semester of 2021. This course is one of the distribution requirement subjects of general education at Kyung Hee University and has been operated as a large online class that can accommodate up to 500 students since the fall semester of 2021. Most of the students were studying full-time. The course consists of pre-recorded video lectures for 16 weeks of the entire fall semester, 7 individual assignments, discussion on 4 topics and printable learning materials related to the video lectures. The course organization and operation were similar to most of other MOOCs offered all over the world, and the same content was given to students regardless of their year of enrollment.
The authors participated in the design, development, and operation of this online course, and helped the instructor to design learning activities and to effectively operate the course. An online survey was conducted for two weeks from the 15th to 16th week of the semester. Among the 496 students enrolled in this course, 215 students responded to the survey, and a total of 203 responses were analyzed after discarding 12 responses with missing values. All survey participants were informed that anonymity is assured as well as that all responses will be compiled together, analyzed as a group, and used only as data for research and course improvement. Only the students who agreed with this participated in the survey. Table 1 summarizes demographic characteristics of the participants of this study.

3.2. Data Collection and Measurement Instrument

This study utilized mainly two different types of data, which are students’ log behaviors during their participation in online course and self-reported survey to measure students’ perceptions of the three types of engagement and three other key factors such as instructor–student interaction, student–student interaction and perceived learning outcome. In order to evaluate students’ log behaviors, data on students’ attendance, assignment completion and discussion participation were collected from the LMS at the end of the course. Other individually identifiable information was not included in the analysis. The following is specific information about the students’ log behaviors collected:
(1)
Attendance: the number of video lecture viewings.
(2)
Assignment completion: the number of assignments submitted.
(3)
Discussion participation: the number of posts in discussion forums.
The self-reported survey measurement is critical in collecting data on students’ subjective perceptions and particularly useful for assessing emotional and cognitive engagement, which are not directly observable [37]. The self-reported questionnaire was developed to investigate the instructor-student interaction, student–student interaction, student engagement and perceived learning outcome. Items for measuring instructor–student interaction and student–student interaction were developed by modifying the survey items utilized in the study of Johnson et al. [29], which were originally parts of Dimensions of Distance Education [38]. The survey items in the study of Sun and Rueda [39], originated from the study of Fredricks and McColskey [12], were utilized for measuring three types of student engagement. The survey items for measuring perceived learning outcome were developed by modifying the items in the study of Sher [40]. The survey questionnaire was composed of twenty-three items. All of the survey items were rated based on a five-point Likert scale, which ranges from 1 indicating “strongly disagree” to 5 “strongly agree”. Cronbach’s alpha coefficients were calculated to ensure the measurement instrument’s internal reliability. Cronbach’s alpha coefficient of the total items was 0.951, and the reliability for each variable was satisfactory as shown in Table 2 as they range from 0.793 to 0.949 [41]. The representative examples of the survey items are summarized in Table 3.

3.3. Data Analysis

The data analysis of this study was conducted in three different steps. First, descriptive statistics and correlations among measurement variables were calculated and analyzed using SPSS (version 25). Second, k-means clustering analysis was performed to classify participants based on their engagement patterns. Before performing k-means clustering, the data was rescaled using Z-score standardization method [42]. It is one of the most popular clustering methods and used when the patterns of data are not known [14]. There are several suggestions in the literature for choosing the right k value after multiple runs of k-means, and we used silhouettes for that. The silhouette is the score of comparing within–cluster distances with between–cluster distances, the greater the difference the better the fit [40]. It can be used as an index to measure the quality of a final clustering, and cluster silhouettes are used to guide a genetic algorithm in the selection of variables that best describe the structure of the data at hand [41]. The silhouette scores lie in the range from −1 to 1. If the silhouette score is close to 1, it means that the data set is well clustered [43]. The k-means function in Orange (version 3.31) was used for the analysis of this study. Orange is an open-source machine learning and data mining software written with Python. It is regarded as one of the best-performing software programs in terms of accuracy according to comparison of several data mining tools based on k-means clustering currently available [44]. It calculates the silhouette score for each cluster and visually presents the results [45], which aids in the selection of the right the number of cluster (k). Lastly, the mean differences between clusters classified based on participants’ engagement patterns were analyzed.

4. Results and Discussion

4.1. Research Question 1: How Are Students’ Log Behaviors Correlated with Behavioral, Emotional and Cognitive Engagement Perceived by Students?

Table 4 shows the descriptive statics and correlations among the measurement variables. It summarizes the values of correlation, mean and standard deviation for measurement variables. Statistically significant correlations are partially found between students’ log behaviors and three types of engagement. However, all Pearson’s simple correlation coefficients are too small to find correlations between students’ actual learning behaviors and their perceptions on the three types of engagement. One interesting thing is that, although correlation between attendance and emotional engagement is weak, the direction of correlation is negative (r = −0.025, p < 0.05).
Consequently, it can be noted from this result that the quantitative indicators on students’ online behaviors were not sufficient evidences to measure the level of student engagement in an online course. These results are in consistence with the results reported in [16]. Specifically, video lectures viewing and submitted assignments can increase the perception of students’ cognitive engagement, but just watching lectures can decrease emotional engagement. As pointed out by several researchers [7,8], this seems to be caused by psychological isolation resulting from the online learning environment. Therefore, it is necessary for instructors to create a positive classroom atmosphere by frequently interacting with students. This can be supported by the work of Kaufmann et al. [22], which insists that the interaction is a critical factor in creating a positive atmosphere in online learning and encouraging student engagement.
It is also noteworthy that the discussion participation, which is one strategy for promoting interactions in online learning, had a positive correlation with behavioral engagement, but had no statistically significant correlation with cognitive and emotional engagement. In the online course considered in this study, discussion was done for the entire class, making it more difficult for student participants to exchange opinions in-depth, in contrast to the case of small group discussion [46]. In particular, mass discussion for the entire class can induce information overload, which may make some students lose their confidence and eventually withdraw themselves from discussion [47]. As a result, it is important that not only observable behavioral indicators but also students’ psychological aspects of their engagement should be considered to encourage student engagement in online courses. Also, small group discussion may be more effective than mass discussion to improve students’ cognitive and emotional engagement. In other words, in order to engage students in discussions and learning activities, instructors should plan learning activities taking into account learners’ psychological engagement by applying a more detailed instructional design. This can be achieved by utilizing various teaching strategies such as providing feedback, scaffolding and hints [48].

4.2. Research Question 2: How Can Students Be Clustered Based on Their Engagement Patterns in an Online Course?

As discussed in the introduction, one of the main purposes of this study is to assign each participant in the online course to a relevant group that shares common engagement patterns. For clustering, a k-means clustering algorithm is adopted, and the perceptions of behavioral, emotional and cognitive engagement are used as features. The scheme of measuring distance is set to “Euclidean”. The number of clusters (k) is internally assigned in Orange, and the silhouette score is computed for each cluster case ranging from 2 to 8. In general, each value of k yields a different overall average silhouette score for the entire plot, and one possible way to choose an appropriate k value is such that the overall average silhouette score is as large as possible [40]. Table 5 lists the silhouette scores evaluated for each number of clusters k, and it indicates that k = 2 provides the largest silhouette score among the seven cases considered (k = 2~8). From this result, clustering with k = 2 was selected and analyzed in this study. Silhouette plot with k = 2 and silhouette score for generated clusters are shown on Figure 1. The averages of silhouette scores of Clusters 1 and 2 are 0.58 and 0.65, respectively.
In the clustering with k = 2, 68 and 135 participants among a total of the 203 participants were classified into Clusters 1 and 2, respectively. In order to compare the levels of student engagement of the two clusters, a t-test was performed, and its results are given in Table 6. The means (M) of the behavioral, emotional, and cognitive engagements in Cluster 1 are generally smaller than those of Cluster 2. This indicates that Cluster 1 is characterized by a relatively low level of engagement patterns compared to Cluster 2. Therefore, Cluster 1 is labeled as “participants with low perception of engagement”, and Cluster 2 as “participants with high perception of engagement”. The result of the table indicates that the mean difference in the emotional engagement was the largest among the three group differences in behavioral, emotional and cognitive engagement.
In previous studies, various online behaviors such as the number of logins, average duration of assignment submissions, content read/access, forum posts, quiz reviews, browsing and social interactions were used as features for clustering [8,15,19]. However, as pointed out by Moubayed et al. [15], these online behaviors are not sufficiently determining factors for identifying the quality of engagement in online learning environments. Regarding research question 1 of this study, it was confirmed that students’ log behaviors are insufficient as evidences that students voluntarily engage in learning itself. Alavi et al. [37] insisted that learning is meaningful only when changes in cognitive and psychological aspects are accompanied with behavioral changes. Therefore, online instructors need to use teaching strategies that can elicit positive emotions and cognitive immersion for facilitating student engagement. Czerkawski and Lyman [47] suggested that instructors are not only content experts, but also experienced learners and mentors in online learning. In online classes, instructors can perform these roles by asking questions leading to critical thinking and deep learning and by providing feedbacks, which can be designed in several different ways such as task feedback, process feedback, self-regulatory feedback, superficial feedback and mediation feedback [48]. In addition to instructor feedback, there are scaffolding and hints that can promote learning and effectively engage students in online learning [49]. Their effects may vary depending on contents of the scaffolding and hints, so they must be used appropriately according to the teaching purpose. In particular, providing strategic or procedural scaffolding [50] can build learners’ confidence to succeed in online courses. In addition, if students are provided with reflective or supportive scaffolding [51], they can increase their awareness of cognitive engagement in learning.

4.3. Research Question 3: Do Differences Exist between Identified Clusters Regarding Students’ Log Behaviors, Instructor–Student Interactions, Student–Student Interactions and Perceived Learning Outcome?

As presented in Table 7, a t-test was conducted to compare the difference between the two clusters in terms of students’ log behaviors, instructor–student interaction, student–student interaction and perceived learning outcome. The results of the table are also plotted in Figure 2. The results of the table and figure show that the two clusters differ significantly in their perceptions of instructor–student interaction (t = 6.601, p < 0.001), student–student interaction (t = 6.848, p < 0.001), and perceived learning outcome (t = 5.997, p < 0.001). These results indicate that students of Cluster 2, a group with relatively higher awareness of engagement, show more interactions with their instructor and peer students than those of Cluster 1. Also, students in Cluster 2 tend to be more aware that the online course can help them acquire knowledge and skills in related fields. However, there is no significant difference between the two clusters in terms of attendance (t = 0.586, p > 0.001), assignments completion (t = 0.439, p > 0.001), and discussion participation (t = 1.445, p > 0.001). This indicates that the students of the two clusters are not so much different in the level of participation of the learning activities required for obtaining credits.
In online learning, it is common for instructors to adopt evaluation criteria such as online lecture watching without skipping, on-time assignment submission and the number of postings in discussion board. However, in the light of the result of this study, it is difficult to insist that these criteria exactly tell whether students are truly immersed and interested in learning contents provided and experience positive emotions through online learning. According to Johnson et al. [29], interactions between instructor and students as well as among students positively affect students’ motivation and allow them to have a sense of belonging to the class. The former helps to create a positive climate in online classes [21], and the latter can foster students’ emotional engagement by strengthening the psychological bond among peer learners [7]. The results of the present study also support these conclusions, emphasizing the importance of interaction on student engagement.
In addition, it was found that students with high awareness of engagement are able to achieve a more valuable learning outcome than those with low awareness of engagement. A positive atmosphere created by instructor–student interaction eventually can result in high level of academic achievement [52]. Instructors can interact with learners by providing feedback on assignments and providing guidelines for learners’ performance [53]. These teaching activities can increase the presence of instructor [33] and raise the perception of cognitive engagement. Increased awareness of cognitive engagement has the effect on facilitating learners’ cognitive learning processes by influencing learning motivation, self-regulated learning and efforts following complex task-solving [13]. In this aspect, it is necessary for instructors to carefully design instructional strategies to provide feedback and appropriate guidelines for learners’ academic performance beyond one-way delivery of online lecture content. Given that student–student interaction positively affects the emotional engagement by creating positive learning environment [54], it is necessary for online course instructors to provide opportunities for learners to exchange opinions or to provide an icebreaking time to get to know each other.

5. Conclusions

As online learning is growing increasingly in the context of higher education, engagement in learning becomes more critical for the effectiveness of online learning. This study classified students enrolled in an asynchronous online course at Kyung Hee University into two clusters based on the level of behavioral, emotional, and cognitive engagement. In addition, differences in attendance, assignment completion, discussion participation, interactions, and perceived learning outcome of the two clusters were analyzed. From the results of this study, the following conclusions can be drawn:
  • Quantitative indicators on students’ online behaviors were not sufficient evidence to measure the level of student engagement. As a result of verification, students’ log behaviors recorded in LMS did not show a positive correlation with the three types of student engagement such as behavioral, emotional, and cognitive engagement. This indicates that students’ psychological, internal, and voluntary participation are essential in achieving meaningful learning in online education.
  • The students enrolled in the asynchronous online course considered in this study were classified into two clusters, designated as Clusters 1 and 2, corresponding to students with low and high engagement perceptions, respectively. Cluster 2 students tended to perceive themselves as behaviorally participating in the online class with positive emotion and to have higher cognitive engagement than Cluster 1 students. This characteristic is important for the success of online education. Therefore, online instructors need to pay attention to Cluster 1 students and carefully manage them in the class.
  • There are group differences between identified clusters regarding instructor–student interaction, student–student interaction and perceived learning outcome. However, there are no significant group differences with attendance, assignment completion and discussion participation. This indicates that students in the group with a high awareness of student engagement have a high awareness of interactions, and value their own online learning performance. Since interaction is closely related to the awareness of learning participation in online classes, instructors need to consider instructor–student interaction strategies such as providing timely feedback, scaffolding, useful hints and guidance for learning. Additionally, they can use peer feedback, small group discussions and ice-breaking as strategies to facilitate student–student interaction. Instructors can encourage students with low engagement perceptions by utilizing these interaction strategies.
For further research, we propose to inspect student engagement in various online learning contexts so that relevant data can be accumulated. In addition, it is necessary to consider both quantitative and qualitative data to measure student engagement in online learning more closely.

Author Contributions

Methodology, S.K. and S.C.; Software, S.K.; Validation, S.K., S.C., J.Y.K. and D.-J.K.; Formal analysis, S.K.; Investigation, S.K., S.C., J.Y.K. and D.-J.K.; Resources, D.-J.K.; Writing—original draft, S.K. and D.-J.K.; Writing—review & editing, D.-J.K.; Funding acquisition, D.-J.K. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by a National Research Foundation of Korea (NRF) grant funded by the Korean government (Ministry of Science, ICT & Future Planning) (No. 2020R1A2C1014806).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Chan, R.Y.; Bista, K.; Allen, R.M. Is online and distance learning the future in global higher education? The faculty perspectives during COVID 19. In Online Teaching and Learning in Higher Education during COVID 19: International Perspectives and Experiences; Chan, R.Y., Bista, K., Allen, R.M., Eds.; Routledge: Abingdon, UK, 2021; pp. 3–12. [Google Scholar]
  2. Cranfield, D.J.; Tick, A.; Venter, I.M.; Blignaut, R.J.; Renaud, K. Higher education students’ perceptions of online learning during COVID-19-A Comparative Study. Educ. Sci. 2021, 11, 403. [Google Scholar] [CrossRef]
  3. Saverino, D. Teaching anatomy at the time of COVID. Clin. Anat. 2020, 34, 1128. [Google Scholar] [CrossRef] [PubMed]
  4. Srinivasan, D.K. Medical students’ perceptions and anatomy teacher’s personal experience using an e-Learning platform for tutorials during the Covid-19 crisis. Anat. Sci. Educ. 2020, 13, 318–319. [Google Scholar] [CrossRef] [PubMed]
  5. Dhonncha, E.N.; Murphy, M. Learning new ways of teaching and assessment: The impact of COVID-19 on undergraduate dermatology education. Clin. Exp. Dermatol. 2020, 46, 170–171. [Google Scholar] [CrossRef] [PubMed]
  6. Martin, F.; Bolliger, D.U. Engagement Matters: Student perceptions on the importance of engagement strategies in the online learning environment. Online Learn. 2018, 22, 205–222. [Google Scholar] [CrossRef]
  7. Dixson, M.D. Creating effective student engagement in online courses: What do students find engage? J. Scholarsh. Teach. Learn. 2010, 10, 1–13. [Google Scholar]
  8. Khalil, M.; Ebner, M. Clustering patterns of engagement in massive open online course (MOOCs): The use of learning analytics to reveal student categories. J. Comput. High. Educ. 2017, 29, 114–132. [Google Scholar] [CrossRef]
  9. Weller, M. The distance from isolation: Why communities are the logical conclusion in e-learning? Comput. Educ. 2007, 49, 148–159. [Google Scholar] [CrossRef]
  10. Keith, T.-S. Early attrition among first time eLearners: A review of factors that contribute to drop-out, withdrawal and non-completion rates of adult learners undertaking eLearning programmes. J. Online Learn. Teach. 2006, 2, 73–85. [Google Scholar]
  11. Connell, J.P.; Welborn, J.G. Competence, autonomy, and relatedness: A motivational analysis of self-system processes. In Self Processes and Development; Gunnar, M., Sroufe, L.A., Eds.; Lawrence Erlbaum, Inc.: Mahwah, NJ, USA, 1991; pp. 43–77. [Google Scholar]
  12. Fredricks, J.A.; McColskey, W. The measurement of student engagement: A comparative analysis of various methods and student self-report instruments. In Handbook of Research on Student Engagement; Christenson, S.L., Reschly, A.L., Wylie, C., Eds.; Springer: New York, NY, USA, 2012; pp. 763–782. [Google Scholar]
  13. Fredricks, J.A.; Blumenfeld, P.C.; Paris, A.H. School engagement: Potential of the concept, state of the evidence. Rev. Educ. Res. 2004, 74, 59–109. [Google Scholar] [CrossRef] [Green Version]
  14. Fall, A.M.; Robert, G. High school dropouts: Interactions between social context, self-perceptions, school engagement, and student dropout. J. Adolesc. 2012, 35, 787–798. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  15. Moubayed, A.; Injadat, M.; Shami, A.; Lutfiyya, H. Student engagement level in an e-Learning environment: Clustering using K-means. Am. J. Distance Educ. 2020, 34, 137–156. [Google Scholar] [CrossRef]
  16. Parker, S.W.; Hansen, M.A.; Bernadowski, C. COVID-19 campus closures in the United States: American student perceptions of forced transition to remote learning. Soc. Sci. 2021, 10, 62. [Google Scholar] [CrossRef]
  17. Ober, J.; Kochmanska, A. Remote learning in higher education: Evidence from Poland. Int. J. Environ. Res. Public Health 2022, 19, 14479. [Google Scholar] [CrossRef]
  18. Atapattu, T.; Falkner, K. Impact of lecturer’s discourse for students’ video engagement: Video learning analytics case study of moocs. J. Learn. Anal. 2018, 5, 182–197. [Google Scholar] [CrossRef] [Green Version]
  19. Yoon, M.; Lee, J.; Jo, I.-H. Video learning analytics: Investigating behavioral patterns and learner clusters in video-based online learning. Internet High. Educ. 2021, 50, 100806. [Google Scholar] [CrossRef]
  20. Peng, W. Research on model of student engagement in online learning. EURASIA J. Math. Sci. Technol. Educ. 2017, 13, 2869–2882. [Google Scholar] [CrossRef]
  21. Cole, A.W.; Lennon, L.; Weber, N.L. Student perceptions of online active learning practices and online learning climate predict online course engagement. Interact. Learn. Environ. 2021, 29, 866–880. [Google Scholar] [CrossRef]
  22. Kaufmann, R.; Sellnow, D.D.; Frisby, B.N. The development and validation of the online learning climate scale (OLCS). Commun. Educ. 2016, 65, 307–321. [Google Scholar] [CrossRef]
  23. Banna, J.; Lin, M.-F.G.; Stewart, M.; Fialkowski, M.K. Interaction matters: Strategies to promote engaged learning in an online introductory nutrition course. J. Online Learn. Teach. 2015, 11, 249–261. [Google Scholar]
  24. Coates, H. Student Engagement in Campus-Based and Online Education: University Connections; Routledge: London, UK, 2006. [Google Scholar]
  25. Trowler, V. Student engagement literature review. High. Educ. Acad. 2010, 11, 1–15. [Google Scholar]
  26. Reeve, J. A self-determination theory perspective on student engagement. In Handbook of Research on Student Engagement; Springer: Boston, MA, USA, 2012; pp. 149–172. [Google Scholar]
  27. Lei, H.; Cui, Y.; Zhou, W. Relationships between student engagement and academic achievement: A meta-analysis. Soc. Behav. Personal. 2018, 46, 517–528. [Google Scholar] [CrossRef]
  28. Ahshan, R.A. Framework of implementing strategies for active student engagement in remote/online teaching and learning during the COVID-19 pandemic. Educ. Sci. 2021, 11, 483. [Google Scholar] [CrossRef]
  29. Johnson, S.D.; Aragon, S.R.; Shaik, N.; Palma-Rivas, N. Comparative analysis of learner satisfaction and learning outcomes in online and face-to-face learning environments. J. Interact. Learn. Res. 2000, 11, 29–49. [Google Scholar]
  30. Kim, S.; Kim, D.-J. Structural relationship of key factors for student satisfaction and achievement in asynchronous online learning. Sustainability 2021, 13, 6734. [Google Scholar] [CrossRef]
  31. Bergdahl, N.; Bond, M. Negotiating (dis-)engagement in K-12 blended learning. Educ. Inf. Technol. 2022, 27, 2635–2660. [Google Scholar] [CrossRef]
  32. Mutalib, A.A.A.; Akim, A.M.; Jaafar, M.H. A systematic review of health sciences students’ online learning during the COVID-19 pandemic. BMC Med Educ. 2022, 22, 1–34. [Google Scholar] [CrossRef]
  33. Moore, M.G.; Kearsley, G. Distance Education: A Systemic View; Wadsworth: New York, NY, USA, 2005. [Google Scholar]
  34. Kwon, K.; Hong, R.Y.; Laffey, J.M. The educational impact of metacognitive group coordination in computer-supported collaborative learning. Comput. Hum. Behav. 2013, 29, 1271–1281. [Google Scholar] [CrossRef]
  35. Carini, R.M.; Kuh, G.D.; Klein, S.P. Student Engagement and Student Learning: Testing the Linkages. Res. High. Educ. 2006, 47, 1–32. [Google Scholar] [CrossRef] [Green Version]
  36. Gray, J.A.; DiLoreto, M. The effects of student engagement, student satisfaction, and perceived learning in Online learning environments. Int. J. Educ. Leadersh. Prep. 2016, 11, n1. [Google Scholar]
  37. Alavi, M.; Marakas, G.M.; Yoo, Y. A comparative study of distributed learning environments on learning outcomes. Inf. Syst. Res. 2002, 13, 404–415. [Google Scholar] [CrossRef]
  38. Appletion, J.J.; Christenson, S.L.; Kim, D.; Reschly, A.L. Measuring cognitive and psychological engagement: Validation of the student engagement instrument. J. Sch. Psychol. 2006, 44, 427–445. [Google Scholar] [CrossRef]
  39. Sun, J.C.; Rueda, R. Situational interest, computer self-efficacy and self-regulation: Their impact on student engagement in distance education. Br. J. Educ. Technol. 2012, 43, 191–204. [Google Scholar] [CrossRef] [Green Version]
  40. Sher, A. Assessing the relationship of student-instructor and student-student interaction to student learning and satisfaction in Web-based online learning environment. J. Interact. Online Learn. 2009, 8, 102–120. [Google Scholar]
  41. Taber, K.S. The Use of Cronbach’s Alpha When Developing and Reporting Research Instruments in Science Education. Res. Sci. Educ. 2018, 48, 1273–1296. [Google Scholar] [CrossRef] [Green Version]
  42. Mohamad, I.B.; Usman, D. Standardization and its effects on K-means clustering algorithm. Res. J. Appl. Sci. Eng. Technol. 2013, 6, 3299–3303. [Google Scholar] [CrossRef]
  43. Rousseeuw, P.J. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 1986, 20, 53–65. [Google Scholar] [CrossRef]
  44. Lieti, R.; Ortiz, M.C.; Sarabia, L.A.; Sanchez, M.S. Selecting variables for k-means cluster analysis by using a genetic algorithm that optimises the silhouettes. Anal. Chim. Acta 2004, 515, 87–100. [Google Scholar] [CrossRef]
  45. Kodinariya, T.M.; Makwana, P.R. Review on determining number of Cluster in k-means clustering. Int. J. Adv. Res. Comput. Sci. Manag. Stud. 2013, 1, 90–95. [Google Scholar]
  46. Yang, T.; Luo, H.; Sun, D. Investigating the combined effects of group size and group composition in online discussion. Act. Learn. High. Educ. 2020. [Google Scholar] [CrossRef]
  47. Czerkawski, B.C.; Lyman, E.W. An instructional design framework for fostering student engagement in online learning environments. TechTrends 2016, 60, 532–539. [Google Scholar] [CrossRef]
  48. Chen, W.-C. Actual and preferred teacher feedback on student blog writing. Australas. J. Educ. Technol. 2014, 30, 402–414. [Google Scholar] [CrossRef]
  49. Kim, J.Y.; Lim, K.Y. Promoting learning in online, ill-structured problem solving: The effects of scaffolding type and metacognition level. Comput. Educ. 2019, 138, 116–129. [Google Scholar] [CrossRef]
  50. Cagiltay, K. Scaffolding strategies in electronic performance support systems: Types and challenges. Innov. Educ. Teach. Int. 2006, 43, 93–103. [Google Scholar] [CrossRef]
  51. Kim, M.C.; Hannafin, M.J. Scaffolding problem solving in technology-enhanced learning environments (TELEs): Bridging research and theory with practice. Comput. Educ. 2011, 56, 403–417. [Google Scholar] [CrossRef]
  52. Frisby, B.N.; Martin, M.M. Instructor–student and student–student rapport in the classroom. Commun. Educ. 2010, 59, 146–164. [Google Scholar] [CrossRef]
  53. Moore, M. Recent contribution to the theory of distance education. Open Learn. J. Open Distance e Learn. 1990, 5, 10–15. [Google Scholar] [CrossRef]
  54. Gilbert, L.; Moore, D.R. Building interactivity into Web courses: Tools for social and instructional interactions. Educ. Technol. 1998, 38, 29–35. [Google Scholar]
Sustainability 15 02049 g001 550
Figure 1. Silhouette score of a clustering with k = 2.
Figure 1. Silhouette score of a clustering with k = 2.
Sustainability 15 02049 g001
Sustainability 15 02049 g002 550
Figure 2. Mean difference for each measurement variable of Cluster 1 (C1) and Cluster 2 (C2).
Figure 2. Mean difference for each measurement variable of Cluster 1 (C1) and Cluster 2 (C2).
Sustainability 15 02049 g002
Table
Table 1. Statistical summary on survey participants. (N = 203).
Table 1. Statistical summary on survey participants. (N = 203).
Frequency (n)Percentage (%)
Year of admission20218139.9
20203818.7
20192110.3
20182612.8
Before 20173718.3
CollegeHumanities199.4
Politics and Economics4723.2
Management3517.2
Hotel and Tourism Management3818.7
Science199.4
Human Ecology125.9
Medicine199.4
Others 146.9
Total203100.0
Table
Table 2. Measurement instrument.
Table 2. Measurement instrument.
Key FactorNumber of ItemsCronbach’s AlphaScales
Behavioral Engagement (BE)40.7935-point Likert scale
Emotional Engagement (EE)40.949
Cognitive Engagement (CE)40.866
Instructor–Student Interaction (ISI)40.876
Student–Student Interaction (SSI)40.873
Perceived Learning Outcome (PLO)30.886
Total230.951
Table
Table 3. Representative examples of the survey items.
Table 3. Representative examples of the survey items.
Key FactorExamples of Survey Items
Behavioral Engagement“I followed the rules of this course.”
“I completed my homework on time.”
Emotional Engagement“I felt excited with my work in online class.”
“The online classroom was a fun place to be.”
Cognitive Engagement“I read extra materials to learn more about things we did in this course.”
“When I read the course materials, I asked myself questions to make sure that I understand them correctly.”
Instructor–Student Interaction“The instructor encouraged me to become actively involved in learning activities.”
“The instructor provided me feedback on my work through comments.”
Student–Student Interaction“I was able to share the learning experience with other students.”
“I was able to communicate with other students in this course.”
Perceived Learning Outcome“I improved my ability to integrate facts and develop generalization from course materials.”
“I learned concepts and principles in this course.”
Table
Table 4. Descriptive statistics and Correlation analysis results.
Table 4. Descriptive statistics and Correlation analysis results.
VariableAttendanceAssignment CompletionDiscussion ParticipationBEEECEISISSIPLO
Attendance-
Assignment completion0.294 **-
Discussion participation0.159 *0.422 **-
BE0.0580.163 **0.206 **-
EE−0.025 *0.0620.1070.579 **-
CE0.027 *0.147 *0.1320.681 **0.563 **-
ISI0.145 *0.141 *0.0820.681 **0.510 **0.556 **-
SSI0.0230.230 **0.327 **0.677 **0.512 **0.500 **0.704 **-
PLO0.0460.178 *0.210 **0.672 **0.505 **0.520 **0.690 **0.703 **-
Mean13.56.779.324.544.243.964.304.474.59
SD1.2680.7321.7940.5690.8590.8010.7290.6770.606
* p < 0.05, ** p < 0.01 BE: Behavioral Engagement, EE: Emotional Engagement, CE: Cognitive Engagement, ISI: Instructor–Student Interaction, SSI: Student–Student Interaction, PLO: Perceived Learning Outcome.
Table
Table 5. Silhouette scores for different k values.
Table 5. Silhouette scores for different k values.
Number of Cluster (k)Silhouette Score
20.444
30.388
40.376
50.384
60.388
70.398
80.397
Table
Table 6. Comparison of means (M) and standard deviation (SD) of the three students’ log behaviors and three other key factors for the two clusters identified.
Table 6. Comparison of means (M) and standard deviation (SD) of the three students’ log behaviors and three other key factors for the two clusters identified.
VariableClustertp
Cluster 1 (n = 68, 33.5%)Cluster 2 (n = 135, 66.5%)
MSDMSD
BE3.980.6134.820.24110.897 ***0.000
EE3.440.6534.640.51110.610 ***0.000
CE3.190.6594.350.54512.449 ***0.000
*** p < 0.001.
Table
Table 7. Comparison of means (M) and standard deviation (SD) of the three types of engagement for the two clusters identified Means comparison (and SD) of each measurement variable by identified clusters.
Table 7. Comparison of means (M) and standard deviation (SD) of the three types of engagement for the two clusters identified Means comparison (and SD) of each measurement variable by identified clusters.
VariableClustertp
Cluster 1 (n = 68)Cluster 2 (n = 135)
MSDMSD
Attendance13.570.813.481.40.5860.559
Assignment Completion6.740.86.790.70.4390.661
Discussion Participation9.042.09.451.61.4450.151
Instructor–Student Interaction (ISI)3.820.8074.540.5456.601 ***0.000
Student–Student Interaction (SSI)4.010.7654.700.4796.848 ***0.000
Perceived Learning Outcome (PLO)4.200.7454.780.3965.997 ***0.000
*** p < 0.001.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Kim, S.; Cho, S.; Kim, J.Y.; Kim, D.-J. Statistical Assessment on Student Engagement in Asynchronous Online Learning Using the k-Means Clustering Algorithm. Sustainability 2023, 15, 2049. https://doi.org/10.3390/su15032049

AMA Style

Kim S, Cho S, Kim JY, Kim D-J. Statistical Assessment on Student Engagement in Asynchronous Online Learning Using the k-Means Clustering Algorithm. Sustainability. 2023; 15(3):2049. https://doi.org/10.3390/su15032049

Chicago/Turabian Style

Kim, Sohee, Sunghee Cho, Joo Yeun Kim, and Dae-Jin Kim. 2023. "Statistical Assessment on Student Engagement in Asynchronous Online Learning Using the k-Means Clustering Algorithm" Sustainability 15, no. 3: 2049. https://doi.org/10.3390/su15032049

APA Style

Kim, S., Cho, S., Kim, J. Y., & Kim, D. -J. (2023). Statistical Assessment on Student Engagement in Asynchronous Online Learning Using the k-Means Clustering Algorithm. Sustainability, 15(3), 2049. https://doi.org/10.3390/su15032049

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop