Design of an IoT-Based Remote Learning System for Medical Skill Training in the Age of COVID-19: Focusing on CPR Skill Training

Abstract

Medical skill education has been scaled down due to the COVID-19 pandemic. In particular, the decrease in CPR skill training has caused the quality of medical services to deteriorate. While new online education methods have emerged, few studies exist on online teaching and its effects. Since the online teaching of medical skills presents several challenges for instructors, it has not been as effective as face-to-face training. This study designed a new remote system focusing on medical skill education. The proposed video-based application prototype uses an IoT device to measure CPR performance metrics and provides real-time data to users. It was tested using the Kano model on a small group of subjects. The effects of skill training were analyzed quantitatively and qualitatively. A comparative analysis of the remote group and face-to-face group revealed similar average values for appropriate compression depth. In other categories, the remote group fared poorer than the face-to-face group. Considering the high scores given to system usability in the USE survey, remote education shows promise as an alternative to face-to-face education. The significance of this study lies in being the first to develop and test a remote education system for medical skill training in the age of COVID-19.

1. Introduction

1.1. Background

With the global implementation of social distancing measures as a means of preventing COVID-19, there have been significant changes in teaching and learning practices [1,2]. Face-to-face sessions have had to be canceled or postponed, and remote forms like webinars have emerged as a plausible alternative. This has impacted medical education, of which skill training is an essential component. According to OSHA (Occupational Safety and Health Administration), those in the healthcare profession have experienced a serious deterioration of medical quality due to the suspension of medical skill education [3]. In particular, CPR (cardiopulmonary resuscitation) requires regular practice and training, but the suspension of training programs has caused medical personnel to be less proficient, negatively affecting the survival rate of cardiac arrest patients during emergencies [4].

The annual number of fatal cardiac arrests is 350,000 in the United States [5]. The mortality rate of cardiac arrests related to COVID-19 is also on the rise. CPR is the only skill that can be performed by general persons during emergencies and has grown widespread with CPR education being supported by the state. The suspension of CPR education for medical personnel and general persons following the outbreak of COVID-19 has weakened emergency treatment for cardiac arrest patients.

Medical skill training is directly related to saving lives, and the restrictions on face-to-face sessions highlight the need for new educational criteria and methods [6]. ERC (European Resuscitation Council) announced guidelines for internet-based self-directed learning and video education in response to COVID-19 [7]. Likewise, AHA (American Heart Association) presented guidelines for self-directed training and introduced a form of resuscitation training that utilizes a virtual interface [8]. However, both AHA and ERC stress that CPR training must be comprised of intervention by instructions and utilization of feedback equipment [9,10].

Solutions utilizing remote learning and digital tools will provide new opportunities for high-quality CPR skill training in the post-COVID-19 era. This study proposes a quality CPR remote training solution that achieves the goals of existing face-to-face education while complying with social distancing rules.

1.2. Related Work

Online learning is more efficient and convenient than conventional learning approaches as it reduces time and space constraints, thereby offering more opportunities for learners. In general, students may find online learning to be less immersive as it involves fewer interactions compared to face-to-face learning, and personalized interactions, such as determining the intent of individual students and engaging in real-time communication, are restricted [11]. According to a study by Ferrara [12], more than 90% of medical personnel answered that the adoption of e-learning negatively influenced the quality of medical education, and among them, 40% said that the impact was critical. Online teaching is not entirely new as it was used sporadically for medical education in the past [13]. The problem is that the transition is being made without proper research on the online handling of medical skill guidance and evaluation [14]. This has resulted in a sense of frustration towards online education, and some even give up webinars and short-term learning solutions. With the ongoing pandemic preventing us from returning to conventional approaches, there is a need to ponder over new methods that utilize the internet as the main medium [15].

Compared to other medical skills, CPR has a clearly defined protocol, including appropriate compression depth, rate, and recoil. CPR was chosen for this study as the transition to online education was expected to be easier. However, few studies exist on CPR skill training in relation to the pandemic. Some forms of education that have been attempted to date are introduced below.

1.2.1. E-Learning

This is a form of fully asynchronous online education that issues certificates to students who complete a multiple-choice test after watching video lectures related to CPR/AED/emergency treatment [16,17]. However, establishing an environment for simulations is necessary for medical skill training, and instructors should be able to assess student performance. From this perspective, the simple e-learning approach is unable to enhance medical skills as it does not support actual skill training.

1.2.2. Blended Learning

Blended learning, which combines online and face-to-face education, is another option for CPR training. Students learn theories online on the website of an educational institute, and then attend face-to-face sessions under the supervision of an instructor to earn their certificates. In the case of the American Red Cross, students who completed online theory sessions and planned to attend face-to-face sessions within three months were able to obtain pre-certification [18]. This approach places greater emphasis on skill training than e-learning but has been less preferred as face-to-face sessions in indoor environments can increase the risk of COVID-19 infection.

1.2.3. Non-Face-to-Face Simulation Station

Students use the simulation station installed at a training site on their own without the supervision of instructors. In Heart Code [19], a feedback mannequin and display are installed for asynchronous online learning. In addition, a spatial projection AR-based simulation station that provides more immersive audiovisual feedback in conjunction with a feedback manikin was introduced [20]. Students receive a certificate after participating in a hands-on session and completing a test. The students have greater autonomy, but this approach is more suitable for retraining of medical personnel or paramedics, and less so for members of the general public.

1.2.4. Video Platform-Based Remote Education

Instructors conduct CPR lectures through a video platform like Zoom. The reduced spatial constraints help to facilitate group sessions, in which an instructor teaches multiple students. In RSV [21], instructors send mannequins to students’ homes so that they can practice on mannequins and gain high-quality skills. While video platform education has increased following the implementation of social distancing, instructors face challenges in providing accurate, detailed feedback when teaching and evaluating students via the video platform. Having fewer interactions between students and instructors affects the effectiveness of education. As such, increasing the number of interactions can maintain the participatory approach while continuing to motivate students. It should be noted that interaction frequency has a greater influence than time on immersion [22,23].

If face-to-face CPR training is converted to remote training, instructors are likely to experience difficulties in assessing student performance in detail, and real-time communication between instructors and students will not be as smooth. High-quality remote education is not easy to achieve due to its weaker effects and poorer motivation of students [24]. According to Han’s study [25], remote skill training achieved effects similar to face-to-face education when instructor–student interactions were increased and hands-on activities were actively monitored. However, the use of multiple mobile devices increases the cognitive load on instructors, which may lower their level of concentration in student monitoring. Remote education shows promise as a useful alternative to face-to-face education if interactions between instructors and students are enhanced, and advanced technology is utilized to provide quantitative feedback on student performance. The aim of this study is to design an IoT (Internet of Things) based video platform for effective CPR skill training in remote environments, and to verify its educational effects.

2. Materials and Methods

2.1. System Design

To design the remote CPR skill training system, the following factors were considered. First, the elements were designed to provide an experience similar to that of existing face-to-face education. Second, the goals of skill training were defined, and were made achievable by students even through remote education. Third, the form of education was set in a way to maintain, as much as possible, the effects of face-to-face education. Lastly, the detailed functions required for education were specified.

2.1.1. Educational Components

• Real-time interactions between instructors and students: Clear, seamless communication between students and instructors was enabled through real-time video + voice + chatting. For this purpose, instructors’ screen sharing, and camera utilization were actively utilized. The frequency of interactions was increased by making use of video calls, chatting, and feedback conversion functions.
• Prompt, accurate feedback: Since CPR is performed on the human body, a high level of accuracy is needed for compression depth, rate, recoil, and amount of breath. A separate device for monitoring and feedback is used so that instructors can provide immediate, high-quality feedback to students [26,27].
• Debriefing of education results: Debriefing is needed to determine whether the intended goals were achieved as it is difficult for instructors to perform an in-depth assessment of students in remote locations. The app (application) should offer a comprehensive debriefing function based on quantified performance assessment data.
• Platform with less interference: A major challenge for remote education lies in networking technology, including the availability of wired/wireless connections and connecting devices. To support connections from remote users, their internet speeds and data transmission capacities must be considered. Another factor to be taken into account for real-time screen sharing is individual terminal performance. Factors interfering with instructor-student communication can be reduced through selective communication paths, set by classifying content into main content and additional content.

2.1.2. Educational Content

CPR educational content includes the basic resuscitation stage. According to adult CPR guidelines by AHA [28], the rescuer must meet the requirements for compression depth (at least 50 mm, not to exceed 60 mm), rate (100–120 compressions per minute), and complete recoil after compression (less than 10 mm). During artificial respiration, the recommended values for amount of breath (500–600 mL), rate (1 s), and hands-off time (within 10 s) are also important. In face-to-face training, feedback equipment is utilized for the quantified data debrief. Similarly, providing quantified data of similar quality in remote training will lay the foundation for high-quality education.

2.1.3. Form of Education

The form of remote education proposed in this study was designed to reflect, as much as possible, the elements of face-to-face education. The form of education is determined based on modality, pacing, student-instructor ratio, pedagogy, online role of instructors, role of students, online communication synchronization, and feedback elements [29]. The form of education selected for this study was fully online in consideration of the pandemic. The student-instructor ratio was set as 6:1 according to the standard ratio of KACPR’s (Korean Association of Cardiopulmonary Resuscitation) basic educational programs for general CPR. Instructors can actively intervene in online classes, and students can practice their skills through simulations. Communication between the two groups was synchronized to support real-time online interactions. A higher sense of immersion was achieved using simulation equipment and two-channel feedback from instructors.

2.1.4. Specific Functions

The app plays the role of supporting CPR training between students and instructors in the virtual classroom. As such, it is extremely important to understand the requirements of users in the two groups. Two KACPR instructors were invited to participate in function development. They observed a KBLS (Korean Basic Life Support) session, and then determined requirements related to core educational content, process, communication, and other matters in need of support. Specific functions were developed by classifying their comments into usage, organization, general functions, special functions, and other effects as shown in Figure 1.

2.2. System Development

The proposed system consists of an IoT device used to measure students’ performance metrics, a remote platform for real-time video and data sharing, and an application run on user terminals.

2.2.1. Connection with IoT Device for Performance Metric Measurement

The purpose of the hardware feedback device is to deliver details of student performance from remote locations to instructors in real-time. To receive feedback on the performance metrics of students in remote locations, a feedback device capable of wireless data transmission was required. The device selected for this study was I.M.LAB Inc’s AoK (Add on Kit) [30]. The selected kit was installed in CPR mannequins. Real-time measurements for chest compression and artificial respiration performed by students were obtained, and 40 data sets per second were sent to the student app via BLE (Bluetooth Low Energy). It relies on a ToF (Time of Flight) sensor to measure real-time distance (at least 15 fps) for compression values, and a barometer to measure changes in air pressure within the mannequin’s lung bags for breathing values. AoK’s accuracy and reliability have been certified by a public certification institute. The AoK app was utilized in face-to-face training. The app was used to send real-time data on compression depth, rate, and recoil to students’ mobile devices.

2.2.2. Remote Education Platform Design

The platform enables real-time exchange of video and voice data of instructors and students, as well as sensor data sent from student devices as shown in Figure 2. To minimize the influence of user’s network environment on data quality, Server 1 using Amazon SDK [31] is used for real-time exchange of high-quality video data obtained from user terminals and cameras, and Server 2 using Firebase SDK [32] sends and receives feedback data in smaller file sizes (numbers, texts, low-quality images, etc.). In the event of poor network connectivity or multiple connections exceeding the server capacity, data size is adjusted (resolution, cell size, etc.) in real-time to lessen the burden on the network.

2.2.3. Software Development for Students

In the implemented iOS and Android software for the students: (1) Real-time feedback based on the data sent from the sensor module is displayed on the students mobile device through a wireless (Wi-Fi, BLE (Bluetooth)) approach, (2) Such training feedback data, voice and the display of the students will be transferred to the software for the instructor and also the voice and the display of the instructor will be shown to the students simultaneously, (3) After the training is completed, the students can receive video and voice feedback from the instructor along with quantified training data debrief (Figure 3).

For the student application, a module was developed to allow real-time data exchange between the sensor kit and student terminal, which are paired via BLE. The algorithm used in the application derives reference data by processing raw data received from the sensor kit, and this is processed again for visualization of compression data and breathing data. In addition, an evaluation algorithm was developed to convert training data into comprehensive session results and debrief data.

2.2.4. Software Development for Instructors

In the implemented iOS and Android software for the instructor: (1) Students’ information and the voice and the display will be shown, and the instructor can monitor the students’ CPR performance in real time and provide feedback, (2) Through real-time performance data sent from the students’ mobile device connected, the instructor can assess the students’ performance in quantified manner. Considering the cognitive data processing limits the instructor might have during such process, (3) Different data interfaces can be selectively displayed depending on the needs of the instructor, by separating the phase of showing summarized assessment data derived from many different students comprehensively and the phase of showing detailed assessment data derived from a specific individual student (Figure 4).

The instructor application had the same module as the student application for video and real-time data output, and the interface was designed to support simultaneous display of multiple students. Instructors can give commands in the application to start a session by creating a virtual classroom and remotely control the student application screen. They can teach students in remote locations, following the process of step-by-step learning, evaluation, and debrief.

2.3. System Validation

2.3.1. Kano Model System Design

An experiment was carried out on the prototype using the Kano model [33,34,35,36] to assess user requirements and level of satisfaction. A survey was developed for respondents to rate each function through a pair of questions, one positive and one negative. The respondents have to choose one of five categories, “Must-be”, “One-dimensional”, “Attractive”, “Indifferent” or “Reverse”, when answering the questions (Figure 5). “Must-be” features are expected features that do not enhance satisfaction but result in dissatisfied users if not provided. “One-dimensional” features enhance satisfaction if provided, and cause dissatisfaction otherwise. “Attractive” features are unexpected features that enhance satisfaction, but do not lead to dissatisfaction even if unfulfilled. “Indifferent” features have no influence on satisfaction levels. Lastly, “Reverse” features cause dissatisfaction if present, and improve satisfaction if removed.

2.3.2. Subjects and Experimental Environment

The experimental subjects were four instructors affiliated with KACPR and 24 voluntary participants. The instructors were recruited through a private institute, and the call for general participants was made in cooperation with a startup incubating space for IT startups. The student–instructor ratio was 6:1, and the subjects were divided into four groups. The curriculum, except for breathing, was based on BLS (Basic Life Support), and classes were conducted remotely using the prototype app. Finally, the questionnaire was sent to a total of 28 participants and instructors. Questionnaires were distributed via text message to participants for ensuring their anonymity and compliance with the General Data Protection Regulation.

2.3.3. Experimental Results

The results showed that more than half of the app functions were classified as “Attractive” by the subjects. Real-time feedback functions such as compression depth and rate were classified as “One-dimensional. The “Reverse” features were login and profile among students, and video on/off among instructors as shown in

Table 1. Questions for Students (n = 24).

Questions	Category	Satisfaction	Dissatisfaction
1. I use a cell phone during the training.	A	0.64	−0.09
2. I use the app during the training.	A	0.57	−0.09
3. I use 1:1 full screen when interacting with the participant.	I	0.50	−0.18
4. I could see the faces of all participants.	I	0.32	−0.05
5. The main screen provided detailed performance feedback.	A	0.57	−0.13
6. I could select the participant slot to make full screen.	I	0.17	−0.09
7. More participants could join the session.	Q	0.23	−0.15
8. It is better to have real-time feedback, rather than the video.	I	0.38	−0.19
9. There was voice on/off feature.	A	0.67	−0.33
10. There was video on/off feature.	I	0.28	−0.11
11. There was login, profile features.	R	0.44	−0.22
12. There was session logging/stop feature.	A	0.57	0.00
13. There was screen sharing/stop feature.	A	0.57	−0.26
14. The compression depth was shown as a real-time graph.	O	0.92	−0.67
15. The compression rate was shown as a real-time graph.	A	0.92	−0.38
16. The recoil feedback was shown in real-time.	A	0.79	−0.29
17. The hands-off time value was shown in real-time.	A	0.73	−0.18
18. The performance assessment value was shown.	A	0.74	−0.26
19. There were data transfer (message, e-mail, etc.)/export/save features.	A	0.61	−0.04

and

Table 2. Questions for Instructors (n = 4).

Questions	Category	Satisfaction	Dissatisfaction
1. I use a cell phone during the training.	A	0.00	0.00
2. I use the app during the training.	A	0.33	0.00
3. I use 1:1 full screen when interacting with the participant.	I	0.50	−0.25
4. I could see the faces of all participants.	M	0.25	−1.00
5. The main screen provided detailed performance feedback.	R	0.00	0.00
6. I could select the participant slot to make full screen.	A	0.50	−0.25
7. More participants could join the session.	Q	0.00	0.00
8. It is better to have the real-time feedback, rather than the video.	R	0.00	0.00
9. There was a voice on/off feature.	A	0.75	−0.50
10. There was a video on/off feature.	R	0.00	0.00
11. There was a login, profile features.	A, I	0.50	0.00
12. There was session logging/stop feature.	A	0.75	0.00
13. There was screen sharing/stop feature.	A	0.75	0.00
14. The compression depth was shown as a real-time graph.	O	0.75	−1.00
15. The compression rate was shown as a real-time graph.	O	0.75	−1.00
16. The recoil feedback was shown in real-time.	O	0.75	−1.00
17. The hands-off time value was shown in real-time.	A	0.75	−0.50
18. The performance assessment value was shown.	A	0.75	−0.50
19. There were data transfer (message, e-mail, etc.)/export/save features.	A	0.50	−0.50

. In addition, for instructors, the display of each student’s screen in a group session was classified as “Must-be”. The features classified as “Attractive” were the provision of debriefing reports and other convenience features, and those classified as “One-dimensional” were feedback functions for compression depth and rate, which must be included in CPR training. The coefficient indicates a clear priority among the app features evaluated in the study [37]. The coefficient consists of a positive value which is a satisfaction value and a negative value which is a dissatisfaction value, and the coefficient indicates satisfaction or dissatisfaction with the presence or absence of a feature. In the formular, the positive values indicate the satisfaction when a requirement is met, and negative values indicate dissatisfaction when requirements are not met. The coefficient is calculated as ‘Satisfaction = (A + O)/(A + O + M + I)’ and ‘Dissatisfaction = −(O + M)/(A + O + M + I)’ (adapted from the study by Berger et al. [38]).

Other open-ended questions (

Table 3. Questions for Comment Fields.

Questions
1. What is your general impression of the app?
2. What are the advantages of this type of remote education?
3. What are some improvements that can be made to this type of remote education?
4. What did you find inconvenient when using the smartphone/equipment for remote education?
5. Do you have specific opinions about the app?

) provided meaningful insights. Both instructors and students saw the utilization of feedback equipment during remote education as an advantage (

Table 4. Theme of Comment Fields.

Theme	Frequency	Instructors, %	Instructors, n = 4	Students, %	Students, n = 24
1. The real-time feedback feature was useful.	13	75%	3	63%	10
2. The initial settings such as setting the app and connecting the devices were complicated.	10	25%	1	38%	9
3. More detailed guidance would be needed.	6	25%	1	29%	5
4. Technological challenges (network environment, disconnection, connection being unstable).	6	75%	3	13%	3
5. It is easy to use and useful in remote environment.	6	50%	2	17%	4
6. Test features and detailed scoring were useful.	3	25%	1	8%	2
7. Errors occurred while using the app.	3	50%	2	4%	1
8. The app is easy to use after the initial settings.	1	0%		4%	1

). The instructors identified network instability as a weakness, while the students mentioned device connection and initial configuration issues.

2.3.4. System Improvement

We made improvements to the system based on feedback. “One-dimensional” features were functions that users expect the app to have, and the number of “Attractive” features are proportionate to product competitiveness. As such, system improvement was focused on “One-dimensional” and “Attractive” features. The functions perceived as inconvenient by students, specifically, login and profile, were replaced with an independent QR code. This helps students to access virtual classes more easily since they are not required to enter information. Because the instructors wanted the app to display the faces of all attending students, the default connection of all students was switched to video mode. The video on/off function was disabled to keep the video on at all times, allowing instructors and students to interact continuously with one another. Connectivity issues were addressed by designing the system to monitor delays or interruptions in screen information transfer, and to immediately re-connect after disconnection.

3. User Study

3.1. Study Design

To test the educational effects of the proposed system, a randomized controlled trial was designed, with the independent variable being the training effects of face-to-face training and remote training. The dependent variables were the five quantitative metrics collected from feedback devices in the evaluation stage and survey data. Quantitative measurements were obtained for mean compression depth, mean compression rate, correct compression rate, correct compression depth, correct recoil through feedback devices, and artificial respiration was excluded due to COVID-19. According to the general CPR guidelines of KACPR, the ideal values of the individually obtained metrics are at least 50 mm for compression depth, 100~120 CPM (Compression Per Minute) for compression rate per minute, and 10 mm for complete recoil after compression. Relevant data is automatically collected during the evaluation of face-to-face and remote training. The survey was conducted over two sessions. The pre-education survey was used to determine the subjects’ age, gender, experience in BLS training or education, e-learning experience, IT familiarity, and perception of remote training (for the remote group). The post-education survey was a USE survey on a 7-point Likert scale [39], and covered perception of remote training after the program and other opinions. The survey was administered via Google Forms [40].

3.2. Subjects

The subjects were recruited from among the general public in cooperation with a Korea-based fire station and a non-profit institute specializing in emergency care. The four instructors had at least 5 years of experience in BLS teaching, but none in remote education. The students were in their 20s to 40s, did not receive CPR education in the past six months, and had majors unrelated to healthcare. As shown in

Table 5. Demographics of Participants.

Student (n = 48)				Instructor (n = 4)
Division	F-Group 1 (n = 24)	R-Group 2 (n = 24)	p-Value	Division	Value
Age	32 (25~47)	31.6 (27~41)	0.634	Age	40.3 (36–45)
Gender			0.810	Gender
Male (%)	13 (54.2)	14 (58.3)		Male (%)	3 (75)
Female (%)	11 (45.8)	10 (41.7)		Female (%)	1 (25)
Experience in BLS			0.229	Experience in BLS
No experience (%)	24 (100)	24 (100)		Over 5 years (%)	3 (75)
				Over 10 years (%)	1 (25)

¹ F-group: Face-to-face training group; ² R-group: Remote training group.

, 48 students were recruited and participated in this experiment.

3.3. Procedures

For the experiment, the subjects were divided into the remote training group and the face-to-face training group. The face-to-face group and remote group were each comprised of four teams of six as shown in Figure 6. The four instructors were assigned to teach one session to the face-to-face group and one to the remote group. Both groups used a mannequin equipped with a feedback device, and the students belonging to the remote training group received their mannequins by mail. The instructors and students used their own smartphones. The students in the remote group installed the remote app, while those in the face-to-face group installed the AoK app. For the remote training group, all students were given a mobile device stand to reduce limitations in posture evaluation that may come from using the front camera alone. Before the start of the session, a silhouette-like ghost image was displayed as a guide for students to place their cameras in the same position and angle.

The two groups were trained under the general BLS program, which lasted for an hour excluding artificial respiration due to the pandemic. The students in the remote group studied theories while looking at the instructor’s face on their smartphone screens. Both groups followed directions given by instructors for the hands-on practice on the mannequin. Feedback data was displayed only to instructors, and student evaluation was carried out for two minutes. After evaluation, the instructors began debriefing, and held a Q&A session. At the end of the experiment, the subjects were asked to complete and submit the survey (Figure 7).

3.4. Results

3.4.1. Compression Quality Evaluation

Table 6. Data of the two groups collected during compression-related skill evaluation.

Division	F-Group 1 (n = 24)	R-Group 2 (n = 24)	p-Value
Mean compression depth, mm	57 (45–63)	53 (43–62)	0.008
Proportion of correct compression depth, %	83 (68–96)	79 (53–93)	0.81
Mean compression rate, cpm	108 (76–136)	101 (54–133)	0.113
Proportion of correct compression rate, %	46 (31–88)	38 (33–74)	0.631
Proportion of correct recoil, %	82 (50–93)	73 (32–90)	0.004

¹ F-group: Face-to-face training group; ² R-group: Remote training group; Data are shown as mean (interquartile range).

shows the average data of the two groups collected during compression-related skill evaluation including Supplementary Materials Table S1. Considering the size of the two groups, which are independent samples, the Shapiro–Wilk test was employed to conduct a normality test on the average value of each item. While a normal distribution could be assumed for compression rate, equal variance through the F-test could not be assumed, and analysis was thus performed using Welch’s two-sample t-test. A normal distribution could not be assumed for the average values of the remaining four items, and a Mann–Whitney test was carried out. Statistical significance was determined based on a p-value of 0.05, and statistically significant differences were found between the two groups for ‘mean compression depth’ and ‘proportion of correct recoil’. The two groups were not significantly different in terms of ‘proportion of correct compression depth’, ‘mean compression rate’, and ‘proportion of correct compression rate’.

Looking at the analysis results, the mean compression depth of the two groups satisfied the KCPR guidelines. The face-to-face group had a slightly higher average than the remote group for ‘proportion of correct recoil’. The difference was not statistically significant due to the limited samples. The trends for ‘mean compression rate’ show that both groups had values falling in the recommended range, and the face-to-face group had a higher ‘proportion of correct compression rate’. For ‘proportion of correct compression depth’, the face-to-face group again had a higher value.

3.4.2. USE Survey Results

The results of the USE survey, conducted on instructors and students in the remote group, are presented in

Table 7. USE survey result conducted on instructors and students.

Dimension	Item	Question	Student
			Score (%)	Score (%)	Score (%)
Usefulness	U1	It helps me be more effective.	71.4	85.4	80.2
	U2	It helps me be more productive.	92.9
	U3	It is useful.	87.1
	U4	It gives me more control over the activities in my life.	91.4
	U5	It makes the things I want to accomplish easier to get done.	84.3
Ease of use	E1	It is easy to use.	77.1	74.6
	E2	It is simple to use.	65.7
	E3	It is user friendly.	74.3
	E4	It requires the fewest steps possible to accomplish what I want to do with it.	81.4
Ease of learning	L1	I learned to use it quickly.	77.1	79.5
	L2	I easily remember how to use it.	72.9
	L3	It is easy to learn to use it.	88.6
Satisfaction	S1	I am satisfied with it.	78.6	81.1
	S2	I would recommend it to a friend.	88.6
	S3	It is fun to use.	80.0
	S4	It works the way I want it to work.	77.1

. The analysis showed that students gave the highest points to the Usefulness dimension (85.4% by students), and the lowest to Ease of use (74.6% by students). Within the Usefulness dimension, the students gave the highest score to “It helps me be more productive (92.9%). All students saw remote education as allowing them to save time and effort since they could attend classes at a place of their convenience.

The question on perception of remote education received different responses before and after the experiment. In the remote group, the percentage of students with a positive image of remote education increased from 49% to 60%. This also increased for instructors from 43% to 78%. The results indicate that the experience of the remote education program positively influenced the subjects’ perception.

The subjects were asked to provide other related opinions to help improve the system. Students in the remote group felt that there were insufficient interactions with instructors, but this was not mentioned by the instructors. Both instructors and students picked device connectivity and network settings as areas for improvement. The students needed more time to prepare as they experienced difficulties in initial connection, and the instructors said that teaching was interrupted by connectivity issues.

4. Discussion

4.1. Principal Findings

The purpose of this experiment was to determine whether the proposed system can serve as a useful alternative to traditional face-to-face training. The system was tested using the Kano model, and the differences in quantitative evaluation across the two groups were analyzed through a user test.

The hands-on skill practice was observed while conducting the experiment, and interviews were carried out at the end. The two groups did not show a significant difference in correct compression depth, but the face-to-face group had slightly higher values for correct recoil. Although not of statistical significance, both groups had ‘mean compression rate’ satisfying the guidelines, and the face-to-face group had slightly higher values than the remote group in terms of ‘proportion of correct compression depth’ and ‘proportion of correct compression rate’. The remote group utilized feedback equipment and managed to achieve performance levels similar to that of the face-to-face group, but the performance gap was broader among the remote group than the face-to-face group. That is, instructors in the remote environment are likely to have experienced difficulties in accurately assessing the performance of individual students. It appeared that the instructors could monitor student performance in compression rate and recoil more rapidly in the face-to-face environment. On the other hand, in the remote environment, instructors were unable to closely examine individual student performance due to the limited viewing angle of smartphones and network delays.

It appeared that the instructors could monitor student performance in compression rate and recoil more rapidly in the face-to-face environment. On the other hand, in the remote environment, instructors were unable to closely examine individual student performance due to the limited viewing angle of smartphones and network delays. Moreover, the app provided quantitative feedback in the order of depth, rate, and recoil, causing slower monitoring of the latter two. This shows that instructors still face limitations in visually assessing students in the remote environment. Further research is needed to improve the method of feedback provision on the interface. Compression depths were similar between the two groups because the general program requirement of at least 50 mm can be easily fulfilled. Considering the existing online CPR skill training programs without feedback equipment, the proposed system is a significant improvement.

Looking at the USE survey, the subjects in the remote training group experienced less time and space constraints. In addition, the instructors replied that remote education serves as a good alternative to face-to-face education in the event of a crisis, like the pandemic. Despite such advantages, usability issues remain to be addressed. Sufficient support should be provided for students to grow familiar with the new learning environment (how to switch on the camera and microphone, how to connect the feedback device to the smartphone, etc.) and overcome barriers posed by electronic devices and IT. Instructors should also receive support and training to facilitate remote teaching.

The change in perception towards remote education revealed through responses obtained before and after the experiment hint at a positive future for remote education. Remote training is less preferred in CPR skill education because providing accurate guidance is difficult, and those who have not received proper training will be less capable of rescuing lives in emergency situations. Given the conservative nature of medical skill education, integrating new approaches can be quite challenging. Regardless, the experimental results indicate that new forms of education may be possible if instructors’ perception of remote education changes from negative to positive.

4.2. Implicit Findings

As described in the above research procedures, KACPR instructors were interviewed on the functions expected of the app if they were to teach CPR through remote classes, and the app was designed to meet such expectations. In testing the system design of the prototype, the following discoveries were made.

First, during design, the instructors were asked whether they needed to teach multiple students at once using the remote app. For general skill training, they answered that the ideal number of students in each virtual session would be 12. As such, the system was designed to support up to 12 simultaneous connections. The experiment was conducted with six students per instructor, and the survey showed that all instructors chose “Questionable” for teaching more than six students. This is consistent with the George A. Miler model [41], which states that five to six students are cognitively acceptable.

Second, during function development, the instructors were asked to rate the importance of video data and feedback data. The instructors felt that real-time feedback data was more important than video in order for them to provide accurate guidance. The app was designed to allow instructors to selectively view the screen of a student by clicking on his/her cell when poor feedback is observed. However, in the design verification experiment, the instructors preferred to view the screens of all students because they could not fully grasp student performance based on feedback data alone.

Third, the instructors wanted the feedback data displayed on their screens in real-time to be as detailed as possible. The app was designed to provide detailed feedback data of all students connected to the virtual class. However, most of the instructors who participated in the design verification classified it as “Reverse” in the Kano model. Providing the instructors with too much information is likely to have increased their cognitive load, and they would not have been able to keep track of all values on the screen while teaching.

Lastly, the instructors said that session logging was an important function but classified it as an “Indifferent” feature in the Kano model. When it came to ensuring the effectiveness of education for the general public, most instructors felt that providing real-time feedback to students in various network environments was more important than data storage or management.

The above opinions are difficult to generalize as they were obtained from a small number. Even so, it is necessary to determine the causes for the change in perception of instructors, who had sufficient experience in CPR training, before and after the experiment. The biggest reason is presumed to be the higher cognitive load experienced by instructors, who have more to manage and pay attention to in online education than face-to-face classes. For example, showing the feedback data of all students on the screen of a small mobile device results in significant fatigue. The instructors found the app more comfortable when information was presented in intuitive colors or semantically. Some of the functions identified as essential by the instructors during the interview were not used in reality. This was because the instructors were recruited based on their CPR education experience, not their experience in remote education. The impact of online education on the elements necessary for general CPR education may not have been sufficiently considered. Thus, when designing functions related to CPR and other medical skill education, the perspectives of various users must be taken into account.

5. Conclusions

The significance of this study lies in being the first to systematically design and test an IoT-based remote education system for medical skill education in times of COVID-19. In particular, to design a robust and effective system, rapid prototyping and small expert group validation were performed. After that, we conducted a main experiment comparing the face-to-face training with the improvement system reflecting the results of the prototype test. As a result, the newly designed remote system succeeded in addressing the limitations of existing online education and enhanced the effectiveness of remote education to a level similar to face-to-face education. This means that there is a possibility of expanding to new forms of education in the future.

The limitations of this study are as follows. The experiment was conducted in 2021, when Korea was most impacted by COVID-19 and a majority of CPR education was suspended in accordance with the government’s disease control measures. Recruiting subjects was difficult because of the restriction on private gatherings, and medical personnel could not participate as they had to minimize external contact. The experiment was carried out with the help of an institute providing general CPR training for the public, and the subjects were adults in their 20s to 40s. The results of the experiment are thus difficult to generalize, and may not be the same for the entire population. While the proposed system was primarily focused on the educational achievements of students, it was difficult to examine the effectiveness of teaching for instructors due to the limited sample size. Further verification using a larger sample is required on teaching effectiveness from the perspective of instructors.

Based on the results, we plan to conduct follow-up research. First, we will develop an improved system in terms of usability and stability using feedback obtained in this study. The effectiveness of the system will be tested on a broader sample, comprised of general members of the public and medical personnel in various age groups. In the long term, the study will be expanded to remote IoT based learning [42] and e-assessment [43] for medical skills other than CPR.

The need for alternative skill training was highlighted by COVID-19, which caused a suspension of all daily activities. The educational environment underwent rapid changes, and new approaches had to be developed even though established criteria were unavailable. Despite the limitations of this study, we expect our findings to serve as a valuable reference for related research, and to ultimately contribute to the establishment of new norms for remote education. In the process, we hope for medical and healthcare capacities affected by the pandemic to gradually be restored.