Effects of the International Training Program for Enhancing Intelligent Capabilities through Blended Learning on Computational Thinking, Artificial Intelligence Competencies, and Core Competencies for the Future Society in Graduate Students

Abstract

Background: The purpose of this study is to find the effects of the international training program for enhancing intelligent capabilities through blended learning on computational thinking, artificial intelligence (AI) competency, and core competencies for the future society in graduated students enrolled in the Smart Information Communication Technology (SMART ICT) course. The teaching model followed the ADDIE framework. Methods: This study is a quasi-experimental study based on nonequivalent control group design. Study subjects were assigned to an experimental (n = 20) or control group (n = 20). The experimental group participated in the international training program in the blended learning form, real-time online classes (60 min per session for a week, six sessions) and face-to-face classes (4–8 h per session for 9 days, six sessions). The variables were measured with a self-report questionnaire and were evaluated before, right after, and in the 12th week of the program. Results: The AI competency of the experimental group was observed to be significantly changed at the points of time (F = 6.76, p = 0.002), and in comparison with that of a different group (F = 9.77, p = 0.003). Conclusions: This study suggests applying an international training program based on blended learning to strengthen intelligence capabilities such as artificial intelligence capabilities.

1. Introduction

With the advancement of computer devices and the development of software and information and communications technology (ICT), collectable data have been on the explosive rise. Today, in various fields of society, such information is used, and many kinds of intelligent activities come to be automated [1]. In order to keep up with social changes and needs, it is important to train people of talent who have the ability to solve problems on the basis of AI and software [2]. Computers are being applied in various ways across all fields, including language, mathematics, medicine, management, law, politics, and the arts [3]. At this time, a capability is required to extract the key elements of a problem to solve and to automate them with a computing device [4]. That is computational thinking, which is an efficient and systematic problem-solving competency applied to all fields of study [5]. This competency is very suitable and essential to people of talent in the present age when industrial convergence actively appears [4,6].

In line with the advent of the intelligent information society, the Korean government implements software education as part of elementary, middle, and high school curricula, and has been enhancing the software competency of not only software majors but all students through the Software-Centric University Project since 2015 [7,8,9]. As a result, there has been increased awareness of software education and continuous improvement in related curriculums and teaching methods [10,11]. A variety of software learning and experience offered by a university give non-major students confidence, and thus there have been more non-major students who apply for plural majors based on software through a double-major or interdisciplinary-studies system [1,10,12]. Recently, the Ministry of Education announced the introduction of ‘artificial intelligence (AI) education’ in the 2022 Revised Curriculum scheduled to have been applied since 2025 [13]. This education contains a variety of contents, including the understanding and experience of the AI concept and principles, problem-finding through the sharing and analysis of social phenomena, creative problem solving using data and statistics, and AI ethics of its effects on society [14,15,16]. Therefore, it is necessary to provide the subject-based interdisciplinary education to solve various problems by utilizing AI and software. To do that, AI education in a university should focus on the active operation of the programs to find and support student-led activities, rather than the school-led AI and software programs performed a single time [1]. In order to create the practical effects of education, it is required to develop the basic AI and software education program to enhance the associations of learners’ different departments and majors in a high educational environment. To do that, it is a prerequisite to analyze learners by school [17,18,19]. In addition, it is required to raise experts or enhance related competencies not only in domestic curricula but through an international training program, and to give students a change in interdisciplinary studies in the way of exploring different curricula or education methods in other countries [14,20]. Since the 2000s, universities have received new technologies and have exchanged information immediately due to globalization. As a result, their communication has become more efficient, and members of universities have built more collaboration in research and education fields through an international network [21,22]. However, there are not many cases where the international training programs and field experience aimed at undergraduate students or graduate students are objectively analyzed in comparison; their actual conditions or characteristics are determined and used for the expansion of the next international training programs [21].

Blended learning is used to explain learning that mixes various event-based activities, including face- to-face classrooms, real-time or live e-learning, and self-lead learning. NIIT categorizes that the blended learning model is skill-driven learning, attitude-driven learning, and competency-driven learning [23,24,25,26]. In particular, skill-driven blended learning is a useful learning method for developing specific knowledge and skills by combining self-directed learning with instructor support. This study involves the international training program for enhancing intelligent capabilities, seeking to apply blended learning, which is a mixture of real-time classes and face-to-face classes. Real-time classes feature facilitator interaction through discussion forums, instructor overviews, and feedback. And a traditional classroom of face-to-face classes are interactive through a system of instructor outlines, demonstrations, discussions, and feedback [23,24,25,26]. By applying these learning methods, we aim to confirm changes in computational thinking, artificial intelligence, and core competencies.

The ADDIE model consists of five stages of analysis, design, development, implementation, and evaluation, and each process is organically related. Because each step is structured in a detailed and specific logical order and includes important processes for successful achievement of the purpose, instructors should design classes by considering the interdependence between all elements. The five steps can be repeated during the program, so it is cyclical and has experiential characteristics because it allows effective decision making according to the situation based on various data. Above all, the structure and operation method of the ADDIE model is structured to meet various educational requirements in not only traditional education but also real-time online class environments, making it very useful in designing and evaluating learning experiences, courses, and educational content. Therefore, this study involves the international training program for enhancing intelligent capabilities, and it is meaningful to confirm the educational effect by applying a blended learning design based on the ADDIE model [27,28,29].

As the competencies for training people of talent who can respond to changes in the fourth industrial revolution and of the human resources needed in the hyper-connection smart industries, critical thinking, communication skills, creativity, and collaboration ability are suggested. These competencies are not only key competencies for the future society, but are significant competencies for majors. They are considered highly important in a future society [23,24]. Accordingly, this study tries to measure the effectiveness of the international training program for enhancing intelligent capabilities and to provide fundamental material for the expansion of the international training program and its improvement.

The details of the study objectives are as follows:

• First, verify the effect of the international training program for enhancing intelligent capabilities on the subjects’ computational thinking.
• Second, verify the effect of the international training program for enhancing intelligent capabilities on the subject’s artificial intelligence competency.
• Third, verify the effect of the international training program for enhancing intelligent capabilities on core competencies for the future society.

The research hypotheses of this study are as follows:

H1: 

The experimental group participating in the international training program for enhancing intelligent capabilities will score higher in computational thinking than the control group that did not participate.

H2: 

The experimental group participating in the international training program for enhancing intelligent capabilities will score higher in artificial intelligence competency than the control group that did not participate.

H3: 

The experimental group that participated in the international training program for enhancing intelligent capabilities will score higher in future society core competency scores than the control group that did not participate.

2. Materials and Methods

2.1. Study Design

This is a quasi-experimental study based on nonequivalent control group design in order to examine the effects of the international training program for enhancing intelligent capabilities on computational thinking, AI competency, and core competencies for the future society.

2.2. Study Subjects and Sampling Method

The accessible population of this study consisted of graduate students participating in the course of Smart Information Communication Technology (SMART ICT) Employees in a university situated in D city. The selected study subjects were those who understood the content of the questionnaire, directly filled in the questionnaire or were able to reply in writing, never participated in any international training program similar to that in this study, had over 80% of attendance, understood the purpose and procedure of this study, and hoped to make volitional participation. The subjects not selected were those who were absent from the program at least twice, who were participating in any similar program in school, and whose insufficient answers to questions in the questionnaire led to a failure of use as research data.

The sample size of study subjects was calculated using G power 3.1.2 based on the effect size (f² = 1.0), significance level (α = 0.05), and power (1-β = 0.80). The required sample size for each group was determined to be 17 participants. In consideration of a dropout rate, 20 participants were assigned to both the experimental and control groups. All participants successfully completed the program and responded to both pre- and post-surveys. As a result, the dataset of 40 participants was used for a final analysis.

2.3. Study Tools

2.3.1. Computational Thinking (CT)

Computational thinking refers to the procedural thinking to solve problems efficiently according to the fundamental concepts and principles of computers [25]. Computational thinking was measured with the use of the tool developed by Hong et al. [10]. This tool consists of a total of 17 items, 4 sub-factors (decomposition, pattern recognition, abstraction, algorithm), and a Likert 5-point scale. A higher score indicates a higher level of computational thinking. In the study by Hong et al. [10], Cronbach’s α was 0.972, and the reliability of each sub-factor was decomposition, 0.928; pattern recognition, 0.902; abstraction, 0.890; and algorithm, 0.928. In this study, Cronbach’s α was 0.978, and the reliability of each sub-factor was decomposition, 0.937; pattern recognition, 0.919; abstraction, 0.921; and algorithm, 0.933.

2.3.2. Artificial Intelligence (AI) Competency

AI competency refers to the ability to solve problems based on the understanding of artificial intelligence [15,16]. AI competency was measured with the use of the tool developed by Yoo et al. [15]. This tool consists of a total of 17 items, 4 sub-factors (knowledge representation and reasoning, data understanding and learning, machine learning, deep learning, AI ethics), and a Likert 5-point scale. A higher score indicates a higher level of AI competency. In the study by Yoo et al. [15], Cronbach’s α was 0.960. In this study, Cronbach’s α was 0.977, and the reliability of each sub-factor was knowledge representation and reasoning, 0.901; data understanding and learning, 0.907; machine learning, 0.878; deep learning, 0.957; and AI ethics, 0.910.

2.3.3. Core Competencies

The core competencies for future talents refer to the ones considered by the Ministry of Education in the 4th Industrial Revolution Innovative Leading University Project, representing critical thinking, communication skills, creativity, and collaboration. In this study, these core competencies were measured with the use of the tool for measuring the core competencies of future talents of the Innovative Curriculum-based Core Competency Scale developed by Kwon [30]. This tool consists of a total of 19 items, 4 sub-factors (critical thinking, communication skills, creativity, collaboration), and a 7-point Likert scale. A higher score indicates higher levels of critical thinking, communication skills, creativity, and collaboration, which fall under the innovative curriculum-based core competencies. In the study by Kwon [30], Cronbach’s α was 0.85. In this study, Cronbach’s α was 0.976, and the reliability of each sub-factor was critical thinking, 0.942; communication skills, 0.899; creativity, 0.936; and collaboration, 0.916.

2.4. Study Intervention

The Design Model of the International Training Program for Enhancing Intelligent Capabilities

The ICT, which stands for Information Communication Technology, represents information technology and telecommunication technology. It encompasses the software technology required to operate and manage information devices such as computers, media, and audiovisual equipment, as well as all methods of collecting, producing, processing, preserving, transmitting, and utilizing information using such technology [26]. To cultivate talents who have problem-solving skills based on artificial intelligence and software, it is crucial to enhance their practical competencies. The international training program for enhancing intelligent capabilities is able to provide an environment where learners actively participate in practical work and can proactively solve problems. In this study, the instructional design for the international training program is based on the ADDIE model in order to maintain an organic relationship in each stage and providing a field-based program that reflects learners’ needs [27,28,29]. This instructional model consists of five stages: analysis, design, development, implementation, and evaluation (See

Table 1. The instructional model.

Analysis	Design	Development	Implementation	Evaluation
○ Curriculum Analytics ○ Learner Analytics ○ Environment Analysis	○ Learning Objectives Assessment ○ Tool Design ○ Media Selection	○ Teaching Materials ○ Development ○ Material Production	○ Implementation	○ Evaluation of training performance

).

• Analysis stage of ADDIE model

• Curriculum

The Department of Smart ICT Convergence at this university aims to foster core talents leading the intelligence and innovation of the main regional industries. The curriculum designed in the department focuses on information and communication technologies in the smart ICT convergence field, such as wireless communication convergence, AI and big data, cyber security and XR, bio-digital, and intelligent robotics [31]. The international training program for enhancing intelligent capabilities in this provides various opportunities to engage in joint R&D convergence projects with enterprises through the connected activities with global specialized companies or overseas universities with excellent research capabilities. In this way, this program enhances the department’s core competencies: practice and challenge, creative convergence, and problem-solving ability.

2.

Learners’ characteristics

Participants in the Smart ICT Convergence Department at this university are employees working at local companies, who are responsible for various tasks related to artificial intelligence, big data, drones, and more. Therefore, they already feel the necessity of IT convergence and have the basic knowledge and qualifications. Nevertheless, in order to perform R&D convergence projects with companies, it is necessary to explore ways to elevate the understanding, application, utilization, and evaluation levels of the latest technologies among the five main information and communication technologies mentioned earlier [32]. This international training program for enhancing intelligent capabilities encourages participants to freely draw topics in a creative problem-solving way and solve problems through discussion [19]. Participants’ active engagement is essential in this program.

3.

Environment analysis

The process of creative problem solving can stimulate learners’ practice and challenge, creative convergence, and problem-solving ability [33]. Therefore, it should consist of three steps. The first step involves identifying problems and topics and conducting education based on skill levels or areas of interest. The second step includes classifying skill levels by major, providing pre-training, and conducting in-depth learning for creative problem solving. The third step is the evaluation phase, where participants can share their experiences through a creative problem-solving competition.

• Design stage of ADDIE model

• Definition of learning objectives

This program aims to increase learners’ understanding levels in order to improve their knowledge about the understanding, application, utilization, and evaluation of the latest technologies in the ICT field, focusing on enhancing their understanding, application, utilization, and evaluation of these technologies; to find solutions to business problems; or to create new ideas by utilizing the knowledge.

2.

Teaching–learning process plan

This program was designed to apply the blended learning education approach reflecting both real-time online classes and face-to-face classes, and to be divided into general and advanced courses according to learners’ levels [24,25,32]. The real-time online classes have six sessions over one week, each of which lasts 90 min. The face-to-face classes have six sessions over nine days, each of which lasts 4–8 h. Real-time online classes include lectures and presentations from U.S. professors and experts in the ICT industry, along with discussions. The face-to-face classes comprise special lectures and presentations from U.S. professors and experts, group discussions, one-on-one feedback on technological challenges, field trips related to the major, cultural exploration, and individual project presentations for creative problem solving. The detailed teaching–learning process plan is shown in Figure 1.

3.

Learning environment plan

The real-time online classes of this program utilize Zoom Video Communication to create a real-time environment for interactive discussions among professors, experts in the industry, and learners. Learners are formed into small-sized groups. Individuals facing similar problems or topics are grouped together for effective presentations and discussions [24,25,32].

Of the face-to-face classes, the field trip component involves planning visits to various local community organizations, companies, universities, and exhibitions in the United States that have close relevance to information and communication technology. This allows participants to interact with local experts and explore equipment, facilities, organizational culture, and more. Safety plans for participants during field trips are also prepared [24,25,32]. For expert lectures, presentations, and discussion sessions, the classrooms are set up with structures conducive to two-way communication, such as the tables arranged for effective teaching, presentations, and discussions. Cultural exploration activities were planned to enhance the understanding of American culture.

4.

Assessment plan

As for the assessment, an individual evaluation was planned and implemented. The evaluation was made by professors and experts. The evaluation criteria included the application level of knowledge for overcoming technological challenges or solving research problems, problem-solving abilities, presentation skills, and other multifaceted aspects.

• Development stage of ADDIE model

• Development of problems

The main education contents in this program include wireless communication convergence, AI and big data, cyber security and XR, bio-digital, and intelligent robotics [30]. This education program was developed together with two computer engineering professors and two industry experts in order to enhance its validity.

2.

Development of evaluation tool

The evaluation was determined to be based on a report. More specifically, the achievement (completeness) related to the resolution of technical difficulties and issues in line with the training objectives is evaluated. In addition, the quality of the report, presentation skill, and other relevant factors is evaluated. Individual assessments are conducted for each participant.

• Implementation stage of ADDIE model

In this training program, the 1-week real-time online classes had 6 sessions, each of which lasted 90 min; the face-to-face classes took place in the United States over 9 days with 6 sessions, each of which lasted 4–8 h. The detailed class schedule for each day is shown in Appendix A. The real-time online classes covered such topics as ‘artificial intelligence, intelligent robots, and human-centered approaches in AI’. The face-to-face classes included ‘seminars on Data Science, Machine Learning, and AI, the understanding of U.S. patented technologies and IT trends in the US’, as well as general lectures and expert special lectures on ‘transformers, IT project management in the U.S., big data analysis’, and more. Field trips included visits to two U.S. universities related to ICT, system resource computer exhibition, WireBarley, and various local community institutions in the U.S. To enhance the understanding of U.S. culture, which is helpful for development in the ICT field, cultural exploration was incorporated into the program. The final session included individual project presentations, discussions, and an award ceremony to foster creative problem-solving skills.

• Evaluation stage of ADDIE model

As for the evaluation in this program, individual assessments were carried out, involving both report writing and presentations. The evaluation of creative problem solving was conducted individually. It focused on the ability to overcome technological challenges and solve problems in line with the final objective of this global program. The evaluation of the ability to solve technological difficulties and problems was based on the feasibility of the content described in the first report and the verbal skills during the second presentation. The summary of main concepts was also included as an assessment criterion to check the understanding of the program’s objectives.

2.5. Statistical Analysis

The collected data were analyzed with the use of SPSS/WIN 20.0. The normality assumption of dependent variables was examined through the Kolmogorov–Smirnov (K-S) test, histograms, skewness, and kurtosis. As a result, the score for computational thinking right after the intervention and the score for core competencies in the 12th week of the intervention failed to meet the normality assumption (D range of K-S test: 0.15, p < 0.05; 0.15, p < 0.05), but other variables satisfied the normality assumption (D range of K-S test: 0.07~0.13, p > 0.05). The results indicated that the normality assumption was not met for the scores for computational thinking immediately after intervention and core competencies 12 weeks after intervention (K-S test’s D range: 0.15, p< 0.05; 0.15, p < 0.05). However, other variables satisfied the normality assumption (K-S test’s D range: 0.07 to 0.13, p > 0.05). The skewness values for the three variables ranged from −0.82 to 0.06, and kurtosis values ranged from −0.31 to 1.23, all within the −2 to +2 range. Since most variables met the normality assumption, parametric statistics were used to analyze the study results. General characteristics of the participants were analyzed with the uses of a real number and percentage, and the test of homogeneity was performed through the Chi-square test and independent t-test. To verify the effects of the international training program for enhancing intelligent capabilities on computational thinking, artificial intelligence competency, and core competencies the experimental and control groups have, a repeated-measures analysis of variance was conducted. In cases where the sphericity assumption was not met, the Greenhouse–Geisser epsilon correction was applied. Group comparisons at each point of time for changes in dependent variables were analyzed with the use of an independent samples t-test.

2.6. Data Collection Method and Procedure

This study as a research project obtained approval (2023-002-001-07) from the Institutional Review Board at University C. Data had been collected from July to October 2023. It involved the international training program for enhancing intelligent capabilities of SMART ICT graduate students. The study details, including the title, purpose, method, and procedure, were publicly posted on an online bulletin board to encourage voluntary participation of graduate students.

Participants were assigned to either the experimental group (Group 1) or the control group (Group 2). In the participant selection process, there was a notification that Group 1 was determined to participate in the international training program and graduate general course of Smart Information Communication Technology (SMART ICT) for enhancing intelligent capabilities, while Group 2 was determined to participate in the graduate general course of Smart Information Communication Technology (SMART ICT) for enhancing intelligent capabilities at the university.

The SMART ICT convergence major is a field-based master’s and doctoral degree program centered on employed people. In other words, it is a general smart information and communication technology (SMART ICT) graduate school course. The educational goal is to develop practical capabilities that can be used immediately by carrying out projects. The curriculum designed in the department focuses on information and communication technologies in the smart ICT convergence field. A total of 24 credits or more must be completed over 2 years, and the curriculum consists of an in-depth major course (12 credits or more) and a convergence project (12 credits or more).

This international training program for enhancing intelligent capabilities encourages participants to freely draw topics in a creative problem-solving way and solve problems through discussion. The global program is designed based on the ADDIE model and classes are operated in a branded manner. This program was designed to apply the blended learning education approach, reflecting both real-time online classes and face-to-face classes, and to be divided into general and advanced courses according to learners’ levels.

The experimental group participated in a global program using the vacation period after participating in one semester of the SMART ICT graduate school course from March to June 2023. For one week, students spent that week studying in real-time online courses with professors and experts from the United States in Korea, then immediately went to the United States and participated in a face-to-face course for 9 days to strengthen their intelligence capabilities. The control group participated in one semester of the SMART ICT graduate school course from March to June 2023.

The participants in the experimental group, who expressed their voluntary participation, took part in the pre-survey. After their attendance in the real-time online classes (90 min, 6 sessions, 1 week) and the face-to-face classes (4–8 h, 6 sessions, 9 days) and in the 12th week of the international training program, they participated in the questionnaire surveys on computational thinking, artificial intelligence competency, and the innovative curriculum-based core competencies.

The control group participated in surveys only, three times (one pre-survey, two post-surveys) (See Figure 2). For Group 1 and Group 2, an internet resource locator (URL) was posted on the online bulletin board three times and enabled them to respond to the surveys. The first page of the website accessed by the participants voluntarily shows the description of this study and whether to agree to join the study, and the next page presents questionnaire items. The participants who wanted to participate in the study voluntarily were asked to sign the agreement and respond to the questionnaire items related to demographic information, computational thinking, artificial intelligence competency, and the innovative curriculum-based core competencies. The study participants received a gift as a token of appreciation.

3. Results

3.1. The Study Participants’ General Characteristics and Homogeneity Testing

The study participants’ gender, age groups, education levels, and major satisfaction were analyzed. As a result, the two groups had homogeneity (See

Table 2. The study participants’ general characteristics and homogeneity testing (N = 40).

Characteristics	Categories	Exp. (n = 20)	Cont. (n = 20)	x2 or t	p
		n (%) or M ± SD	n (%) or M ± SD
Gender	Male	14 (70.0)	14 (70.0)	0.000	1.000
	Female	6 (30.0)	6 (30.0)
Age (year)	20s	7 (35.0%)	4 (20.0)	3.77	0.287
	30s	6 (30.0)	11 (55.0)
	40s	2 (10.0)	3 (15.0)
	50s	5 (25.0)	2 (10.0)
Education (year)	Bachelor’s degree	17 (85.0)	18 (90.0)	2.36	0.307
	Master’s degree	1 (5.0)	2 (10.0)
	Doctoral degree	2 (10.0)	0 (0.0)
Satisfaction with major	Satisfied	17 (85.0)	16 (80.0)	0.17	0.677
	Moderate	3 (15.0)	4 (20.0)
	Dissatisfied	0 (0.0)	0 (0.0)
Computational thinking		3.72 ± 0.32	3.91 ± 0.55	−1.38	0.175
AI competency		3.13 ± 0.50	3.01 ± 1.01	0.43	0.663
Core competencies		5.23 ± 0.61	5.07 ± 0.91	0.62	0.534

Exp. = experimental group; Cont. = control group; AI = artificial intelligence; M = mean score; SD = standard deviation; x² = Chi-square statistic; t = t-statistic; p = significance level; significance at p < 0.05.

). Regarding gender, the experimental group and the control group had a similar distribution, with a 30.0% female ratio and a 70.0% male ratio. In terms of age groups, the experimental group had a slightly higher proportion of participants in their 20s, accounting for 35.0%, and the control group had the majority of participants in their 30s, accounting for 55.0%. With regard to educational levels, in the experimental group, 85.0% held a bachelor’s degree, 5.0% (one participant) held a master’s degree, and 10.0% (two participants) held a doctoral degree. The control group had a similar distribution, with 90.0% holding a bachelor’s degree and 10.0% (two participants) holding a master’s degree. Regarding satisfaction with their majors, in both the experimental and control groups, over 80.0% expressed satisfaction, 20.0% had a moderate level of satisfaction, and no one was dissatisfied.

The dependent variables—computational thinking, artificial intelligence competency, and core competencies—were not significantly different between the two groups, and thus the groups were confirmed to be homogeneous (See Table 2).

3.2. Verification of the Effects of the International Training Program for Enhancing Intelligent Capabilities

3.2.1. Group Comparison at Each Point of Time for a Change in Computational Thinking

To verify the effects of the international training program for enhancing intelligent capabilities, a repeated measures analysis of variance was carried out to analyze a change in computational thinking of the experimental and control groups at each measurement point of time. The results are as follows (see

Table 3. Effects of the international training program for enhancing intelligent capabilities on computational thinking (N = 40).

Variables	Groups	Pre-Test	Post-Test	Follow-Up Test	Sources	F	p	Differences (Post–Pre)			Differences (Follow-Up–Pre)
		M ± SD	M ± SD	M ± SD				M ± SD	t	p	M ± SD	t	p
Computational thinking	Exp.	3.72 ± 0.32	3.99 ± 0.79	4.02 ± 0.55	G	0.01	0.902	0.26 ± 0.69	1.38	0.175	0.30 ± 0.54	2.12	0.040
	Cont.	3.91 ± 0.55	3.92 ± 0.60	3.83 ± 0.75	T	0.94	0.376	0.00 ± 0.48			−0.08 ± 0.60
					G*T	1.70	0.195
Decomposition	Exp.	3.78 ± 0.44	4.07 ± 0.83	4.17 ± 0.46	G	0.49	0.487	0.29 ± 0.64	1.62	0.113	0.39 ± 0.61	2.27	0.029
	Cont.	3.92 ± 0.63	3.90 ± 0.58	3.86 ± 0.77	T	1.41	0.249	−0.02 ± 0.56			−0.06 ± 0.64
					G*T	2.43	0.095
Pattern recognition	Exp.	3.71 ± 0.36	3.97 ± 0.82	4.00 ± 0.72	G	0.06	0.794	0.26 ± 0.82	0.85	0.397	0.28 ± 0.69	1.65	0.107
	Cont.	3.95 ± 0.67	4.00 ± 0.68	3.86 ± 0.82	T	0.69	0.478	0.05 ± 0.74			−0.08 ± 0.73
					G*T	0.98	0.367
Abstraction	Exp.	3.66 ± 0.38	3.83 ± 0.87	3.88 ± 0.76	G	0.18	0.673	0.17 ± 0.88	0.26	0.790	0.22 ± 0.67	1.03	0.308
	Cont.	3.82 ± 0.51	3.93 ± 0.62	3.82 ± 0.72	T	0.75	0.477	0.11 ± 0.55			0.00 ± 0.70
					G*T	0.44	0.643
Algorithm	Exp.	3.73 ± 0.44	4.08 ± 0.77	4.05 ± 0.47	G	0.19	0.664	035 ± 0.71	2.40	0.021	0.32 ± 0.58	2.33	0.025
	Cont.	3.98 ± 0.68	3.87 ± 0.62	3.80 ± 0.83	T	0.55	0.547	−0.11 ± 0.46			−0.18 ± 0.75
					G*T	2.93	0.069

Exp. = experimental group; Cont. = control group; M = mean score; SD = standard deviation; G = group; T = time; G*T = group by time; F = F-statistic; t = t-statistic; p = significance level; significance at p < 0.05.

). Because computational thinking failed to meet sphericity (W = 0.75, p < 0.005), Greenhouse–Geisser’s epsilon correction (ε = 0.801) was applied. No significant interaction was found between groups, time points, and group–time point (Table 3).

According to the analysis on each subfactor, the decomposition score of the experimental group right after the program increased by 0.29 ± 0.64, compared to the score before the program, whereas that of the control group decreased by −0.02 ± 0.56. The difference in the change was not statistically significant (t = 1.62, p = 0.113). However, the decomposition score of the experimental group in the 12th week of the program increased by 0.39 ± 0.61, compared to the pre-program score, whereas that of the control group decreased by −0.06 ± 0.64. The difference in the change was statistically significant (t = 2.27, p = 0.029). According to the independent samples t-test conducted for the group comparison at each point of time for the change, the experimental group had a greater change between the pre-program and in the 12th week of the program than the control group. The algorithm score of the experimental group right after the program increased by 0.35 ± 0.71, compared to the pre-program score, whereas the control group decreased by −0.11 ± 0.46. Therefore, the difference in the change was statistically significant (t = 2.40, p = 0.021). Additionally, the algorithm score of the experimental group in the 12th week of the program increased by 0.32 ± 0.58, compared to the pre-program score, whereas the control group decreased by −0.18 ± 0.75. Therefore, the difference in the change was statistically significant (t = 2.33, p = 0.025). According to the independent samples t-test conducted for the group comparison at each point of time for the change, the experimental group had a greater change between the pre-program and in the 12th week of the program than the control group.

The pattern recognition score of the experimental group right after the program increased by 0.26 ± 0.82, compared to the pre-program score, and the control group’s score increased by 0.05 ± 0.74. However, the difference in the change was not statistically significant (t = 0.85, p = 0.397). The pattern recognition score of the experimental group in the 12th week of the program increased by 0.28 ± 0.69, compared to the pre-program score, whereas the control group decreased by −0.08 ± 0.73. Nevertheless, the difference in the change was not statistically significant (t = 1.65, p = 0.107). The abstraction score of the experimental group right after the program increased by 0.17 ± 0.88, compared to the pre-program score, and the control group’s score increased by 0.11 ± 0.55. The difference in the change was not statistically significant (t = 0.265, p = 0.790). The abstraction score of the experimental group in the 12th week of the program increased by 0.22 ±0.67, compared to the pre-program score, whereas the control group showed almost no change with a score of 0.00 ± 0.70. The difference in the change was not statistically significant (t = 1.03, p = 0.308) (Table 3).

3.2.2. Group Comparison at Each Point of Time for a Change in AI Competency

AI competency satisfied sphericity (W = 0.88, p = 0.98), and significant interaction was observed between time points (F = 6.76, p = 0.002) and between groups (F = 9.77, p = 0.003). However, there was no significant interaction between a group and time point (F = 2.77, p = 0.069). The AI competency score of the experimental group right after the program increased by 0.79 ± 0.88, compared to that before the program, and the control group’s score increased by 0.26 ± 0.71. The difference in the change was statistically significant (t = 2.11, p = 0.041). Additionally, the experimental group’s AI competency score in the 12th week of the program increased by 0.81 ± 0.68, compared to the pre-program score, and the control group’s score increased by 0.10 ± 1.36. The difference in the change was statistically significant (t = 2.06, p = 0.048). According to the independent samples t-test conducted for the group comparison at each point of time for the change, the experimental group had a greater change than the control group (

Table 4. Effects of the international training program for enhancing intelligent capabilities on AI competency (N = 40).

Variables	Groups	Pre-Test	Post-Test	Follow-Up Test	Sources	F	p	Differences (Post–Pre)			Differences (Follow-Up–Pre)
		M ± SD	M ± SD	M ± SD				M ± SD	t	p	M ± SD	t	p
AI competency	Exp.	3.13 ± 0.50	3.92 ± 0.84	3.94 ± 0.47	G	9.77	0.003	0.79 ± 0.88	2.11	0.041	0.81 ± 0.68	2.06	0.048
	Cont.	3.01 ± 1.01	3.28 ± 0.87	3.12 ± 0.80	T	6.76	0.002	0.26 ± 0.71			0.10 ± 1.36
					G*T	2.77	0.069
Knowledge inference	Exp.	3.07 ± 0.56	3.82 ± 1.01	3.77 ± 0.76	G	5.42	0.025	0.75 ± 1.10	1.89	0.066	0.70 ± 1.03	1.49	0.144
	Cont.	3.02 ± 1.08	3.10 ± 1.08	3.17 ± 0.94	T	3.25	0.044	0.07 ± 1.15			0.15 ± 1.28
					G*T	1.79	0.173
Data understanding and learning	Exp.	3.43 ± 0.51	4.15 ± 0.74	4.05 ± 0.50	G	6.98	0.012	0.71 ± 0.89	1.93	0.061	0.61 ± 0.63	1.83	0.074
	Cont.	3.36 ± 1.10	3.60 ± 0.93	3.28 ± 0.90	T	4.23	0.025	0.23 ± 0.64			−0.07 ± 1.55
					G*T	2.31	0.116
Machine learning	Exp.	3.18 ± 0.52	3.90 ± 0.87	3.97 ± 0.42	G	10.46	0.003	0.71 ± 0.94	1.41	0.166	0.78 ± 0.67	1.88	0.068
	Cont.	2.98 ± 1.17	3.28 ± 0.92	3.07 ± 0.84	T	5.25	0.007	0.30 ± 0.90			0.08 ± 1.52
					G*T	2.15	0.123
Deep learning	Exp.	2.93 ± 0.76	3.83 ± 1.03	3.96 ± 0.53	G	11.09	0.002	0.90 ± 1.10	1.61	0.115	1.02 ± 0.093	2.35	0.024
	Cont.	2.78 ± 1.19	3.13 ± 0.96	2.88 ± 0.84	T	6.84	0.002	0.35 ± 1.05			0.10 ± 1.49
					G*T	3.12	0.050
AI ethics	Exp.	3.01 ± 0.85	3.93 ± 0.80	3.95 ± 0.62	G	6.42	0.016	0.91 ± 1.03	1.87	0.069	0.93 ± 0.98	1.68	0.101
	Cont.	2.93 ± 1.18	3.28 ± 0.94	3.20 ± 0.88	T	7.92	0.001	0.35 ± 0.87			0.26 ± 1.46
					G*T	2.01	0.140

AI = artificial intelligence; Exp. = experimental group; Cont. = control group; M = mean score; SD = standard deviation; G = group; T = time; G*T = group by time; F = F-statistic; t = t-statistic; p = significance level; significance at p < 0.05.

).

According to the analysis on each subfactor, the deep learning score of the experimental group right after the program increased by 0.90 ± 1.10, compared to the pre-program score, and the control group’s score increased by 0.35 ± 1.05. The difference in the change was not statistically significant (t = 1.61, p = 0.115). The experimental group’s deep learning score in the 12th week of the program increased by 1.02 ± 10.93, compared to the pre-program score, and the control group’s score increased by 0.10 ± 1.49. Since the experimental group has a greater change than the control group, the change was statistically significant (t = 2.35, p = 0.024). According to the independent samples t-test conducted for the group comparison at each point of time for the change, compared to the control group, the experimental group had a greater change in the 12th week of the program than before the program.

The knowledge inference score of the experimental group right after the program increased by 0.75 ± 1.10, compared to the pre-program score, and the control group’s score increased by 0.07 ± 1.15. However, the difference in the change was not statistically significant (t = 1.89, p = 0.066). The knowledge inference score of the experimental group in the 12th week of the program increased by 0.70 ± 1.03, compared to the pre-program score, and the control group’s score increased by 0.15 ± 1.28. However, the difference in the change was not statistically significant (t = 1.49, p = 0.144). The data comprehension learning score of the experimental group right after the program increased by 0.71 ± 0.89, compared to the pre-program score, and the control group’s score increased by 0.23 ± 0.64. The difference in the change was not statistically significant (t = 1.93, p = 0.061). Additionally, the experimental group’s data comprehension learning score in the 12th week of the program increased by 0.61 ± 0.63, compared to the pre-program score, whereas the control group’s score decreased by −0.07 ± 1.55. The difference in the change was not statistically significant (t = 1.83, p = 0.074). The experimental group’s machine learning score right after the program increased by 0.71 ± 0.94, compared to the pre-program score, and the control group’s score increased by 0.30 ± 0.90. The difference in the change was not statistically significant (t = 1.41, p = 0.166). The experimental group’s machine learning score in the 12th week of the program increased by 0.78 ± 0.67, compared to the pre-program score, and the control group’s score increased by 0.10 ± 1.49. The difference in the change was not statistically significant (t = 1.88, p = 0.068). The experimental group’s AI ethics score right after the program increased by 0.91 ± 1.03, compared to the pre-program score, and the control group’s score increased by 0.35 ± 0.89. The difference in the change was not statistically significant (t = 1.87, p = 0.069). The experimental group’s AI ethics score in the 12th week of the program increased by 0.93 ± 0.98, compared to the pre-program score, and the control group’s score increased by 0.26 ± 1.46. However, the difference in the change was not statistically significant (t = 1.68, p= 0.101) (Table 4).

3.2.3. Group Comparison at Each Point of Time for Changes in Core Competencies

Core competencies did not satisfy sphericity (W = 0.65, p < 0.005), so Greenhouse–Geisser epsilon correction (e = 0.744) was applied. The difference between time points (F = 7.04, p = 0.004) was observed to be significant, but there was no significant interaction between groups, and a group and time point (

Table 5. Effects of the international training program for enhancing intelligent capabilities on core competencies (N = 40).

Variables	Groups	Pre-Test	Post-Test	Follow-Up Test	Sources	F	p	Differences (Post–Pre)			Differences (Follow-Up–Pre)
		M ± SD	M ± SD	M ± SD				M ± SD	t	p	M ± SD	t	p
Core competence	Exp.	5.23 ± 0.61	5.98 ± 1.13	5.52 ± 0.78	G	3.11	0.086	0.74 ± 0.93	1.46	0.151	0.28 ± 0.87	1.51	0.138
	Cont.	5.07 ± 0.91	5.46 ± 1.09	4.89 ± 1.24	T	7.04	0.004	0.38 ± 0.60			−0.18 ± 1.09
					G*T	1.11	0.320
Critical thinking	Exp.	4.53 ± 0.89	5.38 ± 1.32	4.98 ± 0.96	G	1.62	0.210	0.85 ± 1.50	1.83	0.074	0.45 ± 1.05	1.23	0.222
	Cont.	4.60 ± 1.15	4.72 ± 0.97	4.60 ± 1.12	T	3.13	0.049	0.12 ± 0.91			0.00 ± 1.23
					G*T	1.76	0.178
Communication	Exp.	4.86 ± 1.07	5.60 ± 1.10	5.45 ± 0.99	G	5.73	0.022	0.74 ± 1.31	1.60	0.117	0.59 ± 0.98	2.27	0.029
	Cont.	4.69 ± 1.14	4.83 ± 0.98	4.52 ± 1.11	T	2.50	0.089	0.13 ± 0.98			−0.16 ± 1.09
					G*T	2.09	0.131
Creativity	Exp.	5.32 ± 0.81	5.69 ± 1.11	5.58 ± 0.95	G	1.54	0.222	1.79 ± 1.51	1.20	0.237	0.26 ± 0.87	1.54	0.130
	Cont.	5.33 ± 0.97	5.31 ± 1.07	5.03 ± 1.31	T	0.61	0.545	1.30 ± 0.98			−0.30 ± 1.36
					G*T	1.09	0.339
Collaboration	Exp.	5.47 ± 0.73	5.82 ± 1.01	6.06 ± 0.72	G	1.97	0.168	0.35 ± 1.15	0.67	0.503	0.59 ± 0.80	2.04	0.048
	Cont.	5.44 ± 1.07	5.54 ± 1.30	5.28 ± 1.54	T	0.79	0.454	0.10 ± 1.18			−0.15 ± 1.43
					G*T	1.78	0.176

Exp. = experimental group; Cont. = control group; M = mean score; SD = standard deviation; G = group; T = time; G*T = group by time; F = F-statistic; t = t-statistic; p = significance level; significance at p < 0.05.

).

According to the analysis on each subfactor, the experimental group’s critical thinking score right after the program increased by 0.85 ± 1.50, compared to the pre-program score, and the control group’s score increased by 0.12 ± 0.91. The difference in the change was not statistically significant (t = 1.83, p = 0.074). The experimental group’s critical thinking score in the 12th week of the program increased by 0.45 ± 1.05, but the control group’s score had almost no change with a score of 0.00 ± 1.23. The difference in the change was not statistically significant (t = 1.23, p = 0.222). The experimental group’s communication score right after the program increased by 0.74 ± 1.31 compared to the pre-program score, and the control group’s score increased by 0.13 ± 0.98. The difference in the change was not statistically significant (t = 1.60, p = 0.117). In addition, the experimental group’s communication score in the 12th week of the program increased by 0.59 ± 0.98, compared to the pre-program score, but the control group’s score decreased by −0.16 ± 1.09. The difference in change was statistically significant (t = 2.27, p = 0.029). The experimental group’s creativity score right after the program increased by 1.79 ± 1.51, compared to the pre-program score, and the control group’s score increased by 1.30 ± 0.98. The difference in the change was not statistically significant (t = 1.20, p = 0.237). The experimental group’s creativity score right in the 12th week of the program increased by 0.26 ± 0.87, compared to the pre-program score, and the control group’s score increased by –0.30 ± 1.36. The difference in the change was not statistically significant (t = 1.54, p = 0.130) (Table 5).

4. Discussion

This study aimed to provide foundational data for expanding and enhancing the ADDIE-model-based international training program for enhancing intelligent capabilities by applying the 2-week program to graduate students participating in the course of Smart Information Communication Technology (SMART ICT) Employees and examining the changes in their computational thinking, AI competency, and core competencies.

The first hypothesis, “the experimental group participating in the international training program for enhancing intelligent capabilities will score higher in computational thinking than the control group that did not participate” was rejected. The computational thinking of the experimental group participating in the international training program continuously improved, albeit to a minimal extent. In contrast, in the control group, all the scores for computing thinking were similar to the pre-scores, or decreased. Although the score for computing thinking increased continuously, the increase was quite less than that in the score for AI competency. The reason for the minimal change was that the program mainly focused on the AI field. It is interpreted that the participant group was made up of graduate students and employees working in companies, and was already equipped with fundamental abilities in terms of computational thinking. The sample size of this study was sufficient for a repeated measures analysis. Nevertheless, it is necessary to size up samples and test them through repeated research.

The second hypothesis, “the experimental group participating in the international training program for enhancing intelligent capabilities will score higher in artificial intelligence competency than the control group that did not participate” was supported. After the international training program was applied, the changes in the scores for AI competency were significantly different. The experimental group, which participated in the international training program, showed a continuous increase in the score for AI competency. In contrast, in the control group, all the scores for AI competency were similar to the pre-scores, or decreased. This suggests that the international training program in this study had a particularly significant effect on AI competency. In the study by Yoo et al. [15], who applied a 30 h AI class to university students for 15 weeks, their AI competency also improved. The Germo [33] study also showed similar results. Although the class operation procedure in the study was different from that in this study, the result indicates that systematic organization and operation of educational content contribute to program effectiveness. In this study, the international training program for enhancing intelligent capabilities was designed on the basis of the ADDIE model. It incorporates a blended learning approach with online and offline intensive education [27,34]. The real-time online classes consisted of six sessions for one week, each of which lasted 90 min. The face-to-face classes consisted of six sessions for nine days, each of which lasted 4–8 h face-to-face in the U.S. [35,36]. The program securing over 45 h of education provided enough opportunity for knowledge acquisition and extension. In particular, this program enabled participants to deepen online learning during offline sessions and divided the course into general and in-depth sections to offer tailored education based on learners’ levels [26,37]. All the special lectures and presentations on information and communication technology were based on cases. In order to enhance their practical skills, participants had field experiences in the ICT sector. Above all, presentations, small group discussions, and especially continuous one-on-one feedback were applied to help learners find solutions to technological challenges not only in their personal dimension, but in their corporate dimension. It is considered that these diverse efforts positively influenced the increase in the AI competency score and the lasting of the score that had increased from the start of the program to the 12th week of the program. The participants in this study were employees working in various positions related to AI or big data in local companies. They were required to make continuous improvements in knowledge and application abilities related to the latest technologies in their fields rather than to obtain sporadic knowledge. However, the AI competency score of the participants in this study was similar to that (4.07) of university students in previous research [38]. Therefore, it is necessary to provide education in the ICT field continuously.

The third hypothesis, “the experimental group that participated in the international training program for enhancing intelligent capabilities will score higher in future society core competency scores than the control group that did not participate” was rejected. The core competencies’ score was on the rise until the point of time right after the intervention, but then decreased. So, in comparison with the control group, it showed no significant change. In this program, discussions were included in classes, in consideration of the participants’ improvements in core competencies, such as critical thinking, communication, creativity, and collaboration skills. In addition, they were required to present individual projects to enhance creative problem-solving abilities [20,22,39]. One-on-one expert feedback was provided as much as possible in order to help them to address any technological challenges. But these results are the same as those seen in computational thinking. It is interpreted that the participant group was made up of graduate students and employees working in companies, and was already equipped with fundamental abilities in terms of core competencies.

Computational thinking is the problem-solving process based on algorithms and logic. AI is the process of solving problems with data-based reasoning and learning. These two processes are not separate but very closely related, and have many things in common [39]. Previous research already confirmed the positive correlations between AI competency, computational thinking, and core competencies [15,40]. In the world we live in at present, a variety of social fields are interconnected through information infrastructure, and a diversity of intelligent activities are automated based on AI technology [32]. In short, the current society is an intelligence information society. The ideal qualifications for talents in such a society include not only expertise in various fields, but also computational thinking to view and solve problems from a future-oriented perspective, interpret rationally, and collect and utilize information with the use of ICT; AI knowledge; critical thinking, communication, and a collaborative attitude crucial for problem solving [1,22,25,31]. In other words, it is required to obtain AI knowledge, the knowledge of computer application, and expertise in various fields [25,38]. Therefore, in order to improve AI competency, it is crucial to include the educational contents in consideration of the expansion of computational thinking and operate the project-based education with practical outputs.

This study tried to design and operate the program systematically in order to give the program participants an opportunity to reorganize their knowledge, thinking processes, and technological skills, considering that they as advanced learners already have a lot of experiences and knowledge. The results indicated that the international training program was effective for increasing the AI competency score, and that its effect lasted until the 12th week of the program. This study is meaningful in the point that the international training program for enhancing intelligent capabilities had the possibility to become a useful intervention program. Therefore, it is suggested that the international training program for enhancing intelligent capabilities can be applied to or utilized by undergraduate students and various age groups as well as graduate students. However, there are several limitations in interpreting the results. Firstly, the study was conducted in a single institution, and arbitrary allocation, rather than random sampling, was used to control the diffusion effect of the program. Secondly, although participants were chosen according to selection criteria, and demographic homogeneity between the experimental and control groups was secured, it is impossible to rule out completely the possibility of confounding variables due to the lack of random allocation. Thirdly, the study is meaningful in the point that it verified the short-term effectiveness of the international training program, because its effects on AI competency were evaluated right after the program and in the 12th week of the program. However, its long-term effectiveness failed to be defined. Therefore, in a follow-up study, it is necessary to examine the long-term effect through repeated research. Finally, in this study, the program, which could have been performed a single time, was provided with real-time online classes (six sessions for one week, 60 min per session) and face-to-face classes (six sessions for nine days, 4–8 h per session) that amounted to 30–45 h of education and were equivalent to the hours for 2–3 credits. Future AI education is learner-centric, continuous, and creative education, rather than single-time education. It emphasizes enough education hours and systematic contents. Therefore, it is suggested that the international training program for enhancing intelligent capabilities in this study can be used as a fundamental material when interdisciplinary education attempts are made in various fields.

5. Conclusions

This study was to apply a 2-week, 45 h international training program for enhancing intelligence capabilities to 40 graduate students participating in the SMART ICT course. As a result, it was confirmed that the artificial intelligence competency scores of the experimental group maintained as improved scores from immediately after the program up to 12 weeks compared to the control group that did not participate in the program.

We attribute these results to three strategies. First, the international training program applied the ADDIE instructional model to the curriculum design of the program, which is suitable for increasing knowledge, skills, and application of the latest technologies in the ICT field. Second, customized training was applied to reflect the needs of participants identified during the analysis process of the ADDIE model design phase. Third, the blended learning method with real-time online classes and face-to-face classes was applied to enhance the effectiveness of the classes for sustainable education.

Therefore, the educational significance of this study is that it involves a branded learning program designed based on the ADDIE instructional model that provides a model of a circular education system rather than one-time education. It can also be used as a long-term research model to strengthen AI capabilities. In order to implement a long-term research model, we propose to apply an educational program that can be linked to the international training program of this study and to improve the quality of education by continuing the process of evaluating artificial intelligence capabilities to produce results.