Engineering Design and Evaluation of the Process Evaluation Method of Auto Repair Professional Training in Virtual Reality Environment

Abstract

The rapid development of information technology and Internet technology has a far-reaching impact on vocational education. It is possible to accurately and objectively evaluate the training of learners by recording the process data of learners’ realization. The teaching evaluation of traditional vocational skill training requires time, workforce, and educational resources. Due to the limitations of experimental conditions, it is easy to ignore the procedural characteristics of skill training and difficult to implement the procedural evaluation. Based on the above problems, combined with virtual reality and the parts of vocationally skilled auto repair training specialty, using machine learning methods, engineering design of process evaluation method for skilled auto repair training, and takes the secondary vocational auto repair specialty as an example, constructs an evaluation index model based on KSA theoretical model, and evaluates three dimensions: knowledge acquisition, skill mastery, and ability cultivation (knowledge, skill, ability, KSA). The experimental verification of the process evaluator is carried out in the theoretical training evaluation auto repair system (TTE) based on virtual reality. The experimental results can effectively evaluate the practical training of students. The research results of this paper provide a new perspective and reference for the learning evaluation of skill-based training majors.

1. Introduction

The emergence of virtual reality technology has created many possibilities for education development. It can also make a greater breakthrough in vocational education that needs a practical operation. Virtual reality (VR), as a powerful technology that can simulate the real-world experience in the virtual world, has received extensive attention in the past decades due to its application in education. Support learners by creating 3D spatial representations, multi-sensory and intuitive interactions, and immersion in virtual environments [1]. This technology is widely used in education. The impact of VR on education is first reflected in the improvement of the learning experience. VR supports immersive learning, and students can experience scenes that are difficult or impossible to experience in real life “on the spot”. VR technology can reproduce real scenes and has strong interactivity, enabling interaction and creating a personalized learning environment. VR can make learning more efficient. It can “occupy” students’ multiple sensory channels such as vision, hearing, and touch. Multi-sensory participation can improve learning efficiency [2]. An important aspect of VR in education is that virtual reality content will enable students to identify and explore even abstract or difficult-to-observe knowledge in a risk-free environment [3,4]. In most skill-based majors, learning and practical training activities are often too complex, dangerous, or time-consuming to complete in traditional classrooms. The development of traditional classrooms to virtual reality classrooms can solve this problem well. The skill-based major defined in this paper refers to a major that requires hands-on practice. Simple text question tests and training reports cannot accurately evaluate the learner’s real-acquired skills. Instead, there needs to be focus on the learner’s hands-on operation process; this skill has the characteristics of tool mastery proficiency and skill process proficiency. Vocational education and general education are two “different types” of education. Different from general education, learning with knowledge as the core, vocational education is centered on learning vocational skills, adhering to the technical concept of embodiment and the value orientation of cultivating the spirit of craftsmen [5]. The existing process evaluation of skill operation learning is mainly carried out by teachers relying on their own experience, through observation and judgment, which has strong subjectivity and individual differences, and it is difficult to guarantee the scientificity and consistency of evaluation [6]. Evaluation can also use the paper-and-pencil test of general education, and draw lessons from the foreign COMET (Competence Development and Assessment in TVET) ability assessment program, the German Chamber of Commerce and Industry (IHK), the German Handicrafts Association (HWK) Vocational Education Graduation Exam, and other methods [7]. Mantovani et al. [8] argue that VR technology is a powerful and reliable tool for training employees, experts, and managers. Shi, B. et al. [9] pointed out that there are problems in the training teaching of secondary vocational auto repair majors, such as relatively single training teaching, mismatch of training station resources and training needs, limited professional teacher resources, and loss of training equipment resources. Teachers are often too busy with management to pay more attention to the process of students’ technical operation. Therefore, the traditional training teaching evaluation focuses on the results of a single skill operation, and the students’ dynamic learning behavior cannot be traced, resulting in the subjective evaluation of technical standards, lack of process and objectivity, which is not conducive to the formation of students’ comprehensive literacy [10]. In the traditional training evaluation, the teaching evaluation ignores the process evaluation, although the process evaluation has been paid more and more attention. However, in practical application, the teaching evaluation of domestic vocational colleges is still dominated by summative evaluation, and the evaluation is still mainly oriented to text questions and training reports [11]. This leads to the fact that in practical training, teachers’ evaluation of students is highly subjective, ignoring the procedural nature of students’ practical training operations, and cannot accurately and objectively evaluate each student. The training evaluation of skill-based majors needs to pay attention to the process of its operation, and only in this way will the evaluation be more accurate and objective.

To solve various limitations in skilled professional training, including the limitations of teacher resources and classroom space resources and the inaccurate assessment of learners, this paper takes the secondary vocational auto repair major as an example to do the following work:

(1)

This paper combines the professional characteristics of virtual reality technology and secondary vocational skill-based auto repair training and adopts a machine learning method to carry out the engineering design of the process evaluation method for skill-based auto repair training.

(2)

Through expert interviews and literature research, a new process evaluation model is constructed, with three dimensions of evaluation indicators: knowledge acquisition, skill mastery, and ability development (knowledge, skill, ability, KSA).

(3)

The TTES system was designed and developed, and the advanced performance of the model was evaluated through the comparison of multiple groups of experiments and the ten-fold cross-validation process.

The rest of this paper is organized as follows: Section 2 focuses on related research to describe the current network evaluation platform, the application of virtual reality systems in education, and the evaluation form of skill-based training in vocational education; Section 3 introduces the design of the model of the procedural data evaluation system for vocational skill training; Section 4 takes the auto repair profession as an example to determine the evaluation indicators in TTES; Section 5 is designed to evaluate the accuracy of the evaluator in the model; Section 6 concludes the paper with remarks and directions for future research.

2. Related Words

This section is elaborated on three aspects. The first aspect is the research situation of the network evaluation system platform. The second aspect is the application of a virtual reality system in education. The third aspect is the evaluation form of skill-based training in vocational education.

2.1. Network Evaluation System Platform

With the development of technology, many scholars will analyze the data on the network platform from different angles and conduct evaluation research. Zheng Q. et al. [12] conducted a study on data-based online teaching evaluation, aiming at students’ comprehensive evaluation through theoretical deduction and expert interviews, a five-dimensional comprehensive evaluation reference theoretical model with input, completion, regulation, connectivity, and initiative as the core was constructed. A corresponding calculation model was constructed by aggregating characteristic variables through learning behavior data. Hongtao Sun [13] constructed a course evaluation model to evaluate online learning courses from five dimensions of media technology, learning resources, learning behavior, learning support and connectivity, and evaluate teachers’ performances through promotion, engagement, and connectivity. Qiu, F., Zhang, G., Sheng, X. et al. [14] classified the behavior data of the network platform and found the correlation between the behaviors, which not only can supervise the learning process of the learners in real-time, but also predict the learning performance of the learners on the network platform. Macfadyen [15] took the learning process data of the online biology course based on the Blackboard platform as the research object. Then, 118 students were selected as samples, and the statistical values of various learning behaviors that occurred on the platform and the final academic performance were regressed. Forum participation, the number of emails sent, and the number of exams completed were found to be key variables significantly correlated with students’ academic performance. A regression model was built to predict students’ academic performance. Wei, S. [16] took the data-driven perspective and started from the existing data status of the learning platform, determined the main scoring modules and dimensions, and constructed a student’s online learning performance model and an evaluation index system for the online course implementation process. Zhu, K. et al. [17] carried out the secondary development of the SAKAI system and integrated the data analysis tool SPSS to carry out data analysis on the electrical courses of 107 students; the system can acquire, store, analyze, and report the data of students and learning environment, and provide a reference for teachers’ analysis and decision-making. Based on the characteristics of learners’ behavior, Zhang Beisi [18] designed a comprehensive evaluation model for learners with multi-source data to comprehensively evaluate learners’ learning processes. The above researchers use different methods to evaluate from different aspects and different perspectives, as shown in

Table 1. Summary of the network evaluation system platform.

Authors	Database Source	Focus
Zheng Q et al. (2016) [12]	E-Education Research	Online teaching evaluation is based on data, the reference theoretical model of the five-dimensional comprehensive evaluation, and the corresponding calculation model is constructed.
Hongtao Sun et al. (2016) [13]	E-Education Research	A curriculum evaluation model is constructed to evaluate online learning courses from five dimensions: media technology, learning resources, learning behavior, learning support, and connectivity, and to evaluate teachers’ performance from promotion, engagement, and connectivity.
Feiyue Qiu et al. (2022) [14]	Scientific Reports	Classifying the online platform behavior data and finding the correlation between behaviors can not only monitor learners’ learning process in real time but also predict learners’ learning performance on the online platform.
Macfadyen et al. (2010) [15]	Computers & Education	Based on the data of the Blackboard platform, the key variables that are significantly related to student’s academic performance are found, and a regression model is established to predict students’ academic performance.
Shunping W et al. (2016) [16]	Journal of Open Learning	From the data-driven perspective, starting from the existing data of the learning platform, this paper constructs the students’ online learning performance model and the evaluation index system of the online course implementation process.
Zhu, K et al. (2013) [17]	China Educational Technology	Carried out the secondary development of the SAKAI system and integrated the data analysis tool SPSS to carry out data analysis on the electrical courses of 107 students; the system can acquire, store, analyze and report the data of students and learning environment, and provide a reference for teachers’ analysis and decision-making.
Zhang Beisi et al. (2022) [18]	Information Science	Designed a comprehensive evaluation model for learners with multi-source data to comprehensively evaluate learners’ learning process.

, but for vocational trainees, the process data of skill operation cannot be collected through this ordinary online learning system; the learning form of this skill operation also determines that the evaluation method of learning is also different from other courses. The ideal skill operation learning evaluation requires teachers to pay attention to students’ operation process in real-time and give students comments and feedback in time, to improve the efficiency and quality of skill acquisition [6]. However, the unique operational and action attributes of skill operation learning make it difficult to realize its procedural evaluation. To evaluate their training more accurately and objectively, we need to collect the data they left in the operation. Only by collecting the learners’ training process data can we achieve the goal of more accurate and effective evaluation.

2.2. Application of Virtual Reality System in Education

In 1994, Burdea G and Philippe Coiffet proposed three basic characteristics of virtual reality as Imagination, Interaction, and Immersion [19]. The application of virtual reality technology in education can be traced back to the 1980s when the application and effectiveness in education and training have been researched. Still, the attention was not high at that time [20]. Later, with the development of virtual reality technology, based on the characteristics of virtual reality, some bottlenecks in education can be broken through. For example, Omlor et al. [21], through comparative experiments, found that using virtual reality technology to realize online immersive video teaching enhanced the classroom learning atmosphere and the comprehensibility of knowledge. Jiang et al. [22] designed an interactive teaching platform based on virtual reality technology and realized human–computer interaction by establishing a virtual three-dimensional scene, aiming at improving the problems of expensive hardware facilities of the existing VR system and the inability of teachers and students to interact in real-time. Zhang et al. [23] based on the cognitive heuristic model and virtual reality technology, designed and developed interactive courses for international Chinese education and built a mobile learning platform, and found that it can achieve good interactive effects and improve learners’ academic performance. Ott and Freina [24] found that the use of VR in education makes it possible for people to experience inaccessible (in time or space) or problematic (dangerous or unethical) situations. It covers interdisciplinary applications in various disciplines such as architecture, linguistics, educational games, and science courses. In the field of education, compared with video learning, the VR immersive environment improves students’ participation. Vice versa, the interaction between students and learning resources also stimulates their potential learning motivation and interest [25,26]. Arif Farrukh [27] applied VR technology in basic civil engineering education; through quantitative tests, it is found that learners show higher concentration ability in a VR environment. Hwang, G. J. et al. [28] applied VR-based pottery production to junior high school labor technology education and investigated its impact on students’ creativity and learning participation. The results show that the VR-based “observation-do-reflection” curriculum design promotes students’ creative performance and cognitive engagement. Virtual reality training systems have been implemented in many disciplines such as medicine, military, engineering, flight simulation, automotive, aerospace, and manufacturing [29]. A validation evaluation was conducted to determine if VR training led to acquiring driving skills for powered wheelchair maneuvers over no training or desktop training [30]. Another work by Wang et al. [31] includes the development of a tissue-protection-based virtual reality training system for robotic catheter surgery to prevent collateral damage from collisions. The evaluation results demonstrate the effectiveness of the tissue protection mechanism in this system, with reduced tissue damage and reduced collision frequency. Khan, Noman et al. [32] developed a VR system that allows children to accept and participate in road safety exercises and is verified through experiments that the system positively improves children’s road crossing behavior. Zhao, K. et al. [33] discussed the application of VR technology in football training. Additionally, they analyzed the feasibility of VR technology in football applications, providing theoretical support for the development and research of a virtual football training system. Ho Nicholas et al. [34] proposed and evaluated a new VR training system for the assembly manipulation of biological and mechanical components (i.e., hybrid medical device assembly). They provide trainees with effective, efficient, risk-free, and low-cost training through this virtual reality training system, thereby addressing potential training issues in hybrid medical device assembly. Soliman, Maged et al. [35] reviewed previous studies and found that VR technology is an excellent tool in engineering education, which can achieve positive teaching effects and help students experience experimental environments that cannot be accessed in reality. S. Borsci [36] et al. used a virtual reality-based system for the training of car assembly tasks but did not have a module to evaluate trainee acquisition. J. Osterlund et al. [37] applied virtual reality technology to spacecraft assembly and training. Christian Seufer et al. [38] studied whether learning classroom management in a virtual environment can improve the classroom management ability of pre-service teachers. Through experiments on 55 pre-service teachers, they found that a virtual reality environment can improve the classroom management ability of pre-service teachers. Jessica Ulmer et al. [39] studied the specific training of VR Gamification. Compared with the non-Gamification VR group, the Gamification VR group has a lower error rate in training and makes fewer errors in real assembly. Through VR Gamification training, we can get better learning results and improve learners’ self-evaluation. Mondragón Bernal et al. [40] developed a substation operation training system and used virtual reality technology to conduct professional training in the power system in the form of serious games. The usability of the system was evaluated by using the System Availability Scale (SUS) and Game Participation Questionnaire (GEQ), and good experiences and effects were obtained. Hong et al. [41] built a virtual costume catwalk scene with an immersive experience based on the Unity 3D game engine and got a stronger user experience and visual viewing feeling. As shown in

Table 2. Summary of the application of virtual reality systems in education.

Authors	Database Source	Focus
Omlor et al. (2022) [21]	Medical Education Online	Through comparative experiments, found that using virtual reality technology to realize online immersive video teaching enhanced the classroom learning atmosphere and the comprehensibility of knowledge.
Jiang et al. (2021) [22]	Complexity	Based on virtual reality technology, an interactive teaching platform is designed, and human–computer interaction is realized by establishing virtual 3D scenes.
Zhang et al. (2022) [23]	Computational Intelligence and Neuroscience	Based on the cognitive heuristic model and virtual reality technology, designed and developed interactive courses for international Chinese education and built a mobile learning platform, and found that it can achieve good interactive effects and improve learners’ academic performance.
Ott and Freina et al. (2015) [24]	eLearning and Software for Education	In the field of education, compared with video learning, VR immersive environment enhances students’ participation, whereas the interaction between students and learning resources also stimulates their potential learning motivation and interest.
Arif. Farrukh et al. (2021) [27]	Education and Information Technologies	Applying VR technology to basic education of civil engineering, through a quantitative test, it is found that learners show higher concentration ability in a VR environment.
Hwang, G. J. et al. (2021) [28]	Interactive Learning Environments	This paper applies VR-based ceramic production to junior high school labor and technical education and investigates its influence on students’ creativity and learning participation. The results show that the curriculum design of “observation-doing-reflection” based on VR promotes students’ creativity and cognitive input.
John et al. (2017) [30]	IEEE transactions on visualization and computer graphics	A validation evaluation was conducted to determine if VR training led to acquiring driving skills for powered wheelchair maneuvers over no training or desktop training.
Wang et al. (2017) [31]	Medical & Biological Engineering & Computing	A virtual reality training system based on tissue protection is developed for robotic catheter surgery. The evaluation results prove the effectiveness of the tissue protection mechanism in this system.
Khan, Noman et al. (2021) [32]	Sensors	The VR system is developed for children to accept and participate in road safety exercises, and the experiment proves that this system has a positive effect on improving children’s crossing behavior.
Zhao, K. et al. (2022) [33]	Journal of Sensors	This paper discusses the application of VR technology in football training and analyzes the feasibility of VR technology in football applications.
Ho Nicholas et al. (2018) [34]	Multimedia tools and applications	A new VR training system is proposed, which can be used for the assembly of biological and mechanical components (i.e., hybrid medical equipment assembly) to solve the potential training problems of hybrid medical equipment assembly.
Maged. et al. (2021) [35]	Applied Sciences	By reviewing previous studies, it is found that VR technology is an excellent tool in engineering education, which can achieve positive teaching effects and help students experience experimental environments that they cannot enter in reality.
S. Borsci (2016) [36]	Virtual Reality	The system based on virtual reality is used for the training of automobile assembly tasks, but there is no module to evaluate the students’ acquisition.
J. Osterlund (2012) [37]	Acta Astronautica	Applying virtual reality technology to spacecraft assembly and training.
Christian Seufer et al. (2022) [38]	Computers & Education	This paper studies whether learning classroom management in a virtual reality environment can improve pre-service teachers’ classroom management ability, and finds out through experiments that a virtual reality environment can improve pre-service teachers’ classroom management ability.
Jessica Ulmer et al. (2022) [39]	International Journal of Human–computer Studies	The specific training of VR gamification is studied, and it is found that the error rate of the gamification VR group in training is reduced, better learning results can be obtained, and learners’ self-evaluation can be improved through the gamification training of VR.
Mondragón Bernal et al. (2022) [40]	Applied Sciences	A substation operation training system is developed, and the professional training of the power system is carried out by using virtual reality technology in the form of serious games, and good experience and effects are obtained.
Hong et al. (2022) [41]	Computational Intelligence and Neuroscience	Built a virtual costume catwalk scene with an immersive experience based on the Unity 3D game engine and got a stronger user experience and visual viewing feeling.

, from the work of these researchers, it is found that virtual reality technology has a great effect on education, especially in skill-based majors that require practical training. The current virtual reality technology in practical training mainly provides a virtual environment for learners to acquire skills through immersion, however, very few virtual reality systems collect data on students’ operation process, so most of them are a “learning-training” method. This paper will realize a “learning-training-assessment” method through VR technology, collect students’ operation data in the virtual reality environment, and make a more objective evaluation.

2.3. Evaluation Form of Skill-Based Training in Vocational Education

Vocational education pays attention to the cultivation of students’ professional ability, and the evaluation content should be compatible with the knowledge and skills of the vocational skills imparted [42]. For skill-based training courses, summative evaluation cannot evaluate students comprehensively, accurately, and objectively. Because too much emphasis is placed on static, quantifiable, and shallow learning outcomes, the evaluation content is one-sided and the evaluation method is single, which is not conducive to the overall development of learners. Process evaluation pays attention to the dynamic, difficult-to-quantify, high-level learning processes and learning effects. By recording learners in real-time during the learning process, integrate the evaluation process with the learning process to provide learners with timely feedback, guidance, motivation, adjustment, and other aspects of learning support [43]. The process evaluation based on the analysis of the learning process is more scientific, systematic, and intelligent for the evaluation of the teaching process, and it can then realize “learning-based teaching and learning-based guidance” driven by data [44]. In vocational training teaching, the traditional evaluation method has only results and no process; whether the results represent the hard work of the students and whether the process effectively supports the results, that is to say, whether the result is inevitable or accidental in the process cannot be reflected from the development. Traditional paper-based tests have been described as poor predictors of job performance because learners often resort to cramming, problem-finding, and short-term memory without a deep understanding of course content. Changwei Qi [10] found the problem of auto repair skill-based teaching evaluation because the teaching of secondary vocational auto repair specialty in practical training has high requirements on the site. There are risks in operation in real scenarios. In the teaching process, teachers are busy with management and teaching, and ignore the process of students’ skill operation, thus the students’ operation process cannot be traced, resulting in a lack of procedural and subjective evaluation. In recent years, with the continuous development of virtual reality technology, the technology has begun to be widely used in teaching, described in detail above. However, most of the technology is designed for practical training experience and rarely for evaluation. Most of the rare VR evaluation systems are also designed for a single knowledge point in the course. However, there are few cases of applying virtual reality technology to the whole-course teaching evaluation. Therefore, based on the above problems, combined with the characteristics of skills training and teaching, we use the method of machine learning to design a process evaluation method, using the learner’s operational process data in the VR system, the predictor model is trained, so that the system can intelligently evaluate the learner’s process.

3. Design of the Process Data Evaluation Method for Technical Training in Secondary Vocational Training

Taylor believes that “continuous feedback evaluation can form a virtuous cycle system between students and learning, so that teaching can be mutually beneficial and closely linked with future learning development”. The learners leave procedural data during skill learning and training, and through the processing and analysis of the procedural data, the learners can be evaluated objectively, accurately, and comprehensively. Based on the current research, learning under e-learning supports the comprehensive collection of process information, self-assessment and mutual assessment, and various feedback forms [45]. Combined with the professional characteristics of virtual reality and vocational skill-based training, the process evaluation of vocational skills-based training in the context of virtual reality should have the following features:

(1)

Support the professionalism and typicality of training operations

Because the vocational training major has practical needs compared with other majors, we should start by analyzing the characteristics of professional activities [46]. We should determine the scope and level of professional competence, find out the core operational skills, relevant knowledge, work attitude, application equipment, tools, and materials necessary for the occupational position (population). Indeed, we should also reproduce and simulate the workplace situation, reflect and enhance professionalism and sense of the times. However, during the training, what is needed is the training of the important content and skills, and energy should be concentrated on the necessary, typical key skills that need to be mastered. Too much training will also bring a burden to the learners.

(2)

Support the contextualization of training operations

The core purpose of practical training is to cultivate comprehensive professional ability, and the cultivation of complete professional ability needs to be situational. Virtual reality practical training teaching can use modern methods to create situations and transform the traditional unilateral teaching mode into an organic interactive one. The VR training environment is a profession that requires practical training, and the evaluation of the procedural data obtained through the situational experience will be objective and accurate. It is more conducive to students’ understanding, memory, and formation of skilled motor skills.

(3)

The evaluation of the training process and the training process are integrated promptly

The training process evaluation and the training process occur simultaneously, which is the procedural data left in the learner’s training process, which can timely and objectively reflect the learner’s training situation. Detailed data are left in the process of learner learning and training. The whole process evaluation is based on the learner training data, so the evaluation results are more accurate and objective.

(4)

Emphasis on both process and goal of training evaluation

Process evaluation not only pays attention to the value judgment of learning outcomes but also pays attention to the learning process, which is an important aspect of reflecting the level of learning quality.

(5)

Accuracy and personalization of training process evaluation

The training process evaluation can accurately, comprehensively, and objectively evaluate the learners. The training operation evaluation is to test the learner’s mastery of knowledge and the acquisition of skills, which cannot be tested by traditional evaluation methods. Traditionally, teacher evaluation is highly subjective, but modern society needs innovative talents with applied, compound, and technical skills [47]. This has new requirements for the evaluation of teaching objectives. The data of the process evaluation records the learners’ entire training in detail. By collecting and processing these data, the learners are given a comprehensive and objective evaluation of the training process.

The characteristics of these three aspects form a cyclical development process supported by training evaluation information processing and evaluation results application process. It can not only accurately and comprehensively evaluate the learner’s training, but also provide accurate feedback for theta collection and learning, thereby promoting the generation of a learner-centered personalized training learning model.

The development of VR provides technical support for the evaluation of secondary vocational training. Combined with the above characteristic analysis, this paper proposes a model of the procedural evaluation system for vocational technical training in the VR environment. As shown in Figure 1, the system is mainly composed of four subsystems based on the evaluation method to collect VR training process activity records, data and storage, training data analysis, and evaluation. The learner training data follows the process of “data-processing and storage-feature fusion and analysis-evaluation-feedback and intervention” so that the overall training effect of learners is better. Through the collection and analysis of data, from the most essential records to predict future trends, to an accurate and objective evaluation of the learners’ training, timely guidance and adjustment are part of the learners’ training processes.

(1)

Virtual reality training process activity recording subsystem

The learner will leave many process data on the VR platform. The data are based on the learner, and the virtual simulation training is the carrier. The process activities in the recording subsystem are divided into resource learning, interactive communication, peer evaluation, teacher evaluation, learning reflection, self-reflection, training simulation exercises, remote training guidance, training simulation assessment, training simulation evaluation, and other core activities. The system records the process data of learners in training; mainly including learners in it is the operational data during training. Among system records, the data records of resource learning, interactive communication, peer evaluation, teacher evaluation, learning reflection, and self-reflection mainly record the complete learning process of students. The training simulation exercise, remote training guidance, training simulation assessment, and training simulation evaluation are the differences in the data collected by this system. In addition to the way the previous platform collected data, this platform uses the data left by the students’ training experience to evaluate the learners’ training. This multi-dimensional data evaluation makes the evaluation more comprehensive and objective, and the data recorded by the recording subsystem is used for the following processing and storage subsystem to provide data sources.

(2)

Process data processing and storage subsystem

The data processing and storage subsystem mainly include data acquisition, cleaning, and storage. Among them, the data acquisition module realizes two functions of “collection”, and “collection” realizes the pertinence, value, and accuracy of the data provided by the recording system. “Set” aggregates data according to specific rules and filtering criteria. If the collected data is irrelevant, it is difficult to evaluate the learners accurately and objectively, and it is impossible to find out the key indicators of the training evaluation to give feedback. The function of the data cleaning module is to filter out “junk information” and ensure the correctness of the stored data as much as possible. The main task of data storage is to keep the data set submitted by the transformation module into the database according to the table structure defined by the data model to prevent data loss.

(3)

Training process data analysis subsystem

The data analysis subsystem in the training process mines valuable information from the fusion data from multiple dimensions and draws a portrait of the individual learning process of the learners. First, it is necessary to clean the data of the training operation process obtained from the system. There is no unified process in the data cleaning process, and the method must be selected according to the actual situation of the data to deal with missing values, duplicate values, and outliers. At the same time, the student operation data recorded in the system is multi-dimensional, and the operation data of different dimensions are often not comparable in value, so feature selection cannot be performed. Feature selection is to select the features that have a greater impact on the training model from all features, thereby reducing the feature dimension and improving the operating efficiency and interpretability of the model. Because of the professional training operation, experts have criteria and methods for judging learners, and the key indicators and calculation methods of the training are determined through the calculation of the Delphi method. The algorithms are then compared through existing machine learning methods, such as linear regression, decision tree regression, support vector machine regression (SVM regression), KNN regression, random forest regression, etc. This is done to find the machine learning algorithm that is closest to the expert evaluation as the evaluator algorithm.

(4)

Training process evaluation subsystem

Process evaluation “embeds” evaluation into the learning process and conducts a comprehensive evaluation of the learner’s training for professional knowledge acquisition, skill mastery, and ability development. Knowledge, skills, and attitudes (KSA) are considered part of the three domains identified in the learning activities, and educational style [48]. Benjamin Bloom is the one who connects these to the learning process. He believes that knowledge refers to the cognitive process of mental skills. Attitudes are related to affective areas that process emotions or emotions and skills, i.e., psychomotor processes of physical or physical skills [49]. The comprehensive three-dimensional data gives the learners an overall score. In view of the characteristics of the vocational skill-based training major, combined with the procedural data of the training and the evaluation of the learners by the teachers, the evaluation is more accurate and objective but also more diverse and three-dimensional.

This research will conduct a multivariate evaluation. The first-level indicator is the total training score, and the second-level indicator is evaluated from three aspects: knowledge acquisition, skill mastery, and ability development. The data left by the learners on the VR platform will be divided into secondary indicators as specific data sources, which will be explained in detail later by taking the auto repair major as an example.

The application of the procedural data evaluation model of vocational skill-based training in a VR environment involves the process training data of learners being obtained through the VR training platform. Further mining and analyzing these data can provide learners and teachers with an accurate and objective evaluation plan. There are many problems to be solved in vocational training skill-based majors, and this model can play a role to a certain extent. The training data left by the learner, through the deep mining of the procedural data by the model, can further make a specific evaluation of the learner; learners can more clearly discover their particular problems in the training, and better test their skills. Teaching evaluation is an important part of classroom teaching activity. The process data evaluation model constructed in this paper has a certain reference value for teachers to guide and intervene in students’ practical training. At the same time, through the process of evaluation of practical training, we can also reflect on our own teaching situation and make timely adjustments.

4. Evaluation Index of Auto Repair Process Evaluation in TTES System

In this section, to verify the rationality of the designed evaluation model, the process of determining the evaluation index is introduced by taking the auto repair major of a secondary vocational school as an example.

Evaluate the learners according to the training requirements of “automobile maintenance and inspection training” [50] and “technical specifications for automobile maintenance, inspection, and diagnosis” GB/t18344) [51]. In the early stage, according to these standards and outlines, suggestions given by experts to determine the key indicators of the evaluation to determine these process data sources. We need to classify, summarize, and analyze to make the assessment more accurate and more targeted. A learner’s “learning skill” is a qualitative variable that is difficult to measure directly. However, we can use some quantitative variables such as time, the number of errors, and test scores to calculate how well students learn in VR environments [52,53]. Like the research of Qiu et al. [54], by exploring the relationship between behavior data and performance, we can mine adequate E-learning behavior feature space.

According to the skills assessment syllabus, literature reading, learning objective analysis, questionnaire survey, and interview methods, the evaluation indicators of skills learning of the virtual training system are constructed. Through the target analysis method, the evaluation indicators are deconstructed from the skills learning objectives of the virtual training system and are designed into questionnaires. The respondents mainly include VR technical experts in the field of educational technology, graduate students in related research directions, as well as experts and teachers in the field of auto repair who are engaged in the research of this maintenance specialty. A total of 30 questionnaires were distributed and 28 were recovered with a recovery rate of 93.3%. Finally, the skill learning evaluation index of the virtual training system is optimized and processed. These indicators can be divided into three sub-goals: knowledge acquisition, skill mastery, and ability development (knowledge, skill, ability, KSA), with skills as the core and diversified knowledge as the carrier [55]. Because of this,

Table 3. Process evaluation index table.

First Level Evaluation	Secondary Evaluation	Data Sources
Total score (Score)	Knowledge acquisition K	Courseware click-through rate e, video viewing duration g, selection error times w, viewing resource times o, test knowledge point error rate c and other data
	Skill mastery S	expected maintenance duration a, actual maintenance duration f, response duration m, part operation position l, tool selection times h, tool use method d and other data
	Ability development A	return learning times b, device location data i, post times q, reply times t and other data

is finally summarized. According to the three sub-dimensions, we adopt the three-level hierarchical structure of the evaluation index system. The first-level criterion is the abstract classification of the second-level indicators, the second-level indicators constitute the evaluation factors of the observable learning process measurement, and the third-level data source is the learning process data obtained by the VR system. These indicators are the processes and results of data of learners’ learning and operation in the virtual training system. The specific dimensions and indicators are divided as follows.

In addition, through interviews with experts, the evaluation scope is given for the key indicators of each dimension, the evaluation formulas of specific indicators are jointly agreed upon by the R&D design team and the experts of secondary vocational teachers, and the relevant definitions are as follows:

4.1. Knowledge Acquisition ($K$)

Knowledge acquisition refers to the amount of knowledge that learners can use in the virtual training system, and this amount of knowledge is used to deal with problem-solving and skill training. It is a key part of the learning effect of skill training and an essential factor to stimulate and maintain the internal motivation of learners. The secondary indicators are as follows:

The success rate of maintenance selection ($y_1$) refers to the selection operation data of tools and parts, specifically referring to the number of wrong selections of tools and parts w in the operation and learning of the virtual training system. Compare the number of times to obtain the scoring standard of this index, specifically:

$$y_1=\left\{\begin{array}{ll}10\hfill & \left(w=0\right)\hfill \\ 5\hfill & \left(1\le w\le 3\right)\hfill \\ 2\hfill & (3<w\le 5)\hfill \\ 0\hfill & (w>5)\hfill \end{array}\right\}$$

(1)

Among them, according to the survey of experts, the number of mistakes in selecting tools and parts can reflect the students’ knowledge mastery in practice. If the number of mistakes in selecting tools and parts is 0 times, and students with good knowledge mastery can get full marks. If the number of mistakes is 1 to 3 times, the score is 5. If the number of mistakes is more than 5 times, the score is 0, which indicates that students’ knowledge of relevant content is not good.

The use of maintenance reminders ($y_2$) refers to the information of Supporting Materials such as feedback on operation steps that students view during the operation assessment process. This system specifically refers to the number of times to view learning tips o. The scoring standard is as follows:

$$y_2=10-2\ast o \left(o is a positive integer and o ϵ \left[0, 5\right]\right)$$

(2)

where the number of viewing resource times can reflect students’ knowledge grasp, and students’ scores will decrease with the increase of checking learning prompts.

4.2. Skill Mastery ($S$)

Skill mastery refers to the results of maintenance skills training. It verifies whether the complex activity with a high simulation degree has contributed to the transformation of knowledge and experience. Its proficiency determines whether the trainee can achieve fluency in operation in the learning process. It is also the active investment and participation of learners in virtual learning activities, which reflects the learning behavior and process. One of the secondary indicators is maintenance efficiency ($y_3$), which refers to the ratio of the duration of the students’ maintenance operation process to the specified maintenance duration. Further, the operation and maintenance process is divided into grading according to the step-by-step stages. For the maintenance event of a step, the specified maintenance time a, the maintenance response time m, and the actual maintenance time f of each step are obtained. The expert finally gives the calculation formula of maintenance efficiency, and determines the interval combined with the data given by the Bayesian network, specifically as follows:

$$y_3=\left\{\begin{array}{ll}10\hfill & \left(\left(m+f\right)/a\le 1\right)\hfill \\ 8\hfill & (1<\left(m+f\right)/a\le 1.2)\hfill \\ 5\hfill & (1.2<\left(m+f\right)/a<1.5)\hfill \\ 0\hfill & (\left(m+f\right)/a>1.5)\hfill \end{array}\right\}$$

(3)

Among them, when the ratio of learner’s response duration plus the actual maintenance duration and the expected maintenance duration is less than or equal to 1, it indicates that the learner’s operating proficiency is higher and the score is higher. A higher ratio indicates that learners need more time in the response process or in the maintenance process, indicating that the operation is not skilled enough.

The maintenance success rate ($y_4$) refers to the parts operation position l and tool usage method d in the system log file during the maintenance operation, of which the full score is 10 points. The two scores are weighted to obtain the score of the maintenance success rate. Parts of operation position l and tool use method d, with two items have equal weight in the success and effectiveness of maintenance, and the score weight ratio being 1:1. Among them, the part operation position l refers to the trigger range of the tool and parts to achieve the maintenance effectiveness during the system setting operation process, and the set threshold value. Tool use method d includes whether the tool is used correctly, including the direction of use, angle, and the number of operations, etc. When any of these do not meet the training rules for system construction, the corresponding tool use method d is 0. Conversely, when the tool is used correctly, the corresponding d is (1) The specific ratings are:

$$y_4=l+d\left(l\in \left\{\begin{array}{ll}5\hfill & \left(l\le 20\%\right)\hfill \\ 2.5\hfill & (20\%<l\le 30\%)\hfill \\ 0\hfill & (l>30\%)\hfill \end{array}\right\},d\in \left\{\begin{array}{ll}5\hfill & \left(d=1\right)\hfill \\ 0\hfill & \left(d=0\right)\hfill \end{array}\right\}\right)$$

(4)

where, the parts operating position within 20% of the trigger range gets full marks, more than 30% is not scored. Full marks are given for correct tool use, but no marks are given for incorrect tool use. The success rate of maintenance is equal to the sum of the two scores.

4.3. Ability Development ($A$)

Ability development is an important factor affecting students’ learning effects, and it is the internal motivation of students to achieve efficient learning. A good learning attitude, self-efficacy, and reflection affect the depth of students’ learning. The secondary indicators are the maintenance rework rate ($y_5$), which refers to the number of times students restart maintenance or return to learning scenarios due to unskilled operation during the maintenance process. The more times, b, the lower the self-efficacy of students; that is, students have low confidence in whether they can use their knowledge and skills to complete learning tasks. The specific ratings are:

$$y_5=10-2\ast b \left(b is a positive integer and b ϵ \left[0, 5\right]\right)$$

(5)

Among them, the return learning times indicates that they are uncertain about their own mastery. The more times they return to learning scenarios, the more uncertain they are about their own mastery and the lower their confidence level.

Maintenance specification ($y_6$) refers to whether students conform to the 7S management concept of automobile maintenance during and after the operation, including but not limited to the cleanliness of the maintenance site, operation safety specifications, etc. It mainly characterizes the position data i of parts and equipment in their original positions. The specific ratings are:

$$y_6=10-2\ast i \left(i is a positive integer and i ϵ \left[0, 5\right]\right)$$

(6)

where the positioning of tools and parts can reflect the operation specifications of learners after the training. When all the items meet the conditional standards, the corresponding maintenance specifications will be scored 10 points. If any item is violated, 2 points will be deducted.

Data processing measures include data collection, mining, and analysis [56]. The original learning data collected from students is often multi-source, heterogeneous, and stored in different subsystems; it is the basis of a scientific and comprehensive evaluation to restore as much as possible the process that occurs during students’ training and learning. This requires a comprehensive and complete understanding of the data of each platform, and the data is processed. The specific processing method is as follows. Suppose we have a group of students, denoted as S = {s₁, s₂, …, s_n}, and the data recorded by the system during a certain virtual reality auto repair training operation of student’s, denoted as V = {v₁, v₂, …, v_n}. Each piece of operation process data v recorded by the system includes the number of equipment selection errors w, the number of viewing resources o, the expected maintenance time a, the actual maintenance time f, the response time m, and the number of return learning b during the auto repair training process, device location data h and other detailed data. Figure 2 is the process of systematically processing the data collected by the system, and the specific steps are as follows:

• Data normalization

The quality of students’ training operation data directly affects the effect of the evaluator. Therefore, the data of the training operation process obtained from the system should be cleaned first. There is no unified process in the data cleaning process, and the method must be selected according to the actual situation of the data to deal with missing values, duplicate values, and outliers. At the same time, the student operation data recorded in the system is multi-dimensional, and the operation data of different dimensions are often not comparable in value, so feature selection cannot be performed. This model solves this problem by normalizing the Z-score of the training operation data of different dimensions.

Define students’ training operation original data set with V = {v₁, v₂, …, v_n} and standardized data set with V = {v′₁, v′₂, …, v′_n}. Among them, v_nm represents the n_th operation behavior of student m in the system, and v′_nm represents the nth operation behavior of student m after normalization. The calculation formula of v′_mn is as follows:

$${v^{\prime }}_{nm}=\frac{v_{nm}-u_{v_n}}{{\sigma }_{v_n}}$$

(7)

Among them, $u_{v_n}$ represents the average value of the nth operation behavior data, and ${\sigma }_{v_n}$ represents the variance of the n_th operation behavior data.

• Feature selection

Feature selection is to select the features that have a greater impact on the training model from all features, thereby reducing the feature dimension and improving the operating efficiency and interpretability of the model. In this paper, the variance filtering method is used to perform feature dimension reduction selection on the auto repair operation data. This is a class for filtering features by their own variance. For example, the variance of a feature itself is very small, which means that the samples have basically no difference in this feature. Most of the values in the feature may be the same, or even the value of the entire feature is the same. Then, this feature has no effect on sample distinction. Therefore, it is necessary to preferentially eliminate features with a variance of 0. The Variance Threshold has an important parameter threshold, which represents the threshold of variance, which means that all features with variance less than the threshold are discarded. If not filled, the default is 0, that is, all records with the same feature are deleted.

Define the eigenvalue variance set T = {t′₁, t′₂, …, t′_n} of each operation of auto repair, where t′_n represents the eigenvalue variance of the n_th auto repair operation of all students, v′_nm represents the eigenvalue of the n_th auto repair operation of student m, and the calculation formula is as follows:

$${t^{\prime }}_n=\frac{{{\displaystyle \sum }}_{i=1}^m{\left({v^{\prime }}_{nm}-{\mu }_{{v^{\prime }}_n}\right)}^2}{n}$$

(8)

Among them, ${\mu }_{{v^{\prime }}_n}$ represents the average value of the n_th standard auto repair operation data. Traverse the elements in V and compare them with the variance threshold. If the current auto repair operation feature value is greater than the threshold, add the corresponding auto repair operation feature to the core set. Do not add it otherwise.

5. Experiment Design-Accuracy Verification of Process Evaluation Method

This section elaborates on two aspects, the first aspect being the experimental design, and the second aspect is the experimental results and analysis.

5.1. Experimental Design

This experiment is based on our proposed process evaluation algorithm as shown in Figure 2, using real experimental data to train a variety of machine learning process evaluation predictors, 80% of the data is used for training the model, 20% is used for prediction verification, and then through the three MAPE, RMSE, and R Squared indicators for comparative analysis. The proposed model is verified by numerical value [57].

Experimental environment: The desktop computer builds the experimental environment. The hardware equipment consists of an AMD Ryzen 3600x processor which is manufactured by Superway Semiconductor Corporation in Santa Clara, CA, USA, and sourced from Suzhou Industrial Park in Jiangsu Province, 32G memory, and 1T hard disk. In terms of software, we use the Windows 10 operating system, its version number is 21H2 and the Python 3.8 programming language, and conduct experiments on the Jupyter lab 3.0.1programming tool.

• Data Sources

By putting the Lishi Evaluation Auto Repair System into Hangzhou X Secondary Vocational School, a training experiment was carried out. A total of 69 students in two classes used the system to learn and collected their training process data. Use the techniques in Section 4 for data cleaning and use this data as the dataset required for the experiment.

The experiment uses linear regression, decision tree regression, support vector machine regression (SVM regression), KNN regression, random forest regression five machine learning algorithms for testing. The performance of various machine learning evaluators is evaluated through experiments, mainly through three indicators: Mean Absolute Percentage Error (MAPE, Mean Absolute Percentage Error), Root Mean Squared Error (RMSE, Root Mean Squared Error), and R Squared (r2 score). The final experiments show that KNN regression has the most stable performance, and it is used to train the evaluator model and deploy it into the system. Algorithm 1 shows the pseudo-code of the prediction algorithm of this system.

Algorithm 1 Procedual evaluator of students’ practical operation performance

Input:V{v1, v2, … …, vn}, THRESHOLD, EVADATA Output: EVALUATIONY  // First, Eliminate outliers and fill in missing values   1: B{b1, b2,… …, bN} ← V{v1, v2,… …, vn}  // Features with a training-set variance lower than the threshold should be removed 2: For each column features bi∈B do 3: bi_var ← POW (x − MEAN(bi), 2) for x in bi 4: if bi_var < threshold then 5:   continue 6: else 7:   Add bi to B_fsvar  // Initialization Original Data (OD) 8: OD ← B_fsvar  // Standardization for data 9: For (i = 1 to number of features of (OD) do 10:    MEANi ← The mean of column i of OD 11:    SDi ← The standard deviation of column i of OD 12:    STDi ← ( ODi - MEANi )/ SDi  // Dividing the standardized data into validation and test sets 13: trainData, testData ← train_test_split(STD, test_size=0.2)  // Take out training data and training labels 14: xTrain ← the first column to penultimate column of trainData 15: yTrain ← the last cloumn of trainData  // KNeighborsRegressor Is a KNN regression function interface in sklearn 16: modelKNN ← KNeighborsRegressor()  // Model Training 17: modelKNN.fit(xTrain, yTrain )  // the true data which need to evaluate 18:  import EVADATA  // Evaluation the true data’s score 19:  evaluationY ← model.predict(EVADATA )  // evaluationY is the output of the predicted score 20: Return evaluation Y

Procedurally evaluate the performance of students’ training operations. First, remove outliers from the original data and fill in the default values, and then remove eigenvalues with small correlation coefficients through feature selection. The data after feature selection is initialized and standardized, the training set and the test set are divided by a random function, and the evaluator is obtained by fitting the K-nearest neighbor algorithm. Finally, the real data to be evaluated is input into the evaluator, and the corresponding predicted evaluation score Y is output.

5.2. Experimental Result and Analysis

For the practical training evaluation prediction model composed of five machine learning algorithms, linear regression, decision tree regression, support vector machine regression (SVM regression), KNN regression, and random forest regression, we seriously analyzed the value of R Squared (r2 score), RMSE (Root Mean Squared Error), and MAPE (Mean Absolute Percentage Error) of the prediction model based on these five algorithms.

The value of MAPE (Mean Absolute Percentage Error) is analyzed in detail below. AVG in Figure 3, Figure 4 and Figure 5 represent the average value of five algorithm prediction models (linear regression, decision tree regression, SVR, KNN, and random forest regression). As shown in Figure 3, the R-squared index values of five algorithm prediction models are described. The value of R Squared index is distributed between 92.91% and 98.71%. The average value AVG of the R squared index of five commonly used machine algorithm prediction models is 96.50%, indicating that the fitting degree of the experimental group and the control group is very high, indicating that the evaluator has high accuracy in evaluating learners, and the R squared score value of KNN is the highest, indicating that the fitting degree of KNN algorithm model is higher and the prediction is more accurate. Figure 4 describes the RMSE values of the five algorithm prediction models, the value of RMSE ranges from 31.61% to 74.05%, the average AVG is 48.75%, it can be seen that the RMSE value of SVR is the largest, and the RMSE value of KNN is the smallest. The root mean square error is used to measure the deviation between the observed value and the true value, indicating that except for the prediction model of SVM and decision tree algorithm, the difference between the predicted value and the real result of other models is small, while the deviation between the predicted value and the real result of the prediction model of KNN algorithm is the smallest. Figure 5 describes the MAPE values of the five algorithm prediction models. MAPE itself is often used as a statistical indicator to measure the prediction accuracy. MAPE can describe the accuracy. The value range of MAPE of the prediction model of the five algorithms is 0.13–0.41%, and the average AVG is 0.24%, indicating that the prediction error of the evaluation model is very small. The KNN algorithm prediction model also has the smallest MAPE value, indicating that the KNN algorithm model has a smaller error in the prediction score and the original expert scoring.

Further analysis from Figure 3, Figure 4 and Figure 5 show that, among the five algorithm prediction models, the experimental group and the control group have a high degree of fit, and the average absolute percentage error is also small, which indicates that the evaluation model designed in this study can effectively assist teachers to evaluate students more accurately. Among them, KNN has the largest R Squared Score, and RMSE and MAPE have the smallest value. According to specific indicators, KNN algorithm has the smallest error in the prediction model, which is closer to the expert rating. Therefore, KNN algorithm has the strongest stability in this system model. Finally, KNN algorithm is deployed in the system. In the later stage, more labeled real data will be added to the evaluator model, and the iterative procedural evaluator model will be constantly updated to make its performance more powerful and its evaluation more accurate.

6. Conclusions and Future Work

In the context of the networked era, students are engaged in richer learning, so the evaluation of students should be more multimodal, especially in the field of vocational education. In this paper, we focus on the field of practical training education and build a model of a secondary skill-based practical training process data evaluation system in a virtual reality environment, which consists of four subsystems. These are namely VR practical training process activity recording, process data processing and storage, practical training process data analysis, and practical training process evaluation. The evaluation index system of the process evaluator is constructed in the subsystem of the process evaluation model of practical training, and the learners are evaluated from three dimensions on their knowledge acquisition, skill mastery, and ability development. When a new user uses this evaluation system, the system can score the user’s practical training. In this paper, we train an evaluator model by machine learning and demonstrate through experiments that our approach can accurately evaluate the process of the trainers by taking the practical training of secondary vocational auto mechanics as an example.

In the process of algorithm validation of the evaluator model, this study mainly adopts five algorithms suitable for this model: Linear Regression, Decision Tree Regressor, SVR, KNN, and Random Forest Regressor. In the actual validation process we found that the KNN algorithm is more effective in predicting the overall model. We found that the KNN algorithm was more effective in predicting the overall model, and its accuracy rate reached over 90%. Moreover, the evaluator model can be continuously updated and iterated with more real data in the future, and its process evaluation of students’ practical training operations will be more accurate. This study allows teachers and students to more accurately identify problems in practical training operations by providing accurate feedback on the skill-based practical training process in auto mechanics. Teachers can better plan their teaching priorities and students can better master their professional skills and improve them in a targeted manner.

In this study, our proposed model of a process evaluation system in a virtual reality environment can indeed solve the problem of inaccurate and non-objective evaluation in skill-based practical training, and the method has been verified to accurately evaluate learners’ practical training by evaluating it in actual teaching. However, the proposed model is only a small-scale study with the example of practical training in auto mechanics, and the limitation of small sample size limits the generalization of its findings. In the future, it is necessary to study the expansion to different grade levels as well as other practical training specialties. It can promote the wide application and model innovation of process-based assessment services for skill-based practical training learning.