Design of Augmented Reality Training Content for Railway Vehicle Maintenance Focusing on the Axle-Mounted Disc Brake System

Abstract

Light maintenance training for electric multiple-unit components of the organization of railway operations is generally conducted using maintenance manuals and work videos, following the guidelines of each organization. These manuals are in the form of booklets, complicated and inconvenient for maintenance operators to carry. Therefore, training content that visualizes maintenance procedures in a three-dimensions (3D) space is necessary to overcome the drawbacks of booklet-type training. In this study, we developed augmented reality (AR)-based training content for railway vehicle maintenance to increase training efficiency. Providing warning signs for risky procedures reduces human error, and transparency control makes trainees check the product hierarchy. A virtual experience based on the maintenance manual is provided to improve maintenance proficiency. An axle-mounted disc brake system maintenance manual is implemented in AR to reflect the requirements of maintenance operators. The convenience of this tool is improved by loading the AR content on a mobile device. Two methods of verification were used: the system usability scale (SUS) survey and training efficiency evaluation. The resulting SUS grade was B (excellent), and the training efficiency improved by 34%.

1. Introduction

1.1. Research Background

Railway vehicle maintenance plays a key role in railway asset management and contributes to operational performance. Because it is performed in a work environment that requires high adaptability and expertise, systemic training is essential [1]. Light maintenance training for electric multiple-unit components of railway-operating organizations is typically conducted using maintenance manuals and work videos, following the guidelines of each organization. However, the drawings and maintenance manuals of lower-body components/parts are so heavy that difficult to carry, making it difficult to share information among team members in the field. Furthermore, for novices, there is no way to learn how to perform difficult or dangerous maintenance work owing to the age of existing workers. As a solution, a 3D visualization of the disassembling/assembling procedure before the commencement of maintenance work can improve readability compared to conventional manuals [2]. A study has shown that the 3D visualization of the maintenance procedure improves training efficiency by approximately 90% compared to the document method [3]. Moreover, it has been reported that realistic (real behavior) training improves training efficiency by a factor of 2.7 and the concentration by a factor of 2 compared to analog training [4]. Therefore, to overcome the drawbacks of booklet-type training and to improve proficiency in the maintenance of components, it is important to develop AR-based training content.

1.2. Related Work

1.2.1. AR-Based Maintenance Training Content

AR-based maintenance training content has been researched in various fields to determine an efficient method of maintenance. A framework that forms a systemic procedure for railway vehicle maintenance was presented in the work of [5]. This framework provides the necessary data for a realistic maintenance situation. In addition, a case study targeting 28 participants demonstrated that this framework enhances the maintenance capabilities and skills of the operator. An aircraft maintenance manual that applied AR technology was presented in the work of [6]. This study’s manual used the leakage procedure of a hydraulic valve as an example procedure. As a result, it proved that the manual improved maintenance efficiency. Ortega et al. [7] presented a system called Mantra, which was combined with AR technologies. They demonstrated the effectiveness of the system based on real use cases by comparing it with an existing method and applying the system to industrial maintenance. Borro et al. [8] presented a bus maintenance system based on an AR system and wearable devices. They tested the system under actual operating conditions and demonstrated the excellent performance of the system. In addition, this system is convenient because it can be connected directly to the database server.

1.2.2. Effects of AR Content

Research has continuously attempted to verify the effects of the AR content. A platform called PlanetarySystemGo was designed to improve the learning of planetary systems [9]. This platform received a positive response from participants because it was easy to use, and the researchers concluded that participants would use this platform. Nadeem et al. [10] presented an AR application for laboratory orientation. Three surveys were conducted before and after the implementation of the application. According to the surveys, participants got used to the laboratory quickly and responded positively to the convenience of use. Christopoulous et al. [11] presented a theoretical framework for improving the education of students. The main discussion topics were parameters and constructions for improving the effectiveness of education. The conclusion was that this framework could bring out the full potential of students. Wijaya et al. [12] presented a markerless AR system used in food advertising. A SUS survey of 20 participants showed a mean score of 51 points, which leads us to conclude that the system is still unstable to use. An evaluation of acquiring spatial knowledge by using mobile maps is presented in the work of [13]. According to the results of the SUS survey, the interface for wayfinding contributes to the improvement of usability. In addition, the participants felt comfortable when they used the interface with their mobile devices. Fernandes et al. [14] presented an application called NitLabEduca, which is a useful tool in spinal cord education. To demonstrate its usefulness, two groups of 40 participants performed learning tasks. The SUS score was grade C, which means that the system was adequately useful.

1.3. Purpose of Study

Various studies have used feedback systems to verify the effectiveness of AR-based content. In the railway vehicle field, however, no research has applied AR technology to the maintenance of the main product using a feedback system through a training evaluation or SUS survey. The purpose of this study is to develop AR-based training content for railway vehicle components (i.e., axle-mounted disc brake systems) to increase the efficiency of maintenance training. In addition, this study suggests a way to improve training efficiency and reduce human error. Two methods of verification were used: a SUS survey and a training efficiency evaluation.

2. AR Content Development

2.1. AR for Railway Vehicles

AR is a technology that projects virtual images in the real world. Users can interact with a virtual object overlaid on a real image [15]. AR-based maintenance content has been used in a limited range of fields (e.g., industrial sites, the aviation industry, and the automobile industry) [5,6,7,8]. However, AR content specialized for railway vehicle maintenance is very scarce. Therefore, it is necessary to develop content using AR technology for railway vehicle maintenance.

2.2. Requirement Determination

The survey was conducted twice, divided into a first phase (criteria for product selection) and a second phase (system priority) to determine requirements for designing AR content for railway vehicle maintenance. The targets of the surveys were railway experts working for Korean metropolitan railway-operating organizations or Korean railway vehicle manufacturing companies.

The first survey was conducted over 18 days with 10 considerations to set the criteria for product selection (from 8 to 26 June 2020). The response rate was 70%, with 38 of the 50 participants responding. As a result, three criteria were selected: difficulty, frequency of use, and risk of work. Figure 1 represents the weights of the criteria for product selection.

The second survey was also conducted over 18 days to set the priority of the system for effective use of content (from 6 to 24 July 2020). This survey targeted 18 types of products classified as EN 15380-2: 2006 [16]. The response rate was 81%, with 73 of 90 participants responding. Figure 2a shows the top six products as a survey result; the first-place product was a brake with 31%. An additional survey assessed the training priority for the hierarchy of brakes. Figure 2b represents the priority of light/heavy training for the hierarchy of brakes. The first-place priority was the light maintenance of the axle-mounted disc brake system at 43%.

2.3. Axle-Mounted Disc Brake System

The axle-mounted disc brake system transmits the brake force to both sides of the disc installed on the axle by causing friction with the brake pad as shown Figure 3. This system has no vibration in the vehicle body and is suitable for high-speed railways. It is used on trailer bogies because they have sufficient space to accommodate such a system [17]. Additionally, it is hard to conduct a visual examination since the axle-mounted disc brake system is installed at the bottom of the railway vehicle. This product is a driving safety product that requires disassembly/assembly of the disk when the problem breaks out. As a light maintenance procedure for the axle-mounted disc brake system, there are three examination steps: daily examination in service, step 1 (every 3500 km, 7 days), and step 2 (every 60,000 km, 4 months). An examination is performed in which the technician checks the brake release and performs a limited inspection that measures the wear of the brake disc.

2.4. Designing AR System

Figure 4 shows a flow chart of the AR content for railway vehicle maintenance. The axle-mounted disc brake system implemented in this content was converted CAD modeling from Sap 3D Enterprise Author [18] to mesh modeling. Each modeling was produced by merging it in Blender [19]. The implemented modeling was optimized to reduce the number of draw calls. In the case of the axle-mounted disc brake system, the modeling implementation took 3 weeks, and the size of the file was about 60 MB, which is compressed approximately 48% compared to CAD modeling. According to the result of three times of the self-output test for each file size, it was possible to make the stable output of mesh modeling files, which is below 100 MB. The AR display of the content is composed of the main scene, which identifies the product, and the content scene, which shows the product hierarchy. In this context, “trainee” denotes a maintenance operator who uses this content.

As shown in Figure 5a, the main scene used marker-based technology allows the trainees to identify the product when the QR code in the image library on the screen of the camera is recognized. As shown in Figure 5b, the content scene used a markerless technology that allows trainees to access the manual of each product that appears on the AR display through plain detection of the object by using the AR foundation package of Unity [20]. The maintenance procedures are divided into steps, and trainees can access a detailed procedure and play an animation for a given step.

Additionally, this content adopted an operation method that can control the position, scale, and rotation of a 3D model through the touch method.

Table 1. Details of 3D model transformation.

Value to Change	Corresponding Touch Gesture		Explanation
Position	Swipe		Move to the desired position by touching and sliding
Scale	Pinch/Spread		Touch two fingers to increase/decrease the gap Change the size of the 3D model
Rotation	Twist		Touch two fingers to rotate in a circle Rotate the model

shows the detailed functions of 3D model transformation. Trainees can use content without AR glass in the work site using the mobile device with such operation method. Therefore, the trainee can easily access the maintenance manual with a mobile device (tablet) placed in the work site or their own mobile device (smartphone).

Figure 6 shows the user interface, which is a basic screen with a scrollbar format. Trainees can access the maintenance procedure through the “Quick Menu” classified according to the maintenance manual.

Figure 7 shows the maintenance manual implemented by the tap menu. The tap menu offers a list of each simplified maintenance manual. Furthermore, the trainee is able to conveniently switch between each list. This system enables the trainees to receive repetitive training through simplified maintenance manuals by selecting detailed maintenance procedures. This training produces the same effect as a private lesson.

2.5. Features

2.5.1. Reducing Human Error

Figure 8 shows an example of AR content designed to reduce human error during the light maintenance procedure of the axle-mounted disc brake system. Figure 8a,b represents the dangerous and difficult maintenance procedure that occurs when components are disassembled from cases of actual accidents. A warning sign for dangerous/difficult maintenance procedures appears directly to the trainee, as shown in Figure 8a,b, which makes trainees more alert. Figure 8c illustrates the knowledge of veteran maintenance operators. Knowledge visualized with an orange comment/tag prevents risks caused by the lack of transmitted experience. Furthermore, this leads to a reduction in human error.

2.5.2. Improving Training Efficiency

Figure 9 shows the functions of the content to improve training efficiency. Figure 9a shows a checklist displayed on the maintenance manual. Trainees can experience an effect comparable to prior learning if the visualized checklist is offered before beginning work. As Figure 9b shows, a screen that indicates the current maintenance location in the product hierarchy is shown in the upper right corner. This helps the trainee understand the entire maintenance procedure. Additionally, the trainee can control the transparency of the product subcomponents during maintenance, as shown in Figure 9c. In this way, even novices can easily understand the connections between product subcomponents.

2.5.3. Virtual Experience Based on Maintenance Manuals

Figure 10 shows an example of virtual experience implementation for improving maintenance proficiency. The trainee can experience measuring the wear of the brake disc, as shown in Figure 10a. Visualizing the location/number of dimension measurements based on the maintenance manual can reduce the uncertainty in the dimension measurement. The functional check procedure of the maintenance manual is visualized in Figure 10b,c. In AR, the trainee can experience the tasks described in the maintenance manual in a way that is similar to a real working environment.

2.6. Differentiation

Figure 11 shows a method for measuring the frame rate of AR content for railway maintenance. As the figure shows, trainees can access the object in AR in real time because the average FPS is above 30, which makes the output stable [21].

Figure 12a illustrates the method of training the brake disc exchange procedure by loading the content on a tablet, and Figure 12b shows the method of training the axle-mounted disc brake system check procedure by loading the content on a mobile device. As Figure 12 shows, the trainee can learn the maintenance procedure individually in the worksites by using mobile devices. AR content can replace the process of searching information in a booklet-type manual and supplement the inefficiency of existing collective education, which is constrained by time and space. Additionally, the user interface of this content is designed to respond to the aspect ratio of each device. This interface supports various resolutions and can be used easily on Android-based devices. Trainee can recognize AR environment space and markers using its own camera equipped with the flash of the mobile device without using an additional light device. Rapid movement of the camera screen can cause the missing object’s tracking. Thus, users have to be careful when moving the screen for smooth AR implementation. Additionally, to prevent damages to mobile devices due to falling and impact, it is recommended to use protective gears such as protective films and covers to secure solidity.

3. Usability Verification

3.1. SUS

This study used the SUS survey to evaluate the usability of the AR-based training content. SUS, developed by John Brooke [22], evaluates the usability of the content with 10 questions, including both positive and negative questions. The response to each question was organized using the Likert scale: a score of 5 means “strongly agree”, and a score of 1 means “strongly disagree”. The SUS questionnaire is presented in

Table 2. SUS questions.

Questions	Content
Q1	I think that I would like to use this system frequently
Q2	I found the system unnecessarily complex
Q3	I thought the system was easy to use
Q4	I think that I would need the support of a technical person to be able to use this system
Q5	I found the various functions in this system were well integrated
Q6	I thought there was too much inconsistency in this system
Q7	I would imagine that most people would learn to use this system very quickly
Q8	I found the system very awkward to use
Q9	I felt very confident using the system
Q10	I needed to learn a lot of things before I could get going with this system

.

The SUS score is determined by Equations (1)–(3) [12]:

Score for Q1, Q3, Q5, Q7, Q9 = Scale Score − 1

(1)

Score for Q2, Q4, Q6, Q8, Q10 = 5 − Scale Score

(2)

SUS Score = Sum of Scores × 2.5

(3)

Equation (1) is used to calculate the results of the positive questions: Q1, Q3, Q5, Q7, and Q9. Equation (2) is used to calculate the results of the negative questions: Q2, Q4, Q6, Q8, and Q10. Equation (3) is used to obtain the SUS score.

3.2. Verification Conditions

The overall information of the survey participants is presented in

Table 3. Overall information of survey participants.

Total Number of Participants	Average Age	Average Career Length	Maintenance Operators	Experienced Operators
100 people	32.8 years	4.7 years	69 people	21 people

. The total number of participants was 100, of which 69 were maintenance operators working at a Korean metropolitan railway-operating organization. A total of 21 out of 69 (30.4%) were experienced in light maintenance of the axle-mounted disc brake system. The average age was 32.8, and the average career length was 4.7 years.

Table 4. The age distribution of survey participants.

20–29 Years	30–39 Years	40–49 Years	50–59 Years	60–69 Years
30 people	30 people	20 people	10 people	10 people

represents the age distribution of survey participants. The participants were who have various ages and experiences, including managers, veterans, and novice maintenance workers. Through this, surveys can collect opinions from all ages.

The survey participants responded to 10 SUS questions after undergoing training for the light maintenance procedure of the axle-mounted disc brake system implemented in AR using mobile devices for 30 min.

3.3. Results

Figure 13a shows the score distribution of the survey participants’ SUS evaluation results. As a result of SUS evaluation, 24 percent of the participants yielded scores under 60, the average minimum score to pass general tests, and 76 percent of the participants yielded scores over 60. This means that most of the participants were very satisfied with the usability of the content. Figure 13b shows the mean score for each question. The mean scores of the positive questions (Q1, 3, 5, 7, and 9) was 84.12, and that of the negative questions (Q2, 4, 6, 8, 10) was 62.02. The mean score of all questions was 73.36, which is higher than the average SUS score of 68. SUS scores are divided into seven adjective ratings. This content had more than 71.4 points and thus obtained a B grade (Excellent), which means that the system is “useful” [23,24].

Figure 14 shows the SUS evaluation results as a stacked graph. In terms of question 6 (uniformity), around 39% of participants responded with “strongly disagree”. Therefore, it was confirmed that the content lacked consistency. In addition, it is necessary to improve the convenience of use, as around 35% of participants responded with “strongly disagree” to question 8 (convenience). In contrast, roughly 70% of participants responded with “strongly agree” to question 3 (simplicity). This confirms that the content is intuitively structured. Moreover, approximately 57% of participants responded with “strongly agree” to questions 5 (integration) and 7 (learnability). This indicates that this content has a well-organized user interface, making it easy to perform training.

According to the results, the usability verification shows that most users were satisfied with the content. These results also demonstrate that the content is convenient to use and suitable for railway maintenance training.

4. Training Efficiency Evaluation and Statistical Analysis

Following the previous usability verification, a training efficiency evaluation was conducted to verify the suitability of the content. A t-test was used to analyze the evaluation results. The mean scores of the two groups were compared and statistically analyzed.

4.1. Evaluation Conditions and Questionnaire

4.1.1. Evaluation Conditions

The targets of evaluation were 60 college students majoring in the railway vehicle field. To exclude the effect of prior experience, the participants were college students who did not have experience in railway vehicle maintenance.

Table 5. Overall information of evaluation participants.

Total Number of Participants	Average Age	Sex
60 people	25.1 years	Male: 52	Female: 8

shows the overall information of the participants. The average age of the participants was 25.1, and most participants (52) were men.

According to the central limit theorem [25], the evaluation was conducted for 12 days (from 15 to 26 March), dividing 60 participants into two groups (A and B). Group A, with 30 participants, received a 30 min collective education about the light maintenance procedure of the axle-mounted disc brake system through the booklet-type manual used in the operating organization. Group B, with the remaining 30 participants, was trained regarding the light maintenance procedure of the axle-mounted disc brake system through AR using mobile devices for 30 min. Subsequently, the participants in each group responded to the evaluation questions. Figure 15 shows the method of teaching the maintenance process in two groups (A and B).

4.1.2. Evaluation Questions

The criteria for evaluation questions were extracted from the guidelines of the electric vehicle brake system maintenance procedure of the Korea National Competency Standards (NCS) [26]. Evaluation questions were formed based on the advice of railway professors and practitioners with more than 20 years of experience in railway vehicle maintenance. In addition, the survey was conducted to identify whether the participants were acquainted with the procedures and guidelines for the light maintenance of the axle-mounted disc brake system. The survey has a total of 20 questions, consisting of 6–7 questions each for three content features. Each question is worth up to five points, and the maximum score is 100. The reliability of the questions was estimated using Cronbach’s alpha, which measures the degree of consistency between questions. This value is determined by Equation (4).

Cronbach’s alpha = (k × r)/1 + (k − 1) × r

(4)

K is the number of questions, and r is the average correlation coefficient for the questions. The reliability of the questions was graded as “excellent” with Cronbach’s alpha = 0.91 compared to the work of Rovai et al. (0.79), a similar study [27,28].

4.2. Evaluation Results

Figure 16a is a box plot that represents the mean scores of the evaluation results. In group A, the minimum and maximum values were 25 and 80, respectively, and the mean score was 47.2. In group B, the minimum and maximum values were 40 and 90, respectively, and the mean score was 63.5. As a result, both the minimum and maximum values of group B were more than 10 points higher compared to those of group A. Moreover, the mean score increased by 16.2, showing a 34% improvement in training efficiency compared to group A. Figure 16b is an acquired score graph for each content feature. Regarding the reduction in human error, the total acquired score of group B was 17.8, which was 37% higher than that of group A (12.9). It seems that the training efficiency was improved by visualizing the knowledge of veteran operators as a comment/tag in AR content. In terms of improving training efficiency, group B had an efficiency score of 29.2, an increase of 44% compared to group A. This is because the trainee can check the product hierarchy using the transparency control technique of the product subcomponents. In the virtual experience based on the manual, group B received a score of 16.4, which increased by 15% compared to 14.2 in group A. This is because the trainees can experience the functional check procedures given in the maintenance manual in AR.

4.3. Data Analysis

The t-test analysis is a method that tests statistical significance by comparing the mean scores between two groups [29]. In this study, a t-test analysis was conducted to verify the statistical significance of the training efficiency when using the AR content. Figure 17 shows the results of the normal distribution test for the evaluation score of each group using Minitab 20 [30]. The Anderson–Darling (AD) values of the two groups were 0.337 and 0.356, respectively. The probability values (p-value) of the two groups were 0.48 and 0.43, respectively; both values are higher than 0.05. Therefore, the evaluation scores of the two groups were normally distributed.

In this study, the results were analyzed using Student’s t-test [31] of an independent two-sample t-test, which was used to compare the mean scores between the two groups. The null hypothesis was “there is no difference between the means of the two groups”, and the alternative hypothesis was “the means of the two groups differed”.

Table 6. Analysis results of t-test.

Sample	StDev	SE Mean	T-Value	DF	p-Value
Booklet-type	14.03	2.6	4.36	58	<0.0001
AR	14.95	2.7

shows the results of the t-test analyzed using Minitab 20. The standard deviation (StDev) values of the two groups (A and B) were 14.03 and 14.95, respectively; these results are almost the same. The standard error mean (SE mean) values for the two groups (A, B) were 2.6 and 2.7; these results are also almost the same. The test-statistic (T-value) was 4.36, and the degree of freedom (DF) was 58. The p-value was less than 0.0001, which is less than the significance level (0.05), and the alternative hypothesis of “the means of the two groups differed” was validated. Therefore, the mean score increase in group B was statistically significant. This indicates that training efficiency will be improved if AR content is used for railway vehicle maintenance training.

5. Discussion

In this study, AR content was developed, which can be used in various Android-based mobile devices for the maintenance of railway vehicle products.

Table 7. Key features of the AR content.

Features	Actions
Reducing human error	(a) Warning of dangerous and difficult maintenance procedures
	(b) Sharing knowledge of existing veteran maintenance operators
Improving training efficiency	(c) Offering visualized checklist
	(d) Indicating connectivity within product hierarchy
Virtual experience based on manual	(e) Providing virtual experience of measuring dimensions
	(f) Providing virtual experience of functional check operations

shows the key features of the AR content. This AR content visualizes not only simple disassembly and assembly procedures but also the precautions in the manual, product hierarchy, functional checks, and so forth. This is the result of modifying the existing booklet-type manual to reflect the requirements of maintenance operators and to suit the work sites. Therefore, maintenance operators are able to efficiently understand the maintenance process by using this customized information and receiving experience-based knowledge.

In this study, a SUS survey was also used to verify the usability of this content. The SUS survey evaluates the usability of the system performance with 10 questions, including both positive and negative questions. As a result of analyzing the mean scores of all 10 questions and the scores for each question, most participants showed high satisfaction with the usability of AR content. In addition, a test was conducted to evaluate training efficiency by dividing participants into two groups (A and B). By analyzing the evaluation scores of 20 questions using the t-test analysis method, it was confirmed that the increase in the mean score was statistically significant. Moreover, it was indicated that the training efficiency improved in all features by analyzing the earned score for each feature of the content.

However, the survey participants were unevenly distributed, as this feedback work mainly targeted novices who had less than 10 years of experience. Furthermore, this method had a limitation in that there were no analysis results for the elderly because the interview was based on a limited number of participants. Regardless of these limitations, this content has a higher training efficiency for novices who are not used to maintenance procedures. Therefore, it is expected that this content will provide higher training efficiency if it is used as training for the duties of veteran maintenance operators or encouraging maintenance operators to diversify their skills.

6. Conclusions

Prior case studies related to the maintenance of railway vehicle products are insufficient. In this study, AR-based training content for railway vehicle maintenance was developed to overcome the drawbacks of booklet-type manuals and improve the proficiency of maintenance operators. Additionally, AR technology was applied to the maintenance procedure of the axle-mounted disc brake system, which is an important component of railway vehicles. Feedback collection was conducted in two ways: a SUS survey and an evaluation of training efficiency. The conclusions obtained from these efforts are as follows:

• The axle-mounted disc brake system maintenance manual was loaded on the mobile devices, reflecting the requirements of railway vehicle maintenance operators and real situations of working sites. Therefore, maintenance operators can proceed with individual training at a working site without using an existing booklet-type manual.
• To reduce human error, dangerous/difficult maintenance procedures are presented in a way that raises the awareness of operators. Visualizing the knowledge of veteran maintenance operators makes it easier to transfer experience.
• To improve training efficiency compared to that of existing booklet-type manuals, the checklist and top view screen are displayed. Transparency control is applied to the product to display the connections within the product hierarchy.
• The measurement operation and functional check operation are visualized to reduce the uncertainty of the dimension measurement and to provide indirect experience of the working environment. As a result, maintenance operators can experience the maintenance manual in AR, leading to improved maintenance proficiency.
• A SUS survey was conducted to evaluate the usability of the AR content. The mean score was 73.36, and a grade of B (Excellent) was obtained, which means that this content is suitable for use in training for railway vehicle maintenance.
• A test with 20 questions was conducted with two groups to evaluate training efficiency. As a result, the mean score of the group using AR content was 16.2 points higher and showed a 34% improvement in training efficiency.

In future studies, it will be necessary to develop AR content for brake operating units and air compressors and maximize the training efficiency of railway vehicle maintenance.