A Rule-Based System for Human Performance Evaluation: A Case Study

Abstract

The issues of many organizations are related to the proper evaluation of human performance and efficient decision-making. The expert system application within the decision-making process is not a novelty, but the widespread of its implementation regarding performance evaluation has not been recognized yet. To overcome this problem, a case study of rule-based expert system application in the decision-making process regarding human performance in periodical technical inspection stations in Croatia is presented. The rule-based expert system improves the quality of traditional decision-making as designed rules provide a visual, transparent, and accurate comparison of observed values with the expected values. Moreover, it provides easy problem identification. Therefore, rules regarding periodical technical inspection inspectors’ performance are designed and embedded in the expert system architecture. However, more effort should be made into data analysis to define parameters and their relations for the purpose of designing rules. Thus, the binary logistic regression and an ANOVA statistical test were conducted to identify which parameters can be assumed as relevant indicators regarding the performance of periodical technical inspection inspectors. In this study, the expert system application has resulted in faster response, greater efficiency, and increased objectivity. That is of utmost importance for providing an efficient and transparent periodical technical inspection system.

1. Introduction

According to [1], organizations put significant effort into improving the quality of their products and services. Efficient and objective performance evaluation could lead to better and higher quality processes [2]. The performance of organizations is significantly affected by employee performance. Individual performance is a key indicator of the company and contributes to the company’s productivity and competitiveness [3]. Human performance evaluation is an extremely complex problem within most organizations since human judgment has subjective characteristics. Therefore, in order to provide valid decision-making, it is of utmost importance to provide an efficient, transparent, and objective performance evaluation system. The best way to achieve these objectives is to propose the implementation of modern technology and reasoning into the process itself. Modern technology is driven by exact expectations, and it entirely alters the way processes usually flow. This leads to the importance of making real-time decisions. In the past, persons having experience within a certain area were the ones that made the decisions. That ability made them professionals in specific areas of work, i.e., experts. However, the tendency was to inject such capacities of thinking and reasoning into a computer to enable decision-making to less qualified persons. Those were the reasons why back-up systems came into existence. These systems are called decision-making support systems (DMSS). They were a basis for the development of expert systems (ES).

The ES was originally introduced as a branch of applied artificial intelligence by Edward Albert Feigenbaum in the 1970s [4]. They are defined as software apps that are capable of solving specific problems in a way human experts do. Thus, their ability is to perform at a level close to the human expert performance [5]. They are, therefore, highly useful information systems for making decisions in different areas, including in the field of mechanical engineering, science, healthcare, and others. ES are mostly based on rules that represent knowledge from a particular problem area, and it is entered by the user in the knowledge base. In this manner, the user is provided with significant assistance in implementing the expert system within the process since the only task when applying such a system is to add knowledge in the form of rules [6]. The most popular types of expert systems currently applied include Visual Basic, JESS, CLIPS, MATLAB, PyKE, and VisiRule. VisiRule represents the expert system shell [7]. It is a part of the software that contains the user interface, the format for displaying the knowledge base within the specific domain, as well as the inference engine. The inference engine, as the most important ES component, makes a decision when a certain rule needs to be fired. Its purpose is to compare each rule stored in the knowledge base (rule base) with the facts stored in the database. There are two methods of executing a rule-forward chaining and backward chaining, which fire the rules in different manners [6]. Forward chaining is based on data, and it applies rules to make conclusions or new facts by using a top-down approach. Backward chaining is based on objectives to determine which data should be highlighted, using a bottom-up approach [7].

This paper proposes the application of a rule-based ES within a process of making decisions regarding the performance of periodical technical inspection (PTI) inspectors in Croatia. The study is a continuation of the preliminary research conducted in [8], which concluded that by designing rule-based ES, it is possible to achieve the standardization of the process and reduce or even eliminate all types of lean wastes within the process. Moreover, it was found that VisiRule ES shell application within the decision-making process can ease the implementation process and achieve greater efficiency. It was emphasized that the data analysis has a fairly significant influence on the parameter definition as it is necessary to ascertain how many parameters have been defined and their causal relationship with the final outcome.

In order to determine the significance of the research at the very beginning, it is necessary to conduct an adequate statistical analysis to determine the significance and correlation of the observed parameters with regard to the vehicle’s roadworthiness. The main purpose of the proposed ES is to improve the decision-making regarding employee work performance in order to provide rapid and objective decisions. As the objectivity of the PTI system is extremely important for road safety, the work of PTI inspectors, who are the key factor in making a decision regarding the vehicle’s roadworthiness, needs to be analyzed. Legal regulations and professional instructions are the basis for evaluating PTI inspectors’ performance. Thus, the aim of the proposed ES is a proposition of correct decision regarding PTI inspectors’ performance based on entered work performance values, all in order to reduce potential manipulations and unprofessional work to a minimum. Furthermore, the proposed ES could determine PTI inspectors’ work quality by comparing performance values regarding average PTI duration, average vehicle age, and PTI failure rate (Figure 1) with their reference values.

The rest of the paper is organized as follows. The second section provides a literature review regarding ES application. The aim is to determine whether the ES has already been developed and applied in this research area. The third section describes the method used to develop ES. The fourth section presents the design and implementation of the proposed ES in the case of Croatian PTI inspectors’ performance evaluation. Furthermore, the results and ES validation are presented within. In the fifth section, the results are discussed, and in the final section, some final conclusions are provided.

2. Literature Review on Expert System Application

DMSS as pioneers of ES are widely used. Researches mainly focus on practical examples. However, when applying the DMSS, it is necessary that the design and integration of algorithms consider human perspective, not only technological [9]. For example, authors in [10] designed simulation for train drivers training by conducting cognitive task analysis with experts in the domain. Research conducted in [11] reports the design of simulator training within Swedish dynamic decision-making training facilities. Although ES application was not novel, the research area has been very scarce for a long time. The paper [12] provides a review of the rule-based ES application in various manufacturing processes. According to established findings, their advantages are related to easy operation, human error reduction, and possible usage by low and semi-skilled employees. In addition, the field of healthcare is a broad area for the application of ES and artificial intelligence. Especially in the field of diagnostics and decision-making based on predefined parameters. Research [13] was conducted to improve the response rate of general practitioners to poisoning cases in the Philippines. It enabled the diagnosis with an accuracy of 96% by CLIPS ES shell application and a web interface design. It provided a quick response, which is extremely important for a system with some parts dislocated. The same goal regarding providing remote consultation and monitoring of pediatric patients at places distant from medical service was highlighted in [14]. The authors discuss rule-based mobile ES application for pediatric patients. The performance accuracy of the system was 0.86 in the first-round tests and 1 in the second-round tests, demonstrating the great efficiency of the proposed system. Authors in [15] propose multilayered rule-based ES for diagnosing uveitis in order to help other ophthalmologists provide a precise and fast diagnosis. Its multilayered structure presents a quality educational tool that provides system flexibility by allowing easy expansion in the future. Preliminary results of a fuzzy rule-based system for decision support regarding cardiovascular diseases shown in [16] present the effectiveness of the system itself. However, providing more parameters and increasing the knowledge base could provide more accurate decisions and the system’s effectiveness.

The application of the technique for order preference by similarity to ideal solution (TOPSIS) ES based on fuzzy logic in combination with neural networks is presented in the paper [17] on the example of designing a new perfume bottle. It enables easy and successful designing in the process of product design, while it enables the optimal combination of parameters for creating a new product. Preliminary research results of ES application in weapons system operation are presented in the paper [18], which indicate that the introduction of an ES in the processes appears to garner approval. However, some aspects need to be improved, and the application of fuzzy logic can significantly affect the process improvement if there is a lack of specific information. Authors in [19] confirm the above by presenting fuzzy ES for earthquake prediction with 47% accuracy. Due to the lack of application of ESs based on fuzzy logic and designed in the web interface, the paper [20], by using the example of assessment of student success in enrolling into the next academic year, showed the design and advantages of their application. Microsoft Visual Studio and Microsoft SQL Server have been used to design the application system, enabling the implementation of the ES within the process, without the additional investment costs required for the shells. Research in [21] proposed ES that applies the failure mode and effects analysis (FMEA) method for quality control in brake disc production lines. The ES was developed using the Java language and Apache NetBeans platform and included two modules, designed FMEA (DFMEA) as the main module and component FMEA. The research resulted in intelligent ES that provides the operator to carry out a quality control system and to make decisions easily. However, the disadvantage is the impossibility of performing self-updates regarding new data in the knowledge base. The paper [7], by using the example of flood forecasting, determined that developing an ES for a particular purpose is more efficient than generating a procedural code. By applying the backward chaining method, the authors concluded that in the case of flood forecasting characterized by setting objectives by the flood manager, this method is more efficient than the forward chaining method. Authors in [22] developed a rule-based M-DCocoa ES that provided diagnostic of infected cocoa plants. In that case, the symptoms of the disease presented the set of known facts. Thus, the use of the forward chaining method is logically imposed. Moreover, research in [23] proposed rule-based ES for detecting a warning of coffee disease. The system achieved an accuracy of 66.67% by using a forward chaining method. The authors indicated that the proposed system is the first ES that uses not only expert knowledge but also technical reports for designing rules in the coffee rust problem.

Performance evaluation is a very important process for every organization’s success. Human performance evaluation is a more complex problem. Although there is a large number of methods for assessing the performance of individuals during their work, they are mostly observed from the context of human resource management with a certain amount of subjectivity in human judgments. The 360-degree method is one of the most well-known methods, and the authors in [24] combined it with the application of an ES based on fuzzy logic. Although it is an excellent method for assessing employee performance, the ES developed in MATLAB software interface has only provided results in the form of employee performance levels. One of the few studies regarding ES implementation within employee performance evaluation is shown in [25] on public university employee evaluation examples. The authors emphasize that there are so far only a few developed systems that provide just partial evaluations. The authors in [26] designed fuzzy ES for teachers’ performance evaluation. In most organizations, human performance is based on qualitative facts, which are difficult to measure. Therefore, the linguistic terms were numerically weighted. However, both papers do not provide the ES application results and their impact on the performance results. The paper [27] presents a belief rule-based ES (BRBES) for student performance prediction. A belief rule base (BRB) provides a novel approach that increases the accuracy of prediction by capturing uncertain data. The same technique is shown in [28]. The authors implemented BRBES in the e-government of Bangladesh. The system as well provides more accurate and reliable results.

The paper [1] proposed the use of software technologies and mathematical and statistical methods for assessing and analyzing competencies in human resource information systems. Regarding their easy application in processes and contribution to human resource information systems, some papers implement ES in order to solve personnel selection problems [4]. Authors in [29] emphasize the ES advantage to support complex managerial decision-making and provide competitive advantages for organizations. They also indicate that the most appropriate domains to successfully build ES include planning, job analysis, recruitment, selection, performance evaluation, compensation, training, and labor-management relations. The application of ES in a personnel selection process is shown in [30]. In order to develop rule-based ES called BOARDEX, the PROLOG computer language was used. The goal was to educate and train officers in US Army Command and General Staff College. Another example of ES application for personnel selection problems is shown in [31]. The authors conclude that some of the important features of ES are flexibility in the knowledge base and the ability of the presentation of explanations about reasoning and decisions. Authors in [32] apply rule-based ES in the personnel selection framework. By combining ES with data mining techniques, it was possible to extract hidden information from large volumes of personal data. Therefore, it was possible to have a better visualization of knowledge. Moreover, regarding the analysis of a rule-based ES for career selection presented in [33], it is emphasized that ES usage offers greater efficiency and correct decisions.

The use of VisiRule ES shell in combination with the IoT technologies is presented in the paper [34] through a case study of a smart beverage dispenser. The paper emphasizes the advantage of using the VisiRule tool due to the fact that the source code can be executed and implemented within other programs and processes as well. The paper [35] proposes the use of Flex shell in combination with the VisiRule tool in the field of building a knowledge system for the lean supply chain management. In the paper [36], the authors demonstrate the successful implementation of the VisiRule with the example of legal decision-making on acquiring immovable property in India. They enable the adoption of correct decisions based on an empirical approach integrated with the rules and legal regulations.

The literature review has pointed to the fact that the application of ES is carried out in various research areas. By conducting a detailed content analysis of 311 ES case studies dating from 1984 to 2016, the author in [37] indicates that ES implementation within human resources as a functional area has one of the highest average impact factors. In the area of human resources management, the emphasis is mostly on ES application within the personnel selection process. However, the possibility of applying an ES when evaluating the performance of employees has not been significantly explored in detail yet. Therefore, the recent literature is scarce, although some authors advocate such application [29] and others recognize its potential [25,26,27,28].

3. Materials and Methods

In this research, we propose ES application in the field of employee performance evaluation with the aim of assessing their work quality, using a case study as a type of research. According to [38], case studies are especially applicable for the analysis of longitudinal change processes. Therefore, this paper concerns developing an ES to assess the performance of inspectors who conduct periodical technical inspections of vehicles in Croatia (PTI inspectors work quality appraisal ES—PTII WQA ES), with respect to the paper [7]. The results of PTII WQA ES have been used for educational and training purposes. The research was conducted from 2016 to 2017 in order to analyze the impact of the PTII WQA ES application on PTI inspectors’ performance. It was carried out by comparing primarily collected data with the data collected one year after the ES was designed and implemented. The data from 2018 and 2019 were not included in the research because of new regulations regarding a Croatian PTI system that came into force then, which is therefore not comparable.

3.1. PTII WQA ES Design

The PTII WQA ES application has covered seven phases, which include the following:

• process analysis
• data collection
• data analysis and parameter definition
• limitation definition
• rules design
• ES design within software
• verification and validation

The process was first analyzed by the author through active field research to include all the activities of the PTI of vehicles process. Therefore, it was possible to determine which data had to be collected. Afterward, the data were analyzed to determine which parameters have an impact on the vehicle’s roadworthiness. The binary logistic regression and an ANOVA statistical test were conducted to gain insight regarding parameter relationships. In addition, experts defined the intervals of certain parameters that were key for the appraisal of the quality of the PTI inspector work. Moreover, the legal limitations and the prescribed working standards have also been considered in the rules design phase. By designing the rules, the ES was simultaneously designed. In order to ascertain that PTII WQA ES achieves an acceptable level of performance [5], verification and validation were performed on the basis of previously established facts about the quality of the work of 11 PTI inspectors within the observed sample.

3.2. Data Collection

The study was based on secondary data collected by the expert organization, the Centre for Vehicles of Croatia. During the PTI, data on the vehicle are being recorded and logged in the databases. The collected data refer to 11 PTI inspectors and vehicles that underwent PTI from 2016 to 2017 in one observed PTI station in Zagreb in Croatia. Data analyzed in this research include the M1 category vehicles since M1 category vehicles (passenger cars) comprised around 76% of total registered vehicles in the Republic of Croatia in the period 2008–2017. In order to make a decision about the PTI inspector’s work quality at the PTI station, the data on their work were the prerequisite for performing an evaluation.

3.3. Used Software Package

The PTII WQA ES was developed using the tool of visual technology, known as VisiRule ES shell. The shell applies the forward chaining method, which is based on provided data. VisiRule provides a drawing environment for decision charts that can be immediately executed and verified. The main constructs in VisiRule are nodes that represent questions and/or computable functions and expressions that guard the various paths through the network. It generates code in the form of Flex rules that can be executed, inspected, and exported for embedding in external applications [39].

4. Case Study of PTI Inspectors’ Performance Evaluation in Croatia

4.1. Periodical Technical Inspection of Vehicles in Croatia

The PTI system is an independent activity of general interest designed to keep vehicles in a safe and environmentally acceptable condition. The PTI stations are special institutions that must meet the requirements set out by the legal regulations in order to perform vehicle technical inspection in a proper and professional manner [40]. The PTI is performed regularly according to the prescribed procedure, which includes the inspection of vehicle assemblies that must be inspected, all in accordance with the relevant legal regulations that describe in detail the procedure of performing the inspection thereof. During that period (2016–2017), applicable regulations mandated that new vehicles undergo PTI in the last month in which the 24-month period from the first technical inspection and registration expired. Vehicles that were two or more years old had to undergo PTI every twelfth month since the last PTI. PTI starts with the application for PTI and finishes by entering results into the information system that are in addition being logged in the database. By inspecting the parts and assemblies on a vehicle, the inspector determines their condition and if there are any deficiencies. If deficiencies are established to a large extent, the vehicle is declared unroadworthy, and it cannot pass the PTI, i.e., the PTI result is 0—fail. As the schedule of activities often varies due to the differences in the layout of the devices and equipment on the testing line where the PTI is carried out, the flowchart is not universal, and for this reason, is not shown.

Although most work procedures have been designed to minimize the subjectivity of individuals, while performing PTI, anomalies may emerge on behalf of manipulation with PTI results and incorrectly made decisions on the basis of misinterpreted facts. Therefore, the performance evaluation of PTI inspectors proved to be a great case for the ES application due to the importance of providing an efficient and transparent PTI system.

4.2. Deciding on the Quality of PTI Inspectors’ Work

The performance of PTI inspectors is usually assessed regarding the norms defined in the legal regulations and professional instructions. The evaluation is conducted by the person in a managerial position, i.e., an expert. For the purpose of this paper, the intervals that the parameters must meet in accordance with the company’s business rules and policies have been accordingly defined. To assess the quality of PTI inspectors’ work by experience, three parameters were relevant. The final decision is made depending on their value, individual and in relation to the average value at the level of the entire Republic of Croatia, which represents the reference value. The three parameters for assessing the quality of PTI inspector work are, in order:

• the average duration of the PTI of the inspected vehicles
• the average age of the inspected vehicles
• the PTI failure rate

The duration of the PTI is defined by professional instructions with regard to the vehicle category. Accordingly, the standard for the duration of the PTI of the passenger car category-M1 is 38 min. According to the legal regulation, the duration of the PTI must not be shorter than 10 min, and on the basis of experience, inspection lasting more than 60 min may point to possible issues in work.

The average vehicle age parameter refers to the age of the fleet, which the PTI inspector tests. The reference average value is the average age of all vehicles of a certain category tested over a period of one year at the level of the entire Republic of Croatia or the average age of the vehicles tested over a period of one year at an observed PTI station. If a PTI inspector tests a fleet that is too young (up to 5 years of age) or a fleet that is too old (more than the average reference value), they do not perform the work in the manner described in the instructions.

The PTI failure rate represents the value that shows the number of unroadworthy vehicles in relation to the total number of inspected vehicles. In practice, the PTI failure rate of an inspector is compared with the average reference value at the observed PTI station level or at the level of the entire Republic of Croatia. However, during the analysis of the PTI failure rate, the average age is also a key factor, and the decision on the work quality is made with regard to their mutual relation.

4.3. Data Analysis for Defining Significant Parameters

The data included all M1 vehicles that had been inspected during PTI in Croatia in 2016 at the observed PTI station where 11 PTI inspectors were employed. The sample included 24,035 vehicles, and it was preprocessed to remove all extreme values (outliers) from the sample related to irregularities in the PTI duration. The new sample size included 22,804 vehicles. An analysis of the results from 11 PTI inspectors was taken to show what influenced whether the car passed or failed the inspection. It provided an insight into which parameters are relevant and which may have a significant effect on decision-making regarding the technical condition of the vehicle.

As it was mentioned before, the result of PTI can take two values, 1—pass or 0—fail. In other words, the result of PTI is a dichotomous variable. Therefore, binary logistic regression was conducted to analyze the relationship between the PTI result (dependent dichotomous variable), vehicle age in years, and PTI duration time in minutes (two independent continuous variables). It was important to define whether the association between independent variables and the dependent one is statistically significant. The question was posed as to whether the age of the vehicle and the duration of the PTI affect the ultimate result of the vehicle technical inspection. Thus, the aim was to determine the relevance of the parameters for PTII WQA ES design purposes.

At the very beginning of the binary logistic regression conduction, all four assumptions were checked to ensure that none of them have been violated. The statistical analysis was carried out in a software package Statistica by Tibco company.

Table 1. The results of binary logistic regression.

	Estimate	Odds Ratio	Standard Error	Wald Stat.	Lower CL 95%	Upper CL 95%	p-Value
Intercept	−2.55109	0.077997	0.059292	1851.228	−2.66731	−2.43488	0.000000
Vehicle age (years)	0.08476	1.088456	0.002739	957.838	0.07939	0.09013	0.000000
PTI duration (min)	0.00601	1.006028	0.001523	15.555	0.00302	0.00899	0.000080

presents the results of binary logistic regression and the contribution of each independent variable to the model and its statistical significance. It is possible to conclude that changes in both independent variables are associated with changes in the probability that the event of failing on PTI occurs. Both variables are statistically significant at the significance level of 0.05, where the p-value of vehicle age is p = 0.000000 and of PTI duration is p = 0.000080. The positive estimate values of both variables indicate that the event of failing on PTI becomes more likely as the independent value increases, holding other independent variables constant. The estimate (i.e., regression coefficient or B) of variable vehicle age is 0.08476, which implies that for every one-year increase of vehicle age, the likelihood that the vehicle fails on PTI increases by approximately 1.09 times. The regression coefficient of variable PTI duration is 0.00601. Therefore, it means that for every one-minute increase of PTI duration time, it is approximately 1.01 times more likely for a vehicle to fail on PTI.

In addition, an ANOVA test was conducted to determine the association between 11 PTI inspectors regarding PTI duration, as the value is prescribed by the legal regulations and monitored during the inspection of the PTI stations by competent authorities.

Table 2. ANOVA analysis of the duration of periodical technical inspection (PTI) performed by PTI inspectors.

Source	Sum of Squares	Df	Mean Square	F-Ratio	p-Value
Between groups	72,247.3	10	7224.73	63.83	0.0000
Within groups	2.57987 × 106	22,793	113.187
Total (corr.)	2.65212 × 106	22,803

indicates the relationship between them and determines whether there are differences in the duration of the PTI when performed by the 11 inspectors in the observed PTI station. The results indicate the evident fact that the difference exists, i.e., the difference is significant (p = 0.0000, p < 0.05).

Moreover, a multiple range test (

Table 3. Multiple range test results.

	Count	Mean	Homogeneous Groups
Duration_PTII7	2548	26.7551	X 1
Duration_PTII10	2378	27.3061	X
Duration_PTII1	2219	29.2686	X
Duration_PTII4	1844	29.2739	X
Duration_PTII3	2350	29.5128	X
Duration_PTII11	2041	30.1837	X
Duration_PTII2	2207	30.4187	X
Duration_PTII8	2224	30.7783	X
Duration_PTII6	2025	31.6158	X
Duration_PTII5	1883	32.2443	XX
Duration_PTII9	1085	33.0286	X

¹ levels containing X’s form a group of means within which there are no statistically significant differences.

) indicated five homogeneous groups and determined that inspectors differ in regard to the duration of the PTI. As we can see in the contrast table correlated with Table 3, an asterisk has been placed next to 46 pairs, indicating that these pairs show statistically significant differences at the 95.0% confidence level (see

Table A1. Contrast table correlated with Table 3.

Contrast	Sig.	Difference	+/− Limits
Duration_PTII1–Duration_PTII2	*	−1.15008	0.626863
Duration_PTII1–Duration_PTII3		−0.244177	0.617227
Duration_PTII1–Duration_PTII4		−0.00527172	0.657069
Duration_PTII1–Duration_PTII5	*	−2.9757	0.653343
Duration_PTII1–Duration_PTII6	*	−2.34721	0.640831
Duration_PTII1–Duration_PTII7	*	2.51349	0.605468
Duration_PTII1–Duration_PTII8	*	−1.50974	0.625661
Duration_PTII1–Duration_PTII9	*	−3.75998	0.772455
Duration_PTII1–Duration_PTII10	*	1.96245	0.61546
Duration_PTII1–Duration_PTII11	*	−0.915144	0.639516
Duration_PTII2–Duration_PTII3	*	0.905902	0.618089
Duration_PTII2–Duration_PTII4	*	1.14481	0.65788
Duration_PTII2–Duration_PTII5	*	−1.82562	0.654157
Duration_PTII2–Duration_PTII6	*	−1.19713	0.641662
Duration_PTII2–Duration_PTII7	*	3.66357	0.606347
Duration_PTII2–Duration_PTII8		−0.359659	0.626512
Duration_PTII2–Duration_PTII9	*	−2.6099	0.773145
Duration_PTII2–Duration_PTII10	*	3.11253	0.616325
Duration_PTII2–Duration_PTII11		0.234934	0.640349
Duration_PTII3–Duration_PTII4		0.238905	0.648704
Duration_PTII3–Duration_PTII5	*	−2.73153	0.644929
Duration_PTII3–Duration_PTII6	*	−2.10304	0.632251
Duration_PTII3–Duration_PTII7	*	2.75766	0.596379
Duration_PTII3–Duration_PTII8	*	−1.26556	0.61687
Duration_PTII3–Duration_PTII9	*	−3.51581	0.765352
Duration_PTII3–Duration_PTII10	*	2.20663	0.606521
Duration_PTII3–Duration_PTII11	*	−0.670968	0.630918
Duration_PTII4–Duration_PTII5	*	−2.97043	0.683158
Duration_PTII4–Duration_PTII6	*	−2.34194	0.671202
Duration_PTII4–Duration_PTII7	*	2.51876	0.637526
Duration_PTII4–Duration_PTII8	*	−1.50447	0.656734
Duration_PTII4–Duration_PTII9	*	−3.75471	0.797832
Duration_PTII4–Duration_PTII10	*	1.96772	0.647023
Duration_PTII4–Duration_PTII11	*	−0.909872	0.669947
Duration_PTII5–Duration_PTII6		0.628489	0.667554
Duration_PTII5–Duration_PTII7	*	5.48919	0.633684
Duration_PTII5–Duration_PTII8	*	1.46596	0.653005
Duration_PTII5–Duration_PTII9		−0.78428	0.794765
Duration_PTII5–Duration_PTII10	*	4.93815	0.643238
Duration_PTII5–Duration_PTII11	*	2.06056	0.666292
Duration_PTII6–Duration_PTII7	*	4.8607	0.620777
Duration_PTII6–Duration_PTII8	*	0.837475	0.640487
Duration_PTII6–Duration_PTII9	*	−1.41277	0.784512
Duration_PTII6–Duration_PTII10	*	4.30966	0.630526
Duration_PTII6–Duration_PTII11	*	1.43207	0.654028
Duration_PTII7–Duration_PTII8	*	−4.02323	0.605104
Duration_PTII7–Duration_PTII9	*	−6.27347	0.755901
Duration_PTII7–Duration_PTII10		−0.551038	0.59455
Duration_PTII7–Duration_PTII11	*	−3.42863	0.619419
Duration_PTII8–Duration_PTII9	*	−2.25024	0.77217
Duration_PTII8–Duration_PTII10	*	3.47219	0.615102
Duration_PTII8–Duration_PTII11		0.594594	0.639172
Duration_PTII9–Duration_PTII10	*	5.72243	0.763928
Duration_PTII9–Duration_PTII11	*	2.84484	0.783439
Duration_PTII10–Duration_PTII11	*	−2.87759	0.62919

* denotes a statistically significant difference.

). It was possible to determine who differs from whom, as well as which inspectors perform the PTI in the same manner. Therefrom, according to the experience of one of the authors, it can be a possibility that an inspector has trained another inspector and introduced them to work, or that they often consult each other during work or work in the same shift. However, in order to confirm that possibility, the shift schedule should have been provided.

The same analysis regarding differences in PTI duration was conducted one year after PTII WQA ES application within PTI inspectors’ performance evaluation and is shown in subchapter 4.7.

4.4. Knowledge—Rules Design

The basis for generating ES rules that represent knowledge was primarily legal regulations, standards prescribed by professional instructions, and average values of the observed sample. In addition, the intervals of individual parameters based on experience have also been taken into consideration. Based on the established and designed rules, a support model was designed within the decision-making process about the PTI inspectors’ performance evaluation. Within the VisiRule shell, a total of 14 rules were designed, consisting of three levels and suggesting 47 possible decisions (see Figure 2).

The first level included the entry of the average value of the duration of the PTI performed by an individual PTI inspector. The entered value was compared with the average reference value of the PTI duration. The second level included the entry of average vehicle age of the observed PTI inspector and its comparison with the reference average age at the level of an observed PTI station. Finally, the third level included a comparison of the PTI failure rate with the reference value of the PTI failure rate of the observed PTI station as well. For the purpose of comparing the value of an individual PTI inspector with the average reference value, in this case study, the average value of the observed station was taken as the reference value. This type of comparison suggests whose results differ from those of others, which was the goal of PTII WQA ES regarding employee performance evaluation.

4.5. Pseudocode

Simultaneously with the rule design in the form of a flowchart, the source code and rules in the form of the IF-THEN structure were automatically generated. By running the generated source code, the window for entering the first of the required parameters was prompted. Below is a part of the rules in the form of pseudocode that represents the PTII WQA ES operating mode.

START! Rule 0: Average PTI duration value entered value =? If entered value is < 10  write decision_1 Else entered value is [10…20]  call Rule_1 Else entered value is [20…average PTI station value]  call Rule_2 Else entered value is > average PTI station value call Rule_3 Else entered value is > 60  write decision_2 end Rule 1: The average age of the inspected vehicles entered value =? If entered value is < 5  write decision_3 Else entered value is [5…10]  call Rule_4 Else entered value is [10…average PTI station value]  call Rule_5 Else entered value is > average PTI station value  call Rule_6 end Rule 4: The PTI failure rate entered value =? If entered value is < 10  write decision_6 Else entered value is [10…15] and [15…20]  write decision_7 Else entered value is [20…average PTI station value]  write decision_8 Else entered value is > average PTI station value  write decision_9 end

4.6. Verification and Validation of the ES

In order to determine whether the PTII WQA ES had proposed a correct decision, it was necessary to verify and validate the model based on the entered parameters. The ES verification was achieved by successfully running the generated prolog code. Generally, ES has been validated by running a number of test cases through the system and comparing known results or expert opinions with the system result [5]. For this purpose, for each of the 11 PTI inspectors, it was necessary to first make a decision in a conventional manner by comparing the values from

Table 4. Average values of defined parameter per PTII.

		PTI Duration (min)	Average Age (Years)	PTI Failure Rate (%)	Average Deviation from the PTI Station
					Vehicle Technical Inspection Duration (min)	Average Age (Years)	PTI Failure Rate (%)
PTII	PTII1	29.27	10.93	22.85	−0.51	−0.54	1.90
	PTII2	30.42	10.07	21.02	0.64	−1.4	0.07
	PTII3	29.51	9.53	20.72	0.27	−1.94	−0.23
	PTII4	29.27	9.06	20.01	−0.51	−2.41	−0.94
	PTII5	32.24	13.36	21.46	2.46	1.89	0.51
	PTII6	31.62	11.23	19.21	1.84	−0.24	−1.74
	PTII7	26.75	11.08	21.31	−3.03	−0.39	0.36
	PTII8	30.78	13.15	21.99	1	1.68	1.04
	PTII9	33.03	14.06	19.91	3.25	2.59	−1.04
	PTII10	27.31	12.89	20.61	−2.47	1.42	−0.34
	PTII11	30.18	12.05	20.58	0.4	0.58	−0.37
PTI station average		29.78	11.47	20.95

with the average reference value of the observed PTI station.

The deviations are ranked as follows:

• Text on a green background—indicates a better quality of work than the average for the observed parameter
• Text on a gray background—indicates a poorer quality of work than the average for the observed parameter.

From Table 4, by comparing the parameters, it was determined that PTII9 has the greatest deviation from the average reference values. That suggests the possibility of poor work quality and performance. For the purpose of validating the designed model within the ES, the parameters of PTII9 were taken as an example. After all the required parameter values have been entered, the generated code has produced the message containing a decision proposal, as indicated in Figure 3. Obviously, successful verification and validation of the model has been achieved, and the ES has recognized that there is a possibility of poor quality and unprofessional work by PTII9.

4.7. The Analysis of the Impact of the PTII WQA Systems Application on the Quality of PTI Inspector Work

One of the objectives of this paper was to define the advantages and disadvantages of the actual application of the designed ES. For this purpose, data on the work of the same 11 PTI inspectors were collected for a one-year period. The data also referred to the M1 category vehicles that had been inspected for the purpose of the PTI in 2017 in the same observed PTI station.

A hypothesis has been set that one year after the application of PTII WQA ES, better uniformity in the work of 11 PTI inspectors has been achieved with regard to the PTI duration. For that reason, the ANOVA test was conducted with the same question: do the PTI inspectors differ with regard to the duration of the PTI?

Table 5. ANOVA analysis of the duration of PTI performed by different inspectors after the application of the PTII WQA.

Source	Sum of Squares	Df	Mean Square	F-Ratio	p-Value
Between groups	50,861.0	10	5086.1	45.95	0.0000
Within groups	2.4268 × 106	21,923	110.697
Total (corr.)	2.47766 × 106	21,933

indicates that inspectors still significantly differ with regard to the duration of the PTI (p = 0.0000, p < 0.05).

However, the results indicated in

Table 6. Multiple range test after the application of the PTII WQA.

	Count	Mean	Homogeneous Groups
Duration_PTII7	2293	26.7457	X 1
Duration_PTII10	2219	28.014	X
Duration_PTII3	2281	28.7979	X
Duration_PTII8	2193	29.3092	XX
Duration_PTII4	2222	29.3267	XX
Duration_PTII6	1914	29.7576	X
Duration_PTII11	1961	30.4345	X
Duration_PTII1	2158	30.6803	XX
Duration_PTII2	1123	31.3785	XX
Duration_PTII9	1362	31.6593	X
Duration_PTII5	2208	31.8886	X

¹ levels containing X’s form a group of means within which there are no statistically significant differences.

show that, on the basis of statistical significance, a larger number of inspectors with regard to a larger number of homogeneous groups (seven groups) have a more uniform duration of PTI in 2017 compared to 2016. In the contrast table correlated with Table 6, an asterisk has been placed next to 45 pairs, indicating that these pairs show statistically significant differences at the 95.0% confidence level (see

Table A2. Contrast table correlated with Table 6.

Contrast	Sig.	Difference	+/− Limits
Duration_PTII1–Duration_PTII2		−0.698191	0.758759
Duration_PTII1–Duration_PTII3	*	1.88236	0.619256
Duration_PTII1–Duration_PTII4	*	1.35353	0.62324
Duration_PTII1–Duration_PTII5	*	−1.20833	0.624213
Duration_PTII1–Duration_PTII6	*	0.922684	0.647476
Duration_PTII1–Duration_PTII7	*	3.93451	0.618468
Duration_PTII1–Duration_PTII8	*	1.37109	0.625267
Duration_PTII1–Duration_PTII9	*	−0.979065	0.71363
Duration_PTII1–Duration_PTII10	*	2.66629	0.623448
Duration_PTII1–Duration_PTII11		0.245787	0.64335
Duration_PTII2–Duration_PTII3	*	2.58055	0.751724
Duration_PTII2–Duration_PTII4	*	2.05172	0.755009
Duration_PTII2–Duration_PTII5		−0.510136	0.755813
Duration_PTII2–Duration_PTII6	*	1.62087	0.775136
Duration_PTII2–Duration_PTII7	*	4.6327	0.751075
Duration_PTII2–Duration_PTII8	*	2.06929	0.756684
Duration_PTII2–Duration_PTII9		−0.280874	0.831191
Duration_PTII2–Duration_PTII10	*	3.36448	0.755181
Duration_PTII2–Duration_PTII11	*	0.943978	0.771693
Duration_PTII3–Duration_PTII4		−0.528837	0.614657
Duration_PTII3–Duration_PTII5	*	−3.09069	0.615643
Duration_PTII3–Duration_PTII6	*	−0.95968	0.639217
Duration_PTII3–Duration_PTII7	*	2.05215	0.609817
Duration_PTII3–Duration_PTII8		−0.51127	0.616712
Duration_PTII3–Duration_PTII9	*	−2.86143	0.706146
Duration_PTII3–Duration_PTII10	*	0.783925	0.614867
Duration_PTII3–Duration_PTII11	*	−1.63658	0.635039
Duration_PTII4–Duration_PTII5	*	−2.56185	0.61965
Duration_PTII4–Duration_PTII6		−0.430843	0.643078
Duration_PTII4–Duration_PTII7	*	2.58098	0.613862
Duration_PTII4–Duration_PTII8		0.0175671	0.620712
Duration_PTII4–Duration_PTII9	*	−2.33259	0.709643
Duration_PTII4–Duration_PTII10	*	1.31276	0.618879
Duration_PTII4–Duration_PTII11	*	−1.10774	0.638924
Duration_PTII5–Duration_PTII6	*	2.13101	0.644021
Duration_PTII5–Duration_PTII7	*	5.14284	0.61485
Duration_PTII5–Duration_PTII8	*	2.57942	0.621689
Duration_PTII5–Duration_PTII9		0.229262	0.710497
Duration_PTII5–Duration_PTII10	*	3.87462	0.619859
Duration_PTII5–Duration_PTII11	*	1.45411	0.639873
Duration_PTII6–Duration_PTII7	*	3.01183	0.638454
Duration_PTII6–Duration_PTII8		0.44841	0.645042
Duration_PTII6–Duration_PTII9	*	−1.90175	0.731019
Duration_PTII6–Duration_PTII10	*	1.74361	0.643279
Duration_PTII6–Duration_PTII11	*	−0.676896	0.662586
Duration_PTII7–Duration_PTII8	*	−2.56342	0.61592
Duration_PTII7–Duration_PTII9	*	−4.91358	0.705455
Duration_PTII7–Duration_PTII10	*	−1.26822	0.614073
Duration_PTII7–Duration_PTII11	*	−3.68872	0.63427
Duration_PTII8–Duration_PTII9	*	−2.35016	0.711423
Duration_PTII8–Duration_PTII10	*	1.2952	0.620921
Duration_PTII8–Duration_PTII11	*	−1.12531	0.640902
Duration_PTII9–Duration_PTII10	*	3.64535	0.709825
Duration_PTII9–Duration_PTII11	*	1.22485	0.727368
Duration_PTII10–Duration_PTII11	*	−2.4205	0.639127

* denotes a statistically significant difference.

). We can conclude, in 2017, we have fewer PTII pairs whose mode of operation differs from each other. The smaller number of contrast groups affects the increase of process quality. That was the result of the PTII WQA ES application, which due to the relatively short time application of the expert system, indicated the minimal but positive impact of the ES on the PTI system.

5. Discussion

One of the key phases when designing an ES is to conduct adequate data analysis. That is the main difference compared to preliminary research conducted in [8], which has proved ES as a successful tool for waste elimination within processes. In this paper, the focus is on the parameter definition. The data analysis is performed in two senses in order to determine the relationship between observed parameters and to determine associations within a certain parameter for 11 PTI inspectors before and after the PTII WQA ES application. The purpose of the data analysis was to determine which parameters are indeed crucial for designing rules within ES and for their future monitoring. Therefore, the question was posed whether the observed parameters affect the PTI result. A binary logistic regression has shown that PTI result can be predicted based on the vehicle age with a p-value of 0.000000 and duration of the PTI with a p-value of 0.000080 at the significant level of 0.05, indicating the fact that both variables are significant and relevant for designing rule-based PTII WQA ES. An ANOVA analysis pointed to the fact that PTI inspectors significantly differ in regard to the duration of the PTI. In this case, the binary logistic regression and an ANOVA test have confirmed the significance of experiential parameters that were assessed in the standard evaluation system.

In the ES design process, the rules were generated not only considering the experience of the expert but focusing on standards prescribed by legal regulations, professional instructions, and average values of the observed sample. A similar method for designing rules is shown in [23], where authors used technical reports. Thus, the subjectivity of the ES and proposed decisions is reduced.

In order to demonstrate the positive impact of PTII WQA ES application within the process, the hypothesis was proposed. The application of the PTII WQA ES has resulted in the improvement of the work results as soon as in the first year of application. The PTII WQA ES has resulted in reducing the difference in the average PTI duration of 11 PTI inspectors. An ANOVA test has established that the number of homogeneous groups increased from five (before application) to seven after the PTII WQA system application. In addition, more uniform performance of PTI inspectors has been achieved, which confirmed the hypothesis.

The goal of the ES implementation in this area is not to replace an expert in making decisions regarding employee performance but to help him to make efficient decisions in real-time and to help other non-experts to understand them. Furthermore, the purpose of the PTII WQA ES was not to sanction the inspectors who deviate from reference values but to educate and train them, all in order to establish a more objective and efficient PTI system. Research conducted in [26] yielded similar findings. The authors as well emphasized that the proposed ES provides a foundation for performance evaluation and great support to managers regarding decision-making.

The limitation of the conducted research primarily relates to the lack of benchmarking in the research area regarding ES implementation within the process of human performance evaluation, as indicated in [25]. However, it is possible to overcome this limitation by applying the proposed model or similar ones in other organizations or in other countries within their PTI system. That will provide the possibility to compare obtained results and established findings with other cases. Subsequently, the performance of the rule-based PTII WQA ES was not quantified relative to the standard evaluation system. Future research should therefore focus on finding the most adequate method for providing a relationship between them. Furthermore, for the purpose of further research, there is a tendency for the PTII WQA ES to be improved by incorporating certainty factors and fuzzy logic, which is of great importance in solving the subjectivity issues.

6. Conclusions

In this paper, a new rule-based ES to evaluate human performance was proposed. Considering the literature review, it is clear that ES could be applied for various purposes. The above includes control, prediction, diagnosis, evaluation, selection, and comparison. The main advantage of ES, as established in this paper, is the ability to imitate human thinking and identify the facts that are of paramount importance regarding decision-making. The ES also provides the reduction of errors, enables greater transparency, and increases process efficiency. The ES is easily understood and reviewed regarding easy problem solving by using if-then rules. On the other hand, shortcomings relate to the possible subjectivity of the ES due to the fact that it is based on the knowledge and experience of an individual expert. Moreover, the ES can be just a phase within the decision-making process. Thus, the generated code may not be able to predict all possible situations, especially regarding human factor performance.

The practical implementation of an ES is of major importance in processes where subjectivity as a human factor is predominant, as is the case in the Croatian PTI system. This research provided a successful implementation of ES within the process of human performance evaluation in the case of PTI inspectors’ performance. Moreover, the hypothesis was confirmed due to a well-established model that was based on previously conducted data analysis. The data analysis provided a precise and correct definition of key parameters. Precisely for this reason, the ES can facilitate procedures of determining parameter values that can indirectly point to errors and deficiencies within the process. Therefore, it is possible to determine anomalies of human performance regarding manipulation, bias, or human error. Furthermore, the application of ES such as PTII WQA could be carried out in different areas regardless of whether it is a manufacturing, service, or public sector.