Inverse Trigonometric Fuzzy Preference Programming to Generate Weights with Optimal Solutions Implemented on Evaluation Criteria in E-Learning

Abstract

Nielsen’s heuristics are widely recognized for usability evaluation, but they are often considered insufficiently specific for assessing particular domains, such as e-learning. Currently, e-learning plays a pivotal role in higher education because of the shift in the educational paradigm from a teacher-centered approach to a student-centered approach. The criteria utilized in multiple sets of heuristics for evaluating e-learning are carefully examined based on the definitions of each criterion. If there are similarities in meaning among these criteria, they are consolidated into a single criterion, resulting in the creation of 20 new criteria (spanning three primary aspects) for the evaluation of e-learning. These 20 new criteria encompass key aspects related to the user interface, learning development, and motivation. Each aspect is assigned a weight to facilitate prioritization when implementing improvements to evaluate e-learning, which is especially beneficial for institutions with limited resources responsible for the relevant units. In terms of weighting, there is room for enhancement to attain more optimal weighting outcomes by employing a Fuzzy Preference Programming method known as Inverse Trigonometric Fuzzy Preference Programming (ITFPP). The higher the assigned weight, the greater the priority for implementing improvements.

1. Introduction

Expert research has primarily focused on usability, which is becoming increasingly important. Heuristic evaluation is currently a sought-after evaluation technique. Heuristic evaluation, according to Nielsen and Molich [1,2], is a method of addressing usability problems in user interface design so that they can be resolved through an iterative design process. Many heuristic evaluation techniques exist, including Nielsen’s Ten Heuristics, Ben Schneiderman’s Eight Fundamental Rules of Interface Design in 1998, and Alan Cooper’s Fundamentals of Interaction Design in 2014. However, Nielsen’s 10 heuristics, developed in 1990, are the most commonly used methods in usability-based user interface design [3,4,5]. Nielsen’s heuristics are designed for user interface design to detect usability issues. Many advantages encourage researchers to use this assessment method: it is low cost, does not require extensive planning, encourages everyone to use it, and more importantly, it can be used from the early stages of development as well as during the evaluation phase [1]. However, Nielsen’s ten heuristics have been criticized for being too focused on the system rather than the user experience [6,7]. Therefore, many experts have created new heuristics based on Nielsen [8,9,10] to overcome these shortcomings. However, these new heuristics also have limitations, such as lack of validity [11] or ambiguity in adding heuristics [12]. The weaknesses in Nielsen’s heuristics and their modified versions have encouraged researchers to develop heuristics to address these shortcomings. In this study, we utilized nine criteria from Nielsen’s heuristics and compiled criteria previously used by researchers to evaluate e-learning. As a result, we obtained 20 criteria. The subsequent challenge lies in presenting the evaluation results to e-learning developers, especially those in higher education institutions (as well as decision-makers). It is crucial, particularly for universities with limited resources, to address the entire set of 20 criteria quickly and directly. Therefore, the focus of our article is on prioritizing the 20 heuristic criteria used to evaluate higher education e-learning.

A crucial skill required by a decision-maker is the capacity to prioritize a number of possibilities chosen based on a number of factors. However, when decision-makers are confronted with several criteria and pieces of information, the situation becomes problematic. By implementing the AHP technique, which asks experts to rank their preferences for criteria and options on a scale of 1 to 9, research in [13,14] has simplified this issue. Expert comparisons were then poured into a pairwise comparison matrix as components. The AHP approach has been widely applied [15,16,17,18]. However, the AHP approach suffers greatly from subjectivity and ambiguity during evaluation. The FAHP approach was first proposed in [19] to address these constraints. The FAHP approach traditionally converts a numerical scale into fuzzy triangular numbers, which may be examined in the FAHP literature [20,21,22,23,24]. Since the FAHP aims to obtain weights from a fuzzy pairwise comparison matrix, two types of weights can be derived. As seen in [25], the former is weighted by fuzzy numbers and the latter by hard numbers. A representative method that can provide significant weights is the FPP method proposed in the Extent Analysis (EA) method [19,26,27].

The EA approach is more extensively used than the FPP method owing to its simplicity and ease of use in practical applications. The shortcomings of the majority of FAHP applications employing the EA technique discovered by [19] are frequently disregarded. Consequently, the weights derived using this technique do not accurately reflect the relative weights of the decision-making criteria or alternatives [24,28]. This issue arises from the priority vector that calculates the procedure’s zero-weighting of the selection criteria or alternatives. Therefore, multiple weights were not considered in the choice analysis. These circumstances cause inaccurate findings. Because of this, the EA method’s application, which has been used in several previous studies [29,30,31,32,33,34,35,36,37,38,39], has been incorrect. The FPP technique adds a fuzzy membership function, in contrast to the EA method, which is converted into a restricted nonlinear optimization problem for weighted derivatives of crisp numbers. Decision-maker satisfaction is maximized using this strategy. However, occasionally, when there were significant discrepancies between the fuzzy judgments, the FPP approach described in [27] produced the best option. Some of the responses provided false conclusions. Additionally, ref. [28] noted that ref. [27]’s fuzzy membership initiative may result in membership degrees that are negative. By substituting the triangular fuzzy numbers used in [27] with logarithmic triangular fuzzy judgment, the study in [28] overcame this issue. Consequently, ref. [28] formulated a novel restricted optimization known as LFPP, which has a logarithmic term. The nonlinear nature of the optimization issue necessitates a precise numerical optimization process in order to solve it. In addition, LFPP contains settings that may be changed. According to this study, these factors affect the behavior of the LFPP model. The priority gained is the same as that provided by the EA technique; that is, higher-value parameters are utilized. Logarithmic functions may result in overflow during the reduction process from the perspective of numerical computation.

This study proposed a new Fuzzy Preference Programming method for determining priorities. The proposed method was formulated by modifying the natural logarithmic function used in the LFPP method with trigonometric inverse functions. Several elements reveal that the proposed method obtains an optimal solution, namely the weight of the criteria and alternative priority scores. Furthermore, computational performance is provided in the comparison stage with the LFPP methodology.

The remainder of this paper is structured as follows. A brief theory of LFPP is provided in Section 2 (Related Work). The programming of the new trigonometric inverse fuzzy preferences and analytical studies are presented in Section 3 (Research Methodology). Section 4 contains data obtained from the implementation phase (Results and Discussion). Finally, the conclusions are presented in Section 5.

2. Related Work

The priority-based optimization problem initiated by [27] aims to find sharp priority vectors $\stackrel{\rightharpoonup }{w}={(w_1,w_2,\dots ,w_n)}^T,w_i>0,$ for $i=1,\dots ,n$ such that ${\displaystyle \sum _{i=1}^n{w}_i=1},$ from the pairwise comparison matrix.

Formally, it can be stated as Case 1.

Case 1

Assume below ${\tilde{a}}_{ij}=\left(l_{ij},m_{ij},u_{ij}\right)$ is the triangular fuzzy number defined in [27] and $\tilde{A}=\left\{{\tilde{a}}_{ij}\right\}=\left[\begin{array}{cccc}1& {\tilde{a}}_{12}& \cdots & {\tilde{a}}_{1n}\\ {\tilde{a}}_{21}& 1& \cdots & {\tilde{a}}_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ {\tilde{a}}_{n1}& {\tilde{a}}_{n2}& \cdots & 1\end{array}\right]$ is a fuzzy pairwise comparison matrix, where $a_{ij}=(1/u_{ij},1/m_{ij},1/l_{ij}).$

Find $\overrightarrow{w}={\left(w_1,w_2,\dots ,w_n\right)}^T$ from $\tilde{A}$.

To solve case 1, the researcher in [27] initiated the fuzzy membership function for $\widehat{w}\le m_{ij}$ and ${\mu }_{ij}\left(\widehat{w}\right)=\frac{\left(u_{ij}-\widehat{w}\right)}{u_{ij}-m_{ij}}$ for $\widehat{w}\ge m_{ij}$ with $\widehat{w}=\frac{w_i}{w_j}$.

From the fuzzy membership function, Mikhailov [27] formulated the possible domain as the intersection of all membership functions and let $\lambda =\min \left\{{\mu }_{ij}\left(w_i/w_j\right):i=1,\dots ,n-1;j=i+1,\dots ,n\right\}$.

To maximize the satisfaction of the decision-maker(s), a nonlinear optimization problem is formulated in [27], given as Model (1):

Maximize $\lambda$

$$s.t.\{\begin{array}{l}(m_{ij}-l_{ij})\lambda w_j-w_i+l_{ij}{w}_j\le 0\\ \left(u_{ij}-m_{ij}\right)\lambda w_j+w_i-u_{ij}{w}_j\le 0\\ {\displaystyle \sum _{k=1}^n{w}_k=1,w_k}>0,k=1,2,\dots n\\ i=1,2,\dots ,n-1,j=2,3,\dots ,n,j>i.\end{array}$$

(1)

The optimal solution ${\lambda }^{\ast }$ is obtained for $W^{\ast }=(w_1,w_2,w_n^{\ast })$ when ${\lambda }^{\ast }>0$. The author of [28] revealed that nonlinear optimization problems obtain more than one optimal solution and can produce ${\lambda }^{\ast }<0$. Wang and Chin [28] attempted to overcome this obstacle by incorporating the logarithmic function into fuzzy triangular numbers. The logarithm of the triangular fuzzy numbers ${\tilde{a}}_{ij}$ can be expressed as $\ln {\tilde{a}}_{ij}\approx (\ln l_{ij},\ln m_{ij},\ln u_{ij}) \mathrm{and} i,j=1,\dots ,n$.

This shows that the nonlinear optimization problem is transformed into a logarithmic fuzzy optimization problem.

Maximize $\lambda$

$$s.t.\{\begin{array}{l}{\mu }_{ij}\left(\ln \left(\frac{w_i}{w_j}\right)\right)\ge \lambda ,\\ i=1,n-1;\\ j=i+1,\\ n,w_i\ge 0,1=1,\dots ,n\end{array}$$

(2)

However, there is no guarantee that the logarithmic fuzzy optimization problem in Model (2) always produces a lambda $(\lambda )$ greater than 0. Therefore, the variable is non-negative ${\delta }_{ij}$ and ${\eta }_{ij},$ for $i=1,\dots ,n-1$ and $j=i+1,\dots ,n,$ included in the logarithmic fuzzy optimization problem model.

Because the variable deviation values ${\delta }_{ij}$ and ${\eta }_{ij}$ decrease, the objective function can be formulated as

$$J={(1-\lambda )}^2+M\cdot {\displaystyle \sum _{i=1}^{n-1}{\displaystyle \sum _{j=1+1}^n\left({\delta }_{ij}^2+{\eta }_{ij}^2\right)}}.$$

where M is the included parameter to ensure that the weights obtained are always within the 16 fuzzy scoring intervals. Thus, the programming of fuzzy logarithmic preferences for decreasing FAHP weights is defined in Model (3) as follows:

Minimize $J={(1-\lambda )}^2+M\cdot {\displaystyle \sum _{i=1}^{n-1}{\displaystyle \sum _{j=1+1}^n\left({\delta }_{ij}^2+{\eta }_{ij}^2\right)}}$

$$s.t.\{\begin{array}{l}{x}_i-x_j-\lambda \ln \left(m_{ij}/l_{ij}\right)+{\delta }_{ij}\ge \ln l_{ij},i=1,\dots ,n-1;j=i+1,\dots ,n,\\ -x_i+x_j-\lambda \ln \left(u_{ij}/m_{ij}\right)+{\eta }_{ij}\ge -\ln u_{ij},i=1,\dots ,n-1;j=i+1,\dots ,n,\\ \lambda ,x_i\ge 0,i=1,\dots ,n,\\ {\delta }_{ij},{\eta }_{ij}\ge 0,i=0,\dots ,n-1;j=i+1,\dots ,n,\end{array}$$

(3)

where $i=1,\dots ,n-1;j=i+1,\dots ,n$ for constraints 1, 2, and 4. The variables in the LFPP problem are $x_i=\ln w_i,i=1,\dots ,n$. It can be seen that ${\displaystyle \sum _{i=1}^n{w}_i=1}$ is not included in the LFPP model to avoid complexity in the computational stage.

3. Research Methodology

The research approach employed to develop a novel framework for usability heuristics in this study encompasses a sequence of steps as depicted in Figure 1. Insights gleaned from a comprehensive literature review informed our selection and adaptation of Nielsen’s ten heuristics for our heuristic compilation, alongside the development of additional sub-heuristics. Expert participants conducted a heuristic assessment on two chosen systems using solely Nielsen’s set, aiding the subsequent validation of our suggested heuristics. The ITFPP method was applied to the sub-criteria to derive a hierarchy of priorities, facilitating the filtration of superfluous data and the definition of pertinent sub-heuristics. Integrating these findings with Nielsen’s original heuristics and the newly derived sub-heuristics resulted in the preliminary version of our usability heuristics. Subsequently, an ITFPP analysis was reiterated to ascertain the hierarchical ranking for this initial compilation of our heuristics, leading to the formulation of their definitive version. This ultimate set of heuristics underwent validation through expert assessments of the initially selected systems. The outcomes of these assessments were juxtaposed against those from the initial application of Nielsen’s heuristics.

3.1. Data Collection

Heuristic evaluation is the data collection strategy used in this study. Experts do an extensive system evaluation to identify issues that the average user would not generally notice. The purpose of a heuristic evaluation is to identify important usability problems that users may encounter and to recommend design improvements to the system before they are implemented. As a result, a panel of five to eight experts, each with a particular field of knowledge and ages ranging from 30 to 46, was assembled. These experts comprised four in the field of human–computer interaction/software engineering, two in learning development, and two in psychology.

3.2. Experiment

We provided a questionnaire sheet containing 4 paired comparison matrices to 8 experts. The first comparison matrix comprised comparisons for three main criteria, and the three paired comparison matrices contained comparisons for sub-criteria from the first main criterion, sub-criteria from the second main criterion, and sub-criteria from the third main criterion. Before experts began assigning weights to each paired comparison matrix, we ensured that they fully understood our instructions. We used a 1–9 scale. Following the experts’ assignment of weights to all paired comparison matrices, we proceeded with normalization and calculated the average values from all experts, utilizing the ITFPP formula (Equation (4)).

The Inverse Trigonometric Fuzzy Preference Programming (ITFPP) method encompasses a systematic procedure involving various sequential steps to facilitate the decision-making process. The initial step involved identifying relevant and crucial criteria for the decision at hand, in this case, criteria used to evaluate e-learning. Subsequently, fuzzy preferences were elicited from experts (involving 8 experts), manifested in fuzzy sets summarizing the level of importance or subjective preference assigned to each criterion. Next, the ITFPP method was employed to conduct nuanced assessments of the weights associated with these criteria. This assessment involved transforming fuzzy preferences into quantitative weight values through the arc sin function, thereby enhancing accuracy and objectivity in the decision model. Once criteria were assigned weights, alternative assessments were conducted, evaluating each alternative against the identified criteria based on the transformed fuzzy preferences. The subsequent step involved calculating aggregate scores for each alternative, achieved by multiplying the assigned criterion weights with the corresponding alternative assessments. As a result, alternatives were ranked based on their aggregate scores, ultimately designating the option with the highest score as the optimal choice. Additionally, the application of sensitivity analysis enriched the decision-making process, highlighting the potential impact of changes in fuzzy preferences or criterion weights on the final outcomes. This provides stakeholders with valuable insights into the robustness and reliability of the decision model. Validation and consistency of the results were then scrutinized, ensuring the integrity of fuzzy preferences and criterion weights. Presentation of the results was carried out in an understandable format, using tables or graphs to facilitate a clear understanding of the decision outcomes.

3.3. Inverse Trigonometric Fuzzy Preference Programming

In this section, a new type of Fuzzy Preference Programming (FPP) is proposed, which aims to improve the effectiveness of the LFPP problem.

The use of the natural number logarithm function in the LFPP method can produce a solution for determining the importance weight value, but it is not yet known whether the results obtained are optimal or not (minimizing the value of J); therefore, it is necessary to conduct experiments using other mathematical functions. Nonlinear equations can be obtained from finite sets, including trigonometric function arguments, logarithms, and exponential functions. Therefore, the ln function limits the objective function LFPP and needs to be changed to, for example, trigonometric functions. From several experiments using trigonometric inverse functions, the function $\arcsin \left(\frac{x}{1+x}\right)-\arcsin \left(\frac{1}{2}\right)$ is obtained because its characteristics are similar to the graph of the ln function, especially in quadrant 1 (if the x and y values are positive).

3.4. Example

It is known that $f\left(x\right)$, if $1<x\le 15$; when calculated using Maple Software 18, with the Chart Scatter type, a graph is produced as shown in Figure 1a,b.

Function A is $f(x)=\ln (x)$, and Function B is $f(x)=\arcsin \left(\frac{x}{1+x}\right)-\arcsin \left(\frac{1}{2}\right)$.

By calculating the numbers, you can see the difference in

Table 1. Calculation of each function.

Num.	Function A	Function B
1	0	0
2	0.693147181	0.206128881
3	1.098612289	0.324463303
4	1.386294361	0.403696442
5	1.609437912	0.461512008
6	1.791759469	0.506098025
7	1.945910149	0.541837041
8	2.079441542	0.571315302
9	2.197224577	0.596170739
10	2.302585093	0.617497885
11	2.397895273	0.636059689
12	2.48490665	0.652406431
13	2.564949357	0.666946345
14	2.63905733	0.679989531
15	2.708050201	0.69177635

.

Table 1 shows the differences in values starting from number 2, which are thought to cause scores ${\lambda }^{\ast }$ to be not optimal, although the pairwise comparison matrices are consistent. Therefore, to maximize the value of ${\lambda }^{\ast }$, we propose replacing the function $\ln$ with the function $\arcsin \left(\frac{x}{1+x}\right)-\arcsin \left(\frac{1}{2}\right)$ at its limits, as shown in model (4) as follows:

Minimize

$$J={(1-\lambda )}^2+M\cdot {\displaystyle \sum _{i=1}^{n-1}{\displaystyle \sum _{j=1+1}^n\left({\delta }_{ij}^2+{\eta }_{ij}^2\right)}}$$

for

$$\begin{array}{c}\mathrm{f}\left(\frac{\frac{w_i}{w_j}}{1+\frac{w_i}{w_j}}\right)=a,\left(\frac{m_{ij}}{1+m_{ij}}\right)=b,\\ \left(\frac{l_{ij}}{1+l_{ij}}\right)=c,\left(\frac{u_{ij}}{1+u_{ij}}\right)=d.\end{array}$$

$$s.t.\left\{\begin{array}{c}\mathit{arcsin}a-\lambda \left[\mathit{arcsin}b-\mathit{arcsin}c\right]\ge \mathit{arcsin}c\\ -\mathit{arcsin}a-\lambda \left[\mathit{arcsin}d-\mathit{arcsin}b\right]\ge -\mathit{arcsin}d\\ {\displaystyle \sum _{i=1}^k}{w}_i=1\\ \lambda ,{\delta }_{ij},{\eta }_{ij}\ge 0\end{array}\right.$$

(4)

As discussed in the previous section, the LFPP model is implemented to derive the priority weights of the fuzzy pairwise comparison matrices. However, the LFPP model contains p or M parameters. The p and M values used affect the optimum solution. Therefore, the results are relative.

Based on the facts mentioned before, we propose a trigonometric function in the form of arc sin. This is aimed at obtaining more optimal weighting results by minimizing the value of J and increasing the value of lambda ($\lambda$) closer to 1.

4. Results and Discussion

4.1. Result Comparison

Prior to applying our proposed method to assess the priority levels of criteria in e-learning evaluation, we conducted a comparative analysis between the proposed method (ITFPP) and the previous approach [28]. We employed a matrix, as used by prior researchers [28], denoted as matrix T.

$$T=\left[\begin{array}{cccc}\left(1,1,1\right)& \left(\frac{1}{5},\frac{1}{2},1\right)& \left(\frac{1}{6},\frac{2}{5},\frac{3}{4}\right)& \left(\frac{1}{7},\frac{1}{3},\frac{3}{5}\right)\\ \left(1,2,5\right)& \left(1,1,1\right)& \left(\frac{1}{2},\frac{4}{5},\frac{5}{4}\right)& \left(\frac{3}{7},\frac{2}{3},1\right)\\ \left(\frac{4}{3},\frac{5}{2},6\right)& \left(\frac{4}{5},\frac{5}{4},2\right)& \left(1,1,1\right)& \left(\frac{4}{7},\frac{5}{6},\frac{6}{5}\right)\\ \left(\frac{5}{3},3,7\right)& \left(1,\frac{3}{2},\frac{7}{3}\right)& \left(\frac{5}{6},\frac{6}{5},\frac{7}{4}\right)& \left(1,1,1\right)\end{array}\right]$$

Utilizing the previous method (using Equation (3)) yields the result J = 0.312590239154607374, whereas employing the ITFPP method (proposed method) and using Equation (4) yields the result J = 0.0379230382716251938. A detailed presentation of the results is available in

Table 2. Comparison of ITFPP and previous studies.

	ITFPP	Previous Study
J	0.03792303	0.31259023
λ	0.95166599	0.66190648
W1	0.11236095	0.11299603
W2	0.19410871	0.18520212
W3	0.34157342	0.35384105
W4	0.35195690	0.34796078

. This table provides a detailed breakdown of the outcomes derived from both the LFPP and ITFPP methodologies, facilitating a thorough understanding of the contrasting results obtained through these approaches. The juxtaposition of these values underscores the distinctive impact each methodology has on the calculated objective function.

The T matrix presented is structured as a 4 × 4 matrix, indicative of the four criteria under comparison. Criterion 1 undergoes evaluations against criteria 2, 3, and 4, a pattern that is repeated for each criterion. The weighting results produce four values (W₁ to W₄), summing to 1 collectively. Examining Table 2 reveals notable disparities in the obtained J values. The study by [28] defines optimal weighting as the minimum J value. Simultaneously, expert consistency is gauged through the λ variable, where proximity to 1 indicates a high level of expert coherence. The presented matrix elucidates the intricate interplay of criterion evaluations and their impact on the overall weighting structure. The consistency of experts serves as an indicator of the reliability and stability of decision-makers’ judgments throughout the decision-making process. It is a measure that quantifies the degree of coherence in pairwise comparisons, reflecting the extent to which decision-makers maintain consistent preferences across various criteria or alternatives. In the realm of decision sciences, a higher consistency is generally perceived as indicative of a more robust and dependable decision model.

4.2. Implementation of the Proposed Method

We applied this method to determine the priorities of the modified usability heuristic to evaluate the e-learning website. Heuristic evaluation is a usability inspection method that involves the evaluation of an interface against a set of established usability principles known as heuristics. This method is widely used in the field of human–computer interaction (HCI) because it provides a quick and relatively low-cost method to identify usability problems. Heuristic evaluation is widely used in various fields and has recently been found to be useful in evaluating things related to health, such as research conducted by [40,41,42]. In this study, we developed three main criteria: the user interface and experience, learning development, and motivational learning. From these three main criteria, 20 sub-criteria were derived that can be used as principles to be followed when evaluating e-learning: (1) visibility system status; (2) match between the system and the real world; (3) user control and freedom; (4) consistency and standard; (5) error prevention; (6) recognition rather than recall; (7) flexibility and efficiency of use; (8) aesthetic and minimalist design; (9) help and documentation; (10) timeliness; (11) learning design; (12) instructional assessment; (13) instructional material; (14) collaborative learning; (15) learner control; (16) feedback and assessment; (17) diversity of learning content; (18) relevancy; (19) clarity of the purpose and objectives; and (20) motivation to learn.

The data collection for determining the priority level was conducted using a questionnaire and involved eight experts with different backgrounds, consisting of two psychologists, four UI/UX/usability experts, and two experts with expertise in learning development. After the experts filled out the questionnaire, normalization was carried out, followed by calculations using the ITFPP method, resulting in priority rankings.

The consistency index values were calculated and confirmed that the expert assessments were consistent, as shown in

Table 3. The results of the main criteria.

Code	Criteria	Priority Level
A	User Interface and Experience	3
B	Learning Development	2
C	Motivational Learning (Psychology)	1

,

Table 4. The results of sub-criteria A.

Code	Criteria	Priority Level
A1	Visibility System Status	9
A2	Match Between System and the Real World	1
A3	User Control and Freedom	8
A4	Consistency and Standards	7
A5	Error Prevention	6
A6	Recognizing Rather than Recall	2
A7	Flexibility and Efficiency of Use	5
A8	Aesthetic and Minimalist Design	4
A9	Help and Documentation	3

,

Table 5. The result of sub-criteria B.

Code	Sub-Criteria	Priority Level
B1	Timeliness	9
B2	Learning Design	8
B3	Instructional Assessment	7
B4	Instructional Material	6
B5	Collaborative Learning	5
B6	Learner Control	4
B7	Feedback and Assessment	3
B8	Diversity of Learning Content	2
B9	Relevancy	1

and

Table 6. The results of sub-criteria C.

Code	Sub-Criteria	Priority Level
C1	Clarity of the Purpose and Objectives (Goals)	2
C2	Motivation to Learn	1

below.

In this study, we have four matrices: (1) matrix of main criteria, (2) matrix of sub-criteria A, (3) matrix of sub-criteria B, and (4) matrix of sub-criteria C.

Matrix of main criteria

$$MC=\left[\begin{array}{ccc}\left(1,1,1\right)& \left(1.89,2.27,2.65\right)& \left(2.89,3.40,3.90\right)\\ \left(0.38,0.44,0.53\right)& \left(1,1,1\right)& \left(3.79,4.56,5.38\right)\\ \left(0.26,0.29,0.35\right)& \left(0.19,0.22,0.26\right)& \left(1,1,1\right)\end{array}\right]$$

The results can be seen in Table 3.

In the main criteria, the highest to lowest priority order is the criteria with codes C (motivational learning), B (learning development), and A (user interface and experience). According to experts, the psychological aspect of motivational learning is the most important criterion compared with the other two criteria.

Table 3 shows the results of the expert assessment calculation using the ITFPP method, where the results obtained indicate that sub-criteria A, which is the most important, is the match between the system and the real-world sub-criteria. The next most important sub-criteria are recognizing rather than recall, help and documentation, aesthetic and minimalist design, flexibility and efficiency of use, error prevention, consistency and standard, user control and freedom, and visibility system status.

The results of the implementation method for the matrix of sub-criteria A are listed in Table 4.

$$\begin{gathered} \mathrm{Matrix} \mathrm{of} \mathrm{Sub-Criteria} \mathrm{A} \\ \begin{array}{l}SA=\left[\begin{array}{ccccccccc}\left(1,1,1\right)& \left(2.77,3.40,4.03\right)& \left(2.20,2.73,3.30\right)& \left(2.78,3.28,3.79\right)& \left(2.53,2.90,3.28\right)& \left(3.38,4.13,4.88\right)& \left(1.95,2.35,2.79\right)& \left(2.90,3.41,3.91\right)& \left(2.53,2,91,3.17\right)\\ \left(\frac{1}{4.03},\frac{1}{3.40},\frac{1}{2.77}\right)& \left(1,1,1\right)& \left(1.78,2.29,2.80\right)& \left(2.78,3.28,3.79\right)& \left(2.41,2.79,3.17\right)& \left(2.16,2.54,2.93\right)& \left(1.65,2.03,2.41\right)& \left(1.91,2.41,2.92\right)& \left(2.66,3.16,3.54\right)\\ \left(\frac{1}{3.30},\frac{1}{2.73},\frac{1}{2.20}\right)& \left(\frac{1}{2.80},\frac{1}{2.29},\frac{1}{1.78}\right)& \left(1,1,1\right)& \left(2.53,3.16,3.79\right)& \left(1.24,1.53,1.91\right)& \left(2.17,2.56,2.96\right)& \left(0.95,1.22,1.49\right)& \left(1.30,1.68,2.07\right)& \left(1.93,2.31,2.57\right)\\ \left(\frac{1}{3.79},\frac{1}{3.28},\frac{1}{2.78}\right)& \left(\frac{1}{3.79},\frac{1}{3.28},\frac{1}{2.78}\right)& \left(\frac{1}{3.79},\frac{1}{3.16},\frac{1}{2.53}\right)& \left(1,1,1\right)& \left(2.20,2.59,2.98\right)& \left(3.27,3.90,4.53\right)& \left(2.54,3.05,3.56\right)& \left(2.39,3.01,3.64\right)& \left(3.38,4.00,4.50\right)\\ \left(\frac{1}{3.28},\frac{1}{2.90},\frac{1}{2.53}\right)& \left(\frac{1}{3.17},\frac{1}{2.79},\frac{1}{2.41}\right)& \left(\frac{1}{1.91},\frac{1}{1.53},\frac{1}{1.24}\right)& \left(\frac{1}{2.98},\frac{1}{2.59},\frac{1}{2.20}\right)& \left(1,1,1\right)& \left(3.30,3.94,4.59\right)& \left(2.05,2.44,2.83\right)& \left(2.15,2.78,3.41\right)& \left(2.53,3.03,3.54\right)\\ \left(\frac{1}{4.88},\frac{1}{4.13},\frac{1}{3.38}\right)& \left(\frac{1}{2.93},\frac{1}{2.54},\frac{1}{2.16}\right)& \left(\frac{1}{2.96},\frac{1}{2.56},\frac{1}{2.17}\right)& \left(\frac{1}{4.53},\frac{1}{3.90},\frac{1}{3.27}\right)& \left(\frac{1}{4.59},\frac{1}{3.94},\frac{1}{3.30}\right)& \left(1,1,1\right)& \left(1.82,2.20,2.59\right)& \left(1.80,2.30,2.81\right)& \left(2.05,2.68,3.32\right)\\ \left(\frac{1}{2.79},\frac{1}{2.35},\frac{1}{1.95}\right)& \left(\frac{1}{2.41},\frac{1}{2.03},\frac{1}{1.65}\right)& \left(\frac{1}{1.49},\frac{1}{1.22},\frac{1}{0.95}\right)& \left(\frac{1}{3.56},\frac{1}{3.05},\frac{1}{2.54}\right)& \left(\frac{1}{2.83},\frac{1}{2.44},\frac{1}{2.05}\right)& \left(\frac{1}{2.59},\frac{1}{2.20},\frac{1}{1.82}\right)& \left(1,1,1\right)& \left(2.40,3.03,3.66\right)& \left(3.04,3.55,3.94\right)\\ \left(\frac{1}{3.91},\frac{1}{3.41},\frac{1}{2.90}\right)& \left(\frac{1}{2.92},\frac{1}{2.41},\frac{1}{1.91}\right)& \left(\frac{1}{2.07},\frac{1}{1.68},\frac{1}{1.30}\right)& \left(\frac{1}{3.64},\frac{1}{3.01},\frac{1}{2.39}\right)& \left(\frac{1}{3.41},\frac{1}{2.78},\frac{1}{2.15}\right)& \left(\frac{1}{2.81},\frac{1}{2.30},\frac{1}{1.80}\right)& \left(\frac{1}{3.66},\frac{1}{3.03},\frac{1}{2.40}\right)& \left(1,1,1\right)& \left(3.54,4.31,5.13\right)\\ \left(\frac{1}{3.17},\frac{1}{2.91},\frac{1}{2.53}\right)& \left(\frac{1}{3.54},\frac{1}{3.16},\frac{1}{2.66}\right)& \left(\frac{1}{2.57},\frac{1}{2.31},\frac{1}{1.93}\right)& \left(\frac{1}{4.50},\frac{1}{4.00},\frac{1}{3.38}\right)& \left(\frac{1}{3.54},\frac{1}{3.03},\frac{1}{2.53}\right)& \left(\frac{1}{3.32},\frac{1}{2.68},\frac{1}{2.05}\right)& \left(\frac{1}{3.94},\frac{1}{3.55},\frac{1}{3.04}\right)& \left(\frac{1}{5.13},\frac{1}{4.31},\frac{1}{3.54}\right)& \left(1,1,1\right)\end{array}\right]\end{array} \end{gathered}$$

Table 4 shows the priority results for sub-criteria B from the highest to the lowest order: relevance, diversity of learning content, feedback and assessment, learner control, collaborative learning, instructional material, instructional assessment, learning design, and finally timeliness. According to the experts who were the respondents in this study, the timeliness sub-criterion has the smallest weight, making it the lowest priority level.

$$\begin{gathered} \mathrm{Matrix} \mathrm{of} \mathrm{Sub-Criteria} \mathrm{B} \\ SB=\left[\begin{array}{ccccccccc}\left(1,1,1\right)& \left(1.13,1.30,1.54\right)& \left(1.93,2.45,2.97\right)& \left(1.20,1.34,1.49\right)& \left(1.36,1.65,2.03\right)& \left(1.33,1.60,1.88\right)& \left(2.20,2.59,3.04\right)& \left(1.72,2.23,2.73\right)& \left(1.08,1.22,1.38\right)\\ \left(\frac{1}{1.54},\frac{1}{1.30},\frac{1}{1.13}\right)& \left(1,1,1\right)& \left(4.00,4.75,5.38\right)& \left(2.08,2.36,2.69\right)& \left(3.28,3.79,4.30\right)& \left(2.15,2.66,3.16\right)& \left(1.57,1.85,2.16\right)& \left(2.51,2.89,3.27\right)& \left(1.82,2.22,2.66\right)\\ \left(\frac{1}{2.97},\frac{1}{2.45},\frac{1}{1.93}\right)& \left(\frac{1}{5.38},\frac{1}{4.75},\frac{1}{4.00}\right)& \left(1,1,1\right)& \left(2.17,2.56,2.96\right)& \left(2.07,2.47,2.91\right)& \left(1.56,1.94,2.33\right)& \left(2.17,2.80,3.44\right)& \left(2.51,3.02,3.52\right)& \left(2.79,3.30,3.81\right)\\ \left(\frac{1}{1.49},\frac{1}{1.34},\frac{1}{1.20}\right)& \left(\frac{1}{2.69},\frac{1}{2.36},\frac{1}{2.08}\right)& \left(\frac{1}{2.96},\frac{1}{2.56},\frac{1}{2.17}\right)& \left(1,1,1\right)& \left(1.94,2.47,3.03\right)& \left(1.90,2.41,2.91\right)& \left(2.06,2.58,3.10\right)& \left(3.14,3.77,4.39\right)& \left(2.06,2.58,3.10\right)\\ \left(\frac{1}{2.03},\frac{1}{1.65},\frac{1}{1.36}\right)& \left(\frac{1}{4.30},\frac{1}{3.79},\frac{1}{3.28}\right)& \left(\frac{1}{2.91},\frac{1}{2.47},\frac{1}{2.07}\right)& \left(\frac{1}{3.03},\frac{1}{2.47},\frac{1}{1.94}\right)& \left(1,1,1\right)& \left(2.41,2.92,3.43\right)& \left(2.27,2.64,3.02\right)& \left(2.40,2.78,3.17\right)& \left(3.04,3.54,4.05\right)\\ \left(\frac{1}{1.88},\frac{1}{1.60},\frac{1}{1.33}\right)& \left(\frac{1}{3.16},\frac{1}{2.66},\frac{1}{2.15}\right)& \left(\frac{1}{2.33},\frac{1}{1.94},\frac{1}{1.56}\right)& \left(\frac{1}{2.91},\frac{1}{2.41},\frac{1}{1.90}\right)& \left(\frac{1}{3.43},\frac{1}{2.92},\frac{1}{2.41}\right)& \left(1,1,1\right)& \left(1.91,2.41,2.92\right)& \left(3.29,3.92,4.43\right)& \left(1.76,2.01,2.27\right)\\ \left(\frac{1}{3.04},\frac{1}{2.59},\frac{1}{2.20}\right)& \left(\frac{1}{2.16},\frac{1}{1.85},\frac{1}{1.57}\right)& \left(\frac{1}{3.44},\frac{1}{2.80},\frac{1}{2.17}\right)& \left(\frac{1}{3.10},\frac{1}{2.58},\frac{1}{2.06}\right)& \left(\frac{1}{3.02},\frac{1}{2.64},\frac{1}{2.27}\right)& \left(\frac{1}{2.92},\frac{1}{2.41},\frac{1}{1.91}\right)& \left(1,1,1\right)& \left(3.39,3.89,4.27\right)& \left(2.03,2.29,2.42\right)\\ \left(\frac{1}{2.73},\frac{1}{2.23},\frac{1}{1.72}\right)& \left(\frac{1}{3.27},\frac{1}{2.89},\frac{1}{2.51}\right)& \left(\frac{1}{3.52},\frac{1}{3.02},\frac{1}{2.51}\right)& \left(\frac{1}{4.39},\frac{1}{3.77},\frac{1}{3.14}\right)& \left(\frac{1}{3.17},\frac{1}{2.78},\frac{1}{2.40}\right)& \left(\frac{1}{4.43},\frac{1}{3.92},\frac{1}{3.29}\right)& \left(\frac{1}{4.27},\frac{1}{3.89},\frac{1}{3.39}\right)& \left(1,1,1\right)& \left(1.82,2.10,2.41\right)\\ \left(\frac{1}{1.38},\frac{1}{1.22},\frac{1}{1.08}\right)& \left(\frac{1}{2.66},\frac{1}{2.22},\frac{1}{1.82}\right)& \left(\frac{1}{3.81},\frac{1}{3.30},\frac{1}{2.79}\right)& \left(\frac{1}{2.83},\frac{1}{2.44},\frac{1}{2.06}\right)& \left(\frac{1}{4.05},\frac{1}{3.54},\frac{1}{3.04}\right)& \left(\frac{1}{2.27},\frac{1}{2.01},\frac{1}{1.76}\right)& \left(\frac{1}{2.42},\frac{1}{2.29},\frac{1}{2.03}\right)& \left(\frac{1}{2.41},\frac{1}{2.10},\frac{1}{1.82}\right)& \left(1,1,1\right)\end{array}\right] \end{gathered}$$

For the matrix of sub-criteria B, the results of the implementation method are shown in Table 5. In sub-criteria B, the highest priority is B9 (relevancy), and the lowest priority is B1 (timeliness).

In sub-criteria C, as shown in Table 6 above, the motivation to learn sub-criteria is a top priority over the clarity of the purpose and objectives sub-criteria.

$$\begin{gathered} \mathrm{Matrix} \mathrm{of} \mathrm{Sub-Criteria} \mathrm{C} \\ SC=\left[\begin{array}{cc}\left(1,1,1\right)& \left(3.03,3.54,3.92\right)\\ \left(0.26,0.28,0.33\right)& \left(1,1,1\right)\end{array}\right] \end{gathered}$$

The results of the implementation method can be seen in Table 5.

The calculation using the ITFPP method involved the variable M whose magnitude was not explained in previous research, and to determine the effect on the consistency index value, a trial was carried out with the results shown in

Table 7. Trial of consistency index calculation with various M variables.

M	Main Criteria	Sub Criteria A	Sub Criteria B	Sub Criteria C
0.1	0.9996	0.9823	0.9901	1.0000
1	0.9959	0.8316	0.9044	1.0000
2	0.9918	0.6812	0.8162	1.0000
3	0.9877	0.5459	0.7346	1.0000

.

The determination of the value of M used was 0.1 because from the calculation of the consistency index, a more optimal value was obtained for the value of M = 0.1. In addition, Table 5 shows that the more parameters or criteria that are compared, the smaller the consistency index value.

Figure 2 and Figure 3 show that the suggested M variable is 0.1, and the best index consistently reaches 1 for sub-criteria C.

5. Conclusions and Future Work

The Nielsen Heuristic has achieved widespread adoption; however, it is frequently regarded as insufficiently specific for evaluating particular domains, such as e-learning. In the field of prioritization methods, the fuzzy analytic hierarchy process has found extensive application, generating priority weights through a pairwise comparison matrix. This study introduces the Inverse Trigonometric Fuzzy Preference Programming (ITFPP) method, employing a specific function (arc sin) to enhance the solution. The proposed method is utilized to calculate the priority level of e-learning evaluation criteria. The prioritization outcome of the modified usability heuristic can serve as a recommendation to assist interested parties in the development of their e-learning websites.

This investigation contributes not only to the theoretical underpinnings of decision science but also to the practical application of these methodologies in real-world scenarios. By elucidating the superiority of ITFPP over previous studies concerning the consistency index, this study provides valuable insights for researchers, practitioners, and decision-makers seeking optimal solutions in multi-criteria decision-making contexts for enhanced decision quality and reliability.

In our ongoing work, we are exploring ways to integrate a more dynamic perspective into our optimization models. This includes considering factors that might change over time and adapting our models to better align with the complexity of real-world educational settings. We aim to strike a balance between the need for optimization and the need for models that reflect the nuanced and evolving nature of educational scenarios.