A Novel Integration Method for D Numbers Based on Horizontal Comparison

Abstract

D numbers theory is an extension of Dempster–Shafer evidence theory. It eliminates the constraints of mutual exclusion and completeness under the frame of discernment of Dempster–Shafer evidence theory, so it has been widely used to deal with uncertainty modelling, but if it cannot effectively deal with the problem of missing information, sometimes unreasonable conclusions will be drawn. This paper proposes a new type of integration representation of D numbers, which compares the data of multiple evaluation items horizontally, and can reasonably fill in missing information. We apply this method to the user experience evaluation problem of online live course platform to verify the effectiveness of this method.

1. Introduction

The theory of multiple criteria decision making (MCDM) originated from the Pareto optimal concept, and was introduced into the field of decision science as a standardized decision-making method in the 1960s. In 1981, Hwang and Yoon [1] divided MCDM into two types: multiple objective decision making (MODM) and multiple attribute decision making (MADM) based on different decision spaces. The decision space of MODM is continuous, and the goal of decision-making is to plan and design the optimal plan; MADM corresponds to the discrete decision space, and the goal of decision-making is to use the evaluation information of multiple attributes, and through certain calculation or processing, several attribute values are converted into the comprehensive evaluation value of the scheme, so as to reasonably rank the limited alternatives. In the actual decision-making process of MADM problem, due to the complexity of the actual decision-making problem, the evaluation information of the project attributes often needs to be obtained with the cognition of domain experts, and the cognition of the experts participating in the evaluation is often difficult to accurately describe the problem, which brings great uncertainty. On the one hand, this uncertainty comes from the vagueness of the expert’s own cognition, and on the other hand, it comes from the inherent uncertainty of realistic decision-making problems. Moreover, for complex decision-making problems, a single expert has insufficient ability to deal with it. Therefore, uncertain multi-attribute group decision-making has become a research hotspot in decision theory in recent years. The main research methods include: grey correlation analysis [2], Dempster–Shafer evidence theory (D–S evidence theory) [3,4,5], fuzzy set theory [6,7,8], and rough sets theory [9,10,11], etc., among them, D–S evidence theory can well describe the uncertainty and incompleteness of evaluation information without prior probability. At the same time, the Dempster combination rule can effectively integrate data from multiple information sources, which makes it widely used in fields such as information fusion [12], fault diagnosis [13], reliability evaluation [14], human reliability analysis [15], etc.

An important extension of the D–S evidence theory is the D numbers theory [16]. The D number theory is mainly used for the representation and processing of uncertain information. It can be regarded as a special form of random set. It relaxes the assumptions of mutual independence between elements in the frame of discernment of D–S evidence theory, and relaxes the constraints of the completeness of the BPA function. Compared with fuzzy theories such as intuitionistic fuzzy sets and hesitation fuzzy sets and their extended forms, D numbers theory can directly express uncertainty by assigning probabilities to multiple subsets, which greatly improves the ability to represent and process uncertain information. However, the D numbers theory also has its own limitations. It is essentially a method of gathering multi-attribute information, which does not solve the shortcomings of the D–S evidence theory [16]. Therefore, scholars continue to improve the D numbers theory. For example, for multi-level and multi-attribute evaluation data, Deng et al. [17,18] combined the D numbers theory with analytic hierarchy process (AHP) and proposed the D-AHP method. Aiming at the problem that the combination rule cannot satisfy the associative law, Wang et al. [19] developed an improved multiple D numbers combination method, and Guan X et al. [20] proposed a new combination rule. In view of the problem that both qualitative and quantitative data exist at the same time, through the conversion model, all quantitative and qualitative data are transformed into the same qualitative description level, forming a D numbers decision matrix of the evaluation criteria, and using the D numbers integration method to integrate information under different criteria [21]. When the elements in the frame of discernment are not mutually exclusive, the distance between two D numbers is defined according to the definition form of the distance between two BPA in D–S evidence theory, so that the distance between two D numbers is smaller than the distance between D–S evidence theory under the same identification framework [22]. For data with incomplete information, Wang et al. [23] defined an improved D numbers integrated representation method to solve the MADM problem and so on. The D numbers approach is widely applied in different fields, including human reliability analysis [24], green supply chain management [25], and emergency management [26], university scientific research evaluation [27], failure model and effects analysis [28,29], renewable energy resources evaluation [30], and health-care waste treatment technology assessment [31]. Despite there are the above advantages for D numbers theory in dealing with uncertain information issues, some disadvantages still exist when transforming D numbers into the real value of D numbers’ integration rules. In this paper, improvements on the disadvantages of D numbers’ integration rules in incomplete information will be made, etc.

In the previous integrated representation of the D numbers, if the information is incomplete, the missing part of the assessment is directly ignored, and the result may violate common sense. In order to overcome this shortcoming, Wang et al. [23] proposed an improved D numbers integrated representation method, which proportionally distributes the missing information to the decision according to the original value of D numbers. However, this method does not consider the evaluation object globally, and the results may be counterintuitive. For instance, ten experts make an assessment of program A and program B. For program A, seven experts give eight points, and the other three experts make no assessment as they are unfamiliar with A, and this part is regarded as missing information, and the result can be expressed by $D_A=\left\{\left(8, 0.7\right)\right\}$. Based on the integrated method from the literature [16], it can be transformed into the real number $I\left(D_A\right)=8\times 0.7=5.6$, that is to say, when D numbers are integrated, 0 is given as the three experts do not make an assessment. According to the literature [23], when the D numbers are integrated, the missing part of information should be allocated to the given results of assessment in a certain proportion. There is only eight points for the given results, so the missing part should be assigned eight points, and the integration result is expressed by $I^{\prime }\left(D_A\right)=8\times 0.7+8\times 0.3=8$. The different ways the two integration results treat missing information leads to vastly different integration results. When it comes to the integration of $D_A$, we should also consider the assessment on program B made by the experts, and the integration results obtained can be more comparable. Based on [16,23], this paper proposes a new type of D numbers integrated representation, which can deal with the lack of information more reasonably. Then the proposed method is applied to the user experience evaluation problem of the online live course platform to verify the effectiveness of the method.

The rest of this paper is organized as follows. Section 2 first briefly introduces the basic concepts. Section 3 introduces our D numbers integration method (stepwise comparison and replacement integration (SCRI) method). Section 4 studies the implementation of our proposed method on the user experience evaluation problem of the online live course platform. Section 5 discusses and summarizes.

2. Preliminaries

2.1. Dempster–Shafer Theory

Aiming at the uncertainty and incompleteness of information, how to use expert group experience and collected relevant data as decision-making information to reduce the uncertainty of decision-making and improve the credibility of the results is the key content in the research of multi-attribute group decision-making methods. D–S evidence theory [3,4,5] provides a general framework for constructing uncertain reasoning models, which can satisfy an axiom system that is weaker than probability, can describe the uncertainty and incompleteness of evaluation information well. It also has a synthesis rule for synthesizing the opinions of multiple experts, so it is an important and effective group decision-making method.

Definition 1 

[3,4]. (basic probability assignment, BPA) Let $\Theta =\left\{{\theta }_1,{\theta }_2,\cdots ,{\theta }_n\right\}$ be a nonempty finite set of hypotheses or propositions, which is called the frame of discernment. A basic probability assignment (BPA) is a function $m:2^{\Theta }\to \left[0, 1\right]$, which is called a mass function, satisfying:

$$m\left(\varphi \right)=0$$

(1)

$$0\le m\left(A\right)\le 1, \forall A\subseteq \Theta$$

(2)

$${{\displaystyle \sum }}_{A\subseteq \Theta }m\left(A\right)=1$$

(3)

where $\varphi$ is an empty set and $2^{\Theta }$ is the power set of $\Theta$.

Definition 2 

[3,4]. (Dempster’s combination rule) Suppose $m_1$ and $m_2$ are two independent BPAs from different sources, the Dempster combination rule, denoted by $m=m_1\oplus m_2$,is defined as follows:

$$m\left(A\right)=\{\begin{array}{c}0, A=\varphi \\ \begin{array}{c}\\ \frac{1}{1-k}{\displaystyle {\displaystyle \sum }_{B{{\displaystyle \cap }}^{ }C=A}}{m}_1\left(B\right)m_2\left(C\right), \forall A\subseteq \Theta , A\ne \varphi \end{array}\end{array}$$

(4)

with

$$k={\displaystyle \sum }_{B{{\displaystyle \cap }}^{ }C=\varphi }{m}_1\left(B\right)m_2\left(C\right)$$

(5)

where $k$ is a normalization constant, called conflict coefficient of two BPAs. When$k=1$, it means that $m_1$ and $m_2$ completely conflict, and they cannot be combined by Dempster’s combination rule.

2.2. D Numbers Theory

D numbers theory [16] is an extension of Dempster–Shafer evidence theory. It eliminates the constraints of mutual exclusion and completeness under the frame of discernment. The general definition and combination rules of D numbers are given below.

Definition 3 

[16]. Let $\Omega$ be a finite nonempty set, D number is a mapping $D:\Omega \to \left[0, 1\right]$, such that

$${\displaystyle \sum }_{B\subseteq \Omega }D\left(B\right)\le 1 \mathrm{and} D\left(\varphi \right)=0$$

(6)

where $\varphi$ is an empty set and $B$ is a subset of $\Omega$. Note that different from the concept of frame of discernment in Dempster–Shafer theory, in set $\Omega$ the elements don’t require mutually exclusive.

There is a special situation of D numbers.

Definition 4 

[16]. For a discrete set $\Omega =\left(b_1,b_2,\cdots ,b_n\right)$, where$b_i\in {\mathrm{N}}^{+}$ ($i=1,2,\cdots ,n$), and $b_i\ne b_j$ if $i\ne j$, a special form of D numbers can be expressed by

$$\begin{gathered} D\left(\left\{b_1\right\}\right)=v_1 \\ D\left(\left\{b_2\right\}\right)=v_2 \\ \cdots \\ D\left(\left\{b_n\right\}\right)=v_n \end{gathered}$$

Simple notes for$D=\left\{\left(b_1,v_1\right),\left(b_2,v_2\right),\cdots ,\left(b_n,v_n\right)\right\}$, where$v_i>0$and${\displaystyle \sum }_{i=1}^n{v}_i\le 1$. If${\displaystyle \sum }_{i=1}^n{v}_i=1$,the information is said to be complete; If${\displaystyle \sum }_{i=1}^n{v}_i<1$, the information is said to be incomplete.

Definition 5 

[16]. (D numbers theory combination rule) Suppose $D_1$ and $D_2$ are two D numbers, denoted by

$$\{\begin{array}{c}{D}_1=\left\{\left(b_1^1,v_1^1\right),\cdots ,\left(b_i^1,v_i^1\right),\cdots ,\left(b_n^1,v_n^1\right)\right\}\\ D_2=\left\{\left(b_1^2,v_1^2\right),\cdots ,\left(b_j^2,v_j^2\right),\cdots ,\left(b_m^2,v_m^2\right)\right\}\end{array}\begin{array}{c}\\ \end{array}$$

The combination of$D_1$and$D_2$, indicated by$D=D_1\oplus D_2$, is defined by

$$D\left(b\right)=v$$

(7)

with

$$b=\frac{b_i^1+b_j^2}{2}$$

(8)

$$v=\frac{v_i^1+v_j^2}{2}/C$$

(9)

$$C=\{\begin{array}{cc}{\displaystyle \sum _{j=1}^m{\displaystyle \sum _{i=1}^n(\frac{v_i^1+v_j^2}{2}),}}& \hspace{0.17em}{\displaystyle \sum _{i=1}^n{v}_i^1}=1\hspace{0.17em}\hspace{0.17em}and\hspace{0.17em}{\displaystyle \sum _{j=1}^m{v}_j^2}=1\hspace{0.17em};\hspace{0.17em}\\ {\displaystyle \sum _{j=1}^m{\displaystyle \sum _{i=1}^n(\frac{v_i^1+v_j^2}{2})+{\displaystyle \sum _{j=1}^m(\frac{v_c^1+v_j^2}{2})},}}& \hspace{0.17em}{\displaystyle \sum _{i=1}^n{v}_i^1}<1\hspace{0.17em}\hspace{0.17em}and\hspace{0.17em}{\displaystyle \sum _{j=1}^m{v}_j^2}=1\hspace{0.17em};\\ {\displaystyle \sum _{j=1}^m{\displaystyle \sum _{i=1}^n(\frac{v_i^1+v_j^2}{2})+{\displaystyle \sum _{i=1}^n(\frac{v_i^1+v_c^2}{2})},}}& \hspace{0.17em}\hspace{0.17em}{\displaystyle \sum _{i=1}^n{v}_i^1}=1\hspace{0.17em}\hspace{0.17em}and\hspace{0.17em}{\displaystyle \sum _{j=1}^m{v}_j^2}<1\hspace{0.17em};\hspace{0.17em}\\ {\displaystyle \sum _{j=1}^m{\displaystyle \sum _{i=1}^n(\frac{v_i^1+v_j^2}{2})+{\displaystyle \sum _{j=1}^m(\frac{v_c^1+v_j^2}{2})}+{\displaystyle \sum _{i=1}^n(\frac{v_i^1+v_c^2}{2})}+\frac{v_c^1+v_c^2}{2},}}& \hspace{0.17em}\hspace{0.17em}{\displaystyle \sum _{i=1}^n{v}_i^1}<1\hspace{0.17em}\hspace{0.17em}and\hspace{0.17em}{\displaystyle \sum _{j=1}^m{v}_j^2}<1\hspace{0.17em};\hspace{0.17em}\end{array}$$

(10)

where

$$V_c^1=1-{\displaystyle {{\displaystyle \sum }}_{i=1}^n}{v}_i^1 \mathrm{and} V_c^2=1-{\displaystyle {{\displaystyle \sum }}_{j=1}^m}{v}_j^2.$$

We can treat D numbers as “special” discrete random variables. Taking$D_1=\left\{\left(b_1^1,v_1^1\right),\cdots ,\left(b_i^1,v_i^1\right),\cdots ,\left(b_n^1,v_n^1\right)\right\}$for example,$b_i^1$($i=1,2,\cdots ,n$) is the value for it, and$v_i^1$($i=1,2,\cdots ,n$) stands for the corresponding probability. It is a “special” discrete random variable as it requires the sum of probabilities to be less than or equal to 1, that is,${\displaystyle \sum }_{i=1}^n{v}_i\le 1$. It is stated by the definition that two D numbers, namely$D_1$and$D_2$, can be integrated into a new D number, that is,$D=D_1\oplus D_2$. Its value is$b=\frac{b_i^1+b_j^2}{2}$and the corresponding probability is$v=\frac{v_i^1+v_j^2}{2}/C$, of which$C$can be regarded as a regulation constant, and the value of$C$is determined by (10). In this case, it is guaranteed that the sum of all probabilities in new D numbers is still less than or equal to 1.

Definition 6 

[16]. (D numbers’ Integration) Let $D=\left\{\left(b_1,v_1\right),\left(b_2,v_2\right),\cdots ,\left(b_i,v_i\right),\cdots \left(b_n,v_n\right)\right\}$ be a D number, the integration representation of D is defined as

$$I\left(D\right)={\displaystyle \sum }_{i=1}^n{b}_i{v}_i$$

(11)

where $I\left(D\right)$ is real number.

3. Limitations and Proposed Method

3.1. Limitations

According to Deng [16], the integration representation of D numbers is defined as $I\left(D\right)={\displaystyle \sum }_{i=1}^n{b}_i{v}_i$. If the information given by experts is incomplete, namely ${\displaystyle \sum }_{i=1}^n{v}_i<1$, then the missing part of evaluation is directly excluded. Wang et al. [23] argued that this may lead to unreasonable outcomes of decision-making.

Example 1 

(Wang et al. [23]). According to Wang et al. [23], it is assumed that 10 experts are invited to evaluate the automobile performance of A and B, and the evaluation system is set $\Omega =\left\{1,2,3,4,5,6,7,8,9\right\}$, in which “9” means “very good” and “1” means “very bad”. When the 10 experts give an evaluation on the performance of car A, six of them choose “7”, two choose “8”, and the other two do not give an evaluation as they are not familiar with the car. The evaluation result can be expressed as $D_A=\left\{\left(7,0.6\right),\left(8,0.2\right)\right\}$. When the same 10 experts evaluate the performance of car B, they all choose “6”. The expression of D numbers is $D_B=\left\{\left(6, 1.0\right)\right\}$. Based on the D integrated representation of D numbers, the final evaluation results of the two cars are respectively calculated as:

$$I\left(D_A\right)=7\times 0.6+8\times 0.2=5.8$$

$$I\left(D_B\right)=6\times 1.0=6$$

In terms of the integrated results, car B gets a better score than car A, as two experts are uncertain about the score when evaluating A and do not give an evaluation, and incomplete information is abandoned in integration of D numbers. That is to say, if the experts give no evaluation, 0 is applied to deal with it, thus making the final result of car A relatively low.

Wang et al. [23] put forward a method to improve the integration of D numbers. They argued that for experts with incomplete cognition, they could refer to the evaluation results given by experts familiar with the evaluation results, and the incomplete part of incomplete information should be assigned to the evaluation results given in line with certain rules. For the given numbers, $D=\left\{\left(b_1,v_1\right),\left(b_2,v_2\right),\cdots ,\left(b_i,v_i\right),\cdots \left(b_n,v_n\right)\right\}$, $b_i$, ($\mathrm{i}=1,2,\cdots ,\mathrm{n}$) is the evaluation level given by experts, $v_i$, ($i=1,2,\cdots ,n$) is the credibility of corresponding evaluation level $b_i$, and $\frac{v_i}{{{\displaystyle \sum }}_i{v}_i}$ stands for the percentage of credibility of evaluation level $b_i$. Wang et al. [23] believed that the incomplete part of incomplete information should be assigned to the known evaluation level $b_i$($i=1,2,\cdots ,n$) according to the percentage of credibility.

Definition 7 

(Wang et al. [23]). For $D=\left\{\left(b_1,v_1\right),\left(b_2,v_2\right),\cdots ,\left(b_i,v_i\right),\cdots \left(b_n,v_n\right)\right\}$. The overall assessment is denoted by,

$$I^{\prime }\left(D\right)={\displaystyle \sum }_{i=1}^n\left\{b_i\times \left[v_i+\left(1-{\displaystyle \sum }_i{v}_i\right)\times \frac{v_i}{{{\displaystyle \sum }}_{i=1}{v}_i}\right]\right\}$$

(12)

Obviously, if the information is complete, that is, when${\displaystyle \sum }_{i=1}^n{v}_i=1$, the proposed method is degenerated as the traditional D numbers’ integration.

The improved integration representation of D numbers is adopted to reevaluate the performance of car A and car B in Example 1. For car B, the information is complete, thus:

$$I^{\prime }\left(D_B\right)=I\left(D_B\right)=6\times 1.0=6$$

For car A, $D_A=\left\{\left(7,0.6\right),\left(8,0.2\right)\right\}$ is incomplete, and the incomplete part of 0.2 should be assigned to the known evaluation level based on the percentage of credibility, thereby:

$$\begin{array}{c}{I}^{\prime }\left(D_A\right)=7\times \left\{0.6+\left(1-0.6-0.2\right)\times \frac{0.6}{0.6+0.2}\right\}+8\times \left\{0.2+\left(1-0.6-0.2\right)\times \frac{0.2}{0.6+0.2}\right\}=\frac{29}{4} \\ \end{array}$$

Obviously, when the improved integration representation of D numbers is adopted, car A gets a higher total score than car B, which is contrary to the traditional integrated results of D numbers.

Nevertheless, we also see that the improved integration representation of D numbers can also bring about some counterintuitive results, as shown in the following example:

Example 2.

It is still assumed that 10 experts are invited to evaluate the automobile performance of A and B, and the evaluation system is set $\Omega =\left\{1,2,3,4,5,6,7,8,9\right\}$. When the 10 experts evaluate the performance of car A, seven of them choose “8” and three of them do not evaluate as they are not familiar with the car. Thu, the evaluation of car A is expressed as $D_A=\left\{\left(8, 0.7\right)\right\}$. When the same 10 experts evaluate the performance of car B, eight of them choose “8” and the rest two choose “7”. Thus, the expression is $D_B=\left\{\left(8, 0.8\right),\left(7,0.2\right)\right\}$. For car A, $D_A=\left\{\left(8, 0.7\right)\right\}$ is incomplete. The incomplete part of 0.3 should be assigned to the known evaluation level in line with the percentage of credibility, while the known evaluation level is only “8”. Thereby:

$$\begin{array}{c}{I}^{\prime }\left(D_A\right)=8\times \left\{0.7+\left(1-0.7\right)\times \frac{0.7}{0.7}\right\}=8 \\ \end{array}$$

For car B, the information is complete, thus:

$$I^{\prime }\left(D_B\right)=8\times 0.8+7\times 0.2=7.8$$

Obviously, the evaluation result obtained through this method is that car A is in the ascendant, but from the original evaluation result, the number of experts who give a score of “8” to car B is more than that of experts who give score of “8” to car A, and complete information is obtained for the evaluation result of car B. The reason for this lies in the fact that in certain information of $D_A$, the score of “8” accounts for 100%, and when the improved integration representation of D numbers is adopted, the incomplete information remained, namely opinions from the three experts who do not give and evaluation, are replaced with “8”. If the above operation is conducted when the evaluation results of car B are not fully considered, the final score of car A might be made too high. In this case, a more effective integration representation of D numbers needs to be proposed.

3.2. Proposed Method (Stepwise Comparison and Replacement Integration (SCRI) Method)

In this section, a new method of D numbers integration is proposed, which not only considers the assessment results of a single program, but also horizontally compares with the assessment results of other programs. It can fill in the missing information in a more reasonable way. Compared with existing methods, it is more effective.

It assumes that there are N objects $O_1,O_2,\cdots ,O_N$,to be evaluated, and $M$ experts in total who participate in the evaluation. The stepwise representation of the proposed method is given as follows. The flowchart of the proposed method is shown in Figure 1.

Step 1 for the object $O_i$, the evaluation grade from each expert is arranged in descending order, and if there is no evaluation, the temporary record is 0. That is to say, the evaluation result from experts on the object $O_i$ arranged in descending order is viewed as a row vector, thus forming the original evaluation matrix $R$ of $N\times M$.

Step 2 there is a total of $M$ experts participating in the evaluation, and the corresponding credibility from each expert is considered as $1/M$. Each element in the original evaluation matrix $R$ is multiplied by $1/M$, and it is concluded that the initial integrated value matrix is $R_{Int}$.

Step 3 observe whether there is a column in which elements are all 0. If so, enter step 4; otherwise, go directly to step 5.

Step 4 check each column from the first column to see whether there exists element 0. It assumes that 0 appears in the $s+1$ column for the first time, that is to say, the corresponding elements for the first $\mathrm{s}$ columns are initial integrated value obtained based on the definite evaluation and calculation from experts. Add up each element of each row in the first $s$ columns and then take the average of their sum, that is to divide the sum of each row by $s$, which is called reference value for basic evaluation. As for the columns where all the column elements are 0, replace the element 0 in this column with the reference value for basic evaluation of the corresponding row, that is to assign this part of incomplete information to the evaluation results clearly given in the previous $\mathrm{s}$ columns in proportion. Then it comes to the matrix $R_{Int-r}$.

Step 5 check whether there is element 0 in each column. If so, make a comparison of the non-zero elements in this column, and substitute the minimum value for the element 0. That is to say, the incomplete information is replaced with the minimum evaluation value corresponding to the evaluation object through horizontal comparison, until all elements in the matrix are nonzero, which is called the modified integrated value matrix $R_{Int}^{*}$.

Step 6 add up elements of each row in the matrix, and then it comes to the value, which is the integrated value of D numbers corresponding to the evaluation objects $O_1,O_2,\cdots ,O_N$.

Let us use this method to operate Example 2.

Step 1 set up an original evaluation matrix $R=\left(\begin{array}{cccccccccc}8& 8& 8& 8& 8& 8& 8& 0& 0& 0\\ 8& 8& 8& 8& 8& 8& 8& 8& 7& 7\end{array}\right)$.

Step 2 multiply each element by $1/M$ , here $M=10 \mathrm{is} \mathrm{the} \mathrm{number} \mathrm{of} \mathrm{experts}$, and then it comes to the initial integrated value matrix. The weights of experts are the same $1/10$.

$$R_{Int}=\left(\begin{array}{cccccccccc}8/10& 8/10& 8/10& 8/10& 8/10& 8/10& 8/10& 0& 0& 0\\ 8/10& 8/10& 8/10& 8/10& 8/10& 8/10& 8/10& 8/10& 7/10& 7/10\end{array}\right).$$

Step 3 no column with all 0 elements, then go to step 5.

Step 5 checking whether element 0 appears in each column. If so, make a comparison of the non-zero elements in this column and replace element 0 with the minimum value.

$$R_{Int}^{*}=\left(\begin{array}{cccccccccc}8/10& 8/10& 8/10& 8/10& 8/10& 8/10& 8/10& 8/10& 7/10& 7/10\\ 8/10& 8/10& 8/10& 8/10& 8/10& 8/10& 8/10& 8/10& 7/10& 7/10\end{array}\right)$$

Step 6 add up elements of each row in the matrix, and then it comes to the integrated value of D numbers corresponding to evaluation objects A and B, which are all 7. The integration results of the two cars are the same, which is more in line with our intuition.

Example 3.

It still assumes that 10 experts are invited to evaluate three objects A, B, and C, and set $\Omega =\left\{1,2,3,4,5,6,7,8,9\right\}$ is adopted as the evaluation system. When these 10 experts make evaluation of A, 4 choose “9”, 2 choose “8”, and 3 choose “7”, and as the other one expert is not familiar with A, no evaluation is given. Thus, evaluation for A is expressed as $D_A=\left\{\left(9, 0.4\right),\left(8, 0.2\right),\left(7, 0.3\right)\right\}$. Similarly, when the 10 experts evaluate the automobile performance of B, 3 choose “9”, 5 choose “8”, and the other 2 experts do not give evaluation as they are not familiar with B. Thus, the expression is $D_B=\left\{\left(9, 0.3\right),\left(8,0.5\right)\right\}$. When the 10 experts evaluate the automobile performance of C, 4 choose “9”, 3 choose “8”, and the other 3 do not give evaluation as they are not familiar with C. The expression for D numbers is $D_C=\left\{\left(9, 0.4\right),\left(8,0.3\right)\right\}$.

Step 1 set up an original evaluation matrix $R=\left(\begin{array}{cccccccccc}9& 9& 9& 9& 8& 8& 7& 7& 7& 0\\ 9& 9& 9& 8& 8& 8& 8& 8& 0& 0\\ 9& 9& 9& 9& 8& 8& 8& 0& 0& 0\end{array}\right)$

Step 2 multiply each element by $1/M$ , here $M=10$, and then it comes to the initial integrated value matrix

$$R_{Int}=\left(\begin{array}{cccccccccc}9/10& 9/10& 9/10& 9/10& 8/10& 8/10& 7/10& 7/10& 7/10& 0\\ 9/10& 9/10& 9/10& 8/10& 8/10& 8/10& 8/10& 8/10& 0& 0\\ 9/10& 9/10& 9/10& 9/10& 8/10& 8/10& 8/10& 0& 0& 0\end{array}\right)$$

Step 3 observe whether there exists a column in which all column elements are 0. Herein, the elements are all 0 in the last column, then go to step 4.

Step 4 0 appears in the 8th column for the first time, that is, the elements corresponding to the first seven columns are interval integrated values obtained based on the definite evaluation given by the experts. The elements in each row of the first seven columns are added up and then averaged, and $59/70$,$59/70$,$60/70$ are obtained respectively. Replace these values with the elements in the last column of the initial integrated value matrix, and then it comes to the matrix $R_{Int-r}$.

$$R_{Int-r}=\left(\begin{array}{cccccccccc}9/10& 9/10& 9/10& 9/10& 8/10& 8/10& 7/10& 7/10& 7/10& 59/70\\ 9/10& 9/10& 9/10& 8/10& 8/10& 8/10& 8/10& 8/10& 0& 59/70\\ 9/10& 9/10& 9/10& 9/10& 8/10& 8/10& 8/10& 0& 0& 60/70\end{array}\right)$$

Step 5 check whether element 0 appears in each column. If so, make a comparison of the non-zero elements in this column and replace element 0 with the minimum value. As for $min\left(\frac{7}{10},\frac{8}{10}\right)=\frac{7}{10}$ in the 8th column, replace 0 with $7/10$, and in the 9th column, replace 0 with $7/10$ to obtain the modified integrated value matrix $R_{Int}^{*}$:

$$R_{Int}^{*}=\left(\begin{array}{cccccccccc}9/10& 9/10& 9/10& 9/10& 8/10& 8/10& 7/10& 7/10& 7/10& 59/70\\ 9/10& 9/10& 9/10& 8/10& 8/10& 8/10& 8/10& 8/10& 7/10& 59/70\\ 9/10& 9/10& 9/10& 9/10& 8/10& 8/10& 8/10& 7/10& 7/10& 60/70\end{array}\right)$$

Step 6 add up elements of each row in the matrix, and then it comes to the integrated value of D numbers corresponding to evaluation objects A, B and C, which are 8.143, 8.243 and 8.257 respectively. Thereby, we consider that object $\mathrm{C}\gg \mathrm{B}\gg \mathrm{A}$, in which the symbol $‘\gg ’$ means ‘superior’.

4. Applications in MADM

In this section, we will put forward the method of D numbers integration that is adopted to make an assessment of user experience of the online live course platform, and account for the rationality of the proposed method by data collection based on the evaluation index system in the literature [32].

During the rapid transmission of the COVID-19 pandemic, online learning has become an effective way to implement “suspended class, ongoing learning”, and carried out all over the world. Since there are reasons such as technology and network bandwidth, etc., some telecommuting and conferencing software has undertaken the role of online live course platforms. However, due to the original design purpose and the characteristics of the software, the software cannot fully meet the needs of schools in online live courses. In this case, it is prominently significant to optimize the user experience of the online live course platforms [32].

In Wang et al. [32], qualitative and quantitative methods were combined to establish evaluation indicators by integrating data obtained through the literature research, interviews and network research from the perspective of user experience. Delphi method and analytic hierarchy process were utilized to establish the weight of each indicator and set the scoring grade. T-test method was adopted to establish the comparative analysis method of model application, and the standard procedure of model application was designed to build a complete evaluation system.

However, there is an important issue that has not been well resolved in [32]: when evaluating different platforms, some experts may not give definite scores as they are not familiar with the scoring items. In this case, the scores made reluctantly may greatly affect the final evaluation results. The SCRI method proposed by us can solve this in a good way, and it can deal with possible uncertainty, ignorance and missing information in a simple and effective way.

In Wang et al. [32], the main factors affecting the user experience of the online live course platforms are divided into four primary indicators, which are respectively Ease of Operation, Functional Completeness, Interface Rationality, Technical Reliability. Each primary indicator is divided into several secondary indicators. The hierarchical structure and corresponding weights of user experience evaluation of the online live course platforms are shown in the Figure 2. below.

In the hierarchical structure, a total of 16 secondary indicators are included. Experts are invited to evaluate all the influencing factors. To keep consistent with the requirements of the scale, the evaluation results are divided into 9 grades, $\Omega =\left\{1,2,3,4,5,6,7,8,9\right\}$, in which “9” means “very good” and “1” means “very bad”. If the experts do not have a good understanding of some factors, they do not need to score, and finally we obtain evaluation data with missing information.

All the results are expressed in the form of D numbers herein. For instance, 10 experts are invited to evaluate the user experience of the platform. When evaluating a certain influencing factor, 4 experts give 8 scores, 3 experts give 7 scores, 2 experts give 5 scores, and the other one expert does not score as he or she is unfamiliar with the factor. Hence, the evaluation result can be expressed as $\left\{\left(8, 0.4\right),\left(7,0.3\right),\left(5,0.2\right)\right\}$, which is not a complete D number. Ding Talk, Tencent Classroom and Tencent QQ are adopted in this paper, and the evaluation results are shown in the following

Table 1. Assessment data for Ding Talk, Tencent Classroom and Tencent QQ are represented by D numbers.

Basic Attribute	Ding Talk	Tencent Classroom	Tencent QQ
Y1	{(9, 0.2), (8, 0.5), (7, 0.3)}	{(8, 0.5), (7,0.2), (6, 0.2), (5, 0.1)}	{(7, 0.5), (6, 0.2), (5, 0.2), (4,0.1)}
Y2	{(9, 0.2), (8, 0.5), (6, 0.2)}	{(8, 0.2), (7, 0.2), (6, 0.2), (5, 0.2)}	{(8, 0.2), (7, 0.3), (6, 0.1), (5, 0.1)}
Y3	{(9, 0.2), (8, 0.2), (7, 0.3), (6, 0.1)}	{(8, 0.4), (7, 0.3), (6, 0.2)}	{(8, 0.2), (7, 0.4), (5, 0.1)}
Y4	{(8, 0.2), (7, 0.2), (6, 0.4)}	{(8, 0.2), (7, 0.2), (6, 0.6)}	{(7, 0.2), (6, 0.4), (5, 0.2), (4, 0.1)}
Y5	{(9, 0.6), (8, 0.2), (7, 0.2)}	{(8, 0.2), (7, 0.2), (6, 0.1), (5, 0.4)}	{(8, 0.4), (7, 0.2), (5, 0.2)}
Y6	{(7, 0.2), (6, 0.2), (5, 0.1), (4, 0.2)}	{(7, 0.2), (5, 0.4), (4, 0.2)}	{(9, 0.2), (8, 0.2), (7, 0.3), (5, 0.1)}
Y7	{(8, 0.2), (6, 0.2), (5, 0.4)}	{(9, 0.2), (7, 0.2), (6, 0.4)}	{(7, 0.4), (6, 0.2), (5, 0.2)}
Y8	{(9, 0.4), (8, 0.4), (6, 0.2)}	{(9, 0.2), (8, 0.3), (6, 0.2), (5, 0.2)}	{(8, 0.4), (7, 0.1), (5, 0.2), (4, 0.1)}
Y9	{(9, 0.1), (8, 0.4), (6, 0.3)}	{(8, 0.1), (7, 0.6), (6, 0.1)}	{(8, 0.4), (7, 0.3), (4, 0.2)}
Y10	{(9, 0.3), (7, 0.2), (6, 0.4)}	{(8, 0.5), (7, 0.2), (6, 0.2)}	{(9, 0.2), (8, 0.6)}
Y11	{(6, 0.6), (5, 0.2)}	{(8, 0.2), (7, 0.2), (6, 0.2),(4, 0.1), (3, 0.1)}	{(8, 0.4), (6, 0.2), (4, 0.1)}
Y12	{(8, 0.3), (7, 0.4), (5, 0.1)}	{(9, 0.2), (8, 0.2), (6, 0.3), (5, 0.1)}	{(8, 0.3), (7, 0.2), (6, 0.2), (5, 0.1)}
Y13	{(9, 0.4), (8, 0.2), (7, 0.2), (5, 0.2)}	{(8, 0.4), (7, 0.2), (6, 0.2), (5, 0.2)}	{(9, 0.2), (7, 0.4), (6, 0.2)}
Y14	{(9, 0.2), (8, 0.4), (6, 0.3)}	{(7, 0.4), (6, 0.4), (5, 0.2)}	{(8, 0.2), (7, 0.2), (6, 0.2), (5, 0.1), (4, 0.1)}
Y15	{ 9, 0.2), (8, 0.4), (7, 0.2), (5, 0.2)}	{(8, 0.2), (7, 0.2), (6, 0.6)}	{(8, 0.2), (7, 0.2), (6, 0.2), (5, 0.4)}
Y16	{(8, 0.4), (7, 0.1), (6, 0.4)}	{(8, 0.3), (7, 0.2), (6, 0.4)}	{(7, 0.5), (6, 0.1),(5, 0.3)}

:

Then we adopted the SCRI method to transform the evaluation results from all experts into a clear form of real numbers. For instance, in the

Table 2. Assessment data for Y2 of the three platforms are represented by D numbers.

Y2

{(9, 0.2), (8, 0.5), (6, 0.2)}

{(8, 0.2), (7, 0.2), (6, 0.2), (5, 0.2)}

{(8, 0.2), (7, 0.3), (6, 0.1), (5, 0.1)}

, the evaluation results on Y2 made by the ten experts are as follows:

Step 1, establish the original evaluation matrix $R=\left(\begin{array}{cccccccccc}9& 9& 8& 8& 8& 8& 8& 6& 6& 0\\ 8& 8& 7& 7& 6& 6& 5& 5& 0& 0\\ 8& 8& 7& 7& 7& 6& 5& 0& 0& 0\end{array}\right)$;

Step 2, multiply each element by$1/10$ to obtain the initial integrated value matrix

$$R_{Int}=\left(\begin{array}{cccccccccc}9/10& 9/10& 8/10& 8/10& 8/10& 8/10& 8/10& 6/10& 6/10& 0\\ 8/10& 8/10& 7/10& 7/10& 6/10& 6/10& 5/10& 5/10& 0& 0\\ 8/10& 8/10& 7/10& 7/10& 7/10& 6/10& 5/10& 0& 0& 0\end{array}\right)$$

Step 3, elements in the last column are all 0, then go to step 4;

Step 4, 0 appears in the 8th column for the first time, that is, the elements corresponding to the first 7 columns are initial integrated values obtained based on the definite evaluation and calculation given by the experts. Add up the elements in each row of the first 7 columns and calculate the arithmetic mean, and $58/70$,$47/70$,$48/70$ are obtained respectively. Replace these with the elements of the last column in the matrix ${\mathrm{R}}_{\mathrm{Int}}$, and it is concluded that:

$$R_{Int-r}=\left(\begin{array}{cccccccccc}9/10& 9/10& 8/10& 8/10& 8/10& 8/10& 8/10& 6/10& 6/10& 58/70\\ 8/10& 8/10& 7/10& 7/10& 6/10& 6/10& 5/10& 5/10& 0& 47/70\\ 8/10& 8/10& 7/10& 7/10& 7/10& 6/10& 5/10& 0& 0& 48/70\end{array}\right)$$

Step 5, check whether element 0 appears in each column. If so, make a comparison of the non-zero elements in this column and replace element 0 with the minimum value. As for $min\left(\frac{6}{10},\frac{5}{10}\right)=\frac{5}{10}$ in the 8th column, replace 0 with $5/10$, and in the 9th column, replace 0 with $6/10$ to obtain the modified integrated value matrix ${\mathrm{R}}_{\mathrm{Int}}^{*}$:

$$R_{Int}^{*}=\left(\begin{array}{cccccccccc}9/10& 9/10& 8/10& 8/10& 8/10& 8/10& 8/10& 6/10& 6/10& 58/70\\ 8/10& 8/10& 7/10& 7/10& 6/10& 6/10& 5/10& 5/10& 6/10& 47/70\\ 8/10& 8/10& 7/10& 7/10& 7/10& 6/10& 5/10& 5/10& 6/10& 48/70\end{array}\right)$$

Step 6, add up the elements of each row in the matrix and convert the evaluation results from all experts into a clear form of real numbers, as follows:

7.8286, 6.4714 and 6.5857.

The evaluation results of all secondary indicators are converted into real numbers as shown in

Table 3. Integration of the assessment results.

Basic Attribute	Ding Talk	Tencent Classroom	Tencent QQ
Y1	7.9000	7.1000	6.1000
Y2	7.8286	6.4714	6.5857
Y3	7.4857	7.2571	6.8000
Y4	6.4000	6.6000	5.8000
Y5	8.4000	6.3000	6.8000
Y6	5.4143	5.2857	7.5714
Y7	6.0000	7.0000	6.2500
Y8	8.0000	7.0000	6.4000
Y9	7.0375	6.7000	6.8125
Y10	7.2375	7.3500	8.0250
Y11	5.7714	6.2143	6.4714
Y12	7.1250	7.1250	6.8750
Y13	7.6000	6.8000	6.8000
Y14	7.3000	6.2000	6.1000
Y15	7.4000	6.6000	6.2000
Y16	7.0000	6.8889	6.2222

:

Finally, multiply these numbers by the corresponding weights and total them up to obtain the comprehensive evaluation value of user experience from the three platforms, as well as the scores of each primary indicator in

Table 4. The overall assessments and the ranking of the three platforms.

Primary Indicators	Ding Talk	Tencent Classroom	Tencent QQ
Ease of operation	2.9405	2.6771	2.4509
Functional completeness	0.6510	0.5799	0.6504
Interface rationality	1.0617	1.0836	1.1410
Technical reliability	2.5977	2.3451	2.2468
Aggregate score	7.2508	6.6856	6.4891
ranking	1	2	3

:

5. Conclusions and Future Work

As an extension of D–S evidence theory, D numbers theory has recently attracted the attention from scholars, but it is not quite satisfactory while dealing with incomplete information. In this paper, a new integration method SCRI is proposed, of which the advantages are as follows: reasonably filling the missing information by comparing the data of several assessment programs in a horizontal way, to a certain extent, makes up for the decision-making mistakes which may be led by missing information for the D numbers theory in integrated data. The possible limitation of this method is that the information filled in is only the inference of the missing part, which is not guaranteed to be completely accurate. This method can be combined with other MADM classical methods to deal with decision-making issues in uncertain environments such as fuzziness and incompleteness, which is also the direction of subsequent efforts.