Algorithm for Option Number Selection in Stochastic Paired Comparison Models

Abstract

In this paper, paired comparison models with a stochastic background are investigated and compared from the perspective of the option numbers allowed. As two-option and three-option models are the ones most frequently used, we mainly focus on the relationships between two-option and four-option models and three-option and five-option models, and then we turn to the general s- and $(s+2)$-option models. We compare them from both theoretical and practical perspectives; the latter are based on computer simulations. We examine, when it is possible, mandatory, or advisable how to convert four-, five-, and $(s+2)$-option models into two-, three-, and s-option models, respectively. The problem also exists in reverse: when is it advisable to use four-, five-, and $(s+2)$-option models instead of two-, three-, and s-option models? As a result of these investigations, we set up an algorithm to perform the decision process.

1. Introduction

Comparisons in pairs are frequently used in a broad spectrum of problems, across all areas of life. Without claiming to be exhaustive, we mention education [1], management [2], psychology [3], marketing [4], construction [5], social science [6], group decisions [7], engineering [8], medicine [9], economics [10], resource allocation [11], and sports [12] as areas of their application. A paper by Vaidya and Kumar [13] presents more than 150 of the applications that they had been used for before 2003.

Paired comparisons play a significant role in decision-making [14,15]. This method allows for a detailed analysis of the differences and similarities between objects, properties, and aspects, aiding in the selection of the best choice. It is particularly useful in situations where multiple factors must be considered and a clear ranking is necessary [13,16]. By applying paired comparisons, complex problems can be simplified, leading to more objective decisions.

Paired comparisons means that, instead of using a scale, the objects to be evaluated are compared directly to each other, and the decision of the observer (evaluator) is based on the relative merits of the two objects. These opinions are generally more reliable than evaluations based on scales, as it is easier to determine which of two objects, properties, aspects, etc., is ‘better’, ‘more appropriate’, ‘more important’, etc., than to describe them with a single number. This is especially true for evaluations based on subjective criteria. In the case of paired comparisons, their data represent the relations between the observed objects. These types of data require methods that are different from conventional statistical procedures.

The most popular method based on comparisons in pairs is the Analytic Hierarchy Process (AHP) associated with Saaty [17]. This single paper has more than 10 thousand citations; some of these articles provide criticism, see for example [18]. The starting point of the model is a pairwise comparison matrix (PCM), the elements of which show how many times one object is stronger than another. The matrix $A=\left(a_{i,j}\right)$ is reciprocal–symmetric, that is $a_{i,j}={\displaystyle \frac{1}{a_{i,j}}}$. According to Saaty’s original model [17], $a_{i,j}\in \{1/9,1/8,\dots ,1/2,1,2,\dots ,8,9\}$. The observers usually express their decisions verbally by voting for ‘equally preferred’, ‘moderately preferred’, ‘strongly preferred’, ’very strongly preferred’, or ‘extremely preferred’. These verbal expressions help decision-makers articulate their preferences in a qualitative manner, which are converted to quantitative values; 1, 3, 5, 7, or 9, respectively. Intermediate values are used for comparisons that fall between these levels. We mention that the number of options may be reduced in light of the problem under consideration. Their quantitative results are analyzed using different methods. Usually, every object is compared to every other object; that is, the PCM does not contain missing elements. In this case, the number of pairs under comparison increases quadratically with the number of objects examined. If the number of objects is 10, then the number of pairs equals ${\displaystyle \frac{10·9}{2}}=45$, which can be quite overwhelming for an observer. The reliability of their opinions and the measure of the contradictions between their decisions are checked by the inconsistency index of the PCM. The definition of consistency is given in [17] for complete cases; it was recently generalized to incomplete cases in [19]. There are numerous ways to quantify the degree of inconsistency seen [20,21]. Some research papers present procedures for reducing the level of inconsistency [22,23,24,25]. Most measurements are for complete cases, but they can also be generalized to incomplete cases [26]. The most frequently used measurement is related to the eigenvector method [17,24], which is the most common evaluation method, but the logarithmic least squares method (LLSM) is also readily used for evaluations, although mainly in incomplete cases [19,27,28].

Not only can the evaluation method change, but so can the way the PCM is constructed. Maintaining its reciprocal–symmetric property, the matrix can also be formed in another way than how Saaty constructed it. In [29], the authors define the elements of the PCM as the ratio of the number of wins to losses. In a socio-economic study [30], the author used the ratio of certain economic characteristics as the elements of the matrix. In this case, the number of possible options that require a decision is infinitely many. Both the eigenvector method (with some modification) and LLSM work for these PCM cases under the same conditions: the PCM matrix has to be irreducible, i.e., the representing graph must be connected [27,28].

We note that other methods, such as the Preference Ranking Organization Method for Enrichments Evaluations (PROMETHEE) and Elimination and Choice Translating Reality (ELECTREE, ELECTRE) are often applied, mainly when focusing on multi-criteria decision making [31,32,33].

Another branch of the models connected to comparisons in pairs are the Thurstone motivated models, which have stochastic background. Based on Thurstone’s brilliant idea, these models envision the latent probabilistic variables behind each object [34]. The starting points were psychological problems but, based on Thurstone motivated models, it is also possible to solve the ranking and rating of objects related to other research questions [35,36,37,38]. Thurstone assumed the latent variables had a Gaussian distribution, while Bradley and Terry assumed a logistic distribution [39]. In [34,39], two options were allowed for the decisions of the observers, but the models were generalized to three options, allowing ties [40,41]. Models allowing more than three options are presented in [42]. The ranking and rating of these objects are determined by the expectation of latent random variables. There are numerous possibilities for estimating their parameters. In the case of a Gaussian distribution, least squares methods are usually applied, but in cases of logistic distribution, maximum likelihood estimations are widespread. One publication [42] also applied least squares estimations based on relative frequencies and a cumulative distribution function. In [43], a general model for applying a Gaussian distribution and allowing more than two options is presented. The parameter estimation method is the maximum likelihood method, as it contains many inherent possibilities.

The maximum likelihood estimation (MLE) method is based on the maximization of a multi-variate function, and it requires the clarification of the conditions under which the maximum exists and is unique. In [43], the authors provide a sufficient condition for its existence and uniqueness in the case of a Gaussian distribution, and in [44] they generalize this for the case of general strictly log-concave distributions. Previously known conditions are used in [45], assuming a logistic distribution in the case of the two-option model, and in [46] they are used to investigate a modified model in the case of three choices. This condition, in the case of two options, is a necessary and sufficient condition. The condition given by Davidson is a sufficient condition but not necessary. It has been significantly generalized in [47] for the case of two and three options within choices, and the new system of conditions is a generalization of the set of conditions in [44]. The general conditions are met much more frequently [47], making them more useful for both theoretical research and practical applications. This set of conditions and its generalization for more than three options are used in this paper during comparisons of option numbers from a theoretical perspective.

During this research, the applied number of options is usually two or four, but there are cases when further numbers of options are allowed: four options are used in [43] for evaluating women tennis players, five options are used in [48], six options are used in [38], and seven options are used in [49]. In these studies, the numbers of options were fixed before data collection, based on preliminary empirical observations. If we fix the number of options, then the data will be available to us in a corresponding structure. It seems natural to use a fixed number of options during an evaluation. But data can also be transformed. In the past, ties were transformed into half a win and half a loss. Which option number is the best choice for an evaluation? Is this method better than an evaluation using a three-option model? Is it possible that by using a different number of options and transforming the data, we could obtain a model that fits the data better? In which cases is this possible? What can be said about the choice of the number of options? After studying the literature, we did not find an answer to this question. We intend this paper to fill this gap. As a result, we develop an algorithm by which the appropriate number of options can be selected. This algorithm relies on both theoretical and experimental aspects. Theoretical aspects concern evaluability considerations, while experimental aspects concern more accurate estimations and are supported by a large number of computer simulations.

This paper is structured as follows: In Section 2 we introduce a general model allowing for s options with an MLE parameter estimation. Then, we turn to the theoretical aspects of evaluability. First we investigate the relationship between two- and four-option models, and then the general case of s and $s+2$ follows. The preliminary results for the case $s=3$ and $s+2=5$ are contained in the conference paper [50]. We can see that if data are evaluable in the s-option model and the data’s transformation satisfies a natural condition, then the data can also be evaluated in the $(s+2)$-option model. After these theoretical considerations, in Section 4 we present the results of a large number of computer simulations. The runs investigate how often the s-option model ($s=2,3$) and the $(s+2)$-option model ($s+2=4,5$) are evaluable. If both are evaluable, we investigate which one fits better with the data, and that is the one worth using for evaluations. We define a measurement for fitting which helps us decide on the choice of the number of options. In the subsequent section, we elaborate on and present an algorithm for performing the decision process, and the paper then concludes with our final remarks.

2. The Investigated Models

Let the objects to be rated and ranked be denoted by the numbers $1,2,\dots ,n$. The models with a stochastic background assume that the current performances of the objects are random variables denoted by ${\xi }_i$, $i=1,2,\dots ,n$. Let the expectations be E(${\xi }_i$) = $m_i$, which expresses the expected strengths of the object i. Decisions comparing two objects i and j are related to the difference between these latent random variables, i.e., ${\xi }_i-{\xi }_j$. We can separate the expectations as follows:

$${\xi }_i-{\xi }_j=m_i-m_j+{\eta }_{i,j}$$

(1)

where ${\eta }_{i,j}$ are independent identically distributed random variables with the cumulative distribution function F. F is supposed to be three times continuously differentiable, $0<F<1$, its probability density function is symmetric to zero, i.e., $f(-x)=f\left(x\right)$, $x\in \mathbb{R}$, and its logarithm is strictly concave. If ${\eta }_{i,j}$ are normally distributed, then we use the nomination of the Thurstone model, if their distribution is logistic, we use the Bradley–Terry model. These models were originally defined for two-option choices, ‘worse’ or ’better’, and then were generalized for three options: ‘worse’, ‘better’, or ‘equal’. In the case of the two-option model, the differences are compared to zero: the decision ‘better’/‘worse’ indicates whether the difference is positive/negative, respectively. In the three-option model, the value of the difference is compared to the parameter $0<d$; if it is in [$-d,d$], we can consider it a tie. More than three, (s), options can be allowed too, as follows: The options within choices are denoted by $C_k$, $k=1,2,\dots ,s$ ($s=5$: ‘much worse’, ‘slightly worse’, ‘equal’, ‘slightly better’, ‘much better’). The real line $\mathbb{R}$ is divided into disjoint sub-intervals $I_1$, $I_2$, …, $I_s$. $I_1=(-\infty ,d_1)$, $I_2=[d_1,d_2)$, …, $I_s=[d_{s-1},\infty )$. Decisions have a kind of symmetry, i.e., if i is ‘better’/‘much better’/…than j, then j is ‘worse’/‘much worse’/…than i. This means that $d_{s-1}=-d_1$ and, in general, $d_{s-i}=-d_i$, $i=1,2,\dots ,[s/2]$. The figure belonging to the model allowing s options can be seen in Figure 1.

If $s=2$, then no parameter is needed to assign the endpoints of the intervals. If $s=3,4$, then we need one positive parameter; we will denote it using $0<d=d_{s-1}$. If $s=5,6$, we need two parameters, denoted by $0<d=d_{s-2}<D=d_{s-1}$, to assign the intervals, etc. These parameters will be estimated on the basis of the comparison results, which are called data.

The data are included into a three-dimensional ($nxnxs$) data matrix $A^{\left(s\right)}$. Its elements $A_{i,j,k}^{\left(s\right)}$ are the number of comparisons in which decision $C_k$ is the result when we compare object i and j. Of course, $A_{i,i,k}^{\left(s\right)}=0$, $i=1,\dots ,n$, $k=1,\dots ,s$ and $A_{i,j,k}^{\left(s\right)}=A_{j,i,s+1-k}^{\left(s\right)}$, $i=1,\dots ,n$, $j=1,\dots ,n$, $k=1,\dots ,s$. The upper index ^(s) indicates the number of options.

The probability that the difference between the random variables belonging to the objects i and j is in the interval $I_k$ can be expressed as follows, in the case of $2<s$:

$$p_{i,j,1}=P({\xi }_i-{\xi }_j\in I_1)=F(d_1-(m_i-m_j))$$

(2)

$$p_{i,j,k}=P({\xi }_i-{\xi }_j\in I_k)=F(d_k-(m_i-m_j))-F(d_{k-1}-(m_i-m_j)),k=2,\dots ,s-1$$

(3)

and

$$p_{i,j,s}=P({\xi }_i-{\xi }_j\in I_s)=1-F(d_{s-1}-(m_i-m_j)).$$

(4)

If $s=2$, $d_1=0.$ In this case, instead of (2) and (4), the appropriate probabilities can be expressed by

$$p_{i,j,1}=P({\xi }_i-{\xi }_j\in I_1)=F(0-(m_i-m_j))$$

(5)

and

$$p_{i,j,2}=P({\xi }_i-{\xi }_j\in I_2)=1-F(0-(m_i-m_j)).$$

(6)

Assuming independent decisions, the probability of the data, i.e., the likelihood function, is

$$L\left(A^{\left(s\right)}\right|m_1,\dots ,m_n,d_1,\dots ,d_{\left[\right(s-1)/2]})=\prod _{k=1}^s\prod _{i=1}^{n-1}\prod _{j=i+1}^n{p}_{i,j,k}^{A_{i,j,k}^{\left(s\right)}}.$$

(7)

We estimate the parameters by maximizing the likelihood function (7), or, equivalently, its logarithm:

$$logL\left(A^{\left(s\right)}\right|m_1,m_2,\dots ,m_n,d_1,\dots ,d_{\left[\right(s-1)/2]})=\sum _{k=1}^s\sum _{i=1}^{n-1}\sum _{j=i+1}^n{A}_{i,j,k}^{\left(s\right)}·log\left(p_{i,j,k}\right).$$

(8)

The maximum likelihood estimation of the parameters $\underline{m}=(m_1,\dots ,m_n)$ and $\underline{dv}=(-d_1,\dots ,-d_{\left[\right(s-1)/2]})$ is the argument for the maximal value of (8); that is

$$({\widehat{\underline{m}}}^{\left(s\right)},{\underline{\widehat{dv}}}^{\left(s\right)})=\underset{\underline{m}\in {\mathbb{R}}^n,0<-d_{\left[{\displaystyle \frac{s-1}{2}}\right]}<\dots <-d_1}{argmax}L\left(A^{\left(s\right)}\right|\underline{m},\underline{dv}).$$

(9)

We note that the probabilities (2), (3), (4), (5), and (6) depend only on the differences between their expectations; therefore, one coordinate of the parameter vector $\underline{m}$, for example $m_1$, can be fixed at zero or the constraint ${\sum }_{i=1}^n{m}_i=0$ can be assumed. Naturally, the maximum value is not necessarily always attained, but some conditions for the data can guarantee the existence and uniqueness of a maximizer. These conditions are different in the case of different options. Hereinafter, data will be referred to as evaluable in an s-option model if the maximum likelihood estimate of the parameters (9) exists and is unique.

3. Comparison of Models Allowing Different Options in Choices from a Theoretical Aspect

3.1. The Case of s = 2 and s + 2 = 4

The early models of paired comparisons allowed for only two options: ‘worse’ or ‘better’ [34,39]. Thurstone himself argued that, based on experience, people are able to make decisions between these two options and there is not necessarily a need for more. In some sports, such as tennis or basketball, matches can only end in a win or a loss. However, knowing the result, we can sometimes add nuance: for example, the victory was decisive, or one side barely defeated the other. In other words, there may be justification for additional decision options, such as a narrow victory (‘slightly better’/‘slightly defeated’) or a decisive victory (‘much better’/‘much worse’). These models, which allow for and process more information, could yield more nuanced results.

The data’s structure is contained in the data matrices but the data matrices $A^{\left(4\right)}$ and $A^{\left(2\right)}$ can be transformed into each other. We follow the following natural transformation rules:

$$A_{i,j,1}^{\left(2\right)}=A_{i,j,1}^{\left(4\right)}+A_{i,j,2}^{\left(4\right)}$$

(10)

$$A_{i,j,2}^{\left(2\right)}=A_{i,j,3}^{\left(4\right)}+A_{i,j,4}^{\left(4\right)}$$

(11)

This means that if we want to transform the four-option model into a two-option model, we merge the options ‘slightly better’ and ‘much better’ into the ‘better’ option. In the opposite case, we split the option ‘better’ into two parts, and the numbers of decisions will be divided into two parts. We require that $0<A_{i_1,j_1,3}^{\left(4\right)}$ and $0<A_{i_2,j_2,4}^{\left(4\right)}$ for at least one pair of the indices $(i_1,j_1)$ and $(i_2,j_2)$, respectively. This requirement ensures that the data $A_{i,j,k}^{\left(4\right)}$ actually belong to the four-option model.

Now, we compare these models based on their evaluability. First, we track which theoretical cases are possible:

(A)

Data are evaluable only in the two-option model.

(B)

Data are evaluable only in the four-option model.

(C)

Data are evaluable in both models.

(D)

Data are not evaluable in either model.

(Case A) We can prove that this theoretical case may not happen. For this, we summarize the conditions for evaluability in the case of the two-option and the four-option models. For this, we need two graph definitions.

Definition 1

(The graph belonging to $A^{\left(2\right)}$). The nodes are the objects. There is a directed edge from i to j (denoted by →) if $0<A_{i,j,2}^{\left(2\right)}$, i.e., there is a decision according to which object i is ‘better’ than object j. This graph will be denoted by $G{C}^{\left(A^{\left(2\right)}\right)}$.

Definition 2

(The graph belonging to $A^{\left(4\right)}$). The nodes are the objects. Two types of directed edges are contained in the graph. There is a directed edge from i to j (denoted by → and called the ‘slightly better’ edge) if $0<A_{i,j,3}^{\left(4\right)}$, i.e., there is a decision according to which object i is ‘slightly better’ than object j. The other type of directed edge is defined as follows: there is a double directed edge from i to j, denoted by ↠ and called the ‘much better edge’, if $0<A_{i,j,4}^{\left(4\right)}$. In other words, there exists a decision according to which object i is ‘much better’ than object j. This graph will be denoted by $G{C}^{\left(A^{\left(4\right)}\right)}$.

In the case of the two-option model, the necessary and sufficient condition for evaluability, given by Ford, for logistic distributions and generalized for strictly log-concave distributions in [47] is the following:

• C.2.1. For every nonempty partition of S and $\overline{S}$ of the set {1, 2, …, n} ($S\cup \overline{S}=\{1,2,\dots ,n\}$, $S\cap \overline{S}=\varnothing$), there are two (not necessarily different) index pairs $(i_1,j_1)$ and $(i_2,j_2)$, $i_1,i_2\in S$, $j_1,j_2\in \overline{S}$, for which $0<A_{i_1,j_1,1}^{\left(2\right)}$ and $0<A_{i_2,j_2,2}^{\left(2\right)}$. In other words, these subsets are connected in both directions.

These types of graphs are known as strongly connected graphs in graph theory. This property is equivalent to the existence of a directed path between any two nodes.

In the case of the four-option model, the following set of conditions has been proven sufficient for evaluability:

• C.4.1. Both types of edges are included in the graph $G{C}^{\left(A^{\left(4\right)}\right)}$.
• C.4.2. For every nonempty partition of S and $\overline{S}$ ($S\cup \overline{S}=\{1,2,\dots ,n\}$, $S\cap \overline{S}=\varnothing$), there are two (not necessarily different) index pairs $(i_1,j_1)$ and $(i_2,j_2)$, $i_1,i_2\in S$, $j_1,j_2\in \overline{S}$, for which

  $$0<A_{i_1,j_1,4}^{\left(4\right)}\mathrm{and}0<A_{i_2,j_2,1}^{\left(4\right)}$$

  (12)

  are satisfied or there exists such an index pair $(i_3,j_3)$ for which

  $$0<A_{i_3,j_3,2}^{\left(4\right)}\mathrm{or}0<A_{i_3,j_3,3}^{\left(4\right)}$$

  (13)

  is satisfied.
• C.4.3. There exists a cycle in the graph $G{C}^{\left(A^{\left(4\right)}\right)}$ along the ‘slightly better’ and ‘much better’ edges which contains at least one ‘much better’ edge. In other words, there exists a cycle ($i_1,i_2,\dots ,i_k,i_1$), $i_l,l=1,2,\dots ,k$ for some $k=2,\dots ,n$, where $i_1,i_2,\dots ,i_k$ are distinct, for which $0<A_{i_l,i_{l+1},3}^{\left(4\right)}$ or $0<A_{i_l,i_{l+1},4}^{\left(4\right)}$, and there exists at least one index pair in the cycle for which $0<A_{i_l,i_{l+1},4}^{\left(4\right)}$. We note that this cycle might contain only two different nodes.

Using the above set of conditions for evaluability, C.2.1., as well as C.4.1., C.4.2., and C.4.3., we can prove the following theorem:

Theorem 1.

Suppose that there exists at least one index pair $(i_1,j_1)$ for which $0<A_{i_1,j_1,3}^{\left(4\right)}$ is satisfied and there exists an index pair $(i_2,j_2)$ for which $0<A_{i_2,j_2,4}^{\left(4\right)}$ holds. Redefine the data in the two-option model using (10) and (11). If condition C.2.1.is satisfied, then $A^{\left(4\right)}$ is evaluable in the four-option model.

Proof. 

We check the conditions C.4.1., C.4.2., and C.4.3.

C.4.1. is already required in the assumptions.

C.4.2. Consider an arbitrary partition S, $\overline{S}$. As there exists an index pair $(i_1,j_1)$$i_1\in S$$j_1\in \overline{S}$ for which $0<A_{i_1,j_1,2}^{\left(2\right)}$, there exists an index pair for which $0<A_{i_1,j_1,3}^{\left(4\right)}$ or $0<A_{i_1,j_1,4}^{\left(4\right)}$. Similarly, as there exists an index pair $(i_2,j_2)$$i_2\in S$$j_2\in \overline{S}$ for which $0<A_{i_2,j_2,1}^{\left(2\right)}$, there exists an index pair for which $0<A_{i_2,j_2,2}^{\left(4\right)}$ or $0<A_{i_2,j_2,1}^{\left(4\right)}$. These mean that (12) or (13) is satisfied.

C.4.3. Let us choose a node from which a ‘much better’ edge starts ($i_1$). It goes to the object $i_2$. C.2.1. implies that there is a path along the directed edges from $i_2$ to $i_1$ in $G{C}^{\left(A^{\left(2\right)}\right)}$. We denote it by saying that $(i_2,i_3,\dots ,i_1)$ and $i_j$ are different. These edges are also included as ‘slightly better’ or ‘much better’ edges in $G{C}^{\left(A^{\left(4\right)}\right)}$. Therefore, the cycle $(i_1,i_2,\dots ,i_1)$ forms a cycle along the ‘slightly better’ and ‘much better’ edges in $G{C}^{\left(A^{\left(4\right)}\right)}$ and it contains at least one ‘much better’ edge.

We proved that all three conditions, C.4.1., C.4.2., and C.4.3., hold; consequently, data matrix $A^{\left(4\right)}$ is evaluable.    □

(Case B) This case may occur. A simple example can be seen in Figure 2. The justification is as follows:

Data are evaluable in the four-option model as

C.4.1. is obviously satisfied; $A_{1,2,4}^{\left(4\right)}=1$ and $A_{2,3,2}^{\left(4\right)}=1$.

C.4.2. In the case of every partition, the subsets are connected either by ‘much better’ edges, both forward and backward, or a ‘slightly better’ edge. More exactly, if $S=\left\{1\right\}$, $\overline{S}=\{2,3\}$, or $S=\{1,3\}$, $\overline{S}=\left\{2\right\}$, then $A_{1,2,4}^{\left(4\right)}=1$, $A_{2,1,4}^{\left(4\right)}=A_{1,2,1}^{\left(4\right)}=1$. If $S=\{1,2\}$ and $\overline{S}=\left\{3\right\}$, then $A_{3,2,3}^{\left(4\right)}=A_{2,3,1}^{\left(4\right)}=1$. When S and $\overline{S}$ are changed, the situation remains the same.

C.4.3. The required cycle is (1,2,1).

Data cannot be evaluated using the two-option model as C.2.1. does not hold. See $S=\{1,2\}$ and $\overline{S}=\left\{3\right\}$; moreover, take into the consideration that $A_{1,2,1}^{\left(2\right)}=A_{1,2,2}^{\left(2\right)}=2$, $A_{2,3,1}^{\left(2\right)}=A_{3,2,2}^{\left(2\right)}=1$, $A_{2,3,2}^{\left(2\right)}=A_{3,2,1}^{\left(2\right)}=0$, $A_{1,3,1}^{\left(2\right)}=A_{3,1,2}^{\left(2\right)}=A_{1,3,2}^{\left(2\right)}=A_{3,1,1}^{\left(2\right)}=0$.

In this case, the situation is clear; there is no choice.

(Case C) The data given by $A^{\left(4\right)}$ and its transformed version $A^{\left(2\right)}$ can be evaluated in the four- and two-option model, respectively; see Figure 3. This is a very common case. This simple example is a modification of the data presented in Figure 2. The comparison results in Figure 2 are supplemented by a ‘slightly better’ edge from object 2 to object 3; see Figure 3. The data are obviously evaluable in the four-option model. Moreover, C.2.1. is also satisfied; hence, data can be evaluated in the two-option model too.

In this case, we can decide which model will be used for the data evaluation. In Section 4, we investigate these cases’ practical aspects using computer simulations.

(Case D) Data cannot be evaluated in either the two-option or the four-option model.

Let $A_{1,2,1}^{\left(4\right)}=$$A_{1,2,2}^{\left(4\right)}=$$A_{1,2,3}^{\left(4\right)}=$$A_{1,2,4}^{\left(4\right)}=1$, $A_{2,3,1}^{\left(4\right)}=A_{2,3,2}^{\left(4\right)}=A_{2,3,3}^{\left(4\right)}=0$, $A_{2,3,4}^{\left(4\right)}=1$, $A_{1,3,k}^{\left(4\right)}=0$, and $k=1,2,3,4$. The graph of the results belonging to the data matrix $A^{\left(4\right)}$ can be seen in Figure 4.

The reason why the data are not evaluable is clear: object 3 is ‘worse’ than objects 2 and 1, but we do not know by how much they differ. Formally, conditions C.2.1. and C.4.2. do not hold: they take the partition $S=\{1,2\}$ and $\overline{S}=\left\{3\right\}$ in both cases.

3.2. Comparison of Models Allowing Different Options within Choices: The General Case

The comparison of three-option and five-option cases is included in the conference paper [50]; therefore, we do not provide details here. Instead, in this section, we turn to the general case, which includes the analysis in Section 3.1 and the analysis of the cases s = 3 and $s+2$ = 5.

Data will be denoted by $A_{i,j,k}^{\left(s\right)}$, ($k=1,2,\dots ,s$) and $A_{i,j,k}^{(s+2)}$, ($k=1,2,\dots ,s+2$) in the s and (s + 2)-option models, respectively. Data conversion is performed using the following formulas:

$$A_{i,j,1}^{\left(s\right)}=A_{i,j,1}^{(s+2)}+A_{i,j,2}^{(s+2)}$$

(14)

$$A_{i,j,s}^{\left(s\right)}=A_{i,j,s+1}^{(s+2)}+A_{i,j,s+2}^{(s+2)}$$

(15)

and

$$A_{i,j,k}^{\left(s\right)}=A_{i,j,k+1}^{(s+2)},k=2,3,\dots ,s-1$$

(16)

This means that the two outermost decisions are merged and the ‘internal’ decisions remain unchanged. The appropriate graph definition is the following:

Definition 3

(The graph belonging to $A^{\left(s^{*}\right)}$). The nodes are the objects. Next to the edges belonging to ties are two types of directed edge: The first one is a ‘simple’ directed edge from i to j (denoted by → and called the ‘better to some extent’ edge) that exists if $0<A_{i,j,k}^{\left(s^{*}\right)}$$k=\left[{\displaystyle \frac{s^{*}+1}{2}}\right]+1,\dots ,s^{*}-1$; i.e., there is a decision according to which object i is ‘better to some extent’ than object j. The other is an ‘extremely better’ directed edge: it exists from i towards j if $0<A_{i,j,s^{*}}^{\left(s^{*}\right)}$ and it is denoted by ↠. Expressed in words, there is an extreme decision according to which i exceeds j. This graph will be denoted by $G{C}^{\left(A^{\left(s^{*}\right)}\right)}$. We substitute $s^{*}=s$ and $s^{*}=s+2$.

The set of conditions that guarantees data evaluability in the model allowing $s^{*}$ options is as follows:

• C.s*.1. For every value of $k=1,2,\dots ,s^{*}$ there exists an index pair $i_k,j_k$ for which $0<A_{i_k,j_k,k}^{\left(s^{*}\right)}$.
  We note that this condition is necessary. If it does not hold, the number of options has to be reduced.
• C.s*.2. For every nonempty partition of S and $\overline{S}$ ($S\cup \overline{S}=\{1,2,\dots ,n\}$, $S\cap \overline{S}=\varnothing$), there are two (not necessarily different) index pairs $(i_1,j_1)$ and $(i_2,j_2)$$i_1,i_2\in S$, $j_1,j_2\in \overline{S}$ for which

  $$0<A_{i_1,j_1,1}^{\left(s^{*}\right)}\mathrm{and}0<A_{i_2,j_2,s^{*}}^{\left(s^{*}\right)}$$

  (17)

  are satisfied or there exists such an index pair $(i_3,j_3)$ for which

  $$0<A_{i_3,j_3,k}^{\left(s^{*}\right)},\mathrm{for}\mathrm{some}k=2,3,\dots ,s^{*}-1$$

  (18)

  is satisfied. Expressed in words, the S and $\overline{S}$ are connected either by a non-extreme decision or by two extreme decisions made back and forth.
  We note that this condition is also a necessary condition, not just sufficient.
• C.s*.3. There exists a cycle in the graph $G{C}^{\left(A^{\left(s^{*}\right)}\right)}$ along the ‘better to some extent’ and ‘extremely better’ edges which contains at least one ‘extremely better’ edge. In other words, there exists a cycle ($i_1,i_2,\dots ,i_h,i_1$) for which $i_1,i_2,\dots ,i_h$ are different, and $0<A_{i_l,i_{l+1},k_l}^{\left(s^{*}\right)}$, where $k_l\in \{\left[{\displaystyle \frac{s^{*}+1}{2}}\right]+1,\dots ,s^{*}\}$, and there exists at least one index pair in the cycle for which $k_l=s^{*}$. We note that it may be the case that this cycle contains only two different nodes.

As a generalization of Theorem 4 in [47], one can prove that conditions C.s*.1., C.s*.2., and C.s*.3. guarantee the evaluability of the data matrix $A^{\left(s^{*}\right)}$ in the $s^{*}$-option model. We will apply this statement for $s^{*}$ = s and $s^{*}$ = s + 2.

Comparison Based on Evaluability

(A)

Data are evaluable only in the s-option model.

(B)

Data are evaluable only in the (s+2)-option model.

(C)

Data are evaluable in both models.

(D)

Data are not evaluable in either model.

(Case A) This case cannot occur. We can state the following Theorem:

Theorem 2.

Suppose that there exists at least one index pair $(i_1,j_1)$ for which $0<A_{i_1,j_1,s+1}^{(s+2)}$ is satisfied and there exists an index pair $(i_2,j_2)$ for which $0<A_{i_2,j_2,s+2}^{(s+2)}$ holds. Redefine the data in the s-option model using (14), (15), and (16). If conditions C.s.1., C.s.2, and C.s.3. are satisfied, then $A^{(s+2)}$ is evaluable in the $(s+2)$-option model.

The proof of this can be easily carried out by following the steps of $s=2$ cases step by step in Section 3.1.

(Case B) This case can happen; for example, take the following data matrix $A^{(s+2)}$, which is a modification of the comparison results presented in Figure 2: $A_{1,2,k}^{(s+2)}=A_{2,1,k}^{(s+2)}=1$, $k=1,2,\dots ,s+2$, $A_{2,3,2}^{(s+2)}=A_{3,2,s+1}^{(s+2)}=1$, and the other values are zero. This implies that $0<A_{1,2,k}^{\left(s\right)}$, $k=1,2,\dots ,s$ and $A_{2,3,1}^{\left(s\right)}=A_{3,2,s}^{\left(s\right)}=1$. Similarly to the case s = 2, in the general s-option model, object 3 exceeds object 2 but we do not know by how much. By increasing its expectation, its likelihood function increases. C.s.2. does not hold if $S=\{1,2\}$ and $\overline{S}=\left\{3\right\}$.

In this case, it is clear which model should be used for the evaluation; there is no choice between them.

(Case C) Similar to the case where s = 2, it may be the case that the data can be evaluated using both numbers of options. As an example, consider the previous data matrix A, and supplement it with an ‘extremely better’ edge between the objects 2 and 3. That is, let $A_{1,2,k}^{(s+2)}=A_{2,1,k}^{(s+2)}=1$, $k=1,2,\dots ,s+2$, $A_{2,3,2}^{(s+2)}=A_{3,2,s+1}^{(s+2)}=A_{2,3,s+2}^{(s+2)}=A_{3,2,1}^{(s+2)}=1$, and the other values be zero. One can easily check that C.s.1, C.s.2., C.s,3., C.s + 2.1., C.s + 2.2., and C.s + 2.3. are all satisfied. Consequently, the data can be evaluated by both models.

In this case, we can choose which model to use for evaluating the data.

(Case D) The data cannot be evaluated using either the s-option or the $(s+2)$-option model.

An easily understandable example is a modification of the data presented in Figure 4. Let $A_{1,2,k}^{(s+2)}=A_{2,1,k}^{(s+2)}=1$, $k=1,2,\dots ,s+2$, and $A_{2,3,s+2}^{(s+2)}=1$, while the other values are zero. Now, $0<A_{1,2,k}^{\left(s\right)}=A_{2,1,s+1-k}^{\left(s\right)}$$k=1,2,\dots ,s$ and $A_{2,3,s}^{\left(s\right)}=A_{3,2,1}^{\left(s\right)}=1$, while the other values are zero. In this case, neither C.s.2. nor C.s + 2.2. holds, as $S=\{1,2\}$ and $\overline{S}=\left\{3\right\}$.

In this case, further data have to be collected; that is, new comparisons have to be made.

4. Simulation Results

In this chapter, we deal with the two most commonly used option numbers: s = 2 and s = 3. First, we present the relationship between the two- and four-option models, and then we turn to the relationship between the three- and five-option models. We analyze these relationships by applying computer simulations. We point out when it is more appropriate to use which model in order to generate the best possible retrieval of the strength of the objects.

In these cases, we predefined the strength (expectation) of each object within certain bounds and the separation parameters for each object, including $0<d$ for the three-, four- and five-option models and $d<D$ for the five-option model. We then generated comparison results for different comparison numbers using these fixed strengths ($m_i$) and values d, D. We checked whether the data set could be evaluated using the s- and $(s+2)$-option models. If it was possible, we evaluated the data sets using both the s- and $(s+2)$-option models and compared the estimated strengths obtained by the models with the original strengths $m_i,i=1,2,\dots ,n$. We used a large number of simulations (${10}^5$) in our work, applying logistic distribution. The simulation program was implemented in C#.

4.1. The Cases s = 2 and s + 2 = 4

The two-option and four-option models are compared in this subsection.

Table 1. Comparison of the simulation results of the two- and four-option models using the parameters $m_i$: $0-2,d=1.35$.

$\left|\mathit{A}\right|$	2 and 4	Only 4	Neither	$\overline{{\mathit{e}}^{\left(2\right)}}$	$\overline{{\mathit{e}}^{\left(4\right)}}$	${\mathit{R}}^{\left(2\right),\left(4\right)}$
10	23	4006	95,971	3.887	6.059	−0.359
11	85	8611	91,304	4.566	6.405	−0.287
12	279	14,819	84,902	4.880	6.823	−0.285
13	708	21,872	77,420	5.168	7.231	−0.285
14	1473	29,086	69,441	5.313	6.917	−0.232
15	2646	35,962	61,392	5.421	6.736	−0.195
16	4195	42,121	53,684	5.576	6.655	−0.162
17	6401	47,080	46,519	5.709	6.465	−0.117
18	8981	50,959	40,060	5.839	6.332	−0.078
19	11,999	53,683	34,318	5.924	6.179	−0.041
20	15,499	55,244	29,257	5.946	5.970	−0.004
21	19,284	55,878	24,838	5.971	5.811	0.027
22	23,220	55,720	21,060	5.939	5.643	0.052
23	27,247	54,962	17,791	5.921	5.456	0.085
24	31,453	53,502	15,045	5.882	5.284	0.113
25	35,552	51,612	12,836	5.808	5.090	0.141
30	54,845	39,613	5542	5.424	4.296	0.263
40	79,977	18,949	1074	4.441	3.151	0.410
50	91,265	8514	221	3.569	2.417	0.477
60	96,142	3797	61	2.898	1.929	0.503
70	98,224	1763	13	2.388	1.590	0.502
80	99,167	831	2	2.011	1.343	0.497
90	99,587	412	1	1.729	1.163	0.487
100	99,781	219	0	1.510	1.023	0.476
200	99,999	1	0	0.650	0.460	0.412
300	100,000	0	0	0.414	0.297	0.394
400	100,000	0	0	0.304	0.219	0.387
500	100,000	0	0	0.240	0.174	0.382
600	100,000	0	0	0.199	0.144	0.381
700	100,000	0	0	0.169	0.123	0.378
800	100,000	0	0	0.148	0.107	0.376
900	100,000	0	0	0.131	0.095	0.374
1000	100,000	0	0	0.117	0.085	0.373

shows an example for the simulation results of the two models. We present a case in Table 1 where the probabilities of an object falling into the options available are approximately equal. In the presented situation, there are eight objects to compare, their expectations are between 0 and 2, and $d=1.35$. In the case of the four-option model, the ‘slightly better’ and ‘much better’ options can be distinguished: a ‘slightly worse’ or ‘much worse’ decision is a ‘slightly better’ or ‘much better’ decision from the perspective of the other object. In this case, the ratio of ‘much better’ decisions to all decisions was 0.5186, while the ratio of ‘slightly better’ was 0.4814.

The first column in Table 1 shows the number of comparisons denoted by $\left|A\right|$. The second column shows the number of cases in which we were able to use both models. The third column shows the number of cases for which we were only able to evaluate objects using the four-option model and the fourth column shows the number of cases for which we were not able to make an evaluation using either model. The fifth and sixth columns contain the average sum of the squared differences between the exact and the estimated expectations found by the two-option model and the four-option model, respectively. The quantities

$$e^{(s*)}=\sum _{i=1}^n{({\widehat{m}}_i^{(s*)}-m_i)}^2$$

(19)

measure how close the estimated expected values are to the exact expected values in the model allowing $s*$ options. They are zero if, and only if, the estimated parameters coincide with the exact parameters. If we take the average of these values, we get a measure of the model’s accuracy, denoted by $\overline{e^{(s*)}}$.

The seventh (last) column shows their relative relation, which is defined as follows:

$$R^{\left(2\right),\left(4\right)}={\displaystyle \frac{\overline{e^{\left(2\right)}}-\overline{e^{\left(4\right)}}}{\overline{e^{\left(4\right)}}}}.$$

(20)

In these formulas, as previously, the upper index refers to the option number of the model used for the evaluation. As the quantities (19) are non-negative, the sign of (20) characterizes which model is better: if (20) is positive, then the four-option model is better; if it is negative, then the two-option model is. It can clearly be seen that in the case of few comparisons—i.e., when the number of comparisons conducted is less than 21—the two-option model is able to approximate the exact expectations more accurately. Meanwhile, in the case of more comparisons, the more complex four-option model performs better. It can also clearly be seen that as more comparisons are included, we can make better evaluations with both models and the average errors $\overline{e^{\left(2\right)}}$ and $\overline{e^{\left(4\right)}}$ become smaller and smaller. Due to space limitations, we have only presented the case $m_i$: $0-2,d=1.35$, but in other cases the results are similar: if the number of comparisons reaches 21, the more complex four-option model performs better than for 8 compared objects. We conducted simulations with different numbers of objects to be evaluated and we observed a linear relationship between the number of objects evaluated and the number of comparisons when the model with more options performs better, on average. The critical number is the twice of the number of the objects plus 5 in the case of s = 2. This means that if $2·n+5\le \left|A\right|$, then the complicated model is advisable.

This phenomenon can be explained as follows: if there are few comparisons, randomness plays a significant role. The more options there are, the greater the variability. As the number of comparisons increases, the generated data increasingly follow the probabilities derived by the model. The parameters become easier to estimate accurately. Models with more options contain more information, allowing for more precise estimates.

4.2. The Cases s = 3 and s + 2 = 5

Now we compare the three-option and five-option models. The reason for choosing $s=3$ as the general case is that it is often used, and particularly in sports applications. In numerous sports, the possible outcomes of matches are a draw, a victory, or a defeat. Examples of such sports include football and handball. However, if we want to evaluate the results using a three-option model, we cannot do so. One possible reason for this could be that one team defeats every other team. This happened in the 2024 UEFA European Championship: Spain won all of its matches. If we evaluate these results using the five-option model, the evaluation can be carried out. Many other examples can be given of this phenomenon. These also highlight the importance of examining the problem carefully.

We conducted computer simulations. In the presented example, the number of compared objects was eight. The expectations were chosen from the interval $[0,2]$ and d and D were fixed: $0<d<D$, $d=0.1$, $D=1.35$. In this case, we wanted to show an extreme scenario in which there are almost no ‘equal’ decisions, while the number of ‘slightly better’ and ‘much better’ decisions is approximately equal. The ratios of the decisions ‘equal’, ‘slightly better’, and ‘much better’ are 0.0540/0.4768/0.4692, respectively. The simulation results are shown in

Table 2. Comparison of the simulation results of the three- and five-option models using the parameters $m_i$: $0-2,d=0.1,D=1.35$.

$\left|\mathit{A}\right|$	3 and 5	Only 5	Neither	$\overline{{\mathit{e}}^{\left(3\right)}}$	$\overline{{\mathit{e}}^{\left(5\right)}}$	${\mathit{R}}^{\left(3\right),\left(5\right)}$
10	39	2283	97,678	5.715	7.765	−0.264
11	168	4899	94,933	6.203	9.178	−0.324
12	454	8313	91,233	6.508	8.666	−0.249
13	936	12,437	86,627	6.838	8.495	−0.195
14	1788	16,617	81,595	6.870	8.342	−0.177
15	3041	20,513	76,446	6.959	8.112	−0.142
16	4738	23,837	71,425	7.091	7.820	−0.093
17	6766	26,470	66,764	7.074	7.563	−0.065
18	9171	28,704	62,125	7.045	7.260	−0.030
19	11,901	30,146	57,953	6.913	6.949	−0.005
20	14,931	30,912	54,157	6.847	6.706	0.021
30	46,295	21,860	31,845	5.583	4.466	0.250
40	66,621	10,441	22,938	4.310	3.181	0.355
50	77,156	4594	18,250	3.370	2.404	0.402
60	82,770	1964	15,266	2.688	1.896	0.417
70	86,060	887	13,053	2.206	1.559	0.415
80	88,263	412	11,325	1.857	1.316	0.411
90	89,784	191	10,025	1.593	1.137	0.401
100	90,892	92	9016	1.391	1.001	0.390
200	95,497	0	4503	0.607	0.452	0.341
300	96,976	0	3024	0.387	0.292	0.327
400	97,762	0	2238	0.285	0.215	0.323
500	98,232	0	1768	0.225	0.171	0.319
600	98,530	0	1470	0.186	0.142	0.317
700	98,714	0	1286	0.159	0.121	0.315
800	98,867	0	1133	0.138	0.105	0.314
900	98,991	0	1009	0.122	0.093	0.314
1000	99,105	0	895	0.110	0.084	0.314

.

The structure of Table 2 follows the structure of Table 1; the columns’ names and reports change according to the option numbers of the models. Here, as in the two-option model, the accuracy of estimating expected values is better for the simpler model if the numbers of comparisons are small, while, for larger numbers of comparisons, the more complex model performs better. As more and more comparisons are made between random objects, the errors become smaller and smaller. Based on lots of further experiments, we can ascertain that, in the case of 8 objects and at least 20 comparisons, the five-option model estimates the expectations $m_i(i=1,2,\dots ,n$) better, on average; therefore, it is recommended.

Of course, the number of comparisons depends on the number of objects to be evaluated when the model that allows more options is able to use its additional information. By conducting a large number of simulations with different object numbers, we can conclude that five-option model performs better than the three-option model if $2·n+4\le \left|A\right|$.

5. Algorithm for Model Choice

In this section, based on the results of the previous sections, we present an algorithm that allows us to select which model to use; see Algorithm 1.

The function $Ev(A,om)$ specifies, for a given A (data set), whether A can be evaluated with an $om$-option model. As previously, we denote the number of comparisons as $\left|A\right|$. The number of ‘equal’ choices is denoted by $A_{equal}$. The auxiliary variable k that we use takes only positive integer values. The function $Calc(A,om)$ evaluates the data set A in an $om$-option model. $Mdc\left(\right)$ represents additional data collection. $oc$ denotes the number of objects in the comparisons, while $Cm\left(om\right)$ is the constant multiplier in each case where $om=s+2$. For  $s+2=4$ and $s+2=5$, the $Cm\left(om\right)$ is 2. The $od$ is a constant value for each model; it equals 5 for $s+2=4$ and it equals 4 for $s+2=5$.

We present the algorithm described below in the flowchart in Figure 5.

Algorithm 1 Option Number Selection

1: if  $s_2=2k+1$  and  $A_{equal}=0$  then 2:     $s_2:=s_2-1$ 3: end if 4: while not $Ev(A,s_2)$ and  $s_2>3$do 5:     $s_2:=s_2-2$ 6: end while 7: if $s_2>3$ then 8:     $s:=s_2-2$ 9:     if $Ev(A,s)$ then 10:         if $\left|A\right|\ge Cm\left(s_2\right)·oc+od$ then 11:            $Calc(A,s_2)$ 12:         else 13:            $Calc(A,s)$ 14:         end if 15:     else 16:         $Calc(A,s_2)$ 17:     end if 18: else 19:     if $Ev(A,s_2)$ then 20:         $Calc(A,s_2)$ 21:     else 22:         $Mdc\left(\right)$ 23:     end if 24: end if

6. Conclusions

The selection of the number of allowed options was studied to improve evaluability and provide a better approximation of the strength of the objects compared. An algorithm was created to decide which option number is best to use for data evaluations. The algorithm includes theoretical results for evaluability and empirical observations based on simulations. Based on theoretical considerations, we can assert that if the data can be evaluated in an s-option model, then it can also be evaluated in an $(s+2)$-option model, provided that there are comparison results for every option. However, the statement is not true in the reverse scenario: there are situations where only the model with a higher number of options is suitable for the evaluation. If both models are suitable for the evaluation, we have provided a metric to determine which model allows for a more accurate estimation of the objects’ strengths. We can conclude that, in the case of a small number of comparisons, the simpler model is more suitable. However, beyond a certain number of comparisons, the model that allows for more options is better, as it can utilize its additional information. This critical number is approximately twice the number of objects to be evaluated plus five for the $s+2$ = 4-option model and four for the $s+2$ = 5-option model. Further investigations are needed for cases where $s+2$ = 6, 7, and beyond.