Combining Different Stakeholders’ Opinions in Multi-Criteria Decision Analyses Applying Partial Order Methodology

Abstract

Multi-criteria decision analyses (MCDA) for prioritizations may be performed applying a variety of available software, e.g., methods such as Analytic Network Process (ANP) and Elimination Et Choice Translating Reality (ELECTRE III) as recently suggested by Kalifa et al. In addition to a data matrix, usually based on indicators and designed for describing the parts of the framework intended for the MCDA, these methods require input of a variety of other parameters that are not necessarily immediately obtainable. Often the indicators are simply combined by a weighted sum to obtain a ranking score, which is supposed to reflect the opinion of a multitude of stakeholders. A single ranking score facilitates the decision as a unique ordering is obtained; however, such a ranking score masks potential conflicts that are expressed by the values of the single indicators. Beyond hiding the inherent conflicts, the problem arises that the weights, needed for summing up the indicator values are difficult to obtain or are even controversially discussed. Here we show a procedure, which takes care of potential different weighting schemes but nevertheless does not mask any inherent conflicts. Two examples are given, one with a small (traffic) system and one with a pretty large data matrix (food sustainability). The results show how decisions can be facilitated even taking a multitude of stakeholder opinions into account although conflicts are not necessarily completely eliminated as demonstrated in the second case.

1. Introduction

1.1. Ranking

Ranking is a means to support decisions. The advantage of a ranking approach is that even if a best solution is found but not realizable, ranking provides other acceptable solutions. PROMETHEE [1,2,3], AHP [4], ANP [5], ELECTRE-family [6], and others are methods to obtain a ranking based on a data matrix and additional supporting information, such as weights, associated with indicators (ELECTRE, PROMETHEE). On one side, these methods are highly sophisticated; however, for a public understanding, they are rarely understandable. On the other side, partial order can explore the data matrix without complicated algebraic/arithmetic transformations but will not in all cases allow a unique decision, because the values of the single indicators lead to many conflicts which may be important for their own right, but clearly hampering a decision.

In the literature on application of partial ordering in decision making, many attempts can be found where—while keeping the framework of partial order—the degree of comparability (in a ranking the degree of comparability gets its maximum) is enhanced. Here, we present a procedure called General Linear Aggregation (GLA) [7] which not only enhances the degree of comparability, but also simultaneously covers the fact that different stakeholders may have different weightings in mind. The general idea with GLA is not to let the stakeholders try to harmonize their different opinions but to allow their own ones in the analysis. With GLA, the toolbox of partial order enrichments gets a further tool, which may be applied in selected cases. Some of these tools—including GLA—will be mentioned in subsequent sections of this paper.

1.2. Model Studies

The GLA is elucidated by two examples. (i) In a recent paper [8], two multi-criteria decision methods (MCDA), i.e., Analytic Network Process (ANP) and Elimination Et Choice Translating Reality (ELECTRE III), were applied to study and prioritize four paratransit transportation variants, BB: BodaBoda, Coa: Coaster, Kam: Kamunya, and TT: TukTuk, in Kampala, Uganda. The analyses were based on a total of 15 indicators covering B: benefits (8 indicators), C: costs (3 indicators), and R: risks (4 indicators) (cf. [8] Table 4). For the actual analyses, the indicators were aggregated into 3 main indicators (B, C, R) applying weight for the single indicators (cf. [8], Table 3). In both cases (ANP and ELECTRE) identical prioritizations were obtained, i.e., Coa > Kam > TT > BB, respectively. What is the influence of the weights, when simply weighted sums are applied for an aggregation and how to extend partial order methods to cope with cases such as here, whereby far more indicators than objects (here transportation variants) are applied? (ii) In the second example, a larger and thus more complex data matrix is investigated. Hence, the diet composition for 78 countries is to be compared with respect to food sustainability [9], which is not immediately quantifiable, but described by four indicators. In this study, four weighting schemes are adopted and GLA is applied to visualize what the combined effect of these four weighting schemes is.

It should be noted that the inclusion of the first example is important to demonstrate that the here-presented leads to an identical ranking.

1.3. Semantics

The advantages of the partial order-based approach may lose some value when the number of incomparabilities is increasing. To be clear: partial order may or even will disclose incomparabilities resulting from a multi-indicator system, which indicates the presence of conflicts within the set of indicators (details below).

It is appropriate to clarify the semantics of “importance of indicators” and “weighting schemes”. Within partial ordering, the concept “importance” has unfortunately often been associated with exchangeability of indicators expressing their low mutual importance. In connection with “weighting schemes” concerning the indicators importance is simply to stress the contextual importance of the single indicators.

2. Materials and Methods

2.1. Partial Ordering—Basics

The basics of partial ordering is the relation between objects (in our paper, either the four single transport systems or the 78 countries) to be ordered. The set of objects is called X. A priori, the data are analyzed without any pretreatments such as, e.g., aggregation of the single indicators. The only mathematical term applied in this context is “≤” (cf. e.g., [10,11]). By this, a relational instead of a numerical point of view is taken (cf. discussion). Two objects, x and y, are connected with each other if and only if the relation x ≤ y holds (see below, Equation (1)).

2.1.1. Main Equation, Compensation, Binary Relations

Object, x, characterized by the a set of indicators rs(x), s = 1, ..., m, can be compared with object y, characterized by the same set of indicators rs(y), if

rs(x) ≤ rs(y) for all s = 1, …, m

(1)

Application of Equation (1) needs a convention about the orientation of the single indicators, i.e., the larger the value of an indicator, the better a non-measurable quantity (a “latent construct”) by indicators being mutually co-monotone. Equation (1) is the basis for a comparison of objects. As the indicator values are not numerically combined, the problem of compensation (a “good” value of an indicator may compensate a “bad” one of another indicator) is avoided, which is one of the main advantages of partial ordering [12,13]. By several steps, the binary relations due to Equation (1) are prepared for a representation by a graph, the Hasse diagram [10,14]. Independent of a graphical representation, the result of the application of Equation (1) is a partially order set (poset) of the objects.

2.1.2. Hasse Diagram, Incomparabilities, Chains

In the Hasse diagram, comparable objects are connected by a sequence of lines [10,14]. If Equation (1) is not fulfilled for some objects x, y, then x is incomparable with y, denoted by x‖y. Generally, incomparability expresses that the data lead to conflicts between the objects. A subset X’ of X where for every x, y a ≤ - relation can be found is a chain, the number of objects constituting a chain is denoted its length. A subset X’’ of X, where for every x, y ∈ X’’ x‖y, is called an antichain. The number of incomparabilities, U, is an important characterizing quantity.

If for X a chain emerges, then U = 0 and the typical ranking (a linear order) is obtained.

2.1.3. Extension/Enrichment

An extension/enrichment of a partial order is transforming incomparabilities of the original poset into comparabilities (within the new poset) by maintaining the already given comparabilities.

2.2. Framework of Enrichments

As mentioned in Section 1, there are many attempts to enhance the degree of comparability without leaving the principles of partial order. As a detailed description is by far outside the scope of the paper, a table may be sufficient at this stage (Table 1). At the top is just the poset approach directly applied to data.

In the middle, there are more involved techniques to enrich the posets, and finally at the bottom, there are linear orders or construction of composite indicators, which condense the complexity inherent in data matrices to unidimensional representation. In that context, it is worth citing Arcagni et al. [15]: “Admittedly, however compressing the input posets into a simple linear order can be somewhat artificial and misleading…”.

Table 1. Attempts to enhance the degree of comparability.

Method	U (Number of Incomparisons)	Remark	References
Application of Equation (1)	May be very large. Data matrix analyzed without any pretreatment by partial order leading to an “input poset”.	No external information needed beyond the data matrix	[14,16]
Weights not as a sharp number but elements of certain intervals	U will be reduced	Stakeholders have to find intervals for the weights	[17]
Different weighting systems	U will be reduced	GLA (more details below)	[18]
Matrix of mutual ranking probability (MRP)	U will be reduced	Dominance structure of posets	[15,19,20]
Bucket order	U will be reduced	A systematic procedure to reduce U until the value 0	[15,21,22,23]
POSAC (Partial Order Scalogram Analysis by coordinates)	U will be reduced	A bidimensional representation is searched keeping as much as possible the original comparabilities	[24]
Ranking due to mean of different heights	U = 0	Any partial order can be equivalently described by a set of linear orders. The vertical position of an object within a linear order is called its height.	[25,26,27,28]: Concept of Composite, synthetic, indicator

Table 1. Attempts to enhance the degree of comparability.

Method	U (Number of Incomparisons)	Remark	References
Application of Equation (1)	May be very large. Data matrix analyzed without any pretreatment by partial order leading to an “input poset”.	No external information needed beyond the data matrix	[14,16]
Weights not as a sharp number but elements of certain intervals	U will be reduced	Stakeholders have to find intervals for the weights	[17]
Different weighting systems	U will be reduced	GLA (more details below)	[18]
Matrix of mutual ranking probability (MRP)	U will be reduced	Dominance structure of posets	[15,19,20]
Bucket order	U will be reduced	A systematic procedure to reduce U until the value 0	[15,21,22,23]
POSAC (Partial Order Scalogram Analysis by coordinates)	U will be reduced	A bidimensional representation is searched keeping as much as possible the original comparabilities	[24]
Ranking due to mean of different heights	U = 0	Any partial order can be equivalently described by a set of linear orders. The vertical position of an object within a linear order is called its height.	[25,26,27,28]: Concept of Composite, synthetic, indicator

2.3. Enrichment of the MIS—Generalized Linear Aggregation

Although the procedure is explained in detail in [7], we give a brief explanation for the sake of convenience of the reader.

2.3.1. Need of Normalization/Data Pretreatment

Whereas the original partial order method does not need a column wise normalization, the intended numerical combination of weights and indicator values (cf. Equation (4)) makes it favorable to ensure that both the indicator values and the weights are in the same order of amount. Hence, when indicators of different dimensionality are to be combined by weights, they both must be normalized to a common [0, 1] scale. Still worse is when indicators are on an ordinal scale, then the step of normalization is a critical and important step and there will be a trade-off: Either try to analyze all the incomparabilities, based on the original ordinal indicators, or perform a transformation which—in the last consequence—is a perhaps an acceptable data manipulation.

2.3.2. Orientation

In the first example, the 15 criteria values are oriented in both directions. In order to obtain equal orientation of the MIS (co-monotony of the single indicators) the six criteria c2, c9, c11, and c13–c15 (cf. Table 1 below) were all multiplied with −1 to make sure that for all criteria the higher values the better) A normalization of the indicators is performed as follows:

$${\widehat{r}}_s=\{\begin{array}{l}\left(r_s\left(x\right)-min\left(r_s(\dots )\right)\right)/(max\left(r_s(\dots )\right)-min\left(r_s(\dots )\right)if{r}_s\left(x\right)>0\\ -1\cdot \left(abs\left(r_s\left(x\right)\right)-min\left(abs\left(r_s(\dots )\right)\right)\right)/(max\left(abs\left(r_s(\dots )\right)\right)-min\left(abs\left(r_s(\dots )\right)\right)if{r}_s\left(x\right)<0\end{array}$$

(2)

One may consider the application of Equation (2) as an introduction of preference functions within the framework of partial order.

Therein, min or max of r_s(…) and abs(r_s(…)), respectively, is to be taken over the considered objects.

2.3.3. Aggregation Process

When the MIS (denoted as “MIS(old)” in order to emphasize the role of the aggregation process) is written in the form, where r_r are the indicators and e_i the studied elements

$$MIS\left(old\right)=\begin{array}{c}\\ e_1\\ e_2\\ \cdots \\ e_m\end{array}\begin{array}{c}{r}_1 r_2 \dots r_n\\ \left[\begin{array}{ccc}\hspace{1em}& \hspace{1em}& \hspace{1em}\\ \hspace{1em}& \hspace{1em}& \hspace{1em}\\ \hspace{1em}& \hspace{1em}& \hspace{1em}\end{array}\right]\end{array}$$

(3)

then the aggregation to a single scalar, Cf (composite indicator), which serves as a ranking index can be formulated by means of the weights for one single stakeholder, i.e., one single weight scheme, (Equation (4))

Cf⋯=⋯(g₁⋯g₂⋯…⋯g_m) ⋯ MIS(old), i.e., Cf = ∑g_i∙r_i^old

(4)

where the selection of weights is responsible for the composite indicator Cf but is based on a system of indicators riold, where riold refers to the members of MIS(old).

Thus, performing the matrix-multiplication Equation (4), where a row of m entries is acting on each column of matrix MIS, the traditional weighted sum as expression for the aggregation process is obviously obtained. The difficulty in Equation (4) is not its mathematics, but the way how the weights can be found.

The weights bear important information concerning the roles played by the single indicators of an original MIS. There is no need for any equalizing of indicators as pointed out in [29]. The aggregation to a set of single scalar can be formulated as:

$$MIS\left(new\right)=\left[\begin{array}{cccc}{g}_{11}& g_{21}& \dots & g_{m1}\\ g_{12}& g_{22}& \cdots & g_{m2}\\ \dots & \cdots & \cdots & \dots \end{array}\right]·MIS\left(old\right)$$

(5)

Equation (5) describes the calculation of a new MIS by a conventional matrix multiplication of a weight matrix, called G (having m columns, corresponding to the m indicators of the original MIS and as many rows as alternative weighting models that are/can be constructed) and the matrix MIS(old). Each row of matrix G is denoted as a weighting-scheme. Equation (5) can be more formally written as shown in in Equation (6), where the role of G as an operator Ĝ is stressed.

Ĝ*MIS(old) = MIS(new)

(6)

As mentioned above, in practical application of Equation (5), it is more convenient to accept any number for the weights, and to normalize them before Equation (5) is applied. It should be noted that preference functions are needed in other MCDA too and most MCDA further need weights, ignoring the inherent uncertainty in weight findings.

The advantages of the procedure, formulated by Equation (6) are that

(a)

the system of weight-regimes can be checked; for example, the matrix G can be analyzed by correlation measures, or even by posetic tools.

(b)

Equation (5) can also be written as a mapping, performed by an operator Ĝ. Ĝ can be applied to set of indicators of the MIS(old) leading to a set of new indicators, MIS(new) (Equation (7)).

Ĝ {r₁, r₂, …, r_m} = {r′1, r′₂, …, r′_m}

(7)

2.3.4. The Number of Incomparabilities as a Controlling Quantity

This manner of consideration allows a partitioning: G can be thought of as consisting of submatrices, where each of the submatrices describes the opinions of a group of stakeholders. Correspondingly, Ĝ can be seen as obtained from sub-operators Ĝ(A), Ĝ(B), and Ĝ(C), when for instance three groups of stakeholders A, B, and C are considered. Then MIS(new) is just the combination of MIS(new,A), MIS(new,B), and MIS(new,C), respectively, and each of these groups of new indicators is obtained by application of Equation (5) with one of the submatrices. Each of these subsets of new indicators can be the basis for a visualization by a Hasse diagram and for each subset several incomparabilities U(A) (for example, when sub-operator Ĝ(A) is applied) can be obtained. Then, for U based on MIS(new), denoted as U(MIS(new)), the inequality.

U(MIS(old) ≥ U( MIS(new)) ≥ U(Y).

(8)

results, wherein Y stands for a subgroup of stakeholders, e.g., A, or B or C, respectively.

The ≥-sign indicates that the combination of subsets itself can generate new incomparabilities. The number of incomparabilities in a new MIS cannot be larger than that of the old MIS and will typically be lower.

Equation (8) allows to interpret the role of the sub-operators and hence of the different groups of stakeholders. When for instance selected Ĝ(Y) (Y is associated with any subgroups of stakeholders) in that manner that it is approximately just an m*m—unit matrix, then the corresponding MIS(new,Y) is just the MIS(old). Then, with respect to the incomparabilities, the role of other subsets no longer plays any role.

As mentioned above, one advantage of the procedure presented (Equations (5)–(8)) is that the multitude of stakeholders’ opinions can be mapped onto a matrix, that subsequently can be evaluated in a holistic manner. Here, the correlation measures characterizing G, i.e., the mutual correlations between the weight schemes, may be checked. Hence, if, e.g., the correlation between two regimes is very high, then one of the two rows (of matrix G) apparently is redundant and could be but must not be excluded. A perfect correlation (i.e., value 1 between two weight regimes) indicates that one of the two weight regimes could be ignored.

2.4. Software

All partial order analyses were carried out using the PyHasse software [30]. PyHasse is programmed using the interpreter language Python (version 2.6) [30]. Today, the software package contains more than 100 specialized modules and is available upon request from the developer, Dr. R. Bruggemann ([email protected]).

3. Results

3.1. Evaluation Transportation Variants in Kampala

3.1.1. Normalization

The original MIS is adopted from Kalifa et al. [8] (

Table 2. Original MIS adopted from Kalifa et al. [8].

Cluster	Criteria	Unit	BodaBoda	Coaster	Kamunye	TukTuk
Benefit	c1	Score 1–9	9	3	5	7
	c2	Year	1.5	3	2.5	2
	c3	Score 1–9	9	3	5	7
	c4	Score 1–9	3	9	7	5
	c5	Seats	1	29	14	3
	c6	Score 1–9	3	9	7	5
	c7	Score 1–9	3	9	7	5
	c8	Year	5	15	15	5
Cost	c9	kg/100 km·passenger	3.8	0.76	1	1.63
	c10	$/day	20	40	35	25
	c11	$/passenger·year	165	23.3	29.3	87.9
Risk	c12	Score 1–9	3	9	7	5
	c13	Score 1–9	9	1	3	5
	c14	g CO2/pkm	69	20	31	47
	c15	kJ/pkm	2200	430	550	1000

) and the eventual normalized MIS is given in

Table 3. Normalized Kalifa MIS with equal orientation.

Criteria	nBB	nCoa	nKam	nTT
c1	1	0	0.333	0.667
c2	0	−1	−0.667	−0.333
c3	1	0	0.333	0.667
c4	0	1	0.667	0.333
c5	0	1	0.464	0.071
c6	0	1	0.667	0.333
c7	0	1	0.667	0.333
c8	0	1	1.000	0.000
c9	−1	0	−0.079	−0.286
c10	0	1	0.750	0.250
c11	−1	0	−0.042	−0.456
c12	0	1	0.667	0.333
c13	−1	0	−0.250	−0.500
c14	−1	0	−0.224	−0.551
c15	−1	0	−0.068	−0.322

.

It is emphasized that for the criteria c2, c9, c11, c13, c14, and c15, the lower the values the better, whereas for the remaining 9 criteria, the higher the better is valid.

3.1.2. Application of Two Different G Matrices

In the present study, we applied two different G-matrices, comprising 4 and 6 weight-regimes, respectively. It should be noted that the weight-regimes applied here are partly artificially generated for illustrative purposed. In real life cases, the different weight-regimes are typically offered by the participants of the decision process. Thus, the number of weight-regimes is practically determined by stakeholders’ opinions.

The first G matrix includes in addition to the regime, where all weights have the value 1, the weights reported [8] (or) and subjectively chosen weights by the authors of the present paper (br and ca). The weight matrix G1 is shown in

Table 4. Weight-regimes of indifferent-weights (“cw1”), Kalifa et al. (“or”), Bruggemann (“br”) and Carlsen (“ca”).

Regimes	c1	c2	c3	c4	c5	c6	c7	c8	c9	c10	c11	c12	c13	c14	c15
cw1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
or	2.7	5.4	5.4	2.7	5.4	5.4	2.7	5.4	7.5	15	7.5	13	8.8	6.6	6.6
br	3	5	6	7	7	8	1	6	12	15	15	20	5	5	2
ca	2	10	7	4	10	7	5	7	9	10	10	3	6	2	10

. Base on the Pearson correlations between the single regimes it is concluded that they are all mutually only to a minor extent correlated and as such that all should be taken into account.

The correlation (unsquared) is relatively large for “or” and “br”, thus the two weight schemes will not describe too different ideas as to how the coefficients in the weighted sum are to be selected (cf.

Table 5. Pearson correlation coefficients between the four weight-regimes.

cw1-or	na
cw1-br	na
cw1-ca	na
or-br	0.7589
or-ca	0.2291
br-ca	0.1378

). The other two correlations are positive too, but remarkably lower. Hence, one could expect that the weight scheme “ca” infers another conceptual idea. As mentioned above, the possible presence of strongly correlated weight-regimes as such do not pose a problem. Thus, in such cases it will be enough to include only one, i.e., reduced the G matrix, as including highly correlated weight-regimes will yield no new information.

Based on the G1 matrix (Table 4) and the normalized original MIS (Table 3) a new MIS is generated (

Table 6. The new MIS based on the generalized aggregation method (Equation (3)).

Transportation Mode	cw1	or	Br	ca
BB	−0.200	−0.289	−0.256	−0.274
Coa	0.400	0.442	0.504	0.353
Kam	0.281	0.291	0.343	0.239
TT	0.036	0.000	0.038	0.022

), remembering that the criteria c2, c9, c11, c13, c14, and c15 are multiplied by −1 to secure identical orientation.

The same ranking is obtained as was shown in [8]. Although four different weight schemes are applied, with correlation coefficients varying from 0.1378 to 0.7589, the result is an invariant, namely a chain BB < TT < Kam < Coa. Its visualization as Hasse diagram is shown in Figure 1.

The second G matrix was generated based on 6 randomly generated weight-regimes, rd1–rd6. The random selection was [0, 25], respectively. The resulting G matrix is given in

Table 7. Weight-regimes rd1 to rd6.

Regime	c1	c2	c3	c4	c5	c6	c7	c8	c9	c10	c11	c12	c13	c14	c15
rd1	8	8	20	14	20	1	8	15	9	1	15	22	9	16	15
rd2	3	9	18	6	0	5	1	22	0	17	6	19	21	22	8
rd3	12	19	21	1	15	9	22	20	21	14	21	4	3	20	1
rd4	1	5	1	8	22	11	1	11	14	4	10	18	11	6	9
rd5	25	25	7	11	8	5	6	3	8	5	7	11	10	5	0
rd6	18	14	2	11	2	23	9	19	1	18	4	12	14	25	5

. A Pearson correlation showed that the 6 random weights display only very low correlation (in absolute terms) (

Table 8. Pearson correlation coefficients between the 6 weight regimes.

rd1–rd2	0.2295
rd1–rd3	−0.0201
rd1–rd4	0.3906
rd1–rd5	−0.1483
rd1–rd6	−0.4426
rd2–rd3	−0.0804
rd2–rd4	−0.0891
rd2–rd5	−0.2134
rd2–rd6	0.447
rd3–rd4	−0.2693
rd3–rd5	0.0107
rd3–rd6	−0.1064
rd4–rd5	−0.2319
rd4–rd6	−0.2567
rd5–rd6	0.1221

).

As can be seen, in absolute terms the maximum correlation coefficient is found for the weight regimes rd1 and rd6 (−0.4426).

Based on the G2 matrix (Table 6) and the normalized MIS (Table 3) based on the originally adopted data from [27] (Table 2), a new MIS is generated (

Table 9. The new MIS based on the generalized aggregation method (Equation (3)).

Transportation Modes	rd1	rd2	rd3	rd4	rd5	rd6
BB	−0.199	−0.229	−0.163	−0.364	0.015	−0.164
Coa	0.403	0.389	0.325	0.530	0.176	0.452
Kam	0.281	0.289	0.256	0.308	0.161	0.326
TT	0.028	−0.015	0.019	−0.041	0.093	0.04

). The resulting Hasse diagram is identical to the one displayed in Figure 1. It seems as if a differentiation of the 15 indicators by weights is not important. Further, it is noted that the GLA method described here leads to the same ranking as previously reported by Kalifa et al. [8].

3.2. Evaluation of Food Sustainability within 78 Countries

The data for this part of the study were adopted from the Food Sustainability Index (FSI) 2021 [9] and comprises the fraction of the FSI that deals with diet composition ([9], sect. 8.1). This sub study includes 4 indicators (

Table 10. Indicators of the food sustainability study.

	Indicator
r1	Pct. of sugar in diets	Percent sugar in diet
r2	Meat consumption levels	Difference in meat consumption (g/capita(day)) from daily recommended intake (90 g/capita/day)
r3	Saturated fat consumption	g/capita/day
r4	Salt consumption	Average g/day sodium consumption

).

“All indicator scores are normalized to a 0 to 100 scale, where 100 indicates the highest sustainability and greatest progress towards meeting environmental, social and economic key performance indicators (KPI) and 0 represents the lowest” ([9], cf. Excel Workbook: Methodology). Hence, for all four indicators, the higher the indicator value, the better.

A fifth indicator (consumption of fruit and vegetables was left out due to the missing information concerning the weights of this indicators. The study comprises four possible weights, i.e., experts, uniform, outcome, and politics (vide infra). In total, 78 countries were included in the study [9]. In

Table 11. Data matrix of the food sustainability. A total of 78 countries are characterized by the numerical values of four indicators.

	ID	r1	r2	r3	r4
Algeria	DZA	47.3	86.9	91	24.9
Angola	AGO	65.8	91.2	57.3	72.9
Argentina	ARG	12.2	7.4	14.9	59.2
Australia	AUS	13.3	11.4	6	48
Austria	AUT	22.4	41.1	28.3	33.8
Bangladesh	BGD	84.2	69.4	88.6	44.8
Belgium	BEL	15.7	76.9	31.9	47.2
Brazil	BRA	21.7	26.8	26.2	29.5
Bulgaria	BGR	40.9	70.8	69.4	42.6
Burkina Faso	BFA	78.2	79.4	80.8	62.5
Cameroon	CMR	74.9	78.2	79.8	83.6
Canada	CAN	19.6	35.3	48.8	40.2
China	CHN	84.6	68.2	15	10.2
Colombia	COL	27.5	69.4	30.5	30
Cote d’Ivoire	CIV	77.2	76.1	48	64.6
Croatia	HRV	4.1	48.7	40.2	40.2
Cyprus	CYP	46.8	52.2	58.1	30.8
Czech Republic	CZE	33.9	44.3	43.7	33
Dem. Rep. of Congo	COG	69.4	80.1	79.8	74.8
Denmark	DNK	7.5	49.2	29.8	52
Egypt	EGY	43.8	99.1	90.8	41
Estonia	EST	46.9	60.7	48	33.8
Ethiopia	ETH	77.5	72.5	94.3	78.8
Finland	FIN	40.7	50.6	0	36.5
France	FRA	25.4	49.6	8.3	38.6
Germany	DEU	22.6	49.9	26.1	44.8
Ghana	GHA	78	81.7	95.6	76.7
Greece	GRC	45.9	56.1	71.2	38.6
Hungary	HUN	31.2	44.1	25.8	26.3
India	IND	45.5	69.2	83.4	39.9
Indonesia	IDN	56.8	78.5	73.1	49.6
Ireland	IRE	25.4	50.8	33.9	39.4
Israel	ISL	53.8	28.2	43.9	38.1
Italy	ITA	42.3	46.6	31.4	21.2
Japan	JPN	41.9	82.1	77	8.6
Jordan	JOR	15.8	99.8	64.8	29
Kenya	KEN	49.6	82.1	90.1	100
Latvia	LVA	40.8	58.3	50.9	27.3
Lebanon	LBN	7.3	98.3	88.6	55.8
Lithuania	LTU	28	44.3	33.9	30.6
Luxembourg	LUX	49.1	46.6	11.1	30.6
Madagascar	MDG	74.2	79.5	94.1	80.7
Malawi	MWI	68.6	78.8	95.5	95.2
Mali	MLI	77.2	88	97.6	55.2
Malta	MLT	16.9	50.9	63.6	29.8
Mexico	MEX	10.2	60.6	35.2	65.7
Morocco	MAR	35.9	100	88.1	24.1
Mozambique	MOZ	60.2	75.4	60.5	79.6
Netherlands	NLD	32.3	60.7	34.7	50.7
Niger	NER	100	74.3	92.4	61.4
Nigeria	NGA	74	73	69.4	64.1
Pakistan	PAK	37.8	82.8	41.1	34.9
Philippines	PHL	43.7	96.8	48.2	24.7
Poland	POL	18.7	39	7.2	36.7
Portugal	PRT	52.3	32	30.4	26
Romania	ROU	50.4	65.5	63.8	29.2
Russia	RUS	19.8	53.4	57.9	27.9
Rwanda	RWA	70.4	73.3	80.4	96.8
Saudi Arabia	SAU	42.4	87.3	22.3	53.9
Senegal	SEN	54.2	81.4	74.8	55.2
Sierra Leone	SLE	84.9	74.4	52	72.4
Slovakia	SVK	29.1	73.9	65.1	26.3
Slovenia	SVN	45.3	60.2	64.1	26.3
South Africa	ZAF	30.4	66.4	57	73.2
South Korea	KOR	31.3	58.5	33.1	0
Spain	ESP	39	27.4	54.9	31.9
Sudan	SDN	23.8	88.3	95.5	76.1
Sweden	SWE	27.8	56.2	4.1	41.8
Tanzania	TZA	70	76.6	82.1	66
Tunisia	TUN	36	96.5	80.5	20.9
Turkey	TUR	46.9	94.1	43.4	29.8
United Arab Emirates	ARE	29.3	61.8	68.1	41.3
Uganda	UGA	62.7	78	69.2	83.1
United Kingdom	GBR	40.9	51.6	31.7	42.9
United States	USA	0	0	41.5	43.2
Vietnam	VNM	75.6	65.2	24.6	16.6
Zambia	ZMB	66	85.7	87.1	78.8
Zimbabwe	ZWE	27.9	85.3	100	56.6

, the indicator values for the 78 countries are given.

Based on the above data (Table 11), a Hasse diagram was constructed (Figure 2).

Immediately, the diagram has a ‘broad’ structure and is highly complex and dominated by a high number of incomparabilities (1955) and a relative low number of comparabilities (1048). Although the number of incomparabilities is very high (around 66% percentage of all possible binary relations), there are nevertheless chains of length 8. By a special program of the software package PyHasse, details about chains can be found. Here, however, a more detailed discussion is out of the scope of the paper. The main point is that the Hasse diagram is complex; however, there are tools to unfold the jungle of lines.

In the analog study from 2017 [31], four different weighting schemes were discussed corresponding to the relative weighting of the four indicators by four different stakeholders (Expert, Political, Outcome, and Uniform). The four weight schemes are given in

Table 12. Four weighting schemes, as defined within the food sustainability study [31].

Indicators:	Expert	Political	Outcome	Uniform
r1: Percentage of sugar in diets	0.375	0.143	0.400	0.250
r2: Meat consumption levels	0.250	0.286	0.200	0.250
r3: Saturated fat consumption	0.163	0.286	0.200	0.250
r4: Salt consumption	0.213	0.286	0.200	0.250

.

Applying the GLA methodology using the Table 11 and Table 12 gives rise to a new data matrix incorporation the original data as well as the four stakeholder opinions/weight schemes, the new indicators being denoted r1GLA, r2GLA, r3GLA, and r4GLA, respectively (

Table 13. The resulting new MIS following GLA.

	ID	r1GLA	r2GLA	r3GLA	r4GLA
Algeria	DZA	0.595	0.647	0.595	0.625
Angola	AGO	0.723	0.727	0.706	0.718
Argentina	ARG	0.214	0.250	0.212	0.234
Australia	AUS	0.190	0.206	0.184	0.197
Austria	AUT	0.305	0.327	0.296	0.314
Bangladesh	BGD	0.728	0.700	0.742	0.718
Belgium	BEL	0.403	0.468	0.375	0.429
Brazil	BRA	0.254	0.267	0.252	0.261
Bulgaria	BGR	0.534	0.581	0.529	0.559
Burkina Faso	BFA	0.756	0.748	0.758	0.752
Cameroon	CMR	0.784	0.797	0.783	0.791
Canada	CAN	0.327	0.383	0.327	0.360
China	CHN	0.533	0.388	0.525	0.445
Colombia	COL	0.390	0.410	0.370	0.394
Cote d’Ivoire	CIV	0.695	0.649	0.686	0.665
Croatia	HRV	0.288	0.375	0.275	0.333
Cyprus	CYP	0.466	0.470	0.469	0.470
Czech Republic	CZE	0.379	0.394	0.378	0.387
Dem. Rep. of Congo	COG	0.749	0.770	0.747	0.760
Denmark	DNK	0.310	0.385	0.292	0.346
Egypt	EGY	0.647	0.722	0.637	0.687
Estonia	EST	0.477	0.474	0.473	0.474
Ethiopia	ETH	0.793	0.812	0.801	0.808
Finland	FIN	0.357	0.307	0.337	0.320
France	FRA	0.315	0.312	0.295	0.305
Germany	DEU	0.347	0.377	0.332	0.359
Ghana	GHA	0.815	0.837	0.820	0.830
Greece	GRC	0.510	0.540	0.515	0.530
Hungary	HUN	0.325	0.319	0.317	0.319
India	IND	0.564	0.615	0.567	0.595
Indonesia	IDN	0.633	0.656	0.630	0.645
Ireland	IRE	0.361	0.391	0.350	0.374
Israel	ISL	0.425	0.392	0.436	0.410
Italy	ITA	0.371	0.344	0.368	0.354
Japan	JPN	0.506	0.539	0.503	0.524
Jordan	JOR	0.476	0.576	0.450	0.524
Kenya	KEN	0.750	0.849	0.743	0.805
Latvia	LVA	0.439	0.448	0.436	0.443
Lebanon	LBN	0.536	0.704	0.515	0.625
Lithuania	LTU	0.336	0.351	0.330	0.342
Luxembourg	LUX	0.384	0.322	0.373	0.344
Madagascar	MDG	0.801	0.833	0.805	0.821
Malawi	MWI	0.812	0.868	0.813	0.845
Mali	MLI	0.785	0.798	0.790	0.795
Malta	MLT	0.357	0.436	0.356	0.403
Mexico	MEX	0.387	0.476	0.364	0.429
Morocco	MAR	0.579	0.658	0.568	0.620
Mozambique	MOZ	0.682	0.702	0.672	0.689
Netherlands	NLD	0.437	0.464	0.421	0.446
Niger	NER	0.841	0.795	0.856	0.820
Nigeria	NGA	0.709	0.696	0.709	0.701
Pakistan	PAK	0.490	0.508	0.469	0.492
Philippines	PHL	0.537	0.547	0.514	0.534
Poland	POL	0.257	0.264	0.241	0.254
Portugal	PRT	0.381	0.327	0.386	0.352
Romania	ROU	0.518	0.525	0.519	0.522
Russia	RUS	0.361	0.426	0.358	0.398
Rwanda	RWA	0.784	0.816	0.783	0.802
Saudi Arabia	SAU	0.528	0.528	0.497	0.515
Senegal	SEN	0.646	0.681	0.640	0.664
Sierra Leone	SLE	0.743	0.689	0.737	0.709
Slovakia	SVK	0.456	0.514	0.447	0.486
Slovenia	SVN	0.480	0.495	0.482	0.490
South Africa	ZAF	0.528	0.605	0.515	0.568
South Korea	KOR	0.317	0.306	0.308	0.307
Spain	ESP	0.372	0.382	0.384	0.383
Sudan	SDN	0.627	0.777	0.615	0.709
Sweden	SWE	0.340	0.331	0.315	0.325
Tanzania	TZA	0.728	0.742	0.729	0.737
Tunisia	TUN	0.551	0.617	0.540	0.585
Turkey	TUR	0.545	0.545	0.522	0.536
United Arab Emirates	ARE	0.463	0.531	0.460	0.501
Uganda	UGA	0.719	0.748	0.711	0.733
United Kingdom	GBR	0.425	0.419	0.416	0.418
United States	USA	0.160	0.242	0.169	0.212
Vietnam	VNM	0.521	0.412	0.515	0.455
Zambia	ZMB	0.771	0.813	0.767	0.794
Zimbabwe	ZWE	0.601	0.731	0.595	0.675

).

The corresponding Hasse diagram is shown in Figure 3.

In Figure 3, we see a significantly enriched and much ‘slimmer’ Hasse diagram with only 285 incomparabilities and 2718 comparabilities.

Despite the obvious enrichment of the diagram, there are now chains of the length 30, we are not in this system achieving a strict linear order as was the above example studying various transportation forms. However, here, the opinions of all four stakeholders are simultaneously taken into account. Furthermore, the remaining incomparabilities, surviving the effect of the G*-operator motivate to further investigations, which is causing the conflicts. The relatively high number of comparabilities (compared to the original poset) indicates that the stakeholder opinions do not oppose each other but give gradually some more importance to the single indicators of the 4-indicator set. Hence, a possible subsequently decision process is remarkably facilitated.

It is here worth mentioning that simply the shape of Hasse diagrams can be a valuable tool in the analysis of complex data structures [32]. The input poset (i.e., the poset obtained by Equation (1)) of the food sustainability is broad and rather flat, whereas the Hasse diagram after GLA is rather slim and has a remarkable vertical range (cf. chain of lengths of 30).

A subsequent calculation of the average rank [26,33,34] makes much more sense than based on the diagram in Figure 2 although here the essential role of conflicts is no more visualized. In

Table 14. Results of a linearization by LPOMext [33].

Objects.	LPOMext	Rank
Top 10
MWI	77.556	1
GHA	77.275	2
MDG	75.467	3
NER	73.75	4
ETH	73.47	5
RWA	73.387	6
KEN	72	7
MLI	71.619	8
ZMB	70.762	9
CMR	69.708	10
Bottom 10
HUN	9.748	69
HRV	8.305	70
AUT	7.844	71
FRA	7.109	72
KOR	6.728	73
BRA	4.5	74.5
POL	4.5	74.5
ARG	3	76
AUS	1.5	77.5
USA	1.5	77.5

, the top 10 and bottom 10 countries based on an average ranking are shown.

Looking at the data shown in Table 14, it is immediately clear that USA, AUS, and ARG are virtually non-sustainable based on meat consumption (i2). Moreover, in the indicators i1 and i3, these three countries display non-sustainability, whereas in the case of salt consumption (i4), the three countries display values around 50.

4. Discussion

There are some points which should be stated in a clear manner:

(a)

Partial order takes a relational point of view, even if numerical algorithms, as indicated by Equation (5) are applied. Hence, the MIS(new) will once again analyzed in terms of a graph, indicating comparabilities and incomparabilities. Consequently, the data are only used to decide whether a ≤ -relation can be established. This is seen as some zooming out; however, clearly numerical details must be a posteriori analyzed.

(b)

In Equation (5), needs weights are combined with indicator values and then summed up. In a strict mathematical reasoning, this can only be done when the scaling level is metric. If this is not the case, or when the indicator values have very different ranges, which may depend on the used unit of measurements, then a normalization is needed. A normalization in turn requires metric values; when MIS(old) contains ordinal indicators, then a normalization is a crucial step which needs a carefully justification.

(c)

The characterization of the weight scheme (matrix G) can be performed in many ways, as, e.g., different correlation measures can be applied. Further, G itself can be investigated by partial order methods to disclose whether some weight regimes dominate some others.

5. Conclusions and Outlook

As mentioned above, there are constellations where partial ordering is insufficient and delivers only antichains or has an uncomfortable high degree of incomparabilities. Then, the generalized aggregation (Equation (5)) is still simple and remains within the theoretical framework of partial order methodology, since based on the normalized MIS (Equation (2)) a new MIS is generated, which is important on its own right, because stakeholder opinions and (measured) data are included. The new MIS still may have conflicts that require a deeper contextual discussion. In the most general case, Equation (5) cannot be applied without introducing preference functions, which in the framework of partial order are simple [0, 1] linear transformations (whereby it is not excluded that sometimes other preference models could be considered). Thus, in the results section, the effect whereby normalization the ranges of weight and of indicator values is made comparable is discussed. In the example of the Kampala transportation variants, a linear order is obtained, which is the most comfortable case for decision making. That a linear order is not necessarily the result when the matrix G has more than one row, shows the second example of food. Certainly, on the one side, a decision is not that easy (because both, the data of the original indicators (MIS(old)) and the weight regimes must be checked), but on the other side, the incomparabilities indicate that there are still conflicts, which must be discussed in a fair decision process.

However, accepting Equation (5) which frees the decision makers from the process of finding weights in a crisp manner (i.e., as a number with possibly some decimals) seems to be simple enough to be of help in public decision makings.

5.1. Limitations and Future Work

Obviously, there is a need of future work. Thus, the interpretation of ordinal data as metric ones is a big and crucial step that needs attention. However, not in every case such a step can be justified. In that case, GLA breaks down. Therefore, there is a need to find an enrichment procedure, where stakeholder opinions can be collected and can influence the poset. It seems as if the methodological way should start with the set of linear orders, representing the input poset. Hereto, a good basic material is given by Patil and Joshi [35]. The attempts presented therein aim at a final linear order, whereas our intention is an enrichment procedure, which keeps the most important conflicts. Another starting point could be the paper of Arcagni et al. [15], where especially the bucket poset approach seems to be attractive.

5.2. The Novelty of the Here-Presented Approach

Modern MCDA methods such as, e.g., PROMETHEE and members of the ELECTRE family, are typically based on an intricate combination of stakeholders opinions and arithmetic operations according to the methodology used. However, these close interactions make it very difficult to judge the actual effect of the stakeholders and effects that can be assigned to the pure arithmetic procedure.

The here-presented method separates completely the role of stakeholders from that of the procedure. Thus, the outcome is that statistical measures, made as Pearson correlations are available as a further tool to understand the stakeholders’ opinions as a whole. This task is facilitated by the—admitted—simplicity of the partial order methodology itself. The new here-described method is based on conventional matrix multiplication combined with the previously well-described partial order methodology cf., e.g., [7].