Assessing Students’ Mathematical Knowledge with Fuzzy Logic

Abstract

Assessing student mathematical knowledge is an important factor in the mathematics learning process because students obtain important feedback to improve their knowledge and learning. Despite the importance of student assessment, several researchers have shown that student grades comprise noncognitive and metacognitive factors and teachers’ prejudices and beliefs. One method to obtain a more objective view of student mathematical knowledge is through standardized assessments. In this paper, we analyze two methods of assessing student mathematical knowledge by considering their written and oral grades and achievements on the Italian National Assessment of Knowledge (INVALSI). The final grade was produced using the fuzzy logic inference system. It was tested on a sample of 2279 Grade 13 Italian high school students, who had both an oral and written grade in mathematics and who took the INVALSI assessment in the school year 2020–2021. Both tested fuzzy-logic-based assessment methods lowered the mean grades.

1. Introduction

Although the assessment of student knowledge is a key element of the pedagogical process [1,2], student grades do not reflect only student academic knowledge; they include information on students’ noncognitive, metacognitive, and affective elements, and their participation in class, attendance, frequency and readiness in delivering homework, and teachers’ prejudices [3,4,5]. In this case, unobjective grades could lead educators and school authorities to have a distorted or even a false picture of the quality of the students’ real knowledge and abilities [6]. For this reason, it is important to develop more objective techniques to assess student knowledge [7].

Another problem related to grades is grade inflation (i.e., the phenomenon that occurs when teachers tend to grade student knowledge with higher grades than those students would obtain in standardized tests of knowledge) [8]. Some researchers [1,9] have proposed using a standardized assessment of student knowledge to normalize excessively high grades and provide a more objective picture of student learning. A study [1] compared teacher-given mathematics grades to student achievements on standardized national assessments of mathematical knowledge, finding that the hypothetical grades students would obtain by considering only their achievements on the national mathematics assessment differ from their actual grades. Similarly, other studies compared teacher-given mathematics grades to student achievements on standardized assessments, finding that these two variables are correlated but not very strongly [3,10]. Similar results have also been reported for Italy, where teacher-given grades weakly correlate with student achievements on the Italian national assessment of mathematical knowledge [11].

From the mentioned research, we understand that teacher-given grades are a subjective measurement of student mathematical competencies, whereas national assessments represent a more objective method of evaluating student mathematical knowledge. Both measures should be considered to determine a more objective method of determining student grades in mathematics [1,9]. However, the current literature has not yet extensively investigated how to implement these two achievements to reduce grade inflation and normalize student grades. To this end, in this paper, we explore a novel method of assessing student mathematical knowledge by considering both teacher-given grades and student achievements on standardized mathematical tests. We use the mathematical theory of fuzzy logic, which is briefly described in the following sections. The results are discussed, and suggestions for educators and policymakers are also presented.

Background

When Italian students finish middle school (i.e., Grade 8), they can choose to continue their secondary education (high school) between four school typologies: scientific lyceums (SLs), other lyceums (OLs), technical schools (TSs), and vocational schools (VSs). The main aim of these four school typologies is to provide students with different knowledge and abilities; thus, the school curriculum differs between the four school types. There is a common curriculum that each school typology must follow, and it is determined on a national basis. Lyceums provide a higher-level academic education specializing in humanities, sciences, arts, and languages, whereas TSs provide more general education and more qualified technical specialization. Moreover, VSs offer students working knowledge, preparing them for the job market [12]. Thus, the different focuses of these four kinds of high schools differentiate the school mathematics programs. In particular, mathematics represents the leading subject in SLs, and it is studied for at least four hours weekly. In OLs (e.g., linguistic), it is not the main subject, and the number of weekly hours is reduced (e.g., two or three [13]).

In Italian high schools, grades are numerical and range from 1 to 10, with 1 being the lowest possible grade and 10 the highest, representing excellency. All grades from 1 to 5 are failing grades, and those from 6 to 10 are passing grades [14]. According to Italian law [14,15], the final student grades are determined by the Class Council, comprising all teachers that teach in a specific class, based on the suggestions of the subject teacher.

Moreover, the School Council, comprising all teachers from a school, decides how grades are obtained [14]. There are two main categories of grades: written and oral [16]. Written grades comprise all grades of the so-called “compiti in classe” (class tests), which are written tests and are protected by special laws. Oral grades comprise all oral assessments, portfolios, assignments, and other written tests not legally recognized as compiti in classe. In some schools, students receive two grades in mathematics at the end of the first semester (i.e., written and oral grades), whereas students receive one grade (e.g., only a written or oral grade) in other schools. However, at the end of the school year, students receive only one grade, a weighted average of all available grades [17].

The Italian national assessment of mathematical knowledge is a standardized assessment, which is prepared, organized, and corrected by the National Institute to evaluate the instruction, formative, and educative system (INVALSI). All students must take a compulsory test of knowledge of Italian (or another minority language), mathematics, and English in Grades 2, 5, 8, 10, and from the school year 2018–19 on in Grade 13 [14]. In particular, the tests for students in Grades 8, 10, and 13 are computer-based tests [14,18]. The mathematical topics assessed by the INVALSI are present in the “Indicazioni per la scuola secondaria di secondo grado” (Indications for high schools), which is a document that presents the topics that teachers should teach in high schools. The INVALSI tests include several questions, varying each year, that are closed-ended (e.g., multiple-choice questions) or open-ended questions (e.g., short answers and long answers). As high school students solve computer-based tests, the questions are chosen randomly from a database; nevertheless, all tests are equally difficult and are comparable [18]. Each student is assessed with a numerical value on a quantitative scale (Rasch), where the average of all Italian students is 200, and the standard deviation is 40 [19,20]. The method is also used by several international standardized tests (e.g., the PISA and TIMSS) and is detailed in the official documents [21,22]. The INVALSI assessment is believed to be an objective measure of student knowledge [23], although it cannot measure some factors, especially metacognitive and cognitive ones, involved in learning mathematics [18].

2. Review of Literature

2.1. Fuzzy Logic

Fuzzy logic is part of mathematical logic, developed by the Iranian mathematician Lofti A. Zadeh in 1965 [24]. Fuzzy logic represents a mathematically precise way to deal with uncertain quantities, normally verbal descriptions [25]. One of the main concepts in fuzzy logic is fuzzy sets and, related to this, membership functions. A membership function is a function that determines the level of membership of a certain element in a set. In classical logic, an element x might be part of set A (for which we write x ∈ A) or not (x ∉ A); there is no other possibility. In fuzzy logic, such sharp boundaries are “softened,” and we consider the possibility that an element is partially an element of set A. The membership function determines the membership level to a set [26].

Formally, we define a fuzzy set in the following way. We let U be the universal set (of all considered elements) and ${\mu }_A:U\to \left[0,1\right]$ be the membership function associated with each element of the universal set of its grade of membership in set A. Then, set $A=\left\{\left(x,{\mu }_A\left(x\right)\right);x\in U\right\}$ is called a fuzzy set. The membership function is defined by the user [27]. Several families of membership functions are used frequently:

• triangular function ([26]; a < b < c):

$$\mathrm{Trian}\left(x,a,b,c\right)=\{\begin{array}{cc}0& x\le a, x\ge c\\ \frac{x-a}{b-a}& a<x<b\\ \frac{c-x}{c-b}& b\le x<c\end{array}$$

• trapezoidal function ([26]; a < b < c < d):

$$\mathrm{Trap}\left(x,a,b,c,d\right)=\{\begin{array}{c}\begin{array}{cc}0& x\le a, x\ge d\end{array}\\ \begin{array}{cc}\frac{x-a}{b-a}& a<x\le b\end{array}\\ \begin{array}{c}\begin{array}{cc}1& b<x\le c\end{array}\\ \begin{array}{cc}\frac{d-x}{d-c}& c<x\le d\end{array}\end{array}\end{array}$$

• Gaussian function [28]:

$$\mathrm{Gauss}\left(x,\mathsf{\mu },\mathsf{\sigma }\right)=e^{-\frac{{\left(x-\mathsf{\mu }\right)}^2}{2{\mathsf{\sigma }}^2}}.$$

If we want to use fuzzy logic in real-life applications, we follow three phases, which are known as the “fuzzy process” ([26]; see Figure 1):

• fuzzification: We convert clear (crisp) values into fuzzy values via the membership function;
• inference: We use a set of inference rules set by the user, permitting conversion of the fuzzy input values into fuzzy output values; and
• defuzzification: We convert fuzzy output values into clear (crisp) output values using some defuzzification methods.

The fuzzification of crisp values represents the first step in the fuzzy process. This process comprises two steps [25]: determine the membership function for both the input and output data and convert crisp data into verbal variables. Membership functions have different shapes and are chosen by the user, basing the decision on the output. For instance, Gaussian functions are used when greater precision is needed, whereas trapezoidal and triangle functions are used in some rapidly changing dynamic systems [25]. After the fuzzification of crisp data, the inference rule must be defined. It is a logical sequence of the form IF-THEN. The user defines these rules based on everyday experience [25]. The last step is the defuzzification of the data. The result of the inference step combines fuzzy input data and must be converted into crisp data again. Several methods of defuzzification exist. The most used methods are the following [25]:

• Mean of maximum (MOM) is defined as follows:

  $$MOM\left(A\right)=\frac{1}{\left|T\right|}\cdot {\displaystyle \sum }_{x\in T}x,$$

  where $T=\left\{x:{\mu }_A\left(x\right)=“maximal grade of membership”\right\}$ is the set of all data x which have the maximal grade of membership, and |T| is the cardinality of the set (i.e., the number of elements that composes the set T);
• Center of Gravity (COG) is defined as follows:

  $$COG\left(A\right)=\frac{{{\displaystyle \sum }}_x{\mu }_A\left(x\right)\cdot x}{{{\displaystyle \sum }}_x{\mu }_A\left(x\right)},$$

  if the value x is discrete and as:

  $$\mathit{COG}\left(A\right)=\frac{{\displaystyle \int }{\mu }_A\left(x\right)\cdot x\mathrm{dx}}{{\displaystyle \int }{\mu }_A\left(x\right)\mathit{dx}},$$

  if the value x is continuous.

2.2. Assessing Students’ Knowledge with Fuzzy Logic

The reasons to use fuzzy logic for the assessment are presented by several authors [27,29], summarized as follows. We consider a grading criterion where students obtain a “sufficient” (or “passing”) grade at 50% of the points of a test. We consider two students who achieved 50.1% and 49.9%. Considering the difference in percentages, students have very similar achievements; nevertheless, the first student passes the exam, whereas the second does not. Hence, considering the classical discrimination between passing and not passing might be unfair [27]. Fuzzy logic might soften the boundary between passing and not passing. Another problem in several countries is that the final student grades are determined by averaging all student grades. In most cases, grades are ordinal variables, and no average should be computed [30]. Thus, fuzzy logic represents a solution to this problem because it can be inferred from verbal data.

Fuzzy logic for assessing student work has been studied worldwide, and several models have been proposed. For instance, ref. [31] presented an example of determining the final student grade by considering two written exams and a practical exam. Through a fuzzification of the two written exams and the practical exam with triangular membership functions, ref. [31] inferred the first fuzzy student grade from the two written exams and combined it with the fuzzified grade of the practical exam by defining an inference rule, considering five levels of student achievements. Moreover, they considered the achievements of 20 students and analyzed their final grades, which were obtained using the fuzzy process. The results indicated that the fuzzy process produced lower grades for better-achieving students, whereas it graded lower-achieving students more leniently.

Similarly, ref. [32] proposed a model of assessing student knowledge by considering several factors that influence the final grade. In particular, the author considered four factors. Student achievements were fuzzified using generalized bell-shaped curves, and final grades were obtained using the COG method. In the study, ref. [32] found that, in an experiment conducted with 33 students, grades obtained using the fuzzy process were similar to those grades students obtained using the classical grading methods.

In addition, ref. [26] used student grades from the middle and end terms (i.e., the first and second semesters) to determine the final student grade. Student grades were fuzzified with triangle membership functions, and three levels were considered (i.e., low, middle, and high). After defining the inference rule, they used the continuous COG defuzzification method.

Furthermore, ref. [33] considered a method of assessing student knowledge considering three elements (i.e., student attendance of lectures, internal examination, and external examination (end-of-term examination)). Researchers found that in a sample of 54 students, no statistically significant differences exist between traditional grades and those obtained with the fuzzy process.

Similar research, including student grades and lecture attendance, was proposed by [34]. This research found that students graded using the fuzzy process had significantly higher grades than those graded using the traditional method.

Similar results were found by [29], who found that the average grade of the students graded using the fuzzy process was significantly higher than the average of the students graded using the traditional methods. Nevertheless, these two variables are positively and statistically significantly correlated. In contrast, ref. [35] found the opposite: it lowers student achievements. Moreover, ref. [29] also presented some negative aspects of using fuzzy logic for the assessment of student knowledge:

• inference rules are defined based on experience;
• it is impossible to predict the final result; and
• experience is again used to define membership functions.

Thus, considering these limitations, the major issue with the assessment using fuzzy logic is the subjectivity of the definition of the inference rules and membership functions. Nevertheless, fuzzy logic permits teachers to assess students more objectively because several factors are included [36].

2.3. The Proposed Model

According to the discussed research, assessing student knowledge using fuzzy logic is possible, and several models have been proposed. In the present work, we aim to extend the literature concerning student assessment with fuzzy logic by considering a model that aims to normalize student grades and lower the grade-inflation issue.

The model of assessing student knowledge we propose considers two teacher-given grades and student achievements on the Italian national assessment of mathematical knowledge INVALSI. Final student grades (also “hypothetical grades”) are the combination of the written and oral grades and their achievements on the INVALSI test (Figure 2). We transformed the written and oral grades into fuzzy grades using inference Rule 1 (Fuzzy logic 1). Then, we retrieved the final fuzzified grades through a final inference rule (Fuzzy logic 2).

3. Materials and Methods

3.1. Aims of the Research

The present paper aims to analyze a novel method of assessing student mathematical knowledge through fuzzy logic. The goal is to generate final student grades from three inputs: (1) written grades; (2) oral grades; and (3) achievements on the Italian national assessment of mathematical knowledge INVALSI. The research questions are the following:

• RQ1: How are the COG hypothetical grades different from school grades?
  • RQ1.1: Are the COG, student grades, and INVALSI scores correlated?
  • RQ1.2: Is there a difference between COG hypothetical grades and student grades?
  • RQ1.3: Do the COG grades differ between the four high school typologies?
• RQ2: How are the MOM hypothetical grades different from the school grades?
  • RQ2.1: Are the MOM, student grades, and INVALSI scores correlated?
  • RQ2.2: Is there a difference between MOM hypothetical grades and student grades?
  • RQ2.3: Do the MOM grades differ between the four high school typologies?
• RQ3: Is there any difference between the COG and MOM hypothetical grades?

3.2. Methodology

To answer the research questions, we used the nonexperimental quantitative research method.

3.3. Sample

In this study, we considered a sample of all Grade 13 Italian students who took the compulsory national assessment of mathematical knowledge INVALSI in the school year 2020–2021. The original sample was retrieved from the INVALSI Statistical Office web page [37] with the previous registration. The official INVALSI sample comprised 20,281 Grade 13 Italian high school students, among which 18,726 (92.3%) had an oral grade in mathematics, 1555 (8.7%) students had a missing oral grade in mathematics; 2500 (12.3%) students had a written grade in mathematics, and 17,781 (87.8%) had a missing written grade in mathematics. From the initial sample, we included only those students with both a written and oral grade in mathematics at the end of the first semester. The sample comprised 2279 Grade 13 Italian high school students, among which 1242 (54.5%) were females and 1037 (45.5%) were males. Moreover, 290 (12.7%) of the sample were students of a SL, 767 (33.7%) were students of an OL, 623 (27.3%) were students of a TS, and 599 (26.3%) were students of a VS.

3.4. Data Collection

In the official sample retrieved from the Servizio Statistico web page [37], several pieces of information are available:

• school typology (i.e., SL, OL, TS, and VS);
• oral and written grades in mathematics; and
• achievement on the INVALSI mathematics test.

These are the only variables of interest. The data were analyzed by the INVALSI Institute and represent a reliable source [20,38]. There is still a slight possibility that the retrieved data contain minor errors, such as in the transcription of oral or written mathematics grades, which is normally the duty of the school [39].

3.5. Procedure

We present the procedure of the fuzzification, inference, and defuzzification of crisp data. To define the membership functions for the written and oral grades, we distinguished five levels: (1) extremely low, (2) low, (3) average, (4) good, and (5) very good. In

Table 1. Definition of the membership functions for written and oral grades.

Level	Membership Function
Extremely low (EL)	$\mathrm{Trap}\left(x,1,1,2,4\right)$
Low (L)	$\mathrm{Trian}\left(x,2,4,6\right)$
Average (A)	$\mathrm{Trian}\left(x,4,6,8\right)$
Good (G)	$\mathrm{Trian}\left(x,6,8,10\right)$
Very good (VG)	$\mathrm{Trian}\left(x,8,10,10\right)$

, we present the definition of each membership function [26,31]. In Figure 3, we present a graphical representation of such membership functions.

We define the inference rule depicted in

Table 2. The inference rule for Fuzzy logic 1.

		Written Grade
		EL	L	A	G	VG
Oral grade	EL	EL	EL	L	L	A
	L	EL	L	L	A	A
	A	L	L	A	G	G
	G	L	A	G	G	VG
	VG	A	G	G	VG	VG

Notes: EL: extremely low, L: low, A: average, G: good, VG: very good.

[26,31].

The INVALSI scores were obtained through standardization with an average $\mu =200$ and standard deviation $\sigma =40$; thus, we used Gaussian membership functions to describe student achievements on the INVALSI test with five levels, defined in

Table 3. Definition of membership functions for student achievements on the INVALSI test (Fuzzy logic 2).

Level	Membership Function
Extremely low (EL)	$\mathrm{Gauss}\left(x,120,40\right)$
Low (L)	$\mathrm{Gauss}\left(x,160,40\right)$
Average (A)	$\mathrm{Gauss}\left(x,200,40\right)$
Good (G)	$\mathrm{Gauss}\left(x,240,40\right)$
Very good (VG)	$\mathrm{Gauss}\left(x,280,40\right)$

and represented in Figure 4.

We defined the inference rule for Fuzzy logic 1 (i.e., the written and oral grades) and the INVALSI achievements, as presented in

Table 4. The inference rule for Fuzzy logic 2.

		Fuzzy Logic 1
		EL	L	A	G	VG
INVALSI	EL	EL	EL	L	L	A
	L	EL	L	L	A	A
	A	L	L	A	G	G
	G	L	A	G	G	VG
	VG	A	G	G	VG	VG

Notes: EL: extremely low, L: low, A: average, G: good, VG: very good.

.

The defuzzification process requires a membership function, presented in

Table 5. Definition of the membership functions for the defuzzification process.

Level	Membership Function
Extremely low (EL)	$\mathrm{Trap}\left(x,1,1,2,4\right)$
Low (L)	$\mathrm{Trian}\left(x,2,4,6\right)$
Average (A)	$\mathrm{Trian}\left(x,4,6,8\right)$
Good (G)	$\mathrm{Trian}\left(x,6,8,10\right)$
Very good (VG)	$\mathrm{Trian}\left(x,8,10,10\right)$

. We used the COG and MOM defuzzification methods.

The defuzzified grades were approximated to the nearest integer, representing the final student grade (or hypothetical grade). In the following sections, “COG hypothetical grades” denote outcomes obtained using the COG defuzzification method, and “MOM hypothetical grades” represent outcomes obtained using the MOM method.

For example, in

Table 6. Example of the fuzzy logic system.

Oral Grade	COG Fuzzy Logic 1 *	COG Fuzzy Logic 2 *	COG Final Grade	MOM Fuzzy 1 *	MOM Fuzzy 2 *	MOM Final Grade
1	3.24	5.71	6	3.00	5.00	5
2	3.24	5.71	6	3.00	5.00	5
3	3.24	5.71	6	3.00	5.00	5
4	4.00	5.87	6	4.02	6.04	6
5	5.00	5.98	6	5.01	7.00	7
6	5.00	5.98	6	5.01	7.00	7
7	6.00	6.69	7	6.00	8.02	8 **
8	7.00	6.81	7	6.99	7.00	7
9	7.00	6.81	7	6.99	7.00	7
10	8.00	7.77	8	7.98	8.02	8

* Approximated to two decimals. ** See the Discussion section.

, we present how Fuzzy logic 1 (combination of oral and written grades) and Fuzzy logic 2 (final grade) vary for a student with a written grade of 5 and an INVALSI score of 235 among all possible oral grades. We also approximate the outcome of Fuzzy logic 2 to the nearest integer and present the final student grades.

3.6. Data Analysis

Crisp (i.e., clear, original) grades were fuzzified using MATLAB R2020b software. We used the Fuzzy Logic Toolbox, which has been used in research related to fuzzy logic [40]. The data were analyzed using the methods of descriptive and inferential statistics with the statistical software Jamovi. We applied the Shapiro–Wilk test to verify that the normality assumption, and nonparametric tests were used because all data significantly deviated from the normal distribution. We employed the Wilcoxon W-test for paired samples to compare two variables, Spearman’s’ ρ correlation coefficient, and the Kruskal–Wallis χ²-test to check the differences between groups. As a post hoc test, we used the Dwass–Steel–Crichlow–Fligner (DSCF) pairwise comparison. Whenever possible, we present the Cohen’s d measure of effect size [41].

4. Results

4.1. Student Grades and Achievements on INVALSI

An initial screening of the raw data was performed to gain an overview of the written and oral grades and achievements on the INVALSI national assessment of mathematical knowledge (

Table 7. Descriptive statistics of written grades, oral grades and the INVALSI scores.

	Written Grade	Oral Grade	INVALSI
Mean	6.50	6.59	178
Median	6	7	175
Standard deviation	1.43	1.41	35.4
Minimum	1	2	90.1
Maximum	10	10	311
Skewness	−0.102	−0.176	0.512
Kurtosis	−0.214	0.0209	0.239

).

The Shapiro–Wilk test of normality indicates that the written grades (W = 0.988; p < 0.001), oral grades (W = 0.987; p < 0.001), and INVALSI scores (W = 0.985; p < 0.001) depart significantly from normality. Thus, nonparametric tests were used. The Wilcoxon W-test for paired samples revealed that oral grades are significantly higher than written grades (W = 136,705; p < 0.001; Cohen’s d = −0.116), with a mean difference of 0.0878 (SE = 0.0158). Further analysis of the data demonstrated a positive and strong correlation between the written and oral mathematics grades (ρ = 0.856; p < 0.001). A significant moderate correlation exists between written grades and INVALSI scores (ρ = 0.384; p < 0.001) and between oral grades and INVALSI achievements (ρ = 0.380; p < 0.001). Moreover, differences in grades and INVALSI achievements were checked for the four typologies of high schools. The Kruskal–Wallis test indicated statistically significant differences between the four school typologies in the written grade (χ² = 49.4; p < 0.001; ε² = 0.0217), oral grade (χ² = 78.2; p < 0.001; ε² = 0.0343), and INVALSI score (χ² = 591.9; p < 0.001; ε² = 0.2598), as presented in

Table 8. Grades and INVALSI scores between the four school typologies.

Variable	School Typology	Mean	Standard Deviation	Median
Written grade	SL	6.65	1.40	7
	OL	6.65	1.42	7
	TS	6.58	1.50	7
	VS	6.15	1.33	6
Oral grade	SL	6.73	1.40	7
	OL	6.74	1.31	7
	TS	6.75	1.53	7
	VS	6.16	1.30	6
INVALSI	SL	219.35	35.25	-
	OL	175.47	30.44	-
	TS	183.41	29.83	-
	VS	155.41	26.28	-

Notes: SL: scientific lyceum; OL: other lyceum; TS: technical school; VS: vocational school.

.

The DSCF pairwise comparison highlighted some statistically significant differences in student achievements between the four school typologies, listed in

Table 9. Dwass–Steel–Crichlow–Fligner pairwise comparison for school grades and INVALSI scores between school typologies.

			VS	TS	SL
Written grades	OL	W	−9.25	−1.73	−0.189
		p-value	<0.001	0.614	0.999
	VS	W	-	6.93	7.08
		p-value	-	<0.001	<0.001
	TS	W		-	1.25
		p-value		-	0.815
Oral grades	OL	W	−11.35	0.0152	0.180
		p-value	<0.001	1.000	0.999
	VS	W	-	9.85	8.45
		p-value	-	<0.001	<0.001
	TS	W		-	0.089
		p-value		-	1.000
INVALSI	OL	W	−16.87	7.02	23.17
		p-value	<0.001	<0.001	<0.001
	VS	W	-	22.04	29.39
		p-value	-	<0.001	<0.001
	TS	W		-	19.15
		p-value		-	<0.001

Notes: SL: scientific lyceum; OL: other lyceum; TS: technical school; VS: vocational school.

.

4.2. Center of Gravity Fuzzy Logic and Hypothetical Grades

We calculated the final student grade (i.e., the hypothetical grade) and rounded the real number to the closest integer following the procedure described earlier.

Table 10. Descriptive statistics of the COG hypothetical grades.

	Hypothetical Grades
Mean	5.73
Median	6
Standard deviation	1.19
Minimum	3
Maximum	9
Skewness	−0.329
Kurtosis	−0.0879

presents the descriptive statistics of the hypothetical grades.

Table 10 reveals that no student scored lower than 3 or scored a 10. The Shapiro–Wilk test of normality reveals a violation of such an assumption (W = 0.938; p < 0.001); thus, the data are not normally distributed. The hypothetical grades strongly and positively correlate with written grades (ρ = 0.772; p < 0.001), oral grades (ρ = 0.768; p < 0.001), and achievements on the INVALSI test (ρ = 0.750; p < 0.001). Further investigations confirmed that hypothetical grades (M = 5.73; SD = 1.19; Mdn = 6) are lower than the written grades (M = 6.50; SD = 1.43; Mdn = 6) with a mean difference of 0.766 (SE = 0.0192). Such differences are statistically significant (W = 1.25 $\times {10}^6$; p < 0.001; Cohen’s d = 0.835). Moreover, hypothetical grades are also lower than oral grades (M = 6.59; SD = 1.41; Mdn = 7), with a mean difference of 0.854 (SE = 0.0192). These differences are statistically significant (W = 1.38 $\times {10}^6$; p < 0.001; Cohen’s d = 0.931). Moreover, differences in hypothetical grades were checked between the four typologies of high schools. The Kruskal–Wallis test indicated statistically significant differences between the four school typologies (χ² = 289; p < 0.001; ε² = 0.127; see

Table 11. Center of gravity hypothetical grades between the four school typologies.

Variable	School Typology	Mean	Standard Deviation	Median
COG hypothetical grade	SL	6.53	1.15	6
	OL	5.77	1.11	6
	TS	5.90	1.10	6
	VS	5.14	1.11	5

Notes: SL: scientific lyceum; OL: other lyceum; TS: technical school; VS: vocational school.

).

The DSCF pairwise comparison in

Table 12. Dwass–Steel–Crichlow–Fligner pairwise comparison of center of gravity hypothetical grades between school typologies.

		VS	TS	SL
OL	W	−14.62	2.57	13.14
	p-value	<0.001	0.264	<0.001
VS	W	-	16.11	21.25
	p-value	-	<0.001	<0.001
TS	W		-	10.89
	p-value		-	<0.001

demonstrates that all differences between school typologies (except the OL and TS comparison) are statistically significant.

4.3. Mean of Maximum Fuzzy Logic and Hypothetical Grades

We calculated the final student grade (i.e., the hypothetical grade) and rounded the real number to the closest integer following the procedure described earlier.

Table 13. Descriptive statistics of the MOM hypothetical grades.

	Hypothetical Grades
Mean	5.56
Median	5
Standard deviation	1.79
Minimum	1
Maximum	10
Skewness	0.314
Kurtosis	−0.225

presents the descriptive statistics of the hypothetical grades.

The Shapiro–Wilk test of normality indicated that the MOM grades are not normally distributed (W = 0.929; p < 0.001). The hypothetical grades moderately and positively correlate with the written grades (ρ = 0.671; p < 0.001) and oral grades (ρ = 0.676; p < 0.001), and strongly correlate with achievement on the INVALSI test (ρ = 0.814; p < 0.001). Further investigations confirmed that hypothetical grades (M = 5.56; SD = 1.76; Mdn = 5) are lower than the written grades (M = 6.50; SD = 1.43; Mdn = 6) with a mean difference of 0.941 (SE = 0.0267). Such differences are statistically significant (W = 1.12 $\times {10}^6$; p < 0.001; Cohen’s d = 0.738). Moreover, hypothetical grades are also lower than the oral grades (M = 6.59; SD = 1.41; Mdn = 7), with a mean difference of 1.03 (SE = 0.0267). These differences are statistically significant (W = 1.19 $\times {10}^6$; p < 0.001; Cohen’s d = 0.806). Moreover, differences in hypothetical grades were checked between the four typologies of high schools. The Kruskal–Wallis test demonstrated statistically significant differences between the four school typologies (χ² = 354; p < 0.001; ε² = 0.155;

Table 14. Mean of maximum (MOM) hypothetical grades between the four school typologies.

Variable	School Typology	Mean	Standard Deviation	Median
MOM hypothetical grade	SL	6.96	1.73	7
	OL	5.57	1.64	5
	TS	5.80	1.70	6
	VS	4.61	1.38	4

Notes: SL: scientific lyceum; OL: other lyceum; TS: technical school; VS: vocational school.

).

The DSCF pairwise comparisons in

Table 15. Dwass–Steel–Crichlow–Fligner pairwise comparison for mean of maximum hypothetical grades between school typologies.

		VS	TS	SL
OL	W	−15.76	3.00	15.15
	p-value	<0.001	0.147	<0.001
VS	W	—	17.34	24.43
	p-value	—	<0.001	<0.001
TS	W		—	12.48
	p-value		—	<0.001

Notes: SL: scientific lyceum; OL: other lyceum; TS: technical school; VS: vocational school.

reveal that all differences between school typologies are statistically significant, except for the OL and TS comparison.

4.4. Comparing the Two Fuzzy Grading Methods

As illustrated, both the COG and MOM methods of grading students produce grades statistically lower than the student grades. However, a difference of 0.175 (SE = 0.0197) exists between them. The COG (M = 5.73; SD = 1.19; Mdn = 6) method is less strict than the MOM (M = 5.56; SD = 1.76; Mdn = 5) grading method (W = 57,1125; p < 0.001; Cohen’s d = 0.185). Thus, the grades obtained using the MOM method are statistically lower than those obtained using the COG method. The correlation between the two variables is strong and positive (ρ = 0.809; p < 0.001). Checking the possible differences between the four school typologies, we applied the Wilcoxon W-test for each school, finding that the COG grading system produced higher grades in all school typologies except SL students (

Table 16. Center of gravity (COG) and mean of maximum (MOM) hypothetical grades between four school typologies.

School Typology	Higher Grade	W	p	Cohen’s d
SL	MOM	2513	<0.001	0.481
OL	COG	68,382	<0.001	0.211
TS	COG	39,709	<0.001	0.103
VS	COG	54,143	<0.001	0.617

).

5. Discussion and Conclusions

The present research aimed to analyze how the two proposed fuzzy-logic-based assessment methods differ from the traditional teacher assessment. Before doing so, we screened the raw sample data to gain a clearer view of student achievements. Our preliminary analysis reveals that written grades, oral grades, and achievements on the INVALSI test are not normally distributed. Oral and written grades are strongly and positively correlated, indicating that students with higher written grades have higher oral grades and vice versa. A deeper investigation demonstrates that oral grades are generally higher than written grades. The underlying reason for this difference might be explained by considering what comprises oral grades. For instance, oral grades include information about oral examinations, written tests that are not compiti in classe, homework, project work, and attendance. Thus, several elements are included in this information [3,4,5]. In contrast, written grades consist solely of in-class written tests compiti in classe, which do not include other information about the students.

Although the correlation between written and oral grades is strong, the correlation between written grades and achievements on the INVALSI assessment and between oral grades and INVALSI is moderate. The results indicate that students with higher school grades have higher achievements on the Italian national assessment and vice versa. The correlation agrees with previous research in the Italian context [11]; however, it is much lower than the correlations found in the international literature [3,10]. This fact indicates that several factors influence teacher-given grades, not solely student academic knowledge. The INVALSI assessment, although an objective measure of student mathematical outcomes [23], cannot measure some metacognitive and cognitive factors involved in learning and understanding mathematics [18]. Thus, the fact that the correlations are not strong might be explained by considering what comprises each grade and the peculiarity of the INVALSI assessment.

Moreover, from the initial analysis, the average score on the INVALSI was M = 178, which is lower than the national average of M = 200, and the standard deviation SD = 35.4 is lower than the national SD = 40 [18,19]. Thus, additional care should be taken when data are interpreted and generalized to the whole population of Grade 13 Italian students.

Furthermore, we analyzed student grades between different typologies of high schools, finding statistically significant differences concerning written and oral grades and the INVALSI assessment. Students from SLs have the highest achievements on the INVALSI assessment, followed by TSs, OLs, and, finally, VSs. Such differences have also been found by the INVALSI Institute [18,19] and other research [11] and might be understood considering the different focuses of the schools [12]. Students from SLs have the most hours of mathematics per week [13]; thus, they are more likely to study some topics in detail that are assessed by the INVALSI tests. However, VSs aim to provide students practical knowledge, and mathematics represents a marginal subject. Students from all schools have similar grades, except for VSs, which have the lowest achievements concerning student written and oral grades. This fact might be explained by considering that individual teachers use different assessment methods and criteria, leading to different grades between the four high school typologies.

After a general analysis of the sample, we applied the procedure of fuzzification, inference, and defuzzification using both the COG and MOM methods. First, we checked the final student (hypothetical) grades found using the COG method. The minimal grade was 3, and the maximal grade was 9. No student obtained an excellent (10) grade.

Our first research question regarded student COG hypothetical grades and whether they are different from traditional school grades. First, we aimed to determine whether a correlation exists between hypothetical grades and student grades and INVALSI scores. The correlational analysis found that hypothetical grades are positively and strongly correlated with written and oral grades and achievements on the national test, suggesting that students with higher grades or achievements on the INVALSI test obtained a higher hypothetical grade. This result is unsurprising because hypothetical grades are created using student school grades and scores on the INVALSI test.

Second, a deeper analysis of the differences between traditional teacher-given grades and hypothetical grades reveals a statistically significant difference between these variables. Hypothetical grades are lower than written and oral grades. Considering the Cohen’s d measure of the effect size (d = 0.835 for written grades and d = 0.931 for oral grades), substantial differences might exist between hypothetical grades and student grades. Thus, the fuzzy logic assessment method is stricter than the traditional grading system. Considering a student with oral and written grades of 10 and INVALSI of 280 (i.e., very good), the student would obtain a 9.36 as a Fuzzy logic 1 output and an 8.45 (i.e., a score of 9) as the final output, which does not correspond to what one would expect for excellent performance. Moreover, a student with oral and written grades of 10 and an INVALSI score of 311 (the maximum for the INVALSI in this sample) would obtain a total grade of 8.65 (i.e., a score of 9 once again). A student with a 100 on the INVALSI test (i.e., very low) and a 1 for the oral and written grades would obtain a 2.06 for the Fuzzy logic 1 output and a 2.53 (i.e., a score of 3) as the final grade. Thus, from the presented limitations, the proposed method penalizes excellent students and is a more lenient grading method for extremely low-achieving students. The proposed COG method is unfair considering the three models of grading [42]. Considering a fair grade as the achievement students receive as a reward or punishment for learning or failing to learn course content or institutional values, we understand that excellent students are penalized because they master the course topics (their school grades are excellent) and other institutional material (the grade on the INVALSI test is excellent). Hence, excellent grades are unjustly lowered by at least one grade level in this specific case.

Finally, an analysis of the grade distribution between the four high school typologies was considered. The results indicate a statistically significant difference between the four school typologies. Students from the SLs had the highest average, followed by students from TSs, OLs, and VSs. Although no statistically significant difference was found in oral and written grades for the school typology, with the only exception of VS with the lowest mean, hypothetical grades no longer reflect such a distribution. Nevertheless, hypothetical grades reflect the situation depicted concerning the INVALSI test, where students from the SLs outperformed students from all other school typologies. Thus, the hypothetical grades, reflecting student performances on the INVALSI test, maintain such differences between the four school typologies, with the only exception in comparing OLs and TSs, which have a statistically nonsignificant difference in hypothetical grades.

Our second research question regarded student MOM hypothetical grades and whether they are different from traditional school grades. First, we aimed to understand the correlation between the MOM hypothetical grades and student grades and INVALSI scores. A correlational analysis found that hypothetical grades are positively and statistically significantly moderately correlated with school grades and are strongly correlated with achievements on the INVALSI test. Once again, the result is unsurprising because the MOM hypothetical grades also include information from both student grades and achievements on the national assessment of mathematical knowledge.

Second, we verified whether a difference exists between traditional grades and MOM hypothetical grades. The results demonstrated that hypothetical grades are statistically significantly lower than written and oral grades. The interpretation of the Cohen’s d measures of the effect size (d = 0.738 for written grades and d = 0.806 for oral grades) revealed that the differences between the traditional and novel methods of assessing student knowledge are substantial; hence, the MOM hypothetical grades are generally stricter than traditional ones. A student with oral and written grades of 10 and an INVALSI of 280 (i.e., very good) would obtain a 10 as a Fuzzy logic 1 output and a 10 as a final output, which corresponds to what one would expect from excellent performance. Moreover, a student with oral and written grades of 10 and an INVALSI score of 311 (the maximum possible INVALSI score in this sample) would obtain a total grade of 9.78 (i.e., a 10). In contrast, a student with a 100 on the INVALSI test (i.e., very low) and a 1 for oral and written grades would obtain a 1.50 (i.e., a score of 2) as a Fuzzy logic 1 output and a 1.59 (i.e., a score of 2) as a final grade. Hence, the MOM hypothetical grade does not penalize excellent students as much as the COG method. Thus, the MOM hypothetical grade might be considered a fairer method than the COG, despite some anomalies (e.g., those presented in Table 6 (**)). These anomalies are related to the way data are fuzzified and defuzzified. A graphical example is presented in Figure 5, where the surface of the COG Fuzzy logic 2 (denoted by fuzzy2) is depicted. The anomalies are present due to the waves and irregularities of the surface.

Finally, the results indicate a statistically significant difference in student MOM hypothetical grades between the four high school typologies. A deeper analysis confirmed that students from SLs have the highest grades, followed by TSs, OLs, and VSs. All differences in grades are statistically significant, except the OL and TS difference in grades. Hence, the results reveal that the MOM hypothetical grades, similarly to the COG method, discriminate between the school typology similarly to the INVALSI score.

This fact partially answers the last research question (i.e., whether differences exist between the COG and MOM hypothetical grades). Both methods privilege SL students, who had the highest scores on the INVALSI assessment [11,18]. Nevertheless, students from VSs have the lowest hypothetical grades. Thus, both fuzzy methods create a gap in achievements between students from the four high school typologies. Students from SLs have a stronger theoretical basis and a higher-level academic preparation [12]; thus, it is unsurprising that their scores on the INVALSI test are the highest; however, SL students in the sample had written and oral grades similar to those of students from other school typologies. Thus, although their levels of mathematical knowledge measured through the INVALSI test are higher than those of students from other schools, their final grade, which universities can later use to select future students [43,44] or employers to select employees [45], might provide incomplete information about their real knowledge and competencies in mathematics. Thus, including information about student performance on the national assessment (or, in general, other standardized assessments) might contribute to a clearer view of student knowledge and competencies [1,9].

The results demonstrated that the COG and MOM methods produced lower grades than the grades the students obtained on their report cards, both written and oral. Further analysis identified the MOM method as statistically stricter than the COG method, although the latter did not produce any grade below 3 or higher than 9. The correlation between the two kinds of hypothetical grades is strong and positive, indicating that higher grades of one kind would produce higher grades of the other. An analysis of student grades between the four school typologies reveals that SL students received higher grades when graded using MOM methods, whereas students from any other school typology received statistically significantly higher grades using the COG method.

Overall, this research highlights that lower achievements are expected when student scores on the INVALSI test are added to student school grades. Higher-achieving students were penalized using the COG defuzzification method, whereas lower-achieving student grades increased, similar to the findings by [31,35]. Thus, the COG method is unfair in this case. The MOM defuzzification method represents a fairer grading method, despite some anomalies detected due to the definition of the membership functions and inference rules [29]. Nevertheless, the proposed grading system also considers student achievements on standardized assessments, promoting the objectivity of the final student grade [36].