Entropy “2”-Soft Classification of Objects

Abstract

A proposal for a new method of classification of objects of various nature, named “2”-soft classification, which allows for referring objects to one of two types with optimal entropy probability for available collection of learning data with consideration of additive errors therein. A decision rule of randomized parameters and probability density function (PDF) is formed, which is determined by the solution of the problem of the functional entropy linear programming. A procedure for “2”-soft classification is developed, consisting of the computer simulation of the randomized decision rule with optimal entropy PDF parameters. Examples are provided.

1. Introduction

The problem of object classification is highly relevant in contemporary theoretical and applied science. Objects, which are subject to classification, can be text documents, audio, video and graphic objects, events, etc. The "m"-soft classification means the attribution of the object to the appropriate m class with a certain probability, unlike the "m"-hard classification, when no alternative distribution of objects by classes is performed. Here, we consider “2”-soft classification, which is the basis for the classification by m classes.

“2”-soft classification is useful in many applied problems, with the exception, perhaps, of “ideal” ones, where all classes, object specification, and data are absolutely accurate. The fact is that real classification problems are immersed in significantly undefined environments. When it comes to data, they are received with errors, omissions, questionable reliability, and different timelines. The formation of decision rule models and their parameterization is not a formalized and subjective process, depending on the knowledge and experience of a researcher. By minimizing an empirical risk of decision rule model in the learning process, we get parameter evaluations for the existing amounts of data (precedents) and for the accepted parameterized model, i.e., evaluations are conditional. How will they, and consequently results of the classification, would behave with other precedents and with other parameterization model remains unclear. Methods of “2”-soft classification are directed to indicate a possible approach to overcome uncertainty factors. The idea is to make a decision rule randomized, and not arbitrarily randomized, but so that its entropy, as a measure of uncertainty, was at the maximum. It would allow for generating an ensemble of the best solutions at the highest uncertainty.

However, among soft and hard classification, fundamental differences exist: the structures of their procedures are similar and based on a more general concept of machine learning by precedents. A huge amount of work is devoted to this issue. Relevant references to them can be found in monographs [1,2,3,4,5,6,7,8], lectures [9,10] and reviews [11,12,13]. The recent fundamental works [6,14,15] clarify the vast diversity of classification algorithms and its learning procedures.

Within the general concept of machine learning, its modification had been proposed: Randomized Machine Learning (RML) [16]. An idea of the randomization is expanding to data and parameters of decision rules. This means that the model parameters of decision rules and data errors are assumed to be randomized in an appropriate way. The difference from existing machine learning procedures is that RML procedures are built not for optimal evaluations of the model parameters, but their probability density function (PDF) and evaluation of the “worst” data errors of PDF. In the RML, as a criterion of evaluation optimality, generalized information entropy is used, maximization of which was carried out on a set described by the system of empirical balances with collections of learning data.

The principle of maximum entropy has already been used in the domain of machine learning—for example, for speech recognition and text classification problems [17] and even for deep neural net parameters estimation [18]. The main advantage of this technique is its robustness to over-fitting in the presence of data errors within small data sets [19,20]. It is demonstrated by the classification experiments with the additive random noise presented in this paper.

2. Statement of the Problem

Suppose that there are two collections of objects: $\mathcal{E}=\{e_1,\cdots ,e_h\};\mathcal{T}=\{t_1,\cdots t_r\}$, which must be distributed between two classes. Objects in both collections are performed by vectors, whose components are variable features that characterize an object and measured quantitatively:

$$\mathcal{E}=\{{\mathbf{e}}^{\left(1\right)},\cdots ,{\mathbf{e}}^{\left(h\right)}\},\mathcal{T}=\{{\mathbf{t}}^{\left(1\right)},\cdots ,{\mathbf{t}}^{\left(r\right)}\}.$$

(1)

We will assume that dimensions of the vectors in both collections are the same, i.e., $({\mathbf{e}}^{\left(i\right)},{\mathbf{t}}^{\left(j\right)})\in R^n$. Collection $\mathcal{E}$ is used for learning, and collection $\mathcal{T}$ for its classification and testing.

The objects in both collections are marked by belonging to a corresponding class: if object $e_l$ or $t_k$ belongs to the first class, it will be assigned 1, or 0 if it belongs to the second one. Consequently, the learning collection is characterized by a vector of answers $\mathbf{y}=\{y_1,\cdots ,y_h\}$ with components equal to 0 or 1, and a testing collection to vector of responses $\mathbf{z}=\{z_1,\cdots ,z_r\}$ (at the end of last century it was known as learning with a teacher [2]). Numbers of these vector components correspond to object numbers in learning and testing collections.

2.1. Learning

Availability of learning collection allows for hypothesizing about the existence of function (decision rule) $F:\mathcal{E}\times {\mathcal{S}}_2\to \mathbf{y}$. The learning problem is to determine parameterized function $\widehat{F}\left(\mathbf{a}\right)$, which approximately describes function F. Function $\widehat{F}\left(\mathbf{a}\right)$ characterizes the model of decision rule. Being under the “soft” classification, the randomized model occurs as a model of decision rule, i.e., it has randomized parameters $\mathbf{a}$. Its input are vectors $\{{\mathbf{e}}^{\left(1\right)},\cdots ,{\mathbf{e}}^{\left(h\right)}\}$, and the output $\widehat{\mathbf{y}}\left(\mathbf{a}\right)$ depends on randomized parameters of $\mathbf{a}$. As such, we choose a model of a single-layer neural net [21]:

$${\widehat{y}}^{\left(i\right)}\left(\mathbf{a}\right)=\mathrm{sigm}\left(\langle {\mathbf{e}}^{\left(i\right)},\mathbf{b}\rangle \right),i=\overline{1,n},$$

(2)

where

$$\begin{array}{ccc}\hfill \mathrm{sigm}\left(x_i\right)& =& \frac{1}{1+exp[-\alpha (x_i-\Delta )]},\hfill \\ \hfill x_i& =& \left(\langle {\mathbf{e}}^{\left(i\right)},\mathbf{b}\rangle \right),\hfill \\ \hfill \mathbf{a}& =& \{\mathbf{b},\alpha ,\Delta \}.\hfill \end{array}$$

(3)

Figure 1 shows a graph of the sigmoid function with parameters: “slope” $\alpha$ and “threshold” $\Delta$. The function of $\mathrm{sigm}\left(x\right)$ in the interval $[1/2,1]$ corresponds to the first class and the values in the interval $[0,1/2)$-to the second class.

In the randomized Models (2) and (3) parameters $\mathbf{a}=\{a_1,\cdots ,a_{(n+2)}\}$ are of interval type:

$$\begin{array}{ccc}\hfill a_k& \in & {\mathcal{A}}_k=[a_k^{-},a_k^{+}],k=\overline{1,n+2},\hfill \\ \hfill \mathcal{A}& =& \underset{k=1}{\overset{n+2}{⨂}}{\mathcal{A}}_k.\hfill \end{array}$$

(4)

Their probabilistic properties are characterized by PDF $P\left(\mathbf{a}\right)$, which is defined over set $\mathcal{A}$.

Since parameters $\mathbf{a}$ are random, then for each object $e_i$, an ensemble ${\widehat{\mathcal{Y}}}^{\left(i\right)}$ of random numbers ${\widehat{y}}^{\left(i\right)}\left(\mathbf{a}\right)$ occurs for the interval $[0,1]$. We define it as an average:

$$\mathcal{M}\left\{{\widehat{y}}^i\left(\mathbf{a}\right)\right\}={\int }_{\mathcal{A}}P\left(\mathbf{a}\right)\mathrm{sigm}\left(\langle {\mathbf{e}}^{\left(i\right)},\mathbf{a}\rangle \right)d\mathbf{a},i=\overline{1,h}.$$

(5)

Therefore, according to RML [16] general procedure, the problem of “2”-soft classification is represented as follows:

$$\mathcal{H}\left[P\left(\mathbf{a}\right)\right]=-{\int }_{\mathcal{A}}P\left(\mathbf{a}\right)lnP\left(\mathbf{a}\right)d\mathbf{a}\Rightarrow max,$$

(6)

when

$${\int }_{\mathcal{A}}P\left(\mathbf{a}\right)d\mathbf{a}=1,$$

(7)

$${\int }_{\mathcal{A}}P\left(\mathbf{a}\right)\mathrm{sigm}\left(\langle {\mathbf{e}}^{\left(i\right)},\mathbf{a}\rangle \right)d\mathbf{a}=y_i,i=\overline{1,h},$$

(8)

where $P\left(\mathbf{a}\right)$ belongs to class ${\mathbb{C}}^1$ of continuously differentiable functions. This is a problem of functional entropy-linear programming, which has an analytical solution, such as entropy optimal PDF $P^{*}\left(\mathbf{a}\right|\theta )$, parameterized by Lagrange multipliers $\theta$:

$$P^{*}\left(\mathbf{a}\right|\theta )=\frac{exp\left[-{\sum }_{i=1}^h{\theta }_i{\widehat{y}}^{\left(i\right)}\left(\mathbf{a}\right)\right]}{\mathcal{P}\left(\theta \right)},$$

(9)

where ${\widehat{y}}^{\left(i\right)}\left(\mathbf{a}\right)$ is defined by the Equality (2), and

$$\mathcal{P}\left(\theta \right)={\int }_{\mathcal{A}}exp\left[-\sum _{i=1}^h{\theta }_i{\widehat{y}}^{\left(i\right)}\left(\mathbf{a}\right)\right].$$

(10)

Lagrange multipliers are determined from the Equation (8).

2.2. Testing

At this point, a collection of objects $\mathcal{T}$ is used, which are characterized by the vector of responses $\mathbf{z}=\{z_1,\cdots ,z_r\}$ with known objects belonging to grade 1 or 2. Vector $\mathbf{z}$ will be used to evaluate the quality of testing. In the testing procedure itself, only $\mathcal{T}$ objects collection will be used.

The subject of the testing is randomized decision Rules (2) and (3) with entropy optimal PDF function of parameters. At the same time, a trial sequence of Monte Carlo is implemented with volume N, every one of which is generated by a random vector $\mathbf{a}$ with appropriate entropy optimal PDF function $P^{*}\left(\mathbf{a}\right)$ (6)–(8). Assume that, as a result of these tests, it was found that the first object from the testing collection was assigned to the first class $N_1$ times and $N-N_1$ times to the second class; .... k-th object was assigned to the first class $N_k$ times and $(N-N_k)$–to the second class, etc. For a sufficiently large number of tests, the empirical probability can be determined

$$\begin{array}{ccc}{p}_1^{\left(1\right)}\hfill & =\hfill & \frac{N_1}{N},\cdots ,p_1^{\left(k\right)}=\frac{N_k}{N},\cdots ,...\hfill \\ p_2^{\left(1\right)}\hfill & =\hfill & \frac{N-N_1}{N},\cdots ,p_2^{\left(k\right)}=\frac{N-N_k}{N},\cdots ,....\hfill \end{array}$$

(11)

Therefore, the testing algorithm can be represented as follows (where i is the number of the object):

Step 1-i. In accordance with the optimal PDF function, a set of output values is generated with entropy optimal Models (2) and (3), comprising N random numbers from an interval $[0,1]$.

Step 2-i. If the random number from this set is larger than $1/2$, then the object $t^{\left(i\right)}$ belongs to the class 1. If it is less than $1/2$, then it belongs to class 2.

Step 3-i Empirical probabilities are determined (11).

As a result of the functioning of this procedure, any object can be defined in one of two classes with a certain probability, which reflects an uncertainty within data and models of decision rule.

The transition to a hard classification can be accomplished by fixing the threshold of probabilities, an object which above it belongs to the relevant class. The number of objects that can be “hard”-classified depends on the threshold value. It is not difficult to find that when thresholds are greater than 0.5, not all objects are classified, but with more than 0.5 probability. However, at thresholds less than 0.5, all are classified, but with less than 0.5 probability.

3. Model Examples of “2”-Soft Classification

In this section, we present the model experiments conducted in accordance with the proposed learning algorithm. It should be noted that all data sets are synthetic and generated manually with a standard random number generator. The first series of experiments aims to introduce the proposed computational procedure and should be considered as illustrative examples. On the other hand, the last example in the next section is more important and demonstrates the advantages of the probabilistic approach for classification in the presence of data errors.

3.1. Soft “2”-Classification of Four-Dimensional Objects

Consider that the objects characterized by four features are coordinates of vectors $\mathbf{e}$ and $\mathbf{t}$.

3.1.1. Learning

The learning collection consists of three objects, every one of which is described by four attributes, and the values of which are shown in

Table 1. Learning data example.

i	${\mathit{e}}_1^{\left(\mathit{i}\right)}$	${\mathit{e}}_2^{\left(\mathit{i}\right)}$	${\mathit{e}}_3^{\left(\mathit{i}\right)}$	${\mathit{e}}_4^{\left(\mathit{i}\right)}$
1	0.11	0.75	0.08	0.21
2	0.91	0.65	0.11	0.81
3	0.57	0.17	0.31	0.91

.

Randomized model of decision Rules (2) and (3) has parameters: $\alpha =1.0$ and $\Delta =0$. Learning vector (“teacher’s” answers) $\mathbf{y}=\{0.18;0.81;0.43\}$ ($y_i<0.5$ corresponds. to class 2; $y_i\ge 0.5$ corresponds to class 1). Lagrange multipliers for the entropy-optimal PDF (9) have the following values: ${\overline{\theta }}^{*}=\{0.2524;1.7678;1.6563\}$. Parameters $a_i\in [-10,10],i=\overline{1,4}$. Entropy-optimal PDF function for this learning collection is as follows:

$$\begin{array}{ccc}\hfill P^{*}(\mathbf{a},\overline{\theta })& =& \frac{exp\left(-{\sum }_{i=1}^3{\theta }_i{y}_i\left(\mathbf{a}\right)\right)}{\mathcal{P}\left(\overline{\theta }\right)},\hfill \\ \hfill y_i\left(\mathbf{a}\right)& =& {\left(1+exp(-\sum _{k=1}^4{e}_k^{\left(i\right)},a_k)\right)}^{(-1)}.\hfill \end{array}$$

(12)

Figure 2 shows a two-dimensional section PDF $P^{*}(\mathbf{a},{\overline{\theta }}^{*})$.

3.1.2. Testing

At this stage, a set of objects $\mathbb{T}=\{t_1,\cdots ,t_r\}$ is used, where each element of the set is characterized by vector ${\mathbf{t}}^{\left(j\right)}\in R^{\left(4\right)}$. The generated array of $(500\times 4)$ four-dimensional random vectors ${\mathbf{t}}^{\left(i\right)},i=\overline{1,500}$, with independent components evenly distributed in intervals $[0,1]$. Then, the algorithm of “2”-soft classification is applicable. Figure 3 shows the empirical probabilities $p_1^{\left(i\right)},p_2^{\left(i\right)}$ of belonging of $t_i$-object to Classes 1 and 2.

3.2. Two-Dimensional Objects “2”-Soft Classification

Consider the objects characterized by two features that are coordinates of the vectors $\mathbf{e}$ and $\mathbf{t}$.

3.2.1. Learning

The learning collection consists of three objects, every one of which is described by two attributes, and the values of which are shown in Table 1. The values of parameters $\alpha ,\Delta$ and intervals for random parameters $\mathbf{a}$ correspond to Example 1. Lagrange multipliers for the entropy-optimal PDF (9) have the following values: ${\overline{\theta }}^{*}=\{9.6316;-18.5996;16.7502\}$. Entropy-optimal PDF function $P^{*}\left(\mathbf{a}\right|\overline{\theta })$ for this learning collection is as follows:

$$\begin{array}{ccc}\hfill P^{*}\left(\mathbf{a}\right)& =& \frac{exp\left(-{\sum }_{i=1}^3{\theta }_i{y}_i\left(\mathbf{a}\right)\right)}{\mathcal{P}\left(\overline{\theta }\right)},\hfill \\ \hfill y_i\left(\mathbf{a}\right)& =& {\left(1+exp(-\sum _{k=1}^{2}e{\left(i\right)}_k,a_k)\right)}^{(-1)}.\hfill \end{array}$$

(13)

Figure 4 shows function $P^{*}(\mathbf{a},{\overline{\theta }}^{*})$.

3.2.2. Testing

All parameters of this example correspond to Example 2. Figure 5 shows empirical probabilities $p_1^{\left(i\right)},p_2^{\left(i\right)}$ of belonging of the $t_i$-object to Classes 1 and 2 ($i=\overline{1,500}$).

4. Experimental Studies of “2”-Hard/Soft Classifications in Presence of Data Errors

“2”-soft classification, based on entropy randomization decision rules, creates a family of “2”-hard classifications, parameterized by thresholds of belonging probabilities. This family complements “2”-hard classifications based on machine learning methods, particularly using the method of least squares [9].

4.1. Data

The experimental study of the soft and hard classifications was performed on simulated data with model errors. All the data for the objects were labeled in order of belonging to one of two classes. This information was used for learning the model of decision rule, and during the method testing to evaluate the classification accuracy.

The objects of classification are characterized by four-dimensional vectors:

$${\mathbf{u}}^{\left(i\right)}=\{u_1^{\left(i\right)},\cdots ,u_4^{\left(i\right)}\},i=\overline{1,N},N=510.$$

(14)

Vectors of both collections were chosen in a random order, and they were evenly distributed on a four-dimensional unit cube. Data errors are modeled by random and evenly distributed four-dimensional vectors $\overline{\xi }\in [-{\overline{\xi }}^{-},{\overline{\xi }}^{+}]$, where ${\overline{\xi }}^{\pm }=\pm 0.1$.

To mark the belonging of vectors $\overline{u}$ to one of the two classes, the number generation procedure from interval $[0,1]$ is applied as follows:

$$y^{\left(i\right)}=sigm\left(\sum _{s=1}^4{e}_s^{\left(i\right)}{b}_s^0\right)+{\xi }^{\left(i\right)},i=\overline{1,510},$$

(15)

where

$$sigm\left(x\right)=\frac{1}{1+exp(-\alpha (x-\Delta \left)\right)},{\mathbf{b}}^0=\{0.1,0.2,0.3,0.4\},\alpha =3,\Delta =0.5.$$

(16)

Let us recall that the values of the sigmoid function from interval $[0.5,1]$ correspond to the first class, and from interval $[0,0.5)$ to the second class, i.e.,

$$c_i=\left\{\begin{array}{c}1,\mathrm{for}0.5\le y^{\left(i\right)}\le 1,\hfill \\ 2,\mathrm{for}0.0\le y^{\left(i\right)}<0.5.\hfill \end{array}\right.$$

(17)

Thus, numerical characteristics of objects components of $u_s^i,s=\overline{1,4};i=\overline{1,510}$ vectors serve as their features, and $c_i$ values refer them to a certain class.

Two collections were formed from these objects: the learning one $\mathcal{E}=\{{\mathbf{e}}^{\left(1\right)},\cdots ,{\mathbf{e}}^{\left(10\right)}\}$ and the testing one $\mathcal{T}=\{t^{\left(1\right)},\cdots ,t^{\left(500\right)}\}$.

4.2. Randomized Model (Decision Rule)

In the experimental study, we used a model that coincides by its structure with (15) and (16), but has randomized parameters and noises:

$${\widehat{y}}^{\left(i\right)}\left(\mathbf{a}\right)=sigm\left(\sum _{s=1}^4{e}_s^{\left(i\right)}{a}_s\right)+{\zeta }^{\left(i\right)},i=\overline{1,10},$$

(18)

where $\mathbf{a}=\{a_1,\cdots ,a_4\}$ are randomized parameters of the interval type. For example, $\mathbf{a}\in \mathcal{A}=[-1,1]$; ${\zeta }^{\left(i\right)}$ is a random interval type noise, i.e., ${\zeta }^{\left(i\right)}\in {\mathcal{K}}_i=[-0.1,0.1],i=\overline{1,10}$. Noise probabilities ${\zeta }^{\left(i\right)}$ are characterized by PDF $q_i\left({\zeta }^{\left(i\right)}\right)$, which exists as a continuously differentiable function.

The optimization problem of entropy-robust estimation is formulated as follows [16]:

$$H[P\left(\mathbf{a}\right),Q\left(\xi \right)]=-{\int }_{\mathcal{A}}P\left(\mathbf{a}\right)lnP\left(\mathbf{a}\right)d\mathbf{a}-\sum _{i=1}^{10}{\int }_{{\mathcal{K}}_i}{q}_i\left({\zeta }^{\left(i\right)}\right)ln{q}_i\left({\zeta }^{\left(i\right)}\right)d{\zeta }^{\left(i\right)}⟹\underset{P,Q}{max}$$

(19)

under constraints:

$$\int P\left(\mathbf{a}\right)d\mathbf{a}=1,{\int }_{{\mathcal{K}}_i}{q}_i\left({\zeta }^{\left(i\right)}\right)d{\zeta }^{\left(i\right)}=1,i=\overline{1,10};$$

(20)

and

$$\mathcal{M}\left[{\widehat{y}}^{\left(i\right)}\left(\mathbf{a}\right)\right]={\int }_{\mathcal{A}}P\left(\mathbf{a}\right){\widehat{y}}^{\left(i\right)}\left(\mathbf{a}\right)d\mathbf{a}+{\int }_{{\mathcal{K}}_i}{q}_i\left({\zeta }^{\left(i\right)}\right)d{\zeta }^{\left(i\right)}=y^{\left(i\right)},i=\overline{1,10}.$$

(21)

The problem (19)–(21) has an analytical solution in the form of

$$P^{*}(\mathbf{a},\overline{\theta })=G^{-1}\left(\theta \right)W(\mathbf{a},\overline{\theta }),$$

(22)

$$W(\mathbf{a},\overline{\theta })=exp\left(\sum _{i=1}^{10}{\theta }_i{\widehat{y}}^{\left(i\right)}\left(\mathbf{a}\right)\right),G\left(\theta \right)={\int }_{A}exp\left(\sum _{i=1}^{10}{\theta }_i{\widehat{y}}^{\left(i\right)}\left(\mathbf{a}\right)\right)d\mathbf{a},$$

(23)

and

$$q_i^{*}({\zeta }^{\left(i\right)},{\theta }_i)=F_i^{-1}\left({\theta }_i\right)U_i({\zeta }^{\left(i\right)},{\theta }_i),$$

(24)

$$U_i({\zeta }^{\left(i\right)},{\theta }_i)=exp\left({\theta }_i{\zeta }^{\left(i\right)}\right),F_i\left({\theta }_i\right)={\int }_{{\mathcal{K}}_i}exp\left({\theta }_i{\zeta }^{\left(i\right)}\right)d{\zeta }^{\left(i\right)},$$

(25)

where $\theta =\{{\theta }_1,\cdots ,{\theta }_{10}\}$ are Lagrange multipliers.

In this example, the optimization was performed by MATLAB 2015b software (Version 8.6, Build 267246), using the optimization tools (optimization toolbox) and statistical learning (statistic and machine learning toolbox). The values obtained for Lagrange multipliers are: ${\overline{\theta }}^{*}={10}^{-3}\times \{0.1684,0.0275,0.0288,0.0110,0.0326,0.0429,-0.0469,0.0073,-0.0012,-0.0354\}$.

Figure 6 shows a two-dimensional section of entropy-optimal PDF function $P^{*}(\mathbf{a},{\overline{\theta }}^{*})$.

4.3. Testing of Learning Model: Implementation of “2”-Soft Classification

The process of randomized classified algorithm testing involves a generation of a random set of model parameters according to received entropy-optimal density functions. To generate an ensemble of random values, an algorithm of Metropolis–Hastings is used [22].

In this example, the vectors set was generated ${\left[{\mathbf{a}}^{\left(k\right)}\right]}^{*}=\{{\left[a_1^{\left(k\right)}\right]}^{*},\cdots ,{\left[a_4^{\left(k\right)}\right]}^{*}\},k=\overline{1,100}$, with entropy-optimal PDF $P^{*}(\mathbf{a},{\overline{\theta }}^{*})$ (22), defining 100 implementations of decision rule classification models. For each object $\left(i\right)$ from the testing collection, we obtain an ensemble of numbers from the interval $[0,1]$

$${\widehat{y}}_k^{\left(i\right)}\left({\left[{\mathbf{a}}^{\left(k\right)}\right]}^{*}\right)=sigm\left(\sum _{s=1}^4{e}_s^{\left(i\right)}{\left[a_s^{\left(k\right)}\right]}^{*}\right),k=\overline{1,100}.$$

(26)

Along this ensemble, the objects are distributed into classes in accordance with rule

$$c^{\left(i\right)}=\left\{\begin{array}{c}1,\mathrm{for}0.5\le {\widehat{y}}^{\left(i\right)}\le 1,\hfill \\ 2,\mathrm{for}0.0\le {\widehat{y}}^{\left(i\right)}<0.5.\hfill \end{array}\right.$$

(27)

The probabilities of belonging to Classes 1 and 2 are calculated as follows:

$$p_1^{\left(i\right)}=\frac{N_1[c_k^{\left(i\right)}==1]}{K},p_2^{\left(i\right)}=\frac{N_2[c_k^{\left(i\right)}==2]}{K},K=100,$$

(28)

Here, $N_1$ and $N_2$ are the numbers of tests in which the objects were classified as Classes 1 and 2, respectively. Figure 7 shows the value distribution of empirical probability of belonging to Classes 1 and 2 for testing sample objects, which characterize “2”-soft classification with entropy optimal linear decision rule.

It was noted above that with “2”-soft classification, you can generate families of “2”-hard classification, assigning various thresholds $\eta$ of empirical probability. Belonging to a class is defined by the following condition:

$$\mathrm{Object}\left(i\right)\in \mathrm{class}1,\mathrm{if}{p}_1^i\ge \delta .$$

(29)

Let us determine the accuracy of classification as:

$$\eta =\frac{L_{tr}}{L}100\%,$$

(30)

where L is the length of the test collection, and $L_{tr}$ is the number of correct classifications.

The research of the above example shows the existence of dependence between the accuracy $\eta$ and the threshold $\delta$. Figure 8 shows this dependence, where we can see that the quantity $\eta$ attains a maximum value equal to 78.5% at $\delta =0.19$. However, “2” hard classification, which uses the decision rule (26) with non-randomized parameters and the method of least squares to determine their values, yields a 66% accuracy (regardless of the threshold).

This result means that a traditional exponential model widely used for classification problems [6,9] could be improved by statistical interpretation of its output. In this case, the proposed randomized machine learning technique that evolves the generation of entropy-optimal probability functions, and variation of soft decision threshold boosts the accuracy of classification to nearly 80 percent.

5. Conclusions

A method for “2”-soft classification was proposed, which allows for referring objects with calculated empirical probability to one of two classes. The latter is determined by the Monte Carlo method with the use of the entropy-optimal randomized model of the decision rule. A corresponding problem is formulated for the maximization of entropy functional on the set, a configuration of which is determined by balances between the real input data and average output of the randomized model of the decision rule.

The problem of “2”-soft classification for the case of existence of data errors simulated by the additive noise evenly distributed in the parallelogram. The entropy-optimal estimations of probability distribution density for model parameters and for noises, which are the best at the maximum indeterminateness in terms of entropy, were obtained. We performed an experimental comparison of “2”-hard and “2”-soft classifications. An existence of the classification threshold interval was revealed, whereby a precision of “2”-soft classification (the number of correct answers) was increased by 20. Examples illustrating the proposed method are provided.