A New Clustering Method Based on the Inversion Formula

Abstract

Data clustering is one area of data mining that falls into the data mining class of unsupervised learning. Cluster analysis divides data into different classes by discovering the internal structure of data set objects and their relationship. This paper presented a new density clustering method based on the modified inversion formula density estimation. This new method should allow one to improve the performance and robustness of the k-means, Gaussian mixture model, and other methods. The primary process of the proposed clustering algorithm consists of three main steps. Firstly, we initialized parameters and generated a T matrix. Secondly, we estimated the densities of each point and cluster. Third, we updated mean, sigma, and phi matrices. The new method based on the inversion formula works quite well with different datasets compared with K-means, Gaussian Mixture Model, and Bayesian Gaussian Mixture model. On the other hand, new methods have limitations because this one method in the current state cannot work with higher-dimensional data (d > 15). This will be solved in the future versions of the model, detailed further in future work. Additionally, based on the results, we can see that the MIDEv2 method works the best with generated data with outliers in all datasets (0.5%, 1%, 2%, 4% outliers). The interesting point is that a new method based on the inversion formula can cluster the data even if data do not have outliers; one of the most popular, for example, is the Iris dataset.

1. Introduction

Artificial intelligence was first mentioned in 1956, but it was not so widely applied for a long time. Artificial intelligence has been widely used in recent decades. The ever-increasing power of possible computations has driven the high availability, development, and applications of artificial intelligence. Data mining is one of the most critical areas, as it is not limited to business, manufacturing, or other services. For this reason, data research has attracted a large number of researchers. Data clustering is one area of data mining that falls into the class of data mining of unsupervised learning. Cluster analysis divides data into different classes by discovering the internal structure of data set objects and their relationship. Clustering aims to create groups of similar observations/elements. The most similar elements, in this case, will be in one cluster and different elements in separate clusters [1].

With the increasing application of data mining, cluster analysis of data is also being applied in many areas: pattern recognition [2,3], bioinformatics [4,5], environment sciences [6], feature selection [7,8], or to solve different healthcare tasks. Clustering algorithms can be used to detect various diseases [9]. For example, different clustering techniques are used to identify breast cancer [10], Parkinson’s disease [11,12], various psychological and psychiatric disorders [13], heart diseases and diabetes [14], and Alzheimer’s disease [15,16], among many others.

Although there are many clustering methods, this problem is being addressed and remains a complex issue. Different clustering methods often do not work well with all data sets, and different methods are very much needed. Although one of the most widely used algorithms currently used is the k-means, as these methods are fast-acting and work well with certain data sets, there are still a lot of possible improvements to increase this method’s accuracy.

Research focuses a lot on developing new density estimation procedures [17,18]. Moreover, in the last years, different scientists started to propose different robust density estimation methods even based on neural networks. There are different researches on this topic: Parzen neural networks [19], soft constrained neural networks [20], and others [21]. Some time ago, we presented a modified inversion formula for density estimation [22]. In this research, we found that this density estimation works better with different data than multiple density estimators. Therefore, we raised the hypothesis that modified inversion formula density estimation would be suitable for data clustering. Due to these facts, this paper aimed to present a new density clustering method based on the modified inversion formula density estimation. This new method should allow one to improve the performance and robustness of the k-means, Gaussian mixture model, and other methods. The main process of the proposed clustering algorithm consists of three main steps. Firstly, we initialized parameters and generated a T matrix. Secondly, we estimated the densities of each point and cluster. Third, we updated mean, sigma, and phi matrices. To compare results in this paper we used k-means, Gaussian mixture model (GMM), and Bayesian Gaussian mixture model (BGMM) clustering methods.

This paper is organized as follows. In the Section 2, we present introduction of the inversion formula and modified inversion formula density estimations and explain the idea behind these estimations. Then, the process of the proposed algorithm is presented in Section 2. In Section 3, we show empirical results with datasets used in the research, evaluation metrics, and experimental results. Finally, conclusions and future work with the new clustering method are given in Section 4

2. Estimation of the Density of the Modified Inversion Formula

Estimating probability density functions (pdf) is considered one of the most important parts of statistical modeling. This feature allows us to express random variables as a function of other variables while simultaneously allowing the detection of potentially hidden relationships in the data. If distribution density f(x) satisfies the equation

$$f\left(x\right)={\displaystyle \sum }_{k=1}^q{p}_k{f}_k\left(x\right)=f\left(x,\theta \right)$$

(1)

the random vector X Rd satisfies the distribution mixture model. The formula above (1), θ is a multi-dimensional parameter of the model. The function $f_k\left(x\right)$ is a function of the distribution density. X is a d-dimensional random vector with a distribution density $f\left(x\right)$. Additionally, we have independent copies of X (X (1),…, X (n)) (sample of X).

We say that the sample satisfies the mixture model if X (t) satisfies (1). We call the size n the sample size (volume). The parameter q is called the mixture number of components, and $p_k$ is the a priori probability. They meet the following conditions:

$$p_k>0, {\displaystyle \sum }_{k=1}^q{p}_k=1$$

(2)

2.1. Gaussian Mixture and Inversion Density Estimation

It is important to notice that Projection (3) of the observations of the Gaussian mixture (1) is also distributed by the (one-dimensional) Gaussian mixture model:

$$f_{\tau }\left(x\right)={\displaystyle \sum }_{k=1}^q{p}_{k,\tau }{\phi }_{k,\tau }\left(x\right)=f_{\tau }\left(x,{\theta }_{\tau }\right)$$

(3)

here ${\phi }_{k,\tau }\left(x\right)=\phi (x; m_{k,\tau }, {\sigma }_{k,\tau }^2)$—one-dimensional Gaussian density. Multivariate mixture parameter and data projection distribution parameters ${\theta }_{\tau }=(p_{k,\tau }, m_{k,\tau }, {\sigma }_{k,\tau }^2)$, $k=1,\dots ,q$ links equality

$$\begin{array}{c}{p}_{j,\tau }=p_j\\ m_{j,\tau }={\tau }^{\prime }{M}_j\\ {\sigma }_{j,\tau }^2={\tau }^{\prime }{R}_j\tau \end{array}$$

(4)

Using the inversion formula,

$$f\left(x\right)=\frac{1}{{\left(2\pi \right)}^d}\underset{R^d}{{\displaystyle \int }}{e}^{-i{t}^{\prime }x}\psi \left(t\right)dt$$

(5)

where $\psi \left(t\right)=\mathrm{E}{e}^{i{t}^{\prime }x}$ denotes the characteristic function of the random variable X. First, the set of projections directions T is selected. Additionally, the characteristic function is changed using the following formula:

$$\widehat{f}\left(x\right)=\frac{A\left(d\right)}{\#T}{\displaystyle \sum }_{\tau \in T}\underset{0}{\overset{\infty }{{\displaystyle \int }}}{e}^{-iu{\tau }^{\prime }x}{\widehat{\psi }}_{\tau }\left(u\right)u^{d-1}{e}^{-h{u}^2}du$$

(6)

where here and below, # denotes the number of elements in the set. With the formula for the volume of a d-dimensional sphere

$$V_d\left(R\right)=\frac{{\pi }^{\frac{d}{2}}{R}^d}{\mathsf{\Gamma }\left(\frac{d}{2}+1\right)} = \{\begin{array}{cc}\frac{{\pi }^{\frac{d}{2}}{R}^d}{\left(\frac{d}{2}\right)!},& \mathrm{kai} d \mathrm{mod} 2\equiv 0\\ \frac{2^{\frac{d+1}{2}}{\pi }^{\frac{d-1}{2}}{R}^d}{d!!},& \mathrm{kai} d \mathrm{mod} 2\equiv 1\end{array}$$

(7)

one can calculate the constant A(d) depending on the dimension of the data:

$$A\left(d\right)=\frac{{\left(V_d(1)\right)}_R^{\prime }}{{\left(2\pi \right)}^d}=\frac{d{2}^{-d}{\pi }^{-\frac{d}{2}}}{\mathsf{\Gamma }\left(\frac{d}{2}+1\right)}$$

(8)

Simulation studies show that the density estimates of the inversion formula are discontinuous/rough. The multiplier $e^{-h{u}^2}$ in the Formula (6) further smoothes the estimate $\widehat{f}\left(x\right)$ with the Gaussian kernel function. It is worth noting that this form of the multiplier allows analytical calculation of the value of the integral. Furthermore, results from extended Monte Carlo studies have shown that using this multiplier reduces the estimation errors. Formula (6) can be used for various estimates of the characteristic function of the projected data. A parametric estimate of the characteristic function was used in the present case.

$${\widehat{\psi }}_{\tau }\left(u\right)={\displaystyle \sum }_{k=1}^{{\widehat{q}}_{\tau }}{\widehat{p}}_{k,\tau }{e}^{iu{\widehat{m}}_{k,\tau }-u^2{\widehat{\sigma }}_{k,\tau }^2/2}$$

(9)

The chosen form of the smoothing multiplier $e^{-h{u}^2}$ allows us to relate the smoothing parameter h to the variances of the projection clusters.

2.2. Modified Inversion Density Estimation

It is worth noting that the Gaussian mixture Model (1) described by the estimate (where $f_k={\phi }_k$) only estimates the density of the distribution close to it well. This can be seen as a drawback of the inversion formula method (9). The density estimation of the inversion formula often becomes complicated due to a large number of components with low a priori probability when the aim is to approximate the density under study with a mixture of Gaussian distributions. This problem can be solved by using a noise cluster.

We discuss a modified density estimation algorithm based on a multivariate Gaussian mixture model (Algorithm 1). However, first, let us define the parametric estimate of the characteristic function of a uniform distribution density:

$$\widehat{\psi }\left(u\right)=\frac{2}{\left(b-a\right)u}\sin \frac{\left(b-a\right)u}{2}·e^{\frac{iu\left(a+b\right)}{2}}$$

(10)

In the formula for calculating the density estimate, construct the estimate of the characteristic Function 9 as a union of the characteristic functions of a mixture of Gaussian distributions and a uniform distribution with corresponding a priori probabilities.

$${\widehat{\psi }}_{\tau }\left(u\right)={\displaystyle \sum }_{k=1}^{{\widehat{q}}_{\tau }}{\widehat{p}}_{k,\tau }{e}^{iu{\widehat{m}}_{k,\tau }-u^2{\widehat{\sigma }}_{k,\tau }^2/2}+{\widehat{p}}_{0,\tau }\frac{2}{\left(b-a\right)u}\sin \frac{\left(b-a\right)u}{2}·e^{\frac{iu\left(a+b\right)}{2}}$$

(11)

Here, the second term describes the noise cluster with an even distribution, and ${\widehat{p}}_0$ is the weight of the noise cluster,

$$a\left(\tau \right)={\left({\tau }^{\prime }x\right)}_{\min }-\frac{{\left({\tau }^{\prime }x\right)}_{\max }-{\left({\tau }^{\prime }x\right)}_{\min }}{2\left(n-1\right)}$$

(12)

$$b\left(\tau \right)={\left({\tau }^{\prime }x\right)}_{\max }+\frac{{\left({\tau }^{\prime }x\right)}_{\max }-{\left({\tau }^{\prime }x\right)}_{\min }}{2\left(n-1\right)}.$$

(13)

2.3. Modified Inversion Density Clustering Algorithm

This section aims to overview the critical aspects of the new modified inversion density estimation (MIDE) clustering method (Algorithm 1). This clustering algorithm uses the EM (expectation maximization) algorithm. The selection of the initial parameters of the EM algorithm is of particular importance for the clustering results, as each new combination of parameters can steer the cluster in a different direction. Random parameter selection is one of the most commonly used solutions for parameter initialization [23,24]. Random selection of initial parameters is a reasonably simple solution, as it is easy to implement. However, one of the significant disadvantages of this method is that such initialization often results in significant deviations in the clustering results. In addition, the algorithm uses a continuous partition to initialize the initial cluster centers.

$$p\left(x\right)=\frac{1}{b-a}$$

(14)

In addition, another method of initialization presented in the software algorithm solution is the selection of random points. In this case, the initial cluster centers are selected not randomly from the entire space but by randomly selecting a point from the observations in the data set. However, this selection also has several drawbacks, as randomly selected points can be too close to each other, and selected points can also be exceptions in the data.

Hierarchical clustering can also be used to address the potential shortcomings of the random cluster center selection method. For the first time, such a classification algorithm that maintains a Gaussian mixture model to form a cluster tree was described by Fraley in 1998 [25]. Maitra [26] proposed a hierarchical clustering based on mean connectivity to obtain an initial model mean. Moreover, Meila and Heckerman [27] experimentally demonstrated that an algorithm using a pattern-based distance measure is better than a random method. This method is applied to the initial mean of the model. Perhaps the only major drawback it has observed so far is that computer computations take a long time and require a large amount of computer memory if there are many data.

One of the most commonly used methods for selecting cluster centers is the k-means and other heuristic clustering methods. This is one of the most widely used initial parameters selection methods. In the case of the initialization of K-means, firstly, the random cluster centers ${\mu }_1,{\mu }_2,\dots ,{\mu }_k \in {ℛ}^n$ are first selected, and then the procedure is performed until convergence is achieved.

$$c^{\left(i\right)}=\arg mi{n}_j\Vert x^{\left(i\right)}-{\mu }_j{\Vert }^2$$

(15)

$${\mu }_j=\frac{{{\displaystyle \sum }}_{i=1}^{m}1\left\{c^{\left(i\right)}=j\right\}{x}^{\left(i\right)}}{{{\displaystyle \sum }}_{i=1}^m\left\{c^{\left(i\right)}=j\right\}}$$

(16)

Clustering using the modified inversion formula density estimation and the EM algorithm is explained below. If the distribution density X of a random vector has q maxima, then it can be approximated by a mixture of q single-mode distribution densities:

$$f\left(x\right)={\displaystyle \sum }_{k=1}^q{p}_k{f}_k\left(x\right)$$

(17)

Suppose that the distribution of X depends on a random variable v, which acquires the values 1, …, q with the corresponding probabilities p₁, …, p_q. In classification theory, v is interpreted as the number of the class to which the observed object belongs. Thus, the X(t) observations would correspond to v(t), t = 1, …, n. The functions f_k are treated as the density of the conditional distribution X under the condition v = k. Based on this approach, loose clustering of the sample is understood as a posteriori probabilities.

$${\pi }_k\left(x\right)=P\left\{\upsilon =k|X=x\right\}$$

(18)

when all $xϵ\left\{X\left(1\right),\dots ,X\left(n\right)\right\}$. Strict clustering of the sample would be an estimate of the random variables v(1), …, v(n) Take a breakdown into subsets based on equality

$$\widehat{v}\left(t\right)=\arg \underset{k=1,\dots ,q}{max}{\widehat{\pi }}_k\left(X\left(t\right)\right)$$

(19)

The estimates ${\widehat{\pi }}_k$ are obtained by approximating the unknown distribution density components with the density estimates of the inversion formula and using the EM (expectation maximization) algorithm. We briefly describe it as follows. Suppose that Equation (15) holds and f_k is the density function of the inversion formula for the Gaussian mixture model, k = 1, …, q, where q is the number of the clusters. In this case (17), let us denote the right side of the equation by f(x,θ), where θ = (p_k, M_k, R_k, a, b, k = 1, …, q). Equality applies:

$${\pi }_k\left(x\right)=\frac{p_k{f}_k\left(x\right)}{f\left(x,\theta \right)} \mathrm{and} k=\overline{1,q}$$

(20)

Having an estimate of θ, the estimates of the probabilities π_k (k-th cluster probability) are obtained from (20) using the “embedding” method, i. y. replacing the unknown parameters on the right with their statistical estimates. The EM algorithm is a reciprocal procedure for estimating the maximum likelihood

$${\theta }^{*}=\mathit{arg}\underset{\theta }{max}L\left(\theta \right), L\left(\theta \right)={{\displaystyle \prod }}_{t=1}^{n}f\left(X\left(t\right),\theta \right)$$

(21)

and to calculate the corresponding estimates ${\widehat{\pi }}_k$. Several authors have independently proposed this algorithm for Gaussian mixture analysis, including Hasselblad [28] and Behboodian [29]. Its properties were later well examined in refs. [30,31,32] and other works. The EM algorithm has received much attention in various review articles and monographs [33,34,35]. Suppose that after r cycles, we obtain the estimates ${\widehat{\pi }}_k={\widehat{\pi }}_k^{\left(r\right)}$. The new estimate $\widehat{\theta }={\widehat{\theta }}^{\left(r+1\right)}$ is then defined by the equations:

$${\widehat{p}}_k=\frac{1}{n}{\displaystyle \sum }_{t=1}^n{\widehat{\pi }}_k\left(X\left(t\right)\right)$$

(22)

$$\widehat{M}\left(k\right)=\frac{1}{n{\widehat{p}}_k}{\displaystyle \sum }_{t=1}^n{\widehat{\pi }}_k\left(X\left(t\right)\right)\cdot X\left(t\right)$$

(23)

$$\widehat{R}\left(k\right)=\frac{1}{n{\widehat{p}}_k}{\displaystyle \sum }_{t=1}^n{\widehat{\pi }}_k\left(X\left(t\right)\right)[X\left(t\right)-\widehat{M}\left(k\right)]\cdot {[X\left(t\right)-\widehat{M}\left(k\right)]}^{\prime }$$

(24)

where k = 1, …, q. Entering ${\widehat{\theta }}^{\left(r+1\right)}$ to the right of (20), we find ${\widehat{\pi }}^{\left(r+1\right)}\left(X\left(t\right)\right)$, $k=\overline{ 1,\hspace{0.17em}q}$, $t=\overline{1,\hspace{0.17em}n}$. As a result of this recursive procedure, we obtain a non-decreasing sequence $L({\widehat{\theta }}^{\left(r\right)})$, but whether it converges to the point of the global maximum depends very much on the initial estimate ${\widehat{\theta }}^{\left(0\right)}$(or ${\widehat{\pi }}^{\left(0\right)}$).

Algorithm 1: Clustering Algorithm Based on the Modified Inversion Formula Density Estimation (MIDE)
	Input: Data set X= [X1, X2, …, Xn], cluster number K
	Output: C1, C2, _, Ct and $\widehat{M}$, ${\widehat{p}}_k$, $\widehat{R}$ Possible initiation of mean vector: (1) random uniform initialization (2) k-means (3) random point initialization Generate a T matrix. The set T is calculated when the design directions are evenly spaced on the sphere.
1	For i = 1: t do
2	Density estimation for each point and cluster based on (9) Update $\widehat{M}$, ${\widehat{p}}_k$, $\widehat{R}$ values based on (22, 23, 24)
4	End
5	Return C1, C2, _, Ct and $\widehat{M}$, ${\widehat{p}}_k$, $\widehat{R}$

In the case of the high mixture model (GMM), the best number of clusters is selected based on the information criterion. This algorithm’s most commonly used information criteria are AIC, BIC, and others. When these information criteria reach their global minimum or maximum, an optimal number of clusters can be said to have been reached. However, there are also some problems in applying these criteria. First, it is necessary to calculate the global maximum of the function as the maximum value of the local maxima, but sometimes this is performed with exceptions. Therefore, applying any procedure cannot guarantee that such a global maximum will be found in such a case.

On the other hand, applying these criteria assumes that one of the parametric methods being compared is correct. This assumption makes the criterion unstable. The arguments presented to raise the question of whether it may be worthwhile to use nonparametric criteria to test the adequacy of the distribution mix model. Several problems can be encountered if the correct number of clusters is not selected. If the number of components selected is too small, then no clear clusters are formed, and one cluster includes more. Meanwhile, if the selected number of clusters is too large, it is much more challenging to calculate clusters in the first place, and less generalizing clusters are also obtained. An attempt to accurately select the number of clusters was provided by Xie, et al. [36], in which an adaptive selection of components/clusters of the Gaussian mixture model was proposed.

3. Experimental Analyses

This section provides information about the modified inversion function based on the proposed clustering method. This section consists of three parts. The first part provides information on the clustering assessment methods used in the empirical study. The second part of this chapter provides information on the data sets used in the study. Finally, the third part of the chapter presents the study’s main results.

3.1. Evaluation Metrics

This section presents the main evaluation metrics used in the empirical study. In order to evaluate the results of clustering, it is essential to choose the appropriate evaluation metrics, as they can also determine the evaluation of clustering. In this study, clustering methods were used to compare J-Score [37], Normalized Mutual Information (NMI) [38], Adjusted Rand Index (ARI) [39] and Accuracy (ACC) [40], and The Fowlkes–Mallows index (FMI) [41]. These metrics were chosen based on the fact that the actual data clusters are known in advance because if the clusters were not known in advance, then the evaluation metrics could be: Calinski and Harabasz score, also known as Variance Ratio Criterion [42], Davies–Bouldin score [43], or others.

J-score. Ahmadinejad and Liu [37] suggested a new clustering evaluation metric, J-score. The J-score is a simple and robust measure of clustering accuracy. It addresses the matching problem and reduces the risk of overfitting that challenge existing accuracy measures [37]. Bidirectional set matching: Suppose a dataset contains N datapoints belonging to T true classes, and cluster analysis produces K hypothetical clusters. To establish the correspondence between T and K, we first considered each class as reference and identify its best-matched cluster (T→K). Specifically, for a class t∈T, we searched for a cluster k∈K that has the highest Jaccard index,

$$I_t=\underset{k\in K}{\max }\frac{\left|V_t{\displaystyle \cap }{V}_k\right|}{\left|V_t{\displaystyle \cup }{V}_k\right|}$$

(25)

where Vt and Vk are the set of datapoints belonging to class t and cluster k, respectively, and|·| denotes the size of a set. We then considered each cluster as a reference and identified its best-matched class (K→T) using a similar procedure. For a cluster k∈K, we searched for a class t∈T with the highest Jaccard index,

$$I_k=\underset{t\in T}{\max }\frac{\left|V_t{\displaystyle \cap }{V}_k\right|}{\left|V_t{\displaystyle \cup }{V}_k\right|}$$

(26)

Calculating overall accuracy: To quantify the accuracy, we aggregated Jaccard indices of individual clusters and classes, accounting for their relative sizes (i.e., number of data points). We first calculated a weighted sum of It across all classes as $R={\displaystyle \sum }_{t\in T}\left(\frac{V_t}{N}{I}_t\right)$, and a weighted sum of Ik across all clusters as $P={\displaystyle \sum }_{k\in K}\left(\frac{V_k}{N}{I}_k\right)$. We then took their harmonic mean as J score,

$$J= \frac{2\times R\times P}{R+P}$$

(27)

To work with this metric, we implemented the calculation of the J-score metric in the Python programming language. The program code is available in Appendix A.

Normalized Mutual Information (NMI). The mutual information (MI) of two random variables is a measure of the mutual dependence of the two variables. MI normalization was performed for greater comparability and better interpretation, and the NMI metric was obtained. The values of this metric can range from 0 to 1.0; in this case, zero would indicate no relationship between the variables, while one would indicate a perfect correlation.

$$NMI=\frac{MI\left(Y^{\prime },Y\right)}{\sqrt{H\left(Y^{\prime }\right)H\left(Y\right)}}$$

(28)

Here, $Y^{\prime }$ are the predicted labels and $Y$ are the actual classes known in advance. $MI\left(Y^{\prime },Y\right)$ is the mutual information between predicted labels and actual labels. This formula also uses the entropy $H(*)$ of predicted labels and actual labels.

Adjusted Rand Index (ARI). The Rand index evaluates the similarity between two clusters. Pairs of all observations are used to calculate this similarity. When calculating this index, there are observations in the assignment to clusters, and how this coincides with the real labels of the clusters.

$$ARI=\frac{2\left(ad-bc\right)}{\left(a+b\right)\left(b+d\right)+\left(a+c\right)\left(c+d\right)}$$

(29)

In the given formula, a is a number that describes how many data points are correctly assigned to a cluster. B is the number of observations in a pair assigned to the same cluster (predicted and actual cluster values match). Here, c is the number of observations in a pair for which the predicted cluster matches, but the actual cluster values do not match. Finally, D is the number of data points in a pair that neither the predicted case nor the actual case belongs to the same cluster.

Accuracy (ACC). Accuracy is often used to measure the quality of classification. It is also used for clustering. It is calculated as the sum of the diagonal elements of the confusion matrix, divided by the number of samples to obtain a value between 0 and 1.

$$ACC=\frac{1}{N}{\displaystyle \sum }_{i=1}^k{n}_i$$

(30)

where N is the total number of data points in the dataset, n_i is the number of data points correctly divided into the corresponding cluster i, and k is the cluster number.

The Fowlkes–Mallows index (FMI). The Fowlkes–Mallows score FMI is defined as the geometric mean of pairwise precision and recall.

$$FMI=\frac{TP}{\sqrt{\left(TP+FP\right)\left(TP+FN\right)}}$$

(31)

True Positive (TP) is the number of pairs of points belonging to the same clusters (true label = predicted label). False Positive (FP) is the number of the points that belong to the same cluster in the true labels but do not belong to the same cluster in the predicted clusters. False Negative (FN) is the number of the pairs of points that belong in the same clusters in the predicted labels and not in the true labels. The higher the metric value, the better the cluster separation is (the maximum possible value of the metric is 1, and the minimum is 0).

3.2. Experimental Datasets

To test the developed method and compare it with other methods, 25 data sets were used in this study. Data sets can be divided into three categories: synthetic, real, and generated data with outliers. Synthetic data sets are data sets that have been generated by other authors and are often used in research on clustering methods. Actual datasets include datasets such as Iris, Wine, Diabetes, and others, and these datasets are also selected based on datasets used by other authors. The third category of generated data with outliers is generated as Gaussian data, including a certain amount of outliers: 0.5%, 1%, 2%, and 4%. These datasets aim to evaluate how different methods work with data with an appropriate amount of outliers. The table below (see

Table 1. A description of the data set used.

ID	Data Sets	Sample Size (N)	Dimensions (D)	Classes
Synthetic
1	Aggregation	788	2	7
2	Atom	800	3	2
3	D31	3100	2	31
4	R15	600	2	15
5	Gaussians1	100	2	2
6	Threenorm	1000	2	2
7	Twenty	1000	2	20
8	Wingnut	1016	2	2
Real
9	Breast	570	30	2
10	CPU	209	6	4
11	Dermatology	366	17	6
12	Diabetes	442	10	4
13	Ecoli	336	7	8
14	Glass	214	9	6
15	Heart-statlog	270	13	2
16	Iono	351	34	2
17	Iris	150	4	3
18	Wine	178	13	3
19	Thyroid	215	5	3
Generated clusters with outliers
20	2 clusters (0.5% outliers)	1005	2	2
21	2 clusters (1% outliers)	1010	2	2
22	2 clusters (2% outliers)	1020	2	2
23	2 clusters (4% outliers)	1040	2	2
25	3 clusters (0.5% outliers)	1005	2	3
26	3 clusters (1% outliers)	1010	2	3
27	3 clusters (2% outliers)	1020	2	3
28	3 clusters (4% outliers)	1040	2	3

) shows the data sets used.

3.3. Performances of Clustering Methods

To avoid the possible influence of successful parameter initialization on test results, all experiments were performed 10,000 times. For the k-means method, initial cluster centers were randomly selected based on the 100 runs best solutions. For GMM (Gaussian Mixture Model), BGMM (Bayesian Gaussian Mixture Model) and clustering based on modified inversion density estimation (MIDE) initial center were selected based on the k-means centers initialization. The following table provides information on the Accuracy metric values for the different clustering algorithms. Other evaluation metrics like NMI, ARI, FMI, and J-Score can be found in the Appendix B tables.

The accuracy results for different datasets are presented in

Table 2. Different models were comparative (means and standard deviation) based on the accuracy (ACC) for 10,000 runs.

Dataset	K-Means		GMM		BGMM		MIDEv1		MIDEv2
	Mean	Std	Mean	Std	Mean	Std	Mean	Std	Mean	Std
Synthetic
Aggregation	0.857	0.005	0.835	0.075	0.907	0.042	0.889	0.008	0.895	0.009
Atom	0.710	0.002	0.618	0.028	0.637	0.022	0.723	0.002	0.746	0.004
D31	0.972	0.015	0.928	0.028	0.601	0.022	0.721	0.017	0.723	0.013
R15	0.997	0.000	0.979	0.036	0.669	0.011	0.768	0.008	0.855	0.007
Gaussians1	1.000	0.000	1.000	0.000	1.000	0.000	1.000	0.000	1.000	0.000
Threenorm	0.591	0.001	0.612	0.047	0.549	0.006	0.649	0.003	0.679	0.003
Twenty	1.000	0.000	0.985	0.029	0.838	0.075	-	-	-	-
Wingnut	0.909	0.000	0.964	0.000	0.965	0.000	0.876	0.000	0.880	0.000
Real
Breast	0.908	0.003	0.940	0.001	0.933	0.001	-	-	-	-
CPU	0.738	0.008	0.574	0.073	0.590	0.093	0.808	0.007	0.828	0.006
Dermatology	0.739	0.044	0.737	0.080	0.756	0.109	-	-	-	-
Diabetes	0.356	0.010	0.419	0.043	0.439	0.033	0.420	0.008	0.448	0.007
Ecoli	0.649	0.013	0.753	0.018	0.739	0.006	0.714	0.011	0.754	0.009
Glass	0.447	0.016	0.468	0.025	0.483	0.025	0.465	0.013	0.487	0.017
Heart-statlog	0.837	0.002	0.794	0.045	0.791	0.045	-	-	-	-
Iono	0.707	0.000	0.810	0.029	0.803	0.023	-	-	-	-
Iris	0.831	0.007	0.953	0.065	0.838	0.049	0.933	0.006	0.955	0.005
Wine	0.966	0.000	0.953	0.048	0.977	0.038	0.943	0.003	0.953	0.004
Thyroid	0.874	0.000	0.953	0.029	0.917	0.035	0.754	0.007	0.778	0.009
Generated blobs with outliers
2 clusters (0.5% outliers)	0.995	0.000	0.995	0.000	0.995	0.000	0.995	0.000	1.000	0.000
2 clusters (1% outliers)	0.989	0.000	0.990	0.000	0.990	0.000	0.990	0.000	0.996	0.000
2 clusters (2% outliers)	0.979	0.000	0.980	0.000	0.980	0.000	0.981	0.000	0.997	0.000
2 clusters (4% outliers)	0.961	0.000	0.962	0.000	0.962	0.000	0.964	0.000	0.996	0.000
3 clusters (0.5% outliers)	0.994	0.000	0.994	0.000	0.994	0.000	0.994	0.000	0.999	0.000
3 clusters (1% outliers)	0.989	0.000	0.989	0.000	0.989	0.000	0.989	0.000	0.997	0.000
3 clusters (2% outliers)	0.979	0.000	0.979	0.000	0.979	0.000	0.981	0.000	0.997	0.000
3 clusters (4% outliers)	0.961	0.000	0.951	0.000	0.945	0.000	0.965	0.000	0.996	0.000

Bold underlined values indicate best results for each dataset.

. It can be seen that the new method based on the inversion formula works quite well with different datasets compared with K-means, Gaussian Mixture Model (GMM), and Bayesian Gaussian Mixture model (BGMM).

4. Discussion

Research focuses a lot on developing new density estimation procedures [17,18]. Moreover, in the last years, different scientists started to propose different robust density estimation methods even based on neural networks such as Parzen neural networks [19], soft constrained neural networks [20], and others [21]. This paper presented a new clustering method based on the modified inversion formula density estimation (MIDE). This new method improves the performance and robustness of the k-means, Gaussian mixture model, and other methods. Method working: Firstly, we initialized parameters and generated a T matrix. Secondly, we estimated the densities of each point and cluster based on the modified inversion formula. Third, we updated mean, sigma, and phi matrices. Based on the results presented earlier, it is possible to conclude that the newly presented method works well with different clustering datasets even if the datasets do not have any outliers. Results based on the generated clusters data with outliers showed that the newly presented method (MIDEv2) works the best in all situations (0.5%, 1%, 2%, and 4%). Based on the accuracy metric with all of these datasets, accuracy was higher than 0.995. The interesting point is that a new method based on the inversion formula can cluster the data even if data do not have outliers; one of the most popular, for example, is the Iris data set. When we compared the accuracy results in other datasets, it can be mentioned that the MIDE method achieved 0.955 accuracy on the Iris dataset compared with the second-best GMM method with 0.953 accuracy; using the ARI metric for this dataset, MIDE methods as well showed better results compared with other methods. Based on the NMI, J-Score, and FMI metrics (see

Table A1. Comparative table of different models (means and standard deviation) based on Normalized Mutual Information (NMI) for 10,000 runs.

Dataset	K-Means		GMM		BGMM		MIDEv1		MIDEv2
	Mean	Std	Mean	Std	Mean	Std	Mean	Std	Mean	Std
Synthetic
Aggregation	0.836	0.004	0.886	0.035	0.909	0.041	0.779	0.006	0.845	0.005
Atom	0.289	0.003	0.170	0.036	0.194	0.028	0.310	0.004	0.319	0.003
D31	0.969	0.005	0.951	0.008	0.871	0.004	0.791	0.007	0.822	0.006
R15	0.994	0.000	0.989	0.012	0.868	0.014	0.881	0.001	0.909	0.001
Gaussians1	1.000	0.000	1.000	0.000	1.000	0.000	1.000	0.000	1.000	0.000
Threenorm	0.024	0.001	0.047	0.039	0.007	0.002	0.069	0.001	0.076	0.001
Twenty	1.000	0.000	0.996	0.008	0.956	0.026	1.000	0.000	0.988	0.005
Wingnut	0.562	0.000	0.778	0.002	0.779	0.000	0.459	0.000	0.420	0.001
Real
Breast	0.547	0.011	0.659	0.003	0.630	0.003	-	-	-	-
CPU	0.487	0.013	0.398	0.025	0.389	0.033	0.467	0.013	0.529	0.011
Dermatology	0.862	0.009	0.809	0.044	0.862	0.049	-	-	-	-
Diabetes	0.090	0.004	0.084	0.041	0.105	0.017	0.089	0.004	0.106	0.003
Ecoli	0.636	0.004	0.636	0.016	0.639	0.010	0.592	0.004	0.534	0.004
Glass	0.303	0.019	0.327	0.052	0.364	0.042	0.304	0.020	0.369	0.024
Heart-statlog	0.363	0.005	0.270	0.055	0.263	0.058	0.339	0.008	0.308	0.007
Iono	0.125	0.000	0.305	0.052	0.299	0.024	-	-	-	-
Iris	0.657	0.006	0.890	0.04	0.751	0.011	0.841	0.007	0.763	0.008
Wine	0.876	0.000	0.856	0.055	0.926	0.054	0.822	0.001	0.799	0.003
Thyroid	0.559	0.000	0.783	0.059	0.661	0.051	0.382	0.009	0.390	0.008
Generated clusters with outliers
2 clusters (0.5% outliers)	0.976	0.000	0.976	0.000	0.976	0.000	0.977	0.000	1.000	0.000
2 clusters (1% outliers)	0.947	0.000	0.957	0.000	0.957	0.000	0.958	0.000	0.974	0.000
2 clusters (2% outliers)	0.916	0.000	0.925	0.000	0.925	0.000	0.928	0.000	0.976	0.000
2 clusters (4% outliers)	0.867	0.000	0.876	0.000	0.876	0.000	0.886	0.000	0.972	0.000
3 clusters (0.5% outliers)	0.978	0.000	0.978	0.000	0.978	0.000	0.978	0.000	0.993	0.000
3 clusters (1% outliers)	0.964	0.000	0.964	0.000	0.964	0.000	0.964	0.000	0.986	0.000
3 clusters (2% outliers)	0.943	0.000	0.943	0.000	0.943	0.000	0.945	0.000	0.985	0.000
3 clusters (4% outliers)	0.907	0.000	0.901	0.000	0.898	0.000	0.911	0.000	0.982	0.000

Bold underlined values indicate best results for each dataset.

), a better method for the Iris dataset would be GMM. It is hard to compare and use multiple metrics because, in this research, we used accuracy as our main metric. After all, all datasets have labels, and it is possible to calculate accuracy of our clustering methods. Compared with other researchers’ results in the past, Sun et al. were able to achieve 0.925 accuracy with the SVC-KM approach [44], and Hyde and Angelov achieved 0.950 accuracy with DDC (Data Density-Based Clustering) [45]. Additonally, it is notable that the MIDE method has a lower standard deviation than other methods used in this research. It is worth mentioning that this method also has limitations. Based on the experimental study, this one method in the current state can not work with higher dimensional data (d > 15). This occurs due to T matrix generation; as dimensions grow, finding a suitable T matrix becomes harder. This one will be solved in the future versions of the model; we will present more about it in future work. Another method problem is speed; the current stage method is slower than other methods, but this problem can be solved with parallelization of the process on the programming side. The future direction of the newly created method is this method application for deep clustering. It can be seen that MIDEv1 and MIDEv2 methods do not work very well with higher-dimension data. Due to that, the deep clustering method with an encoder structure could solve this problem.