A Review of Fuzzy and Pattern-Based Approaches for Class Imbalance Problems

Abstract

The usage of imbalanced databases is a recurrent problem in real-world data such as medical diagnostic, fraud detection, and pattern recognition. Nevertheless, in class imbalance problems, the classifiers are commonly biased by the class with more objects (majority class) and ignore the class with fewer objects (minority class). There are different ways to solve the class imbalance problem, and there has been a trend towards the usage of patterns and fuzzy approaches due to the favorable results. In this paper, we provide an in-depth review of popular methods for imbalanced databases related to patterns and fuzzy approaches. The reviewed papers include classifiers, data preprocessing, and evaluation metrics. We identify different application domains and describe how the methods are used. Finally, we suggest further research directions according to the analysis of the reviewed papers and the trend of the state of the art.

1. Introduction

In recent years, classification using imbalanced classes has attracted increasing interest because it is a recurrent problem in real-world data, such as financial statement fraud detection [1], bankruptcy prediction [2,3,4], medical decision making [5], fault diagnosis [6,7], card fraud detection [8,9], pattern recognition [10], cancer gene expression [11,12], and telecommunications fraud [13]. The class imbalance problem is presented when the objects are not distributed equally among classes; usually, the popular classifier is biased to the class that has significantly more objects (a.k.a. majority class) and dismisses the class with significantly fewer objects (a.k.a. minority class) [14,15].

Popular classifiers such as support vector machine (SVM) [16], C4.5 [17], and k nearest neighbour (kNN) [18] were originally created for non-imbalanced databases. If the same techniques are used to handle imbalanced databases, these classifiers could obtain poor classification results for the minority class [19].

Many approaches have been developed in recent years and have addressed the imbalance problem with great interest in the scientific community [15,20,21,22]. Unfortunately, some results are obtained by black-box models, and there is a lack of understandability in the model itself. For that reason, the international scientific community has a great interest in creating explainable artificial intelligence (XAI) models [23]. Those models need to be accurate and explainable for experts in a specific area [24].

Among these approaches, the usage of patterns and fuzzy logic has shown more accurate results than other classifiers [25,26]. Pattern-based classifiers provide an expression defined in a particular language that describes a set of objects. Nevertheless, these types of patterns are restrictive in their classifications because their features are limited to crisp values [27]. The hard cut of the data can result in an abrupt classification of the objects. This problem can be solved with fuzzy patterns that have the property of flexibility, and the results are expressed in a language closer to that used by an expert [28,29].

To the best of our knowledge, there is no review of the most recent papers for the class imbalance problem addressed by fuzzy and pattern-based approaches. Hence, in this paper, we provide a systematic review of the state of the art for imbalanced databases that include fuzzy and pattern-based approaches, including theoretical and practical approaches. For the theory, we present common approaches to deal with imbalanced data; these approaches will include data mining, preprocessing, evaluation metrics, and classifiers. For the applications, we gather the most common real-world scenarios where imbalanced data is a significant concern. We cover different research areas so that the reader can have a better understanding of the problem.

The rest of this paper is structured as follows: Section 2 presents the general background. Section 3 describes the research methodology and presents different approaches to deal with class imbalance problems with patterns and fuzzy approaches. Section 4 presents different research areas where class imbalance problems are presented in real-world data. Section 5 presents a taxonomy to group the different approaches. Section 6 explains our thoughts of the current state of the art and presents future directions. Section 7 presents the conclusions of this paper.

2. General Background

In this section, we present a general background that explains the class imbalance problem, common ways to deal with imbalanced data, how pattern-based classifiers works, and preliminaries about pattern-based classification and fuzzy logic.

2.1. Class Imbalance Problem

As we previously stated, the class imbalance problem is a recurrent issue in real-world data, and it occurs when the objects are not distributed equally among the problem’s classes. The objects of the minority class can be described as safe, borderline, rare, and outliers [30], making the classification process even harder for classifiers [31]. In the following list, there is a description for each of them:

• Safe: Data located in the homogeneous regions from one class only (majority or minority).
• Borderline: Data located in nearby decision boundaries between classes. In this scenario, the classifiers need to decide the class of the objects when they are in the decision boundary, which, due to the bias, result in favor of the majority class.
• Rare: Data located inside the majority class is often seen as overlapping. The classifier tends to classify the minority class as part of the majority class. The effect of this has been discussed in different works [32].
• Outliers: Data located far away from the sample space. The minority objects could be treated as noise by the classifier; on the other hand, noise could be treated as minority objects [33]. This happens when there are outlier objects in the database and the data should not be removed because it could be a representation of a minority class.

The presence of small disjuncts and the lack of density in the training data also affect classic classifier performance. These characteristics tend to affect outlier data with a greater impact [19].

2.2. Approaches to Deal with Class Imbalance Problems

To deal with the class imbalance problem, three approaches are commonly used: data level, algorithm level, and cost-sensitive [15,31,34].

• Data level: The objective of this approach is to create a balanced training dataset by preprocessing the data through artificial manipulation. There are three solutions to data sampling: over-sampling, under-sampling, and hybrid-sampling.

  (a)

  Over-sampling: New objects are generated for the minority class. The basic version of this solution is the random oversampling of the objects belonging to the minority class. However, the main drawback is that it can lead to overfitting [35,36].

  (b)

  Under-sampling: Objects are removed from the majority class. The goal is to have the same number of objects in each class. The basic solution of this method is random under-sampling. The disadvantage of using this method is that it can exclude a significant amount of the original data.

  (c)

  Hybrid-sampling: A combination of over-sampling and under-sampling. This approach generates objects for the minority class while it eliminates objects from the majority class.

• Algorithm level: The aim of this type of approach is a specific modification of the classifier. This approach is not flexible for different classification problems because it focuses on a specific classifier with a specific type of database. Nevertheless, the results could lead to good classification results for a particular problem. This type of solution can also combine strengths of different solutions as the NeuroFuzzy Model [37], which combines a fuzzy system trained as a neural network [31].
• Cost-sensitive: The objective is to create a cost matrix that is built with different misclassification costs. The misclassified objects of the minority class have a higher misclassification cost than the misclassified objects of the majority class. One of the main disadvantages is the cost-sensitive problem, which appears because the cost of misclassification is different for each of the classes. Therefore, this type of problem cannot be compared against non-cost-sensitive problems [38].

2.3. Pattern-Based Classifiers

Pattern-based classifiers can explain their results, through patterns, in a language closer to that used by experts, and the results have been shown to be more accurate than other popular classifiers, such as decision trees, nearest neighbor, bagging, boosting, SVM, and naive Bayes [25,26].

The stages of building a pattern-based classifier are:

• Mining stage: At this stage, several patterns are extracted from the training dataset. Mining contrast patterns is a challenging problem due to the high computational cost generated by the exponential number of candidate patterns [12,39,40,41].
• Filtering stage: At this stage, there are set-based filters and quality measures that need to distinguish between patterns that have a high discriminative ability for supervised classification [41]. The quality is usually established by measuring reliability, novelty, coverage, conciseness, peculiarity, diversity, utility, and actionability [42]. All the previous measurements take into account two parameters: if the pattern covers an object and if the object is representative of the class determined by the pattern.
• Classification stage: The last stage is the classification of query objects. At this stage, the classifier combines the patterns and creates a voting scheme. Finally, it is necessary to evaluate the performance of the classifier to determine its quality.

2.4. Fuzzy Logic

Fuzzy logic has been used as a term to refer to degrees of something [43], rather than a Boolean value (i.e., 0 or 1). It is helpful to handle imprecision, and it gives more flexibility in the systems [44]. In 1965, the concept of fuzzy logic theory was introduced by Zadeh [45]. The idea came while working on a problem with natural language, which is complicated to translate to binary values because not everything can be described as absolutely true or absolutely false.

The main purpose of this approach was to overcome many of the problems found in classical logic theory [46], especially the limitations presented with crisp values. With a fuzzy approach, the items have a membership grade value that goes from a lower limit to an upper limit for a given class A; typically, it goes in an interval from [0,1]. The lower limit (membership value zero) represents no membership to class A, while the upper limit (membership value one) represents full membership to class A. The other possible numbers mean that there is a partial degree of membership of the corresponding class. The membership grade is assigned by a membership function that possesses the quality of shape adaptability according to a specific need. Elements with zero degree of membership are usually not listed.

A fuzzy set is defined as:

$$\tilde{A}=\left\{(o,{\mu }_{\tilde{A}}\left(o\right))\right|oϵO\}.$$

where $\tilde{A}$ is the fuzzy set, O is a set of objects, o is the object, and ${\mu }_{\tilde{A}}\left(o\right)$ is the membership grade of o in $\tilde{A}$ [47].

As we mentioned before, fuzzy logic has the primary purpose of overcome the limitations presented with crisp values. This problem is extended to classifiers where the hard cut of the data can result in an abrupt classification of the objects. A common solution is to apply fuzzy logic to different stages of the classifier.

A fuzzy classifier calculates a membership vector $M=\{m_1,m_2,m_3,\dots ,m_n\}$ for an object o. The vector element $m_i\in [0,1]$ is the membership grade value of o in the class $C_i$ [48]. The object o is represented by a vector of t-dimensional features from a universe of discourse U and $C_1,C_2,C_3,\dots ,C_n$ is a set of given classes or classes to be discovered.

Since fuzzy set theory is a broader topic, more information can be consulted in [47]. It details definitions, algebraic operations with fuzzy numbers, fuzzy functions, and applications. We also suggest the work of Orazbayev et al. [49] and Nicolas Werro [50], where they explained in detail the process of developing complex systems in real-world environments with a fuzzy approach.

3. Pattern and Fuzzy Approaches for Imbalance Problems

In this section, we explain our research methodology, which includes the number of reviewed papers, the range of covered years, and the aim of the analyzed papers. Then, we list the most recent data-level and algorithm-level approaches for imbalance problems that use patterns or fuzzy approaches. Finally, we finish the section with a discussion of the reviewed papers.

For the reviewed papers, we perform an overview of their work that includes a summary of their method, the databases, the evaluation metrics, the compared methods and their results.

3.1. Research Methodology

For this review, we performed an in-depth review of papers that include fuzzy and/or pattern-based approaches for imbalanced databases. The search includes 62 documents from 2005 to 2021. The range was determined by the first work that covers fuzzy sets, pattern-based, and imbalance problems. We also include documents that have cited the work of Garcia-Borroto et al. [20]. This case was included because it was the first published work that combined fuzzy sets and pattern classification; unfortunately, it does not cover the imbalance problem. The results that cited Garcia-Borroto et al., showed 22 documents from 2011 to 2020. Therefore, a total of 84 documents were relevant in the scope of our research.

In recent decades, there have been many algorithms to address the data classification problems. However, in this paper, we reduce the scope to the approaches that include fuzzy sets, pattern-based, imbalance problems, and documents that have cited [20]. This decision was made so as to avoid confusion due to the number of methods or approaches that exist in the classification process.

3.2. Data- Level Approaches

In this subsection, we present the most recent data-level approaches presented for imbalance problems that use patterns or fuzzy approaches.

In 2017, Liu et al. [51] proposed a method to handle class imbalance and missing values called fuzzy-based information decomposition (FID). FID created artificial data of the minority class. Their scheme is divided into two parts, weighting, and recovery. The weighting process is made by the fuzzy membership functions. The functions are used to measure how much the data contributes to the missing estimation. For the recovery step, they consider the contribution of the data to determine the missing values. They used a C4.5 [52] classifier with an entropy-based splitting criterion. They used 27 databases combined from different repositories (e.g., PROMISE [53], UCI [54], KEEL [55]). The evaluation metrics used for their experiments are G-mean, AUC, and Matthews correlation coefficients (MCC) [56]. Their results outperform the following crisp methods: Mix [57], KNNI [58], SOM [59], SM [35], CBOS [60], CBUS [61], ROS [62], RUS [62], and MWM [60].

In 2018, Liu et al. [63] proposed a fuzzy rule-based oversampling technique (FRO) to solve the class imbalance problem. FRO uses the training data to create fuzzy rules and determine the weight of each rule. The weight of the rule represents how much the sample belongs to a fuzzy space. After the rules are created, they synthesize the artificial data according to those rules. Another contribution is that FRO can handle missing values with the best matched fuzzy rule, considering the correlation and difference between attributes. They used 55 real-world databases from the UCI [54] and KEEL [55] repository. They used AUC, recall, and F1-measure as their evaluation metrics. Their solution is compared against SMOTE [35], MWMOTE [64], RWO [65], RACOG [66], and the fuzzy method FID [51]. Their results showed that their technique is better or equal to the other compared methods and is robust in cases with overlapping.

In 2020, Pawel Ksieniewicz [67] proposed the construction of the support-domain decision boundary implementing the fuzzy templates method. Their goal was to avoid artificial data created with oversampling techniques. Their approach includes two solutions, one based on fuzzy templates, and the second one includes the standard deviation. They compare the results obtained by RDB, ROS-RDB, FTDB [68], and SDB. In RDB and ROS-RDB, the decision boundary is defined by a line equation. For FDTB, the decision boundary includes the mean support of each class. Finally, for SDB, they included the standard deviation. The databases used for the experiments are from the KEEL [55] repository and have an imbalance degree greater than 1:9 ratio with up to 20 features. They used F1-score, precision, recall, accuracy, and geometric mean to evaluate the performance of their solution. Their results claimed to improve the process of classification against oversampling methods by avoiding an overweight in the minority class.

In 2020, Ren et al. [69] proposed a fuzzy representativeness difference-based oversampling technique called FRDOAC. Their method uses affinity propagation and chromosome theory of inheritance. First, they use an oversampling method, fuzzy representativeness difference (FRD), which finds a representative sample of each class according to the importance of the samples. Then, they used Mahalanobis distance [70] to calculate the representativeness of every sample. In the last step, they created artificial data based on the chromosome theory of inheritance. The comparison was made against other crisp oversampling methods such as: SMOTE [35], MWMOTE [64], RWO [65], RAGOC, and GIR [71]. They used 16 databases from the UCI [54] and KEEL [55] repository. For the evaluation metrics, they used F-measure, G-mean, and AUC. Finally, the results showed better performance than the other methods.

In 2020, Kaur et al. [72] proposed a hybrid data level that overcomes some difficulties presented in different types of data impurities. They define data impurities as data that present noise, different data distributions, or class imbalance. Their approach has three different phases; in the first phase, they clean the data from noise and applied a radial basis kernel for clustering minority and majority classes. In the second phase, the clusters are artificially balanced; they used radial basis kernel fuzzy membership to reduce the majority class and firefly-based SMOTE [73] to increase the minority class. In the last step, they use a DT [52] as their classifier. They tested their method in three synthetic databases and 44 UCI [54] real-world databases. They used AUC to evaluate the performance of the method. They compared their results against ASS [74], ADOMS [74], AHC [75], B_SMOTE [76], ROS [77], SL_SMOTE [78], SMOTE [35], SMOTE_ENN [77], SMOTE_TL [77], SPIDER [79], SPIDER2, CNN [80], CNNTL [81], CPM [82], NCL [83], OSS [84], RUS [85], SBC [86], TL [81], and SMOTE_RSB, which has a fuzzy approach [87]. Finally, their results outperform the other methods especially in databases with a high imbalanced data ratio.

3.3. Algorithm Level

In this subsection, we present the most recent algorithm-level approaches presented for imbalance problems that use patterns and fuzzy logic.

In 2011, Garcia-Borroto et al. [20] proposed a fuzzy emerging pattern technique that overcomes the problem presented with continuous attributes using fuzzy discretization. To extract the fuzzy patterns, they used several fuzzy decision trees. For the induction procedure, they implemented linguistic hedges that solved the problem of the initial fuzzy discretization of continuous attributes. They created a way to test the discriminability of a fuzzy pattern named Trust. Finally, they proposed a new classifier (FEPC) that uses a graph-based strategy for pattern organization. The results outperform popular crisp classifiers such as bagging [88], boosting [88], c4.5 [52], random forest [89], and SJEP [90]; however, the results were similar compared to kNN and SVM, which were tested on 16 databases from the UCI repository [54].

In 2014, Buscema et al. [91] proposed K-Contractive Map (K-CM), which is a neural network that uses the variable connection weights to obtain the z-transforms. Then, they combined k-NN classifiers, which are used for class evaluation, and the connection weights are given by the Learning of the Auto-Contractive Map (Auto-CM) strategy [92]. The authors mentioned how K-CM helps bottom-up algorithms to provide a symbolic explanation of their learning. They also claimed that K-CM is not prone to over-fitting because it works with a complex minimization of the energy function with parallel constraints that includes all the input variable. The experiments were performed in 10 databases. Overall, K-CM showed the best performance when it was compared against k-NN [18], RF [93], PLS-DA [94], MLP [95], CART [96], Logistic [97], LDA [98], and SMO [99]. However, it has a similar performance with k-NN because it has the same modeling principles.

In 2016, Fernandez et al. [100] reviewed evolutionary fuzzy systems (EFSs) for imbalance problems. They introduce some concepts about linguistic fuzzy rules systems. They presented a taxonomy of the methodologies of that moment. Then, they present the most popular methods, until that year, to solve the imbalance problem (i.e., data level, algorithm level, cost-sensitive, and methods embedded into ensemble learning). Finally, to demonstrate the performance of EFSs, they implemented the GP-COACH-H [101] algorithm. The evaluation metric was G-mean and they compared their results against crisp C4.5 with SMOTE + ENN [77]. They used 44 databases from the KEEL repository [55]. In the end, the results are favorable for the EFSs and it outperforms the results of the SMOTE + ENN + C4.5.

In 2016, Fan et al. [102] proposed a fuzzy membership evaluation that assigns the membership value according to the class certainty. Their method is EFSVM and is inspired by FSVM [103]. The method is going to pay more attention to the minority class due to the entropy levels. Their method has three main steps: first, the entropy of the samples in the majority class is calculated. Then, the majority class is divided into subsets. Finally, each sample in the minority class is assigned to a larger fuzzy membership to guarantee the importance of the class. They compared their method, in three synthetic databases, against SVM [104] and the fuzzy method FSVM. Then, they compared their method on 64 real-world databases from the KEEL repository [55], against FSVM, SVM-SMOTE, SVM-OSS- SVM-RUS, EasyEnsemble [105], AdaBoost [106], and 1-NN. The results showed that EFSVM has better results than the compared algorithms in terms of AUC.

In 2017, Zhang et al. [107] introduced KRNN (k rare-class nearest neighbour), which is a modification of the k nearest neighbour (kNN) classifier [18]. KRNN creates groups of objects from the minority class and tries to adjust the induction bias of kNN in correspondence to the size and the distribution of the groups. kRNN also uses the Laplace estimate to adjust the posterior probability estimation for query objects.

In 2017, Zhu et al. [108] proposed an algorithm using an entropy-based fuzzy membership evaluation that enhances the importance of patterns. Their algorithm can handle the imbalance problem as well as matrix patterns. The algorithm is an entropy-based matrix pattern-oriented Ho–Kashyap learning machine with regularization learning (EMatMHKS), and it is based in the MatMHKS [109] algorithm. They divide the classification process into two steps: first, they determine the fuzzy membership for each pattern and then they used the obtained values to solve the criterion function. They used SMOTE [35], ADASYN [110], CBSO [111], EE [105], and BC [112] to preprocess the data (separately), and later they used each of the classifiers. For their experiments, they used ten real-world databases. They used accuracy, sensitivity, specificity, precision, F-measure, and G-mean as their evaluation metrics. They compared their algorithm against SVM, MatMHKS, and the three fuzzy methods FSVM, B-FSVM [113], and FSVM-CIL [114]. In the showed results, EMatMHKS has better overall performance in terms of the evaluation metrics, and it also has lower computational complexity than FSVM and B-FSVM.

In 2017, Pruengkarn et al. [115] proposed an approach to deal with imbalanced data by combining complementary fuzzy support vector machine (CMTFSVM) and synthetic minority oversampling technique (SMOTE). To enhance the classification performance, they used a membership function. The experiments were performed in four benchmark databases from KEEL [55] and UCI repositories [54], and one real-world data database. Their results are compared against crisp NN [116], SVM [16], and the fuzzy method, FSVM [114], showing that combining CMT undersampling technique and an oversampling technique presents the best results.

In 2018, Lee et al. [117] proposed a classifier that handles class imbalance and overlapping problems by geometrically separating the data. Their solution is based on fuzzy support vector machine and k-NN. First, they assign an overlap-sensitive weight scheme that used k-NN and different error cost algorithms. Then, they used an overlap-sensitive margin classifier (OSM) that separates the data into soft and hard overlap regions. For the soft region, the data is classified with the decision boundary of the OSM. For the hard region, they classified the data with 1-NN. Finally, it classified the data using the decision boundary of the separated regions. Their method was tested in 12 synthetics databases and 29 real-world databases from the KEEL [55] repository, where they were evaluated with geometric mean and F1 as their evaluation metrics. Their algorithm was compared against normal SVM, under + SVM, SMOTE + SVM, SDC [118], 1-NN [119], and Boosting SVM [120]. They also compared their algorithm against FSVM-CIL cen [114], FSVM-CIL hyp [114], EFSVM [102], and EMatMHKS [108], in which all of them are fuzzy solutions. All the results were favorable for their proposed method.

In 2019, Gupta et al. [121] proposed two variants of entropy based fuzzy SVM (EFSVM) [102], one that uses least squares, called EFLSSVM-CIL, and another that uses twin support called entropy based fuzzy least squares twin support vector machine (EFLSTWSVM-CIL). In both cases, the fuzzy membership is calculated with the entropy values of each sample. Their algorithms use linear equations to find the decision surface, in contrast with other similar methods that use quadratic programming problem. The performance of the algorithms is evaluated with AUC and the databases used are from the KEEL [55] and UCI [54] repository. The results were tested in linear and non-linear cases, where they compared their methods against TWSVM [122], FTWSVM [122], EFSVM, and NFTWSVM [123], where the last three are fuzzy solutions. Their results showed that EFLSTWSVM-CIL outperforms the other classifiers in terms of learning speed.

In 2019, Arafat et al. [124] proposed an under-sampling method with support vectors. They combined support vector decision boundary points from the majority class with an equal number of points from the minority class. After they obtained a balanced dataset, they used SVM [125] as the base learner for new instances, and they tested their method against C4.5 [52], Naive Bayes [126], Random forest [127], and AdaBoost [128]. The used 13 databases were from the KEEL repository [55]. In their results, they showed that their method outperforms the other classifiers.

In 2019, Liu et al. [129] proposed an algorithm based on adjustable fuzzy classification with a multi-objective genetic strategy based on decomposition. Their algorithm is called AFC-MOGD, which generates fuzzy rules and then those rules are optimized to get new fuzzy rules. They used an optimization algorithm based on decomposition with the purpose of designing a new pattern. They consider the class percentage to determine the class label and the rule weight to get more viable rules. They used 11 databases from the KEEL [55] repository and for the evaluation metric, they used area under the ROC convex hull (AUCH). They compared their algorithm against MOGF-CS [130], C4.5 [52], and E-Algorithm [131]. Their results showed they the new implementation outperforms other classifiers.

In 2019, Cho et al. [132] proposed an instance-based entropy fuzzy support vector machine (IEFSVM). The fuzzy membership of each sample is determined by the entropy information of kNN, which focuses on the importance of each sample. They used polar coordinates to determine the entropy function of each sample. They take into consideration the diversity of entropy to increase the size of the neighbors k. They used 35 databases for the UCI [54] repository and 12 real-world databases. They used AUC to evaluate their performance against SVM [16], undersampling SVM, cost-sensitive SVM, fuzzy SVM [114], entropy fuzzy SVM [102], cost-sensitive AdaBoost [133], cost-sensitive RF [93], EasyEnsemble [105], RUSBoost [85], and weighted ELM [134]. In their results, they proved that IEFSVM has a higher AUC value than the other methods and the results are less sensitive to noise.

In 2019, Sakr et al. [135] proposed a multilabel classification for complex activity recognition. They used emerging patterns and fuzzy sets. Their classification process uses a training dataset of simple activities, where they extract strong jumping emerging patterns (SJEPs). Then, their scoring function uses SJEPs and fuzzy membership values, which result in the labels of the existing activities. They assumed that all the activities are linearly separated. They used three evaluation metrics: precision, recall, and F-measure. The results outperform the other compared crisp methods (i.e., SVM [16], k-NN [18], and HMM [136]).

In 2019, Patel et al. [137] proposed a combination of adaptive K-nearest neighbor (ADPTKNN) and fuzzy K-nearest neighbor. The adaptive K-nearest neighbor part is used to overcome the difficulties presented in fuzzy K-nearest neighbor when it deals with imbalanced databases. Their new addition chooses different values of k for different classes according to their sizes. They tested their work in: ten databases, different values of K (i.e., 5-25 in steps of 5), and evaluates F-measure, AUC, and G-mean. They compared their algorithm (Fuzzy ADPTKNN) with NWKNN [138], Adpt-NWKNN [139], and Fuzzy NWKNN [140]. In most of the presented cases, their proposed method performs better than the other approaches.

In 2019, García-Vico et al. [42] proposed an algorithm for big data environments (BD-EFEP). They claimed it is the first multi-objective evolutionary algorithm for pattern mining in big data. BD-EFEP has the objective of extract high-quality emerging patterns that describes the discriminative characteristics of the data. BD-EFEP follows a competitive-cooperative schema, where the patterns compete with each other, but they cooperate to describe the space’s greatest possible area. To improve efficiency in evaluating the individual without losing quality, they used a MapReduce-based global approach or a token competition-based procedure. In conclusion, BD-EFEP obtains a set of patterns with an improvement in the trade-off between the generality and the reliability of the results. These patterns are extracted faster than other approaches, which makes the algorithm relevant for big data environments.

In 2019, García-Vico et al. [141] presented the effects of different quality measures that are used as objectives for multi-objectives approaches, focused on the extraction of emerging patterns (EPs) in big data environments. EPs are a type of contrast patterns extracted by using the quality measure Growth Rate, which is computed using the mean of the highest ratio between the support of a certain pattern in one class and the support of the same pattern in the other classes [41]. They used eight combinations of quality measures: Jac and TPR, G-mean and Jac, Jac and FPR, G-mean and WRacc, Jac and WRacc, SuppDiff and Jac, TPR and FPR, and WRAcc and SupDiff [142]. They performed the experiments in six large-scale databases from the UCI repository [54]. The number of instances goes up to 11 million, but the number of variables is low. They used the BD-EFEP [42] algorithm, which is one of the two algorithms for the extraction of EPs (i.e., EvAEFP-Spark [143] and the BD-EFEP). In their results, they showed that the combination of TPR and FPR, G-mean and Jac, and Jac and FPR, are the most suitable quality measures for the descriptive rules in emerging pattern mining. Nevertheless, Jac and FPR is shown as the best trade-off in the descriptive rules for EPs.

In 2019, Luna et al. [144] proposed three different subgroup discovery approaches for mining subgroups in multiple instances problems. The proposed approaches were based on an exhaustive search approach (SD-Map [145]), an evolutionary algorithm based on grammar-guided genetic programming (CGBA-SD [146]), and an evolutionary fuzzy system (NMEEF-SD [147]). Those approaches are adapted to multiple instance problems. The approaches were tested in two different steps: first, they used ten benchmarks representing real-world applications, and then, 20 artificial databases generated by themselves. Their results showed that SD-Map-MI has poor performance with low quality and bad trade-off between sensitivity and confidence. CGBA-SD-MI has a good trade-off between sensitivity and confidence, as well as good performance in the different databases. NMEEF-SD-MI also has rules of low quality and bad results on confidence and low sensitivity; however, some databases present a confidence level higher than 83%. Finally, they conclude that they need to keep performing more tests because it can be a promising approach to medicine or Bioinformatics problems.

In 2020, García-Vico et al. [148] proposed a method based on an adaptive version of the NSGA-II algorithm. They proposed a cooperative-competitive multi-objective evolutionary fuzzy system called E2PAMEA. The algorithm is used to extract emerging patterns in big data environments. E2PAMEA uses an adaptive schema that selects different genetic operators. The genetic operators used for diversity are two-point and HUX crossover. Similar to their previous work [42], they used a cooperative-competitive schema along with a token competition-based procedure for reliable results. Finally, E2PAMEA outperforms previous approaches showing that their results are more reliable, and the processing time and physical memory are also improved.

In 2020, Liu [21] presented an extension of the fuzzy support vector machines for class imbalance learning (FSVM-CIL) method. The idea is to overcome the limitation of FSVM-CIL for imbalanced databases with borderline noise (i.e., between-class borderline). To solve this limitation, a new distance measure and a new fuzzy function are introduced. The distance measure characterizes the distance of different data points to the hyperplane, and the Gaussian fuzzy function reduces the impact of the borderline noise. The experiments were performed on several public imbalanced datasets. The results show an improvement over the original FuzzySVM-CIL [114], SVM [16], CS-SVM [149], and SMOTE-SVM [150].

In 2020, Gu et al. [151] proposed a self-adaptive synthetic over-sampling approach (SASYNO). They synthesize data samples near the real data samples, giving priority to the minority class. Their method balances the classes and boosts the performance. They claimed it could be applied to different base algorithms such as SVM, k-NN, DP, rule-based classifiers, DT, and others. They used four databases for their experiments, and the results showed that their approach enhances the overall performance in terms of specificity, F-measure, and overall ranks. The classifiers where they applied their approach were SONFIS [152], SVM [125], and k-NN [153].

In 2020, García-Vico et al. [154] presented a preliminary many objective algorithm for extracting Emerging Fuzzy Patterns (ManyObjective-EFEP). They combined fuzzy logic, for soft-computing, with NSGA-III [155]; the purpose is to increase the search space in emerging pattern mining. The algorithm is compared against an adaptation of NSGA-II [156] and they used 46 databases from the UCI repository [54]. The quality measures used are nP, nV, WRACC, CONF, GR, TPR, and FPR [142]. They presented results of a set of patterns with more interpretability and precision. They conclude that the algorithm could be interesting for real-world applications. However, they need to continue the study and analysis due to the complex space and the trade-off among different quality measures.

Finally, in

Table 1. Algorithm-level approaches with their key merit(s) and disadvantage(s).

Year	Ref.	Key Merit(s)	Disadvantage(s)/Improvements
2011	[20]	-Proposed a fuzzy emerging pattern technique. -Proposed the FEPC classifier.	-Fuzzy emerging patterns are highly dependent on the quality measure Growth Rate.
2014	[91]	-Proposed K-Contractive Map (K-CM).	-Similar performance with a classical k-NN classifier.
2016	[100]	-A study in evolutionary fuzzy systems (EFSs) for imbalance problems.	-A taxonomy of the reviewed methods is missing.
2016	[102]	-Presented EFSVM. A fuzzy membership evaluation that assigns the membership value according to the class certainty.	-Unfortunately, they do not mention possible improvements in their work.
2017	[107]	-Proposed KRNN based on a KNN classifier.	-KRNN could be extended to multiple classes.
2017	[108]	-Proposed EMatMHKS. An algorithm with an entropy-based fuzzy membership and based in MatMHKS.	-An improvement in their function to measure the entropy-based fuzzy membership.
2017	[115]	-Proposed a combination of CMTFSVM and SMOTE. entropy-based fuzzy.	-A comparison with other popular classifiers can enrich the results.
2018	[117]	-Proposed a classifier based on OSM, and inspired on FSVM and k-NN. It geometrically separates the data to solve the imbalance problem.	-It uses a 1-NN algorithm for the hard-overlapping regions, which can result in a high generalization error.
2019	[121]	-Proposed two variants of EFSVM. One uses least squares and the other one uses twin SVM.	-The results could improve with the implementation of heuristics solutions in the method for parameter selection.
2019	[124]	-Proposed a method to generate balanced data with support vectors.	-The usage of real-world data can enrich the results of their method.
2019	[129]	-Proposed AFC-MOGD. An algorithm based on adjustable fuzzy classification with a multi-objective genetic strategy.	-The usage of real-world data can enrich the results of their method.
2019	[132]	-Proposed an instance-based EFSVM.	-The selection of different neighborhood sizes could improve the results due to the better knowledge of the distribution of the data.
2019	[135]	-Proposed a multilabel classification for complex activity recognition.	-A comparison against other fuzzy classifiers would enrich the results.
2019	[137]	-Proposed Fuzzy ADPTKNN. It combines ADPTKNN and fuzzy k-NN.	-The method could be extended to feature-based NN.
2019	[42]	-Proposed BD-EFEP. An algorithm for big data environments.	-Additional comparisons against big data environments can enrich the results.
2019	[141]	-Presents the effects of different quality measures in patterns, focused on Big Data Environments.	-New approaches for efficient extraction of patterns in big data environments are needed.
2019	[144]	-Proposed three approaches for mining subgroups in multiple instances problems.	-More tests are needed to improve their results and to determine the imbalance ratio in which the method is more suitable.
2020	[148]	-Proposed an adaptive version of NSGA-II.	-An optimization of the fuzzy sets could improve the results.
2020	[21]	-Proposed an extension of FSVM-CIL with a new distance measure and a new fuzzy function.	-A comparison against more fuzzy classifiers would enrich the results.
2020	[151]	-Proposed SASYNO. A self-adaptive synthetic over-sampling approach.	-The usage of more databases would enrich the results.
2020	[154]	-Presents a preliminary many objective algorithm for extracting Emerging Fuzzy Patterns.	-The usage of real-world databases would enrich the results.

, we show key merit(s) and disadvantage(s) or possible improvements for each proposal presented in Section 3.3.

3.4. Discussion

In the reviewed documents, the proposed methods with fuzzy pattern-based classifiers have shown better classifier results than other non-fuzzy approaches. Despite the favorable results, fuzzy pattern-based classifiers have not been studied more in detail. To the best of our knowledge, the only classifier based on fuzzy patterns is that proposed in [20]. Additionally, there are no fuzzy pattern-based classifiers specifically designed to deal with class imbalance problems and the approaches that are closer to this type of problem have not been studied enough.

Fuzzy approaches have achieved those results for different reasons; one of them is that the results are similar to the language used by experts. This solution is due to the use of linguistic hedges (e.g., “very”, “less”, “often”) that overcome the problems presented with the discretization of continuous features. Another characteristic is the flexibility the offers the fuzzification of the data. The fuzzification gives a wider window of opportunity in different classification steps (i.e., mining process, filtering process, variable window size).

4. Applications Domains

In this next section, we present the papers that the research was used directly into an application domain. As we see, most research was conducted in the field of medicine, followed by that of the financial market.

In 2017, Gerald Schaefer [157] showed strategies to handle the class imbalance problem in digital pathology. In the paper, it is showed how fuzzy rule-based algorithms can use cost-sensitive functions to include misclassification cost and how to use ensemble classification methods for imbalanced data. They evaluate the methods in the AIDPATH project [157] and are waiting to get results to make them public.

In 2018, Jaafar et al. [158] proposed a classifier named Mahalanobis fuzzy k-nearest centroid neighbor (MFkNCN). First, they calculate the k-nearest centroid neighborhood for each class. Then, they used a fuzzy membership to designate the correct class label of the query sample. They used a real-world database from the Shiraz University, where it has the adult heart sound of 112 subjects (79 healthy people and 33 patients). They used accuracy as their performance metric. They compared their classifier against kNN, kNCN, and two fuzzy approaches, FkNN and MFkCNN. Their results showed that MFkNCN outperforms the other classifiers.

In 2018, Kemal Polat [159] proposed a data preprocessing method to classify Parkinson, hepatitis, Pima Indians, single proton emission computed thoracic heart, and thoracic surgery medical databases. Their method is divided into three main steps. First, they used k-means, fuzzy c-means, and mean shift to calculate the cluster centers. Secondly, they calculated the absolute differences between each attribute and the cluster centers; the average of the differences of each attribute is also calculated. In the last step, they proposed three attribute weighting methods to reduce the variance within the class. If the similarity rate is low, the coefficient will be be high, and if the similarity rate is high, the coefficient will be low. Their method is compared against random sampling and they used Linear discriminant analysis, k-NN, SVM [16], and RT as classification methods. The databases are from the UCI [54] repository. The performance is evaluated with accuracy, precision. Recall, AUC, j-value, and F-measure. Finally, their results showed that their approach has better results than random sampling method. They claimed that the three proposed attribute weighting methods could be used in signal and image classification.

In 2019, Cho et al. [160] proposed an investment decision model for a peer-to-peer (P2P) lending market based on the instance-based entropy fuzzy support vector machine (IEFSVM). The idea of this algorithm is based on the necessity of an effective prediction method that supports decisions in the P2P lending market. First, for the training set, they select the desired features and train the loan status with IEFSVM. Secondly, for the test set, they remove the unselected features and predict the loan status using the same IEFSVM. In the end, the method selects the loans classified as fully paid. The database used is from the Lending Club, which is specialized in P2P loans. They used AUC, precision, predicted negative condition rate as their evaluation metrics. They compared their method against Cost-sensitive adaptive boosting [133], cost-sensitive random forest [93], EasyEnsemble [105], random undersampling boosting [85], weighted extreme learning machine [134], cost-sensitive extreme gradient boosting [161], and one fuzzy approach, EFSVM [102]. Their results showed that their method outperforms the other classifiers for loan status classification.

In 2019, Li et al. [162] proposed a Bayesian Possibilistic C-means (BPCM) clustering algorithm that provides a preliminary cervical cancer diagnostic. They explained that the motivation of the research is due the high lethality and morbidity of the disease. The information used in the diagnostic process could be highly private and many patients decided not to provide this information. To solve the lack of attributes for the classification, they look for patterns from the complete data, and then, they estimated the missing values with the closest representative pattern. In the first stage, they used a fuzzy clustering approach that models the noise and uncertainty of the database. During the second stage, they use a Bayesian approach to get the cluster centroids which can be translated as the representative. They used accuracy and sensitivity as their evaluation metrics. They tested their algorithm in a database with 858 patients. The proposed solution can identify the patterns and is able the predict the missing values with an accuracy of 76%

In 2020, Ambika et al. [22] proposed a method that looks in medical databases to predict the risk of developing hypertension; as they mentioned, most medical databases have the class imbalance problem. Their method is divided into four phases: data collection, data preparation, learning model, and data exploration. For the data collection, they used databases from primary health centers. In the preparation phase, they used mean substitution [163] for missing values and interquartile range [164] to detect outliers. The learning model uses SVM with AdaBoost [165]. Finally, for the exploration phase, they used fuzzy-based association mining rule. They used precision, recall, accuracy, and F-measure to evaluate their performance. They compared their results against ANN, IBK, and NaiveBayes. Their results showed an improvement in medical diagnosis, but they said that it needs further work in classification errors.

In 2020, García-Vico et al. [166] proposed an algorithm, called FEPDS, to extract fuzzy emerging patterns in Data Stream environments. The algorithm can be described in three main steps, a recollection, an adaptive learning step, and a learning multi-objective evolutionary algorithm. First, they collected instances by batches and once the batch is collected, the learning method updates the fuzzy patterns model according to the current batch. Then, they used a multi-objective evolutionary learning algorithm that extracts pattern models according to simplicity and reliability. Finally, the algorithm is continuously evaluating the instance against unseen instances to determine the quality of the new information acquired. The algorithm was tested in the profiles of customers of New York City taxis corresponding to the fare amount; it was also included recurrent behaviors such as payment by card on medium or high fares and the daily travel to work of the citizens. They conclude that FEPDS can adapt the model to abrupt changes without losing performance in terms of execution time. The algorithm also has great scalability and performance in terms of memory consumption, allowing its usage in real-world environments.

5. Taxonomy

In the reviewed papers, we classified the type of approach and presented a table (

Table 2. Taxonomy of similarities and differences among the reviewed papers.

Application	Approach	Refs.	Advantage	Disadvantage
Theoretical	Data level	[51,63,67,69,72]	Data is artificially manipulated to deal with the imbalance problem.	The data created can lead to a bias in the classification process.
	Algorithm level	[20,21,42,91,100,102,107,108,115,117,121,124,129,132,135,137,141,144,148,151,154]	The results from the experiments tend to have positive results due to the fitting process of the problem.	Each solution solves the imbalance problem in their own scenario, which does not present a general solution for most cases.
	Cost-sensitive	-	-	-
Medicine	Data level Algorithm level	[22]	The used methods have been deeply studied and have better results against other learning techniques.	They need to enhance how they handle classification errors.
	Algorithm level	[158]	The usage of Mahalanobis distance in combination with fuzzy kNN, improves the results of normal kNN.	The results are limited to the specific case.
	Data level Algorithm level	[159]	The approach can be used in diseases, signal, and image classification	Optimization could be improved.
	Algorithm level	[162]	The author mentioned that there are room for improvements, and they have promising results.	Accuracy and sensitivity are somehow low, 76% and 79% respectively.
	Data level Cost-sensitive	[157]	Improved robustness and classification performance in contrast of a single lassifier.	The results are not known, but they will report them in future publications.
Financial	Algorithm level	[160]	Robust results for the P2P lending market.	The decision model could be considered too simple.
Urban planning	Algorithm level	[166]	The model can adapt to abrupt changes and has great scalability.	It is unknown how it will perform outside the pilot city.

) with some advantages and disadvantages. The table is divided into theoretical and real-world applications, and subdivided into the three approaches commonly used to deal with class imbalance problems [15,31,34]. This division was used to be concise in the common categories found in the literature and follow the same methodology that we used for the systematic review of the state of the art for class imbalance problems on fuzzy and pattern-based classification (i.e., theoretical and practical approaches). Another advantage is to quickly identify the type of approach with its benefits and drawbacks.

We create a taxonomy that is a summary of the similarities and differences among the different approaches. Most of the papers are focused on the algorithm level due to the possible improvement in their results. However, those approaches are limited to the specific problem to solve. In real-world applications, most of them are related to medicine, with some exceptions in finance and urban planning. Additionally, in real-world applications, it is common to combine different approaches to deal with the problem ahead to get the most of each approach (i.e., data level and algorithm level been to most common combination).

Figure 1 shows the results of the subject area of the 62 papers that cover fuzzy sets, pattern-based, and imbalance problems without real-world applications. Figure 2 shows the number of publications by year.

As is expected, the three main subject areas are Computer Science, Mathematics, and Engineering. Additionally, Figure 2 shows the increase in published documents in the field, where 2018 was the year with the higher number of publications across the 15 reviewed years.

6. Future Directions

From the studies reviewed in this paper, we can see that a prominent approach to solve the class imbalance problem is by using fuzzy classification due to the flexibility that fuzzy approaches provide. Nevertheless, the approaches previously reviewed are extensions of previous solutions. Pattern-based solutions have positive classification results, but they have been limited to use emerging patterns only, which contains several limitations [142].

Analyzing the results of the reviewed work and exploiting the advantages of each approach, we infer that fuzzy pattern-based classifiers specifically designed to deal with class imbalance problems could achieve better results with a statistically significant difference than other popular state-of-the-art fuzzy and pattern-based classifiers.

To carry out this fuzzy pattern-based solution, better algorithms for mining fuzzy patterns in class imbalance problems and new fuzzy pattern-based classifiers for class imbalance problems are still needed. We proposed the following strategies on each stage of the pattern-based classification (i.e., mining, filtering, and classification) to improve the result when dealing with imbalanced data.

In the mining stage, it is needed to extract fuzzy patterns that are representatives of all classes. One approach is by inducing diverse fuzzy decision trees, which has shown better results than other contrast pattern mining approaches [20]. Then it is needed to analyze whether the extracted fuzzy patterns are representatives of all classes. Further, it is important to analyze the sources of bias, mainly focusing on strategies to mitigate fragmentation data and noise.

In the filtering stage, it is needed to obtain a representative subset of patterns for all problem’s classes. The subset needs to be, at least, as good to classify as the set containing all the fuzzy patterns. To obtain the subset, the implementation of strategies based on weights assigned to fuzzy patterns, considering the class imbalance ratio, could be used; with this improvement, the problem of high computational cost generated by the exponential number of candidate patterns [12,39,40,41] could be reduced. Another possible improvement is modifying or developing a quality measure for fuzzy patterns that could improve the extracted patterns.

In the classification stage, the voting scheme needs to be adapted for fuzzy pattern-based classification. To improve this step, we could consider weighting the objects in consequence of the class imbalance ratio presented by the training dataset. For the classifier, the result of each stage is of significant importance due to the interconnected nature of all the stages.

Finally, to the best of our knowledge, fuzzy pattern-based classification has not been studied in detail, and it is a good area of opportunity in the research field.

7. Conclusions

In this paper, we review and analyze the current state-of-the-art of classifiers that involved fuzzy techniques and pattern-based classifiers for solving class imbalance problems. This research was motivated by the renewed interest of the machine learning community in creating explainable artificial intelligence models and the wide interest in applying this type of model to the real world.

Therefore, in this paper, we performed an in-depth review of current state-of-the-art techniques. We reviewed 84 papers that include fuzzy and/or pattern-based approaches for imbalanced databases, as well as papers that have cited the work of Garcia-Borroto M et al. We also included the most common applications domains. Next, we present a taxonomy that includes the type of approach with its advantages and disadvantages. Then, we include future directions of the field according to the reviewed papers. As a final part of the review, we provide the conclusions and some advantages and disadvantages of all the reviewed approaches.

• Advantages:

  -

  Fuzzy and pattern-based approaches attract interest from the research community.

  -

  Fuzzy logic is widely used for its flexibility and understandability of the results.

  -

  Medicine is an area where the imbalance problem is constantly presented and uses the newest techniques.

  -

  Techniques that include fuzzy approaches have shown better classification results in comparison to other classifiers based on non-fuzzy approaches.

  -

  Fuzzy pattern-based approaches are a promising solution to handle the imbalanced data problem. However, this type of classifier should be studied further.

• Disadvantages:

  -

  Despite the flexibility of fuzzy approaches, they can lead to repetitive solutions that are small variations of other ones.

  -

  The quality of fuzzy patterns is highly dependent on the quality of the features of the fuzzification process.

  -

  Fuzzy emerging patterns are highly dependent on the quality measure Growth Rate, which could not provide good patterns as stated in [142] .

  -

  The combination of fuzzy and pattern-based approaches has not been studied in detail, so some research can lead to a dead end.

In summary, we present a survey that covers an in-depth review of papers from 2005 to 2021 that include fuzzy sets, pattern-based, imbalance problems, and documents that have cited [20]. Then, we present our analysis and show possible future directions. Finally, we finish with a conclusion of the solutions to classification for imbalanced data.