Interpretable Lightweight Ensemble Classification of Normal versus Leukemic Cells

Abstract

The lymphocyte classification problem is usually solved by deep learning approaches based on convolutional neural networks with multiple layers. However, these techniques require specific hardware and long training times. This work proposes a lightweight image classification system capable of discriminating between healthy and cancerous lymphocytes of leukemia patients using image processing and feature-based machine learning techniques that require less training time and can run on a standard CPU. The features are composed of statistical, morphological, textural, frequency, and contour features extracted from each image and used to train a set of lightweight algorithms that classify the lymphocytes into malignant or healthy. After the training, these classifiers were combined into an ensemble classifier to improve the results. The proposed method has a lower computational cost than most deep learning approaches in learning time and neural network size. Our results contribute to the leukemia classification system, showing that high performance can be achieved by classifiers trained with a rich set of features. This study extends a previous work by combining simple classifiers into a single ensemble solution. With principal component analysis, it is possible to reduce the number of features used while maintaining a high accuracy.

1. Introduction

Leukocytes are some of the types of cells that compose the human blood. Leukemia is a disease that affects the function and shape of leukocytes and can occur in a chronic or acute form. Acute leukemia is more aggressive, has more intense symptoms, and evolves quickly. Lymphocytes, a type of leukocyte, are called lymphoblasts in their immature form. Acute lymphoblastic leukemia (ALL) is a type of cancer characterized by the accumulation of lymphoblasts within the bone marrow. ALL is the most common childhood leukemia, mainly affecting children between 3 and 7 years old, and 75% of diagnoses occur before the age of 6 [1]. According to the Brazilian Institute of Cancer (INCA), in the year 2020, leukemia killed about 6738 people. An early and more accessible diagnosis could save many of these lives [2].

Usually, leukemia diagnoses are done by microscopic analyses of blood smears. The diagnosis depends on the hematologist’s expertise in distinguishing malignant from healthy lymphocytes. Pattern recognition, combined with image processing techniques, has been used in blood analyses to produce computer-aided diagnosis (CADx) systems that aim to improve the lymphocyte classification performance [3,4].

The Acute Lymphoblastic Leukemia Image Database (ALL-IDB) [5,6] for image processing provides a set of annotated images that can be used in the evaluation of classifiers of ALL cells. This initiative provides two different datasets: ALL-IDB1, consisting of 108 blood smear pictures collected from healthy and leukemic patients containing 510 single lymphocytes; ALL-IDB2, a collection of cropped areas of interest of normal; and malignant lymphocytes that belong to the ALL-IDB1 dataset.

Many studies have assisted hematologists in analyzing blood smear images for ALL recognition. Some of these attempts have considered aspects of lymphocytes such as color, and textural and morphological features. The ALL-IDB dataset was used to train classification models based on techniques such as support vector machines (SVM), k-nearest neighbors (KNN), random forests, and ensemble classifiers. More recent approaches use deep learning algorithms such as convolutional neural networks (CNN) to build models to solve this problem.

Putzu et al. [7] proposed a leukocyte classification method using image features such as color, texture, and shape. An SVM, a KNN, and decision tree models were trained using these features to classify leukocytes and detect malignant cells. The dataset used in his work was ALL-IDB1. Accuracy of 93.63% was achieved using an SVM with a radial basis function kernel to analyze 267 leukocytes of the dataset.

Mishra et al. [8] presented a CADx system for detecting leukemia using the features extracted from a discrete cosine transform (DCT) of grayscale lymphocyte images. This paper proposed using DCT values with an SVM for the lymphocyte classification. The ALL-IDB2 dataset was used in the k-fold cross-validation strategy to split the data into training and testing datasets. His system achieved an accuracy of 89.76%.

MoradiAmin et al. [9] presented a CADx to distinguish between healthy and diseased cells. The proposed system aggregates first and second-order statistical, morphological and geometric features extracted from the nucleus images. These feature sets are used to train an SVM using different kernels. After training, these classifiers are ensembled to build a single classifier based on majority voting. The research team used a private image dataset of 958 lymphocyte images (315 healthy and 643 malignant) divided into test and training datasets. Finally, the author used the k-fold cross-validation strategy to achieve an accuracy of 96.37%.

Shafique and Teshin [10] deployed an AlexNet, a pre-trained classifier, to achieve an accuracy of 99.5%. The authors used the ALL-IDB2 dataset and increased the number of images from 260 to 760 (500 malignant and 260 healthy) using mirroring and rotation operations. The augmented dataset was split into training and test sets, with 60% and 40% of the images respectively.

Moshvash et al. [4] developed a system that used a set of features composed of 32 textural, 15 shape, and six color descriptors. The texture data such as energy, correlation, homogeneity and contrast were extracted from a gray level co-occurrence matrix (GLCM). Each matrix component indicates the probability of two pixels having particular gray levels at a particular spatial relationship. These features were used to train naive Bayes (NB), KNN, decision tree and SVM models. Then these classifiers were combined into an ensemble classifier. The ALL-IDB1 dataset was used to achieve an accuracy of 98.10%.

Mourya et al. [11] used DCT features and a CNN to distinguish malignant and healthy cells in a hybrid classifier called Leukonet. To train this architecture, they developed a dataset with 9211 cancer cells from 65 subjects and 4528 healthy cells from 52 subjects; these images are separated into different folders. Different subjects were divided into training and validation sets. They achieved an accuracy of 89.70% and an F1-score of 91.95% for the cancer cell class.

Many works were developed using small datasets. The use of the ALL-IDB dataset, with only 510 lymphocyte samples, has been commonplace. There are many feature extraction-based approaches. However, none of these studies have combined these features in a single feature vector, and no methodology has ensembled a feature-based neural network with other classifier algorithms. CNNs or deep convolutional architectures are widely used and achieve high accuracy but require considerable amount of training time and more specific, restricted, and expensive hardware. According to Dongyu [12], a convolutional neural network training trial may take a couple of weeks. This time can amount to months of searching for different parameters.

Garcia et al. [13] showed the benefits of combining various classification models to improve the results in complex and imbalanced datasets. This combination of models improves the results as long as the results of individual models are not too close to perfection. Many works combine different algorithms to achieve a single ensemble solution. For breast cancer detection, Abdar [14] proposed a CWV-BANN-SVM that combined two SVM and a boosting ANN to achieve an accuracy of 100%. Hsieh [15] developed, for the same problem, an algorithm that combines a neuro-fuzzy, KNN, and a quadratic classifier to obtain a 97.17% accuracy. Neuro-fuzzy refers to a combination of an artificial neural network and fuzzy logic. These studies presented ensemble classifiers with high-performance using lightweight classifiers. On the other hand, Moon et al. [16] presented a CADx system to classify breast ultrasound images using an ensemble of different deep CNN architectures, including VGGNet, ResNet and DenseNet, obtaining excellent results on different datasets.

Several studies use ensemble classifiers to solve the leukocyte classification problem and similar problems. However, many of these studies combined heavyweight classifiers [11,17,18], increasing the training time even more, and increasing the need for high-performance hardware. Approaches that combine lightweight algorithms on an ensemble classifier generally use the same algorithm with different configurations [4] or only classical classifiers [9] without exploring a lightweight artificial neural network (ANN) on the ensemble solution.

In a previous study [19], we proposed a lightweight neural network classifier to classify images of lymphocytes into malignant or healthy. The classifier used as input a feature vector with 108 low-order statistical, 20 morphological, 75 textural, 1024 DCT, and 160 contour features extracted from the lymphocyte images.

This study extended that previous work [19] by combining the neural network with three traditional lightweight classifiers—KNN, SVM, and NB—into an ensemble classifier to improve the classification results. The methodology achieved state-of-the-art performance using a fraction of the computational time cost necessary to train a convolutional neural network. Finally, a principal component analysis (PCA) was used to select the most important features, and to test the possibility of reducing the number of features required in order to reduce the classifier’s complexity.

2. Materials and Methods

The proposed methodology comprises two steps: feature extraction and ensemble classifier.

In the feature extraction step, we extracted low-order statistical, morphological, DCT, and contour data from images of lymphocytes. These features were combined into a single feature vector, normalized, and used to train a set of classifiers. The feature extraction step is described in Section 3.

The ensemble classifier uses ANN, KNN, SVM, and NB classifiers. The ANN approach was selected from a combination of networks evaluated in a grid search. The best ANN solution was fine-tuned to obtain the final neural network. The KNN, SVM, and NB were trained with the same feature set as the ANN. Finally, we used a PCA to find the most relevant features to the classification process, as an attempt to obtain a light and interpretable model. With these features, a new and simpler classifier was obtained with performance similar to the complete one. The ensemble classifier step is described in Section 4.

The implementation of this methodology is publicly available online at https://github.com/yurifarod/ISBI-2019 (accessed on 8 June 2022). The code was developed in Python 3.7 using machine learning libraries such as Tensorflow, Keras, and Mlxtend; the image processing libraries OpenCV, PIL, and Pyradiomics [20]; and many other libraries responsible for data processing and manipulations, such as Numpy, Scipy, Pandas, CSV, Os, Multiprocessing, Queue, and Timeit.

The dataset used was the publicly available C-NMC 2019 dataset, which is described in the Section 2.1. We used a data augmentation strategy to balance the training and validation image sets, as described in Section 2.2.

2.1. C-NMC 2019 Dataset

The dataset used in this study was provided by the research team of SBILab [21]. The C-NMC 2019 dataset [22] consists of 15,114 images of lymphocytes collected from 118 subjects. These images were split into a training, preliminary, and test sets. Each image set contains single-cell images of healthy or malignant lymphocytes previously labeled by a team of oncologists.

The cells were dyed using the Jenner–Giemsa stain technique [23]. The SBILab team preprocessed these images using segmentation, image enhancement, and normalization techniques [24,25,26]. Individual lymphocytes were segmented from blood smear images and placed in the center of them; each picture has 450 × 450 pixels and a black background. Figure 1 shows samples of both healthy and malignant cells from this dataset.

The data of the final test set were unlabeled and can be used to submit results to the website of the “C-NMC challenge: Classification of Normal versus Malignant Cells in B-ALL White Blood Cancer Microscopic Images” organized by the SBILab [27].

This work used the labeled data, i.e., train and preliminary sets, with 4037 healthy lymphocytes from 41 subjects and 8491 malignant lymphocytes from 60 patients. These images were split into training, validation, and test sets. The pictures of the same patient were placed in the same group, as done by Mourya et al. [11]. The subjects were divided into training–validation–test in the ratio of 7:2:1, as shown in

Table 1. Numbers of samples in the training, validation, and test sets. The number of patients is shown in parenthesis. Reproduced with permission from Ref. [19]. 2021, IEEE.

	Original		Data-Augmentation
	Malignant	Healthy	Malignant	Healthy
Training	5923 (42)	3035 (29)	20,000	20,000
Validation	1531 (12)	506 (8)	5000	5000
Test	1007 (6)	496 (4)	N/A	N/A

.

2.2. Data Augmentation

Sometimes, a classifier may work very well on the training data while performing poorly on previously unseen data. When this happens, we say that the model does not generalize well; i.e., it is overfitted. When a model is complex to the point that it models noise in the training data instead of smooth decision surfaces, it is probably overfitted. The model probably memorized the samples present in the training set instead of learning to generalize from them. To avoid overfitting, we may use several regularization techniques, such as dropout layers in the ANN, lasso, and ridge regression. Another possibility is to use data augmentation. It consists of augmenting the dataset with new samples obtained from the original ones by adding noise or doing some transformation. With data augmentation, it is possible to increase the dataset’s size when it is too small. It is also useful to balance the number of samples of the classes, as an unbalanced training set may generate a biased model [28].

Data augmentation was used to balance the training and validation sets and was not applied to the test images. New images were created using and combining rotation, blurring, mirroring, shearing transformation, and addition of salt-and-pepper noise. Examples of these images appear in Figure 2. Table 1 shows the sizes of the augmented sets.

3. Feature Extraction

The first step of the proposed classification method is the feature extraction. From each image contained in the dataset, we extracted an array of 1387 features. The features used were combinations of several found in previous leukocyte classification studies, and were the same as the ones used in our previous study [19].

We used low-order statistical, textural, morphological, contour, and DCT features extracted from each lymphocyte image.

Table 2. Number of features of each type.

Feature Type	Number
Low-order statistical	108
Textural	75
Morphological	20
Contour	160
DCT	1024
Total	1387

shows the number of features used of each type.

We obtained the low-order statistics from each channel of the images in both RGB and HSV formats. These statistics provide information about the image histograms, such as energy, entropy, skewness, kurtosis, mean, and standard deviation, as defined by the Image Biomarker Standardisation Initiative (IBSI) [29].

The textural features were calculated using the coefficients of co-occurrence matrices. These coefficients represent the different gray level combinations that occur in the image and can be used in image classification tasks [9,30]. We used features obtained from the gray level co-occurrence matrix (GLCM), gray level run length matrix (GLRLM), gray level dependence matrix (GLDM), gray level size zone matrix (GLSZM), and neighboring gray-tone difference matrix (GLDM) [20,29].

The morphological features used—rectangularity, eccentricity, elongation, compactness, etc.—indicate the general shape of a lymphocyte, and also have been used in the cell classification task by other authors [7,9,31].

The contour features were obtained from the discrete Fourier transform of the centroid distance function (CDF) of the lymphocyte. The CDF represents the distance between the lympocyte centroid and each pixel of its contour. This kind of shape signature was first proposed by Cosgriff [32] as a technique to identify objects and has been used to classify cells [33,34].

The discrete cosine transform (DCT), due to its energy compactness in the frequency domain, is widely used in image and video compression [35,36]. In this study, we calculated the DCT from the lymphocyte image converted to grayscale, producing a matrix with 202.500 DCT coefficients. The size of of this matrix was the same as the number of pixels in each image (450 × 450). We mapped the coefficients to a 1D array using a zigzag scan and used only the first 1024 lowest frequency coefficients.

Finally, all the features were combined into a unique vector for each sample image to train the different classifiers. The features from all samples were combined into a matrix with one sample per row and one feature per column. Feature values were normalized by subtracting each value from the column’s mean and dividing it by the column’s standard deviation.

4. Ensemble Classifier

The study of Garcia et al. [13] showed that it is possible to achieve high-performance results by combining different lightweight classifiers into a single solution. These classifiers obtain better results if trained with diverse data and applied to a complex and unbalanced problem. The simple vote scheme is a light and fast method to combine these classifiers. In this type of ensemble solution, it is possible to train all the classifiers with the same data in parallel, saving processing time. The ensemble classification result of a certain input is given by the class with the most votes from the different classifiers [15]. A criterion could be determined to be used in case of ties.

In this study, we combined an NB, a KNN, an SVM, and an ANN into four different ensemble classifiers, each one composed by three classifiers. Our classification problem has only two classes, so an odd number of classification models ensured that there would be no ties.

4.1. Naive Bayes Classifier

The naive Bayes classifier is one of the simplest and most widely used algorithms of pattern recognition. It is a probabilistic approach that calculates, for each possible class, the probability of an object belonging to it. The classification result is the class with the highest probability [37].

This algorithm is based on the Thomas Bayes decision theory, which assumes that the decision problem is posed in probabilistic terms and that all relevant probability values are known [38]. In simpler words, the Bayesian classifier maps decision boundaries based on the information given by labeled data and calculates the probabilities of new objects being allocated into a certain class. In this study, we used a Gaussian classifier with no prior class and a smoothing value of ${10}^{-9}$.

4.2. K-Nearest Neighbor

Proposed in 1951, the KNN is another machine learning algorithm used in many works of supervised classification problems. This method has a simple logical structure and classifies a given object as the most frequent occurring class in its neighborhood [38].

In other words, the KNN determines the class of a sample by finding the most frequent class among the K nearest objects to the sample. These neighbors are the ones used in the training step, and are already labeled. In the particular case where $K=1$, the KNN is equivalent to the nearest neighbor algorithm, and the chosen class is defined by the neighbor closest to the sample to be classified [37].

4.3. Support Vector Machine

The third lightweight classifier trained from the feature vector was the SVM classifier. The central idea of this algorithm is to obtain hyperplanes that separate the samples used for training into their respective classes [38].

The points closest to the discrimination hyperplane are called support vector points, and the distances between these points and a hyperplane are called margins. The support vector machine technique searches for a separation hyperplane that maximizes the margins [37]. In this study, we used a canonical SVM classifier with default parameters and linear discrimination.

4.4. Neural Network Training and Fine-Tuning

The previously extracted feature matrix was fed into an ANN that discriminates the lymphocytes as either malignant or healthy. The classification scheme is represented by Figure 3.

To find the best architecture to solve our problem, we did an extensive search in the hyper-parameter space of our network.

Table 3. Grid-search execution. Reproduced with permission from Ref. [19]. 2021, IEEE.

Parameter	Values	Chosen Value
Hidden Layers	1, 2, 3, 4	1
Batch Size	250, 750, 1000, 1500	250
Dropout	0.1, 0.25, 0.3, 0.5	0.1
Neurons Number	1024, 1536, 2048, 2560	2560
Activation	Prelu, Relu, Sigmoid, Softmax	Relu
Optimizer	Adamax, Adam, SGD	Adam
Kernel Initializer	Random Uniforme, Normal	Normal

shows all the evaluated values and the best parameters found. The grid search executed 25 epochs for each data point to finish the process timely.

After finding the best architecture among all ANN possibilities, a fine-tuning step was implemented to obtain, among other values, the best number of epochs. Since the best optimization method was the Adam function, it was essential to choose the best values for the learning rate, ${\beta }_1$, and ${\beta }_2$. These coefficients are responsible for controlling the exponential decay rates of the moving averages [39]. The values tested were 0.01, 0.001, 0.005, 0.0001, 0.0005 for the learning rate. The values tested for ${\beta }_1$ and ${\beta }_2$ were 0.99, 0.98, and 0.97. The best value found for ${\beta }_1$ and ${\beta }_2$ was 0.97, and for the learning rate the best was 0.001.

Finally, we searched for the minimum number of epochs necessary to maximize the F1-score. A number of epochs that is too big could cause excessive specialization on the training dataset. This condition leads to an incapacity in generalizing and errors when evaluating new images. This phenomenon is called overfitting [38]. The number of epochs started at 50, and at each iteration increased by 50 until the F1-score remained stable. The test showed that 150 is the best number of epochs for this ANN, and after this threshold, additional training could cause overfitting.

4.5. Ensemble Learning

According to Dietterich et al. [40], ensemble learning algorithms differ from other approaches because they do not use a single model to explain the data. Instead, they construct a set of classifiers and combine them in some fashion to classify new data points. As previously mentioned, other ensemble classifiers were proposed to solve the C-NMC problem, but none of them combined a neural network with other lightweight models [4,9] or with dense convolutional networks [11,17,18].

The literature presents several ways to combine a classifier into a single solution. It is possible to combine the results of the classifiers, use them as input to a new classification algorithm, use a function with different weights for each classifier [41], or use more sophisticated approaches, such as alpha-integration [42].

In this study, we trained—using the same vector of features—three of the most notorious and simple classifiers, a Gaussian NB with no prior class, a KNN, and a linear SVM classifier. These three classifiers plus the ANN were combined into four different ensemble classifiers, each one composed by three primitive ones, using a simple voting scheme. This procedure ensures the absence of ties and provides a fast and light solution without giving preference or a bigger weight to a particular classifier. The four ensemble classifiers created were

• ANN + SVM + NB (full ensemble model, with best F1-score).
• ANN + SVM + KNN.
• ANN + KNN + NB.
• SVM + KNN + NB.

Each primitive model used in the ensemble was trained separately, as described in the previous sections, without further optimization or fine-tuning. The result of the ensemble is the class with the most votes from each primitive model after their individual decision, a method known as late hard fusion [43].

4.6. Principal Component Analysis and Interpretable Models

The study of Ruding [44] explains the importance of using interpretable models instead the black-box trained machine learning to better understand the relevant features and to study which characteristics are decisive for the discrimination of classes. This practice can generate better and more applicable solutions to real world scenarios.

As an attempt to find the most important features for the classification task at hand, we started by choosing the full ensemble model, i.e., the ensemble classifier with best F1-score, composed by the ANN, SVM, and NB.

We used the Mlxtend feature selection Python library to find the principal components of our best ANN configuration. The Mlxtend sequential feature selection (SFS) removes one of the features at each iteration, returning a list of features and a score obtained with them. This process was interrupted when we observed the reduction in the quality metric. The minimum number of features selected to minimize the loss of precision of the ANN had about 15% of the total number of features.

Afterwards, we used the Keras features selector to find the 15% most relevant features for the SVM and NB classifiers [45]. The final reduced list of features was obtained from the union of the reduced lists of the individual classifiers. This procedure returned an array with 268 features, as shown in

Table 4. Number of features of each type in the reduced ensemble model.

Feature Type	Number
Low-order statistical	33
Textural	45
Morphological	4
Contour	4
DCT	182
Total	268

. These features are listed with the source code in descending order of importance at https://github.com/yurifarod/ISBI-2019/blob/main/z_interpretable_ensemble_analysis.txt (accessed on 8 June 2022). We used this reduced feature array to train a new, reduced version of our the full ensemble classifier:

• PCA, ANN + SVM + NB (reduced ensemble model).

The new reduced model is a lighter, interpretable, and faster solution. The hyperparameters used to train the ANN with the reduced parameter set were the same as the ones used for the full-set training.

5. Results and Discussion

As the C-NCM 2019 dataset was unbalanced, the obtained accuracy may not reflect reality. F1-score was chosen as the metric to overcome this problem. The metric also allows the comparison with other studies, as the teams who participated in SBILab’s challenge also used the F1-score. Gupta [46] edited a book where the participants’ results can be found.

The optimal hyper-parameters, i.e., the number of hidden layers, the number of neurons per layer, and the optimizer function, were chosen based on the performances of several ANNs generated by combining the parameters presented in Table 3. The table also shows the optimal parameters.

The experiments were done on an Intel Core i7-7500U CPU @ 2.7 GHz × 4, with 32 GB of RAM, without a dedicated GPU, as we aimed to evaluate a low-cost setup.

The published F1-scores obtained from the preliminary test set in SBILab’s challenge are shown in

Table 5. Performance of participants in the C-NMC challenge hosted by the SBILab. Proposed classifiers in boldface.

SBILab Challenger	F1-Score	Methodology
ANN + SVM + NB	93.70%	Feature extraction and ensemble classifier
[47]	92.50%	Transfer learning with ResNets
[48]	91.70%	Transfer learning with VGG16
ANN + SVM + KNN	91.80%	Feature extraction and ensemble classifier
ANN	91.20%	Feature extraction and ANN
ANN + KNN + NB	90.60%	Feature extraction and ensemble classifier
[17]	90.30%	Deep multi-model ensemble network
PCA, ANN + SVM + NB	89.87%	Reduced feature vector and ensemble classifier
[49]	89.47%	Transfer learning with MobileNetV2
[50]	87.89%	ResNeXt50
SVM + KNN + NB	87.60%	Feature extraction and ensemble classifier
[51]	87.58%	Transfer learning with CNN and recurrent ANN
[23]	87.46%	Transfer learning with ResNet18
[52]	86.74%	InceptionV3 + DenseNet + InceptionResNetV2
[53]	85.70%	ResNeXt50 + ResNeXt101
[18]	84.00%	Transfer learning with Inception + ResNets
[54]	81.79%	Transfer learning with ResNets + SENets
SVM	79.53%	Feature extraction and SVM
KNN	76.66%	Feature extraction and KNN
NB	74.25%	Feature extraction and NB

. Compared to these results, the best ensemble classifier trained using the proposed feature extraction method achieved the the highest F1-score. The ANN and two other ensemble classifiers had among the five best F1-scores. Notice that these approaches use less computational power than all convolutional neural network approaches submitted to this challenge. Another interesting point is the high performance achieved by the reduced ensemble model, which evidences its viability.

The time spent in training is omitted in most papers, but it is possible to compare the sizes of the neural networks. The number of parameters in our biggest model is smaller than those in most convolutional networks submitted to the SBILab challenge, and the reduced model is significantly smaller than all approaches. The proposed approach could potentially become a portable solution, and even be used in a low-cost device, such as a smartphone.

Table 6. Size comparison with other network architectures. Proposed classifiers in boldface. Adapted from Ref. [19]. 2021, IEEE.

Network	Number of Parameters
VGG16	138,357,544
ResNet152	60,380,648
InceptionResNetV2	55,873,736
ResNet50	25,636,712
Xception	22,910,480
DenseNet201	20,242,984
ANN and full ensemble	9,177,601
DenseNet121	8,062,504
Reduced ensemble model	2,775,553

indicates the network sizes as computed by Tensorflow.

In comparison with a CNN approach [55], using the same computational setup, all of our ensemble classifiers had similar F1-scores. The best ensemble learning model showed a better F1-score, and far smaller computational time and network size.

Table 7. VGG16 Comparison. Adapted from Ref. [19]. 2021, IEEE.

Metric	Reduced Ensemble	ANN	Full Ensemble	VGG16
Feature extraction time	16 min	1 h 2 min	1 h 2 min	-
Training time	8 min	9 min	9 min	16 h 20 min
Number of parameters	2,775,553	9,177,601	9,177,601	66,358,593
F1-score	89.87%	91.20%	93.70%	92.60%

shows the results of this comparison.

For the malignant class, our best model achieved an F1-score of 93.70%. This result was obtained by combining an NB, an SVM, and a neural network with three layers (one input, one hidden, and one output). The output layer consisted of one neuron and used a sigmoid activation function. The hidden layers contained 2560 neurons each and used the ReLU activation function. The ANN was trained from scratch using an Adam optimizer, a learning rate of 0.001 over 150 epochs, and 0.97 for ${\beta }_1$ and ${\beta }_2$.

The training of all approaches can be executed in parallel on a multi-core CPU; it took a maximum of 9 min in the CPU used in the experiments. Each sample consisted of an array with size 1 × 1387 for the complete classifiers and size 1 × 268 for the reduced approach.

Table 5 shows us that a reduced version of our best ensemble learning classifier (ANN + SVM + NB), although not as good as the version trained with all the features, can achieve a high performance. The F1-score is 4% smaller than our best result, with faster training and a lighter structure than the version trained with all 1387 features.

Table 8. Metric comparison between the full ensemble, reduced ensemble, and ANN.

Metric	Reduced Ensemble	ANN	Full Ensemble
F1-Score	89.87%	91.20%	93.70%
Accuracy	83.19%	86.82%	88.13%
Sensitivity	86.60%	88.11%	95.47%
AUC	75.10%	84.68%	88.36%
Kappa	44.47%	56.45%	67.79%
Precision	94.36%	96.48%	97.32%
Specificity	65.43%	77.92%	80.25%

shows the quality metrics obtained with our best ensemble model, its reduced version, and the ANN.

We evaluated the variability of the best methods by doing a Monte Carlo experiment with 100 repetitions. The training and validation sets were mixed together. At each step, new random training and test sets were obtained and used to train and evaluate new classifiers.

Table 9. Metrics obtained with the Monte Carlo experiment.

Metric	Reduced Ensemble Mean (SD)	Reduced vs. ANN p-Value	ANN Mean (SD)	ANN vs. Full p-Value	Full Ensemble Mean (SD)
F1-Score	89.78% (0.74%)	1.6 × 10−31	91.94% (0.75%)	1.0 × 10−24	93.88% (1.41%)
Accuracy	83.05% (1.16%)	6.7 × 10−32	86.54% (1.19%)	1.1 × 10−24	89.67% (2.32%)
Sensitivity	85.24% (1.13%)	9.8 × 10−28	87.89% (1.18%)	9.2 × 10−25	90.89% (2.21%)
AUC	76.62% (1.99%)	1.3 × 10−32	82.57% (1.94%)	3.2 × 10−17	86.12% (3.02%)
Kappa	40.99% (3.40%)	1.2 × 10−32	51.79% (3.53%)	2.2 × 10−23	61.08% (7.64%)
Precision	94.85% (0.58%)	2.6 × 10−34	96.38% (0.56%)	2.6 × 10−34	97.09% (0.72%)
Specificity	68.01% (3.57%)	2.3 × 10−31	77.25% (3.48%)	2.1 × 10−10	81.35% (4.41%)

shows the results obtained in the Monte Carlo experiment with the full ensemble classifier (ANN + NB + SVC), ANN, and reduced ensemble classifier. A nonparametric Mann–Whitney U-test was used to verify if the ensemble model is significantly better than the ANN. We can see that the full ensemble model always achieved better metrics than the ANN. Besides that, considering a significance level of 0.01, the p-values obtained were all smaller than the threshold for all metrics, which confirms that the improvements in the metrics from the usage of the full ensemble model were statistically significant over using simply the ANN.

Figure 4 and Figure 5 show, respectively, the F1-scores and accuracies obtained in the Monte Carlo experiment. The figures show histograms and boxplots of the metrics obtained.

Figure 6 shows the receiver operating characteristic (ROC) curve of the full ensemble classifier. The proposed test, when set to a specificity of 90%, has a sensitivity of 60%. Diminishing the specificity to 85% raises the sensitivity to 79%. Depending on the usage, we may want a test that is more sensitive or more specific. Consider the situation where a patient is under treatment, and we want to know whether the treatment must proceed or stop. A highly specific test has a low probability of classifying a healthy patient as diseased, which avoids unnecessary procedures, is often invasive, is costly, and is stressful [56]. Thus, in case of a positive result, we may assume that the treatment must proceed. On the other hand, a highly specific test may have low sensitivity or a high probability of classifying a diseased patient as healthy, i.e., a false negative. In the case of a negative result, as stopping the treatment of a diseased patient may cause grave risks, a new test may be done, such as a bone marrow aspirate (BMA), which is more invasive, but has a high performance.

6. Conclusions and Future Work

In this work, we demonstrated that a set of lightweight classifiers combined with a multilayer neural network, associated with a standard image processing feature extraction procedure, works as well as deep convolutional learning models. Our results indicate that the proposed methodology can accurately classify the lymphocytes as healthy or malignant. The rich textural, frequency-domain, and statistical data used by our method can be applied to many other problems besides cell classification. Our study of the PCA gives us a way to select the most relevant features for the classification problem, achieving good performance with a relatively small number of parameters and a short training time.

State-of-the-art techniques typically use deep convolutional neural networks, which may require long training, depending on the computer used. The performances of all proposed methods are comparable to the best approaches in the literature, yet they require a few minutes to train and seconds to run on a simple Core i7 CPU.

Many studies only tested their methodologies on a few sample images or their private datasets. On the other hand, our study was done with a large and public set of images, making our results more general and easily replicated. It must also be noted that images from different patients were presented from the training and test datasets. This procedure emulates a real-life scenario well [11].

Although BMA is the gold standard for leukemia diagnosis, it is an invasive procedure done under anesthesia. Exams done with peripheral blood are less invasive and may sometimes be preferred, even being less accurate than BMA. Recent studies reported very good results obtained with peripheral blood flow cytometry (PBFC) [57]. Lam et al. reported a sensitivity of 99.7% and a specificity of 98.5% obtained with PBFC [58]. A disadvantage of flow cytometry is the requirement of marker reagents that may not be readily available in all laboratories, especially in third world countries [59], so blood smear image analysis may be an alternative. An F1-score of 93.70% is not accurate enough for disease diagnosis but can serve as a tool for assisting oncologists.

Future works may refine our methodology by focusing on adding features similar to the 268 best features chosen for our interpretable model, especially textural and low-order statistical features. We may also test the inclusion of other classification models, such as decision trees, and linear and quadratic discrimination analysis. We may test our approach in a more complex dataset and try to solve a a multi-class classification problem. A possible way to improve our results is to use the scores returned by the classifiers before the decision, i.e., late soft fusion, and to integrate them in different ways, such as using their averages. It is also possible to use more sophisticated techniques, such as alpha-integration, and to optimize the weight of each primitive model in order to minimize the least mean squared error (LMSE) or the minimum probability of error (MPE) [43,60]. The alpha-integration uses a family of alpha-means that generalizes many widely used means, e.g., arithmetic, geometric, and harmonic. The alpha is a continuous value that defines which kind of mean is used and can be used to minimize the alpha-divergence of the distributions of the classifier’s results, improving the performance of the ensemble [61].