Classification of Valvular Regurgitation Using Echocardiography

Abstract

Echocardiography (echo) is a commonly utilized tool in the diagnosis of various forms of valvular heart disease for its ability to detect types of cardiac regurgitation. Regurgitation represents irregularities in cardiac function and the early detection of regurgitation is necessary to avoid invasive cardiovascular surgery. In this paper, we focussed on the classification of regurgitations from videographic echo images. Three different types of regurgitation are considered in this work, namely, aortic regurgitation (AR), mitral regurgitation (MR), and tricuspid regurgitation (TR). From the echo images, texture features are extracted, and classification is performed using Random Forest (RF) classifier. Extraction of keyframe is performed from the video file using two approaches: a reference frame keyframe extraction technique and a redundant frame removal technique. To check the robustness of the model, we have considered both segmented and nonsegmented frames. Segmentation is carried out after keyframe extraction using the Level Set (LS) with Fuzzy C-means (FCM) approach. Performances are evaluated in terms of accuracy, precision, recall, and F1-score and compared for both reference frame and redundant frame extraction techniques. K-fold cross-validation is used to examine the performance of the model. The performance result shows that our proposed approach outperforms other state-of-art machine learning approaches in terms of accuracy, precision, recall, and F1-score.

1. Introduction

Images are captured, preprocessed, and segmented to extract shape, texture, and color information, providing us clear knowledge of images. This will aid in detecting and classifying any abnormalities or diseases that may be present in an image. Medical imaging is one of the most effective tools that can provide proper treatment for any disease. Electrocardiograph (ECG), echocardiogram (echo), and computed tomography (CT) scans or angiograms are few of the medical imaging tools a physician can use to screen out any heart abnormalities [1]. One of the standard methods for diagnosing heart-related abnormalities or diseases is an echo. It is a non-invasive, radiation-free, and cost-effective procedure [2]. Cardiologists often use it to visualize heart structure, including walls, aorta, and other blood vessels [3]. It is used primarily for early diagnosis. Depending on the transducer deployed, echo can be transesophageal or transthoracic, that is, insertion through the throat or placing externally on the chest, respectively [4].

Echo has become a prominent choice used in examining valvular heart disease or regurgitation, one of the most common diseases in the cardiovascular system. Regurgitation is leakage caused in the heart valves, which is caused by incorrect closing of the leaflets where blood flows back, or blood leaks through. It can be congenital or acquired. Acquired cases are mainly caused due to smoking, tobacco consumption, lack of exercise, and others. Aortic regurgitation (AR), mitral regurgitation (MR), and tricuspid regurgitation (TR) are the three most common types of regurgitation, which are shown in Figure 1. When it comes to rheumatic heart disorders and youngsters, MR is the most frequent valvular involvement [5]. It is the most typical type of heart valve disease. In MR, the valve between the left heart chambers does not completely close, which causes blood to leak backward across the valve. It is known that AR occurs less frequently than MR. The disease of the aortic leaflets or the aortic root causes the leaflets to distort and hinder their proper apposition, which causes AR, for which careful analysis of the aortic valve is necessary. Rheumatic fever, bicuspid aortic valve, infective endocarditis, and senile leaflet calcifications are common causes of leaflet abnormalities that lead to AR [6]. TR, on the other hand, is a condition where the valve between the right ventricle and right atrium does not close completely. Blood leaks backward into the upper right chamber as a result [6].

These data are obtained using the transthoracic procedure in the left lateral decubitus [7]. The data are 3D color Doppler videographic images. Using color Doppler, regurgitation can be identified based on the color jets characteristics. Other abnormalities, such as aortic stenosis (AS), mitral stenosis (MS), and tricuspid stenosis (TS), and other cardiovascular disorders, can be distinguished from regurgitation [8]. Echo can be used to access, detect, and diagnose regurgitation at an early stage. Regurgitation can range from mild to severe. Mild and moderate regurgitation can usually be treated without surgery; however, severe regurgitation may require surgery or cauterization [9]. For most of the population, especially in developing and rural economies, this procedure is expensive and unaffordable [10]. Regurgitation can make it difficult for a person to breathe because of the blood’s improper inflow and outflow, which is a concern. A cardiologist can determine whether a person is healthy or has any regurgitation or disorders using visualization and knowledge which needs precise identification of many images before accurately predicting such abnormalities. As a result, if regurgitation can be detected early, proper diagnosis and treatment can be provided. This can be accomplished with the use of automated tools and approaches.

The usage of automated tools is critical since it reduces human effort, reduces the necessity of invasive procedures and allows more precise prediction of various problems in the heart. It has become prominent in medical imaging due to its accessibility, efficiency and effectiveness. Different techniques can achieve automatic disease diagnosis, such as traditional methods, machine learning, deep learning, and reinforcement learning. Significant research and analysis are required before applying such tools in medical imaging. Many researchers are trying to find a suitable application that will make human intervention easier and possibly obsolete. To date, no such application has been established which is applicable in clinics and hospitals. Work in classification and prediction of the type of regurgitation employing automated technologies is ongoing, and traditional approaches or advanced state-of-art techniques are being used.

Here, work is done for classification using a simpler model involving preprocessing, keyframe extraction, segmentation, feature extraction, and classification. This step-by-step procedure is significant for a detailed analysis and prognosis of heart abnormalities as a whole. These are the main steps in automated diagnosis. In addition, keyframe extraction has been included in this work. The pictorial representation of the steps involved is shown in Figure 2. Preprocessing is a phase in image processing that helps to reduce the presence of noise and other undesirable data and artifacts in an image. Preprocessing allows for greater clarity and understanding of the image, resulting in a more effective result. There are many techniques for preprocessing, some of which are filtering techniques, morphological operations, and statistical techniques [11,12]. Keyframe extraction is a crucial step in video analysis which is efficient in indexing [13]. It is usually applied in movies, sports videos, low-quality videos, etc. [14]. It is a step used to extract keyframes to suffice comprehensive analysis of videos by eliminating replication. It is also sometimes used to reduce the number of frames instead of using all of them. Some techniques used are clustering techniques, color-based techniques, hash function, global comparison, motion-based, and others [13,14,15]. Segmentation is one of the most important aspects in diagnosis of disease for detecting a region or regions where abnormalities are located. Techniques commonly used in segmentation are threshold based techniques, clustering techniques, statistical methods [12], and other methods such as deep learning. A ground truth image is required when employing deep learning methodology, which can be a disadvantage in some circumstances where one is not available. Feature extraction is a step to extract useful information based on properties such as texture, shape, and color. This process is essential in classifying images as it provides a more meaningful representation of the image. A specific region can be localized or globalized according to a particular application [12]. Classification is a step that classifies the images or a region into different classes. Classes can be predefined (supervised) or random (unsupervised) [5]. It usually involves data division into training (training phase) and testing (testing phase) data. In some cases validation data can also be taken into account. Different techniques available are K-nearest neighbor (KNN), Naive Bayesian [16], etc. A pictorial representation of this process is shown in Figure 2.

In this work, an automated approach is proposed to classify the three types of regurgitation (AR, MR and TR). This approach is efficient for detecting location as well as prediction. Here, preprocessing is used to remove speckle noise which acts as a barrier and poses a wrong perception about the image, followed by keyframe extraction where the frames are free from redundancy. In addition to redundancy removal, the reference frame is obtained by comparing the frames that has been extracted from all the other frames in a video to an original reference frame. Since no ground truth is available, segmentation is carried out to locate the regions containing regurgitation using an unsupervised deformable model. The Level set method is used as it is well suited in the segmentation of the heart and its different parts [17,18,19]. After segmentation, different texture features are extracted using Gray Level Coocurrence Matrix (GLCM) and Haralick feature extraction methods. Here, the integration of various features is carried out instead of a specific type of feature. Since no prior knowledge of image patterns is required, such features will make the classification more reliable. The use of features here illustrates the differences between each video frame. After that, classification is performed where different heart regurgitation (AR, MR, and TR) are classified. We utilized Random forest for classification because it has been used in various fields of computer vision and medical imaging [20,21,22,23]. Even though deep learning methodologies have taken over the traditional approach due to their performance and training time, we have depended on traditional ways to classify our data because deep learning does not perform well when the data set is not large. Here, traditional methodologies that have achieved effective results in various fields related to medical imaging and have properties that are suited for heart classification are used. Moreover, a comparison to other techniques currently in use in the field is provided.

Contributions

The main contributions of the paper are as follows:

• An automated system is designed to classify valvular regurgitation using echo with all the steps involved, such as preprocessing, keyframe extraction, segmentation, feature extraction, and classification.
• In contrast to most of the existing work where authors use image file format, here, we have used videographic images to classify valvular regurgitation.
• Using videographic images, the number of frames is large, and there may be similar frames. In this work, we have used the keyframe extraction technique, which reduces the number of frames from video and also minimizes redundancy. This is the beginning of the application of keyframe extraction in regurgitation classification. Here, the reference keyframe extraction and redundant frame keyframe extraction techniques have been incorporated.
• The data used are validated by a cardiologist in the case of classification.
• The utilization of videographic images, keyframe extraction, and methodologies such as Level set, Haralick features, and GLCM with Random Forest distinguish this research from others.
• To evaluate the robustness of the model, we have used both segmented images and non-segmented images and evaluate the performance of the model.
• The results of the proposed method are compared to several existing methodologies, and the results show that our implemented method provides higher performance accuracy than other state-of-art techniques.

The rest of the paper is organized as follows: Works related to regurgitation and heart-related classification are described in Section 2. In Section 3, the methodologies used are explained in brief. Section 4 provides the experimental result, and Section 5 includes the conclusion and future work.

2. Related Works

The heart is a sophisticated structure that requires a trained cardiologist for the proper diagnosis of any abnormalities. Some of the works that deal with the heart perspective related to our work are described below.

Pinjari [8] worked with valvular regurgitation using color Doppler images. Two types of regurgitation were considered, namely MR and AR, whereby the images were first converted into YCbCr space. Two filters were used for noise removal, namely the Wiener filter and Gaussian filter. After filtering using Fuzzy K Means and anisotropic diffusion, segmentation was carried out, and Proximal Isovelocity Surface Area Method (PISA) was used for quantification, which classifies regurgitation into mild mitral regurgitation, moderate aortic regurgitation, and severe aortic regurgitation. Allan et al. [12] used 2D echo and proposed an approach for information extraction in apical view. The procedure takes the information of patients and image for classification into volume and label. For finding intensity and label, Joint Independent Component Analysis (JICA) was used. The approach classified moderate MR with an accuracy of 82%.

Varghese and Jayanthi [24] used an approach for segmentation of echo from videos. A closed curve was used to detect heart boundary and Gaussian Mixture Model (GMM) clustering for segmentation. Balaji et al. [3] worked on view classification using echo. Two features were used namely histogram features and statistical features to classify of four standard views, namely, parasternal short axis (PSAX), parasternal long axis (PLAX), apical two chambers (A2C), and apical four chambers (A4C), was carried out. For 200 images, an accuracy of 87.5% was achieved. Another work by Balaji et al. in [25], similar to that in [3], on view classification was done. Mathematical morphology was used before segmentation using Connected Components Labeling (CCL). Classification of three standard cardiac views, namely parasternal short axis (PSAX), apical two chambers (A2C), and four-chamber (A4C) views, was carried out with an accuracy of 94.56%. Nandagopalan [16] has worked on view classification using a Bayesian classifier. Danilov et al. worked on preprocessing using median filter and Ramponi filter and segmentation using active contours. Supha [26] dealt with the left ventricle (LV) segmentation by detecting LV boundary using fuzzy logic and watershed algorithm along with hidden Markov model-based scheme. Oo et al. [17] used a level set method for LV boundary detection. Mazaheri et al. [18] also worked with LV boundary segmentation and reviewed using level set, active contour, and active shape models. In [27], heart valve disease for aortic stenosis (AS) was assessed. A review on machine learning techniques for heart disease prediction can be seen in [28], and work based on heartbeat counts for the classification of heart diseases could be observed in [29].

In the next section, a discussion on the different existing methods and the proposed approach used for the classification of regurgitation into AR, MR and TR can be observed.

3. Methodology

3.1. Existing Methodologies Used in Proposed Methodology

3.1.1. Gray Level Cooccurrence Matrix (GLCM) and Haralick Texture Features

GLCM is a statistical method of extracting texture features that considers the spatial relationship of pixels in an image. A square matrix of size $N\times N$ defines gray level values of pixels at a different position. For the image, $I(i,j)$, the intensity features and texture features that were extracted are Energy, Correlation, Entropy, Contrast, Homogeneity, Mean, Standard Deviation, Root Mean Square (RMS), Skewness, Variance, Smoothness, Kurtosis, shade, prominence and Inverse Difference Normalized, Inverse Difference Moment Normalized (IDM) [30,31]. Haralick texture features are also texture descriptors of an image that specify the spatial relationship among the neighboring pixels in an image [32]. Haralick features are calculated using GLCM and widely used due to their simplicity and perceptive interpretations [33]. The different extracted Haralick features are angular second moment, sum of squares, sum average, sum variance, sum entropy, difference variance, difference entropy, Info measure of correlation 1, and info measure of correlation 2 [34]. Texture features are selected because they provide information about the intensities of an image. Such features partition images into the region of interest and classify them.

3.1.2. Random Forest (RF)

RF is used as it is a stable algorithm having multiple trees and is not biased. It can even take care of missing values in data. RF is a supervised classification technique used in solving classification and regression problems. Two stages are present in the RF algorithm: one is to create the forest, and another is to predict the forest. The RF model calculates a response variable from the randomly created number of decision trees and then puts each feature set to be modeled down each decision tree. The response is then determined by evaluating the responses from all of the trees. At each internal node of the decision tree, entropy is given by the formula in Equation (1) [35].

$$Entropy=-\sum _{i=1}^n{p}_{i}Xlog{p}_i$$

(1)

where n = number of classes and $p_i$ is the probability of each given classes.

3.1.3. Level Set Methodology (LSM)

It is a method developed in the 1980s that has become popular in imaging processing and computer vision that is used to analyze shapes and surfaces [36,37]. It is used for segmentation where a surface (x) is taken, and a contour is obtained as its output. The front (the point where the surface is at its zero levels) is defined as a zero level set (x = 0). This technique provides accurate numerical as well as adhering to topological changes as well [36]. The intersection of x with the plane creates a contour. For a point, p(a,b) the derived quantity p(t) is its position over time. The evolution of the level set function resembles that of the Hamilton Jacobi equation [38]. It is dependent on partial differential equations, where different parameters are combined to obtain an interface for image segmentation [19]. LSM can be defined in Equations (2)–(4) as follows:

$$x(a,b,t)<0 if (a,b)\in \overline{\varphi }$$

(2)

$$x(a,b,t)=0 if (a,b)\in \tau$$

(3)

$$x(a,b,t)>0 if (a,b)\in \overline{\varphi }$$

(4)

where $\varphi$ denotes sub region inside $\tau$ and $\overline{\varphi }$.

3.2. Proposed Methodology

The major computational steps in the proposed approach are image preprocessing, keyframe extraction, unsupervised segmentation (without ground truth), feature extraction, and classification. The overall flowchart of the algorithm used in the proposed methodology is given in Figure 3, and is explained in subsequent sections.

3.2.1. Image Preprocessing

In this paper, the echo obtained are in Audio Video Interleave (AVI) format which were extracted into frames. To improve feature extraction, the image in the dataset is preprocessed before the extraction of features. The first operation is to normalize the image size, and all the images are resized to $224\times 224$ number of pixels. The next step is to convert the RGB image to a grayscale image for better representation of an image. To make the uniform background, a masking operation is performed and finds the region of interest (ROI). Median filtering technique is used to remove the noises and some connected components. Some morphological operations, such as erosion and dilation, are performed to remove the unnecessary pixels, add boundary pixels, and fill small holes in the masked image. After that the following steps are performed:

• Binarize the image; get the membership of each pixel using label_image.
• Using regionprops, the area of each object in an image is calculated. It measures image quantities and features.
• Sort the area and calculate the centroid of the image.
• Using ismember, the desired image is computed.
• Store the frames into a folder for each video.

3.2.2. Image Keyframe Extraction

This method overcomes any irregularities in an image boundary and helps reduce and eliminate any artifacts. The obtained frames are then stored in a folder, and keyframe extraction is applied. Two types of frame extraction are used in this paper. Firstly, using reference frame extraction, a static image captured during videographic echo is taken as a reference frame and compared with the frames obtained from the videographic echo. Mathematically, a reference frame $Re{f}_i$ for each patient $P_i$ is compared to frames $F_1,F_2,\dots ,F_n$ where n is the number of frames present for each patient i. For this, we have used Euclidean distance to obtain a single or few frames from the videographic echo by eliminating images having a lower similarity index [39]. Similar frames will have a low value, and dissimilar frames will have higher values. The top one or three best values are taken and used for our implementation purpose. This is useful when a reference frame is available to compare but not suitable for videos with no frame as a valid reference frame. Since we have an available reference frame for each video we can compare to every frame of a particular video. Mathematically it can be represented as Equation (5):

$$dist(i,j)=\sqrt{\sum _{j=1}^N\sum _{i=1}^M(i-j)}$$

(5)

where i = Reference frame extracted by an expert, j = A single frame in a video, N = Total number of reference frames and M = Total number of frames.

For video frames having no reference frame, uniform sampling is used where skipping every two frames has been done. This helps in removing redundancy between images. For this, for each frame F in $F_1,F_2,\dots ,F_n$ select $F_i+3$ where i = 0 upto n. Mathematically for all videos (Vid) it can be written as shown in Equation (6)

$$\sum _{i=1}^{n}Vi{d}_i=\sum _{j=1}^k{F}_j+3$$

(6)

where j = 1 to k, k = total number of frames present in a video. After keyframe extraction, extraction of features is performed. We have extracted the features from the keyframe in two different forms: one is extraction of features with segmenting the ROI, another one is extraction of features directly from the keyframe.

3.2.3. Image Segmentation

Currently, there are a variety of tools accessible for manually segmenting an image. Such manual work is time-consuming and may potentially result in erroneous results. Here, an unsupervised technique is used for segmentation. Segmentation is carried out using the LSM [17], which is a commonly used deformable model combined with Fuzzy C Means (FCM) [19]. LSM is computationally intensive, but performance-wise has a powerful impact. Segmentation starts with the FCM algorithm, followed by LSM as per [19]. It is used when shapes and contours are involved and can help detect the presence of any regurgitation in an image. Nonetheless, segmentation for nonuniform images is still complex, and research is still being carried out to date.

3.2.4. Feature Extraction Using GLCM Texture Features and Haralick Texture Feature

Different features are extracted from videographic images to describe the image, which is further used for classification. We have used GLCM texture features and Haralick texture feature in our work. These features are selected due to their performance, GLCM is a widely used statistical texture feature and Haralick is an extension to GLCM that develops spatial indices that can be interpreted easily and are correlated. These are inspired by [31,32,40,41]. In this work, a set of 19 statistical texture features are extracted from the segmented and non-segmented images as shown in

Table 1. Features extracted with their mathematical expressions.

Sl. No.	Name of Features	Mathematical Expression
1	Contrast	$f_1={\sum }_{i=1}^N{\sum }_{j=1}^N{(i-j)}^2{P}_{i,j}$
2	Dissimilarity	$f_2={\sum }_{i=1}^N{\sum }_{j=1}^N{P}_{i,j}\left|i-j\right|$
3	Energy	$f_3={\sum }_i{\sum }_j{P}_{i,j}^2$
4	Entropy	$f_4=-{\sum }_{i=1}^N{\sum }_{j=1}^N{P}_{i,j}lg{P}_{i,j}$
5	Correlation	$f_5={\sum }_{i=1}^N\parallel \frac{(i-{\mu }_i)(j-{\mu }_i)}{{\sqrt{\sigma }}_i^2{\sigma }_j^2}\parallel$
6	Homogeneity	$f_6={\sum }_{i=1}^N{\sum }_{j=1}^N\frac{P_{i,j}}{1-P_{i,j}^2}$
7	Variance	$f_7={\sum }_{i=1}^N{P}_{i,j}{(1-\mu )}^2)$
8	Autocorrelation	$f_8={\sum }_{i=1}^{N}i,j{P}_{i,j}$
9	Sum average	$f_9={\sum }_{l=2}^{2N}l.P_{x+y}l$
10	Sum entropy	$f_{10}=-{\sum }_{n=2}^{2N}{P}_{x+y,n}log\left(P_{x+y,n}\right)$
11	Sum variance	$f_{11}={\sum }_{l=2}^{2N}{(l-f_4)}^2{P}_{x+y}l$
12	Difference entropy	$f_{12}=-{\sum }_{l=0}^{N-1}{P}_{x-y}l{P}_{x+y}l$
13	Difference variance	$f_{13}={\sum }_{l=0}^{N-1}{l}^2{P}_{x+y}l$
14	Information measure of correlation 1	$f_{14}=\frac{HXY-HX{Y}_1}{max(HX,HY)}$
15	Information measure of correlation 2	$f_{15}=\sqrt{(}1-exp(-2(HXY2-HXY)))$
16	Cluster Prominence	$f_{16}={\sum }_{i=1}^N{\sum }_{j=1}^N{(i+j-{\mu }_x{\mu }_y)}^4{P}_{i,j}$
17	Cluster Shade	$f_{17}={\sum }_{i=1}^N{\sum }_{j=1}^N{(i+j-{\mu }_x{\mu }_y)}^3{P}_{i,j}$
18	Inverse Difference Normalized	$f_{18}={\sum }_{i=1}^N{\sum }_{j=1}^N\frac{P(i,j)}{1+\frac{k}{n}}$
19	Inverse Difference Moment Normalization	$f_{19}={\sum }_{i=1}^N{\sum }_{j=1}^N\frac{P(i,j)}{1+\frac{k^2}{n^2}}$

with their mathematical expressions [32,33,34].

Due to the lack of segmentation using features such as GLCM and Haralick features, the segmentation can be reliable. These are robust methods. The features are extracted from the segmented and nonsegmented images, and the values are stored in a comma-separated values (CSV) file.

3.2.5. Classification

Classification is carried out using Random Forest (RF). RF is preferred over other classifiers such as SVM as this paper involves multiclass classification, not binary classification where it is known to perform better [42,43,44]. It also gives us the probability of belonging to the class. It can also handle imbalanced data such as those relied upon in this work. It is robust to outliers [44]. RF classifier works well in small datasets and outperform other ensemble classifier as has the ability to tackle overfitting issues.

The CSV file is read using the readcsv command from the panda library. The data is split into training and testing using the k-fold split command, and then this is fed to a random forest classifier that classifies the testing data into three classes 0, 1, and 2 (AR, MR, and TR, respectively).

4. Experimental Results

4.1. Experimental Setup

For running the program, we have used Python programming. It is implemented using Google Colab, an online, freely-available platform. The libraries and modules used are Matplotlib, Keras, Sklearn and PIL. The detailed specifications of Google Colab hardware used are provided in Appendix A. The different results obtained are given next. The proposed technique follow the same block diagram as shown in Figure 3.

4.2. Dataset

For carrying out the experiment and analysis, data were obtained from Hope Clinic, Shillong. A color Doppler data consisting of five patients’ data in video format was used for each regurgitation. The number of data obtained after reference frame extraction is 170, where for 20% testing data data division is 136 for training and 34 for testing, and the data count obtained after redundant frame extraction is 532, where for 20% testing data data division is 424 for training and 106 for testing. Here, data are divided into 10%, 20%, 30%, 40% and 50% from the total obtained data as testing data. Using k-fold cross-validation, the number of folds is 2, 3, 5, and 8 folds cross-validation. K-fold cross-validation helps to avoid overfitting and resampling of data which aids in assessing the performance of a model. The input images after cropping and resizing can be seen in Figure 4.

4.3. Output of Proposed Methodology

4.3.1. Preprocessing and Segmentation Output

After cropping and resizing, preprocessing is carried out. A preprocessed boundary detection has been applied, and the result can be seen in Figure 5. The figure includes the original images and the preprocessed images. After which segmentation is carried out for both redundant and reference frames. The purpose of this is to test whether segmentation provides a better result or not in later phases of diagnosis, mainly in predicting the presence or absence of any regurgitation in an image.

The output showing segmented images to that of the original images is given in Figure 5. Segmentation is not validated as ground truth is not available for videos. However, the segmented region is carried out for feature extraction, where values of each image are obtained. These values provide the differences and similarities of each frame to one another.

4.3.2. Features Extracted

The different GLCM and Haralick features extracted using Table 1 are stored in a CSV file which look like the image provided in Figure 6. The figure shows obtained features for selected frames. The 19 different features are contrast as $contr$, dissimilarity as $dissi$, energy as $energ$, entropy as $entro$, correlation as $corrp$, homogeneity as $homom$, variance as $sosvh$, autocorrelation as $autoc$, sum average as $savgh$, sum entropy as $senth$, sum variance as $svarh$, difference entropy as $denth$, difference variance as $dvarth$, information measure of correlation 1 as $info1$, information measure of correlation 2 as $info2$, cluster prominence as $cprom$, cluster shade as $cshad$, inverse difference normalized as $maxpr$ and inverse difference moment normalized as $idmnc$. With the features, class label is also provided for training phase where 0 is for AR, 1 is for MR and 2 is for TR.

4.3.3. Classification Output

After feature extraction, RF is used to classify the images into three types of regurgitation, namely AR, MR, and TR. Here, two types of classification are carried out which is reference frame classification and redundant frame classification. For each type of classification, two approaches have been used, the segmentation approach and the without segmentation approach. It is crucial to determine whether or not segmentation is an essential factor in classifying regurgitation in the heart. It also gives us a clear idea of validation and analysis of images in the future.

The classification was assessed using four measures based on the confusion matrix. These are Accuracy, Precision, Recall, and F1-score [45]. Classification is implemented using k-fold cross-validation for statistical purposes since data used are not massive.

• Accuracy: It is the measure of correctly classified images as a percentage. It can be calculated using Equation (7).

  $$Accuracy=\frac{TP+TN}{TP+FP+TN+FP}$$

  (7)
• Precision: It is the fraction of True Positives (TP) and False Positives (FP). Precision can be calculated using Equation (8).

  $$Precision=\frac{TP}{TP+FP}$$

  (8)
• Recall: It represents the fraction of True Positives (TP) and False Negatives (FN). It can be calculated using Equation (9).

  $$Recall=\frac{TP}{TP+FN}$$

  (9)
• F1-score: It is a harmonic mean of precision and recall. It is given by Equation (10).

  $$F1-Score=\mathsf{2}\times \frac{\mathsf{Precision}\times \mathsf{Recall}}{\mathsf{Precision}+\mathsf{Recall}}$$

  (10)

The following observations were made based on the output obtained from

Table 2. Reference frame without segmentation.

No. of Fold	%	Precision			Recall			F1-Score
		P0	P1	P2	R0	R1	R2	F0	F1	F2
2-folds	10	100	78.57	100	76.92	100	100	86.95	87.99	100
	20	100	84.61	77.77	100	84.61	77.77	100	84.61	77.77
	30	72.72	81.81	58.33	80.00	75.00	58.33	76.18	78.25	58.33
	40	75.00	81.81	81.61	100	64.28	81.81	85.71	71.99	81.81
	50	84.61	100	77.77	84.61	100	77.77	84.61	100	77.77
3-folds	10	50.00	100	100	100	50.00	100	66.66	66.66	100
	20	100	66.66	87.5	100	80.00	77.77	100	72.72	82.34
	30	75.00	60.00	100	100	100	71.42	85.71	75.00	83.32
	40	100	100	100	100	100	100	100	100	100
	50	100	71.42	75.00	100	71.42	75.00	100	71.42	75.00
5-folds	10	66.66	100	100	100	60.00	100	79.99	75.00	100
	20	100	100	100	100	100	100	100	100	100
	30	100	100	100	100	100	100	100	100	100
	40	100	100	100	100	100	100	100	100	100
	50	100	85.71	100	80.00	100	100	88.88	92.30	100
8-folds	10	100	100	0	100	75.00	0	100	85.71	0
	20	100	100	100	100	100	100	100	100	100
	30	100	100	100	100	100	100	100	100	100
	40	100	100	100	100	100	100	100	100	100
	50	100	100	50.00	100	66.66	100	100	79.99	66.66

,

Table 3. Reference frame with segmentation.

No. of Fold	%	Precision			Recall			F1-Score
		P0	P1	P2	R0	R1	R2	F0	F1	F2
2-folds	10	100	28.57	90.00	55.55	100	75.00	71.42	44.44	81.81
	20	83.33	69.23	77.77	83.33	100	53.84	83.33	81.81	63.62
	30	81.81	72.72	75.00	60.00	100	81.81	69.22	84.20	73.68
	40	75.00	72.72	100	75.00	72.72	100	75.00	72.72	100
	50	100	58.33	100	86.66	100	75.00	92.85	73.68	85.71
3-folds	10	100	50.00	72.72	72.72	100	80.00	84.20	66.66	76.18
	20	88.88	50.00	100	72.72	100	88.88	79.99	66.66	94.11
	30	100	100	100	100	100	100	100	100	100
	40	100	66.66	100	100	100	75.00	100	79.99	85.71
	50	100	100	87.50	88.88	100	100	94.11	100	93.33
5-folds	10	100	100	100	100	100	100	100	100	100
	20	100	66.66	0	55.55	100	0	71.42	79.99	0
	30	100	100	75.00	89.71	100	100	94.57	100	85.71
	40	100	100	60.00	66.66	100	100	79.99	100	75.00
	50	100	100	100	100	100	100	100	100	100
8-folds	10	100	33.33	100	100	100	33.33	100	49.99	49.99
	20	100	100	100	100	100	100	100	100	100
	30	33.33	100	50.00	100	20.00	100	49.99	33.33	66.66
	40	100	100	75.00	66.66	100	100	79.99	100	85.71
	50	50.00	100	100	100	50.00	100	66.66	66.66	100

,

Table 4. Redundant frame without segmentation.

No. of Fold	%	Precision			Recall			F1-Score
		P0	P1	P2	R0	R1	R2	F0	F1	F2
2-folds	10	95.23	96.77	90.90	97.56	90.90	93.75	96.38	93.74	92.30
	20	95.74	99.17	100	97.82	99.58	94.44	96.76	99.37	97.14
	30	88.88	71.42	93.93	86.95	86.95	83.78	87.90	78.42	88.56
	40	85.00	96.77	80.00	91.89	78.94	90.32	88.31	86.95	84.84
	50	94.59	96.66	97.50	100	93.54	95.12	97.21	95.07	96.29
3-folds	10	93.75	88.23	95.65	96.77	88.23	91.66	95.23	88.23	93.61
	20	92.59	90.00	91.66	96.15	85.71	91.66	94.33	87.80	91.66
	30	91.66	90.90	88.46	84.61	90.90	95.83	87.99	90.90	91.99
	40	96.55	95.23	90.47	96.55	95.23	90.47	96.55	95.23	90.47
	50	94.44	92.85	95.23	94.44	86.66	100	94.44	89.65	97.55
5-folds	10	100	100	100	100	100	100	100	100	100
	20	94.11	83.33	92.30	88.88	90.90	92.30	91.42	86.95	92.30
	30	100	100	100	100	100	100	100	100	100
	40	86.66	87.50	95.00	92.85	77.77	95.00	89.64	82.34	95.00
	50	100	100	100	100	100	100	100	100	100
8-folds	10	100	100	100	100	100	100	100	100	100
	20	100	100	100	100	100	100	100	100	100
	30	100	100	100	100	100	100	100	100	100
	40	100	100	100	100	100	100	100	100	100
	50	100	100	100	100	100	100	100	100	100

,

Table 5. Redundant frame with segmentation.

No. of Fold	%	Precision			Recall			F1-Score
		P0	P1	P2	R0	R1	R2	F0	F1	F2
2-folds	10	80.95	64.51	48.27	79.07	50.00	73.68	79.99	56.33	58.87
	20	76.59	72.00	58.82	81.81	56.25	66.66	79.11	63.15	62.49
	30	88.88	71.42	100	95.23	86.95	80.48	91.94	78.42	89.18
	40	75.00	77.41	31.42	62.50	57.14	68.75	68.18	65.74	43.12
	50	81.08	55.17	42.50	62.50	59.25	61.29	70.58	57.13	50.19
3-folds	10	87.50	81.25	86.95	87.50	68.42	100	87.50	74.28	93.01
	20	48.14	45.00	58.33	54.16	39.13	58.33	50.97	41.86	58.33
	30	78.26	63.66	84.61	69.23	73.68	84.61	73.46	68.28	84.61
	40	65.51	71.42	66.66	70.37	62.50	70.00	67.85	66.66	68.28
	50	83.33	71.42	66.66	85.71	52.63	82.35	84.60	60.60	73.67
5-folds	10	77.77	66.66	66.66	77.77	61.53	72.72	77.77	63.99	69.55
	20	82.35	58.33	75.00	77.77	63.63	75.00	79.99	60.86	75.00
	30	85.71	80.00	88.88	80.00	72.72	100	82.75	76.18	94.11
	40	60.00	62.50	36.84	52.94	35.71	63.63	56.24	71.61	46.66
	50	62.50	69.23	84.61	83.83	52.94	84.61	71.61	59.99	84.61
8-folds	10	66.66	71.42	40.00	60.00	50.00	66.66	63.15	58.82	49.99
	20	55.55	71.42	60.00	55.55	55.55	75.00	55.55	62.49	66.66
	30	77.77	62.50	77.77	87.50	62.50	70.00	82.34	62.50	73.68
	40	81.81	90.00	80.00	90.00	81.81	80.00	85.70	85.70	80.00
	50	86.66	80.00	83.33	92.85	80.00	71.42	89.64	80.00	76.91

,

Table 6. Performance metrics for 10%, 20%, 30%, 40% and 50% testing data using Reference frame extraction for with and without segmentation for two, three, five, and eight folds.

Fold	Performance Metrics	With Segmentation					Without Segmentation
		10%	20%	30%	40%	50%	10%	20%	30%	40%	50%
2-folds	Accuracy	67.64	76.47	76.47	82.35	85.29	91.17	88.23	70.58	79.41	88.23
	Precision	72.85	76.77	76.51	82.57	86.11	92.85	87.46	70.95	79.54	87.46
	Recall	76.85	79.05	80.60	82.57	87.22	92.30	87.45	71.11	82.03	87.46
	F1-score	65.89	76.25	75.70	82.57	84.08	91.64	87.46	70.92	79.83	87.46
3-folds	Accuracy	78.26	82.60	100	86.95	95.65	82.60	86.95	82.60	100	82.60
	Precision	74.24	79.62	100	88.88	95.83	83.33	84.72	78.33	100	82.14
	Recall	84.24	87.20	100	91.66	96.29	83.33	85.92	90.47	100	82.14
	F1-score	81.65	80.25	100	88.56	95.81	77.77	85.02	81.34	100	82.14
5-folds	Accuracy	100	69.23	92.30	84.61	100	90.00	100	100	100	92.30
	Precision	100	55.55	91.66	86.66	100	88.88	100	100	100	95.23
	Recall	100	51.85	96.57	88.88	100	86.66	100	100	100	93.33
	F1-score	100	50.47	93.42	84.99	100	84.99	100	100	100	93.72
8-folds	Accuracy	75.00	100	50.00	87.50	75.00	87.50	100	100	100	87.50
	Precision	77.77	100	77.77	91.66	83.33	66.66	100	100	100	83.33
	Recall	77.77	100	73.33	88.88	83.33	58.33	100	100	100	88.88
	F1-score	66.66	100	49.99	88.56	77.77	61.90	100	100	100	82.21

and

Table 7. Performance metrics for 10%, 20%, 30%, 40% and 50% testing data using Redundant frame extraction for with and without segmentation for two, three, five, and eight folds.

Fold	Performance Metrics	With Segmentation					Without Segmentation
		10%	20%	30%	40%	50%	10%	20%	30%	40%	50%
2-folds	Accuracy	66.66	69.81	87.73	61.32	61.32	94.34	98.76	85.84	86.79	96.26
	Precision	64.57	69.13	86.76	61.27	59.58	94.30	98.30	84.74	87.25	96.25
	Recall	67.58	68.24	87.55	62.79	61.01	94.07	97.28	85.89	87.05	96.22
	F1-score	65.06	68.25	86.51	59.01	59.30	94.14	97.75	84.96	86.70	96.19
3-folds	Accuracy	85.91	50.70	76.05	67.60	76.05	93.05	91.54	90.27	94.36	94.36
	Precision	85.23	50.49	75.51	67.86	73.80	92.54	91.41	90.34	94.08	94.17
	Recall	85.30	50.54	75.84	67.62	73.56	92.22	91.17	90.44	94.08	93.70
	F1-score	84.93	50.38	75.45	67.59	72.92	92.35	91.26	90.29	94.08	93.88
5-folds	Accuracy	71.42	73.17	85.71	50.00	71.42	100	90.47	100	90.69	100
	Precision	70.36	71.89	84.86	53.11	72.11	100	89.91	100	89.72	100
	Recall	70.67	72.13	84.24	50.59	73.79	100	90.69	100	88.54	100
	F1-score	70.43	71.95	84.34	58.17	72.07	100	90.22	100	90.22	100
8-folds	Accuracy	57.69	61.53	73.07	84.61	84.61	100	100	100	100	100
	Precision	66.02	62.32	72.68	83.93	83.33	100	100	100	100	100
	Recall	58.88	62.03	73.33	83.93	81.42	100	100	100	100	100
	F1-score	57.32	61.56	72.84	83.80	82.18	100	100	100	100	100

.

• Using reference frame extraction, two-fold cross-validation highest accuracy obtained is 85.29% and 91.17% for with segmentation and without segmentation, respectively, and three-, five-, and eight-fold cross-validation highest accuracy is 100% in both cases.
• Using redundant frame extraction, the best accuracy for two-fold is 87.73% and 98.76%, three-fold is 85.91% and 94.36%, five-fold is 85.71% and 100%, and eight-fold is 84.61% and 100% for with segmentation and without segmentation, respectively.
• The overall accuracy for two-fold is 76.64% and 83.52%, three-fold is 88.69% and 86.95%, five-fold is 89.22% and 96.46%, and eight-fold is 77.50% and 95% for with segmentation and without segmentation, respectively, in the case of the reference frame, and two-fold is 69.36% and 92.39%, three-fold is 71.26% and 92.71%, five-fold is 63.14% and 96.23%, and eight-fold is 72.30% and 100% for with segmentation and without segmentation respectively in the case of the redundant frame. The overall accuracy of two-, three-, five-, and eight-fold for with and without segmentation for reference frame is 83.56% and 90.48%, respectively. The overall accuracy of two-, three-, five-, and eight-fold for with and without segmentation for reference frame is 69.01% and 95.33%, respectively.
• Using five folds provided the best result when testing data are divided into 10%, 20%, 30%, 40%and 50%.
• Based on the output obtained on the accuracy, precision, recall, and F1-score, it can be seen that in most cases, without segmentation gives a better result compared to with segmentation approach. The result that reflects this can be visualized using eight-fold cross-validation, a smaller data distribution using a reference frame.
• It can be concluded that without using segmentation can also be applied to classify regurgitation in the heart. This might not be the case for all types of diagnosis. It might not be valid for cases of data having ground truth segmentation as well. This can be true for fully unsupervised segmentation techniques and not for supervised or semisupervised segmentation. Using the redundant frame approach provides better accuracy in the case of without segmentation.

4.4. Classification Comparison with Existing Methodologies

To classify AR, MR and TR, the results are compared with SVM and PCA-SVM, which were previously used to classify normal and abnormal images or types of regurgitations. The results of our model is compared with the two methodologies as shown in

Table 8. Comparison table for all the methodologies in the classification of heart abnormalities.

Author	Methodologies Used	Types of Classification	Type of Images	Accuracy (%)
Allan [12] (2017)	JICA, PCA, SVM	Types of regurgitation	Static	82
Kumar [11] (2010)	Affine transform, Histogram, Pyramid matching, SVM	Normal or abnormal (hypokinesis)	Static	90.5
Proposed	Binarization, Levelset method, Haralick and GLCM, Random Forest	Types of regurgitation	Videographic	95.33 (Highest obtained accuracy in case of reference frame without segmentation)

. For comparison with other methodologies, taking the average best accuracy of our method, it can be observed that the proposed method is better than the existing method as shown in Table 8. Other parameters are not compared as they are not available in other papers. Moreover, using our data, a comparison is made with PCA + SVM and SVM as a method. Classification is performed using five folds for the existing methodologies, based on the output obtained from Table 6 and Table 7 for the proposed methodology, in which five folds show promising results compared to other folds. The result is provided in

Table 9. Performance metrics for 10%, 20%, 30%, 40% and 50% testing data using Reference frame as *Ref and redundant frame as *Red for with and without segmentation.

Method	Performance Metrics	With Segmentation					Without Segmentation
		10%	20%	30%	40%	50%	10%	20%	30%	40%	50%
SVM (Ref)	Accuracy	0.88	0.66	0.50	0.51	0.50	1	0.57	0.47	0.42	0.34
	Precision	0.90	0.66	0.56	0.52	0.56	1	0.55	0.46	0.44	0.34
	Recall	0.90	0.73	0.50	0.48	0.17	1	0.60	0.53	0.53	0.51
	F1-score	0.88	0.63	0.50	0.48	0.43	1	0.56	0.43	0.43	0.73
PCA-SVM (Ref)	Accuracy	0.70	0.71	0.71	0.64	0.57	0.85	0.57	0.52	0.50	0.51
	Precision	0.66	0.69	0.75	0.66	0.58	0.89	0.72	0.51	0.52	0.51
	Recall	0.50	0.83	0.81	0.74	0.58	0.89	0.49	0.57	0.64	0.70
	F1-score	0.89	0.63	0.50	0.48	0.43	0.86	0.64	0.52	0.50	0.46
SVM (Red)	Accuracy	0.33	0.40	0.42	0.40	0.33	0.90	0.88	0.90	0.90	0.94
	Precision	0.33	0.33	0.66	0.58	0.46	0.77	0.79	0.83	0.83	0.92
	Recall	0.16	0.16	0.27	0.26	0.25	0.91	0.90	0.92	0.92	0.94
	F1-score	0.22	0.22	0.38	0.37	0.32	0.84	0.84	0.87	0.87	0.93
PCA-SVM (Red)	Accuracy	0.54	0.58	0.47	0.51	0.47	1	0.93	0.95	0.93	0.69
	Precision	0.51	0.56	0.47	0.40	0.48	1	0.93	0.94	0.91	0.73
	Recall	0.54	0.57	0.47	0.40	0.48	1	0.93	0.96	0.94	0.80
	F1-score	0.53	0.56	0.46	0.50	0.48	1	0.93	0.95	0.92	0.77
Proposed (Ref)	Accuracy	1	0.69	0.92	0.84	1	0.90	1	1	1	0.92
	Precision	1	0.55	0.91	0.86	1	0.88	1	1	1	0.95
	Recall	1	0.51	0.96	0.88	1	0.86	1	1	1	0.93
	F1-score	1	0.50	0.93	0.84	1	0.84	1	1	1	0.93
Proposed (Red)	Accuracy	0.71	0.73	0.85	0.50	0.71	1	0.90	1	0.90	1
	Precision	0.70	0.71	0.84	0.53	0.72	1	0.89	1	0.89	1
	Recall	0.70	0.72	0.84	0.50	0.73	1	0.90	1	0.88	1
	F1-score	0.70	0.71	0.84	0.58	0.72	1	0.90	1	0.90	1

.

The following summary has been observed after implementing the SVM, PCA-SVM, and the proposed approach.

• From Table 9, it can be observed that our method provides a better result in most cases compared to PCA-SVM and SVM without segmentation and better accuracy to both the methods in the case when using segmentation.
• In the majority of cases, using 10% testing data provides more promising results than when using 20% or more data. This is mainly due to the data size, which is small in our case.
• The highest accuracy obtained in SVM, PCA-SVM and the proposed approach is 100% which occurs once in SVM and PCA-SVM and eight times using the proposed approach.
• Another observation is made where it can be seen that using the redundant frame extraction method, the result is better when using without any segmentation while using reference frame it is better to use with segmentation. This shows that segmentation is more valid and reliable with a known parameter like ground truth than without any ground truth. Furthermore, this is an important aspect of clinical usage.

4.5. Benefits and Limitations of the Proposed Approach

An expert has validated the classification output. Although no deep learning methodologies are used, based on the few datasets using RF as a classifier has overcome that of the SVM classifier, whereas, for binary classification, SVM has been the most used classification technique. RF is used here since it has the capability of distinguishing multiple classes. Overall, the proposed model works as the primary purpose is classification. Segmentation, in this case, is a step that help reduces the size of our data that can be used for feature extraction. Frame extraction, on the other hand, helps reduces slices of data which is crucial as we want only essential slices and not every slice. Our data already consisted of a frame where the exact location of regurgitation appears, but no ground truth was obtained from it.

Data is not large, and no data augmentation is used to test the model’s capability. This deteriorates the performance of a model like deep learning; therefore, a machine learning RF classifier with texture features is used instead of CNN and other relevant models. This work may not be applicable for larger data or any other data as methodologies used are specifically for heart-related field like the LS method. Segmentation output is fully unsupervised, which is not viable for detecting the region of interest. This model is not clinically validated.

Additionally, this effort is technical and not clinical. Clinical usage of a method needs to be precise with efficiency and effectiveness. If more data are fed, the system might not perform well, and we may require a denser model, like deep-learning model that can handle this easily.

5. Conclusions

The use of automated techniques in echo to classify heart regurgitation and its diagnosis has recently emerged in cardiology. In most of the work, researchers identified regurgitation from the image file. In this work, we have designed an automated machine learning-based technique that can be useful in echo regurgitation detection and diagnosis from a video file. Three different types of regurgitation, namely AR, MR, and TR, are considered in this work. One of the advantages of this approach is that the generation of keyframes from the video file which reduces the total number of frames by a large margin. From the results, it is observed that in some folds using segmentation and without segmentation in both reference and redundant frames, the accuracy obtained accuracy is 100% with an overall highest accuracy of 95.33%. In comparison to other techniques proposed by the researchers, as shown in Table 9, the output obtained using our approach is much higher in terms of performance accuracies. Identifying regurgitation from real-time video images in clinics is a crucial future aspect in this area. Though the model gives higher performance accuracies, the dataset used to train and validate the results is small. Using this model in a real-time clinic requires a large number of data to train and validate the model. So, dataset collection based on real-time data is essential in this area.

Using such algorithms will provide a solution to the cardiologist, which will aid in diagnosis and may even assist or replace human efforts in the future. A traditional approach is applied here, which can be studied with additional advanced techniques in the future. Future work also includes the classification of regurgitation severity and early detection of regurgitation, which will reduce the chances of damage to the valves. Also, work with more videographic images and experimentation using deep learning and reinforcement learning models, which are not explored in valvular disease, can be an important aspect of research in this field. These state-of-the-art techniques can tackle and provide higher accuracy, scalability, and efficiency when used correctly.