A Novel Inertial Viscosity Algorithm for Bilevel Optimization Problems Applied to Classification Problems

Abstract

Fixed-point theory plays many important roles in real-world problems, such as image processing, classification problem, etc. This paper introduces and analyzes a new, accelerated common-fixed-point algorithm using the viscosity approximation method and then employs it to solve convex bilevel optimization problems. The proposed method was applied to data classification with the Diabetes, Heart Disease UCI and Iris datasets. According to the data classification experiment results, the proposed algorithm outperformed the others in the literature.

1. Introduction

Many real-world hierarchical problems can be modeled mathematically as bilevel problems and appear in many practical applications. They are often encountered in the fields of production and capacity planning [1,2], traffic and transportation [3,4], chemistry [5,6] and management science [7,8], as well as energy networks and markets [9,10]. In addition, Nimana et al. [11] proposed an algorithm combining the incremental proximal gradient method with a smooth penalization technique to solve convex bilevel problems and applied it to image inpainting and binary classification problem.

Nowadays, we are in a world with various types of big data. In order to obtain the benefits of such data, we need to integrate advanced knowledge concerning both theory and methods from many areas, such as mathematics, computer science, statistics, medicine, etc. In mathematics, optimization plays a very important role in classifying and predicting large amounts of data because it can provide deep machine learning algorithms with high accuracy. Among optimization models for machine learning, the bilevel optimization model approach is an efficient one that makes it possible to create intelligent machine learning algorithms for data prediction and classification.

In this work, we study a bilevel problem that is an optimization problem where the constraint is another optimization problem. This problem is formulated as follows:

$$\begin{array}{c}\hfill \underset{x\in S_{*}}{min}\omega \left(x\right),\end{array}$$

(1)

where $\omega :{\mathbb{R}}^n\to \mathbb{R}$ is assumed to be strongly convex and differentiable, while $S_{*}$ is a nonempty set of inner-level problem minimizers given by

$$\begin{array}{c}\hfill \underset{x\in {\mathbb{R}}^n}{min}\{f\left(x\right)+g\left(x\right)\},\end{array}$$

(2)

where $f:{\mathbb{R}}^n\to \mathbb{R}$ is a differentiable and convex function such that $\nabla f$ is $L_f$-Lipschitz-continuous and $g:{\mathbb{R}}^n\to \mathbb{R}\cup \{\infty \}$ is a lower-semicontinuous, proper, convex function.

It can be observed that the above bilevel optimization model contains both inner- and outer-level minimization problems (Equations (1) and (2)). Normally, the minimization problem in Equation (2) can be applied to data prediction and classification; see [12,13]. However, among the solutions to the inner-level problem (Equation (2)), we use the objective function $\omega$ to select solutions that are minimizers of $\omega$. This method can provide more accuracy for data prediction and classification than Equation (2) alone.

The inner-level optimization problem is a constraint on the outer-level optimization problem. There are several algorithms for solving the problem in Equation (2); see [12,14,15].

The proximal gradient (PG) method, also known as the proximal forward–backward technique, is the basic algorithm used to solve the problem in Equation (2) (see [16,17]). It is defined by

$$\begin{array}{c}\hfill x_{n+1}=pro{x}_{{\alpha }_{n}g}(I-{\alpha }_n\nabla f)\left(x_n\right),\end{array}$$

(3)

where ${\alpha }_n>0$ is the step size, $pro{x}_g$ is the proximity operator of g and $\nabla f$ is the gradient of f. The algorithm in Equation (3), which is also known as the forward–backward splitting algorithm (FBSA) [14], is suitable to solve Equation (2) if $\nabla f$ is L-Lipschitz-continuous. The FBSA is also called an iterative denoising method [18] or a fixed-point continuation algorithm [19].

One of the most well-known first-order optimization schemes is the fast iterative shrinkage-thresholding algorithm (FISTA). Beck and Teboulle [15] proposed the FISTA to solve the problem in Equation (2) by using an inertial technique as follows:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{w}_n=pro{x}_{\frac{1}{L}g}(I-\frac{1}{L}\nabla f)\left(x_n\right),\hfill \\ p_{n+1}=(1+\sqrt{1+4p_n^2})/2,\hfill \\ x_{n+1}=y_n+\left(\frac{p_n-1}{p_{n+1}}\right)(w_n-w_{n-1}),n\ge 1,\hfill \end{array}\right.\end{array}$$

(4)

where $x_1=w_0\in {\mathbb{R}}^N$ and $p_1=1$. They applied the FISTA to image restoration problems and showed that the rate of convergence of the FISTA was better than other existing algorithms. The generated sequence’s weak convergence was then proved by Liang and Schonlieb [20], who modified the FISTA by setting $p_{n+1}=(u+\sqrt{v+s{p}_n^2})/2$, where $u,v>0$ and $0<s\le 4$.

It may be noticed that the convex minimization problem and fixed-point problem are related. If $0<\alpha <2/L$, then we know that a forward–backward operator $T:=pro{x}_{\alpha g}(I-\alpha \nabla f)$ is nonexpansive. It is known that $Fix\left(T\right)=argmin\left\{f\right(x)+g(x\left)\right\}$. Fixed-point problems with nonexpansive mappings have been investigated by many authors using the method of viscosity approximation [21,22,23,24]. This method provides a strong convergence result and it is defined by the following:

$$\begin{array}{c}\hfill x_{n+1}={\beta }_{n}S\left(x_n\right)+(1-{\beta }_n)T{x}_n,n\ge 1,\end{array}$$

(5)

where $x_1\in H$, $S:H\to H$ is a contraction when H is a Hilbert space and $\left\{{\beta }_n\right\}\in (0,1)$. We can also call Equation (5) the viscosity forward–backward algorithm if $T:=pro{x}_{\alpha g}(I-\alpha \nabla f)$.

In 2014, Beck et al. [25] introduced a new, direct first-order method to solve the problem in Equation (1) and established its convergence results under some suitable conditions, as well as the rate of convergence of the sequence of function values. After that, Sabach and Shtern [26] proposed the following algorithm, called the bilevel gradient sequential averaging method (BiG-SAM), to solve the problems in Equations (1) and (2). The iterative process can be defined as follows

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{u}_n=pro{x}_{cg}(x_{n-1}-c\nabla f\left(x_{n-1}\right)),\hfill \\ v_n=x_{n-1}-\lambda \nabla \omega \left(x_{n-1}\right),\hfill \\ x_{n+1}={\gamma }_n{v}_n+(1-{\gamma }_n)u_n,n\ge 1,\hfill \end{array}\right.\end{array}$$

(6)

where $x_0\in {\mathbb{R}}^n,$$c\in (0,1/L_f],$$\lambda \in (0,2/(L_{\omega }+\sigma )],$ in which $L_f$ and $L_{\omega }$ are the Lipschitz constants of $\nabla f$ and $\nabla \omega$, and $\left\{{\gamma }_n\right\}$ satisfies certain conditions from [22]. In terms of the values of the inner objective function, the authors of [22] studied and analyzed the convergence behavior of the BiG-SAM with a nonasymptotic $O(1/n)$ global rate of convergence.

In 2019, Shehu et al. [27] introduced an inertial extrapolation step into BiG-SAM (Equation (6)), calling the result the inertial bilevel gradient sequential averaging method (iBiG-SAM), to solve the problems in Equations (1) and (2). This iterative scheme is defined by

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{s}_n=x_n+{\theta }_n(x_n-x_{n-1}),\hfill \\ u_n=pro{x}_{cg}(I-c\nabla f)\left(s_n\right),\hfill \\ v_n=s_n-\lambda \nabla \omega \left(s_n\right),\hfill \\ x_{n+1}={\gamma }_n{v}_n+(1-{\gamma }_n)u_n,n\ge 1.\hfill \end{array}\right.\end{array}$$

(7)

In their study, they presented a strong convergence analysis of an inertial algorithm that can be used to approximate fixed points of nonexpansive mappings in infinite-dimensional real Hilbert spaces. Furthermore, they converted the bilevel optimization problems into a fixed-point problem of nonexpansive mappings and showed its convergence under certain conditions.

In 2022, Duan and Zhang [28] introduced an alternated inertial step into BiG-SAM to create an alternated inertial bilevel gradient sequential averaging method (aiBiG-SAM) for solving convex bilevel optimization problems. It is defined as

$$\begin{array}{c}\hfill s_n=\left\{\begin{array}{c}{x}_n\mathrm{if}n\mathrm{is}\mathrm{even},\hfill \\ x_n+{\theta }_n(x_n-x_{n-1}) \mathrm{if}n\mathrm{is}\mathrm{odd},\hfill \end{array}\right.\end{array}$$

and

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{u}_n=pro{x}_{cg}(I-c\nabla f)\left(s_n\right),\hfill \\ v_n=s_n-\lambda \nabla \omega \left(s_n\right),\hfill \\ x_{n+1}={\gamma }_n{v}_n+(1-{\gamma }_n)u_n,n\ge 1.\hfill \end{array}\right.\end{array}$$

(8)

They proved that the aiBiG-SAM converges strongly to a solution for the problem and extended the method into a more general alternating inertial acceleration method.

Recently, in [29,30], the authors proposed new bilevel optimization methods within the framework of Hilbert spaces and proved the strong convergence of their algorithms using the viscosity approximation technique.

In this paper, motivated by these results, we present a novel accelerated algorithm using the viscosity approximation method and the inertial parameter of the FISTA to solve the convex bilevel optimization problem. We then demonstrate the efficacy of this algorithm in solving data classification problems.

The paper is organized as follows. In Section 2, we present the preliminaries in terms of definitions, notations and lemmas for proving the main results. The new accelerated viscosity-type algorithm is introduced and studied, and then we apply it to solving the convex bilevel optimization problems described in Section 3. Then, in Section 4, we present mathematical models for the classification of datasets and the application of the results obtained in the previous section, and we provide numerical experimental results in Section 5. Finally, we present the conclusions and future work in Section 6.

2. Preliminaries

In this section, we present fundamental ideas and principles that will be utilized in the the rest of the research.

Throughout the present paper, H denotes a real Hilbert space with norm $\parallel ·\parallel$ and inner product $\langle ·,·\rangle$, $\mathbb{R}$ is the set of real numbers and $\mathbb{N}$ is the set of positive integers. I denotes the identity operator on H. Let C be a nonempty subset of H and let T be a mapping of C into itself. The strong convergence of a sequence $\left\{x_n\right\}$ in H to $x\in H$ is denoted by $x_n\to x$, weak convergence by $x_n\rightharpoonup x$ and $Fix\left(T\right)$ symbolizes the set of all fixed points of T.

For this work, nonlinear mappings from the following classes were essentially needed.

Definition 1.

The mapping $T:C\to C$ is said to be L-Lipschitz with $L\ge 0$ if

$$\parallel Tu-Tv\parallel \le L\parallel u-v\parallel$$

for all $u,v\in C$.

An L-Lipschitz mapping T is said to be a contraction mapping if $L\in [0,1)$, and it is nonexpansive if $L=1$.

Definition 2

([31]). Let T be a nonexpansive mapping of C into itself and let $T_n:C\to C$ be a family of nonexpansive mappings such that $\varnothing \ne Fix\left(T\right)\subset \Gamma :={\bigcap }_{n=1}^{\infty }Fix\left(T_n\right)$, where $Fix(T_n$) is the set of all fixed points of $T_n$ for each $n\ge 1$. Then, $\left\{T_n\right\}$ is said to satisfy the NST-condition (I) with T if, for any bounded sequence $\left\{x_n\right\}\subset C$,

$$\underset{n\to \infty }{lim}\parallel x_n-T_n{x}_n\parallel =0\mathit{implies}\underset{n\to \infty }{lim}\parallel x_n-T{x}_n\parallel =0.$$

Definition 3

([32,33]). For any bounded sequence $\left\{u_n\right\}$ in H, a family $\left\{T_n\right\}$ of nonexpansive mappings $T_n:C\to C$ with ${\bigcap }_{n=1}^{\infty }Fix\left(T_n\right)\ne \varnothing$ is said to satisfy the condition (Z) if

$$\underset{n\to \infty }{lim}\parallel u_n-T_n{u}_n\parallel =0.$$

Then, every weak cluster point of $\left\{u_n\right\}$∈${\bigcap }_{n=1}^{\infty }Fix\left(T_n\right)$.

Using the demicloseness of $I-T$ where $T:C\to C$ is a nonexpansive mapping, we obtain the following remark.

Remark 1.

Let T be a nonexpansive mapping. Then, $\left\{T_n\right\}$ satisfies the condition (Z) if $\left\{T_n\right\}$ is a family of nonexpansive mappings that satisfies NST-condition (I) with T.

The metric projection $P_C$ from H onto C is defined by

$$P_{C}x=argmin\{\parallel x-y\parallel :y\in C\}$$

for all $x\in H$, where C is a nonempty closed convex subset of H. It is known that $v=P_{C}x$ if and only if $\langle x-v,y-v\rangle \le 0$ for all $y\in C$.

Let us recall the definition of the proximity operator and its properties.

Definition 4

([34,35]). Let $f:H\to \mathbb{R}\cup \{\infty \}$ be a convex, proper and lower-semicontinuous function. The proximity operator of f, denoted by $pro{x}_f$, is defined as follows:

$$\begin{array}{c}\hfill pro{x}_f=\underset{y\in H}{min}f\left(y\right)+\frac{1}{2}{\parallel x-y\parallel }^2\end{array}$$

and it can be formulated in the equivalent form:

$$pro{x}_f={(I+\partial f)}^{-1},$$

where $\partial f$ is the subdifferential of f defined by

$$\partial f\left(x\right):=\{v\in H:f(x)+\langle v,u-x\rangle \le f(u)\mathit{for}\mathit{all}u\in H\}$$

for all $x\in H$. For $\rho >0$, we also know that $pro{x}_{\rho f}$ is firmly nonexpansive and

$$Fix\left(pro{x}_{\rho f}\right)=Argminf:=\{v\in H:f\left(v\right)\le f\left(u\right)\mathit{for}\mathit{all}u\in H\}.$$

Let C be closed convex with $\varnothing \ne C\subset H$. In particular, if $f:=i_C$, the indicator function on C is defined by

$$\begin{array}{c}\hfill i_C\left(x\right)=\left\{\begin{array}{c}0\mathrm{if}x\in C,\hfill \\ \infty \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

Then, $pro{x}_{\rho f}=P_C$.

The following lemmas are well known; see [13,36,37].

Lemma 1

([13]). Let $\mathfrak{g}:H\to \mathbb{R}\cup \{\infty \}$ be a lower-semicontinuous, proper and convex function and let $\mathfrak{f}:H\to \mathbb{R}$ be differentiable and convex such that $\nabla \mathfrak{f}$ is L-Lipschitz-continuous. Let

$$T_n:=pro{x}_{{\rho }_n\mathfrak{g}}(I-{\rho }_n\nabla \mathfrak{f})\mathrm{and}T:=pro{x}_{\rho \mathfrak{g}}(I-\rho \nabla \mathfrak{f}),$$

where ${\rho }_n,\rho \in (0,2/L)$ with ${\rho }_n\to \rho$ as $n\to \infty$. Then, $\left\{T_n\right\}$ satisfies the NST-condition (I) with T.

Lemma 2

([36]). Let $\eta ,\mu \in H$ and $\zeta \in [0,1]$. Then, the following properties hold for H:

(1)

${\parallel \eta +\mu \parallel }^2\le {\parallel \eta \parallel }^2+2\langle \mu ,\eta +\mu \rangle$;

(2)

${\parallel \eta \pm \mu \parallel }^2={\parallel \eta \parallel }^2\pm 2\langle \eta ,\mu \rangle +{\parallel \mu \parallel }^2$;

(3)

${\parallel \zeta \eta +(1-\zeta )\mu \parallel }^2={\zeta \parallel \eta \parallel }^2+{(1-\zeta )\parallel \mu \parallel }^2-\zeta (1-\zeta ){\parallel \eta -\mu \parallel }^2$.

Lemma 3

([37]). Let $\left\{c_n\right\}\subset {\mathbb{R}}_{+},$$\left\{b_n\right\}\subset \mathbb{R}$ and $\left\{t_n\right\}\subset (0,1)$ such that ${\sum }_{n=1}^{\infty }{t}_n=\infty$. Suppose that

$$\begin{array}{c}\hfill c_{n+1}\le (1-t_n)c_n+t_n{b}_n\end{array}$$

for all $n\in \mathbb{N}$. If ${lim\; sup}_{i\to \infty }{b}_{n_i}\le 0$, and for any subsequence $\left\{c_{n_i}\right\}$ of $\left\{c_n\right\}$ satisfying

$$\begin{array}{c}\hfill \underset{i\to \infty }{lim\; inf}(c_{n_i+1}-c_{n_i})\ge 0,\end{array}$$

then ${lim}_{n\to \infty }{c}_n=0$.

In the next section, we introduce an inertial viscosity modified SP algorithm and its application to the convex bilevel optimization problem.

3. Proposed Method

We first present a new inertial viscosity algorithm and prove a strong convergence theorem under mild conditions as follows.

Let C be closed convex with $\varnothing \ne C\subset H$ and the mapping $S:C\to C$ be a k-contraction where $0<k<1$. Let $\left\{T_n\right\}$ be a family of nonexpansive mappings of C onto itself satisfying the condition (Z) such that $\mathsf{\Gamma }:={\bigcap }_{n=1}^{\infty }Fix\left(T_n\right)\ne \varnothing$.

Many mathematicians often use inertial-type extrapolation [38,39] in optimization problems to speed up the convergence of iterative methods by using the technical term ${\theta }_n(x_n-x_{n-1})$. The momentum $x_n-x_{n-1}$ is controlled by the parameter ${\theta }_n$, also known as an inertial parameter.

In 2012, Phuengrattana and Suantai [40] introduced an SP algorithm and showed that its convergence behavior is better than Mann and Ishikawa iterations [41,42]. By using the idea of the SP algorithm, in this paper, we introduce an inertial viscosity modified SP algorithm (IVMSPA) for obtaining a common fixed point for $\left\{T_n\right\}$ as follows.

The following theorem establishes strong convergence for the proposed algorithm.

Theorem 1.

A sequence $\left\{x_n\right\}$ generated by Algorithm 1 converges strongly to an element $\breve{a}\in \mathsf{\Gamma }$, where $\breve{a}=P_{\mathsf{\Gamma }}S\left(\breve{a}\right)$, provided that the sequences $\left\{{\alpha }_n\right\}$, $\left\{{\beta }_n\right\}$, $\left\{{\gamma }_n\right\}$ and $\left\{{\tau }_n\right\}$ satisfy the following conditions:

(C1)

$0<a_1\le {\beta }_n\le a_2<1$;

(C2)

$0<{\alpha }_n$, ${\gamma }_n<1$, ${lim}_{n\to \infty }{\alpha }_n=0$ and ${\sum }_{n=1}^{\infty }{\alpha }_n=\infty$;

(C3)

${lim}_{n\to \infty }{\tau }_n=0.$

Algorithm 1 An Inertial Viscosity Modified SP Algorithm (IVMSPA)

Initialization: Let $\left\{{\alpha }_n\right\}$, $\left\{{\beta }_n\right\}$, $\left\{{\gamma }_n\right\}$ and $\left\{{\tau }_n\right\}$ be sequences of positive real  numbers. Take $x_0$, $x_1\in H$ arbitrarily.  Iterative steps: For $n\ge 1,$ calculate $x_{n+1}$ as follows:  Step 1. Compute an inertial parameter $$\begin{array}{c}\hfill {\theta }_n=\left\{\begin{array}{cc}min\left\{\frac{p_n-1}{p_{n+1}},\frac{{\alpha }_n{\tau }_n}{\parallel x_n-x_{n-1}\parallel }\right\}\hfill & \mathrm{if}{x}_n\ne x_{n-1},\hfill \\ \frac{p_n-1}{p_{n+1}}\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$  where $p_1=1$ and $p_{n+1}=\frac{1+\sqrt{1+4p_n^2}}{2}$.  Step 2. Compute $$\begin{array}{c}\hfill \begin{array}{c}{y}_n=x_n+{\theta }_n(x_n-x_{n-1}),\hfill \\ z_n=(1-{\alpha }_n)y_n+{\alpha }_{n}S\left(y_n\right),\hfill \\ w_n=(1-{\beta }_n)z_n+{\beta }_n{T}_n{z}_n,\hfill \\ x_{n+1}=(1-{\gamma }_n)w_n+{\gamma }_n{T}_n{w}_n.\hfill \end{array}\end{array}$$

Proof. 

Let $\breve{a}=P_{\mathsf{\Gamma }}S\left(\breve{a}\right)$. Then, $\breve{a}\in {\bigcap }_{n=1}^{\infty }F\left(T_n\right)$. First of all, we show that $\left\{x_n\right\}$ is bounded. From Algorithm 1, we have

$$\begin{array}{cc}\hfill \parallel w_n-\breve{a}\parallel & \le (1-{\beta }_n)\parallel z_n-\breve{a}\parallel +{\beta }_n\parallel T_n{z}_n-\breve{a}\parallel \hfill \\ \hfill & \le (1-{\beta }_n)\parallel z_n-\breve{a}\parallel +{\beta }_n\parallel z_n-\breve{a}\parallel \hfill \\ \hfill & =\parallel z_n-\breve{a}\parallel \hfill \end{array}$$

(9)

and

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\breve{a}\parallel & \le {\gamma }_n\parallel w_n-\breve{a}\parallel +(1-{\gamma }_n)\parallel T_n{w}_n-\breve{a}\parallel \hfill \\ \hfill & \le {\gamma }_n\parallel w_n-\breve{a}\parallel +(1-{\gamma }_n)\parallel w_n-\breve{a}\parallel \hfill \\ \hfill & =\parallel w_n-\breve{a}\parallel \hfill \\ \hfill & \le \parallel z_n-\breve{a}\parallel .\hfill \end{array}$$

(10)

From the definition of $y_n$ and $z_n$, we obtain

$$\begin{array}{cc}\hfill \parallel z_n-\breve{a}\parallel & \le {\alpha }_n\parallel S\left(y_n\right)-\breve{a}\parallel +(1-{\alpha }_n)\parallel y_n-\breve{a}\parallel \hfill \\ \hfill & \le {\alpha }_n\parallel S\left(y_n\right)-S\left(\breve{a}\right)\parallel +{\alpha }_n\parallel S\left(\breve{a}\right)-\breve{a}\parallel +(1-{\alpha }_n)\parallel y_n-\breve{a}\parallel \hfill \\ \hfill & \le {\alpha }_{n}k\parallel y_n-\breve{a}\parallel +{\alpha }_n\parallel S\left(\breve{a}\right)-\breve{a}\parallel +(1-{\alpha }_n)\parallel y_n-\breve{a}\parallel \hfill \\ \hfill & =\left[1-{\alpha }_n(1-k)\right]\parallel y_n-\breve{a}\parallel +{\alpha }_n\parallel S\left(\breve{a}\right)-\breve{a}\parallel \hfill \\ \hfill & \le \left[1-{\alpha }_n(1-k)\right]\left[\parallel x_n-\breve{a}\parallel +{\theta }_n\parallel x_{n-1}-x_n\parallel \right]+{\alpha }_n\parallel S\left(\breve{a}\right)-\breve{a}\parallel \hfill \\ \hfill & \le \left[1-{\alpha }_n(1-k)\right]\parallel x_n-\breve{a}\parallel +{\alpha }_n\left(\frac{{\theta }_n}{{\alpha }_n}\parallel x_{n-1}-x_n\parallel +\parallel S\left(\breve{a}\right)-\breve{a}\parallel \right).\hfill \end{array}$$

From (C3), we have

$$\begin{array}{c}\hfill \frac{{\theta }_n}{{\alpha }_n}\parallel x_{n-1}-x_n\parallel \to 0\mathrm{as}n\to \infty .\end{array}$$

(11)

From Equation (11), we know that there exists $M>0$ such that $\frac{{\theta }_n}{{\alpha }_n}\parallel x_{n-1}-x_n\parallel \le M$ for all $n\ge 1$. Thus,

$$\begin{array}{cc}\hfill \parallel z_n-\breve{a}\parallel & \le \left[1-(1-k){\alpha }_n\right]\parallel x_n-\breve{a}\parallel +(1-k){\alpha }_n\left(\frac{\parallel S\left(\breve{a}\right)-\breve{a}\parallel +M}{1-k}\right)\hfill \\ \hfill & \le max\left\{\parallel x_n-\breve{a}\parallel ,\frac{\parallel S\left(\breve{a}\right)-\breve{a}\parallel +M}{1-k}\right\}.\hfill \end{array}$$

From Equation (10) and the above inequality, we obtain

$$\begin{array}{c}\hfill \parallel x_{n+1}-\breve{a}\parallel \le max\left\{\parallel x_n-\breve{a}\parallel ,\frac{\parallel S\left(\breve{a}\right)-\breve{a}\parallel +M}{1-k}\right\}.\end{array}$$

Using mathematical induction, we have

$$\begin{array}{c}\hfill \parallel x_n-\breve{a}\parallel \le max\left\{\parallel x_1-\breve{a}\parallel ,\frac{\parallel S\left(\breve{a}\right)-\breve{a}\parallel +M}{1-k}\right\}\end{array}$$

for all $n\ge 1$. It follows that $\left\{x_n\right\}$ is bounded and, hence, $\left\{z_n\right\}$ is bounded. According to part (3) of Lemma 2, we obtain

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\breve{a}{\parallel }^2& ={\gamma }_n\parallel T_n{w}_n-\breve{a}{\parallel }^2+(1-{\gamma }_n)\parallel w_n-\breve{a}{\parallel }^2-(1-{\gamma }_n){\gamma }_n{\parallel w_n-T_n{w}_n\parallel }^2\hfill \\ \hfill & \le (1-{\gamma }_n)\parallel w_n-\breve{a}{\parallel }^2+{\gamma }_n{\parallel w_n-\breve{a}\parallel }^2\hfill \\ \hfill & =\parallel w_n-\breve{a}{\parallel }^2\hfill \end{array}$$

(12)

and

$$\begin{array}{cc}\hfill \parallel w_n-\breve{a}{\parallel }^2& ={\beta }_n\parallel T_n{z}_n-\breve{a}{\parallel }^2+(1-{\beta }_n)\parallel z_n-\breve{a}{\parallel }^2-(1-{\beta }_n){\beta }_n{\parallel z_n-T_n{z}_n\parallel }^2\hfill \\ \hfill & =\parallel z_n-\breve{a}{\parallel }^2-(1-{\beta }_n){\beta }_n{\parallel z_n-T_n{z}_n\parallel }^2.\hfill \end{array}$$

(13)

Using Lemma 2, we obtain

$$\begin{array}{cc}\hfill \parallel z_n-\breve{a}{\parallel }^2& \le \parallel (1-{\alpha }_n)(y_n-\breve{a})+{\alpha }_n(S\left(y_n\right)-S\left(\breve{a}\right)){\parallel }^2+2{\alpha }_n\langle S\left(\breve{a}\right)-\breve{a},z_n-\breve{a}\rangle \hfill \\ \hfill & \le (1-{\alpha }_n)\parallel y_n-\breve{a}{\parallel }^2+{\alpha }_n{\parallel S\left(y_n\right)-S\left(\breve{a}\right)\parallel }^2+2{\alpha }_n\langle S\left(\breve{a}\right)-\breve{a},z_n-\breve{a}\rangle \hfill \\ \hfill & \le (1-{\alpha }_n)\parallel y_n-\breve{a}{\parallel }^2+{\alpha }_{n}k{\parallel y_n-\breve{a}\parallel }^2+2{\alpha }_n\langle S\left(\breve{a}\right)-\breve{a},z_n-\breve{a}\rangle \hfill \\ \hfill & =\left[1-{\alpha }_n(1-k)\right]{\parallel y_n-\breve{a}\parallel }^2+2{\alpha }_n\langle S\left(\breve{a}\right)-\breve{a},z_n-\breve{a}\rangle \hfill \end{array}$$

(14)

and

$$\begin{array}{cc}\hfill \parallel y_n-\breve{a}{\parallel }^2& \le \parallel x_n-\breve{a}{\parallel }^2+2{\theta }_n\parallel x_n-\breve{a}\parallel \parallel x_{n-1}-x_n\parallel +{\theta }_n^2{\parallel x_{n-1}-x_n\parallel }^2.\hfill \end{array}$$

(15)

From Equations (12)–(15), we obtain

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\breve{a}{\parallel }^2& \le \parallel z_n-\breve{a}{\parallel }^2-(1-{\beta }_n){\beta }_n{\parallel z_n-T_n{z}_n\parallel }^2\hfill \\ \hfill & =\left[1-(1-k){\alpha }_n\right]{\parallel y_n-\breve{a}\parallel }^2+2{\alpha }_n\langle S\left(\breve{a}\right)-\breve{a},z_n-\breve{a}\rangle \hfill \\ \hfill & -(1-{\beta }_n){\beta }_n{\parallel z_n-T_n{z}_n\parallel }^2\hfill \\ \hfill & \le \left[1-(1-k){\alpha }_n\right]\parallel x_n-\breve{a}{\parallel }^2+2{\theta }_n\parallel x_n-\breve{a}\parallel \parallel x_{n-1}-x_n\parallel +{\theta }_n^2{\parallel x_{n-1}-x_n\parallel }^2\hfill \\ \hfill & +2{\alpha }_n\langle S\left(\breve{a}\right)-\breve{a},z_n-\breve{a}\rangle -(1-{\beta }_n){\beta }_n{\parallel z_n-T_n{z}_n\parallel }^2\hfill \\ \hfill & =\left[1-(1-k){\alpha }_n\right]\parallel x_n-\breve{a}{\parallel }^2-(1-{\beta }_n){\beta }_n{\parallel z_n-T_n{z}_n\parallel }^2+(1-k){\alpha }_n{b}_n,\hfill \end{array}$$

(16)

where

$$\begin{array}{cc}\hfill b_n& =\frac{1}{1-k}\left[2\langle S\left(\breve{a}\right)-\breve{a},z_n-\breve{a}\rangle +{\theta }_n\parallel x_{n-1}-x_n\parallel \left(\frac{{\theta }_n}{{\alpha }_n}\parallel x_{n-1}-x_n\parallel \right)\right.\hfill \\ \hfill & \left.+2\parallel x_n-\breve{a}\parallel \left(\frac{{\theta }_n}{{\alpha }_n}\parallel x_{n-1}-x_n\parallel \right)\right].\hfill \end{array}$$

It follows that

$$\begin{array}{c}\hfill (1-{\beta }_n){\beta }_n\parallel z_n-T_n{z}_n{\parallel }^2\le \parallel x_n-\breve{a}{\parallel }^2-{\parallel x_{n+1}-\breve{a}\parallel }^2+(1-k){\alpha }_n{B}^{{}^{\prime }},\end{array}$$

(17)

where $B^{{}^{\prime }}=sup\{b_n:n\in \mathbb{N}\}$.

We next show that $\left\{x_n\right\}$ converges strongly to $\breve{a}$. To apply Lemma 3, we set $a_n:={\parallel x_n-\breve{a}\parallel }^2$ and $t_n:={\alpha }_n(1-k)$. From Equation (16), we obtain

$$\begin{array}{c}\hfill a_{n+1}\le (1-t_n)a_n+t_n{b}_n.\end{array}$$

Suppose that $\left\{a_{n_i}\right\}$ is a subsequence of $\left\{a_n\right\}$ such that ${lim\; inf}_{i\to \infty }(a_{n_i+1}-a_{n_i})\ge 0$. Using Equation (17) and (C2), we obtain

$$\begin{array}{cc}\hfill \underset{i\to \infty }{lim\; sup}{\beta }_{n_i}\left(1-{\beta }_{n_i}\right){\parallel z_{n_i}-T_{n_i}{z}_{n_i}\parallel }^2& \le \underset{i\to \infty }{lim\; sup}\left(a_{n_i}-a_{n_i+1}+{\alpha }_{n_i}(1-k)B^{{}^{\prime }}\right)\hfill \\ \hfill & =\underset{i\to \infty }{lim\; sup}(a_{n_i+1}-a_{n_i})\hfill \\ \hfill & \le 0.\hfill \end{array}$$

(18)

From (C1) and Equation (18), we obtain

$$\begin{array}{c}\hfill \underset{i\to \infty }{lim}\parallel z_{n_i}-T_{n_i}{z}_{n_i}\parallel =0.\end{array}$$

(19)

Next, we show that ${lim\; sup}_{i\to \infty }{b}_{n_i}\le 0$. Obviously, it suffices to show that

$$\begin{array}{c}\hfill \underset{i\to \infty }{lim\; sup}〈S\left(\breve{a}\right)-\breve{a},z_{n_i}-\breve{a}〉\le 0.\end{array}$$

Since $\left\{z_{n_i}\right\}$ is bounded, there exists a subsequence $\{z_{n_{i_j}}\}$ of $\{z_{n_i}\}$ and $y\in H$ such that $z_{n_{i_j}}\rightharpoonup y$ as $j\to \infty$ and

$$\underset{i\to \infty }{lim\; sup}〈S\left(\breve{a}\right)-\breve{a},z_{n_i}-\breve{a}〉=\underset{j\to \infty }{lim}〈S\left(\breve{a}\right)-\breve{a},z_{n_{i_j}}-\breve{a}〉.$$

Since $\left\{T_n\right\}$ satisfies the condition (Z) and Equation (19), we obtain $y\in \mathsf{\Gamma }$. Using $\breve{a}=P_{\mathsf{\Gamma }}S\left(\breve{a}\right)$, we get

$$\underset{j\to \infty }{lim}〈S\left(\breve{a}\right)-\breve{a},z_{n_{i_j}}-\breve{a}〉\le 0.$$

So, we have

$$\underset{i\to \infty }{lim\; sup}〈S\left(\breve{a}\right)-\breve{a},z_{n_i}-\breve{a}〉\le 0.$$

Thus, in view of Lemma 3, $\left\{x_n\right\}$ converges to $\breve{a}$ as required.    □

We next establish our inertial bilevel gradient modified SP algorithm (IBiG-MSPA) to solve the convex bilevel optimization problems in Equations (1) and (2) by applying Algorithm 1 and present its strong convergence. We use the following assumptions in order to solve this problem:

• $f:H\to \mathbb{R}$ is a convex and differentiable function such that $\nabla f$ is Lipschitz-continuous with constant $f>0$ and $g:H\to (-\infty ,\infty ]$ are proper, lower-semicontinuous and convex functions;
• $\omega :{\mathbb{R}}^n\to \mathbb{R}$ is strongly convex with a parameter $\sigma$ such that $\nabla \omega$ is $L_{\omega }$-Lipschitz continuous and $s\in (0,\frac{2}{L_{\omega }+\sigma })$.

Our IBiG-MSPA algorithm is defined as follows.

The following useful result was proved by Sabach and Shtern [26].

Proposition 1.

Suppose that $\omega :{\mathbb{R}}^n\to \mathbb{R}$ is strongly convex with $\sigma >0$ and $\nabla \omega$ is Lipschitz-continuous with constant $L_{\omega }$. Then, the mapping defined by $S_s=I-s\nabla \omega$ is a contraction for all $s\in (0,\frac{2}{\sigma +L_{\omega }})$. Thus,

$$\begin{array}{c}\hfill \parallel x-s\nabla \omega \left(u\right)-(v-s\nabla \omega \left(v\right))\parallel \le \sqrt{1-\frac{2s\sigma L_{\omega }}{\sigma +L_{\omega }}}\parallel u-v\parallel \end{array}$$

for all $u,v\in {\mathbb{R}}^n$.

Combining Theorem 1 and Proposition 1, we obtain the following result.

Theorem 2.

Let Λ be the set of all solutions to Equation (1) and $\breve{a}=P_{S_{*}}(I-s\nabla \omega )\left(\breve{a}\right)$ and let (C1)–(C3) in Theorem 1 hold. Then, $\left\{x_n\right\}$ generated by Algorithm 2 converges strongly to $\breve{a}\in \mathsf{\Lambda }.$

Algorithm 2 An Inertial Bilevel Gradient Modified SP Algorithm (IBiG-MSPA)

Initialization: Let $\left\{{\alpha }_n\right\}$, $\left\{{\beta }_n\right\}$, $\left\{{\gamma }_n\right\}$, $\left\{{\tau }_n\right\}$ and $\left\{c_n\right\}$ be sequences of positive  real numbers. Take $x_0$, $x_1\in H$ arbitrarily.  Iterative steps: For $n\ge 1,$ calculate $x_{n+1}$ as follows:  Step 1. Compute an inertial parameter $$\begin{array}{c}{\theta }_n=\left\{\begin{array}{cc}min\left\{\frac{p_n-1}{p_{n+1}},\frac{{\alpha }_n{\tau }_n}{\parallel x_n-x_{n-1}\parallel }\right\}\hfill & \mathrm{if}{x}_n\ne x_{n-1},\hfill \\ \frac{p_n-1}{p_{n+1}}\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$  where $p_1=1$ and $p_{n+1}=\frac{1+\sqrt{1+4p_n^2}}{2}$.  Step 2. Compute $$\begin{array}{c}{y}_n=x_n+{\theta }_n(x_n-x_{n-1}),\hfill \\ z_n=(1-{\alpha }_n)y_n+{\alpha }_n(I-s\nabla \omega )y_n,\hfill \\ w_n=(1-{\beta }_n)z_n+{\beta }_{n}pro{x}_{c_{n}g}(I-c_n\nabla f)z_n,\hfill \\ x_{n+1}=(1-{\gamma }_n)w_n+{\gamma }_{n}pro{x}_{c_{n}g}(I-c_n\nabla f)w_n.\hfill \end{array}$$

Proof. 

Set $S=I-s\nabla \omega$ and $T_n=pro{x}_{c_{n}g}(I-c_n\nabla f)$. Then, according to Proposition 1, S is a contraction mapping. We also know that $T_n$ is nonexpansive. Using Theorem 1, we can conclude that $x_n\to \breve{a}\in \mathsf{\Gamma }$, where $\breve{a}=P_{\mathsf{\Gamma }}S\left(\breve{a}\right)$. It can be noted that, in this case, $\mathsf{\Gamma }={\bigcap }_{n=1}^{\infty }F\left(T_n\right)=S_{*}$. Then, for all $x\in S_{*}$, we have

$$\begin{array}{cc}\hfill 0& \ge \langle S\left(\breve{a}\right)-\breve{a},x-\breve{a}\rangle =\langle \breve{a}-s\nabla \omega \left(\breve{a}\right)-\breve{a},x-\breve{a}\rangle =\langle -s\nabla \omega \left(\breve{a}\right),x-\breve{a}\rangle .\hfill \end{array}$$

Dividing the above inequalities by $-s$, we obtain

$$\begin{array}{c}\hfill \langle \nabla \omega \left(\breve{a}\right),x-\breve{a}\rangle \ge 0\end{array}$$

for all $x\in S_{*}$. Then, $x_n\to \breve{a}\in \mathsf{\Lambda }$. This completes the proof. □

Using our main results (Theorems 1 and 2), we apply the IBiG-MSPA in the next section to solve a classification problem.

4. Applications with Classification Problems

There are several mathematical models used for the classification of datasets. For this paper, we use the extreme learning machine model and present the advantages of this model as follows.

The advantages of feedforward neural networks have led to their widespread use in diverse fields over the past few decades. Stated concisely, feedforward neural networks allow for the approximation of complex nonlinear mappings directly from input samples and provide models for numerous natural and artificial phenomena that are difficult to deal with using classical parametric techniques. However, the rendering of feedforward neural networks is time-consuming due to the dependence of the parameters of the different layers and the requirement to configure all of the parameters. One of the widely used feedforward neural networks is the single hidden layer feedforward neural network (SLFN). It has been widely studied in terms of both theory and application because of its learning ability and anti-error ability (see [43,44,45] for more detail).

In order to increase the effectiveness of SLFNs, a development model of a neural learning algorithm called the extreme learning machine (ELM) [46] was recently established. The advantage of the ELM is that hidden node learning parameters, such as input weights and biases, are generated at random and do not need to be adjusted, whereas the output weights can be obtained analytically by using a simply generalized inverse operation. The ELM has been effectively used in several real-world applications, including regression and classification problems [47,48].

We next examine some aspects of the ELM regarding the classification of data.

Let $\{(x_l,t_l):x_l\in {\mathbb{R}}^n$, $t_l\in {\mathbb{R}}^m$, $l=1,2,\cdots ,N\}$ be a set of training data taken from different samples with a total sample size N, where $x_l=[x_{l1},x_{l2},\cdots ,x_{ln}]\in {\mathbb{R}}^n$ and $t_l=[t_{l1},t_{l2},\cdots ,t_{lm}]\in {\mathbb{R}}^m$ are the input data and target data, respectively. The mathematical formula for an ELM with M hidden nodes is as follows:

$$\sum _{r=1}^M{K}_r\mathfrak{E}(\langle v_r,x_s\rangle +a_r)=o_s,s=1,\cdots ,N,$$

where $\mathfrak{E}\left(x\right)$ represents the activation function, $K_r={[K_{r1},K_{r2},\cdots ,K_{rm}]}^T$ is the weight that links the r-th output node and the hidden node, $v_r={[v_{r1},v_{r2},\cdots ,v_{rn}]}^T$ is the weight that links the r-th input nodes and the hidden node and $a_r$ is a bias.

The purpose of SLFNs is the prediction of N output nodes that satisfy ${\sum }_{s=1}^N\parallel o_s-t_s\parallel =0$. This means that there exist $K_r$, $v_r$ and $a_r$ such that

$$\sum _{r=1}^M{K}_r\mathfrak{E}(\langle v_r,x_s\rangle +a_r)=t_s,s=1,\cdots ,N.$$

From the above system of linear equations, we can rewrite the following:

$$\mathbf{H}K=\mathbf{T},$$

where

$$\mathbf{H}=\left[\begin{array}{ccc}\mathfrak{E}(\langle v_1,x_1\rangle +a_1)& \cdots & \mathfrak{E}(\langle v_M,x_1\rangle +a_M)\\ ⋮& \ddots & ⋮\\ \mathfrak{E}(\langle v_1,x_N\rangle +a_1)& \cdots & \mathfrak{E}(\langle v_M,x_N\rangle +a_M)\end{array}\right],$$

$K={[K_1^T,\cdots ,K_M^T]}_{m\times M}^T,$ and $\mathbf{T}={[t_1^T,\cdots ,t_N^T]}_{m\times N}^T.$

As the Moore–Penrose generalized inverse $\breve{H}$ of $\mathbf{H}$ exists, K can be obtained from $K=\breve{H}\mathbf{T}$ (see [46]). If $\breve{H}$ does not exist, then it could be impossible to find K using this approach. To solve this issue, we determine K as a minimizer of the ordinary least squares minimization problem (OLS):

$$\begin{array}{c}\hfill \underset{K}{min}{\parallel \mathbf{H}K-\mathbf{T}\parallel }_2^2,\end{array}$$

(20)

where $\mathbf{H}\in {\mathbb{R}}^{N\times M}$ is called the hidden layer output matrix, $K\in {\mathbb{R}}^{M\times m}$ is the weight of the output layer, $\mathbf{T}\in {\mathbb{R}}^{N\times m}$ is the training data target matrix, M is the number of hidden nodes and N is the number of training samples.

However, in a real situation, the use of OLS (Equation (20)) may cause an overfitting problem. To overcome such problems, the output weight K can be approximated with the least absolute shrinkage and selection operator (lasso) (see [49]):

$$\begin{array}{c}\hfill \underset{K}{min}{\parallel \mathbf{H}K-\mathbf{T}\parallel }_2^2+\lambda {\parallel K\parallel }_1,\end{array}$$

(21)

where $\lambda$ is a regularization parameter. Now, let $S_{*}$ be the set of all solutions to Equation (21). Among the solutions in $S_{*}$, we would like to select a solution $K^{*}\in S_{*}$ in such a way that $K^{*}$ is a minimizer of

$$\begin{array}{c}\hfill \underset{K\in S_{*}}{min}\frac{1}{2}{\parallel K\parallel }^2.\end{array}$$

(22)

Our aim in the next section is to employ the IBiG-MSPA to solve the convex bilevel optimization problems in Equations (21) and (22) and to use the obtained optimal weight for classification of the Diabetes [50], Heart Disease UCI [51] and Iris datasets [52]. These databases are widely used as benchmarks in many research works in the area of data classification.

5. Numerical Experiments

In this section, we present the experimental results from applying our proposed algorithm to classify the Diabetes, Heart Disease UCI and Iris datasets.

We employed our algorithm (IBiG-MSPA) to solve the convex bilevel optimization problems in Equations (21) and (22) by setting $\omega \left(K\right)=\frac{1}{2}{\parallel K\parallel }_2^2$, $f\left(K\right)={\parallel \mathbf{H}K-\mathbf{T}\parallel }_2^2$, $g\left(K\right)={\lambda \parallel K\parallel }_1$ and $\mathfrak{E}\left(x\right)$ as sigmoid.

The parameters selected for this experiment are shown in

Table 1. Parameter selection for the IBIG-MSPA, BiG-SAM, iBiG-SAM and aiBiG-SAM.

Methods	Settings
IBiG-MSPA	$s=0.01$, $c_n=\frac{1}{L_f}$, ${\alpha }_n=\frac{1}{50n}$, ${\beta }_n={\gamma }_n=0.5$, ${\tau }_n=\frac{{10}^{20}}{n}$
BiG-SAM	$\lambda =0.01$, $c=\frac{1}{L_f}$, ${\gamma }_n=\frac{2\left(0.1\right)}{1-\frac{2+c{L}_f}{4}}$
iBiG-SAM	$\lambda =0.01$, $c=\frac{1}{L_f}$, $\alpha =3$, ${\gamma }_n=\frac{2\left(0.1\right)}{1-\frac{2+c{L}_f}{4}}$, ${\beta }_n=\frac{{\gamma }_n}{n^{0.01}}$ ${\theta }_n=\left\{\begin{array}{c}min\left\{\frac{n}{n+\alpha -1},\frac{{\beta }_n}{\parallel x_n-x_{n-1}\parallel }\right\}\mathrm{if}{x}_n\ne x_{n-1},\hfill \\ \frac{n}{n+\alpha -1} \mathrm{otherwise}.\hfill \end{array}\right.$
aiBiG-SAM	$\lambda =0.01$, $c=\frac{1}{L_f}$, ${\gamma }_n=\frac{2\left(0.1\right)}{1-\frac{2+c{L}_f}{4}}$, ${\beta }_n=\frac{{\gamma }_n}{n^{0.01}}$${\theta }_n=\left\{\begin{array}{c}min\left\{\frac{n}{n+\alpha -1},\frac{{\beta }_n}{\parallel x_n-x_{n-1}\parallel }\right\}\mathrm{if}{x}_n\ne x_{n-1},\hfill \\ \frac{n}{n+\alpha -1} \mathrm{otherwise}.\hfill \end{array}\right.$

, where $L_f$ = ${2\parallel \mathbf{H}\parallel }^2$. We measured the efficiency of each algorithm using the output data accuracy as follows:

$$\begin{array}{c}\hfill \mathrm{accuracy}=100\times \frac{\mathrm{correct} \mathrm{prediction}}{\mathrm{total} \mathrm{cases}}.\end{array}$$

Next, we used the Diabetes, Heart Disease UCI and Iris datasets for classification, as described below.

Diabetes dataset [50]: There are nine features in the dataset. We categorized two data classes in this dataset.

Heart Disease UCI dataset [51]: The dataset has 14 features. Patients’ heart disease data are presented in this dataset. We wanted to categorize the data into two groups.

Iris dataset [52]: The dataset contains four features and three classes. We aimed to classify the data into three types of iris plants.

Testing and training data are shown in

Table 2. Diabetes, Heart Disease UCI and Iris datasets, with 30% of each dataset used for testing and 70% for training.

Datasets	Features	Sample
		Testing Set	Training Set
Diabetes	9	230	538
Heart Disease UCI	14	90	213
Iris	4	45	105

.

For each dataset, the numbers of iterations and hidden nodes were determined as follows

Table 3. Numbers of iterations and hidden nodes specified for each data collection.

Datasets	Number of Iterations ($\overline{\mathit{I}}$)	Number of Hidden Nodes (M)
Diabetes	200	100
Heart Disease UCI	100	60
Iris	300	30

:

The number of iterations for each dataset was chosen to produce the best results for each method under consideration, as can be seen in Table 2.

We conducted experiments to compare the efficiency of the IBiG-MSPA with other algorithms under consideration; namely, the BiG-SAM, iBiG-SAM and aiBiG-SAM.

As representations of the accuracy of testing and training, we use the terms Ac.Test and Ac.Train, respectively, in

Table 4. Accuracy of predictions using various algorithms.

Dataset	IBiG-MSPA		BiG-SAM		iBiG-SAM		aiBiG-SaM
	Ac.Train	Ac.Test	Ac.Train	Ac.Test	Ac.Train	Ac.Test	Ac.Train	Ac.Test
Diabetes	77.11	81.08	71.98	76.13	72.34	76.58	70.88	73.42
Heart Disease UCI	85.71	83.87	74.76	74.19	82.38	78.49	83.81	79.57
Iris	99.05	100.00	94.29	95.56	94.29	95.56	94.29	95.56

.

The results from Table 4 show that our proposed method, the IBiG-MSPA, performed better than the BiG-SAM, iBiG-SAM and aiBiG-SAM in terms of training and testing accuracy for each dataset. Therefore, based on our study, the IBiG-MSPA could classify the chosen datasets with greater accuracy than the other methods.

We can observe that, for the Heart Disease UCI dataset, the accuracies of the existing algorithms were around 70%, while our proposed algorithm achieved accuracy over 80%. In a practical scenario, even small improvements in classification accuracy can have significant effects. For instance, in the case of medical diagnoses, for which the Heart Disease UCI dataset is often used as a benchmark, a slight increase in accuracy can lead to more reliable predictions and better patient outcomes. It may help identify more individuals at risk or improve the overall efficiency of the classification process, leading to appropriate treatments. This observation applies equally well to the other two datasets and datasets similar to them.

6. Conclusions

We first introduced an inertial viscosity modified SP algorithm (IVMSPA). Then, the strong convergence of the IVMSPA was proved under mild conditions with the control sequence. Next, we proposed the inertial bilevel gradient modified SP algorithm (IBiG-MSPA) to solve the convex bilevel optimization problem. Finally, we applied our method to classifying the Diabetes, Heart Disease UCI and Iris datasets. The numerical experiments showed that the IBiG-MSPA had higher efficiency than the BiG-SAM, iBiG-SAM and aiBiG-SAM.

The performances of the algorithms discussed in this paper depend in part on the characteristics of the datasets. In order to improve the accuracy, one needs to address issues related to the preprocessing of the data, such as feature selection, missing data and dataset imbalances. The goal of our future research is to develop new techniques or processes that can improve the efficiency of algorithms in classifying imbalanced and big datasets.