Group Logistic Regression Models with l_p,q Regularization

Abstract

In this paper, we proposed a logistic regression model with $l_{p,q}$ regularization that could give a group sparse solution. The model could be applied to variable-selection problems with sparse group structures. In the context of big data, the solutions for practical problems are often group sparse, so it is necessary to study this kind of model. We defined the model from three perspectives: theoretical, algorithmic and numeric. From the theoretical perspective, by introducing the notion of the group restricted eigenvalue condition, we gave the oracle inequality, which was an important property for the variable-selection problems. The global recovery bound was also established for the logistic regression model with $l_{p,q}$ regularization. From the algorithmic perspective, we applied the well-known alternating direction method of multipliers (ADMM) algorithm to solve the model. The subproblems for the ADMM algorithm were solved effectively. From the numerical perspective, we performed experiments for simulated data and real data in the factor stock selection. We employed the ADMM algorithm that we presented in the paper to solve the model. The numerical results were also presented. We found that the model was effective in terms of variable selection and prediction.

1. Introduction

For regression models, the categorical variables are important for applications and the explanatory variables are always thought of as grouped. Considering the interpretability and the accuracy of the models, the information regarding the group should be considered for the modeling, especially for the high-dimensional settings where sparsity and variable selection can play a very important role in estimation accuracy. Generally speaking, the regression model with the penalized regularizations gives a good result for variable-selection problems; we found lots of research in the literature for this kind of problem [1,2,3,4,5,6,7,8,9,10,11,12]. When the explanatory variables have a group structure, penalized regularization also plays an important role, such as with group least absolute shrinkage and selection operators (LASSOs) [13], the group smoothly clipped absolute deviation (SCAD) penalty [14] and the group minimax concave penalty (MCP) [15] models.

As we all know, the $l_p(0<p<1)$ norm is seen as a good approximation of the $l_0$ norm, and it can recover a more sparse solution than the $l_1$ norm [16]. For the group sparsity of variables with a group structure, the $l_{p,q}$ norm plays an important role in the sparse aspect. The $l_{p,q}$ norm with a group structure is described as follows:

$$\begin{array}{c}\hfill {\parallel \beta \parallel }_{p,q}:={\left(\sum _{i=1}^r{\parallel {\beta }_{{\mathcal{G}}_i}\parallel }_p^q\right)}^{\frac{1}{q}},\end{array}$$

(1)

where $\beta :={({\beta }_{{\mathcal{G}}_1}^T,\cdots ,{\beta }_{{\mathcal{G}}_r}^T)}^T$, and $\{{\beta }_{{\mathcal{G}}_i}\in R^{n_i}:i=1,\cdots ,r\}$ is the grouping of the variable $\beta$. Here ${\displaystyle \sum _{i=1}^r}{n}_i=d+1$, and ${\mathcal{G}}_i$ denotes the index set corresponding to the i-th group. We denote $\mathcal{S},\mathcal{N}$ to be the index sets, and ${\mathcal{G}}_{\mathcal{S}}$ is to denote the index set $\{{\mathcal{G}}_i:i\in \mathcal{S}\}$.

For the linear regression problem with $l_{p,q}$ regularization, the oracle inequality and the global recovery bound were established in the paper [17]. The goodness of $l_{p,q}$ regularization was very obvious. For the logistic regression model, we employed penalized regularization in the variable-selection problems.

We assume that $\beta ={({\beta }_0,{\beta }_1,\cdots ,{\beta }_d)}^T\in R^{d+1}$ are the coefficients of the explanatory variables. The matrix X denotes the explanatory variables, and it is given as follows:

$$X=\left(\begin{array}{cccc}1& X_{11}& \cdots & X_{1d}\\ 1& X_{21}& \cdots & X_{2d}\\ ⋮& ⋮& \ddots & ⋮\\ 1& X_{n1}& \cdots & X_{nd}\end{array}\right)\in R^{n\times (d+1)}.$$

$X_i$ denotes the $i-$th row of the matrix X, $y={(y_1,\cdots ,y_n)}^T$ are the categorical variables and $y_i\in \{0,1\}$.

In this paper, we considered the logistic regression model with the $l_{p,q}$ norm, described as follows:

$$\begin{array}{ccc}\hfill \underset{\beta }{min}& & f\left(\beta \right)+{\lambda \parallel \beta \parallel }_{p,q}^q,\hfill \end{array}$$

(2)

where $f\left(\beta \right):=-\frac{1}{n}{y}^{T}X\beta +\frac{1}{n}{\displaystyle \sum _{i=1}^n}ln(1+exp\left(X_i\beta \right))$ is the loss function and ${\parallel ·\parallel }_{p,q}$ is defined by formulation (1). Moreover, we know $f\left(\beta \right)>0$ from the properties of the logistic regression model, and ${\parallel \beta \parallel }_{p,q}^q={\displaystyle \sum _{i=1}^r}{\parallel {\beta }_{{\mathcal{G}}_i}\parallel }_p^q$, $p\ge 1,0<q<1$ and $\lambda$ is the penalized parameter.

The group LASSO for the logistic regression model [18] is able to perform variable selection on groups of variables, and the model has the following form:

$$\begin{array}{c}\hfill \underset{\beta }{min}f\left(\beta \right)+\lambda \sum _{i=1}^{r}s\left(d{f}_{{\mathcal{G}}_i}\right){\parallel {\beta }_{{\mathcal{G}}_i}\parallel }_2,\end{array}$$

(3)

where $\lambda \ge 0$ controls the amount of penalization, and $s\left(d{f}_{{\mathcal{G}}_i}\right)$ is used to rescale the penalty with respect to the dimensionality of the parameter vector ${\beta }_{{\mathcal{G}}_i}$.

Moreover, a quite general composite absolute penalty for the group sparsity problem is considered in [19], and this model includes the group LASSO as a special case. The group LASSO is an important extension of regularization, and it proposes an $l_2$ regularization for each group and, ultimately, gives the sparsity in a group manner. This property can be found in the numerical experiments.

We found that models (2) and (3) were logistic regression models with different penalized regularizations. They aimed to give a solution with group sparsity. The logistic regression model with $l_{p,q}$ regularization is different with the LASSO logistic regression, and it can give a more sparse solution within a group or between groups by adjusting the values of p and q. This could be found in the numerical experiments.

To illustrate the goodness of model (2), we also introduced the logistic regression problem with the elastic net penalty that could give a sparse solution of the problem, and it is described as follows:

$$\begin{array}{c}\hfill \underset{\beta }{min}f\left(\beta \right)+{\lambda }_1{\parallel \beta \parallel }_1+{\lambda }_2{\parallel \beta \parallel }_2^2,\end{array}$$

(4)

where ${\lambda }_1>0$ and ${\lambda }_2>0$ are the penalized parameters. Model (4) was not good at group sparsity, and we showed this in the numerical parts.

This paper is organized as follows. In Section 2, we introduce the inequalities of $l_{p,q}$ regularization, the properties of the loss function for the logistic regression model, the $(p,q)$-group restricted eigenvalue condition relative to $(S,N)$ ($(p,q)$-GREC(S,N)) and establish the oracle inequality and the global recovery bound for model (2). In Section 3, we apply the ADMM to solve model (2), and we found that the subproblems of the algorithm could be solved efficiently. In Section 4, we employ two numerical experiments that are always used for variable-selection problems to show the goodness of model (2) and the ADMM algorithm. The results for the LASSO logistic regression model (3) and the logistic regression model with the elastic net penalty (4) are compared with those of model (2), and we determine the advantages of model (2). We also give the results for model (2), which produced the real data in Section 5, and the results showed the effectiveness of the model and the algorithm that we gave in the paper. The last section draws conclusions and presents future work.

We introduce some notations that we will use in the following analysis. Let $\mathcal{S}:=\{i\in \{1,\cdots ,r\}:{\beta }_{{\mathcal{G}}_i}\ne 0\}$ be the index set of nonzero groups of $\beta$, ${\mathcal{S}}^c:=\{1,\cdots ,r\}\setminus \mathcal{S}$ be the complement of $\mathcal{S}$, $S:=|\mathcal{S}|$ be the group sparsity of $\beta$. For a variable $\beta \in R^{d+1}$ and $\mathcal{S}\subseteq \{1,\cdots ,d+1\}$, we employ ${\beta }_{\mathcal{S}}$ to denote the subvector of $\beta$ corresponding to $\mathcal{S}$. For a group ${\beta }_{{\mathcal{G}}_i}$, we employ ${\beta }_{{\mathcal{G}}_i}=0$ to describe a zero group, where ${\beta }_{{\mathcal{G}}_i}=0$ means that ${\beta }_j=0$ for all $j\in {\mathcal{G}}_i$. We give $S\le N\ll r$, $\beta \in R^{d+1}$ and $\mathcal{J}\subseteq \{1,\cdots ,r\}$, and we use rank${}_i\left(\beta \right)$ to denote the rank of $\parallel {\beta }_{{\mathcal{G}}_i}{\parallel }_p$ among $\{\parallel {\beta }_{{\mathcal{G}}_j}{\parallel }_p:j\in {\mathcal{J}}^c\}$ (in a decreasing order). We employ $\mathcal{J}(\beta ;N)$ to denote the index set of the first N largest groups in the value of $\parallel {\beta }_{{\mathcal{G}}_i}{\parallel }_p$ among $\{\parallel {\beta }_{{\mathcal{G}}_j}{\parallel }_p:j\in {\mathcal{J}}^c\}$, which means

$$\mathcal{J}(\beta ;N):=\{i\in {\mathcal{J}}^c:{\mathrm{rank}}_i\left(\beta \right)\in \{1,\cdots ,N\}\}.$$

Moreover, we let $R:=\lceil \frac{r-|\mathcal{J}|}{N}\rceil$, and we denote

$$\begin{array}{c}\hfill {\mathcal{J}}_k(x;N):=\left\{\begin{array}{cc}\{i\in {\mathcal{J}}^c:ran{k}_i\left(\beta \right)\in \{kN+1,\cdots ,(k+1)N\}\},& k=1,\cdots ,R-1,\\ \{i\in {\mathcal{J}}^c:ran{k}_i\left(\beta \right)\in \{RN+1,\cdots ,r-|\mathcal{J}|\}\},& k=R.\end{array}\right.\end{array}$$

(5)

2. Theoretical Analysis

In this section, we analyze the oracle property and the global recovery bound of the penalized regression model (2). Firstly, we introduce the following inequalities of the $l_{p,q}$ norm and the properties of the loss function $f\left(\beta \right)$.

Lemma 1

([17], p. 8). Let $0<q\le p\le 2$, $\beta \in R^{d+1}$ and K be the smallest integer such that $2^{K-1}q\ge 1$. Then the following relation holds:

$${\parallel \beta \parallel }_{p,q}^q\le r^{1-2^{-K}}{\parallel \beta \parallel }_{p,2}^q.$$

Lemma 2

([17], p. 9). Let $0<q\le 1\le p$ and ${\beta }_1,{\beta }_2\in R^{d+1}$. Then we have

$$\parallel {\beta }_1{\parallel }_{p,q}^q-\parallel {\beta }_2{\parallel }_{p,q}^q\le {\parallel {\beta }_1-{\beta }_2\parallel }_{p,q}^q.$$

Lemma 3

([17], p. 13). Let $0<q\le 1\le p$, $\tau \ge 1$ and $\beta \in R^{d+1}$, $\mathcal{N}:=\mathcal{J}(\beta ;N)\bigcup \mathcal{J}$ and ${\mathcal{J}}_k:={\mathcal{J}}_k(\beta ;N)$ for $k=1,\cdots ,R$. Then the following inequalities hold:

$$\parallel {\beta }_{{\mathcal{G}}_{{\mathcal{N}}^c}}{\parallel }_{p,\tau }\le \sum _{k=1}^R\parallel {\beta }_{{\mathcal{G}}_{{\mathcal{J}}_k}}{\parallel }_{p,\tau }\le N^{\frac{1}{\tau }-\frac{1}{q}}{\parallel {\beta }_{{\mathcal{G}}_{{\mathcal{J}}^c}}\parallel }_{p,q}.$$

Moreover, the following properties are about the Lipschitz continuity and the convexity of the loss function $f\left(\beta \right)$.

Proposition 1.

For ${\beta }_1,{\beta }_2\in R^{d+1}$, we have

$$\begin{array}{c}\hfill |f\left({\beta }_1\right)-f\left({\beta }_2\right){|\le 2n\parallel X\parallel }_2{\parallel {\beta }_1-{\beta }_2\parallel }_2.\end{array}$$

(6)

Proof. 

For ${\beta }_1,{\beta }_2,\tilde{\beta }\in R^{d+1}$, based on the differential mean value theorem and the properties of the norms we can get

$$\begin{array}{ccc}\hfill |f\left({\beta }_1\right)-f\left({\beta }_2\right)|& =& |-\frac{1}{n}{y}^{T}X({\beta }_1-{\beta }_2)+\frac{1}{n}\sum _{i=1}^n(ln(1+exp\left(X_i{\beta }_1\right))-ln(1+exp\left(X_i{\beta }_2\right)))|\hfill \\ & =& |-\frac{1}{n}{y}^{T}X({\beta }_1-{\beta }_2)+\frac{1}{n}\sum _{i=1}^n\frac{exp\left(X_i\tilde{\beta }\right)X_i^T}{1+exp\left(X_i\tilde{\beta }\right)}({\beta }_1-{\beta }_2)|\hfill \\ & \le & |-\frac{1}{n}{y}^{T}X({\beta }_1-{\beta }_2)|+|\frac{1}{n}\sum _{i=1}^n{X}_i^T({\beta }_1-{\beta }_2)|\hfill \\ & \le & {(n+\parallel y\parallel }_2{)\parallel X\parallel }_2{\parallel {\beta }_1-{\beta }_2\parallel }_2\hfill \\ & \le & {2n\parallel X\parallel }_2{\parallel {\beta }_1-{\beta }_2\parallel }_2.\hfill \end{array}$$

(7)

Hence, we obtain our desirable result.    □

Proposition 2.

For $\beta \in R^{d+1}$, the function $f\left(\beta \right)$ is convex.

Proof. 

From the definition of the function $f\left(\beta \right)$, we know the Hessian matrix of the function $f\left(\beta \right)$ is

$$H_f\left(\beta \right)=\frac{1}{n}\sum _{i=1}^n{X}_i^T{X}_i\frac{exp\left(X_i\beta \right)}{{(1+exp\left(X_i\beta \right))}^2}.$$

Here, we find for $i=1,\cdots ,n$ the matrix $X_i^T{X}_i⪰0$ and $\frac{exp\left(X_i\beta \right)}{{(1+exp\left(X_i\beta \right))}^2}>0$. Thus, the Hessian matrix $H_f\left(\beta \right)$ is a positive semi-definite matrix. Hence, the function $f\left(\beta \right)$ is convex.    □

The above lemmas and properties state the inequalities for $l_{p,q}$ regularization, and they will help the proof of the oracle inequality and the global recovery bound. The oracle inequalities for predicition error were discussed in [20,21], and they were derived without restricted eigenvalue conditions for LASSO-type estimators or sparsity. Morever, for the group LASSO problems, the oracle inequalities were discussed in [22,23,24] under the restricted eigenvalue assumption. For linear regression with $l_{p,q}$ regularizaiton, the oracle inequality was established in [17] with the help of the $(p,q)$-GREC(S,N).

Moreover, the $(p,q)$-GREC(S,N) was very important for the analysis of the oracle property and the global recovery bound of the $l_{p,q}$ norm. We define $\gamma$ to be the smallest non-zero eigenvalue of the Hessian matrix of the function $f\left(\beta \right)$. We introduce it in the following definition.

Definition 1.

Let $0<q\le p\le 2$. The $(p,q)$-GREC(S,N) is said to be satisfied if

$${\varphi }_{p,q}(S,N):=min\left\{\frac{{\gamma \parallel \beta \parallel }_2}{\parallel {\beta }_{{\mathcal{G}}_{\mathcal{N}}}{\parallel }_{p,2}}:|\mathcal{J}|\le S,\parallel {\beta }_{{\mathcal{G}}_{\mathcal{J}}^c}{\parallel }_{p,q}\le {\parallel {\beta }_{{\mathcal{G}}_{\mathcal{J}}}\parallel }_{p,q},\mathcal{N}=\mathcal{J}(\beta ,N)\bigcup \mathcal{J}\right\}>0.$$

The oracle property is an important property for the variable selection, and it gives an upper bound on the square error of the logistic regression problem and the violation of the true nonzero groups for each point in the level set of the objective function of problem (2).

For $\overline{\beta }\in R^{d+1}$, the level set is given as follows:

$$Le{v}_F\left(\overline{\beta }\right):=\{\beta \in R^{d+1}:f\left(\beta \right)+{\lambda \parallel \beta \parallel }_{p,q}^q\le \lambda \parallel \overline{\beta }{\parallel }_{p,q}^q\}.$$

From the definition of the level set in ([25], p. 8), we know that many properties of the optimization problem (2) relate to the level set $Le{v}_F\left(\overline{\beta }\right)$.

Theorem 1.

Let $0<q\le 1\le p$, $S>0$ and let the $(p,q)$-$GREC(S,S)$ hold. Let $\overline{\beta }$ be the unique solution of $\underset{\beta }{min}f\left(\beta \right)$ at a group sparsity level S, and $\mathcal{S}$ be the index set of nonzero groups of $\overline{\beta }$. Let K be the smallest integer such that $2^{K-1}q\ge 1$. Then, for any ${\beta }^{*}\in le{v}_F\left(\overline{\beta }\right)$, which means that $f\left({\beta }^{*}\right)+\lambda \parallel {\beta }^{*}{\parallel }_{p,q}^q\le \lambda {\parallel \overline{\beta }\parallel }_{p,q}^q$, the following oracle inequality holds:

$$\begin{array}{c}\hfill f\left({\beta }^{*}\right)-f\left(\overline{\beta }\right)+\lambda {\parallel {\beta }_{{\mathcal{G}}_{S^c}}^{*}\parallel }_{p,q}^q\le 2^{\frac{q}{2-q}}{\lambda }^{\frac{2}{2-q}}{\gamma }^{\frac{q}{2-q}}{S}^{\frac{2(1-2^{-K})}{2-q}}/{\varphi }_{p,q}^{\frac{2q}{2-q}}(S,S).\end{array}$$

(8)

Moreover, letting ${\mathcal{N}}_{*}:=\mathcal{S}\bigcup \mathcal{S}({\beta }^{*};S)$, we have

$$\parallel {\beta }_{{\mathcal{G}}_{{\mathcal{N}}_{*}}}^{*}-{\overline{\beta }}_{{\mathcal{G}}_{{\mathcal{N}}_{*}}}{\parallel }_{p,2}^2\le 2^{\frac{2}{2-q}}{\gamma }^{\frac{2}{2-q}}{\lambda }^{\frac{2}{2-q}}{S}^{2(1-2^{-K})/(2-q)}/{\varphi }_{p,q}^{\frac{4}{2-q}}(S,S).$$

Proof. 

Let ${\beta }^{*}\in Le{v}_F\left(\overline{\beta }\right)$, and by the definition of the level set $Le{v}_F\left(\overline{\beta }\right)$ we have

$$f\left({\beta }^{*}\right)+\lambda \parallel {\beta }^{*}{\parallel }_{p,q}^q\le \lambda {\parallel \overline{\beta }\parallel }_{p,q}^q.$$

Then by Lemmas 1 and 2 and the fact $f\left(\overline{\beta }\right)>0$, one has the following formulation:

$$\begin{array}{ccc}\hfill f\left({\beta }^{*}\right)-f\left(\overline{\beta }\right)+\lambda {\parallel {\beta }_{{\mathcal{G}}_{{\mathcal{S}}^c}}^{*}\parallel }_{p,q}^q& \le & f\left({\beta }^{*}\right)+\lambda {\parallel {\beta }_{{\mathcal{G}}_{{\mathcal{S}}^c}}^{*}\parallel }_{p,q}^q\hfill \\ & \le & \lambda (\parallel {\overline{\beta }}_{{\mathcal{G}}_{\mathcal{S}}}{\parallel }_{p,q}^q-\parallel {\beta }_{{\mathcal{G}}_{\mathcal{S}}}^{*}{\parallel }_{p,q}^q)\hfill \\ & \le & \lambda \parallel {\overline{\beta }}_{{\mathcal{G}}_{\mathcal{S}}}-{\beta }_{{\mathcal{G}}_{\mathcal{S}}}^{*}{\parallel }_{p,q}^q\hfill \\ & \le & \lambda S^{1-2^{-K}}{\parallel {\overline{\beta }}_{{\mathcal{G}}_{\mathcal{S}}}-{\beta }_{{\mathcal{G}}_{\mathcal{S}}}^{*}\parallel }_{p,2}^q.\hfill \end{array}$$

(9)

Moreover, we find

$$\parallel {\beta }_{{\mathcal{G}}_{{\mathcal{S}}^c}}^{*}-{\overline{\beta }}_{{\mathcal{G}}_{{\mathcal{S}}^c}}{\parallel }_{p,q}^q-\parallel {\beta }_{{\mathcal{G}}_{\mathcal{S}}}^{*}-{\overline{\beta }}_{{\mathcal{G}}_{\mathcal{S}}}{\parallel }_{p,q}^q\le \parallel {\beta }_{{\mathcal{G}}_{{\mathcal{S}}^c}}^{*}{\parallel }_{p,q}^q-(\parallel {\overline{\beta }}_{{\mathcal{G}}_{\mathcal{S}}}{\parallel }_{p,q}^q-\parallel {\beta }_{{\mathcal{G}}_{\mathcal{S}}}^{*}{\parallel }_{p,q}^q)=\parallel {\beta }^{*}{\parallel }_{p,q}^q-{\parallel \overline{\beta }\parallel }_{p,q}^q\le 0.$$

Then, the $(p,q)$-$GREC(S,S)$ implies the following:

$$\parallel {\overline{\beta }}_{{\mathcal{G}}_{\mathcal{S}}}-{\beta }_{{\mathcal{G}}_{\mathcal{S}}}^{*}{\parallel }_{p,2}\le (\gamma \parallel {\beta }^{*}-\overline{\beta }{\parallel }_2)/{\varphi }_{p,q}(S,S).$$

From the expansion of the Taylor formulation, we obtain the following relationship:

$$\begin{array}{ccc}\hfill f\left({\beta }^{*}\right)-f\left(\overline{\beta }\right)& =& \nabla f{\left(\overline{\beta }\right)}^T({\beta }^{*}-\overline{\beta })+\frac{1}{2}{({\beta }^{*}-\overline{\beta })}^T{\nabla }^{2}f\left(\tilde{\beta }\right)({\beta }^{*}-\overline{\beta })\hfill \\ & \ge & \frac{1}{2}{({\beta }^{*}-\overline{\beta })}^T{\nabla }^{2}f\left(\tilde{\beta }\right)({\beta }^{*}-\overline{\beta })\hfill \\ & \ge & \frac{1}{2}\gamma {\parallel {\beta }^{*}-\overline{\beta }\parallel }_2^2,\hfill \end{array}$$

(10)

where $\tilde{\beta }\in \{\tilde{\beta }:\parallel \tilde{\beta }-\overline{\beta }{\parallel }_2\le \parallel {\beta }^{*}-\overline{\beta }{\parallel }_2\}$. The first inequality of formulation (10) is based on the fact that $\overline{\beta }$ is the unique optimal solution of ${min}_{\beta }f\left(\beta \right)$ at group sparsity level $\mathcal{S}$. Moreover, because of the uniqueness of $\overline{\beta }$, we obtain that the smallest eigenvalue of the Hessian matrix of the function $f\left(\beta \right)$ is positive. Hence, we obtain the second inequality.

Then, we get

$$\begin{array}{c}\hfill \gamma \parallel {\beta }^{*}-\overline{\beta }{\parallel }_2^2\le 2(f\left({\beta }^{*}\right)-f\left(\overline{\beta }\right)).\end{array}$$

(11)

Combining this with formulation (9), we get

$$\begin{array}{c}\hfill f\left({\beta }^{*}\right)-f\left(\overline{\beta }\right)+\lambda {\parallel {\beta }_{{\mathcal{G}}_{{\mathcal{S}}^c}}^{*}\parallel }_{p,q}^q\le 2^{\frac{q}{2}}\lambda {\gamma }^{\frac{q}{2}}{S}^{1-2^{-K}}{(f\left({\beta }^{*}\right)-f\left(\overline{\beta }\right))}^{\frac{q}{2}}/{\varphi }_{p,q}^q(S,S).\end{array}$$

(12)

Thus, we have

$$\begin{array}{c}\hfill f\left({\beta }^{*}\right)-f\left(\overline{\beta }\right)\le 2^{\frac{q}{2-q}}{\gamma }^{\frac{q}{2-q}}{\lambda }^{\frac{2}{2-q}}{S}^{2(1-2^{-K})/(2-q)}/{\varphi }_{p,q}^{\frac{2q}{2-q}}(S,S).\end{array}$$

(13)

Hence, by formulations (12) and (13), we obtain the oracle inequality (8). Moreover, from the definition of ${\mathcal{N}}_{*}$, the $(p,q)$-$GREC(S,S)$ implies that

$$\parallel {\beta }_{{\mathcal{G}}_{{\mathcal{N}}_{*}}}^{*}-{\overline{\beta }}_{{\mathcal{G}}_{{\mathcal{N}}_{*}}}{\parallel }_{p,2}^2\le 2\gamma (f\left({\beta }^{*}\right)-f\left(\overline{\beta }\right))/{\varphi }_{p,q}^2(S,S)\le 2^{\frac{2}{2-q}}{\gamma }^{\frac{2}{2-q}}{\lambda }^{\frac{2}{2-q}}{S}^{2(1-2^{-K})/(2-q)}/{\varphi }_{p,q}^{\frac{4}{2-q}}(S,S).$$

Thus, the proof is complete.    □

In the following, we established the global recovery bound for the $l_{p,q}$ regularization problem (2). The global recovery bound shows that the sparse solution $\overline{\beta }$ could be recovered by any point ${\beta }^{*}$ in the level set $le{v}_F\left(\overline{\beta }\right)$. Here ${\beta }^{*}$ is a global optimal solution of problem (2) when the penalized parameter $\lambda$ is small enough. We show the global recovery bound for the $l_{p,q}$ regularization problem (2) in the next theorem.

Theorem 2.

Let $0<q\le 1\le p\le 2$, $S>0$ and suppose the $(p,q)$-GREC(S,S) holds. Let $\overline{\beta }$ be the unique solution of $\underset{\beta }{min}f\left(\beta \right)$ at group sparsity level S, and $\mathcal{S}$ be the index set of nonzero groups of $\overline{\beta }$. Let K be the smallest integer such that $2^{K-1}q\ge 1$. Then, for any ${\beta }^{*}\in le{v}_F\left(\overline{\beta }\right)$, the following global recovery bound for problem (2) holds:

$$\begin{array}{c}\hfill \parallel {\beta }^{*}-\overline{\beta }{\parallel }_2^2\le 2\ast 2^{\frac{2}{2-q}}{\lambda }^{\frac{4}{q(2-q)}}{\gamma }^{\frac{2}{2-q}}{S}^{\frac{q-2}{q}+\frac{4(1-2^{-K})}{q(2-q)}}/{\varphi }_{p,q}^{\frac{4}{2-q}}(S,S).\end{array}$$

(14)

More precisely,

$$\begin{array}{c}\hfill \parallel {\beta }^{*}-\overline{\beta }{\parallel }_2^2\le \left\{\begin{array}{cc}\mathcal{O}\left({\lambda }^{\frac{4}{q(2-q)}}{\gamma }^{\frac{2}{2-q}}S\right),& 2^{K-1}q=1,\\ \mathcal{O}\left({\lambda }^{\frac{4}{q(2-q)}}{\gamma }^{\frac{2}{2-q}}{S}^{\frac{4-q}{2-q}}\right),& 2^{K-1}q>1.\end{array}\right.\end{array}$$

(15)

Proof. 

We suppose ${\mathcal{N}}_{*}:=\mathcal{S}\bigcup \mathcal{S}({\beta }^{*};S)$ as defined in Theorem 1. Since $p\le 2$, from Lemma 3 and Theorem 1, we get

$$\parallel {\beta }_{{\mathcal{G}}_{{\mathcal{N}}_{*}^c}}^{*}{\parallel }_2^2\le \parallel {\beta }_{{\mathcal{G}}_{{\mathcal{N}}_{*}^c}}^{*}{\parallel }_{p,2}^2\le S^{1-2/q}{\parallel {\beta }_{{\mathcal{G}}_{{\mathcal{S}}^c}}^{*}\parallel }_{p,q}^2\le 2^{\frac{2}{2-q}}{\lambda }^{\frac{4}{q(2-q)}}{\gamma }^{\frac{2}{2-q}}{S}^{\frac{q-2}{q}+\frac{4(1-2^{-K})}{q(2-q)}}/{\varphi }_{p,q}^{\frac{4}{2-q}}(S,S).$$

Furthermore, from Theorem 1 and the fact $2^{K-1}q\ge 1$, we get

$$\begin{array}{ccc}& & \parallel {\beta }^{*}-\overline{\beta }{\parallel }_2^2=\parallel {\beta }_{{\mathcal{G}}_{{\mathcal{N}}_{*}}}^{*}-{\overline{\beta }}_{{\mathcal{G}}_{{\mathcal{N}}_{*}}}^{*}{\parallel }_2^2+{\parallel {\beta }_{{\mathcal{G}}_{{\mathcal{N}}_{*}^c}}^{*}\parallel }_2^2\hfill \\ & \le & 2^{\frac{2}{2-q}}{\gamma }^{\frac{2}{2-q}}{\lambda }^{\frac{2}{2-q}}{S}^{2(1-2^{-K})/(2-q)}/{\varphi }_{p,q}^{\frac{4}{2-q}}(S,S)+2^{\frac{2}{2-q}}{\lambda }^{\frac{4}{q(2-q)}}{\gamma }^{\frac{2}{2-q}}{S}^{\frac{q-2}{q}+\frac{4(1-2^{-K})}{q(2-q)}}/{\varphi }_{p,q}^{\frac{4}{2-q}}(S,S)\hfill \\ & \le & 2\ast 2^{\frac{2}{2-q}}{\lambda }^{\frac{4}{q(2-q)}}{\gamma }^{\frac{2}{2-q}}{S}^{\frac{q-2}{q}+\frac{4(1-2^{-K})}{q(2-q)}}/{\varphi }_{p,q}^{\frac{4}{2-q}}(S,S).\hfill \end{array}$$

(16)

Hence, formulation (14) holds.

Moreover, if $2^{K-1}q=1$, we have $\frac{q-2}{q}+\frac{4(1-2^{-K})}{q(2-q)}=1$ and we get

$$\parallel {\beta }^{*}-\overline{\beta }{\parallel }_2^2\le \mathcal{O}({\lambda }^{\frac{4}{q(2-q)}}{\gamma }^{\frac{2}{2-q}}S).$$

If $2^{K-1}q>1$, we know $q>2^{1-K}$. Hence $\frac{q-2}{q}+\frac{4(1-2^{-K})}{q(2-q)}\le \frac{4-q}{2-q}$, and we get

$$\parallel {\beta }^{*}-\overline{\beta }{\parallel }_2^2\le \mathcal{O}({\lambda }^{\frac{4}{q(2-q)}}{\gamma }^{\frac{2}{2-q}}{S}^{\frac{4-q}{2-q}}).$$

Thus, formulation (15) holds.    □

Remark 1.

From Proposition 2, we know that $f\left(\beta \right)$ is convex. By the convexity of the function $f\left(\beta \right)$, we know it is suitable for the assumption for the variable $\overline{\beta }$. The conditions of the variable $\overline{\beta }$ help us to obtain desirable results for Theorems 1 and 2.

3. ADMM Algorithm

In this section, we give an algorithm based on the ADMM algorithm [26,27] for solving the logistic regression model with $l_{p,q}$ regularization (2). The ADMM algorithm performs very well for problems where the variables can be separated. Model (2) can be equivalently described as follows:

$$\begin{array}{ccc}\hfill \underset{\beta }{min}& & f\left(\beta \right)+{\lambda \parallel r\parallel }_{p,q}^q,\hfill \\ \hfill s.t.& & \beta =r.\hfill \end{array}$$

(17)

The augmented Lagrange function of the above model is as follows:

$$\begin{array}{c}\hfill L_{\rho }(\beta ,r,u)=f\left(\beta \right)+{\lambda \parallel r\parallel }_{p,q}^q+u^T(\beta -r)+\frac{\rho }{2}{\parallel \beta -r\parallel }_2^2,\end{array}$$

(18)

where $u\in R^{d+1}$ is the dual variable, and $\rho >0$ is the augmented Lagrange multiplier.

Generally speaking, the structure of the ADMM algorithm is given as follows:

$$\begin{array}{ccc}\hfill {\beta }^{k+1}& =& {\mathrm{argmin}}_{\beta }f\left(\beta \right)+{(\beta -r^k)}^T{u}^k+\frac{\rho }{2}{\parallel \beta -r^k\parallel }_2^2,\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill r^{k+1}& =& {\mathrm{argmin}}_r{\lambda \parallel r\parallel }_{p,q}^q+{({\beta }^{k+1}-r)}^T{u}^k+\frac{\rho }{2}{\parallel {\beta }^{k+1}-r\parallel }_2^2,\hfill \end{array}$$

(20)

$$\begin{array}{ccc}\hfill u^{k+1}& =& u^k+\rho ({\beta }^{k+1}-r^{k+1}).\hfill \end{array}$$

(21)

Based on the above structure and Propositions 1 and 2, the subproblem (19) is an unconstrained convex optimization problem and the objective function is Lipschitz continuous. Based on these good properties of the subproblem (19), we found that it could be effectively solved by many optimal algorithms, such as the trust region algorithm, the sequential quadratic programming algorithm, the algorithm based on the gradient and so on. Moreover, the first order optimal conditions of problem (19) are given by the following formulation:

$$\frac{\partial \left(f\left(\beta \right)+u^{kT}(\beta -r^k)+\frac{\rho }{2}{\parallel \beta -r^k\parallel }_2^2\right)}{\partial \beta }=0_{d+1}.$$

Thus, we obtain the optimal solution for subproblem (19) by solving the following nonlinear equations:

$$\begin{array}{c}\hfill n\rho \beta -\sum _{i=1}^n\frac{X_i^T}{1+exp\left(X_i\beta \right)}=n\rho r^k-n{u}^k+X^T(y-e_n),\end{array}$$

(22)

where $e_n={({\underbrace{1,1,\cdots ,1}}_n)}^T\in R^n$.

From this fact we know that the variable $\beta$ is group sparse. Hence, we can divide the variables ${\beta }^{k+1}$ and $u^k$ by the group structure. ${\beta }_{{\mathcal{G}}_i}^{k+1}$ and $u_{{\mathcal{G}}_i}^{k+1}$ denote the ${\mathcal{G}}_i$th group variables of ${\beta }^{k+1}$ and $u^k$, respectively. Subproblem (20) is solved by employing a group structure. Then, for $i=1,\cdots ,r$, $r_{{\mathcal{G}}_i}^{k+1}$ can be given by solving the following optimization problem:

$$\begin{array}{c}\hfill r_{{\mathcal{G}}_i}^{k+1}={\mathrm{argmin}}_{r^m}\lambda \parallel r_{{\mathcal{G}}_i}{\parallel }_{p,q}^q+{({\beta }_{{\mathcal{G}}_i}^{k+1}-r_{{\mathcal{G}}_i})}^T{u}_{{\mathcal{G}}_i}^k+\frac{\rho }{2}{\parallel {\beta }_{{\mathcal{G}}_i}^{k+1}-r_{{\mathcal{G}}_i}\parallel }_2^2.\end{array}$$

(23)

We can then obtain the solution for subproblem (20) by the following:

$$\begin{array}{c}\hfill r^{k+1}={(r_{{\mathcal{G}}_1}^{k+1T},\cdots ,r_{{\mathcal{G}}_r}^{k+1T})}^T.\end{array}$$

(24)

Moreover, we find that the problem (23) can be equivalently solved by the following one:

$$\begin{array}{c}\hfill {\left(r_{{\mathcal{G}}_i}\right)}^{k+1}={\mathrm{argmin}}_{r_{{\mathcal{G}}_i}}\parallel r_{{\mathcal{G}}_i}-({\beta }_{{\mathcal{G}}_i}^{k+1}+\frac{u_{{\mathcal{G}}_i}^k}{\rho }){\parallel }_2^2+\frac{2\lambda }{\rho }{\parallel r_{{\mathcal{G}}_i}\parallel }_{p,q}^q.\end{array}$$

(25)

The proximal gradient method given in [17] has proven very useful for solving (25). Based on the above analysis, we found that the ADMM algorithm was effective for solving the logistic regression problem with $l_{p,q}$ regularization, and we describe the structure of Algorithm 1 in the following.

Algorithm 1: ADMM algorithm for solving (2).

Step 1: Initialization: give ${\beta }^0$,$u^0$,$r^0$, $\rho >0$, $\lambda >0$, and set $k=0$; Step 2: for $k=0,1,\cdots$, if the stop criteria is satisfied, the algorithm is stopped; otherwise go to Step 3; Step 3: Update ${\beta }^{k+1}$: ${\beta }^{k+1}$ is given by solving nonlinear Equation (22); Step 4: Update $r^{k+1}$: for $i=1,\cdots ,r$, we employ the proximal gradient method to solve the optimization problem (25) and give $r_{{\mathcal{G}}_i}^{k+1}$. Then, we have $$r^{k+1}={(r_{{\mathcal{G}}_1}^{k+1T},\cdots ,r_{{\mathcal{G}}_r}^{k+1T})}^T.$$ Step 5: Update $u^{k+1}$: $u^{k+1}=u^k+\rho ({\beta }^{k+1}-r^{k+1})$.

4. Simulation Examples

In this section, we employ the simulation data to illustrate the efficiency of the logistic regression model with $l_{p,q}$ regularization and the ADMM algorithm. $l_{1/2}$ regularization is shown to give a more sparse optimal solution than the $l_1$ norm [28]. Hence, we employed $q=1/2$ to do our numerical experiments, and we used the ADMM algorithm that we designed in Section 3 to solve model (2). The environment for the simulations was Python 3.7.

In order to verify the effect of the prediction and the classification for penalized logistic regression model (2), we designed two simulation experiments with different data structures. At the same time, we employed the LASSO logistic regression model and the logistic regression model with the elastic net penalty to solve the numerical problems. The numerical results illustrated the advantages of the logistic regression model with $l_{p,q}$ regularization.

We mainly considered two aspects of the effects of the models: the ability of the model to select variables; the ability of the model to test the effects of the classifications and predictions for the penalized logistic regression models. The evaluation indexes for the models in this section mainly included the following:

• P: the number of non-zero coefficients in the variables that the model gives.
• $TP$: the number of coefficients predicted to be non-zero which are actually non-zero.
• $TN$: the number of coefficients predicted to be zero which are actually zero.
• $FP$: the number of coefficients predicted to be non-zero but which are actually zero.
• $FN$: the number of coefficients predicted to be zero but which are actually non-zero.
• $PSR$: the ratio of the number of non-zero coefficients in the variables for the predicted case to that of the true case, which is calculated by the following:

  $$PSR=\frac{TP}{p}.$$
• $Accuracy$: the accuracy of the prediction for the test data, which is calculated by the following formulation:

  $$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}.$$
• $AUC$: area under the curve.

A P value close to $TP$ shows that the model is good. The greater the $Accuracy$ and $AUC$, the better the model. $PSR$ being close to 1 shows the goodness of the model.

Moreover, for the penalized parameter $\lambda$, we used the test set verification method to choose it. Firstly, we selected a value of $\lambda$ that made all coefficients equal to 0, and we set it as ${\lambda }_{max}$. Secondly, we chose a number that was very close to 0, such as 0.0001, and set it as ${\lambda }_{min}$. We chose $\lambda \in [{\lambda }_{min},{\lambda }_{max}]$ to do the numerical experiments. Finally, we gave $\lambda \in [{\lambda }_{min},{\lambda }_{max}]$, and it produced a maximum value for the AUC. The augmented Lagrange multiplier $\rho$ did not influence the convergence of the algorithm, but the proper one would give a fast convergence rate. When we performed the numerical experiments, we just chose a better one, which meant it made the algorithm converge quickly.

4.1. Simulation Experiment with Non-Sparse Variables in the Group

Firstly, we constructed the group structure features with similar features in the group and different features between different groups, and we obtained the data as it is given in [1]. The data was generated according to the following model:

$$y_i=\frac{1}{1+exp(-X_i\beta +\epsilon )}.$$

Here, the explanatory variables for the groups followed the multivariate normal distributions, which meant $X_i\sim N(0,{\Sigma }_{p_s})$, and the error followed the standard normal distribution, which meant $\epsilon \sim N(0,1).$ The correlation coefficient of the variables ${\left(X_{{\mathcal{G}}_i}\right)}_i$ and ${\left(X_{{\mathcal{G}}_i}\right)}_j$ was ${\rho }_s^{|i-j|}$. Generally speaking, we employed ${\rho }_s=0.2$ or ${\rho }_s=0.7$ to denote the weak correlation or strong correlation for the variables in the group.

For this simulation experiment, we generated data using 10 groups independently, and each group contained five variables. Hence, the total number of variables was 50. There were three groups that were significant, and the other seven groups were not significant. The correlation coefficients within the groups were 0.2 and 0.7, respectively. The sample size was 500. We selected $80\%$ of the data for the training set and the others were the test set. The experimental simulation was repeated 30 times. For this example, the penalized parameter was $\lambda =0.001$ for the LASSO logistic regression model. For model (2) the penalized parameter was $\lambda =0.02$ and the augmented Lagrange multiplier was chosen as $\rho =0.1$. For the logistic regression model with the elastic net penalty, the parameters were set as ${\lambda }_1=0.0009$ and ${\lambda }_2=0.0001$.

The following table gives the numerical results for this example with different models and different correlation coefficients.

According to

Table 1. The results for the simulations in experiment one.

The Correlation Coefficient	Models	P	$\mathit{TP}$	$\mathit{PSR}$	$\mathit{Accuracy}$	$\mathit{AUC}$
${\rho }_s=0.2$	Elastic Net	33.07	15.00	1.0000	0.9520	0.9503
	LASSO	32.17	15.00	1.0000	0.9503	0.9497
	$p=2,q=\frac{1}{2}$	15.00	15.00	1.0000	0.9657	0.9658
	$p=1,q=\frac{1}{2}$	15.20	14.93	0.9956	0.9580	0.9582
${\rho }_s=0.7$	Elastic Net	31.33	15.00	1.0000	0.9577	0.9571
	LASSO	29.47	14.83	0.9889	0.9577	0.9571
	$p=2,q=\frac{1}{2}$	15.00	15.00	1.0000	0.9710	0.9716
	$p=1,q=\frac{1}{2}$	16.03	14.90	0.9933	0.9637	0.9634

, the logistic regression model with the elastic net penalty, the LASSO logistic regression model and the logistic regression model with $l_{p,q}$ regularization could perform variable selection. When the correlation of variables was different, the criteria of the logistic regression model with $l_{p,q}$ regularization were better than the other two models. According to the indexes P and $TP$, it could be seen that all variables with non-zero coefficients in the logistic regression model with $l_{2,1/2}$ regularization were screened out when the correlation was different. The value of P was closer to that of $TP$, and all the selected variables were variables with non-zero coefficients. When choosing different parameters, the logistic regression model with $l_{p,q}$ regularization selected more variables with true non-zero coefficients than that of the other two models, and the logistic regression model with $l_{p,q}$ regularization gave a solution that was closer to the number of variables with true non-zero coefficients. The $PSR$ was closer to 1 and could select significant variables. From the prediction effect of the model, the AUC and accuracy of the logistic regression model with $l_{p,q}$ regularization showed better prediction effects with different correlation coefficients.

According to the variable-selection effect and the prediction effect of the models, the logistic regression model with $L_{2,1/2}$ regularization was better than those of the logistic regression model with the elastic net penalty and the LASSO logistic regression model. Since the data for this simulation experiment was designed to be zero or non-zero in one group, the logistic regression model with $L_{2,1/2}$ regularization performed better than the logistic regression model with $L_{1,1/2}$ regularization. The logistic regression model with $L_{2,1/2}$ regularization could select a set of variables or not, and it had an ideal group variable-selection effect. The logistic regression model with the elastic net penalty and the LASSO logistic regression model compressed the variables to achieve a sparse effect, but because they did not make full use of the group structure information of the variables, they selected too many variables with zero coefficients during variable selection, so the performance of the two models was not good.

4.2. Simulation Experiment with the Sparse Variables in the Group

The data give were similar to those in the above experiment. The difference between these two simulation experiments are given as follows. A total of six groups of variables with intragroup correlation were simulated. Each group contained 10 variables. A total of two groups of 12 variables were significant. One group of variables was completely significant, and the other group contained two significant variables. The sample size was 500. The ratio of the training set to the test set was $8:2$. The correlation coefficients within the group were 0.2 and 0.7, respectively. For this example, the penalized parameter was $\lambda =0.001$ for the LASSO logistic regression model, and for model (2) the penalized parameter was $\lambda =0.032$ and the augmented Lagrange multiplier was $0.1$. When ${\rho }_s=0.2$, the parameters for the logistic regression model with the elastic net penalty were ${\lambda }_1=0.0018$ and ${\lambda }_2=0.0002$. Moreover, when ${\rho }_s=0.7$, the parameters for the logistic regression model with the elastic net penalty were ${\lambda }_1=0.0021$ and ${\lambda }_2=0.0009$.

The following table gives us the numerical results for this example with different models and different correlation coefficients.

According to

Table 2. The results for the simulations in experiment two.

The Correlation Coefficient	Models	P	$\mathit{TP}$	$\mathit{PSR}$	$\mathit{Accuracy}$	$\mathit{AUC}$
${\rho }_s=0.2$	Elastic Net	29.83	12.00	1.0000	0.9467	0.9467
	LASSO	36.00	12.00	1.0000	0.9480	0.9478
	$p=2,q=\frac{1}{2}$	14.67	12.00	1.0000	0.9580	0.9650
	$p=1,q=\frac{1}{2}$	12.37	12.00	1.0000	0.9651	0.9673
${\rho }_s=0.7$	Elastic Net	24.36	12.00	1.0000	0.9570	0.9561
	LASSO	29.40	12.00	1.0000	0.9590	0.9589
	$p=2,q=\frac{1}{2}$	17.33	12.00	1.0000	0.9640	0.9635
	$p=1,q=\frac{1}{2}$	12.90	12.00	1.0000	0.9700	0.9704

, the logistic regression model with $L_{1,1/2}$ regularization performed very well. From the perspective of variable selection, the logistic regression model with the elastic net penalty, the LASSO logistic regression model and the logistic regression model with $l_{p,q}$ regularization could screen out all the variables with non-zero coefficients. The logistic regression model with the elastic net penalty and the LASSO logistic regression model screened out too many variables with non-zero coefficients. However, the logistic regression model with $l_{p,q}$ regularization not only gave all the variables with non-zero coefficients, but also the values of P and $TP$ were very close, which meant the variables given by the logistic regression model with $l_{p,q}$ regularization were close to the predicted ones. For this kind of data, we found that the logistic regression model with $l_{2,1/2}$ regularization tended to compress a group of variables to be zero or non-zero at the same time. Therefore, the logistic regression model with $l_{2,1/2}$ regularization selected a significant difference, but it did not filter out the important variables in the group. In terms of the prediction effect of the model, the logistic regression model with $l_{1,1/2}$ regularization performed well for different correlation coefficients, and its prediction ability also improved compared with the univariate selection model.

Combining the effects of variable selection and model prediction, the logistic regression model with $l_{p,q}$ regularization performed well when the variables were sparse in the group. Due to the “all in all out” mechanism, the logistic regression model with $l_{2,1/2}$ regularization could only screen out important variable groups. The important variables in the group could not be screened out, and the variable-selection ability was not good. But the logistic regression model with $l_{1,1/2}$ regularization could overcome this disadvantage.

From the above two simulation experiments, we found the goodness of the logistic regression model with $l_{p,q}$ regularization for variable selection and prediction for the data with a group structure. Moreover, for different data, we adjusted the value of p to adapt to the problems.

5. Real-Data Experiment

Data description and preprocessing are described as follows. In this section, the real data are considered. The data came from the excellent mining quantitative platform, and the website is https://uqer.datayes.com/ (acdcessed on 31 October 2021) The data were the factor data and the yield data of constituent stocks for the Shanghai and Shenzhen 300 index in China’s stock market from 1 January 2010 to 31 December 2020. The advantages for using these data were good performance, large scale, high liquidity and active trading in the market. In order to ensure the accuracy and rationality of the analysis results, we needed to select and correct the range of samples. According to the development experience of China’s stock market and previous research experience, the empirical part located the sample starting point in 2010. Secondly, because the capital of companies in the financial industry has the characteristics of high leverage and high debt, the construction of some financial indicators is quite different from the other listed companies. Therefore, we excluded some financial companies based on our experience. In addition, ST and PT stocks in the market have abnormal financial conditions and performance losses. We also excluded such kinds of stocks with weak comparability. Among the 243 stock factors visible on the excellent mining quantitative platform, 34 factors that belong to nine groups were selected to evaluate model (2). The data were daily data, but in practice, if daily transaction data were used for investment, the frequent transactions would lead to a significant increase in transaction costs. The rise of transaction costs would affect the annualized rate of return. In order to reduce the impact of transaction costs, we used monthly transaction data for modeling. Hence, it was necessary to conduct a monthly average processing for each factor’s data, and we used monthly data for stock selection. In the division of the data set, the data from 1 January 2010 to 30 April 2018 were selected as the training set, and the data from 1 May 2018 to 31 December 2020 were used for back testing.

In order to ensure the quality of subsequent factor screening and the effect of stock rise and fall prediction, we needed to preprocess the data before the analysis. The methods for data preprocessing included noise cleaning, missing value processing, data standardization, lag processing and so on.

Using the above factors and data processing methods to generate the stock factor matrix, we employed the logistic regression model with $l_{p,q}$ regularization and the ADMM algorithm proposed in Section 3 to estimate the coefficients of the factors. We also calculated the posterior probability of the stock, sorted the probability from large to small, and bought the top ten stocks with equal weight.

In the following, we introduce the historical back test evaluation index. When we performed the back test analysis for the solutions, in order to perform the objective reflective and comprehensive evaluation of the solutions, it was necessary to give the evaluation indicators. We selected nine indicators to do this, and they were the return rate of the year, the return rate of the benchmark year, the sharp ratio, the volatility, the return unrelated to the market fluctuations ($\alpha$), the sensitivity to market changes ($\beta$), the information ratio, the maximum pullback, the turnover rate of the year, etc.

Based on the above evaluation criteria, we used the selected data to back test and verify the effectiveness of model (2), and we employed the above model to predict the stock trend. Moreover, we sorted the predicted yield data, and selected the top 10 stocks with the highest rise in probability as the stock portfolio for the month, according to the equal weight reconstruction portfolio. We held it until the end of the month to calculate the month’s income. The initial capital was set at 10 million yuan, the tax for buying was 0.003, and the tax for selling was 0.0013. The sliding point was 0. Moreover, the parameters for model (2) were given as $\lambda =0.004$. $\lambda =0.003$ was chosen for the LASSO logistic regression model. We adopted ${\lambda }_1={\lambda }_2=0.02$ for the logistic regression model with the elastic net penalty. We employed the ADMM algorithm that we gave in Section 3 to solve this example, and the Lagrange multipliers for the ADMM algorithm to solve these models were chosen as $\rho =0.1$. After the calculation, we listed the following transaction back test results and the cumulative yield figure, which are listed in

Table 3. The back test results.

The Logistic Regression with Different Penalties	The ${\mathit{l}}_{1,1/2}$ Norm	LASSO	Elastic Net
The return rate of the year	$22.7\%$	$20.5\%$	$18.8\%$
The return rate of the benchmark year	$13.3\%$	$13.3\%$	$13.3\%$
$\alpha$	$10.5\%$	$6.2\%$	$8.0\%$
$\beta$	0.89	0.92	0.64
The sharp ratio	0.84	0.72	0.41
The volatility	$22.8\%$	$23.5\%$	$23.7\%$
The information ratio	0.68	0.54	0.41
The maximum pullback	$18.8\%$	$27.7\%$	$25.1\%$
The turnover rate of the year	9.14	7.67	9.33

and Figure 1, respectively.

From Table 3, we found that the return rates for the year for these three models were all higher than the return rate of the benchmark year, $13.3\%$. For the return rate of the benchmark year, the logistic regression model with $l_{1,1/2}$ regularization was the highest. Model (2) gave a good strategy from the perspective of excess return $\alpha$ and the sharp ratio. Under the same risk coefficient, the investment strategy based on the logistic regression model with $l_{1,1/2}$ regularization could help investors make effective investment decisions and obtain higher yields. Moreover, the model could give a strategy with an acceptable range in the maximum pullback, and the strategy could also more effectively prevent the pullback risk.

From the graph of the cumulative rate of return, the line given by the logistic regression model with $l_{1,1/2}$ regularization was basically always above the benchmark annualized rate of the return curve, which indicated that the return of the portfolio constructed with the group information was always stable. The portfolio constructed based on the logistic regression model with $l_{1,1/2}$ regularization could not only screen out the important factor types affecting stock returns, but could also screen out the important factor indicators in the group, so as to more accurately predict the probability of stock returns rising. When constructing the portfolio based on this model, it had certain advantages over the other two regression models.

6. Conclusions

Combined with the data requirements, this paper proposed a logistic regression model with $l_{p,q}$ regularization. We showed the properties of the $l_{p,q}$ norm and the loss function of the logistic regression problem. Moreover, the oracle inequality for the $l_{p,q}$ norm and the global recovery bound for the penalized regression model were established with the help of the $(p,q)$-group restricted eigenvalue condition. These properties were important for variable selection. In Section 3, we showed the framework for the ADMM algorithm for solving the penalized logistic regression model. For the algorithm, we gave a method for solving the subproblems, so as to reduce the difficulty and complexity of solving the model (2).

By the numerical simulation results, since the logistic regression with $l_{p,q}$ regularization comprehensively considered the group structure information of variables, compared with the univariate selection method considering only a single variable, it could eliminate more redundant variables, so as to screen out the more important characteristics of dependent variables. It also had higher accuracy when we performed the test on the test set. Moreover, for the real-data experiment, it was found that the logistic regression model with $l_{p,q}$ regularization could show a more stable effect in variable selection and prediction.

In future work, we could extend the group logistic regression model with $L_{p,q}$ regularization. In the theoretical part, we could do more analysis, such as local recovery bounds and design a more appropriate algorithm, which could give a convergence solution. Moreover, in the numerical parts, we could choose more p and q to illustrate the goodness of the model (2).