A Linearly Involved Generalized Moreau Enhancement of ℓ_2,1-Norm with Application to Weighted Group Sparse Classification

Abstract

This paper proposes a new group-sparsity-inducing regularizer to approximate ${\ell }_{2,0}$ pseudo-norm. The regularizer is nonconvex, which can be seen as a linearly involved generalized Moreau enhancement of ${\ell }_{2,1}$-norm. Moreover, the overall convexity of the corresponding group-sparsity-regularized least squares problem can be achieved. The model can handle general group configurations such as weighted group sparse problems, and can be solved through a proximal splitting algorithm. Among the applications, considering that the bias of convex regularizer may lead to incorrect classification results especially for unbalanced training sets, we apply the proposed model to the (weighted) group sparse classification problem. The proposed classifier can use the label, similarity and locality information of samples. It also suppresses the bias of convex regularizer-based classifiers. Experimental results demonstrate that the proposed classifier improves the performance of convex ${\ell }_{2,1}$ regularizer-based methods, especially when the training data set is unbalanced. This paper enhances the potential applicability and effectiveness of using nonconvex regularizers in the frame of convex optimization.

1. Introduction

In recent decades, sparse reconstruction has become an active topic in many areas, such as in fields of signal processing, statistics, and machine learning [1]. By reconstructing a sparse solution from a linear measurement, we can obtain a certain expression of high-dimensional data as a vector with only a small number of nonzero entries. In practical applications, the data of interest can often be assumed to have a certain special structure. For example, in microarray analysis of gene expression [2,3], hyperspectral image unmixing [4,5,6], force identification in industrial applications [7], classification problems [8,9,10,11,12,13], etc., the solution of interest often possesses group-sparsity structure, namely the solution has a natural grouping of its coefficients and nonzero entries only occur in few groups.

This paper focuses on the estimation of group sparse solution, which is related to the Group LASSO (least absolute shrinkage and selection operator) [14]. Suppose $\mathit{x}={[{\mathit{x}}_1^{\top },{\mathit{x}}_2^{\top },\cdots ,{\mathit{x}}_g^{\top }]}^{\top }\in {\mathbb{R}}^n$ is a group sparse signal, where ${\mathit{x}}_i\in {\mathbb{R}}^{n_i}$, ${\sum }_{i=1}^g{n}_i=n$ and g is the number of groups. Just as with the use of ${\ell }_0$ pseudo-norm for evaluation of the sparsity, the group sparsity of $\mathit{x}$ can be evaluated with the ${\ell }_{2,0}$ pseudo-norm, i.e., ${\parallel \mathit{x}\parallel }_{2,0}={∥\left(\parallel {\mathit{x}}_1{\parallel }_2,\parallel {\mathit{x}}_2{\parallel }_2,\cdots ,{\parallel {\mathit{x}}_g\parallel }_2\right)∥}_0$, where ${\parallel ·\parallel }_2$ is the Euclidean norm, and ${\parallel ·\parallel }_0$ is the ${\ell }_0$ pseudo-norm which counts the number of nonzero entries in the vector in ${\mathbb{R}}^g$.

The group sparse regularized least squares problem can be modeled as

$${\displaystyle \underset{\mathit{x}\in {\mathbb{R}}^n}{minimize}\frac{1}{2}{\parallel \mathit{y}-\mathit{A}\mathit{x}\parallel }_2^2+\lambda {\parallel \mathit{x}\parallel }_{2,0},}$$

(1)

where $\mathit{y}\in {\mathbb{R}}^m$ and $\mathit{A}\in {\mathbb{R}}^{m\times n}$ are known, and $\lambda >0$ is the regularization parameter. However, the employment of the pseudo-norm ${\ell }_{2,0}$ makes (1) NP-hard [15]. Most studies in the application replace the nonconvex regularizer ${\ell }_{2,0}$ with its tightest convex envelope ${\ell }_{2,1}$ [16] (or its weighted variants), and the following regularized least squares problem has been proposed known as the Group LASSO [14],

$${\displaystyle \underset{\mathit{x}\in {\mathbb{R}}^n}{minimize}\frac{1}{2}{\parallel \mathit{y}-\mathit{A}\mathit{x}\parallel }_2^2+\lambda \sum _{i=1}^g{w}_i{\parallel {\mathit{x}}_i\parallel }_2,}$$

(2)

where $w_i>0$ ($i=1,\cdots ,g$) in the regularization term

$${\parallel \mathit{x}\parallel }_{w,2,1}:=\sum _{i=1}^g{w}_i{\parallel {\mathit{x}}_i\parallel }_2$$

(3)

(this is ${\ell }_{w,2,1}$-norm of $\mathit{x}$, i.e., a separable weighted version [17] of ${\ell }_{2,1}$-norm ${\parallel \mathit{x}\parallel }_{2,1}={\sum }_{i=1}^g{\parallel {\mathit{x}}_i\parallel }_2$) are used to adjust for group sizes with $w_i=\sqrt{n_i}$ in [14,18]. We give a simple but clear explanation in Appendix A, to show the bias of ${\ell }_{2,1}$-norm caused by group size in the application of group sparse classification (GSC).

Although the convex optimization problem (2) has been used as a standard model for group sparse estimation applications, the convex regularizer ${\ell }_{w,2,1}$ does not necessarily promote group sparsity sufficiently, mainly due to the fact that ${\ell }_{w,2,1}$-norm is just an approximation of ${\ell }_{2,0}$ pseudo-norm within the severe restriction of the convexity. To promote the group sparsity more effectively than convex regularizers, nonconvex regularizers such as group SCAD (smoothly clipped absolute deviation) [3], group MCP (minimax concave penalty) [18,19], ${\ell }_{p,q}$ regularization (${\parallel \mathit{x}\parallel }_{p,q}:={\left({\sum }_{i=1}^g{\parallel {\mathit{x}}_i\parallel }_p^q\right)}^{1/q}$, $0<q<1\le p$) [20], iterative weighted group minimization [21], and ${\ell }_{2,0}$[22] have been used for group sparse estimation problems. However, they lose the overall convexity (In [23], a nonconvex regularizer which can preserve the overall convexity was proposed, but the fidelity term of the optimization model is $\frac{1}{2}{\parallel \mathit{y}-\mathit{x}\parallel }_2^2$ (limited to the case of $\mathit{A}={\mathit{I}}_n$, where ${\mathit{I}}_n\in {\mathbb{R}}^{n\times n}$ is the identity matrix), which cannot be applied to (1) for general $\mathit{A}\in {\mathbb{R}}^{m\times n}$.) of the optimization problems, which results in their algorithms of no guarantee of convergence to global minimizers of the overall cost functions.

In this paper, we propose a generalized weighted group sparse estimation model based on the linearly involved generalized-Moreau-enhanced (LiGME) approach [24] that uses nonconvex regularizer while maintaining the overall convexity of the optimization problem. Our contributions can be summarized as follows:

• We show in Proposition 2 that the generalized Moreau enhancement (GME) of ${\ell }_{w,2,1}$, i.e., ${(\parallel ·\parallel }_{w,2,1}{)}_{\mathit{B}}$ (see (11)), can bridge the gap between ${\ell }_{w,2,1}$ and ${\ell }_{2,0}$. For the non-separable weighted ${\ell }_{2,1}$, i.e., ${\parallel \mathit{W}·\parallel }_{2,1}$, its GME can be expressed as LiGME of ${\ell }_{2,1}$ in the case of weight matrix $\mathit{W}$ has full row-rank.
• We present a convex regularized least squares model with a nonconvex group sparsity promoting regularizer based on LiGME. It can be served as a unified model of many types of group sparsity related applications.
• We illustrate the unfairness of ${\ell }_{2,1}$ regularizer in unbalanced classification and then apply the proposed model to reduce the unfairness of it in GSC and weighted GSC (WGSC) [11].

The remainder of this paper is organized as follows. In Section 2, we give a brief review of LiGME model and WGSC method. In Section 3, we present our group sparse enhanced representation model and its mathematical properties. In Section 4, we apply the proposed model to group-sparsity-based classification problems. The conclusion is given in Section 5.

A preliminary short version of this paper was presented at a conference [25].

2. Preliminaries

2.1. Review of Linearly Involved Generalized-Moreau-Enhanced (LiGME) Model

We first give a brief review of linearly involved generalized-Moreau-enhanced (LiGME) models, which is closely related to our method. Although the convex function ${\ell }_1$-norm (or nuclear norm) is the most frequently adopted regularizer for sparsity (or low-rank) pursuing problems, it tends to yield underestimation for high-amplitude value (or large singular value) [26,27]. The convexity-preserving nonconvex regularizers have been widely explored in [24,28,29,30,31,32,33], which promote sparsity (or low-rank) more effectively than convex regularizers without losing the overall convexity. Among them, the generalized minimax concave (GMC) function in [31] does not rely on certain strong assumptions in the least squares term and has great potential for dealing with nonconvex variations of ${\parallel ·\parallel }_1$. Motivated by GMC function, the LiGME model [24] provides a general framework for constructing linearly involved nonconvex regularizers for sparsity (or low-rank) regularized linear least squares while maintaining the overall convexity of the cost function.

Let $(\mathcal{X},{\langle ·,·\rangle }_{\mathcal{X}},\parallel ·{\parallel }_{\mathcal{X}})$, $(\mathcal{Y},{\langle ·,·\rangle }_{\mathcal{Y}},\parallel ·{\parallel }_{\mathcal{Y}})$, $(\mathcal{Z},{\langle ·,·\rangle }_{\mathcal{Z}},\parallel ·{\parallel }_{\mathcal{Z}})$, and $(\tilde{\mathcal{Z}},{\langle ·,·\rangle }_{\tilde{\mathcal{Z}}},\parallel ·{\parallel }_{\tilde{\mathcal{Z}}})$ be finite-dimensional real Hilbert spaces. Let a function $\Psi \in {\Gamma }_0\left(\mathcal{Z}\right)$ be coercive with $\mathrm{dom}\Psi =\mathcal{Z}$. Here ${\Gamma }_0\left(\mathcal{Z}\right)$ is the set of proper (i.e., $\mathrm{dom}\Psi :=\{\mathit{z}\in \mathcal{Z}|\Psi \left(\mathit{z}\right)<\infty \}\ne ⌀$) lower semicontinuous (i.e., ${\mathrm{lev}}_{\le a}\Psi :=\{z\in \mathcal{Z}|\Psi \left(\mathit{z}\right)\le a\}$ is closed for $\forall a\in \mathbb{R}$) convex function (i.e., $\Psi (\theta {\mathit{z}}_1+(1-\theta ){\mathit{z}}_2)\le \theta \Psi \left({\mathit{z}}_1\right)+(1-\theta )\Psi \left({\mathit{z}}_2\right))$ for $\forall {\mathit{z}}_1,{\mathit{z}}_2\in \mathrm{dom}\Psi ,0\le \theta \le 1$) from $\mathcal{Z}$ to $(-\infty ,\infty ]$; a function $\Psi \in {\Gamma }_0\left(\mathcal{Z}\right)$ is called coercive if ${\parallel \mathit{z}\parallel }_2\to \infty \Rightarrow \Psi \left(\mathit{z}\right)\to \infty$. For $\Psi \in {\Gamma }_0\left(\mathcal{Z}\right)$, the proximity operator of $\Psi$ is defined by ${\mathrm{Prox}}_{\Psi }:\mathcal{Z}\to \mathcal{Z}:\mathit{z}\mapsto arg{min}_{\mathit{v}\in \mathcal{Z}}\Psi \left(\mathit{v}\right)+\frac{1}{2}{\parallel \mathit{v}-\mathit{z}\parallel }_{\mathcal{Z}}^2.$

The generalized Moreau enhancement (GME) of $\Psi$ with $\mathit{B}\in \mathcal{B}(\mathcal{Z},\tilde{\mathcal{Z}})$ is defined as

$${\Psi }_{\mathit{B}}(·):=\Psi (·)-\underset{v\in \mathcal{Z}}{min}\left[\Psi \left(v\right)+\frac{1}{2}{\parallel \mathit{B}(·-v)\parallel }_{\tilde{\mathcal{Z}}}^2\right],$$

(4)

where $\mathit{B}$ is a tuning matrix for the enhancement. Then the LiGME model defined as the minimization of

$$J_{{\Psi }_{\mathit{B}}\circ \mathfrak{L}}:\mathcal{X}\to \mathbb{R}:\mathit{x}\mapsto \frac{1}{2}{\parallel \mathit{y}-\mathit{A}\mathit{x}\parallel }_{\mathcal{Y}}^2+\lambda {\Psi }_{\mathit{B}}\circ \mathfrak{L}\left(\mathit{x}\right),$$

(5)

where $(\mathit{A},\mathfrak{L},\lambda )\in \mathcal{B}(\mathcal{X},\mathcal{Y})\times \mathcal{B}(\mathcal{X},\mathcal{Z})\times {\mathbb{R}}_{+}$.

Please note that GMC [31] can be seen as a special case of (5) with $\Psi ={\parallel ·\parallel }_1$ and $\mathfrak{L}=\mathrm{Id}$, where Id is the identity operator. Model (5) can also be seen as an extension of [32,33].

Although the GME function ${\Psi }_B$ in (4) is not convex in general for $\mathit{B}\ne {\mathit{O}}_{\mathcal{B}(\mathcal{Z},\tilde{\mathcal{Z}})}$, where ${\mathit{O}}_{\mathcal{B}(\mathcal{Z},\tilde{\mathcal{Z}})}\in \mathcal{B}(\mathcal{Z},\tilde{\mathcal{Z}})$ is the zero operator, the overall convexity of the cost function (5) can be achieved with $\mathit{B}$ designed to satisfy the following convexity condition.

Proposition 1

([24], Proposition 1). The cost function $J_{{\Psi }_{\mathit{B}}\circ \mathfrak{L}}$ in (5) belongs to ${\Gamma }_0\left(\mathcal{X}\right)$ for any $\mathit{y}\in \mathcal{Y}$, if the GME regularizer ${\Psi }_{\mathit{B}}$ in (4) satisfies that

$${\mathit{A}}^{*}\mathit{A}-\lambda {\mathfrak{L}}^{*}{\mathit{B}}^{*}\mathit{B}\mathfrak{L}⪰O_{\mathcal{X}},$$

(6)

where ${\mathit{A}}^{*}$ denotes the adjoint of $\mathit{A}$ and ${\mathit{O}}_{\mathcal{X}}\in \mathcal{B}(\mathcal{X},\mathcal{X})$ is the zero operator. In particular, when Ψ is a certain norm over the vector space $\mathcal{Z}$, $J_{{\Psi }_{\mathit{B}}\circ \mathfrak{L}}\in {\Gamma }_0\left(\mathcal{X}\right)$ if and only if (6) is satisfied.

A method of designing $\mathit{B}$ satisfying (6) for $\mathcal{X}={\mathbb{R}}^n$ is provided in [24]; see Proposition A1 in Appendix B. For any $\Psi \in {\Gamma }_0\left(\mathcal{Z}\right)$ that is coercive, even symmetry and prox-friendly (Even symmetry means $\Psi \circ (-\mathrm{Id})=\Psi$; prox-friendly means ${\mathrm{Prox}}_{\gamma \Psi }$ is computable ($\forall \gamma \in {\mathbb{R}}_{++}$).) with $\mathrm{dom}\Psi =\mathcal{Z}$, [24] provides a proximal splitting algorithm (see Proposition A2 in Appendix B) of guaranteed convergence to a globally optimal solution of model (5) under the overall-convexity condition (6).

2.2. Basic Idea of Weighted Group Sparse Classification (WGSC)

As a relatively simple but typical scenario for the application of the proposed idea in this paper, we introduce the main idea of weighted group sparse classification (WGSC). Classification is one of fundamental tasks in the field of the signal and image processing and pattern recognition. For a classification problem with g classes of subjects, the training samples can formulate a dictionary matrix $\mathit{A}=[{\mathit{A}}_1,{\mathit{A}}_2,\cdots ,{\mathit{A}}_g]\in {\mathbb{R}}^{m\times n}$, where ${\mathit{A}}_i=[{\mathit{a}}_{i1},{\mathit{a}}_{i2},\cdots ,{\mathit{a}}_{i{n}_i}]\in {\mathbb{R}}^{m\times n_i}$ is the subset of the training samples from subject i, ${\mathit{a}}_{ij}$ is the j-th training sample from the i-th class, $n_i$ is the number of training samples from class i, and $n={\sum }_{i=1}^g{n}_i$ is the number of total training samples. The aim is to correctly determine which class the input test sample $\mathit{y}\in {\mathbb{R}}^m$ belongs to. Although deep learning is very popular and powerful for classification tasks, it requires a very large-scale training set and computation resources for numerous parameters training with complicated back-propagation.

Wright et al. proposed the sparse representation-based classification (SRC) [34] for face recognition. With the assumption that samples of a specific subject lie in a linear subspace, a valid test sample $\mathit{y}$ is expected to be approximated well by a linear combination of the training samples from the same class, which leads to a sparse representation coefficient over all training samples. Specifically, the test sample $\mathit{y}$ is approximated by the linear combination of the dictionary items, i.e., $\mathit{y}\approx \mathit{A}\mathit{x}$, where $\mathit{x}$ is the coefficient vector. A simple minimization model with sparse representation can be ${\mathrm{minimize}}_{\mathit{x}\in {\mathbb{R}}^n}\frac{1}{2}{\parallel \mathit{y}-\mathit{A}\mathit{x}\parallel }_2^2+\lambda {\parallel \mathit{x}\parallel }_0$. In most SRC-based approaches, ${\ell }_0$ regularizer is relaxed to ${\ell }_1$, and the model becomes the well-known LASSO model [35] in statistics.

The label information of the dictionary atoms is not used in the simple model of SRC, hence the regression is based solely on the structure of each sample. When the subspaces spanned by different classes are not independent, SRC may lead the test image to be represented by training samples from multiple different classes. Considering ideal situation where the test image should only be approximated well by the training samples from one class corresponding to the correct one, in [8,9,10], the authors divided training samples into groups by prior label information and used group-sparsity regularizers. Naturally, the coefficient vector $\mathit{x}$ has group structure $\mathit{x}={[{\mathit{x}}_1^{\top },{\mathit{x}}_2^{\top },\cdots ,{\mathit{x}}_g^{\top }]}^{\top }\in {\mathbb{R}}^n$, where ${\mathit{x}}_i={[x_{i1},x_{i2},\cdots ,x_{i{n}_i}]}^{\top }\in {\mathbb{R}}^{n_i}$$(i=1,2,\cdots ,g)$. This kind of group sparse classification (GSC) approach aims to represent the test image using the minimum number of groups, and thus an ideal model is (1) which is NP-hard. As stated in Section 1, a convex approximation of ${\ell }_{2,0}$, i.e., ${\ell }_{2,1}$-norm, has been used widely as a best convex regularizer to incorporate the class labels.

More generally, the non-separable weighted ${\ell }_{2,1}$-norm, i.e., ${\parallel \mathit{W}·\parallel }_{2,1}$ has also been used as the regularizer in GSC [11,36,37]. For example, Tang et al. [11] proposed a weighted GSC (WGSC) model as follows, by involving the information of the similarity between query sample and each class as well as the distance between query sample and each training sample,

$${\displaystyle \underset{\mathit{x}\in {\mathbb{R}}^n}{minimize}\frac{1}{2}{\parallel \mathit{y}-\mathit{A}\mathit{x}\parallel }_2^2+\lambda \sum _{i=1}^g{w}_i{\parallel {\mathit{d}}_i\odot {\mathit{x}}_i\parallel }_2,}$$

(7)

where ${\mathit{d}}_i=[d_{i1},d_{i2},\cdots ,d_{i{n}_i}]\in {\mathbb{R}}^{n_i}$ penalizes the distance between $\mathit{y}$ and each training sample of i-th class, $w_i$ is set to assess the relative importance of training samples from i-th class for representing the test sample, and here ⊙ denotes element-wise multiplication. Specifically, the weights are computed by

$$d_{ij}=\exp \left(\frac{\parallel \mathit{y}-{\mathit{a}}_{ij}{\parallel }_2}{{\sigma }_1}\right)\mathrm{and}{w}_i=\exp \left(\frac{r_i-r_{\min }}{{\sigma }_2}\right),$$

(8)

where ${\sigma }_1$ and ${\sigma }_2$ are bandwidth parameters, ${\mathit{x}}_i^{*}=arg{min}_{{\mathit{x}}_i}{\parallel \mathit{y}-{\mathit{A}}_i{\mathit{x}}_i\parallel }_2^2$, $r_i=\parallel \mathit{y}-{\mathit{A}}_i{\mathit{x}}_i^{*}\parallel$ computes the distance from $\mathit{y}$ to the individual subspace generated by ${\mathit{A}}_i$, and $r_{\min }$ denotes the minimum reconstruction error of ${\left\{r_i\right\}}_{i=1}^g$. The regularizer in (7) can be written as a non-separable weighted ${\ell }_{2,1}$, i.e., ${\parallel \mathit{W}\mathit{x}\parallel }_{2,1}$, where

$$\mathit{W}=\mathrm{BlockDiag}({\mathit{W}}_1,{\mathit{W}}_2,\cdots ,{\mathit{W}}_g)\mathrm{and}{\mathit{W}}_i=\mathrm{Diag}(w_i{d}_{i1},w_i{d}_{i2},\cdots ,w_i{d}_{i{n}_i}).$$

(9)

For the aforementioned methods, after obtaining the optimal solution (denoted by $\widehat{\mathit{x}}={[{\widehat{\mathit{x}}}_1^{\top },{\widehat{\mathit{x}}}_2^{\top },\cdots ,{\widehat{\mathit{x}}}_g^{\top }]}^{\top }$), they assign $\mathit{y}$ to the class that minimizes the class reconstruction residual defined by $\parallel \mathit{y}-{\mathit{A}}_i{\widehat{\mathit{x}}}_i{\parallel }_2$.

Although ${\ell }_{2,1}$ regularizer and its weighted variants are widely used in GSC and WGSC-based methods, they not only suppress the number of selected classes, but also suppress significant nonzero coefficients within classes. The later may lead to underestimation of high-amplitude elements and adversely affect the performance. The nonconvex regularizers such as ${\ell }_{2,p}$ ($0<p<1$) [37] and group MCP [38] make the corresponding optimization problems nonconvex. Therefore, we hope to use a regularizer which can reduce the bias and approximate ${\ell }_{2,0}$ better than ${\ell }_{2,1}$ while ensuring the overall convexity of the problem.

3. LiGME Model for Group Sparse Estimation

3.1. GME of Weighted ${\ell }_{2,1}$-Norm and Its Properties

Although ${\ell }_{2,1}$-norm (or its weighted variants) acts as the favorable approach to approximate ${\ell }_{2,0}$ in the literature of group sparse estimation, it has large bias and does not promote group sparsity as effective as ${\ell }_{2,0}$. Since GME provides an approach to better approximate direct discrete measures (e.g., ${\ell }_0$ for sparsity, matrix rank for low-rankness) than their convex envelopes, we propose to use it for designing group-sparsity pursuing regularizers.

More generally, let us consider the GME of ${\parallel ·\parallel }_{w,2,1}$ in (3). Clearly, ${\parallel ·\parallel }_{w,2,1}\in {\Gamma }_0\left({\mathbb{R}}^n\right)$ is coercive, even symmetry and prox-friendly, whose proximity operator can be computed by

$${\mathrm{Prox}}_{{\gamma \parallel ·\parallel }_{w,2,1}}:{\mathbb{R}}^n\to {\mathbb{R}}^n:\mathit{x}\mapsto {\left\{\left(1-\frac{\gamma w_i}{max\{\parallel {\mathit{x}}_i{\parallel }_2,\gamma w_i\}}\right){\mathit{x}}_i\right\}}_{i=1}^g,$$

(10)

where $\mathit{x}={[{\mathit{x}}_1^{\top },{\mathit{x}}_2^{\top },\cdots ,{\mathit{x}}_g^{\top }]}^{\top }\in {\mathbb{R}}^n$ is a signal with group structure, ${\mathit{x}}_i\in {\mathbb{R}}^{n_i}(i=1,2,\cdots ,g)$ and ${\sum }_{i=1}^g{n}_i=n$.

Actually, the GME of ${\parallel ·\parallel }_{w,2,1}$ with $\mathit{B}\in {\mathbb{R}}^{b\times n}$ (see (4)):

$${\left({\parallel ·\parallel }_{w,2,1}\right)}_{\mathit{B}}\left(\mathit{x}\right)=\sum _{i=1}^g{w}_i{\parallel {\mathit{x}}_i\parallel }_2-\underset{\mathit{v}\in {\mathbb{R}}^n}{min}\left\{\sum _{i=1}^g{w}_i\parallel {\mathit{v}}_i{\parallel }_2+\frac{1}{2}{\parallel \mathit{B}(\mathit{x}-\mathit{v})\parallel }_2^2\right\},$$

(11)

where ${\mathit{v}}_i\in {\mathbb{R}}^{n_i}(i=1,2,\cdots ,g)$ and $\mathit{v}={[{\mathit{v}}_1^{\top },{\mathit{v}}_2^{\top },\cdots ,{\mathit{v}}_g^{\top }]}^{\top }\in {\mathbb{R}}^n$, can serve as a parametric bridge between ${\parallel ·\parallel }_{2,0}$ and ${\parallel ·\parallel }_{w,2,1}$.

Proposition 2.

(GME of ${\parallel ·\parallel }_{w,2,1}$ can bridge the gap between ${\parallel ·\parallel }_{2,0}$ and ${\parallel ·\parallel }_{w,2,1}$.) Let ${\mathit{B}}_{\gamma }:=\mathrm{BlockDiag}(\frac{w_1}{\sqrt{\gamma }}{\mathit{I}}_{n_1},\frac{w_2}{\sqrt{\gamma }}{\mathit{I}}_{n_2},\cdots ,\frac{w_g}{\sqrt{\gamma }}{\mathit{I}}_{n_g})$ for $\gamma >0$, where $w_i>0$ is the weight in (3) for $i=1,\cdots ,g$. Then, for any $\mathit{x}\in {\mathbb{R}}^n$,

$$\underset{\gamma \downarrow 0}{lim}\frac{2}{\gamma }{\left({\parallel ·\parallel }_{w,2,1}\right)}_{{\mathit{B}}_{\gamma }}\left(\mathit{x}\right)={\parallel \mathit{x}\parallel }_{2,0}.$$

(12)

Together with the fact that ${\left({\parallel ·\parallel }_{w,2,1}\right)}_{{\mathit{O}}_{n\times n}}\left(\mathit{x}\right)={\parallel \mathit{x}\parallel }_{w,2,1}$ where ${\mathit{O}}_{n\times n}\in {\mathbb{R}}^{n\times n}$ is the zero matrix, the regularization term $\frac{2}{\gamma }{\left({\parallel ·\parallel }_{w,2,1}\right)}_{{\mathit{B}}_{\gamma }}\left(\mathit{x}\right)$ can serve as a parametric bridge between ${\parallel ·\parallel }_{2,0}$ and ${\parallel ·\parallel }_{w,2,1}$. As a special case, the GME of ${\parallel ·\parallel }_{2,1}$ can serve as a parametric bridge between ${\parallel ·\parallel }_{2,0}$ and ${\parallel ·\parallel }_{2,1}$.

Proof. 

The regularization term $\frac{2}{\gamma }{\left({\parallel ·\parallel }_{w,2,1}\right)}_{{\mathit{B}}_{\gamma }}\left(\mathit{x}\right):{\mathbb{R}}^n\to \mathbb{R}:{[{\mathit{x}}_1^{\top },{\mathit{x}}_2^{\top },\cdots ,{\mathit{x}}_g^{\top }]}^{\top }\mapsto {\sum }_{i=1}^g\frac{2}{\gamma }{\phi }_i\left({\mathit{x}}_i\right)$, where

$${\phi }_i\left({\mathit{x}}_i\right):=w_i{\parallel {\mathit{x}}_i\parallel }_2-\underset{{\mathit{v}}_i\in {\mathbb{R}}^{n_i}}{min}\left\{w_i\parallel {\mathit{v}}_i{\parallel }_2+\frac{{w_i}^2}{2\gamma }{\parallel {\mathit{x}}_i-{\mathit{v}}_i\parallel }_2^2\right\}$$

for $i=1,\cdots ,g$. By ([39], Example 24.20), we obtain

$$\frac{2}{\gamma }{\phi }_i\left({\mathit{x}}_i\right)=\left\{\begin{array}{cc}\frac{2w_i}{\gamma }\parallel {\mathit{x}}_i{\parallel }_2-\frac{{w_i}^2}{{\gamma }^2}{\parallel {\mathit{x}}_i\parallel }_2^2,\hfill & \mathrm{if}\parallel {\mathit{x}}_i{\parallel }_2\le \frac{\gamma }{w_i}\hfill \\ 1,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(13)

Then, we obtain

$$\underset{\gamma \downarrow 0}{lim}\frac{2}{\gamma }{\phi }_i\left({\mathit{x}}_i\right)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}\parallel {\mathit{x}}_i{\parallel }_2=0,\hfill \\ 1,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(14)

and

$$\begin{array}{cc}\hfill \underset{\gamma \downarrow 0}{lim}\frac{2}{\gamma }{\left({\parallel ·\parallel }_{w,2,1}\right)}_{{\mathit{B}}_{\gamma }}& =\underset{\gamma \downarrow 0}{lim}\sum _{i=1}^g\frac{2}{\gamma }{\phi }_i\left({\mathit{x}}_i\right)={\parallel \mathit{x}\parallel }_{2,0}.\hfill \end{array}$$

(15)

□

Figure 1 illustrates simple examples of ${\parallel \mathit{x}\parallel }_{2,1}$ and ${(\parallel ·\parallel }_{2,1}{)}_{\mathit{B}}\left(\mathit{x}\right)$ when $g=1$, $n=2$ and $\mathit{B}={\mathit{I}}_2$. As we can see, ${(\parallel ·\parallel }_{2,1}{)}_{{\mathit{I}}_2}\left(\mathit{x}\right)$ can approximate ${\parallel \mathit{x}\parallel }_{2,0}$ better than ${\parallel \mathit{x}\parallel }_{2,1}$.

Of course, as reviewed in Section 2.1, we can minimize $J_{{\left({\parallel ·\parallel }_{w,2,1}\right)}_{\mathit{B}}\circ \mathrm{Id}}$ (see (5)) with the algorithm in (A3) in Proposition A2, under the overall-convexity condition ${\mathit{A}}^{\top }\mathit{A}-\lambda {\mathit{B}}^{\top }\mathit{B}⪰{\mathit{O}}_{n\times n}$.

In the following, we consider the GME of non-separable weighted ${\ell }_{2,1}$-norm ${\parallel \mathit{W}·\parallel }_{2,1}$, where $\mathit{W}\in {\mathbb{R}}^{l\times n}$ is not necessarily a diagonal matrix. This is because in some applications, such as classification problems [11,36,37] stated in Section 2.2, and also heterogeneous feature selection [40], weights are introduced inside groups as well (i.e., the weight of every entry can be different) to improve the estimation accuracy. The GME of ${\parallel \mathit{W}·\parallel }_{2,1}$ with $\tilde{\mathit{B}}\in {\mathbb{R}}^{b\times n}$ is well-defined (The lack of coercivity requires slight modification from min to inf.) as

$${\left({\parallel \mathit{W}·\parallel }_{2,1}\right)}_{\tilde{\mathit{B}}}\left(\mathit{x}\right)={\parallel \mathit{W}\mathit{x}\parallel }_{2,1}-\underset{\mathit{v}\in {\mathbb{R}}^n}{inf}\left\{{\parallel \mathit{W}\mathit{v}\parallel }_{2,1}+\frac{1}{2}{\parallel \tilde{\mathit{B}}(\mathit{x}-\mathit{v})\parallel }_2^2\right\},$$

(16)

and therefore we can formulate

$${\displaystyle \underset{\mathit{x}\in {\mathbb{R}}^n}{minimize}{J}_{{(\parallel \mathit{W}·\parallel }_{2,1}{)}_{\tilde{\mathit{B}}\circ \mathrm{Id}}}=\frac{1}{2}{\parallel \mathit{y}-\mathit{A}\mathit{x}\parallel }_2^2{+\lambda (\parallel \mathit{W}·\parallel }_{2,1}{)}_{\tilde{\mathit{B}}}\left(\mathit{x}\right).}$$

(17)

However, we should remark that ${\parallel \mathit{W}·\parallel }_{2,1}\in {\Gamma }_0\left({\mathbb{R}}^n\right)$ is even symmetric but not necessarily coercive or prox-friendly (As found in ([39], Proposition 24.14), it is known that for $\Psi \in {\Gamma }_0\left(\mathcal{Z}\right)$ and $\mathfrak{L}\in \mathcal{B}(\mathcal{X},\mathcal{Z})$ satisfying $\mathfrak{L}{\mathfrak{L}}^{*}=\mu \mathrm{Id}$ with some $\mu \in {\mathbb{R}}_{++}$, we have ${\mathrm{Prox}}_{\Psi \circ \mathfrak{L}}\left(\mathit{x}\right)=\mathit{x}+{\mu }^{-1}{\mathfrak{L}}^{*}({\mathrm{Prox}}_{\mu \Psi }\left(\mathfrak{L}\mathit{x}\right)-\mathfrak{L}\mathit{x})$ for $\mathit{x}\in \mathcal{X}$. In such a special case, if $\Psi$ is prox-friendly, $\Psi \circ \mathfrak{L}$ is also prox-friendly. However, for general $\mathfrak{L}\in \mathcal{B}(\mathcal{X},\mathcal{Z})$ not necessarily satisfying such standard conditions, we have to discuss the prox-friendliness of $\Psi \circ \mathfrak{L}$ case by case.). Fortunately, by Proposition 3 below, if $\mathrm{rank}\left(\mathit{W}\right)=l$ and $\tilde{\mathit{B}}$ can be expressed as $\tilde{\mathit{B}}=\mathit{B}\mathit{W}$ for some $\mathit{B}\in {\mathbb{R}}^{b\times l}$, we can show the useful relation

$${\left({\parallel \mathit{W}·\parallel }_{2,1}\right)}_{\tilde{\mathit{B}}}\left(\mathit{x}\right)={\left({\parallel ·\parallel }_{2,1}\right)}_{\mathit{B}}\circ \mathit{W}\left(\mathit{x}\right),$$

(18)

which implies that the GME ${\left({\parallel \mathit{W}·\parallel }_{2,1}\right)}_{\tilde{\mathit{B}}}$ of ${\parallel \mathit{W}·\parallel }_{2,1}$ can be handled as the LiGME ${\left({\parallel ·\parallel }_{2,1}\right)}_{\mathit{B}}\circ \mathit{W}$ of ${\parallel ·\parallel }_{2,1}$.

Proposition 3.

For $\Psi \in {\Gamma }_0\left(\mathcal{Z}\right)$ which is coercive and $\mathit{B}\in \mathcal{B}(\mathcal{Z},\tilde{\mathcal{Z}})$, assume $\mathfrak{L}\in \mathcal{B}(\mathcal{X},\mathcal{Z})$ has full row-rank. Then for any $\mathit{x}\in \mathcal{X}$,

$${\left(\Psi \circ \mathfrak{L}\right)}_{\mathit{B}\circ \mathfrak{L}}\left(\mathit{x}\right)={\Psi }_{\mathit{B}}\circ \mathfrak{L}\left(\mathit{x}\right),$$

(19)

where ${\left(\Psi \circ \mathfrak{L}\right)}_{\mathit{B}\circ \mathfrak{L}}(·):=\Psi (\mathfrak{L}·)-{inf}_{v\in \mathcal{X}}\left\{\Psi \left(\mathfrak{L}\mathit{v}\right)+\frac{1}{2}{\parallel \mathit{B}\mathfrak{L}(·-\mathit{v})\parallel }_2^2\right\}$ and ${\Psi }_{\mathit{B}}(·):=\Psi (·)-{min}_{\mathit{v}\in \mathcal{Z}}\left\{\Psi \left(\mathit{v}\right)+\frac{1}{2}{\parallel \mathit{B}(·-\mathit{v})\parallel }_2^2\right\}$.

Proof. 

On one hand, by the definition of GME, we have

$$\begin{array}{cc}\hfill {\left(\Psi \circ \mathfrak{L}\right)}_{\mathit{B}\circ \mathfrak{L}}\left(\mathit{x}\right)& =\Psi \left(\mathfrak{L}\mathit{x}\right)-\underset{v\in \mathcal{X}}{inf}\left\{\Psi \left(\mathfrak{L}\mathit{v}\right)+\frac{1}{2}{\parallel \mathit{B}\mathfrak{L}(\mathit{x}-\mathit{v})\parallel }_2^2\right\}\hfill \\ & =\Psi \left(\mathfrak{L}\mathit{x}\right)-h\left(\mathit{B}\mathfrak{L}\mathit{x}\right),\hfill \end{array}$$

where $h\left(\mathit{z}\right):\tilde{\mathcal{Z}}\to \mathbb{R}$ is given by

$$\begin{array}{cc}\hfill h\left(\mathit{z}\right)& =\underset{v\in \mathcal{X}}{inf}\left\{\Psi \left(\mathfrak{L}\mathit{v}\right)+\frac{1}{2}{\parallel \mathit{z}-\mathit{B}\mathfrak{L}\mathit{v}\parallel }_2^2\right\}\hfill \\ & =\underset{\mathit{u}\in {\left(\mathrm{null}\mathfrak{L}\right)}^{\perp }}{inf}\underset{\widehat{\mathit{u}}\in \mathrm{null}\mathfrak{L}}{inf}\left\{\Psi \left(\mathfrak{L}(\mathit{u}+\widehat{\mathit{u}})\right)+\frac{1}{2}{\parallel \mathit{z}-\mathit{B}\mathfrak{L}(\mathit{u}+\widehat{\mathit{u}})\parallel }_2^2\right\}\hfill \\ & =\underset{\mathit{u}\in {\left(\mathrm{null}\mathfrak{L}\right)}^{\perp }}{inf}\left\{\Psi \left(\mathfrak{L}\mathit{u}\right)+\frac{1}{2}{\parallel \mathit{z}-\mathit{B}\mathfrak{L}\mathit{u}\parallel }_2^2\right\}\hfill \\ & =\underset{\mathit{u}\in \mathrm{range}{\mathfrak{L}}^{*}}{inf}\left\{\Psi \left(\mathfrak{L}\mathit{u}\right)+\frac{1}{2}{\parallel \mathit{z}-\mathit{B}\mathfrak{L}\mathit{u}\parallel }_2^2\right\}\hfill \\ & =\underset{\mathit{v}\in \mathcal{Z}}{inf}\left\{\Psi \left(\mathfrak{L}{\mathfrak{L}}^{*}\mathit{v}\right)+\frac{1}{2}{\parallel \mathit{z}-\mathit{B}\mathfrak{L}{\mathfrak{L}}^{*}\mathit{v}\parallel }_2^2\right\}\hfill \\ & =\underset{\mathit{v}\in \mathcal{Z}}{inf}\left\{\Psi \left(\mathfrak{L}{\mathfrak{L}}^{*}{\left(\mathfrak{L}{\mathfrak{L}}^{*}\right)}^{-1}\mathit{v}\right)+\frac{1}{2}{\parallel \mathit{z}-\mathit{B}\mathfrak{L}{\mathfrak{L}}^{*}{\left(\mathfrak{L}{\mathfrak{L}}^{*}\right)}^{-1}\mathit{v}\parallel }_2^2\right\}\hfill \\ & =\underset{\mathit{v}\in \mathcal{Z}}{inf}\left\{\Psi \left(\mathit{v}\right)+\frac{1}{2}{\parallel \mathit{z}-\mathit{B}\mathit{v}\parallel }_2^2\right\}.\hfill \end{array}$$

Therefore, ${\left(\Psi \circ \mathfrak{L}\right)}_{\mathit{B}\circ \mathfrak{L}}\left(\mathit{x}\right)=\Psi \left(\mathfrak{L}\mathit{x}\right)-{inf}_{\mathit{v}\in \mathcal{Z}}\left\{\Psi \left(\mathit{v}\right)+\frac{1}{2}{\parallel \mathit{B}\mathfrak{L}\mathit{x}-\mathit{B}\mathit{v}\parallel }_2^2\right\}$.

On the other hand, ${\Psi }_{\mathit{B}}\circ \mathfrak{L}\left(\mathit{x}\right)=\Psi \left(\mathfrak{L}\mathit{x}\right)-{min}_{\mathit{v}\in \mathcal{Z}}\left\{\Psi \left(\mathit{v}\right)+\frac{1}{2}{\parallel \mathit{B}(\mathfrak{L}\mathit{x}-\mathit{v})\parallel }_2^2\right\}$ by definition. Thus, we obtain the conclusion. □

In the rest of the paper, we focus on LiGME model of ${\ell }_{2,1}$-norm.

3.2. LiGME of ${\ell }_{2,1}$-Norm

For simplicity as well as for effectiveness in application to GSC and WGSC, we focus on the LiGME model of ${\parallel ·\parallel }_{2,1}$ with an invertible linear operator $\mathit{W}$,

$$\underset{\mathit{x}\in {\mathbb{R}}^n}{\mathrm{minimize}}{J}_{{(\parallel ·\parallel }_{2,1}{)}_{\mathit{B}}\circ \mathit{W}}=\frac{1}{2}{\parallel \mathit{y}-\mathit{A}\mathit{x}\parallel }_2^2+\lambda {\left({\parallel ·\parallel }_{2,1}\right)}_{\mathit{B}}\circ \mathit{W}\left(\mathit{x}\right).$$

(20)

In this case, for achieving $J_{{(\parallel ·\parallel }_{2,1}{)}_{\mathit{B}}\circ \mathit{W}}\in {\Gamma }_0\left({\mathbb{R}}^n\right)$, we can simply design $\mathit{B}\in {\mathbb{R}}^{m\times n}$, in a way similar to ([31], (48)), as in the next proposition.

Proposition 4.

For an invertible $\mathit{W}\in {\mathbb{R}}^{n\times n}$, let

$$\mathit{B}=\sqrt{\theta /\lambda }\mathit{A}{\mathit{W}}^{-1},0\le \theta \le 1,$$

(21)

then for the LiGME model in (20), $J_{{(\parallel ·\parallel }_{2,1}{)}_{\mathit{B}}\circ \mathit{W}}\in {\Gamma }_0\left({\mathbb{R}}^n\right)$.

Proof. 

By ${\mathit{A}}^{\top }\mathit{A}-\lambda {\mathit{W}}^{\top }{\mathit{B}}^{\top }\mathit{B}\mathit{W}={\mathit{A}}^{\top }\mathit{A}-\lambda {\mathit{W}}^{\top }{\left(\sqrt{\theta /\lambda }\mathit{A}{\mathit{W}}^{-1}\right)}^{\top }\left(\sqrt{\theta /\lambda }\mathit{A}{\mathit{W}}^{-1}\right)\mathit{W}=(1-\theta ){\mathit{A}}^{\top }\mathit{A}⪰{\mathit{O}}_{n\times n}$ and Proposition 1, $J_{{(\parallel ·\parallel }_{2,1}{)}_{\mathit{B}}\circ \mathit{W}}\in {\Gamma }_0\left({\mathbb{R}}^n\right)$ is ensured. □

Model (20) can be applied to many different applications that conform to group-sparsity structure.

4. Application to Classification Problems

4.1. Proposed Algorithm for Group-Sparsity Based Classification

Since ${\ell }_{2,1}$ regularizer in GSC is unfair for classes of different sizes (see Appendix A) while ${\ell }_{2,0}$-regularizer is not, our purpose is to use a better approximation of ${\ell }_{2,0}$ as the regularizer. Therefore, we apply model (20) to group-sparsity-based classification. Following GSC, we can set $\mathit{W}={\mathit{I}}_n$ in (20).

Inspired by WGSC [11] which well designs weights to enforce locality and similarity information of samples, we can also set the weight matrix $\mathit{W}$ according to (9). The classification algorithm is summarized in Algorithm 1.

The ${\ell }_{2,1}$-norm regularized least squares problem in WGSC can be solved by a proximal gradient method [41]. Compared with it, the step 2 in Algorithm 1 for solving (20) requires at each update only one additional computation for ${\mathrm{Prox}}_{{\gamma \parallel ·\parallel }_{2,1}}$ (see (10) with $w_i=1$).

4.2. Experiments

First, by setting $\mathit{W}={\mathit{I}}_n$, we conduct the experiments on a relatively simple dataset to investigate the influence by bias of ${\ell }_{2,1}$ regularizer on the classification problem (especially when training set is unbalanced), and verify the performance improvement using ${(\parallel ·\parallel }_{2,1}{)}_{\mathit{B}}$ as the regularizer by conducting the experiments on a relatively simple dataset. The USPS handwritten digit database [42] has 11,000 samples of digits “0” through “9” (1100 samples per class). The dimension of each sample is $16\times 16$. In our classification experiments, we vectorized them to 256-D vectors. The number of training samples for each class is not necessarily equal, which varies from 5 to 50 (the size of test set is fixed to 50 images per class).

We set $\mathit{W}={\mathit{I}}_n$ (the initialization of $\mathit{W}$ should be modified in Algorithm 1) for the proposed model (20) and compared it with GSC (with ${\ell }_{2,1}$ regularizer) [10]. We set $\mathit{B}=\sqrt{\theta /\lambda }\mathit{A}$ and fix $\theta =0.9$ to achieve the overall convexity of proposed method, and set $\kappa =1.1$, $\iota =\parallel (\kappa /2){\mathit{A}}^{\top }\mathit{A}+\lambda {\mathit{I}}_n{\parallel }_{\mathrm{spec}}+(\kappa -1)$, $\tau ={(\kappa /2+2/\kappa )\lambda \parallel \mathit{B}\parallel }_{\mathrm{spec}}^2+(\kappa -1)$. The initial estimate is set as $({\mathit{x}}^{\left(0\right)},{\mathit{u}}^{\left(0\right)},{\mathit{w}}^{\left(0\right)})=({\mathit{O}}_{n\times 1},{\mathit{O}}_{n\times 1},{\mathit{O}}_{n\times 1})$, and the stopping criterion is set to either $\parallel ({\mathit{x}}^{\left(k\right)},{\mathit{u}}^{\left(k\right)},{\mathit{w}}^{\left(k\right)})-({\mathit{x}}^{(k+1)},{\mathit{u}}^{(k+1)},{\mathit{w}}^{(k+1)}){\parallel }_2<{10}^{-4}$ or steps reaching 10,000.

Figure 2 shows an example of unbalanced training set (digits “0” through “4” have 5 samples per class and “5” through “9” have 25 samples per class). The input (an image of digit “0”) was misclassified (into digital “6”) by GSC while classified correctly by proposed method. The obtained coefficient vectors by GSC and proposed method (both with $\lambda =4$) are illustrated respectively, and some samples corresponding to nonzero coefficients are also displayed in Figure 2. It can be seen that the samples from digit “6” made the greatest contribution to the representation in GSC, and samples from “5” and “0” also made small contribution. In our method, samples from the correct class “0” made the biggest contribution and led to correct result. It is reasonable, because our method did not suppress the high value coefficients too much whereas ${\ell }_{2,1}$ did. The big suppression of ${\ell }_{2,1}$ made the coefficients of the correct class cannot be large enough, and thus easily led to misclassification.

Algorithm 1: The proposed group-sparsity enhanced classification algorithm

Input: A matrix of training samples $\mathit{A}=[{\mathit{A}}_1,{\mathit{A}}_2,\cdots ,{\mathit{A}}_G]\in {\mathbb{R}}^{m\times n}$ grouped by  class information, a test sample vector $\mathit{y}\in {\mathbb{R}}^m$, parameters $\lambda$, ${\sigma }_1$ and ${\sigma }_2$.  1. Initialization: Let $({\mathit{x}}^{\left(0\right)},{\mathit{u}}^{\left(0\right)},{\mathit{v}}^{\left(0\right)})\in {\mathbb{R}}^n\times {\mathbb{R}}^n\times {\mathbb{R}}^n$.   Compute the weight matrix $\mathit{W}$ by (9).   Choose $\mathit{B}$ satisfying ${\mathit{A}}^{\top }\mathit{A}-\lambda {\mathit{W}}^{\top }{\mathit{B}}^{\top }\mathit{B}\mathit{W}⪰{\mathit{O}}_{n\times n}$.   Choose $(\iota ,\tau ,\kappa )\in {\mathbb{R}}_{++}\times {\mathbb{R}}_{++}\times (1,+\infty )$ satisfying $$\iota {\mathit{I}}_n-\frac{\kappa }{2}{\mathit{A}}^{\top }\mathit{A}-\lambda {\mathit{W}}^{\top }\mathit{W}⪰{\mathit{O}}_{n\times n}\mathrm{and}\tau \ge \left(\frac{\kappa }{2}+\frac{2}{\kappa }\right)\lambda {\parallel \mathit{B}\parallel }_{\mathrm{spec}}^2.$$ (22)  2. For $k=0,1,2,\cdots$, compute $$\begin{array}{cc}\hfill {\mathit{x}}^{(k+1)}=& \left[{\mathit{I}}_n-\frac{1}{\iota }({\mathit{A}}^{\top }\mathit{A}-\lambda {\mathit{W}}^{\top }{\mathit{B}}^{\top }\mathit{B}\mathit{W})\right]{\mathit{x}}^{\left(k\right)}-\frac{\lambda }{\iota }{\mathit{W}}^{\top }{\mathit{B}}^{\top }\mathit{B}{\mathit{u}}^{\left(k\right)}-\frac{\lambda }{\iota }{\mathit{W}}^{\top }{\mathit{v}}^{\left(k\right)}+\frac{1}{\iota }{\mathit{A}}^{\top }\mathit{y},\hfill \\ \hfill {\mathit{u}}^{(k+1)}=& {\mathrm{Prox}}_{\frac{\lambda }{\tau }{\parallel ·\parallel }_{2,1}}\left[\frac{2\lambda }{\tau }{\mathit{B}}^{\top }\mathit{B}\mathit{W}{\mathit{x}}^{(k+1)}-\frac{\lambda }{\tau }{\mathit{B}}^{\top }\mathit{B}\mathit{W}{\mathit{x}}^{\left(k\right)}+({\mathit{I}}_n-\frac{\lambda }{\tau }{\mathit{B}}^{\top }\mathit{B}){\mathit{u}}^{\left(k\right)}\right],\hfill \\ \hfill {\mathit{v}}^{(k+1)}=& 2\mathit{W}{\mathit{x}}^{(k+1)}-\mathit{W}{\mathit{x}}^{\left(k\right)}+{\mathit{v}}^{\left(k\right)}-{\mathrm{Prox}}_{{\parallel ·\parallel }_{2,1}}\left(2\mathit{W}{\mathit{x}}^{(k+1)}-\mathit{W}{\mathit{x}}^{\left(k\right)}+{\mathit{v}}^{\left(k\right)}\right)\hfill \end{array}$$   until the stopping criterion is fulfilled.  3. Compute the class label $i^{★}$ of $\mathit{y}$ by $$i^{★}=arg\underset{i}{min}{\parallel \mathit{y}-{\mathit{A}}_i{\mathit{x}}_i^{(k+1)}\parallel }_2.$$ (23) Output: The class label $i^{★}$ corresponding to $\mathit{y}$.

(For example, any $\kappa >1$, $\iota =\parallel (\kappa /2){\mathit{A}}^{\top }\mathit{A}+\lambda {\mathit{W}}^{\top }{\mathit{W}\parallel }_{\mathrm{spec}}+(\kappa -1)$ and $\tau ={(\kappa /2+2/\kappa )\lambda \parallel \mathit{B}\parallel }_{\mathrm{spec}}^2+(\kappa -1)$ can satisfy (22).)

Table 1. Recognition results on the USPS database.

Method	Training Set Size ($\mathit{\alpha }={max}_{\mathit{i}}\left\{{\mathit{n}}_{\mathit{i}}\right\}$, $\mathit{\beta }={min}_{\mathit{i}}\left\{{\mathit{n}}_{\mathit{i}}\right\}$)
	$\mathit{\alpha }$	10		25		50
	$\mathit{\beta }$	5	10	5	25	25	50
GSC(with $\lambda =1.5$)		81.4%	86.6%	73.6%	91.4%	88.4%	93.2%
Proposed ($\lambda =1.5$)		82.0%	87.2%	79.0%	92.2%	89.4%	93.0%
Proposed ($\lambda =3$)		82.6%	87.8%	80.8%	92.2%	90.6%	93.4%

summarizes the recognition accuracy of GSC and the proposed method with $\mathit{W}={\mathit{I}}_n$. The training set includes digits “0” through “4” $\beta$ samples per class and “5” through “9” $\alpha$ samples per class. Through numerical experiments, we found that GSC with $\lambda ={\lambda }_{\mathrm{GSC}}=1.5$ and the proposed method with $\lambda ={\lambda }_{\mathrm{prop}}=3$ perform well on this dataset. We also experimented the proposed method of using ${\lambda }_{\mathrm{GSC}}$, which did not degrade too much compared with using ${\lambda }_{\mathrm{prop}}$. We see that the GSC model degrades when the training set is unbalanced, and the proposed method outperforms GSC especially in such case.

Next, we conduct the experiments on a classic face dataset to verify the validity of the proposed linearly involved model by setting the weight matrix $\mathit{W}$ according to (9). The ORL Database of Faces [43] contains 400 images from 40 distinct subjects (10 images per subject) with variations in lighting, facial expressions (open or closed eyes, smiling or not smiling) and facial details (glasses or no glasses). In our experiments, following [44], all images were downsampled from $112\times 92$ to $16\times 16$ and then formed 256-D vectors. The number of training samples for each class is not necessarily equal, which varies from 4 to 8 (test set is fixed to 2 images per class).

We compared the proposed model (20) ($\mathit{W}={\mathit{I}}_n$ and $\mathit{W}$ by (9) respectively) with GSC [10] and WGSC [11]. In order to achieve the overall convexity, we set $\mathit{B}=\sqrt{\theta /\lambda }\mathit{A}{\mathit{W}}^{-1},0\le \theta \le 1$ and fix $\theta =0.9$ for proposed method. Settings of $(\iota ,\tau ,\kappa )$, initial estimate and stopping criterion are the same as those in the previous experiment. When the parameter $\lambda$ is assigned too small, the obtained coefficient vector is not group sparse; when the parameter ${\sigma }_1$ or ${\sigma }_2$ is assigned too small, the information of locality or similarity plays a decisive role. We found that $\lambda =0.05$ for ${\ell }_{2,1}$ regularizer-based methods (i.e., GSC and WGSC), $\lambda =0.2$ for the proposed method and ${\sigma }_1\in [2,4],{\sigma }_2\in [0.5,2]$ for weights involved methods (i.e., WGSC and proposed method with $\mathit{W}$ by (9)) work well on this dataset.

Figure 3 shows a classification result of WGSC and proposed method ($\mathit{W}$ by (9)) (both with ${\sigma }_1=4$, ${\sigma }_2=2)$) when training set is unbalanced (20 subjects have 8 samples per class and the others have 6 samples per class). The input is an image of subject 10 which was misclassified into subject 8 by WGSC while classified correctly by proposed method.

Table 2. Recognition results on the ORL database.

Method	Training Set Size ($\mathit{\alpha }={max}_{\mathit{i}}\left\{{\mathit{n}}_{\mathit{i}}\right\}$, $\mathit{\beta }={min}_{\mathit{i}}\left\{{\mathit{n}}_{\mathit{i}}\right\}$)
	$\mathit{\alpha }$	4	6		8
	$\mathit{\beta }$	4	4	6	4	6	8
GSC		86.3%	85.0%	91.3%	85.0%	92.5%	93.8%
Proposed ($\mathit{W}={\mathit{I}}_n$)		88.8%	86.3%	93.8%	86.3%	93.8%	95.0%
WGSC		90.6%	87.5%	95.0%	88.8%	93.8%	96.3%
Proposed ($\mathit{W}$ by (9))		91.3%	89.4%	95.6%	91.9%	94.4%	96.3%

summarizes the recognition accuracy of GSC, the proposed method with $\mathit{W}={\mathit{I}}_n$, WGSC and the proposed method with $\mathit{W}$ computed by (9). Training set setting is that 20 subjects have $\beta$ samples per class and the others have $\alpha$ samples per class. With the strategically designed matrix (9), WGSC achieves a significant improvement over GSC. By using the proposed method with $\mathit{W}$ computed by (9), the performance can be further improved, especially when the training set is unbalanced.

5. Conclusions

In this paper, the potential applicability and effectiveness of using nonconvex regularizers in convex optimization framework was explored. We proposed a generalized Moreau enhancement (GME) of weighted ${\ell }_{2,1}$ function and analyzed its relationship with the linearly involved GME of ${\ell }_{2,1}$-norm. The proposed regularizer is nonconvex and promotes group sparsity more effectively than ${\ell }_{2,1}$ while maintaining the overall convexity of the regression model at the same time. The model can be used in many applications and we applied it to classification problems. Our model makes use of the grouping structure by class information and suppresses the tendency of underestimation of high-amplitude coefficients. Experimental results showed that the proposed method is effective for image classification.