Review of Quaternion-Based Color Image Processing Methods

Abstract

Images are a convenient way for humans to obtain information and knowledge, but they are often destroyed throughout the collection or distribution process. Therefore, image processing evolves as the need arises, and color image processing is a broad and active field. A color image includes three distinct but closely related channels (red, green, and blue (RGB)). Compared to directly expressing color images as vectors or matrices, the quaternion representation offers an effective alternative. There are several papers and works on this subject, as well as numerous definitions, hypotheses, and methodologies. Our observations indicate that the quaternion representation method is effective, and models and methods based on it have rapidly developed. Hence, the purpose of this paper is to review and categorize past methods, as well as study their efficacy and computational examples. We hope that this research will be helpful to academics interested in quaternion representation.

1. Introduction

Image processing is a fundamental task in data science, and color image processing is particularly important because it contains more color information. Although many image processing methods have been proposed, mainstream works still represent images in the form of vectors and matrices. Despite these methods being able to achieve competitive results, in recent years, methods based on quaternion representation have been widely used and proven to yield better results [1,2,3,4], such as quaternion-based models for image segmentation [5], restoration [6,7], watermarking [8], face recognition [9,10,11], classification [12,13,14,15], super-resolution [16], etc.

With such a wide range of applications, one wonders what a quaternion is. In fact, the quaternion was invented by Hamilton in 1843 [17]. Similar to a complex number $x=a+bi\in \mathbb{C}$ ($a,b\in \mathbb{R}$, i is the imaginary unit, and $x\in \mathbb{C}$ is a complex number), the quaternion number can be denoted as $\dot{x}=x_0+x_{1}i+x_{2}j+x_{3}k\in \mathbb{H}$ ($x_0,x_1,x_2,x_3\in \mathbb{R}$, $i,j,k$ are imaginary units, and $\dot{x}\in \mathbb{H}$ is a quaternion number). The quaternion number $\dot{x}$ can be understood as an extension of complex numbers, where a and b in x are complex numbers. For a more detailed explanation, please refer to the following reference:

$$\begin{array}{cc}\hfill a+bi& =(x_0+x_{2}j)+(x_1-x_{3}j)i\hfill \\ \hfill & =x_0+x_{2}j+x_{1}i-x_{3}ji\hfill \\ \hfill & =x_0+x_{1}i+x_{2}j+x_{3}k,\hfill \end{array}$$

(1)

where $a=x_0+x_{2}j$, $b=x_1-x_{3}j$ are complex numbers, and $ji=-k$, the detailed rule of the quaternion number, will be introduced in Section 2. In color image processing, we usually represent a color image as a matrix or vector. However, in this case, the color information between the color channels cannot be represented well due to a color image having three channels, i.e., red, green, and blue (RGB). How to denote a color image in a holistic way to avoid errors in handling color image processing is a challenge. Note that, mathematically, a color pixel can be denoted as $(u_r,u_g,u_b)$. Considering the quaternion number with three imaginary parts may be a better way to represent the color image. Then we can represent a color pixel as a quaternion number, i.e., $\dot{u}=u_0+u_{r}i+u_{g}j+u_{b}k$. Please see Figure 1 for a better understanding. As quaternions have a real part u₀ and imaginary parts (u_r, u_g, u_b), the quaternion representation for color pixels should be the pure quaternion u = u_ri + u_gj + u_bk. However, in basic tasks such as image denoising or other tasks with simple methods, e.g., [18], we do not affect the real part of the quaternion. Therefore, for better understanding, we use the quaternion representation u = u₀ + u_ri + u_gj + u_bk. In some works, such as singular value decomposition, the real part of the quaternion is iterated to be nonzero; thus, a zero constraint is needed for the real part.

By representing color images holistically, the relationship between color channels can be preserved, and artifacts in the results can be avoided. Based on this conclusion, many models have been extended to the quaternion domain, including variation models [19,20], sparse representation-based models [21,22,23,24], and low-rank models [25,26,27,28,29]. Moreover, quaternion modules are widely used in deep convolutional neural networks (CNNs). The original convolution kernels merge the color channels by summing up the convolution results and outputting a single channel per kernel. However, with the quaternion representation, the complicated interrelationship between color channels and some important structural information can be well preserved. This can reduce the degrees of freedom to the learning of convolution kernels and the number of neural parameters, thus decreasing the risk of over-fitting. Based on these hypotheses, quaternion-based deep CNN (QCNN) and transformer (QTrans) models have been proposed, e.g., the QCNN [30,31,32,33,34,35,36], QTrans [37,38,39,40], etc.

With the use of quaternion representation, it is possible to better preserve the information and relationships between color channels, leading to more satisfactory results. This paper provides an overview of recent quaternion-based traditional and deep learning models with their applications in image processing, as well as the challenges that still need to be addressed. To provide a comprehensive overview of all related work on quaternion representation in color image processing, we searched literature databases, including journals, conferences, and book chapters in Web of Science. Since research is ongoing, without loss of generality, we set the search time to 1 March 2023. A summary of our findings can be seen in Figure 2, organized by task, method, and year.

There have already been several reviews of quaternion-based color image processing methods. Barthelemy et al. [41] provided an overview of traditional models and algorithms using quaternion sparse representation. In 2020, Garc’ıa-Retuerta et al. [1] discussed the challenges in QCNN models and quaternion applications in neural networks. Similarly, Parcollet et al. [2] provided a review of QCNN and its applications in various domains, with a more detailed description of QCNN fundamentals. In 2021, Eduardo [3] surveyed the quaternion applications from the aspect of quaternion algebra. One can learn geometric algebra and the rotation property of the quaternion number for applications such as kinematics, tracking, and the control of robotics. Moreover, some packages and links for the mentioned applications can be found in [3]. Each of these surveys provides a comprehensive review of quaternion-based models from different perspectives. However, none of them includes a detailed review of color image processing with quaternion representation. In contrast, we cover traditional, deep learning, and hybrid quaternion-based color image processing models. The contributions of this work include:

• A detailed overview of color image processing with a quaternion representation;
• A comprehensive survey of the different algorithms along with their benefits and limitations;
• A summary of each algorithm in detail, including objectives, goals, and weaknesses, and a discussion of recent challenges and their possible solutions.

The main objective of this paper is to comprehensively analyze the potential applications of quaternion representation and provide an overview of recent research advancements. The rest of the paper is organized as follows: Section 2 provides a review of the basic theory of quaternion and its related definitions. In Section 3, we discuss the successful implementation of traditional variation models. Section 4 reviews the variants used to connect quaternion ideas within neural networks, with a focus on the significant breakthroughs. Promising research directions and their associated challenges are discussed in Section 5. Finally, the paper concludes in Section 6.

2. Basic Theory

In this section, we first present the fundamental of color image processing, then provide a brief introduction of quaternion and some related definitions.

2.1. Color Image Processing Model

Color image processing is generally an extension of gray image processing. Since the image will be corrupted by noise and blur, the general image degradation model is

$$f=Au+b,$$

(2)

where f is the observation, u is the desired image, b is additive noise, and A is a linear operator. For the image deblurring task, A is the blur operator related to the blur kernel; for the image denoising task, A is the identical operator; for the image super-resolution task, A is the downsampling operator; for medical image reconstruction, A is the sampling operator; and for image inpainting, A is a projection operator. There are many methods that can restore the desired image u from f. One classical method is the total variation (TV) model [42,43,44]:

$$u=\underset{u}{argmin}\frac{\lambda }{2}{\parallel Au-f\parallel }_2^2+{\parallel \nabla u\parallel }_1,$$

(3)

where $\lambda >0$ is a trade-off parameter and ∇ is the gradient operator. ${\parallel \nabla u\parallel }_1$ is the TV term defined by

$${(\nabla u)}_{j,k}=\left({(\nabla u)}_{j,k}^x,{(\nabla u)}_{j,k}^y\right)$$

with

$${(\nabla u)}_{j,k}^x=\left\{\begin{array}{cc}{u}_{j+1,k}-u_{j,k}\hfill & \mathrm{if}j<n,\hfill \\ 0\hfill & \mathrm{if}j=n,\hfill \end{array}\right.$$

$${(\nabla u)}_{j,k}^y=\left\{\begin{array}{cc}{u}_{j,k+1}-u_{j,k}\hfill & \mathrm{if}k<n,\hfill \\ 0\hfill & \mathrm{if}k=n,\hfill \end{array}\right.$$

for $j,k=1,\dots ,n$. Here, $u_{j,k}$ refers to the $(jn+k)$ th entry of the vector u (it is the $(j,k)$th pixel location of the image). The discrete TV of u is defined by

$${\parallel \nabla u\parallel }_1:=\sum _{1\le j,k\le n}{\left|{(\nabla u)}_{j,k}\right|}_2=\sum _{1\le j,k\le n}\sqrt{{\left|{(\nabla u)}_{j,k}^x\right|}^2+{\left|{(\nabla u)}_{j,k}^y\right|}^2}$$

and ${|·|}_2$ is the Euclidean norm in ${\mathbb{R}}^2$.

We can obtain the solution to model (3) with many algorithms; the classical one is the alternating direction method of multipliers (ADMM) [45,46]. By introducing the auxiliary variable p, (3) can be reformulated as

$$\begin{array}{cc}\hfill u=& \underset{u}{argmin}\frac{\lambda }{2}{\parallel Au-f\parallel }_2^2+{\parallel p\parallel }_1,\\ \hfill & \mathrm{s}.\mathrm{t}.p=\nabla u.\hfill \end{array}$$

(4)

The augmented Lagrangian function is given by attaching multiplier $\xi$

$$\mathcal{L}(u,p;\xi )=\frac{\lambda }{2}{\parallel Au-f\parallel }_2^2+{\parallel p\parallel }_1+\frac{\beta }{2}{\parallel p-\nabla u\parallel }_2^2+\langle \xi ,p-\nabla u\rangle ,$$

(5)

where $\beta$ is the penalty parameter for linear constraints to be satisfied. From Algorithm 1, the final solution $u^{k+1}$ can be obtained. If we extend the whole algorithm and model into the quaternion system, at the very least, the norm, gradient, multiple, and division should be defined. In Section 2.2, we will provide these definitions in quaternion by referring to the quaternion theory.

Algorithm 1 ADMM for solving (3)

Initialization Let $u^0=f$, $\xi =0$ be the initial input data for  $k=0\to K$  do    Update $p^{k+1}$ with $$\begin{array}{ll}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{p}^{k+1}& =\underset{p}{argmin}{\parallel p\parallel }_1+\frac{\beta }{2}{\parallel p-\nabla u^k\parallel }_2^2+\langle {\xi }^k,p-\nabla u^k\rangle \\ & =\underset{p}{argmin}{\parallel p\parallel }_1+\frac{\beta }{2}\parallel p-\nabla u^k+\frac{{\xi }^k}{\beta }{\parallel }_2^2\\ & =max\left(\parallel \nabla u^k-\frac{{\xi }^k}{\beta }{\parallel }_2-\frac{1}{\beta },0\right)\frac{\nabla u^k-\frac{{\xi }^k}{\beta }}{\parallel \nabla u^k-\frac{{\xi }^k}{\beta }{\parallel }_2}\end{array}$$    Update $u^{k+1}$ with $$\begin{array}{ll}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{u}^{k+1}& =\underset{u}{argmin}\frac{\lambda }{2}{\parallel Au-f\parallel }_2^2+\frac{\beta }{2}{\parallel p^{k+1}-\nabla u\parallel }_2^2+\langle {\xi }^k,p^{k+1}-\nabla u\rangle \\ & =\frac{\lambda A^{*}f+\beta {\nabla }^{*}(p^{k+1}+\frac{{\xi }^k}{\beta })}{\lambda A^{*}A+\beta \nabla {\nabla }^{*}}\end{array}$$    Update ${\xi }^{k+1}$ with ${\xi }^{k+1}={\xi }^k-\beta (p^{k+1}-\nabla u^{k+1})$    $k=k+1$. end for return  $u^{k+1}$

Owing to the powerful feature representation capabilities of deep learning, deep convolutional neural network (CNN)-based image processing methods have been developed and have shown remarkable performance [47,48,49]. The CNN learns by discovering intricate structures in training data, which are mainly composed of convolution layers, pooling layers, and fully connected layers. The convolution layer extracts features from high-dimensional data by using a set of convolution kernels. After obtaining these features, they are used for classification by first proceeding to the feature section in the pooling layer, dividing the features into disjoint regions, and taking the mean (or maximum) feature activation over these regions to obtain the pooled convolved features. The fully connected layer is then used for classification. For color images, there are also some quaternion-based CNN (QCNN) models that are considered better than real-valued CNNs in color preservation and parameter reduction [40,50,51,52]. The basic model in the quaternion system will be introduced in Section 2.3.

2.2. Quaternion

Quaternion was proposed by Hamilton in 1843 [17]. As mentioned before, the quaternion number system is an extension of complex numbers. A quaternion number $\dot{x}$ is usually represented as a linear combination of a real part and three imaginary parts, i.e.,

$$\dot{x}=x_0+x_{1}i+x_{2}j+x_{3}k,$$

(6)

where $x_0$ is the real part and $x_1,x_2,x_3$ are the imaginary parts of the quaternion number $\dot{x}$, $i,j,k$ are the fundamental quaternion units that satisfy

$$i^2=j^2=k^2=ijk=-1$$

and

$$ij=k,jk=i,ki=j,ik=-j,kj=-i,ji=-k.$$

The quaternion unit rules infer that multiplication is not commutative. The Hamilton product of two quaternions, $\dot{x}=x_0+x_{1}i+x_{2}j+x_{3}k$ and $\dot{y}=y_0+y_{1}i+y_{2}j+y_{3}k$, is

$$\begin{array}{cc}\hfill \dot{x}\dot{y}=& x_0{y}_0-x_1{y}_1-x_2{y}_2-x_3{y}_3\hfill \\ \hfill +& (x_0{y}_1+x_1{y}_0+x_2{y}_3-x_3{y}_2)i\hfill \\ \hfill +& (x_0{y}_2-x_1{y}_3+x_2{y}_0+x_3{y}_1)j\hfill \\ \hfill +& (x_0{y}_3+x_1{y}_2-x_2{y}_1+x_3{y}_0)k.\hfill \end{array}$$

(7)

Physically, $\dot{x}\dot{y}$ is rotation $\dot{x}$ followed by rotation $\dot{y}$. The multiple of $\dot{x}\dot{y}$ can also be written as

$$\dot{x}\dot{y}=\left[\begin{array}{cccc}{x}_0& -x_1& -x_2& -x_3\\ x_1& x_0& -x_3& x_2\\ x_2& x_3& x_0& -x_1\\ x_3& -x_2& x_1& x_0\end{array}\right]\left[\begin{array}{c}{y}_0\\ y_1\\ y_2\\ y_3\end{array}\right].$$

(8)

The dot product of $\dot{x}$ and $\dot{y}$ is

$$\langle \dot{x},\dot{y}\rangle =x_0{y}_0+x_1{y}_1+x_2{y}_2+x_3{y}_3.$$

(9)

The conjugate of a quaternion number $\dot{x}$ is ${\dot{x}}^{*}=x_0-x_{1}i-x_{2}j-x_{3}k$, the modulus is $|\dot{x}|=\sqrt{\dot{x}{\left(\dot{x}\right)}^{*}}=\sqrt{x_0^2+x_1^2+x_2^2+x_3^2}$ (one can check by Equation (8)), the inverse is ${\dot{x}}^{-1}=\frac{{\dot{x}}^{*}}{|\dot{x}{|}^2}$, and the inverse of $\dot{x}$ and $\dot{y}$ is

$${\left(\dot{x}\dot{y}\right)}^{-1}=\frac{{\left(\dot{x}\dot{y}\right)}^{*}}{|\dot{x}\dot{y}{|}^2}=\frac{{\dot{y}}^{*}{\dot{x}}^{*}}{|\dot{y}{|}^2{\left|\dot{x}\right|}^2}=\frac{{\dot{y}}^{*}}{|\dot{y}{|}^2}\frac{{\dot{x}}^{*}}{|\dot{x}{|}^2}={\dot{y}}^{-1}{\dot{x}}^{-1}.$$

(10)

If $|\dot{x}|=1$, we call $\dot{x}$ a unit quaternion number. If $\Re \left(\dot{x}\right)=x_0=0$, we call $\dot{x}$ a pure quaternion number, where $\Re (·)$ denotes the real part of a quaternion number. The required rules of quaternion matrix derivatives for image processing are listed in

Table 1. Derivatives of the functions of type $f\left(\dot{q}\right)$.

$\mathit{f}\left(\dot{\mathit{q}}\right)$	${\mathcal{D}}_{\dot{\mathit{q}}}\mathit{f}$	Note
$\dot{q}$	1	$\dot{q}\in \mathbb{H}$
$\dot{\mu }\dot{q}$	$\dot{\mu }$	$\forall \dot{\mu }\in \mathbb{H}$
$\dot{q}\dot{\nu }$	$\Re \left(\dot{\nu }\right)$	$\forall \dot{\nu }\in \mathbb{H}$, $\Re \left(\dot{\nu }\right)$ denotes the real part of $\dot{\nu }$
$\dot{\mu }\dot{q}\dot{\nu }+\dot{\tau }$	$\dot{\mu }\Re \left(\dot{\nu }\right)$	$\forall \dot{\mu },\dot{\nu },\dot{\tau }\in \mathbb{H}$
${\dot{q}}^{*}$	$-\frac{1}{2}$	${\dot{q}}^{*}$ denotes the conjugation of $\dot{q}$
$\dot{\mu }{\dot{q}}^{*}$	$-\frac{1}{2}\dot{\mu }$	$\forall \dot{\mu }\in \mathbb{H}$
${\dot{q}}^{*}\dot{\nu }$	$-\frac{1}{2}{\dot{\nu }}^{*}$	$\forall \dot{\nu }\in \mathbb{H}$
$\dot{\mu }{\dot{q}}^{*}\dot{\nu }+\dot{\tau }$	$-\frac{1}{2}\dot{\mu }{\dot{\nu }}^{*}$	$\forall \dot{\mu },\dot{\nu },\dot{\tau }\in \mathbb{H}$
${\dot{q}}^{-1}$	$-{\dot{q}}^{-1}\Re \left({\dot{q}}^{-1}\right)$	${\dot{q}}^{-1}$ denotes the reciprocal of $\dot{q}$
${\left({\dot{q}}^{*}\right)}^{-1}$	$\frac{1}{2|\dot{q}{|}^2}$	–
${(\dot{\mu }\dot{q}\dot{\nu }+\dot{\tau })}^2$	$\dot{\mathrm{g}}\dot{\mu }\Re \left(\dot{\nu }\right)+\dot{\mu }\Re \left(\dot{\nu }\dot{\mathrm{g}}\right)$	$\dot{\mathrm{g}}=\dot{\mu }\dot{q}\dot{\nu }+\dot{\tau }$
${(\dot{\mu }{\dot{q}}^{*}\dot{\nu }+\dot{\tau })}^2$	$-\frac{1}{2}\dot{\mathrm{g}}\dot{\mu }{\dot{\nu }}^{*}-\frac{1}{2}\dot{\mu }{\left(\dot{\nu }\dot{\mathrm{g}}\right)}^{*}$	$\dot{\mathrm{g}}=\dot{\mu }{\dot{q}}^{*}\dot{\nu }+\dot{\tau }$
$|\dot{\mu }\dot{q}\dot{\nu }+\dot{\tau }|$	$\frac{{\dot{\mathrm{g}}}^{*}}{2|\dot{\mathrm{g}}|}\dot{\mu }\Re \left(\dot{\nu }\right)-\frac{1}{4|\dot{\mathrm{g}}|}{\dot{\nu }}^{*}{\left({\dot{\mu }}^{*}\dot{\mathrm{g}}\right)}^{*}$	$\dot{\mathrm{g}}=\dot{\mu }\dot{q}\dot{\nu }+\dot{\tau }$
$|\dot{\mu }{\dot{q}}^{*}\dot{\nu }+\dot{\tau }|$	$\frac{\dot{\mathrm{g}}}{2|\dot{\mathrm{g}}|}{\dot{\nu }}^{*}\Re \left({\dot{\mu }}^{*}\right)-\frac{1}{4|\dot{\mathrm{g}}|}\dot{\mu }{\left(\dot{\nu }{\dot{\mathrm{g}}}^{*}\right)}^{*}$	$\dot{\mathrm{g}}=\dot{\mu }{\dot{q}}^{*}\dot{\nu }+\dot{\tau }$
$|\dot{\mu }\dot{q}\dot{\nu }+\dot{\tau }{|}^2$	${\dot{\mathrm{g}}}^{*}\dot{\mu }\Re \left(\dot{\nu }\right)-\frac{1}{2}{\dot{\nu }}^{*}{\left({\dot{\mu }}^{*}\dot{\mathrm{g}}\right)}^{*}$	$\dot{\mathrm{g}}=\dot{\mu }\dot{q}\dot{\nu }+\dot{\tau }$
$|\dot{\mu }{\dot{q}}^{*}\dot{\nu }+\dot{\tau }{|}^2$	$\dot{\mathrm{g}}{\dot{\nu }}^{*}\Re \left({\dot{\mu }}^{*}\right)-\frac{1}{2}\dot{\mu }{\left(\dot{\nu }{\dot{\mathrm{g}}}^{*}\right)}^{*}$	$\dot{\mathrm{g}}=\dot{\mu }{\dot{q}}^{*}\dot{\nu }+\dot{\tau }$

. We refer the reader to [53] for details. With the derivative rules of the quaternion function, we can solve the quaternion-based model directly. Suppose the nuclear norm model was extended to the quaternion domain as

$$\underset{\dot{u}}{min}\frac{\lambda }{2}\parallel \dot{A}\dot{u}-\dot{f}{\parallel }_2^2+{\parallel \dot{u}\parallel }_{★},$$

(11)

where $\dot{u}$ is the desired image, $\dot{A}$ is the linear operator, f is the observation, and $\parallel \dot{u}{\parallel }_{★}$ is the nuclear norm of $\dot{u}$, which sums the singular values of $\dot{u}$. Before solving (11), we give the definition of the quaternion singular value decomposition (SVD) as follows. Let $\dot{S}\in {\mathbb{H}}^{m\times n}$, then there exist two unitary quaternion matrices $\dot{U}\in {\mathbb{H}}^{m\times m}$ and $\dot{V}\in {\mathbb{H}}^{n\times n}$, such that $\dot{U}\dot{S}{\dot{V}}^{*}=\sum$, where $\sum =diag\left({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_s\right)$, ${\sigma }_i\ge 0$ are the singular values of $\dot{S}$ and $s=min(m,n)$.

Let $\dot{p}=\dot{u}$, with the ADMM algorithm; one can obtain the augmented Lagrangian function, which is similar to (5)

$$\mathcal{L}(\dot{u},\dot{p};\dot{\xi })=\frac{\lambda }{2}\parallel \dot{A}\dot{u}-\dot{f}{\parallel }_2^2+\parallel \dot{p}{\parallel }_{★}+\frac{\beta }{2}{\parallel \dot{p}-\dot{u}\parallel }_2^2+\langle \dot{\xi },\dot{p}-\dot{u}\rangle .$$

(12)

Then we have

$$\left\{\begin{array}{cc}{\dot{u}}^{k+1}\hfill & =\underset{\dot{u}}{min}\frac{\lambda }{2}\parallel \dot{A}\dot{u}-\dot{f}{\parallel }_2^2+\frac{\beta }{2}\parallel \dot{u}-({\dot{p}}^k+\frac{{\dot{\xi }}^k}{\beta }){\parallel }_2^2,\hfill \\ {\dot{p}}^{k+1}\hfill & =\underset{\dot{p}}{min}\parallel \dot{p}{\parallel }_{★}+\frac{\beta }{2}\parallel \dot{p}-({\dot{u}}^{k+1}-\frac{{\dot{\xi }}^k}{\beta }){\parallel }_2^2,\hfill \\ {\dot{\xi }}^{k+1}\hfill & ={\dot{\xi }}^k-\beta ({\dot{p}}^{k+1}-{\dot{u}}^{k+1}).\hfill \end{array}\right.$$

(13)

According to Table 1, we have the optimization condition

$$\begin{array}{cc}\hfill & \frac{\lambda }{2}\left({(\dot{A}\dot{u}-\dot{f})}^{*}\dot{A}-\frac{1}{2}\left({\left({\dot{A}}^{*}(\dot{A}\dot{u}-\dot{f})\right)}^{*}\right)\right)+\frac{\beta }{4}\left(\dot{u}-{({\dot{p}}^k+\frac{{\dot{\xi }}^k}{\beta })}^{*}\right)\hfill \\ \hfill =& \frac{\lambda }{2}\left({(\dot{A}\dot{u}-\dot{f})}^{*}\dot{A}-\frac{1}{2}\left({(\dot{A}\dot{u}-\dot{f})}^{*}\dot{A}\right)\right)+\frac{\beta }{4}\left(\dot{u}-{({\dot{p}}^k+\frac{{\dot{\xi }}^k}{\beta })}^{*}\right)\hfill \\ \hfill =& \frac{\lambda }{4}\left({(\dot{A}\dot{u}-\dot{f})}^{*}\dot{A}\right)+\frac{\beta }{4}\left(\dot{u}-{({\dot{p}}^k+\frac{{\dot{\xi }}^k}{\beta })}^{*}\right)=0,\hfill \end{array}$$

(14)

thus, the solution is

$${\dot{u}}^{k+1}=\frac{\lambda {\dot{A}}^{*}\dot{f}+\beta ({\dot{p}}^k+\frac{{\dot{\xi }}^k}{\beta })}{\lambda {\dot{A}}^{*}\dot{A}+\beta }.$$

(15)

For $\dot{p}$-subproblem, the QSVD can directly give the closed-form solution. If the regularizer is the TV term, we first rewrite the $\dot{p}$-subproblem as

$$\underset{\dot{p}}{min}\parallel \dot{p}{\parallel }_1+\frac{\beta }{2}\parallel \dot{p}-\nabla {\dot{u}}^k+\frac{{\dot{\xi }}^k}{\beta }{\parallel }_2^2.$$

(16)

Let ${\dot{p}}_i$ be the i-th element of $\dot{p}$, then we have

$$E=\parallel {\dot{p}}_i{\parallel }_1+\frac{\beta }{2}\parallel {\dot{p}}_i-(\nabla {\dot{u}}_i^k-\frac{{\dot{\xi }}_i^k}{\beta }){\parallel }_2^2,$$

(17)

and

$$\frac{\partial E}{\partial {\dot{p}}_i}=\lambda \frac{{\dot{p}}_i-(\nabla {\dot{u}}_i^k-\frac{{\dot{\xi }}_i^k}{\beta })}{4}+\frac{\overline{\dot{p_i}}}{4|{\dot{p}}_i|}.$$

(18)

Let $\frac{\partial E}{\partial {\dot{p}}_i}=0$ and $\nabla {\dot{u}}_i^k-\frac{{\dot{\xi }}_i^k}{\beta }={\dot{y}}_i$, then $\frac{\overline{\dot{p_i}}}{|{\dot{p}}_i|}=\frac{\overline{\dot{y_i}}}{|{\dot{y}}_i|}$. By discussing $|{\dot{y}}_i|>\lambda$ or $\le \lambda$, we have

$${\dot{p}}_i=\frac{{\dot{y}}_i}{\left|{\dot{y}}_i\right|}·max\left(\left|{\dot{y}}_i\right|-\lambda ,0\right).$$

(19)

The visual performance of the quaternion-based method is given in Figure 3 and Figure 4. We use the codes https://github.com/Huang-chao-yan/QWNNM (accessed on 11 December 2020) of [54]. We add the average blur with blur kernel 9 and Gaussian noise with noise level σ = 20 in Figure 3, and the Gaussian blur with blur kernel [25,1.6] and Gaussian noise with noise level 20 in Figure 4. The results show that color spots are still visible in the output of the real-valued weighted nuclear norm minimization (WNNM)-based method proposed in [55]. The quaternion-basedWNNM [54] can better preserve the detailed structure of the image.

2.3. Quaternion Modules

The convolution process is defined in a real-valued space by convolving a filter matrix with a vector. In QCNN, a quaternion filter matrix and quaternion vector are convoluted. We denote the quaternion weight filter matrix as $\dot{w}=w_0+w_{1}i+w_{2}j+w_{3}k$ and quaternion input as $\dot{u}=u_0+u_{1}i+u_{2}j+u_{3}k$; then the quaternion convolution is defined as the Hamilton product

$$\begin{array}{cc}\hfill \dot{w}\otimes \dot{u}=& \left(w_0{u}_0-w_1{u}_1-w_2{u}_2-w_3{u}_3\right)\hfill \\ \hfill +& \left(w_0{u}_1+w_1{u}_0+w_2{u}_3-w_3{u}_2\right)i\hfill \\ \hfill +& \left(w_0{u}_2-w_1{u}_3+w_2{u}_0+w_3{u}_1\right)j\hfill \\ \hfill +& \left(w_0{u}_2+w_1{u}_2-w_2{u}_1+w_3{u}_0\right)k.\hfill \end{array}$$

(20)

In CNN, the fully connected layer is defined as $f=\varphi (wu+b)$. In QCNN, the quaternion fully connected layer is usually defined as

$$\dot{f}=\varphi (\dot{w}\dot{u}+\dot{b}),$$

(21)

where $\dot{b}$ is the bias and $\varphi (·)$ is any activation function. Suppose that with the rectified linear unit (ReLU) activation function, the final result is

$$f=\mathrm{ReLU}\left(z_0\right)+\mathrm{ReLU}\left(z_1\right)i+\mathrm{ReLU}\left(z_2\right)j+\mathrm{ReLU}\left(z_3\right)k$$

(22)

with $z=z_0+z_{1}i+z_{2}j+z_{3}k=\dot{w}\dot{u}+\dot{b}$. Due to the linear combination property, the conditionality of $\dot{w}$ is $1/4\left|w\right|$; then, QCNNs can be built with $1/4$ of the parameters required by their real-valued counterparts. In this case, the design of a quaternion-based activation function and other general quaternion modules may help to further improve the QCNN models. Overall, the benefits of QCNNs can be summarized as follows:

• Reduced network size: QCNNs can represent weights using fewer parameters than traditional CNNs, thereby reducing the overall size of the network.
• Improved performance: QCNNs outperform traditional CNNs on numerous tasks, particularly those involving 3D data, such as video analysis and computer vision.
• Efficient computation: Quaternion operations can be efficiently implemented using GPUs, resulting in fast training and inference times.

3. Traditional Methods

Based on the aforementioned quaternion rules and definitions, the quaternion representation is widely applied in color image processing. In this section, we will provide an overview of the main contributions in six aspects. Firstly, TV-based methods are discussed in Section 3.1; secondly, low-rank-based and sparse-based models are reviewed in Section 3.2; thirdly, moment-based models are introduced in Section 3.3; fourthly, decomposition-based models are presented in Section 3.4; fifthly, transformation-based models are reviewed in Section 3.5; finally, other significant models are summarized in Section 3.6.

3.1. TV-Based Models

As mentioned in Section 2.1, the total variation (TV)-based model is defined by Equation (3). Due to the model’s effectiveness, many works have improved it, such as non-local TV and TVp models. There are also references for quaternion-based image processing. For example, Liu et al. [56] extended the fractional order TV with $l_p$ norm to the quaternion domain for image super-resolution. The non-local TV was extended to the quaternion domain with unit transformation for image denoising [57]. Jia et al. [58] applied quaternion representation in the HSV color space and proposed a saturation value-TV (SV-TV) model for image denoising and deblurring. Voronin et al. [59] proposed an automated segmentation analysis based on the modified Chan and Vese method using the quaternion anisotropic TV algorithm under Merced data (https://vision.ucmerced.edu/datasets/, accessed on 1 March 2023). Wu et al. [18] extended the $l_1/l_2$ regularizer to the quaternion domain for image segmentation under the Weizmann (https://www.wisdom.weizmann.ac.il/vision/Seg_Evaluation_DB/dl.html, accessed on 1 March 2023) and Berkeley (https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/, accessed on 1 Mar. 2023) datasets. Their works extended the original TV-based model into the quaternion domain, and some works explored the theoretical properties of their proposed quaternion models. Furthermore, the experimental results illustrated the superiority of the quaternion representation. By representing color images as a whole, the color information between color channels can be well-preserved.

3.2. Low-Rank-Based and Sparse-Based Models

The basic low-rank minimization problem is

$$\begin{array}{cc}\hfill & \underset{u}{min}{\parallel u-f\parallel }_2^2,\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\mathrm{rank}\left(u\right)\le r,\hfill \end{array}$$

(23)

where $\mathrm{rank}\left(u\right)$ is the rank of matrix u and r is a positive number. The above constriction was reformulated to other rank functions to better represent the low-rank properties, such as the Schatten-$\gamma$ norm, nuclear norm, logarithm, Laplace, and Geman function. In the quaternion representation, Chen, Xiao, and Zhou [60] extended the low-rank regularizer-based model (Laplace, Geman, weighted Schatten-$\gamma$) into the quaternion domain, showing that the quaternion representation with low-rank is better than real-valued low-rank methods in color image in painting and denoising. In [54,61], the weighted nuclear norm was extended to the quaternion domain for image denoising and deblurring. Other low-rank versions of the quaternion representation can be found in [62,63].

Let D be the dictionary matrix and $\alpha$ be a sparse coefficient matrix. The core sparse representation problem involves finding the sparsest $\alpha$ that satisfies $u=D\alpha$. Mathematically, the sparse decomposition problem becomes

$$\begin{array}{cc}\hfill & \underset{\alpha }{min}{\parallel \alpha \parallel }_0,\hfill \\ & {\mathrm{s}.\mathrm{t}.\parallel u-D\alpha \parallel }_2^2\le {ϵ}^2,\hfill \end{array}$$

(24)

where ${\parallel \alpha \parallel }_0=\#\{i:{\alpha }_i\ne 0,i=1,2,\dots ,k\}$ is the $l_0$ norm of $\alpha$ and counts the non-zero number of $\alpha$. Here, k denotes the number of elements. Based on this equation, the sparse representation theory was developed by Elad et al. [64] for learned dictionaries. To preserve color information, Yu et al. [65] proposed a quaternion online dictionary-learning model for image super-resolution. Reference [66] extended the collaborative representation-based classification (CRC) and sparse representation-based classification (SRC) to the quaternion domain for face recognition. Xu et al. [67] proposed a quaternion sparse representation with the dictionary learning algorithm K-QSVD (generalized k-means clustering for quaternion singular value decomposition) and QOMP (quaternion orthogonal matching pursuit) for color image reconstruction, denoising, inpainting, and super-resolution. Following this work, Wu et al. [68] improved the sparse representation by combining the TV regularizer for image denoising. Meanwhile, the TV term was replaced by a more efficient regularizer, SV-TV, in [69]. They also improved the sparse prior by training the dictionary with the DIV2K dataset (https://data.vision.ee.ethz.ch/cvl/DIV2K/, accessed on 1 March 2023). Moreover, Liu et al. [70] combined the quaternion-based total variation and sparse dictionary learning for super-resolution in the Infrared LTIR dataset (http://www.cvl.isy.liu.se/research/datasets/ltir/version1.0/, accessed on 1 March. 2023) and IRData (http://www.dgp.toronto.edu/nmorris/data/IRData/, accessed on 1 March 2023). Furthermore, the block sparse representation was extended into the quaternion domain for face recognition in [71]. A sparse quaternion Welsch estimator was introduced to measure the quaternion residual error [72]; the estimator in the quaternion domain can largely suppress the impact of large data corruption and outliers.

On the other hand, principal component analysis (PCA) is another popular technique used for sparse and low-rank minimization problems, which can analyze large datasets containing a large number of dimensions. In the PCA theory, the observed image f is generated as $f=u+s$, where u is the target low-rank matrix and s is a sparse matrix that usually acts as the corruption datum. Under the above assumption, one can recover u by solving

$$\begin{array}{cc}\hfill & \underset{u,s}{min}{\parallel u\parallel }_{★}+\lambda {\parallel s\parallel }_1,\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.u+s=f,\hfill \end{array}$$

(25)

where ${\parallel ·\parallel }_{★}$ is the nuclear norm, ${\parallel ·\parallel }_1$ is the $l_1$ norm, and $\lambda$ is a positive parameter. Equation (25) was extended to the quaternion domain in [73] for image inpainting with a theoretical guarantee. Shi and Funt [74] extended the PCA to the quaternion domain and then derived a low-dimensional basis for color texture segmentation. Their model demonstrated the advantage of representing and analyzing an image as a single entity. Due to the competitive performance of quaternion-based PCA and the theoretical guarantee, there are many related works [75]. For example, Wang et al. [76] proposed a robust subspace learning method with PCA for face recognition under the Georgia Tech face dataset (https://computervisiononline.com/dataset/1105138700, accessed on 1 Mar. 2023) and the color FERET (https://www.nist.gov/itl/iad/imagegroup/color-feret-database, accessed on 1 Mar. 2023) dataset. Sun et al. [77] suggested modified two-dimensional principal component analysis (2DPCA) and bidirectional principal component analysis (BDPCA) methods based on the quaternion matrix to recognize and reconstruct face images. Jia et al. [78] presented the quaternion-based two-dimensional principal component analysis (2DPCA) for face recognition.

3.3. Moment-Based Models

Owing to their image descriptions and invariance properties, moments are scalar quantities widely used in image processing. Various types of moment functions have been constructed, such as orthogonal moments, Zernike moments, exponent moments, and Chebyshev–Fourier moments. Due to the effectiveness of moment-based models, they have been extended to the quaternion domain as well [79]. For example, Wang et al. [80] proposed a robust watermarking model with local quaternion exponent moments. In [81], the Chebyshev moment was extended to the quaternion domain by designing a quaternion radial-substituted Chebyshev moment. Moreover, the quaternion-weighted spherical Bessel–Fourier moment (QSBFM) was proposed in [82] and the application of color image reconstruction and object recognition in the CVG-UGR dataset (http://decsai.ugr.es/cvg/dbimagenes/index.php, accessed on 1 March 2023), Amsterdam Library (https://ccia.ugr.es/cvg/dbimagenes/index.php, accessed on 1 March 2023), and Columbia Library (https://www.cs.columbia.edu/CAVE/software/softlib/coil-100.php, accessed on 1 March 2023) illustrated the effectiveness of the quaternion representation. In [83], the discrete orthogonal moment was applied to neural networks for color face recognition. Other moment-based models, such as the quaternion Fourier–Mellin moment [84] and the quaternion radial moment [85], demonstrated the competitiveness of quaternion-based models. However, the computational costs of quaternion moments are high. Deriving a fast quaternion moment algorithm may be a significant challenge.

3.4. Decomposition-Based Models

For color image processing, one effective method is to regard the color image as a matrix. With the help of the matrix analysis, the desired image can be solved. The QR decomposition was used to handle the linear least squares problem. The QR decomposition is a matrix factorization technique that decomposes an m × n matrix A into the product of two matrices: an orthogonal matrix Q and an upper triangular matrix R. The QR decomposition of A is given by

$$A=QR$$

(26)

where Q is an m × n orthogonal matrix (i.e., Q^TQ = I) and R is an m × n upper triangular matrix. For color image processing, QR decomposition is also widely used. Later, the quaternion QR decomposition was extended for better handling of color images, where the matrices in (26) are in the quaternion domain. The main difference between quaternion QR decomposition and real-valued QR decomposition is that quaternion QR decomposition factorizes a quaternionic matrix into the product of an orthogonal quaternion matrix and an upper triangular quaternion matrix. Generally, the quaternion QR decomposition is more computationally expensive than the real-valued QR decomposition due to the additional complexity introduced by working with quaternion numbers. However, the quaternion QR decomposition has applications in color image processing and is usually better than the real-valued QR decomposition. In [86], the QR decomposition in the quaternion domain was applied in watermarking. The blind watermarking in the quaternion domain with QR decomposition was proposed in [87]. Similarly, the Schur decomposition and singular value decomposition (SVD) are also extended to quaternion [88,89,90]. In particular, He et al. [91] applied the matrix decomposition for quaternion in the control system and for watermarking. A face recognition method using wavelet decomposition and quaternion correlation filters was proposed in [92]. Kumar et al. [93] proposed a medical image super-resolution model with quaternion wavelet transform (QWT) and SVD. Miao et al. [94] proposed a quaternion higher-order SVD method for image fusion and denoising, and demonstrated its effectiveness on the MFFW (https://www.researchgate.net/profile/Xu-Shuang-3/publication/350965471_MFFW/data/607d2a6d881fa114b411103c/MFFW.zip, accessed on 1 March 2023) and Lytro (http://clim.inria.fr/IllumDatasetLF/index.html, accessed on 1 March 2023) datasets.

3.5. Transformation-Based Models

One of the classical quaternion representations is the quaternion unit transform [95]. A unit vector $\dot{p}$ is defined as

$$\begin{array}{cc}\hfill \dot{p}& =cos\theta +\frac{1}{\sqrt{3}}\mu sin\theta \hfill \\ \hfill & =cos\theta +\frac{1}{\sqrt{3}}[(sin\theta )i+(sin\theta )j+(sin\theta )k]\hfill \end{array}$$

(27)

where $\mu =i+j+k$ is the pure imaginary axis. Then the unit transform for a color image $\dot{u}=u_{r}i+u_{g}j+u_{b}k$ is defined as

$$\begin{array}{cc}\hfill \dot{t}& =\dot{p}\dot{u}{\dot{p}}^{*}\hfill \\ \hfill & =\left[cos\theta +\frac{1}{\sqrt{3}}sin\theta (i+j+k)\right](u_{r}i+u_{g}j+u_{b}k)\left[cos\theta -\frac{1}{\sqrt{3}}sin\theta (i+j+k)\right]\hfill \\ \hfill & =cos2\theta (u_{r}i+u_{g}j+u_{b}k)+\frac{2}{3}\mu {sin}^2\theta (u_r+u_g+u_b)\hfill \\ & +\frac{1}{\sqrt{3}}sin2\theta [(u_b-u_g)i+(u_r-u_b)j+(u_g-u_r)k]\hfill \\ \hfill & =Y_{RGB}+Y_{\Delta }+Y_I,\hfill \end{array}$$

(28)

where $Y_{RGB}=cos2\theta (u_{r}i+u_{g}j+u_{b}k)$ represents the RGB space component, $Y_I=\frac{2}{3}\mu {sin}^2\theta (u_r+u_g+u_b)$ is the intensity, and $Y_{\Delta }=\frac{1}{\sqrt{3}}sin2\theta [(u_b-u_g)i+(u_r-u_b)j+(u_g-u_r)k]$ denotes the color difference. With the quaternion unit transformation (28), many excellent image processing works have been proposed. Geng, Hu, and Xiao [96] proposed a quaternion-switching filter for impulse noise reduction. They employed the color difference to detect whether the center pixel in a filtering window is noisy or not. Later, a two-stage method with the quaternion unit transform was proposed for removing impulse noise [97]. In 2019, Li, Zhou, and Zhang [57] extended the classical non-local total variation to the quaternion domain with the unit transform for color image denoising. A similar representation was also applied in [98] for color image enhancement.

With the quaternion Fourier transform (QFT), Bas, Bihan, and Chassery [99] presented an image watermarking scheme. Later, a blind color image watermarking method based on the quaternion Fourier transform and least squares support vector machine was proposed in [100]. An image watermarking approach based on quaternion discrete Fourier transform and an improved uniform log-polar mapping was introduced in [101]. Combining the superpixel image segmentation and QWT, Niu et al. [102] proposed a novel image watermarking approach. Grigoryan and Agaian [103] proposed an image restoration model with the Wiener filter and quaternion Fourier transform, which can handle denoising and deblurring tasks. Wang et al. [104] applied the QWT for a no-reference stereoscopic image quality assessment. Other transforms, such as the quaternion polar harmonic transform [105] and discrete wavelet transform [106], were also extended to the quaternion domain with better performance.

3.6. Other Models

Instead of the above models, there are other standard works in the quaternion domain. For example, the quaternion Gabor filter (QGF) [107,108] was introduced to extract the local orientation information. Later, Li et al. [109] improved a multiscale QGF to describe texture attributes. Zou et al. [110] utilized the linear regression classification and collaborative representation in the quaternion domain for face recognition on SCface (https://www.scface.org/, accessed on 1 March 2023), AR (https://www2.ece.ohio-state.edu/aleix/ARdatabase.html, accessed on 1 March 2023), and Caltech (https://www.vision.caltech.edu/datasets/caltech_10k_webfaces/, accessed on 1 March 2023) datasets. Liu et al. [111] proposed a quaternion-based maximum margin criterion (QMMC) algorithm for face recognition.

4. Deep Learning

Deep convolutional neural networks (CNNs) have shown great potential in computer vision. Quaternion-based convolutional neural networks (QCNNs) have also shown great potential. In [112], the authors proposed a quaternion-based approach for unsupervised feature learning that enables the joint encoding of intensity and color information. Later, they introduced unsupervised learning of quaternion feature filters and feature encoding [113]. Combining the traditional PCA theory, Zeng et al. [114] proposed a quaternion PCA network (QPCANet) for color image classification. In [50], the basic modules, such as the convolution layer and fully connected layer, were designed in the quaternion domain, which helped establish fully-quaternion convolutional neural networks. Later, a quaternion weight initialization scheme and algorithms for quaternion batch normalization were introduced [115]. Yin et al. [116] derived quaternion batch normalization and pooling operations, and incorporated the attention mechanism to boost the performance of QCNNs.

The classification and forensics results of the Uncompressed Colour Image Database (UCID) (https://qualinet.github.io/databases/image/uncompressed_colour_image_database_ucid/, accessed on 1 March 2023) illustrates the efficiency of a quaternion-based network. A similar idea was shown in [117]. By independently learning both internal and external relations, and with fewer parameters than a real-valued convolutional encoder–decoder, Reference [118] investigated the impact of the Hamilton product on a color image reconstruction task. The results on the Kodak dataset (https://r0k.us/graphics/kodak/, accessed on 1 March 2023) showed that the quaternion convolutional encoder–decoder can perfectly reconstruct unseen color information. Jin et al. [119] incorporated deformable quaternion Gabor filters into the convolutional neural network and applied the proposed model in facial expression recognition on Oulu-CASIA https://www.v7labs.com/opendatasets/oulu-casia, accessed on 1 March 2023), MMI (https://mmifacedb.eu/, accessed on 1 March 2023) and SFEW (https://cs.anu.edu.au/few/AFEW.html, accessed on 1 March 2023) datasets. At the same time, Zhou et al. [120] proposed a deep CNN with a Gabor attention module for facial expression recognition. Later, quaternion representations were added to attention networks for classification [38]. More specifically, axial-attention modules were supplemented with quaternion input representations to improve image classification accuracy in the ImageNet300k dataset (https://deepai.org/machine-learning-glossary-and-terms/imagenet, accessed on 1 Mar. 2023). Considering different types of noise, Cao et al. [121] proposed a convolutional attention-denoising network to remove random-valued impulse noise. Classical real-valued CNN models were also extended to the quaternion domain. For example, EdgeNet [122] was modified by proposing an end-to-end trainable quaternion-based super-resolution network (QSRNet) [123]. The experiment on image super-resolution demonstrates that the local and global interrelationships between the channels can be better maintained with fewer parameters.

In [124], a quaternion residual unit was employed to capture the interdependencies in a multidimensional input in the DCASE19 (https://zenodo.org/record/2589280#.ZBLAuOxBzJx, accessed on 1 March 2023) and DCASE20 (https://zenodo.org/record/3670167#.ZBLAEOxBzJw, accessed on 1 March 2023) datasets. As a result, quaternion encoding can increase accuracy with fewer parameters. Frants et al. [125] proposed a quaternion-based multi-stage multiscale neural network with a self-attention module for rain streak removal. They replaced all convolutional layers with the quaternion convolution layer and replaced the ReLU activation layer with its quaternion split version. Later, they proposed a single-image dehazing model based on quaternion neural networks [126]. EI et al. [83] added quaternion discrete orthogonal moments to the deep neural network to extract compact and pertinent features. Their recognition performance on datasets, such as Faces94 (https://cmp.felk.cvut.cz/spacelib/faces/faces94.html, accessed on 1 Mar. 2023), Faces95 (https://cmp.felk.cvut.cz/spacelib/faces/faces95.html, accessed on 1 Mar. 2023), Faces96 (https://cmp.felk.cvut.cz/spacelib/faces/faces96.html, accessed on 1 March 2023), Grimace (https://cmp.felk.cvut.cz/spacelib/faces/grimace.html, accessed on 1 March 2023), Georgia Tech Face (https://computervisiononline.com/dataset/1105138700, accessed on 1 March 2023), and FEI (https://fei.edu.br/cet/facedatabase.html, accessed on 1 March 2023) showed the superior performance of the quaternion-based deep neural network. Zhou et al. [127] designed a non-iterative quaternion routing algorithm to integrate quaternion-valued capsule networks. Xu et al. [128] proposed a plug-and-play model for image denoising and inpainting by combing the FFDNet [129] and low-rank (Laplace) function in the quaternion domain. In order to solve the proposed hybrid non-convex model, the ADMM and difference of the convex algorithm (DCA) were used. Moreover, the generative adversarial networks were extended to quaternion in [130].

5. Discussion

The aforementioned quaternion-based methods are classical and representative, and some typical methods are listed in

Table 2. Typical quaternion-based image processing models. The model structure, model prior, the used algorithm, testing data, and tasks are listed.

	Method	Model Structure	Prior	Algorithm	Testing Data	Task
2018–2011	QGmF [107]	–	Gabor Filter + hypercomplex exponential basis functions	Closed Form Solution	Common Used	Denoising/Inpainting/ Segmentation
	Xu et al. [67]	Sparse	Dictionary	KQSVD/QOMP	Animal Images	Reconstruction /Denoising/Inpainting/Super-resolution
	Zou et al. [66]	Sparse	CRC + Sparse RC	ADMM	AR/Caltech/SCface/ FERET/LFW	Face Recognition
	Kumar et al. [93]	Low-rank	QWT + QSVD	–	Biomedical Images	Super-resolution
	QPCANet [114]	Deep Network	Principal Component Analysis Network	Deep Learning	Caltech-101/Georgia Tech face/UC Merced Land Use	Classification
	QCNN [50]	Deep Network	Quaternion Representation	Deep Learning	COCO/Oxford flower102	Classification/Denoising
2019	QNLTV [57]	Non-local	Non-local Total Variation	Splitting Bargeman	Common Used	Denoising
	LRQA [60]	Low-rank	Laplace/Geman/Weighted Schatten-$\gamma$	Difference of Convex	Common Used	Denoising/Inpainting
	QWNNM [61]	Low-rank	Nuclear Norm	QSVD	Berkeley	Denoising
	QPHTs [105]	–	Chaotic System+Polar Harmonic Transform	-	Whole Brain Atlas	Watermarking
	QMC [73]	Low-rank	Nuclear Norm/${\ell }_1$ Norm	ADMM	Berkeley	Inpainting
	HOGS4 [70]	Sparse	Total Variation+High-order Overlapping Group Sparse	ADMM	Infrared LTIR/IRData	Super-resolution
	QSBFM [82]	Orthogonal Moment	Weighted Spherical Bessel–Fourier Moment	QSBFM	CVG-UGR/Amsterdam Library/Columbia Library	Reconstruction /Recognition
	QCROC [110]	Linear Regression Classification	Linear Regression Classification+Collaborative Representation	Collaborative Representation Optimized Classification	SCface/AR/Caltech	Recognition
	Yin et al. [116]	Deep Network	QCNN+Attention Mechanism	Deep Learning	UCID	Classification/Forensics
2020	Li et al. [86]	–	Discrete Fourier Transform+QR decomposition	Wavelet Transform+Just-noticeable Difference	Common Used	Watermarking
	Voronin et al. [59]	–	Modified Chan and Vese Method	Anisotropic Gradient Calculation	Merced	Segmentation
	F-2D-QPCA [76]	Low-rank+Sparse	F-norm	Principal Component Analysis	Georgia Tech Face/FERET	Face Recognition
	DQG-CNN [119]	Deep Network	Deformable Gabor Filter	Deep Learning	Oulu-CASIA/MMI/SFEW	Facial Expression Recognition
	Zhou et al. [120]	Deep Network	Gabor Attention	Deep Learning	Oulu-CASIA/MMI/SFEW	Facial Expression Recognition
2021	QBSR [71]	Sparse	Block Sparse Representation	ADMM	AR/SCface	Recognition
	Huang et al. [69]	Sparse	Dictionary+Total Variation	ADMM	DIV2K	Denoising/Deblurring
	Shahadat et al. [38]	Deep Network	Axial-attention Modules	Deep Learning	ImageNet300k	Classification
	RQSVR [72]	Sparse	Welsch Estimator	HQS/ADMM	AR/SCface	Reconstruction/Recognition
	He et al. [91]	Low-rank	Matrix Decomposition	PSVD	Common Used	Watermarking
2022	QSRNet [123]	Deep Network	Edge-Net	Deep Learning	DIV2K/Flickr2K /Set5/Set14/BSD100 /Urban100/UEC100	Super-resolution
	Yang et al. [62]	Low-rank	Logarithmic Norm	FISTA/ADMM	Common Used/Berkeley	Inpainting
	Wu et al. [18]	Total Variation	$l_1/l_2$-norm	spADMM	Weizmann/Berkeley	Segmentation
	QSAM-Net [125]	Deep Network	QCNN+Self Attention	Deep Learning	LOL	Rain Streak Removal
	QDOMNN [83]	Deep Network	Discrete Orthogonal Moments	Deep Learning	Faces94/Faces95 /Faces96/Grimace /Georgia Tech Face/FEI	Recognition
2023	RQNet [124]	Deep Network	Residual CNN	Deep Learning	DCASE19/DCASE20	Classification
	QHOSVD [94]	Low-rank	Higher-order SVD	Matrix Decomposition	Lytro/MFFW	Image Fusion/Denoising
	DLRQP [128]	Plug-and-play	FFDNet/Laplace	ADMM/DCA	Common Used	Denoising/Inpainting

. Overall, quaternion-based algorithms can be more memory-efficient than traditional methods, which is important when dealing with large datasets. In some cases, quaternion-based methods can provide more accurate results than traditional methods, especially for problems that require rotation-invariant features. Although the quaternion-based models show better performance, there are disadvantages. First of all, the previous description shows that most quaternion-based methods directly extend real-valued methods to the quaternion domain. A model that can better reflect the advantages of the quaternion should be proposed, such as the unit transform-based model, in which an image can be divided into two parts according to the properties of the quaternion. Secondly, the quaternion representation denotes the color image as an entirety, which can keep the interrelationships between color channels and reduce the parameters in CNN-based models. However, this could slow the computation and make it more costly, which could limit their use in real-time applications. An accelerated algorithm should be proposed to optimize the quaternion-based methods. Thirdly, while quaternions are well-suited for representing three dimensions, they may not be as useful in higher dimensions.

6. Conclusions

In this paper, we reviewed classical and representative quaternion-based methods in image processing according to their model types. We divided these models into two types: traditional and deep learning. Specifically, we introduced TV-based, low-rankbased, sparse-based, moment-based, decomposition-based, transformation-based, and deep learning-based models. We believe that this survey can help academics better understand quaternion-based models and further advance this topic.

Furthermore, quaternion representation has several potential future research directions in color image processing. Firstly, developing new color spaces based on quaternion representation could improve the accuracy and efficiency of color image processing algorithms. For instance, a quaternion-based color space may better capture the spatial and chromatic information in an image compared to traditional color spaces. Secondly, data augmentation techniques, such as rotation and scaling, are commonly used in deep learning to increase the size of training datasets. Using quaternion representations to perform these transformations could improve the robustness and generalization of models trained on color image datasets. Thirdly, QCNNs can use quaternion representations to learn features from color images more effectively than traditional CNNs. Future research could explore the performance of QCNNs by designing the quaternion modules.