Blind Image Quality Assessment Based on Classification Guidance and Feature Aggregation

Abstract

In this work, we present a convolutional neural network (CNN) named CGFA-CNN for blind image quality assessment (BIQA). A unique two-stage strategy is utilized which firstly identifies the distortion type in an image using Sub-Network I and then quantifies this distortion using Sub-Network II. Different from most deep neural networks, we extract hierarchical features as descriptors to enhance the image representation and design a feature aggregation layer in an end-to-end training manner applying Fisher encoding to visual vocabularies modeled by Gaussian mixture models (GMMs). Considering the authentic distortions and synthetic distortions, the hierarchical feature contains the characteristics of a CNN trained on the self-built dataset and a CNN trained on ImageNet. We evaluated our algorithm on four publicly available databases, and the results demonstrate that our CGFA-CNN has superior performance over other methods both on synthetic and authentic databases.

1. Introduction

Digital pictures may occur different distortions in the procedure of acquisition, transmission, and compression, leading to an unsatisfactory perceived visual quality or a certain level of annoyance. Thus, it is crucial to predict the quality of digital pictures in many applications, such as compression, communication, printing, display, analysis, registration, restoration, and enhancement [1,2,3]. Generally, image quality assessment approaches can be classified into three kinds according to the additional information needed. Specifically, full-reference image quality assessment (FR-IQA) [4,5,6,7] and reduced-reference image quality assessment (RR-IQA) [8,9,10] need full and partial information of reference images, respectively, while blind image quality assessment (BIQA) [11,12,13,14] performs quality measure without any information from the reference image. Thus, BIQA methods are more attractive in many practical applications because the reference image usually is not available or hard to derive.

Early studies mainly focused on one or more specific distortion types, such as Gaussian blur [15], blockiness from JPEG compression [16], or ringing arising from JPEG2000 compression [17]. However, images may be affected by unknown distortion in many practical scenarios. In contrast, general BIQA methods aim to work well for arbitrary distortion, which can be classified into two categories according to the features extracted, i.e., Natural Scene Statistics (NSS)-based methods and training-based methods.

NSS-based methods [18] assume that the natural image with non-distorted obeys certain perceptually relevant statistical laws that are violated by the presence of common image distortions, and they attempt to describe an image utilizing its scene statistics from different domains. For example, BIRSQUE [19] derives features from the locally normalized luminance coefficients in the spatial domain. M3 [20] utilizes the joint local contrast features from the gradient magnitude (GM) map and the Laplacian of Gaussian (LOG) response. Later, a perceptually motivated and feature-driven model is deployed in FRIQUEE [21], in which a large collection of features defined in various complementary, perceptually relevant color and transform-domain spaces are drawn from among the most successful BIQA models produced to date.

However, knowledge-driven feature extraction and data-driven quality prediction are separated in the above methods. It has been demonstrated that training-based methods outperform NSS-based methods by a large margin because a fully data-driven BIQA solution becomes possible. For example, CORNIA [22] constructs a codebook in an unsupervised manner, using raw image patches as local descriptors and using soft-assignment for encoding. Considering that the feature set generally adopted in previous methods are from zero-order statistics and insufficient for BIQA, HOSA [23] constructs a much smaller codebook using K-means clustering [24] and introducing higher-order statistics. In contrast, the above methods capture spatially normalized coefficients and codebook-based features which are learned automatically from beginning to end by using CNNs. For example, TCSN [25] aims to learn the complicated relationship between visual appearance and perceived quality via a two-stream convolutional neural network. DIQA [26] defines two separated CNN branches to learn objective distortion and human visual sensitivity, respectively.

In this work, we propose an end-to-end BIQA based on classification guidance and feature aggregation, which is accomplished by two sub-networks with shared features in the early layers. Due to the lack of training data, we construct a large-scale dataset by means of synthesizing distortions and pre-train Sub-Network I to identify an image into a specific distortion type from a set of pre-defined categories. We find the proposed method will be much harder to achieve high accuracies on authentic images if only it is exposed to synthetic distortions during training. Then, we extract hierarchical features from the shared layers of two-subnetworks and another CNN (VGG-16 [27]) pre-trained on ImageNet [28], in which pictures occur as a natural consequence of photography and a unified feature group is formed.

Sub-Network II takes the hierarchical features and the classification information as inputs to predict the perceptual quality. The combination of two sub-networks enables the learning framework to have the probability of favorable quality perception and proper parameter initialization in an end-to-end training manner. We design a feature aggregation layer that could convert arbitrary input seizes to a fixed-length representation. Then, a fully connected layer is exploited as a linear regression model to map the high-dimensional features into the quality scores. This allows the proposed CGFA-CNN to accept an image of any size as the input, thus there is no need to perform any transformation of images (including cropping, scaling, etc.), which would affect perceptual quality scores.

The paper is structured as follows. In Section 2, previous work on CNN-based BIQA related to our work is briefly reviewed. In Section 3, details of the proposed method are described. In Section 4, experimental results on the public IQA databases and the corresponding analysis are presented. In Section 5, the work of this paper is concluded.

2. Related Work

In this section, we provide a brief survey about the major solutions to the lack of training data in BIQA and a review of recent studies related to our work.

Due to the number of parameters to be trained on CNN is usually very large, the training set needs to contain sufficient data to avoid over-fitting. However, the number of samples and image contents in the public quality-annotated image databases are rather limited, which cannot meet the need for end-to-end training of a deep network. Currently, there are two main methods to tackle this challenge.

The first method is to train the model based on image patches. For example, deepIQA [29] randomly samples image patches from the entire image as inputs and predicts the quality score on local regions by assigning the subjective mean score (MOS) of the pristine image to all patches within it. Although taking small patches as inputs for data augmentation is superior to using the whole image in a given dataset, this method still suffers from limitations because local image quality with context varies across spatial locations even when the distortion is homogeneous. To resolve this problem, BIECON [30] makes use of the existing FR-IQA algorithms to assign quality labels for sampled image patches, but the performance of such a network depends highly on that of FR-IQA models. Other methods such as dipIQ [31] attempting to generate discriminable image pairs by involving FR-IQA models may suffer from similar problems.

The second method is to pre-train a network with large-scale datasets in other fields. For each pre-trained architecture, two types of back-end training strategies are available: replacing the last layer of the pre-trained CNN model with the regression layer and fine-tuning it with the IQA database to conduct image quality prediction or using SVR to regress the extracted features through the pre-trained networks onto subjective scores. For instance, DeepBIQ [32] reports on the use of different features extracted from pre-trained CNNs for different image classification tasks via ImageNet [28] and Places365 [33] as a generic image description. Kim et al. [34] selected the well-known deep CNN models AlexNet [35] and ResNet50 [36] as the architectures of the baseline models, which have been pre-trained for the task of image classification on ImageNet [28]. These methods directly inheriting the weights from the pre-trained models for general image classification tasks have a defect of low relevance to BIQA but unnecessary complexity.

To better address the training data shortage problem, MEON [37] proposes a cascaded multi-task framework, which firstly trains a distortion type identification network by large-scale pre-defined samples. Then, a quality prediction network is trained subsequently, taking advantage of distortion information obtained from the first stage. Furthermore, DB-CNN [38] not only constructs a pre-training set based on Waterloo Exploration Database [39] and PASCAL VOC [40] for synthetic distortions, but also uses ImageNet [28] to pre-train another CNN for authentic distortions. Motivated by the previous studies on MEON [37] and DB-CNN [38], we construct a pre-training set based on Waterloo Exploration Database [39] and PASCAL VOC [40] for synthetic distortions. Besides, both distortion type and distortion level are considered at the same time, which results in better quality-aware initializations and distortion information.

Although previous DNN-based BIQA methods have achieved significant performance, all of these methods usually comprise convolutional layers and pooling layers for feature extraction and employ fully connected layers for regression, which would suffer three limitations. First, such techniques as averaging or maximum pooling are too simple to be accurate for long sequences. Second, a fully connected layer is destructive to the high-dimensional disorder and spatial invariance of the local feature. Third, such CNNs typically require a fixed image size. To feed into the network, images have to be resized or cropped to a fixed size, and either scaling or cropping would cause the perceptual difference with the assigned quality labels. To tackle these challenges, we explore more sophisticated pooling techniques based on clustering approaches such as Bag-of-visual-words (BOW) [41], Vector of Locally aggregated Descriptors (VLAD) [42] and Fisher Vectors [43]. Studies have shown that integrating VLAD as a differentiable module in a neural network can significantly improve the aggregated representation for the task of place recognition [44] and video classification [45]. Our proposed feature aggregation layer acts as a pooling layer on top of the convolutional layers, which converts arbitrary input seizes to a fixed-length representation. Afterward, using a fully connected layer for regression does not require any preprocessing of the input image.

3. The Proposed Method

The framework CGFA-CNN is illustrated in Figure 1. Sub-Network I aims to classify an image into a specific distortion type and initialize the shared layers for a further learning process, which is firstly pre-trained on a self-built dataset. Sub-Network II predicts the perceptual quality of the same image, which is fine-tuned with the IQA databases and takes advantage of distortion information obtained from Sub-Network I. The feature aggregation layer (FV layer) and classification-guided gating unit (CGU) are described in Section 3.3 and Section 3.4.

3.1. Distortion Type Identification

3.1.1. Construction of the Pre-Training Dataset

Due to the deficiency of the available quality-annotated samples, we firstly construct a large-scale dataset based on Waterloo Database [39] and PASCAL VOC Database [40]. The former contains 4744 images and can be loosely categorized into seven classes. The latter contains 17,125 images covering 20 categories. In this paper, we merge the two databases and obtain 21,869 pristine images with various contents. Then, nine types of distortion are introduced: JPEG compression, JPEG2000 compression, Gaussian blur, white Gaussian noise, contrast stretching, pink noise, image quantization with color dithering, over-exposure, and under-exposure. We synthesize each image with five distortion levels following Ma et al. [39] except for over-exposure and under-exposure, where only three levels are generated according to Ma et al. [46]. The constructed dataset consists of 896,629 images, which are organized into 41 subcategories according to the distortion type and degradation level. We label these images by the subcategory they belong to.

3.1.2. Sub-Network I Architecture

Inspired by the VGG-16 network architecture [27], we design a similar structure subject to some modifications identifying the distortion type of the input image. Details are given in

Table 1. Architecture of Sub-Network I.

Layer Name	Type	Patch Size	Stride	Output Size
Conv 1-1	Conv + ReLU + BN	$3\times 3\times 48$	1	$H\times W\times 48$
Conv 2-1	Conv + ReLU + BN	$3\times 3\times 48$	2	$\frac{H}{2}\times \frac{W}{2}\times 48$
Conv 2-2	Conv + ReLU + BN	$3\times 3\times 64$	1	$\frac{H}{2}\times \frac{W}{2}\times 64$
Conv 3-1	Conv + ReLU + BN	$3\times 3\times 64$	2	$\frac{H}{4}\times \frac{W}{4}\times 64$
Conv 3-2	Conv + ReLU + BN	$3\times 3\times 64$	1	$\frac{H}{4}\times \frac{W}{4}\times 64$
Conv 4-1	Conv + ReLU + BN	$3\times 3\times 64$	2	$\frac{H}{8}\times \frac{W}{8}\times 64$
Conv 4-2	Conv + ReLU + BN	$3\times 3\times 128$	1	$\frac{H}{8}\times \frac{W}{8}\times 128$
Conv 5-1	Conv + ReLU + BN	$3\times 3\times 128$	2	$\frac{H}{16}\times \frac{W}{16}\times 128$
Conv 5-2	Conv + ReLU + BN	$3\times 3\times 128$	1	$\frac{H}{16}\times \frac{W}{16}\times 128$
Pool	MaxPool	$1\times 1\times 128$	1	$1\times 1\times 128$
FC-1	FC + ReLU	$1\times 1\times 256$	1	$1\times 1\times 256$
FC-2	FC + ReLU	$1\times 1\times 256$	1	$1\times 1\times 256$
FC-3	FC	$1\times 1\times 41$	1	$1\times 1\times 41$
Classifier	Soft-max	$1\times 1\times 41$	1	$1\times 1\times 41$

. The tailored VGG-16 network comprises a stack of convolutions (Conv) for feature extraction, one maximum pooling (MaxPool) for feature fusion, and three fully connected layers (FC) for feature regression. All hidden layers are equipped with the Rectified Linear Unit (ReLU) [35] and Batch Normalization (BN) [47]. We denote the input mini-batch training data by ${\left\{\left({\mathbf{X}}^{\left(n\right)},{\mathbf{p}}^{\left(\mathbf{n}\right)}\right)\right\}}_{n=i}^N$, where ${\mathbf{X}}^{\left(n\right)}$ is the nth input image and ${\mathbf{p}}^{\left(n\right)}$ is a multi-class indicator vector of the ground truth distortion type. We append the soft-max layer at the end and define the soft-max function as

$${\widehat{p}}_i^{\left(n\right)}({\mathbf{X}}^{\left(n\right)};\mathbf{W})=\frac{exp\left(y_i^{\left(n\right)}({\mathbf{X}}^{\left(n\right)};\mathbf{W})\right)}{{\sum }_{j=1}^{C}exp\left(y_i^{\left(n\right)}({\mathbf{X}}^{\left(n\right)};\mathbf{W})\right)},$$

(1)

where ${\widehat{\mathbf{p}}}_{}^{\left(n\right)}={\left[{\widehat{p}}_1^{\left(n\right)},\cdots ,{\widehat{p}}_C^{\left(n\right)}\right]}^T$ is a C-dimensional probability vector of the nth input in a mini-batch, indicating the probability of each distortion type. Model parameters of Sub-Network I are collectively denoted as $\mathbf{W}$. A cross-entropy loss is used to train this sub-network

$${\ell }_s(\left\{{\mathbf{X}}^{\left(n\right)}\right\};\mathbf{W})=-\sum _{n=1}^N\sum _{i=1}^C{p}_i^{\left(n\right)}log{\widehat{p}}_i^{\left(n\right)}({\mathbf{X}}^{\left(n\right)};\mathbf{W}).$$

(2)

Notably, in the fine-tuning phase, except for the shared layers, the rest of Sub-Network I only participates in the forward propagation and the parameters are fixed.

3.2. Feature Extraction and Fusion

In Figure 2, we can see that the representation of different distortion types varies in each convolution. Therefore, only using features extracted from the last convolution is not enough to predict the quality of an image. Inspired by the idea of combining the complementary features and hierarchical feature extraction strategy in our previous work [48], we resort to extracting features from low-level, middle-level and high-level convolutional layers as descriptors by rescaling and concatenating them. We design Sub-Network I to identify a given image’s distortion type pre-trained on a synthesized dataset. We find this takes advantage of synthetic images but fails to handle those authentically distorted. More details can be found in Section 4.5. Then, we model synthetic and authentic distortions by two separated CNNs and fuse the two feature sets into a unified representation for final quality prediction. The tailored VGG-16 pre-trained on ImageNet that contains many realistic natural images of different perceptual quality is added to extract relevant features for authentic images. In the proposed CGFA-CNN index, we take a raw image of $H\times W\times 3$ as input and predict its perceptual quality. Then, the fused feature group acquired is with the size of $\frac{H}{16}\times \frac{W}{16}\times D$. Here, D is the channel of hierarchical features. Sub-Network II takes the fused feature group and the estimated probability vector ${\widehat{\mathbf{p}}}_{}^{\left(n\right)}$ as inputs.

3.3. Feature Aggregation Layer and Encoding

In this paper, we design a feature aggregation layer that employs the Fisher Vectors (FV) [43] to perform the feature aggregation and encoding procedures. Because GMM [49] and FV are non-differentiable and fail to achieve theoretically valid backpropagation, we define a FV layer to yield a quality-aware feature vector $\mathit{f}$. The implementation is shown in Figure 3.

As illustrated in Figure 1, the fused feature group is a $\frac{H}{16}\times \frac{W}{16}\times D$ map, which can be considered as a set of D-dimensional descriptors extracted at $\frac{H}{16}\times \frac{W}{16}$ spatial locations. Then, we utilize GMM to obtain the cluster centers $\mathit{C}$ of K components and encoding vector $\mathit{f}$ of the image descriptors-X.

3.3.1. GMM Clustering

A Gaussian mixture model $p\left(\mathbf{x}\right|\theta )$ is a mixture of K multivariate Gaussian distributions [49], which can be formulated as

$$p\left(\mathbf{x}\right|\theta )=\sum _{k=1}^{K}p\left(\mathbf{x}\right|{\mu }_k,{\sum }_k){\pi }_k,$$

(3)

$$p\left(\mathbf{x}\right|{\mu }_k,{\sum }_k)=\frac{e^{-\frac{1}{2}{(\mathbf{x}-{\mu }_k)}^T{\sum }_k^{-1}(\mathbf{x}-{\mu }_k)}}{\sqrt{{\left(2\pi \right)}^{D}det{\sum }_k}},$$

(4)

$$\mathbf{\theta }=({\pi }_1,{\mu }_1,{\sum }_1,\cdots ,{\pi }_K,{\mu }_K,{\sum }_K),$$

(5)

where $\mathbf{\theta }$ is the vector of parameters of the model. For each Gaussian component, ${\pi }_k$ is the prior probability value, ${\mu }_k$ is the means, and ${\sum }_K$ is the diagonal covariance matrices. The parameters are learned from a training set of descriptors ${\mathbf{x}}_1,\dots ,{\mathbf{x}}_N$. The GMM defines the assignments $q_{ki}(k=1,\cdots ,$$K,i=1,\cdots ,N)$ of the N descriptors to the K Gaussian components

$$q_{ki}=\frac{p\left({\mathbf{x}}_i\right|{\mu }_k,{\sum }_k){\pi }_k}{{\sum }_{j=1}^{K}p\left({\mathbf{x}}_i\right|{\mu }_j,{\sum }_j){\pi }_j},k=1,\cdots ,K.$$

(6)

3.3.2. Fisher Encoding

Fisher encoding captures both the first- and second-order differences between the image descriptors and the centers of a GMM. The construction of the encoding begins by learning a GMM model $\theta$. For each $k=1,\cdots ,K$, define the vectors

$${\mathbf{u}}_k=\frac{1}{N\sqrt{{\pi }_k}}\sum _{i=1}^N{q}_{ki}\sum _k^{-\frac{1}{2}}({\mathbf{x}}_i-{\mu }_k),$$

(7)

$${\mathbf{v}}_k=\frac{1}{N\sqrt{2{\pi }_k}}\sum _{i=1}^N{q}_{ki}[({\mathbf{x}}_i-{\mu }_k)\sum _k^{-1}({\mathbf{x}}_i-{\mu }_k)-1].$$

(8)

The Fisher encoding of the set of local descriptors is then given by the concatenation of ${\mu }_k$ and ${\mathbf{v}}_k$ for all K components, giving an encoding of size $2\times D\times K$

$${\mathbf{f}}_{Fisher}={[{\mathbf{u}}_1^T,\cdots {\mathbf{u}}_K^T,{\mathbf{v}}_1^T,\cdots ,{\mathbf{v}}_K^T]}^T.$$

(9)

To integrate Fisher vector as a differentiable module in a neural network, we write the descriptor $x_i$ hard assignment to the cluster k as a soft assignment

$$a_k\left(x_i\right)=\frac{e^{-\alpha {∥x_i-c_k∥}^2}}{{\sum }_{j=1}^K{e}^{-\alpha {∥x_i-c_j∥}^2}}.$$

(10)

Then, we can write the FV representation as

$$FV{1}_{j,k}=\sum _{i=1}^N{a}_k\left(x_i\right)\left(\frac{x_i\left(j\right)-c_k\left(j\right)}{{\sigma }_k\left(j\right)}\right),$$

(11)

$$FV{2}_{j,k}=\sum _{i=1}^N{a}_k\left(x_i\right)\left({\left(\frac{x_i\left(j\right)-c_k\left(j\right)}{{\sigma }_k\left(j\right)}\right)}^2-1\right),$$

(12)

where $FV1$ and $FV2$ are capturing the first- and second-order statistics, respectively. $x_i\left(j\right)$ is the jth dimensions of the ith descriptor and $c_k\left(j\right)$ is the kth cluster centers. $c_k$ and ${\sigma }_k$ ($k\in \left[1,K\right]$) are the learnable clusters and the clusters’ diagonal covariance. We define $\alpha$ as positive ranging between 0 and 1.

Let ${\omega }_k=2\alpha c_k$ and $b_k=-\alpha {∥c_k∥}^2$; Equation (10) can then be written as

$$a_k\left(x_i\right)=\frac{e^{{\omega }_k^T{x}_i+b_k}}{{\sum }_{j=1}^K{e}^{{\omega }_j^T{x}_i+b_j}}.$$

(13)

where $\left\{{\omega }_k\right\}$, $\left\{b_k\right\}$, and $\left\{c_k\right\}$ are sets of trainable parameters for each cluster k.

3.3.3. Beyond the FV Aggregation

The source of discontinuities in the traditional Bag-of-visual-words (BOW) [41] and Vector of Locally aggregated Descriptors (VLAD) [42] are the hard assignments $q_{ki}$ of descriptors $\mathbf{x}$ to cluster centers $c_k$. To make this operation differentiable, we replace it with the descriptor $x_i$ hard assignment to the cluster as a soft assignment and reuse the same soft assignment established in Equation (12) to obtain differentiable representation. We denote them as VLAD layer and BOW layer, respectively. The differentiable BOW representation and VLAD representation can be written as

$$BO{W}_k=\sum _{i=1}^N{a}_k\left(x_i\right),$$

(14)

$$VLA{D}_{j,k}=\sum _{i=1}^N{a}_k\left(x_i\right)\left(x_i\left(j\right)-c_k\left(j\right)\right),$$

(15)

where $a_k\left(x_i\right)$ denotes the membership of the descriptor $x_i$ to cluster k. BOW is the histogram of the number of image descriptors assigned to each visual word. Therefore, it produces a K-dimensional vector, while VLAD is a simplified non-probabilistic version of the FV and produces a $D\times K$-dimensional vector.

The soft assignment $a_k\left(x_i\right)$ can be regarded as a two-step process: (i) Perform a $1\times 1$ convolution with a set of K filters ${\omega }_k$ and bias $b_i$. Then, the output produced is ${\omega }_k^T{x}_i+b_k$. (ii) Follow a soft-max function to obtain soft assignment of the descriptor $x_i$ to the cluster k. Notably, for BOW encoding, there is no need to store the sum of residuals for each visual word, which is the difference vector between the descriptor and its corresponding cluster center.

The advantage of the BOW aggregation is that it aggregates the descriptor into a more compact representation, and fewer parameters are trained in a discriminative manner only including $\left\{{\omega }_k\right\}$ and $\left\{b_k\right\}$. The drawback is that significantly more clusters are needed to obtain a rich representation. The VLAD computes the first-order residuals between the descriptors and the cluster centers, making the richness of representation relatively sufficient, and parameters to be learned are moderate, including $\left\{{\omega }_k\right\}$, $\left\{b_k\right\}$ and $\left\{c_k\right\}$. In contrast, the FV aggregation concatenates both the first- and second-order aggregated residuals, but too many parameters need to be learned, including $\left\{{\omega }_k\right\}$, $\left\{b_k\right\}$, $\left\{c_k\right\}$ and $\left\{{\sigma }_k\right\}$.

As discussed Section 4.5, we also experimented with averaging and maximum pooling of the image descriptor-X. The results show that FV proves itself to be superior to the reference BOW and VLAD approach. Additionally, simply using averaging or maximum pooling results in poor performance.

3.4. Classification-Guided Gating Unit and Quality Prediction

We pre-trained Sub-Network I to identify the distortion type of the input, and Sub-Network II takes the estimated probability vector $\widehat{\mathbf{p}}$ from Sub-Network I as partial input. To introduce this prior information of the classification, a Classification-guided Gating Unit (CGU) is utilized to emphasize informative features and suppress less useful ones. The CGU combines $\widehat{\mathbf{p}}$ and $\mathbf{f}$ to produce a score vector $\widehat{\mathbf{f}}$

$$\widehat{\mathbf{f}}=\widehat{\mathbf{p}}·\sigma \left(W·{\mathbf{f}}_{\mathbf{Fisher}}+b\right),$$

(16)

where $\sigma$ is a linear mapping and $\left(W,b\right)$ are the learnable parameters. Then, a linear mapping is followed to yield an overall quality score q. To increase nonlinearity, two fully connected layers are applied as the linear mapping.

For Sub-Network II, the $L_1$ function is used as the empirical loss

$$\ell =\frac{1}{N}\sum _{i=1}^N∥q_i-\widehat{q_i}∥,$$

(17)

where $q_i$ is the MOS of the ith image in a mini-batch and $\widehat{q_i}$ is the predicted quality score by CGFA-CNN.

4. Experimental Results and Discussions

4.1. Database Description and Experimental Protocol

(1) IQA databases: These experiments were confirmed on three singly distorted synthetic IQA databases, namely LIVE [50], CSIQ [51], and TID2013 [52], and an authentic LIVE Challenge database [53]. LIVE contains five distortion types—JPEG compression (JPEG), JPEG-2000 compression (JP2K), White noise (GN), Gaussian blurring (GB), and Fast-fading error (FF)—at 7–8 degradation levels. CSIQ contains six distortion types—JPEG compression (JPEG), JPEG-2000 compression (JP2K), global contrast decrements (GC), additive pink Gaussian noise (PN), additive white Gaussian noise (WN), and Gaussian blurring (GB)—at 3–5 degradation levels. TID2013 contains 24 sceptic distortion types: additive Gaussian noise, additive noise in color components, spatially correlated noise, masked noise, high-frequency noise, impulse noise, quantization noise, Gaussian blur, image denoising, JPEG compression, JPEG2000 compression, JPEG transmission errors, non-eccentricity pattern errors, local bock-wise distortions, mean shift, contrast change, change of color saturation, multiplicative Gaussian noise, comfort noise, lossy compression of noisy images, color quantization with dither, chromatic aberrations, sparse sampling and reconstruction, which are denoted as #01–#24, respectively.

(2) Evaluation Criteria: Two evaluation criteria are adopted as follows to benchmark BIQA models:

• Spearman’s rank-order correlation coefficient (SRCC) is a nonparametric measure

  $$\mathrm{SRCC}=1-\frac{6{\sum }_i{d}_i^2}{I\left(I^2-1\right)},$$

  (18)

  where I is the test image number and $d_i$ is the rank difference between the MOS and the model prediction of the ith image.
• Pearson linear correlation coefficient (PLCC) is a nonparametric measure of the linear correlation

  $$\mathrm{PLCC}=\frac{{\sum }_i\left(q_i-q_m\right)\left({\widehat{q}}_i-{\widehat{q}}_m\right)}{\sqrt{{{\sum }_i\left(q_i-q_m\right)}^2}\sqrt{{{\sum }_i\left({\widehat{q}}_i-{\widehat{q}}_m\right)}^2}},$$

  (19)

  where $q_i$ and ${\widehat{q}}_i$ stand for the MOS and the model prediction of the ith image, respectively.

For synthetic databases LIVE, CSIQ and TID2013, we divided the distorted images into two splits of non-overlapping content—80% of which were used as fine-tuning samples and the other 20% were left as testing samples. For the LIVE Challenge database, the distorted images were divided into two groups—80% for training and 20% for testing. This random process was repeated ten times, and the average SRCC and PLCC are reported as the final results. Besides, the three synthetic databases were selected for cross-database experiments, using one database as the training sets while the other as the testing.

We compared the proposed CGFA-CNN against several state-of-the-art BIQA methods, including three based on NSS (BRISQUE [19], M3 [20], and FRIQUEE [21]), two based on manual feature learning (CORNIA [22] and HOSA [23]), and eight based on CNN (BIECON [30], dipIQ [31], deepIQA [29], ResNet50+ft [34], MEON [48], DIQA [26], TSCN [25], and DB-CNN [38]). Due to the source codes of some methods are not available to the public, we only copy the metrics from the corresponding papers.

4.2. Experimental Settings

Parameters in Sub-Network I were initialized by He’s method [54], and Adam was adopted as optimizer with the default parameters with a mini-batch of 64. The learning rate was initialized as a decay logarithmically from $\left[{10}^{-4},{10}^{-6}\right]$ in 30 epochs. The construction details of the pre-training dataset are described in Section 3.1. The datasets were randomly divided into two subsets, 80% for training and 20% for testing. All images were firstly scaled to $256\times 256\times 3$ and then cropped to $224\times 224\times 3$ as inputs. The top-1 and top-5 errors were 3.842% and 0.026%, respectively.

In the fine-tuning phase, the shared layers were directly initialized with the parameters of Sub-Network I. Adam was used as optimizer with the default parameters for 20 epochs and the learning rate was set to ${10}^{-5}$. Except for the LIVE database, images were input without any pre-processing during training with a mini-batch of 8. Since the LIVE database contains images in different size, images were randomly cropped to $320\times 320$ during training in a mini-batch, whose quality annotated were assigned from the corresponding image. All of the images were input without any preprocessing during testing. We implemented all of our models using PyTorch 0.4.1 deep learning framework and the numerical calculations presented in this paper were performed on the supercomputing system at the Supercomputing Center of Wuhan University. We will release the code at https://github.com/Cwp1107/CGFA-CNN.

4.3. Consistency Experiment

We investigated the effectiveness of CGFA-CNN on LIVE, TID2013, CSIQ and LIVE Challenge databases and the results are presented in

Table 2. Demographic prediction performance comparison by two evaluation metrics. The top three SRCC and PLCC results are highlighted in red, green, and blue, respectively.

Method	LIVE		CSIQ		TID2013		LIVE Challenge
	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC
BRISQUE [19]	0.940	0.945	0.777	0.817	0.573	0.651	0.603	0.641
M3 [20]	0.950	0.954	0.804	0.835	0.679	0.705	0.595	0.620
FRIQUEE [21]	0.948	0.955	0.844	0.889	0.668	0.705	0.694	0.710
CORNIA [22]	0.943	0.946	0.730	0.800	0.550	0.613	0.618	0.665
BIECON [30]	0.958	0.960	0.815	0.823	0.717	0.762	0.595	0.613
deepIQA [29]	0.960	0.972	—	—	0.803	0.821	0.671	0.680
ResNet50+ft [34]	0.950	0.954	0.876	0.905	0.712	0.756	0.819	0.849
DIQA [26]	0.975	0.977	0.884	0.915	0.825	0.850	0.703	0.704
TSCN [25]	0.969	0.972	—	—	—	—	—	—
DB-CNN [38]	0.968	0.971	0.946	0.959	0.816	0.865	0.851	0.869
CGFA-CNN	0.971	0.973	0.953	0.965	0.841	0.858	0.837	0.846

. The results of each specific distortion type on LIVE, CSIQ, and TID2013 databases are reported in

Table 3. Average SRCC and PLCC results of individual distortion types across ten sessions on LIVE database. The top three SRCC and PLCC results are highlighted in red, green, and blue, respectively.

SRCC	JPEG	JP2K	WN	GB	FF
BRISQUE [19]	0.965	0.929	0.982	0.964	0.828
M3 [20]	0.966	0.930	0.986	0.935	0.902
FRIQUEE [21]	0.947	0.919	0.983	0.937	0.884
CORNIA [22]	0.947	0.924	0.958	0.951	0.921
HOSA [23]	0.954	0.935	0.975	0.954	0.954
dipIQ [31]	0.969	0.956	0.975	0.940	—
DIQA [26]	0.961	0.976	0.988	0.962	0.912
TCSN [25]	0.966	0.950	0.979	0.963	0.911
DB-CNN [38]	0.972	0.955	0.980	0.935	0.930
CGFA-CNN	0.973	0.975	0.986	0.968	0.912
PLCC	JPEG	JP2K	WN	GB	FF
BRISQUE [19]	0.971	0.940	0.989	0.965	0.894
M3 [20]	0.977	0.945	0.992	0.947	0.920
FRIQUEE [21]	0.955	0.935	0.991	0.949	0.943
CORNIA [22]	0.962	0.944	0.974	0.961	0.943
HOSA [23]	0.967	0.949	0.983	0.967	0.967
dipIQ [31]	0.980	0.964	0.983	0.948	—
DIQA [26]	—	—	—	—	—
TCSN [25]	0.966	0.963	0.995	0.950	0.949
DB-CNN [38]	0.986	0.967	0.988	0.956	0.961
CGFA-CNN	0.972	0.976	0.981	0.974	0.947

,

Table 4. Average SRCC and PLCC results of individual distortion types across ten sessions on CSIQ database. The top three SRCC and PLCC results are highlighted in red, green, and blue, respectively.

SRCC	JPEG	JP2K	WN	GB	PN	CC
BRISQUE [19]	0.806	0.840	0.732	0.820	0.378	0.804
M3 [20]	0.740	0.911	0.741	0.868	0.663	0.770
FRIQUEE [21]	0.869	0.846	0.748	0.870	0.753	0.838
CORNIA [22]	0.513	0.831	0.664	0.836	0.493	0.462
HOSA [23]	0.733	0.818	0.604	0.841	0.500	0.716
dipIQ [31]	0.936	0.944	0.904	0.932	—	—
MEON [48]	0.948	0.898	0.951	0.918	—	—
DIQA [26]	0.835	0.931	0.927	0.893	0.870	0.718
DB-CNN [38]	0.940	0.953	0.948	0.947	0.940	0.870
CGFA-CNN	0.950	0.939	0.956	0.941	0.952	0.897
PLCC	JPEG	JP2K	WN	GB	PN	CC
BRISQUE [19]	0.828	0.887	0.742	0.891	0.496	0.835
M3 [20]	0.768	0.928	0.728	0.917	0.717	0.787
FRIQUEE [21]	0.885	0.883	0.778	0.905	0.769	0.864
CORNIA [22]	0.563	0.883	0.778	0.905	0.632	0.543
HOSA [23]	0.759	0.899	0.656	0.912	0.601	0.744
dipIQ [31]	0.975	0.959	0.927	0.958	—	—
MEON [48]	0.979	0.925	0.958	0.846	—	—
DIQA [26]	—	—	—	—	—	—
DB-CNN [38]	0.982	0.971	0.956	0.969	0.950	0.895
CGFA-CNN	0.972	0.953	0.969	0.955	0.942	0.893

and

Table 5. Average SRCC results of individual distortion types across ten sessions on TID2013 database. The top three SRCC results are highlighted in red, green, and blue, respectively.

Method	#01	#02	#03	#04	#05	#06	#07	#08	#09	#10	#11	#12
BRISQUE [19]	0.852	0.709	0.491	0.575	0.753	0.630	0.798	0.813	0.586	0.852	0.893	0.315
M3 [20]	0.748	0.591	0.769	0.491	0.875	0.693	0.833	0.878	0.721	0.823	0.872	0.400
FRIQUEE [21]	0.730	0.573	0.866	0.345	0.345	0.847	0.730	0.764	0.881	0.839	0.813	0.498
CORNIA [22]	0.756	0.750	0.7i 27	0.726	0.769	0.767	0.016	0.921	0.832	0.874	0.910	0.686
HOSA [23]	0.833	0.551	0.842	0.468	0.897	0.809	0.815	0.883	0.854	0.891	0.730	0.710
MEON [48]	0.813	0.722	0.926	0.728	0.911	0.901	0.888	0.887	0.797	0.860	0.891	0.746
DIQA [26]	0.915	0.755	0.878	0.734	0.939	0.843	0.858	0.920	0.788	0.892	0.912	0.861
DB-CNN [38]	0.790	0.700	0.826	0.646	0.879	0.708	0.825	0.859	0.865	0.894	0.916	0.772
CGFA-CNN	0.812	0.804	0.851	0.845	0.910	0.794	0.867	0.933	0.866	0.914	0.922	0.763
Method	#13	#14	#15	#16	#17	#18	#19	#20	#21	#22	#23	#24
BRISQUE [19]	0.359	0.145	0.224	0.124	0.040	0.109	0.724	0.008	0.685	0.764	0.616	0.784
M3 [20]	0.731	0.190	0.318	0.119	0.224	−0.121	0.701	0.202	0.664	0.886	0.648	0.915
FRIQUEE [21]	0.660	0.076	0.032	0.254	0.585	0.589	0.704	0.318	0.641	0.768	0.737	0.891
CORNIA [22]	0.805	0.286	0.219	0.065	0.182	0.081	0.644	0.534	0.862	0.272	0.792	0.862
MEON [48]	0.716	0.116	0.500	0.177	0.252	0.684	0.849	0.406	0.772	0.857	0.779	0.855
DIQA [26]	0.812	0.659	0.407	0.299	0.687	−0.151	0.904	0.655	0.930	0.936	0.756	0.909
DB-CNN [38]	0.773	0.270	0.444	0.646	0.548	0.631	0.711	0.752	0.860	0.833	0.732	0.902
CGFA-CNN	0.757	0.335	0.649	0.441	0.573	0.657	0.819	0.785	0.897	0.940	0.711	0.938

. The top three SRCC and PLCC results are highlighted in red, green, and blue, respectively.

Based on the results in Table 2, we have the following observations. First, DIQA [26] achieves state-of-the-art accuracies which surpasses CGFA-CNN by about 0.004 in SRCC and PLCC, and most methods take great advantages in indexes on LIVE. However, their results on CSIQ and TID2013 are rather diverse. Second, CGFA-CNN achieves comparable accuracies on LIVE Challenge compared with DB-CNN [38] and ResNet50+ft [34], which are pre-trained on ImageNet [28] databases. This suggests that CNNs pre-trained on ImageNet [28] could extract relevant features for authentically distorted images.

Performance on individual distortion types on LIVE, CSIQ, and TID2013 are shown in Table 3, Table 4 and Table 5. On LIVE, we also find that CGFA-CNN is superior to other methods in most distortions, except Fast-fading error, which is not introduced into the pre-training dataset because there is no open-source or detailed description of it. On CSIQ, CGFA-CNN has obvious advantages compared with other methods, especially in contrast change and pink noise. On TID2013, CGFA-CNN achieves state-of-the-art performance in 10 of the 24 distortions and the whole effect standout accuracies other methods. In addition, we find that CGFA-CNN performs well when the distortion shares similar artifacts with the distortion synthesized in the pre-training dataset. For example, additive Gaussian noise, additive noise in color components, and high-frequency noise are all grainy noise; quantization noise and image color quantization with dither exhibit similar appearances; and Gaussian blur, image denoising, and sparse sampling and reconstruction all introduce blur effects on the image. Therefore, although the pre-training dataset constructed in this paper does not cover all distortion types, CGFA-CNN still achieves impressive gains in performance.

4.4. Cross-Database Experiment

To analyze the generalization ability of the proposed method, we trained CGFA-CNN on one full database and evaluated it on another database. Specifically, a model was trained on CSIQ and evaluated on either LIVE or TID2013. The results are reported in

Table 6. SRCC comparison on cross-database. The top three SRCC results are highlighted in red, green, and blue, respectively.

Method	CSIQ		TID2013
	LIVE	TID2013	LIVE	CSIQ
BRISQUE [19]	0.847	0.454	0.790	0.590
M3 [20]	0.797	0.328	0.873	0.605
FRIQUEE [21]	0.879	0.463	0.755	0.635
CORNIA [22]	0.853	0.312	0.846	0.672
HOSA [23]	0.773	0.329	0.594	0.462
DB-CNN [38]	0.877	0.540	0.891	0.807
CGFA-CNN	0.891	0.533	0.898	0.774

. It can be concluded that CGFA-CNN can easily be generalized to distortions that have not been seen during training.

4.5. Comparison among Different Experimental Settings

In this section, we first work with the performance of different feature aggregation layers investigated in this paper and number of GMM components K. Experiments were conducted on LIVE and the results are shown in Figure 4. We observe that SRCC gradually increases and eventually keeps stability as K increases. Besides, CGFA-CNN FV, CGFA-CNN VLAD, and CGFA-CNN BOW attain highly competitive prediction accuracy when K is set to 32, 64 and 1024, respectively. By contrast, CGFA-CNN FV is superior to CGFA-CNN VLAD and CGFA-CNN BOW.

Additionally, we report ablation studies to evaluate the design rationality of CGFA-CNN and the following comparative set of experiments were conducted: (1) to evaluate the effectiveness of the proposed FV layer, we used the maximum pooling (denoted as CGFA-CNN (MaxPool)) and average pooling (denoted as CGFA-CNN (AvgPool)) instead; (2) to examine the validity of the CGU described in this work, we predicted the quality score directly by regressing the output feature vector without CGU (denoted as CGFA-CNN (w/o CGU)); (3) to verify the necessity of the hierarchical feature extraction, we extracted features only from high-level (Conv 5-2 of shared layers and Conv 4-3 of VGG-16) convolutional layers as descriptors (denoted as CGFA-CNN (single feature)); (4) to discuss the optimal settings of for feature aggregation layer, we set the BOW with $K=1024$ (denoted as CGFA-CNN (BOW layer ($K=1024$))), VLAD with $K=64$ (denoted as CGFA-CNN (VLAD layer ($K=1024$))), and FV with $K=32$ (denoted as CGFA-CNN (proposed)); and (5) to demonstrate the prediction accuracies on authentic distortions by involving VGG-16, we only included Sub-Network I pre-trained on self-built dataset to extract features (denoted as CGFA-CNN (w/o VGG-16)). The results are demonstrated in

Table 7. SRCC with different settings.

Method	LIVE	CSIQ	TID2013	LIVE Challenge
CGFA-CNN (MaxPool)	0.915	0.893	0.778	0.766
CGFA-CNN (AvgPool)	0.909	0.876	0.755	0.761
CGFA-CNN (w/o CGU)	0.948	0.919	0.783	0.799
CGFA-CNN (single feature)	0.931	0.890	0.757	0.765
CGFA-CNN (BOW layer (K = 1024))	0.955	0.936	0.808	0.791
CGFA-CNN (VLAD layer (K = 64))	0.966	0.945	0.819	0.810
CGFA-CNN (w/o VGG-16)	0.970	0.950	0.836	0.672
CGFA-CNN (proposed)	0.973	0.953	0.841	0.837

. We empirically found that the proposed CGFA-CNN could achieve state-of-the-art prediction accuracies on both synthetic and authentic distortion image quality databases. Besides, CGFA-CNN (w/o VGG-16) can only deliver promising performance on synthetic databases and its results on LIVE Challenge are inferior to CGFA-CNN (proposed), suggesting that authentic distortions cannot be fully fitted by synthetic distortions.

5. Conclusions

In this work, we propose an end-to-end learning framework for BIQA based on classification guidance and feature aggregation, which is named as CGFA-CNN. In the fine-tuning phase, except for the shared convolutional layers, the rest of Sub-Network I only participates in the forward propagation, and the parameters are fixed. The fused feature group is aggregated and encoded by the FV layer to obtain a fisher vector. Then, the fisher vector is corrected by the CGU to obtain a quality-ware feature, which is mapped to a quality score by the regression model. In the test phase, only forward propagation is required to obtain the quality score. The results on the four publicly IQA databases demonstrate that the proposed method indeed benefited image quality assessment. However, CGFA-CNN is not a unified learning framework because it takes two steps to pre-train and fine-tune. The promising future direction is to optimize CGFA-CNN for both distortion identification and quality prediction at the same time. For example, we think that the autoencoder method could be designed to perform k-mean clustering. A VAE framework can be introduced to decode. This approach can replace the two-stage procedure. We also look forward to designing a potential objective function that could in principle reduce the necessity to rely on external procedures.

CGFA-CNN is versatile and extensible. For example, more distortion types and levels can be added to the pre-training dataset, and it could fuse with other approaches to achieve a new backbone network.