Grouping Bilinear Pooling for Fine-Grained Image Classification

Abstract

Fine-grained image classification is a challenging computer visual task due to the small interclass variations and large intra-class variations. Extracting expressive feature representation is an effective way to improve the accuracy of fine-grained image classification. Bilinear pooling is a simple and effective high-order feature interaction method. Compared with common pooling methods, bilinear pooling can obtain better feature representation by capturing complex associations between high-order features. However, the dimensions of bilinear representation are often up to hundreds of thousands or even millions. In order to get compact bilinear representation, we propose grouping bilinear pooling (GBP) for fine-grained image classification in this paper. Firstly, by dividing the feature layers into different groups, and then carrying out intra-group bilinear pooling or inter-group bilinear pooling. The representation captured by GBP can achieve the same accuracy with less than 0.4% parameters compared with full bilinear representation when using the same backbone. This extreme compact representation largely overcomes the high redundancy of the full bilinear representation, the computational cost and storage consumption. Besides, it is because GBP compresses the bilinear representation to the extreme that it can be used with more powerful backbones as a plug-and-play module. The effectiveness of GBP is proved by experiments on the widely used fine-grained recognition datasets CUB and Stanford Cars.

1. Introduction

Computer vision is an important field of artificial intelligence, and its goal is to realize the extension of human vision function based on intelligent computing. Image classification is the most basic task in computer vision. In general, Image classification can be divided into two categories: coarse-grained image classification and fine-grained image classification. Recognizing images of different classes (cats, dogs, flowers, planes, etc.) is the goal of coarse-grained image classification, and fine-grained visual classification aims at classifying sub-classes of category [1,2]. Fine-grained visual classification is a challenging visual task due to the small interclass variations and large intra-class variations. Obtaining more discriminative feature representation is the focus of fine-grained image classification. Convolutional Neural Network (CNN) has been widely used in computer vision tasks [3,4,5]. By stacking different functional layers, CNN can obtain powerful feature extraction and generalization capabilities. Extracting expressive feature representation is an effective way to improve the accuracy of fine-grained image classification; many enlightening and meaningful works are proposed to make better use of the extracted CNN semantic features.

Most studies obtain the feature representation of an input image by pooling high-order features [6,7], extracting high-value information through attention methods [8,9] or aggregating the features of different levels [10], then use these expressive representation to subsequent tasks. In addition, some other studies use the high-order statistical information of features to obtain better feature representation, such as Fisher Vector [11,12], VLAD [13], spatial pyramids [14] and full bilinear pooling [15,16]. Among them, the full bilinear pooling (FBP) [15] captures the complex correlation between paired features and uses the bilinear representation for classification which makes remarkable achievements in fine-grained image classification. However, the bilinear representation is as high as hundreds of thousands or even millions of dimensions. It is far higher than the aggregating representation obtained by common pooling methods, resulting in huge computational load and memory consumption, and limiting the expansion of the model structure.

In order to reduce the dimension of the bilinear representation and obtain compact representation, refs. [16,17,18,19,20] simplify the bilinear pooling in different ways. In [17], the image classification network based on bilinear pooling is regarded as a linear kernel machine, and it is proved that bilinear pooling enables the linear classifier to have the recognition ability of second-order kernel machine. Then, Random Maclaurin (RM) [21] and Tenor Sketch (TS) [22] are used for low-dimensional approximation, and a compact bilinear pooling (CBP) is established. Ref. [18] uses the Hadamard product instead of bilinear pooling, the Hadamard product also achieves a low-dimensional approximation to bilinear pooling. Refs. [16,19] reduces the dimensions of representation by carrying out feature mapping on high-order feature layers or simplifying bilinear pooling by matrix factorization to get compact representation. The learnable semantic grouping module is introduced to reduce the computation of bilinear pooling in [20].

Besides, some papers focus on what else compact bilinear pooling can do [23,24,25,26]. Ref. [23] uses multimodal compact bilinear pooling in visual question answering which performs well. In [24], bilinear pooling is used to combine global features and local features for person re-identification. Ref. [25] aggregates local CNN features by carrying out bilinear pooling to obtain 3D object representation and achieves remarkable results in 3D object recognition tasks. In [26], bilinear blocks are proposed to obtain rich modality-temporal representation for RGB-D Action Recognition.

Bilinear pooling has potential in different visual tasks. Inspired by various compact bilinear pooling methods, we want to simplify bilinear pooling in a simple way and compress the bilinear representation to the extreme. We first re-analyze bilinear pooling in Section 2, it is shown intuitively that the bilinear representation is actually a low-rank self-correlation and cross-correlation representation. Then, the reason for high redundancy of bilinear representation is analyzed. To compress the bilinear representation to the extreme, we propose grouping bilinear pooling (GBP) to minimize the dimensions of bilinear representation. Comparing with different compact bilinear pooling methods, GBP reaches the best accuracy with the fewest parameters. In addition, GBP can be embedded into different models as a plug-and-play module and can achieve a competitive performance compared with other state-of-the-art approaches. Experiments on the CUB-200-2011 [1] dataset and the Stanford Cars [2] dataset show the effectiveness of GBP.

2. Analysis of Bilinear Pooling

In BCNN [15], in order to get the full bilinear representation Z of the input image, CNN is used to extract the higher-order feature representation X of the input image, $X\in {\mathbb{R}}^{C\times H\times W}$, where C is the number of feature layers, H and W are the height and the width of feature layers, respectively. For each feature layer, there are $H\times W$ different locations. The architecture of BCNN is illustrated in Figure 1.

Here, we define local descriptor $x_i^T=\left[x_i^1,x_i^2,\dots ,x_i^C\right]\in {\mathbb{R}}^C(i\in [1,\dots ,HW])$, $x_i^n$ represents the value of ith position on the $nth$ feature layer. Besides, for convenience, considering $x_i$ as a vector the dimension of which is C. The bilinear representation Z is as follows:

$$Z=\frac{1}{HW}\sum _i^{HW}{x}_i{x}_i^T=\frac{1}{HW}\left[\begin{array}{ccc}\sum _i^{HW}{x}_i^1{x}_i^1& \dots & \sum _i^{HW}{x}_i^1{x}_i^C\\ & \dots & \\ \sum _i^{HW}{x}_i^1{x}_i^C& \dots & \sum _i^{HW}{x}_i^C{x}_i^C.\end{array}\right]$$

(1)

Vectorizing Z, vector$\left(Z\right)\in {\mathbb{R}}^{C^2}$. Assuming C is 512, vector$\left(Z\right)$ will be up to 250 K dimensions. The high dimensions of bilinear representation result in high computation and storage costs.

The representation Z is used for classification after passing through the full-connection layer,

$$Output=Z{W}_C+b=\frac{1}{HW}\left(\sum _i^{HW}{x}_i{x}_i^T\right)W_C+b_C,$$

(2)

where $W_C\in {\mathbb{R}}^{C^2\times N}$ is the weight matrix of the full-connection layer, $b_C\in {\mathbb{R}}^k$, $Output$$\in {\mathbb{R}}^N$, N is the number of categories. In general, $C^2\gg N$, the rank of $W_C$ is as follows:

$$rank\left(W_C\right)\le min\left(C^2,N\right)=N.$$

(3)

Vectorizing the ith feature layer: $f_i^T=\left[x_1^i,x_2^i,\dots ,x_{HW}^i\right]\in {\mathbb{R}}^{HW}$, $F^T=\left[f_1,f_2,\dots ,f_C\right]\in {\mathbb{R}}^C$,

$$Z=\frac{1}{HW}\left[\begin{array}{ccc}{f}_1^T{f}_1& \dots & f_1^T{f}_c\\ & \dots & \\ f_1^T{f}_c& \dots & f_c^T{f}_c\end{array}\right]=\frac{1}{HW}F{F}^T.$$

(4)

$$Output=Z{W}_C+b=\frac{1}{HW}\left(F{F}^T\right)W_C+b_C.$$

(5)

In [17], RM [21] is used to sample the feature layers, then bilinear pooling is carried out. In fact, the representation obtained by [17] is the recombination of part of the element in Z. The representation obtained by [18], using the low-dimensional approximation of Hadamard product, is the elements on the diagonal of Z. This low-rank approximation actually abandons the vast majority of information of Z. The loss of information is inevitable, although the dimensions of representation are reduced.

X contains C different feature layers f; the bilinear representation Z in Equation (1) is a symmetric matrix. The elements on the diagonal of Z are the dot product sum of the corresponding positions of the feature layer itself. The scalar obtained by the point-wise product can be regarded as the pixel-level self-correlation of the feature layer to some extent. Similarly, the elements on the non-diagonal of Z are the dot product sum of the corresponding positions of different feature layers, which can be regarded as the cross-correlation between feature layers.

Since the bilinear representation obtained by bilinear pooling is a correlation representation with extremely high dimensions between high-order feature layers (increasing with the square of the number of feature layers), it will greatly increase the parameters to be learned by the full-connection layers even if there is only a single-layer full-connection layer. Furthermore, nearly half of the calculations in the symmetric matrices Z are repeated; obviously, it greatly reduces computational efficiency and results in redundancy of the model.

3. Grouping Bilinear Pooling

According to Equation (3), the dimension of Z is often up to hundreds of thousands or even millions, far more than the dimension of $Output$, hoping that the full-connection layer will establish high-efficient contact and find high value information between Z and $Output$ is impractical. Minimizing the huge gap of dimensions between Z and $Output$ is a highly cost-effective way to improve bilinear pooling.

We propose grouping bilinear pooling (GBP) that by grouping the feature layers X and performing intra-group bilinear pooling (Intra-GBP) or inter-group bilinear pooling (Inter-GBP), the information of the original full bilinear representation can be greatly preserved, and extreme compact bilinear feature representation is available. The differences between GBP with other bilinear pooling methods [15,17,19] are shown in Figure 2, and detailed comparisons are described in the experimental section.

3.1. Intra-Group Bilinear Pooling

Define F is the high-order feature, $F^T=\left[f_1,f_2,\dots ,f_C\right],F_k\in {\mathbb{R}}^{\frac{C}{g}}$, F contains C different feature layers. Then, dividing F into g groups. Noting that the maximun of g is C while carrying out Intra-GBP. $G=\left[G_1,G_2,\dots ,G_{C/g}\right]\in {\mathbb{R}}^{\frac{C}{g}}$, for the kth group $G_k^T=\left[f_{g(k-1)+1},f_{g(k-1)+2},\dots ,f_{g(k-1)+C/g}\right],G_k\in {\mathbb{R}}^{\frac{C}{g}}$, Intra-GBP is as follows:

$$Z_k=\frac{1}{HW}{G}_k{G}_k^T$$

(6)

$$Z_G=Concat\left(Z_1,Z_2,\dots ,Z_k\right)$$

(7)

$$Output=Z_G{W}_G+b_G$$

(8)

$$rank\left(W_G\right)\le min\left(\frac{C^2}{g},N\right)$$

(9)

where $vector\left(Z_k\right)\in {\mathbb{R}}^{{\left(\frac{C}{g}\right)}^2},k\in [1,2,\dots ,g],Z_G\in {\mathbb{R}}^{\frac{C^2}{g}},W_G\in {\mathbb{R}}^{\frac{C^2}{g}\times N},b_G\in {\mathbb{R}}^N$.

For FBP, when the full bilinear representation Z is used for classification, the parameters that the full-connection layer needs to learn are as high as $C^{2}N$. While for the intra-group bilinear pooling (Intra-GBP), the parameters that the full-connection layer needs to learn reduce to $C^{2}N/g$.

When g is small, this bilinear operation after grouping still requires large computational resources (Intra-GBP at $g=1$ is equivalent to FBP), and the dimension of Intra-GBP is still a larger number. As g gets bigger and bigger, it will bring huge benefits. The influence of changing g will be explained in the experiment section.

The representation $Z_k$ in Equation (6) has similar properties to the representation Z obtained by FBP, that is, $Z_k$ is also a symmetric matrix and there is still computational redundancy. Thus, we propose inter-group bilinear pooling for further improvement.

3.2. Inter-Group Bilinear Pooling

The same as with Intra-GBP, dividing F into g groups, $G^{\prime }=\left[G_1,G_2,\dots ,G_{C/g}\right]\in {\mathbb{R}}^{\frac{C}{g}}$. The difference is that bilinear pooling is carried out between two different grouped feature layer groups $G_a$ and $G_b$ ($G_a,G_b\in G{}^{\prime },G_a\ne G_b$) in Inter-GBP. $G_a$ and $G_b$ are from grouped g groups, and each group is selected only once. Inter-GBP is as follows:

$$Z_k{}^{\prime }=\frac{1}{HW}{G}_a{G}_b^T$$

(10)

$$Z_G{}^{\prime }=Concat\left(Z_1{}^{\prime },Z_2{}^{\prime },\dots ,Z_k{}^{\prime }\right)$$

(11)

$$Output{}^{\prime }=Z_G{}^{\prime }{W}_G{}^{\prime }+b_G{}^{\prime }$$

(12)

$$rank\left(W_G{}^{\prime }\right)\le min\left(\frac{C^2}{2g},N\right)$$

(13)

where $vector\left(Z_k{}^{\prime }\right)\in {\mathbb{R}}^{{\left(\frac{C}{g}\right)}^2},k\in [1,2,\dots ,g/2],Z_G{}^{\prime }\in {\mathbb{R}}^{\frac{C^2}{2g}},W_G{}^{\prime }\in {\mathbb{R}}^{\frac{C^2}{2g}\times N},b_G{}^{\prime }\in {\mathbb{R}}^N$.

Similar to Intra-GBP, Inter-GBP will yield huge benefits when g is large enough. Besides, g should be a multiple of 2 due to the specific group selection method of Inter-GBP, the maximum and minimum of g are $C/2$ and 2. The full connection layer needs to learn the parameters: $C^{2}N/2g$.

In experiments, the selection method of $G_a$ and $G_b$ does not affect the experimental results. So, to simplify the operation of Inter-GBP, feature layers are grouped in order, and Inter-GBP selects adjacent feature layer groups sequentially for bilinear pooling.

4. Experiment

4.1. Datasets, Backbone and Experiment Configurations

4.1.1. Datasets

We conduct experiments on two widely used fine-grained image classification datasets: CUB [1] and Stanford Cars [2]. In all experiments, only category labels of images and images themselves were used for end-to-end model training. The details of the CUB dataset and the Stanford Cars dataset are shown in

Table 1. Dataset Details.

Dataset	Training	Testing	Category
CUB [1]	5994	5794	200
Stanford Cars [2]	8144	8041	196

.

4.1.2. Backbone

In order to compare with compact bilinear pooling methods and the state-of-the-art approaches using different methods, we use VGG-16 [27], ResNet-50 [28], ResNet-101 [28] and ResNet-152 [28] pretrained on the ImageNet [29] image classification dataset as our backbone networks respectively (the feature extraction networks are retained, removing the final pooling layers and full connection layers, using the GBP pooling layer and the new full-connection layer instead).

4.1.3. Experimental Configurations

The experiments were carried out on the server of ubuntu system, the code was written by Python and PyTorch [30] deep learning framework. Four NVIDIA GTX $1080Ti$ GPUs were used for distributed model training and testing. The size of the input image is $448\times 448$ and the data augmentation follows the commonly used methods. During the training, the pre-training weight of the model on ImageNet [29] was first loaded and frozen, and the parameters of the full-connection layer between GBP representation and outputs were fine-tuned. During the fine-tuning, the initial learning rate was 0.0003, and the Adam optimizer [31] was adopted to dynamically adjust the learning rate with factor = 0.15, patience = 2, cooldown = 4. After fine-tuning, all frozen parameters were unfreezing, then the learning rate was adjusted to 0.0001. The loss function is multi-classification cross entropy loss function. In all experiments, the same experimental configurations were followed.

4.2. Evaluation

First, convolutional feature extraction network of VGG-16 [27] is used as the backbone network to perform Intra-GBP and Inter-GBP on CUB [1] dataset and Stanford Cars [2] dataset respectively. Then, after grouping bilinear pooling, the obtained representation was used to classify directly by the full-connection layer. The high-order feature layer X extracted from VGG-16 has 512 feature channels, which was divided into g groups, $g\in [1,2,4,8,16,32,64,128,256]$; noting that the Intra-GBP degenerates to FBP [15] at $g=1$, and the minimum of g is 2 in Inter-GBP.

The original FBP achieved an accuracy of 84.01% on the CUB dataset, and our reimplementation achieved an accuracy of 83.43%. The experiment results of Intra-GBP and Inter-GBP on the CUB dataset are shown in Figure 3 and Figure 4. With the increasing of $g$, the accuracy of intra-GBP and Inter-GBP generally increases first and then decreases, the size of the model and the amount of calculation for pooling and classification show a decreasing trend. Intra-GBP achieved the best accuracy of 83.64% at $g=32$ on CUB and the best accuracy of 91.42% on Stanford Cars; Inter-GBP achieved the best accuracy of 83.66% at $g=64$ on CUB and the best accuracy of 92.49% on Stanford Cars.

In general, both Intra-GBP and Inter-GBP can effectively reduce the dimension of full bilinear representation, and the model based on the two methods can reduce the size of the model and the amount of calculation without causing a loss of model performance within a certain number of $g$. As discussed, comparing with the Intra-GBP which has inherent redundancy, Inter-GBP can obtain feature representation with less redundancy. In experiments with different backbone networks and datasets, Inter-GBP always performs better. Therefore, in the subsequent experiments, we mainly show the performance of Inter-GBP unless specifically stated.

In order to further verify the performance and effectiveness of GBP with different backbones, we used ResNet-50, ResNet-101 and ResNet-152 as the backbone to perform GBP respectively. With a more powerful backbone, GBP performs better.

Table 2. The performances of Inter-GBP based on ResNet-50 on CUB dataset and Stanford Cars dataset.

The Groups	2	4	8	16	32	64	128	256	512	1024
CUB (%)	83.79	84.19	85.13	85.28	85.23	85.11	85.07	85.21	85.32	85.54
Model size (MB)	987.81	497.81	297.81	197.81	147.81	122.81	110.31	104.06	100.94	99.37
Stanford Cars (%)	92.11	92.34	92.35	92.29	92.33	92.49	92.61	92.75	92.74	92.86
Model size (MB)	971.81	489.81	293.81	195.81	146.81	122.31	110.06	103.94	100.87	99.34

shows the performances of Inter-GBP based on ResNet-50 on CUB and Stanford Cars.

Due to the number of high-order features, the layer is 2048(C = 2048); when using ResNet-50, the range of $g$ goes up, $g\in [2,4,8,16,32,64,128,256,512,1024]$. As can be seen from Table 2, the accuracy of Inter-GBP keeps improving as the number of $g$ keeps increasing. When g = 1024, the accuracy of Inter-GBP on CUB and Stanford Cars reaches the highest accuracy of 85.54% and 92.86% respectively, which are 1.5 and 0.4 percentage points higher than Inter-GBP based on VGG-16. When g goes from 2 to 1024, the model size of Inter-GBP goes from 987.81 MB and 971.81 MB to 99.37 MB and 99.34 MB on CUB and Stanford Cars, respectively.

Noting that the accuracies reach the highest at g = 1024 (maximum number of grouping). It is because feature information extracted from a deep network is more stable compared with a shallow network, and the Inter-GBP representation is sufficiently robust when the maximum number of grouping is used. In all experiments with ResNet as the backbone, more groupings often represent a better performance.

4.3. Comparing with Other Compact Bilinear Pooling

In this section, GBP is compared with other compact bilinear pooling methods [15,17,19,32] in detail. Assuming that the categories to be classified are N, and bilinear pooling is carried out on the feature layers with the size of $c\times h\times w$, where c is the feature channels, the height and width of feature layers are h and w (VGG-16: h = w = 28, c = 512; ResNet-50: h = w = 14, c = 2048). For a more intuitive comparison, taking the experiment on CUB as an example, the input size of the images is $448\times 448$. The configurations of the comparative experiment are as follows: g = 128 (VGG-16), g = 1024 (ResNet-50), m = 100, r = 8, d = 8192, N = 200.

The detailed comparisons of different compact bilinear pooling methods are shown in

Table 3. Comparison of different compact bilinear pooling methods. We used VGG-16 and ResNet-50 as the backbone respectively to compare the computational complexity, representation dimensions and the parameters needed to be learned (excluding the backbone network).

Backbone	Method	Dimension	Computing			Parameters
			Pooling	Classifying	Total	Projection	Classifier	Total
VGG-16	FBP [15]	$c^2$ [262 K]	$\mathrm{O}\left({hwc}^2\right)$	$\mathrm{O}\left({Nc}^2\right)$	257,949,696	0	${Nc}^2$	200 MB
	iFBP [32]	$c^2$ [262 K]	$\mathrm{O}\left({hwc}^2\right)$	$\mathrm{O}\left({Nc}^2\right)$	257,949,696	0	${Nc}^2$	200 MB
	CBP-TS [17]	d [10 K]	$\mathrm{O}\left(hw\right(c+dlogd\left)\right)$	$\mathrm{O}\left(Nd\right)$	85,532,672	$2c$	$Nd$	8 MB
	CBP-RM [17]	d [10 K]	$\mathrm{O}\left(hwcd\right)$	$\mathrm{O}\left(Nd\right)$	3,289,972,736	$2cd$	$Nd$	48 MB
	LRBP-I [19]	$mhw$ [78 K]	$\mathrm{O}\left(hwcm\right)$	$\mathrm{O}\left(Nrmhw\right)$	165,580,800	$cm$	$Nrm$	$0.8$ MB
	LRBP-II [19]	$m^2$ [10 K]	$\mathrm{O}\left(hw(cm+m^2)\right)$	$\mathrm{O}\left(Nr{m}^2\right)$	63,980,800	$cm$	$Nrm$	$0.8$ MB
	Intra-GBP (ours)	$c^2/g$ [2 K]	$\mathrm{O}({hwc}^2/g^2)$	$\mathrm{O}({Nc}^2/g)$	422,144	0	${Nc}^2/g$	$1.6$ MB
	Inter-GBP $\left(ours\right)$	$c^2/2g$ [1 K]	$\mathrm{O}({hwc}^2/{2g}^2)$	$\mathrm{O}({Nc}^2/2g)$	211,072	0	${Nc}^2/2g$	$0.8$ MB
ResNet-50	FBP [15]	$c^2$ [4194 K]	$\mathrm{O}\left({hwc}^2\right)$	$\mathrm{O}\left({Nc}^2\right)$	1,660,944,384	0	${Nc}^2$	3200 MB
	Intra-GBP $\left(ours\right)$	$c^2/g$ [4 K]	$\mathrm{O}({hwc}^2/g^2)$	$\mathrm{O}({Nc}^2/g)$	819,984	0	${Nc}^2/g$	$3.2$ MB
	Inter-GBP $\left(ours\right)$	$c^2/2g$ [2 K]	$\mathrm{O}({hwc}^2/{2g}^2)$	$\mathrm{O}({Nc}^2/2g)$	409,992	0	${Nc}^2/2g$	$1.6$ MB

. The comparison contents include the dimensions of representation, computational complexity of pooling and classifying, and the number of parameters. Under the same configurations, the classification accuracy of different compact bilinear pooling methods based on VGG-16 is shown in

Table 4. The performances of different compact bilinear pooling methods on CUB dataset and Stanford Cars dataset. (Based on VGG-16).

Method	FBP [15]	iFBP [32]	CBP-TS [17]	CBP-RM [17]	LRBP [19]	Intra-GBP	Inter-GBP
CUB (%)	84.01	85.80	84.00	83.86	84.21	83.64	83.66
Cars (%)	91.18	92.10	90.19	89.54	90.92	91.42	92.49

.

As can be seen from Table 3, among all bilinear pooling methods, GBP has the lowest representation dimension. When VGG-16 is used as a backbone, the representation dimension of Inter-GBP is only 0.4% of FBP, the pooling computation is reduced by four orders of magnitude, and the parameters needed to learn are reduced by 99.6%. When ResNet-50 is used as a backbone, the representation dimension of FBP will up to 4.19 million and the parameters needed to learn will reach 3200 MB, while Inter-GBP are 2048 and 1.6 MB respectively. It can be seen intuitively that GBP has a huge advantage in the compression of representation.

Table 4 shows that the performance of GBP is not weaker than or even better than other compact bilinear pooling methods when using the same backbone. The Inter-GBP based on VGG-16 reaches the best accuracy of 92.49% on the Stanford Cars dataset. As an earlier compact bilinear pooling method [17], CBP improves bilinear pooling to some extent and inspires a series of subsequent approaches. iFBP [32] proposes the matrix square-root normalization to improve the performance of the bilinear model, and achieves the best performance on the CUB dataset, but it still requires a lot of computing and storage space. LRBP [19] obtains statistics without performing bilinear pooling, instead using the Frobenius norm as the classification score, which reduces the computation to a certain extent and has advantages in reducing parameters, but it requires the learning of additional projection parameters. Compared with [19], although the performance of GBP based on VGG-16 on the CUB dataset is lower, the pooling computation of GBP is reduced by two orders of magnitude, and the dimension of feature representation obtained by GBP is lower.

4.4. Comparison with the State-of-the-Art

Generally, it is hard to balance accuracy and the complexity of the model when bilinear pooling is applied. With the extreme compression, GBP is able to use more powerful backbones to improve performance. We embedded GBP in different backbones and compared it with other methods. The performances of baselines, full bilinear pooling based methods, compact bilinear pooling based methods, GBP (ours) methods and other state-of-the-art methods relating to channels are shown in

Table 5. The performances of different methods on CUB dataset and Stanford Cars dataset. From top to bottom, the five blocks respectively list the baselines, full bilinear pooling based methods, compact bilinear pooling based methods, other state-of-the-art methods relating to channels and our method.

Method	Backbone	Dimension	Parameters	CUB (%)	Stanford Cars (%)
VGG-16 [27]	-	25 K	20 MB	74.59	85.05
ResNet-50 [28]	-	2 K	1.6 MB	82.15	92.19
ResNet-101 [28]	-	2 K	1.6 MB	82.58	92.56
ResNet-152 [28]	-	2 K	1.6 MB	82.74	92.64
FBP [15]	VGG-16	260 K	200 MB	84.01	91.18
iFBP [32]				85.80	92.10
MoNet-FBP [33]				86.40	91.80
CBP [17]	VGG-16	10 K	8 MB	84.00	90.19
LRBP [19]		10 K	0.8 MB	84.21	90.90
MoNet-TS [33]		10 K	8 MB	85.70	90.80
FBC [34]		8 K	6.4 MB	84.30	-
SBP-EN [35]		10 K	8 MB	84.50	90.90
SWP [36]	VGG-16	-	-	-	90.70
	ResNet-50	-	-	-	92.30
	ResNet-101	-	-	-	93.10
HBPASM [37]	ResNet-34	-	-	86.80	92.80
HBP [16]	VGG-16	24 K	19 MB	87.01	93.70
SEF [38]	VGG-16	-	-	81.10	88.30
	ResNet-50	-	-	87.30	94.00
MC-loss [39]	ResNet-50	-	-	87.30	93.70
Inter-GBP	VGG-16	1 K	0.8 MB	83.66	92.49
	ResNet-50	2 K	1.6 MB	85.54	92.86
	ResNet-101	2 K	1.6 MB	86.10	93.76
	ResNet-152	2K	1.6 MB	86.31	94.22

.

Compared with VGG-16, ResNet-50, ResNet-101 and ResNet-152, Inter-GBP improved the accuracy of the CUB dataset by 9, 3.4, 3.5 and 3.6 percentage points respectively after being applied to these backbone networks, and also improved the accuracy of the Stanford Cars dataset by 7.4, 0.7, 1.2 and 1.6 percentage points, respectively. Comparing with the methods based on bilinear pooling and compact bilinear pooling, GBP is the most compact method and achieves the best performance. Especially on the Stanford Cars dataset, Inter-GBP improves the best performance of compact bilinear pooling from 91.80% to 94.22%.

In addition, due to lack of simplicity and convenience, few papers apply compact bilinear pooling approaches to high-performing backbones. We also tried to perform other compact BP on better backbones, but the result is not good enough. GBP compresses the bilinear representation to the extreme so that it can be used with more powerful backbones to achieve competitive performance.

The grouping operation of GBP is performed at the channel level. Compared with other state-of-the-art methods relating to channels [16,36,37,38,39], GBP shows a performance as good as or even better than these methods.

The spatially weighted pooling (SWP) [36] strategy was proposed to improve the robustness and effectiveness of the feature representation, compared with SWP, GBP can improve the performance of the model better when combined with different backbones. Ref. [37] devised a novel model Hierarchical Bilinear Pooling with Aggregated Slack Mask (HBPASM) to generate a RoI-aware image feature representation for better performance, ref. [16] first proposed to obtain a better feature representation by adjusting channel dimensions and performing the Hadamard product between different hierarchical feature layers, but this approach only performs well when using VGG as the backbone, the generalization performance is poor. Mutual-channel loss [39] achieved the state-of-the-art performance when implemented on top of common base networks. Channel permutation and weighted combination regularization in [38] also showed its effectiveness. These methods achieved the state-of-the-art in different ways. Compared with these methods, GBP is simple and easy to implement and has good generalization ability; the Inter-GBP achieves 94.22% accuracy on the Stanford Cars dataset, which is the best accuracy.

In general, GBP not only achieves the best experimental results among methods based on bilinear pooling, but also shows a competitive performance and greater potential compared with other fine-grained image classification methods.

4.5. Visualization

To visually demonstrate the recognition of fine-grained images by the GBP based model, as shown in Figure 5, we visualized the model’s response to the input images of CUB dataset. We compute the magnitude of feature activations averaged across feature channels as the attention of the model and superimpose it to the input image.

Figure 5 shows the model’s visual recognition of images by taking three randomly selected subclasses of birds as examples. It is shown that the model tends to ignore features in the cluttered background and focus on the most discriminative parts of the birds. For example, when recognizing the Northern Waterthrush, similar features such as color and texture exist in the background, but the model focuses primarily on the bird and the distinguishing part of the bird.

5. Conclusions

In order to get compact bilinear representation, we propose grouping bilinear pooling (GBP) for fine-grained image classification in this paper. By dividing the feature layers into different groups and carries out intra-group bilinear pooling or inter-group bilinear pooling, GBP can obtain extreme compact representation. Compared with other compact bilinear methods, GBP achieves the-state-of-the-art. Besides, few papers use a more powerful backbone to achieve bilinear pooling because it is hard to balance accuracy and the complexity of model, but the experiments show that GBP can be embedded into different models as a plug-and-play module and perform well.

With the development of computer vision, a series of peaks that bilinear pooling never reached were achieved by the new methods. As an improvement of bilinear pooling, GBP achieves a competitive performance compared with other approaches. It is worth noting that GBP does not conflict with other fine-grained image classification methods, which means GBP has much more potential. In the future work, we plan to further explore GBP by combining it with other methods in different tasks.