Image-Caption Model Based on Fusion Feature

Abstract

The encoder–decoder framework is the main frame of image captioning. The convolutional neural network (CNN) is usually used to extract grid-level features of the image, and the graph convolutional neural network (GCN) is used to extract the image’s region-level features. Grid-level features are poor in semantic information, such as the relationship and location of objects, while regional features lack fine-grained information about images. To address this problem, this paper proposes a fusion-features-based image-captioning model, which includes the fusion feature encoder and LSTM decoder. The fusion-feature encoder is divided into grid-level feature encoder and region-level feature encoder. The grid-level feature encoder is a convoluted neural network embedded in squeeze and excitation operations so that the model can focus on features that are highly correlated to the title. The region-level encoder employs node-embedding matrices to enable models to understand different node types and gain richer semantics. Then the features are weighted together by an attention mechanism to guide the decoder LSTM to generate an image caption. Our model was trained and tested in the MS COCO2014 dataset with the experimental evaluation standard Bleu-4 score and CIDEr score of 0.399 and 1.311, respectively. The experimental results indicate that the model can describe the image in detail.

1. Introduction

With the rapid development of multimedia technology and computer networks, the multi-modal character of data has become more prominent, and the amount of data has exploded. There is a growing demand for multi-modal data processing in academia and business circles.

Farhadi et al. [1] proposed the task of image captioning. Their task was to realize modal conversion from the image mode to the text mode through the model, which was embodied as a binary group (I, S). The model completed the modal conversion from image mode I (Image) to text mode S (Sentence). Image captioning is a cross-task between Computer Vision (CV) and Natural Language Processing (NLP). This task is straightforward for humans with particular life experiences. However, it is a considerable challenge for computers, which not only requires computers to understand the content of images but also requires computers to generate sentences with human language habits. Image captioning is essential in image understanding, human–computer interaction, assisting visually impaired people, and journalism. Because of its importance and challenge, it is becoming a hotspot in artificial intelligence.

In the early stages of task studies, most researchers used machine-learning methods that required much human effort to annotate, extract manually, and evaluate datasets, resulting in slow progress, poor model performance, and poor caption quality. With the rise of deep learning, image captioning, for which the tasks have significantly progressed, has become a mainstream method. This method employs CNN as the encoder to extract image features. Then the model sends them to Long Short-Term Memory Network (LSTM), which decodes features to caption. This approach avoids wasting resources and produces more flexible and high-quality captions. Recently, GCN has been used as an encoder in many works to improve the model’s performance. The difference between CNN and GCN is that CNN extracts the grid-level features of the image. Grid-level features contain contextual and fine-grained information about the image, but some semantic information, such as the interrelationships and location relationships between multiple objects, is not included. GCN extracts the regional features of the image and relies on target detection techniques to accurately capture semantic information such as targets, target attributes, and the interrelationship between targets. However, it is object-oriented and ignores the fine-grained and contextual information of the rest of the image.

In order to solve this problem, this paper proposes an image caption-model based on the fusion feature. This model employs a fused feature approach, where an attention mechanism dynamically captures grid-level features and region-level features related to captioning, and they are weighted and fused. The fusion features are more capable of capturing the semantic relationships between objects and retain a certain degree of fine-grained and contextual information. Our captions are richer in information than traditional captions and can more fully express the relationships between objects in the images. The main contributions of this paper are as follows:

(1)

The grid-level features of an image are extracted by a Res-Net101 with squeeze and excitation operations. The GCN extracts the region-level features. The feature-fusion module weights fusion of the two features of the joint attention mechanism to learn image features and semantic information better.

(2)

Fusion-feature vectors are processed by a two-layer LSTM, which adds an attention mechanism to obtain the context information of the heading and dramatically improves the quality of the heading.

(3)

The algorithm was trained and tested, and the experimental results show that the algorithm achieved excellent performance.

The remaining part of the article is structured as follows. Section 2 describes related work, Section 3 details the model and method, Section 4 presents the experimental results, and Section 5 concludes the paper.

2. Related Work

2.1. Image Captioning

The methods of image captioning are mainly divided into three basic approaches. (1) The first is the template-based approach [1,2], which was proposed by Farhadi et al. in 2010. According to the syntax specification, this method sets the sentence template and the triad “object, action, scene”. The model detects all possible values of the scene, object, object’s attributes, and action in the image by target detection in computer vision. The model then uses the Conditional Random Field (CRF) algorithm to predict the correct three-element filling in the template to form the basic structure of the title; next, it uses the relevant algorithm to fill the rest of the template to create the image caption. The method requires the manual design of syntactic templates and relies on the visual concept of hard decoding. The caption generated by this method has a single grammatical form and an insufficient variety of captions. (2) The second is Kuznetsova et al.’s retrievation-based approach [3,4]. Based on the input image, the method generates an image caption driven by the whole database, retrieves similar images, and synthesizes phrases describing the image, which are then selectively combined to create image captions. The caption quality depends on the similarity between the input image and the image in the database. It is not easy to guarantee the semantic correctness of the title. (3) The third is the encode–decode-based approach [5,6,7,8,9,10,11], which was proposed by Vinyals et al. in 2015. The model uses CNN and Long Short-Term Memory Network (LSTM) as the encoder and decoder, respectively. The model obtains the input image; it extracts the image feature by CNN, initializes the decoder using the image feature, then generates the word chronologically, and then combines the text into image captions. These captions are semantically rich, syntactically flexible, and aligned with human language logic. The encode–decode-based approach is the current mainstream image captioning method.

2.2. Grid-Level Feature

CNN, as an encoder, extracts grid-level features of images. To improve the image-caption model, research focuses on the attention mechanism. Xu et al. [7] first applied the attention mechanism in the image-caption field. They proposed an image-caption model based on the attention mechanism, which includes “soft attention” and “hard attention”. The soft attention assigns weights to all grids for each decoding, ranging from 0 to 1, and uses backward propagation for training. The hard attention mechanism focuses on only one graph grid for each decoding. It uses one-hot coding, which takes less time but is not differentiable, and it generally uses Monte Carlo sampling to estimate the gradient first and then backward propagation for training. The attention mechanism commonly used in image captioning is mainly a “soft” attention mechanism. Attention-mechanism models optimize the CNN extracted image features from global to grid features. Before generating words, the model calculates the correlation between the words generated and each grid in the image by the attention mechanism to select the grid features with high correlation. It passes them to the decoder and guides the decoder to generate the image captions. The introduction of the attention mechanism gives the image caption model the ability to focus on the key grid. However, this attention mechanism will force words to correspond to the going grid, and dummy words such as “of” and “the” will also force them to correspond to the images’ grids, resulting in wasted arithmetic power. Therefore, Lu et al. [8] proposed an adaptive attention mechanism. This attention mechanism introduces the “Visual Sentinel” vector, which indicates the relevance of the generated words to the visual information between 0 and 1. When the words are directly related to the image, the value is 1, and the model will focus on the image grid and generate the words. When words, such as “of” and “the,” are generated, they are inferred directly by the language model. In recent years, various attention mechanisms have been applied to grid-level features to improve model performance continuously.

2.3. Region-Level Feature

Since the proposed encoder–decoder model, the research on image high-level semantics extraction has been slow due to technical limitations. Before Kipf et al. [12] proposed GCN, GCN [13] showed a high performance in extracting non-Euclidean data signatures (as shown in the structure) and remained high when untrained. Graph structure has the advantage that other data structures cannot compare to in representing semantic information. It can express the high-level semantics of images, such as objects in an image, their properties, and the interrelationship between objects.

Yao et al. [14] proposed a GCN–LSTM architecture for an image-captioning model, the first GCN application to the image-caption-generation field. The model relies on target detection techniques such as Faster-RCNN. It first detects the relationship between object, object attribute, and object in the image by target-detection techniques; then it constructs a graph structure. GCN was used to extract the graph structure’s features to guide the LSTM to generate the caption. Graph convolutional neural networks are widely used in image captioning. It is worth mentioning that, with the advent of graph convolutional neural networks, the features of the image have been changed from the direct extraction of grid-level features to the extraction of region-level features of the image by the target-detection technique first.

Yao et al. [13] proposed the Hierarchy Parsing (HIP) architecture. It was developed by combining Faster-RCNN and Mask-RCNN techniques for region-level and instance-level segmentation of images, constructing the image into a tree structure, where I represents the image, and R represents the region-level object. M represents the instance-level object, representing the relationship in the tree structure, and then using GCN to extract the features of the tree structure, passing the features into the up-to-down attention mechanism to compute the most relevant few objects. The model adopts the Analytic Hierarchy Process (AHP) to extract the semantics of images at three levels. The acquired semantics are richer, and the model’s generalization is more robust than the regular model. However, the tree structure has some limitations in expressing the complex relationship of objects in the image. Therefore, Shi et al. [15] proposed a framework for Caption-Guided Visual Relationship Graph (CGVRG). The framework first takes objects from the image by Faster-RCNN and then by text scene graphical parsers to extract relational triads from the caption. Then CGVRG is constructed by using weak-supervised learning to correspond to objects and predicates. The CGVRG is input to GCN, and GCN extracts the features and context vectors of CGVRG. The model uses the features and context vectors of the graph structure to guide the decoder to generate captions so that the model has better semantic information than the regular model.

3. Models and Methods

The image-caption model based on the fusion feature, which shown in Figure 1, is designed to generate fluent and semantically informative sentences based on a specified image, I. The model adopts an encoder–decoder structure, which consists of a fusion of characteristic encoder and decoder. The encoder and decoders are described in Section 3.1 and Section 3.2, respectively.

3.1. Encoder

The fused-feature encoder proposed in this paper consists of three parts: squeeze and excitation module (SE), node embedding module (NE), and feature-fusion module based on the attention mechanism. The SE module can extract the image’s grid-level features, and the NE module extracts region-level features of the image. Then they are weighted and fused by attention mechanisms so that the model obtains more fine-grained and richer semantic information.

3.1.1. Squeeze and Excitation Module (SE)

The frame of the SE module is shown in Figure 2. The encoder dynamically captures the dependency of each channel’s feature map and selects highly dependent feature maps to guide the decoder.

Inspired by Hu et al. [16], we embedded the SE module in front of the convolutional operation of ResNet-101. The feature of the l-th convolutional layer is expressed as $x^l=\left\{x_1^l,\dots ,x_C^l\right\}$, where the feature $x^l$ is composed of several feature maps, $x_m^l$, and the size of $x_m^l$ is h*w. C represents the number of channels in the feature map. The calculation of $SE$ is as follows:

$$z_m=F_{sq}(x_m^l)=\frac{1}{H\times W}{\displaystyle \sum }_{i=1}^H{\displaystyle \sum }_{j=1}^W{x}_m^l\left(i,j\right)$$

(1)

First, we add each value of x and divide it by its size. We compress feature map x into a value. Then $z_m$ and the dual activation function are used to calculate the weight of the feature map in the m-th channel. The calculation is as follows:

$$s_m=F_{ex}\left(z_m,W\right)=\sigma \left(W_2\delta \left(W_1{z}_m\right)\right)=\frac{1}{1+e^{W_{2}max\left(0,W_1{z}_m\right)}}$$

(2)

where $s_m$ is the grid-level feature attention weight of $z_m; \delta$ is the ReLU function that ensures a positive output; $W_1$ and $W_2$ are weight parameters to learn; and $\sigma$ is the Sigmoid activation function that maps the grid-level feature attention weight between 0 and 1. The dual fully connected layers enable the model to capture the dependency of feature maps better.

Finally, the module multiplies the dependency of the feature map, $s_m$, with the feature values in the feature map $x_m^l$. The calculation is as follows:

$$f_m=F_{scale}\left(x_m^l , s_m\right)=s_m \ast x_m^l$$

(3)

where $f_m$ represents the m-th channel’s feature map after an SE operation. After SE operation, the feature of the l-th convolutional layer is expressed as $f^l=\left\{f_1^l,\dots ,f_C^l\right\}$. Then we employ the convolution operation on $f^l$ to obtain the feature maps of $x^{l+1}$. The calculation is as follows:

$$x^{l+1}=Residual\left(f^l\right)$$

(4)

So, the grid-level feature is $F_g=\left\{s_1,\dots ,s_C\right\}$, which is the output of the final layer.

3.1.2. Node Embedding (NE) Module

The NE module of the model is shown in Figure 3, which relies on the MS COCO dataset processed by the target detection technique. The processed dataset includes the object information in the image, object attributes, the relationship between objects, and the corresponding box position. The model employs this information to construct a directional graph structure that initially encodes the graph nodes as X = {x₁, _, x_V}, in which x_i is the visual feature of the node and V is the number of nodes.

Unlike the traditional GCN of visual features of direct extraction nodes, this model incorporates the node-embedding operation in this module. Through the node embedding of different types of nodes, the model can distinguish the type of the node, as well as obtain the semantic information of the node and its neighbors better, and the formula of node embedding is as follows:

$$x_i^{\left(0\right)}=\left\{\begin{array}{c}{v}_i⨀W_r\left[0\right], if i \in o;\hfill \\ v_i⨀\left(W_r\left[1\right]+pos\left[i\right]\right), if i \in a;\hfill \\ v_i⨀W_r\left[2\right], if i \in r.\hfill \end{array}\right.$$

(5)

where o, a, and r represent object node, attribute node, and relationship node, respectively; $W_r$ ∈ $ℝ$^3×d is the role-embedding matrix; d is the feature dimension; $W_r$[k] denotes the k-th row of $W_r$; and $pos\left[i\right]$ is the positional embedding to distinguish the order of the different attribute nodes connected to the same object.

After node embedding, the model obtains the graph structure $G=(V,\epsilon ,R)$. The model then encodes the graph context in G by using GCN as follows:

$$X_i^{\left(l+1\right)}=\sigma (W_0^{\left(l\right)}{x}_i^{\left(l\right)}+{\displaystyle \sum }_{\tilde{r}\in R}{\displaystyle \sum }_{j\in N_i^{\tilde{r}}}\frac{1}{\left|N_i^{\tilde{r}}\right|}{W}_{\tilde{r}}^{\left(l\right)}{x}_i^{\left(l\right)})$$

(6)

where $N_i^{\tilde{r}}$ denotes the neighbors of the i-th node under the relation $\tilde{r}$ ∈ $R$, σ is the ReLU activation function, and $W_{\ast }^{\left(l\right)}$ is the parameter to be learned by the l-th GCN layer. Using one layer brings context for each node from its direct neighbor nodes, while stacking multiple layers can encode a broader range of contexts in the graph. The model superimposes the L layer and then uses the output of the final L layer as the final node embedded in X. The regional features of the image are obtained by averaging X.

3.1.3. Feature-Fusion Module

The feature-fusion module employs the attention mechanism to weigh the fusion of two levels of features. In our fusion method, we chose the vector-splicing approach to avoid the fusion-noise problem after the fusion of two levels of features.

First, the model uses the following equations to perform vector concatenate for two levels of the feature:

$$F=concat\left(F_c , F_r\right)$$

(7)

The model computes the attention weights for the features with the following equation:

$$\beta =\sigma \left(W_2\delta \left(W_{1}F\right)\right)$$

(8)

where $W_1$ and $W_2$ are the parameters of learning. Finally, multiply and merge the image features with their attention weights by taking the following actions:

$$F=\beta \otimes F$$

(9)

3.2. Decoder

The model adopts a double-layer memory network with an attention mechanism, aiming to increase the depth of the model, extract deeper features, improve the prediction accuracy, and add an attention weight vector to the LSTM operation, which is calculated as follows:

$$h_t={ \sigma }_h\left(W_{xh}{x}_t+W_{hh}{h}_{t-1}+b_n\right)$$

(10)

$$y_t={\sigma }_y\left(W_{hy}{h}_t+b_y\right)$$

(11)

$$f_t ={\sigma }_g\left(W_f{x}_t+U_f{h}_{t-1}+b_f\right)$$

(12)

$$i_t ={\sigma }_g\left(W_i{x}_t+U_i{h}_{t-1}+b_i\right)$$

(13)

$$o_t ={\sigma }_g\left(W_o{x}_t+U_o{h}_{t-1}+b_o\right)$$

(14)

$$\widetilde{C_t}=\tanh \left(W_C{x}_t+U_C{h}_{t-1}+b_C\right)$$

(15)

where $x_t$ is the input vector of the LSTM, and $h_{t-1}$ is the output vector of the LSTM of the previous time step. The LSTM module of each time step consists of an Att–LSTM and a Lan–LSTM. The input of the Att–LSTM at the moment t in the LSTM is $x_t^1=\left[h_{t-1}^1,W_{t-1}\right]$, where $h_{t-1}^1$ is the output of the Lan–LSTM at the previous time step, and $W_{t-1}$ is the embedding vector of the output word of the previous time step. Finally, the two are spliced to obtain $x_t^1$. The output $h_t^a$ is obtained by feeding it into the Att–LSTM. The input of the Lan-LSTM is $x_t^2=\left[F,h_t^a\right]$, which consists of $h_t^a$ and the image features obtained above, and the output is obtained as $h_t^2$. Using the notation $y_{1:T}$ to denote the sequence of words (y¹, …, y^T), the probability distribution of the output words at time step t is given by the following equation:

$$p\left(y_t\right|y_{1:t-1})=softmax (W_p{h}_t^2+b_p)$$

(16)

where $W_p$ and $b_p$ are the learned weights and bias parameters, respectively.

3.3. Dataset, Evaluation, and Loss Function

We performed experiments on the MS COCO2014 [17] dataset, which aims to improve the state-of-the-art in object recognition by placing the object-recognition problem in the context of a broader scene of understanding the problem and by collecting images containing common objects in natural environments. The dataset uses professional agencies to artificially describe the images with 5 or 15 reference descriptions per image, and the annotation set is typically saved in JSON format. The dataset has over 330,000 images, 200,000 annotated descriptions containing 91 classes of objects, and a total of 2.5 million tagged instances in 328,000 images, making it the largest semantic segmentation dataset available.

BLEU (Bilingual Evaluation Understudy) [18] and CIDEr [19] were used to evaluate this algorithm. The BLEU algorithm rates the difference between the captions generated and those annotated manually, scoring outputs between 0 and 1. The algorithm has become one of image description’s most widely used algorithms. For an image, the image description model is used to evaluate a set of 5 descriptors generated for the image and manually marked:

$$C{P}_n (C,S)=\frac{{{\displaystyle \sum }}_i{{\displaystyle \sum }}_{k}min\left(h_k\left(c_i\right),ma{x}_{j\in m}{h}_k\left(s_{ij}\right)\right)}{{{\displaystyle \sum }}_i{{\displaystyle \sum }}_k{h}_k\left(c_i\right)}$$

(17)

$$b\left(C,S\right)=\left\{\begin{array}{c}1 , if l_C>l_S\\ e^{1-l_S/l_C} , if l_C<l_S\end{array}\right.$$

(18)

$$BLE{U}_N\left(C,S\right)=b\left(C,S\right)exp({{\displaystyle \sum }}_{n=1}^N{\omega }_{n}lgC{P}_n\left(C,S\right))$$

(19)

The number of occurrences of the n-tuple, ${\omega }_k$, in the manual reference caption, S_ij, and the caption, $c_i$, to be evaluated is denoted as $h_k\left(s_{ij}\right)$ and $h_k\left(c_i\right)$. Moreover, $l_S$ is the length of the caption to be evaluated, and l_C is the length of the manual reference caption. The higher the BLEU score, the better the performance.

The METEOR metric considers the effect of recall, which calculates the matching relationship between synonyms, root words, and affixes, and the evaluation results are more correlated with the results of the manual evaluation, which is calculated as follows:

$$F_{means}=\frac{PR}{\alpha P+\left(1-\alpha \right)R}$$

(20)

$$P=\frac{m}{c}$$

(21)

$$R=\frac{m}{r}$$

(22)

$$Pen=\frac{\#chunks}{m}$$

(23)

$$METEOR=\left(1-Pen\right) ·F_{means}$$

(24)

where R is the recall rate; P is the accuracy rate, m is the total number of matched pairs; c is the length of the candidate caption; r is the length of the reference caption; and Pen is the penalty factor, which is to consider the order between words, if in two sentences, the words that match each other are adjacent. They are defined as the same chunk, the total number of #chunks.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of evaluation metrics proposed by Lin, mainly including ROUGE-N, ROUGE-L, ROUGE-S, ROUGE-W, and ROUGE-SU. Users can choose the appropriate evaluation metrics according to their needs. ROUGE-L evaluation metrics are generally used to evaluate the model’s performance in the image-captioning field. ROUGE-L is mainly optimized for the problem that BLEU evaluation metrics ignore the recall rate. Similar to BLEU, ROUGE-L is also based on the n-gram evaluation metrics, which calculate the tuple overlap between generated captions and reference captions to measure the quality of captions. BLEU is the same, except that the recall factor is added to the algorithm, and its calculation formula is as follows:

$$\mathrm{ROUGE}-L=\frac{\left(1+{\beta }^2\right)R_{lcs}{P}_{lcs}}{R_{lcs}+{\beta }^2{P}_{lcs} }$$

(25)

$$R_{lcs}=\frac{LCS\left(X,Y\right)}{m}$$

(26)

$$P_{lcs}=\frac{LCS\left(X,Y\right)}{n}$$

(27)

where X denotes the candidate title, Y denotes the reference title, LCS(X,Y) denotes the length of the longest common subsequence of the candidate caption and the reference caption, m denotes the length of the reference caption, and n denotes the length of the candidate caption.

CIDEr is specially designed to evaluate the image-captioning model; it obtains the similarity between the captions to be evaluated and the reference captions by calculating the TF-IDF weights of each n-tuple to evaluate the effectiveness of the image captioning.

The number of occurrences of the n-tuple, ${\omega }_k$, in the manual reference caption, S_ij, and the caption $c_i$ to be evaluated is denoted as $h_k\left(s_{ij}\right)$ and $h_k\left(c_i\right)$, and the TF-IDF weight, $g_k\left(S_{ij}\right)$, of the n-tuple is shown below:

$$g_k\left(S_{ij}\right)=\frac{h_k\left(S_{ij}\right)}{\sum {\omega }_l \in \Omega }lg\left(\frac{\left|I\right|}{{{\displaystyle \sum }}_{I_p \in I}min\left(1,{{\displaystyle \sum }}_q{h}_k(s_{pq})\right)}\right)$$

(28)

$$CIDE{r}_n\left(c_i,S_i\right)=\frac{1}{m}{\displaystyle \sum }_j\frac{g^n\left(c_i\right)·g^n\left(s_{ij}\right)}{||g^n\left(c_i\right)||·||g^n\left(s_{ij}\right)||}$$

(29)

$$CIDEr\left(c_i,S_i\right)={\displaystyle \sum }_{n=1}^N{\omega }_{n}CIDE{r}_n\left(c_i,S_i\right)$$

(30)

The higher the CIDEr score, the better the resulting discourse quality.

Based on the given sequence of real artificial descriptions, $y_{1:T}^{\ast }$, and the parameters, θ, obtained from the training of the model in this paper, a cross-entropy function is used to minimize the loss, calculated according to Equation (31):

$$L\left(\theta \right)=-{\displaystyle \sum }_{t=1}^{T}log(P_{\theta }(y_t^{*}|y_{1:t-1}^{*}))$$

(31)

where $y_t^{\ast }$ denotes the t-th true artificial description.

4. Results

4.1. Experimental Environment

The experimental environment is based on Ubuntu 18.04, Inteli9-9900 k CPU, NVIDIA GeForce RTX 3080 GPU, 12 GB RAM, PyTorch deep-learning environment with Python 3.8 + CUDA 11.1. In processing the annotation file, the non-letter characters were deleted. The remaining characters were converted to lowercase letters, and all words that appeared less than five times were replaced by the unique word UNK, making the 10,942 words in the MS COCO dataset the final use of the corpus. The maximum output length of the algorithm is set at 16, the algorithm scores the highest for the evaluation index. The dropout method is used to prevent excess surplus, and the parameter is set to 0.5. The model is trained with the Adam optimizer, with a learning rate set at 3 × 10⁻⁴, weight decay set at 1 × 10⁻⁶, batch size set at 128, and training rounds set at 30.

4.2. Experimental Results and Analysis

Firstly, the validity of the model is verified. In this paper, five state-of-the-art-level image-captioning models in the last three years are selected for comparison, and the results are shown in

Table 1. Model performance on the MS COCO dataset.

	BLEU-1	BLEU-4	CIDEr	METEOR	ROUGE-L
High-Level Attention, 2019 [20]	0.746	0.317	1.103	0.265	0.535
CGVRG, 2020 [15]	0.814	0.386	1.267	0.286	0.588
M2-Transformer, 2020 [21]	0.816	0.397	1.293	0.294	0.592
RDN, 2019 [22]	0.775	0.368	1.153	0.272	0.568
Entangle-Transformer, 2019 [23]	0.776	0.378	1.193	0.284	0.574
DLCT, 2021 [24]	0.816	0.392	1.297	0.298	0.598
Ours	0.821	0.399	1.311	0.297	0.601

. the scoring results are all on the MS COCO dataset with the condition of five reference captions. (1) High-Level Attention is the image caption generation model proposed by Ding et al. [20]. The model adopts a CNN–LSTM architecture with an embedded bottom-up attention mechanism. The model can score low-level features (contrast, sharpness, and clarity) and high-level features (face impact) of image regions and combine the scores to determine the regions to which attention should be directed. (2) CGVRG is an image-caption-generation model based on a caption-guided visual relationship graph (CGVPR) proposed by Shi et al. [15]. The model uses the features and context vectors of the graph structure to guide the decoder to generate captions so that the model has better semantic information. (3) M²-Transformer is a model proposed by Cornia et al. [21], adopting the transformer architecture. The framework learns the multilevel representation of the relationship between image regions. It uses grid-like connections to exploit the properties of the lower and higher levels in the decoding phase, again reducing the complexity of the model and mitigating the semantic gap. (4) RDN (Reflective Decoding Network) is a model proposed by Ke et al. [22]. The CNN–LSTM architecture is embedded in the RDN. The framework enhances the decoder’s long-sequence dependence and improves the decoder’s long-sequence modeling capability. (5) Entangle-Transformer is a model proposed by Li et al. [23]. The transformer [24,25] model is also adopted. The framework alleviates the semantic gap. (6) DCLT is a Transformer-based model proposed by Luo et al. [26] that uses two Transformers to extract two levels of features and fuse them.

Compared with the above five models, the results of the five evaluation metrics consistently show that our work is favorable with other state-of-the-art techniques, which include CNN-based (High-Level Attention and RDN), GCN-based (CGVRG), and Transformer-based models. Our model’s CIDEr score and BLEU-4 score can reach 131% and 39.9%, which are 2.4% and 0.5% better than the best comparison models, respectively. Furthermore, inspired by the fact that all five better models contain attention mechanisms, the model adds a fusion-feature attention mechanism at a later stage to improve the performance. From the analysis of the model architecture and its performance, the current image-caption-generation models, mainly GCN and Transformer, have gradually replaced the mainstream encoder position of CNN because of their excellent performance. We performed a second set of ablation experiments to further explore the model’s advantages over the traditional CNN and GCN, as shown in

Table 2. Performance of frame structure.

	BLEU-1	BLEU-4	CIDEr	METEOR	ROUGE-L
CNN–LSTM	0.666	0.246	0.862	0.201	-
CNN–LSTM–Attention	0.726	0.310	0.920	0.235	-
GCN–LSTM	0.808	0.387	1.253	0.285	0.585
GCN–LSTM–Attention	0.816	0.393	1.279	0.288	0.590
Grid + Regionadd	0.801	0.344	1.237	0.273	0.577
Grid + Regionmul	0.811	0.389	1.271	0.280	0.591
Grid + Regionconcat	0.821	0.399	1.311	0.297	0.601

.

In order to investigate the superiority of this model over the traditional model, several ablation experiments were carried out. First, a CNN–LSTM architecture with no embedded attention mechanism was used as the baseline results. The first set of ablation experiments in this study was CNN–LSTM architecture based on an embedded MLP attention mechanism. The results showed that the ablation model based on the embedded attention mechanism has good scoring performance. GCN–LSTM was the framework for the second set of experiments. The data show that GCN has a significant performance advantage over CNN, and CNN–LSTM–Attention, with its CIDEr score of 39.1% and 33.3% higher than CNN–LSTM and CNN–LSTM–Attention, respectively. Due to the excellent performance of the attention mechanism, we embedded the attention mechanism in the GCN–LSTM architecture, and its performance in the CIDE metric increased by another 2.6%, further validating the attention mechanism in the image-caption-generation model. Next, we performed three sets of comparison experiments to verify the effectiveness of the fusion properties. Firstly, we verified the add-fusion of region features and channel features by Grid + Region^add. The model performance outperformed the traditional CNN–LSTM–Attention architecture, which verified the feasibility of fused features. Due to the fusion noise problem in add-fusion and multiplicative fusion, we obtained the best performance on the concatenate approach Grid + Region^concat measured by CIDE scores. It was 3.2% higher than in the best-class comparison experiments.

The above ablation experiment verifies the superiority of the image-caption-generation model based on fusion features. To further explore the role and importance of each part of the model, we performed a third set of ablation experiments at baseline, using a pure CNN + GCN + LSTM architecture, without adding the SE module, node embedding modules, attention mechanism fusion module, and pure vector stitching fusion vectors, as shown in

Table 3. Comparison of the performance of each part of the model.

	BLEU-4	CIDEr
Baseline	0.792	1.183
Grid + Region + SE	0.797	1.199
Grid + Region + NE	0.802	1.217
Grid + Region + Att	0.811	1.233
Grid + Region + SE + NE	0.807	1.229
Grid + Region + SE + Att	0.818	1.253
Grid + Region + NE + Att	0.816	1.267
Grid + Region + SE + NE + Att	0.821	1.311

. SE refers to the SE channel feature-extraction module, NE refers to the node-embedded regional feature module, and Att refers to the feature-fusion module based on the attention mechanism.

As can be seen from the above experimental results’ graphs and images, the attention mechanism of the feature-fusion module improves model performance the most among the various components of the model, by 1% and 3.4%, respectively, compared to the baseline of BLEU-4 and CIDE. In the first two cases alone, SE and NE are added. While more semantic and contextual information is obtained, the lack of attention to the mechanisms’ fusion process results in noise and poor performance of guidance information. Therefore, we embedded the attention mechanism into the module with SE and NE alone, and the experiments were conducted again. It is found that a more significant increase in both increases the performance compared to the case without the attention mechanism. In the SE + NE + Att case, the model can obtain rich semantic and contextual information and attention mechanism to mitigate the fusion-noise problem and obtain the best performance.

After verifying the model’s performance through various experiments, two images were randomly selected and fed into this paper’s training and the NIC model models, respectively, resulting in a caption, as shown in Figure 4. As can be seen from the generated captions, the image captions generated by the model are more detailed and richer in semantic information.

5. Conclusions

This paper presents an image-description algorithm based on fusion features. The model innovatively integrates grid-level and region-level features so that the extracted image features have a visual focus. The streamlined model parameters can effectively reduce the training time. The algorithm further unites the cross-module features between visual image and language comprehension. Experiments show that the algorithm has an excellent performance in all evaluation metrics. In future work, we will further study the combination of the graphical convolutional neural network and transformer to strengthen the connection between targets in the image.