Associating Images with Sentences Using Recurrent Canonical Correlation Analysis

Abstract

Associating images with sentences has drawn much attention recently. Existing methods commonly represent an image by indistinctively describing all its contents in an one-time static way, which ignores two facts that (1) the association analysis can only be effective for partial salient contents and the associated sentence, and (2) visual information acquisition is a dynamical rather than static process. To deal with this issue, we propose a recurrent canonical correlation analysis (RCCA) method for associating images with sentences. RCCA includes a contextual attention-based LSTM-RNN which can selectively attend to salient regions of an image at each time step, and then represent all the salient contents within a few steps. Different from existing attention-based models, our model focuses on the modelling of contextual visual attention mechanism for the task of association analysis. RCCA also includes a conventional LSTM-RNN for sentence representation learning. The resulting representations of images and sentences are fed into CCA to maximize linear correlation, where parameters of LSTM-RNNs and CCA are jointly learned. Due to the effective image representation learning, our model can well associate images with sentences with complex contents, and achieve better performance in terms of image annotation and retrieval.

1. Introduction

Associating images with sentences plays an important role in many applications, e.g., finding sentences given an image query for image annotation and caption, and retrieving images with a sentence query for image search. Recently, various methods have been proposed to deal with this problem. Most of them first extract representations for images and sentences, and then use either a structured objective [1,2] or a canonical correlation objective [3,4] to learn a latent joint space for association analysis, where matched images and sentences have small distance or high correlation.

Obtaining effective representations for images is crucial for the association analysis. An image usually contains very abundant contents, but its associated sentence can only describe partial salient contents due the limited size. Thus the following association analysis can only be effective for these salient contents and their corresponding textual descriptions since only they share the same semantics. As a result, the rest undescribed contents in the image are redundant, and tend to bring in noisy side effects in the association analysis. However, most existing methods [1,3,4,5,6,7,8] ignore this fact and represent an image by simultaneously processing its entire contents. Although other methods [2,9,10] consider to associate the salient contents (regions) with the corresponding descriptions (words) in the image and sentence, respectively, they have to pre-establish the salient regions using manual annotations or detection algorithms. In addition, the existing methods usually represent the visual information in an one-time static process, which is in contrast to the neuroscience fact [11] that visual information acquisition is actually a dynamic rather than static process. Selective attention mechanism plays an essential role in this process. In particular, human visual system does not focus its attention on an entire image for only once. Instead, it selectively attends to salient regions at different locations to collect all important information in a sequential way.

Recently, some attention-based models have been proposed in various vision applications [12,13,14,15], but the modelling of attention mechanism in the context of association analysis is still under investigated. In fact, existing models mostly focus on the problems of recognizing salient image regions using dense supervision information, which makes them less generally applicable to the association analysis that aims to fuse the information of all the salient regions to obtain a unified representation. Moreover, they rarely consider the modelling of context [16] which actually has the prominent effect on attention mechanism. Ba et al. [17] consider to exploit the context information only at the first time step during the attention process, but they ignore the fact that context information modulates the attention mechanism during all the time steps [18].

In this work, we propose a recurrent canonical correlation analysis (RCCA) method for associating images with sentences using the visual attention. The proposed RCCA includes a contextual attention-based LSTM-RNN for image representation learning, which can selectively focus on salient regions of an image by referring to context information at each time step, and sequentially aggregate local features corresponding to the regions at all time steps to obtain the final image representation. RCCA also exploits a conventional LSTM-RNN to learn representation for the associated sentence by orderly taking its contained words as sequential inputs. Then the obtained representations of images and sentences are fed into CCA for association analysis, where parameters of LSTM-RNNs and CCA are jointly learned by maximizing linear correlation between images and sentences. To demonstrate the effectiveness of the proposed RCCA, we perform several experiments of image annotation and retrieval on three publicly available datasets, and achieve satisfactory results.

Our main contributions can be summarized as follows. We propose the recurrent canonical correlation analysis for associating images with sentences, which models the contextual visual attention mechanism and learns image representation in a biologically-plausible dynamical process. We systematically study the modelling of context in conjunction with attention mechanism, and demonstrate that the use of dense context information does help much. We achieve better performance in the tasks of image annotation and retrieval.

2. Related Work

2.1. Association between Images and Sentences

Existing methods for associating images with sentences usually include two steps: (1) feature extraction for images and sentences, and (2) association analysis based on the obtained features. Many recent methods [2,3,4,6,7,8,9,10,19] exploit CNNs [20,21] to obtain image features, i.e., the 4096-dimensional output vector in the last fully-connected layer. While for sentences, since they are sequential data, most methods use LSTM-RNNs [22] to model their long-range temporal dependency, and exploit the hidden states at the last step as their representations. After obtaining features, they use either a structured objective [1,2,7] or a canonical correlation objective [3,4,9] to learn a latent joint space, in which matched image-sentence pairs have small distance or high correlation.

It is believed that obtaining good image representations, such as by CNNs and fisher vector (FV) [4,23], can greatly facilitate the following association analysis. However, these methods indistinctively describe all the contents of an image, which might be suboptimal since only salient contents and their associated textual descriptions can be effective in association analysis. Although other works [2,9,10] consider to associate these salient regions with their descriptions, they have to explicitly use object detectors or manual annotations to pre-establish the salient regions. Different from them, our model can automatically select and attend to salient regions in the image by modelling the contextual attention mechanism, which is more efficient and biologically-plausible.

The most related work to ours is [24], but it has to perform an one-to-one similarity measurement during testing, which makes it much more time-consuming than ours. Note that we do not make comparisons with recent state-of-the-art technologies [25,26,27], since they all use advance image representation modules like ResNet [28] and bottom-up features [29].

2.2. Attention-Based Models

There have been several works simulating the attention mechanism. Alex Graves [30] exploits RNNs and differentiable Gaussian filters to simulate the attention mechanism, and applies it to handwriting synthesis. The model is able to predict a word conditioned on an auxiliary annotation sequence. Gregor et al. [12] introduce the deep recurrent attentive writer for image generation, which develops a novel spatial attention mechanism based on 2-dimensional Gaussian filters to mimic the foveation of human eyes. Larochelle and Hinton [31] propose high-order Boltzmann machines to learn how to accumulate information from an image over several fixations. Mnih et al. [32] present a RNN-based model to select a sequence of regions and then only process the selected regions from an image to extract information. Ba et al. [17] present a recurrent attention model for multiple objects recognition. By training with reinforcement learning, the model can attend to some most label-relevant regions of an image. Bahdanau et al. [13] propose a neural machine translator which can search for relevant parts of a source sentence to predict a target word, without explicit alignment. Inspired by it, Xu et al. [15] develop an attention-based caption model which can automatically learn to fix gazes on salient objects in an image and generate the corresponding annotated words. In addition, this work is also related to recent works [33,34,35] focusing on semantic segmentation based on attention-based networks. To the best of our knowledge, we make the first attempt to consider the attention mechanism in the context of association between images and sentences, and systematically study the modelling of contextual attention mechanism.

3. Association between Images and Sentences via Recurrent Canonical Correlation Analysis

In this section, we will introduce the proposed recurrent canonical correlation analysis (RCCA) for associating images with sentences, by describing its included three modules in details: (1) a contextual attention-based LSTM-RNN for image representation learning, (2) a conventional LSTM-RNN for sentence representation learning, and (3) canonical correlation analysis based on the resulting image and sentence representations for model learning.

3.1. Image Representation Using Contextual Attention-Based LSTM-RNN

The contextual attention-based LSTM-RNN is illustrated in Figure 1 (at the bottom). Given an image, we first use a CNN [36] to extract two kinds of features: (1) multiple local features ${\left\{{\mathbf{a}}_l|{\mathbf{a}}_l\in {\mathbb{R}}^F\right\}}_{l=1,\cdots ,L}$, i.e., feature maps from the last convolutional layer, where ${\mathbf{a}}_l$ describes l-th local region in the image and L is the total number of local regions, and (2) a context feature $\mathbf{m}\in {\mathbb{R}}^D$, i.e., a feature vector from the last fully-connected layer, describing the global context information of the image. Based on these features, we exploit a contextual attention scheme to obtain sequential representations ${\left\{{\mathbf{e}}_t\right\}}_{t=1,\cdots ,T}$ for the static image, where ${\mathbf{e}}_t$ is a combined local feature describing the attended salient regions at the t-th time step. Then we use LSTM-RNN to model the long-rang temporal dependency in the sequential representations, by separately taking them as inputs to hidden states at different time steps ${\left\{{\mathbf{h}}_t\in {\mathbb{R}}^{D_x}\right\}}_{t=1,\cdots ,T}$. As time goes on, all the hidden states propagate their values along the time axis until the end, and thus the hidden state at the last time step ${\mathbf{h}}_T$ indicates the fused representation of all the attended salient regions.

The details of the contextual attention scheme at the t-th time step can be formulated as follows. As illustrated in Figure 2a, given context feature $\mathbf{m}$, local feature set ${\left\{{\mathbf{a}}_l|{\mathbf{a}}_l\in {\mathbb{R}}^F\right\}}_{l=1,\cdots ,L}$ and hidden state at previous time step ${\mathbf{h}}_{t-1}$, we compute ${\left\{p_{t,l}|p_{t,l}\in [0,1]\right\}}_{l=1,···,L}$ for all the local regions, where $p_{t,l}$ indicates the probability that the l-th region will be attended to at the t-th time step:

$$p_{t,l}=\frac{e^{{\widehat{p}}_{t,l}}}{{\sum }_{l=1}^L{e}^{{\widehat{p}}_{t,l}}},{\widehat{p}}_{t,l}=f_{gated}\left([{\mathbf{h}}_{t-1},\mathbf{m},{\mathbf{a}}_l]\right)$$

(1)

where $f_{gated}(·)$ is a gated fusion unit [37] to fuse all the context information in terms of ${\widehat{\mathbf{h}}}_{t-1}$, $\mathbf{m}$ and ${\mathbf{a}}_l$ as input, and ${\widehat{\mathbf{h}}}_{t-1}$ is obtained by modulating previous hidden state ${\mathbf{h}}_{t-1}$ with context information $\mathbf{m}$. Note that the usage of context information is necessary, since it can not only provide the complementary global information about the current attended regions, but also encourage the model to focus on other unexplored regions under the principle of information maximization.

After obtaining the probabilities, we exploit the “soft attention” scheme [13] to compute the attended local feature ${\mathbf{e}}_t$. We perform element-wise multiplication between each local feature ${\mathbf{a}}_l$ and its corresponding probability $p_{t,l}$, and sum all the products together:

$${\mathbf{e}}_t=\sum _{l=1}^L{p}_{t,l}{\mathbf{a}}_l$$

(2)

Then, we can use LSTM-RNN [38] to model the temporal dependency in the sequential representations ${\left\{{\mathbf{e}}_t\right\}}_{t=1,\cdots ,T}$ as shown in Figure 2b, where (initial) memory state (${\mathbf{c}}_0$) ${\mathbf{c}}_t$, (initial) hidden state (${\mathbf{h}}_0$) ${\mathbf{h}}_t$, input gate ${\mathbf{i}}_t$, forget gate ${\mathbf{f}}_t$ and output gate ${\mathbf{o}}_t$ can be computed as follows:

$${\mathbf{c}}_0=f_{MLP,\mathbf{c}}(\frac{1}{L}\sum _{l=1}^L{\mathbf{a}}_l),{\mathbf{h}}_0=f_{MLP,\mathbf{h}}(\frac{1}{L}\sum _{l=1}^L{\mathbf{a}}_l)$$

(3)

$${\mathbf{i}}_t=\sigma (W_{\mathbf{e}\mathbf{i}}{\mathbf{e}}_t+W_{\mathbf{h}\mathbf{i}}{\mathbf{h}}_{t-1}+{\mathbf{b}}_{\mathbf{i}}),{\mathbf{f}}_t=\sigma (W_{\mathbf{e}\mathbf{f}}{\mathbf{e}}_t+W_{\mathbf{h}\mathbf{f}}{\mathbf{h}}_{t-1}+{\mathbf{b}}_{\mathbf{f}})$$

(4)

$${\mathbf{c}}_t={\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}+{\mathbf{i}}_t\odot tanh(W_{\mathbf{e}\mathbf{c}}{\mathbf{e}}_t+W_{\mathbf{h}\mathbf{c}}{\mathbf{h}}_{t-1}+{\mathbf{b}}_{\mathbf{c}})$$

(5)

$${\mathbf{o}}_t=\sigma (W_{\mathbf{e}\mathbf{o}}{\mathbf{e}}_t+W_{\mathbf{h}\mathbf{o}}{\mathbf{h}}_{t-1}+{\mathbf{b}}_{\mathbf{o}}),{\mathbf{h}}_t={\mathbf{o}}_t\odot tanh\left({\mathbf{c}}_t\right)$$

(6)

where $f_{MLP,\mathbf{c}}(·)$ and $f_{MLP,\mathbf{h}}(·)$ are two multilayer perceptrons associated with $\mathbf{c}$ and $\mathbf{h}$, respectively, $\sigma$ denotes the sigmoid activation function, and ⊙ denotes element-wise multiplication.

3.2. Sentence Representation Using Conventional LSTM-RNN

Considering that sentences are naturally sequential data consisting of ordered words, and conventional LSTM-RNN can well model long-range contextual information, so we directly exploit the conventional LSTM-RNN to obtain sentence representations. As shown in Figure 1 (at the top), conventional LSTM-RNN orderly takes a word as the input to the corresponding hidden state at each time step, and all the hidden states propagate their values along the time axis. The hidden state at the last time step collects all the information in the sentence, which can thus be regarded as the representation of the entire sentence.

We denote $[{\mathbf{w}}_1,\cdots ,{\mathbf{w}}_t,\cdots ,{\mathbf{w}}_{T^{\prime }}]$ as a sentence associated with the image, ${\mathbf{w}}_t\in {\left\{0,1\right\}}^M$ as the t-th word in the sentence, $T^{\prime }$ as the number of words in the sentence (also the number of time steps in the conventional LSTM-RNN), M as the size of word vocabulary, and ${\left\{{\mathbf{h}}_t\in {\mathbb{R}}^{D_y}\right\}}_{t=1,\cdots ,T^{\prime }}$ as the hidden states at all time steps. Since the original representations of words are high-dimensional “one-hot” vectors, we embed them into low-dimensional real-valued vectors using an embedding matrix $E\in {\mathbb{R}}^{U\times M}$. The details of the conventional LSTM-RNN can be formulated as follows:

$${\mathbf{i}}_t=\sigma (W_{\mathbf{w}\mathbf{i}}E{\mathbf{w}}_t+W_{\mathbf{h}\mathbf{i}}{\mathbf{h}}_{t-1}+{\mathbf{b}}_{\mathbf{i}}),{\mathbf{f}}_t=\sigma (W_{\mathbf{w}\mathbf{f}}E{\mathbf{w}}_t+W_{\mathbf{h}\mathbf{f}}{\mathbf{h}}_{t-1}+{\mathbf{b}}_{\mathbf{f}})$$

(7)

$${\mathbf{c}}_t={\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}+{\mathbf{i}}_t\odot tanh(W_{\mathbf{w}\mathbf{c}}E{\mathbf{w}}_t+W_{\mathbf{h}\mathbf{c}}{\mathbf{h}}_{t-1}+{\mathbf{b}}_{\mathbf{c}})$$

(8)

$${\mathbf{o}}_t=\sigma (W_{\mathbf{w}\mathbf{o}}E{\mathbf{w}}_t+W_{\mathbf{h}\mathbf{o}}{\mathbf{h}}_{t-1}+{\mathbf{b}}_{\mathbf{o}}),{\mathbf{h}}_t={\mathbf{o}}_t\odot tanh\left({\mathbf{c}}_t\right)$$

(9)

Different from the contextual attention-based LSTM-RNN, here we do not include any explicit definition of the initial memory state and the hidden state, similar to [38].

3.3. Model Learning with Canonical Correlation Analysis

We regard the last hidden states of two modality-specific LSTM-RNNs as the representations of images and sentences, denoted by $X\in {\mathbb{R}}^{D_x\times N}$ and $Y\in {\mathbb{R}}^{D_y\times N}$, respectively, where N is the number of given image-sentence pairs, and $D_x$ and $D_y$ are the dimensions of obtained representations of images and sentences, respectively. We feed these representations to canonical correlation analysis (CCA) to learn two linear projections $W_x$ and $W_y$ for X and Y, respectively, so that their correlation is high:

$$(W_x^{*},W_y^{*})=\underset{W_x,W_y}{argmax}corr(X,Y)=\underset{W_x,W_y}{argmax}\frac{{W_x}^T{\sum }_{xy}{W}_y}{\sqrt{{W_x}^T{\sum }_{xx}{W}_x{W_y}^T{\sum }_{yy}{W}_y}}$$

(10)

The covariances ${\sum }_{xx}$, ${\sum }_{yy}$ and ${\sum }_{xy}$ are estimated as: ${\sum }_{xx}=\frac{1}{N-1}\overline{X}{\overline{X}}^T+{\lambda }_{x}I$, ${\sum }_{yy}=\frac{1}{N-1}\overline{Y}{\overline{Y}}^T+{\lambda }_{y}I$ and ${\sum }_{xy}=\frac{1}{N-1}\overline{X}{\overline{Y}}^T$, respectively, where $\overline{X}$ and $\overline{Y}$ are centered representations, and ${\lambda }_{x}I$ and ${\lambda }_{y}I$ are regularization terms to ensure the positive definiteness of ${\sum }_{xx}$ and ${\sum }_{yy}$.

Since the objective is invariant to the scaling of $W_x$ and $W_y$, the projections can be constrained to have unit variance. Then by assembling the top k projection vectors into the columns of projection matrices, denoted by ${\widehat{W}}_x$ and ${\widehat{W}}_y$, the objective can be written as:

$$\underset{{\widehat{W}}_x,{\widehat{W}}_y}{max}tr\left({{\widehat{W}}_x}^T{\sum }_{xy}{\widehat{W}}_y\right),s.t.,{{\widehat{W}}_x}^T{\sum }_{xx}{\widehat{W}}_x={{\widehat{W}}_y}^T{\sum }_{yy}{\widehat{W}}_y=1$$

(11)

Then according to [39], the optimum of this objective can be attained at:

$$({\widehat{W}}_x^{*},{\widehat{W}}_y^{*})=({{\sum }_{xx}}^{-1/2}{U}_k,{{\sum }_{yy}}^{-1/2}{V}_k)$$

(12)

where $U_k$ and $V_k$ are matrices of the first k left- and right- singular vectors of $T={{\sum }_{xx}}^{-1/2}{\sum }_{xy}{{\sum }_{yy}}^{-1/2}$, respectively. As a result, the objective is equivalent to the sum of the top k singular values of T, i.e., the trace norm:

$$corr(X,Y)={∥T∥}_{tr}=tr\left({\left(T^{T}T\right)}^{1/2}\right)$$

(13)

It should be noted that the parameters of two modality-specific LSTM-RNNs and CCA are jointly learned, where the parameters of LSTM-RNNs are adaptively trained to optimize the CCA objective using stochastic gradient descent. The gradients of $corr(X,Y)$ with respect to parameters of LSTM-RNNs can be computed by first obtaining the gradients with respect to X and Y as follows, and then performing backpropagation.

$$\frac{\partial corr(X,Y)}{\partial X}=\frac{1}{N-1}(2{\nabla }_{xx}\overline{X}+{\nabla }_{xy}\overline{Y}),\frac{\partial corr(X,Y)}{\partial Y}=\frac{1}{N-1}(2{\nabla }_{yy}\overline{Y}+{\nabla }_{yx}\overline{X})$$

$${\nabla }_{xx}=-\frac{1}{2}{{\sum }_{xx}}^{-1/2}UD{U}^T{{\sum }_{xx}}^{-1/2},{\nabla }_{xy}={{\sum }_{xx}}^{-1/2}U{V}^T{{\sum }_{yy}}^{-1/2}$$

$${\nabla }_{yy}=-\frac{1}{2}{{\sum }_{yy}}^{-1/2}VD{V}^T{{\sum }_{yy}}^{-1/2},{\nabla }_{yx}={{\sum }_{yy}}^{-1/2}V{U}^T{{\sum }_{xx}}^{-1/2}$$

where $UD{V}^T$ is the singular value decomposition (SVD) of T.

Similar to [40,41], although the objective of CCA is a function of the entire training set, we perform mini-batch based optimization due to the high computational cost. Such implementations often lead to the fact that, directly optimizing parameters of LSTM-RNNs from random initialization cannot obtain good results. To provide a better initialization for the parameters, we exploit a structured objective for parameter pretraining, which has been widely used in cross-modal learning [2,7,10,41]:

$$\begin{array}{c}\hfill W_{LSTM-RNN}^{*}=\underset{W_{LSTM-RNN}}{argmin}\sum _{i=1}^N\sum _{k=1}^{K}max\left\{0,m-d({\mathbf{x}}_i,{\mathbf{y}}_i)+d({\mathbf{x}}_i,{\mathbf{y}}_k)\right\}\\ \hfill +max\left\{0,m-d({\mathbf{x}}_i,{\mathbf{y}}_i)+d({\mathbf{x}}_k,{\mathbf{y}}_i)\right\}\end{array}$$

(14)

where $W_{LSTM-RNN}$ denotes all the parameters of two LSTM-RNNs, $\left({\mathrm{x}}_i,{\mathrm{y}}_i\right)$ are the obtained representations of the i-th matched images and sentences, ${\mathbf{x}}_k$ is a mismatched (contrastive) image representation for ${\mathbf{y}}_i$, and vice-versa with ${\mathbf{x}}_k$, K is the total number of contrastive samples, m is a tuning parameter and $d(·,·)$ is a cosine similarity measurement. This structured objective encourages representations of matched images and sentences to be more similar than those of mismatched ones. Such initial representations are helpful for the following CCA optimization, since maximizing the correlation between two more similar representations is much easier.

In addition, we add a doubly stochastic regularization [15] to the objective, which forces the contextual attention-based LSTM-RNN to pay equal attention to every local region of the image during the dynamical attention process:

$$R=\lambda \sum _{l=1}^L\left(1-\sum _{t=1}^T{p}_{t,l}\right)$$

(15)

where $\lambda$ is a balancing parameter. In our experiments, we find that using this regularization can further improve the performance.

4. Experimental Results

To demonstrate the effectiveness of the proposed RCCA, we perform experiments of image annotation and retrieval on three publicly available datasets.

4.1. Datasets and Protocols

The three evaluation datasets and their corresponding experimental protocols are described as follows. (1) Flickr8k [42] consists of 8000 images collected from the Flickr website, each of which may contain people or animal performing actions, and is accompanied with five sentences describing the content of the image. This dataset provides the standard training, validation and testing splits, with 6000, 1000 and 1000 images, respectively. (2) Flickr30k [43] is an extension of Flicker8K, which consists of 31,783 images collected from the Flickr website. Each image is also accompanied with five sentences, and the sentences are annotated in a similar way as those in Flicker8K. We use the public training, validation and testing splits, which contains 28,000, 1000 and 1000 images, respectively. (3) Microsoft COCO [44] consists of 82,783 training and 40,504 validation images, each of which is associated with five sentences. We use the public training, validation and testing splits [45], with 82,783, 4000 and 1000 images, respectively.

4.2. Experimental Details

During model learning, we use the Stochastic Gradient Descent (SGD) to optimize the parameters, in which the learning rate is 0.005, batch size is 128 and gradient clipping is [−10, 10]. The model is trained for 30 epochs to guarantee its convergence, based on the Pytorch platform [46] and accelerated with a NVIDIA Titan X GPU.

The task of image annotation and retrieval can be jointly formulated as cross-modal retrieval, i.e., given an image, the goal is to retrieve highly matched sentences for annotation, and vice versa. The commonly used evaluation criterions are “$\mathrm{R}@1$”, “$\mathrm{R}@5$” and “$\mathrm{R}@10$”, i.e., recall rate at the top 1, 5 and 10 results. Another one is “Med r” which is the median rank of the first ground truth result. We also compute an additional criterion “Sum” to evaluate the overall performance for both image annotation and retrieval.

4.3. Image and Sentence Matching

To systematically study the modelling of contextual attention mechanism, as shown in Figure 3, we propose four variants of RCCA: (a) RCCA-nc does not use any context information at all the time steps in the contextual attention-based LSTM-RNN, (b) RCCA-fc uses the context information only at the first time step, (c) RCCA-lc uses the context information only at the last time step, and (d) RCCA-na dose not use the attention mechanism but only the context information. We denote RCCA-ac as the original model in Figure 1 which uses the context information at all the time steps. We also develop an ensemble model ${\mathrm{RCCA} }^{*}$ in a similar way as [8], by summing five cross-modal similarity matrices generated from the five RCCAs together,

For all the RCCAs, we use the 19-layer VGG Network [21] to extract 512 feature maps (with a size of 14 × 14) in “conv5-4” layer as multiple local features, and a feature vector in “fc7” layer as the context feature. Therefore, the dimensions of local and context features are F = 512 and D = 4096, respectively, and the total number of local regions is L = 196. Some hyperparameters of RCCAs are illustrated as follows: $D_x$ = 1024, $D_y$ = 1024, ${\lambda }_x$ = $0.1$, ${\lambda }_y$ = $0.1$, $\lambda$ = 10, K = 100, U = 300 and m = $0.2$. For the pretraining, we find that using 3 pretraining epoches can always achieve satisfactory performance. The number of time steps in the contextual attention-based LSTM-RNN is T=3 (more details in Section 4.4).

We compare RCCAs with several previous state-of-the-arts on the three datasets in

Table 1. Comparison results of image annotation and retrieval on the Flickr8K dataset ($*$ indicates ensemble or multi-model methods).

Method	Image Annotation				Image Retrieval				Sum
	R@1	R@5	R@10	Med r	R@1	R@5	R@10	Med r
DeViSE [1]	4.8	16.5	27.3	28.0	5.9	20.1	29.6	29	104.2
SDT-RNN [5]	6.0	22.7	34.0	23.0	6.6	21.6	31.7	25	122.6
Deep Fragment [2]	12.6	32.9	44.0	14	9.7	29.6	42.5	15	171.3
RVP (T+I) [6]	11.7	34.8	48.6	11.2	11.4	32.0	46.2	11	184.7
m-RNN [45]	14.5	37.2	48.5	11	11.5	31.0	42.4	15	185.0
DCCA [3]	17.9	40.3	51.9	9	12.7	31.2	44.1	13	197.9
DVSA (BRNN) [10]	16.5	40.6	54.2	7.6	11.8	32.1	44.7	12.4	199.9
MNLM [7]	18.0	40.9	55.0	8	12.5	37.0	51.5	10	214.9
NIC [47]	20.0	-	61.0	6	19.0	-	64.0	5	-
m-CNN-st [8]	18.1	44.1	57.9	7	14.6	38.5	53.5	9	226.7
${\mathrm{m-CNN} }^{*}$ [8]	24.8	53.7	67.1	5	20.3	47.6	61.7	5	275.2
FV(HGLMM) [4]	28.5	58.4	71.7	4	20.6	49.4	64.0	6	292.6
${\mathrm{FV} }^{*}$ [4]	31.0	59.3	73.7	4	21.3	50.0	64.8	5	300.1
Ours:
RCCA-nc	19.5	44.8	58.2	7	14.7	39.9	53.0	9	230.1
RCCA-na	22.8	48.9	63.3	6	16.6	41.1	54.0	9	246.7
RCCA-fc	23.3	49.4	64.0	6	17.0	41.2	53.5	8	248.4
RCCA-lc	25.6	51.9	65.4	5	17.7	41.2	53.7	9	255.5
RCCA-ac	26.5	55.0	67.9	4	18.2	45.3	58.2	7	271.1
${\mathrm{RCCA} }^{*}$-f30k	28.1	56.1	66.3	4	18.5	43.8	59.6	7	272.4
${\mathrm{RCCA} }^{*}$-coco	29.0	57.7	68.2	4	19.7	45.3	61.1	6	281.0
${\mathrm{RCCA} }^{*}$	30.3	59.6	69.7	3	20.6	47.9	62.1	6	290.2

,

Table 2. Comparison results of image annotation and retrieval on the Flickr30K dataset ($*$ indicates ensemble or multi-model methods, and † indicates using manual annotations).

Method	Image Annotation				Image Retrieval				Sum
	R@1	R@5	R@10	Med r	R@1	R@5	R@10	Med r
DeViSE [1]	4.5	18.1	29.2	26	6.7	21.9	32.7	25	113.1
SDT-RNN [5]	9.6	29.8	41.1	16	8.9	29.8	41.1	16	160.3
RVP (T+I) [6]	12.1	27.8	47.8	11	12.7	33.1	44.9	12.5	178.4
Deep Fragment [2]	14.2	37.7	51.3	10	10.2	30.8	44.2	14	188.4
DCCA [3]	16.7	39.3	52.9	8	12.6	31.0	43.0	15	195.5
NIC [47]	17.0	-	56.0	7	17.0	-	57.0	7	-
DVSA (BRNN) [10]	22.2	48.2	61.4	4.8	15.2	37.7	50.5	9.2	235.2
MNLM [7]	23.0	50.7	62.9	5	16.8	42.0	56.5	8	251.9
LRCN [48]	-	-	-	-	17.5	40.3	50.8	9	-
m-RNN [45]	35.4	63.8	73.7	3	22.8	50.7	63.1	5	309.5
FV (HGLMM) [4]	34.4	61.0	72.3	3	24.4	52.1	65.6	5	309.8
${\mathrm{FV} }^{*}$ [4]	35.0	62.0	73.8	3	25.0	52.7	66.0	5	314.5
m-CNN-st [8]	27.0	56.4	70.1	4	19.7	48.4	62.3	6	283.9
${\mathrm{m-CNN} }^{*}$ [8]	33.6	64.1	74.9	3	26.2	56.3	69.6	4	324.7
${\mathrm{RTP} }^{†*}$ [9]	37.4	63.1	74.3	-	26.0	56.0	69.3	-	326.1
Ours:
RCCA-nc	27.5	53.3	66.9	5	20.9	46.7	58.8	7	274.1
RCCA-na	34.4	61.6	71.2	3	23.9	51.1	61.8	5	304.0
RCCA-fc	32.2	60.4	72.3	3	23.7	53.1	65.0	5	306.7
RCCA-lc	32.2	61.2	72.4	3	23.9	53.4	65.8	5	308.9
RCCA-ac	36.0	65.8	75.6	3	25.8	53.9	65.7	5	322.8
${\mathrm{RCCA} }^{*}$	39.3	68.7	78.2	2	28.7	57.2	69.8	4	341.9

and

Table 3. Comparison results of image annotation and retrieval on the Microsoft COCO dataset ($*$ indicates ensemble or multi-model methods).

Method	Image Annotation				Image Retrieval				Sum
	R@1	R@5	R@10	Med r	R@1	R@5	R@10	Med r
STD (bi-skip) [49]	32.7	67.3	79.6	3	24.2	57.1	73.2	4	334.1
${\mathrm{STD} }^{*}$ [49]	33.8	67.7	82.1	3	25.9	60.0	74.6	4	344.1
m-RNN [45]	41.0	73.0	83.5	2	29.0	42.2	77.0	3	345.7
FV (HGLMM) [4]	37.7	66.6	79.1	3	24.9	58.8	76.5	4	343.6
${\mathrm{FV} }^{*}$ [4]	39.4	67.9	80.9	2	25.1	59.8	76.6	4	349.7
DVSA [10]	38.4	69.9	80.5	1	27.4	60.2	74.8	3	351.2
MNLM [7]	43.4	75.7	85.8	2	31.0	66.7	79.9	3	382.5
m-CNN-st [8]	38.3	69.6	81.0	2	27.4	63.4	79.5	3	359.2
${\mathrm{m-CNN} }^{*}$ [8]	42.8	73.1	84.1	2	32.6	68.6	82.8	3	384.0
Ours:
RCCA-nc	37.4	70.3	81.5	2	29.7	65.3	79.8	3	364.0
RCCA-na	40.7	71.0	84.6	2	32.9	68.8	81.0	3	379.0
RCCA-fc	41.9	73.5	84.1	2	33.4	68.1	81.8	3	382.8
RCCA-lc	42.3	77.8	87.9	2	34.3	68.4	81.0	3	391.7
RCCA-ac	44.9	79.6	87.7	2	35.8	71.2	83.3	2	402.5
${\mathrm{RCCA} }^{*}$	49.4	80.1	89.5	2	37.9	73.5	84.9	2	415.3

, respectively. All the compared methods can be classified into two classes: (1) global matching and (2) local matching. The global matching methods [1,3,4,6,7,45,47,48] directly learn global representations for image and sentence, and then associate them by computing the global similarity in an one-to-one manner. While the local matching methods [2,5,8,9,10] first detect local objects and words in the image and sentence, and then summarize their local similarities to finally obtain the global similarity.

From Table 1, we can see that the pretraining results can already perform better than most compared methods. Then after CCA optimization, the performance of all the RCCAs can be further improved. Note that our best ${\mathrm{RCCA} }^{*}$ still performs a little worse than ${\mathrm{FV} }^{*}$. It is mainly attributed to the factor that our model has a large number of learning parameters which cannot be well fitted using only the limited number of training data on the Flickr8k dataset.

When exploiting large-scale training data on the Flickr30k (an extension of Flickr8k) and Microsoft COCO datasets in Table 2 and Table 3, ${\mathrm{RCCA} }^{*}$ can achieve much better performance than all the compared methods. Our best single model RCCA-ac outperforms both the best single model FV (HGLMM) on the Flickr30k dataset, and the best single model m-CNN-st on the Microsoft COCO dataset. These observations demonstrate that and dynamically learning representations for images is more suitable for association analysis.

When comparing among all the RCCAs, we can conclude as follows. (1) Using context information to modulate attention mechanism is very necessary, since RCCA-fc, RCCA-lc and RCCA-ac all perform much better than RCCA-nc. (2) Exploiting only context information without the attention mechanism, RCCA-na achieve worse results than RCCA-fc, RCCA-lc and RCCA-ac, which verifies the effectiveness of the attention mechanism. (3) RCCA-ac performs the best among all the RCCA variants, which indicates that the best way of modelling the context is using dense context information at all the time steps. (4) The ensemble of the five RCCA variants ${\mathrm{RCCA} }^{*}$ can greatly improve the performance.

To evaluate the cross-dataset generalization of the proposed model, we use the train splits of Flickr30K and Microsoft COCO datasets to train two models: ${\mathrm{RCCA} }^{*}$-f30k and ${\mathrm{RCCA} }^{*}$-coco, respectively, and then perform test on the test split of Flickr8K dataset. We can see that either ${\mathrm{RCCA} }^{*}$-f30k or ${\mathrm{RCCA} }^{*}$-coco achieves worse performance than the original ${\mathrm{RCCA} }^{*}$. It mainly results from the fact that the image and sentence contents are quite different between the two datasets. ${\mathrm{RCCA} }^{*}$-coco performs much better than ${\mathrm{RCCA} }^{*}$-f30k, because the train split of Microsoft COCO has a larger size than that of Flickr30K.

4.4. Analysis of the Number of Time Steps

For a sequential sentence, the number of time steps in conventional LSTM-RNN is naturally the length of the sentence. While for a static image, we have to manually set the number of time steps T. Ideally, T should roughly equal to the number of salient objects appeared in the image, so that the model can separately attend to these objects within T steps to collect all information. In the following, we gradually increase T from 1 to 9, and testify the impact of different numbers of time steps on the performance of RCCA-ac and RCCA-nc.

From

Table 4. Different numbers of time steps on the Microsoft COCO dataset. T: number of time steps in contextual attention-based LSTM-RNN.

Method	Image Annotation				Image Retrieval				Sum
	R@1	R@5	R@10	Med r	R@1	R@5	R@10	Med r
RCCA-ac:
$T=1$	42.5	74.9	86.0	2	32.1	67.7	81.4	3	384.6
$T=3$	44.9	79.6	87.7	2	35.8	71.2	83.3	2	402.5
$T=5$	44.0	78.0	86.8	2	35.4	71.0	83.1	2	398.3
$T=7$	44.1	76.9	86.7	2	35.5	71.1	83.2	2	397.5
$T=9$	43.2	76.5	87.1	2	34.7	71.0	83.1	2	395.6
RCCA-nc:
$T=1$	38.5	70.1	80.9	2	28.0	63.2	76.1	3	356.8
$T=3$	37.4	70.3	81.5	2	29.7	65.3	79.8	3	364.0
$T=5$	34.9	67.1	78.9	3	28.8	64.5	77.5	3	351.7
$T=7$	36.2	65.6	77.9	3	28.9	64.0	77.2	3	350.9
$T=9$	32.6	64.7	76.6	3	27.1	62.6	75.7	3	339.2

we can observe that both RCCA-ac and RCCA-nc achieve their best performance when the number of time steps is 3. It indicates that they can get all the important information in an image by iteratively visiting the image three times. Intuitively, most images usually contain at most three objects, which is in consistent with the optimal number of time steps. Note that when T becomes larger than 3, the performance of RCCA-ac and RCCA-nc slightly drops, which results from the fact that an overly complex network architecture could lead to overfitting. When comparing RCCA-ac with RCCA-nc, we find that the usage of context information can greatly improve the performance in all cases, regardless of the number of time steps. Especially for image annotation, with the aid of context information, RCCA-ac seems to exploit more time steps to attend to more contents of the image so that it can produce more accurate results.

4.5. Visualization of Dynamical Attention Maps

To verify whether the proposed model can attend to salient regions of an image at different time steps, we visualize the predicted dynamical attention maps on the contextual attention-based LSTM-RNN. For the predicted probabilities at the t-th time step ${\left\{p_{t,l}\right\}}_{l=1,\cdots ,196}$, we first reshape it to a probabilistic map, with the size of 14 × 14. Its layout is equivalent to that of feature maps extracted in “conv5-4” of the VGG Network. Then we resize the probabilistic map to the same size as its corresponding original image, so that each probability in the resized map measures the importance of an image pixel at the same location. We thus perform element-wise multiplication between the resized probabilistic map and its corresponding image to obtain an attention map, where lighter areas indicate attended regions.

Dynamical attention maps at three successive time steps by RCCA-ac and RCCA-nc are shown in

Table 5. Visualization of dynamical attention maps on the Microsoft COCO dataset.

Input Image	RCCA-ac			RCCA-nc
	1st Step	2nd Step	3rd Step	1st Step	2nd Step	3rd Step

. We can see that RCCA-ac attends to different objects in the images at different time steps. Note that without the aid of context information, RCCA-nc cannot produce accurate dynamical attention maps as those of RCCA-ac. Particularly, it cannot well attend to the “cow” and “clock” in the two images, respectively. In fact, RCCA-nc always finishes attending to salient objects within the first two steps, and does not focus on meaningful regions at the third step any more. Different from it, RCCA-ac uses more steps to focus on more salient objects, which is also inconsistent with the observation in Section 4.4. These evidences again demonstrate that the modelling of context can greatly facilitate the attention mechanism.

4.6. Error Analysis

By analyzing the results by our proposed model, we find it cannot well generalize to those images containing very complex content. To illustrate this, we present the results of image retrieval by sentence queries in Figure 4, where the numbers in the top left corner are returned ranks (the smaller, the better) of groundtruth images. We can see that the ranks of left three examples are very high and the corresponding image contents are relatively simple. While the ranks of right three examples are very high, which indicates that our model cannot find matched images. According to a recent state-of-the-art [50], it mainly because the complex image contents include many few-shot objects and attributes, which cannot be well associated with few-shot words in the sentences.

5. Conclusions and Future Work

In this paper, we have proposed the recurrent canonical correlation analysis (RCCA) method for associating images with sentences. Our main contribution is the modelling of contextual attention mechanism for dynamically learning image representations. We have applied our model to image annotation and retrieval, and achieved better results.

In the future, we will exploit the similar attention-based scheme to learn good representations for sentences, and further verify its effectiveness on more datasets. In addition, we will try to combine the two tasks of image and sentence matching and image phrase localization into a joint framework to exploit the complementary advantages of global and local matching. To presuit higher performance, we will also consider to exploit the more detailed annotations in Flickr30k Entity [9] between objects and words for fine-grained image and sentence matching.