Self-Supervised Video Representation and Temporally Adaptive Attention for Audio-Visual Event Localization

Abstract

Localizing the audio-visual events in video requires a combined judgment of visual and audio components. To integrate multimodal information, existing methods modeled the cross-modal relationships by feeding unimodal features into attention modules. However, these unimodal features are encoded in separate spaces, resulting in a large heterogeneity gap between modalities. Existing attention modules, on the other hand, ignore the temporal asynchrony between vision and hearing when constructing cross-modal connections, which may lead to the misinterpretation of one modality by another. Therefore, this paper aims to improve event localization performance by addressing these two problems and proposes a framework that feeds audio and visual features encoded in the same semantic space into a temporally adaptive attention module. Specifically, we develop a self-supervised representation method to encode features with a smaller heterogeneity gap by matching corresponding semantic cues between synchronized audio and visual signals. Furthermore, we develop a temporally adaptive cross-modal attention based on a weighting method that dynamically channels attention according to the time differences between event-related features. The proposed framework achieves state-of-the-art performance on the public audio-visual event dataset and the experimental results not only show that our self-supervised method can learn more discriminative features but also verify the effectiveness of our strategy for assigning attention.

1. Introduction

Teaching machines to use captured signals, such as visual and acoustic signals, to understand their surroundings is essential for constructing artificial intelligence. At present, studies in scene perception such as action recognition [1,2,3] and sound event detection [4,5], are mainly based on the use of unimodal signals. However, in more realistic situations, it may not be sufficient to characterize certain scenarios merely with unimodal information. To cope with the ambiguity caused by the unimodal data, the task of audio-visual event (AVE) localization is proposed to investigate how to understand video content by jointly leveraging audio and visual information in neural networks [6]. Specifically, an AVE refers to an event that is both visible and audible in a video segment. Localizing an AVE in a video requires predicting its temporal boundary and identifying its content. For example, as shown in Figure 1, a car appears in the beginning but can only be heard in the latter half of the video after its engine is started, thereby only the last three segments are categorized as ‘car’ while the other segments are background.

Due to the natural heterogeneity gap that exists between visual and acoustic signals, the key challenge in AVE localization is how to effectively fuse the information contained in the two modalities. To address the challenge, previous works [6,7,8,9,10,11,12,13] regarded primary visual and audio features extracted by unimodal backbones as tokens and fed them into attention modules to model the cross-modal relationships. However, the features they used are encoded in different unimodal spaces since they are extracted separately by convolutional neural networks (CNNs) trained solely on single-modal datasets [14,15]. Thus, there is a large heterogeneity gap between these features, which makes it difficult to model the relationship between modalities with only a few sets of scalar weights in previous attention networks. Unlike these deep learning techniques that use primary features learned on separate audio and visual data, the studies in [16,17,18] show that the human perceptual system naturally receives signals from multiple sensory streams and associates multimodal information collaboratively at an early stage of the learning process. Therefore, our approach is motivated to train feature encoders with both audio and visual data and associate their features in the same semantic space to narrow down the heterogeneity gap.

Another concern laid in existing localization frameworks is the attention module, which plays a pivotal role in learning cross-modal relationships. Generally, using the attention module to obtain a joint representation fusing audio and visual information can facilitate localization accuracy [8,10,11,12,13]. Nevertheless, these existing approaches often ignored the interference caused by the interaction between event-unrelated visual and audio features that are generated from background segments. Therefore, the approach in [19] applied a threshold on the cross-modal connections to filter out semantically similar feature pairs that share event-related information and used them to obtain a better-fused representation. However, the thresholding method treats all relevant feature pairs equally, ignoring that some of these pairs are temporally asynchronous. We argue that the asynchronous audio and visual feature pairs should also be distinguished since they inherently describe different states of the event. In fact, studies [20,21] on psychophysics have already proven that the temporal misalignment between vision and hearing can lead to the misinterpretation of one modality by another. Furthermore, the study [22] on multisensory representation learning also shows that judging whether visual and audio signals are synchronized can be used to learn representative multimodal features. Hence, our idea is to adaptively assign attention to event-related audio-visual connections by considering the time difference between modalities.

In light of the above analysis, this paper proposes a novel AVE localization framework consisting of two parts: a self-supervised audio-visual representation method called audio-visual Barlow Twins (AV-BT), and temporally adaptive cross-modal attention (TACMA). On the one hand, AV-BT takes both audio and visual data as inputs and trains the feature encoders using the intrinsic relation between audio and visual signals as supervision. It helps to narrow the heterogeneity gap between the encoded features by matching the corresponding semantic information between synchronized audio and visual embeddings. On the other hand, a weighting method is introduced in TACMA to adaptively model the cross-modal relations by assigning different weights to audio-visual pairs with different time differences and semantic similarities. Specifically, the most attention is assigned between the synchronized visual and audio features which describe the same state of the event since they are the most relevant, while the association between those asynchronous signals (even if both contain the information of the same event) is relatively weakened because they are less relevant. The main contributions of this work are concluded as follows:

• We propose a self-supervised audio-visual representation method that encodes features in the same latent space and matches the semantic information contained in the audio and visual modalities. Such a design can narrow the heterogeneity gap between different modal features and benefit subsequent modeling of cross-modal relationships;
• We propose a weighting-based cross-modal attention module that dynamically weakens the connections between different modal features that are unrelated to events or temporally asynchronous;
• The proposed methods are combined in a framework to perform the event localization task on the public audio-visual event dataset. When directly classifying features learned by our self-supervised method for event localization, significant improvements are achieved in both unimodal and multimodal cases. When further combining these features with the proposed cross-modal attention, our overall approach achieves state-of-the-art localization accuracy.

The rest of this paper is organized as follows: Section 2 briefly introduces several related works about our method. In Section 3, the detailed system flow and discussions of AV-BT and TACMA are presented. Both the quantitative and qualitative results of our methods are given in Section 4. Conclusions and future directions are presented in Section 5.

2. Related Works

Based on the task and method of this paper, related works are briefly reviewed in three aspects: (1) Self-supervised audio-visual representation learning. (2) Multimodal fusion. (3) AVE localization.

2.1. Self-Supervised Audio-Visual Representation Learning

Self-supervised representation learning aims to train representation networks in the absence of human labeling. Existing self-supervised methods [23,24,25,26,27,28] fed two randomly distorted versions of an image into a backbone and maximized the similarity of their projected features. Their ideas of learning representations that are both invariant to random augmentations and distinctive to semantically different targets, inspire many works on self-supervised audio-visual learning [22,29,30,31,32,33,34].

Mimicking the self-supervised method for single-modal representation learning, some existing methods focus on designing pretext tasks, trying to exploit the cross-modal correlation effectively. Considering the co-occurrence of hearing and sounding, Owens et al. [22] proposed to predict whether video frames and audio are temporally aligned based on fused multisensory representations. By solving the pretext problem of audio-visual synchronization, Cheng et al. [29] trained a co-attention network in a self-supervised manner. Sarkar et al. [30] find that by relaxing the temporal synchronicity between modalities, more generalized representations can be learned. Different from defining a pretext task, Patrick et al. [31] shifted their attention to imposing transformations on multimodal data and combining them with existing noise-contrastive learning methods and achieved astonishing results on several downstream tasks. These studies show that a carefully designed self-supervised framework can help the network effectively uncover the joint representation of multimodal information by exploiting the general connection between different modal data. In our work, we develop a framework that matches the corresponding semantic information between synchronized audio and visual signals to obtain multimodal features that have a smaller heterogeneity gap.

2.2. Multimodal Fusion

In the real world, many realistic tasks involve inputs of multimodal information that are complementary to each other. For example, the combination of speaking tone, the content of speech, and facial expressions are critical to properly judging a person’s emotion. Therefore, integrating cues from different modalities, known as multimodal fusion, is necessary for building a stronger multimodal neural network to tackle real-world problems. Here we mainly review some of the commonly used fusion techniques in neural networks.

For information fusion in neural networks, many simple operations can be exploited, such as direct concatenation [35,36], element-wise multiplication [37,38], and weighted sums [39]. Since useful information about the data is often expressed in a highly abstract fashion in neural networks (which can be linearly related to the output predictions), applying these simple operations before the classification layer to directly combine high-level information from different sources can be effective. However, these simple operations often play limited roles because the complex relationships that may exist between different modalities can affect the representation of the data. Hence, modules that utilize the attention mechanism attract more interest [40,41]. Computed from multimodal cues, attention blocks that employ sets of scalar weights are more capable when modeling both inter-modal and intra-modal relationships. Lu et al. [42] proposed a co-attention model to extract correlated information from both visual and text modalities. Dou et al. [43] developed a merged-attention mechanism to ease the computational cost while also avoiding modeling the redundant information between modalities. In our solution, we do not restrict the information fusion process to a particular module. In early fusion, we encode informative multimodal features by self-supervised representation learning. For late fusion, we modify the process of cross-modal attention to make it more attentive to useful parts. In the classification layer, we directly employ summation for decision-level fusion.

2.3. AVE Localization

Nowadays, with the development of machine learning and deep learning, there are more and more cases of neural network techniques being used in practical applications [44]. As single-modal deep learning is becoming increasingly mature in applications [45,46,47,48], the machine learning community has turned to multimodal cases. For video content understanding based on visual and audio modalities, Tian et al. [6] first defined the AVE localization task, which is to detect whether an event that is both visible and audible happened in a video segment. Other than task definition, they also propose a complete set of solutions and initially study several basic fusion strategies of features. As the encoded feature of a video is sequential, Lin et al. [9] employ a bidirectional long short-term memory (Bi-LSTM) [49] network to better fuse the concatenated audio-visual feature. Although the localization task is built on the segment level, Wu et al. [7] consider the potential intra-modal misalignment within the segment so they propose a dual attention-matching module to model high-level global information over a longer range. Note that in their method, to compute global features, background segments need to be additionally supervised during training. Furthermore, in [10,11], Ramaswamy introduced several attention-based modules to interact between audio and visual both locally and globally. Xuan et al. [12] presented a different network with multiple attention modules to deal with the task under temporal inconsistency. Duan et al. [13] took a further step on multimodal attention and utilized joint co-attention for modeling both inter-modal and intra-modal relations, aiming to learn robust fused features by recursively stacking attention blocks. Lin et al. [50] complete the encoding of temporal features and information fusion simultaneously through their newly proposed audio-visual transformer. The above works emphasize a variety of attention-based designs, resulting in a large increase in computational cost when improving performance. Zhou et al. [19] revisit the localization task from a more concise perspective and propose a positive sample propagation module to only aggregate information from high-relevance audio and visual features. This cross-modal attention-based module selects positive audio-visual pairs that are of high attention scores through threshold operation. However, it cannot be ignored that most of the features that carry event-related information are inconsistent in time. Since this inconsistency reflects the fact that features from different modalities describe different states of the event, these asynchronous features should be distinguished from those feature pairs that are both synchronous and associated. Therefore, we apply weights on the attention map based on the time difference to integrate information more effectively by weakening the less relevant connections caused by inter-modal asynchrony.

3. The Proposed Framework

The whole scheme of our framework is depicted in Figure 2, in which our network is trained in two stages. First, the two-stream representation network is trained in a self-supervised manner to encode spatial–temporal features with the proposed AV-BT. At this point, the heterogeneity gap between features from different modalities is narrowed. Then, the extracted features are fed into the proposed TACMA to adaptively integrate useful information from each modality. To start with the description, the task statement is outlined in Section 3.1. Next, the detailed system flow and discussion of the AV-BT and TACMA are described in Section 3.2 and Section 3.3, respectively.

3.1. Task Statement

The meanings of the symbols used throughout this paper are provided in

Table 1. Main symbols used throughout the paper.

Symbols	Definition
$S_a$/$S_v$	Audio/visual portion of the video
${\mathit{y}}^t$	Annotation of the $t^{}$-th video segment
$\mathit{A}$/$\mathit{V}$	Preprocessed audio/visual data
${\mathit{f}}_a$/${\mathit{f}}_v$	Encoded audio/visual feature
${\mathit{Z}}_a$/${\mathit{Z}}_v$	Projected audio/visual embedding
$\mathcal{C}$	Cross-correlation matrix of ${\mathit{Z}}_a$/${\mathit{Z}}_v$
${\mathit{W}}_a^1$/${\mathit{W}}_v^1$	Parameters for computing audio/visual query and key
${\mathit{M}}^{av}$/${\mathit{M}}^{va}$	Cross-modal attention matrix
${\mathit{W}}^{mv}$/${\mathit{W}}^{ma}$	Weight matrix for ${\mathit{M}}^{av}$/${\mathit{M}}^{va}$
${\mathit{W}}^{av}$/${\mathit{W}}^{va}$	Weighted attention matrix
${\mathit{W}}_a^2$/${\mathit{W}}_v^2$	Parameters for computing audio/visual value
${\mathit{A}}^{fuse}$/${\mathit{V}}^{fuse}$	Attended audio/visual feature
${\mathit{W}}_a^3$/${\mathit{W}}_v^3$	Parameters for ${\mathit{A}}^{fuse}$/${\mathit{V}}^{fuse}$
${\mathit{f}}_{a\leftarrow v}$/${\mathit{f}}_{v\leftarrow a}$	Fused audio/visual feature
$\mathit{p}\mathit{r}\mathit{e}\mathit{d}$	Prediction of the input video

. Following [6], an AVE is defined as an event in a video clip that is both visible and audible. Performing the AVE localization task involves both predicting the temporal boundaries and identifying the content information. Specifically, given a synchronous audio-visual video sequence $S=\left(S_a,S_v\right)$, where $S_a$ and $S_v$ denote the auditory portion and visual portion, respectively. $S$ is continuously divided into $T$ nonoverlapping segments of equal duration. As in [19,50], the annotation of the $t$-th segment $S^t=(S_a^t,{\mathrm{S}}_v^t$) is defined as ${\mathit{y}}^t=\left\{y_c^t|y_c^t\in \left\{0, 1\right\},{\displaystyle \sum }_c{y}_c^t=1\right\}\in {ℝ}^C$, where C denotes the number of categories. If no event happens in a video segment, it will be labeled as background. Note that in our method, we directly classify the background as one of the categories and use no additional supervision to help the network distinguish it from events. Compared to an AVE, it is noticeably more challenging to identify a background because its audio and visual content are not correlated with each other.

3.2. Self-Supervised Audio-Visual Representation

According to the above definition, the localization of an AVE requires a combination of information from both visual and audio signals. To extract useful information from the raw signal, we followed existing methods [6,7,8,9,10,11,12,13] to extract spatial features in video frames and audio spectrograms using two-dimensional CNN. Considering that the video is sequential, Bi-LSTMs are then used for the modeling of temporal relations.

Due to the scarcity of audio-visual video samples that can be used for training, the performance of training feature encoders from scratch is often unsatisfying, making it necessary to pretrain the CNN on large-scale datasets [14,15]. However, this pretraining also poses a problem: there is a huge heterogeneity gap that exists between the features extracted by the pretrained CNN and it naturally hinders information fusion. To overcome this problem, our solution is to adapt the CNN features from different modalities to the same semantic space. In practice, we accomplish the feature adaptation in a self-supervised manner by exploiting the relationship between vision and hearing as supervision. Specifically, there are two general cross-modal relationships in AVEs, namely, audio-visual co-occurrence and correspondence. On the one hand, co-occurrence refers to the fact that when an AVE happens in a video segment, its visual and auditory stimuli are provided simultaneously. For example, a baby’s cry is accompanied by a sad facial expression. Audio-visual correspondence, on the other hand, means that the appearance of a particular object is associated with its characteristic sound. For example, when a bell appears in a scene, the neural network should match it to the ringing sound in the audio stream at that moment. Utilizing these two attributes of AVE means that the network should match the relevant semantic information carried by the synchronized audio and visual signal pairs.

3.2.1. Audio-Visual Barlow Twins

Based on the above consideration, we modified the pipeline in [23] and design an AV-BT that trains the audio-visual encoder in a self-supervised manner to narrow the heterogeneity gap between different modal data. Figure 2 illustrates the full AV-BT pipeline, with the three steps involved detailed below.

Preprocessing. For each video segment, we convert the raw audio into a log-Mel spectrogram and randomly sample one frame from the visual stream. The resulting spectrograms and extracted frames of the whole video are denoted as $\mathit{A}\in {ℝ}^{T\times H_a\times W_a}$ and $\mathit{V}\in {ℝ}^{T\times H_v\times W_v\times C}$, respectively. Since random data augmentations can boost the performance of self-supervised learning and help to fight against overfitting [23,24,25,26,27,28,51], we employ similar strategies in AV-BT. Specifically, cropping and Gaussian blurring are randomly applied to the extracted visual frames. To simplify notation, $\mathit{V}$ is assumed to be augmented. In addition, the random frame sampling technique also plays a role in data augmentation, as it can provide slight variations in visual content during each training epoch and helps the feature encoder capture more information along the timeline when the visual content remains stable. An in-depth analysis of the random sampling technique is provided in the next subsection.

Feature encoding. $\mathit{A}$ and $\mathit{V}$ are then sent to a two-stream audio-visual encoder simultaneously to learn spatial-temporal features. The output features are denoted as ${\mathit{f}}_a\in {ℝ}^{T\times d_l}$ and ${\mathit{f}}_v\in {ℝ}^{T\times d_l}$, where $d_l$ denotes the feature dimension.

Projecting. Eventually, two three-layer multi-layer perceptrons (MLP) are used as projectors to map the ${\mathit{f}}_a$ and ${\mathit{f}}_v$ into the same latent space, where we compute their cross-correlation matrix afterward to explore the audio-visual relationships. We assign d as the output dimension of these projectors, so the output embeddings are denoted as ${\mathit{Z}}_a\in {ℝ}^{T\times d}$ and ${\mathit{Z}}_v\in {ℝ}^{T\times d}$. To simplify notations, ${\mathit{Z}}_a$ and ${\mathit{Z}}_v$ are assumed to be normalized along the time dimension after projection. It is necessary to note that these two projectors share the same parameters so that they can force the encoders to learn joint representations carrying similar semantic information about the video.

To calculate the objective function of AV-BT, we first compute the cross-correlation matrix $\mathcal{C}\in {ℝ}^{d\times d}$ of the normalized embeddings ${\mathit{Z}}_a$ and ${\mathit{Z}}_v$ along the time dimension. Specifically, the element in $\mathcal{C}$ is obtained by the following equation:

$${\mathcal{C}}_{ij}\triangleq \frac{{{\displaystyle \sum }}_t{Z}_a{}_{t,i}{Z}_v{}_{t,j}}{\sqrt{{{\displaystyle \sum }}_t{(Z_a{}_{t,i})}^2}\sqrt{{{\displaystyle \sum }}_t{(Z_v{}_{t,j})}^2}},$$

(1)

where t indexes the time dimension and $i$, $j$ indexes the feature dimension of ${\mathit{Z}}_a$ and ${\mathit{Z}}_v$ individually. Then we optimize the network by minimizing the difference between $\mathcal{C}$ and the identity matrix, and the objective function is written as:

$${\mathcal{L}}_{AVBT}\triangleq \underset{invariance term}{\underbrace{{{\displaystyle \sum }}_i{(1-{\mathcal{C}}_{ii})}^2}}+\underset{redundancy reduction term}{\underbrace{\lambda {{\displaystyle \sum }}_i{{\displaystyle \sum }}_{j\ne i}{\mathcal{C}}_{ij}{}^2}},$$

(2)

where $\lambda$ is the same hyperparameter in the original method [23] and is used to trade off the importance between the first term (invariance term) and the second term (redundancy reduction term). Note that Equation (1) is defined for each video, and the total loss of a training step is obtained based on the average $\mathcal{C}$ over all videos in a batch. By minimizing ${\mathcal{L}}_{AVBT}$, the representation network is trained to bridge the semantic information between synchronized audio and visual data. Another key point in Equation (1) is that our approximative identity correlation matrix is computed on the video level rather than the segment level. This is because of the existence of the background segment where audio and visual signals are not semantically associated with each other. In-depth analysis of how the objective function ${\mathcal{L}}_{AVBT}$ guides the network to encode informative audio-visual representations is discussed in the next subsection.

The pseudocode of our AV-BT is shown in Algorithm 1.

Algorithm 1 PyTorch-style pseudocode for Audio-Visual Barlow Twins.
# # # # # # # # #	f: two-stream audio-visual encoder and projectors p: preprocess the video lambda: weight on the off-diagonal term N: batch size D: dimensionality of projected embeddings transpose: transpose the last two dimensions of a tensor bmm: matrix-matrix multiplication with batch dimension at first off_dia: off-diagonal elements of a matrix eye: identity matrix
for video in loader: # load a batch with N videos           # extract frames randomly and converts raw audio           spectrograms, frames = p (video)           # encode feature and project           z_a, z_v = f (spectrograms, frames) # [N, T, D]           # normalize along the time dimension           z_a_norm = (z_a – − z_a.mean (1) )/z_a.std (1) # [N, T, D]           z_v_norm = (z_v – − z_v.mean (1) )/z_v.std (1) # [N, T, D]           # compute cross-correlation matrix           c = bmm (transpose (z_a_norm), z_v_norm) # [N, D, D]           c = c.sum (0)/N # [D, D]           # loss           diff = (c − eye (D) ).pow (2) # [D, D]           off_dia (diff) .mul_(lambda)           loss = diff.sum ()           # optimize           loss.backward ()           optimizer.step ()

3.2.2. Discussion on Audio-Visual Barlow Twins

Insight of the${\mathcal{L}}_{\mathit{A}\mathit{V}\mathit{B}\mathit{T}}$. First, we need to briefly explain the original design in [23]. The original objective function was applied to the projected embeddings of two identical networks fed with different distorted versions of an image. The outputs were forced to be invariant to the distortions by the invariance term while the redundancy reduction term decorrelated different components of the embedding so that the outputs comprised as much information as possible. Although our objective function is very close in form to the original one, there are differences and connections in many aspects.

First, the information from both audio and visual modalities needs to be considered for judging an AVE. Therefore, our cross-correlation matrix is computed through different modalities in the same latent space, while the original method only needs to consider one modality. Second, similarly to the different distorted versions of an image, vision and sound can be regarded as two heterogeneous expressions of the same event. Hence in our ${\mathcal{L}}_{AVBT}$, the invariance term makes the features extracted from audio and visual streams both carry semantic information about the underlying target. Note that such a “target” refers to an object that may potentially have both visual and aural attributes. For example, for a person who appears in the scene, the semantic representation of the appearance extracted by the visual encoder should be tightly bound to the representation of the speaking voice extracted by the audio encoder. This is like how humans naturally associate the sound of speech when they see a person talking. In contrast, a table does not make any characteristic sound, so it is not audibly perceptible despite its visual characteristics. Therefore, an object like a table is treated as visual background and the encoder naturally ignores it. Third, since ambient noise and visual background in the environment do not carry meaningful semantic information across modalities, the redundancy reduction term forces the representations of both modalities to contain as little information about irrelevant interference as possible. This is analogous to the idea of the original method.

Examination of random frame sampling technique. It is worth noting that in Equation (1), the cross-correlation matrix is calculated along the time dimension for each sample in a batch (while it was computed along the batch dimension in [23]), which means that only the similarity of visual and audio embeddings from the same moment is measured. This is general to exploit audio-visual co-occurrences but inadequate for AVE representation owing to the possible contextual relevance between asynchronous audio and visual signals. For example, if the $t$-th segment and the ${\left(t-1\right)}^{th}$ segment contain the same AVE, obviously ${\mathit{Z}}_{a,t}$ and ${\mathit{Z}}_{v,t-1}$ are correlated. To fully describe our technique for compensating for this defect, an example is given as follows.

We first denote the $t$-th audio-visual segment pair as $(S_a^t,{\mathrm{S}}_v^t$), and there are N frames in ${\mathrm{S}}_v^t$, denoted as $f_n^t$ where $n\in \left\{1,2, \dots ,N\right\}$. Suppose that we extract $f_1^t$ at random during a training epoch, such that the network is taught to match the information shared between $f_1^t$ and $S_a^t$. Similarly, the network will learn to model pairs such as $S_a^t,f_1^t$ and $S_a^{t-1},f_N^{t-1}$ by training with abundant epochs. Note that $f_1^t$ and $f_N^{t-1}$ are indeed two adjacent frames and only a minor difference exists between them (if visual content has not changed much in this period), such that the association within $S_a^t,f_N^{t-1}$ can be broadcast through $f_1^t$. This technique allows the network to model the relation between adjacent segments to a certain extent. An intuitive example is demonstrated in Figure 3. Moreover, processing only one frame for each segment can help to fit the video memory of the GPU and this random sampling technique also helps to prevent overfitting.

3.3. Temporally Adaptive Cross-Modal Attention

The attention mechanism was first proposed to deal with natural language processing tasks, and it is very capable of modeling the complex relationships within serialized tokens [52]. By considering features from each segment in the video as tokens, the attention module can also be modified to learn cross-modal relationships in AVE [8,10,11,12,13]. Furthermore, since task-related information is often contained in only some of the tokens, a threshold-based approach can better model the relationships between modalities sparsely by obtaining all the semantically similar feature pairs at all times [19]. However, unlike sentences, where words that are far apart may still be highly correlated, the changing state of events in the video is often continuous, namely, the audio (visual) signal at one moment is primarily correlated with its synchronized visual (audio) signal. Therefore, this paper designs a TACMA that makes one modality adaptively assign attention to useful information in another modality by considering their temporal difference. In such a way, the multimodal information is integrated along the timeline in a more fine-grained manner. The detailed structure of our adaptive attention is depicted in Figure 4.

Given the fixed ${\mathit{f}}_a$ and ${\mathit{f}}_v$ output from feature encoders as input, the first step is to compute the attention matrices ${\mathit{M}}^{av}\in {ℝ}^{T\times T}$ and ${\mathit{M}}^{va}\in {ℝ}^{T\times T}$ by the scaled dot-product along the feature dimension:

$${\mathit{M}}^{av}=({\mathit{f}}_a{\mathit{W}}_a^1){({\mathit{f}}_v{\mathit{W}}_v^1)}^{{\rm T}}/\sqrt{d_l}, {\mathit{M}}^{va}={\left({\mathit{M}}^{av}\right)}^{{\rm T}},$$

(3)

where ${\mathit{W}}_a^1$ and ${\mathit{W}}_v^1\in {ℝ}^{d_l\times d_h}$ are learnable weight matrices implemented as linear layers. Note that we adopt a similar strategy as in [19] that uses the same set of queries and keys (which are both the linear projections of input features) in the computation of the attention matrices. This can help to reduce computational costs and ensure efficiency. ReLU activation is applied on ${\mathit{M}}^{av}$ and ${\mathit{M}}^{v\mathrm{a}}$ afterward to filter out negatively correlated audio and visual feature pairs. Then, the softmax function is performed along each row of ${\mathit{M}}^{av}$ and ${\mathit{M}}^{v\mathrm{a}}$ to rescale the values between 0 and 1. The larger the elements in ${\mathit{M}}^{av}$ and ${\mathit{M}}^{v\mathrm{a}}$ are, the more event-related information that is shared between their corresponding audio and visual features.

To weaken the connections between features that share no event-related information or are temporally asynchronous, we weigh the scores of all the feature pairs in the attention matrices. Particularly, given $M_{i,j}^{av}$ and $M_{i,j}^{va}$ as the elements in the $i$-th row and the $j$-th column of the attention matrices, where $i$, $j$ also index the time dimension of audio and visual features, respectively, their corresponding weights $W_{i,j}^{mv}$ and $W_{i,j}^{ma}$ are

$$\{\begin{array}{c}{W}_{ij}^{mv}\triangleq exp\left(-\theta \left|i-j\right|/M_{i,j}^{av}\right) , \\ W_{ij}^{ma}\triangleq exp\left(-\theta \left|i-j\right|/M_{i,j}^{va}\right) ,\end{array}$$

(4)

where $\theta$ is a hyperparameter and the magnitudes of $W{ }_{i,j}^{mv}$ and $W_{i,j}^{ma}$ are adjusted by the ratio of $\theta$ to the corresponding element in ${\mathit{M}}^{av}$ and ${\mathit{M}}^{va}$. Hence, the overall weighting method is expressed as the element-wise product, written as:

$$\{\begin{array}{c}{\mathit{W}}^{av}={\mathit{W}}^{mv}\odot {\mathit{M}}^{av} ,\\ {\mathit{W}}^{va}={\mathit{W}}^{ma}\odot {\mathit{M}}^{va},\end{array}$$

(5)

where ${\mathit{W}}^{mv}\in {ℝ}^{T\times T}$ and ${\mathit{W}}^{ma}\in {ℝ}^{T\times T}$ are the weighting matrices, and ${\mathit{W}}^{av}\in {ℝ}^{T\times T}$ and ${\mathit{W}}^{va}\in {ℝ}^{T\times T}$ are the resulting attention matrices. With the above equations, we generally weaken those connections between asynchronous audio and visual signals, but also retain the consideration of the semantic similarity between features. That is to say when an event in the video continues for a long time, i.e., when the features of different modalities carry similar information over a long period, we do not weaken these connections very substantially (even if some of the audio and visual features are far apart). In contrast, if adjacent features of different modalities do not share useful information about the event, then the connections between them are still significantly weakened.

Once the attention matrices are adjusted, ${\mathit{W}}^{av}$ and ${\mathit{W}}^{va}$ are used to update the linear projected audio and visual features (which are also known as the value in the attention mechanism) by matrix multiplication, and features from each modality are summed together to further reduce their differences in amplitude and direction. The vectors that carry fused information are updated as

$$\{\begin{array}{c}\begin{array}{c}{\mathit{V}}^{fuse}\end{array}={\mathit{W}}^{va}\left({\mathit{f}}_v{\mathit{W}}_v^2\right)+{\mathit{f}}_a ,\\ {\mathit{A}}^{fuse}={\mathit{W}}^{av}\left({\mathit{f}}_a{\mathit{W}}_a^2\right)+{\mathit{f}}_v ,\end{array}$$

(6)

where ${\mathit{W}}_a^2$, ${\mathit{W}}_v^2\in {ℝ}^{d_l\times d_l}$ and ${\mathit{V}}^{fuse}$, ${\mathit{A}}^{fuse}\in {ℝ}^{T\times d_l}$. Before being sent to the classification layer, we linear transform ${\mathit{V}}^{fuse}$ and ${\mathit{A}}^{fuse}$ followed by layer normalization, written as:

$$\{\begin{array}{c}{\mathit{f}}_{v\leftarrow a}=layernorm\left({\mathit{V}}^{fuse}{\mathit{W}}_v^3\right),\\ {\mathit{f}}_{a\leftarrow v}=layernorm\left({\mathit{A}}^{fuse}{\mathit{W}}_a^3\right) \end{array}$$

(7)

where ${\mathit{W}}_a^3$, ${\mathit{W}}_v^3\in {ℝ}^{d_l\times d_e}$ and ${\mathit{f}}_{v\leftarrow a}$, ${\mathit{f}}_{a\leftarrow v}\in {ℝ}^{T\times d_e}$. Compared to AV-BT, TACMA can further improve the overall performance especially when the visual content changes rapidly since it constructs all pair relationships by attention matrices.

3.4. Classification and Optimization

After the information from both modalities is further fused by TMCMA, we simply add ${\mathit{f}}_{v\leftarrow a}$ and ${\mathit{f}}_{a\leftarrow v}$ together and the final prediction is made by a three-layer MLP, written as follows:

$$\mathit{p}\mathit{r}\mathit{e}\mathit{d}=MLP\left({\mathit{f}}_{v\leftarrow a}+{\mathit{f}}_{a\leftarrow v}\right)$$

(8)

where $\mathit{p}\mathit{r}\mathit{e}\mathit{d}\in {ℝ}^{T\times C}$. Since the ground truth is denoted as ${\mathit{Y}}^{gt}=\left[{\mathit{y}}^0,{\mathit{y}}^1,\dots ,{\mathit{y}}^{T-1}\right] \in {ℝ}^{T\times C}$, we minimize the cross entropy (CE) loss between $\mathit{p}\mathit{r}\mathit{e}\mathit{d}$ and ${\mathit{Y}}^{gt}$ to optimize our adaptive attention network during training, written as

$${\mathcal{L}}_{CE}=\frac{1}{TC}{\displaystyle \sum }_{t=1}^T{\displaystyle \sum }_{c=1}^C{\mathit{Y}}_{t,c}^{gt}log\left(\mathit{p}\mathit{r}\mathit{e}{\mathit{d}}_{t,c}\right)$$

(9)

where $t$ and $c$ index the time dimension and category, respectively. Compared to the threshold-based attention in [19], our method does not require additional loss items to supervise the challenging background segment but only uses the basic CE loss.

4. Experiments

4.1. Experiment Setup

Dataset. For the AVE localization task, we conduct our experiments on the AVE dataset which is introduced in [6] and consists of 4143 videos, covering 28 event categories (e.g., human activities, instrument playing, vehicle sounds, etc.). All videos are temporally labeled with audio-visual boundaries on the segment level. We follow the default split where the train/validation/test sets are 3339/402/402 clips, respectively. All clips are divided into 10 segments, with each segment lasting for one second. Whether training AV-BT or TACMA, both networks only have access to the training set.

Evaluation. We follow the routing in [7,10,11,13,19,50] and consider global segment-wise classification accuracy as the evaluation metric. Specifically, for evaluating TACMA, the predictions made by the classification layer are used for evaluation. To evaluate the audio-visual feature learned by AV-BT, we train another three-layer MLP to directly produce predictions based on the fixed feature without any other late fusion modules. All of our results are reported on the test set.

Architecture. During preprocessing before feature extraction, we sample 160 RGB frames at a resolution of 256 × 256 from each 10 s video. Then for each 1 s long segment that contains 16 frames, we randomly pick one frame for each training epoch. For raw audio, we sample it at 16,000 Hz with an equal duration of 10 s for each video. After that, a short-time Fourier transform (STFT) is performed and the signal in the frequency domain is then transformed into Mel-scale filter banks with 64 bins. The resulting spectrogram is 96 × 64 for every segment. For the random augmentations applied to the frames, we use the same settings as in [23].

Considering the size of the AVE dataset is too small to train the feature encoder from scratch, ResNet-50 [53] (removing the final fully connected layer and retaining the 512-dimensional representation) pretrained on ImageNet [14], and VGGish [54] (outputting 128-dimensional representation) pretrained on AudioSet [15], are employed as CNN backbones for extracting spatial features. In AV-BT, the projector has three linear layers, and the output dimensions of each are 2048/4096/4096. For TACMA, the parameter $\theta$ in Equation (4) is set to 0.03 to achieve the best performance.

Optimization. For training AV-BT, we use the LARS optimizer [55] and train 100 epochs with a batch size of 128. The learning rates of weights and biases in the representation network are the same in [23] except that we multiply them by 2e-4. We reduce the learning rate by a factor of 10 using a cosine decay schedule after using 10 epochs of warm-up period [56]. In Equation (2), $\lambda$ is set to 5e-3. For training TACMA, we use an initial learning rate of 1e-4 and multiply it by 0.98 after every 10 epochs, and the batch size is 128 using the Adam [57] optimizer.

4.2. Results

4.2.1. Evaluating AV-BT

AVEL in [6] adopts an audio-guided visual attention module (AGVA) which interacts between the audio and the visual encoder. It uses audio information to help the visual encoder focus on the sounding target to better filter out interference caused by the visual background. Apart from the AGVA module, the structures of our encoders are quite similar to those of AVEL. Particularly, the CNN backbone used for audio feature extraction is identical while the backbone for visual features has comparable capabilities. Even if AVEL is trained with full labels in an end-to-end manner, we still think it is reasonable to compare with it since the following research [12,13,19] directly adopted it to extract features.

Comparison with existing methods. Analogous to the protocols in self-supervised image representation learning [23,24,25,26,27,28], we directly train a classifier on top of the fixed representations produced by the encoders trained with AV-BT. Unlike using a single linear layer as the prediction head for single-modal representations, sequential audio-visual representations are much more complex, so we adopt a three-layer MLP (which has the same structure as the classification layer in Figure 4) to make predictions. The results are reported in

Table 2. Evaluation of AV-BT. Our results are in bold. Note that we use no extra module for fusion at this point, but simply add audio and visual features together to make classifications. For comparison, the result of AVEL in [6] is obtained by end-to-end fully supervised training and they used the same strategy of addition before prediction.

Method	Accuracy (%)
Audio only [6]	59.5
Visual only [6]	55.3
AVEL [6]	71.3
Ours (Audio only) Ours (Visual only) Ours (Audio-Visual)	63.5 61.0 75.6

, where our AV-BT significantly outperforms the supervised method in both unimodal and multimodal cases. Specifically, in the case where the CNN backbone used for audio feature extraction is identical, our method surpasses AVEL by 4% when using only audio data. This improvement in the unimodal case suggests that by maximizing the similarity of different modal features in the same latent space, features from each modality are enhanced to be more discriminative. An intuitive explanation is that by reducing the differences between different modal representations, both the visual and auditory properties of an object can be considered more collaboratively. For example, mandolins and guitars may be very close in appearance, but they are very different in sound. Combining both the visual and audio characteristics of the mandolin allows it to be better distinguished from the guitar. Moreover, the results also show that the improved separability of the unimodal feature also further enhances the performance in the multimodal case.

4.2.2. Evaluating TACMA

PSP in [19] is one of the state-of-the-art solutions for AVE localization. It adopts a threshold-based attention style module to aggregate information from all the similar audio and visual features. This module is highly reliant on applying additional supervision for the background to force the network to learn appropriate features for thresholding.

Comparison with state-of-the-art. By training the proposed TACMA based on fixed representations produced by AV-BT, we decoupled multimodal data representation and late fusion in the AVE localization task. From this perspective, it increases the difficulty of adapting from the source domain to the target domain while the solutions we compare are all combined late fusion networks with feature encoders, and are trained in an end-to-end manner supervised with full labels. Nevertheless, our overall method still achieves state-of-the-art performance on the AVE localization task, as shown in

Table 3. Comparing the proposed method to state-of-the-art methods. Our result is in bold.

Method	Accuracy (%)
CMAN [12]	73.3
AVRB [10] AVIN [11] MPN [8] JCAN [13] PSP [19] AVT [50] AV-BT+TACMA (Ours)	74.8 75.2 75.2 76.2 76.6 * 76.8 77.2

* For a fair comparison, the result of PSP is obtained by training without audio-visual pair similarity loss, which applies additional supervision for differentiating the background.

. This result suggests that channeling attention according to both the time difference and semantic similarity between features encoded from AV-BT can further improve the quality of fused features.

Detailed comparison with AV-BT. In addition to comparing with the latest methods, we would also like to know the detailed performance improvement of TACMA compared to AVBT. Hence, we show the confusion matrices of our solutions in Figure 5. From Figure 5a we can observe that most of the categories can be well represented by AV-BT. However, there are still many misclassifications between events and the background. After further fusing information with TACMA, the misclassification of events as the background improves for almost all categories, indicating that TACMA can grasp more event-related information since it dynamically considers all connections between different modal features to integrate useful cues, as shown in the bottom row in Figure 5b. In addition, for certain categories with very few samples, TACMA also improves the accuracy based on AV-BT. For example, index 11: “Truck”, index 12: “Shofar”, index 13: “Motorcycle”, index 22: “Cat” and index 23: “Horse”.

4.3. Ablations

In this section, we delve into the designs of our method and explore their influences. It is necessary to clarify that we employ our results reported in the previous subsection as the baseline of the ablation study.

4.3.1. Architecture of AV-BT

Weight sharing. The two projectors share the same parameters by default in AV-BT. One may wonder how heterogenous audio and visual features can be mapped to the same latent space by a single MLP. For this problem, we cut off the weight sharing between two projectors to deeply explore our AV-BT, denoted as “w/o sharing projector” in

Table 4. Ablations on the architecture of the representation network. The best performance is in bold. Note that when applying an interaction module such as AVGA, the feature extraction network is no longer able to encode features under single modal data input.

Modification	Audio-Visual	Audio Only	Visual Only
Baseline w/AGVA w/o sharing projector	75.6 71.5 72.5	63.5 - 63.0	61.0 - 60.4

.

Attention. Sound source localization on the visual portion is one of the optional demands in the AVE localization task. Most existing approaches ambiguously localize the event-related sounding object by a heatmap generated in the attention block as a byproduct. Although the AVE dataset does not have bounding boxes as annotations (so the results of localization cannot be quantified for comparison) and localizing sound sources is not in the scope of this paper, we still wonder if we could improve feature quality by integrating attention modules such as AGVA. We denote this setting as “w/AGVA” in Table 4.

Analysis. As discussed above, the architecture of AV-BT is adjusted in two ways and Table 4 recapitulates these modifications tested along with their results. When disabling the weight sharing between projectors, the performance dropped in the unimodal case. In addition, the quality of audio-visual joint representation is significantly worse. We argue that the point here is not about the method of mapping heterogeneous data. Using the same MLP for projection forces the encoders to learn representations that carry as similar semantic information as they possible can. Another phenomenon that can be observed from Table 4 is that spatial attention modules such as AGVA, which are successful in other methods, conversely fail in our case. We argue that because our representation network is trained in a self-supervised way, the attention block is not guided by event labels, thereby the network is uncertain about which sounding objects should be focused on. Especially when encountering a background segment, AGVA may further corrupt the representations since it may attend to an interfering area. Multi-head co-attention may be a potential solution to the visual localization problem because it allows the construction of complex many-to-many audio-visual relationships spatially, and we will investigate such a design in the future.

4.3.2. Data Augmentations in AV-BT

Data augmentations. Popular self-supervised methods such as [23,26,27] are competitive with supervised learning partly thanks to their random data augmentations. Random cropping, horizontal flipping, color jittering, Gaussian blurring, solarizing, and converting to grayscale are among the most used methods. We also study their influences in our self-supervised method. Multiple representative experimental results are presented in

Table 5. Ablations on adding random data augmentations in our AV-BT. The best result is in bold. Note that * denotes that the crop scale is adjusted to be relatively smaller. The original intention for this was to prevent cropping out the entire sounding target, resulting in breaking the agreed rule of an AVE, i.e., visible and audible at the same time.

Crop	Flip	Solarize	Grayscale	Jitter	Blur	Accuracy (%)
✓ *	✓	✓	✓	✓	✓	74.0
✓	✓	✓	✓	✓	✓	74.5
✓	✕	✓	✓	✓	✓	74.2
✓	✕	✕	✓	✓	✓	74.3
✓	✕	✕	✕	✓	✓	74.4
✓	✕	✕	✕	✕	✓	75.6
✓	✕	✕	✕	✕	✕	74.3
✓	✓	✕	✕	✕	✓	74.7
✓	✓	✓	✓	✕	✓	74.5
✕	✕	✕	✕	✕	✕	74.4

.

Analysis. Compared to unimodal self-supervised learning cases, our results in Table 5 suggest that most of the commonly used augmentations (except color jittering) do not make a huge difference in our case. Different from a single image, AVE has a stricter definition that requires the target to be both seen and heard. Adding strong interruptions on data may break the agreed rules. An extreme case in Table 5 is that when applying color jittering, performance is badly damaged. We argue that altering the exposure and contrast ratio of a visual frame too much would make the target very likely to be undetectable in a noisy environment. For example, a black church bell ringing in almost complete darkness in an attic. Before the experiment, we suspect that cropping too many regions might result in losing the target in a frame. However, as the results shown in the first two rows of Table 5 indicate, a larger cropping ratio helps to train stronger encoders. After delving deeper into the dataset, we found that certain kinds of events were still defined as visible when only a portion of the sounding object was observed, such as the nose of an airliner. Ultimately, the best result is achieved by randomly adopting cropping and Gaussian blurring.

4.3.3. Ablation on $\theta$ in TACMA

In this section, we further study how the proposed weighting operation influences the performance of TACMA by gradually changing the hyperparameter $\theta$ in Equation (4). The corresponding results are shown in Figure 6. The overall localization accuracy reaches a high level when $\theta$ is set in the range from 0.01 to 0.06, and peak performance is achieved when $\theta$ = 0.03, indicating that the proposed weighting method contributes a lot to modeling cross-modal relations by discriminating those less-relevant audio-visual connections that are unrelated to the event or temporally asynchronous. In addition, when $\theta$ = 0, the proposed weighting method is completely disabled and the whole design degrades to standard cross-modal attention which has a very limited contribution to the localization performance by only improving by 0.4% compared to the result in Table 2. This shows that such a cross-modal attention module, without considering the time difference and semantic similarity, is not very capable of fusing audio and visual information.

The curvature of Equation (4) is controlled by $\theta$ and the similarity of the feature pair together so that $\theta$ is playing a similar role of thresholding that filters out those feature pairs with smaller similarity scores. This is close to the idea in PSP [19], while our weighting operation works in conjunction with the similarity determined by the feature itself instead of counting on an external loss item to guide it. This makes our attention module more adaptive, and our results verify the effectiveness.

4.4. Visualization

Feature visualization. To verify that our methods can learn discriminative audio-visual features, we visualize several classes of the features learned by AV-BT and TACMA with the high-dimensional data visualization tool, t-SNE [58], as shown in Figure 7. Specifically, we selected features from categories covering diverse targets, such as people, animals, vehicles, and environmental objects, to demonstrate the adaptability of our approach to a variety of situations. Despite its relatively small sample numbers for a certain class, such as index 23: “Horse”, good within-class cohesion is still achieved in both audio and visual features. In addition, the discrimination between classes is also greatly enhanced after further information fusion by TACMA, and this is particularly remarkable for the visual stream.

Qualitative results. We evaluated our whole method with several interesting samples from the AVE dataset to verify the design. Predictions made by our networks are presented in Figure 8. Through these representative samples, we can intuitively gain a better understanding of our method, especially how TACMA coordinates with AV-BT for superior performance.

First, we observed that both of our proposed methods can detect various AVEs in the real world, including human speaking, vehicle sounds, environmental object activities, etc., as shown in in Figure 8b–d. Even if some samples are exceptionally challenging, for example, the visual quality of (e) in Figure 8 is very poor and the sound of the helicopter is almost completely distorted by extreme wind noise, our method still holds up to a certain extent. Generally, based on AV-BT, more accurate predictions can be made after fusion by TACMA.

Second, our networks only represent the data from a semantic perspective. This conclusion can be obtained from a typical case in Figure 8a, where there is a ringing across the whole timeline while only the first appearing bell is emitting sound. When the camera’s view is turned to the second bell, both AV-BT and TACMA still treat the sounding objects as visible (since a bell is still visible even though it’s not the one ringing at this moment). We argue that adopting mono audio signals for feature encoding makes it difficult to solve this kind of problem because of the lack of information for tracking sounding targets spatially.

Last, we designed TACMA to integrate more useful information from the audio-visual pairs that are of various time intervals, and Figure 8f proves that our design meets such expectations to a certain extent. In that example, a man is trimming branches on a tree with a chainsaw and the video is recorded from a first-person perspective. Since the visual content is very unstable and the chainsaw is only captured in very few frames, it is very difficult for AV-BT to deal with such rapidly changing visual content (resulting in predicting all segments as background). After further aggregating useful cues with TACMA, the network can grasp more event-related information.

5. Conclusions

In this paper, we proposed a self-supervised audio-visual representation method and temporally adaptive cross-modal attention for AVE localization. When representing video data, the audio and visual features are encoded in the same space with their association explored in a self-supervised manner. When modeling cross-modal relationships, attention is channeled adaptively by considering both the time difference and semantic similarity between features. Our proposed framework performs better than previous event localization methods on the public AVE dataset and the results show that: (1) more discriminative features can be encoded by narrowing the heterogeneity gap between modalities with our self-supervised method. (2) Our strategy for adaptively assigning attention by considering the temporal asynchrony of event-related features can effectively fuse the information across modalities.

6. Future Direction

Although our proposed framework achieves competitive AVE localization accuracy and the effectiveness of our design is verified by experiments, there is still room for further improvement. Specifically, the visualization result shows that both AV-BT and TACMA focus on representing the multimodal data and cross-modal relationships from a semantic perspective, which makes our network lack the ability to track the sound source spatially. This limits the event localization performance, especially in the face of multiple objects that may emit sound. A potential solution such as multi-head co-attention which can construct complex audio-visual relations spatially can be studied in the future. Moreover, as the task of AVE localization provides an intuitive form of multimodal video analysis, we will also investigate whether our proposed self-supervised method with improved encoder structures can be transferred to more tasks such as video parsing and object segmentation.