A Self-Paced Multiple Instance Learning Framework for Weakly Supervised Video Anomaly Detection

Abstract

Weakly supervised video anomaly detection (WS-VAD) is often addressed as a multi-instance learning problem in which a few fixed number of video segments are selected for classifier training. However, this kind of selection strategy usually leads to a biased classifier. To solve this problem, we propose a novel self-paced multiple-instance learning (SP-MIL) framework for WS-VAD. Given a pre-trained baseline model, the proposed SP-MIL can enhance its performance by adaptively selecting video segments (from easy to hard) and persistently updating the classifier. In particular, for each training epoch, the baseline classifier is firstly used to predict the anomaly score of each segment, and their pseudo-labels are generated. Then, for all segments in each video, their age parameter is estimated based on their loss values. Based on the age parameter, we can determine the self-paced learning weight (hard weight with values of 0 or 1) of each segment, which is used to select the subset of segments. Finally, the selected segments, along with their pseudo-labels, are used to update the classifier. Extensive experiments conducted on the UCF-Crime, ShanghaiTech, and XD-Violence datasets demonstrate the effectiveness of the proposed framework, outperforming state-of-the-art methods.

1. Introduction

As monitoring systems become more widespread, timely and accurate detection of abnormal behaviors or emergencies in surveillance videos is crucial for building smart cities and maintaining public safety. An abnormal event in a video typically refers to an unusual appearance or motion property, or to a normal appearance or motion property occurring at an anomalous time or in an anomalous space. The task of video anomaly detection involves identifying temporal or spatial anomalies within the video. In general, based on the granularity of available labels, the video anomaly detection methods can be typically classified into supervised learning [1], unsupervised learning [2,3], and weakly supervised learning [4,5]. Since manual frame-by-frame labeling of abnormal events in videos is extremely labor-intensive, supervised learning-based video anomaly detection methods are prohibitively costly. In contrast, unsupervised video anomaly detection methods rely on the similarities among samples to perform clustering, or modeling the distribution of normal samples. During testing, video frames that deviate significantly from the learned normal pattern are identified as anomalies. However, due to the absence of prior knowledge of anomalies, the effectiveness of unsupervised methods is generally limited [6]. Consequently, this paper primarily focuses on weak supervision-based video anomaly detection methods, which rely solely on video-level labels (normal and abnormal) during the training process.

The WS-VAD task was initially proposed by Sultani et al. [7] and formulated as a multi-instance learning (MIL) problem. In particular, normal and abnormal videos are considered negative and positive bags, respectively, and a series of non-overlapping video clips of equal length are treated as instances within each bag. This refers to a scenario in which only video-level labels are provided during training, while frame-by-frame anomaly detection must be performed during testing. If at least one anomalous instance is detected in a video, the video is labeled anomalous (i.e., a positive bag); otherwise, it is classified as normal (i.e., a negative bag). The objective of the WS-VAD task is to train an instance-level anomaly classifier using only video-level labels. During training, the most representative instances from both positive and negative bags are initially identified, and these selected instances, along with their corresponding bag-level labels, are then used to train the anomaly detection classifier. The training process forcibly reduces the predicted abnormal score for each selected representative segment in a normal video, while increasing the score of one or more selected segments with the highest abnormal scores in an abnormal video. During testing, the classifier predicts the anomaly score for each video segment, with all frames within a segment being assigned the same score, thereby enabling frame-by-frame anomaly detection. Based on this MIL framework, numerous studies have been conducted, leading to significant advancements [8,9].

However, as a result of this training approach, the classifier is typically sensitive only to the most representative segments of both normal and abnormal videos, while neglecting other segments within these videos. Consequently, classifiers trained in this manner often yield biased results [10], tending to recognize only the most conspicuous anomalies, while subtle anomalies are frequently overlooked. For example, the classifier was able to identify anomalies when a car was vandalized but missed subtle anomalous activity by the person prior to the vandalism. The moment the car is destroyed can be seen as an easy anomaly, while the subtle abnormal activity by a person prior to the destruction can be seen as a difficult anomaly. A biased classifier may generally recognize only the easy anomalies while failing to recognize the difficult ones. In contrast, an unbiased classifier should be able to identify both easy and difficult anomalies in most cases. The root cause of biased predictions in MIL stems from its training scheme, which involves biased sample selection. A natural idea is to train the classifier to utilize a broader set of video clips rather than only the top-k segments with the highest anomaly scores. To this end, previous studies [1,10,11] have made significant progress toward this goal, often by training classifiers on the most confident or reliable video clips along with their respective pseudo-labels. For example, Zhong et al. [1], using the 10-crop method, performed data augmentation on each video segment and then selected the segment with the least variance among the ten scores as the most confident one for training. Similarly, Zhang et al. [11] employed Monte Carlo dropout for uncertainty estimation on each video segment, with each segment generating multiple distinct scores via dropout. A specific proportion of segments with the lowest variance is selected, and their corresponding average score is used to train the classifier. In addition, Lv et al. [10] considered the video clips with the most stable historical scores as confident sets, while the remaining clips were categorized as ambiguous sets. They employed supervised learning for clips within the confident sets and unsupervised clustering for clips within the ambiguous sets to facilitate classifier training. Although the aforementioned methods have demonstrated some success, they typically pre-define the proportion or number of reliable and confident samples to be selected, which lacks flexibility in adapting to different datasets, potentially impacting the generalization performance of the classifier.

To address this issue, we propose a general self-paced multiple-instance learning (SP-MIL) framework inspired from the idea of self-paced learning [12]. To illustrate our framework, the comparative flow diagram is shown in Figure 1. For the baseline model, when given a sequence of video segments, segment-level features are typically derived via feature extraction and temporal modeling, after which a randomly initialized classifier is used to predict the abnormal score of each segment. Subsequently, the top-k segments with the highest anomaly scores are selected and aggregated to form video-level features. The classifier is then trained using these video-level features along with corresponding video-level labels to obtain the baseline model. It is worth noting that k is usually a pre-set fixed value that does not change with the training epochs. This setting makes the trained classifier biased and inaccurate in judging the abnormal degree of other segments in the video. Different from the baseline model, the proposed SP-MIL framework seeks to enhance the classifier’s prediction ability by progressively and adaptively incorporating more representative video segments into each epoch of the whole training process, advancing from easy to hard segments through a self-paced learning strategy. Specifically, in each training epoch, the difficulty degree of each video segment is initially calculated based on its current anomaly score and its corresponding pseudo-label. Subsequently, based on the calculated complexity, the easiest video segment at this epoch is selected for training. As time progresses, the number of selected segments gradually increases in a progression from easy to hard. Furthermore, more segments will be adaptively selected and integrated into the classifier’s training process. This enables the classifier to progressively incorporate more video segments, thereby enhancing its ability to distinguish between anomalous and normal cases, resulting in unbiased predictions.

The main contributions of this paper can be summarized as follows:

• We propose a general self-paced multiple-instance learning (SP-MIL) framework for the task of WS-VAD, which can significantly enhance the performance of widely used models (e.g., DeepMIL, RTFM, UR-DMU). To the best of our knowledge, this is the first work in which self-paced learning is used for WS-VAD.
• Unlike the widely used top-k instance selection strategy, which may result in a biased classifier, we propose to adaptively select the video instances (i.e., segments) from easy to hard according to the principle of self-paced learning. By alternatively updating the subset of segments used for training and parameters of the classifier, we can achieve an enhanced classifier with the ability of unbiased prediction.
• Extensive experiments conducted on the UCF-Crime, ShanghaiTech, and XD-Violence datasets demonstrate the effectiveness of the proposed SP-MIL framework across different video features and baseline models, and the proposed framework achieved the best performance compared with the state-of-the-art methods.

2. Related Work

2.1. Weakly Supervised Video Anomaly Detection

The weakly supervised setting offered a balanced solution for the VAD task, requiring less annotation efforts than dense frame-level labeling in a fully supervised setting, while achieving superior performance compared to the unsupervised setting. Current WS-VAD approaches can be broadly categorized into two classes [13]. One class includes encoder-agnostic methods, which utilize a pre-trained feature extractor to obtain segment feature and train the classifier to detect abnormal segments. The second category involves encoder-based methods that train both the feature extractor and classifier. In the first category of methods, Sultani et al. [7] modeled this task as a multi-instance learning problem, where the segment with the highest abnormal score was used as the video-level prediction, and a ranking loss was employed to increase the margin between normal and abnormal video predictions. Chang et al. [14] proposed a contrastive attention mechanism, assigning importance weights to each video segment and weighting segment features to obtain video-level features, which were then used with video-level labels to train the classifier. Zhang et al. [15] improved the loss function by introducing a new inner-bag loss, which encouraged the highest and lowest scores in normal videos to converge as closely as possible, while driving the highest and lowest scores in abnormal videos apart. However, some researchers argued that the feature extractors used in this category were usually pre-trained on large-scale trimmed video datasets, which presented a domain gap when applied to the WS-VAD task [9].

In the second category of methods, some researchers [5,9,16] performed temporal modeling on the segment features produced by the feature extractor. They used temporal modeling networks to further extract temporal context features and then trained the temporal modeling network alongside the classifier. For example, Tian et al. [5] proposed a multi-scale temporal network (MTN), which captured both multi-resolution local and global temporal dependencies between video segments. Wu and Liu [16] introduced a causal temporal relation (CTR) module, designed to aggregate relevant information from historical and current features through temporal attention. In contrast, Feng et al. [9] trained the feature extractor and classifier simultaneously, without using a temporal modeling network. They proposed an attention-based self-guided encoder to learn task-specific representations and generated clip-level pseudo-labels to guide network optimization. However, both the above two types of methods typically used only a few fixed number of segments with top-k abnormal scores from both normal and abnormal videos to update the classifier. The classifier obtained by this kind of segment selection and training strategy usually results in biased predictions [10]; that is, only simple and obvious abnormal behaviors can be detected, potentially producing false alarms for more subtle anomalies. Based on this finding, utilizing only video-level labels to achieve unbiased prediction without additional human annotations has become an important problem to address.

2.2. Self-Paced Learning

The self-paced learning (SPL) theory [12] is inspired by the cognitive process of human beings, where samples are gradually incorporated into the training process from easy to hard. In recent years, self-paced learning methods have been successfully applied to different computer vision tasks [17,18,19]. For example, Zhou et al. [17] proposed a deep self-paced learning algorithm for the person re-identification task, which utilized high-confidence samples while suppressing low-confidence noisy samples in the early stages of training. This approach allowed the neural network to be trained more stably by gradually involving reliable samples from easy to hard, thereby improving the accuracy of person re-identification. For weakly supervised object detection tasks, Sangineto et al. [18] proposed a training method based on self-paced learning, which selects the most reliable subset of images and boxes in each iteration and trains the network in a fully supervised manner to develop robust detectors. Zhang et al. [19] proposed a self-paced fine-tuning network-based framework for the weakly supervised object segmentation task, which combined a self-paced learning mechanism with deep neural networks to enhance the model’s segmentation capability.

Inspired by the successful application of SPL in related tasks, we propose to combine SPL with multiple-instance learning (MIL) for the WS-VAD task. This combination aims to aid in the selection of more reliable video segments, thereby facilitating the training of more robust models. To the best of our knowledge, we are the first to use SPL for the task of WS-VAD. The work most related to ours is Lv et al. [10], in which the authors proposed an unbiased multiple-instance learning (UMIL) approach. In each training iteration, the video clips are first divided into confident and ambiguous sets. The confident set is used to train the classifier through segment-wise supervised learning, while the ambiguous set assists the classifier training by reducing the abnormal context bias within the confident set through unsupervised clustering. This method allows the classifier to be trained on a broader range of video clips, ultimately leading to a more reliable and unbiased classifier. However, their approach typically involves presetting the proportion of confident video clips used for training, which limits its flexibility in various practical scenarios, particularly for videos with differing durations of abnormal events. Therefore, in this paper, inspired by the success of self-paced learning, we propose a SP-MIL framework, the core idea being to adaptively select the subset of segments used for training in a easy-to-hard way and dynamically update the classifier to persistently enhance its performance through continuous self-training.

3. Proposed Method

3.1. Overview

An overview of the proposed general SP-MIL framework is shown in Figure 2. For each input video, it is first split into non-overlapped, consecutive segments of unequal lengths. Then, the pre-trained features of these segments are extracted, and temporal modeling is applied to derive temporal context features. It is important to note that temporal modeling is a optional module. The temporal context features of the segments are then fed into a randomly initialized classifier to predict the abnormal score of each segment. Subsequently, the most representative segments are selected along with video-level labels for training this classifier. Once the training is complete, the trained classifier is employed as the baseline model of our proposed SP-MIL framework, which is then refined to obtain an enhanced classifier for unbiased prediction. The pseudo-code of the proposed SP-MIL framework is provided in Algorithm 1. Specifically, the anomaly scores of segments are first predicted and then converted into pseudo-labels via threshold binarization. The difficulty of each segment is then calculated based on its loss. Next, during each epoch of training, the age parameter of each video is first evaluated, reflecting the overall difficulty of the video and each segment, and is determined by the loss of all segments in the video. Then, the evaluated age parameter is used as a threshold to determine the self-paced learning weight of each segment, which is a binary hard weight with values of 0 or 1. Finally, the segment subset for training is selected based on the self-paced learning weights, and the selected segments along with their pseudo-labels are used to update the classifier. This process is repeated T times to obtain the enhanced classifier $f_T$ with the optimal parameter $\mathit{\omega }$${}_T$.

Algorithm 1: Pseudo-code of the proposed SP-MIL framework

3.2. SP-MIL Framework

3.2.1. Baseline Model Initialization

Let {$\mathit{X}$${}_k{\}}_{k=1}^K$ denote the given K training videos which include $K_{+}$ abnormal videos and $K_{-}$ normal videos, and the index set of all abnormal videos is $I_{+}=\{1,2,\cdots ,K_{+}\}$ while the negative ones $I_{-}=\{K_{+}+1,K_{+}+2,\cdots ,K\}$. Given a video ${\mathit{X}}_k={\left\{{\mathit{x}}_{ki}\right\}}_{i=1}^N$ with N continuous and non-overlapped segments, the corresponding weak label $y_k\in \{0,1\}$ indicates whether ${\mathit{X}}_k$ contains abnormal events. The baseline model consists of two primary components: a feature extractor g and a classifier f. For each video ${\mathit{X}}_k$, the feature extractor g processes video segments ${\left\{{\mathit{x}}_{ki}\right\}}_{i=1}^N$ as input, generating segment features ${\left\{{\mathit{z}}_{ki}\right\}}_{i=1}^N$, which are subsequently fed into the classifier f to determine the anomaly scores ${\left\{s_{ki}\right\}}_{i=1}^N$. The top-1 or top-k segments with the highest anomaly scores are selected and used to train the classifier using video-level labels.

1.

Feature extractor $\mathit{g}$. Feature extractors used in the existing WS-VAD task can be generally categorized into two types. The first type extractors rely solely on a pre-trained network $h_0$ for feature extraction, such as convolutional 3D neural networks (C3D) [20], Inflated 3D ConvNets (I3D) [21], and contrastive language-image pre-training (CLIP) [22]. The output of the feature extractor is generally represented as ${\mathit{z}}_{ki}=g\left({\mathit{x}}_{ki}\right)=h_0\left({\mathit{x}}_{ki}\right)$. The second type further refines features using a temporal modeling network $h_1$ built upon the pre-trained networks $h_0$, such as graph convolutional network (GCN) [1], multi-scale temporal network (MTN) [5], and temporal self-attention (TSA) [23], etc. In this case, the output of the feature extractor can be represented as ${\mathit{z}}_{ki}=g\left({\mathit{x}}_{ki}\right)=h_1\left(h_0\left({\mathit{x}}_{ki}\right)\right)$. Despite the variations in feature extraction methods and output dimensions among different pre-trained networks, each video segment can be consistently represented as a corresponding feature vector, ensuring the compatibility between $h_1$ to $h_0$ modeling. To demonstrate the generality of our SP-MIL framework, we evaluate the effectiveness of two different pre-trained feature extractors (I3D and CLIP) along with DeepMIL [7], which relies solely on the pre-trained network $h_0$ for feature extraction, RTFM [5], and UR-DMU [24], which additionally incorporate the temporal modeling network $h_1$.

2.

Classifier $\mathit{f}$. The standard classifier f used in the WS-VAD task typically comprises a fully connected network. Each video segment feature ${\mathit{z}}_{ki}$ is fed into f, producing its corresponding anomaly score $s_{ki}=f\left({\mathit{z}}_{ki}\right)=f\left(g\left({\mathit{x}}_{ki}\right)\right)$. The widely used classification objective function is as follows:

$$L_{ce}=-y_k\log p_k-(1-y_k)\log (1-p_k),$$

(1)

where $p_k=\underset{i\in \{1,2,\cdots ,N\}}{max}{s}_{ki}$ or $p_k={\displaystyle \frac{1}{k}}{\sum }_{i\in {\mathrm{\Omega }}_k\left({\mathit{X}}_k\right)}{s}_{ki}$, with ${\mathrm{\Omega }}_k\left({\mathit{X}}_k\right)$ denotes the segment index set of video ${\mathit{X}}_k$ with the top-k largest anomaly scores. This indicates that one or more of the most representative video segments are typically selected using the video-level label to supervise the training of the classifier f. The regularization terms commonly used in baseline loss functions typically include the ranking loss (i.e., hinge loss), smooth loss, and sparse loss, with their respective expressions given as follows:

$$L_{hinge}=max(0,1-\underset{k\in I_{+}}{max}f\left(g\left({\mathit{x}}_{ki}\right)\right)+\underset{k\in I_{-}}{max}f\left(g\left({\mathit{x}}_{ki}\right)\right)),$$

(2)

$$L_{sparse}=\sum _{i=1}^N\left|f\left(g\left({\mathit{x}}_{ki}\right)\right)\right|,$$

(3)

$$L_{smooth}=\sum _{i=1}^{N-1}{(f\left(g\left({\mathit{x}}_{ki}\right)\right)-f\left(g\left({\mathit{x}}_{k,i+1}\right)\right))}^2.$$

(4)

The overall objective function of the baseline model is expressed as follows:

$$L_{baseline}=\alpha L_{hinge}+\beta L_{sparse}+\gamma L_{smooth}+L_{ce},$$

(5)

where $\alpha$, $\beta$ and $\gamma$ are hyperparameters.

3.2.2. Classifier Enhancement Using Self-Paced Learning

The SP-MIL framework models the WS-VAD task as a segment-level supervised learning problem. The main idea is to initially learn easy segments and then gradually transfer the acquired knowledge to recognize harder ones. By gradually exposing the classifier to more segments and their pseudo-labels during training, the classifier can identify difficult anomalies more effectively than the baseline model, thereby achieving more unbiased prediction. Specifically, the SP-MIL framework consists of four primary steps: segment-level score prediction, segment-level pseudo-label generation, adaptive segment subset selection, and classifier updating.

For illustrative purposes, let superscript t denote the t-th epoch of training. For each video ${\mathit{X}}_k={\left\{{\mathit{x}}_{ki}\right\}}_{i=1}^N$ with N segments, the segment features ${\left\{{\mathit{z}}_{ki}\right\}}_{i=1}^N$ are used to generate the anomaly scores ${\left\{s_{ki}^t\right\}}_{i=1}^N$ using the classifier $f_{t-1}$ from the $(t-1)$-th epoch of training. These anomaly scores are then binarized to produce segment-level pseudo-labels ${\left\{y_{ki}^t\right\}}_{i=1}^N$. Subsequently, the segment subset with index set $I_k^t$ is adaptively updated based on the estimated age parameter ${\lambda }_k^t$ and segment-level self-paced learning weights ${\left\{v_{ki}^t\right\}}_{i=1}^N$. The age parameter vector for all K videos is represented as ${\mathit{\lambda }}_t=[{\lambda }_1^t,\cdots ,{\lambda }_K^t]\in {\mathbb{R}}^K$, where ${\lambda }_k^t$ represents the age parameter shared for all segments in the k-th video during the t-th epoch of training. The self-paced learning weight matrix for all segments of all videos during the t-th epoch of training is represented as ${\mathit{V}}_t={[{\mathit{v}}_1^t,\cdots ,{\mathit{v}}_K^t]}^T\in {\mathbb{R}}^{K\times N}$, where ${\mathit{v}}_k^t=[v_{k1}^t,v_{k2}^t,\cdots ,v_{kN}^t]$ and $v_{ki}^t$ represents the self-paced learning weight of i-th segment in the k-th video during the t-th epoch of training, which is a hard weight and takes the binary values 0 or 1, indicating whether the video segment is selected or not. Finally, these segments in the selected subset with their corresponding pseudo-labels are used for supervised training, refining the classifier $f_{t-1}$ to obtain $f_t$. More details for each step are introduced as follows. Additionally, the details of parameter updates at each step are illustrated in Figure 3.

1.

Segment-level score prediction. For each video ${\mathit{X}}_k$ during the t-th epoch of training, the classifier $f_{t-1}$ takes video segments ${\left\{{\mathit{x}}_{ki}\right\}}_{i=1}^N$ as input and output segment-level anomaly scores ${\left\{s_{ki}^t\right\}}_{i=1}^N$ by using the following formulation:

$$s_{ki}^t=f(g\left({\mathit{x}}_{ki}\right);{\mathit{\omega }}_{t-1}),\forall {\mathit{x}}_{ki}\in {\mathit{X}}_k,\forall k\in \{1,2,\cdots ,K\},$$

(6)

where ${\mathit{\omega }}_{t-1}$ are network parameters of classifier $f_{t-1}$.

2.

Segment-level pseudo-label generation. For each segment-level anomaly score $s_{ki}^t$, its segment-level pseudo-label $y_{ki}^t$ is generated using a pre-defined binarization threshold $\eta$. The calculation formula is as follows:

$$y_{ki}^t=\left\{\begin{array}{c}1(s_{ki}^t>\eta ),\forall k\in I_{+}\hfill \\ 0,\forall k\in I_{-}\hfill \end{array}\right.$$

(7)

where 1 is an indicator function. Equation (7) demonstrates that during the t-th epoch of training, the segment-level pseudo-label is assigned to a value of 1 for segments in abnormal videos if their anomaly score exceeds the threshold $\eta$; otherwise, it is set to 0. It is worth noting that for segments in normal videos, the segment-level pseudo-label is consistently set to 0.

3.

Adaptive segment subset selection. The purpose of segment selection is to adaptively identify the most easily classified subset of segments from each video in every training epoch. The processes of segment subset selection and classifier updating are iterative. During the t-th epoch of training, this self-paced learning-based segment selection and classifier-updating strategy can be formulated as:

$$\underset{{\mathit{\omega }}_t,{\mathit{V}}_t}{min}E({\mathit{\omega }}_t,{\mathit{V}}_{\mathit{t}};{\mathit{\lambda }}_t)=\sum _{k=1}^K\sum _{i\in I_k^t}{v}_{ki}^t{l}_{ki}^t+h({\mathit{V}}_t;{\mathit{\lambda }}_t),$$

(8)

where ${\mathit{\omega }}_t$ represents the model parameter of $f_t$, ${\mathit{V}}_t$ denotes the self-paced learning weight matrix, and $v_{ki}^t$ represents the self-paced learning weight of i-th segment in the k-th video. ${\mathit{\lambda }}_t$ represents the age parameter vector, and ${\lambda }_k^t$ denotes the age parameter shared for all segments in the k-th video, which control the difficulty of the video. $I_k^t$ is the easiest subset with the index of segments selected in the k-th video. For each video segment and its generated pseudo-label during the t-th epoch of training, $l_{ki}^t$ denotes the segment-level loss, which can be calculated using the following formula:

$$l_{ki}^t=-y_{ki}^t\log \left(f(g\left({\mathit{x}}_{ki}\right);{\mathit{\omega }}_{t-1})\right)-(1-y_{ki}^t)\log (1-f(g\left({\mathit{x}}_{ki}\right);{\mathit{\omega }}_{t-1})).$$

(9)

$h({\mathit{V}}_t;{\mathit{\lambda }}_t)$ is the self-paced learning regularizer; according to [25], it can be defined as $h({\mathit{V}}_t;{\mathit{\lambda }}_t)=-{\sum }_{k=1}^K{\sum }_{i=1}^N{\lambda }_k^t{v}_{ki}^t$. To solve the optimization problem (8), we firstly need to estimated the age parameter ${\mathit{\lambda }}_t$ according to the loss values of the video segments during the t-th epoch of training. In particular, for all segments ${\left\{{\mathit{x}}_{ki}\right\}}_{i=1}^N$ and their corresponding pseudo-labels ${\left\{{\mathit{y}}_{ki}\right\}}_{i=1}^N$ of video ${\mathit{X}}_k$, ${\lambda }_k^t$ can be estimated as follows. We first compute their corresponding loss values ${\left\{l_{ki}^t\right\}}_{i=1}^N$. It is assumed that the easier the segment, the lower the corresponding loss value. According to this assumption, we sort the segment loss values in ascending order. Let $\mathit{L}=[l_{k1}^t,l_{k2}^t,\cdots ,l_{kN}^t]$ be a vector of the sorted loss values. Then, ${\lambda }_k^t$ can be defined as the R-th element of $\mathit{L}$, that is,

$${\lambda }_k^t=\mathit{L}\left(R\right),$$

(10)

where R denotes the number of selected segments for the video ${\mathit{X}}_k$ and is defined as follows:

$$R=\lfloor R_0+{\displaystyle \frac{t}{T}}\times (R_T-R_0)\rfloor ,$$

(11)

where $\lfloor ·\rfloor$ denotes the floor function, $R_0$ and $R_T$ represent the number of segments selected for the subset with index $I_k^t$ during the initial and final training epochs, respectively, and T represents the maximum number of training epochs. It is worth noting that the age parameter describes the difficulty of each segment. According to the definition of ${\lambda }_k^t$, it will increase gradually with the training epochs t. Then, more difficult segments will be added to the training set for classifier updating. This is also the key of self-paced learning for the strategy of easy-to-hard sample selection. Once all ${\lambda }_k^t$ has been well defined, the optimization problem (8) can be solved using the alternative convex search method. Specifically, when ${\mathit{\omega }}_{t-1}$ is fixed, the optimal ${\mathit{V}}_t$ can be calculated via Equation (12) [12], and when ${\mathit{V}}_t$ is fixed, existing off-the-shelf supervised learning methods can be used to determine the optimal ${\mathit{\omega }}_t$.

$$v_{ki}^t=1(l_{ki}^t\le {\lambda }_k^t),\forall {\mathit{x}}_{ki}\in {\mathit{X}}_k,\forall k\in \{1,2,\cdots ,K\},$$

(12)

where 1 is an indicator function, Equation (12) shows that when a segment’s loss is less than the given age parameter ${\lambda }_k^t$, it is considered as an easy segment and will be selected (i.e., $v_{ki}^t=1$); otherwise, it will be unselected (i.e., $v_{ki}^t=0$). As ${\lambda }_t$ increases gradually over time, more segments will be selected into the subset with index $I_k^t$. This allows the classifier to progressively access more segments, moving from easier segments to more difficult ones, thereby facilitating unbiased prediction.

4.

Classifier updating. The classifier updating and segment selection processes are performed alternately. During this stage, ${\mathit{V}}_t$ is fixed while optimizing the classifier parameter ${\mathit{\omega }}_t$ by using the selected segment subset with index $I_k^t$, and the segment-level classification objective is defined as follows:

$$L_{sce}=\sum _{k=1}^K\sum _{i\in I_k^t}{l}_{ki}^t,$$

(13)

where $l_{ki}^t$ corresponds to Equation (9). Similarly, we employ the terms of hinge loss, smooth loss, and sparse loss, and the total loss function of the proposed SP-MIL framework is defined as follows:

$$L_{SP-MIL}=\alpha L_{hinge}+\beta L_{sparse}+\gamma L_{smooth}+L_{sce},$$

(14)

where the loss functions $L_{hinge}$, $L_{sparse}$, and $L_{smooth}$ correspond to Equations (2)–(4), respectively. When combined with the previously described four-step iterative process, this forms a single training epoch of the proposed SP-MIL framework. Repeating this process over T epochs results in the final enhanced classifier $f_T$ with the optimal parameter ${\mathit{\omega }}_T$.

3.2.3. Testing Phase

When a test video is provided, the pre-trained features of video segments are first extracted using the feature extractor g, followed by direct prediction of abnormal scores for each segment by using the SP-MIL framework enhanced classifier. Drawing inspiration from ensemble learning, classifiers enhanced through different pre-trained feature extractors can complement each other. Thus, we propose a straightforward yet effective late-fusion strategy. Specifically, when I3D and CLIP are used as pre-trained feature extractors, the enhanced classifiers from both are fused at the score level, further boosting the performance. The fused segment-level anomaly score is calculated using the following formula:

$$s_{ki}=\left\{\begin{array}{c}{s}_1,\mathrm{if}|s_1-c|\ge |s_2-c|\hfill \\ s_2,\mathrm{otherwise}\hfill \end{array}\right.$$

(15)

where $s_1$ and $s_2$ represent the enhanced classifiers obtained using I3D and CLIP as the pre-trained feature extractor, respectively, and c denotes the threshold for distinguishing anomaly scores from the two enhanced classifiers.

4. Experiments

4.1. Datasets and Evaluation Metrics

We conducted the WS-VAD experiments on three widely used datasets: ShanghaiTech [26], UCF-Crime [7], and XD-Violence [27]. (1) ShanghaiTech: A medium-scale dataset derived from campus street video surveillance, encompassing 13 distinct background scenes, comprising 437 videos—330 normal and 107 abnormal. Following the standard splits established in [1] for WS-VAD, we partitioned this dataset into two subsets: a training set with 63 abnormal and 175 normal videos, and a testing set with the remaining 44 abnormal and 155 normal videos. (2) UCF-Crime: A large-scale dataset comprising 1900 videos, evenly split between 950 abnormal and 950 normal videos. The dataset encompasses 13 realistic anomalies, including abuse, arrest, arson, assault, accident, burglary, explosion, fighting, robbery, shooting, stealing, shoplifting, and vandalism. (3) XD-Violence: A dataset comprising 217 hours of audio-visual content collected from both movies and real-world scenes, containing 4754 untrimmed videos, with 3954 videos for training and 800 videos for testing.

For ShanghaiTech and UCF-Crime, the area under the frame-level receiver operating characteristic curve (AUC) is used as the evaluation metric. For XD-Violence, the area under the precision-recall curve, referred to as frame-level average precision (AP), is used as the evaluation metric.

4.2. Baseline Models and Implementation Details

4.2.1. Baseline Models

Since the SP-MIL framework does not alter the baseline model structure, to verify the effectiveness of our proposed general SP-MIL framework, we employed three of the most widely used WS-VAD methods as our baseline models: DeepMIL [7], RTFM [5], and UR-DMU [24].

• DeepMIL model: This model was introduced by Sultani et al. [7] to solve the WS-VAD problem. In this approach, the pre-trained features of video segments are first extracted using a pre-trained feature extractor. These pre-trained features are then directly fed into the classifier to generate anomaly scores, and the segment with the highest anomaly score is selected to train the classifier using the video-level label. The objective function is defined as follows:

  $$L_{DeepMIL}=\alpha L_{hinge}+\beta L_{sparse}+\gamma L_{smooth},$$

  (16)

  where $L_{hinge}$, $L_{sparse}$, and $L_{smooth}$ correspond to Formulas (2)–(4), respectively.
  RTFM model: This model, introduced by Tian et al. [5], first extracts pre-trained features of video segments through a pre-trained feature extractor. These pre-trained features are then processed by a multi-scale temporal network to derive temporal context features, which are subsequently fed into the classifier to obtain anomaly scores, and the top-k segments with the highest temporal context feature magnitudes are selected to train the classifier with video-level labels. The objective function is defined as follows:

  $$L_{RTFM}=\alpha L_{hinge}+\beta L_{sparse}+\gamma L_{smooth}+L_{ce}$$

  (17)

Here, $L_{hinge}$, $L_{sparse}$, $L_{smooth}$, and $L_{ce}$ correspond to Formulas (2)–(5), respectively. Notably, the hinge loss in the RTFM model differs slightly from that in Formula (2) as it is applied to the magnitudes of the temporal context features rather than the anomaly scores.

• UR-DMU model: This model was introduced by Zhou et al. [24], it first extracts pre-trained features of video segments through a pre-trained feature extractor. These pre-trained features are subsequently processed by a global and local multi-head self-attention module within the transformer network to obtain more expressive embeddings, which are then fed into dual memory units (DMU) to learn more discriminative features. The embeddings and features output by the DMU are then incorporated into the classifier training process. For further details, please refer to [24].

4.2.2. Implementation Details

Following [7,24], each video is divided into 32 or 200 segments, i.e., N = 32 for DeepMIL and RTFM (N = 200 for UR-DMU). The hyperparameters in the loss function are set to $\alpha$ = $1\times {10}^{-4}$, $\beta$ = $8\times {10}^{-3}$, and $\gamma$ = $8\times {10}^{-4}$. The maximum number of epochs T is set to 100, $R_0$ = 3 and $R_T$ = 32 for DeepMIL and RTFM ($R_T$ = 200 for UR-DMU). The threshold value $\eta$ is set to 0.5, and the late-fusion threshold value c is set to 0.4. The SP-MIL framework is trained on a single NVIDIA RTX 3090 GPU (NVIDIA Corporation, Santa Clara, CA, USA) using PyTorch (version 2.1). The Adam optimiser is employed with a weight decay of 0.0005 and a learning rate of 0.001. For DeepMIL, the fully connected layers described in our model have 512, 32, and 1 nodes, each followed by a ReLU activation function and a dropout function with a dropout rate of 0.7. For RTFM and UR-DMU, the number of nodes is 512, 128, and 1, respectively. It is worth noting that for fair comparison, we also reported the results of DeepMIL, RTFM, and UR-DMU reproduced by our framework by using different pre-trained features.

4.3. The Effectiveness of the SP-MIL Based Classfier Enhancement Strategy

In this section, we verify the general effectiveness of the SP-MIL based classfier enhancement strategy using various baseline models (DeepMIL [7], RTFM [5], UR-DMU [24]), and the generality across different feature extractors (I3D, CLIP).

Table 1. The effectiveness of the SP-MIL for different baseline models (I3D feature is used).

Method	Feature Extractor	UCF-Crime AUC (%)	ShanghaiTech AUC (%)	XD-Violence AP (%)
DeepMIL *	I3D	78.32	92.71	73.24
DeepMIL+SP-MIL	I3D	82.68	97.04	74.44
RTFM *	I3D	83.50	97.39	78.07
RTFM+SP-MIL	I3D	85.20	97.74	80.81
UR-DMU *	I3D	85.02	97.54	80.05
UR-DMU+SP-MIL	I3D	86.53	97.76	80.77

* means that we re-implemented this method.

and

Table 2. The effectiveness of the SP-MIL for different baseline models (CLIP feature is used).

Method	Feature Extractor	UCF-Crime AUC (%)	ShanghaiTech AUC (%)	XD-Violence AP (%)
DeepMIL *	CLIP	80.91	94.96	74.03
DeepMIL+SP-MIL	CLIP	84.14	97.54	75.28
RTFM *	CLIP	84.45	97.70	79.11
RTFM+SP-MIL	CLIP	85.50	97.94	81.52
UR-DMU *	CLIP	86.76	97.56	80.64
UR-DMU+SP-MIL	CLIP	87.13	97.90	81.24

* means that we re-implemented this method.

present the comparative results between the baseline models and the enhanced classifiers on the ShanghaiTech, UCF-Crime, and XD-Violence datasets, with DeepMIL, RTFM, and UR-DMU as the baseline models and I3D and CLIP as the feature extractors, respectively. From Table 1, it can be observed that when I3D is used as the feature extractor, the performance of three baseline models is significantly improved by further training with the SP-MIL framework. Specifically, DeepMIL+SP-MIL achieved performance gains of 4.36%, 4.33%, and 1.20% on the UCF-Crime, ShanghaiTech, and XD-Violence over DeepMIL. Similarly, RTFM+SP-MIL achieved improvements of 1.70%, 0.35%, and 2.74% over RTFM, and UR-DMU+SP-MIL demonstrated improvements of 1.51%, 0.22%, and 0.72% over UR-DMU on the same datasets. Table 2 exhibits a comparable trend based on the CLIP video features. Table 1 and Table 2 demonstrate that the SP-MIL framework enables the classifier to initially focus on easily identifiable video segments and progressively tackle more challenging examples near the classification boundary. Consequently, the classifier gradually accumulates crucial prior knowledge for the challenging segments, thereby enhancing its discriminative capability.

To further confirm this observation, we use t-SNE to visualize the features extracted from the output of the second layer of the classifier in both the baseline models and the baselines+SP-MIL on the UCF-Crime dataset. The results are shown in Figure 4. It illustrates that classifier features trained by the three baseline models remain intermixed. However, after additional training with the SP-MIL framework, normal and abnormal video features become distinctly separated, demonstrating the effectiveness of the SP-MIL framework.

4.4. The Effectiveness of Late-Fusion Strategy for SP-MIL Framework

Drawing inspiration from ensemble learning, we propose a straightforward yet effective score-level late-fusion strategy, where we first use I3D and CLIP as feature extractors, respectively. Next, we train the respective classifiers, enhanced by the SP-MIL framework, and test the video segments using these classifiers to obtain the corresponding anomaly scores. The anomaly scores of video segments are subsequently fused using Formula (15) to produce the final anomaly scores. The comparison results are presented in

Table 3. The effectiveness of the late-fusion strategy for SP-MIL framework.

Method	Feature Extractor	UCF-Crime AUC (%)	ShanghaiTech AUC (%)	XD-Violence AP (%)
DeepMIL *	I3D+CLIP	80.96	94.97	75.07
DeepMIL+SP-MIL	I3D+CLIP	84.81	97.81	76.63
RTFM *	I3D+CLIP	84.65	97.76	80.80
RTFM+SP-MIL	I3D+CLIP	86.35	98.10	84.42
UR-DMU *	I3D+CLIP	86.89	97.78	82.97
UR-DMU+SP-MIL	I3D+CLIP	87.76	97.92	84.32

* means that we re-implemented this method.

. Notably, we integrate these two classifiers only during the testing phase, as the entire pipeline has already been trained. From Table 3, we can find that the fusion results of the SP-MIL framework demonstrate significant improvements over both baseline models, which indicates that the widely used I3D and CLIP features have much complementary information for the WS-VAD task.

4.5. Effect of Parameters $\eta$ and R

For the pseudo-label threshold $\eta$, we employ the RTFM model as the baseline model and CLIP as the feature extractor to conduct ablation experiments on the UCF-Crime dataset. The experimental results are presented in

Table 4. Ablation study on parameter $\eta$ on the UCF-Crime dataset.

$\mathit{\eta }$	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
AUC (%)	83.94	84.60	84.68	85.40	85.50	85.49	85.41	84.80	83.58

. Based on the experimental results, we can find that when $\eta$ is set to 0.5, the model achieves the best performance. This may be because anomaly detection of video frames is a binary classification task, and the threshold is naturally associated with fluctuations around the median value.

The sample selection hyperparameter R, which pertains to training in epoch t, represents the number of samples selected for training in each video during the current epoch. We use the RTFM model and CLIP feature extractor to perform ablation experiments on the UCF-Crime dataset. The results are presented in

Table 5. Ablation study on parameter R on the UCF-Crime dataset.

R	3	5	8	10	15	20	25	30	32
AUC (%)	84.43	84.68	85.07	85.50	84.23	83.67	82.99	81.24	80.75

. Based on the experimental results, we conclude that 10 is the most suitable value of R for the UCF-Crime dataset. For ShanghaiTech and XD-Violence datasets, we also conducted similar experiments and we find that 8 and 15 are the most suitable values for R, respectively. The results above demonstrate that our SP-MIL framework can adaptively select the optimal number of training samples for different datasets. Notably, when UR-DMU is used as the baseline model, the most suitable values of R for the ShanghaiTech, UCF-Crime, and XD-Violence datasets are 15, 21, and 33, respectively.

4.6. Comparison with the Prior SOTA

In this section, we primarily compare the performance of our proposed method DeepMIL+SP-MIL, RTFM+SP-MIL and UR-DMU+SP-MIL with other state-of-the-art methods.

Results on UCF-Crime. Table 6 shows that UR-DMU+SP-MIL achieved competitive results. For comparison, He et al. [28] focused on addressing the inter-video data imbalance problem by proposing an adversarial and focus training method. However, their performance was 2.69% lower than that of UR-DMU+SP-MIL. Similarly, Zhang et al. [11] introduced a method that focuses on pseudo-label completeness and uncertainty, employing Monte Carlo dropout to select confident video clips and enhance classifier performance. However, their results were slightly lower than those of the RTFM+SP-MIL method. In addition, our UR-DMU+SP-MIL method is comparable to that of Yang et al. [29], who incorporated additional textual information to enhance the performance of their model. Compared to MIST [9], which utilized a two-stage training strategy to improve the accuracy for pseudo-labels, the UR-DMU+SP-MIL method not only outperformed it by 5.46% but also demonstrated its versatility by improving the capabilities of classifiers with various baseline models.

• Results on ShanghaiTech. Table 7 presents the AUC results of different WS-VAD methods on the ShanghaiTech dataset. It can be observed that RTFM+SP-MIL achieves an AUC of 98.10%. This may be attributed to the fact that our framework can enhance the classifier’s discriminability through self-paced learning strategy. Compared to the noisy label cleaning strategy in GCN [1], our model demonstrates an AUC improvement of 13.66%, highlighting the benefit of refining segment-level pseudo-labels via the SP-MIL framework. Additionally, Ye et al. [30] and Wu et al. [16] improved the temporal modeling of pre-trained features. However, their performance remains slightly lower than that of our RTFM+SP-MIL method.

  Table 6. Comparisons of the frame-level AUC scores with other methods on the UCF-Crime dataset.

  Method	Source	Feature	AUC (%)
  DeepMIL [7]	CVPR 2018	C3D RGB	75.41
  DeepMIL * [7]	CVPR 2018	I3D RGB	78.32
  DeepMIL * [7]	CVPR 2018	CLIP	80.91
  GCN [1]	CVPR 2019	TSN RGB	82.12
  CLAWS [31]	ECCV 2020	C3D RGB	83.03
  MIST [9]	CVPR 2021	I3D RGB	82.30
  RTFM [5]	ICCV 2021	I3D RGB	84.30
  RTFM * [5]	ICCV 2021	I3D RGB	83.50
  RTFM * [5]	ICCV 2021	CLIP	84.45
  Chang et al. [14]	TMM 2021	I3D RGB	84.62
  Wu et al. [16]	TIP 2021	I3D RGB	84.89
  BN-SVP [32]	CVPR 2022	I3D RGB	83.39
  TCA-VAD [33]	ICME 2022	I3D RGB	83.75
  MSL [6]	AAAI 2022	I3D RGB	85.30
  Thakare et al. [34]	PR 2023	I3D RGB	83.56
  Liu et al. [13]	TNNLS 2023	I3D RGB	85.42
  Sun et al. [35]	ICME 2023	I3D RGB	85.88
  Cho et al. [36]	CVPR 2023	I3D RGB	86.10
  Zhang et al. [11]	CVPR 2023	I3D RGB	86.22
  He et al. [28]	PR 2024	I3D RGB	85.07
  AlMarri et al. [37]	WACV 2024	I3D RGB+FLOW	85.47
  Yang et al. [29]	CVPR 2024	CLIP (RGB+Text)	87.79
  DeepMIL+SP-MIL		I3D+CLIP	84.81
  RTFM+SP-MIL		I3D+CLIP	86.35
  UR-DMU+SP-MIL		I3D+CLIP	87.76

  * means that we re-implemented this method.

  Table 6. Comparisons of the frame-level AUC scores with other methods on the UCF-Crime dataset.

  Method	Source	Feature	AUC (%)
  DeepMIL [7]	CVPR 2018	C3D RGB	75.41
  DeepMIL * [7]	CVPR 2018	I3D RGB	78.32
  DeepMIL * [7]	CVPR 2018	CLIP	80.91
  GCN [1]	CVPR 2019	TSN RGB	82.12
  CLAWS [31]	ECCV 2020	C3D RGB	83.03
  MIST [9]	CVPR 2021	I3D RGB	82.30
  RTFM [5]	ICCV 2021	I3D RGB	84.30
  RTFM * [5]	ICCV 2021	I3D RGB	83.50
  RTFM * [5]	ICCV 2021	CLIP	84.45
  Chang et al. [14]	TMM 2021	I3D RGB	84.62
  Wu et al. [16]	TIP 2021	I3D RGB	84.89
  BN-SVP [32]	CVPR 2022	I3D RGB	83.39
  TCA-VAD [33]	ICME 2022	I3D RGB	83.75
  MSL [6]	AAAI 2022	I3D RGB	85.30
  Thakare et al. [34]	PR 2023	I3D RGB	83.56
  Liu et al. [13]	TNNLS 2023	I3D RGB	85.42
  Sun et al. [35]	ICME 2023	I3D RGB	85.88
  Cho et al. [36]	CVPR 2023	I3D RGB	86.10
  Zhang et al. [11]	CVPR 2023	I3D RGB	86.22
  He et al. [28]	PR 2024	I3D RGB	85.07
  AlMarri et al. [37]	WACV 2024	I3D RGB+FLOW	85.47
  Yang et al. [29]	CVPR 2024	CLIP (RGB+Text)	87.79
  DeepMIL+SP-MIL		I3D+CLIP	84.81
  RTFM+SP-MIL		I3D+CLIP	86.35
  UR-DMU+SP-MIL		I3D+CLIP	87.76

  * means that we re-implemented this method.

  Table 7. Comparisons of the frame-level AUC scores with other methods on the ShanghaiTech dataset.

  Method	Source	Feature	AUC (%)
  DeepMIL * [7]	CVPR 2018	I3D RGB	92.71
  DeepMIL * [7]	CVPR 2018	CLIP	94.96
  GCN [1]	CVPR 2019	TSN RGB	84.44
  CLAWS [31]	ECCV 2020	C3D RGB	89.67
  Chang et al. [14]	TMM 2021	I3D RGB	92.25
  MIST [9]	CVPR 2021	I3D RGB	94.83
  RTFM [5]	ICCV 2021	I3D RGB	97.21
  RTFM * [5]	ICCV 2021	I3D RGB	97.39
  RTFM * [5]	ICCV 2021	CLIP	97.70
  Wu et al. [16]	TIP 2021	I3D RGB	97.48
  BN-SVP [32]	CVPR 2022	C3D RGB	96.00
  S3R [38]	ECCV 2022	I3D RGB	97.48
  MSL [6]	AAAI 2022	I3D RGB	96.08
  Liu et al. [13]	TNNLS 2023	I3D RGB	97.54
  Liu et al. [13]	TNNLS 2023	I3D RGB+FLOW	97.76
  Cho et al. [36]	CVPR 2023	I3D RGB	97.60
  Sun et al. [35]	ICME 2023	I3D RGB	97.92
  Tan et al. [39]	WACV 2024	I3D RGB+FLOW	97.54
  Ye et al. [30]	ICASSP 2024	I3D RGB	98.00
  DeepMIL+SP-MIL		I3D+CLIP	97.81
  RTFM+SP-MIL		I3D+CLIP	98.10
  UR-DMU+SP-MIL		I3D+CLIP	97.92

  * means that we re-implemented this method.

  Table 7. Comparisons of the frame-level AUC scores with other methods on the ShanghaiTech dataset.

  Method	Source	Feature	AUC (%)
  DeepMIL * [7]	CVPR 2018	I3D RGB	92.71
  DeepMIL * [7]	CVPR 2018	CLIP	94.96
  GCN [1]	CVPR 2019	TSN RGB	84.44
  CLAWS [31]	ECCV 2020	C3D RGB	89.67
  Chang et al. [14]	TMM 2021	I3D RGB	92.25
  MIST [9]	CVPR 2021	I3D RGB	94.83
  RTFM [5]	ICCV 2021	I3D RGB	97.21
  RTFM * [5]	ICCV 2021	I3D RGB	97.39
  RTFM * [5]	ICCV 2021	CLIP	97.70
  Wu et al. [16]	TIP 2021	I3D RGB	97.48
  BN-SVP [32]	CVPR 2022	C3D RGB	96.00
  S3R [38]	ECCV 2022	I3D RGB	97.48
  MSL [6]	AAAI 2022	I3D RGB	96.08
  Liu et al. [13]	TNNLS 2023	I3D RGB	97.54
  Liu et al. [13]	TNNLS 2023	I3D RGB+FLOW	97.76
  Cho et al. [36]	CVPR 2023	I3D RGB	97.60
  Sun et al. [35]	ICME 2023	I3D RGB	97.92
  Tan et al. [39]	WACV 2024	I3D RGB+FLOW	97.54
  Ye et al. [30]	ICASSP 2024	I3D RGB	98.00
  DeepMIL+SP-MIL		I3D+CLIP	97.81
  RTFM+SP-MIL		I3D+CLIP	98.10
  UR-DMU+SP-MIL		I3D+CLIP	97.92

  * means that we re-implemented this method.
  Results on XD-Violence.

  Table 8. Performance comparison of state-of-the-art methods on the XD-Violence dataset.

  Method	Source	Feature	AP (%)
  DeepMIL * [7]	CVPR 2018	I3D RGB	73.24
  DeepMIL * [7]	CVPR 2018	CLIP	74.03
  Wu et al. [27]	ECCV 2020	C3D RGB	67.19
  MSL [6]	AAAI 2022	C3D RGB	75.53
  Wu et al. [16]	TIP 2021	I3D RGB	75.90
  RTFM [5]	ICCV 2021	I3D RGB	77.81
  RTFM * [5]	ICCV 2021	I3D RGB	78.07
  RTFM * [5]	ICCV 2021	CLIP	79.11
  MSL [6]	AAAI 2022	I3D RGB	78.28
  Sun et al. [35]	ICME 2023	I3D RGB	77.92
  MGFN [40]	AAAI 2023	I3D RGB	79.19
  Thakare et al. [34]	PR 2023	I3D RGB	79.89
  DMU [24]	AAAI 2023	I3D RGB	81.66
  Liu et al. [13]	TNNLS 2023	I3D RGB	79.00
  Zhang et al. [11]	CVPR 2023	I3D RGB	78.74
  Zhang et al. [11]	CVPR 2023	I3D+VGGish	81.43
  Cho et al. [36]	CVPR 2023	I3D RGB	81.30
  He et al. [28]	PR 2024	I3D RGB	80.07
  Tan et al. [39]	WACV 2024	I3D RGB	82.10
  Yang et al. [29]	CVPR 2024	CLIP	83.68
  DeepMIL+SP-MIL		I3D+CLIP	76.63
  RTFM+SP-MIL		I3D+CLIP	84.42
  UR-DMU+SP-MIL		I3D+CLIP	84.32

  * means that we re-implemented this method.

  shows the performance of our framework on the XD-Violence dataset compared to those of state-of-the-art (SOTA) methods. As can be seen from the table, RTFM+SP-MIL achieved the best results. With the CLIP feature extractor, the AP value of our RTFM+SP-MIL was 0.74% higher than that of Yang et al. [29]. The MGFN [40] network enhances the multi-scale temporal network based on the RTFM model, but its result is 5.23% lower than our RTFM+SP-MIL. Similarly, the AFT method [28] focus on recognizing abnormal events and also employ a late-fusion strategy, but our RTFM+SP-MIL results are 4.35% higher than those of He et al. [28]. These results demonstrate the effectiveness of our framework.

4.7. Qualitative Results

In this section, we present the qualitative results of our RTFM+SP-MIL and three baseline models (DeepMIL, RTFM, UR-DMU) on the UCF-Crime, ShanghaiTech, and XD-Violence datasets. Figure 5, Figure 6 and Figure 7 show that our framework can locate abnormal events more effectively than the baseline models. For the UCF-Crime dataset, Figure 5a–c demonstrates that the positioning curve of our framework closely aligns with the ground truth. Even when the abnormal events occur intermittently in Figure 5a, our detecting results are only slightly delayed or advanced. This highlights the robustness and accuracy of our framework in detecting abnormal events, such as arson and robbery. Additionally, Figure 5d illustrates that our framework has a low false positive rate. Similar conclusions can be drawn from the positioning curves of the ShanghaiTech and XD-Violence datasets. In addition, Figure 6b illustrates that the classifiers in RTFM and DeepMIL exhibit a bias in localizing anomalies, resulting in delayed anomaly detection, while the classifiers in UR-DMU cause significant false alarms. In contrast, our method effectively identifies the location of anomalies. This highlights the effectiveness of our SP-MIL strategy in providing unbiased predictions. Although our framework exhibits a few false positives in Figure 6a and Figure 7c, overall, our detecting results accurately promptly reflect the real anomalies in the videos.

5. Conclusions

In this paper, we model the WS-VAD task as a segment-level supervised learning problem and propose a general self-paced multiple-instance learning framework. The proposed framework can adaptively select training segments in each training epoch, progressing from easy to hard video segments, and alternately updates the self-paced learning weights and the classifier parameters to enhance the baseline classifier’s capability through unbiased prediction. Our SP-MIL framework can be applied to various baseline models (e.g., DeepMIL, RTFM, UR-DMU) and feature extractors (e.g., I3D, CLIP), demonstrating broad applicability and effectiveness. Furthermore, when using RTFM as the baseline model, our proposed SP-MIL framework achieved state-of-the-art results on the ShanghaiTech and XD-Violence datasets. When using UR-DMU as the baseline model, our framework achieved competitive results on UCF-Crime dataset.

6. Limitation and Future Work

This study has several limitations that open avenues for future research: (1) Text information integration. Although our current approach leverages pre-trained CLIP features, it lacks text information related to the video anomaly class that could enhance the model’s performance. Future work could investigate integrating text-based features to exploit the rich semantic context of video annotations, possibly resulting in more robust anomaly detection. (2) Sample selection mechanism. In our self-paced learning method, the number of samples involved in training increases with each epoch. While this approach allows for adaptive learning, current mechanisms for selecting segments could be refined. Exploring more sophisticated sample selection strategies, such as dynamic weighting or reinforcement learning-based methods, could optimize the learning process and improve the model’s ability to detect more subtle anomalies. (3) Baseline model diversity. Our study primarily utilizes three common baseline models, leaving many other potential model architectures unexplored. Future research should explore a broader range of model architectures to validate their effectiveness within the SP-MIL framework. This exploration could uncover new insights and further enhance the generalizability of our approach.