Deep Active Learning for Computer Vision Tasks: Methodologies, Applications, and Challenges

Abstract

Active learning is a label-efficient machine learning method that actively selects the most valuable unlabeled samples to annotate. Active learning focuses on achieving the best possible performance while using as few, high-quality sample annotations as possible. Recently, active learning achieved promotion combined with deep learning-based methods, which are named deep active learning methods in this paper. Deep active learning plays a crucial role in computer vision tasks, especially in label-insensitive scenarios, such as hard-to-label tasks (medical images analysis) and time-consuming tasks (autonomous driving). However, deep active learning still has some challenges, such as unstable performance and dirty data, which are future research trends. Compared with other reviews on deep active learning, our work introduced the deep active learning from computer vision-related methodologies and corresponding applications. The expected audience of this vision-friendly survey are researchers who are working in computer vision but willing to utilize deep active learning methods to solve vision problems. Specifically, this review systematically focuses on the details of methods, applications, and challenges in vision tasks, and we also introduce the classic theories, strategies, and scenarios of active learning in brief.

1. Introduction

With the rapid development of deep learning, the performance of computer vision tasks has achieved breakthroughs benefiting from large-scale annotated datasets, such as ImageNet [1], Cityscapes [2], and AbdomenCT-1K [3]. These datasets provide direct supervision for model training. Meanwhile, there are useless, uninformative examples, which serve as risks to overwhelm the training. Apart from the noise inside the labeled data, there are always scenarios where unlabeled data is abundant. However, manual labeling is high cost, such as medical image analysis, autonomous driving, anomaly detection, and related issues in computer vision tasks. Specifically, taking the Cityscapes [2] dataset as an example, the pixel-wise annotation will cost more than 90 min per image in the urban street segmentation task. Similarly, in the medical image tumor segmentation task, it is more challenging for medical experts to detect the mm-size objects from 3D volume data, which is more time-consuming and medical knowledge–demanding.

Under the above conditions, maximizing the model’s performance with a limited annotation budget becomes the primary concern. In order to figure out this problem, active learning becomes a promising solution to improve data efficiency and relieve the high annotation burden. As a subfield of machine learning, the core idea of active learning [4] is to find the most valuable samples through some heuristic strategies, so the model can achieve or even exceed the expected performance with as few labeled samples as possible. The intuition of active learning is that not all samples in a dataset have the same significance for model training. For example, some samples contain more noise that hinders training. Besides this, some samples are too similar to be worth labeling. Therefore, the goal of active learning is to select as few valuable or ambiguous samples as possible via the designed strategy and promote the performance with the selected samples interactively. The above iterative training process between optimization and annotator is the primary active learning mechanism, and human annotations exist in each training interaction. Consequently, the essence of active learning is the Human-in-the-Loop (HITL) computing systems, where human expertise is joint in the computer-based systems [5]. Humans (such as doctors in clinical diagnosis) are part of the intelligent system and participate in the model training process by providing judgments or domain-knowledge that influence the final output of the system [6]. More details are introduced in Algorithm 1 and other surveys [5,6].

Algorithm 1: The pool-based active learning workflow

Settles’s active learning literature survey [4] systematically summarized classic active learning methods in 2004. More basic definitions and formulations can be found in this survey [4]. Active learning has been rapidly growing and booming with various novel methodologies, applications and challenges in recent years. Settles’s survey provided the basic theory for active learning, especially the traditional AL. Differently, our work focuses on deep learning–based active learning theories in computer vision tasks, which is named deep active learning in this paper.

Apart from this, we summarize the latest surveys [6,7,8,9] about deep learning in

Table 1. The latest surveys about deep active learning.

Title	Main Contents	Publication
Samuel Budd et al. A survey on active learning and human-in-the-loop deep learning for medical image analysis [6]	⊳ Investigate the active learning in the medical image analysis. ⊳ Propose the considerations in the deep learning–based active learning, including noisy oracles, weakly supervised learning, multi-task learning, annotation interface, and variable learning costs. ⊳ Discuss the future prospective and unanswered questions in the medical image analysis.	Medical Image Analysis. 2021, 71, 102062
Punit Kumar et al. Active Learning Query Strategies for Classification, Regression, and Clustering: A Survey [7]	⊳ Summarize the active learning query strategies for three tasks, including classification, regression, and clustering. ⊳ Classify the query strategies under classification into: informative-based, representative-based, informative-and-representative-based, and others. ⊳ Summarize the empirical evaluation of active learning query strategies. ⊳ Present the implementation, application, and challenges of the active learning in brief.	Journal of Computer Science and Technology. 2020, 35, 913–945.
Pengzhen Ren et al. A Survey of Deep Active Learning [8]	⊳ Classify the existing works in the deep active learning. ⊳ Summarize the deep active learning applications, including vision and NLP. ⊳ Especially, in the visual data processing tasks, it discusses image classification and recognition, object detection and semantic segmentation, and video processing.	ACM Computing Surveys. 54.9 (2021): 1–40.
Xueying Zhan et al. A Comparative Survey of Deep Active Learning [9]	⊳ Categorize deep active learning sampling methods and querying strategies. ⊳ Compare deep active learning algorithms across common used datasets. ⊳ Conduct experiments to explore influence factors of deep active learning ⊳ Release a deep active learning toolkit, named DeepAL+.	arXiv:2203.13450, 2022.

. Previous surveys systematically introduced the deep active learning in many fields, such as CV and NLP. After studied the existing surveys about deep active learning and their references, we found that there was not any review designed for CV researcher. Hence, we decided to re-organize existing works and introduce latest research from a CV-related perspective in this manuscript. The biggest difference between this manuscript and above-mentioned works is that the expected audience of this review are researchers who are working in computer vision but willing to utilize deep active learning methods to solve CV problems. Active learning is still new to them. As such, we organized this manuscript from the perspective of a CV researcher, and introduced the deep active learning from CV-related methodologies and corresponding applications. This CV-friendly survey is our new contribution to the community.

The remainder of this review is as follows: First, we argue that it is necessary to introduce the basic concepts and methodologies of traditional active learning for the newcomers. Then, Section 2 introduces the three basic candidate selection strategies in active learning and give responding examples, and then we detail the pool-based strategy. Section 3 introduces the common querying scenarios in active learning. Then, we focus on the methodologies integrated deep learning and active learning. Section 4 details the recent methodologies for deep learning-based active learning. Section 5 details the applications of deep active learning, especially in computer vision tasks. Finally, Section 6 summarizes the existing challenges of deep active learning in computer vision tasks, which are the future works in this field. Section 7 concludes the survey.

2. Candidate Selection Strategies in Active Learning

In the classic active learning framework, one of the two most important components is how to develop a criterion for evaluating the “worthiness” of unlabeled samples. After evaluation, the selection strategies decide whether one sample is worthy of being labeled by the annotator according to its worthiness. The selected samples are regarded as candidates. Finally, an appropriate selection strategy reduces the labeling cost and thus has important implications in active learning. Due to this knowledge being beyond the main concern of this review,

Table 2. Summary of candidate selection strategies in active learning.

Strategies	Methodologies	Typical Works
Random selection	⊳ Random sampling is to use random numbers to select samples from the unlabeled dataset for labeling.	N.A.
	⊳ $\mathbb{X}=\underset{x\in U}{Random}\left(x\right)$
Uncertainty-based selection	⊳ Least confidence is to select the sample with the smallest probability of the top1 predicted class. In practice, the opposite of the maximum predicted probability is often taken as the uncertainty score of the sample.	Li et al. [10]
	⊳ $\mathbb{X}=\underset{x\in U}{\arg \max }\left[1-P(\widehat{y}\mid x)\right]=\underset{x\in U}{\arg \min }P(\widehat{y}\mid x)$	Agrawal et al. [11,12]
	⊳ Margin sampling is to calculate the difference between the probabilities of the top1 and the top2 predicted class.Then the samples with the smallest difference are defined as hard-to-classify samples for labeling.	Ajay J. et al. [13]
	⊳ $\mathbb{X}=\underset{x\in U}{\arg \min }\left(P\left({\widehat{y}}_1\mid x\right)-P\left({\widehat{y}}_2\mid x\right)\right)$	Zhou et al. [14]
	⊳ Multi-class level uncertainty is to select the two samples that are the farthest from the classification hyperplane of multi-class and take their distance difference as the score.	Gu et al. [15]
	⊳ $\mathbb{X}=\underset{x\in U}{\arg \min }\left\{D\left(x,{\widehat{y}}_1\right)-{\max }_{y\ne {\widehat{y}}_1}D\left(x,y\right)\right\}$	Yang et al. [16]
	⊳ Maximize entropy is to utilize the methodology that larger entropy denotes higher uncertainty. The sample with the largest entropy is selected as candidate.	Yu et al. [17]
	⊳ $\mathbb{X}=\underset{x\in U}{\arg \max }{E}_x=\underset{x\in U}{\arg \min }\left\{{\sum }_{i=1}^{Y}P(y_i\mid x)\times \log P(y_i\mid x)\right\}$	Ozdemir et al. [18]
Diversity-based selection	⊳ Angle-based measurement is to to measure diversity by calculating the undirected angles between the induced hyper-planes.	Brinker et al. [19]
	⊳ $\left|\cos \left(\angle \left(h_i,h_j\right)\right)\right|=\frac{\left|〈\varphi \left(x_i\right),\varphi \left(x_j\right)〉\right|}{∥\varphi \left(x_i\right)∥∥\varphi \left(x_j\right)∥}=\left|k\left(x_i,x_j\right)\right|/\sqrt{k\left(x_i,x_i\right)k\left(x_j,x_j\right)}$
	⊳ Redundancy-based measurement is to measure the diversity as the redundancy between unlabeled points via symmetric KL divergence [20] between the two vectors of probability values.	Shayok et al. [21]
	⊳ $R(x_i,x_j)={\sum }_y^{\left|Y\right|}\left[P\left(y\mid x_i\right)-P\left(y\mid x_j\right)\right]\log \left[P\left(y\mid x_i\right)/P\left(y\mid x_j\right)\right]$	Zhou et al. [22]
Committee-based selection [23]	⊳ Vote entropy–based measurement is to select the hard sample voted by the Committee. The models in the committee distinguish samples into different classes. The predicted results toward one sample with the largest entropy is classified as hard sample and needs to vote for labeling.	Yan et al. [24]
	⊳ $\mathbb{X}=\underset{x\in U}{\arg \min }\left\{-\frac{1}{N}{\sum }_{i=1}^Y\left[Vote\left(y_i\right)\log \frac{Vote\left(y_i\right)}{N}\right]\right\}$	Dagan et al. [25]
	⊳ Average KL divergence–based measurement is to measure the deviation of those unlabeled samples via calculating the average KL divergence of the committee $\mathcal{C}$.	Dagan et al. [25]
	$\mathbb{X}=\underset{x\in U}{\arg \max }\frac{1}{N}{\sum }_{i=1}^{N}KL\left(P_{{\mathcal{M}}_i}\parallel P_{\mathcal{C}}\right)=\underset{x\in U}{\arg \max }\frac{1}{N}{\sum }_{i=1}^N{\sum }_j^{\left|Y\right|}{P}_{{\mathcal{M}}_j}\left(y_j\mid x\right)\log \frac{P_{{\mathcal{M}}_i}\left(y_j\mid x\right)}{P_{\mathcal{C}}\left(y_j\mid x\right)}$

Note: $\mathbb{X}$ represents the selected sample, U is the unlabeled data pool, $\widehat{y_1}$ is the predicted class with max probability, $P(\widehat{y_1}\mid x)$ is the conditional probability where the input is x, and the predicted class is $\widehat{y_1}$. $\widehat{y_2}$ denotes the top2 predicted class. $D(x,y)$ represents the distance from the sample x to the classification hyperplane of class y. $\varphi \left(x_i\right)$ is the normalization function, $k(x_i,x_j)=〈\varphi \left(x_i\right),\varphi \left(x_j\right)〉$ is the kernel function. |Y| is the total number of classes, $P(y\mid x)$ is the conditional probability where the input is x_i, and the predicted class is y. Vote(y_i) denotes the number of models that voted for the current class y_i, and ${\sum }_{i=1}^{\mid Y\mid }$ Vote(y_i) = N. ${\mathcal{M}}_i$ represents a specific model in the committee. KL(P₁∥P₂) is the KL divergence between two distributions P₁ and P₂.

introduces four basic selection strategies in brief, and more details can be found in existing active learning surveys [6,7,8,9].

2.1. Random Selection Strategies

Random sampling uses random numbers to select samples from the unlabeled dataset for labeling. There is no interaction with the model’s prediction in the above process, which is the most naive selection strategy. Consequently, it is often used as the basic comparison experiment in active learning.

2.2. Uncertainty-Based Selection Strategies

The uncertainty-based selecting method is the simplest and most common strategy, which assumes that the samples closest to the classification hyperplane have richer information than others. It selects the most uncertain samples according to the predicted value of the samples by the current model.

Typical uncertainty-based selection strategies includes Least confidence, Margin sampling, Multi-class level uncertainty, and Maximize entropy. This survey briefly summarized the above strategies in Table 2. More details can be found in existing surveys [6,7,8,9]. Consequently, the machine learning model can quickly improve its performance by learning the labels of the samples with substantial uncertainty.

2.3. Diversity-Based Selection Strategy

The above uncertainty-based selection strategy can effectively sample a single candidate for annotation, but it is often ineffective when there are multiple candidates. At this time, the selection strategy based on the diversity of sample feature distribution comes into being. Diversity tends to reflect the prediction consistency among the samples, i.e., higher diversity values denote more inconsistency between the candidate sample and the entire unlabeled pool [26]. Typical diversity-based measurement strategies includes angles-based and redundancy-based perspectives. We also summarized above strategies in Table 2 and provided the comparison between uncertainty-based and diversity-based selections in Figure 1.

2.4. Committee-Based Selection Strategy

Committee-based selection strategy [23] is based on version space reduction, and its core idea is to preferentially select unlabeled samples that can reduce the version space to the greatest extent. The committee-based selection strategy’s motivation is that the most informative selections are the samples where the committee predicts the most inconsistent. Typical committee-based selection strategies include vote entropy and average KL divergence, which are listed in Table 2. There are four basic steps in the committee-based selection strategy:

• Multiple models $\left\{{\mathcal{M}}_1,\dots ,{\mathcal{M}}_N\right\}$ are used to construct a committee $\mathcal{C}$ for voting, i.e., $\mathcal{C}=\left\{{\mathcal{M}}_1,\dots ,{\mathcal{M}}_N\right\}$.
• The models in the committee $\mathcal{C}$ are then trained on the labeled dataset L and get different parameters.
• All models in the committee make predictions separately on unlabeled samples from $\mathcal{U}$. The samples with the richest information are voted.
• The samples which obtain the most disagreements are selected as candidates for labeling.

3. Common Querying Scenarios in Active Learning

According to the application scenarios, active learning methods can be divided into three types: query synthesis scenario, stream-based scenario, and pool-based scenario. We briefly summarize the above querying scenarios in

Table 3. Summary of common querying scenarios in active learning.

Scenarios	Concepts	Limitations	Publications
Membership query synthesis	⊳ The membership query synthesis is to generate new unlabeled instances for querying by itself instead of selecting samples from the real-world distribution [27].	⊳ It may encounter troubles when the generated data is too arbitrary for the annotator to recognize or does not contains any semantic information.	[28,29]
Stream-based sampling	⊳ The stream-based scenario [30] is to sample from the natural distribution instead of the synthesized one. ⊳ In this scenario, the selection process is similar to a pipeline. The unlabeled sample is firstly input into the model one by one. ⊳ Then, the active learning strategy needs to decide whether to pass it to the annotator for labeling or reject it directly.	⊳ It is necessary for the model to immediately decide based on a single input rather than the comprehensive consideration of this batch. ⊳ The active learning system may suffer from the absence of knowledge of unseen areas.	[31,32,33]
Pool-based sampling	⊳ The pool-based sampling scenario is to selects the most valuable samples from an unlabeled data pool for labeling according to the informativeness [34]. ⊳ The unlabeled data pool is sampled from the natural distribution instead of synthesized samples.	⊳ It is computationally expensive because every iteration requires the informativeness evaluation for the whole pool.	[35,36,37]

and introduce the pool-based scenario in detail.

Among the above mainstreams, pool-based active learning is more compatible with batch-based modern deep learning training mechanisms. Compared with the stream-based selective sampling, the pool-based scenario is able to consider every sample based on this batch comprehensively. Consequently, it has become the most common scenario in computer vision tasks. Moreover, the most related works introduced in this review also belong to the pool-based active learning scenario.

Figure 2 is a classic pool-based active learning framework. A single batch of unlabeled samples is input to the model from the unlabeled data pool during the training process. Then, the query strategy selects the most valuable samples for labeling according to the informativeness. After that, these labeled samples are added to the labeled dataset to continue training the model.

Sequentially, we formally define the pool-based active learning method in Algorithm 1. The End conditions include that the performance of the model meets requirements, or the budgets for annotation run out, or the selected samples are hard for annotators to label.

As shown in Figure 2, there are the labeled pool $L=\{(x_1^l,y_1^l),\cdots ,(x_{N_l}^l,y_{N_l}^l)\}$ and the unlabeled pool $\mathcal{U}=\{x_1^u,\cdots ,x_{N_u}^u\}$ at the beginning. $N_l$ and $N_u$ are the number of labeled and unlabeled samples, respectively. Then, the data from the labeled pool is fed into the machine learning model for supervised training. After that, the well-trained model is utilized to extract the representation ${\mathcal{R}}^u=\{(r_1^u),\cdots ,(r_{N_u}^u)\}$ of the unlabeled pool data. Based on ${\mathcal{R}}^u$, the informativeness is calculated according to the query strategy. The top-K samples $\mathcal{K}=\{x_1^k,\cdots ,x_{N_k}^k\}$ are selected out to the oracle (human annotators) and obtain the labels $Y^{\mathcal{K}}=\{y_1^k,\cdots ,y_{N_k}^k\}$. Finally, the labeled pool L will be updated to $L=\{(x_1^l,y_1^l),\cdots ,(x_{N_l}^l,y_{N_l}^l),(x_1^k,y_1^k),\cdots ,(x_{N_k}^k,y_{N_k}^k)\}$. With the updated L, the machine learning model will promote the performance in return. Meanwhile, the size of unlabeled pool $\mathcal{U}$ is reduced to $N_u-N_k$. The above loop will be repeated until one of the end conditions is met. Since the selected samples $\mathcal{K}$ from the unlabeled pool $\mathcal{U}$ are the most significant ones for training, the size of L to learn a model can often be much smaller than the size required in classic supervised learning without active learning.

4. Deep Active Learning Methods

Recent developments are dedicated to multi-label active learning, hybrid active learning, and deep learning–based active learning. In the upcoming sections, we will detail deep learning–based active learning.

4.1. Deep Active Learning for CNNs

As we introduced in Section 2.2, uncertainty is one of the most used metrics to select candidate in active learning. We summarize the uncertainty estimation methods in

Table 4. Summary of uncertainty estimation methods in deep active learning.

Type	Methodology	Equation
MC dropout	⊳ In practice, the MC dropout usually trains the CNN with the labeled data pool L with dropout. ⊳ After training, it generates a new dropout mask for the model parameters ${\mathcal{M}}_t$ and performing T forward inference. ⊳ The output is the average of T results.	$\begin{array}{l}p\left(y\mid x,L\right)=\\ \frac{1}{T}{\sum }_{t=1}^{T}p\left(y\mid x,{\mathcal{M}}_t\right)\end{array}$
Deep Ensembles	⊳ The ensemble-based approaches design N neural networks $\{{\mathcal{M}}_1\cdots {\mathcal{M}}_N\}$ at first. ⊳ These networks share same architecture but initialized from different weights. ⊳ Then networks are trained with the labeled data pool L. ⊳ The average of the outputs of the N networks is the final output.	$\begin{array}{l}p\left(y\mid x,L\right)=\\ \frac{1}{N}{\sum }_{i=1}^{N}p\left(y\mid x,{\mathcal{M}}_i\right)\end{array}$

. At the same time, Bayesian methods are known for their ability to capture underlying model uncertainty. The classic Bayesian active learning framework consists of an unlabeled data pool $\mathcal{U}$, the labeled data pool L, and a Bayesian model $\mathcal{M}$ trained on the current L. The output of the Bayesian model $\mathcal{M}$ is $p\left(y\right|x,\mathcal{M},L)$, where the input data is x and the prediction $y\in \{1,...,c\}$ in the classification tasks. In Bayesian deep learning, the model $\mathcal{M}$ is replaced by a Convolutional Neural Network (CNN) with prior probability distributions. Gal et al. [38] proposed a Bernoulli-based approximate variational inference method. After that, they [39] proposed to capture model uncertainty by using the Monte Carlo dropout regularization as a variational Bayesian approximation. Consequently, it is natural to utilize Bayesian methods to actively select valuable candidates.

Gal et al. [40] introduced the Bayesian Convolutional Neural Networks into the active learning framework. They demonstrated that such combination improved performance over existing active learning methods on the image classification dataset MNIST [41] (achieving 5% test error) and skin cancer diagnosis dataset ISIC 2016 [42] (achieving 0.71/0.75 AUC). Bayesian Active Learning by Disagreement (BALD) [43] was proposed to be the basic selection strategy, where Shannon entropy [44] was utilized to measure the “information content”. The discrepancy of Shannon entropy denoted the difference between the information entropy of a certain sample and the average information entropy of the dataset. The larger the difference, the more information the sample contained relative to the average. Finally, the BALD strategy pushed the samples with the largest Shannon entropy. The above process is formulated as follows.

$$\mathbb{X}=\underset{\mathbf{x}\in \mathcal{U}}{\arg \max }\mathbb{I}\left(x\right)=\underset{\mathbf{x}\in \mathcal{U}}{\arg \max }SE(\mathbf{y}\mid \mathbf{x},L)-\underset{\theta \sim P(\theta \mid L)}{\mathbb{E}}\left[SE(\mathbf{y}\mid \mathbf{x},\mathcal{M})\right],$$

(1)

where $\mathbb{I}(·)$ is the mutual information. Higher mutual information means the model’s predictions are more uncertain. $SE(\mathbf{y}\mid \mathbf{x},L)$ and $SE(\mathbf{y}\mid \mathbf{x},\mathcal{M})$ are represented by the Shannon Entropy of the prediction $P_{\mathcal{M}}(\mathbf{y}\mid \mathbf{x})$ and the mean distribution $P_{\mathcal{M}}(\mathbf{y}\mid L)$, respectively.

Ensemble learning is a well-known technique in machine learning that improves performance by integrating different models and combining their results [45]. Ref. [46] explored the difference between ensemble-based methods against Monte Carlo dropout methods on image classification tasks MNIST [41] (achieving 90% test set accuracy with roughly 12,200 labeled images), CIFAR-10 [47] (achieving 91.5% accuracy) and diabetic retinopathy (DR) detection (https://www.kaggle.com/competitions/diabetic-retinopathy-detection/rules, accessed on 9 July 2022) (achieving 0.983 AUC). They conducted extensive experiments from 11 aspects and found that the former outperformed the latter and was a more reliable measure of uncertainty.

Bayesian-based methods addressed the problem of uncertainty-based candidate selection strategies, but there was another obstacle that needed to be solved. The biggest difference of CNN-based deep learning (DL) methods and traditional active learning (AL) methods is that AL methods query candidates one by one while DL methods load a batch size of samples at the same time. Ozan Sener and Silvio Savarese [48] conducted experiments on vision tasks with traditional active learning methods and found that previous AL works did not perform well for CNN-based vision tasks due to the batch settings during model training. They attributed this ineffectiveness to batch sampling. In order to solve it, they defined the active learning as a Core-Set selection problem, where the algorithm aims to train on a small mount of labeled samples instead of the whole dataset such that the trained model is able to get competitive performance over the model trained on the whole dataset. They defined the core-set selection problem as the following optimization:

$$\underset{L^1:\left|L^1\right|\le b}{\min }\left|\frac{1}{n}\sum _{i\in \left[n\right]}loss\left({\mathbf{x}}_i,y_i;{\mathcal{M}}_{L^0\cup L^1}\right)-\frac{1}{\left|L^0+L^1\right|}\sum _{j\in L^0\cup L^1}loss\left({\mathbf{x}}_j,y_j;{\mathcal{M}}_{L^0\cup L^1}\right)\right|,$$

(2)

where $L^0$ represents the randomly selected samples at the beginning, $L^1$ represents newly selected samples from the entire dataset under budget b. n is the number of samples in the entire dataset. ${\mathcal{M}}_{L^0\cup L^1}$ denotes the trained model under the subset $L^0$ and $L^1$. $loss$ is the loss function, where the authors suggested the cross-entropy loss for effective training. In order to prove their hypothesis, they conducted experiments on active learning for fully supervised models and weakly supervised models. Specifically, they used dataset CIFAR10/100 [47] for image classification and dataset SVHN [49] for digit classification.

Based on the Core-Set strategy, the combination of batch-based active learning and deep learning has been a researcher topic in the community. the goal of batch-based deep active learning is to select the most informative batch or mini-batch ${\mathcal{B}}^{*}=\left\{x_1^{*},x_2^{*},\dots ,x_n^{*}\right\}$ from the loaded batches $\mathcal{B}$, where $\mathcal{B}$ belongs to unlabeled pool $\mathcal{U}$, and n is the batch size. We formulate above process as follows.

$${\mathcal{B}}^{*}=\underset{\mathcal{B}\subseteq U}{\arg \max }\mathsf{\Phi }\left(\mathcal{B},\mathcal{M}\left(L\right)\right),$$

(3)

where $\mathsf{\Phi }(·)$ is the score function to measure the informativeness of the batch $\mathcal{B}$, L is the labeled data pool, $\mathcal{M}$ is the trained model.

After that, most of the related research was devoted to the innovation of the scoring function $\mathsf{\Phi }(·)$. David Janz et al. [50] adopted the idea of Bayesian Active Learning by Disagreement (BALD) [43] into scoring function. Specifically, they utilized the mutual information $\mathbb{I}(·)$ as score function and selected samples with the maximum gain of information, where ${\mathsf{\Phi }}_{BALD}(\mathcal{B},\mathcal{M}\left(L\right))={\sum }_{i=1}^n\mathbb{I}\left(y_i;\mathcal{M}\mid x_i,L\right)$. However, BALD just considered the mutual information between one single sample $x_i$ and model parameters $\mathcal{M}\left(L\right)$, and did not take the relationship between samples in an batch. As an extension of BALD, BatchBALD [51] promoted the BALD by estimating the mutual information $\mathbb{I}(·)$ between all samples in an batch and the model, which was formulated as ${\mathsf{\Phi }}_{BatchBALD}(\mathcal{B},\mathcal{M}\left(L\right))=\mathbb{I}\left(y_{1:n};\mathcal{M}\mid x_{1:n},L\right)$.

Yoo et al. [52] proposed a novel loss prediction module into the target model. This module consisted of global average pooling (GAP), full connected layer (FC) and ReLU, capturing multi-level features. Then the features was concentrated and calculated the loss prediction. All unlabeled samples were evaluated by this module via mini-batch. The batch with the top-K predicted losses selected as candidates and then labeled to update training set. the proposed module was evaluated in Image Classification task CIFAR-10 [47] (achieving accuracy of 0.9101), Object Detection task PASCAL VOC [53] (achieving 0.7338 mAP), and Human Pose Estimation task MPII [54] (achieving 0.8046 [email protected]).

4.2. Generative Adversarial Active Learning

According to the analysis of [55], the Core-Set strategy [48] is very inefficient in high-dimensional representation learning due to its inherent distance-based computation. This obstacle is well addressed by leveraging GAN or VAE for representation learning from high-dimensional data.

Generative Adversarial Networks (GAN) is a novel representation learning method proposed by Goodfellow [56]. Its core idea is “Generative” and “Adversarial”. The GAN network structure contains two models. One is the generator $\mathbb{G}$ and the other is the discriminator $\mathbb{D}$. The generator generally uses a deconvolutional neural network or a fully connected neural network to synthesize new data (e.g., images). At the same time, the discriminator is a CNN-based binary classifier to distinguish whether the input is from the natural distribution or synthesized one from the generator.

The discriminator is trained first to make a good judgment so that the real and generated samples can be better distinguished. Then the parameters of the generator can be updated more accurately. The goal of discriminator is that $P\left(\mathbb{D}\right(x\left)\right|x\in real)=1$ while $P\left(\mathbb{D}\right(x\left)\right|x\in fake)=0$. Then the generator and discriminator in GANs are trained against each other in a two-player game. The weights and biases of the discriminator and generator are trained through back-propagation until they reach a dynamic equilibrium state with unlabeled samples. In order to discriminate samples and classify them, the discriminator usually utilizes the cross-entropy loss to measure similarity, which is formulated as follows [56]:

$${\mathcal{L}}_{\mathbb{D}}=\frac{1}{N}\sum _{i=1}^N\left[\log \mathbb{D}\left({\mathit{x}}^{\left(i\right)}\right)+\log \left(1-\mathbb{D}\left(\mathbb{G}\left({\mathit{z}}^{\left(i\right)}\right)\right)\right)\right]$$

(4)

$${\mathcal{L}}_{\mathbb{G}}=\frac{1}{N}\sum _{i=1}^N\log \left(1-\mathbb{D}\left(\mathbb{G}\left({\mathit{z}}^{\left(i\right)}\right)\right)\right)$$

(5)

$\mathit{z}\sim p_{\mathit{z}}\left(\mathit{z}\right)$ is the generated distribution and $\mathit{x}\sim p_{\mathrm{data}}\left(\mathit{x}\right)$ is the real distribution. N denotes the batch size. As a consequence, the objective function of GAN is shown as follows [56]:

$$\mathcal{L}=\arg \underset{\mathbb{G}}{\min }\underset{\mathbb{D}}{\max }V(\mathbb{G},\mathbb{D})={\mathbb{E}}_{\mathit{x}\sim p_{\mathrm{data}}\left(\mathit{x}\right)}\left[\log \mathbb{D}\left(\mathit{x}\right)\right]+{\mathbb{E}}_{\mathit{z}\sim p_{\mathit{z}}\left(\mathit{z}\right)}\left[\log (1-\mathbb{D}\left(\mathbb{G}\left(\mathit{z}\right)\right))\right],$$

(6)

where $V(\mathbb{G},\mathbb{D})$ is the difference between the generated distribution $p_{\mathit{z}}$ and the actual distribution $p_{\mathrm{data}}$. max is to train the discriminator to discriminate the sample to the greatest extent. min is to train the generator to minimize the difference between the generated sample and the actual sample. When the algorithm converges, the samples generated by the generator can confuse the discriminator and cannot distinguish right from wrong. In other words, the generator should try its best to generate more realistic results to deceive the discriminator. The discriminator should try its best to distinguish the truth from the false and not be deceived by the generator. The network reaches the ideal state when the generated result is actual (discrimination probability is 0.5).

Zhu et al. [57] firstly proposed a novel query synthesis-based active learning method GAAL fused with GAN. GAAL was trained on datasets MNIST [41] (achieving accuracy 70.44%) and CIFAR-10 [47] while tested on the dataset USPS [58]. The workflow of GAAL can be concluded as Algorithm 2.

Algorithm 2: The synthesis-based active learning method workflow

GAAL inspired GAN-based generative adversarial methods in active learning. Consequently, the latter works were devoted to studying pool-based Generative Adversarial Active Learning. Tran et al. [59] proposed a framework of Bayesian Generative Active Learning (BGAL) to solve multi-classification tasks when the amount of labeled data is small. The proposed BGAL was validated on MNIST [41] (achieving accuracy 99.68%), CIFAR-10/100 [47] (achieving accuracy 91.13%), and SVHN [49] (achieving accuracy 69.69%). Mayer et al. [60] proposed a pool-based active learning strategy(ASAL). Compared to traditional pool-based strategies for exhaustive uncertainty search from unlabeled pools, ASAL utilized GAN to generate the most representative samples from unlabeled pools, resulting in more efficient active learning techniques. ASAL was validated on the datasets MNIST [41] (reducing 300 labeled samples), CIFAR-10 [47] (reducing 500 labeled samples), CelebA [61] (reducing 750 labeled samples), SVHN [49], and LSUN Scenes [62]. Sinha et al. [63] proposed a pool-based semi-supervised active learning algorithm VAAL. VAAL obtained great improvement in experimental results on classification and segmentation. VAAL achieved great improvement in experimental results, including CIFAR10/100 [47] (achieving accuracy of 90.16%/63.14%), Caltech-256 [64] (achieving 1.01% improvement on Core-set), ImageNet [1], Cityscapes [2] (achieving mIoU 62.95), and BDD100K [65] (achieving mIoU 44.95).

The above methods were devoted to directly generating the representative samples by solving some optimization problems, and then improving the efficiency of screening samples for active learning. More details are summarized in

Table 5. Summary of generative methods in deep active learning.

Methods	Innovation	Architecture	Comments
GAAL [57]	⊳ The first novel query synthesis-based active learning method GAAL fused with GAN. ⊳ GAAL combined query synthesis with the uncertainty sampling principle and adaptively synthesized training instances for querying to increase learning speed. ⊳ The DCGAN was implemented to replace the unlabeled pool in previous work.	⊳ Generator: CNN ⊳ Discriminator: CNN ⊳ Predictor: SVM ⊳ Score-function: Uncertainty	⊳ GAAL was the first work integrated active learning and generative methods. ⊳ GAAL provided rich representation training samples for active learning via GAN. ⊳ GAAL was limited by the generated abnormal instances if the GAN was not optimized correctly. ⊳ GAAL is limited by the binary classification setting.
BGAL [59]	⊳ BGAL integrated deep active learning and data augmentation methods to generate informative samples and expand the labeled data set to improve the accuracy of model classification. ⊳ BGAL also integrated ACGAN [67] and VAE-GAN [68] into a novel generative model named VAE-ACGAN, where the VAE decoder was the generator of the GAN. ⊳ VAE-ACGAN generated new synthetic instances on the query samples. ⊳ The learner and the VAE-ACGAN were jointly trained in this work.	⊳ Generator: VAE ⊳ Discriminator: Bayesian CNN ⊳ Predictor: Resnet18 ⊳ Score-function: MC-dropout	⊳ BGAL extended the GAAL by combined more robust data augmentation techniques. ⊳ The combination of data augmentation and active learning obtained consistent improvement on classification than single methods. ⊳ The computation efficiency need to be improved due to the computational cost is high.
ASAL [60]	⊳ ASAL consists of uncertainty sampling, adversarial sample generation, and sample matching. ⊳ In order to approximate the underlying data distribution from the unlabeled data pool, ASAL utilized a GAN to generate adversarial samples. ⊳ ASAL designed an efficient matching algorithm, where an uncertainty score was calculated to measure the similarity between the unlabeled samples and the generated samples. ⊳ ASAL selected the most similar samples from the pool and performs annotation.	⊳ Generator: CNN with matching ⊳ Discriminator: CNN ⊳ Predictor: CNN ⊳ Score-function: Entropy	⊳ ASAL was the first pool-based generative active learning method. ⊳ The main contribution of ASAL was to select the most similar sample from pool instead of directly annotating it via a matching algorithm. ⊳ ASAL utilized the entropy for uncertainty estimation and was applied in the multi-label classification.
VAAL [63]	⊳ VAAL utilized adversarial learning to promote active learning. ⊳ A variational autoencoder (VAE) was used to extract image features, and then a discriminator decided whether the image was labeled or unlabeled. ⊳ The VAE hoped to trick the discriminator into judging all samples as labeled data, but the discriminator hoped to accurately distinguish unlabeled samples in the data pool. ⊳ The annotator labeled the unlabeled samples selected based on this method.	⊳ Generator: VAE ⊳ Discriminator: MLP ⊳ Predictor: VGG16 ⊳ Score-function: Confidence	⊳ VAAL provided a computational efficient sampling method with the best accuracy and time cost.

. Apart from that, Huijser et al. [66] firstly proposed to use a GAN to generate a batch of samples along the decision boundary of the current classifier. Next, they determined the location where the category changed from the generated samples through visualization and added it to the set of samples to be labeled. Finally, the method’s effectiveness was verified by a large number of image classification experiments. The method can reduce the burden of data annotation by requiring the annotator to label decision boundaries instead of samples.

4.3. Semi-Supervised Active Learning

Vision tasks based on supervised learning require a large amount of labeled data for model training. These labeled data not only provide direct supervision signals but also limit the generalization ability of the model [69]. At the same time, the acquisition of these data is challenging due to the cost and time of annotation [26]. To alleviate this limitation, methods based on semi-supervised learning have become another mainstream. It studies model training under the premise of limited labeled data. It expects higher performance to balance the dilemma between performance and cost, which is a perfect fit for active learning, thus bringing a lot of research and practical value.

Semi-supervised learning (SSL) can train the model with a small amount of labeling cost. Unlike active learning, SSL methods usually select samples with confident prediction results instead of uncertain samples and then label them directly by the model instead of annotators. However, it is still impossible to guarantee that these high-confidence prediction results do not contain noise or erroneous results due to model prediction accuracy errors. Thus, these predictions could not directly participate in model training because the pseudo-labels may make the model abnormal. In contrast, active learning selects the samples with the most uncertain prediction results to be labeled by annotators, which can be used as the ground truth without noise and thus ensure the quality of labels. Therefore, the combination of semi-supervised learning methods and active learning methods can complement each other to a certain extent.

McCallumzy et al. [70] firstly proposed the above idea that combined Query-by-Committee active learning and Expectation-Maximization (EM) algorithms, using the naive Bayes method as a classifier and conducting experiments on text classification tasks. Subsequently, Muslea et al. [71] promoted the above work and proposed the joint testing method (Co-Testing), where two classifiers were trained in different perspectives. After that, samples were jointly selected for annotation. Finally, new labeled data were joined into the expectation-maximization (Co-EM) algorithm. Sequentially, Zhou et al. [72] combined the above semi-supervised learning and active learning methods and then validated that both of them are beneficial to the image retrieval task.

In addition, the self-training algorithm is one of the primary methods in SSL, and its core steps are shown in Algorithm 3. Firstly, the self-training algorithm initializes the model with a small number of labeled samples to ensure the initial performance of the model. Then, the algorithm selects appropriate samples and calculates their corresponding predicted labels based on the predicted representations. Finally, the labeled dataset is updated with new pseudo-labeled samples and the model is trained again in the next iteration. The main challenge of the semi-supervised learning algorithm is that SSL is easy to introduce a large number of noise samples during the training process, so the model cannot learn the correct information. Some researchers mitigate noisy samples by constructing Co-Training [73] and Tri-Training [74] algorithms of multiple classifiers.

Algorithm 3: The workflow of basic self-training algorithm

Apart from this, the authors of Refs. [75,76,77] integrated semi-supervised learning and active learning skillfully. They combined uncertainty-based selection strategies and self-training methods and made full use of their respective advantages while making up for their shortcomings. Consequently, their works achieved remarkable results.

However, the above semi-supervised active learning methods have not dealt with the noisy samples effectively, so it still had a significant impact on the model.

Song et al. [78] combined active Learning and semi-supervised Learning with inconsistent prediction and utilized data augmentations. These works achieved remarkable performance in CIFAR-10/100 [47] (improving 1.47%/1.16% accuracy) and SVHN [49] (improving 0.43% accuracy) classification tasks. However, they still suffered from data augmentation because there were too little data augmentations to estimate inconsistency. Sequentially, Guo et al. [79] proposed REVIVAL method and obtained more semantic distribution information via learning the continuous local distribution of unlabeled samples from feature space.

Despite the progress, most active learning algorithms suffer from data waste problems because they ignore that most of the data in unlabeled pool is not actively annotated, which can further enhance the performance via semi-supervised learning (SSL).

4.4. Active Contrastive Learning

Semi-supervised learning still needs some labeled data to carry out training, but self-supervised learning extracts representation by mining data instead of annotation. Contrastive learning is one of the most successful paradigms for self-supervised learning. The key idea of contrastive learning is to learn its relationship by comparing the similarity of different samples in the dataset. Thus, how to define the positive pairs (similar samples) and negative pairs (dissimilar samples) is the main issue in contrastive learning. The workflow of basic contrastive active learning is reported in the Algorithm 4. For arbitrary data x, the goal of contrastive learning is to learn an encoder $f(·)$ such that:

$$\mathrm{score}\left(f\left(x\right),f\left(x^{+}\right)\right)>>\mathrm{score}\left(f\left(x\right),f\left(x^{-}\right)\right),$$

(7)

where $f\left(x\right)$ denotes the global features. $f\left(x^{+}\right)$ denotes the local features from positive samples. $f\left(x^{-}\right)$ denotes the local features from negative samples. The $\mathrm{score}(\overrightarrow{X_1},\overrightarrow{X_2})$ is the function to measure the similarity of vectors $\overrightarrow{X_1}$ and $\overrightarrow{X_2}$. Euclidean distance and cosine similarity are two classical score functions, which are formulated as follows.

$$Euclidean\left(\overrightarrow{X_1},\overrightarrow{X_2}\right)={∥F\left(\overrightarrow{X_1}\right)-F\left(\overrightarrow{X_2}\right)∥}_2$$

(8)

$$Cosine\left(\overrightarrow{X_1},\overrightarrow{X_2}\right)=\frac{\overrightarrow{X_1}{}^T\overrightarrow{X_2}}{|\overrightarrow{X_1}||\overrightarrow{X_2}|}$$

(9)

After that, it optimizes an objective that pulls the positive pairs together while pushing the negative pairs away in the representation space. The loss function InfoNCE [80] is usually used in the related research of contrastive learning, which is formulated as follows:

$$L_{NCE}=-{\mathbf{E}}_X\left[\log \frac{score\left(f\left(x_i\right),f\left(x_i^{+}\right)\right)}{score\left(f\left(x_i\right),f\left(x_i^{+}\right)\right)+{\sum }_{j\ne i}score\left(f\left(x_i\right),f\left(x_j^{-}\right)\right)}\right],$$

(10)

where the corresponding sample x has one positive and N − 1 negative pairs. By minimizing the InfoNCE loss, contrastive learning is to make the features of $f\left(x\right)$ more similar to the features of positive samples $f\left(x^{+}\right)$, and less similar to the features of N − 1 negative samples $f(x_j^{-})$($j\in 1,\cdots ,N-1$). Finally, it can maximize a lower bound on the mutual information between $f\left(x\right)$ and $f\left(x^{+}\right)$ [81].

McAllester et al. [82] analyzed the theoretical shortcomings of contrastive learning, where they argued that the learned representations of contrastive learning were high relative to the size of negative samples. For example, MoCo [83,84,85] and SimCLR [86,87] obtained success due to the various data argumentation with large memory bank and extremely large batch size, respectively. However, Saunshi et al. [88] validated that a larger size of negative samples does not always learn better representations, leading to better performance. They found that the larger batch size would likely generate more redundant samples, thus affecting the efficiency of contrastive learning.

In order to address the above problems, active learning was introduced into contrastive learning by assisting the selection of negative samples [89]. When they carried out the cross-modal contrastive representation learning, uncertainty and diversity were used to sample the negative samples, thus reducing the redundancy actively.

Furthermore, previous active learning works assume that the labeled and unlabeled data pools follow the same class distribution. When applying these works to mismatched class distribution tasks, it suffered from performance degradation sharply. Du et al. [90] focused on this problem. They firstly defined the score function:

$$score=\theta \left(x_i,x_j\right)=\exp \left(\frac{f{\left(x_i\right)}^{\top }f\left(x_j\right)}{\tau ·∥f\left(x_i\right)∥∥f\left(x_j\right)∥}\right)$$

(11)

Then, they used contrastive learning to select semantic and distinctive features and then selected the most informative unlabeled samples $v^{★}$ with minimax selection scheme for querying.

$$v^{★}=\underset{v_i\in \mathcal{V}-L}{\mathrm{argmin}}\underset{v_j\in \mathcal{N}\left(i\right)}{\max }d\left(x_i,x_j\right),$$

(12)

where $d(x_i,x_j)$ calculates the Euclidean distance between embeddings of two nodes $x_i$ and $x_j$, $\mathcal{N}\left(i\right)$ represents the neighbor set of node i, $\mathcal{V}$ and L denotes the node set and labeled set, respectively. Finally, they generalized contrastive learning to active learning with the following modified loss function:

$$L_{CAL}\left(x_i,x_i^{\prime }\right)=-\log \frac{\theta \left(x_i,x_i^{\prime }\right)+\lambda ·{\sum }_{x_p\in \mathcal{P}\left(i\right)}\theta \left(x_i,x_p\right)}{\theta \left(x_i,x_i^{\prime }\right)+\lambda ·{\sum }_{x_p\in \mathcal{P}\left(i\right)}\theta \left(x_i,x_p\right)+{\sum }_{j\ne i}\left[\theta \left(x_i,x_j^{\prime }\right)+\theta \left(x_i,x_j^{\prime }\right)\right]},$$

(13)

where $\mathcal{P}\left(i\right)$ denotes the set of positive embedding whose label is the same with node $v_i$.

Algorithm 4: The workflow of basic contrastive active learning algorithm

Zhu et al. [91] integrated graph neural networks (GNNs)-based active learning with contrastive learning. They denoised the selected candidates by considering the neighborhood propagation scheme in GNNs. Krishnan et al. [92] proposed the featuresim score, which selected balanced, diverse, and informative samples (samples in-between clusters and from edge of clusters) from each class. Gao et al. [93] applied active learning and contrastive learning to the fine-grained off-road semantic segmentation task. They used different semantic attributes for weak supervision and defined the image patches that share the same label as positive pairs while the rest were negative pairs. Besides this, they proposed a risk evaluation method to evaluate high-risk predictions and selected for additional annotation.

4.5. Other Deep Active Learning

Unsupervised domain adaptation (UDA) is a type of cross-domain transfer learning, where the source samples are annotated, and the labels of target samples are absent during training [94]. The goal of UDA is to minimize the discrepancy in distribution between the source domain and the target domain and to learn a robust generalized representation without target annotations [95]. At present, only a few works [96,97] have utilized active learning methods to solve domain adaptation challenges and to improve the performance of the source domain model in the target domain. Recently, Ning et al. [98] first proposed a novel framework that combines active learning and unsupervised domain adaptation to assist cross-domain semantic segmentation tasks. Specifically, they clustered multiple anchors from the source domain to adopt the multi-center distribution. After that, they queried from the unlabeled target samples to the most complimentary samples from the source domain as candidates. The active learning method modeled the data distribution in both the source and target domain and, thus, captured more comprehensive information for domain adaptation.

5. Applications

Recently, computer vision is achieving a breakthrough with deep learning and booms wide applications, which require large amounts of labeled data. Meanwhile, it is impossible to abandon labels entirely or give up unlabeled data in practical applications. Under this condition, active learning can provide a more reasonable expedient, i.e., to annotate valuable data instead of all data.

According to the detailed introductions of deep active learning in the previous sections, we can conclude that the deep active learning methods can play a significant role in the label-intensive vision tasks, helping to reduce labeling costs. In other words, active learning applications are for such conditions, i.e., how to save the workload of labeling and make the model achieve satisfactory performance under a large amount of unlabeled data. Here, we only introduce some extensive-studied applications related to active learning, especially deep active learning.

5.1. Deep Learning-Based Autonomous Driving

In supervised deep learning, a large amount of labeled data needs to be collected for training [99,100], especially in the scorching field of autonomous driving. In this field, the perception of the environment of unmanned vehicles is particularly important [101,102]. The perception of the model directly affects the quality of decision making and plays a vital role in the safety of unmanned vehicles [103,104]. However, there are many complexities environments in autonomous driving scenes. In order to ensure the performance of the model, most companies need to collect the images, point clouds, and radar data from the actual scene for training. Such massive amounts of data are challenging to collect and label. Nevertheless, active learning is able to select the most valuable samples (or via the uncertainty estimation of the current model prediction) and then manually label them. Finally, we can carry out continued model training, thereby improving the model’s performance as much as possible, improving stability and security. In this section, we introduce and compare the applications on deep active learning–based autonomous driving. The overview is summarized in the

Table 6. Summary of applications on deep learning-based autonomous driving.

Applications	Comments	Implementation	Evaluation
Autonomous navigation [105]	⊳ Proposed a framework for learning autonomous policies for navigation tasks from demonstrations.	⊳ Network: 3 × (Conv + Pool) + FC. ⊳ Score-function Entropy: $E\left(X\right)=-{\sum }_{i=1}^{N}P\left(x_i\right)\log P\left(x_i\right)$	⊳ Reach the flag: error rate = 2.48%. ⊳ Follow the line: error rate = 4.06%. ⊳ Reach the correct object: error rate = 0.86%. ⊳ Eat all disks: error rate = 1.70%
Weather and light classification [106]	⊳ Released the first public dataset for weather and light level classification focused on autonomous driving.	⊳ Target network: Resnet18 ⊳ Loss-prediction module [52]: 4 × (GAP + FC + ReLU) + Concat + FC. ⊳ Selection strategy: High loss samples.	⊳ Weather1 [107]: accuracy = 98.80% ⊳ Weather2 [108]: F1 score = 0.872 ⊳ Proposed dataset [106]: F1 score = 0.772
3D object detection [109]	⊳ The first work that introduced active learning into 3D object detection in autonomous driving.	⊳ 3D Detector: VoxelNet ⊳ Score-function: Diversity: ${\min }_{\mathcal{B}\subseteq \mathcal{U}}{\max }_{{\mathbf{x}}_i\in \mathcal{B}}{\min }_{{\mathbf{x}}_j\in \mathcal{B}\cup L}$ $d_{temp+spat+feat}\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)$ $\mathrm{s}.\mathrm{t}.cost\left(\mathcal{B}\right)\le budget$	⊳ nuScenes [110]: mAP = 45.02.
Lane detection [111]	⊳ The first work that introduced active learning into lane detection in autonomous driving.	⊳ Student model: ResNet-122 (for PLN [112]) ResNet-18 (for UFLD [113]) ⊳ Teacher model: SENet-154 (for PLN [112]) ResNet-101 (for UFLD [113]) ⊳ Score-function: Combined the uncertainty and diversity metrics.	⊳ CULane [114] and LLAMAS [115]. (F1 score not reported)
Crowd counting [116]	⊳ The first work that used predictive uncertainty for sample selection pertaining to crowd counting task.	⊳ Local feature block: VGG16 ⊳ Non-local feature block: Transformer ⊳ Score-function: Informativeness difference: $\mathrm{Diff}\left(X_i\right)=$ ${\left(Mea{n}_i\left(X_i,{\mathcal{M}}_1\right)-Mea{n}_i\left(X_i,{\mathcal{M}}_2\right)\right)}^2$	⊳ UCF-QNRF [117]: MAE = 86; MSE = 146. ⊳ UCF CC [118]: MAE = 210; MSE = 305.4. ⊳ ShanghaiTech-A [119]: MAE = 61.5; MSE = 103.4. ⊳ ShanghaiTech-B [119]: MAE = 7.5; MSE = 11.9. ⊳ NWPU [120]: MAE = 78; MSE = 448.
Crowd counting [121]	⊳ Proposed a partition-based sample selection with weights (PSSW) strategy to actively select and annotate both diverse and dissimilar samples for network training.	⊳ Backbone: VGG16 pretrained by imagenet ⊳ Score-function: Diverse in density and dissimilar to previous selections.	⊳ ShanghaiTech-A [119]: MAE = 80.4; MSE = 138.8. ⊳ ShanghaiTech-B [119]: MAE = 12.7; MSE = 20.4. ⊳ UCF CC [118]: MAE = 318.7; MSE = 421.6. ⊳ Mall [122]: MAE = 3.8; MSE = 5.4. ⊳ TRANCOS [123]: MAE = 7.5. ⊳ DCC [124]: MAE = 4.5.

.

Hussein et al. [105] introduced active learning into the autonomous navigation application. In order to address the challenge of generalizing a model over unseen data, they utilized the entropy to measure the confidence of prediction and then labeled the low-confident samples for iterative training. Dhananjaya et al. [106] focused on the harsh weather and low light conditions during driving. They proposed a related dataset containing 60k images from videos, which consisted of various weather conditions (clear, rain, and snow), light levels (bright, moderate, and low), and street types (asphalt, grass, and cobblestone). Under the proposed dataset, previous deep learning–based autonomous driving algorithms suffered from accuracy degradation. The authors introduced an active learning framework to reduce the redundancy from adjacent frames in the video and find the optimal subset for training. Peng et al. [111] designed a novel metric combined with uncertainty and diversity to measure the informativeness of samples. The uncertainty was utilized to estimate the valuable knowledge and noise, while the diversity was used to reduce data redundancy. Liang et al. [109] took the advantage of the multimodal information provided in LiDAR point clouds, and proposed a diversity-based acquisition function that enforces spatial and temporal diversity in the selected samples. Besides this, they investigated the cold-start problem of active learning and demonstrated that the proposed diversity-based methods was able to select better initial batch at early batches, resulting in better performance. Ranjan et al. [116] focused on the domain adaptation of crowd counting. Based on the Query-By-Committee sampling strategy, they constructed the committee with two CTN networks and estimated the density and uncertainty of predictions from committee. Afterwards, they selected the informative samples from the target domain for active learning. Zhao et al. [121] selected the most informative samples via diverse in density and dissimilar to previous selections. The diversity was evaluated by separating the unlabeled set into different density partitions. The dissimilarity was evaluated by considering local crowd density and global crowd count.

5.2. Intelligent Medical Assisted Diagnosis

In the medical field, the development of deep learning has brought revolutionary development to many aspects, including diagnosis [125,126]. However, the above data-driven methods inevitably require a large amount of labeled data [127,128]. However, labeling medical images is time-consuming and labor-intensive, which also requires specific professional knowledge [129,130]. Therefore, it is efficient to use active learning to select samples that are difficult to predict by the model for selective labeling. There is much research studying active learning in the medical field. We summarized the most typical works in the

Table 7. Summary of applications on deep learning-based intelligent medical assisted diagnosis.

Applications	Comments	Implementation	Evaluation
Medical image detection and classification [22,26]	⊳ Combined active learning, incremental fine-tuning, and transfer learning.	⊳ Network: AlexNet pretrained by imagenet ⊳ Selection strategy: Entropy: $e_i^j=-{\sum }_{k=1}^{\left|Y\right|}{p}_i^{j,k}\log p_i^{j,k}$ Diversity: $d_i(j,l)={\sum }_{k=1}^{\left|Y\right|}\left(p_i^{j,k}-p_i^{l,k}\right)\log \frac{p_i^{j,k}}{p_i^{l,k}}$	⊳ polyp detection: ↓ 86% labels. ⊳ pulmonary embolism detection: ↓ 80% labels. ⊳ colonoscopy frame classification: ↓ 82% labels. ⊳ scene classification: ↓ 35% labels.
COVID-19 Lung Ultrasound Multi-symptom Classification [131]	⊳ The first work that introduced active learning into ultrasound classification for COVID-19-assisted diagnosis.	⊳ Backbone: ResNet50 pretrained by imagenet ⊳ Score-function: Least confidence: $\mathrm{LC}\left(x\right)={\max }_{1\le i\le l}p\left(l_i\mid x\right)$t Multi-label entropy: $\mathrm{MLE}\left(x\right)={\sum }_{i=1}^l(p\left(l_i\mid x\right)\log p\left(l_i\mid x\right)+$ $p(\overline{l_i}\mid x)\log p(\overline{l_i}\mid x))$	⊳ COVID19-LUSMS v1: ↓ 80% labels.
Brain tumor Classification [132]	⊳ Sampling candidates by discarding subsets of training samples with the highest and lowest uncertainty scores.	⊳ Network: AlexNet pretrained by imagenet ⊳ Score-function: Combined entropy and Kullback–Leibler (KL) divergence: $E\left(X\right)=-{\sum }_{i=1}^{N}P\left(x_i\right)\log P\left(x_i\right)\mathrm{D}(\mathrm{p}\parallel \mathrm{q})=$ ${\sum }_{i=1}^{n}P\left(x_i\right)\log \frac{P\left(x_i\right)}{Q\left(x_i\right)}$	⊳ MICCAI BRATS [133,134,135]: ↓ 40% labels.
Diabetic retinopathy classification [136]	⊳ The first work that introduced active learning into lane detection in autonomous driving.	⊳ Bayesian convolutional neural network (BCNN): Monte-Carlo drop-out ⊳ Teacher model: SENet-154 (for PLN [112]) ResNet-101 (for UFLD [113]) ⊳ Score-function: entropy.	⊳ APTOS 2019 [137]: AUC = 0.99 (multi-class classification) Accuracy = 92% (multi-class classification) Accuracy = 85% (BCNN in Active Learning)
Histo-pathology image analysis [138]	⊳ The first work that proposed an AL framework (PathAL) to dynamically identify important samples to annotate and to distinguish noisy from hard samples in the training set.	⊳ Backbone: EfficientNet-B0 [139] ⊳ Noisy sample detector: O2U-Net [140] Curriculum Sample Classification: CurriculumNet [141] ⊳ Score-function: Distinguished noisy samples from hard ones, and selected the most informative samples to be annotated.	⊳ PANDA [142]: quadratic weighted kappa = 89.5.
Gastric adenocarcinoma and colorectal cancer [143]	⊳ The first work that explored the identification of the most informative region of patches and proposed a patch location system to select patches.	⊳ Backbone: ResNet-18 ⊳ Loss-prediction module [52]: 4 × (GAP + FC + ReLU) + Concat + FC. ⊳ Score-function: $\Delta {\mathbf{p}}_{*}^{\mathcal{G}}=\arg {\max }_{\Delta {\mathbf{p}}^{\mathcal{G}}}\mathcal{H}\left(\Delta {\mathbf{p}}^{\mathcal{G}}+{\mathbf{p}}^{\mathcal{G}}\right).$	⊳ TCGA [144,145]: AUC = 0.933. accuracy = 92.7%.

.

Zhou et al. [22,26] introduced transfer learning and data enhancement into active learning. By measuring the uncertainty and diversity, the proposed AIFT framework achieved SOTA performance in the biomedical image analysis. Liu et al. [131] introduced active learning into ultrasound classification for COVID19-assisted diagnosis. In order to actively reduce the labeling efforts, the proposed method combined least confidence and entropy selection strategies. Hao et al. [132] combined entropy and Kullback–Leibler(KL) divergence for uncertainty-based sampling. Apart from the active learning, they also carried out transfer learning from imagenet-pretrained AlexNet [146] to MRI (MICCAI BRATS 2019 dataset) [133,134,135]. The proposed transfer learning framework reduced the annotation cost while maintaining the stability and robustness of the model performance for brain tumor classification.

Ahsan et al. [136] integrated Bayesian-based CNN and uncertainty-based active learning method, where active learning was applied to the pool-based sampling and query by committee scenarios. Wang et al. [147] formulated the active learning as a Markov decision process and introduced a deep reinforcement learning algorithm for the selection of the most informative samples. The proposed method was validated in four kinds of lung disease detection with CT images (chestCT (https://tianchi.aliyun.com/competition/entrance/231724/introduction, accessed on 11 July 2022)) and diabetic retinopathy in digital color fundus photograph (Retinopathy (https://www.kaggle.com/competitions/diabetic-retinopathy-detection, accessed on 11 July 2022)). Smit et al. [148] pretrained the active learning framework with contrastive learning and utilized the cosine similarity to classify unseen images. The proposed method was validated in the eight common chest observations in X-ray images (CheXpert [149]). Shen et al. [143] first explored the identification of the most informative region of patches and proposed a patch location system to select patches. The proposed method was validated in three gastric adenocarcinoma and colorectal cancer datasets from The Cancer Genome Atlas (TCGA [144]). After that, they continued to explore the whole-slide histopathology image annotation with active learning. In this work [150], they incorporated spatial distribution representation and histopathology tissue informativeness for uncertainty sampling. Li et al. [138] adopted the semi-supervised idea that selected confident samples from the unlabeled set and automatically utilized their corresponding predictions as pseudo-labels for training. They proposed the PathAL framework, where annotators and co-training label the other “informative” sample with the above pseudo-labels.

6. Challenges

Although the motivation of deep active learning is to reduce the amount of annotation in practical applications and provide an efficient learning solution for deep learning, the current active learning methods still have some challenges in practical application, which can be summarized into the following four aspects.

6.1. Inefficient Serial Human-in-the-Loop Collaboration

The essence of active learning is still a process of continuous interaction between computers and annotators, which will undoubtedly cause inconvenience in interaction. The process of most active learning methods is still to select a batch of candidates and send them to annotators for labeling and expect annotators to label them as soon as possible and return the labeled samples back, and finally, the model continues to train and then select candidates again. This is a serial process, which means that when annotators are labeling, the model cannot be trained or perform any other operations. It is necessary to wait for the end of manual labeling before the next round of iterative training can be performed.

For example, we assume that there is an active learning labeling system in the medical scene. For the computer, the strategy first selects some samples and sends them to the doctor for labeling, and then is in the idle period waiting for labeling. For the doctor, after receiving the samples, it is time-consuming to label and then return it to the model training, and wait for the subsequent feedback from the model. In this way, the doctor and the model wait for each other’s operations, reducing efficiency and convenience. Consequently, an efficient parallel strategy for active learning is expected to be proposed in the future.

6.2. Dirty Data and Noisy Oracle

Most of the existing deep learning research assumes that the data is independent and identically distributed and uses publicly available datasets. These datasets contain little to no dirty data (noise, imbalance). However, in industrial practice, data sources are far from the ideal dataset with more dirty data. For example, there are categories with fewer samples or fewer categories with more samples (sample category distribution imbalance). The uncertainty selection strategy is widely used in active learning, but it is hard to evaluate the uncertainty of noisy samples. At the same time, the oracle’s annotation is considered ground truth, but it may also contain errors [151]. Consequently, it is unconfident when these noisy samples or labels are used for active learning. Such samples may not improve the model’s performance but even worsen the performance.

6.3. Difficult to Cross-Domain Transfer

No matter what selection strategy is used in the existing active learning, it is based on the current data distribution of the source domain. Industrial practice requires a more general and generalizable active learning strategy, so that they can transfer between different domains and tasks with considerable performance.

As a sub-field of transfer learning, cross-domain adaptation has been extensively studied in the recent years [152,153]. Prabhu et al. [154] demonstrated that existing model uncertainty-based or diversity-based active learning methods based solely on are ineffective for domain adaptation. Xie et al. [155] introduced an energy-based strategy to select the most representative and informative target data to assist the adaptation. However, we are disappointed to find that most active learning strategies are domain-designed, and there is no guarantee that the active learning strategy can achieve competitive performance when cross-domain transfer. For example, there is already an active learning method designed for cat and dog classification tasks based on the uncertainty selection strategy, and it has achieved better performance. Now, if we transfer it into a new task for husky and labrador classification, the performance may degrade. If the new task is organ or tumor classification in medical images, redesigning a new active learning method is more recommended than using the previous method, but it wastes time and cost. Fu et al. [156] proposed the transferable query selection (TQS) strategy to select the most informative samples under domain shift. The TQS consists of transferable committee, transferable uncertainty, and transferable domainness. Besides these, rare works have studied the unsupervised domain adaptation with active learning. Consequently, an active learning strategy with robust cross-domain transferring ability is expected to be proposed in the future to solve this challenge.

6.4. Unstable Performance

The biggest challenge that hinders the practical application of active learning methods is the unstable performance. As introduced in previous sections, active learning is to select candidates according to some strategy. These selected samples are significant for the sequential training and evaluation, especially at the beginning.

As we expected, deep active learning usually outperforms random sampling, especially when high-redundant data distribution. However, we have to admit that current active learning may still perform worse than random sampling in the early stage when the data distribution is diverse and has low redundancy. Random sampling can collect more representative samples than active learning under this condition, and the model receptive field is more comprehensive, thus obtaining better initialization. This phenomenon is named the cold-start problem in active learning, which is shown in Figure 3. When the application scenario of active learning has the data distribution mentioned above, it must afford the cost of additional selection samples than random sampling in the early stage of training. If the performance is worse than random sampling, this part of the cost has already been invested and cannot be recovered.

Therefore, the industry has stricter requirements for active learning in practical applications, and it is almost necessary to work if the designed strategy is directly applied. If not, those selected samples are still marked, and time and money are lost. Such harsh requirements and unstable performance lead people to prefer to save this cost and turn to directly adopting random sampling, but design a better model or use a better optimization strategy to achieve more stable performance.

Zhou et al. [22] explored the cold-start problem and found the reasons were the scarcity of labeled dataset and the instability of the model at the beginning. They addressed this problem by cooperating with the random sampling method. They obtained better performance in early stages and improvement during sequential steps. Another solution is pretrained active learning, which means that, before carrying out active learning, we initialized the model with pretrained weights and gave the stability to the model. Typical self-supervised pretraining methods such as MoCo [83] or Genesis [157] utilize the unlabeled data pool and have the potential to address the cold start problems in active learning.

7. Conclusions

This paper reviewed the fundamental theories of active learning, including the candidate selection strategies and querying scenarios. Besides this, we conducted a comprehensive analysis of deep learning–based active learning, including generative adversarial active learning, semi-supervised active learning, active contrastive learning, and unsupervised active domain adaptation. Meanwhile, active learning applications in computer vision tasks were detailed, such as deep learning-based autonomous driving and intelligent medical assisted diagnosis. Lastly, we summarized some challenges in current deep active learning methods for future research.