S²AC: Self-Supervised Attention Correlation Alignment Based on Mahalanobis Distance for Image Recognition

Abstract

Susceptibility to domain changes for image classification hinders the application and development of deep neural networks. Domain adaptation (DA) makes use of domain-invariant characteristics to improve the performance of a model trained on labeled data from one domain (source domain) on an unlabeled domain (target) with a different data distribution. But existing DA methods simply use pretrained models (e.g., AlexNet, ResNet) for feature extraction, which are convolutional models that are trapped in localized features and fail to acquire long-distance dependencies. Furthermore, many approaches depend too much on pseudo-labels, which can impair adaptation efficiency and lead to unstable and inconsistent results. In this research, we present S²AC, a novel approach for unsupervised deep domain adaptation, that makes use of a stacked attention architecture as a feature map extractor. Our method can fuse domain discrepancy with minimizing a linear transformation of the second statistics (covariances) extended by the p-norm, while simultaneously designing pretext tasks on heuristics to improve the generality of the learning representation. In addition, we have developed a new trainable relative position embedding that not only reduces the model parameters but also enhances model accuracy and expedites the training process. To illustrate our method’s efficacy and controllability, we designed extensive experiments based on the Office31, Office_Caltech_10, and OfficeHome datasets. To the best of our knowledge, the proposed method is the first attempt at incorporating attention-based networks and self-supervised learning for image domain adaptation, and has shown promising results.

1. Introduction

Neural networks have transformed the area of machine learning, enabling the development of powerful models capable of learning complicated patterns and making accurate predictions. However, one of the major challenges in employing neural networks is their sensitivity to distribution shift of the data on which they are trained. Figure 1 shows that the data used to train a model may not be indicative of the data encountered by the model in the real-world. Domain adaptation seeks to address this issue by adapting the model to the target domain, allowing it to generalize more effectively and generate more accurate predictions.

Domain adaptation (DA) is a sub-field of transfer learning that aims to bridge the source and target domains by deriving domain-invariant features from labeled data in the source domain and unlabeled data in the target domain. There has been a surge of interest in employing neural networks for DA in recent years. Many novel algorithms, ranging from simple techniques to more complicated ones, have been presented. These approaches have yielded promising results in a variety of applications, such as computer vision, natural language processing, and speech recognition.

Basically, recent advances in DA are divided mainly into two different types of DA methods: shallow methods and deep methods. The shallow method can be applied to homogeneous and heterogeneous visual tasks, where visual features are extracted from the images. According to [1], features generated by deep models can be repurposed to new tasks and outperform the shallow DA methods even without any adaptation. Inspired by this phenomenon, a commonly accepted paradigm considers the deep model to extract vectorial features as representations for the input image, which can then be used within the shallow DA methods to mitigate the domain discrepancy.

In this work, we present an End2End deep domain adaptation model S²AC, which stands for self-supervised attention CORAL based on Mahalanobis distance. There are two main ideas proposed to improve the previous methods.

First, in our model, the attention architecture relies entirely on attention mechanisms. We stack self-attention blocks and linear layers to draw global dependencies, and insert position encoding between them.

Second, the model leverages recent advances in self-supervised learning to learn representations that can be used for solving other downstream tasks of interest, but requires no label data.

The experimental results illustrate that our method can achieve state-of-the-art performance comparable with previously carefully tuned baselines.

The rest of this paper is organized as follows: Section 3 surveys the theoretical background related to our model; Section 4 describes the proposed S²AC model and provides a theoretical exposition; Section 5 shows experimental results; Section 6 presents the conclusions.

2. Related Work

Shallow DA Methods: Intensive efforts have been made in domain adaptation. Borgwardt et al. [2] introduce the maximum mean discrepancy (MMD) for structured biological data to estimate the differences in distribution between two molecular graph datasets. H. Shimodaira [3] discusses that the inference based on the maximum likelihood estimate (MLE) can be improved by weighting the covariate in the log-likelihood function. In order to adapt the existing classifiers across various video domains, Yang et al. [4] propose adaptive support vector machines (A-SVMs), which adapt the original and the adapted classifier through minimizing an SVM-like objective function.

Deep DA Methods: Deep learning’s advancement has heralded a new era for domain adaptation technology. DDC [5], an acronym for deep domain confusion, discovers the domain confusion metric, which can be utilized to estimate the dimensionality and ideal position of the domain adaptation layer; therefore, a domain confusion loss is introduced to reduce discrepancy between the source and target domains. Conditional domain adversarial networks (CDANs) [6] enhance the classic adversarial domain adaptation by including a conditional branch that takes class label information into account during training. The conditional branch directs the domain classifier in learning domain-invariant characteristics unique to each class. They have been found to be useful in a variety of applications, including sentiment analysis and object identification. In [7], they focus on reconciling the centroid-hypothesis conflict between two loss functions within source-free domain adaptation: the cross-entropy loss inclines to the probability implied by pseudo-labels, but nevertheless, the entropy minimization loss is prone to the class implied by hypothesis. Song et al. [8] introduce stable diffusion and score distillation sampling into a Style-GAN2 to guide the generator towards a target domain under the text prompt. Instead of using the text prompt, d-SNE [9] learns domain-agnostic latent space by using the stochastic neighborhood embedding techniques and a modified Hausdorffian distance. To learn adaptive classifiers, the residual transfer network (RTN) [10] plugs a residual block into CNN architectures to bridge tightly the source classifier and target classifier. To enable feature adaptation, they embed the fused features of multiple layers into Hilbert spaces to match distributions.

Attention Mechanisms: Attention mechanisms, especially self-attention (also called intra-attention), have become an integral part in transformer-related model, and present impressive improvements in various tasks [11,12,13,14,15]. Attention mechanisms are usually used to reweight the features of fully connected or convolutional layers. In [15], they show that inserting an attention layer into AlexNet (a convolutional neural network) used as the backbone network of deep CORAL achieves improved performance. Vaswani et al. [11] introduce a transformer based entirely on attention mechanisms to compute representations of its input on the WMT 2014 English-to-German translation tasks. The attention vectors are invariant during the training process in terms of sequence ordering, otherwise addition of position information can be considered [16]. GANs is a great tool in image-to-image transduction [17,18] and text-to-image translation tasks. Zhang et al. [13] employ an intra-attention module to implement long-range interaction across image regions.

Self-Supervised Model: Recent work shows that self-supervised learning [19] methods can significantly improve the accuracy on image classification tasks for most deep learning cases. Models trained on predefined surrogate tasks learn useful image semantics using only unsupervised data that can be used for solving other downstream tasks of interest. S⁴L [20] (self-supervised semi-supervised learning), proposed by Zhai et al., reports two techniques: S⁴L-Rotation rotates input images into four degrees $\left\{0^{\circ },{90}^{\circ },{180}^{\circ },{270}^{\circ }\right\}$ and predicts the rotation degree for each of them, and S⁴L-exemplar learns a visual invariant representation to image transformations, such as HSV-space color randomization, random mirroring, etc. A key issue for GAN training instability is catastrophic forgetting, which can be countered through introducing temporal memory to the training process. Noroozi and Favaro [21] propose that combining adversarial and self-supervised learning relieves the impact of discriminator forgetting.

3. Theoretical Background

3.1. Mahalanobis Distance-Based CORAL Loss ($L_{MD-CORAL}$)

Mahalanobis distance (MD) is a measure that can be seen as a modification of Euclidean distance, with the problem of inconsistent and correlated dimensional scales corrected.

Given two vectors $\overrightarrow{X}$ and $\overrightarrow{Y}$, the squared MD $D_M^2$ is:

$$\begin{array}{cc}\hfill D_M^2& ={\left(\frac{x_1-y_1}{\sqrt{{\lambda }_1}}\right)}^2+{\left(\frac{x_2-y_2}{\sqrt{{\lambda }_2}}\right)}^2+\cdots +{\left(\frac{x_n-y_n}{\sqrt{{\lambda }_n}}\right)}^2\hfill \\ \hfill & =\left(x_1-y_1,x_2-y_2,\cdots ,x_n-y_n\right)\left[\begin{array}{cccc}\sqrt{{\lambda }_1}& & & \\ & \sqrt{{\lambda }_2}& & \\ & & \cdots & \\ & & & \sqrt{{\lambda }_n}\end{array}\right]\left(\begin{array}{c}{x}_1-y_1\\ x_2-y_2\\ \cdots \\ x_n-y_n\end{array}\right)\hfill \\ & ={\left(\overrightarrow{X}-\overrightarrow{Y}\right)}^{{}^{\prime }}{\mathsf{\Sigma }}^{-1}\left(\overrightarrow{X}-\overrightarrow{Y}\right)\hfill \end{array}$$

(1)

where ${\mathsf{\Sigma }}^{-1}$ stands for the inverse of the covariance matrix between $\overrightarrow{X}$ and $\overrightarrow{Y}$. Correlation alignment (CORAL) [22] proves that aligning second-order statistics (covariances) across domains facilitates learning domain-invariant features that are less affected by domain shift. P-ADC [15] attempted to extend CORAL loss with the idea of p-norm to balance the offset caused by the classification loss, but this led to dilemmas of accuracy and computation time.

$$\begin{array}{cccc}\hfill & \mathrm{CORAL}\mathrm{loss}\hfill & \hfill & L=\underset{\omega }{min}{L}_{Class}\left(D_S;\omega \right)+\alpha L_{CORAL}\hfill \end{array}$$

(2)

$$\begin{array}{cccc}\hfill & \mathrm{P}-\mathrm{norm}\mathrm{loss}\hfill & \hfill & L=\underset{\omega }{min}{L}_{Class}\left(D_S;\omega \right)+\sum _p{\lambda }_p{L}_p\hfill \end{array}$$

(3)

The Mahalanobis distance-based loss function (2-norm) is:

$$L_{MD-CORAL}^{2-norm}=\frac{1}{4d^2}{\left(C_{D_S}-C_{D_{T_i}}\right)}^{{}^{\prime }}{\mathsf{\Sigma }}^{-1}\left(C_{D_S}-C_{D_{T_i}}\right)$$

(4)

According to [15], as p increases, p-norm loss consumes a large amount of computational resources, leading to a drastic decrease in computing speed. We designed a linear transform to alleviate the conflict between accuracy and computational complexity, and to compensate for the loss of accuracy due to the neglect of higher-order terms.

$$\begin{array}{cc}\hfill L_{MD-CORAL}& ={\beta }_0+{\beta }_1{L}_{MD-CORAL}^{1-norm}+{\beta }_2{L}_{MD-CORAL}^{2-norm}+{\beta }_3{L}_{MD-CORAL}^{3-norm}\hfill \\ \hfill & \mathrm{where}\left\{\begin{array}{c}{\beta }_0=1+C_{D_S}^{{}^{\prime }}{C}_{D_{T_i}}\hfill \\ {\beta }_1=C_{D_S}^{{}^{\prime }}{C}_{D_{T_i}}\hfill \\ {\beta }_2=\frac{\sum \left(C_{D_S}+C_{D_{T_i}}\right)}{2n}\hfill \\ {\beta }_3=\frac{1}{3}\hfill \end{array}\right.\hfill \end{array}$$

(5)

3.2. Self-Attention

Attention almost entirely replaces recurrence by allowing for a focus on important regions and modeling long-distance dependencies within input or output images. In practice, the queries, keys, and values, packed together into matrices Q, K, and V, are computed as linearly transformed input images x with parameter matrices $W^Q\in R^{\frac{C}{8}\times C},W^K\in R^{C\times \frac{C}{8}},W^V\in R^{\frac{C}{8}\times C}$, respectively. The self-attention function is:

$$\begin{array}{cc}\hfill Self-Attention\left(Q,K,V\right)& =softmax\left(\frac{Q{K}^{{}^{\prime }}}{\sqrt{d_k}}\right)V\hfill \\ \hfill & =softmax\left(\frac{\left(W^{Q}x\right){\left(W^{K}x\right)}^{{}^{\prime }}}{\sqrt{d_k}}\right)\left(W^{V}x\right)\hfill \end{array}$$

(6)

where the input images $x=\left\{x_1,x_2,\cdots ,x_C\right\},x_i\in R^{W\times H}$ are converted to two matrices to compute the attention after being normalized by a softmax function. $d_k$ denotes the variance of $Q{K}^{{}^{\prime }}$. Figure 2 shows an example of a 64-channel self-attention block.

3.3. Positional Embedding

Self-attention raises the permutation equivalence without positional information, which limits the expressiveness of the model for visual tasks [23]. Positional embeddings refer to injecting some positional information of the tokens into the features according to different rules. There are two different strategies based on embedded location information. The attention with absolute positional embeddings, which utilizes the absolute position of pixels in an image, is called absolute attention, such as sinusoidal embedding [11], which uses sine and cosine functions to encode each dimension to form a geometric progression whose wavelength is from $2\pi$ to $10,000·2\pi$. The attention based on relative positional embeddings, called relative attention, which calculates the relative distance to each location $\left(x,y\right)\in \mathcal{N}\left(i,j\right)$ to receive two offsets for the horizontal and vertical coordinate axes, respectively, i.e., $\left(x-i,y-j\right)$, results in improved performance [23]. The relative attention function is:

$$Self-Attentio{n}_{relative}(Q,K,V)=softmax\left(\frac{Q^{{}^{\prime }}K+Q^{{}^{\prime }}\left(r_{x-i},r_{y-j}\right)}{\sqrt{d_k}}\right)V$$

(7)

4. Methods

In this section, we present our S²AC approach for unsupervised domain adaptation (UDA) problem. Let a model $q_{\omega }\left(y|x\right)$ trained on a labeled source dataset $D_s$ composed of source images $x_s$ and corresponding labels $y_s$, $D_s=\left\{\left(x_s,y_s\right)|x_s\in R^{H\times W\times C},y_s\in {\left\{0,1\right\}}^{H\times W\times K}\right\}$, where $W,H,C$ are the width, height, and channel of the source domain images, respectively; K denotes the number of classes, to be tested over multiple unlabeled target datasets composed of target domain images, $D_{T_1},D_{T_2},\cdots ,D_{T_n}$, where $D_{T_i}={\left\{x_j^T\right\}}_{j=1}^{N_{T_i}}$ is the $i^{th}$ target dataset and $N_{T_i}$ denotes the scale of the target domain $D_{T_i}$.

Our method assumes that $q_{\omega }\left(y|x\right)$ is trained in an unsupervised manner; meanwhile, it utilizes only the pretrained model $q_{\omega }\left(y|x\right)$, which works well in the source domain, and an unlabeled target dataset $D_{T_i}$.

Our framework mainly comprises the following three components (see Figure 3):

• Attention-based backbone network module;
• Self-supervised learning module;
• Relative positional embedding.

4.1. Attention-Based Backbone Network Module

An attention-based backbone network module maps features to the space where joint training with the classification loss, the Mahalanobis distance-based CORAL loss, and the self-supervised loss are applied. We consider the total loss function in this study to have the following form:

$$L=\underset{\omega }{min}{L}_{Class}\left(D_S;\omega \right)+\alpha L_{MD-CORAL}\left(D_S,D_{T_i};\omega \right)+\beta L_{self}\left(D_S;\omega \right)$$

(8)

where $L_{Class}$ is a standard cross-entropy loss of all labeled data in the source domain. $L_{MD-CORAL}$ stands for a linear transformation of the second statistics (covariances) extended by the p-norm to align the distributions of the source and target features. $L_{self}$ is the self-supervised loss that encourages the model to learn useful information. The corresponding self-supervised task is to predict the index of the chosen permutation in this paper.

The backbone network is composed of a stack of N = 5 identical layers, followed by layer position encoding and two linear layers. Each layer has two sub-layers. The first is a self-attention mechanism and the second is max-pooling layer. We employ a residual connection inside the self-attention mechanism. That is, the output of each layer is $H_i=Max\left(ReLU(x_{i-1}+Self-Attention(Q,K,V))\right)$ (see Figure 4), where $Max(·)$ is the function implemented by the max-pooling layer itself. To leverage these residual connections, the dimensions of input and output must be equal for all self-attention modules; thus, we set $kernel\_size=1$, $stride=1$, $padding=0$ for query, key, and value matrices in convolution-based attention.

4.2. Self-Supervised Learning Module

A self-supervised auxiliary task (called Random Context Prediction) module that transforms both given source and target data samples along with two prominent self-supervised techniques was adopted in our framework: context prediction + random rotation. Following [19,21], context prediction, as a “pretext”, forces the model to understand scenes and objects. The main idea is to learn general-purpose representation by exploiting the relative spatial location as labels that come for “free” within images. Doersch et al. [19]’s sample randomly pairs of patches in a three-by-thre grid. In their approach, one tile is the pivot kept in middle of a three-by-three grid; the other can be any of the eight remaining available locations for each pair. In contrast, ref. [21] observes all nine tiles through associating with the Jigsaw puzzle reassembly problem. A set of jigsaw puzzle permutations is defined, as well as the corresponding index for each item. To learn image representation, the Jigsaw puzzle solver shuffles the input tiles base on the set of permutations to predict the relevant index. However, since the number of possible permutations of nine tiles, which is $9!$, is too numerous to be applied directly, a subset must be established, which is also time-consuming (see Section 5). In this study, we combine two strategies to create a novel, harder, and more efficient pretext task. Following [21], we generate the subset using “random” sampling rather than Hamming distance. To achieve end-to-end processing, it is also feasible to omit subset creation and just perform random selection in the original matrix. We crop a region from an image at random and divide it into nine patches, which are rearranged according to a randomly selected permutation. At the same time, each patch rotates at a “random” degree, which is generally picked from $\left\{0^{\circ },{90}^{\circ },{180}^{\circ },{270}^{\circ }\right\}$ (see Figure 5).

The self-supervised loss is:

$$\begin{array}{cc}\hfill L_{self}\left(x,m;\omega ,\mu \right)& =\frac{1}{N_M{N}_{D_S\cup D_{T_i}}}\sum _{m\in M}\sum _{x\in D_S\cup D_{T_i}}{L}_{CE}\left(p\left(f_{\omega }\left(x^m\right);\mu \right),g\left(m\right)\right)\hfill \\ \hfill & =-\frac{1}{N_M{N}_{D_S\cup D_{T_i}}}\sum _{m\in M}\sum _{x\in D_S\cup D_{T_i}}g\left(m\right)lnp\left(f_{\omega }\left(x^m\right);\mu \right)\hfill \end{array}$$

(9)

where M stands for the predefined permutation set. $x^r$ is the input x reordered via a randomly chosen permutation from m and rotated via a randomly chosen degree from $\left\{0^{\circ },{90}^{\circ },{180}^{\circ },{270}^{\circ }\right\}$. $g\left(·\right)$ is the mapping of the jigsaw puzzle permutations to $\left\{1,2,\cdots ,9!\right\}$, e.g., $\left(6,3,7,2,9,1,4,8,5\right)\to 53$; N denotes the number of candidates. $D_{T_i}$ is the target domain. $f_{\omega }\left(·\right)$ illustrates the model with parameters $\omega$. $p\left(·,\mu \right)$ denotes the joint probability as $p_{i,j}\left(x;\mu \right)=\frac{e^{{\mu }_{i,j}^{{}^{\prime }}x}}{{\sum }_{k,l}{e}^{{\mu }_{k,l}^{{}^{\prime }}x}}$.

4.3. Relative Positional Embedding

It has been proven in [11,23] that employing positional embeddings results in more efficient and accurate models. However, the definition of early relative position embeddings only takes into account relative distance, i.e., row offset and column offset, which is irrelevant to the picture content. Unlike earlier literature, our strategy attaches the relative position to the model, converting it into a trainable type that can be tuned during parameter optimization. Using $\left(x_i,y_j\right)$ as the base point, the relative distance of and other point $\left(x_j,y_j\right)$ in the image to $\left(x_i,y_j\right)$ is $\left(x_d,y_d\right)$ (see Figure 6). The relative distance $\left(x_d,y_d\right)$ is associated with embeddings $\left(r_{x_d},r_{y_d}\right)$.

$$x_i=\sigma \left(p_1\right)\left(w_2-w_1\right)+w_1,0\le w_1<w_2\le W$$

(10)

$$y_i=\sigma \left(p_2\right)\left(h_2-h_1\right)+h_1,0\le h_1<h_2\le H$$

(11)

$$x_d=x_j-x_i$$

(12)

$$y_d=y_j-y_i$$

(13)

where $x_i$, $y_i$ stand for the horizontal and vertical coordinates of the base point. $x_j$, $y_j$ are the horizontal and vertical coordinates of any point on feature map. $p_1$, $p_2$ represent trainable parameters that are optimized during model training. $\sigma \left(x\right)=\frac{1}{1+e^{-x}}$, is the sigmoid function. For example, given a $5\times 5$ feature map implying $W=H=5$ and also $x_i=2,y_i=3$ after backpropagation computed from Equations (10) and (11), the positional embedding matix is: $\left[\begin{array}{ccccc}(0,0)& (0,1)& (0,2)& (0,3)& (0,4)\\ (1,0)& (1,1)& (1,2)& (1,3)& (1,4)\\ (2,0)& (2,1)& (2,2)& (2,3)& (2,4)\\ (3,0)& (3,1)& (3,2)& (3,3)& (3,4)\\ (4,0)& (4,1)& (4,2)& (4,3)& (4,4)\end{array}\right]\underset{h_1=w1=2,h_2=w_2=3}{\overset{x_i=2,y_i=3}{\to }}\left[\begin{array}{ccccc}(-2,-3)& (-2,-2)& (-2,-1)& (-2,0)& (-2,1)\\ (-1,-3)& (-1,-2)& (-1,-1)& (-1,0)& (-1,1)\\ (0,-3)& (0,-2)& (0,-1)& (0,0)& (0,1)\\ (1,-3)& (1,-2)& (1,-1)& (1,0)& (1,1)\\ (2,-3)& (2,-2)& (2,-1)& (2,0)& (2,1)\end{array}\right]$.

5. Experimental Results

5.1. Setup

In this study, we perform experiments on a computer that has an Intel(R) Xeon(R) Silver 4210R CPU, 64-bit 2.4GHz (two processors), and an NVIDIA Tesla T4 GPU. We evaluate our model on three image datasets: Office31 [24], Office_Caltech_10 [1], OfficeHome [25].

Office31 is a classic collection of images of objects from 31 distinct categories commonly encountered in office environments, including headphones, bottles, speakers, etc. It has 4652 images in total, with an average of 150 images per category. Three domains in the dataset are Amazon (A), DSLR (D), and Webcam (W).

Office_Caltech_10 is divided into two parts: the office part and the Caltech part. The office part contains images of office scenes, while the Caltech part contains images of Caltech scenes. It contains ten overlapping categories shared by four domains: Amazon (A), Caltech (C), DSLR (D), and Webcam (W).

OfficeHome contains images from four different workplace and home contexts. It includes images of objects from four different domains: Art (A), Clipart (C), Product (P), and Real-World (R). The dataset includes 15,588 images from 65 different categories, with a total of 4652 images from each domain.

Our model takes images of $224\times 224$ pixels as input, and the weight decay and momentum are set to $5\times {10}^{-4}$ and $0.9$, respectively; the initial learning rate is $1\times {10}^{-3}$ as well. Because of GPU memory limitations, both our train_batch size and test_batch size are set to 8.

5.2. Selection of Criteria for Generating Permutation Subset

The original $9!$-item permutation set in [21] is too large to be directly used in the computation; thus, the jigsaw puzzle solver must employ an external process, which is a non-end-to-end process, to calculate a subset of the permutations by hamming distance. We suggest using uniform random sampling to create the permutation subset in real time, which will increase the model’s efficacy.

We designed this experiment to show the time difference between executing three criteria, i.e., Mean Hamming distance, Max Hamming distance, and Random sampling, to generate the set of permutations. We discover that as the number of permutations increases, computing the Hamming distance has a substantial impact on running time when compared with random sampling, while the improvement in test accuracy is not significant. The runs are shown in

Table 1. The running time and test accuracy analysis on three criteria: mean Hamming distance, max Hamming distance, and random sampling, to generate the permutation set. # means “the number of”. $\mathsf{\Sigma }$ is “the sum of”. $Mean$ stands of the mean Hamming distance. $Max$ indicates the max Hamming distance. $Random$ is the random sampling.

# Permutations	Mean Hamming Distance				Max Hamming Distance				Random Sampling
	$\mathsf{\Sigma }$ Mean	Running Time (s)	Test Accuracy (%)		$\mathsf{\Sigma }$ Max	Running Time (s)	Test Accuracy (%)		Running Time (s)	Test Accuracy (%)
6	5.333	0.411	45.6		6.000	0.366	45.4		0.056	45.5
7	6.222	0.583	45.4		7.000	0.482	45.6		0.071	45.6
15	13.333	1.740	45.8		14.400	1.449	45.7		0.165	45.6
25	22.222	3.794	46.1		23.525	3.349	46.1		0.195	45.8
35	32.111	6.895	47.4		32.571	6.034	46.6		0.282	46.5
55	48.889	15.412	48.9		50.527	14.289	47.2		0.409	47.6
73	64.889	26.863	49.32		66.651	24.137	48.8		0.597	48.6
85	75.556	35.069	49.9		77.381	32.536	49.4		0.688	48.9
100	88.889	47.602	50.1		90.791	42.939	49.5		0.718	49.4

. Based on the experiment results, we conclude that the use of random sampling can significantly reduce the running time without significantly affecting the accuracy.

Figure 7 displays us three graphs created from shift $D\to W$ to help us analyze S²AC. In Figure 7a, we show the classification loss, the Mahalanobis distance-based CORAL loss, the self-supervised learning loss, and the total loss for the training phrase. The classification loss is initially significantly bigger than both the Mahalanobis distance-based CORAL loss and the self-supervised learning loss, but as iterations progress, the classification loss becomes minimal at around $epochs=50$ and progressively stabilizes and converges. The Mahalanobis distance-based CORAL loss exhibits a notable growth between iteration cycles 20 and 40, but beyond 50, it steadily declines and converges. In Figure 7b, we visualize the average loss for the test phrase in the source and target domains. The plot shows that the average loss is generally on the decline in both the source and target domains. The fluctuation of the training (source) and test (target) accuracy is shown in Figure 7c. Figure 7d shows the performance of Frobenius distance, Mahalanobis distance, earth mover’s distance (EMD), Chebyshev distance, and cosine distance matrices. The EMD fluctuations are excessively large, and the Frobenius distance-based loss function is weaker than our method in this scenario, while the Chebyshev distance-based loss and the cosine distance-based loss continue to rise in the latter phases, causing the total loss to be poorly converged (see Figure 7e–g).

5.3. Ablation Studies

We designed an ablation analysis of the influence of the attention-based backbone network, the self-supervised learning module, and positional embedding. Figure 8 shows the classification accuracy curves. For convenience, ‘-S’, and ‘-T’ denote training and test accuracy, respectively. And incorporation of only the AlexNet [26], instead of the attention-based backbone network, pretrained on ImageNet [27], is called S²AC-AlexNet-S/T pretrained. The corresponding settings are shown in

Table 2. Ablation study of the attention-based backbone network ($D\to W$).

Methods	Attention-Based Backbone Network	AlexNet	Pretrained	Accuracy(%)
S2AC	✓			66.5 ± 0.6
S2AC-AlexNet		✓		63.6 ± 0.7
S2AC-AlexNet pretrained		✓	✓	61.3 ± 0.5

. The table shows that if AlexNet was replaced by the proposed attention-based backbone network, the test accuracy would increase by about 3% points, mainly benefiting from learning long-range dependencies, which is a main challenge for convolutional neural networks. Furthermore, if the extractor is pretrained, it will converge quicker in both the training and testing phases. But nevertheless, the final accuracy in the training phase is 0.3% points lower for S²AC-AlexNet-S pretrained than S²AC-AlexNet-S, and the test accuracy is about 1.5% points lower for S²AC-AlexNet-S pretrained than S²AC-AlexNet-S. This demonstrates that pre-training has no significant effects on the model’s identification accuracy, but only on its convergence speed. The pre-training procedure cannot be used for S²AC since the brand new backbone network for the weights pre-trained on ImageNet does not exist.

When we compare test accuracy alone, S²AC-AlexNet-S pretrained fluctuates the most after convergence, while S²AC is more stable than S²AC-AlexNet-S. Nevertheless, S²AC has the highest test accuracy when compared with S²AC-AlexNet-S.

The validity of positional embedding has been demonstrated [23], and the purpose of this experiment is to see if positional embedding is more effective for the domain adaptation problem when employed in the source and target domains individually, or in both domains. For convenience, not performing the positional embedding operation in both source and target domains is called S²AC-∼S&∼T; adding location information in the source domain but not in the target domain is called S²AC-S&∼T; and vice versa is referred to as S²AC-∼S&T.

Figure 9 illustrates that positional embedding brings a significant rise of the accuracy, consequently resulting in an increase in F₁ score from 73.33% to 85.74% (see

Table 3. Ablation study of positional embeddings. "✓" denotes the execution of self-supervised learning in the corresponding domain. ($D\to W$).

Methods	Source Domain	Target Domain	Accuracy(%)	F1(%)
S2AC-S&T	✓	✓	43.2 ± 0.7	65.00
S2AC-∼S&∼T			45.5 ± 0.9	73.33
S2AC-S&∼T	✓		53.4 ± 1.6	84.23
S2AC-∼S&T		✓	61.6 ± 1.7	85.74

). This highlights the effectiveness of location information in domain adaptation tasks heavy with the ability to present a more detailed representation of this information. Nevertheless, adding positional information to each domain does not improve operational outcomes compared with no location information. In fact, superior recognition accuracy requires only positional embedding in the target domain.

Last but not least, we attempt to tweak the self-supervised learning module in order to compare and analyze its influence on the model’s performance. Similarly, we set up the self-supervised learning module in the source and target domains, respectively, and compare the test accuracy. As for the legends of tables and figures, S²AC-S&T represents the execution of self-supervised learning in both the source and target domains; S²AC-S&!T denotes performing only the self-supervised module in the source domain; and S²AC-!S&T is the exact opposite, executed only in the target domain.

As observed in Figure 10, the self-supervised learning module has made a noteworthy contribution to the enhancement of the model’s test accuracy. In terms of convergence speed and stability of the algorithm, the two models, S²AC-S&T and S²AC-!S, perform the best, with S²AC-S&T exhibiting a faster convergence rate than S²AC-!S. Although S²AC-!S initially displayed the highest test accuracy, S²AC-S&T tends to outperform the previous one towards the end of the iteration. The trend is further evidenced by the F₁ scores, where S²AC-S&T’s final score of 83.21% is about 5 points higher than S²AC-!S&T’s score of 78.67%; see

Table 4. Ablation study of the self-supervised learning module. ($D\to W$).

Methods	F1(%)
S2AC-S& !T	68.57
S2AC-S&T	83.21
S2AC-!S&T	78.67
S2AC-!S& !T	54.33

.

5.4. The Network Visualization and Impacts of Training Dataset Size

Although understanding the mechanism of deep neural networks can be arduous, certain visual clues reveal that S²AC is capable of fusing two domains to achieve domain adaptation. In order to guarantee fair comparison, we trained both models for 160 epochs. Figure 11 shows feature maps extracted by the model. One can see that the model learned the contour information of the object compared with AlexNet during the training phase (see Figure 11a), while also exhibiting competitive feature extraction capacity during the test phase (see Figure 11b). The first figure of Figure 11a) shows that two regions are activated corresponding to the top and bottom of the object, respectively, in the left, while only the region corresponding to the bottom of the object in the right (extracted by AlexNet). For the test phase, the second figure of Figure 11b demonstrates that S²AC (left) activates the majority of the light-colored areas, and the dark-colored region activated weakly can basically distinguish the shape of the object, whereas the right (AlexNet) cannot represent the outline of the object due to the lack of activated regions.

We also investigated the influence of dataset size on domain adaptation. Five training datasets of varying sizes were created and used for evaluation. In addition to the whole dataset, 100, 200, 300, and 400 images were randomly chosen from the source domain to create a new dataset. The corresponding F₁ score curves are shown in Figure 12b. From the figure, we conclude that the model’s performance improves as the size of the training set increases. The F₁ curve shows a trend of fast forward and slow backward, indicating that the number of images is less than 350 and the F₁ score grows rapidly as the number increases. As the dataset size, 350, increases, the performance improvement slows down and tends to saturate.

Table 5. The test performance with different size of datasets on Office31 ($D\to W$).

Dataset Size	Recall (%)	Precision (%)	F1 (%)	AP (%)
100	44.31	46.13	43.56	40.64
200	51.85	52.65	51.53	49.89
300	62.42	65.53	63.04	64.61
400	75.38	76.93	76.65	77.92
500	81.74	80.19	81.91	83.72

depicts F₁, recall, precision score, and the micro AP. The corresponding P-R curves are shown in Figure 12a.

5.5. Discussion

In this experiment, we chose one of the domains at random as the source domain for training and another as the target domain. The labels of the source domain are inherently known, while the labels of all target domains remain unknown. To compare the effectiveness of our method, in addition to a variant of S²AC with AlexNet, we evaluated popular algorithms, such as deep domain confusion (DDC), conditional domain adversarial networks (CDANs), etc., on the Office31, Office_Caltech_10, and OfficeHome datasets.

As seen in Table 6, our method (S²AC) obtained higher average performance than the other baseline approaches on the Office31 dataset. S²AC had the greatest running accuracy in four out of the six shifts. CDAN(+E) received the highest scores in the other two shifts.

In Table 7, we evaluated our method on Office_Caltech_10 compared with other baseline algorithms. The Office_Caltech_10 dataset contains four domains: Caltech, DLSR, Amazon, and Webcam. Three of them, i.e., DLSR, Amazon, and Webcam, are from the Office31 Dataset. To avoid duplication of effort, Table 7 simply compares the shifts of Caltech with the other three domains (DLSR, Amazon, and Webcam); hence, Table 7 only includes six shifts ($A\to C$, $D\to C$, etc.). In three out of the six shifts, our algorithm performs best, and the gap between S²AC and the best baseline method in the other three shifts is extremely small (⩽2 approximately), which illustrates its effectiveness on this dataset.

The results on OfficeHome are reported in Table 8, in most transfer tasks, the S²AC-model beats other baseline methods, demonstrating high performance and potential for improvement. One can see that OfficeHome contains more categories and a larger volume than the other two datasets, which are visually more distinct from each other, and hence are more difficult to classify because of their substantially lower classification accuracy within the domain [28]. Since the prior work’s feature extraction is distance-agnostic, the extracted features are unsuitable for classification in the presence of a large number of categories.

In the tables, AttNet-5 represents a five-layer stack, and it is interesting to note that although the number of layers in the stack is less than ResNet-34, AlexNet, S²AC outperforms them in most cases. Since the number of layers of AttNet-5 is significantly smaller than ResNet-34, AlexNet, its convergence is faster and it is easier to train. In addition, considering the hardware environment, we did not include ResNet-50, ResNet-101 in the Table 6, Table 7 and Table 8; researchers who are able to do so might use it as a starting point for their study.

Table 6. Performance comparison for all six domain shifts on Office31. A: Amazon, D: DLSR, W: Webcam.

Method	Backbone	A→W	A→D	W→A	W→D	D→A	D→W
No Adaptation	AlexNet [29]	35.3 ± 0.5	22.3 ± 1.2	30.2 ± 0.5	37.4 ± 1.6	21.2 ± 0.6	39.5 ± 0.3
	ResNet-34 [30]	37.6 ± 0.7	25.5 ± 0.6	32.8 ± 0.4	39.4 ± 0.4	24.3 ± 0.9	39.3 ± 0.5
DDC [5]	Deep CNN [5]	43.3 ± 0.8	30.2 ± 1.4	35.6 ± 0.7	45.6 ± 1.3	25.3 ± 0.5	65.4 ± 0.4
CDAN [6]	AlexNet [29]	41.4 ± 1.9	36.2 ± 1.6	35.1 ± 1.5	75.2 ± 0.6	38.4 ± 1.5	65.6 ± 1.6
	ResNet-34 [30]	44.7 ± 0.4	37.4 ± 0.6	38.1 ± 0.7	76.7 ± 0.9	40.6 ± 1.6	66.3 ± 0.8
CDAN+E [6]	AlexNet [29]	45.5 ± 0.6	37.4 ± 1.6	38.2 ± 0.6	72.4 ± 1.5	35.7 ± 0.43	68.4 ± 1.5
	ResNet-34 [30]	47.3 ± 0.5	40.5 ± 0.6	39.7 ± 1.6	74.7 ± 0.7	40.1 ± 1.1	71.5 ± 0.7
Deep CORAL [31]	AlexNet [29]	50.5 ± 0.7	38.1 ± 1.5	37.9 ± 0.6	71.4 ± 0.6	38.6 ± 0.8	60.2 ± 0.7
P-ADC [15]	AlexNet [29] with an attention layer	51.4 ± 0.9	40.3 ± 0.7	39.5 ± 0.6	73.9 ± 0.5	40.1 ± 0.7	63.8 ± 0.5
S2AC-AlexNet	AlexNet [29]	56.8 ± 0.7	44.0 ± 1.2	39.8 ± 0.9	74.7 ± 1.7	40.9 ± 1.1	63.6 ± 0.7
S2AC	AttNet-5	56.8 ± 1.1	45.6 ± 1.4	40.2 ± 0.5	75.9 ± 0.7	41.5 ± 0.6	61.6 ± 1.6

Table 6. Performance comparison for all six domain shifts on Office31. A: Amazon, D: DLSR, W: Webcam.

Method	Backbone	A→W	A→D	W→A	W→D	D→A	D→W
No Adaptation	AlexNet [29]	35.3 ± 0.5	22.3 ± 1.2	30.2 ± 0.5	37.4 ± 1.6	21.2 ± 0.6	39.5 ± 0.3
	ResNet-34 [30]	37.6 ± 0.7	25.5 ± 0.6	32.8 ± 0.4	39.4 ± 0.4	24.3 ± 0.9	39.3 ± 0.5
DDC [5]	Deep CNN [5]	43.3 ± 0.8	30.2 ± 1.4	35.6 ± 0.7	45.6 ± 1.3	25.3 ± 0.5	65.4 ± 0.4
CDAN [6]	AlexNet [29]	41.4 ± 1.9	36.2 ± 1.6	35.1 ± 1.5	75.2 ± 0.6	38.4 ± 1.5	65.6 ± 1.6
	ResNet-34 [30]	44.7 ± 0.4	37.4 ± 0.6	38.1 ± 0.7	76.7 ± 0.9	40.6 ± 1.6	66.3 ± 0.8
CDAN+E [6]	AlexNet [29]	45.5 ± 0.6	37.4 ± 1.6	38.2 ± 0.6	72.4 ± 1.5	35.7 ± 0.43	68.4 ± 1.5
	ResNet-34 [30]	47.3 ± 0.5	40.5 ± 0.6	39.7 ± 1.6	74.7 ± 0.7	40.1 ± 1.1	71.5 ± 0.7
Deep CORAL [31]	AlexNet [29]	50.5 ± 0.7	38.1 ± 1.5	37.9 ± 0.6	71.4 ± 0.6	38.6 ± 0.8	60.2 ± 0.7
P-ADC [15]	AlexNet [29] with an attention layer	51.4 ± 0.9	40.3 ± 0.7	39.5 ± 0.6	73.9 ± 0.5	40.1 ± 0.7	63.8 ± 0.5
S2AC-AlexNet	AlexNet [29]	56.8 ± 0.7	44.0 ± 1.2	39.8 ± 0.9	74.7 ± 1.7	40.9 ± 1.1	63.6 ± 0.7
S2AC	AttNet-5	56.8 ± 1.1	45.6 ± 1.4	40.2 ± 0.5	75.9 ± 0.7	41.5 ± 0.6	61.6 ± 1.6

Table 7. Performance comparison for six domain shifts on Office_Caltech_10. C: Caltech, D: DLSR, A: Amazon, W: Webcam.

Method	Backbone	$\mathit{A}\to \mathit{C}$	$\mathit{D}\to \mathit{C}$	$\mathit{W}\to \mathit{C}$	$\mathit{C}\to \mathit{A}$	$\mathit{C}\to \mathit{D}$	$\mathit{C}\to \mathit{W}$
No Adaptation	AlexNet [29]	42.0 ± 0.4	26.5 ± 1.5	35.0 ± 0.9	45.3 ± 0.9	50.5 ± 0.5	49.2 ± 0.6
	ResNet-34 [30]	45.8 ± 0.7	31.4 ± 0.6	46.2 ± 0.3	47.5 ± 0.4	46.5 ± 0.6	51.3 ± 1.1
DDC [5]	Deep CNN [5]	64.4 ± 0.8	54.4 ± 0.9	68.4 ± 1.2	74.5 ± 1.4	52.4 ± 0.4	64.8 ± 0.7
CDAN [6]	AlexNet [29]	62.5 ± 0.9	60.2 ± 1.9	71.4 ± 1.9	73.6 ± 0.6	53.4 ± 0.5	64.3 ± 1.7
	ResNet-34 [30]	65.2 ± 0.4	63.3 ± 0.4	69.6 ± 0.9	74.3 ± 1.3	58.4 ± 0.6	68.4 ± 0.6
CDAN+E [6]	AlexNet [29]	64.9 ± 1.5	63.4 ± 1.1	71.3 ± 0.2	75.6 ± 0.8	60.4 ± 0.9	70.5 ± 1.2
	ResNet-34 [30]	67.2 ± 0.5	67.8 ± 0.7	73.2 ± 0.7	80.0 ± 1.2	63.6 ± 0.5	69.1 ± 0.8
Deep CORAL [31]	AlexNet [29]	74.8 ± 0.9	66.6 ± 0.6	67.4 ± 0.3	81.9 ± 0.7	60.2 ± 0.9	70.5 ± 1.3
P-ADC [15]	AlexNet [29] with an attention layer	76.5 ± 0.2	68.5 ± 0.1	70.7 ± 0.7	84.6 ± 0.1	63.8 ± 0.3	72.4 ± 0.2
S2AC-AlexNet	AlexNet [29]	72.5 ± 0.1	61.5 ± 0.1	69.6 ± 0.3	85.2 ± 0.2	55.7 ± 0.3	74.4 ± 0.2
S2AC	AttNet-5	79.3 ± 0.4	67.4 ± 0.2	71.1 ± 0.2	83.9 ± 0.3	69.8 ± 0.3	75.7 ± 0.5

Table 7. Performance comparison for six domain shifts on Office_Caltech_10. C: Caltech, D: DLSR, A: Amazon, W: Webcam.

Method	Backbone	$\mathit{A}\to \mathit{C}$	$\mathit{D}\to \mathit{C}$	$\mathit{W}\to \mathit{C}$	$\mathit{C}\to \mathit{A}$	$\mathit{C}\to \mathit{D}$	$\mathit{C}\to \mathit{W}$
No Adaptation	AlexNet [29]	42.0 ± 0.4	26.5 ± 1.5	35.0 ± 0.9	45.3 ± 0.9	50.5 ± 0.5	49.2 ± 0.6
	ResNet-34 [30]	45.8 ± 0.7	31.4 ± 0.6	46.2 ± 0.3	47.5 ± 0.4	46.5 ± 0.6	51.3 ± 1.1
DDC [5]	Deep CNN [5]	64.4 ± 0.8	54.4 ± 0.9	68.4 ± 1.2	74.5 ± 1.4	52.4 ± 0.4	64.8 ± 0.7
CDAN [6]	AlexNet [29]	62.5 ± 0.9	60.2 ± 1.9	71.4 ± 1.9	73.6 ± 0.6	53.4 ± 0.5	64.3 ± 1.7
	ResNet-34 [30]	65.2 ± 0.4	63.3 ± 0.4	69.6 ± 0.9	74.3 ± 1.3	58.4 ± 0.6	68.4 ± 0.6
CDAN+E [6]	AlexNet [29]	64.9 ± 1.5	63.4 ± 1.1	71.3 ± 0.2	75.6 ± 0.8	60.4 ± 0.9	70.5 ± 1.2
	ResNet-34 [30]	67.2 ± 0.5	67.8 ± 0.7	73.2 ± 0.7	80.0 ± 1.2	63.6 ± 0.5	69.1 ± 0.8
Deep CORAL [31]	AlexNet [29]	74.8 ± 0.9	66.6 ± 0.6	67.4 ± 0.3	81.9 ± 0.7	60.2 ± 0.9	70.5 ± 1.3
P-ADC [15]	AlexNet [29] with an attention layer	76.5 ± 0.2	68.5 ± 0.1	70.7 ± 0.7	84.6 ± 0.1	63.8 ± 0.3	72.4 ± 0.2
S2AC-AlexNet	AlexNet [29]	72.5 ± 0.1	61.5 ± 0.1	69.6 ± 0.3	85.2 ± 0.2	55.7 ± 0.3	74.4 ± 0.2
S2AC	AttNet-5	79.3 ± 0.4	67.4 ± 0.2	71.1 ± 0.2	83.9 ± 0.3	69.8 ± 0.3	75.7 ± 0.5

Table 8. Performance comparison for 12 domain shifts on OfficeHome. A: Art, C: Clipart, P: Product, R: Real-World.

Method	Backbone	$\mathit{A}\to \mathit{C}$	$\mathit{A}\to \mathit{P}$	$\mathit{A}\to \mathit{R}$	$\mathit{C}\to \mathit{A}$	$\mathit{C}\to \mathit{P}$	$\mathit{C}\to \mathit{R}$	$\mathit{P}\to \mathit{A}$	$\mathit{P}\to \mathit{C}$	$\mathit{P}\to \mathit{R}$	$\mathit{R}\to \mathit{A}$	$\mathit{R}\to \mathit{C}$	$\mathit{R}\to \mathit{P}$
No Adaptation	AlexNet [29]	35.6 ± 0.3	33.6 ± 1.3	40.0 ± 0.4	27.5 ± 0.5	33.1 ± 0.3	29.6 ± 0.2	37.4 ± 0.8	32.7 ± 1.0	28.4 ± 0.6	30.3 ± 0.7	41.5 ± 0.3	37.3 ± 0.4
	ResNet-34 [30]	34.9 ± 0.5	38.4 ± 0.6	45.7 ± 0.4	38.0 ± 1.0	35.8 ± 0.5	32.7 ± 0.6	39.1 ± 1.3	39.1 ± 0.5	34.6 ± 1.1	34.8 ± 0.7	49.6 ± 0.2	41.5 ± 0.9
DDC [5]	Deep CNN [5]	41.0 ± 1.4	54.5 ± 0.6	63.4 ± 0.6	53.4 ± 0.6	66.9 ± 0.4	45.8 ± 1.0	49.5 ± 0.5	55.9 ± 0.7	39.4 ± 0.3	48.1 ± 1.6	71.5 ± 0.8	66.4 ± 0.6
CDAN [6]	AlexNet [29]	51.9 ± 1.7	53.7 ± 0.9	65.1 ± 2.1	71.4 ± 1.1	63.3 ± 0.6	51.4 ± 1.7	51.8 ± 0.7	58.5 ± 1.4	47.8 ± 0.4	49.7 ± 2.1	73.1 ± 0.7	67.1 ± 0.1
	ResNet-34 [30]	52.4 ± 0.5	55.3 ± 0.3	65.7 ± 1.0	69.3 ± 0.6	64.1 ± 0.9	57.2 ± 0.6	54.9 ± 0.7	59.1 ± 0.9	49.2 ± 0.8	50.1 ± 1.4	75.6 ± 0.9	68.5 ± 0.7
CDAN+E [6]	AlexNet [29]	55.3 ± 0.6	60.4 ± 1.5	67.1 ± 1.6	75.3 ± 1.0	65.2 ± 0.8	55.7 ± 1.1	61.7 ± 0.7	61.4 ± 2.0	49.9 ± 0.9	51.5 ± 0.7	75.8 ± 0.8	69.5 ± 0.7
	ResNet-34 [30]	60.2 ± 0.6	58.1 ± 0.9	66.9 ± 0.7	76.1 ± 0.6	66.9 ± 0.6	59.5 ± 0.6	63.1 ± 0.6	62.7 ± 0.9	50.6 ± 0.6	53.5 ± 0.6	74.0 ± 0.6	69.9 ± 0.7
Deep CORAL [31]	AlexNet [29]	54.6 ± 0.8	62.4 ± 1.3	67.1 ± 0.5	76.7 ± 0.5	67.5 ± 0.3	60.6 ± 0.7	60.5 ± 0.8	62.6 ± 0.6	45.7 ± 0.5	54.4 ± 1.4	73.1 ± 0.5	69.6 ± 0.5
P-ADC [15]	AlexNet [29] with an attention layer	55.2 ± 0.5	62.1 ± 0.8	65.6 ± 1.2	75.0 ± 0.8	68.1 ± 0.2	61.1 ± 0.4	62.7 ± 1.4	61.9 ± 0.7	47.1 ± 0.4	56.1 ± 0.9	72.2 ± 1.3	70.2 ± 0.9
S2AC-AlexNet	AlexNet [29]	56.3 ± 1.7	61.3 ± 0.7	68.1 ± 0.5	76.7 ± 0.6	68.0 ± 0.9	65.3 ± 0.5	61.3 ± 1.1	62.6 ± 0.3	50.2 ± 0.6	56.5 ± 0.3	73.7 ± 0.7	70.3 ± 1.3
S2AC	AttNet-5	56.9 ± 0.8	62.6 ± 0.6	68.1 ± 0.6	77.6 ± 1.0	68.5 ± 0.6	64.5 ± 0.7	62.1 ± 0.4	63.6 ± 1.1	53.2 ± 0.5	57.6 ± 0.6	76.4 ± 1.2	71.8 ± 0.9

Table 8. Performance comparison for 12 domain shifts on OfficeHome. A: Art, C: Clipart, P: Product, R: Real-World.

Method	Backbone	$\mathit{A}\to \mathit{C}$	$\mathit{A}\to \mathit{P}$	$\mathit{A}\to \mathit{R}$	$\mathit{C}\to \mathit{A}$	$\mathit{C}\to \mathit{P}$	$\mathit{C}\to \mathit{R}$	$\mathit{P}\to \mathit{A}$	$\mathit{P}\to \mathit{C}$	$\mathit{P}\to \mathit{R}$	$\mathit{R}\to \mathit{A}$	$\mathit{R}\to \mathit{C}$	$\mathit{R}\to \mathit{P}$
No Adaptation	AlexNet [29]	35.6 ± 0.3	33.6 ± 1.3	40.0 ± 0.4	27.5 ± 0.5	33.1 ± 0.3	29.6 ± 0.2	37.4 ± 0.8	32.7 ± 1.0	28.4 ± 0.6	30.3 ± 0.7	41.5 ± 0.3	37.3 ± 0.4
	ResNet-34 [30]	34.9 ± 0.5	38.4 ± 0.6	45.7 ± 0.4	38.0 ± 1.0	35.8 ± 0.5	32.7 ± 0.6	39.1 ± 1.3	39.1 ± 0.5	34.6 ± 1.1	34.8 ± 0.7	49.6 ± 0.2	41.5 ± 0.9
DDC [5]	Deep CNN [5]	41.0 ± 1.4	54.5 ± 0.6	63.4 ± 0.6	53.4 ± 0.6	66.9 ± 0.4	45.8 ± 1.0	49.5 ± 0.5	55.9 ± 0.7	39.4 ± 0.3	48.1 ± 1.6	71.5 ± 0.8	66.4 ± 0.6
CDAN [6]	AlexNet [29]	51.9 ± 1.7	53.7 ± 0.9	65.1 ± 2.1	71.4 ± 1.1	63.3 ± 0.6	51.4 ± 1.7	51.8 ± 0.7	58.5 ± 1.4	47.8 ± 0.4	49.7 ± 2.1	73.1 ± 0.7	67.1 ± 0.1
	ResNet-34 [30]	52.4 ± 0.5	55.3 ± 0.3	65.7 ± 1.0	69.3 ± 0.6	64.1 ± 0.9	57.2 ± 0.6	54.9 ± 0.7	59.1 ± 0.9	49.2 ± 0.8	50.1 ± 1.4	75.6 ± 0.9	68.5 ± 0.7
CDAN+E [6]	AlexNet [29]	55.3 ± 0.6	60.4 ± 1.5	67.1 ± 1.6	75.3 ± 1.0	65.2 ± 0.8	55.7 ± 1.1	61.7 ± 0.7	61.4 ± 2.0	49.9 ± 0.9	51.5 ± 0.7	75.8 ± 0.8	69.5 ± 0.7
	ResNet-34 [30]	60.2 ± 0.6	58.1 ± 0.9	66.9 ± 0.7	76.1 ± 0.6	66.9 ± 0.6	59.5 ± 0.6	63.1 ± 0.6	62.7 ± 0.9	50.6 ± 0.6	53.5 ± 0.6	74.0 ± 0.6	69.9 ± 0.7
Deep CORAL [31]	AlexNet [29]	54.6 ± 0.8	62.4 ± 1.3	67.1 ± 0.5	76.7 ± 0.5	67.5 ± 0.3	60.6 ± 0.7	60.5 ± 0.8	62.6 ± 0.6	45.7 ± 0.5	54.4 ± 1.4	73.1 ± 0.5	69.6 ± 0.5
P-ADC [15]	AlexNet [29] with an attention layer	55.2 ± 0.5	62.1 ± 0.8	65.6 ± 1.2	75.0 ± 0.8	68.1 ± 0.2	61.1 ± 0.4	62.7 ± 1.4	61.9 ± 0.7	47.1 ± 0.4	56.1 ± 0.9	72.2 ± 1.3	70.2 ± 0.9
S2AC-AlexNet	AlexNet [29]	56.3 ± 1.7	61.3 ± 0.7	68.1 ± 0.5	76.7 ± 0.6	68.0 ± 0.9	65.3 ± 0.5	61.3 ± 1.1	62.6 ± 0.3	50.2 ± 0.6	56.5 ± 0.3	73.7 ± 0.7	70.3 ± 1.3
S2AC	AttNet-5	56.9 ± 0.8	62.6 ± 0.6	68.1 ± 0.6	77.6 ± 1.0	68.5 ± 0.6	64.5 ± 0.7	62.1 ± 0.4	63.6 ± 1.1	53.2 ± 0.5	57.6 ± 0.6	76.4 ± 1.2	71.8 ± 0.9

5.6. Computational Complexity

We calculated the average training time over all the cross-domain scenarios on the Office31 dataset to compare the time complexity of our approach to other baseline algorithms, as shown in

Table 9. The average training time of each approach on the Office31 dataset with epoch = 1, batch_size = 128 (Seconds).

Method	DDC [5]	Deep CORAL [31]	DANN [32]	CDAN [6]	CDAN + E [6]	P-ADC [15]	S2AC
Computational Time	34.8	11.9	17.6	18.01	18.06	12.3	12.7

. When comparing to other algorithms, Deep CORAL has the lowest computational complexity. Overall, the adversarial-based approach is more computationally difficult than the distance-based approach. Both S²AC and P-ADC are located in the 12-s average computing time echelon, just behind Deep CORAL.

6. Conclusions and Future Work

Our study suggests a self-supervised attention CORAL based on a Mahalanobis distance approach for domain adaptation, based on the attention mechanism. This technique might eliminate certain influences from data insufficiency, enhance local extraction reliance, and increase the test accuracy of base algorithms, such as deep CORAL. Several ideas are applied to attain these improvements. To begin with, the attention architecture facilitates attribute extraction, which aids in the learning to long-distance dependencies and allows for enhanced attribute reuse. The second is that the incorporation of self-supervised learning enables S²AC to acquire high-quality representations even with limited labeled data, which can be utilized to learn new tasks that lack sufficient data. Integating these modules, S²AC constructs an adaptation model that is completely independent of pretrained models for feature extraction, with a global receptive field and that are pseudo-label-free.

Numerous experimental results show that it is effective and exceeds other compared algorithms in terms of test accuracy and time efficiency.

We hope that our work stimulates researchers in other fields to consider using this approach to broaden their research horizons, as well as domain adaptation researchers to be inspired by the great number of innovative methods that have lately been proposed.

In the future, category information will be considered to improve the domain alignment performance, particularly the utilization of object and background information in the same category in different domains. Moreover, in the absence of knowledge about the target domain, information about multiple source domains will be studied and generalized to unseen domains in diverse environments.