Unsupervised Domain Adaptation with Coupled Generative Adversarial Autoencoders

Abstract

When large-scale annotated data are not available for certain image classification tasks, training a deep convolutional neural network model becomes challenging. Some recent domain adaptation methods try to solve this problem using generative adversarial networks and have achieved promising results. However, these methods are based on a shared latent space assumption and they do not consider the situation when shared high level representations in different domains do not exist or are not ideal as they assumed. To overcome this limitation, we propose a neural network structure called coupled generative adversarial autoencoders (CGAA) that allows a pair of generators to learn the high-level differences between two domains by sharing only part of the high-level layers. Additionally, by introducing a class consistent loss calculated by a stand-alone classifier into the generator optimization, our model is able to generate class invariant style-transferred images suitable for classification tasks in domain adaptation. We apply CGAA to several domain transferred image classification scenarios including several benchmark datasets. Experiment results have shown that our method can achieve state-of-the-art classification results.

1. Introduction

Large-scale well-annotated datasets such as Microsoft COCO [1], ImageNet [2] and KITTI [3] have played a vital role in the recent success of deep learning based models on computer vision tasks such as image classification, target detection, semantic segmentation and so on. However, models trained with large datasets still cannot generalize well to novel datasets when these datasets have different feature distributions. The typical solution is to further fine-tune these models on the task specific datasets. However, creating such datasets can be expensive and time-consuming. Unsupervised domain adaptation offers a solution to this problem by learning a mapping between a labeled dataset (source domain) and an unlabeled dataset (target domain) or by learning domain invariant features. Conventional domain adaptation approaches for image classification are usually developed in two separate steps: designing and extracting fixed features and then training models to reduce their differences in either the marginal distributions or the conditional distributions between domains [4,5,6,7]. Recent deep learning based domain adaptation approaches avoid the difficulty of feature design by extracting features automatically through convolutional neural networks [8,9,10,11,12,13].

Among all kinds of deep neural network based domain adaptation approaches, generative adversarial network (GAN) [14] has become a popular branch. A typical GAN trains a generator and a discriminator to compete against each other. The generator is trained to produce synthetic images as real as possible, whereas the discriminator is trained to distinguish the synthetic and real images. When applying GAN to domain adaptation for image classification, there are two major types of approaches. The first type trains a GAN to generate unlabeled target domain images, thus enlarging the data volume to train a more robust image classifier [15,16,17]. In these methods, the training strategy of the final classifier need to be carefully designed since the newly generated images have no label. The other type of approaches generate labeled target domain images directly by transferring the source domain images into target domain style and have achieved some state-of-the-art results, such as CoGAN [18] and UNIT [19]. These methods are based on the shared latent space assumption, which assumes that the differences of the source domain and the target domain are primarily low-level, and that the two domains share a common high-level latent space. This assumption works well for simple scenarios such as digits adaptation between MNIST [20] and USPS [21] but faces challenges when the semantic features are more complex. When shared high-level latent space in different domains does not exist or such latent space is not as ideal as assumed, these methods will fail [18].

In this paper, we propose an unsupervised domain adaptation method for image classification by combining generative adversarial networks with autoencoders. We call our proposed network architecture Coupled Generative Adversarial Autoencoders (CGAA). Our work is perhaps most similar to CoGAN and UNIT, but we try to solve the aforementioned shortcomings of these methods by the following designs: CGAA consists of a pair of generative adversarial networks (GAN) and a domain adaptive classifier. The architecture of the generator in GAN is designed based on the autoencoder. During training, part of the layers in the generators are forced to share their weights, which gives our model the ability to learn the domain transformation in an unsupervised manner and generate synthetic target domain images with label. By decoupling the highest level layer, we give our model the capacity to tolerant the differences of high-level features between the domains. The classifier provides a class-invariant loss to help the generator produce more suitable images for the classification task in domain adaptation. The main contributions of this work are:

• We propose an unsupervised domain adaptation method for image classification. Our method trains a pair of coupled generative adversarial networks in which the generator has an encoder-decoder structure.
• We force part of the layers in the generator to share weights during training to generate labeled synthetic images, and make the highest level layer decoupled for different high-level representations.
• We introduce a class consistent loss into the GAN training, which is calculated from the output of a stand-alone domain adaptive classifier. It can help the generator to generate more suitable images for domain adaptation.

2. Related Work

The goal of unsupervised domain adaptation is to transfer knowledge from a labeled source dataset to a target dataset where labeled data is not available. Recent studies have tried to learn transferable features with deep neural networks. The DDC method [11] learned domain invariant representations by introducing an adaptation layer and a Maximum Mean Discrepancy (MMD) domain confusion loss. The work in [22] extended the MMD to jointly mitigate the gaps of marginal and conditional distributions between source and target domain. The DAN method [9] embedded task-specific layers in a reproducing kernel Hilbert space to enhance the feature transferability. The DANN method [8,23] suggested that the features suitable for domain adaptation should be both discriminative and domain-invariant and added a domain classifier at the end of the feature extractor to learn domain invariant features. CAN [24] suggested that some characteristic information from target domain data may be lost after learning domain-invariant features with DANN. Therefore, CAN introduced a set of domain classifiers into multiple blocks to learn domain-informative representations at lower blocks and domain-uninformative representations at higher blocks. The work of [25] proposed to learn a representation that transfered the semantic structure from a well labeled source domain to the sparsely labeled target domain by adding a domain classifier and a domain confusion loss. The DRCN [12] proposed a model which had two pipelines: The first was label prediction for the source domain and the second was data reconstruction for the target domain. ADDA [26] learns the representation of the source domain and then maps the target data to the same space through a domain-adversarial loss.

Other works have attempted to use GANs [14] into image-to-image translation and domain adaptation. The “pix2pix” framework [27] used a conditional generative adversarial network to learn a mapping from input to output images with paired images. CycleGAN [28] learned the mapping without paired training examples using a cycle consistency loss. The method in [29] used GAN to translate unpaired images between domains while remain high level semantic information aligned by introducing attention consistent loss. CoGAN [18] learned a joint distribution of images without corresponding supervision by training two GANs to generate the source and target images respectively given the same noise input and tying the high-level layer parameters of the two GANs. Instead of generating images from noise vectors, PixelDA [30] generated style-transferred images conditioned on the source images. CoGASA [31] integrated a stacked autoencoder with the CoGAN, and UNIT [19] proposed an image-to-image translation framework based on CoGAN and VAE [32].

3. Proposed Approach

In this section, we introduce the model structure of CGAA and explain our training strategy. As illustrated in Figure 1, CGAA contains seven sub-networks. Two image encoders $EN{C}_S$ and $EN{C}_T$, two image decoders $DE{C}_S$ and $DE{C}_T$, two adversarial discriminators $D_S$ and $D_T$, and a classifier C.

3.1. Image Reconstruction and Autoencoder

The encoder $EN{C}_S$ and decoder $DE{C}_S$ constitute an autoencoder for the source domain $X^S$. The $EN{C}_S$ maps an input image $x^S\in X^S$ to a code in a latent space and based on this code, the $DE{C}_S$ reconstructs the input image as $x^{S\to S}$. Similarly, $EN{C}_T$ and $DE{C}_T$ constitute an autoencoder for the target domain $X^T$. The aim of these two autoencoders is to reconstruct images as similar as possible to their input images in each domain. We use the mean squared error as the loss function to penalize the differences between inputs and outputs:

$${\mathcal{L}}_{RE{C}_S}(EN{C}_S,DE{C}_S)={\mathbb{E}}_{X^S}[\frac{1}{k}\parallel x^S-DE{C}_S\left(EN{C}_S\left(x^S\right)\right){\parallel }_2^2]$$

(1)

$${\mathcal{L}}_{RE{C}_T}(EN{C}_T,DE{C}_T)={\mathbb{E}}_{X^T}[\frac{1}{k}\parallel x^T-DE{C}_T\left(EN{C}_T\left(x^T\right)\right){\parallel }_2^2]$$

(2)

where k is the number of pixels in input $x^S$ and $\parallel ·\parallel$ is the squared $L_2$-norm.

3.2. Style Transfer and GAN

Style-transferred synthetic images can be generated by changing the combination of encoders and decoders. More specifically, let $DE{C}_T$ take the output of $EN{C}_S$, and let $DE{C}_S$ take the output of $EN{C}_T$, thus we are able to change the style of images between domains.

When training an autoencoder, element level penalties such as squared error, is the classic choice. However, as discussed in [34], they are actually not ideal for image generation, and the generated images are always blurred. Therefore, we combine autoencoder with GAN in our method. By jointly training an autoencoder and a GAN, we can generate better images with the feature level metric expressed by the discriminator. In our method CGAA, the $EN{C}_S$, $DE{C}_T$ and $D_T$ constitute a generative adversarial network. During training, $DE{C}_T$ takes the output of $EN{C}_S$, mapping an input source domain image $x^S$ into a target domain style synthetic image $x^{S\to T}$, and discriminator $D_T$ is trained to distinguish between synthetic images $x^{S\to T}$ and real images $x^T$ from the target domain. Similarly, $EN{C}_T$ and $DE{C}_S$ generate synthetic source-style images $x^{T\to S}$ conditioned on the target domain images $x^T$ and $D_S$ is trained to distinguish between real source domain images $x^S$ and synthetic images $x^{T\to S}$. With this pair of GANs, our goal is to minmax the following object:

$$\begin{array}{cc}\hfill \underset{EN{C}_S,EN{C}_T,DE{C}_S,DE{C}_T,C}{min}\underset{D_S,D_T}{max}& \alpha {\mathcal{L}}_{GA{N}_S}(EN{C}_T,DE{C}_S,D_S)+\beta {\mathcal{L}}_{RE{C}_S}(EN{C}_S,DE{C}_S)\hfill \\ & \alpha {\mathcal{L}}_{GA{N}_T}(EN{C}_S,DE{C}_T,D_T)+\beta {\mathcal{L}}_{RE{C}_T}(EN{C}_T,DE{C}_T)\hfill \end{array}$$

(3)

where $\alpha$ and $\beta$ are weights that balance the GAN loss and the reconstruction loss. ${\mathcal{L}}_{GA{N}_S}$ and ${\mathcal{L}}_{GA{N}_T}$ represent the GAN loss:

$${\mathcal{L}}_{GA{N}_S}(EN{C}_T,DE{C}_S,D_S)={\mathbb{E}}_{X^S}[log{D}_S\left(x^S\right)]+{\mathbb{E}}_{X^T}[log(1-D_S\left(DE{C}_S\left(EN{C}_T\left(x^T\right)\right)\right))]$$

(4)

$${\mathcal{L}}_{GA{N}_T}(EN{C}_S,DE{C}_T,D_T)={\mathbb{E}}_{X^T}[log{D}_T\left(x^T\right)]+{\mathbb{E}}_{X^S}[log(1-D_T\left(DE{C}_T\left(EN{C}_S\left(x^S\right)\right)\right))]$$

(5)

3.3. Weight Sharing

Previous shared latent space assumption based methods such as CoGAN [18] and UNIT [19] are able to conduct the domain transfer training without paired images in different domains by sharing weights in the generators. They assume that images from different domains only have low-level semantic differences due to noise, resolution, illumination and color, etc. Furthermore, a pair of corresponding images in two domains share the same high-level concepts. Therefore, layers responsible for high level representation are forced to share their weights. However, these methods are based on the existence of shared high-level representations in the two domains. If the high-level semantic features are complex and such shared representations do not exist or are hard to find, these methods will not work out well. To this end, our method extends the previous works by only sharing part of the high-level layers and decoupling the rest. More specifically, we do not share the weights of last layer in the encoder, the first layer in decoder, and the last two layers in the discriminator, as shown is Figure 2. Under this structure, the generative models will, to some extent, tolerate different high level representations in different domains.

3.4. Domain Adapted Classifier

The focus of the unsupervised domain adaptation method described in this paper is to extend a classifier’s generalization ability on two domains, originally trained on the source domain that generalizes to the unlabeled target domain. To this end, we train a classifier C with the source domain images and the synthetic target domain images generated by $\{EN{C}_S,DE{C}_T\}$. Unlike some other domain adaptation works where the discriminator is modified as classifier, our classifier has a stand-alone structure, shown in Figure 1, which is easy to be detached from the whole network for future training. We do not describe the detailed architecture of the classifier in Figure 2 because it is task specific. During training, we use the typical cross-entropy loss to optimize C:

$${\mathcal{L}}_C(EN{C}_S,DE{C}_T,C)={\mathbb{E}}_{x^S,y^S}[-y^{S}logC\left(DE{C}_T\left(EN{C}_S\left(x^S\right)\right)\right)-y^{S}logC\left(x^S\right)]$$

(6)

In addition, the classifier C has another function in CGAA, that is being a part of the optimization of the generator $\{EN{C}_S,DE{C}_T\}$ with a class-consistency loss. When training the generator, C assigns a label $\widehat{y}$ the generated image $x^{S\to T}$, and the class-consistency loss is defined as:

$${\mathcal{L}}_{CC}(EN{C}_S,DE{C}_T,C)={\mathbb{E}}_{x^S,y^S}[-y^{S}logC\left(DE{C}_T\left(EN{C}_S\left(x^S\right)\right)\right)]$$

(7)

where $y^S$ is the class label of the input $x^S$. The class-consistency loss makes sure the output image $x^{S\to T}$ remains class-invariant, which is essential for the classification task in domain adaptation. With ${\mathcal{L}}_C$ and ${\mathcal{L}}_{CC}$, our final optimization object becomes:

$$\begin{array}{cc}\hfill \underset{EN{C}_S,EN{C}_T,DE{C}_S,DE{C}_T,C}{min}\underset{D_S,D_T}{max}& \alpha {\mathcal{L}}_{GA{N}_S}(EN{C}_T,DE{C}_S,D_S)+\beta {\mathcal{L}}_{RE{C}_S}(EN{C}_S,DE{C}_S)\hfill \\ & \alpha {\mathcal{L}}_{GA{N}_T}(EN{C}_S,DE{C}_T,D_T)+\beta {\mathcal{L}}_{RE{C}_T}(EN{C}_T,DE{C}_T)+\gamma {\mathcal{L}}_{CC}(EN{C}_S,DE{C}_T,C)\hfill \\ & {\mathcal{L}}_C(EN{C}_S,DE{C}_T,C)\hfill \end{array}$$

(8)

The minmax optimization of Equation (8) is achieved by two alternative steps. During the first step, we keep the discriminators and the classifier fixed, optimize the generators and at the same time, minimize the reconstruction losses and the class consistent loss. During the second step, we keep the generators fixed and optimize the discriminators and the classifier.

4. Experiment Results and Evaluation

To evaluate our method, we conduct experiments on various domain adaptation scenarios and compare our results with other recently reported methods.

4.1. Facial Expression Recognition

We first evaluate our method on cross domain facial expression recognition task with three publicly available facial expression datasets: JAFFE, MMI and CK+. The images in these datasets have different resolutions and illuminations, and the subjects vary in gender, age and cultural background. Figure 3 shows some of the sample images from these datasets.

JAFFE dataset [35,36] contains 213 facial expression images. These images are from 10 Japanese females with seven expressions (angry, disgust, fear, happy, sad, surprise and neutral). We use all of the images in JAFFE in our experiments.

MMI dataset [37,38] consists of over 2900 videos as well as still images of 75 subjects, in which 235 videos have emotional labels. We choose the peak frame of each video that has the six basic emotions (angry, disgust, fear, happy, sad and surprise) and the first frame of these videos as neutral emotion images. In total we use 242 images from MMI.

CK+ dataset [39] consists of 593 image sequences from 123 subjects, 327 sequences of which have emotional labels. The dataset labels seven expressions including angry, disgust, fear, happy, sad, surprise, and contempt. We only choose the peak frame from the sequences labelled with the first six expressions. In addition, we choose the first frame from some of the sequences as neutral samples. In total we use 363 images from CK+.

In this experiment, the network structure of our method is shown in Figure 2. Since the facial expression datasets are rather small, to avoid over-fitting, we use the Alexnet model pre-trained on ImageNet as the base model of the classifier and fine-tune it in our experiment.

Table 1. Recognition accuracy evaluation for domain adaptation on facial expression datasets. JAFFE(J), MMI(M), CK+(C). J→M means J is the source and M is the target. Bold numbers are the best results.

Training Method	Decoupled Layers			J→M	M→J	C→M	M→C	J→C	C→J
	$\mathbf{ENC}$	$\mathbf{DEC}$	D
Source				0.330	0.362	0.428	0.697	0.634	0.437
UNIT	−	−	−	0.507	0.470	0.567	0.719	0.733	0.526
CGAA	√	√	−	0.521	0.460	0.581	0.736	0.744	0.559
	√	√	√	0.521	0.498	0.581	0.736	0.769	0.573

shows experiment results of the classifier’s accuracy tested on the target domain. In Table 1, the source model is trained with only the labeled source dataset. As for the adapted model, to evaluate the effectiveness of our proposed method, we train with three different settings. In all three settings, the parameters of the low-level layers in encoders, decoders and discriminators are not shared, which are the first layer of the encoder, the last layer of the decoder and the first layer of the discriminator. As for the high-level layers, the first experiment shares the weights of all of these layers, which is a similar structure to UNIT described in [19]. The second experiment has decoupled high-level layers in $ENC$ and $DEC$, which means we do not share the last layer of the two encoders and the first layer of the two decoders. The last one is to have decoupled high-level layers in $ENC$, $DEC$ as well as D, which is the same setting described in Figure 2. A decoupled D means we do not share the last two layers in the discriminators. Figure 4 shows examples of the style-transferred images generated by UNIT and the last experiment setting of CGAA.

In each domain adaptation, we use all available source examples and target examples and resize them to 224 × 224 pixels to train the generative adversarial networks. Only the label information of the source dataset is used to train the classifier. Optimization is done on Pytorch using Adam with the learning rate set as 0.0001 and the weight decay as 0.0001. The $\alpha$, $\beta$ and $\gamma$ in Equation (8) are set as 1.0, 1.0 and 0.1. Figure 4 shows some of the style-transferred images generated in the C→J domain adaptation under our last kind of network setting. The experiment results in Table 1 show that our CGAA model with partially-decoupled high level layers outperforms the model with all the high level layers tied-up in all six domain adaptations. In addition, we find that decoupling the encoder-decoder can lead to a significant increase of recognition accuracy, whereas decoupling the discriminator has only a small impact on the experiment result. Therefore, in other experiments described in this paper, we use the last setting in Table 1 as CGAA for evaluation. We visualize the feature distribution of the two domains before and after the adaptation (J→C), as shown in Figure 5. Figure 5 proves that our model can make the distribution of the features from the two domains much closer, which brings about a higher accuracy in the classification.

To further evaluate the effectiveness of our method, we compare the confusion matrices of the class-wise classification accuracy on target domain before and after adaptation.

As shown in Figure 6, the blue ones are the confusion matrices when only source domain images are used for training, the green ones are matrices when UNIT is used for domain adaption and the red ones are matrices when our method is used for domain adaptation. When trained on source domain only, the model have difficulties separating Angry and Neutral between CK+ and JAFFE (see Figure 6a,b, and also cannot seperate Angry and Sad between MMI and CK+ (see Figure 6c,d). When trained on MMI and tested on JAFFE, the model misclassifies a lot of images as Surprised (see Figure 6e) whereas when trained on JAFFE and tested on MMI, the model misclassifies most of the images as Angry (see Figure 6f). These misclassifications are caused by the semantic gap between domains. Figure 6 shows that our domain adaptation method can help the model to cross the semantic gap between domains and increase the class-wise classification accuracies.

4.2. Office Dataset

In this experiment, we evaluate our method on the Office dataset [41]. This is the most popular benchmark dataset for object recognition in the domain adaptation field. This dataset has 4410 images across 31 classes of everyday objects in three domains: amazon (A), webcam (W), and dslr (D). The amazon contains product pictures with no background from the Amazon website, and images in webcam and dslr contains similar real-world objects with different resolution. Following previous domain adaptation work [26], we use ResNet-50 as the model structure for the classifier. Other sub-parts of the model are the same as those shown in Figure 2. We adopt the common “fully-transductive” training protocol [8,9,26], using all available labeled source examples and unlabeled target examples. Optimization is done on Pytorch using Adam for 10 epochs with the learning rate set as 0.0001 and the weight decay set as 0.0001. The $\alpha$, $\beta$ and $\gamma$ in Equation (8) are 1.0, 1.0 and 0.1. We evaluate our method on all six domain adaptations and compare our method with other reported domain adaptation approaches (some of them only have experiment results on three domain adaptations). We also implement two other methods on Pytorch based on share-latent space assumption, which are CoGAN and UNIT. Note that in the original papers of these two methods, the classifier is gained by attaching a softmax layer to the last hidden layer of the discriminator. Whereas in our implementation of these methods, we train a stand-alone classifier with the same structure of our method for a fair comparison. The experiment results in

Table 2. Recognition accuracy evaluation for domain adaptation on the Office dataset. Amazon (A), webcam (W), dslr (D). A→W indicates A is the source dataset and W is the target dataset. Bold numbers are the best results.

Training Method	A→W	W→A	W→D	D→W	A→D	D→A
Source	0.670	0.498	0.952	0.941	0.689	0.515
DDC	0.594	−	0.917	0.925	−	−
DAN	0.685	0.531	0.990	0.960	0.670	0.540
DAH	0.683	0.530	0.988	0.961	0.665	0.555
DRCN	0.687	0.549	0.990	0.964	0.668	0.560
DANN	0.730	−	0.992	0.964	−	−
RTN	0.733	0.510	0.996	0.968	0.710	0.505
ADDA	0.751	−	0.996	0.970	−	−
CoGAN	0.745	0.549	0.996	0.968	0.710	0.560
UNIT	0.751	0.566	0.992	0.968	0.715	0.568
CGAA	0.752	0.575	0.996	0.957	0.723	0.572

show that our method is a competitive method and achieve state-of-the-art results compared with previously-reported methods, except for D→W. The method proposed in this paper aims to solve the problem of domain adaptation when the high-level features in the two domains are different and the shared high-level latent space cannot be established. As shown in Figure 7, the images of webcam (W) and dslr (D) are actually very similar, only having differences in the illumination and the image resolution. In other words, their high-level features are the same. Therefore, our method did not achieve better results than other methods in this particular task, but obtained better results in other more challenging tasks with obvious high-level feature differences, such as W→A and D→A.

4.3. Office-Home Dataset

Finally, we test our model on Office-home dataset [13]. This is a newer, larger and more challenging dataset compared to the classic Office dataset. It has about 15,500 images cross 4 domains, with each domain containing images from 65 classes of everyday objects. As shown in Figure 8, the four domains are: Art, Clipart, Product and Real-world. Images in Art are artistic depictions of objects and Clipart contains clipart images. Product consists of images of objects without background and Real-World consists of images of objects captured with a camera. We conduct this experiment with the same setting as the classic Office dataset experiment.

Table 3. Recognition accuracy evaluation for domain adaptation on the office-home dataset. Art (A), Clipart (C), Product (P), Real-World (R). A→C indicates A is the source dataset and C is the target dataset. Bold numbers are the best results.

Method	A→C	A→P	A→R	C→A	C→P	C→R	P→A	P→C	P→R	R→A	R→C	R→P
DAN 1	0.307	0.422	0.541	0.328	0.476	0.498	0.291	0.341	0.567	0.436	0.383	0.627
DANN 1	0.333	0.430	0.544	0.322	0.491	0.498	0.305	0.381	0.568	0.447	0.427	0.647
DAH	0.316	0.408	0.517	0.347	0.519	0.528	0.299	0.396	0.607	0.450	0.451	0.625
CoGAN	0.399	0.545	0.672	0.471	0.570	0.579	0.478	0.406	0.635	0.580	0.489	0.728
UNIT	0.404	0.554	0.670	0.480	0.572	0.583	0.509	0.412	0.658	0.599	0.503	0.726
CGAA	0.434	0.571	0.676	0.499	0.577	0.591	0.517	0.435	0.662	0.612	0.517	0.749

¹ Results reproduced from [13].

shows that our results outperform competitors in all of the domain adaptations in this experiment.

5. Conclusions

In this paper, we proposed an unsupervised domain adaptation method called coupled generative adversarial autoencoders. The weight-sharing training strategy proposed in this paper extends the shared high-level latent space assumption and improves the tolerance of the model to the differences in high-level semantic features between domains. Under this training strategy, our model can generate style-transferred images with unpaired images in the two domains and domain adaptation is done by training a classifier with the target-style images generated from the source images. With this proposed method, we achieve state-of-the-art experiment results on various domain adaptation scenarios including popular benchmark datasets.

References