Research on Named Entity Recognition for Spoken Language Understanding Using Adversarial Transfer Learning

Abstract

In this paper, we propose an adversarial transfer learning method to solve the lack of data resources for named entity recognition (NER) tasks in spoken language understanding. In the framework, we use bi-directional long short-term memory with self-attention and conditional random field (BiLSTM-Attention-CRF) model which combines character and word information as the baseline model to train source domain and target domain corpus jointly. Shared features between domains are extracted by a shared feature extractor. This paper uses two different sharing patterns simultaneously: full sharing mode and private sharing mode. On this basis, an adversarial discriminator is added to the shared feature extractor to simulate generative adversarial networks (GAN) and eliminate domain-dependent features. This paper compares ordinary adversarial discriminator (OAD) and generalized resource-adversarial discriminator (GRAD) through experiments. The experimental results show that the transfer effect of GRAD is better than other methods. The F1 score reaches 92.99% at the highest, with a relative increase of 12.89%. It can effectively improve the performance of NER tasks in resource shortage fields and solve the problem of negative transfer.

1. Introduction

Spoken language understanding, as the leading task of dialogue system, aims to transform users’ spoken language into structured information. It can be used to provide a guarantee for downstream tasks of dialogue system. Spoken language understanding consists of three key subtasks, which are domain determination (DD), intent determination (ID) and NER [1]. Among them, NER identifies specific entities in users’ oral expression, such as name, address, organization, institution, etc., to help the machine understand spoken language. Initially, NER mainly used rule-based and statistical machine learning methods [2,3,4]. Because deep learning has strong modeling ability and can automatically mine regular knowledge contained in data, many researchers began to apply it to the task of NER [5,6,7] and showed excellent performance.

In fact, deep learning methods rely on large-scale and high-quality labeled data, while task-based dialogue systems usually only have access to a small amount of data in the development stage. In addition, the text data obtained is generally short, such as abbreviation, colloquial, and semantically ambiguous. For example, article comments on social media such as Twitter or Weibo write “no idea” as “no eye deer” and “xswl” means “smile to death”. Therefore, it is difficult to understand such short texts through their literal meaning, and hard to collect enough labeled training data. Adopting deep learning methods in this field of NER may lead to poor model generalization and unstable performance. Transfer learning focuses on knowledge transfer across domains, and aims to improve the performance of target learners on the target domain by transferring knowledge contained in different but related source domains, to reduce the dependence of constructing target learners on a large amount of target domain data. It is a promising machine learning method to solve the above problems [8]. This paper applies transfer learning to the NER task of spoken language understanding. The main work and contributions of this paper can be summarized as follows:

(1)

For the first time, we applied transfer learning to the task of spoken language understanding. Aiming at the shortage of labeled training data for the task of NER in spoken language understanding, the source domain and target domain data are trained jointly, and shared features of different domains are extracted by sharing feature extractor. Building on the reference [9], we extend to two sharing modes: full sharing mode and private sharing mode.

(2)

Referring to the work [10], a NER method using adversarial transfer learning is proposed to solve the problem that unique features of the source domain in shared features have a negative impact on the target task. Therefore, an adversarial discriminator is added to the shared feature extractor. Two kinds of adversarial discriminators, OAD and GRAD, are investigated. We combine the two sharing modes to form a total of four combinations. The experimental results show that the method of adversarial transfer learning based on GRAD and private sharing mode can effectively improve the system performance of NER tasks in spoken language understanding.

2. Related Work

This paper mainly studies the application of transfer learning to the task of NER in spoken language understanding. The related content includes traditional NER methods and NER methods using transfer learning.

Traditional NER methods include the hidden markov model (HMM) [11], maximum entropy model (MEM) [12], support vector machine (SVM) [13], conditional random fields (CRF) [14], etc. In recent years, NER tasks have generally adopted deep learning models based on multi-layer neural networks. Since LSTM can alleviate long-term dependency problems, some researchers have integrated it with CRF and achieved advanced recognition accuracy on NER tasks. For example, Habibi [15] used LSTM-CRF for NER tasks in the biomedical field, and used generated character embeddings and word embeddings concatenation as input of the model. On this basis, WANG [16] proposed to use the BiLSTM-CRF model for entity recognition in the field of traditional Chinese medicine. The BiLSTM-CRF model has also become one of the common methods for NER tasks. Zeng [17] incorporated a self-attention mechanism into the BiLSTM-CRF model to capture global dependencies of the entire sentence and learn the internal structural features of the sentence in NER of electronic medical records, and obtained better results. Although the above methods achieve high recognition accuracy, these methods use supervised training methods and require a large amount of labeled data to train the model. For some specific fields, it is very difficult to collect enough data with manual annotation, such as military, small languages and other fields.

To solve the scarcity of labeled data resources, some scholars have used transfer learning for NER tasks [18]. Traditional methods are mainly divided into two types: data-based method and model-based method. Data-based methods are mainly designed to reduce the distribution difference between samples in source and target domains but are only used for cross-language transferring [18]. Model-based methods are mainly use chosen features or trained model parameters in the source domain to improve the task performance in the target domain. To improve the performance of various tasks including NER, Ando and Zhang [19] proposed a transfer learning model that can share structural parameters among multiple tasks. To solve the classification problem, Tommasi [20] proposed a single-model knowledge transfer (SMKL) method, which is based on least squares SVM to select a pre-acquired binary decision function from the source domain, and then migrate its parameters to the target domain model.

With the development of deep learning, many researchers use it to build transfer learning models. Some methods aim to reuse some pre-trained deep neural networks of the source domain and transfer them to the target domain model. Wang [21] performed a feature to transfer through label-aware maximum mean discrepancy (MMD) and realized a cross-medical professional NER system. Yang [22] proposed a sequence labeling transfer learning framework based on RNN (RNN-TL) to improve system performance by sharing feature representations and model parameters. Although these methods have obtained some achievements, there are differences between different domain resources. Forcing source and target domains to share the same features and model parameters may lead to poor model generalization. Inspired by GAN [23], researchers introduce adversarial techniques, which aim to remove some domain-dependent features to alleviate the negative transfer problem. Cao [9] applied adversarial transfer learning to the NER task for the first time. They made full use of the richer word boundary information of Chinese word segmentation (CWS) and filtered the unique information of CWS through task discriminator and adversarial loss function to improve the performance of Chinese NER. Zhou [10] proposed a dual adversarial transfer network (DATNet). They introduced GRAD and adversarial training based on a general deep transfer unit, which plays a role in solving the difference and imbalance of domain resources and improving the generalization performance of the model. Based on the above methods, this paper applies adversarial transfer learning to spoken language understanding tasks and explores a transfer learning method that can solve the problems of scarcity of labeled data resources and irregular data representation in NER tasks in this field.

3. Method

The main research of this paper is to use adversarial transfer learning to solve the problem of data scarcity for NER tasks in spoken language understanding. We achieve knowledge transfer by extracting word-level shared features between domains. Figure 1 shows two word-level feature sharing modes used in this paper, which are full sharing mode (MTL-F) and private sharing mode (MTL-P). Full sharing mode is shown in Figure 1a, which is mainly composed of five parts, embedding layer, shared word-level feature extractor, self-attention mechanism, adversarial discriminator, and label decoder CRF. The model first inputs text into the embedding layer of the source domain and target domain to obtain a hybrid representation of word embeddings and character-level feature representations. Then the model uses the BiLSTM framework as a word-level feature extractor to extract contextual word features. For full sharing mode, the two domains are jointly trained for learning task-sharing features. Using attention mechanisms to capture semantic and grammatical connections between different words in sentences. Considering the difference between the source domain and target domain data and the unique features of the source domain, we add an adversarial discriminator to the shared feature extractor. Finally, the results are output through the CRF layer of the tag decoder in their respective fields. The private sharing mode is shown in Figure 1b. To consider different resource representations of word features, it adds a private word-level feature extractor for each field. The private word-level feature extractor is trained independently for each field to extract specific characteristics of the task. In the following sections, we will state each part of this model in detail.

3.1. Embedding Layer

To inject a richer semantic representation into the pre-training of the model, this study uses a hybrid representation in the distributed representation, including word embeddings and character-level feature representations. Character-level feature representations are obtained from character embeddings by CNN, which is a character-level feature extractor [7]. The process is shown in Figure 2. Given a Chinese sentence $s=\{w_1,w_2,\mathrm{\dots },w_n\}$, where $n$ is the number of its words, each word is composed of several characters, $w_i=\{c_1,\mathrm{\dots },c_p\}$. We put the word $w_i$ into character-level CNN to extract its character-level feature representation $x_i^{char}$, then concatenate character-level feature representation and word embedding of each word to obtain a hybrid representation $x_i=[x_i^{char},e_i^{word}]$, and finally input them into word-level feature extractor. Both word embeddings and character embeddings are initialized using word embedding table and character embedding table pre-trained by the Word2Vec model.

3.2. Word-Level Feature Extractor

In this paper, we use BiLSTM as a word-level feature extractor to make full use of past and future information. The hidden layer vector of BiLSTM is denoted as Equations (1)–(3), where x_i is the output in Section 3.1, $\overrightarrow{h_i}$ and $\overleftarrow{h_i}$ are hidden vectors of forward and backward LSTM at the position $i$, and then $\overrightarrow{h_i}$ and $\overleftarrow{h_i}$ are concatenated to obtain hidden layer vector $h_i$ of BiLSTM.

$$\overrightarrow{h_i}=\overrightarrow{\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}}({\overrightarrow{h}}_{i-1},x_i)$$

(1)

$$\overleftarrow{h_i}=\overleftarrow{\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}}({\overleftarrow{h}}_{i+1},x_i)$$

(2)

$$h_i=[\overrightarrow{h_i},\overleftarrow{h_i}]$$

(3)

For the two shared modes used in this paper, we introduce two different transferable word-level feature extractors, namely shared word-level feature extractor and private word-level feature extractor. The shared feature extractor is jointly trained with source and target domains data to learn shared features of the task, while the private word-level feature extractor is trained independently for each domain to extract task-specific features. In full sharing mode, there is only one shared word-level feature extractor, while in private sharing mode, one shared word-level feature extractor and one private word-level feature extractor are assigned to source and target tasks. For any sentence in the dataset of task $k\in${source domain, target domain}, hidden states $f_i^k$ and $p_i^k$ of shared word-level feature extractor and private word-level feature extractor are denoted as Equations (4) and (5):

$$f_i^k=\mathrm{B}\mathrm{i}\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}(x_i^k,f_{i-1}^k;{\theta }_f)$$

(4)

$$p_i^k=\mathrm{BiLSTM}(x_i^k,p_{i-1}^k;{\theta }_p)$$

(5)

where ${\theta }_f$ and ${\theta }_p$ denote shared BiLSTM parameters and private BiLSTM parameters, respectively.

3.3. Self-Attention

Using an attention mechanism can fully consider semantic and grammatical connections between different words in sentences. This paper adopts a multi-head self-attention mechanism, and the calculation is shown in Equation (6):

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{Q{K}^T}{\sqrt{d}})V$$

(6)

where $Q$, $K$, and $V$ are query matrix, key matrix and value matrix. We set $Q=K=V=H$. Multi-head self-attention mechanism first projects a query matrix, key matrix, and value matrix for $h$ times by using different linear projections. Then, the $h$ projections are executed in parallel. Finally, the $h$ self-attentions are concatenated and re-projected to obtain the final result. Multi-head attention can be denoted as Equations (7) and (8):

$${\mathrm{head}}_i=\mathrm{Attention}(Q{W}_i^Q,K{W}_i^K,V{W}_i^V)$$

(7)

$$H^{\prime }=({\mathrm{head}}_i\oplus \mathrm{\dots }\oplus {\mathrm{head}}_h)W_o$$

(8)

where $W_i^Q$, $W_i^K$, and $W_i^V$ are projection parameters and $W_o$ is a trainable parameter. For the sentences in the dataset of task $k$, the shared self-attention output vector ${F^{\prime }}^k$ and private self-attention output vector ${P^{\prime }}^k$ can be obtained by Equations (6)–(8). For full sharing mode, the shared self-attention output vector ${F^{\prime }}^k$ is input to the CRF layer. The private sharing mode is to connect two self-attention output vectors ${F^{\prime }}^k$ and ${P^{\prime }}^k$ to obtain the input of the CRF layer in their respective fields. The calculation can be denoted as Equation (9).

$${H^{″}}^k=({F^{\prime }}^k\oplus {P^{\prime }}^k)$$

(9)

3.4. Adversarial Discriminator

Inspired by Cao [9] and Zhou [10], we further add an adversarial discriminator on shared feature extractor by considering the difference between data in source and target domains and the unique features of the source domain. We use a shared feature extractor as a generator and an adversarial discriminator as a discriminator to simulate GAN [23] to ensure that obtained shared features are pure and there is no source domain-specific information. The core part is the adversarial discriminator, which is used to identify the domain of shared features. The features are marked with domain identification before entering the shared feature extractor, and then back-propagate and train the shared feature extractor according to the identification of the source of shared features. This produces shared features that cannot be identified by the adversarial discriminator. Thus, an adversarial network composed of an adversarial discriminator and a shared feature extractor is formed. After multiple rounds of training, a balance is reached. That is, the adversarial discriminator cannot identify which domain shared features come from, and finally obtains pure domain-independent features. This method can alleviate the negative transfer problem. This paper uses two adversarial discriminators, which are OAD and GRAD. The following two subsections will introduce in detail.

3.4.1. Ordinary Adversarial Discriminator

OAD judges the source of a sentence according to the idea of generative adversarial network discriminator. The identification process is shown in Figure 3. OAD identifies the domain of feature through the max-pooling layer and softmax layer, which can be denoted as Equations (10) and (11):

$$S=\mathrm{Maxpooling}({F^{\prime }}^k)$$

(10)

$$D(S;{\theta }_d)=\mathrm{softmax}(W_{d}S+b_d)$$

(11)

where ${F^{\prime }}^k$ represents the output of shared multi-head self-attention in Section 3.3, ${\theta }_d$ represents parameters of the adversarial discriminator, and $W_d$ and $b_d$ are trainable parameters. During training, the function of the gradient descent algorithm is to make the discriminative ability of the adversarial discriminator stronger. Next, we use the gradient reversal layer to make it reverse, so that the adversarial discrimination ability of the discriminator becomes worse. Eventually the adversarial discriminator will not be able to identify the domain of shared features.

We introduce an adversarial loss function $l_{AD}$ to prevent the unique information in the source domain from entering shared space, as shown in Equation (12): $l_{AD}$ train a shared feature extractor to generate shared features such that the adversarial discriminator cannot judge the domain of features reliably.

$$l_{AD}=\underset{{\theta }_f}{\min }(\underset{{\theta }_d}{\max }{\displaystyle \sum _{k=1}^K{\displaystyle \sum _{i=1}^{T_k}\log D(E_s(x_k^{(i)}))}})$$

(12)

where ${\theta }_f$ is the trainable parameter of the shared feature extractor, $E_s$ represents shared feature extractor, $T_k$ is the number of training instances for task $k$, and $x_k^{(i)}$ is the ith instance of the task $k$. We complete optimization by adding a gradient inversion layer before the softmax layer, and train model parameters to make the discriminative ability of the adversarial discriminator weaker. When the shared feature extractor and adversarial discriminator reach a balance point, the adversarial discriminator cannot discriminate the domain of the shared feature.

3.4.2. Generalized Resource Adversarial Discriminator

Although OAD can remove the impurities in shared features, it does not consider the problem of data scale imbalance between the source domain and target domain. If this imbalance is not considered during training, stochastic gradient descent optimization will make the model more biased toward source domain [24]. This paper uses GRAD [10] to solve this problem. GRAD is shown in Figure 4.

GRAD also uses a gradient inversion layer to reduce its judgment ability. Shared feature representations extracted in source and target domains are made more compatible, GRAD cannot distinguish the source of features, and the output of the shared feature extractor is domain-independent. To compute the loss function of GRAD, the output sequence of the shared feature extractor is first encoded into a single vector via a self-attention mechanism [25], and then projected onto a scalar $r$ through a linear transformation, as shown in Equation (13).

$$l_{\mathrm{GARD}}=-{\displaystyle \sum _i\{I_{i\in D_S}\alpha {(1-r_i)}^{\gamma }\log r_i+I_{i\in D_T}(1-\alpha )r_i^{\gamma }\log (1-r_i)\}}$$

(13)

The weight $\alpha$ is used to balance the influence of the large difference in training scale between high and low resources, and provide adaptive weights for each sample, so that the focus of model training is on hard samples. $I_{i\in D_S}$ indicates that feature comes from the source domain, and $I_{i\in D_T}$ indicates that the feature comes from the target domain, both of which are identification functions. The parameter $\gamma$ measures the comparison of the loss contribution of hard and easy samples, and ${(1-r_i)}^{\gamma }$ (or $r_i^{\gamma }$) controls the loss contribution of each sample by measuring the difference between the predicted value and true label. Weights $\alpha$ and ${(1-r_i)}^{\gamma }$ (or $r_i^{\gamma }$) reduce the loss contribution of high-resource samples and simple samples, respectively.

3.5. Label Decoder

This paper uses a linear chain CRF [13] as the label decoder. Given a Chinese sentence $s=\{w_1,w_2,\mathrm{\dots },w_n\}$ and its corresponding sequence of predicted labels $y=\{y_1,y_2,\mathrm{\dots },y_n\}$ for each word, linear chain CRF can be written as Equation (14):

$$p(\tilde{y}|{H^{\prime }}_{1:n})=\frac{1}{Z(h_{1:n})}\exp \{{\displaystyle \sum _{t=2}^n{\theta }_{y_{\mathrm{t}-1},y_t}+{\displaystyle \sum _{t=1}^n{W}_{y_t}{H^{\prime }}_t}}\}$$

(14)

where $H^{\prime }$ represents the output of the attention mechanism and $\tilde{y}$ is the sequence of predicted labels, $\tilde{y}=y_{1:n}$. $\theta$ represents transition distribution between $t-1$ and $t$. $Z(h_{1:n})$ is the normalization term. The calculation process is as Equation (15); $t_k$ and $s_l$ represent the transition feature function and state feature function, respectively, $t_k$ is determined by the previously hidden node $y_{i-1}$ and currently hidden node $y_i$, and $s_l$ is determined by currently hidden node $y_i$; ${\lambda }_k$ and ${\mu }_l$ represent corresponding weights of two feature functions respectively.

$$Z(x)={\displaystyle \sum _y\exp ({\displaystyle \sum _{i,k}{\lambda }_k{t}_k(y_{i-1},y_i,x,i)+{\displaystyle \sum _{i,l}{\mu }_l{s}_l(y_i,x,i)}})}$$

(15)

The model parameters are optimized using the negative log-likelihood function [25] as the objective function of the model. The loss function $l$ is defined as shown in Equation (16):

$$l=-{\displaystyle \sum \log p(\tilde{y}|{H^{\prime }}_{1:n})}$$

(16)

Since different domains have different types of labels, the two shared modes add a specific CRF layer for each domain. Define source domain loss function $l_S$ and target domain loss function $l_T$, as shown in Equations (17) and (18).

$$l_S=-{\displaystyle {\sum }_i\log p(\tilde{y}|{H^{″}}_{1:n}^S)}$$

(17)

$$l_T=-{\displaystyle {\sum }_i\log p(\tilde{y}|{H^{″}}_{1:n}^T)}$$

(18)

3.6. Training

Our model can be trained by end-to-end mode, which uses the backpropagation algorithm by minimizing loss. For OAD, the loss function of the model is shown in Equation (19):

$$l=l_S\cdot I(i)+l_T\cdot (1-I(i))+\lambda l_{AD}$$

(19)

where $\lambda$ is a hyperparameter and $I(i)$ is a domain discriminant function, as shown in Equation (20), used to identify whether a sentence comes from the source domain or target domain. In the formula, $D_S$ and $D_T$ are source domain and target domain datasets, respectively.

$$I(i)=\{\begin{array}{ccc}1,& if& i\in D_T\\ 0,& if& i\in D_S\end{array}$$

(20)

For GRAD, the loss function of the model is shown in Equation (21).

$$l=l_{\mathrm{GARD}}+l_S+l_T$$

(21)

4. Experiments

In this section, we first introduce the dataset and preprocessing scheme used in the experiment, then give the parameter settings for model training and performance evaluation indicators, and finally compare and analyze the experimental results.

4.1. Data Set

This experiment uses CLUENER2020 Chinese Fine Grained NER Dataset [26] and flight information domain dataset [27] as source domain datasets. CLUENER is a dataset for NER tasks in the news domain with fine-grained entity classification. The flight information domain dataset is the data inquiring about flight information in the dialogue system, which is part of short text data. The dataset includes 19 entity types, which have more fine-grained entity types than the CLUENER dataset. Both datasets are annotated with BIO. BIO labeling is a kind of joint labeling. Specifically, B, I, and O represent Begin, Inner, and Other, respectively. BIO annotation is to label each element as “B-X”, “I-X” or “O”. Among them, “B-X” indicates that the segment where this element is located belongs to type X and this element is at the beginning of this segment, “I-X” indicates that the segment where this element is located belongs to type X and this element is in the middle of this segment, “O” indicates that it is not of any type. Details are shown in

Table 1. Source domain dataset.

Dataset	Sentences	Entities	Classes	Entity Types
CLUENER	12,091	25,244	10	Address, Book, Company, Game, Name, Government, Movie, Scene, Organization, Position
Flight Information	5871	16,575	19	Departure, Destination, Time, Airport Name, Fare Range, Airline, Flight Number, Seat Classes, etc.

.

Our work uses SMP2020-ECDT Chinese Human-Machine Dialogue Technology Evaluation Dataset [28] as the target domain dataset, which is also part of the short text dataset. The dataset has a total of 5024 sentences, involving 45 domains, with 81 named entity types and 8787 entities. In addition, we select four sub-domains with appropriate data volume from the SMP2020-ECDT dataset as target domain datasets, which are recipes, music, news, and trains, to verify the impact of different domains on the transfer effect. The amount of data in other fields is too small, and overfitting is prone to occur during the experiment. The information on the selected four domain datasets is shown in

Table 2. SMP2020 subdomain dataset.

Domain	Sentences	Entities	Classes	Entity Types
Cookbook	438	450	4	dishName, utensil, ingredient, keyword
Music	189	195	3	song, artist, category
News	197	254	8	datetime_time, datetime_date, category, country, province, city, area, keyword
Train	171	366	8	startLoc_province, startLoc_city, startLoc_area, endLoc_province, endLoc_city, endLoc_area, startDate_date, category

.

We use the hold-out method to divide the dataset into training and testing sets with a ratio of 7:3. We first clean the dataset and then use Jieba tools for word segmentation. To prevent specific nouns from being incorrectly segmented, all entity words are stored in the user dictionary in advance. The data is labeled according to the BIO labeling method. Finally, we input data into the model.

4.2. Parameter Settings

In this experiment, the dimension of character embeddings and word embeddings is set to 300, the batch size is 16, the learning rate is 0.001, the learning decay rate is 0.7, the dropout is 0.5, and adam optimizer is used as the gradient descent algorithm. The dimensions of LSTM hidden layers of baseline model and full sharing mode are set to 200, and the dimensions of LSTM hidden layers of source, shared and target domains are set to 100 in private sharing mode, respectively. Hyperparameter $\lambda$ of OAD model is 0.06, and hyperparameters $\alpha$ and $\gamma$ of GRAD model are 0.25 and 2.

4.3. Evaluation Metrics

This experiment uses F1 score as the evaluation metric, which considers precision and recall, and the calculation is shown in Equations (22)–(24):

$$\mathrm{Precision}=\frac{TP}{TP+FP}$$

(22)

$$\mathrm{Recall}=\frac{TP}{TP+FN}$$

(23)

$$\mathrm{F}1-\mathrm{score}=2\times \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$

(24)

where TP (true positive) means the number of correct entities identified. FP (false positive) means the number of wrong entities identified. FN (false negative) means the number of unidentified entities. For NER tasks, the precision rate represents the ratio of the number of correctly labeled entities in a given dataset to the total number of labeled entities. The recall rate represents the ratio of the number of correctly labeled entities in a given dataset to the total number of entities in the dataset.

4.4. Results and Analysis

In this paper, the BiLSTM-Attention-CRF baseline model fused with character and word information is first tested on five target domains, and the results are used as the baseline for comparison experiments to detect the transfer effect of the model. Then, private sharing mode (MTL-P) and full sharing mode (MTL-F) are used for transfer learning. Knowledge on the two source domains is transferred to five target domains. The results are later compared with the ones of the model used to detect the impact of adversarial discriminators on transfer learning. Finally, we add an OAD and a GRAD to the shared feature extractor, and the final comparison results of the experimental F1 score are shown in

Table 3. Experimental results of NER for spoken language understanding using transfer learning (F1-score/%).

Source Domain	Model	SMP2020	Cookbook	Music	News	Train
	baseline	86.75	83.87	74.29	88.46	87.50
CLUENER	MTL-P	87.58	83.52	78.95	88.89	87.94
	MTL-F	88.18	84.44	75.68	87.72	87.32
	OAD-P	87.96	87.64	81.08	90.57	89.04
	OAD-F	88.39	86.36	78.05	91.55	88.73
	GRAD-P	92.59	88.17	86.42	92.57	90.78
	GRAD-F	91.20	86.96	83.33	91.76	89.38
Flight Information	MTL-P	88.72	85.39	73.17	89.29	88.73
	MTL-F	87.76	84.09	73.68	90.91	88.59
	OAD-P	88.12	89.13	82.67	92.59	89.21
	OAD-F	88.27	89.89	82.93	90.20	89.51
	GRAD-P	92.99	91.40	87.18	92.86	90.14
	GRAD-F	90.22	88.89	84.21	92.31	92.31

.

Due to a large number of named entities in the SMP2020 dataset, there are situations in which the same entity type is used in different fields, the same word has different entity types in different fields, and the number of sentences in the other four subfields is too small. However, the baseline model adopts a supervised training method, which cannot be well converged due to the influence of the data, which will result in a low F1 score. After adding a shared feature extractor, the F1 score obtained by the model transfer from the flight information domain dataset, as the source domain is generally higher than that of CLUENER. The model transfer effect of using MTL-P mode is generally better than that of MTL-F mode, probably because private sharing mode has more private features than full sharing mode, which provides a richer feature representation for the target domain. It can be seen that the MTL-P mode is more suitable for transfer learning across domains. Although the performance of the model is improved after adding a shared feature extractor, the improvement effect is limited, and even the problem of negative transfer occurs. For example, when the flight information domain is used as the source domain, the F1 score transferred to the music target domain is not as high as the baseline. It may be that the source domain and target domain data are quite different, and there are a large number of features unique to the source domain in the shared feature extractor which lead to negative effects.

Inspired by GAN, this experiment adds an adversarial discriminator to the shared feature extractor to ensure that shared features do not have source-specific features. The two modes of private sharing and full sharing continue to be used, and OAD and GRAD are added. They are represented by OAD-P, OAD-F, GRAD-P, and GRAD-F, respectively. It can be seen from Table 3 that, consistent with the previous analysis, the private sharing mode still performs better. In addition, the F1 score of OAD in all target domains exceeds the baseline, which proves that adversarial discriminator can alleviate negative transfer problems. The transfer effect of using GRAD is the best, and the F1 score exceeds that of the baseline model, NER model using model transfer, and OAD. It can alleviate the data scale imbalance between the source domain and the target domain.

From the perspective of transfer effect, the music domain has the best effect, and the F1 score is increased by 12.89%, but the F1 score of the baseline of the music domain is the lowest at 74.29%. The smallest improvement was in the news domain, where the F1 score improved by 4.4%, while the F1 score of baseline was the highest at 88.46%. The highest F1 score is obtained from the SMP2020 dataset. To sum up, it can be seen that the domain with a lower baseline has a better transfer effect, that is, a domain with fewer data and less labeled data has a better transfer effect. This proves our method can solve the problem that resource shortage of NER tasks in spoken language understanding.

5. Conclusions

This paper proposes to use an adversarial transfer learning method to solve the lack of resources for NER tasks in spoken language understanding. It adds a shared feature extractor and adversarial discriminator to the BiLSTM-Attention-CRF model that combines character and word information. We use the shared feature extractor as a generator and an adversarial discriminator as a discriminator to simulate GAN. This can eliminate the feature of the source domain in shared features and improve the purity of shared features. The experimental results show that the method in this paper has a better transfer effect in a domain where the baseline F1 score is lower. The transfer effect is better in fields with fewer data and fewer labeled data. This method can effectively improve the performance of NER tasks in resource shortage fields.

Although this research has made some progress, the negative transfer is still the biggest obstacle to transfer learning. We can alleviate this problem by looking for correlation metrics between source and target domains or introducing adversarial transfer learning, but these methods have their limitations. Regarding this latter point, more research is still in process. Concurrently, the NER task only considers one subtask in spoken language understanding. This field also includes two tasks: intent determination and domain determination. These tasks are related; considering a single task may provide limited improvement in model performance. In the future, we will consider combining multiple tasks for transfer learning to improve the performance of NER tasks in resource shortage fields.