Learning Class-Transductive Intent Representations for Zero-shot Intent Detection

1 Introduction

In recent years, smart devices with built-in personal assistants like Google Assistant and Siri are becoming omnipresent. Behind these systems, a key question is how to identify the underlying intent of a user utterance, which has triggered a large amount of work on intent detection [Ravuri and Stolcke, 2015; Liu and Lane, 2016; Nam et al., 2016]. Most existing intent detection systems are built on models trained on annotated data. However, as user demands and the functions of smart devices continue to grow, collecting supervised data for every new intent becomes labor-intensive.

To address this issue, some studies tackle intent detection in the zero-shot learning (ZSL) manner, attempting to utilize the learned knowledge of seen classes to help detect unseen classes. Recent methods of ZSID can be roughly divided into two categories: The first category [Xia et al., 2018; Liu et al., 2019], referred to as the transformation-based methods, utilizes word embeddings of label names to establish a similarity matrix, which is then used to transfer the prediction space of seen intents to unseen intents. Another line is the compatibility-based methods [Chen et al., 2016; Kumar et al., 2017], which aims to encode the label names and utterances into the same semantic space and then calculate their similarity. However, in both kinds of methods, most existing ZSID methods are inductive, which do not consider any information about the unseen classes in the training stage. Consequently, the representations of unseen intents cannot be learned, resulting in two limitations.

∗ Peng Fu and Zheng Lin are the corresponding authors.

1 The code, datasets and Appendix are available at https://github. com/PhoebusSi/CTIR

First, the ZSID methods are not good at modeling the relationship between seen and unseen intents, especially when the label names are given in the form of raw phrases or sentences. For the transformation-based methods, word embeddings of label names are inadequate to associate the connections between seen and unseen intents. For example, "BookRestaurant" is similar to "RateBook" when measured by word embeddings. However, the meaning of these two intents are not that relevant. For the compatibility-based methods, since the unseen intents are not included in learning the semantic space shared by utterance and label names, it cannot effectively detect unseen intents during the test stage, especially when the expressions of utterances are diverse.

Second, the vanilla ZSL methods are not applicable to generalized zero-shot intent detection (GZSID), where the models (at test time) are presented with not only unseen class utterances but also seen class utterances. In GZSID, existing ZSL models usually suffer from the domain shift [Fu et al., 2015a] problem, in which utterances from unseen intents are almost always mistakenly classified into seen intents.

The two limitations are caused by inadequate learning of unseen intents. Naturally, the label name provides a proper sketch of the intent meaning. Existing models use it during the test stage. In contrast, we introduce the class-transductive [Wang et al., 2019; Xian et al., 2019] setting into ZSID, which uses semantic information about the unseen classes (e.g., the label names) for model training. Specifically, we include the unseen intents into the prediction space during training, with the label names serving as the pseudo utterances. This allows the model to learn a rough boundary of each seen and unseen class in the semantic space. Under this framework, we introduce an assistant task that forces the model to find the distinction between seen and unseen intents, thereby alleviating the domain-shift problem. On this basis, we refine the word embedding based similarity matrix by averaging the representations of all corresponding (seen intent) utterances and (unseen intent) label names. As a result, we can better capture the intent meanings and the similarity matrix reflects more accurate intent connections. In summary, our contribution is three-fold:

• In response to the limitations of existing ZSID systems, we propose a class-transductive framework that makes use of unseen label names in the training stage.
• Under the framework, we present a multi-task learning objective to find the inter-intent distinctions and a similarity scorer to associate the inter-intent connections.
• Empirical results on ZSID and GZSID in two benchmarks show that CTIR can bring improvement to a wide range of ZSID systems with different zero-shot learning strategies and model architectures.

2 ProblemFormulation

Zero-shot Intent Detection. In ZSID, the model is trained on the annotated dataset {(x, y)}, where y ∈ Yseen = {ys 1 , ys 2 , . . . , ys I } is the intent label and x is the utterance. At test time, the goal is to identify the intent of an utterance, which belongs to one of the J unseen intents Yunseen = {yu 1 , yu 2 , . . . , yu J }, where Yseen ∩ Yunseen = ∅.

Generalized Zero-shot Intent Detection. In GZSID, the model is presented with utterances from either seen or unseen intents, and the prediction space is Yseen ∪ Yunseen.

Unknown Intent Detection (UID). In UID [Lin and Xu, 2019], the training set is the same as ZSID and GZSID. During testing, the model is expected to detect seen intents and decide whether an utterance belongs to the unknown intents. The prediction space is {ys 1 , ys 2 , . . . , ys I , yunseen}, where all the unknown intents are grouped into a single class yunseen. The GZSID can be solved by breaking it into UID and ZSID tasks(as shown in Figure1 lower).

Simplified Unknown Intent Detection (SUID). The prediction space is reduced to {yseen, yunseen} in SUID. In our multi-task learning, SUID serves as an assistant task.

3 CTIR Framework

In this section, we describe how to integrate CTIR into transformation-based, compatibility-based methods and twostage GZSID method respectively. Figure1 gives an overview. The core idea is to expand the prediction space during training to include unseen classes, with the unseen label names serving as pseudo utterances. During test time, the trained model can be applied to both ZSID and GZSID settings.

3.1 Transformation-basedMethods

Feature Extractor. The feature extractor transforms an input text into a sequence of hidden vectors H = (h1, h2, . . . , hT ) ∈ RT×DH , where DH is the hidden dimension and T is the sequence length. Note that the architecture of feature extractor is a free choice in CTIR framework. Specially, on top of the H produced by a Bi-LSTM, CapsNet [Xia et al., 2018] applies a multi-head attention layer. Each attention head represents a unique semantic feature, which gives rise to {m1,m2, . . .mR} ∈ RR×DH , where R is the number of attention heads. We follow this operation when combining CapsNet with CTIR.

Intent Classifier. There are two commonly used classifiers in intent detection: 1) the linear classifier and 2) the capsule networks [Sabour et al., 2017]. The linear classifier predicts the intent probabilities as:

where W ∈ RDH×K is the weight matrix, and K = I + J is the total number of seen and unseen classes.

In capsule networks, the Dynamic Routing algorithm [Sabour et al., 2017] is used to aggregate the low-level features {m1,m2, . . .mR} into higher-level representations:

where ckr is the coefficient that determines how much the rth semantic feature contributes to intent yk. Following sk is the squash function, which gives rise to the activation vectors V = (v1, v2, . . . , vK). The 2-norms of these vectors are then used as the probability of different intent classes.

Multi-task Learning Objective. The assistant task SUID has two class labels, namely yseen and yunseen. In order to compute the probabilities for yseen and yunseen, we sum up the probabilities of all intent classes in the respective categories. For linear classifiers, we compute the sum of the first I dimensions of the vector vtr for yseen and the last J dimensions for yunseen. Then, the linear classifier is trained with cross-entropy loss:

where zk ∈ {0, 1} indicates whether the kth intent is true, and pk is the predicted probability of the kth intent. z′n and Pn are respectively the ground truth and the predicted probability of each class in SUID. α is a down-weighting coefficient.

For capsule networks, we sum up the 2-norms of the activation vectors (v1, v2, . . . , vI) and (vI+1, vI+2, . . . , vK), respectively. Then, the max-margin loss for capsule networks is:

where Tk = 1 when the kth intent is ground-truth, and otherwise Tk = 0. T ′n is defined in the same way for SUID. λ and λ′ are the down-weighting coefficients, m+, m- and m′+, m′- are the margins. In addition, a regularization term is added to Lmargin to encourage the discrepancy among attention heads [Xia et al., 2018].

Similarity Scorer. Similarity Scorer, which measures the connections between intent classes, is a key component for transformation-based methods. Inspired by Chao [2016], we average the representations of all utterances to represent the seen intents. For the unseen intents, we use the representations of label names. The representation of each utterance or label name is obtained by averaging over different time steps or attention heads (for CapsNet). In practice, the representations are computed during the last training epoch, which considers the entire training set (i.e., the parameters of feature extractors are updated in this process).

After the averaging operation, we have the intent representations {g1, g2, . . . gK} ∈ RK×DH , which is used to compute the similarity matrices Lzsl ∈ RK×J for ZSID and Lgzsl ∈ RK×K for GZSID. The similarity between intent k1 and k2 is computed as Lk1k2 = Cosine (gk1 , gk2 ), where gk1, gk2 are the representations of k1 and k2, respectively.

Inference Process. During inference, a test utterance is first encoded into a prediction vector of K dimensions, in the same way as the training process. Based on this vector, we further employ the Similarity Scorer to obtain the final prediction. In terms of the linear classifier, we have:

where L refers to Lzsl or Lgzsl. When it comes to the capsule networks:

where Ljk is the kth entry in the j th row of L. After Dynamic Routing, we can get J activation vectors for ZSID and K activation vectors for GZSID.

3.2 Compatibility-basedMethods

Feature Extractor. Similar to the transformation-based methods, the first step of compatibility-based methods is to encode the utterance or label name into a dense vector. In this paper we study two kinds of compatibility-based methods: Zero-shotDNN [Kumar et al., 2017] and CDSSM [Chen et al., 2016], which extract the text representation with a tanhactivated nonlinear layer and CNN respectively.

Intent Classifier. With the representations of an utterance and all label names, compatibility-based methods compute the cosine similarity between the utterance u and each intent, which results in S = {sim(u, y)|y ∈ Yseen ∪ Yunseen} ∈ RK . Then, u can be classified into a particular intent class according to vtr = Softmax(S).

Multi-task Learning Objective. To perform SUID with the compatibility-based classifier, we sum up the first I and last J positions in S as ∑i=1:I Si and ∑i=I+1:K Si, the result of which is passed to a binary softmax function to obtain the final probabilities for yseen and yunseen. The multi-task learning objective is defined in the same way as Equation 3.

Inference Process. The inference process of compatibilitybased methods is basically the same as the classification process during training. Given an input utterance, we compute its similarity with each candidate intent in the learned representation space, and classify it to the most similar intent.

3.3 Two-stage GZSIDMethod

The main idea of two-stage method is to first determine whether an utterance belongs to unseen intents (i.e., Yseen), and then classify it into a specific intent class. This method bypasses the need to classify an input sentence among all the seen and unseen intents, thereby alleviating the domain shift problem. To verify the performance of integrating CTIR into the two-stage method, we design a new two-stage pipeline. In Phase1, a test utterance is classified into one of the classes from Yseen ∪ {yunseen} using the UID classifier. In practice, we use the density-based algorithm LOF (LMCL) [Lin and Xu, 2019] to perform UID. In Phase2, we perform ZSID for the utterances that have been classified into yunseen, using the methods described in Section 3.1 and Section 3.2.

4 Experiments

4.1 Datasets and Experimental Setup

We conduct experiments on two benchmarks for intent detection. SNIPS [Coucke et al., 2018] is a corpus to evaluate the performance of voice assistants, which contains 5 seen intents and 2 unseen intents. CLINC [Larson et al., 2019] includes out-of-scope queries and 22,500 in-scope queries covering intent classes from 10 domains. We use the in-scope data to build our dataset with 50 seen intents and 10 unseen intents. For more details of the datasets, please refer to Appendix A1 .

Dataset Processing. For ZSID, we use all utterances from seen intents to construct the training set and those from unseen intents to construct the test set. For the training set of GZSID, we randomly select 70% utterances of each seen intent and replicate the unseen label names to roughly the same number. Although our focus is the zero-shot problem, utterances from seen intents still account for the majority in real-world applications. In light of this phenomenon, we balance the sample number of unseen and seen classes to build the test set: selecting the remaining 30% utterances of each seen intent and 30% random utterances of each unseen intent (30% seen and 100% unseen intent utterance in the setting of existing works). Besides, we use the raw label names without modification while most existing work modified the label names slightly to achieve better performance, e.g. from "SearchScreeningEvent" to "SearchMovie".

Experimental Setup. We use the test set for hyperparameter(Appendix B1 ) tuning, which is the same with most ZSID work. We run the experiments for five times with different random seeds. We consider two evaluation metrics: Accuray (Acc), and F1, both of which are computed with the average value weighted by their support on each class.

4.2 Baselines

We integrate CTIR with representative ZSID systems. For transformation-based methods with linear classifier, we explore the use of CNN, LSTM, and BERT [Devlin et al., 2019] as the feature extractor, which are denoted as CNN, LSTM and BERT, respectively. We find that use BERT to compute the similarity matrix leads to poor results, as the label names are very short. Therefore, we combine BERT feature extractor with CNN-CTIR computed similarity matrix for the BERT baseline. For capsule network classifier, we adopt CapsNet [Xia et al., 2018] as the baseline system. In terms of the compatibility-based methods, we study the combination of CTIR with Zero-shotDNN [Kumar et al., 2017] and CDSSM [Chen et al., 2016].

We also introduce two strong baselines targeting the two limitations related to ZSID and GZSID respectively, as discussed in Section 1. For ZSID, we replace the original label names with manually selected keywords (denoted as + key), which reflect the inter-intent relationships more accurately. For GZSID, we build a two-stage approach (denoted as + LOF) as described in Section 3.3.

ReCapsNet [Liu et al., 2019] and the two-stage method SEG [Yan et al., 2020], which achieve the SOTA performance, are not open-sourced. Different from our two-stage method, SEG ensembles UID and ZSID models in Phase1 and conducts seen intent prediction in Phase2. To compare with them, we follow their way of data processing and run our model.

4.3 Results and Analysis

Performance in ZSID. As can be seen in Table 1, the performance of CDSSM, ZSDNN and CapsNet can be significantly improved with the manually composed keyword labels. This demonstrates the first limitation that existing ZSID methods are not good at associating the relationship between seen and unseen intents when the label names are given in raw phrases or sentences. Our proposed framework, regardless of the backbone network and ZSID strategy, consistently outperforms the baselines with comfortable margin (11.49 ∼ 33.82 absolute F1 improvement). More importantly, the CTIR-enhanced models also achieve better results than the manually selected keywords, with an encouraging improvement of 11.28 F1 score for the CDSSM baseline. This shows the effectiveness of the proposed method to alleviate the effect of poor-quality label names, dispensing with the need of manual modification.

Performance in GZSID. From the results in Table 2, we can derive four observations: 1) The baselines work well on the seen intents, while they fail to detect the unseen intents. This phenomenon attests to the second limitation that the ZSID systems cannot effectively work in the GZSID scenario. 2) The two-stage framework (+LOF) brings significant improvement on the unseen intents, successfully alleviating the domain-shift problem. 3) Our CTIR framework improves the performance to a larger extent, which on average outperforms +LOF with 10.04 F1 on unseen intents and 3.37 F1 in terms of overall performance. 4) In terms of Acc, CTIR performs better in seen intents while LOF achieves higher scores in unseen intents. We analyze this trade-off phenomenon in Appendix C1 . 5) CTIR can further promote the strong twostage baseline: Enhancing the Phase2 ZSID model with CTIR (+LOF+CTIR) results in a substantial improvement of +LOF on the unseen intents, where the Acc is increased by 19.77 on average and F1 score is increased by 16.52 on average. The overall performance is thereby also improved.

Relieving of two limitations. We qualitatively analyze the effectiveness of CTIR in relieving the two limitations by answering the following questions. Q1: Can CTIR model a better inter-intent relationship in ZSID? Q2: Can CTIR distinguish between the samples of seen and unseen intents?

Q1 A good inter-intent relationship is crucial to transform the model from seen intent to unseen intent. To illustrate the problem of word embedding based inter-intent similarity, we compute the similarity with the sum of original label word embeddings (denoted as CapsNet Intent Labels), the keyword embedding (denoted as Keyword Intent Labels), and our similarity scorer (CapsNet+CTIR), which is shown in Figure2. When using the original label names, "BookRestaurant" is much more similar to "RateBook" than the other labels, as a result of the shared word "Book". In comparison, the keyword-based similarity matrix and our similarity scorer successfully avoid the pitfall of "Book" polysemy, and the similarities among other intents are also reasonable (e.g., "AddToPlaylist" and "PlayMusic"). This demonstrates that the intent representation of CapsNet+CTIR is well-learned, which can better associate the inter-intent relationship and dispenses with manually selecting keywords. We also find that for CapsNet+CTIR the two unseen classes are more similar to each other than to the seen classes. This is partly due to the introduction of the simplified unseen intent detection task, which aims to distinguish between seen classes and unseen classes.

Q2 To examine whether the CTIR framework can actually learn to distinguish between the unseen and seen intent representations, we visualize the utterance representations using the feature extractors of four methods (see Figure 3). We can see that the two unseen intents "AddToPlaylist" (red) and "RateBook" (yellow) are entangled with "SearchCreativeWork" (blue) and "PlayMusic" (green) in the CapsNet representation space. For ZSDNN representations, yellow dots and blue dots appear as a single cluster, so it is with red dots and green dots. These entangled representations make it difficult to perform GZSID. In comparison, the CapsNet+CTIR successfully learns an independent sub-space for "AddToPlaylist" and disentangles a part of "RateBook" from the seen intents. ZSDNN+CTIR pulls the two unseen classes as a whole out from the seen classes, with a vague but identifiable boundry in between.

Comparison with SOTA methods. Table 3 presents the comparison of the proposal with SOTA methods under their experimental settings as described in the Baselines. As we can see, +CTIR methods clearly outperform ReCapsNet in ZSID, and the +LOF+CTIR methods outperform SEG, which is also a two-stage approach, in GZSID. The +LOF method is used as our stronger baseline and its performance is comparable with SEG in GZSID. One may find that ZSDNN+LOF+CTIR is better than ZSDNN+CTIR in Table 3, while lags behind in our GZSID setting. The reason is that their dataset splitting assigns less importance to the seen intents, where the performance of +LOF is relatively poor.

Ablation Study. We can observe from Table 4 that: 1) Our similarity scorer clearly outperforms the word embedding based intent similarity, especially in the ZSID setting. The advantage of SS over ES is less obvious in GZSID because the model trained with CTIR can directly perform GZSID, which makes the effect of the similarity matrix less significant. This can also explain for the phenomenon that w/o SS outperforms CNN+CTIR in CLINC under GZSID setting. 2) In spite of this, removing the similarity scorer has a negative impact on the model performance. 3) Multi-task learning consistently benefits the CTIR framework in all circumstances. For GZSID, the improvement directly comes from the disentangling effect of MT. For ZSID, we attribute the improvement to the well-learned relationship between seen and unseen intents, which is a side-effect of increasing the distance between unseen and seen intent representations. We also investigate the performance variation with the increase of α and λ′ , please see Appendix D1 for details.

5 RelatedWork

Zero-shot Learning. In computer vision, ZSL is a welldeveloped sub-field, where a common approach is to relate unseen classes with seen classes through visual attributes [Farhadi et al., 2009; Parikh and Grauman, 2011] or word2vec representations of the class names [Mikolov et al., 2013; Frome et al., 2013; Socher et al., 2013]. For ZSID, transformation-based methods [Xia et al., 2018; Liu et al., 2019] calculate the inter-intent similarity based on the word embeddings of intent labels, and use it to transform the predictions from seen intents to unseen intents. Compatibility-based methods [Chen et al., 2016; Kumar et al., 2017] attempt to learn a shared semantic space for label names and utterances from seen data, and then measure the similarity between a test utterance and each unseen label in this space. There are also studies resorting to external knowledge, e.g., label ontologies [Ferreira et al., 2015] or human-defined attributes [Yazdani and Henderson, 2015; Zhang et al., 2019], which, however, are laborious to obtain.

Generalized Zero-shot Learning. Socher [2013] and Zhang [2016] design different model architectures to consider the probability of each sample coming from an unseen class before final classifying. In the task of intent detection, ReCapsNet-ZS [Liu et al., 2019] enhanced CapsNet [Xia et al., 2018] in GZSID by modeling the correlation between the dimensions of word embeddings, which can find a more accurate connection between seen and unseen intents. On the basis of ReCapsNet-ZS, Yan [2020] proposed a two-stage GZSID method that combines unknown intent detection and ZSID, which successfully resolves the domain-shift problem. However, this is at the cost of the performance in seen intents, which is the majority in real-world applications.

Class-transductive Zero-shot Learning. Class- transductive ZSL supports the use of semantic information (typically textual descriptions) about the unseen classes during training. In the CV field, class-transductive methods are used to infer the relationship between seen and unseen classes [Fu et al., 2015b; Fu et al., 2018] or directly predict the parameters of unseen intent classifiers [Elhoseiny et al., 2013; Wang et al., 2018]. In comparison, CTIR uses the unseen label names as training instances to learn unseen intent representations, which takes advantage of the fact that label names and utterances both come from the textual domain.

6 Conclusions

In this paper, we propose a class-transductive framework, CTIR, to overcome the limitations of existing ZSID models. CTIR utilizes the unseen label names as input utterances and includes the unseen classes into the prediction space during training. Under this framework, we present a multitask learning objective in the training stage to encourage the model to learn the distinctions between unseen and seen intents. In the inference stage, we develop a similarity scorer, which can better associate the inter-intent connections based on the learned representations. Experiments on two benchmarks show that CTIR can bring considerable improvement to ZSID systems with different ZSL strategies and backbone networks.

Acknowledgments

This work was supported by National Natural Science Foundation of China (No. 61976207, No. 61906187)