{"type":"ScholarlyArticle","title":"Towards Joint Intent Detection and Slot Filling via Higher-order Attention","authors":[{"name":"Dongsheng Chen","slug":"dongsheng-chen","given":"Dongsheng","family":"Chen","affiliation":"School of ECE , Peking University , China"},{"name":"Zhiqi Huang","slug":"zhiqi-huang","given":"Zhiqi","family":"Huang","affiliation":"School of ECE , Peking University , China"},{"name":"Xian Wu","slug":"xian-wu","given":"Xian","family":"Wu","affiliation":"Tencent Medical AI Lab , China"},{"name":"Shen Ge","slug":"shen-ge","given":"Shen","family":"Ge","affiliation":"Tencent Medical AI Lab , China"},{"name":"Yuexian Zou","slug":"yuexian-zou","given":"Yuexian","family":"Zou","affiliation":"School of ECE , Peking University , China"}],"abstract":"Recently, attention-based models for joint intent detection and slot filling have achieved state-of-the-art performance. However, we think the conventional attention can only capture the first-order feature interaction between two tasks and is insufficient. To address this issue, we propose a unified BiLinear attention block, which leverages bilinear pooling to synchronously explore both the contextual and channel-wise bilinear attention distributions to capture the second-order interactions between the input intent and slot features. Higher-order interactions are constructed by combining many such blocks and exploiting Exponential Linear activations. Furthermore, we present a Higher-order Attention Network (HAN) to jointly model them. The experimental results show that our approach outperforms the state-of-the-art results. We also conduct experiments on the new SLURP dataset, and give a discussion on HAN’s properties, i.e., robustness and generalization.","doi":"10.24963/ijcai.2022/565","url":"https://ijcai.org/proceedings/2022/565","pdf":"https://ijcai.org/proceedings/2022/0565.pdf","markdown":"https://ijcai.org/proceedings/2022/565.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/565","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":565,"pages":{"first":4072,"last":4078},"keywords":["Natural Language Processing: Applications","Natural Language Processing: Interpretability and Analysis of Models for NLP","Natural Language Processing: Dialogue and Interactive Systems"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986972","length_sec":80,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985115","length_sec":632,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.","title":"Layer normalization","authors":"Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E Hinton","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barron, 2017] Jonathan T. Barron. Continuously differen- tiable exponential linear units. CoRR, abs/1704.07483, 2017.","title":"Continuously differentiable exponential linear units","authors":"Jonathan T Barron","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bastianelli et al., 2020] Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. Slurp: A spoken language understanding resource package. arXiv preprint arXiv:2011.13205, 2020.","title":"Slurp: A spoken language understanding resource package","authors":"Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, Verena Rieser","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2022] Dongsheng Chen, Zhiqi Huang, and Yuexian Zou. Leveraging bilinear attention to improve spoken language understanding. In ICASSP, 2022.","title":"Leveraging bilinear attention to improve spoken language understanding","authors":"Dongsheng Chen, Zhiqi Huang, Yuexian Zou","year":"2022","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cornia et al., 2020] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image captioning. In CVPR, 2020.","title":"Meshed-memory transformer for image captioning","authors":"Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, Rita Cucchiara","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Coucke et al., 2018] Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. Snips voice platform: an embedded spoken lan- guage understanding system for private-by-design voice interfaces. CoRR, abs/1805.10190, 2018.","title":"Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces","authors":"Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, Joseph Dureau","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cui and Zhang, 2019] Leyang Cui and Yue Zhang. Hierarchically-refined label attention network for se- quence labeling. arXiv preprint arXiv:1908.08676, 2019.","title":"Hierarchically-refined label attention network for sequence labeling","authors":"Zhang Cui, Leyang Cui, Yue Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fang et al., 2019] Pengfei Fang, Jieming Zhou, Soumava Kumar Roy, Lars Petersson, and Mehrtash Harandi. Bilinear attention networks for person retrieval. In CVPR, 2019.","title":"Bilinear attention networks for person retrieval","authors":"Pengfei Fang, Jieming Zhou, Soumava Kumar Roy, Lars Petersson, Mehrtash Harandi","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goo et al., 2018] Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, and Yun-Nung Chen. Slot-gated modeling for joint slot filling and intent prediction. In ACL, 2018.","title":"Slot-gated modeling for joint slot filling and intent prediction","authors":"Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, Yun-Nung Chen","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hakkani-Tür et al., 2016] Dilek Hakkani-Tür, Gökhan Tür, Asli C ¸elikyilmaz, Yun-Nung Chen, Jianfeng Gao, Li Deng, and Ye-Yi Wang. Multi-domain joint semantic frame parsing using bi-directional RNN-LSTM. In IN- TERSPEECH, 2016.","title":"Multi-domain joint semantic frame parsing using bi-directional RNN-LSTM","authors":"Dilek Hakkani-Tür, Gökhan Tür, Yun-Nung Asli C ¸elikyilmaz, Jianfeng Chen, Li Gao, Ye-Yi Deng, Wang","year":"2016","venue":"IN-TERSPEECH","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hemphill et al., 1990] Charles T. Hemphill, John J. God- frey, and George R. Doddington. The ATIS spoken lan- guage systems pilot corpus. In HLT, 1990.","title":"The ATIS spoken language systems pilot corpus","authors":"Charles T Hemphill, John J Godfrey, George R Doddington","year":"1990","venue":"HLT","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim et al., 2018] Jin-Hwa Kim, Jaehyun Jun, and Byoung- Tak Zhang. Bilinear attention networks. arXiv preprint arXiv:1805.07932, 2018.","title":"Bilinear attention networks","authors":"Kim","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kumari and Ekbal, 2021] Rina Kumari and Asif Ekbal. Amfb: Attention based multimodal factorized bilinear pooling for multimodal fake news detection. Expert Sys- tems with Applications, 184:115412, 2021.","title":"Amfb: Attention based multimodal factorized bilinear pooling for multimodal fake news detection","authors":"Rina Kumari, Asif Ekbal","year":"2021","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2018] Changliang Li, Liang Li, and Ji Qi. A self- attentive model with gate mechanism for spoken language understanding. In EMNLP, 2018.","title":"A selfattentive model with gate mechanism for spoken language understanding","authors":"Changliang Li, Liang Li, Ji Qi","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2015] Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. Bilinear CNN models for fine-grained visual recognition. In ICCV, 2015.","title":"Bilinear CNN models for fine-grained visual recognition","authors":"Tsung-Yu Lin, Aruni Roychowdhury, Subhransu Maji","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Niu et al., 2019] Peiqing Niu, Zhongfu Chen, Meina Song, et al. A novel bi-directional interrelated model for joint intent detection and slot filling. arXiv preprint arXiv:1907.00390, 2019.","title":"A novel bi-directional interrelated model for joint intent detection and slot filling","authors":"Niu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pan et al., 2020] Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. X-linear attention networks for image captioning. In CVPR, 2020.","title":"X-linear attention networks for image captioning","authors":"Pan","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qin et al., 2019] Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. A stack-propagation frame- work with token-level intent detection for spoken language understanding. In EMNLP/IJCNLP, 2019.","title":"A stack-propagation framework with token-level intent detection for spoken language understanding","authors":"Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, Ting Liu","year":"2019","venue":"EMNLP/IJCNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Qin et al., 2021] Libo Qin, Tailu Liu, Wanxiang Che, Bing- bing Kang, Sendong Zhao, and Ting Liu. A co-interactive transformer for joint slot filling and intent detection. In ICASSP, 2021.","title":"A co-interactive transformer for joint slot filling and intent detection","authors":"Libo Qin, Tailu Liu, Wanxiang Che, Bingbing Kang, Sendong Zhao, Ting Liu","year":"2021","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2020] Donghang Yu, Haitao Guo, Qing Xu, Jun Lu, Chuan Zhao, and Yuzhun Lin. Hierarchical attention and bilinear fusion for remote sensing image scene classi- fication. Journal of Selected Topics in Applied Earth Ob- servations and Remote Sensing, 13:6372-6383, 2020.","title":"Hierarchical attention and bilinear fusion for remote sensing image scene classification","authors":"Donghang Yu, Haitao Guo, Qing Xu, Jun Lu, Chuan Zhao, Yuzhun Lin","year":"2020","venue":"Journal of Selected Topics in Applied Earth Observations and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2019] Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip S Yu. Joint slot filling and intent de- tection via capsule neural networks. In ACL, 2019.","title":"Joint slot filling and intent detection via capsule neural networks","authors":"Zhang","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2020] Linhao Zhang, Dehong Ma, Xiaodong Zhang, Xiaohui Yan, and Houfeng Wang. Graph lstm with context-gated mechanism for spoken language under- standing. In AAAI, 2020.","title":"Graph lstm with context-gated mechanism for spoken language understanding","authors":"Zhang","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhu et al., 2020] Su Zhu, Ruisheng Cao, and Kai Yu. Dual learning for semi-supervised natural language understand- ing. IEEE/ACM TASLP, 2020.","title":"Dual learning for semi-supervised natural language understanding","authors":"Su Zhu, Ruisheng Cao, Kai Yu","year":"2020","venue":"IEEE/ACM TASLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}