{"type":"ScholarlyArticle","title":"Submodularity-Inspired Data Selection for Goal-Oriented Chatbot Training Based on Sentence Embeddings","authors":[{"name":"Mladen Dimovski","slug":"mladen-dimovski","given":"Mladen","family":"Dimovski","affiliation":"School of Computer and Communication Sciences , EPFL , Switzerland"},{"name":"Claudiu Musat","slug":"claudiu-musat","given":"Claudiu","family":"Musat","affiliation":"Artificial Intelligence Group -Swisscom AG"},{"name":"Vladimir Ilievski","slug":"vladimir-ilievski","given":"Vladimir","family":"Ilievski","affiliation":"School of Computer and Communication Sciences , EPFL , Switzerland"},{"name":"Andreea Hossman","slug":"andreea-hossman","given":"Andreea","family":"Hossman","affiliation":"Artificial Intelligence Group -Swisscom AG"},{"name":"Michael Baeriswyl","slug":"michael-baeriswyl","given":"Michael","family":"Baeriswyl","affiliation":"Artificial Intelligence Group -Swisscom AG"}],"abstract":"Spoken language understanding (SLU) systems, such as goal-oriented chatbots or personal assistants, rely on an initial natural language understanding (NLU) module to determine the intent and to extract the relevant information from the user queries they take as input. SLU systems usually help users to solve problems in relatively narrow domains and require a large amount of in-domain training data. This leads to significant data availability issues that inhibit the development of successful systems.\r\nTo alleviate this problem, we propose a technique of data selection in the low-data regime that enables us to train with fewer labeled sentences, thus smaller labelling costs.\r\n\r\nWe propose a submodularity-inspired data ranking function, the ratio-penalty marginal gain, for selecting data points to label based only on the information extracted from the textual embedding space. We show that the distances in the embedding space are a viable source of information that can be used for data selection. Our method outperforms two known active learning techniques and enables  cost-efficient training of the NLU unit. Moreover, our proposed selection technique does not need the model to be retrained in between the selection steps, making it time efficient as well.","doi":"10.24963/ijcai.2018/559","url":"https://ijcai.org/proceedings/2018/559","pdf":"https://ijcai.org/proceedings/2018/0559.pdf","markdown":"https://ijcai.org/proceedings/2018/559.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/559","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":559,"pages":{"first":4019,"last":4025},"keywords":["Natural Language Processing: Tagging, chunking, and parsing","Natural Language Processing: Embeddings","Natural Language Processing: NLP Applications and Tools","Natural Language Processing: Natural Language Processing","Machine Learning: Active Learning"],"videos":[],"references":[{"position":1,"raw":"[Angeli et al., 2014] Gabor Angeli, Sonal Gupta, Melvin Jose, Christopher D Manning, Christopher Ré, Julie Tib- shirani, Jean Y Wu, Sen Wu, and Ce Zhang. Stanford's 2014 slot filling systems. TAC KBP, 695, 2014.","title":"Stanford's 2014 slot filling systems","authors":"Gabor Angeli, Sonal Gupta, Melvin Jose, Christopher D Manning, Christopher Ré, Julie Tibshirani, Jean Y Wu, Sen Wu, Ce Zhang","year":"2014","venue":"TAC KBP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Culotta and McCallum, 2005] Aron Culotta and Andrew McCallum. Reducing labeling effort for structured pre- diction tasks. In AAAI, volume 5, pages 746-751, 2005.","title":"Reducing labeling effort for structured prediction tasks","authors":"Mccallum Culotta, Aron Culotta, Andrew Mccallum","year":"2005","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fu et al., 2013] Yifan Fu, Xingquan Zhu, and Bin Li. A sur- vey on instance selection for active learning. Knowledge and information systems, pages 1-35, 2013.","title":"A survey on instance selection for active learning","authors":"Yifan Fu, Xingquan Zhu, Bin Li","year":"2013","venue":"Knowledge and information systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hakkani-Tür et al., 2016] Dilek Hakkani-Tür, Gökhan Tür, Asli Celikyilmaz, Yun-Nung Chen, Jianfeng Gao, Li Deng, and Ye-Yi Wang. Multi-domain joint semantic frame parsing using bi-directional rnn-lstm. In INTER- SPEECH, pages 715-719, 2016.","title":"Multi-domain joint semantic frame parsing using bi-directional rnn-lstm","authors":"Dilek Hakkani-Tür, Gökhan Tür, Asli Celikyilmaz, Yun-Nung Chen, Jianfeng Gao, Li Deng, Ye-Yi Wang","year":"2016","venue":"INTER-SPEECH","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huang et al., 2015] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991, 2015.","title":"Bidirectional lstm-crf models for sequence tagging","authors":"Zhiheng Huang, Wei Xu, Kai Yu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jaech et al., 2016] Aaron Jaech, Larry Heck, and Mari Os- tendorf. Domain adaptation of recurrent neural net- works for natural language understanding. arXiv preprint arXiv:1604.00117, 2016.","title":"Domain adaptation of recurrent neural networks for natural language understanding","authors":"Aaron Jaech, Larry Heck, Mari Ostendorf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kirchhoff and Bilmes, 2014] Katrin Kirchhoff and Jeff Bilmes. Submodularity for data selection in statistical ma- chine translation. In 2014 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 131-141, 2014.","title":"Submodularity for data selection in statistical machine translation","authors":"Bilmes Kirchhoff, Katrin Kirchhoff, Jeff Bilmes","year":"2014","venue":"Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krause and Golovin, 2014] Andreas Krause and Daniel Golovin. Submodular function maximization., 2014.","title":"Submodular function maximization","authors":"Golovin Krause, Andreas Krause, Daniel Golovin","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kurata et al., 2016] Gakuto Kurata, Bing Xiang, Bowen Zhou, and Mo Yu. Leveraging sentence-level information with encoder lstm for semantic slot filling. arXiv preprint arXiv:1601.01530, 2016.","title":"Leveraging sentence-level information with encoder lstm for semantic slot filling","authors":"Gakuto Kurata, Bing Xiang, Bowen Zhou, Mo Yu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lample et al., 2016] Guillaume Lample, Miguel Balles- teros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recog- nition. arXiv preprint arXiv:1603.01360, 2016.","title":"Neural architectures for named entity recognition","authors":"Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, Chris Dyer","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin and Bilmes, 2011] Hui Lin and Jeff Bilmes. A class of submodular functions for document summarization. In Proceedings of the 49th Annual Meeting of the Associ- ation for Computational Linguistics: Human Language Technologies-Volume 1, pages 510-520. Association for Computational Linguistics, 2011.","title":"A class of submodular functions for document summarization","authors":"Hui Lin, Jeff Bilmes","year":"2011","venue":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin and Bilmes, 2012] Hui Lin and Jeff A Bilmes. Learn- ing mixtures of submodular shells with application to doc- ument summarization. arXiv preprint arXiv:1210.4871, 2012.","title":"Learning mixtures of submodular shells with application to document summarization","authors":"Hui Lin, Jeff A Bilmes","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu and Lane, 2016] Bing Liu and Ian Lane. Attention- based recurrent neural network models for joint intent de- tection and slot filling. arXiv preprint arXiv:1609.01454, 2016.","title":"Attentionbased recurrent neural network models for joint intent detection and slot filling","authors":"Lane ; Liu, Bing Liu, Ian Lane","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2013a] Jingjing Liu, Panupong Pasupat, Scott Cyphers, and Jim Glass. Asgard: A portable architecture for multilingual dialogue systems. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 8386-8390. IEEE, 2013.","title":"Asgard: A portable architecture for multilingual dialogue systems","authors":"Liu","year":"2013","venue":"Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2013b] Jingjing Liu, Panupong Pasupat, Yining Wang, Scott Cyphers, and Jim Glass. Query understand- ing enhanced by hierarchical parsing structures. In Au- tomatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, pages 72-77. IEEE, 2013.","title":"Query understanding enhanced by hierarchical parsing structures","authors":"Liu","year":"2013","venue":"Automatic Speech Recognition and Understanding (ASRU)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ma and Hovy, 2016] Xuezhe Ma and Eduard Hovy. End- to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354, 2016.","title":"Xuezhe Ma and Eduard Hovy. Endto-end sequence labeling via bi-directional lstm-cnns-crf","authors":"Hovy Ma","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[McCallum et al., 1998] Andrew McCallum, Kamal Nigam, et al. Employing em and pool-based active learning for text classification. In ICML, volume 98, pages 350-358, 1998.","title":"Employing em and pool-based active learning for text classification","authors":"Andrew Mccallum, Kamal Nigam","year":"1998","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mesnil et al., 2015] Grégoire Mesnil, Yann Dauphin, Kaisheng Yao, Yoshua Bengio, Li Deng, Dilek Hakkani- Tur, Xiaodong He, Larry Heck, Gokhan Tur, Dong Yu, et al. Using recurrent neural networks for slot filling in spoken language understanding. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 23(3):530-539, 2015.","title":"Using recurrent neural networks for slot filling in spoken language understanding","authors":"Grégoire Mesnil, Yann Dauphin, Kaisheng Yao, Yoshua Bengio, Li Deng, Dilek Hakkani-Tur, Xiaodong He, Larry Heck, Gokhan Tur, Dong Yu","year":"2015","venue":"IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nemhauser et al., 1978] George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. An analysis of approxima- tions for maximizing submodular set functions-i. Math- ematical Programming, 14(1):265-294, 1978.","title":"An analysis of approximations for maximizing submodular set functions-i","authors":"Laurence A George L Nemhauser, Marshall L Wolsey, Fisher","year":"1978","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pagliardini et al., 2017] Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. Unsupervised learning of sentence embeddings using compositional n-gram features. arXiv preprint arXiv:1703.02507, 2017.","title":"Unsupervised learning of sentence embeddings using compositional n-gram features","authors":"Matteo Pagliardini, Prakhar Gupta, Martin Jaggi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wei et al., 2014] Kai Wei, Yuzong Liu, Katrin Kirchhoff, Chris Bartels, and Jeff Bilmes. Submodular subset se- lection for large-scale speech training data. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE Inter- national Conference on, pages 3311-3315. IEEE, 2014.","title":"Submodular subset selection for large-scale speech training data","authors":"Kai Wei, Yuzong Liu, Katrin Kirchhoff, Chris Bartels, Jeff Bilmes","year":"2014","venue":"Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wei et al., 2015] Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International Conference on Machine Learning, pages 1954-1963, 2015.","title":"Submodularity in data subset selection and active learning","authors":"Kai Wei, Rishabh Iyer, Jeff Bilmes","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhai et al., 2017] Feifei Zhai, Saloni Potdar, Bing Xiang, and Bowen Zhou. Neural models for sequence chunking. In AAAI, pages 3365-3371, 2017.","title":"Neural models for sequence chunking","authors":"Feifei Zhai, Saloni Potdar, Bing Xiang, Bowen Zhou","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang and Wang, 2016] Xiaodong Zhang and Houfeng Wang. A joint model of intent determination and slot fill- ing for spoken language understanding. In IJCAI, pages 2993-2999, 2016.","title":"A joint model of intent determination and slot filling for spoken language understanding","authors":"Xiaodong Zhang, Houfeng Wang","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu and Yu, 2017] Su Zhu and Kai Yu. Encoder-decoder with focus-mechanism for sequence labelling based spo- ken language understanding. In Acoustics, Speech and Sig- nal Processing (ICASSP), 2017 IEEE International Con- ference on, pages 5675-5679. IEEE, 2017.","title":"Encoder-decoder with focus-mechanism for sequence labelling based spoken language understanding","authors":"Yu ; ] Zhu, Zhu Su, Kai Yu","year":"2017","venue":"Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}