{"type":"ScholarlyArticle","title":"“Think Before You Speak”: Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs","authors":[{"name":"Shuo Zhang","slug":"shuo-zhang","given":"Shuo","family":"Zhang","affiliation":"MOE KLINNS Lab , Xi'an Jiaotong University , Xi'an 710049 , P. R. China"},{"name":"Junzhou Zhao","slug":"junzhou-zhao","given":"Junzhou","family":"Zhao","affiliation":"MOE KLINNS Lab , Xi'an Jiaotong University , Xi'an 710049 , P. R. China"},{"name":"Pinghui Wang","slug":"pinghui-wang","given":"Pinghui","family":"Wang","affiliation":"MOE KLINNS Lab , Xi'an Jiaotong University , Xi'an 710049 , P. R. China"},{"name":"Yu Li","slug":"yu-li","given":"Yu","family":"Li","affiliation":"MOE KLINNS Lab , Xi'an Jiaotong University , Xi'an 710049 , P. R. China"},{"name":"Yi Huang","slug":"yi-huang","given":"Yi","family":"Huang","affiliation":"JIUTIAN Team , China Mobile Research , Beijing 100053 , P. R. China"},{"name":"Junlan Feng","slug":"junlan-feng","given":"Junlan","family":"Feng","affiliation":"JIUTIAN Team , China Mobile Research , Beijing 100053 , P. R. China"}],"abstract":"Multi-action dialog policy (MADP), which generates multiple atomic dialog actions per turn, has been widely applied in task-oriented dialog systems to provide expressive and efficient system responses. Existing MADP models usually imitate action combinations from the labeled multi-action dialog samples. Due to data limitations, they generalize poorly toward unseen dialog flows. While interactive learning and reinforcement learning algorithms can be applied to incorporate external data sources of real users and user simulators, they take significant manual effort to build and suffer from instability. To address these issues, we propose Planning Enhanced Dialog Policy (PEDP), a novel multi-task learning framework that learns single-action dialog dynamics to enhance multi-action prediction. Our PEDP method employs model-based planning for conceiving what to express before deciding the current response through simulating single-action dialogs. Experimental results on the MultiWOZ dataset demonstrate that our fully supervised learning-based method achieves a solid task success rate of 90.6%, improving 3% compared to the state-of-the-art methods. The source code and the appendix of this paper can be obtained from https://github.com/ShuoZhangXJTU/PEDP.","doi":"10.24963/ijcai.2022/626","url":"https://ijcai.org/proceedings/2022/626","pdf":"https://ijcai.org/proceedings/2022/0626.pdf","markdown":"https://ijcai.org/proceedings/2022/626.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/626","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":626,"pages":{"first":4510,"last":4516},"keywords":["Natural Language Processing: Applications","Natural Language Processing: Dialogue and Interactive Systems"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986958","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985101","length_sec":670,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Budzianowski et al., 2018] Paweł Budzianowski, Tsung- Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. MultiWOZ: A large-scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. arXiv:1810.00278, 2018.","title":"MultiWOZ: A large-scale multi-domain wizard-of-oz dataset for taskoriented dialogue modelling","authors":"Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Milica Osman Ramadan, Gašić","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Gašić et al., 2015] M Gašić, N Mrkšić, Pei-hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. Policy committee for adaptation in multi-domain spoken dialogue systems. In ASRU, pages 806-812. IEEE, 2015.","title":"Policy committee for adaptation in multi-domain spoken dialogue systems","authors":"Gašić, Pei-Hao Mrkšić, David Su, Tsung-Hsien Vandyke, Steve Wen, Young","year":"2015","venue":"ASRU","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gordon-Hall et al., 2020] Gabriel Gordon-Hall, Philip John Gorinski, and Shay B Cohen. Learning dialog policies from weak demonstrations. arXiv:2004.11054, 2020.","title":"Learning dialog policies from weak demonstrations","authors":"Gabriel Gordon-Hall, Philip John Gorinski, Shay B Cohen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Huang et al., 2020] Xinting Huang, Jianzhong Qi, Yu Sun, and Rui Zhang. Semi-supervised dialogue policy learning via stochastic reward estimation. arXiv:2005.04379, 2020.","title":"Semi-supervised dialogue policy learning via stochastic reward estimation","authors":"Huang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jang et al., 2016] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv:1611.01144, 2016.","title":"Categorical reparameterization with gumbel-softmax","authors":"Eric Jang, Shixiang Gu, Ben Poole","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jhunjhunwala et al., 2020] Megha Jhunjhunwala, Caleb Bryant, and Pararth Shah. Multi-action dialog policy learning with interactive human teaching. In SIGdial, pages 290-296, 2020.","title":"Multi-action dialog policy learning with interactive human teaching","authors":"Megha Jhunjhunwala, Caleb Bryant, Pararth Shah","year":"2020","venue":"SIGdial","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2017] Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. End-to-end task- completion neural dialogue systems. arXiv:1703.01008, 2017.","title":"End-to-end taskcompletion neural dialogue systems","authors":"Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, Asli Celikyilmaz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2020] Ziming Li, Julia Kiseleva, and Maarten de Rijke. Rethinking supervised learning and rein- forcement learning in task-oriented dialogue systems. arXiv:2009.09781, 2020.","title":"Rethinking supervised learning and reinforcement learning in task-oriented dialogue systems","authors":"Ziming Li, Julia Kiseleva, Maarten De Rijke","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liu and Lane, 2018] Bing Liu and Ian Lane. Adver- sarial learning of task-oriented neural dialog models. arXiv:1805.11762, 2018.","title":"Adversarial learning of task-oriented neural dialog models","authors":"Lane ; Liu, Bing Liu, Ian Lane","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Peng et al., 2018] Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, Kam-Fai Wong, and Shang-Yu Su. Deep Dyna-Q: Integrating planning for task-completion dia- logue policy learning. arXiv:1801.06176, 2018.","title":"Deep Dyna-Q: Integrating planning for task-completion dialogue policy learning","authors":"Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, Kam-Fai Wong, Shang-Yu Su","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Qin et al., 2021] Xiaoyu Qin, Xiaowang Zhang, Muham- mad Qasim Yasin, Shujun Wang, Zhiyong Feng, and Guo- hui Xiao. Suma: A partial materialization-based scalable query answering in owl 2 dl. Data Science and Engineer- ing, 6(2):229-245, 2021.","title":"Suma: A partial materialization-based scalable query answering in owl 2 dl","authors":"Xiaowang Xiaoyu Qin, Muhammad Zhang, Shujun Qasim Yasin, Zhiyong Wang, Guohui Feng, Xiao","year":"2021","venue":"Data Science and Engineering","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rastogi et al., 2020] Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In AAAI, volume 34, pages 8689-8696, 2020.","title":"Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset","authors":"Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, Pranav Khaitan","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Shu et al., 2019] Lei Shu, Hu Xu, Bing Liu, and Piero Molino. Modeling multi-action policy for task-oriented dialogues. arXiv:1908.11546, 2019.","title":"Modeling multi-action policy for task-oriented dialogues","authors":"Lei Shu, Hu Xu, Bing Liu, Piero Molino","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Su et al., 2018] Shang-Yu Su, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Yun-Nung Chen. Discriminative deep dyna-q: Robust planning for dialogue policy learning. arXiv:1808.09442, 2018.","title":"Discriminative deep dyna-q: Robust planning for dialogue policy learning","authors":"Shang-Yu Su, Xiujun Li, Jianfeng Gao, Jingjing Liu, Yun-Nung Chen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Suzuki and Ogata, 2020] K. Suzuki and T. Ogata. Stable Deep Reinforcement Learning Method by Predicting Un- certainty in Rewards as a Subtask. ICONIP, Proceedings, Part II, 2020.","title":"Stable Deep Reinforcement Learning Method by Predicting Uncertainty in Rewards as a Subtask","authors":"K Suzuki, T Ogata","year":"2020","venue":"ICONIP, Proceedings","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Takanobu et al., 2019] Ryuichi Takanobu, Hanlin Zhu, and Minlie Huang. Guided dialog policy learning: Re- ward estimation for multi-domain task-oriented dialog. arXiv:1908.10719, 2019.","title":"Guided dialog policy learning: Reward estimation for multi-domain task-oriented dialog","authors":"Ryuichi Takanobu, Hanlin Zhu, Minlie Huang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tian et al., 2020] Shan Tian, Songsong Mo, Liwei Wang, and Zhiyong Peng. Deep reinforcement learning-based ap- proach to tackle topic-aware influence maximization. Data Science and Engineering, 5(1):1-11, 2020.","title":"Deep reinforcement learning-based approach to tackle topic-aware influence maximization","authors":"Shan Tian, Songsong Mo, Liwei Wang, Zhiyong Peng","year":"2020","venue":"Data Science and Engineering","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2016] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with expe- rience replay. arXiv:1611.01224, 2016.","title":"Sample efficient actor-critic with experience replay","authors":"Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, Nando De Freitas","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2020] Sihan Wang, Kaijie Zhou, Kunfeng Lai, and Jianping Shen. Task-completion dialogue policy learn- ing via monte carlo tree search with dueling network. In EMNLP, pages 3461-3471, 2020.","title":"Task-completion dialogue policy learning via monte carlo tree search with dueling network","authors":"Sihan Wang, Kaijie Zhou, Kunfeng Lai, Jianping Shen","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2019] Yuexin Wu, Xiujun Li, Jingjing Liu, Jian- feng Gao, and Yiming Yang. Switch-based active deep dyna-q: Efficient adaptive planning for task-completion dialogue policy learning. In AAAI, volume 33, pages 7289-7296, 2019.","title":"Switch-based active deep dyna-q: Efficient adaptive planning for task-completion dialogue policy learning","authors":"Yuexin Wu, Xiujun Li, Jingjing Liu, Jianfeng Gao, Yiming Yang","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2020] Zheng Zhang, Ryuichi Takanobu, Qi Zhu, Minlie Huang, and Xiaoyan Zhu. Recent ad- vances and challenges in task-oriented dialog systems. Science China Technological Sciences, pages 1-17, 2020.","title":"Recent advances and challenges in task-oriented dialog systems","authors":"Zheng Zhang, Ryuichi Takanobu, Qi Zhu, Minlie Huang, Xiaoyan Zhu","year":"2020","venue":"Science China Technological Sciences","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2021] Shuo Zhang, Junzhou Zhao, Pinghui Wang, Nuo Xu, Yang Yang, Yiting Liu, Yi Huang, and Junlan Feng. Learning to check contract inconsistencies. In AAAI, volume 35, pages 14446-14453, 2021.","title":"Learning to check contract inconsistencies","authors":"Shuo Zhang, Junzhou Zhao, Pinghui Wang, Nuo Xu, Yang Yang, Yiting Liu, Yi Huang, Junlan Feng","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu et al., 2020] Qi Zhu, Zheng Zhang, Yan Fang, Xiang Li, Ryuichi Takanobu, Jinchao Li, Baolin Peng, Jianfeng Gao, Xiaoyan Zhu, and Minlie Huang. Convlab-2: An open-source toolkit for building, evaluating, and diagnos- ing dialogue systems. arXiv:2002.04793, 2020.","title":"Convlab-2: An open-source toolkit for building, evaluating, and diagnosing dialogue systems","authors":"Qi Zhu, Zheng Zhang, Yan Fang, Xiang Li, Ryuichi Takanobu, Jinchao Li, Baolin Peng, Jianfeng Gao, Xiaoyan Zhu, Minlie Huang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}