{"type":"ScholarlyArticle","title":"Building Personalized Simulator for Interactive Search","authors":[{"name":"Qianlong Liu","slug":"qianlong-liu","given":"Qianlong","family":"Liu","affiliation":"Fudan University , China"},{"name":"Baoliang Cui","slug":"baoliang-cui","given":"Baoliang","family":"Cui","affiliation":"Alibaba Group , China"},{"name":"Zhongyu Wei","slug":"zhongyu-wei","given":"Zhongyu","family":"Wei","affiliation":"Fudan University , China"},{"name":"Baolin Peng","slug":"baolin-peng","given":"Baolin","family":"Peng","affiliation":"The Chinese University of Hong Kong , Hong Kong"},{"name":"Haikuan Huang","slug":"haikuan-huang","given":"Haikuan","family":"Huang","affiliation":"Alibaba Group , China"},{"name":"Hongbo Deng","slug":"hongbo-deng","given":"Hongbo","family":"Deng","affiliation":"Alibaba Group , China"},{"name":"Jianye Hao","slug":"jianye-hao","given":"Jianye","family":"Hao","affiliation":"Tianjin University , China"},{"name":"Xuanjing Huang","slug":"xuanjing-huang","given":"Xuanjing","family":"Huang","affiliation":"Fudan University , China"},{"name":"Kam-Fai Wong","slug":"kam-fai-wong","given":"Kam-Fai","family":"Wong","affiliation":"The Chinese University of Hong Kong , Hong Kong"}],"abstract":"Interactive search, where a set of tags is recommended to users together with search results at each turn, is an effective way to guide users to identify their information need. It is a classical sequential decision problem and the reinforcement learning based agent can be introduced as a solution. The training of the agent can be divided into two stages, i.e., offline and online. Existing reinforcement learning based systems tend to perform the offline training in a supervised way based on historical labeled data while the online training is performed via reinforcement learning algorithms based on interactions with real users. The mis-match between online and offline training leads to a cold-start problem for the online usage of the agent. To address this issue, we propose to employ a simulator to mimic the environment for the offline training of the agent. Users' profiles are considered to build a personalized simulator, besides, model-based approach is used to train the simulator and is able to use the data efficiently. Experimental results based on real-world dataset demonstrate the effectiveness of our agent and personalized simulator.","doi":"10.24963/ijcai.2019/710","url":"https://ijcai.org/proceedings/2019/710","pdf":"https://ijcai.org/proceedings/2019/0710.pdf","markdown":"https://ijcai.org/proceedings/2019/710.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/710","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":710,"pages":{"first":5109,"last":5115},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Asri et al., 2016] Layla El Asri, Jing He, and Kaheer Sule- man. A sequence-to-sequence model for user simulation in spoken dialogue systems. CoRR, abs/1607.00070, 2016.","title":"A sequence-to-sequence model for user simulation in spoken dialogue systems","authors":"Layla El Asri, Jing He, Kaheer Suleman","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chandramohan et al., 2011] Senthilkumar Chandramohan, Matthieu Geist, Fabrice Lefevre, and Olivier Pietquin. User simulation in dialogue systems using inverse rein- forcement learning. In Twelfth Annual Conference of the International Speech Communication Association, 2011.","title":"User simulation in dialogue systems using inverse reinforcement learning","authors":"Senthilkumar Chandramohan, Matthieu Geist, Fabrice Lefevre, Olivier Pietquin","year":"2011","venue":"Twelfth Annual Conference of the International Speech Communication Association","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Shi-Yong Chen, Yang Yu, Qing Da, Jun Tan, Hai-Kuan Huang, and Hai-Hong Tang. Stabilizing reinforcement learning in dynamic environment with ap- plication to online recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 1187-1196. ACM, 2018.","title":"Stabilizing reinforcement learning in dynamic environment with application to online recommendation","authors":"Yang Shi-Yong Chen, Qing Yu, Jun Da, Hai-Kuan Tan, Hai-Hong Huang, Tang","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Choi et al., 2018] Sungwoon Choi, Heonseok Ha, Uiwon Hwang, Chanju Kim, Jung-Woo Ha, and Sungroh Yoon. Reinforcement learning based recommender system using biclustering technique. CoRR, abs/1801.05532, 2018.","title":"Reinforcement learning based recommender system using biclustering technique","authors":"Choi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Clark, 2015] Jack Clark. Google turning its lucrative web search over to ai machines. Bloomberg Technology, 26, 2015.","title":"Google turning its lucrative web search over to ai machines","authors":"Clark ;, Jack Clark","year":"2015","venue":"Bloomberg Technology","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Ji He, Jianshu Chen, Xiaodong He, Jian- feng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with a natural language ac- tion space. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1621-1630, 2016.","title":"Deep reinforcement learning with a natural language action space","authors":"Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, Mari Ostendorf","year":"2016","venue":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jiang et al., 2015] Nan Jiang, Alex Kulesza, and Satinder Singh. Abstraction selection in model-based reinforce- ment learning. In International Conference on Machine Learning, pages 179-188, 2015.","title":"Abstraction selection in model-based reinforcement learning","authors":"Nan Jiang, Alex Kulesza, Satinder Singh","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2011] Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual- bandit-based news article recommendation algorithms. In Proceedings of the fourth ACM international conference on Web search and data mining, pages 297-306. ACM, 2011.","title":"Unbiased offline evaluation of contextualbandit-based news article recommendation algorithms","authors":"Lihong Li, Wei Chu, John Langford, Xuanhui Wang","year":"2011","venue":"Proceedings of the fourth ACM international conference on Web search and data mining","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2017] Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. End-to-end task- completion neural dialogue systems. In Proceedings of the Eighth International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), volume 1, pages 733-743, 2017.","title":"End-to-end taskcompletion neural dialogue systems","authors":"Li","year":"2017","venue":"Proceedings of the Eighth International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2017] Shichen Liu, Fei Xiao, Wenwu Ou, and Luo Si. Cascade ranking for operational e-commerce search. In Proceedings of the 23rd ACM SIGKDD Inter- national Conference on Knowledge Discovery and Data Mining, pages 1557-1565. ACM, 2017.","title":"Cascade ranking for operational e-commerce search","authors":"Liu","year":"2017","venue":"Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lu et al., 2018] Keting Lu, Shiqi Zhang, and Xiaoping Chen. Learning to dialogue via complex hindsight experi- ence replay. arXiv preprint arXiv:1808.06497, 2018.","title":"Learning to dialogue via complex hindsight experience replay","authors":"Lu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Peng et al., 2017] Baolin Peng, Xiujun Li, Lihong Li, Jian- feng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. Composite task-completion dialogue policy learn- ing via hierarchical deep reinforcement learning. In Pro- ceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2231-2240, 2017.","title":"Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning","authors":"Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, Kam-Fai Wong","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Peng et al., 2018] Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Kam-Fai Wong. Deep dyna-q: Integrat- ing planning for task-completion dialogue policy learning. In Proceedings of the 56th Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Pa- pers), volume 1, pages 2182-2192, 2018.","title":"Deep dyna-q: Integrating planning for task-completion dialogue policy learning","authors":"Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, Kam-Fai Wong","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Schatzmann et al., 2007] Jost Schatzmann, Blaise Thom- son, Karl Weilhammer, Hui Ye, and Steve Young. Agenda- based user simulation for bootstrapping a pomdp dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Associ- ation for Computational Linguistics; Companion Volume, Short Papers, pages 149-152. Association for Computa- tional Linguistics, 2007.","title":"Agendabased user simulation for bootstrapping a pomdp dialogue system","authors":"Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, Steve Young","year":"2007","venue":"Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Serban et al., 2018] Iulian Vlad Serban, Chinnadhurai Sankar, Michael Pieper, Joelle Pineau, and Yoshua Bengio. The bottleneck simulator: A model-based deep reinforcement learning approach. arXiv preprint arXiv:1807.04723, 2018.","title":"The bottleneck simulator: A model-based deep reinforcement learning approach","authors":"Iulian Vlad Serban, Chinnadhurai Sankar, Michael Pieper, Joelle Pineau, Yoshua Bengio","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Taghipour and Kardan, 2008] Nima Taghipour and Ahmad Kardan. A hybrid web recommender system based on q- learning. In Proceedings of the 2008 ACM Symposium on Applied Computing, SAC '08, pages 1164-1168, New York, NY, USA, 2008. ACM.","title":"Nima Taghipour and Ahmad Kardan. A hybrid web recommender system based on qlearning","authors":"Kardan Taghipour","year":"2008","venue":"Proceedings of the 2008 ACM Symposium on Applied Computing, SAC '08","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wei et al., 2018] Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuanjing Huang, Kam-Fai Wong, and Xiangying Dai. Task-oriented dialogue sys- tem for automatic diagnosis. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), volume 2, pages 201- 207, 2018.","title":"Task-oriented dialogue system for automatic diagnosis","authors":"Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuanjing Huang, Kam-Fai Wong, Xiangying Dai","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yin et al., 2016] Dawei Yin, Yuening Hu, Jiliang Tang, Tim Daly, Mianwei Zhou, Hua Ouyang, Jianhui Chen, Chang- sung Kang, Hongbo Deng, Chikashi Nobata, et al. Rank- ing relevance in yahoo search. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 323-332. ACM, 2016.","title":"Ranking relevance in yahoo search","authors":"Yuening Dawei Yin, Jiliang Hu, Tim Tang, Mianwei Daly, Hua Zhou, Jianhui Ouyang, Changsung Chen, Hongbo Kang, Chikashi Deng, Nobata","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhao et al., 2018] Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. Deep rein- forcement learning for page-wise recommendations. In Proceedings of the 12th ACM Conference on Recom- mender Systems, RecSys '18, pages 95-103, New York, NY, USA, 2018. ACM.","title":"Deep reinforcement learning for page-wise recommendations","authors":"Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, Jiliang Tang","year":"2018","venue":"Proceedings of the 12th ACM Conference on Recommender Systems, RecSys '18","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zheng et al., 2018] Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. Drn: A deep reinforcement learning frame- work for news recommendation. In Proceedings of the 2018 World Wide Web Conference, WWW '18, pages 167-176, Republic and Canton of Geneva, Switzerland, 2018. International World Wide Web Conferences Steer- ing Committee.","title":"Drn: A deep reinforcement learning framework for news recommendation","authors":"Zheng","year":"2018","venue":"International World Wide Web Conferences Steering Committee","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}