{"type":"ScholarlyArticle","title":"Towards Hierarchical Policy Learning for Conversational Recommendation with Hypergraph-based Reinforcement Learning","authors":[{"name":"Sen Zhao","slug":"sen-zhao","given":"Sen","family":"Zhao","affiliation":"Cognitive Computing and Intelligent Information Processing (CCIIP) Laboratory , School of Computer Science and Technology , Huazhong University of Science and Technology; Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL)"},{"name":"Wei Wei","slug":"wei-wei","given":"Wei","family":"Wei","affiliation":"Cognitive Computing and Intelligent Information Processing (CCIIP) Laboratory , School of Computer Science and Technology , Huazhong University of Science and Technology; Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL)"},{"name":"Yifan Liu","slug":"yifan-liu","given":"Yifan","family":"Liu","affiliation":"Cognitive Computing and Intelligent Information Processing (CCIIP) Laboratory , School of Computer Science and Technology , Huazhong University of Science and Technology; Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL)"},{"name":"Ziyang Wang","slug":"ziyang-wang","given":"Ziyang","family":"Wang","affiliation":"Cognitive Computing and Intelligent Information Processing (CCIIP) Laboratory , School of Computer Science and Technology , Huazhong University of Science and Technology; Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL)"},{"name":"Wendi Li","slug":"wendi-li","given":"Wendi","family":"Li","affiliation":"Cognitive Computing and Intelligent Information Processing (CCIIP) Laboratory , School of Computer Science and Technology , Huazhong University of Science and Technology; Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL)"},{"name":"Xian-Ling Mao","slug":"xian-ling-mao","given":"Xian-Ling","family":"Mao","affiliation":"School of Computer Science and Technology , Beijing Institute of Technology"},{"name":"Shuai Zhu","slug":"shuai-zhu","given":"Shuai","family":"Zhu","affiliation":"Ant Group"},{"name":"Minghui Yang","slug":"minghui-yang","given":"Minghui","family":"Yang","affiliation":"Ant Group"},{"name":"Zujie Wen","slug":"zujie-wen","given":"Zujie","family":"Wen","affiliation":"Ant Group"}],"abstract":"Conversational recommendation systems (CRS) aim to timely and proactively acquire user dynamic preferred attributes through conversations for item recommendation. In each turn of CRS, there naturally have two decision-making processes with different roles that influence each other: 1) director, which is to select the follow-up option (i.e., ask or recommend) that is more effective for reducing the action space and acquiring user preferences; and 2) actor, which is to accordingly choose primitive actions (i.e., asked attribute or recommended item) to estimate the effectiveness of the director’s option. However, existing methods heavily rely on a unified decision-making module or heuristic rules, while neglecting to distinguish the roles of different decision procedures, as well as the mutual influences between them. To address this, we propose a novel Director-Actor Hierarchical Conversational Recommender (DAHCR), where the director selects the most effective option, followed by the actor accordingly choosing primitive actions that satisfy user preferences. Specifically, we develop a dynamic hypergraph to model user preferences and introduce an intrinsic motivation to train from weak supervision over the director. Finally, to alleviate the bad effect of model bias on the mutual influence between the director and actor, we model the director’s option by sampling from a categorical distribution. Extensive experiments demonstrate that DAHCR outperforms state-of-the-art methods.","doi":"10.24963/ijcai.2023/273","url":"https://ijcai.org/proceedings/2023/273","pdf":"https://ijcai.org/proceedings/2023/0273.pdf","markdown":"https://ijcai.org/proceedings/2023/273.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/273","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Data Mining","track":"Main Track","number":273,"pages":{"first":2459,"last":2467},"keywords":["Data Mining: DM: Recommender systems"],"videos":[],"references":[{"position":1,"raw":"[Battaglia et al., 2018] Peter W Battaglia, Jessica B Ham- rick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.","title":"Relational inductive biases, deep learning, and graph networks","authors":"Jessica B Peter W Battaglia, Victor Hamrick, Alvaro Bapst, Vinicius Sanchez-Gonzalez, Mateusz Zambaldi, Andrea Malinowski, David Tacchetti, Adam Raposo, Ryan Santoro, Faulkner","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellman and Kalaba, 1957] Richard Bellman and Robert Kalaba. On the role of dynamic programming in statistical communication theory. IRE Transactions on Information Theory, 3(3):197-203, 1957.","title":"On the role of dynamic programming in statistical communication theory","authors":"Kalaba Bellman, Richard Bellman, Robert Kalaba","year":"1957","venue":"IRE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bordes et al., 2013] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.","title":"Translating embeddings for modeling multi-relational data","authors":"Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, Oksana Yakhnenko","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019] Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. Towards knowledge-based recommender dialog sys- tem. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 1803-1813, 2019.","title":"Towards knowledge-based recommender dialog system","authors":"Chen","year":"2019","venue":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chentanez et al., 2004] Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated re- inforcement learning. Advances in neural information processing systems, 17, 2004.","title":"Intrinsically motivated reinforcement learning","authors":"Chentanez","year":"2004","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder ap- proaches. arXiv preprint arXiv:1409.1259, 2014.","title":"On the properties of neural machine translation: Encoder-decoder approaches","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Christakopoulou et al., 2016] Konstantina Chris- takopoulou, Filip Radlinski, and Katja Hofmann. Towards conversational recommender systems. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 815-824, 2016.","title":"Towards conversational recommender systems","authors":"Konstantina Christakopoulou, Filip Radlinski, Katja Hofmann","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Christakopoulou et al., 2018] Konstantina Chris- takopoulou, Alex Beutel, Rui Li, Sagar Jain, and Ed H Chi. Q&r: A two-stage approach toward inter- active recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 139-148, 2018.","title":"Q&r: A two-stage approach toward interactive recommendation","authors":"Konstantina Christakopoulou, Alex Beutel, Rui Li, Sagar Jain, Ed H Chi","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Deng et al., 2021] Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. Unified conversational recommenda- tion policy learning via graph-based reinforcement learn- ing. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1431-1441, 2021.","title":"Unified conversational recommenda-tion policy learning via graph-based reinforcement learning","authors":"Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, Wai Lam","year":"2021","venue":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Feng et al., 2019] Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural net- works. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3558-3565, 2019.","title":"Hypergraph neural networks","authors":"Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, Yue Gao","year":"2019","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han et al., 2018] Xu Han, Shulin Cao, Xin Lv, Yankai Lin, Zhiyuan Liu, Maosong Sun, and Juanzi Li. Openke: An open toolkit for knowledge embedding. In Proceedings of the 2018 conference on empirical methods in natural language processing: system demonstrations, pages 139- 144, 2018.","title":"Openke: An open toolkit for knowledge embedding","authors":"Xu Han, Shulin Cao, Xin Lv, Yankai Lin, Zhiyuan Liu, Maosong Sun, Juanzi Li","year":"2018","venue":"Proceedings of the 2018 conference on empirical methods in natural language processing: system demonstrations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2019] Jianwen Jiang, Yuxuan Wei, Yifan Feng, Jingxuan Cao, and Yue Gao. Dynamic hypergraph neural networks. In IJCAI, pages 2635-2641, 2019.","title":"Dynamic hypergraph neural networks","authors":"Jianwen Jiang, Yuxuan Wei, Yifan Feng, Jingxuan Cao, Yue Gao","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/366"},{"position":13,"raw":"[Kang et al., 2020] Dongyeop Kang, Anusha Balakrishnan, Pararth Shah, Paul Crook, Y Lan Boureau, and Jason We- ston. Recommendation as a communication game: Self- supervised bot-play for goal-oriented dialogue. In 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Nat- ural Language Processing, EMNLP-IJCNLP 2019, pages 1951-1961. Association for Computational Linguistics, 2020.","title":"Recommendation as a communication game: Selfsupervised bot-play for goal-oriented dialogue","authors":"Kang","year":"2019","venue":"Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lei et al., 2018] Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. Sequic- ity: Simplifying task-oriented dialogue systems with sin- gle sequence-to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 1437- 1447, 2018.","title":"Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures","authors":"Lei","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lei et al., 2020a] Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, and Tat-Seng Chua. Estimation-action-reflection: Towards deep interaction between conversational and recommender systems. In Proceedings of the 13th International Con- ference on Web Search and Data Mining, pages 304-312, 2020.","title":"Estimation-action-reflection: Towards deep interaction between conversational and recommender systems","authors":"Lei","year":"2020","venue":"Proceedings of the 13th International Conference on Web Search and Data Mining","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lei et al., 2020b] Wenqiang Lei, Gangyi Zhang, Xiangnan He, Yisong Miao, Xiang Wang, Liang Chen, and Tat-Seng Chua. Interactive path reasoning on graph for conversa- tional recommendation. In Proceedings of the 26th acm sigkdd international conference on knowledge discovery & data mining, pages 2073-2083, 2020.","title":"Interactive path reasoning on graph for conversational recommendation","authors":"Lei","year":"2020","venue":"Proceedings of the 26th acm sigkdd international conference on knowledge discovery & data mining","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2018] Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. Towards deep conversational recommendations. Advances in neural information processing systems, 31, 2018.","title":"Towards deep conversational recommendations","authors":"Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, Chris Pal","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2020] Zeming Liu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, and Ting Liu. Towards con- versational recommendation over multi-type dialogs. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1036-1049, 2020.","title":"Towards conversational recommendation over multi-type dialogs","authors":"Zeming Liu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, Ting Liu","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pei et al., 2022] Jiahuan Pei, Cheng Wang, and György Szarvas. Transformer uncertainty estimation with hier- archical stochastic attention. In Proceedings of the 36th AAAI Conference on Artificial Intelligence, 2022.","title":"Transformer uncertainty estimation with hierarchical stochastic attention","authors":"Pei","year":"2022","venue":"Proceedings of the 36th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Priyogi, 2019] Bilih Priyogi. Preference elicitation strategy for conversational recommender system. In Proceedings of the twelfth ACM international conference on web search and data mining, pages 824-825, 2019.","title":"Bilih Priyogi. Preference elicitation strategy for conversational recommender system","authors":"Priyogi","year":"2019","venue":"Proceedings of the twelfth ACM international conference on web search and data mining","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sun and Zhang, 2018] Yueming Sun and Yi Zhang. Conver- sational recommender system. In The 41st international acm sigir conference on research & development in infor- mation retrieval, pages 235-244, 2018.","title":"Conversational recommender system","authors":"Yueming Sun, Yi Zhang","year":"2018","venue":"The 41st international acm sigir conference on research & development in information retrieval","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tarvainen and Valpola, 2017] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- averaged consistency targets improve semi-supervised deep learning results. Advances in neural information pro- cessing systems, 30, 2017.","title":"Mean teachers are better role models: Weightaveraged consistency targets improve semi-supervised deep learning results","authors":"Valpola Tarvainen, Antti Tarvainen, Harri Valpola","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2022] Ziyang Wang, Wei Wei, Xian-Ling Mao, Guibing Guo, Pan Zhou, and Sheng Jiang. User-based net- work embedding for opinion spammer detection. Pattern Recognition, 125:108512, 2022.","title":"User-based network embedding for opinion spammer detection","authors":"Wang","year":"2022","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wei et al., 2023] Wei Wei, Sen Zhao, and Ding Zou. Rec- ommendation system: A survey and new perspectives. World Scientific Annual Review of Artificial Intelligence, 2023.","title":"Recommendation system: A survey and new perspectives","authors":"Wei Wei, Sen Zhao, Ding Zou","year":"2023","venue":"World Scientific Annual Review of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xia et al., 2022] Lianghao Xia, Chao Huang, Yong Xu, Ji- ashu Zhao, Dawei Yin, and Jimmy Huang. Hypergraph contrastive collaborative filtering. In Proceedings of the 45th International ACM SIGIR conference on research and development in information retrieval, pages 70-79, 2022.","title":"Hypergraph contrastive collaborative filtering","authors":"Lianghao Xia, Chao Huang, Yong Xu, Jiashu Zhao, Dawei Yin, Jimmy Huang","year":"2022","venue":"Proceedings of the 45th International ACM SIGIR conference on research and development in information retrieval","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xie et al., 2021] Zhihui Xie, Tong Yu, Canzhe Zhao, and Shuai Li. Comparison-based conversational recommender system with relative bandit feedback. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1400- 1409, 2021.","title":"Comparison-based conversational recommender system with relative bandit feedback","authors":"Zhihui Xie, Tong Yu, Canzhe Zhao, Shuai Li","year":"2021","venue":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2022] Yiming Zhang, Lingfei Wu, Qi Shen, Yitong Pang, Zhihua Wei, Fangli Xu, Bo Long, and Jian Pei. Multiple choice questions based multi-interest policy learning for conversational recommendation. In Proceed- ings of the ACM Web Conference 2022, pages 2153-2162, 2022.","title":"Multiple choice questions based multi-interest policy learning for conversational recommendation","authors":"Yiming Zhang, Lingfei Wu, Qi Shen, Yitong Pang, Zhihua Wei, Fangli Xu, Bo Long, Jian Pei","year":"2022","venue":"Proceedings of the ACM Web Conference 2022","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2022] Sen Zhao, Wei Wei, Ding Zou, and Xian- ling Mao. Multi-view intent disentangle graph networks for bundle recommendation. In Proceedings of the 36th AAAI Conference on Artificial Intelligence, 2022.","title":"Multi-view intent disentangle graph networks for bundle recommendation","authors":"Sen Zhao, Wei Wei, Ding Zou, Xianling Mao","year":"2022","venue":"Proceedings of the 36th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2020] Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. Im- proving conversational recommender systems via knowl- edge graph based semantic fusion. In Proceedings of the 26th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 1006-1014, 2020.","title":"Improving conversational recommender systems via knowledge graph based semantic fusion","authors":"Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, Jingsong Yu","year":"2020","venue":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}