{"type":"ScholarlyArticle","title":"Solving Continual Combinatorial Selection via Deep Reinforcement Learning","authors":[{"name":"Hyungseok Song","slug":"hyungseok-song","given":"Hyungseok","family":"Song","affiliation":"School of Electrical Engineering , KAIST , Daejeon , South Korea"},{"name":"Hyeryung Jang","slug":"hyeryung-jang","given":"Hyeryung","family":"Jang","affiliation":"Informatics , King's College London , London , United Kingdom"},{"name":"Hai H. Tran","slug":"hai-h-tran","given":"Hai H.","family":"Tran","affiliation":"School of Electrical Engineering , KAIST , Daejeon , South Korea"},{"name":"Se-eun Yoon","slug":"se-eun-yoon","given":"Se-eun","family":"Yoon","affiliation":"School of Electrical Engineering , KAIST , Daejeon , South Korea"},{"name":"Kyunghwan Son","slug":"kyunghwan-son","given":"Kyunghwan","family":"Son","affiliation":"School of Electrical Engineering , KAIST , Daejeon , South Korea"},{"name":"Donggyu Yun","slug":"donggyu-yun","given":"Donggyu","family":"Yun","affiliation":"Naver Corporation , Seongnam , South Korea"},{"name":"Hyoju Chung","slug":"hyoju-chung","given":"Hyoju","family":"Chung","affiliation":"Naver Corporation , Seongnam , South Korea"},{"name":"Yung Yi","slug":"yung-yi","given":"Yung","family":"Yi","affiliation":"School of Electrical Engineering , KAIST , Daejeon , South Korea"}],"abstract":"We consider the Markov Decision Process (MDP) of selecting a subset of items at each step, termed the Select-MDP (S-MDP). The large state and action spaces of S-MDPs make them intractable to solve with typical reinforcement learning (RL) algorithms especially when the number of items is huge. In this paper, we present a deep RL algorithm to solve this issue by adopting the following key ideas. First, we convert the original S-MDP into an Iterative Select-MDP (IS-MDP), which is equivalent to the S-MDP in terms of optimal actions. IS-MDP decomposes a joint action of selecting K items simultaneously into K iterative selections resulting in the decrease of actions at the expense of an exponential increase of states. Second, we overcome this state space explosion by exploiting a special symmetry in IS-MDPs with novel weight shared Q-networks, which provably maintain sufficient expressive power. Various experiments demonstrate that our approach works well even when the item space is large and that it scales to environments with item spaces different from those used in training.","doi":"10.24963/ijcai.2019/481","url":"https://ijcai.org/proceedings/2019/481","pdf":"https://ijcai.org/proceedings/2019/0481.pdf","markdown":"https://ijcai.org/proceedings/2019/481.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/481","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":481,"pages":{"first":3467,"last":3474},"keywords":["Uncertainty in AI: Markov Decision Processes","Machine Learning: Reinforcement Learning","Machine Learning: Recommender Systems","Heuristic Search and Game Playing: Combinatorial Search and Optimisation"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2018] Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi, and Le Song. Neural model-based rein- forcement learning for recommendation. arXiv preprint arXiv:1812.10613, 2018.","title":"Neural model-based reinforcement learning for recommendation","authors":"Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi, Le Song","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Dai et al., 2017] Hanjun Dai, Elias B Khalil, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial opti- mization algorithms over graphs. In NeurIPS, 2017.","title":"Learning combinatorial optimization algorithms over graphs","authors":"Dai","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dauphin et al., 2014] Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In NeurIPS, 2014.","title":"Identifying and attacking the saddle point problem in high-dimensional non-convex optimization","authors":"Razvan Yann N Dauphin, Caglar Pascanu, Kyunghyun Gulcehre, Surya Cho, Yoshua Ganguli, Bengio","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deudon et al., 2018] Michel Deudon, Pierre Cournut, Alexandre Lacoste, Yossiri Adulyasak, and Louis-Martin Rousseau. Learning heuristics for the tsp by policy gradient. In CPAIOR. Springer, 2018.","title":"Learning heuristics for the tsp by policy gradient","authors":"Pierre Michel Deudon, Alexandre Cournut, Yossiri Lacoste, Louis-Martin Adulyasak, Rousseau","year":"2018","venue":"CPAIOR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jason and Devon R Graham, 2018] Hartford Jason and Sia- mak Ravanbakhsh Devon R Graham, Kevin Leyton-Brown. Deep models of interactions across sets. In ICML, 2018.","title":"Deep models of interactions across sets","authors":"Jason, Devon R Graham, Jason, Siamak Ravanbakhsh, Devon R Graham, Kevin Leyton-Brown","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kawaguchi, 2016] Kenji Kawaguchi. Deep learning without poor local minima. In NeurIPS, 2016.","title":"Deep learning without poor local minima","authors":"Kenji Kawaguchi","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kipf and Welling, 2017] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kool et al., 2019] Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn to solve routing problems! In ICLR, 2019.","title":"Attention, learn to solve routing problems! In ICLR","authors":"Wouter Kool, Herke Van Hoof, Max Welling","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kushner and Whiting, 2004] Harold J Kushner and Philip A Whiting. Convergence of proportional-fair sharing algo- rithms under general conditions. IEEE Transactions on Wireless Communications, 3(4):1250-1259, 2004.","title":"Convergence of proportional-fair sharing algorithms under general conditions","authors":"Whiting ; Kushner, J Harold, Philip A Kushner, Whiting","year":"2004","venue":"IEEE Transactions on Wireless Communications","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Laurent and Brecht, 2018] Thomas Laurent and James Brecht. Deep linear networks with arbitrary loss: All local minima are global. In ICML, 2018.","title":"Deep linear networks with arbitrary loss: All local minima are global","authors":"Thomas Laurent, James Brecht","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] Shuai Li, Baoxiang Wang, Shengyu Zhang, and Wei Chen. Contextual combinatorial cascading bandits. In ICML, 2016.","title":"Contextual combinatorial cascading bandits","authors":"Shuai Li, Baoxiang Wang, Shengyu Zhang, Wei Chen","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Maes et al., 2009] Francis Maes, Ludovic Denoyer, and Patrick Gallinari. Structured prediction with reinforcement learning. Machine learning, 77(2-3):271, 2009.","title":"Structured prediction with reinforcement learning","authors":"Francis Maes, Ludovic Denoyer, Patrick Gallinari","year":"2009","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Maron et al., 2019] Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. arXiv preprint arXiv:1901.09342, 2019.","title":"On the universality of invariant networks","authors":"Maron","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Qi et al., 2017] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.","title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","authors":"Hao Charles R Qi, Kaichun Su, Leonidas J Mo, Guibas","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Qin et al., 2014] Lijing Qin, Shouyuan Chen, and Xiaoyan Zhu. Contextual combinatorial bandit and its application on diversified online recommendation. In ICDM. SIAM, 2014.","title":"Contextual combinatorial bandit and its application on diversified online recommendation","authors":"Lijing Qin, Shouyuan Chen, Xiaoyan Zhu","year":"2014","venue":"ICDM. SIAM","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ravanbakhsh et al., 2017a] Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Deep learning with sets and point clouds. In ICLR, workshop track, 2017.","title":"Deep learning with sets and point clouds","authors":"Ravanbakhsh","year":"2017","venue":"ICLR, workshop track","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ravanbakhsh et al., 2017b] Siamak Ravanbakhsh, Jeff Schneider, and Barnabás Póczos. Equivariance through parameter-sharing. In ICML, 2017.","title":"Equivariance through parameter-sharing","authors":"Ravanbakhsh","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ricci et al., 2015] Francesco Ricci, Lior Rokach, and Bracha Shapira. Recommender systems: Introduction and challenges. In Recommender systems handbook. Springer, 2015.","title":"Recommender systems: Introduction and challenges","authors":"Francesco Ricci, Lior Rokach, Bracha Shapira","year":"2015","venue":"Recommender systems handbook","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tampuu et al., 2017] Ardi Tampuu, Tambet Matiisen, Do- rian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4):e0172395, 2017.","title":"Multiagent cooperation and competition with deep reinforcement learning","authors":"Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, Raul Vicente","year":"2017","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Usunier et al., 2017] Nicolas Usunier, Gabriel Synnaeve, Zeming Lin, and Soumith Chintala. Episodic exploration for deep deterministic policies for starcraft micromanage- ment. In ICLR, 2017.","title":"Episodic exploration for deep deterministic policies for starcraft micromanagement","authors":"Usunier","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yarotsky, 2018] Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. arXiv preprint arXiv:1804.10306, 2018.","title":"Universal approximations of invariant maps by neural networks","authors":"Dmitry Yarotsky","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zaheer et al., 2017] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In NeurIPS, 2017.","title":"Deep sets","authors":"Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, Alexander J Smola","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zheng et al., 2018] Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. Drn: A deep reinforcement learning framework for news recommendation. In WWW, 2018.","title":"Drn: A deep reinforcement learning framework for news recommendation","authors":"Zheng","year":"2018","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zinkevich and Balch, 2001] Martin Zinkevich and Tucker Balch. Symmetry in markov decision processes and its implications for single agent and multi agent learning. In ICML, 2001.","title":"Symmetry in markov decision processes and its implications for single agent and multi agent learning","authors":"Balch Zinkevich, Martin Zinkevich, Tucker Balch","year":"2001","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}