{"type":"ScholarlyArticle","title":"Combinatorial Pure Exploration with Continuous and Separable Reward Functions and Its Applications","authors":[{"name":"Weiran Huang","slug":"weiran-huang","given":"Weiran","family":"Huang","affiliation":"Tsinghua University , Beijing , China"},{"name":"Jungseul Ok","slug":"jungseul-ok","given":"Jungseul","family":"Ok","affiliation":"KTH , Stockholm , Sweden"},{"name":"Liang Li","slug":"liang-li","given":"Liang","family":"Li","affiliation":"Ant Financial Group , Hangzhou , China"},{"name":"Wei Chen","slug":"wei-chen","given":"Wei","family":"Chen","affiliation":"Microsoft Research , Beijing , China"}],"abstract":"We study the Combinatorial Pure Exploration problem with Continuous and Separable reward functions (CPE-CS) in the stochastic multi-armed bandit setting. In a CPE-CS instance, we are given several stochastic arms with unknown distributions, as well as a collection of possible decisions. Each decision has a reward according to the distributions of arms. The goal is to identify the decision with the maximum reward, using as few arm samples as possible. The problem generalizes the combinatorial pure exploration problem with linear rewards, which has attracted significant attention in recent years. In this paper, we propose an adaptive learning algorithm for the CPE-CS problem, and analyze its sample complexity. In particular, we introduce a new hardness measure called the consistent optimality hardness, and give both the upper and lower bounds of sample complexity. Moreover, we give examples to demonstrate that our solution has the capacity to deal with non-linear reward functions.","doi":"10.24963/ijcai.2018/317","url":"https://ijcai.org/proceedings/2018/317","pdf":"https://ijcai.org/proceedings/2018/0317.pdf","markdown":"https://ijcai.org/proceedings/2018/317.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/317","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":317,"pages":{"first":2291,"last":2297},"keywords":["Machine Learning: Online Learning","Machine Learning: Learning Theory"],"videos":[],"references":[{"position":1,"raw":"[Audibert and Bubeck, 2010] Jean-Yves Audibert and Sébastien Bubeck. Best arm identification in multi-armed bandits. In COLT, pages 41-53, 2010.","title":"Best arm identification in multi-armed bandits","authors":"Bubeck ; Jean-Yves Audibert, Sébastien Audibert, Bubeck","year":"2010","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 2002a] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fis- cher. Finite-time analysis of the multiarmed bandit problem. Ma- chine learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Auer et al., 2002b] Peter Auer, Nicolo Cesa-Bianchi, Yoav Fre- und, and Robert E Schapire. The nonstochastic multiarmed ban- dit problem. SIAM journal on computing, 32(1):48-77, 2002.","title":"The nonstochastic multiarmed bandit problem","authors":"Auer","year":"2002","venue":"SIAM journal on computing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ballin and Barcaroli, 2013] Marco Ballin and Giulio Barcaroli. Joint determination of optimal stratification and sample alloca- tion using genetic algorithm. Survey Methodology, 39(2):369- 393, 2013.","title":"Joint determination of optimal stratification and sample allocation using genetic algorithm","authors":"Marco Ballin, Giulio Barcaroli","year":"2013","venue":"Survey Methodology","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bethel, 1986] James William Bethel. An optimum allocation algo- rithm for multivariate surveys. 1986.","title":"An optimum allocation algorithm for multivariate surveys","authors":"James William, Bethel","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bradley et al., 1977] Stephen Bradley, Arnoldo Hax, and Thomas Magnanti. Applied mathematical programming. 1977.","title":"Applied mathematical programming","authors":"Stephen Bradley, Arnoldo Hax, Thomas Magnanti","year":"1977","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bubeck et al., 2011] Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in finitely-armed and continuous-armed bandits. Theoretical Computer Science, 412(19):1832-1852, 2011.","title":"Pure exploration in finitely-armed and continuous-armed bandits","authors":"Sébastien Bubeck, Rémi Munos, Gilles Stoltz","year":"2011","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bubeck et al., 2012] Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends R in Machine Learn- ing, 5(1):1-122, 2012.","title":"Regret analysis of stochastic and nonstochastic multi-armed bandit problems","authors":"Sébastien Bubeck, Nicolo Cesa-Bianchi","year":"2012","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Bubeck et al., 2013] Séebastian Bubeck, Tengyao Wang, and Nitin Viswanathan. Multiple identifications in multi-armed bandits. In ICML, pages 258-265, 2013.","title":"Multiple identifications in multi-armed bandits","authors":"Bubeck","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cesa-Bianchi and Lugosi, 2012] Nicolo Cesa-Bianchi and Gábor Lugosi. Combinatorial bandits. Journal of Computer and Sys- tem Sciences, 78(5):1404-1422, 2012.","title":"Nicolo Cesa-Bianchi and Gábor Lugosi","authors":"Cesa-Bianchi, Lugosi","year":"2012","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chen et al., 2014] Shouyuan Chen, Tian Lin, Irwin King, Michael R Lyu, and Wei Chen. Combinatorial pure exploration of multi-armed bandits. In NIPS, pages 379-387, 2014.","title":"Combinatorial pure exploration of multi-armed bandits","authors":"Shouyuan Chen, Tian Lin, Irwin King, Wei Michael R Lyu, Chen","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chen et al., 2016a] Lijie Chen, Anupam Gupta, and Jian Li. Pure exploration of multi-armed bandit under matroid constraints. In COLT, pages 647-669, 2016.","title":"Pure exploration of multi-armed bandit under matroid constraints","authors":"Chen","year":"2016","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chen et al., 2016b] Wei Chen, Wei Hu, Fu Li, Jian Li, Yu Liu, and Pinyan Lu. Combinatorial multi-armed bandit with general re- ward functions. In NIPS, pages 1659-1667, 2016.","title":"Combinatorial multi-armed bandit with general reward functions","authors":"Chen","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Chen et al., 2016c] Wei Chen, Yajun Wang, Yang Yuan, and Qin- shi Wang. Combinatorial multi-armed bandit and its extension to probabilistically triggered arms. The Journal of Machine Learn- ing Research, 17(1):1746-1778, 2016.","title":"Combinatorial multi-armed bandit and its extension to probabilistically triggered arms","authors":"Chen","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Chen et al., 2017] Lijie Chen, Anupam Gupta, Jian Li, Mingda Qiao, and Ruosong Wang. Nearly optimal sampling algorithms for combinatorial pure exploration. arXiv:1706.01081, 2017.","title":"Nearly optimal sampling algorithms for combinatorial pure exploration","authors":"Lijie Chen, Anupam Gupta, Jian Li, Mingda Qiao, Ruosong Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Clifford and Sudbury, 1973] Peter Clifford and Aidan Sudbury. A model for spatial conflict. Biometrika, 60(3):581-588, 1973.","title":"Peter Clifford and Aidan Sudbury. A model for spatial conflict","authors":"Sudbury Clifford","year":"1973","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Combes et al., 2015] Richard Combes, Mohammad Sadegh Talebi Mazraeh Shahi, Alexandre Proutiere, et al. Combinatorial bandits revisited. In NIPS, pages 2116-2124, 2015.","title":"Combinatorial bandits revisited","authors":"Combes","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Gabillon et al., 2011] Victor Gabillon, Mohammad Ghavamzadeh, Alessandro Lazaric, and Sébastien Bubeck. Multi-bandit best arm identification. In NIPS, pages 2222-2230, 2011.","title":"Multi-bandit best arm identification","authors":"Victor Gabillon, Mohammad Ghavamzadeh, Alessandro Lazaric, Sébastien Bubeck","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Gabillon et al., 2012] Victor Gabillon, Mohammad Ghavamzadeh, and Alessandro Lazaric. Best arm identification: A unified ap- proach to fixed budget and fixed confidence. In NIPS, pages 3212-3220, 2012.","title":"Best arm identification: A unified approach to fixed budget and fixed confidence","authors":"Victor Gabillon, Mohammad Ghavamzadeh, Alessandro Lazaric","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Gai et al., 2012] Yi Gai, Bhaskar Krishnamachari, and Rahul Jain. Combinatorial network optimization with unknown variables: Multi-armed bandits with linear rewards and individual ob- servations. IEEE/ACM Transactions on Networking (TON), 20(5):1466-1478, 2012.","title":"Combinatorial network optimization with unknown variables: Multi-armed bandits with linear rewards and individual observations","authors":"Yi Gai, Bhaskar Krishnamachari, Rahul Jain","year":"2012","venue":"IEEE/ACM Transactions on Networking (TON)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Gopalan et al., 2014] Aditya Gopalan, Shie Mannor, and Yishay Mansour. Thompson sampling for complex online problems. In ICML, pages 100-108, 2014.","title":"Thompson sampling for complex online problems","authors":"Aditya Gopalan, Shie Mannor, Yishay Mansour","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Huang et al., 2017] Weiran Huang, Liang Li, and Wei Chen. Parti- tioned sampling of public opinions based on their social dynam- ics. In AAAI, pages 24-30, 2017.","title":"Partitioned sampling of public opinions based on their social dynamics","authors":"Weiran Huang, Liang Li, Wei Chen","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Huang et al., 2018] Weiran Huang, Jungseul Ok, Liang Li, and Wei Chen. Combinatorial pure exploration with continuous and separable reward functions and its applications (extended ver- sion). arXiv:1805.01685, 2018.","title":"Combinatorial pure exploration with continuous and separable reward functions and its applications (extended version)","authors":"Huang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kalyanakrishnan and Stone, 2010] Shivaram Kalyanakrishnan and Peter Stone. Efficient selection of multiple bandit arms: Theory and practice. In ICML, pages 511-518, 2010.","title":"Efficient selection of multiple bandit arms: Theory and practice","authors":"Shivaram Kalyanakrishnan, Peter Stone","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Kalyanakrishnan et al., 2012] Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone. PAC subset selection in stochastic multi-armed bandits. In ICML, volume 12, pages 655- 662, 2012.","title":"PAC subset selection in stochastic multi-armed bandits","authors":"Ambuj Shivaram Kalyanakrishnan, Peter Tewari, Peter Auer, Stone","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Kaufmann and Kalyanakrishnan, 2013] Emilie Kaufmann and Shivaram Kalyanakrishnan. Information complexity in bandit subset selection. In COLT, pages 228-251, 2013.","title":"Information complexity in bandit subset selection","authors":"Emilie Kaufmann, Shivaram Kalyanakrishnan","year":"2013","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Kveton et al., 2014] Branislav Kveton, Zheng Wen, Azin Ashkan, Hoda Eydgahi, and Brian Eriksson. Matroid bandits: Fast combi- natorial optimization with learning. In Proceedings of the Thirti- eth Conference on Uncertainty in Artificial Intelligence, UAI'14, pages 420-429, 2014.","title":"Matroid bandits: Fast combinatorial optimization with learning","authors":"Zheng Branislav Kveton, Azin Wen, Hoda Ashkan, Brian Eydgahi, Eriksson","year":"2014","venue":"Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI'14","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Kveton et al., 2015] Branislav Kveton, Zheng Wen, Azin Ashkan, and Csaba Szepesvari. Tight regret bounds for stochastic com- binatorial semi-bandits. In Artificial Intelligence and Statistics, pages 535-543, 2015.","title":"Tight regret bounds for stochastic combinatorial semi-bandits","authors":"Zheng Branislav Kveton, Azin Wen, Csaba Ashkan, Szepesvari","year":"2015","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Lai and Robbins, 1985] Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4-22, 1985.","title":"Asymptotically efficient adaptive allocation rules","authors":"Tze Leung, Lai, Herbert Robbins","year":"1985","venue":"Advances in applied mathematics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Li et al., 2015] Yanhua Li, Wei Chen, Yajun Wang, and Zhi-Li Zhang. Voter model on signed social networks. Internet Mathe- matics, 11(2):93-133, 2015.","title":"Voter model on signed social networks","authors":"Yanhua Li, Wei Chen, Yajun Wang, Zhi-Li Zhang","year":"2015","venue":"Internet Mathematics","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Soare et al., 2014] Marta Soare, Alessandro Lazaric, and Rémi Munos. Best-arm identification in linear bandits. In NIPS, pages 828-836, 2014.","title":"Best-arm identification in linear bandits","authors":"Marta Soare, Alessandro Lazaric, Rémi Munos","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yildiz et al., 2011] Ercan Yildiz, Daron Acemoglu, Asuman E Ozdaglar, Amin Saberi, and Anna Scaglione. Discrete opinion dynamics with stubborn agents. 2011.","title":"Discrete opinion dynamics with stubborn agents","authors":"Ercan Yildiz, Daron Acemoglu, Asuman E Ozdaglar, Amin Saberi, Anna Scaglione","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou et al., 2014] Yuan Zhou, Xi Chen, and Jian Li. Optimal PAC multiple arm identification with applications to crowdsourcing. In ICML, pages 217-225, 2014.","title":"Optimal PAC multiple arm identification with applications to crowdsourcing","authors":"Yuan Zhou, Xi Chen, Jian Li","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}