{"type":"ScholarlyArticle","title":"Cost-aware Cascading Bandits","authors":[{"name":"Ruida Zhou","slug":"ruida-zhou","given":"Ruida","family":"Zhou","affiliation":"University of Science and Technology of China"},{"name":"Chao Gan","slug":"chao-gan","given":"Chao","family":"Gan","affiliation":"The Pennsylvania State University"},{"name":"Jing Yang","slug":"jing-yang","given":"Jing","family":"Yang","affiliation":"The Pennsylvania State University"},{"name":"Cong Shen","slug":"cong-shen","given":"Cong","family":"Shen","affiliation":"University of Science and Technology of China"}],"abstract":"In this paper, we propose a cost-aware cascading bandits model, a new variant of multi-armed bandits with cascading feedback, by considering the random cost of pulling arms. In each step, the learning agent chooses an {\\it ordered} list of items and \\congr{examines} them sequentially, until certain stopping condition is satisfied. Our objective is then to maximize the expected {\\it net reward} in each step, i.e., the reward obtained in each step minus the total cost incurred in examining the items, by deciding the ordered list of items, as well as when to stop examination. We study both the offline and online settings, depending on whether the state and cost statistics of the items are known beforehand. For the offline setting, we show that the Unit Cost Ranking with Threshold 1 (UCR-T1) policy is optimal. For the online setting, we propose a Cost-aware Cascading Upper Confidence Bound (CC-UCB) algorithm, and show that the cumulative regret scales in $O(\\log T)$. We also provide a lower bound for all $\\alpha$-consistent policies, which scales in $\\Omega(\\log T)$ and matches our upper bound. The performance of the CC-UCB algorithm is evaluated with both synthetic and real-world data.","doi":"10.24963/ijcai.2018/448","url":"https://ijcai.org/proceedings/2018/448","pdf":"https://ijcai.org/proceedings/2018/0448.pdf","markdown":"https://ijcai.org/proceedings/2018/448.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/448","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":448,"pages":{"first":3228,"last":3234},"keywords":["Machine Learning: Online Learning","Uncertainty in AI: Sequential Decision Making","Machine Learning: Cost-Sensitive Learning"],"videos":[],"references":[{"position":1,"raw":"[Anantharam et al., 1987] Venkatachalam Anantharam, Pravin Varaiya, and Jean Walrand. Asymptotically effi- cient allocation rules for the multiarmed bandit problem with multiple plays-part i: Iid rewards. IEEE Transactions on Automatic Control, 32(11):968-976, 1987.","title":"Asymptotically effi-cient allocation rules for the multiarmed bandit problem with multiple plays-part i: Iid rewards","authors":"Pravin Venkatachalam Anantharam, Jean Varaiya, Walrand","year":"1987","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Audibert and Bubeck, 2010] Jean-Yves Audibert and Sébastien Bubeck. Best Arm Identification in Multi- Armed Bandits. In the 23th Conference on Learning Theory, Haifa, Israel, June 2010.","title":"Best Arm Identification in Multi-Armed Bandits","authors":"Bubeck ; Jean-Yves Audibert, Sébastien Audibert, Bubeck","year":"2010","venue":"the 23th Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Badanidiyuru et al., 2013] Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks. In Proceedings of the IEEE 54th Annual Symposium on Foundations of Computer Science, pages 207-216, 2013.","title":"Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks","authors":"Badanidiyuru","year":"2013","venue":"Proceedings of the IEEE 54th Annual Symposium on Foundations of Computer Science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bubeck et al., 2009] Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in multi-armed bandits problems. In Proceedings of the 20th International Con- ference on Algorithmic Learning Theory, pages 23-37, 2009.","title":"Pure exploration in multi-armed bandits problems","authors":"Sébastien Bubeck, Rémi Munos, Gilles Stoltz","year":"2009","venue":"Proceedings of the 20th International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Burnetas and Kanavetas, 2012] Apostolos Burnetas and Odysseas Kanavetas. Adaptive policies for sequential sampling under incomplete information and a cost con- straint. Applications of Mathematics and Informatics in Military Science, page 97-112, 2012.","title":"Adaptive policies for sequential sampling under incomplete information and a cost constraint","authors":"Apostolos Burnetas, Odysseas Kanavetas","year":"2012","venue":"Applications of Mathematics and Informatics in Military Science","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Burnetas et al., 2017] Apostolos Burnetas, Odysseas Kanavetas, and Michael N Katehakis. Asymptotically optimal multi-armed bandit policies under a cost con- straint. Probability in the Engineering and Informational Sciences, 31(3):284-310, 2017.","title":"Asymptotically optimal multi-armed bandit policies under a cost constraint","authors":"Apostolos Burnetas, Odysseas Kanavetas, Michael N Katehakis","year":"2017","venue":"Probability in the Engineering and Informational Sciences","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ding et al., 2013] Wenkui Ding, Tao Qiny, Xu-Dong Zhang, and Tie-Yan Liu. Multi-armed bandit with bud- get constraint and variable costs. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intel- ligence, pages 232-238, 2013.","title":"Multi-armed bandit with budget constraint and variable costs","authors":"Ding","year":"2013","venue":"Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guha and Munagala, 2007] Sudipto Guha and Kamesh Mu- nagala. Approximation algorithms for budgeted learning problems. In Proceedings of the Thirty-ninth Annual ACM Symposium on Theory of Computing, pages 104-113, New York, NY, USA, 2007. ACM. [Int, 2011] \"Internet mathematics\". https: //academy.yandex.ru/events/data_ analysis/relpred2011/, 2011.","title":"Approximation algorithms for budgeted learning problems","authors":"Sudipto Guha, Kamesh Munagala","year":"2007","venue":"Proceedings of the Thirty-ninth Annual ACM Symposium on Theory of Computing","doi":null,"url":"https://academy.yandex.ru/events/data_analysis/relpred2011/,2011","internal_url":null},{"position":10,"raw":"[Kalathil et al., 2014] D. Kalathil, N. Nayyar, and R. Jain. Decentralized learning for multiplayer multiarmed ban- dits. IEEE Transactions on Information Theory, 60(4):2331-2345, April 2014.","title":"Decentralized learning for multiplayer multiarmed bandits","authors":"D Kalathil, N Nayyar, R Jain","year":"2014","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Komiyama et al., 2015] Junpei Komiyama, Junya Honda, and Hiroshi Nakagawa. Optimal regret analysis of thomp- son sampling in stochastic multi-armed bandit problem with multiple plays. In Proceedings of the 32nd Inter- national Conference on Machine Learning, pages 1152- 1161, Lille, France, Jul 2015. Kveton et al., 2015a] Branislav Kveton, Csaba Szepesvari, Zheng Wen, and Azin Ashkan. Cascading bandits: Learn- ing to rank in the cascade model. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 767-776, 2015.","title":"Optimal regret analysis of thompson sampling in stochastic multi-armed bandit problem with multiple plays","authors":"Komiyama","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning (ICML-15)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kveton et al., 2015b] Branislav Kveton, Zheng Wen, Azin Ashkan, and Csaba Szepesvari. Combinatorial cascading bandits. In Advances in Neural Information Processing Systems, pages 1450-1458, 2015.","title":"Combinatorial cascading bandits","authors":"Kveton","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Radlinski et al., 2008] Filip Radlinski, Robert Kleinberg, and Thorsten Joachims. Learning diverse rankings with multi-armed bandits. In In 25th Intl. Conf. on Machine Learning (ICML), pages 784-791, 2008.","title":"Learning diverse rankings with multi-armed bandits","authors":"Filip Radlinski, Robert Kleinberg, Thorsten Joachims","year":"2008","venue":"25th Intl. Conf. on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Slivkins et al., 2013] Aleksandrs Slivkins, Filip Radlinski, and Sreenivas Gollapudi. Ranked bandits in metric spaces: Learning diverse rankings over large document collec- tions. J. Mach. Learn. Res., 14(1):399-436, February 2013.","title":"Ranked bandits in metric spaces: Learning diverse rankings over large document collections","authors":"Aleksandrs Slivkins, Filip Radlinski, Sreenivas Gollapudi","year":"2013","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Streeter and Golovin, 2008] Matthew Streeter and Daniel Golovin. An online algorithm for maximizing submodular functions. In Advances in Neural Information Processing Systems 21, pages 1577-1584. 2008.","title":"An online algorithm for maximizing submodular functions","authors":"Golovin Streeter, Matthew Streeter, Daniel Golovin","year":"2008","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tran-Thanh et al., 2010] Long Tran-Thanh, Archie C. Chapman, Enrique Munoz de Cote, Alex Rogers, and Nicholas R. Jennings. Epsilon-first policies for budget-limited multi-armed bandits. In AAAI, 2010.","title":"Epsilon-first policies for budget-limited multi-armed bandits","authors":"Tran-Thanh","year":"2010","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tran-Thanh et al., 2012] Long Tran-Thanh, Archie Chap- man, Alex Rogers, and Nicholas R. Jennings. Knapsack based optimal policies for budget-limited multi-armed bandits. In Proceedings of the Twenty-Sixth AAAI Con- ference on Artificial Intelligence, pages 1134-1140, 2012.","title":"Knapsack based optimal policies for budget-limited multi-armed bandits","authors":"Tran-Thanh","year":"2012","venue":"Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xia et al., 2015] Yingce Xia, Haifang Li, Tao Qin, Nenghai Yu, and Tie-Yan Liu. Thompson sampling for budgeted multi-armed bandits. In Proceedings of the 24th Interna- tional Conference on Artificial Intelligence, pages 3960- 3966, 2015.","title":"Thompson sampling for budgeted multi-armed bandits","authors":"Yingce Xia, Haifang Li, Tao Qin, Nenghai Yu, Tie-Yan Liu","year":"2015","venue":"Proceedings of the 24th International Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/556"},{"position":19,"raw":"[Xia et al., 2016a] Yingce Xia, Wenkui Ding, Xu-Dong Zhang, Nenghai Yu, and Tao Qin. Budgeted bandit prob- lems with continuous random costs. In Geoffrey Holmes and Tie-Yan Liu, editors, Asian Conference on Machine Learning, volume 45, pages 317-332, Hong Kong, Nov 2016.","title":"Budgeted bandit problems with continuous random costs","authors":"Xia","year":"2016","venue":"Asian Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xia et al., 2016b] Yingce Xia, Tao Qin, Weidong Ma, Nenghai Yu, and Tie-Yan Liu. Budgeted multi-armed ban- dits with multiple plays. In Proceedings of the Twenty- Fifth International Joint Conference on Artificial Intelli- gence, pages 2210-2216. AAAI Press, 2016.","title":"Budgeted multi-armed bandits with multiple plays","authors":"Xia","year":"2016","venue":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2016/315"},{"position":21,"raw":"[Zong et al., 2016] Shi Zong, Hao Ni, Kenny Sung, Nan Rosemary Ke, Zheng Wen, and Branislav Kveton. Cascading bandits for large-scale recommendation prob- lems. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, pages 835-844, 2016.","title":"Cascading bandits for large-scale recommendation problems","authors":"Hao Shi Zong, Kenny Ni, Nan Rosemary Sung, Zheng Ke, Branislav Wen, Kveton","year":"2016","venue":"Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}