{"type":"ScholarlyArticle","title":"Multi-Objective Generalized Linear Bandits","authors":[{"name":"Shiyin Lu","slug":"shiyin-lu","given":"Shiyin","family":"Lu","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Guanghui Wang","slug":"guanghui-wang","given":"Guanghui","family":"Wang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Yao Hu","slug":"yao-hu","given":"Yao","family":"Hu","affiliation":"YouKu Cognitive and Intelligent Lab , Alibaba Group , Beijing 100102 , China"},{"name":"Lijun Zhang","slug":"lijun-zhang","given":"Lijun","family":"Zhang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"}],"abstract":"In this paper, we study the multi-objective bandits (MOB) problem, where a learner repeatedly selects one arm to play and then receives a reward vector consisting of multiple objectives. MOB has found many real-world applications as varied as online recommendation and network routing. On the other hand, these applications typically contain contextual information that can guide the learning process which, however, is ignored by most of existing work. To utilize this information, we associate each arm with a context vector and assume the reward follows the generalized linear model (GLM). We adopt the notion of Pareto regret to evaluate the learner's performance and develop a novel algorithm for minimizing it. The essential idea is to apply a variant of the online Newton step to estimate model parameters, based on which we utilize the upper confidence bound (UCB) policy to construct an approximation of the Pareto front, and then uniformly at random choose one arm from the approximate Pareto front. Theoretical analysis shows that the proposed algorithm achieves an \\tilde O(d\\sqrt{T}) Pareto regret, where T is the time horizon and d is the dimension of contexts, which matches the optimal result for single objective contextual bandits problem. Numerical experiments demonstrate the effectiveness of our method.","doi":"10.24963/ijcai.2019/427","url":"https://ijcai.org/proceedings/2019/427","pdf":"https://ijcai.org/proceedings/2019/0427.pdf","markdown":"https://ijcai.org/proceedings/2019/427.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/427","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":427,"pages":{"first":3080,"last":3086},"keywords":["Machine Learning: Online Learning","Machine Learning: Time-series;Data Streams","Machine Learning Applications: Big data ; Scalability","Machine Learning: Recommender Systems"],"videos":[],"references":[{"position":1,"raw":"[Abbasi-Yadkori et al., 2011] Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for lin- ear stochastic bandits. In Advances in Neural Information Processing Systems 24, pages 2312-2320, 2011.","title":"Improved algorithms for linear stochastic bandits","authors":"Yasin Abbasi-Yadkori, Dávid Pál, Csaba Szepesvári","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agarwal et al., 2014] Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for con- textual bandits. In International Conference on Machine Learning, pages 1638-1646, 2014.","title":"Taming the monster: A fast and simple algorithm for contextual bandits","authors":"Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, Robert Schapire","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Auer et al., 2002a] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Auer et al., 2002b] Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic mul- tiarmed bandit problem. SIAM journal on computing, 32(1):48-77, 2002.","title":"The nonstochastic multiarmed bandit problem","authors":"Auer","year":"2002","venue":"SIAM journal on computing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Auer et al., 2016] Peter Auer, Chao-Kai Chiang, Ronald Ortner, and Madalina Drugan. Pareto front identification from stochastic bandit feedback. In International Confer- ence on Artificial Intelligence and Statistics, pages 939- 947, 2016.","title":"Pareto front identification from stochastic bandit feedback","authors":"Peter Auer, Chao-Kai Chiang, Ronald Ortner, Madalina Drugan","year":"2016","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Auer, 2002] Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.","title":"Using confidence bounds for exploitation-exploration trade-offs","authors":"Peter Auer","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bubeck and Cesa-Bianchi, 2012] Sébastien Bubeck and Ni- colo Cesa-Bianchi. Regret analysis of stochastic and non- stochastic multi-armed bandit problems. Foundations and Trends in Machine Learning, 5(1):1-122, 2012.","title":"Regret analysis of stochastic and nonstochastic multi-armed bandit problems","authors":"Cesa-Bianchi ; Colo Bubeck, Cesa-Bianchi","year":"2012","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bubeck et al., 2009] Sébastien Bubeck, Gilles Stoltz, Csaba Szepesvári, and Rémi Munos. Online optimization in X - armed bandits. In Advances in Neural Information Pro- cessing Systems 22, pages 201-208, 2009.","title":"Online optimization in Xarmed bandits","authors":"Sébastien Bubeck, Gilles Stoltz, Csaba Szepesvári, Rémi Munos","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dani et al., 2008] Varsha Dani, Thomas P. Hayes, and Sham M. Kakade. Stochastic linear optimization under bandit feedback. In Conference on Learning Theory, page 355, 2008.","title":"Stochastic linear optimization under bandit feedback","authors":"Varsha Dani, Thomas P Hayes, Sham M Kakade","year":"2008","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Drugan and Nowe, 2013] MM Drugan and A Nowe. De- signing multi-objective multi-armed bandits algorithms: a study. In International Joint Conference on Neural Net- works, pages 2352-2359, 2013.","title":"Designing multi-objective multi-armed bandits algorithms: a study","authors":"Nowe ; Mm Drugan, Drugan, Nowe","year":"2013","venue":"International Joint Conference on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Drugan and Nowé, 2014] Madalina M Drugan and Ann Nowé. Scalarization based pareto optimal set of arms iden- tification algorithms. In International Joint Conference on Neural Networks, pages 2690-2697, 2014.","title":"Scalarization based pareto optimal set of arms identification algorithms","authors":"M Madalina, Ann Drugan, Nowé","year":"2014","venue":"International Joint Conference on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dudik et al., 2011] Miroslav Dudik, Daniel Hsu, Satyen Kale, Nikos Karampatziakis, John Langford, Lev Reyzin, and Tong Zhang. Efficient optimal learning for contextual bandits. In Conference on Uncertainty in Artificial Intelli- gence, pages 169-178, 2011.","title":"Efficient optimal learning for contextual bandits","authors":"Dudik","year":"2011","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Filippi et al., 2010] Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. In Advances in Neural Information Processing Systems 23, pages 586-594, 2010.","title":"Parametric bandits: The generalized linear case","authors":"Sarah Filippi, Olivier Cappe, Aurélien Garivier, Csaba Szepesvári","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jun et al., 2017] Kwang-Sung Jun, Aniruddha Bhargava, Robert Nowak, and Rebecca Willett. Scalable general- ized linear bandits: Online computation and hashing. In Advances in Neural Information Processing Systems 30, pages 99-109, 2017.","title":"Scalable generalized linear bandits: Online computation and hashing","authors":"Kwang-Sung Jun, Aniruddha Bhargava, Robert Nowak, Rebecca Willett","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kleinberg et al., 2008] Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Multi-armed bandits in met- ric spaces. In Proceedings of the 40th Annual ACM Symposium on Theory of Computing, pages 681-690, 2008.","title":"Multi-armed bandits in metric spaces","authors":"Robert Kleinberg, Aleksandrs Slivkins, Eli Upfal","year":"2008","venue":"Proceedings of the 40th Annual ACM Symposium on Theory of Computing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Langford and Zhang, 2008] John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in Neural Information Processing Systems 21, pages 817-824, 2008.","title":"The epoch-greedy algorithm for multi-armed bandits with side information","authors":"Zhang Langford, John Langford, Tong Zhang","year":"2008","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to per- sonalized news article recommendation. In International Conference on World Wide Web, pages 661-670, 2010.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lu et al., 2010] Tyler Lu, Dávid Pál, and Martin Pál. Con- textual multi-armed bandits. In International Conference on Artificial Intelligence and Statistics, pages 485-492, 2010.","title":"Contextual multi-armed bandits","authors":"Tyler Lu, Dávid Pál, Martin Pál","year":"2010","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lu et al., 2019a] Shiyin Lu, Guanghui Wang, Yao Hu, and Lijun Zhang. Multi-objective generalized linear bandits. arXiv preprint arXiv:1905.12879, 2019.","title":"Multi-objective generalized linear bandits","authors":"Lu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lu et al., 2019b] Shiyin Lu, Guanghui Wang, Yao Hu, and Lijun Zhang. Optimal algorithms for Lipschitz bandits with heavy-tailed rewards. In Proceedings of the 36th International Conference on Machine Learning, pages 4154-4163, 2019.","title":"Optimal algorithms for Lipschitz bandits with heavy-tailed rewards","authors":"Lu","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nelder and Wedderburn, 1972] J. A. Nelder and R. W. M. Wedderburn. Generalized linear models. Journal of the Royal Statistical Society, Series A, General, 135:370-384, 1972.","title":"Generalized linear models","authors":"J A Nelder, R W M Wedderburn","year":"1972","venue":"Journal of the Royal Statistical Society, Series A, General","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rodriguez et al., 2012] Mario Rodriguez, Christian Posse, and Ethan Zhang. Multiple objective optimization in rec- ommender systems. In Proceedings of the sixth ACM conference on Recommender systems, pages 11-18. ACM, 2012.","title":"Multiple objective optimization in recommender systems","authors":"Mario Rodriguez, Christian Posse, Ethan Zhang","year":"2012","venue":"Proceedings of the sixth ACM conference on Recommender systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Slivkins, 2014] Aleksandrs Slivkins. Contextual bandits with similarity information. Journal of Machine Learning Research, 15(1):2533-2568, 2014.","title":"Aleksandrs Slivkins. Contextual bandits with similarity information","authors":"Slivkins","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Turgay et al., 2018] Eralp Turgay, Doruk Oner, and Cem Tekin. Multi-objective contextual bandit problem with similarity information. In International Conference on Ar- tificial Intelligence and Statistics, pages 1673-1681, 2018.","title":"Multi-objective contextual bandit problem with similarity information","authors":"Eralp Turgay, Doruk Oner, Cem Tekin","year":"2018","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yahyaa and Manderick, 2015] Saba Yahyaa and Bernard Manderick. Thompson sampling for multi-objective multi- armed bandits problem. In European Symposium on Arti- ficial Neural Networks, pages 47-52, 2015.","title":"Saba Yahyaa and Bernard Manderick. Thompson sampling for multi-objective multiarmed bandits problem","authors":"Manderick Yahyaa","year":"2015","venue":"European Symposium on Artificial Neural Networks","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2016] Lijun Zhang, Tianbao Yang, Rong Jin, Yichi Xiao, and Zhi-hua Zhou. Online stochastic linear op- timization under one-bit feedback. In International Con- ference on Machine Learning, pages 392-401, 2016.","title":"Online stochastic linear optimization under one-bit feedback","authors":"Lijun Zhang, Tianbao Yang, Rong Jin, Yichi Xiao, Zhi-Hua Zhou","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}