{"type":"ScholarlyArticle","title":"Bernoulli Rank-1 Bandits for Click Feedback","authors":[{"name":"Sumeet Katariya","slug":"sumeet-katariya","given":"Sumeet","family":"Katariya","affiliation":"University of Wisconsin-Madison ,"},{"name":"Branislav Kveton","slug":"branislav-kveton","given":"Branislav","family":"Kveton","affiliation":"Adobe Research ,"},{"name":"Csaba Szepesvári","slug":"csaba-szepesvari","given":"Csaba","family":"Szepesvári","affiliation":"University of Alberta ,"},{"name":"Claire Vernade","slug":"claire-vernade","given":"Claire","family":"Vernade","affiliation":"Telecom ParisTech"},{"name":"Zheng Wen","slug":"zheng-wen","given":"Zheng","family":"Wen","affiliation":"Adobe Research ,"}],"abstract":"The probability that a user will click a search result depends both on its relevance and its position on the results page. The position based model explains this behavior by ascribing to every item an attraction probability, and to every position an examination probability. To be clicked, a result must be both attractive and examined. The probabilities of an item-position pair being clicked thus form the entries of a rank-1 matrix. We propose the learning problem of a Bernoulli rank-1 bandit where at each step, the learning agent chooses a pair of row and column arms, and receives the product of their Bernoulli-distributed values as a reward. This is a special case of the stochastic rank-1 bandit problem considered in recent work that proposed an elimination based algorithm Rank1Elim, and showed that Rank1Elim's regret scales linearly with the number of rows and columns on \"benign\" instances. These are the instances where the minimum of the average row and column rewards mu is bounded away from zero. The issue with Rank1Elim is that it fails to be competitive with straightforward bandit strategies as mu tends to 0. In this paper we propose Rank1ElimKL, which replaces the crude confidence intervals of Rank1Elim with confidence intervals based on Kullback-Leibler (KL) divergences. With the help of a novel result concerning the scaling of KL divergences we prove that with this change, our algorithm will be competitive no matter the value of mu. Experiments with synthetic data confirm that on benign instances the performance of Rank1ElimKL is significantly better than that of even Rank1Elim. Similarly, experiments with models derived from real-data confirm that the improvements are significant across the board, regardless of whether the data is benign or not.","doi":"10.24963/ijcai.2017/278","url":"https://ijcai.org/proceedings/2017/278","pdf":"https://ijcai.org/proceedings/2017/0278.pdf","markdown":"https://ijcai.org/proceedings/2017/278.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/278","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":278,"pages":{"first":2001,"last":2007},"keywords":["Machine Learning: Learning Preferences or Rankings","Machine Learning: Structured Learning","Multidisciplinary Topics and Applications: Personalization and User Modeling","Machine Learning: Online Learning"],"videos":[],"references":[{"position":1,"raw":"[Auer and Ortner, 2010] Peter Auer and Ronald Ortner. UCB revisited: Improved regret bounds for the stochas- tic multi-armed bandit problem. Periodica Mathematica Hungarica, 61(1-2):55-65, 2010.","title":"UCB revisited: Improved regret bounds for the stochastic multi-armed bandit problem","authors":"Ortner Auer, Peter Auer, Ronald Ortner","year":"2010","venue":"Periodica Mathematica Hungarica","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47:235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chuklin et al., 2015] Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. Click Models for Web Search. Morgan & Claypool Publishers, 2015.","title":"Click Models for Web Search","authors":"Aleksandr Chuklin, Ilya Markov, Maarten De Rijke","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Combes et al., 2015] Richard Combes, Stefan Magureanu, Alexandre Proutiere, and Cyrille Laroche. Learning to rank: Regret lower bounds and efficient algorithms. In Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, 2015.","title":"Learning to rank: Regret lower bounds and efficient algorithms","authors":"Richard Combes, Stefan Magureanu, Alexandre Proutiere, Cyrille Laroche","year":"2015","venue":"Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Craswell et al., 2008] Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. An experimental comparison of click position-bias models. In Proceedings of the 1st ACM International Conference on Web Search and Data Min- ing, pages 87-94, 2008.","title":"An experimental comparison of click position-bias models","authors":"Nick Craswell, Onno Zoeter, Michael Taylor, Bill Ramsey","year":"2008","venue":"Proceedings of the 1st ACM International Conference on Web Search and Data Mining","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Garivier and Cappe, 2011] Aurelien Garivier and Olivier Cappe. The KL-UCB algorithm for bounded stochastic bandits and beyond. In Proceeding of the 24th Annual Conference on Learning Theory, pages 359-376, 2011.","title":"The KL-UCB algorithm for bounded stochastic bandits and beyond","authors":"Cappe Garivier, Aurelien Garivier, Olivier Cappe","year":"2011","venue":"Proceeding of the 24th Annual Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gentile et al., 2014] Claudio Gentile, Shuai Li, and Gio- vanni Zappella. Online clustering of bandits. In Pro- ceedings of the 31st International Conference on Machine Learning, pages 757-765, 2014.","title":"Online clustering of bandits","authors":"Claudio Gentile, Shuai Li, Giovanni Zappella","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Katariya et al., 2016] Sumeet Katariya, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. DCM bandits: Learn- ing to rank with multiple clicks. In Proceedings of the 33rd International Conference on Machine Learning, 2016.","title":"DCM bandits: Learning to rank with multiple clicks","authors":"Sumeet Katariya, Branislav Kveton, Csaba Szepesvari, Zheng Wen","year":"2016","venue":"Proceedings of the 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Katariya et al., 2017] Sumeet Katariya, Branislav Kveton, Csaba Szepesvari, Claire Vernade, and Zheng Wen. Stochastic rank-1 bandits. In AISTATS, 2017.","title":"Stochastic rank-1 bandits","authors":"Sumeet Katariya, Branislav Kveton, Csaba Szepesvari, Claire Vernade, Zheng Wen","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kawale et al., 2015] Jaya Kawale, Hung Bui, Branislav Kveton, Long Tran-Thanh, and Sanjay Chawla. Efficient Thompson sampling for online matrix-factorization rec- ommendation. In Advances in Neural Information Pro- cessing Systems 28, pages 1297-1305, 2015.","title":"Efficient Thompson sampling for online matrix-factorization recommendation","authors":"Kawale","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kveton et al., 2015a] Branislav Kveton, Csaba Szepesvari, Zheng Wen, and Azin Ashkan. Cascading bandits: Learn- ing to rank in the cascade model. In Proceedings of the 32nd International Conference on Machine Learning, 2015.","title":"Cascading bandits: Learning to rank in the cascade model","authors":"Kveton","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kveton et al., 2015b] Branislav Kveton, Zheng Wen, Azin Ashkan, and Csaba Szepesvari. Combinatorial cascading bandits. In Advances in Neural Information Processing Systems 28, pages 1450-1458, 2015.","title":"Combinatorial cascading bandits","authors":"Kveton","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lagree et al., 2016] Paul Lagree, Claire Vernade, and Olivier Cappe. Multiple-play bandits in the position-based model. CoRR, abs/1606.02448, 2016.","title":"Multiple-play bandits in the position-based model","authors":"Paul Lagree, Claire Vernade, Olivier Cappe","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2016] Shuai Li, Baoxiang Wang, Shengyu Zhang, and Wei Chen. Contextual combinatorial cascading ban- dits. In Proceedings of the 33rd International Conference on Machine Learning, pages 1245-1253, 2016.","title":"Contextual combinatorial cascading bandits","authors":"Shuai Li, Baoxiang Wang, Shengyu Zhang, Wei Chen","year":"2016","venue":"Proceedings of the 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Maillard and Mannor, 2014] Odalric-Ambrym Maillard and Shie Mannor. Latent bandits. In Proceedings of the 31st International Conference on Machine Learning, pages 136-144, 2014.","title":"Odalric-Ambrym Maillard and Shie Mannor. Latent bandits","authors":"Mannor Maillard","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Markov, 2014] Ilya Markov. Pyclick -click models for web search. https://github.com/markovi/PyClick, 2014.","title":"Ilya Markov. Pyclick -click models for web search","authors":"Markov","year":"2014","venue":null,"doi":null,"url":"https://github.com/markovi/PyClick,2014","internal_url":null},{"position":17,"raw":"[Richardson et al., 2007] Matthew Richardson, Ewa Domi- nowska, and Robert Ragno. Predicting clicks: Estimating the click-through rate for new ads. In Proceedings of the 16th International Conference on World Wide Web, pages 521-530, 2007.","title":"Predicting clicks: Estimating the click-through rate for new ads","authors":"Matthew Richardson, Ewa Dominowska, Robert Ragno","year":"2007","venue":"Proceedings of the 16th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sen et al., 2017] Rajat Sen, Karthikeyan Shanmugam, Mu- rat Kocaoglu, Alexandros G Dimakis, and Sanjay Shakkottai. Contextual bandits with latent confounders: An nmf approach. In AISTATS, 2017.","title":"Contextual bandits with latent confounders: An nmf approach","authors":"Rajat Sen, Karthikeyan Shanmugam, Murat Kocaoglu, Alexandros G Dimakis, Sanjay Shakkottai","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Thompson, 1933] William. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285- 294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"R William, Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yandex, 2013] Yandex personalized web search chal- lenge. https://www.kaggle.com/c/yandex-personalized- web-search-challenge, 2013.","title":"Yandex personalized web search challenge","authors":null,"year":"2013","venue":null,"doi":null,"url":"https://www.kaggle.com/c/yandex-personalized-web-search-challenge,2013","internal_url":null},{"position":21,"raw":"[Zhao et al., 2013] Xiaoxue Zhao, Weinan Zhang, and Jun Wang. Interactive collaborative filtering. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management, pages 1411-1420, 2013.","title":"Interactive collaborative filtering","authors":"Zhao","year":"2013","venue":"Proceedings of the 22nd ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zong et al., 2016] Shi Zong, Hao Ni, Kenny Sung, Nan Rosemary Ke, Zheng Wen, and Branislav Kveton. Cascading bandits for large-scale recommendation prob- lems. In Proceedings of the 32nd Conference on Uncer- tainty in Artificial Intelligence, 2016.","title":"Cascading bandits for large-scale recommendation problems","authors":"Hao Shi Zong, Kenny Ni, Nan Rosemary Sung, Zheng Ke, Branislav Wen, Kveton","year":"2016","venue":"Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}