{"type":"ScholarlyArticle","title":"Cascading Non-Stationary Bandits: Online Learning to Rank in the Non-Stationary Cascade Model","authors":[{"name":"Chang Li","slug":"chang-li","given":"Chang","family":"Li","affiliation":"University of Amsterdam"},{"name":"Maarten de Rijke","slug":"maarten-de-rijke","given":"Maarten","family":"de Rijke","affiliation":"University of Amsterdam"}],"abstract":"Non-stationarity appears in many online applications such as web search and advertising. In this paper, we study the online learning to rank problem in a non-stationary environment where user preferences change abruptly at an unknown moment in time. We consider the problem of identifying the K most attractive items and propose cascading non-stationary bandits, an online learning variant of the cascading model, where a user browses a ranked list from top to bottom and clicks on the first attractive item. We propose two algorithms for solving this non-stationary problem: CascadeDUCB and CascadeSWUCB. We analyze their performance and derive gap-dependent upper bounds on the n-step regret of these algorithms. We also establish a lower bound on the regret for cascading non-stationary bandits and show that both algorithms match the lower bound up to a logarithmic factor. Finally, we evaluate their performance on a real-world web search click dataset.","doi":"10.24963/ijcai.2019/396","url":"https://ijcai.org/proceedings/2019/396","pdf":"https://ijcai.org/proceedings/2019/0396.pdf","markdown":"https://ijcai.org/proceedings/2019/396.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/396","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":396,"pages":{"first":2859,"last":2865},"keywords":["Machine Learning: Learning Preferences or Rankings","Multidisciplinary Topics and Applications: Information Retrieval","Machine Learning: Recommender Systems","Machine Learning: Online Learning"],"videos":[],"references":[{"position":1,"raw":"[Auer et al., 1995] Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert Schapire. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In FOCS, pages 322-331, 1995.","title":"Gambling in a rigged casino: The adversarial multi-armed bandit problem","authors":"Auer","year":"1995","venue":"FOCS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47:235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Besbes et al., 2014] Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non- stationary rewards. In NIPS, 2014.","title":"Stochastic multi-armed-bandit problem with nonstationary rewards","authors":"Omar Besbes, Yonatan Gur, Assaf Zeevi","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"David Blackwell. An analog of the mini- max theorem for vector payoffs. Pacific Journal of Mathe- matics, 6(1):1-8, November 1956.","title":"An analog of the minimax theorem for vector payoffs","authors":"David Blackwell","year":"1956","venue":"Pacific Journal of Mathematics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cesa-Bianchi and Lugosi, 2006] Nicolo Cesa-Bianchi and Gabor Lugosi. Prediction, Learning, and Games. Cam- bridge University Press, 2006.","title":"Nicolo Cesa-Bianchi and Gabor Lugosi","authors":"Cesa-Bianchi, Lugosi","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chapelle and Zhang, 2009] Olivier Chapelle and Ya Zhang. A dynamic Bayesian network click model for web search ranking. In WWW, pages 1-10, 2009.","title":"A dynamic Bayesian network click model for web search ranking","authors":"Chapelle, Zhang, Olivier Chapelle, Ya Zhang","year":"2009","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chuklin et al., 2015] Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. Click Models for Web Search. Morgan & Claypool, 2015.","title":"Click Models for Web Search","authors":"Aleksandr Chuklin, Ilya Markov, Maarten De Rijke","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Craswell et al., 2008] Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. An experimental comparison of click position-bias models. In WSDM, pages 87-94, 2008.","title":"An experimental comparison of click position-bias models","authors":"Nick Craswell, Onno Zoeter, Michael Taylor, Bill Ramsey","year":"2008","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Garivier and Moulines, 2008] Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for non-stationary bandit problems. arXiv preprint arXiv:0805.3415, 2008.","title":"Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for non-stationary bandit problems","authors":"Moulines Garivier","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Garivier and Moulines, 2011] Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for switch- ing bandit problems. In ALT, pages 174-188, 2011.","title":"On upper-confidence bound policies for switching bandit problems","authors":"Moulines Garivier, Aurélien Garivier, Eric Moulines","year":"2011","venue":"ALT","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Guo et al., 2009] Fan Guo, Chao Liu, and Yi Min Wang. Efficient multiple-click models in web search. In WSDM, pages 124-131, 2009.","title":"Efficient multiple-click models in web search","authors":"Fan Guo, Chao Liu, Yi Min, Wang","year":"2009","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jagerman et al., 2019] Rolf Jagerman, Ilya Markov, and Maarten de Rijke. When people change their mind: Off- policy evaluation in non-stationary recommendation envi- ronments. In WSDM, pages 447-455, 2019.","title":"When people change their mind: Offpolicy evaluation in non-stationary recommendation environments","authors":"Rolf Jagerman, Ilya Markov, Maarten De Rijke","year":"2019","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Joachims, 2002] Thorsten Joachims. Optimizing search en- gines using clickthrough data. In SIGKDD, pages 133-142, 2002.","title":"Optimizing search engines using clickthrough data","authors":"Thorsten Joachims","year":"2002","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Katariya et al., 2016] Sumeet Katariya, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. DCM bandits: Learning to rank with multiple clicks. In ICML, pages 1215-1224, 2016.","title":"DCM bandits: Learning to rank with multiple clicks","authors":"Sumeet Katariya, Branislav Kveton, Csaba Szepesvari, Zheng Wen","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kveton et al., 2014] Branislav Kveton, Zheng Wen, Azin Ashkan, Hoda Eydgahi, and Brian Eriksson. Matroid ban- dits: Fast combinatorial optimization with learning. arXiv preprint arXiv:1403.5045, 2014. [Kveton et al., 2015] Branislav Kveton, Csaba Szepesvari, Zheng Wen, and Azin Ashkan. Cascading bandits: Learn- ing to rank in the cascade model. In ICML, pages 767-776, 2015.","title":"Matroid bandits: Fast combinatorial optimization with learning","authors":"Zheng Branislav Kveton, Azin Wen, Hoda Ashkan, Brian Eydgahi, Eriksson ; Kveton","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lagrée et al., 2016] Paul Lagrée, Claire Vernade, and Olivier Cappe. Multiple-play bandits in the position-based model. In NIPS, 2016.","title":"Multiple-play bandits in the position-based model","authors":"Paul Lagrée, Claire Vernade, Olivier Cappe","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2018] Chang Li, Ilya Markov, Maarten de Rijke, and Masrour Zoghi. Merge double Thompson sampling for large scale online ranker evaluation. arXiv preprint arXiv:1812.04412, 2018.","title":"Merge double Thompson sampling for large scale online ranker evaluation","authors":"Chang Li, Ilya Markov, Maarten De Rijke, Masrour Zoghi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2019] Chang Li, Branislav Kveton, Tor Lattimore, Ilya Markov, Maarten de Rijke, Csaba Szepesvári, and Mas- rour Zoghi. BubbleRank: Safe online learning to rerank. In UAI, 2019.","title":"BubbleRank: Safe online learning to rerank","authors":"Chang Li, Branislav Kveton, Tor Lattimore, Ilya Markov, Maarten De Rijke, Csaba Szepesvári, Masrour Zoghi","year":"2019","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2018] Fang Liu, Joohyun Lee, and Ness Shroff. A change-detection based framework for piecewise-stationary multi-armed bandit problem. In AAAI, pages 3651-3658, 2018.","title":"A change-detection based framework for piecewise-stationary multi-armed bandit problem","authors":"Fang Liu, Joohyun Lee, Ness Shroff","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu, 2009] Tie-Yan Liu. Learning to rank for information retrieval. Foundations and Trends in Information Retrieval, 3(3):225-331, 2009.","title":"Learning to rank for information retrieval","authors":"Tie-Yan Liu, Liu","year":"2009","venue":"Foundations and Trends in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Oosterhuis and de Rijke, 2018] Harrie Oosterhuis and Maarten de Rijke. Differentiable unbiased online learning to rank. In CIKM, pages 1293-1302, 2018.","title":"Differentiable unbiased online learning to rank","authors":"Oosterhuis, Harrie De Rijke, Maarten Oosterhuis, De Rijke","year":"2018","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pereira et al., 2018] Fabíola SF Pereira, João Gama, Sandra de Amo, and Gina MB Oliveira. On analyzing user pref- erence dynamics with temporal social networks. Machine Learning, 107(11):1745-1773, 2018.","title":"On analyzing user preference dynamics with temporal social networks","authors":"Pereira","year":"2018","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Qin et al., 2010] Tao Qin, Tie-Yan Liu, Jun Xu, and Hang Li. Letor: A benchmark collection for research on learning to rank for information retrieval. Information Retrieval, 13(4):346-374, 2010.","title":"Letor: A benchmark collection for research on learning to rank for information retrieval","authors":"Tao Qin, Tie-Yan Liu, Jun Xu, Hang Li","year":"2010","venue":"Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Radlinski et al., 2008] Filip Radlinski, Robert Kleinberg, and Thorsten Joachims. Learning diverse rankings with multi-armed bandits. In ICML, pages 784-791, 2008.","title":"Learning diverse rankings with multi-armed bandits","authors":"Filip Radlinski, Robert Kleinberg, Thorsten Joachims","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Slivkins and Upfal, 2008] Aleksandrs Slivkins and Eli Upfal. Adapting to a changing environment: the brownian restless bandits. In COLT, pages 343-354, 2008.","title":"Adapting to a changing environment: the brownian restless bandits","authors":"Upfal Slivkins, Aleksandrs Slivkins, Eli Upfal","year":"2008","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2018] Qingyun Wu, Naveen Iyer, and Hongning Wang. Learning contextual bandits in a non-stationary environment. In SIGIR, pages 495-504, 2018.","title":"Learning contextual bandits in a non-stationary environment","authors":"Qingyun Wu, Naveen Iyer, Hongning Wang","year":"2018","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu and Mannor, 2009] Jia Yuan Yu and Shie Mannor. Piecewise-stationary bandit problems with side observa- tions. In ICML, 2009.","title":"Piecewise-stationary bandit problems with side observations","authors":"Jia Yuan, Yu, Shie Mannor","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zoghi et al., 2016] Masrour Zoghi, Tomáš Tunys, Lihong Li, Damien Jose, Junyan Chen, Chun Ming Chin, and Maarten de Rijke. Click-based hot fixes for underperforming torso queries. In SIGIR, pages 195-204, 2016.","title":"Click-based hot fixes for underperforming torso queries","authors":"Zoghi","year":"2016","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zoghi et al., 2017] Masrour Zoghi, Tomáš Tunys, Moham- mad Ghavamzadeh, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. Online learning to rank in stochastic click models. In ICML, pages 4199-4208, 2017.","title":"Online learning to rank in stochastic click models","authors":"Zoghi","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}