{"type":"ScholarlyArticle","title":"Advancements in Dueling Bandits","authors":[{"name":"Yanan Sui","slug":"yanan-sui","given":"Yanan","family":"Sui","affiliation":"Microsoft"},{"name":"Masrour Zoghi","slug":"masrour-zoghi","given":"Masrour","family":"Zoghi","affiliation":"Microsoft"},{"name":"Katja Hofmann","slug":"katja-hofmann","given":"Katja","family":"Hofmann","affiliation":"Microsoft"},{"name":"Yisong Yue","slug":"yisong-yue","given":"Yisong","family":"Yue","affiliation":"Microsoft"}],"abstract":"The dueling bandits problem is an online learning framework where learning happens ``on-the-fly'' through preference feedback, i.e., from comparisons between a pair of actions. Unlike conventional online learning settings that require absolute feedback for each action, the dueling bandits framework assumes only the presence of (noisy) binary feedback about the relative quality of each pair of actions. The dueling bandits problem is well-suited for modeling settings that elicit subjective or implicit human feedback, which is typically more reliable in preference form. \r\nIn this survey, we review recent  results in the theories, algorithms, and applications of the dueling bandits problem.\r\nAs an emerging domain, the theories and algorithms of dueling bandits have been intensively studied during the past few years. We provide an overview of recent advancements, including algorithmic advances and applications.\r\nWe discuss extensions to standard problem formulation and novel application areas, highlighting key open research questions in our discussion.","doi":"10.24963/ijcai.2018/776","url":"https://ijcai.org/proceedings/2018/776","pdf":"https://ijcai.org/proceedings/2018/0776.pdf","markdown":"https://ijcai.org/proceedings/2018/776.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/776","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Survey track","track":"Survey track","number":776,"pages":{"first":5502,"last":5510},"keywords":["Machine Learning: Online Learning","Machine Learning: Recommender Systems","Machine Learning: Active Learning","Machine Learning: Learning Preferences or Rankings"],"videos":[],"references":[{"position":1,"raw":"[Ailon et al., 2014] Nir Ailon, Zohar Karnin, and Thorsten Joachims. Reducing dueling bandits to cardinal bandits. In ICML, 2014.","title":"Reducing dueling bandits to cardinal bandits","authors":"Zohar Nir Ailon, Thorsten Karnin, Joachims","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic mul- tiarmed bandit problem. SIAM Journal on Computing, 2002.","title":"The nonstochastic multiarmed bandit problem","authors":"Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, Robert E Schapire","year":"2002","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Balsubramani et al., 2016] Akshay Balsubramani, Zohar Karnin, Robert E. Schapire, and Masrour Zoghi. Instance- dependent regret bounds for dueling bandits. In COLT, 2016.","title":"Instancedependent regret bounds for dueling bandits","authors":"Akshay Balsubramani, Zohar Karnin, Robert E Schapire, Masrour Zoghi","year":"2016","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brost et al., 2016] Brian Brost, Yevgeny Seldin, Ingemar J Cox, and Christina Lioma. Multi-dueling bandits and their application to online ranker evaluation. In CIKM, 2016.","title":"Multi-dueling bandits and their application to online ranker evaluation","authors":"Brian Brost, Yevgeny Seldin, Ingemar J Cox, Christina Lioma","year":"2016","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bubeck and Cesa-Bianchi, 2012] Sébastien Bubeck and Nicolò Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning, 2012.","title":"Regret analysis of stochastic and nonstochastic multi-armed bandit problems","authors":"Cesa-Bianchi ; Bubeck, Sébastien Bubeck, Nicolò Cesa-Bianchi","year":"2012","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Busa-Fekete and Hüllermeier, 2014] Róbert Busa-Fekete and Eyke Hüllermeier. A survey of preference-based online learning with bandit algorithms. In ALT. Springer, 2014.","title":"A survey of preference-based online learning with bandit algorithms","authors":"Róbert Busa-Fekete, Eyke Hüllermeier","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cesa-Bianchi and Lugosi, 2006] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cam- bridge university press, 2006.","title":"Nicolo Cesa-Bianchi and Gábor Lugosi","authors":"Cesa-Bianchi, Lugosi","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cesa-Bianchi et al., 2006] Nicolo Cesa-Bianchi, Gábor Lu- gosi, and Gilles Stoltz. Regret minimization under par- tial monitoring. Mathematics of Operations Research, 31(3):562-580, 2006.","title":"Regret minimization under partial monitoring","authors":"Cesa-Bianchi","year":"2006","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chapelle et al., 2012] Olivier Chapelle, Thorsten Joachims, Filip Radlinski, and Yisong Yue. Large-scale validation and analysis of interleaved search evaluation. ACM Trans- actions on Information Systems, 2012.","title":"Large-scale validation and analysis of interleaved search evaluation","authors":"Olivier Chapelle, Thorsten Joachims, Filip Radlinski, Yisong Yue","year":"2012","venue":"ACM Transactions on Information Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen and Frazier, 2017] Bangrui Chen and Peter I. Frazier. Dueling bandits with weak regret. In ICML, 2017.","title":"Dueling bandits with weak regret","authors":"Chen, Frazier ;, Bangrui Chen, Peter I Frazier","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dudík et al., 2015] Miroslav Dudík, Katja Hofmann, Robert E Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In COLT, 2015.","title":"Contextual dueling bandits","authors":"Dudík","year":"2015","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fürnkranz et al., 2012] Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-based reinforcement learning: a formal frame- work and a policy iteration algorithm. Machine learning, 89(1-2):123-156, 2012.","title":"Preference-based reinforcement learning: a formal framework and a policy iteration algorithm","authors":"Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, Sang-Hyeun Park","year":"2012","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gajane et al., 2015] Pratik Gajane, Tanguy Urvoy, and Fab- rice Clérot. A relative exponential weighing algorithm for adversarial utility-based dueling bandits. In ICML, 2015.","title":"A relative exponential weighing algorithm for adversarial utility-based dueling bandits","authors":"Pratik Gajane, Tanguy Urvoy, Fabrice Clérot","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[González et al., 2017] Javier González, Zhenwen Dai, An- dreas Damianou, and Neil D Lawrence. Preferential bayesian optimization. In ICML, 2017.","title":"Preferential bayesian optimization","authors":"Javier González, Zhenwen Dai, Andreas Damianou, Neil D Lawrence","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hofmann et al., 2011] Katja Hofmann, Shimon Whiteson, and Maarten De Rijke. A probabilistic method for infer- ring preferences from clicks. In CIKM, 2011.","title":"A probabilistic method for inferring preferences from clicks","authors":"Katja Hofmann, Shimon Whiteson, Maarten De, Rijke","year":"2011","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hofmann et al., 2013] Katja Hofmann, Anne Schuth, Shi- mon Whiteson, and Maarten de Rijke. Reusing historical interaction data for faster online learning to rank for ir. In Proceedings of the sixth ACM international conference on Web search and data mining, pages 183-192. ACM, 2013.","title":"Reusing historical interaction data for faster online learning to rank for ir","authors":"Katja Hofmann, Anne Schuth, Shimon Whiteson, Maarten De Rijke","year":"2013","venue":"Proceedings of the sixth ACM international conference on Web search and data mining","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jamieson et al., 2015] Kevin Jamieson, Sumeet Katariya, Atul Deshpande, and Robert Nowak. Sparse dueling ban- dits. In AISTATS, 2015.","title":"Sparse dueling bandits","authors":"Kevin Jamieson, Sumeet Katariya, Atul Deshpande, Robert Nowak","year":"2015","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Komiyama et al., 2015] Junpei Komiyama, Junya Honda, Hisashi Kashima, and Hiroshi Nakagawa. Regret lower bound and optimal algorithm in dueling bandit problem. In COLT, 2015.","title":"Regret lower bound and optimal algorithm in dueling bandit problem","authors":"Komiyama","year":"2015","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Komiyama et al., 2016] Junpei Komiyama, Junya Honda, and Hiroshi Nakagawa. Copeland dueling bandit problem: Regret lower bound, optimal algorithm, and computation- ally efficient algorithm. In ICML, 2016.","title":"Copeland dueling bandit problem: Regret lower bound, optimal algorithm, and computationally efficient algorithm","authors":"Komiyama","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kumagai, 2017] Wataru Kumagai. Regret analysis for con- tinuous dueling bandit. In NIPS, 2017.","title":"Regret analysis for continuous dueling bandit","authors":"Wataru Kumagai","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Owen, 1995] Guillermo Owen. Game Theory. Emerald Group Publishing Limited, 3rd edition, 1995.","title":"Game Theory","authors":"Guillermo Owen","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Radlinski and Craswell, 2013] Filip Radlinski and Nick Craswell. Optimized interleaving for online retrieval eval- uation. In Proceedings of the sixth ACM international con- ference on Web search and data mining, pages 245-254. ACM, 2013.","title":"Optimized interleaving for online retrieval evaluation","authors":"Craswell Radlinski, Filip Radlinski, Nick Craswell","year":"2013","venue":"Proceedings of the sixth ACM international conference on Web search and data mining","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Radlinski et al., 2008] Filip Radlinski, Madhu Kurup, and Thorsten Joachims. How does clickthrough data reflect retrieval quality? In Proceedings of the 17th ACM confer- ence on Information and knowledge management, 2008.","title":"How does clickthrough data reflect retrieval quality?","authors":"Filip Radlinski, Madhu Kurup, Thorsten Joachims","year":"2008","venue":"Proceedings of the 17th ACM conference on Information and knowledge management","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ramamohan et al., 2016] Siddartha Y Ramamohan, Arun Rajkumar, and Shivani Agarwal. Dueling bandits: Be- yond condorcet winners to general tournament solutions. In NIPS, 2016.","title":"Dueling bandits: Beyond condorcet winners to general tournament solutions","authors":"Arun Siddartha Y Ramamohan, Shivani Rajkumar, Agarwal","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Robbins, 1952] Herbert Robbins. Some aspects of the se- quential design of experiments. Bulletin of the American Mathematical Society, 1952.","title":"Herbert Robbins. Some aspects of the sequential design of experiments","authors":"Robbins","year":"1952","venue":"Bulletin of the American Mathematical Society","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schuth et al., 2015] Anne Schuth, Katja Hofmann, and Filip Radlinski. Predicting search satisfaction metrics with in- terleaved comparisons. In Proceedings of the 38th Inter- national ACM SIGIR Conference on Research and Devel- opment in Information Retrieval, pages 463-472. ACM, 2015.","title":"Predicting search satisfaction metrics with interleaved comparisons","authors":"Anne Schuth, Katja Hofmann, Filip Radlinski","year":"2015","venue":"Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sokolov et al., 2016] Artem Sokolov, Stefan Riezler, and Tanguy Urvoy. Bandit structured prediction for learning from partial feedback in statistical machine translation. In MT summit, 2016.","title":"Bandit structured prediction for learning from partial feedback in statistical machine translation","authors":"Artem Sokolov, Stefan Riezler, Tanguy Urvoy","year":"2016","venue":"MT summit","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sui and Burdick, 2014] Yanan Sui and Joel Burdick. Clini- cal online recommendation with subgroup rank feedback. In RecSys. ACM, 2014.","title":"Clinical online recommendation with subgroup rank feedback","authors":"Burdick Sui, Yanan Sui, Joel Burdick","year":"2014","venue":"RecSys","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sui et al., 2017a] Yanan Sui, Yisong Yue, and Joel Burdick. Correlational dueling bandits with application to clinical treatment in large decision spaces. In IJCAI, 2017.","title":"Correlational dueling bandits with application to clinical treatment in large decision spaces","authors":"Sui","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/389"},{"position":30,"raw":"[Sui et al., 2017b] Yanan Sui, Vincent Zhuang, Joel Burdick, and Yisong Yue. Multi-dueling bandits with dependent arms. In UAI, 2017.","title":"Multi-dueling bandits with dependent arms","authors":"Sui","year":"2017","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sui et al., 2018] Yanan Sui, Vincent Zhuang, Joel Burdick, and Yisong Yue. Stagewise safe bayesian optimization with gaussian processes. In ICML, 2018.","title":"Stagewise safe bayesian optimization with gaussian processes","authors":"Yanan Sui, Vincent Zhuang, Joel Burdick, Yisong Yue","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Szörényi et al., 2015] Balázs Szörényi, Róbert Busa- Fekete, Adil Paul, and Eyke Hüllermeier. Online rank elicitation for plackett-luce: A dueling bandits approach. In NIPS, 2015.","title":"Online rank elicitation for plackett-luce: A dueling bandits approach","authors":"Balázs Szörényi, Róbert Busa-Fekete, Adil Paul, Eyke Hüllermeier","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Urvoy et al., 2013] Tanguy Urvoy, Fabrice Clerot, Raphael Féraud, and Sami Naamane. Generic exploration and k- armed voting bandits. In ICML, 2013.","title":"Generic exploration and karmed voting bandits","authors":"Urvoy","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wirth et al., 2017] Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods. Jour- nal of Machine Learning Research, 18(136):1-46, 2017.","title":"A survey of preference-based reinforcement learning methods","authors":"Wirth","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wu and Liu, 2016] Huasen Wu and Xin Liu. Double thomp- son sampling for dueling bandits. In NIPS, 2016.","title":"Double thompson sampling for dueling bandits","authors":"Liu ; ] Huasen Wu, Xin Wu, Liu","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yue and Joachims, 2009] Yisong Yue and Thorsten Joachims. Interactively optimizing information retrieval systems as a dueling bandits problem. In ICML, 2009.","title":"Interactively optimizing information retrieval systems as a dueling bandits problem","authors":"Joachims Yue, Yisong Yue, Thorsten Joachims","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yue and Joachims, 2011] Yisong Yue and Thorsten Joachims. Beat the mean bandit. In ICML, 2011.","title":"Beat the mean bandit","authors":"Joachims Yue, Yisong Yue, Thorsten Joachims","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yue et al., 2012] Yisong Yue, Josef Broder, Robert Klein- berg, and Thorsten Joachims. The k-armed dueling ban- dits problem. Journal of Computer and System Sciences, 2012.","title":"The k-armed dueling bandits problem","authors":"Yue","year":"2012","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zoghi et al., 2014a] Masrour Zoghi, Shimon Whiteson, Maarten de Rijke, and Remi Munos. Relative confidence sampling for efficient on-line ranker evaluation. In WSDM, 2014.","title":"Relative confidence sampling for efficient on-line ranker evaluation","authors":"Zoghi","year":"2014","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zoghi et al., 2014b] Masrour Zoghi, Shimon Whiteson, Remi Munos, and Maarten de Rijke. Relative upper con- fidence bound for the k-armed dueling bandit problem. In ICML, 2014.","title":"Relative upper confidence bound for the k-armed dueling bandit problem","authors":"Zoghi","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zoghi et al., 2015a] Masrour Zoghi, Zohar S Karnin, Shi- mon Whiteson, and Maarten de Rijke. Copeland dueling bandits. In NIPS, 2015.","title":"Masrour Zoghi, Zohar S Karnin, Shi","authors":"Zoghi","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zoghi et al., 2015b] Masrour Zoghi, Shimon Whiteson, and Maarten de Rijke. Mergerucb: A method for large-scale online ranker evaluation. In WSDM, 2015.","title":"Mergerucb: A method for large-scale online ranker evaluation","authors":"Zoghi","year":"2015","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zoghi et al., 2016] Masrour Zoghi, Tomáš Tunys, Lihong Li, Damien Jose, Junyan Chen, Chun Ming Chin, and Maarten de Rijke. Click-based hot fixes for underperform- ing torso queries. In SIGIR, 2016.","title":"Click-based hot fixes for underperforming torso queries","authors":"Zoghi","year":"2016","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zoghi et al., 2017] Masrour Zoghi, Tomas Tunys, Moham- mad Ghavamzadeh, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. Online learning to rank in stochastic click models. In ICML, 2017.","title":"Online learning to rank in stochastic click models","authors":"Zoghi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}