{"type":"ScholarlyArticle","title":"Correlational Dueling Bandits with Application to Clinical Treatment in Large Decision Spaces","authors":[{"name":"Yanan Sui","slug":"yanan-sui","given":"Yanan","family":"Sui","affiliation":"California Institute of Technology Pasadena , CA 91125"},{"name":"Joel W. Burdick","slug":"joel-w-burdick","given":"Joel W.","family":"Burdick","affiliation":"California Institute of Technology Pasadena , CA 91125"}],"abstract":"We consider sequential decision making under uncertainty, the optimization over large decision space with noisy comparative feedback. This problem can be formulated as a K-armed Dueling Bandits problem where K is the total number of decisions. When K is very large, existing dueling bandits algorithms suffer huge cumulative regret before converging on the optimal arm. This paper studies the dueling bandits problem with a large number of dependent arms. Our problem is motivated by a clinical decision making process in large decision space. We propose an efficient algorithm CorrDuel for the problem which makes decisions to simultaneously deliver effective therapy and explore the decision space. Many sequential decision making problems with large and structured decision space could be facilitated by our algorithm. After evaluated the fast convergence of CorrDuel in analysis and simulation experiments, we applied it on a live clinical trial of therapeutic spinal cord stimulation. It is the first applied algorithm towards spinal cord injury treatments and experimental results show the effectiveness and efficiency of our algorithm.","doi":"10.24963/ijcai.2017/389","url":"https://ijcai.org/proceedings/2017/389","pdf":"https://ijcai.org/proceedings/2017/0389.pdf","markdown":"https://ijcai.org/proceedings/2017/389.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/389","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":389,"pages":{"first":2793,"last":2799},"keywords":["Machine Learning: Learning Preferences or Rankings","Multidisciplinary Topics and Applications: Computational Biology and e-Health","Machine Learning: Active Learning","Uncertainty in AI: Sequential Decision Making"],"videos":[],"references":[{"position":1,"raw":"[Abbasi-Yadkori et al., 2011] Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems (NIPS), pages 2312-2320, 2011.","title":"Improved algorithms for linear stochastic bandits","authors":"Yasin Abbasi-Yadkori, Dávid Pál, Csaba Szepesvári","year":"2011","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abernethy et al., 2008] Jacob Abernethy, Elad Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In Proceedings of the 21st Annual Conference on Learning Theory (COLT), pages 263-274, 2008.","title":"Competing in the dark: An efficient algorithm for bandit linear optimization","authors":"Jacob Abernethy, Elad Hazan, Alexander Rakhlin","year":"2008","venue":"Proceedings of the 21st Annual Conference on Learning Theory (COLT)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ailon et al., 2014] Nir Ailon, Zohar Karnin, and Thorsten Joachims. Reducing dueling bandits to cardinal bandits. In Inter- national Conference on Machine Learning (ICML), 2014.","title":"Reducing dueling bandits to cardinal bandits","authors":"Zohar Nir Ailon, Thorsten Karnin, Joachims","year":"2014","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Auer et al., 2002a] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fis- cher. Finite-time analysis of the multiarmed bandit problem. Ma- chine learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Auer et al., 2002b] Peter Auer, Nicolo Cesa-Bianchi, Yoav Fre- und, and Robert E Schapire. The nonstochastic multiarmed ban- dit problem. SIAM Journal on Computing, 32(1):48-77, 2002.","title":"The nonstochastic multiarmed bandit problem","authors":"Auer","year":"2002","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bubeck and Cesa-Bianchi, 2012] Sébastien Bubeck and Nicolo Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Ma- chine Learning, 5:1-122, 2012.","title":"Regret analysis of stochastic and nonstochastic multi-armed bandit problems","authors":"Cesa-Bianchi ; Bubeck, Sébastien Bubeck, Nicolo Cesa-Bianchi","year":"2012","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bubeck et al., 2008] Sébastien Bubeck, Rémi Munos, Gilles Stoltz, and Csaba Szepesvári. Online optimization in X-armed bandits. In Advances in Neural Information Processing Systems (NIPS), 2008.","title":"Online optimization in X-armed bandits","authors":"Sébastien Bubeck, Rémi Munos, Gilles Stoltz, Csaba Szepesvári","year":"2008","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cesa-Bianchi et al., 2006] Nicolo Cesa-Bianchi, Gábor Lugosi, and Gilles Stoltz. Regret minimization under partial monitoring. Mathematics of Operations Research, 31(3):562-580, 2006.","title":"Regret minimization under partial monitoring","authors":"Cesa-Bianchi","year":"2006","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chapelle and Li, 2011] Olivier Chapelle and Lihong Li. An em- pirical evaluation of thompson sampling. In Advances in Neural Information Processing Systems (NIPS), 2011.","title":"An empirical evaluation of thompson sampling","authors":"Li ; Chapelle, Olivier Chapelle, Lihong Li","year":"2011","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chapelle et al., 2012] Olivier Chapelle, Thorsten Joachims, Filip Radlinski, and Yisong Yue. Large-scale validation and analysis of interleaved search evaluation. ACM Transactions on Information Systems (TOIS), 30(1):6:1-6:41, 2012.","title":"Large-scale validation and analysis of interleaved search evaluation","authors":"Olivier Chapelle, Thorsten Joachims, Filip Radlinski, Yisong Yue","year":"2012","venue":"ACM Transactions on Information Systems (TOIS)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dani et al., 2008] Varsha Dani, Thomas P. Hayes, and Sham M. Kakade. Stochastic linear optimization under bandit feedback. In Conference on Learning Theory (COLT), 2008.","title":"Stochastic linear optimization under bandit feedback","authors":"Varsha Dani, Thomas P Hayes, Sham M Kakade","year":"2008","venue":"Conference on Learning Theory (COLT)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dudík et al., 2015] Miroslav Dudík, Katja Hofmann, Robert E Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In Conference on Learning Theory (COLT), 2015.","title":"Contextual dueling bandits","authors":"Dudík","year":"2015","venue":"Conference on Learning Theory (COLT)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gajane et al., 2015] Pratik Gajane, Tanguy Urvoy, and Fabrice Clérot. A relative exponential weighing algorithm for adversar- ial utility-based dueling bandits. In International Conference on Machine Learning (ICML), 2015.","title":"A relative exponential weighing algorithm for adversarial utility-based dueling bandits","authors":"Pratik Gajane, Tanguy Urvoy, Fabrice Clérot","year":"2015","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gentile et al., 2014] Claudio Gentile, Shuai Li, and Giovanni Zap- pella. Online clustering of bandits. In ICML, pages 757-765, 2014.","title":"Online clustering of bandits","authors":"Claudio Gentile, Shuai Li, Giovanni Zappella","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Harkema et al., 2011] Susan Harkema, Yury Gerasimenko, Jonathan Hodes, Joel Burdick, Claudia Angeli, Yangsheng Chen, Christie Ferreira, Andrea Willhite, Enrico Rejc, Robert G Gross- man, et al. Effect of epidural stimulation of the lumbosacral spinal cord on voluntary movement, standing, and assisted stepping after motor complete paraplegia: a case study. The Lancet, 377(9781):1938-1947, 2011.","title":"Effect of epidural stimulation of the lumbosacral spinal cord on voluntary movement, standing, and assisted stepping after motor complete paraplegia: a case study","authors":"Susan Harkema, Yury Gerasimenko, Jonathan Hodes, Joel Burdick, Claudia Angeli, Yangsheng Chen, Christie Ferreira, Andrea Willhite, Enrico Rejc, Robert G Grossman","year":"2011","venue":"The Lancet","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jamieson et al., 2015] Kevin Jamieson, Sumeet Katariya, Atul Deshpande, and Robert Nowak. Sparse dueling bandits. In Inter- national Conference on Artificial Intelligence and Statistics (AIS- TATS), 2015.","title":"Sparse dueling bandits","authors":"Kevin Jamieson, Sumeet Katariya, Atul Deshpande, Robert Nowak","year":"2015","venue":"International Conference on Artificial Intelligence and Statistics (AIS-TATS)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kleinberg et al., 2008] Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Multi-armed bandits in metric spaces. In ACM Symposium on Theory of Computing (STOC). Association for Computing Machinery, Inc., May 2008.","title":"Multi-armed bandits in metric spaces","authors":"Robert Kleinberg, Aleksandrs Slivkins, Eli Upfal","year":"2008","venue":"ACM Symposium on Theory of Computing (STOC)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based Monte-Carlo planning. In Ma- chine Learning: ECML, pages 282-293, 2006.","title":"Bandit based Monte-Carlo planning","authors":"Levente Kocsis, Csaba Szepesvári","year":"2006","venue":"Machine Learning: ECML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Komiyama et al., 2015] Junpei Komiyama, Junya Honda, Hisashi Kashima, and Hiroshi Nakagawa. Regret lower bound and opti- mal algorithm in dueling bandit problem. In COLT, 2015.","title":"Regret lower bound and optimal algorithm in dueling bandit problem","authors":"Komiyama","year":"2015","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rejc et al., 2015] Enrico Rejc, Claudia Angeli, and Susan Harkema. Effects of lumbosacral spinal cord epidural stimulation for standing after chronic complete paralysis in humans. PloS one, 10(7):e0133998, 2015.","title":"Effects of lumbosacral spinal cord epidural stimulation for standing after chronic complete paralysis in humans","authors":"Enrico Rejc, Claudia Angeli, Susan Harkema","year":"2015","venue":"PloS one","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Robbins, 1952] Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 1952.","title":"Herbert Robbins. Some aspects of the sequential design of experiments","authors":"Robbins","year":"1952","venue":"Bulletin of the American Mathematical Society","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Russo and Van Roy, 2014] Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Op- erations Research, 39(4):1221-1243, 2014.","title":"Learning to optimize via posterior sampling","authors":"Van Russo, Daniel Roy, Benjamin Russo, Roy Van","year":"2014","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Srinivas et al., 2010] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Inter- national Conference on Machine Learning (ICML), 2010.","title":"Gaussian process optimization in the bandit setting: No regret and experimental design","authors":"Niranjan Srinivas, Andreas Krause, Sham Kakade, Matthias Seeger","year":"2010","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sui and Burdick, 2014] Yanan Sui and Joel Burdick. Clinical on- line recommendation with subgroup rank feedback. In ACM Con- ference on Recommender Systems (RecSys), 2014.","title":"Clinical online recommendation with subgroup rank feedback","authors":"Burdick ; ] Yanan Sui, Joel Sui, Burdick","year":"2014","venue":"ACM Conference on Recommender Systems (RecSys)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu and Liu, 2016] Huasen Wu and Xin Liu. Double thompson sampling for dueling bandits. In Advances in Neural Information Processing Systems, pages 649-657, 2016.","title":"Double thompson sampling for dueling bandits","authors":"Liu ; ] Huasen Wu, Xin Wu, Liu","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yue and Joachims, 2009] Yisong Yue and Thorsten Joachims. In- teractively optimizing information retrieval systems as a dueling bandits problem. In International Conference on Machine Learn- ing (ICML), 2009.","title":"Interactively optimizing information retrieval systems as a dueling bandits problem","authors":"Joachims Yue, Yisong Yue, Thorsten Joachims","year":"2009","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yue and Joachims, 2011] Yisong Yue and Thorsten Joachims. Beat the mean bandit. In International Conference on Machine Learning (ICML), 2011.","title":"Beat the mean bandit","authors":"Joachims Yue, Yisong Yue, Thorsten Joachims","year":"2011","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yue et al., 2012] Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Jour- nal of Computer and System Sciences, 78(5):1538-1556, 2012.","title":"The k-armed dueling bandits problem","authors":"Yue","year":"2012","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zoghi et al., 2014] Masrour Zoghi, Shimon Whiteson, Remi Munos, and Maarten de Rijke. Relative upper confidence bound for the k-armed dueling bandit problem. In International Confer- ence on Machine Learning (ICML), 2014.","title":"Relative upper confidence bound for the k-armed dueling bandit problem","authors":"Zoghi","year":"2014","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zoghi et al., 2015] Masrour Zoghi, Zohar S Karnin, Shimon Whiteson, and Maarten de Rijke. Copeland dueling bandits. In Advances in Neural Information Processing Systems, pages 307- 315, 2015.","title":"Masrour Zoghi, Zohar S Karnin, Shimon Whiteson, and Maarten de Rijke. Copeland dueling bandits","authors":"Zoghi","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}