{"type":"ScholarlyArticle","title":"Optimal Exploitation of Clustering and History Information in Multi-armed Bandit","authors":[{"name":"Djallel Bouneffouf","slug":"djallel-bouneffouf","given":"Djallel","family":"Bouneffouf","affiliation":"IBM Research , Yorktown Heights , NY , USA"},{"name":"Srinivasan Parthasarathy","slug":"srinivasan-parthasarathy","given":"Srinivasan","family":"Parthasarathy","affiliation":"IBM Research , Yorktown Heights , NY , USA"},{"name":"Horst Samulowitz","slug":"horst-samulowitz","given":"Horst","family":"Samulowitz","affiliation":"IBM Research , Yorktown Heights , NY , USA"},{"name":"Martin Wistuba","slug":"martin-wistuba","given":"Martin","family":"Wistuba","affiliation":"IBM Research , Yorktown Heights , NY , USA"}],"abstract":"We consider the stochastic multi-armed bandit problem and the contextual bandit problem with historical observations and pre-clustered arms. The historical observations can contain any number of instances for each arm, and the pre-clustering information is a fixed clustering of arms provided as part of the input. We develop a variety of algorithms which incorporate this offline information effectively during the online exploration phase and derive their regret bounds. In particular, we develop the META algorithm which effectively hedges between two other algorithms: one which uses both historical observations and clustering, and another which uses only the historical observations. The former outperforms the latter when the clustering quality is good, and vice-versa. Extensive experiments on synthetic and real world datasets on Warafin drug dosage and web server selectionfor latency minimization validate our theoretical insights and demonstrate that META is a robust strategy for optimally exploiting the pre-clustering information.","doi":"10.24963/ijcai.2019/279","url":"https://ijcai.org/proceedings/2019/279","pdf":"https://ijcai.org/proceedings/2019/0279.pdf","markdown":"https://ijcai.org/proceedings/2019/279.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/279","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":279,"pages":{"first":2016,"last":2022},"keywords":["Machine Learning: Learning Theory","Machine Learning: Recommender Systems","Machine Learning: Online Learning","Uncertainty in AI: Uncertainty in AI","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Abbasi-Yadkori et al., 2011] Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for lin- ear stochastic bandits. In Advances in Neural Information Processing Systems, pages 2312-2320, 2011.","title":"Improved algorithms for linear stochastic bandits","authors":"Yasin Abbasi-Yadkori, Dávid Pál, Csaba Szepesvári","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 2002] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Balakrishnan et al., 2019] Avinash Balakrishnan, Djallel Bouneffouf, Nicholas Mattei, and Francesca Rossi. Incor- porating behavioral constraints in online AI systems. AAAI 2019, 2019.","title":"Incorporating behavioral constraints in online AI systems","authors":"Avinash Balakrishnan, Djallel Bouneffouf, Nicholas Mattei, Francesca Rossi","year":"2019","venue":"AAAI 2019","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bouneffouf and Rish, 2019] Djallel Bouneffouf and Irina Rish. A survey on practical applications of multi-armed and contextual bandits. CoRR, abs/1904.10040, 2019.","title":"Djallel Bouneffouf and Irina Rish. A survey on practical applications of multi-armed and contextual bandits","authors":"Rish Bouneffouf","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bouneffouf et al., 2017] Djallel Bouneffouf, Irina Rish, Guillermo A. Cecchi, and Raphaël Féraud. Context at- tentive bandits: Contextual bandit with restricted context. In IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 1468-1475, 2017.","title":"Context attentive bandits: Contextual bandit with restricted context","authors":"Bouneffouf","year":"2017","venue":"IJCAI 2017","doi":null,"url":null,"internal_url":"/proceedings/2017/203"},{"position":6,"raw":"[Chu et al., 2011] Wei Chu, Lihong Li, Lev Reyzin, and Robert E. Schapire. Contextual bandits with linear payoff functions. In Geoffrey J. Gordon, David B. Dunson, and Miroslav Dudik, editors, AISTATS, volume 15 of JMLR Proceedings, pages 208-214. JMLR.org, 2011.","title":"Contextual bandits with linear payoff functions","authors":"Wei Chu, Lihong Li, Lev Reyzin, Robert E Schapire","year":"2011","venue":"JMLR Proceedings","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gentile et al., 2017] Claudio Gentile, Shuai Li, Purushot- tam Kar, Alexandros Karatzoglou, Giovanni Zappella, and Evans Etrue. On context-dependent clustering of bandits. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 1253-1262, 2017.","title":"On context-dependent clustering of bandits","authors":"Claudio Gentile, Shuai Li, Purushottam Kar, Alexandros Karatzoglou, Giovanni Zappella, Evans Etrue","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning, ICML 2017","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kocsis and Szepesvári, 2006] Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In ECML, volume 6, pages 282-293. Springer, 2006.","title":"Bandit based monte-carlo planning","authors":"Szepesvári ; Kocsis, Csaba Kocsis, Szepesvári","year":"2006","venue":"ECML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Korda et al., 2016] Nathan Korda, Balázs Szörényi, and Shuai Li. Distributed clustering of linear bandits in peer to peer networks. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 1301-1309, 2016.","title":"Distributed clustering of linear bandits in peer to peer networks","authors":"Nathan Korda, Balázs Szörényi, Shuai Li","year":"2016","venue":"Proceedings of the 33nd International Conference on Machine Learning, ICML 2016","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lai and Robbins, 1985] T. L. Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Ad- vances in Applied Mathematics, 6(1):4-22, 1985.","title":"Asymptotically efficient adaptive allocation rules","authors":"T L Lai, Herbert Robbins","year":"1985","venue":"Advances in Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Langford and Zhang, 2007] John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In John C. Platt, Daphne Koller, Yoram Singer, and Sam T. Roweis, editors, NIPS.","title":"The epoch-greedy algorithm for multi-armed bandits with side information","authors":"Zhang ; ] Langford, John Langford, Tong Zhang","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Curran Associates, Inc., 2007.","title":null,"authors":"Curran Associates, Inc","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to per- sonalized news article recommendation. In Proceedings of the 19th international conference on World wide web, WWW '10, pages 661-670, USA, 2010. ACM.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"Proceedings of the 19th international conference on World wide web, WWW '10","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2018] Baihan Lin, Djallel Bouneffouf, Guillermo A. Cecchi, and Irina Rish. Contextual bandit with adaptive feature extraction. In 2018 IEEE International Conference on Data Mining Workshops, ICDM Workshops, Singapore, Singapore, November 17-20, 2018, pages 937-944, 2018.","title":"Contextual bandit with adaptive feature extraction","authors":"Lin","year":"2018","venue":"2018 IEEE International Conference on Data Mining Workshops, ICDM Workshops","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2010] Yanchi Liu, Zhongmou Li, Hui Xiong, Xuedong Gao, and Junjie Wu. Understanding of internal clustering validation measures. In Proceedings of the 2010 IEEE International Conference on Data Mining, ICDM '10, pages 911-916, Washington, DC, USA, 2010. IEEE Computer Society.","title":"Understanding of internal clustering validation measures","authors":"Yanchi Liu, Zhongmou Li, Hui Xiong, Xuedong Gao, Junjie Wu","year":"2010","venue":"Proceedings of the 2010 IEEE International Conference on Data Mining, ICDM '10","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Maillard and Mannor, 2014] Odalric-Ambrym Maillard and Shie Mannor. Latent bandits. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, pages 136-144, 2014.","title":"Odalric-Ambrym Maillard and Shie Mannor. Latent bandits","authors":"Mannor Maillard","year":"2014","venue":"Proceedings of the 31th International Conference on Machine Learning, ICML 2014","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nguyen and Lauw, 2014] Trong T. Nguyen and Hady Wirawan Lauw. Dynamic clustering of con- textual multi-armed bandits. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014, Shanghai, China, November 3-7, 2014, pages 1959-1962, 2014.","title":"Dynamic clustering of contextual multi-armed bandits","authors":"T Trong, Hady Nguyen, Lauw Wirawan","year":"2014","venue":"Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Noothigattu et al., 2018] Ritesh Noothigattu, Djallel Boun- effouf, Nicholas Mattei, Rachita Chandra, Piyush Madan, Kush R. Varshney, Murray Campbell, Moninder Singh, and Francesca Rossi. Interpretable multi-objective rein- forcement learning through policy orchestration. CoRR, abs/1809.08343, 2018.","title":"Interpretable multi-objective reinforcement learning through policy orchestration","authors":"Ritesh Noothigattu, Djallel Bouneffouf, Nicholas Mattei, Rachita Chandra, Piyush Madan, Kush R Varshney, Murray Campbell, Moninder Singh, Francesca Rossi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pandey et al., 2007] Sandeep Pandey, Deepayan Chakrabarti, and Deepak Agarwal. Multi-armed bandit problems with dependent arms. In Proceedings of the 24th international conference on Machine learning, pages 721-728. ACM, 2007.","title":"Multi-armed bandit problems with dependent arms","authors":"Sandeep Pandey, Deepayan Chakrabarti, Deepak Agarwal","year":"2007","venue":"Proceedings of the 24th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sharabiani et al., 2015] Ashkan Sharabiani, Adam Bress, Elnaz Douzali, and Houshang Darabi. Revisiting warfarin dosing using machine learning techniques. Computational and mathematical methods in medicine, 2015, 2015.","title":"Revisiting warfarin dosing using machine learning techniques","authors":"Ashkan Sharabiani, Adam Bress, Elnaz Douzali, Houshang Darabi","year":"2015","venue":"Computational and mathematical methods in medicine","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shivaswamy and Joachims, 2012] Pannagadatta Shiv- aswamy and Thorsten Joachims. Multi-armed bandit problems with history. In Artificial Intelligence and Statistics, pages 1046-1054, 2012.","title":"Pannagadatta Shivaswamy and Thorsten Joachims. Multi-armed bandit problems with history","authors":"Joachims Shivaswamy","year":"2012","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Thompson and Freede, 1971] Robert C Thompson and Linda J Freede. On the eigenvalues of sums of hermitian matrices. Linear Algebra and Its Applications, 4(4):369- 376, 1971.","title":"On the eigenvalues of sums of hermitian matrices","authors":"Freede ; Thompson, C Robert, Linda J Thompson, Freede","year":"1971","venue":"Linear Algebra and Its Applications","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vermorel and Mohri, 2005] Joannès Vermorel and Mehryar Mohri. Multi-armed bandit algorithms and empirical eval- uation. In Machine Learning, pages 437-448, 2005.","title":"Joannès Vermorel and Mehryar Mohri. Multi-armed bandit algorithms and empirical evaluation","authors":"Mohri Vermorel","year":"2005","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2018] Zhiyang Wang, Ruida Zhou, and Cong Shen. Regional multi-armed bandits. arXiv preprint arXiv:1802.07917, 2018.","title":"Regional multi-armed bandits","authors":"Zhiyang Wang, Ruida Zhou, Cong Shen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}