{"type":"ScholarlyArticle","title":"Context Attentive Bandits: Contextual Bandit with Restricted Context","authors":[{"name":"Djallel Bouneffouf","slug":"djallel-bouneffouf","given":"Djallel","family":"Bouneffouf","affiliation":""},{"name":"Irina Rish","slug":"irina-rish","given":"Irina","family":"Rish","affiliation":""},{"name":"Guillermo Cecchi","slug":"guillermo-cecchi","given":"Guillermo","family":"Cecchi","affiliation":"IBM Thomas J. Watson Research Center , Yorktown Heights , NY USA"},{"name":"Raphaël Féraud","slug":"raphael-feraud","given":"Raphaël","family":"Féraud","affiliation":"Orange Labs , 2 av. Pierre Marzin , 22300 Lannion ( France)"}],"abstract":"We consider a novel formulation of the multi-armed bandit model, which we call the contextual bandit with restricted context, where only a limited number of features can be accessed by the learner at every iteration. This novel formulation is motivated by different online problems arising in clinical trials, recommender systems and attention modeling.Herein, we adapt the standard multi-armed bandit algorithm known as Thompson Sampling to take advantage of our restricted context setting, and propose two novel algorithms, called the Thompson Sampling with Restricted Context (TSRC) and the Windows Thompson Sampling with Restricted Context (WTSRC), for handling stationary and nonstationary environments, respectively. Our empirical results demonstrate advantages of the proposed approaches on several real-life datasets.","doi":"10.24963/ijcai.2017/203","url":"https://ijcai.org/proceedings/2017/203","pdf":"https://ijcai.org/proceedings/2017/0203.pdf","markdown":"https://ijcai.org/proceedings/2017/203.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/203","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":203,"pages":{"first":1468,"last":1475},"keywords":["Machine Learning: Online Learning","Machine Learning: Reinforcement Learning","Machine Learning: Machine Learning","Machine Learning: New Problems"],"videos":[],"references":[{"position":1,"raw":"[Abbasi-Yadkori et al., 2012] Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Online-to- confidence-set conversions and application to sparse stochastic bandits. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2012, pages 1-9, 2012.","title":"Online-toconfidence-set conversions and application to sparse stochastic bandits","authors":"Yasin Abbasi-Yadkori, Dávid Pál, Csaba Szepesvári","year":"2012","venue":"Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2012","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal and Goyal, 2012] Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi- armed bandit problem. In COLT 2012 -The 25th Annual Conference on Learning Theory, June 25-27, 2012, Edinburgh, Scotland, pages 39.1-39.26, 2012.","title":"Analysis of thompson sampling for the multiarmed bandit problem","authors":"Shipra Agrawal, Navin Goyal","year":"2012","venue":"COLT 2012 -The 25th Annual Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Agrawal and Goyal, 2013] Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In ICML (3), pages 127-135, 2013.","title":"Thompson sampling for contextual bandits with linear payoffs","authors":"Shipra Agrawal, Navin Goyal","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Allesiardo et al., 2014] Robin Allesiardo, Raphaël Féraud, and Djallel Bouneffouf. A neural networks committee for the contextual bandit problem. In Neural Information Processing -21st International Conference, ICONIP 2014, Kuching, Malaysia, November 3-6, 2014. Proceedings, Part I, pages 374-381, 2014.","title":"Djallel Bouneffouf. A neural networks committee for the contextual bandit problem","authors":"Allesiardo","year":"2014","venue":"Neural Information Processing -21st International Conference, ICONIP 2014","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Auer et al., 2002] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the mul- tiarmed bandit problem. Machine Learning, 47(2- 3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bastani and Bayati, 2015] Hamsa Bastani and Mohsen Bayati. Online decision-making with high-dimensional covariates. Available at SSRN 2661896, 2015.","title":"Online decision-making with high-dimensional covariates","authors":"Bayati ; ] Hamsa Bastani, Mohsen Bastani, Bayati","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bouneffouf and Féraud, 2016] Djallel Bouneffouf and Raphaël Féraud. Multi-armed bandit problem with known trend. Neurocomputing, 205:16-21, 2016.","title":"Djallel Bouneffouf and Raphaël Féraud. Multi-armed bandit problem with known trend","authors":"Féraud Bouneffouf","year":"2016","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Carpentier and Munos, 2012] Alexandra Carpentier and Rémi Munos. Bandit theory meets compressed sensing for high dimensional stochastic linear bandit. In Proceedings of the Fifteenth International Confer- ence on Artificial Intelligence and Statistics, AISTATS 2012, pages 190-198, 2012.","title":"Bandit theory meets compressed sensing for high dimensional stochastic linear bandit","authors":"Munos Carpentier, Alexandra Carpentier, Rémi Munos","year":"2012","venue":"Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2012","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Durand and Gagné, 2014] Audrey Durand and Chris- tian Gagné. Thompson sampling for combinatorial bandits and its application to online feature selection. 2014.","title":"Thompson sampling for combinatorial bandits and its application to online feature selection","authors":"Durand, Gagné, Audrey Durand, Christian Gagné","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu and Ogihara, 2011] Yajie Hu and Mitsunori Ogi- hara. Nextone player: A music recommendation sys- tem based on user behavior. In ISMIR, pages 103-108, 2011.","title":"Nextone player: A music recommendation system based on user behavior","authors":"Ogihara Hu, Yajie Hu, Mitsunori Ogihara","year":"2011","venue":"ISMIR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lai and Robbins, 1985] T. L. Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Ad- vances in Applied Mathematics, 6(1):4-22, 1985.","title":"Asymptotically efficient adaptive allocation rules","authors":"T L Lai, Herbert Robbins","year":"1985","venue":"Advances in Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Langford and Zhang, 2008] John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in neu- ral information processing systems, pages 817-824, 2008.","title":"The epoch-greedy algorithm for multi-armed bandits with side information","authors":"Zhang Langford, John Langford, Tong Zhang","year":"2008","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. CoRR, 2010.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mary et al., 2015] Jérémie Mary, Romaric Gaudel, and Philippe Preux. Bandits and recommender systems. In Machine Learning, Optimization, and Big Data -First International Workshop, MOD 2015, pages 325-336, 2015.","title":"Bandits and recommender systems","authors":"Jérémie Mary, Romaric Gaudel, Philippe Preux","year":"2015","venue":"Machine Learning, Optimization, and Big Data -First International Workshop","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2014] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual atten- tion. In Advances in Neural Information Processing Systems, pages 2204-2212, 2014.","title":"Recurrent models of visual attention","authors":"Volodymyr Mnih, Nicolas Heess, Alex Graves","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tekin et al., 2015] Cem Tekin, Onur Atan, and Mihaela van der Schaar. Discover the expert: Context-adaptive expert selection for medical diagnosis. IEEE Trans. Emerging Topics Comput., 3(2):220-234, 2015.","title":"Discover the expert: Context-adaptive expert selection for medical diagnosis","authors":"Tekin","year":"2015","venue":"IEEE Trans. Emerging Topics Comput","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Thompson, 1933] William R Thompson. On the like- lihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"William R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Villar et al., 2015] Sofía S Villar, Jack Bowden, and James Wason. Multi-armed bandit models for the op- timal design of clinical trials: benefits and challenges. Statistical science: a review journal of the Institute of Mathematical Statistics, 30(2):199, 2015.","title":"Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges","authors":"S Sofía, Jack Villar, James Bowden, Wason","year":"2015","venue":"Statistical science: a review journal of the Institute of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2014] Jialei Wang, Peilin Zhao, Steven CH Hoi, and Rong Jin. Online feature selection and its applications. IEEE Transactions on Knowledge and Data Engineering, 26(3):698-710, 2014.","title":"Online feature selection and its applications","authors":"Wang","year":"2014","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}