{"type":"ScholarlyArticle","title":"Multi-Feedback Bandit Learning with Probabilistic Contexts","authors":[{"name":"Luting Yang","slug":"luting-yang","given":"Luting","family":"Yang","affiliation":"University of California , Riverside"},{"name":"Jianyi Yang","slug":"jianyi-yang","given":"Jianyi","family":"Yang","affiliation":"University of California , Riverside"},{"name":"Shaolei Ren","slug":"shaolei-ren","given":"Shaolei","family":"Ren","affiliation":"University of California , Riverside"}],"abstract":"Contextual bandit is a classic multi-armed bandit setting, where side information (i.e., context) is available before arm selection. A standard assumption is that exact contexts are perfectly known prior to arm selection and only single feedback is returned. In this work, we focus on multi-feedback bandit learning with probabilistic contexts, where a bundle of contexts are revealed to the agent along with their corresponding probabilities at the beginning of each round. This models such scenarios as where contexts are drawn from the probability output of a neural network and the reward function is jointly determined by multiple feedback signals. We propose a kernelized learning algorithm based on upper confidence bound to choose the optimal arm in reproducing kernel Hilbert space for each context bundle. Moreover, we theoretically establish an upper bound on the cumulative regret with respect to an oracle that knows the optimal arm given probabilistic contexts, and show that the bound grows sublinearly with time. Our simula- tion on machine learning model recommendation further validates the sub-linearity of our cumulative regret and demonstrates that our algorithm outper- forms the approach that selects arms based on the most probable context.","doi":"10.24963/ijcai.2020/427","url":"https://ijcai.org/proceedings/2020/427","pdf":"https://ijcai.org/proceedings/2020/0427.pdf","markdown":"https://ijcai.org/proceedings/2020/427.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/427","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":427,"pages":{"first":3087,"last":3093},"keywords":["Machine Learning: Recommender Systems","Machine Learning: Online Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25299/file_25299.m3u8","length_sec":289,"provider":"hls"}],"references":[{"position":1,"raw":"[Abbasi-Yadkori et al., 2011] Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for lin- ear stochastic bandits. In Advances in Neural Information Processing Systems, pages 2312-2320, 2011.","title":"Improved algorithms for linear stochastic bandits","authors":"Yasin Abbasi-Yadkori, Dávid Pál, Csaba Szepesvári","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Allesiardo et al., 2014] Robin Allesiardo, Raphaël Féraud, and Djallel Bouneffouf. A neural networks committee for the contextual bandit problem. In International Confer- ence on Neural Information Processing, pages 374-381, 2014.","title":"Djallel Bouneffouf. A neural networks committee for the contextual bandit problem","authors":"Allesiardo","year":"2014","venue":"International Conference on Neural Information Processing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic mul- tiarmed bandit problem. SIAM Journal on Computing, 32(1):48-77, 2002.","title":"The nonstochastic multiarmed bandit problem","authors":"Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, Robert E Schapire","year":"2002","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Badanidiyuru et al., 2014] Ashwinkumar Badanidiyuru, John Langford, and Aleksandrs Slivkins. Resourceful contextual bandits. In Conference on Learning Theory, pages 1109-1134, 2014.","title":"Ashwinkumar Badanidiyuru, John Langford, and Aleksandrs Slivkins. Resourceful contextual bandits","authors":"Badanidiyuru","year":"2014","venue":"Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bouneffouf et al., 2017] Djallel Bouneffouf, Irina Rish, Guillermo A Cecchi, and Raphaël Féraud. Context atten- tive bandits: contextual bandit with restricted context. In International Joint Conference on Artificial Intelligence, pages 1468-1475, 2017.","title":"Context attentive bandits: contextual bandit with restricted context","authors":"Bouneffouf","year":"2017","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2017/203"},{"position":6,"raw":"[Chowdhury and Gopalan, 2017] Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. In International Conference on Machine Learning, pages 844-853, 2017.","title":"On kernelized multi-armed bandits","authors":"Gopalan ; Chowdhury, Ray Sayak, Aditya Chowdhury, Gopalan","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deshmukh et al., 2017] Aniket Anand Deshmukh, Urun Dogan, and Clay Scott. Multi-task learning for contextual bandits. In Advances in Neural Information Processing Systems, pages 4848-4856, 2017.","title":"Multi-task learning for contextual bandits","authors":"Aniket Anand Deshmukh, Urun Dogan, Clay Scott","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kirschner and Krause, 2019] Johannes Kirschner and An- dreas Krause. Stochastic bandits with context distribu- tions. In Advances in Neural Information Processing Sys- tems, pages 14090-14099, 2019.","title":"Stochastic bandits with context distributions","authors":"Krause Kirschner","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lai and Robbins, 1985] Tze Leung Lai and Herbert Rob- bins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1):4-22, 1985.","title":"Asymptotically efficient adaptive allocation rules","authors":"Tze Leung, Lai, Herbert Robbins","year":"1985","venue":"Advances in Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Langford and Zhang, 2008] John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in Neural Information Processing Systems, pages 817-824, 2008.","title":"The epoch-greedy algorithm for multi-armed bandits with side information","authors":"Zhang Langford, John Langford, Tong Zhang","year":"2008","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to per- sonalized news article recommendation. In International Conference on World Wide Web, pages 661-670, 2010.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2017] Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear con- textual bandits. In International Conference on Machine Learning, pages 2071-2080, 2017.","title":"Provably optimal algorithms for generalized linear contextual bandits","authors":"Lihong Li, Yu Lu, Dengyong Zhou","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2018] Baihan Lin, Guillermo Cecchi, Djallel Bouneffouf, and Irina Rish. Adaptive representation se- lection in contextual bandit. arXiv:1802.00981, 2018.","title":"Adaptive representation selection in contextual bandit","authors":"Lin","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lu et al., 2019] Shiyin Lu, Guanghui Wang, Yao Hu, and Lijun Zhang. Multi-objective generalized linear ban- dits. In International Joint Conference on Artificial In- telligence, pages 3080-3086, 2019.","title":"Multi-objective generalized linear bandits","authors":"Shiyin Lu, Guanghui Wang, Yao Hu, Lijun Zhang","year":"2019","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/427"},{"position":15,"raw":"[Mary et al., 2015] Jérémie Mary, Romaric Gaudel, and Philippe Preux. Bandits and recommender systems. In In- ternational Workshop on Machine Learning, Optimization and Big Data, pages 325-336, 2015.","title":"Bandits and recommender systems","authors":"Jérémie Mary, Romaric Gaudel, Philippe Preux","year":"2015","venue":"International Workshop on Machine Learning, Optimization and Big Data","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Quinlan, 1986] J. Ross Quinlan. Induction of decision trees. Machine Learning, 1(1):81-106, 1986.","title":"Induction of decision trees","authors":"J, Ross Quinlan","year":"1986","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Roijers et al., 2017] Diederik M Roijers, Luisa M Zintgraf, and Ann Nowé. Interactive thompson sampling for multi- objective multi-armed bandits. In International Confer- ence on Algorithmic Decision Theory, pages 18-34, 2017.","title":"Interactive thompson sampling for multiobjective multi-armed bandits","authors":"Luisa M Diederik M Roijers, Ann Zintgraf, Nowé","year":"2017","venue":"International Conference on Algorithmic Decision Theory","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tang et al., 2013] Liang Tang, Romer Rosales, Ajit Singh, and Deepak Agarwal. Automatic ad format selection via contextual bandits. In International Conference on In- formation & Knowledge Management, pages 1587-1594, 2013.","title":"Automatic ad format selection via contextual bandits","authors":"Liang Tang, Romer Rosales, Ajit Singh, Deepak Agarwal","year":"2013","venue":"International Conference on Information & Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Valko et al., 2013] Michal Valko, Nathan Korda, Rémi Munos, Ilias Flaounas, and Nello Cristianini. Finite-time analysis of kernelised contextual bandits. In Conference on Uncertainty in Artificial Intelligence, pages 654-663, 2013.","title":"Finite-time analysis of kernelised contextual bandits","authors":"Michal Valko, Nathan Korda, Rémi Munos, Ilias Flaounas, Nello Cristianini","year":"2013","venue":"Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Villar et al., 2015] Sofía S Villar, Jack Bowden, and James Wason. Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges. Statistical Sci- ence: A Review Journal of the Institute of Mathematical Statistics, 30(2):199, 2015.","title":"Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges","authors":"S Sofía, Jack Villar, James Bowden, Wason","year":"2015","venue":"Statistical Science: A Review Journal of the Institute of Mathematical Statistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wan, 1990] Eric A. Wan. Neural network classification: A bayesian interpretation. IEEE Transactions on Neural Net- works, 1(4):303-305, 1990.","title":"Neural network classification: A bayesian interpretation","authors":"Eric A Wan","year":"1990","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2016] Huazheng Wang, Qingyun Wu, and Hongning Wang. Learning hidden features for contextual bandits. In International Conference on Information and Knowledge Management, pages 1633-1642, 2016.","title":"Learning hidden features for contextual bandits","authors":"Wang","year":"2016","venue":"International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wanigasekara et al., 2019] Nirandika Wanigasekara, Yux- uan Liang, Siong Thye Goh, Ye Liu, Joseph Jay Williams, and David S Rosenblum. Learning multi-objective rewards and user utility function in contextual bandits for person- alized ranking. In International Joint Conference on Arti- ficial Intelligence, pages 3835-3841, 2019.","title":"Learning multi-objective rewards and user utility function in contextual bandits for personalized ranking","authors":"Wanigasekara","year":"2019","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/532"},{"position":24,"raw":"[Yahyaa and Manderick, 2015] Saba Yahyaa and Bernard Manderick. Thompson sampling for multi-objective multi- armed bandits problem. In European Symposium on Ar- tificial Neural Networks, Computational Intelligence and Machine Learning., pages 47-52, 2015.","title":"Saba Yahyaa and Bernard Manderick. Thompson sampling for multi-objective multiarmed bandits problem","authors":"Manderick Yahyaa","year":"2015","venue":"European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yun et al., 2017] Se-Young Yun, Jun Hyun Nam, Sangwoo Mo, and Jinwoo Shin. Contextual multi-armed bandits un- der feature uncertainty. arXiv:1703.01347, 2017.","title":"Contextual multi-armed bandits under feature uncertainty","authors":"Se-Young Yun, Jun, Hyun Nam, Sangwoo Mo, Jinwoo Shin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2019] Dongruo Zhou, Lihong Li, and Quan- quan Gu. Neural contextual bandits with UCB-based ex- ploration. arXiv:1911.04462, 2019.","title":"Neural contextual bandits with UCB-based exploration","authors":"Zhou","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}