{"type":"ScholarlyArticle","title":"A Practical Semi-Parametric Contextual Bandit","authors":[{"name":"Yi Peng","slug":"yi-peng","given":"Yi","family":"Peng","affiliation":"Alibaba Group , Hang Zhou , China"},{"name":"Miao Xie","slug":"miao-xie","given":"Miao","family":"Xie","affiliation":"Alibaba Group , Hang Zhou , China"},{"name":"Jiahao Liu","slug":"jiahao-liu","given":"Jiahao","family":"Liu","affiliation":"Alibaba Group , Hang Zhou , China"},{"name":"Xuying Meng","slug":"xuying-meng","given":"Xuying","family":"Meng","affiliation":"Institute of Computing Technology , Chinese Academy of Sciences"},{"name":"Nan Li","slug":"nan-li","given":"Nan","family":"Li","affiliation":"Alibaba Group , Hang Zhou , China"},{"name":"Cheng Yang","slug":"cheng-yang","given":"Cheng","family":"Yang","affiliation":"Alibaba Group , Hang Zhou , China"},{"name":"Tao Yao","slug":"tao-yao","given":"Tao","family":"Yao","affiliation":"Alibaba Group , Hang Zhou , China"},{"name":"Rong Jin","slug":"rong-jin","given":"Rong","family":"Jin","affiliation":"Alibaba Group , Hang Zhou , China"}],"abstract":"Classic multi-armed bandit algorithms are inefficient for a large number of arms. On the other hand, contextual bandit algorithms are more efficient, but they suffer from a large regret due to the bias of reward estimation with finite dimensional features. Although recent studies proposed semi-parametric bandits to overcome these defects, they assume arms' features are constant over time. However, this assumption rarely holds in practice, since real-world problems often involve underlying processes that are dynamically evolving over time especially for the special promotions like Singles' Day sales. In this paper, we formulate a novel Semi-Parametric Contextual Bandit Problem to relax this assumption. For this problem, a novel Two-Steps Upper-Confidence Bound framework, called Semi-Parametric UCB (SPUCB), is presented. It can be flexibly applied to linear parametric function problem with a satisfied gap-free bound on the n-step regret. Moreover, to make our method more practical in online system, an optimization is proposed for dealing with high dimensional features of a linear function. Extensive experiments on synthetic data as well as a real dataset from one of the largest e-commercial platforms demonstrate the superior performance of our algorithm.","doi":"10.24963/ijcai.2019/450","url":"https://ijcai.org/proceedings/2019/450","pdf":"https://ijcai.org/proceedings/2019/0450.pdf","markdown":"https://ijcai.org/proceedings/2019/450.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/450","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":450,"pages":{"first":3246,"last":3252},"keywords":["Machine Learning: Online Learning","Heuristic Search and Game Playing: Combinatorial Search and Optimisation"],"videos":[],"references":[{"position":1,"raw":"[Agrawal and Goyal, 2013] Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International Conference on Machine Learning, pages 127-135, 2013.","title":"Thompson sampling for contextual bandits with linear payoffs","authors":"Shipra Agrawal, Navin Goyal","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal et al., 1988] R. Agrawal, M. V. Hedge, and D. Teneketzis. Asymptotically efficient adaptive alloca- tion rules for the multiarmed bandit problem with switch- ing cost. IEEE Transactions on Automatic Control, 33(10):899-906, Oct 1988.","title":"Asymptotically efficient adaptive allocation rules for the multiarmed bandit problem with switching cost","authors":"R Agrawal, M V Hedge, D Teneketzis","year":"1988","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-Time Analysis of the Multiarmed Bandit Problem. Machine Learning, 47(2-3):235-256, 2002.","title":"Finite-Time Analysis of the Multiarmed Bandit Problem","authors":"Auer","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Auer, 2002] Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.","title":"Using confidence bounds for exploitation-exploration trade-offs","authors":"Peter Auer","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2013] Wei Chen, Yajun Wang, and Yang Yuan. Combinatorial multi-armed bandit: General framework, results and applications. In Proceedings of the 30th Inter- national Conference on International Conference on Ma- chine Learning -Volume 28, ICML'13, pages I-151-I- 159. JMLR.org, 2013.","title":"Combinatorial multi-armed bandit: General framework, results and applications","authors":"Wei Chen, Yajun Wang, Yang Yuan","year":"2013","venue":"Proceedings of the 30th International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chu et al., 2011] Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 208-214, 2011.","title":"Contextual bandits with linear payoff functions","authors":"Wei Chu, Lihong Li, Lev Reyzin, Robert Schapire","year":"2011","venue":"Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gai et al., 2012] Yi Gai, Bhaskar Krishnamachari, and Rahul Jain. Combinatorial network optimization with un- known variables: Multi-armed bandits with linear rewards and individual observations. IEEE/ACM Trans. Netw., 20(5):1466-1478, October 2012.","title":"Combinatorial network optimization with unknown variables: Multi-armed bandits with linear rewards and individual observations","authors":"Yi Gai, Bhaskar Krishnamachari, Rahul Jain","year":"2012","venue":"IEEE/ACM Trans. Netw","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ghosh et al., 2017] Avishek Ghosh, Sayak Ray Chowdhury, and Aditya Gopalan. Misspecified linear bandits. In AAAI, pages 3761-3767, 2017.","title":"Misspecified linear bandits","authors":"Avishek Ghosh, Sayak Ray Chowdhury, Aditya Gopalan","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gopalan et al., 2014] Aditya Gopalan, Shie Mannor, and Yishay Mansour. Thompson sampling for complex on- line problems. In International Conference on Machine Learning, pages 100-108, 2014.","title":"Thompson sampling for complex online problems","authors":"Aditya Gopalan, Shie Mannor, Yishay Mansour","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Greenewald et al., 2017] Kristjan Greenewald, Ambuj Tewari, Susan Murphy, and Predag Klasnja. Action centered contextual bandits. In Advances in neural information processing systems, pages 5977-5985, 2017.","title":"Action centered contextual bandits","authors":"Kristjan Greenewald, Ambuj Tewari, Susan Murphy, Predag Klasnja","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hoffman et al., 2011] Matthew D Hoffman, Eric Brochu, and Nando de Freitas. Portfolio allocation for bayesian optimization. In UAI, pages 327-336. Citeseer, 2011.","title":"Portfolio allocation for bayesian optimization","authors":"Eric Matthew D Hoffman, Nando Brochu, De Freitas","year":"2011","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krishnamurthy et al., 2018] Akshay Krishnamurthy, Zhi- wei Steven Wu, and Vasilis Syrgkanis. Semiparamet- ric contextual bandits. arXiv preprint arXiv:1803.04204, 2018.","title":"Semiparametric contextual bandits","authors":"Akshay Krishnamurthy, Zhiwei Steven Wu, Vasilis Syrgkanis","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kveton et al., 2015a] Branislav Kveton, Csaba Szepesvári, Zheng Wen, and Azin Ashkan. Cascading bandits: Learn- ing to rank in the cascade model. In Proceedings of the 32Nd International Conference on International Confer- ence on Machine Learning -Volume 37, ICML'15, pages 767-776. JMLR.org, 2015.","title":"Cascading bandits: Learning to rank in the cascade model","authors":"Kveton","year":"2015","venue":"Proceedings of the 32Nd International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kveton et al., 2015b] Branislav Kveton, Zheng Wen, Azin Ashkan, and Csaba Szepesvári. Combinatorial cascading bandits. In Proceedings of the 28th International Confer- ence on Neural Information Processing Systems -Volume 1, NIPS'15, pages 1450-1458, Cambridge, MA, USA, 2015. MIT Press.","title":"Combinatorial cascading bandits","authors":"Kveton","year":"2015","venue":"Proceedings of the 28th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to per- sonalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web, WWW '10, pages 661-670, New York, NY, USA, 2010. ACM.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"Proceedings of the 19th International Conference on World Wide Web, WWW '10","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2016] Shuai Li, Baoxiang Wang, Shengyu Zhang, and Wei Chen. Contextual combinatorial cascading ban- dits. In Proceedings of the 33rd International Conference on International Conference on Machine Learning -Vol- ume 48, ICML'16, pages 1245-1253. JMLR.org, 2016.","title":"Contextual combinatorial cascading bandits","authors":"Shuai Li, Baoxiang Wang, Shengyu Zhang, Wei Chen","year":"2016","venue":"Proceedings of the 33rd International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ou et al., 2019] Mingdong Ou, Nan Li, Cheng Yang, Shenghuo Zhu, and Rong Jin. Semi-parametric sampling for stochastic bandits with many arms. In Proc. of AAAI Conference on Artificial Intelligence, Hawaii USA, Jan 2019.","title":"Semi-parametric sampling for stochastic bandits with many arms","authors":"Ou","year":"2019","venue":"Proc. of AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"William H Press. Bandit solutions provide unified ethical models for randomized clinical trials and comparative effectiveness research. Proceedings of the National Academy of Sciences, pages pnas-0912378106, 2009.","title":"Bandit solutions provide unified ethical models for randomized clinical trials and comparative effectiveness research","authors":"H William, Press","year":"2009","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qin et al., 2014] Lijing Qin, Shouyuan Chen, and Xiaoyan Zhu. Contextual combinatorial bandit and its appli- cation on diversified online recommendation. In Mo- hammed Javeed Zaki, Zoran Obradovic, Pang-Ning Tan, Arindam Banerjee, Chandrika Kamath, and Srinivasan Parthasarathy, editors, Proceedings of the 2014 SIAM In- ternational Conference on Data Mining, pages 461-469.","title":"Contextual combinatorial bandit and its application on diversified online recommendation","authors":"Lijing Qin, Shouyuan Chen, Xiaoyan Zhu","year":null,"venue":"Proceedings of the 2014 SIAM International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Robbins, 1952] Herbert Robbins. Some aspects of the se- quential design of experiments. Bulletin of the American Mathematical Society, 58(5):527-535, 1952.","title":"Herbert Robbins. Some aspects of the sequential design of experiments","authors":"Robbins","year":"1952","venue":"Bulletin of the American Mathematical Society","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schwartz et al., 2017] Eric M Schwartz, Eric T Bradlow, and Peter S Fader. Customer acquisition via display ad- vertising using multi-armed bandit experiments. Market- ing Science, 36(4):500-522, 2017.","title":"Customer acquisition via display advertising using multi-armed bandit experiments","authors":"Eric M Schwartz, Eric T Bradlow, Peter S Fader","year":"2017","venue":"Marketing Science","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Thompson, 1933] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285- 294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"William R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xie et al., 2015] Miao Xie, Qiusong Yang, Qing Wang, Gao Cong, and Gerard De Melo. Dynadiffuse: A dynamic diffusion model for continuous time constrained influence maximization. In Twenty-Ninth AAAI Conference on Arti- ficial Intelligence, 2015.","title":"Dynadiffuse: A dynamic diffusion model for continuous time constrained influence maximization","authors":"Miao Xie, Qiusong Yang, Qing Wang, Gao Cong, Gerard De, Melo","year":"2015","venue":"Twenty-Ninth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}