{"type":"ScholarlyArticle","title":"AdaLinUCB: Opportunistic Learning for Contextual Bandits","authors":[{"name":"Xueying Guo","slug":"xueying-guo","given":"Xueying","family":"Guo","affiliation":"University of California , Davis"},{"name":"Xiaoxiao Wang","slug":"xiaoxiao-wang","given":"Xiaoxiao","family":"Wang","affiliation":"University of California , Davis"},{"name":"Xin Liu","slug":"xin-liu","given":"Xin","family":"Liu","affiliation":"University of California , Davis"}],"abstract":"In this paper, we propose and study opportunistic contextual bandits - a special case of contextual bandits where the exploration cost varies under different environmental conditions, such as network load or return variation in recommendations. When the exploration cost is low, so is the actual regret of pulling a sub-optimal arm (e.g., trying a suboptimal recommendation). Therefore, intuitively, we could explore more when the exploration cost is relatively low and exploit more when the exploration cost is relatively high. Inspired by this intuition, for opportunistic contextual bandits with Linear payoffs, we propose an Adaptive Upper-Confidence-Bound algorithm (AdaLinUCB) to adaptively balance the exploration-exploitation trade-off for opportunistic learning. We prove that AdaLinUCB achieves O((log T)^2) problem-dependent regret upper bound, which has a smaller coefficient than that of the traditional LinUCB algorithm. Moreover, based on both synthetic and real-world dataset, we show that AdaLinUCB significantly outperforms other contextual bandit algorithms, under large exploration cost fluctuations.","doi":"10.24963/ijcai.2019/336","url":"https://ijcai.org/proceedings/2019/336","pdf":"https://ijcai.org/proceedings/2019/0336.pdf","markdown":"https://ijcai.org/proceedings/2019/336.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/336","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":336,"pages":{"first":2420,"last":2427},"keywords":["Uncertainty in AI: Sequential Decision Making","Machine Learning: Cost-Sensitive Learning","Uncertainty in AI: Uncertainty in AI"],"videos":[],"references":[{"position":1,"raw":"[Abbasi-Yadkori et al., 2011] Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for lin- ear stochastic bandits. In Proceedings of the 24th Interna- tional Conference on Neural Information Processing Sys- tems, NIPS'11, pages 2312-2320, 2011.","title":"Improved algorithms for linear stochastic bandits","authors":"Yasin Abbasi-Yadkori, Dávid Pál, Csaba Szepesvári","year":"2011","venue":"Proceedings of the 24th International Conference on Neural Information Processing Systems, NIPS'11","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal, 1995] Rajeev Agrawal. Sample mean based index policies with o(log n) regret for the multi-armed bandit problem. Advances in Applied Probability, 27(4):1054- 1078, 1995.","title":"Sample mean based index policies with o(log n) regret for the multi-armed bandit problem","authors":"Rajeev Agrawal","year":"1995","venue":"Advances in Applied Probability","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Auer et al., 2002] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Mach. Learn., 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Auer, 2002] Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. J. Mach. Learn. Res., 3:397-422, 2002.","title":"Using confidence bounds for exploitation-exploration trade-offs","authors":"Peter Auer","year":"2002","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bouneffouf et al., 2012] Djallel Bouneffouf, Amel Bouzeghoub, and Alda Lopes Ganc ¸arski. A contextual- bandit algorithm for mobile context-aware recommender system. In International conference on Neural Infor- mation Processing (ICONIP), pages 324-331, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg.","title":"Djallel Bouneffouf, Amel Bouzeghoub, and Alda Lopes Ganc ¸arski. A contextualbandit algorithm for mobile context-aware recommender system","authors":"Bouneffouf","year":"2012","venue":"International conference on Neural Information Processing (ICONIP)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chapelle and Li, 2011] Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. In J. Shawe- Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Pro- cessing Systems 24, pages 2249-2257.","title":"An empirical evaluation of thompson sampling","authors":"Li ; Chapelle, Olivier Chapelle, Lihong Li","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Curran Associates, Inc., 2011.","title":null,"authors":"Curran Associates, Inc","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chu et al., 2011] Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, vol- ume 15 of AISTATS, pages 208-214, 2011.","title":"Contextual bandits with linear payoff functions","authors":"Wei Chu, Lihong Li, Lev Reyzin, Robert Schapire","year":"2011","venue":"Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chuai et al., 2019] Jie Chuai, Zhitang Chen, Guochen Liu, Xueying Guo, Xiaoxiao Wang, Xin Liu, Chongming Zhu, and Feiyi Shen. A collaborative learning based approach for parameter configuration of cellular networks. In IEEE International Conference on Computer Communications (INFOCOM), 2019.","title":"A collaborative learning based approach for parameter configuration of cellular networks","authors":"Jie Chuai, Zhitang Chen, Guochen Liu, Xueying Guo, Xiaoxiao Wang, Xin Liu, Chongming Zhu, Feiyi Shen","year":"2019","venue":"IEEE International Conference on Computer Communications (INFOCOM)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Filippi et al., 2010] Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. In J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems 23, pages 586-594.","title":"Parametric bandits: The generalized linear case","authors":"Sarah Filippi, Olivier Cappe, Aurélien Garivier, Csaba Szepesvári","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Curran Associates, Inc., 2010.","title":null,"authors":"Curran Associates, Inc","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Langford and Zhang, 2008] John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 817-824.","title":"The epoch-greedy algorithm for multi-armed bandits with side information","authors":"Zhang ; ] Langford, John Langford, Tong Zhang","year":"2008","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Curran Associates, Inc., 2008.","title":null,"authors":"Curran Associates, Inc","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to per- sonalized news article recommendation. In the 19th Inter- national Conference on World Wide Web (WWW), 2010.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"the 19th International Conference on World Wide Web (WWW)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2011] Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual- bandit-based news article recommendation algorithms. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining, WSDM '11, pages 297- 306. ACM, 2011.","title":"Unbiased offline evaluation of contextualbandit-based news article recommendation algorithms","authors":"Lihong Li, Wei Chu, John Langford, Xuanhui Wang","year":"2011","venue":"Proceedings of the Fourth ACM International Conference on Web Search and Data Mining, WSDM '11","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rossman, 2015] Rossman. Rossmann Store sales data, 2015. https://www.kaggle.com/c/rossmann-store-sales/ data.","title":"Rossmann Store sales data","authors":"Rossman","year":"2015","venue":null,"doi":null,"url":"https://www.kaggle.com/c/rossmann-store-sales/data","internal_url":null},{"position":17,"raw":"[Valko et al., 2013] Michal Valko, Nathaniel Korda, Rémi Munos, Ilias Flaounas, and Nelo Cristianini. Finite-time analysis of kernelised contextual bandits. arXiv preprint arXiv:1309.6869, 2013.","title":"Finite-time analysis of kernelised contextual bandits","authors":"Valko","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Walraevens et al., 2003] Joris Walraevens, Bart Steyaert, and Herwig Bruneel. Performance analysis of a single- server atm queue with a priority scheduling. Computers & Operations Research, 30(12):1807 -1829, 2003.","title":"Performance analysis of a singleserver atm queue with a priority scheduling","authors":"Joris Walraevens, Bart Steyaert, Herwig Bruneel","year":"2003","venue":"Computers & Operations Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2016] Huazheng Wang, Qingyun Wu, and Hongning Wang. Learning hidden features for contextual bandits. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM '16, pages 1633-1642. ACM, 2016.","title":"Learning hidden features for contextual bandits","authors":"Wang","year":"2016","venue":"Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM '16","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2017] Huazheng Wang, Qingyun Wu, and Hongning Wang. Factorization bandits for interactive rec- ommendation. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, 2017.","title":"Factorization bandits for interactive recommendation","authors":"Wang","year":"2017","venue":"Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2016] Qingyun Wu, Huazheng Wang, Quanquan Gu, and Hongning Wang. Contextual bandits in a collab- orative environment. In Proceedings of the 39th Interna- tional ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, SIGIR '16, pages 529-538. ACM, 2016.","title":"Contextual bandits in a collaborative environment","authors":"Qingyun Wu, Huazheng Wang, Quanquan Gu, Hongning Wang","year":"2016","venue":"Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2018] Huasen Wu, Xueying Guo, and Xin Liu. Adaptive exploration-exploitation tradeoff for opportunis- tic bandits. In ICML, 2018.","title":"Adaptive exploration-exploitation tradeoff for opportunistic bandits","authors":"Wu","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}