{"type":"ScholarlyArticle","title":"Efficient and Robust High-Dimensional Linear Contextual Bandits","authors":[{"name":"Cheng Chen","slug":"cheng-chen","given":"Cheng","family":"Chen","affiliation":"Shanghai Jiao Tong University"},{"name":"Luo Luo","slug":"luo-luo","given":"Luo","family":"Luo","affiliation":"Hong Kong University of Science and Technology"},{"name":"Weinan Zhang","slug":"weinan-zhang","given":"Weinan","family":"Zhang","affiliation":"Shanghai Jiao Tong University"},{"name":"Yong Yu","slug":"yong-yu","given":"Yong","family":"Yu","affiliation":"Shanghai Jiao Tong University"},{"name":"Yijiang Lian","slug":"yijiang-lian","given":"Yijiang","family":"Lian","affiliation":"Baidu"}],"abstract":"The linear contextual bandits is a sequential decision-making problem where an agent decides among sequential actions given their corresponding contexts. Since large-scale data sets become more and more common, we study the linear contextual bandits in high-dimensional situations. Recent works focus on employing matrix sketching methods to accelerating contextual bandits. However, the matrix approximation error will bring additional terms to the regret bound. In this paper we first propose a novel matrix sketching method which is called Spectral Compensation Frequent Directions (SCFD). Then we propose an efficient approach for contextual bandits by adopting SCFD to approximate the covariance matrices. By maintaining and manipulating sketched matrices, our method only needs O(md) space and O(md) updating time in each round, where d is the dimensionality of the data and m is the sketching size. Theoretical analysis reveals that our method has better regret bounds than previous methods in high-dimensional cases. Experimental results demonstrate the effectiveness of our algorithm and verify our theoretical guarantees.","doi":"10.24963/ijcai.2020/588","url":"https://ijcai.org/proceedings/2020/588","pdf":"https://ijcai.org/proceedings/2020/0588.pdf","markdown":"https://ijcai.org/proceedings/2020/588.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/588","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Uncertainty in AI","track":"Main track","number":588,"pages":{"first":4259,"last":4265},"keywords":["Uncertainty in AI: Sequential Decision Making","Machine Learning: Online Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25174/file_25174.m3u8","length_sec":251,"provider":"hls"}],"references":[{"position":1,"raw":"[Abbasi-Yadkori et al., 2011] Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for lin- ear stochastic bandits. In Advances in Neural Information Processing Systems, pages 2312-2320, 2011.","title":"Improved algorithms for linear stochastic bandits","authors":"Yasin Abbasi-Yadkori, Dávid Pál, Csaba Szepesvári","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal and Goyal, 2013] Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In Proceedings of the 30th International Conference on Machine Learning, pages 127-135, 2013.","title":"Thompson sampling for contextual bandits with linear payoffs","authors":"Shipra Agrawal, Navin Goyal","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Auer, 2002] Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.","title":"Using confidence bounds for exploitation-exploration trade-offs","authors":"Peter Auer","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Calandriello et al., 2019] Daniele Calandriello, Luigi Car- ratino, Alessandro Lazaric, Michal Valko, and Lorenzo Rosasco. Gaussian process optimization with adaptive sketching: Scalable and no regret. In Proceedings of the 32nd Annual Conference on Learning Theory, pages 533- 557, 2019.","title":"Gaussian process optimization with adaptive sketching: Scalable and no regret","authors":"Daniele Calandriello, Luigi Carratino, Alessandro Lazaric, Michal Valko, Lorenzo Rosasco","year":"2019","venue":"Proceedings of the 32nd Annual Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carpentier and Munos, 2012] Alexandra Carpentier and Rémi Munos. Bandit theory meets compressed sensing for high dimensional stochastic linear bandit. In Proceedings of the 15th Artificial Intelligence and Statistics, 2012.","title":"Bandit theory meets compressed sensing for high dimensional stochastic linear bandit","authors":"Munos Carpentier, Alexandra Carpentier, Rémi Munos","year":"2012","venue":"Proceedings of the 15th Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chu et al., 2011] Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the 14th International Con- ference on Artificial Intelligence and Statistics, 2011.","title":"Contextual bandits with linear payoff functions","authors":"Wei Chu, Lihong Li, Lev Reyzin, Robert Schapire","year":"2011","venue":"Proceedings of the 14th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dani et al., 2008] Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. In Proceedings of the 21st Annual Con- ference on Learning Theory, 2008.","title":"Stochastic linear optimization under bandit feedback","authors":"Dani Varsha, Thomas P Hayes, M Sham, Kakade","year":"2008","venue":"Proceedings of the 21st Annual Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Desai et al., 2016] Amey Desai, Mina Ghashami, and Jeff M Phillips. Improved practical matrix sketching with guarantees. IEEE Transactions on Knowledge and Data Engineering, 28(7):1678-1690, 2016.","title":"Improved practical matrix sketching with guarantees","authors":"Amey Desai, Mina Ghashami, Jeff M Phillips","year":"2016","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Deshpande and Montanari, 2012] Yash Deshpande and An- drea Montanari. Linear bandits in high dimension and rec- ommendation systems. In 50th Annual Allerton Confer- ence on Communication, Control, and Computing (Aller- ton), pages 1750-1754. IEEE, 2012.","title":"Yash Deshpande and Andrea Montanari. Linear bandits in high dimension and recommendation systems","authors":"Montanari Deshpande","year":"2012","venue":"50th Annual Allerton Conference on Communication, Control, and Computing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dumitrascu et al., 2018] Bianca Dumitrascu, Karen Feng, and Barbara Engelhardt. Pg-ts: Improved thompson sam- pling for logistic contextual bandits. In Advances in Neural Information Processing Systems, pages 4629-4638, 2018.","title":"Pg-ts: Improved thompson sampling for logistic contextual bandits","authors":"Bianca Dumitrascu, Karen Feng, Barbara Engelhardt","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Filippi et al., 2010] Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. In Advances in Neural Information Processing Systems, pages 586-594, 2010.","title":"Parametric bandits: The generalized linear case","authors":"Sarah Filippi, Olivier Cappe, Aurélien Garivier, Csaba Szepesvári","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ghashami et al., 2016] Mina Ghashami, Edo Liberty, Jeff M Phillips, and David P Woodruff. Frequent direc- tions: Simple and deterministic matrix sketching. SIAM Journal on Computing, 45(5):1762-1792, 2016.","title":"Frequent directions: Simple and deterministic matrix sketching","authors":"Ghashami","year":"2016","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009. [Kuzborskij et al., 2019] Ilja Kuzborskij, Leonardo Cella, and Nicolò Cesa-Bianchi. Efficient linear bandits through matrix sketching. Proceedings of the 22th Artificial Intel- ligence and Statistics, 2019.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton, ; Kuzborskij","year":"2009","venue":"Proceedings of the 22th Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Langford and Zhang, 2008] John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in Neural Information Processing Systems, pages 817-824, 2008.","title":"The epoch-greedy algorithm for multi-armed bandits with side information","authors":"Zhang Langford, John Langford, Tong Zhang","year":"2008","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to per- sonalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web, pages 661-670. ACM, 2010.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"Proceedings of the 19th International Conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2017] Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contex- tual bandits. In Proceedings of the 34th International Con- ference on Machine Learning, pages 2071-2080, 2017.","title":"Provably optimal algorithms for generalized linear contextual bandits","authors":"Lihong Li, Yu Lu, Dengyong Zhou","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liberty, 2013] Edo Liberty. Simple and deterministic matrix sketching. In Proceedings of the 19th ACM SIGKDD In- ternational Conference on Knowledge Discovery and Data mining, pages 581-588. ACM, 2013.","title":"Edo Liberty. Simple and deterministic matrix sketching","authors":"Liberty","year":"2013","venue":"Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data mining","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Luo et al., 2016] Haipeng Luo, Alekh Agarwal, Nicolo Cesa-Bianchi, and John Langford. Efficient second order online learning by sketching. In Advances in Neural Infor- mation Processing Systems, pages 902-910, 2016.","title":"Efficient second order online learning by sketching","authors":"Haipeng Luo, Alekh Agarwal, Nicolo Cesa-Bianchi, John Langford","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Luo et al., 2019] Luo Luo, Cheng Chen, Zhihua Zhang, Wu-Jun Li, and Tong Zhang. Robust frequent directions with application in online learning. Journal of Machine Learning Research, 20(45):1-41, 2019.","title":"Robust frequent directions with application in online learning","authors":"Luo Luo, Cheng Chen, Zhihua Zhang, Wu-Jun Li, Tong Zhang","year":"2019","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Russo and Van Roy, 2014] Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research, 2014.","title":"Learning to optimize via posterior sampling","authors":"Van Russo, Daniel Roy, Benjamin Russo, Roy Van","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tang et al., 2013] Liang Tang, Romer Rosales, Ajit Singh, and Deepak Agarwal. Automatic ad format selection via contextual bandits. In Proceedings of the 22nd ACM Inter- national Conference on Information & Knowledge Man- agement, pages 1587-1594. ACM, 2013.","title":"Automatic ad format selection via contextual bandits","authors":"Liang Tang, Romer Rosales, Ajit Singh, Deepak Agarwal","year":"2013","venue":"Proceedings of the 22nd ACM International Conference on Information & Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tewari and Murphy, 2017] Ambuj Tewari and Susan A Murphy. From ads to interventions: Contextual bandits in mobile health. In Mobile Health. Springer, 2017.","title":"From ads to interventions: Contextual bandits in mobile health","authors":"Ambuj Tewari, Susan A Murphy","year":"2017","venue":"Mobile Health","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2017] Xiaotian Yu, Michael R Lyu, and Irwin King. Cbrap: Contextual bandits with random projection. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, 2017.","title":"Cbrap: Contextual bandits with random projection","authors":"Xiaotian Yu, Irwin Michael R Lyu, King","year":"2017","venue":"Proceedings of the 31st AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhao et al., 2014] Tong Zhao, Julian McAuley, and Irwin King. Leveraging social connections to improve person- alized ranking for collaborative filtering. In Proceedings of the 23rd ACM International Conference on Information and Knowledge Management. ACM, 2014.","title":"Leveraging social connections to improve personalized ranking for collaborative filtering","authors":"Tong Zhao, Julian Mcauley, Irwin King","year":"2014","venue":"Proceedings of the 23rd ACM International Conference on Information and Knowledge Management","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}