{"type":"ScholarlyArticle","title":"Marginal Posterior Sampling for Slate Bandits","authors":[{"name":"Maria Dimakopoulou","slug":"maria-dimakopoulou","given":"Maria","family":"Dimakopoulou","affiliation":""},{"name":"Nikos Vlassis","slug":"nikos-vlassis","given":"Nikos","family":"Vlassis","affiliation":""},{"name":"Tony Jebara","slug":"tony-jebara","given":"Tony","family":"Jebara","affiliation":""}],"abstract":"We introduce a new Thompson sampling-based algorithm, called marginal posterior sampling, for online slate bandits, that is characterized by three key ideas. First, it postulates that the slate-level reward is a monotone function of the marginal unobserved rewards of the base actions selected in the slates's slots, but it does not attempt to estimate this function. Second, instead of maintaining a slate-level reward posterior, the algorithm maintains posterior distributions for the marginal reward of each slot's base actions and uses the samples from these marginal posteriors to select the next slate. Third, marginal posterior sampling optimizes at the slot-level rather than the slate-level, which makes the approach computationally efficient. Simulation results establish substantial advantages of marginal posterior sampling over alternative Thompson sampling-based approaches that are widely used in the domain of web services.","doi":"10.24963/ijcai.2019/308","url":"https://ijcai.org/proceedings/2019/308","pdf":"https://ijcai.org/proceedings/2019/0308.pdf","markdown":"https://ijcai.org/proceedings/2019/308.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/308","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":308,"pages":{"first":2223,"last":2229},"keywords":["Machine Learning: Learning Preferences or Rankings","Machine Learning: Online Learning"],"videos":[],"references":[{"position":1,"raw":"[Agrawal and Goyal, 2013] Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International Conference on Machine Learning, pages 127-135, 2013.","title":"Thompson sampling for contextual bandits with linear payoffs","authors":"Shipra Agrawal, Navin Goyal","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal et al., 2017] Shipra Agrawal, Vashist Avadhanula, Vineet Goyal, and Assaf Zeevi. Mnl-bandit: A dynamic learning approach to assortment selection. arXiv preprint arXiv:1706.03880, 2017.","title":"Mnl-bandit: A dynamic learning approach to assortment selection","authors":"Shipra Agrawal, Vashist Avadhanula, Vineet Goyal, Assaf Zeevi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cesa-Bianchi and Lugosi, 2012] Nicolo Cesa-Bianchi and Gábor Lugosi. Combinatorial bandits. Journal of Com- puter and System Sciences, 78(5):1404-1422, 2012.","title":"Nicolo Cesa-Bianchi and Gábor Lugosi","authors":"Cesa-Bianchi, Lugosi","year":"2012","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chapelle and Li, 2011] Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. In Advances in neural information processing systems, pages 2249- 2257, 2011.","title":"An empirical evaluation of thompson sampling","authors":"Li ; Chapelle, Olivier Chapelle, Lihong Li","year":"2011","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2013] Wei Chen, Yajun Wang, and Yang Yuan. Combinatorial multi-armed bandit: General framework and applications. In International Conference on Machine Learning, pages 151-159, 2013.","title":"Combinatorial multi-armed bandit: General framework and applications","authors":"Wei Chen, Yajun Wang, Yang Yuan","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dimakopoulou et al., 2019] Maria Dimakopoulou, Zhengyuan Zhou, Susan Athey, and Guido Imbens. Balanced linear contextual bandits. Thirty-Third AAAI Conference on Artificial Intelligence, 2019.","title":"Balanced linear contextual bandits","authors":"Maria Dimakopoulou, Zhengyuan Zhou, Susan Athey, Guido Imbens","year":"2019","venue":"Thirty-Third AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kale et al., 2010] Satyen Kale, Lev Reyzin, and Robert E Schapire. Non-stochastic bandit slate problems. In Ad- vances in Neural Information Processing Systems, pages 1054-1062, 2010.","title":"Non-stochastic bandit slate problems","authors":"Satyen Kale, Lev Reyzin, Robert E Schapire","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lagrée et al., 2016] Paul Lagrée, Claire Vernade, and Olivier Cappé. Multiple-play bandits in the position-based model. In Advances in Neural Information Processing Sys- tems, pages 1597-1605, 2016.","title":"Multiple-play bandits in the position-based model","authors":"Lagrée","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lai and Robbins, 1985] Tze Leung Lai and Herbert Rob- bins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4-22, 1985.","title":"Asymptotically efficient adaptive allocation rules","authors":"Tze Leung, Lai, Herbert Robbins","year":"1985","venue":"Advances in applied mathematics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to per- sonalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661-670. ACM, 2010.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"Proceedings of the 19th international conference on World wide web","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2016] Shuai Li, Baoxiang Wang, Shengyu Zhang, and Wei Chen. Contextual combinatorial cascading ban- dits. In ICML, volume 16, pages 1245-1253, 2016.","title":"Contextual combinatorial cascading bandits","authors":"Shuai Li, Baoxiang Wang, Shengyu Zhang, Wei Chen","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2017] Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear con- textual bandits. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2071- 2080. JMLR. org, 2017.","title":"Provably optimal algorithms for generalized linear contextual bandits","authors":"Lihong Li, Yu Lu, Dengyong Zhou","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2018] Shuai Li, Yasin Abbasi-Yadkori, Branislav Kveton, S Muthukrishnan, Vishwa Vinay, and Zheng Wen. Offline evaluation of ranking policies with click models. In Proceedings of the 24th ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining, pages 1685-1694. ACM, 2018.","title":"Offline evaluation of ranking policies with click models","authors":"Shuai Li, Yasin Abbasi-Yadkori, Branislav Kveton, S Muthukrishnan, Vishwa Vinay, Zheng Wen","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Qin et al., 2014] Lijing Qin, Shouyuan Chen, and Xiaoyan Zhu. Contextual combinatorial bandit and its application on diversified online recommendation. In Proceedings of the 2014 SIAM International Conference on Data Mining, pages 461-469. SIAM, 2014.","title":"Contextual combinatorial bandit and its application on diversified online recommendation","authors":"Qin","year":"2014","venue":"Proceedings of the 2014 SIAM International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Russo et al., 2018] Daniel J Russo, Benjamin Van Roy, Ab- bas Kazerouni, Ian Osband, Zheng Wen, et al. A tutorial on thompson sampling. Foundations and Trends R in Ma- chine Learning, 11(1):1-96, 2018.","title":"A tutorial on thompson sampling","authors":"J Daniel, Benjamin Russo, Abbas Van Roy, Ian Kazerouni, Zheng Osband, Wen","year":"2018","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Swaminathan et al., 2017] Adith Swaminathan, Akshay Kr- ishnamurthy, Alekh Agarwal, Miro Dudik, John Langford, Damien Jose, and Imed Zitouni. Off-policy evaluation for slate recommendation. In Advances in Neural Information Processing Systems, pages 3632-3642, 2017.","title":"Off-policy evaluation for slate recommendation","authors":"Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miro Dudik, John Langford, Damien Jose, Imed Zitouni","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Thompson, 1933] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285- 294, 1933.","title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","authors":"William R Thompson","year":"1933","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2017] Yingfei Wang, Hua Ouyang, Chu Wang, Jianhui Chen, Tsvetan Asamov, and Yi Chang. Efficient ordered combinatorial semi-bandits for whole-page rec- ommendation. In Thirty-First AAAI Conference on Arti- ficial Intelligence, 2017.","title":"Efficient ordered combinatorial semi-bandits for whole-page recommendation","authors":"Yingfei Wang, Hua Ouyang, Chu Wang, Jianhui Chen, Tsvetan Asamov, Yi Chang","year":"2017","venue":"Thirty-First AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}