{"type":"ScholarlyArticle","title":"Linear Combinatorial Semi-Bandit with Causally Related Rewards","authors":[{"name":"Behzad Nourani-Koliji","slug":"behzad-nourani-koliji","given":"Behzad","family":"Nourani-Koliji","affiliation":"University of Tübingen"},{"name":"Saeed Ghoorchian","slug":"saeed-ghoorchian","given":"Saeed","family":"Ghoorchian","affiliation":"Technical University of Berlin"},{"name":"Setareh Maghsudi","slug":"setareh-maghsudi","given":"Setareh","family":"Maghsudi","affiliation":"University of Tübingen"}],"abstract":"In a sequential decision-making problem, having a structural dependency amongst the reward distributions associated with the arms makes it challenging to identify a subset of alternatives that guarantees the optimal collective outcome. Thus, besides individual actions' reward, learning the causal relations is essential to improve the decision-making strategy. To solve the two-fold learning problem described above, we develop the 'combinatorial semi-bandit framework with causally related rewards', where we model the causal relations by a directed graph in a stationary structural equation model. The nodal observation in the graph signal comprises the corresponding base arm's instantaneous reward and an additional term resulting from the causal influences of other base arms' rewards. The objective is to maximize the long-term average payoff, which is a linear function of the base arms' rewards and depends strongly on the network topology. To achieve this objective, we propose a policy that determines the causal relations by learning the network's topology and simultaneously exploits this knowledge to optimize the decision-making process. We establish a sublinear regret bound for the proposed algorithm. Numerical experiments using synthetic and real-world datasets demonstrate the superior performance of our proposed method compared to several benchmarks.","doi":"10.24963/ijcai.2022/676","url":"https://ijcai.org/proceedings/2022/676","pdf":"https://ijcai.org/proceedings/2022/0676.pdf","markdown":"https://ijcai.org/proceedings/2022/676.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/676","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Uncertainty in AI","track":"Main Track","number":676,"pages":{"first":4878,"last":4884},"keywords":["Machine Learning: Online Learning","Uncertainty in AI: Sequential Decision Making","Machine Learning: Sequence and Graph Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987050","length_sec":91,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985193","length_sec":537,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Auer et al., 2002] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Auer","year":"2002","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bazerque et al., 2013] Juan Andrés Bazerque, Brian Bain- gana, and Georgios B Giannakis. Identifiability of sparse structural equation models for directed and cyclic net- works. In 2013 IEEE Global Conference on Signal and Information Processing, pages 839-842. IEEE, 2013.","title":"Identifiability of sparse structural equation models for directed and cyclic networks","authors":"Bazerque","year":"2013","venue":"IEEE Global Conference on Signal and Information Processing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bridgwater and Bóta, 2021] Alexander Bridgwater and András Bóta. Identifying regions most likely to contribute to an epidemic outbreak in a human mobility network. In 2021 Swedish Artificial Intelligence Society Workshop (SAIS), pages 1-4. IEEE, 2021.","title":"Identifying regions most likely to contribute to an epidemic outbreak in a human mobility network","authors":"Alexander Bridgwater, András Bóta","year":"2021","venue":"Swedish Artificial Intelligence Society Workshop (SAIS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bull, 2021] Martin Bull. The italian government response to covid-19 and the making of a prime minister. Contem- porary Italian Politics, pages 1-17, 2021.","title":"The italian government response to covid-19 and the making of a prime minister","authors":"Martin Bull","year":"2021","venue":"Contemporary Italian Politics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2013] Wei Chen, Yajun Wang, and Yang Yuan. Combinatorial multi-armed bandit: General framework and applications. In International Conference on Machine Learning, pages 151-159. PMLR, 2013.","title":"Combinatorial multi-armed bandit: General framework and applications","authors":"Wei Chen, Yajun Wang, Yang Yuan","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2016] Wei Chen, Yajun Wang, Yang Yuan, and Qinshi Wang. Combinatorial multi-armed bandit and its extension to probabilistically triggered arms. The Journal of Machine Learning Research, 17(1):1746-1778, 2016.","title":"Combinatorial multi-armed bandit and its extension to probabilistically triggered arms","authors":"Wei Chen, Yajun Wang, Yang Yuan, Qinshi Wang","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dong et al., 2019] Xiaowen Dong, Dorina Thanou, Michael Rabbat, and Pascal Frossard. Learning graphs from data: A signal representation perspective. IEEE Signal Process- ing Magazine, 36(3):44-63, 2019.","title":"Learning graphs from data: A signal representation perspective","authors":"Xiaowen Dong, Dorina Thanou, Michael Rabbat, Pascal Frossard","year":"2019","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Giannakis et al., 2018] Georgios B Giannakis, Yanning Shen, and Georgios Vasileios Karanikolas. Topology identification and learning over graphs: Accounting for nonlinearities and dynamics. Proceedings of the IEEE, 106(5):787-807, 2018.","title":"Topology identification and learning over graphs: Accounting for nonlinearities and dynamics","authors":"Yanning Georgios B Giannakis, Georgios Shen, Karanikolas Vasileios","year":"2018","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guaitoli and Pancrazi, 2021] Gabriele Guaitoli and Roberto Pancrazi. Covid-19: Regional policies and local infection risk: Evidence from italy with a modelling study. The Lancet Regional Health-Europe, 8:100169, 2021.","title":"Gabriele Guaitoli and Roberto Pancrazi. Covid-19: Regional policies and local infection risk: Evidence from italy with a modelling study","authors":"Pancrazi Guaitoli","year":"2021","venue":"The Lancet Regional Health-Europe","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huyuk and Tekin, 2019] Alihan Huyuk and Cem Tekin. Analysis of thompson sampling for combinatorial multi- armed bandit with probabilistically triggered arms. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1322-1330. PMLR, 2019.","title":"Analysis of thompson sampling for combinatorial multiarmed bandit with probabilistically triggered arms","authors":"Tekin Huyuk, Alihan Huyuk, Cem Tekin","year":"2019","venue":"The 22nd International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kaplan, 2008] David Kaplan. Structural equation model- ing: Foundations and extensions, volume 10. Sage Publi- cations, 2008.","title":"Structural equation modeling: Foundations and extensions","authors":"David Kaplan","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lattimore et al., 2016] Finnian Lattimore, Tor Lattimore, and Mark D Reid. Causal bandits: learning good inter- ventions via causal inference. In Proceedings of the 30th International Conference on Neural Information Process- ing Systems, pages 1189-1197, 2016.","title":"Causal bandits: learning good interventions via causal inference","authors":"Finnian Lattimore, Tor Lattimore, Mark D Reid","year":"2016","venue":"Proceedings of the 30th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Maghsudi and Hossain, 2016] Setareh Maghsudi and Ekram Hossain. Multi-armed bandits with application to 5g small cells. IEEE Wireless Communications, 23(3):64-73, 2016.","title":"Multi-armed bandits with application to 5g small cells","authors":"Hossain Maghsudi, Setareh Maghsudi, Ekram Hossain","year":"2016","venue":"IEEE Wireless Communications","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mastakouri and Schölkopf, 2020] Atalanti Mastakouri and Bernhard Schölkopf. Causal analysis of covid-19 spread in germany. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural In- formation Processing Systems, volume 33, pages 3153- 3163.","title":"Atalanti Mastakouri and Bernhard Schölkopf. Causal analysis of covid-19 spread in germany","authors":"Schölkopf Mastakouri","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nouvellet et al., 2021] Pierre Nouvellet, Sangeeta Bhatia, Anne Cori, Kylie EC Ainslie, Marc Baguelin, Samir Bhatt, Adhiratha Boonyasiri, Nicholas F Brazeau, Lorenzo Cat- tarino, Laura V Cooper, et al. Reduction in mobility and covid-19 transmission. Nature communications, 12(1):1- 9, 2021.","title":"Reduction in mobility and covid-19 transmission","authors":"Pierre Nouvellet, Sangeeta Bhatia, Anne Cori, Kylie Ec Ainslie, Marc Baguelin, Samir Bhatt, Adhiratha Boonyasiri, Nicholas F Brazeau, Lorenzo Cattarino, Laura V Cooper","year":"2021","venue":"Nature communications","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Robbins, 1952] Herbert Robbins. Some aspects of the se- quential design of experiments. Bulletin of the American Mathematical Society, 58(5):527-535, 1952.","title":"Herbert Robbins. Some aspects of the sequential design of experiments","authors":"Robbins","year":"1952","venue":"Bulletin of the American Mathematical Society","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sardellitti et al., 2017] Stefania Sardellitti, Sergio Bar- barossa, and Paolo Di Lorenzo. On the graph fourier trans- form for directed graphs. IEEE Journal of Selected Topics in Signal Processing, 11(6):796-811, 2017.","title":"On the graph fourier transform for directed graphs","authors":"Stefania Sardellitti, Sergio Barbarossa, Paolo Di, Lorenzo","year":"2017","venue":"IEEE Journal of Selected Topics in Signal Processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sen et al., 2017] Rajat Sen, Karthikeyan Shanmugam, Alexandros G Dimakis, and Sanjay Shakkottai. Iden- tifying best interventions through online importance sampling. In International Conference on Machine Learning, pages 3057-3066. PMLR, 2017.","title":"Identifying best interventions through online importance sampling","authors":"Rajat Sen, Karthikeyan Shanmugam, Alexandros G Dimakis, Sanjay Shakkottai","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Stellato et al., 2020] B. Stellato, G. Banjac, P. Goulart, A. Bemporad, and S. Boyd. OSQP: an operator splitting solver for quadratic programs. Mathematical Program- ming Computation, 12(4):637-672, 2020.","title":"OSQP: an operator splitting solver for quadratic programs","authors":"B Stellato, G Banjac, P Goulart, A Bemporad, S Boyd","year":"2020","venue":"Mathematical Programming Computation","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tang et al., 2017] Shaojie Tang, Yaqin Zhou, Kai Han, Zhao Zhang, Jing Yuan, and Weili Wu. Networked stochastic multi-armed bandits with combinatorial strate- gies. In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), pages 786-793. IEEE, 2017.","title":"Networked stochastic multi-armed bandits with combinatorial strategies","authors":"Shaojie Tang, Yaqin Zhou, Kai Han, Zhao Zhang, Jing Yuan, Weili Wu","year":"2017","venue":"IEEE 37th International Conference on Distributed Computing Systems (ICDCS)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Toni and Frossard, 2018] Laura Toni and Pascal Frossard. Spectral mab for unknown graph processes. In 2018 26th European Signal Processing Conference (EUSIPCO), pages 116-120. IEEE, 2018.","title":"Spectral mab for unknown graph processes","authors":"Toni, Frossard, Laura Toni, Pascal Frossard","year":"2018","venue":"26th European Signal Processing Conference (EUSIPCO)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yu et al., 2020] Tong Yu, Branislav Kveton, Zheng Wen, Ruiyi Zhang, and Ole J Mengshoel. Graphical models meet bandits: A variational thompson sampling approach. In International Conference on Machine Learning, pages 10902-10912. PMLR, 2020.","title":"Graphical models meet bandits: A variational thompson sampling approach","authors":"Tong Yu, Branislav Kveton, Zheng Wen, Ruiyi Zhang, Ole J Mengshoel","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zimmert et al., 2019] Julian Zimmert, Haipeng Luo, and Chen-Yu Wei. Beating stochastic and adversarial semi- bandits optimally and simultaneously. In International Conference on Machine Learning, pages 7683-7692.","title":"Beating stochastic and adversarial semibandits optimally and simultaneously","authors":"Julian Zimmert, Haipeng Luo, Chen-Yu Wei","year":null,"venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"PMLR, 2019.","title":null,"authors":"Pmlr","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}