{"type":"ScholarlyArticle","title":"Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledge","authors":[{"name":"Yupei Yang","slug":"yupei-yang","given":"Yupei","family":"Yang","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , China"},{"name":"Biwei Huang","slug":"biwei-huang","given":"Biwei","family":"Huang","affiliation":"Halicioglu Data Science Institute (HDSI) , University of California San Diego , USA"},{"name":"Shikui Tu","slug":"shikui-tu","given":"Shikui","family":"Tu","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , China"},{"name":"Lei Xu","slug":"lei-xu","given":"Lei","family":"Xu","affiliation":"Department of Computer Science and Engineering , Shanghai Jiao Tong University , China; Guangdong Institute of Intelligence Science and Technology , Zhuhai , China"}],"abstract":"The effectiveness of model training heavily relies on the quality of available training resources. However, budget constraints often impose limitations on data collection efforts. To tackle this challenge, we introduce causal exploration in this paper, a strategy that leverages the underlying causal knowledge for both data collection and model training. We, in particular, focus on enhancing the sample efficiency and reliability of the world model learning within the domain of task-agnostic reinforcement learning. During the exploration phase, the agent actively selects actions expected to yield causal insights most beneficial for world model training. Concurrently, the causal knowledge is acquired and incrementally refined with the ongoing collection of data. We demonstrate that causal exploration aids in learning accurate world models using fewer data and provide theoretical guarantees for its convergence. Empirical experiments, on both synthetic data and real-world applications, further validate the benefits of causal exploration. The source code is available at https://github.com/CMACH508/CausalExploration.","doi":"10.24963/ijcai.2024/591","url":"https://ijcai.org/proceedings/2024/591","pdf":"https://ijcai.org/proceedings/2024/0591.pdf","markdown":"https://ijcai.org/proceedings/2024/591.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/591","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":591,"pages":{"first":5344,"last":5352},"keywords":["Machine Learning: ML: Reinforcement learning","Machine Learning: ML: Active learning","Uncertainty in AI: UAI: Causality, structural causal models and causal inference","Machine Learning: ML: Causality"],"videos":[],"references":[{"position":1,"raw":"[Aubret et al., 2019] Arthur Aubret, Laetitia Matignon, and Salima Hassas. A survey on intrinsic motivation in re- inforcement learning. arXiv preprint arXiv:1908.06976, 2019.","title":"A survey on intrinsic motivation in reinforcement learning","authors":"Aubret","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Burda et al., 2018] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355, 2018.","title":"Large-scale study of curiosity-driven learning","authors":"Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, Alexei A Efros","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ding et al., 2022] Wenhao Ding, Haohong Lin, Bo Li, and Ding Zhao. Generalizing goal-conditioned reinforcement learning with variational causal reasoning. Advances in Neural Information Processing Systems, 35:26532-26548, 2022.","title":"Generalizing goal-conditioned reinforcement learning with variational causal reasoning","authors":"Wenhao Ding, Haohong Lin, Bo Li, Ding Zhao","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duan et al., 2016] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Interna- tional conference on machine learning, pages 1329-1338.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel","year":null,"venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"PMLR, 2016.","title":null,"authors":"Pmlr","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fang et al., 2017] Meng Fang, Yuan Li, and Trevor Cohn. Learning how to active learn: A deep reinforcement learn- ing approach. arXiv preprint arXiv:1708.02383, 2017.","title":"Learning how to active learn: A deep reinforcement learning approach","authors":"Meng Fang, Yuan Li, Trevor Cohn","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2022] Xing Hu, Rui Zhang, Ke Tang, Jiaming Guo, Qi Yi, Ruizhi Chen, Zidong Du, Ling Li, Qi Guo, Yunji Chen, et al. Causality-driven hierarchical structure discovery for reinforcement learning. Advances in Neural Information Processing Systems, 35:20064-20076, 2022.","title":"Causality-driven hierarchical structure discovery for reinforcement learning","authors":"Xing Hu, Rui Zhang, Ke Tang, Jiaming Guo, Qi Yi, Ruizhi Chen, Zidong Du, Ling Li, Qi Guo, Yunji Chen","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2021] Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. arXiv preprint arXiv:2107.02729, 2021.","title":"Adarl: What, where, and how to adapt in transfer reinforcement learning","authors":"Huang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2022] Biwei Huang, Chaochao Lu, Liu Leqi, José Miguel Hernández-Lobato, Clark Glymour, Bernhard Schölkopf, and Kun Zhang. Action-sufficient state repre- sentation learning for control with structural constraints. In International Conference on Machine Learning, pages 9260-9279. PMLR, 2022.","title":"Action-sufficient state representation learning for control with structural constraints","authors":"Huang","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ke et al., 2021] Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Christopher Pal. Systematic evaluation of causal discovery in visual model based reinforcement learning. arXiv preprint arXiv:2107.00848, 2021.","title":"Systematic evaluation of causal discovery in visual model based reinforcement learning","authors":"Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, Christopher Pal","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2018] Hyoungseok Kim, Jaekyeom Kim, Yeon- woo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Exploration with mutual information. arXiv preprint arXiv:1810.01176, 2018.","title":"Emi: Exploration with mutual information","authors":"Kim","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kosoy et al., 2022] Eliza Kosoy, Adrian Liu, Jasmine L Collins, David Chan, Jessica B Hamrick, Nan Rosemary Ke, Sandy Huang, Bryanna Kaufmann, John Canny, and Alison Gopnik. Learning causal overhypotheses through exploration in children and computational models. In Con- ference on Causal Learning and Reasoning, pages 390- 406. PMLR, 2022.","title":"Learning causal overhypotheses through exploration in children and computational models","authors":"Eliza Kosoy, Adrian Liu, Jasmine L Collins, David Chan, Jessica B Hamrick, Nan Rosemary Ke, Sandy Huang, Bryanna Kaufmann, John Canny, Alison Gopnik","year":"2022","venue":"Conference on Causal Learning and Reasoning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Machado et al., 2020] Marlos C Machado, Marc G Belle- mare, and Michael Bowling. Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5125-5133, 2020.","title":"Count-based exploration with the successor representation","authors":"C Marlos, Marc G Machado, Michael Bellemare, Bowling","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mutti et al., 2023] Mirco Mutti, Riccardo De Santi, Emanuele Rossi, Juan Felipe Calderon, Michael Bron- stein, and Marcello Restelli. Provably efficient causal model-based reinforcement learning for systematic gen- eralization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9251-9259, 2023.","title":"Provably efficient causal model-based reinforcement learning for systematic generalization","authors":"Mirco Mutti, Riccardo De Santi, Emanuele Rossi, Juan, Felipe Calderon, Michael Bronstein, Marcello Restelli","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nguyen et al., 2017] Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational contin- ual learning. arXiv preprint arXiv:1710.10628, 2017.","title":"Variational continual learning","authors":"Yingzhen Cuong V Nguyen, Thang D Li, Richard E Bui, Turner","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778-2787.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":null,"venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"PMLR, 2017.","title":null,"authors":"Pmlr","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pathak et al., 2019] Deepak Pathak, Dhiraj Gandhi, and Ab- hinav Gupta. Self-supervised exploration via disagree- ment. In International conference on machine learning, pages 5062-5071. PMLR, 2019.","title":"Self-supervised exploration via disagreement","authors":"Deepak Pathak, Dhiraj Gandhi, Abhinav Gupta","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rahaf and Lucas, 2019] Aljundi Rahaf and Caccia Lucas. Online continual learning with maximally interfered re- trieval. In NIPS, 2019.","title":"Online continual learning with maximally interfered retrieval","authors":"Lucas ; Rahaf, Aljundi Rahaf, Caccia Lucas","year":"2019","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Robine et al., 2023] Jan Robine, Marc Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-based world models are happy with 100k interactions. arXiv preprint arXiv:2303.07109, 2023.","title":"Transformer-based world models are happy with 100k interactions","authors":"Robine","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Seitzer et al., 2021] Maximilian Seitzer, Bernhard Schölkopf, and Georg Martius. Causal influence de- tection for improving efficiency in reinforcement learning. Advances in Neural Information Processing Systems, 34:22905-22918, 2021.","title":"Causal influence detection for improving efficiency in reinforcement learning","authors":"Maximilian Seitzer, Bernhard Schölkopf, Georg Martius","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sekar et al., 2020] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learn- ing, pages 8583-8592. PMLR, 2020.","title":"Planning to explore via self-supervised world models","authors":"Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, Deepak Pathak","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shyam et al., 2019] Pranav Shyam, Wojciech Jaśkowski, and Faustino Gomez. Model-based active exploration. In International conference on machine learning, pages 5779-5788. PMLR, 2019.","title":"Model-based active exploration","authors":"Pranav Shyam, Wojciech Jaśkowski, Faustino Gomez","year":"2019","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Spirtes et al., 2000] Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000.","title":"Causation, prediction, and search","authors":"Peter Spirtes, Richard Clark N Glymour, David Scheines, Heckerman","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tassa et al., 2018] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.","title":"Deepmind control suite","authors":"Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego De Las, David Casas, Abbas Budden, Josh Abdolmaleki, Andrew Merel, Lefrancq","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2022] Zizhao Wang, Xuesu Xiao, Zifan Xu, Yuke Zhu, and Peter Stone. Causal dynamics learn- ing for task-independent state abstraction. arXiv preprint arXiv:2206.13452, 2022.","title":"Causal dynamics learning for task-independent state abstraction","authors":"Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wei et al., 2018] Hua Wei, Guanjie Zheng, Huaxiu Yao, and Zhenhui Li. Intellilight: A reinforcement learning ap- proach for intelligent traffic light control. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2496-2505, 2018.","title":"Intellilight: A reinforcement learning approach for intelligent traffic light control","authors":"Hua Wei, Guanjie Zheng, Huaxiu Yao, Zhenhui Li","year":"2018","venue":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yoo and Kweon, 2019] Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 93-102, 2019.","title":"Donggeun Yoo and In So Kweon. Learning loss for active learning","authors":"Kweon Yoo","year":"2019","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yoon et al., 2021] Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. arXiv preprint arXiv:2106.01085, 2021.","title":"Online coreset selection for rehearsal-based continual learning","authors":"Yoon","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yu et al., 2023] Zhongwei Yu, Jingqing Ruan, and Deng- peng Xing. Explainable reinforcement learning via a causal world model. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 4540-4548. IJCAI, 2023.","title":"Explainable reinforcement learning via a causal world model","authors":"Zhongwei Yu, Jingqing Ruan, Dengpeng Xing","year":"2023","venue":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2023/505"},{"position":36,"raw":"[Zhang et al., 2012] Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based condi- tional independence test and application in causal discov- ery. arXiv preprint arXiv:1202.3775, 2012.","title":"Kernel-based conditional independence test and application in causal discovery","authors":"Kun Zhang, Jonas Peters, Dominik Janzing, Bernhard Schölkopf","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhu et al., 2019] Shengyu Zhu, Ignavier Ng, and Zhitang Chen. Causal discovery with reinforcement learning. arXiv preprint arXiv:1906.04477, 2019.","title":"Causal discovery with reinforcement learning","authors":"Zhu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhu et al., 2022] Zheng-Mao Zhu, Xiong-Hui Chen, Hong- Long Tian, Kun Zhang, and Yang Yu. Offline reinforce- ment learning with causal structured world models. arXiv preprint arXiv:2206.01474, 2022.","title":"Offline reinforcement learning with causal structured world models","authors":"Zheng-Mao Zhu, Xiong-Hui Chen, Hong-Long Tian, Kun Zhang, Yang Yu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}