{"type":"ScholarlyArticle","title":"Hierarchical State Abstraction based on Structural Information Principles","authors":[{"name":"Xianghua Zeng","slug":"xianghua-zeng","given":"Xianghua","family":"Zeng","affiliation":"State Key Laboratory of Software Development Environment , Beihang University"},{"name":"Hao Peng","slug":"hao-peng","given":"Hao","family":"Peng","affiliation":"State Key Laboratory of Software Development Environment , Beihang University"},{"name":"Angsheng Li","slug":"angsheng-li","given":"Angsheng","family":"Li","affiliation":"State Key Laboratory of Software Development Environment , Beihang University; Zhongguancun Laboratory"},{"name":"Chunyang Liu","slug":"chunyang-liu","given":"Chunyang","family":"Liu","affiliation":""},{"name":"Lifang He","slug":"lifang-he","given":"Lifang","family":"He","affiliation":"Department of Computer Science & Engineering , Lehigh University"},{"name":"Philip S. Yu","slug":"philip-s-yu","given":"Philip S.","family":"Yu","affiliation":"Department of Computer Science , University of Illinois at Chicago"}],"abstract":"State abstraction optimizes decision-making by ignoring irrelevant environmental information in reinforcement learning with rich observations. Nevertheless, recent approaches focus on adequate representational capacities resulting in essential information loss, affecting their performances on challenging tasks. In this article, we propose a novel mathematical Structural Information principles-based State Abstraction framework, namely SISA, from the information-theoretic perspective. Specifically, an unsupervised, adaptive hierarchical state clustering method without requiring manual assistance is presented, and meanwhile, an optimal encoding tree is generated. On each non-root tree node, a new aggregation function and condition structural entropy are designed to achieve hierarchical state abstraction and compensate for sampling-induced essential information loss in state abstraction. Empirical evaluations on a visual gridworld domain and six continuous control benchmarks demonstrate that, compared with five SOTA state abstraction approaches, SISA significantly improves mean episode reward and sample efficiency up to 18.98 and 44.44%, respectively. Besides, we experimentally show that SISA is a general framework that can be flexibly integrated with different representation-learning objectives to improve their performances further.","doi":"10.24963/ijcai.2023/506","url":"https://ijcai.org/proceedings/2023/506","pdf":"https://ijcai.org/proceedings/2023/0506.pdf","markdown":"https://ijcai.org/proceedings/2023/506.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/506","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":506,"pages":{"first":4549,"last":4557},"keywords":["Agent-based and Multi-agent Systems: MAS: Applications","Machine Learning: ML: Reinforcement learning","Machine Learning: ML: Deep reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Abel et al., 2016] David Abel, David Hershkowitz, and Michael Littman. Near optimal behavior via approximate state abstraction. In Proceedings of the International Con- ference on Machine Learning, pages 2915-2923. PMLR, 2016.","title":"Near optimal behavior via approximate state abstraction","authors":"David Abel, David Hershkowitz, Michael Littman","year":"2016","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abel et al., 2018] David Abel, Dilip Arumugam, Lucas Lehnert, and Michael Littman. State abstractions for life- long reinforcement learning. In Proceedings of the Inter- national Conference on Machine Learning, pages 10-19. PMLR, 2018.","title":"State abstractions for lifelong reinforcement learning","authors":"David Abel, Dilip Arumugam, Lucas Lehnert, Michael Littman","year":"2018","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Allen et al., 2021] Cameron Allen, Neev Parikh, Omer Gottesman, and George Konidaris. Learning markov state abstractions for deep reinforcement learning. Advances in Neural Information Processing Systems, 34:8229-8241, 2021.","title":"Learning markov state abstractions for deep reinforcement learning","authors":"Cameron Allen, Neev Parikh, Omer Gottesman, George Konidaris","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Baker et al., 2020] Bowen Baker, Ingmar Kanitscheider, Todor M. Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In Proceedings of the International Confer- ence on Learning Representations, pages 1-28, 2020.","title":"Emergent tool use from multi-agent autocurricula","authors":"Bowen Baker, Ingmar Kanitscheider, M Todor, Yi Markov, Glenn Wu, Bob Powell, Igor Mcgrew, Mordatch","year":"2020","venue":"Proceedings of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Belardinelli et al., 2023] Francesco Belardinelli, Angelo Ferrando, and Vadim Malvone. An abstraction-refinement framework for verifying strategic properties in multi-agent systems with imperfect information. Artificial Intelli- gence, page 103847, 2023.","title":"An abstraction-refinement framework for verifying strategic properties in multi-agent systems with imperfect information","authors":"Francesco Belardinelli, Angelo Ferrando, Vadim Malvone","year":"2023","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bellman, 1957] Richard Bellman. A markovian decision process. Journal of Mathematics and Mechanics, pages 679-684, 1957.","title":"A markovian decision process","authors":"Richard Bellman","year":"1957","venue":"Journal of Mathematics and Mechanics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1724-1734, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Collins et al., 2005] Steve Collins, Andy Ruina, Russ Tedrake, and Martijn Wisse. Efficient bipedal robots based on passive-dynamic walkers. Science, 307(5712):1082- 1085, 2005.","title":"Efficient bipedal robots based on passive-dynamic walkers","authors":"Steve Collins, Andy Ruina, Russ Tedrake, Martijn Wisse","year":"2005","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gelada et al., 2019] Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare. Deep- mdp: Learning continuous latent space models for rep- resentation learning. In Proceedings of the Interna- tional Conference on Machine Learning, pages 2170- 2179. PMLR, 2019.","title":"Deepmdp: Learning continuous latent space models for representation learning","authors":"Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, Marc G Bellemare","year":"2019","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learn- ing with a stochastic actor. In Proceedings of the Inter- national Conference on Machine Learning, pages 1861- 1870. PMLR, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hutter, 2016] Marcus Hutter. Extreme state aggregation be- yond markov decision processes. Theoretical Computer Science, 650:73-91, 2016.","title":"Extreme state aggregation beyond markov decision processes","authors":"Marcus Hutter","year":"2016","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ie et al., 2019] Eugene Ie, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Tushar Chandra, and Craig Boutilier. Slateq: A tractable decomposition for reinforcement learning with recommen- dation sets. In Proceedings of the International Joint Con- ference on Artificial Intelligence, pages 2592-2599. Inter- national Joint Conferences on Artificial Intelligence Orga- nization, 2019.","title":"Slateq: A tractable decomposition for reinforcement learning with recommendation sets","authors":"Eugene Ie, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Tushar Chandra, Craig Boutilier","year":"2019","venue":"Proceedings of the International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/360"},{"position":13,"raw":"[Jong and Stone, 2005] Nicholas K Jong and Peter Stone. State abstraction discovery from irrelevant state variables. In Proceedings of the International Joint Conference on Artificial Intelligence, volume 8, pages 752-757.","title":"State abstraction discovery from irrelevant state variables","authors":"K Nicholas, Peter Jong, Stone","year":null,"venue":"Proceedings of the International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2005/228"},{"position":14,"raw":"[Kaiser et al., 2019] Łukasz Kaiser, Mohammad Babaeizadeh, Piotr Miłos, Błażej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model based reinforcement learning for atari. In Proceedings of the International Conference on Learning Representations, 2019.","title":"Model based reinforcement learning for atari","authors":"Łukasz Kaiser, Mohammad Babaeizadeh, Piotr Miłos, Błażej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine","year":"2019","venue":"Proceedings of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Laskin et al., 2020a] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised repre- sentations for reinforcement learning. In Proceedings of the International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.","title":"Curl: Contrastive unsupervised representations for reinforcement learning","authors":"Laskin","year":"2020","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Laskin et al., 2020b] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems, 33:19884-19895, 2020.","title":"Reinforcement learning with augmented data","authors":"Laskin","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lee et al., 2020] Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor- critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Sys- tems, 33:741-752, 2020.","title":"Stochastic latent actorcritic: Deep reinforcement learning with a latent variable model","authors":"Alex X Lee, Anusha Nagabandi, Pieter Abbeel, Sergey Levine","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li and Pan, 2016] Angsheng Li and Yicheng Pan. Struc- tural information and dynamical complexity of networks. IEEE Transactions on Information Theory, 62(6):3290- 3339, 2016.","title":"Angsheng Li and Yicheng Pan. Structural information and dynamical complexity of networks","authors":"Pan Li","year":"2016","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2006] Lihong Li, Thomas J Walsh, and Michael L Littman. Towards a unified theory of state abstraction for mdps. In AI&M, 2006.","title":"Towards a unified theory of state abstraction for mdps","authors":"Lihong Li, Thomas J Walsh, Michael L Littman","year":"2006","venue":"AI&M","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2016] Angsheng Li, Xianchen Yin, and Yicheng Pan. Three-dimensional gene map of cancer cell types: Structural entropy minimisation principle for defining tu- mour subtypes. Scientific Reports, 6:1-26, 2016.","title":"Three-dimensional gene map of cancer cell types: Structural entropy minimisation principle for defining tumour subtypes","authors":"Li","year":"2016","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2018] Angsheng Li, Xianchen Yin, Bingxiang Xu, Danyang Wang, Jimin Han, Yi Wei, Yun Deng, Ying Xiong, and Zhihua Zhang. Decoding topologically associ- ating domains with ultra-low resolution hi-c data by graph structural entropy. Nature Communications, 9:1-12, 2018.","title":"Decoding topologically associating domains with ultra-low resolution hi-c data by graph structural entropy","authors":"Li","year":"2018","venue":"Nature Communications","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2019] Yiwei Liu, Jiamou Liu, Zijian Zhang, Liehuang Zhu, and Angsheng Li. Rem: From structural entropy to community structure deception. Advances in Neural Information Processing Systems, 32, 2019.","title":"Rem: From structural entropy to community structure deception","authors":"Yiwei Liu, Jiamou Liu, Zijian Zhang, Liehuang Zhu, Angsheng Li","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Peng et al., 2021] Hao Peng, Ruitong Zhang, Yingtong Dou, Renyu Yang, Jingyi Zhang, and Philip S. Yu. Re- inforced neighborhood selection guided multi-relational graph neural networks. ACM Transactions on Information Systems (TOIS), 40(4):1-46, 2021.","title":"Reinforced neighborhood selection guided multi-relational graph neural networks","authors":"Hao Peng, Ruitong Zhang, Yingtong Dou, Renyu Yang, Jingyi Zhang, Philip S Yu","year":"2021","venue":"ACM Transactions on Information Systems (TOIS)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Peng et al., 2022] Hao Peng, Ruitong Zhang, Shaoning Li, Yuwei Cao, Shirui Pan, and Philip S. Yu. Reinforced, incremental and cross-lingual event detection from social messages. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 45(1):980-998, 2022.","title":"Reinforced, incremental and cross-lingual event detection from social messages","authors":"Hao Peng, Ruitong Zhang, Shaoning Li, Yuwei Cao, Shirui Pan, Philip S Yu","year":"2022","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tunyasuvunakool et al., 2020] Saran Tunyasuvunakool, Al- istair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continu- ous control. Software Impacts, 6:100022, 2020.","title":"dm control: Software and tasks for continuous control","authors":"Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, Yuval Tassa","year":"2020","venue":"Software Impacts","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2023] Yifei Wang, Yupan Wang, Zeyu Zhang, Song Yang, Kaiqi Zhao, and Jiamou Liu. User: Unsu- pervised structural entropy-based robust graph neural net- work. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.","title":"User: Unsupervised structural entropy-based robust graph neural network","authors":"Yifei Wang, Yupan Wang, Zeyu Zhang, Song Yang, Kaiqi Zhao, Jiamou Liu","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu et al., 2022a] Junran Wu, Xueyuan Chen, Ke Xu, and Shangzhe Li. Structural entropy guided graph hierarchical pooling. In Proceedings of the International Conference on Machine Learning, pages 24017-24030. PMLR, 2022.","title":"Structural entropy guided graph hierarchical pooling","authors":"Wu","year":"2022","venue":"Proceedings of the International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2022b] Junran Wu, Shangzhe Li, Jianhao Li, Yicheng Pan, and Ke Xu. A simple yet effective method for graph classification. Proceedings of the International Joint Conference on Artificial Intelligence, 2022.","title":"A simple yet effective method for graph classification","authors":"Wu","year":"2022","venue":"Proceedings of the International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2022/497"},{"position":30,"raw":"[Yang et al., 2023] Zhenyu Yang, Ge Zhang, Jia Wu, Jian Yang, Quan Z Sheng, Hao Peng, Angsheng Li, Shan Xue, and Jianlin Su. Minimum entropy principle guided graph neural networks. In Proceedings of the Sixteenth ACM In- ternational Conference on Web Search and Data Mining, pages 114-122, 2023.","title":"Minimum entropy principle guided graph neural networks","authors":"Zhenyu Yang, Ge Zhang, Jia Wu, Jian Yang, Z Quan, Hao Sheng, Angsheng Peng, Shan Li, Jianlin Xue, Su","year":"2023","venue":"Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yarats et al., 2021] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10674-10681, 2021.","title":"Improving sample efficiency in model-free reinforcement learning from images","authors":"Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, Rob Fergus","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zang et al., 2022] Hongyu Zang, Xin Li, and Mingzhong Wang. Simsr: Simple distance-based state representa- tions for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8997-9005, 2022.","title":"Simsr: Simple distance-based state representations for deep reinforcement learning","authors":"Hongyu Zang, Xin Li, Mingzhong Wang","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zeng et al., 2023] Xianghua Zeng, Hao Peng, and Ang- sheng Li. Effective and stable role-based multi-agent col- laboration by structural information principles. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, 2023.","title":"Effective and stable role-based multi-agent collaboration by structural information principles","authors":"Xianghua Zeng, Hao Peng, Angsheng Li","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2020] Amy Zhang, Rowan Thomas McAllis- ter, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learn- ing without reconstruction. In Proceedings of the Interna- tional Conference on Learning Representations, 2020.","title":"Learning invariant representations for reinforcement learning without reconstruction","authors":"Amy Zhang, Thomas Rowan, Roberto Mcallister, Yarin Calandra, Sergey Gal, Levine","year":"2020","venue":"Proceedings of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2022] Ruitong Zhang, Hao Peng, Yingtong Dou, Jia Wu, Qingyun Sun, Yangyang Li, Jingyi Zhang, and Philip S. Yu. Automating dbscan via deep reinforce- ment learning. In Proceedings of the 31st ACM Interna- tional Conference on Information & Knowledge Manage- ment, pages 2620-2630, 2022.","title":"Automating dbscan via deep reinforcement learning","authors":"Ruitong Zhang, Hao Peng, Yingtong Dou, Jia Wu, Qingyun Sun, Yangyang Li, Jingyi Zhang, Philip S Yu","year":"2022","venue":"Proceedings of the 31st ACM International Conference on Information & Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhu et al., 2022] Zheng-Mao Zhu, Shengyi Jiang, Yu-Ren Liu, Yang Yu, and Kun Zhang. Invariant action effect model for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 9260-9268, 2022.","title":"Invariant action effect model for reinforcement learning","authors":"Zheng-Mao Zhu, Shengyi Jiang, Yu-Ren Liu, Yang Yu, Kun Zhang","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zou et al., 2023] Dongcheng Zou, Hao Peng, Xiang Huang, Renyu Yang, Jianxin Li, Jia Wu, Chunyang Liu, and Philip S. Yu. Se-gsl: A general and effective graph struc- ture learning framework through structural entropy opti- mization. In Proceedings of the Web Conference, pages 1-12, 2023.","title":"Se-gsl: A general and effective graph structure learning framework through structural entropy optimization","authors":"Zou","year":"2023","venue":"Proceedings of the Web Conference","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}