{"type":"ScholarlyArticle","title":"CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning","authors":[{"name":"Chenyu Sun","slug":"chenyu-sun","given":"Chenyu","family":"Sun","affiliation":"Alibaba-NTU Singapore Joint Research Institute; School of Computer Science and Engineering , Nanyang Technological University; Alibaba Group"},{"name":"Hangwei Qian","slug":"hangwei-qian","given":"Hangwei","family":"Qian","affiliation":"School of Computer Science and Engineering , Nanyang Technological University; Joint NTU - UBC Research Centre of Excellence in Active Living for the Elderly (LILY)"},{"name":"Chunyan Miao","slug":"chunyan-miao","given":"Chunyan","family":"Miao","affiliation":"Alibaba-NTU Singapore Joint Research Institute; School of Computer Science and Engineering , Nanyang Technological University; Joint NTU - UBC Research Centre of Excellence in Active Living for the Elderly (LILY)"}],"abstract":"In reinforcement learning (RL), it is challenging to learn directly from high-dimensional observations, where data augmentation has recently remedied it via encoding invariances from raw pixels. Nevertheless, we empirically find that not all samples are equally important and hence simply injecting more augmented inputs may instead cause instability in Q-learning. In this paper, we approach this problem systematically by developing a model-agnostic Contrastive-Curiosity-driven Learning Framework (CCLF), which can fully exploit sample importance and improve learning efficiency in a self-supervised manner. Facilitated by the proposed contrastive curiosity, CCLF is capable of prioritizing the experience replay, selecting the most informative augmented inputs, and more importantly regularizing the Q-function as well as the encoder to concentrate more on under-learned data. Moreover, it encourages the agent to explore with a curiosity-based reward. As a result, the agent can focus on more informative samples and learn representation invariances more efficiently, with significantly reduced augmented inputs. We apply CCLF to several base RL algorithms and evaluate on the DeepMind Control Suite, Atari, and MiniGrid benchmarks, where our approach demonstrates superior sample efficiency and learning performances compared with other state-of-the-art methods. Our code is available at https://github.com/csun001/CCLF.","doi":"10.24963/ijcai.2022/478","url":"https://ijcai.org/proceedings/2022/478","pdf":"https://ijcai.org/proceedings/2022/0478.pdf","markdown":"https://ijcai.org/proceedings/2022/478.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/478","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":478,"pages":{"first":3444,"last":3450},"keywords":["Machine Learning: Self-supervised Learning","Machine Learning: Reinforcement Learning","Machine Learning: Deep Reinforcement Learning","Machine Learning: Representation learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986613","length_sec":84,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984756","length_sec":335,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Aubret et al., 2019] Arthur Aubret, Laetitia Matignon, and Salima Hassas. A survey on intrinsic motivation in reinforcement learn- ing. arXiv:1908.06976, 2019.","title":"A survey on intrinsic motivation in reinforcement learning","authors":"Arthur Aubret, Laetitia Matignon, Salima Hassas","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Ve- ness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. JAIR, 47, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellemare et al., 2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unify- ing count-based exploration and intrinsic motivation. NeurIPS, 29:1471-1479, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Berlyne, 1960] Daniel E. Berlyne. Conflict, arousal, and curiosity. McGraw-Hill Book Company, 1960.","title":"Conflict, arousal, and curiosity","authors":"E Daniel, Berlyne","year":"1960","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chevalier-Boisvert et al., 2018] Maxime Chevalier-Boisvert, Lu- cas Willems, and Suman Pal. Minimalistic gridworld en- vironment for openai gym. https://github.com/maximecb/ gym-minigrid, 2018. Accessed: 2021-12-05.","title":"Minimalistic gridworld environment for openai gym","authors":"Maxime Chevalier-Boisvert, Lucas Willems, Suman Pal","year":"2021","venue":null,"doi":null,"url":"https://github.com/maximecb/gym-minigrid,2018","internal_url":null},{"position":6,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.","title":"Soft actor-critic algorithms and applications","authors":"Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hessel et al., 2018] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rain- bow: Combining improvements in deep reinforcement learning. In AAAI, volume 32, 2018.","title":"Rainbow: Combining improvements in deep reinforcement learning","authors":"Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Laskin et al., 2020a] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for rein- forcement learning. In ICML, pages 5639-5650. PMLR, 2020.","title":"Curl: Contrastive unsupervised representations for reinforcement learning","authors":"Laskin","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Laskin et al., 2020b] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforce- ment learning with augmented data. NeurIPS, 33, 2020.","title":"Reinforcement learning with augmented data","authors":"Laskin","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2021] Kevin Li, Abhishek Gupta, Ashwin Reddy, Vitchyr H Pong, Aurick Zhou, Justin Yu, and Sergey Levine. Mu- ral: Meta-learning uncertainty-aware rewards for outcome-driven reinforcement learning. In ICML, pages 6346-6356, 2021.","title":"Mural: Meta-learning uncertainty-aware rewards for outcome-driven reinforcement learning","authors":"Kevin Li, Abhishek Gupta, Ashwin Reddy, H Vitchyr, Aurick Pong, Justin Zhou, Sergey Yu, Levine","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liquin and Lombrozo, 2020] Emily G Liquin and Tania Lom- brozo. Explanation-seeking curiosity in childhood. Current Opinion in Behavioral Sciences, 35:14-20, 2020.","title":"Explanation-seeking curiosity in childhood","authors":"G Emily, Tania Liquin, Lombrozo","year":"2020","venue":"Current Opinion in Behavioral Sciences","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Malik et al., 2021] Dhruv Malik, Aldo Pacchiano, Vishwak Srini- vasan, and Yuanzhi Li. Sample efficient reinforcement learning in continuous state spaces: A perspective beyond linearity. In ICML, pages 7412-7422. PMLR, 2021.","title":"Sample efficient reinforcement learning in continuous state spaces: A perspective beyond linearity","authors":"Dhruv Malik, Aldo Pacchiano, Vishwak Srinivasan, Yuanzhi Li","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML, pages 1928-1937. PMLR, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Mnih","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nguyen et al., 2021] Thanh Nguyen, Tung M Luu, Thang Vu, and Chang D Yoo. Sample-efficient reinforcement learning repre- sentation learning with curiosity contrastive forward dynamics model. arXiv preprint arXiv:2103.08255, 2021.","title":"Sample-efficient reinforcement learning representation learning with curiosity contrastive forward dynamics model","authors":"Thanh Nguyen, M Tung, Thang Luu, Chang D Vu, Yoo","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In ICML, pages 2778-2787. PMLR, 2017.","title":"Curiosity-driven exploration by selfsupervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Raileanu et al., 2020] Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmen- tation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006.12862, 2020.","title":"Automatic data augmentation for generalization in deep reinforcement learning","authors":"Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, Rob Fergus","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rakelly et al., 2019] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta- reinforcement learning via probabilistic context variables. In ICML, pages 5331-5340. PMLR, 2019.","title":"Efficient off-policy metareinforcement learning via probabilistic context variables","authors":"Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, Deirdre Quillen","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schaul et al., 2015] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.","title":"Prioritized experience replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schwarzer et al., 2020] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive repre- sentations. In ICLR, 2020.","title":"Data-efficient reinforcement learning with self-predictive representations","authors":"Schwarzer","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Seo et al., 2021] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. arXiv preprint arXiv:2102.09430, 2021.","title":"State entropy maximization with random encoders for efficient exploration","authors":"Seo","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Spielberger and Starr, 2012] Charles D Spielberger and Laura M Starr. Curiosity and exploratory behavior. In Motivation: Theory and research, pages 231-254. 2012.","title":"Curiosity and exploratory behavior","authors":"D Charles, Laura M Spielberger, Starr","year":"2012","venue":"Motivation: Theory and research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sun et al., 2022] Chenyu Sun, Hangwei Qian, and Chunyan Miao. From psychological curiosity to artificial curiosity: Curiosity- driven learning in artificial intelligence tasks. arXiv preprint arXiv:2201.08300, 2022.","title":"From psychological curiosity to artificial curiosity: Curiositydriven learning in artificial intelligence tasks","authors":"Sun","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tunyasuvunakool et al., 2020] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.","title":"dm control: Software and tasks for continuous control","authors":"Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, Yuval Tassa","year":"2020","venue":"Software Impacts","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2020] Kaixin Wang, Bingyi Kang, Jie Shao, and Ji- ashi Feng. Improving generalization in reinforcement learning with mixture regularization. NeurIPS, 33:7968-7978, 2020.","title":"Improving generalization in reinforcement learning with mixture regularization","authors":"Kaixin Wang, Bingyi Kang, Jie Shao, Jiashi Feng","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yarats et al., 2021a] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.","title":"Mastering visual continuous control: Improved data-augmented reinforcement learning","authors":"Yarats","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yarats et al., 2021b] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforce- ment learning from pixels. In ICLR, 2021.","title":"Image augmentation is all you need: Regularizing deep reinforcement learning from pixels","authors":"Yarats","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yarats et al., 2021c] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sam- ple efficiency in model-free reinforcement learning from images. In AAAI, 2021.","title":"Improving sample efficiency in model-free reinforcement learning from images","authors":"Yarats","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao and Tresp, 2019] Rui Zhao and Volker Tresp. Curiosity- driven experience prioritization via density estimation. arXiv preprint arXiv:1902.08039, 2019.","title":"Curiositydriven experience prioritization via density estimation","authors":"Rui Zhao, Volker Tresp","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}