{"type":"ScholarlyArticle","title":"IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL","authors":[{"name":"Nirbhay Modhe","slug":"nirbhay-modhe","given":"Nirbhay","family":"Modhe","affiliation":"Georgia Institute of Technology"},{"name":"Prithvijit Chattopadhyay","slug":"prithvijit-chattopadhyay","given":"Prithvijit","family":"Chattopadhyay","affiliation":"Georgia Institute of Technology"},{"name":"Mohit Sharma","slug":"mohit-sharma","given":"Mohit","family":"Sharma","affiliation":"Georgia Institute of Technology"},{"name":"Abhishek Das","slug":"abhishek-das","given":"Abhishek","family":"Das","affiliation":"Georgia Institute of Technology"},{"name":"Devi Parikh","slug":"devi-parikh","given":"Devi","family":"Parikh","affiliation":"Georgia Institute of Technology; Facebook AI Research {nirbhaym ,"},{"name":"Dhruv Batra","slug":"dhruv-batra","given":"Dhruv","family":"Batra","affiliation":"Georgia Institute of Technology; Facebook AI Research {nirbhaym ,"},{"name":"Ramakrishna Vedantam","slug":"ramakrishna-vedantam","given":"Ramakrishna","family":"Vedantam","affiliation":"Facebook AI Research {nirbhaym ,"}],"abstract":"We propose a novel framework to identify sub-goals useful for exploration in sequential decision making tasks under partial observability. We utilize the variational intrinsic control framework (Gregor et.al., 2016) which maximizes empowerment -- the ability to reliably reach a diverse set of states and show how to identify sub-goals as states with high necessary option information through an information theoretic regularizer. Despite being discovered without explicit goal supervision, our sub-goals provide better exploration and sample complexity on challenging grid-world navigation tasks compared to supervised counterparts in prior work.","doi":"10.24963/ijcai.2020/280","url":"https://ijcai.org/proceedings/2020/280","pdf":"https://ijcai.org/proceedings/2020/0280.pdf","markdown":"https://ijcai.org/proceedings/2020/280.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/280","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":280,"pages":{"first":2022,"last":2028},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Deep Reinforcement Learning","Machine Learning: Unsupervised Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25306/file_25306.m3u8","length_sec":300,"provider":"hls"}],"references":[{"position":1,"raw":"[Achiam et al., 2018] Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv:1807.10299, 2018.","title":"Variational option discovery algorithms","authors":"Joshua Achiam, Harrison Edwards, Dario Amodei, Pieter Abbeel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alemi et al., 2016] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In ICLR, 2016.","title":"Deep variational information bottleneck","authors":"Ian Alexander A Alemi, Joshua V Fischer, Kevin Dillon, Murphy","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chalk et al., 2016] Matthew Chalk, Olivier Marre, and Gasper Tkacik. Relevant sparse codes with variational in- formation bottleneck. 2016.","title":"Relevant sparse codes with variational information bottleneck","authors":"Matthew Chalk, Olivier Marre, Gasper Tkacik","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chechik et al., 2005] Gal Chechik, Amir Globerson, Naf- tali Tishby, and Yair Weiss. Information bottleneck for gaussian variables. J. of Machine Learning Research, 6(Jan):165-188, 2005.","title":"Information bottleneck for gaussian variables","authors":"Chechik","year":"2005","venue":"J. of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chevalier-Boisvert et al., 2018] Maxime Chevalier- Boisvert, Lucas Willems, and Suman Pal. Min- imalistic gridworld environment for openai gym. https://github.com/maximecb/gym-minigrid, 2018.","title":"Minimalistic gridworld environment for openai gym","authors":"Maxime Chevalier-Boisvert, Lucas Willems, Suman Pal","year":null,"venue":null,"doi":null,"url":"https://github.com/maximecb/gym-minigrid,2018","internal_url":null},{"position":6,"raw":"[Cover and Thomas, 1991] Thomas M Cover and Joy A Thomas. Elements of information theory. Wiley Series in Telecommunications, 1991.","title":"Elements of information theory","authors":"M Thomas, Joy A Cover, Thomas","year":"1991","venue":"Wiley Series in Telecommunications","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Eysenbach et al., 2018] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. CoRR, abs/1802.06070, 2018.","title":"Diversity is all you need: Learning skills without a reward function","authors":"Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goyal et al., 2019] Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, and Sergey Levine. InfoBot: Transfer and exploration via the information bottleneck. arXiv [stat.ML], January 2019.","title":"InfoBot: Transfer and exploration via the information bottleneck","authors":"Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, Sergey Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gregor et al., 2016] Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07507, 2016.","title":"Variational intrinsic control","authors":"Karol Gregor, Danilo Jimenez Rezende, Daan Wierstra","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hayes-Roth and Hayes-Roth, 1979] Barbara Hayes-Roth and Frederick Hayes-Roth. A cognitive model of planning. Cognitive science, 3(4):275-310, 1979.","title":"Barbara Hayes-Roth and Frederick","authors":"Hayes-Roth, Hayes-Roth ; Hayes-Roth","year":"1979","venue":"Cognitive science","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Higgins et al., 2017] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained varia- tional framework. In ICLR, 2017.","title":"beta-VAE: Learning basic visual concepts with a constrained variational framework","authors":"Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, Alexander Lerchner","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lopes et al., 2012] Manuel Lopes, Tobias Lang, Marc Tou- ssaint, and Pierre-Yves Oudeyer. Exploration in model- based reinforcement learning by empirically estimating learning progress. In NIPS, 2012.","title":"Exploration in modelbased reinforcement learning by empirically estimating learning progress","authors":"Manuel Lopes, Tobias Lang, Marc Toussaint, Pierre-Yves Oudeyer","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Machado et al., 2017] Marios Machado, Marc Bellemare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning. In ICML, 2017.","title":"A laplacian framework for option discovery in reinforcement learning","authors":"Marios Machado, Marc Bellemare, Michael Bowling","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[McGovern and Barto, 2001] Amy McGovern and An- drew G. Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. In ICML, 2001.","title":"Automatic discovery of subgoals in reinforcement learning using diverse density","authors":"Barto Mcgovern, G Barto","year":"2001","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 16-17, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"Proc. of the IEEE Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Polani et al., 2006] Daniel Polani, Chrystopher L Nehaniv, Thomas Martinetz, and Jan T Kim. Relevant information in optimized persistence vs. progeny strategies. In In: Ar- tificial Life X: Proc. of the Tenth International Conference on the Simulation and Synthesis of Living Systems. Mit Press, 2006.","title":"Relevant information in optimized persistence vs. progeny strategies","authors":"Daniel Polani, L Chrystopher, Thomas Nehaniv, Jan T Martinetz, Kim","year":"2006","venue":"Artificial Life X: Proc. of the Tenth International Conference on the Simulation and Synthesis of Living Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Salge et al., 2013] Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment -an introduction. October 2013.","title":"Empowerment -an introduction","authors":"Christoph Salge, Cornelius Glackin, Daniel Polani","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schmidhuber, 1990] Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the First International Con- ference on Simulation of Adaptive Behavior on From An- imals to Animats, pages 222-227, Cambridge, MA, USA, 1990. MIT Press.","title":"A possibility for implementing curiosity and boredom in model-building neural controllers","authors":"Jürgen Schmidhuber","year":"1990","venue":"Proc. of the First International Conference on Simulation of Adaptive Behavior on From Animals to Animats","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Still et al., 2004] Susanne Still, William Bialek, and Léon Bottou. Geometric clustering using the information bot- tleneck method. In S Thrun, L K Saul, and B Schölkopf, editors, NIPS. 2004.","title":"Geometric clustering using the information bottleneck method","authors":"Susanne Still, William Bialek, Léon Bottou","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Strouse and Schwab, 2017] D J Strouse and David J Schwab. The information bottleneck and geometric clus- tering. December 2017.","title":"The information bottleneck and geometric clustering","authors":"D J Strouse, David J Schwab","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Strouse et al., 2018] D J Strouse, Max Kleiman-Weiner, Josh Tenenbaum, Matt Botvinick, and David Schwab. Learning to share and hide intentions using information regularization. August 2018.","title":"Learning to share and hide intentions using information regularization","authors":"D J Strouse, Max Kleiman-Weiner, Josh Tenenbaum, Matt Botvinick, David Schwab","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Teh et al., 2017] Yee Teh, Victor Bapst, Wojciech M Czar- necki, John Quan, James Kirkpatrick, Raia Hadsell, Nico- las Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In NIPS, 2017.","title":"Distral: Robust multitask reinforcement learning","authors":"Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, Razvan Pascanu","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tishby et al., 1999] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. In The 37th annual Allerton Conf. on Communication, Con- trol, and Computing, pages 368-377, 1999.","title":"The information bottleneck method","authors":"Naftali Tishby, Fernando C Pereira, William Bialek","year":"1999","venue":"The 37th annual Allerton Conf. on Communication, Control, and Computing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[van Dijk and Polani, 2011] Sander G van Dijk and Daniel Polani. Grounding subgoals in information transitions. In IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning, pages 105-111, 2011.","title":"Grounding subgoals in information transitions","authors":"Van Dijk, ; ] Polani, G Sander, Daniel Van Dijk, Polani","year":"2011","venue":"IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zaslavsky et al., 2018] Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. Efficient compression in color naming and its evolution. Proc. Natl. Acad. Sci.","title":"Efficient compression in color naming and its evolution","authors":"Noga Zaslavsky, Charles Kemp, Terry Regier, Naftali Tishby","year":null,"venue":"Proc. Natl. Acad. Sci","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"U. S. A., 115(31):7937-7942, July 2018.","title":"U. S. A","authors":null,"year":"2018","venue":"U. S. A","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ziebart et al., 2008] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"Aaai","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}