{"type":"ScholarlyArticle","title":"Flow-based Intrinsic Curiosity Module","authors":[{"name":"Hsuan-Kung Yang","slug":"hsuan-kung-yang","given":"Hsuan-Kung","family":"Yang","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Po-Han Chiang","slug":"po-han-chiang","given":"Po-Han","family":"Chiang","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Min-Fong Hong","slug":"min-fong-hong","given":"Min-Fong","family":"Hong","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Chun-Yi Lee","slug":"chun-yi-lee","given":"Chun-Yi","family":"Lee","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"}],"abstract":"In this paper, we focus on a prediction-based novelty estimation strategy upon the deep reinforcement learning (DRL) framework, and present a flow-based intrinsic curiosity module (FICM) to exploit the prediction errors from optical flow estimation as exploration bonuses.  We propose the concept of leveraging motion features captured between consecutive observations to evaluate the novelty of observations in an environment. FICM encourages a DRL agent to explore observations with unfamiliar motion features, and requires only two consecutive frames to obtain sufficient information when estimating the novelty.  We evaluate our method and compare it with a number of existing methods on multiple benchmark environments, including Atari games, Super Mario Bros., and ViZDoom.  We demonstrate that FICM is favorable to tasks or environments featuring moving objects, which allow FICM to utilize the motion features between consecutive observations.  We further ablatively analyze the encoding efficiency of FICM, and discuss its applicable domains comprehensively. See here for our codes and demo videos.","doi":"10.24963/ijcai.2020/286","url":"https://ijcai.org/proceedings/2020/286","pdf":"https://ijcai.org/proceedings/2020/0286.pdf","markdown":"https://ijcai.org/proceedings/2020/286.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/286","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":286,"pages":{"first":2065,"last":2072},"keywords":["Machine Learning: Deep Reinforcement Learning","Machine Learning Applications: Applications of Reinforcement Learning","Machine Learning Applications: Game Playing"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24663/file_24663.m3u8","length_sec":285,"provider":"hls"}],"references":[{"position":1,"raw":"[Beier and Neely, 1992] Thaddeus Beier and Shawn Neely. Feature-based image metamorphosis. In Special Interest Group on Computer Graphics (SIGGRAPH), pages 35-42, Jul. 1992.","title":"Feature-based image metamorphosis","authors":"Neely ; Beier, Thaddeus Beier, Shawn Neely","year":"1992","venue":"Special Interest Group on Computer Graphics (SIGGRAPH)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2013] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. J. Artificial Intelligence Research (JAIR), 47:253-279, May 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"G Marc, Yavar Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"J. Artificial Intelligence Research (JAIR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellemare et al., 2016] Marc G. Bellemare, Sriram Srini- vasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and in- trinsic motivation. In Advances in Neural Information Pro- cessing Systems (NeurIPS), pages 1471-1479, Dec. 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"G Marc, Sriram Bellemare, Georg Srinivasan, Tom Ostrovski, David Schaul, Rémi Saxton, Munos","year":"2016","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Burda et al., 2019a] Yuri Burda, Harrison Edwards, Deepak Pathak, Amos J. Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In Proc. Int. Conf. Learning Representation (ICLR), May 2019.","title":"Large-scale study of curiosity-driven learning","authors":"Burda","year":"2019","venue":"Proc. Int. Conf. Learning Representation (ICLR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Burda et al., 2019b] Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In Proc. Int. Conf. Learning Representations (ICLR), May 2019.","title":"Exploration by random network distillation","authors":"Burda","year":"2019","venue":"Proc. Int. Conf. Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fischer et al., 2015] Philipp Fischer, Alexey Dosovitskiy, and Eddy Ilg. et al. FlowNet: Learning optical flow with convolutional networks. In Proc. IEEE Int. Conf. Com- puter Vision (ICCV), pages 2758-2766, May 2015.","title":"FlowNet: Learning optical flow with convolutional networks","authors":"Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg","year":"2015","venue":"Proc. IEEE Int. Conf. Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Greydanus et al., 2018] Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and under- standing atari agents. In Int. Conf. Machine Learning (ICML), pages 1787-1796, Jun. 2018.","title":"Visualizing and understanding atari agents","authors":"Samuel Greydanus, Anurag Koul, Jonathan Dodge, Alan Fern","year":"2018","venue":"Int. Conf. Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Houthooft et al., 2016] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: Variational information maximizing exploration. In Proc. Advances in Neural Information Processing Sys- tems (NeurIPS), pages 1109-1117, Dec. 2016.","title":"VIME: Variational information maximizing exploration","authors":"Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, Pieter Abbeel","year":"2016","venue":"Proc. Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ilg et al., 2017] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. FlowNet 2.0: Evolution of optical flow estimation with deep networks. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pages 1647-1655, Dec. 2017.","title":"FlowNet 2.0: Evolution of optical flow estimation with deep networks","authors":"Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, Thomas Brox","year":"2017","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lotter et al., 2017] William Lotter, Gabriel Kreiman, and David D. Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv:1605.08104, Mar. 2017.","title":"Deep predictive coding networks for video prediction and unsupervised learning","authors":"William Lotter, David D Gabriel Kreiman, Cox","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Meister et al., 2018] Simon Meister, Junhwa Hur, and Ste- fan Roth. Unflow: Unsupervised learning of optical flow with a bidirectional census loss. In Proc. of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI- 18), pages 7251-7259, 2018.","title":"Unflow: Unsupervised learning of optical flow with a bidirectional census loss","authors":"Simon Meister, Junhwa Hur, Stefan Roth","year":"2018","venue":"Proc. of the Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, and David Silver. et al. Human-level control through deep reinforcement learning. Nature, vol. 518, no. 7540, pp. 529-533, Feb. 2015.","title":"Human-level control through deep reinforcement learning","authors":"Mnih","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adrià Puigdomènech Badia, and Mehdi Mirza. et al. Asynchronous methods for deep reinforcement learning. In Proc. Int. Conf. Machine Learning (ICML), pages 1928-1937, Jun. 2016.","title":"Volodymyr Mnih, Adrià Puigdomènech Badia, and Mehdi Mirza","authors":"Mnih","year":"2016","venue":"Proc. Int. Conf. Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ostrovski et al., 2017] Georg Ostrovski, Marc G. Belle- mare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In Proc. Int. Conf. Machine Learning (ICML), pages 2721-2730, Jun. 2017.","title":"Count-based exploration with neural density models","authors":"Georg Ostrovski, Marc G Bellemare, Aäron Van Den Oord, Rémi Munos","year":"2017","venue":"Proc. Int. Conf. Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proc. Int. Conf. Machine Learning (ICML), pages 2778-2787, May 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"Proc. Int. Conf. Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv:1707.06347, Jul. 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Stadie et al., 2015] Bradly C. Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv:1507.00814, Nov. 2015.","title":"Incentivizing exploration in reinforcement learning with deep predictive models","authors":"Bradly C Stadie, Sergey Levine, Pieter Abbeel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wydmuch et al., 2018] M. Wydmuch, M. Kempkaand, and W. Jaśkowski. ViZDoom competitions: Playing Doom from pixels. IEEE Trans. Games, Oct. 2018.","title":"ViZDoom competitions: Playing Doom from pixels","authors":"M Wydmuch, M Kempkaand, W Jaśkowski","year":"2018","venue":"IEEE Trans. Games","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xue et al., 2016] Tianfan Xue, Jiajun Wu, Katherine L. Bouman, and Bill Freeman. Visual dynamics: Proba- bilistic future frame synthesis via cross convolutional net- works. In Proc. Advances in Neural Information Process- ing Systems (NeurIPS), pages 91-99, Dec. 2016.","title":"Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks","authors":"Xue","year":"2016","venue":"Proc. Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang et al., 2016] Marvin Zhang, Zoe McCarthy, Chelsea Finn, Sergey Levine, and Pieter Abbeel. Learning deep neural network policies with continuous memory states. In Proc. Int. Conf. Robotics and Automation (ICRA), pages 520-527, May 2016.","title":"Learning deep neural network policies with continuous memory states","authors":"Marvin Zhang, Zoe Mccarthy, Chelsea Finn, Sergey Levine, Pieter Abbeel","year":"2016","venue":"Proc. Int. Conf. Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}