{"type":"ScholarlyArticle","title":"Virtual-to-Real: Learning to Control in Visual Semantic Segmentation","authors":[{"name":"Zhang-Wei Hong","slug":"zhang-wei-hong","given":"Zhang-Wei","family":"Hong","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Yu-Ming Chen","slug":"yu-ming-chen","given":"Yu-Ming","family":"Chen","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Hsuan-Kung Yang","slug":"hsuan-kung-yang","given":"Hsuan-Kung","family":"Yang","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Shih-Yang Su","slug":"shih-yang-su","given":"Shih-Yang","family":"Su","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Tzu-Yun Shann","slug":"tzu-yun-shann","given":"Tzu-Yun","family":"Shann","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Yi-Hsiang Chang","slug":"yi-hsiang-chang","given":"Yi-Hsiang","family":"Chang","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Brian Hsi-Lin Ho","slug":"brian-hsi-lin-ho","given":"Brian Hsi-Lin","family":"Ho","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Chih-Chieh Tu","slug":"chih-chieh-tu","given":"Chih-Chieh","family":"Tu","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Tsu-Ching Hsiao","slug":"tsu-ching-hsiao","given":"Tsu-Ching","family":"Hsiao","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Hsin-Wei Hsiao","slug":"hsin-wei-hsiao","given":"Hsin-Wei","family":"Hsiao","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Sih-Pin Lai","slug":"sih-pin-lai","given":"Sih-Pin","family":"Lai","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Yueh-Chuan Chang","slug":"yueh-chuan-chang","given":"Yueh-Chuan","family":"Chang","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"},{"name":"Chun-Yi Lee","slug":"chun-yi-lee","given":"Chun-Yi","family":"Lee","affiliation":"Elsa Lab , Department of Computer Science , National Tsing Hua University , Hsinchu , Taiwan"}],"abstract":"Collecting training data from the physical world is usually time-consuming and even dangerous for fragile robots, and thus, recent advances in robot learning advocate the use of simulators as the training platform. Unfortunately, the reality gap between synthetic and real visual data prohibits direct migration of the models trained in virtual worlds to the real world. This paper proposes a modular architecture for tackling the virtual-to-real problem. The proposed architecture separates the learning model into a perception module and a control policy module, and uses semantic image segmentation as the meta representation for relating these two modules.  The perception module translates the perceived RGB image to semantic image segmentation.  The control policy module is implemented as a deep reinforcement learning agent, which performs actions based on the translated image segmentation. Our architecture is evaluated in an obstacle avoidance task and a target following task.  Experimental results show that our architecture significantly outperforms all of the baseline methods in both virtual and real environments, and demonstrates a faster learning curve than them.  We also present a detailed analysis for a variety of variant configurations, and validate the transferability of our modular architecture.","doi":"10.24963/ijcai.2018/682","url":"https://ijcai.org/proceedings/2018/682","pdf":"https://ijcai.org/proceedings/2018/0682.pdf","markdown":"https://ijcai.org/proceedings/2018/682.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/682","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Robotics","track":"Main track","number":682,"pages":{"first":4912,"last":4920},"keywords":["Robotics: Vision and Perception","Robotics: Robotics","Machine Learning: Reinforcement Learning","Robotics: Learning in Robotics","Robotics: Robotics and Vision","Robotics: Behavior and Control","Robotics: Cognitive Robotics","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Biswas and Veloso, 2012] Joydeep Biswas and Manuela Veloso. Depth camera based indoor mobile robot local- ization and navigation. In Proc. Int. Conf. Robotics and Automation (ICRA), pages 1697-1702, May 2012.","title":"Depth camera based indoor mobile robot localization and navigation","authors":"Joydeep Biswas, Manuela Veloso","year":"2012","venue":"Proc. Int. Conf. Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2016] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. DeepLab: Semantic image segmentation with deep con- volutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Analysis and Machine Intel- ligence, Jun. 2016.","title":"DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2016","venue":"IEEE Trans. Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2017] Yi-Hsin Chen, Wei-Yu Chen, Yu-Ting Chen, Bo-Cheng Tsai, Yu-Chiang Frank Wang, and Min Sun. No more discrimination: Cross city adaptation of road scene segmenters. In Proc. Int. Conf. Computer Vi- sion (ICCV), pages 2011-2020, Oct. 2017.","title":"No more discrimination: Cross city adaptation of road scene segmenters","authors":"Yi-Hsin Chen, Wei-Yu Chen, Yu-Ting Chen, Bo-Cheng Tsai, Yu-Chiang Frank, Wang, Min Sun","year":"2017","venue":"Proc. Int. Conf. Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cordts et al., 2016] Marius Cordts, Mohamed Omran, Se- bastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene under- standing. In Proc. IEEE Conf. Computer Vision and Pat- tern Recognition (CVPR), pages 3213-3223, Jun. 2016.","title":"The Cityscapes dataset for semantic urban scene understanding","authors":"Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele","year":"2016","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Finn and Levine, 2017] Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In Proc. Int. Conf. Robotics and Automation (ICRA), pages 2786- 2793, May 2017.","title":"Deep visual foresight for planning robot motion","authors":"Levine Finn, Chelsea Finn, Sergey Levine","year":"2017","venue":"Proc. Int. Conf. Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ghadirzadeh et al., 2017] Ali Ghadirzadeh, Atsuto Maki, Danica Kragic, and Mårten Björkman. Deep predic- tive policy training using reinforcement learning. In Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), pages 2351-2358, Sep. 2017.","title":"Deep predictive policy training using reinforcement learning","authors":"Ali Ghadirzadeh, Atsuto Maki, Danica Kragic, Mårten Björkman","year":"2017","venue":"Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gupta et al., 2017] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pages 7272-7281, Jul. 2017.","title":"Cognitive mapping and planning for visual navigation","authors":"Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, Jitendra Malik","year":"2017","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep con- volutional networks for visual recognition. Trans. Pattern Analysis and Machine Intelligence (TPAMI), vol. 37, no.9, pp. 1904-1916, Sep. 2015.","title":"Spatial pyramid pooling in deep convolutional networks for visual recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":"Trans. Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pages 770-778, Jun. 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[James and Johns, 2016] Stephen James and Edward Johns. 3D simulation for robot arm control with deep Q-learning. arXiv:1609.03759, Sep. 2016.","title":"3D simulation for robot arm control with deep Q-learning","authors":"Stephen James, Edward Johns","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. Int. Conf. Learning Representations (ICLR), May 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"Proc. Int. Conf. Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In Proc. Advances in Neu- ral Information Processing Systems (NIPS), pages 1097- 1105. Dec. 2012.","title":"ImageNet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Proc. Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pages 3431-3440, Jul. 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maier et al., 2012] Daniel Maier, Armin Hornung, and Maren Bennewitz. Real-time navigation in 3D environ- ments based on depth camera data. In Proc. Int. Conf. Hu- manoid Robots (Humanoids), pages 692-697, Nov. 2012.","title":"Real-time navigation in 3D environments based on depth camera data","authors":"Daniel Maier, Armin Hornung, Maren Bennewitz","year":"2012","venue":"Proc. Int. Conf. Humanoid Robots (Humanoids)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In Proc. Int. Conf. Machine Learning (ICML), pages 1928- 1937, Jul. 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"Proc. Int. Conf. Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pan et al., 2017] Xinlei Pan, Yurong You, Ziyan Wang, and Cewu Lu. Virtual to real reinforcement learning for au- tonomous driving. In Proc. The British Machine Vision Conference (BMVC), Sep. 2017.","title":"Virtual to real reinforcement learning for autonomous driving","authors":"Pan","year":"2017","venue":"Proc. The British Machine Vision Conference (BMVC)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Parisotto and Salakhutdinov, 2018] Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. In Proc. Int. Conf. Learning Representations (ICLR), Apr.-May 2018.","title":"Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning","authors":"Salakhutdinov Parisotto","year":"2018","venue":"Proc. Int. Conf. Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Paszke et al., 2016] Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello. ENet: A deep neu- ral network architecture for real-time semantic segmenta- tion. arXiv:1606.02147, Jun. 2016.","title":"ENet: A deep neural network architecture for real-time semantic segmentation","authors":"Adam Paszke, Abhishek Chaurasia, Sangpil Kim, Eugenio Culurciello","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Peng et al., 2018] Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real trans- fer of robotic control with dynamics randomization. In Proc. Int. Conf. Robotics and Automation (ICRA), May 2018.","title":"Sim-to-real transfer of robotic control with dynamics randomization","authors":"Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, Pieter Abbeel","year":"2018","venue":"Proc. Int. Conf. Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Richter et al., 2017] Stephan R Richter, Zeeshan Hayder, and Vladlen Koltun. Playing for benchmarks. In Proc. Int. Conf. Computer Vision (ICCV), pages 2232-2241, Oct. 2017.","title":"Playing for benchmarks","authors":"Zeeshan Stephan R Richter, Vladlen Hayder, Koltun","year":"2017","venue":"Proc. Int. Conf. Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rusu et al., 2016] Andrei A Rusu, Matej Vecerik, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Had- sell. Sim-to-real robot learning from pixels with progres- sive nets. In Proc. Conf. Robot Learning (CoRL), pages 262-270, Nov. 2016.","title":"Sim-to-real robot learning from pixels with progressive nets","authors":"Andrei A Rusu, Matej Vecerik, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, Raia Hadsell","year":"2016","venue":"Proc. Conf. Robot Learning (CoRL)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sadeghi and Levine, 2017] Fereshteh Sadeghi and Sergey Levine. CAD 2 RL: Real single-image flight without a sin- gle real image. In Proc. Robotics: Science and Systems (RSS), Jul. 2017.","title":"CAD 2 RL: Real single-image flight without a single real image","authors":"Levine Sadeghi, Fereshteh Sadeghi, Sergey Levine","year":"2017","venue":"Proc. Robotics: Science and Systems (RSS)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Smolyanskiy et al., 2017] Nikolai Smolyanskiy, Alexey Kamenev, Jeffrey Smith, and Stan Birchfield. Toward low-flying autonomous MAV trail navigation using deep neural networks for environmental awareness. In Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), pages 4241-4247, Sep. 2017.","title":"Toward low-flying autonomous MAV trail navigation using deep neural networks for environmental awareness","authors":"Nikolai Smolyanskiy, Alexey Kamenev, Jeffrey Smith, Stan Birchfield","year":"2017","venue":"Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tobin et al., 2017] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural net- works from simulation to the real world. In Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), pages 23-30, Sep. 2017.","title":"Domain randomization for transferring deep neural networks from simulation to the real world","authors":"Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, Pieter Abbeel","year":"2017","venue":"Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. J. Machine learning, vol. 8, no. 3-4, pp. 229-256, May 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"J. Machine learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu and Tian, 2017] Yuxin Wu and Yuandong Tian. Train- ing agent for first-person shooter game with actor-critic curriculum learning. In Proc. Int. Conf. Learning Repre- sentations (ICLR), Apr. 2017.","title":"Training agent for first-person shooter game with actor-critic curriculum learning","authors":"Tian ; ] Yuxin Wu, Yuandong Wu, Tian","year":"2017","venue":"Proc. Int. Conf. Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yan et al., 2017] Mengyuan Yan, Iuri Frosio, Stephen Tyree, and Jan Kautz. Sim-to-real transfer of accurate grasping with eye-in-hand observations and continuous control. arXiv:1712.03303, Dec. 2017.","title":"Sim-to-real transfer of accurate grasping with eye-in-hand observations and continuous control","authors":"Yan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2017] Fangyi Zhang, Jürgen Leitner, Michael Milford, and Peter Corke. Sim-to-real transfer of visuo-motor policies for reaching in clutter: Domain randomization and adaptation with modular networks. arXiv:1709.05746, Sep. 2017.","title":"Sim-to-real transfer of visuo-motor policies for reaching in clutter: Domain randomization and adaptation with modular networks","authors":"Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao et al., 2017a] Hengshuang Zhao, Xiaojuan Qi, Xiaoy- ong Shen, Jianping Shi, and Jiaya Jia. ICNet for real- time semantic segmentation on high-resolution images. arXiv:1704.08545, Apr. 2017.","title":"ICNet for realtime semantic segmentation on high-resolution images","authors":"Zhao","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2017b] Hengshuang Zhao, Jianping Shi, Xiao- juan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pages 2881-2890, Jul. 2017.","title":"Pyramid scene parsing network","authors":"Zhao","year":"2017","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou et al., 2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pages 5122-5130, Jul. 2017.","title":"Scene parsing through ADE20K dataset","authors":"Zhou","year":"2017","venue":"Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhu et al., 2017] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In Proc. Int. Conf. Robotics and Automation (ICRA), pages 3357-3364, May 2017.","title":"Target-driven visual navigation in indoor scenes using deep reinforcement learning","authors":"Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, Ali Farhadi","year":"2017","venue":"Proc. Int. Conf. Robotics and Automation (ICRA)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}