{"type":"ScholarlyArticle","title":"Learning Unmanned Aerial Vehicle Control for Autonomous Target Following","authors":[{"name":"Siyi Li","slug":"siyi-li","given":"Siyi","family":"Li","affiliation":"Department of Computer Science and Engineering , HKUST"},{"name":"Tianbo Liu","slug":"tianbo-liu","given":"Tianbo","family":"Liu","affiliation":"Department of Electronic and Computer Engineering , HKUST"},{"name":"Chi Zhang","slug":"chi-zhang","given":"Chi","family":"Zhang","affiliation":"Department of Computer Science and Engineering , HKUST"},{"name":"Dit-Yan Yeung","slug":"dit-yan-yeung","given":"Dit-Yan","family":"Yeung","affiliation":"Department of Computer Science and Engineering , HKUST"},{"name":"Shaojie Shen","slug":"shaojie-shen","given":"Shaojie","family":"Shen","affiliation":"Department of Electronic and Computer Engineering , HKUST"}],"abstract":"While deep reinforcement learning (RL) methods have achieved unprecedented successes in a range of challenging problems, their applicability has been mainly limited to simulation or game domains due to the high sample complexity of the trial-and-error learning process. However, real-world robotic applications often need a data-efficient learning process with safety-critical constraints. In this paper, we consider the challenging problem of learning unmanned aerial vehicle (UAV) control for tracking a moving target. To acquire a strategy that combines perception and control, we represent the policy by a convolutional neural network. We develop a hierarchical approach that combines a model-free policy gradient method with a conventional feedback proportional-integral-derivative (PID) controller to enable stable learning without catastrophic failure. The neural network is trained by a combination of supervised learning from raw images and reinforcement learning from games of self-play. We show that the proposed approach can learn a target following policy in a simulator efficiently and the learned behavior can be successfully transferred to the DJI quadrotor platform for real-world UAV control.","doi":"10.24963/ijcai.2018/685","url":"https://ijcai.org/proceedings/2018/685","pdf":"https://ijcai.org/proceedings/2018/0685.pdf","markdown":"https://ijcai.org/proceedings/2018/685.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/685","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Robotics","track":"Main track","number":685,"pages":{"first":4936,"last":4942},"keywords":["Robotics: Learning in Robotics","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Abbeel et al., 2007] Pieter Abbeel, Adam Coates, Morgan Quigley, and Andrew Y Ng. An application of reinforcement learning to aerobatic helicopter flight. In NIPS, 2007.","title":"An application of reinforcement learning to aerobatic helicopter flight","authors":"Pieter Abbeel, Adam Coates, Morgan Quigley, Andrew Y Ng","year":"2007","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abbeel et al., 2010] Pieter Abbeel, Adam Coates, and Andrew Y Ng. Autonomous helicopter aerobatics through apprenticeship learning. The International Journal of Robotics Research, 2010.","title":"Autonomous helicopter aerobatics through apprenticeship learning","authors":"Pieter Abbeel, Adam Coates, Andrew Y Ng","year":"2010","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Aijun Bai, 2017] Stuart Russell Aijun Bai. Efficient reinforcement learning with hierarchies of machines by leveraging internal tran- sitions. In IJCAI, 2017.","title":"Efficient reinforcement learning with hierarchies of machines by leveraging internal transitions","authors":"Stuart Russell, Aijun Bai","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/196"},{"position":4,"raw":"[Bachrach et al., 2012] Abraham Bachrach, Samuel Prentice, Rui- jie He, Peter Henry, Albert S Huang, Michael Krainin, Daniel Maturana, Dieter Fox, and Nicholas Roy. Estimation, planning, and mapping for autonomous flight using an RGB-D camera in GPS-denied environments. The International Journal of Robotics Research, 31(11):1320-1343, 2012.","title":"Estimation, planning, and mapping for autonomous flight using an RGB-D camera in GPS-denied environments","authors":"Abraham Bachrach, Samuel Prentice, Ruijie He, Peter Henry, Albert S Huang, Michael Krainin, Daniel Maturana, Dieter Fox, Nicholas Roy","year":"2012","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Barry and Tedrake, 2015] Andrew J Barry and Russ Tedrake. Push- broom stereo for high-speed navigation in cluttered environments. In ICRA, 2015.","title":"Pushbroom stereo for high-speed navigation in cluttered environments","authors":"J Andrew, Russ Barry, Tedrake","year":"2015","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bills et al., 2011] Cooper Bills, Joyce Chen, and Ashutosh Saxena. Autonomous MAV flight in indoor environments using single image perspective cues. In ICRA, 2011.","title":"Autonomous MAV flight in indoor environments using single image perspective cues","authors":"Cooper Bills, Joyce Chen, Ashutosh Saxena","year":"2011","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deisenroth et al., 2015] Marc Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data-efficient learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(2):408-423, 2015.","title":"Gaussian processes for data-efficient learning in robotics and control","authors":"Marc Peter Deisenroth, Dieter Fox, Carl Edward Rasmussen","year":"2015","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Duan et al., 2016] Yan Duan, Xi Chen, Rein Houthooft, John Schul- man, and Pieter Abbeel. Benchmarking deep reinforcement learn- ing for continuous control. In ICML, 2016.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Giusti et al., 2016] Alessandro Giusti, Jérôme Guzzi, Dan C Cires ¸an, Fang-Lin He, Juan P Rodríguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro, et al. A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters, 1(2):661-667, 2016.","title":"A machine learning approach to visual perception of forest trails for mobile robots","authors":"Alessandro Giusti, Jérôme Guzzi, Dan C Cires ¸an, Fang-Lin He, Juan P Rodríguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro","year":"2016","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2017] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipu- lation with asynchronous off-policy updates. In ICRA, 2017.","title":"Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates","authors":"Gu","year":"2017","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2015] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee, 2011] Taeyoung Lee. Geometric tracking control of the atti- tude dynamics of a rigid body on so (3). In ACC, 2011.","title":"Geometric tracking control of the attitude dynamics of a rigid body on so (3)","authors":"Lee ;, Taeyoung Lee","year":"2011","venue":"ACC","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor poli- cies. Journal of Machine Learning Research, 17(39):1-40, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li and Yeung, 2017] Siyi Li and Dit-Yan Yeung. Visual object tracking for unmanned aerial vehicles: A benchmark and new motion models. In AAAI, 2017.","title":"Visual object tracking for unmanned aerial vehicles: A benchmark and new motion models","authors":"Li, Yeung ; ] Siyi Li, Dit-Yan Yeung","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lillicrap et al., 2016] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep rein- forcement learning. In ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mellinger and Kumar, 2011] Daniel Mellinger and Vijay Kumar. Minimum snap trajectory generation and control for quadrotors. In ICRA, 2011.","title":"Minimum snap trajectory generation and control for quadrotors","authors":"Kumar Mellinger, Daniel Mellinger, Vijay Kumar","year":"2011","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mohajerin and Waslander, 2014] Nima Mohajerin and Steven L Waslander. Modular deep recurrent neural network: Application to quadrotors. In SMC, 2014.","title":"Modular deep recurrent neural network: Application to quadrotors","authors":"Waslander Mohajerin, Nima Mohajerin, Steven L Waslander","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mueller et al., 2012] Fabian L Mueller, Angela P Schoellig, and Raffaello D'Andrea. Iterative learning of feed-forward corrections for high-performance tracking. In IROS, 2012.","title":"Iterative learning of feed-forward corrections for high-performance tracking","authors":"Angela P Fabian L Mueller, Raffaello D' Schoellig, Andrea","year":"2012","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peters and Schaal, 2006] Jan Peters and Stefan Schaal. Policy gra- dient methods for robotics. In IROS, 2006.","title":"Policy gradient methods for robotics","authors":"Schaal ; Peters, Stefan Peters, Schaal","year":"2006","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Punjani and Abbeel, 2015] Ali Punjani and Pieter Abbeel. Deep learning helicopter dynamics models. In ICRA, 2015.","title":"Deep learning helicopter dynamics models","authors":"Abbeel Punjani, Ali Punjani, Pieter Abbeel","year":"2015","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ross et al., 2013] Stéphane Ross, Narek Melik-Barkhudarov, Ku- mar Shaurya Shankar, Andreas Wendel, Debadeepta Dey, J An- drew Bagnell, and Martial Hebert. Learning monocular reactive UAV control in cluttered natural environments. In ICRA, 2013.","title":"Learning monocular reactive UAV control in cluttered natural environments","authors":"Stéphane Ross, Narek Melik-Barkhudarov, Shaurya Kumar, Andreas Shankar, Debadeepta Wendel, Andrew Dey, Martial Bagnell, Hebert","year":"2013","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sadeghi and Levine, 2017] Fereshteh Sadeghi and Sergey Levine. (CAD) 2 RL: Real single-image flight without a single real image. In RSS, 2017.","title":"CAD) 2 RL: Real single-image flight without a single real image","authors":"Fereshteh Sadeghi, Sergey Levine","year":"2017","venue":"RSS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schmid et al., 2013] Korbinian Schmid, Teodor Tomic, Felix Ruess, Heiko Hirschmüller, and Michael Suppa. Stereo vision based indoor/outdoor navigation for flying robots. In IROS, 2013.","title":"Stereo vision based indoor/outdoor navigation for flying robots","authors":"Korbinian Schmid, Teodor Tomic, Felix Ruess, Heiko Hirschmüller, Michael Suppa","year":"2013","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shen et al., 2013] Shaojie Shen, Yash Mulgaonkar, Nathan Michael, and Vijay Kumar. Vision-based state estimation and trajectory control towards high-speed flight with a quadrotor. In RSS, 2013.","title":"Vision-based state estimation and trajectory control towards high-speed flight with a quadrotor","authors":"Shen","year":"2013","venue":"RSS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Determin- istic policy gradient algorithms. In ICML, 2014.","title":"Deterministic policy gradient algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sun et al., 2017] Ting Sun, Shengyi Nie, Dit-Yan Yeung, and Shao- jie Shen. Gesture-based piloting of an aerial robot using monocular vision. In ICRA, 2017.","title":"Gesture-based piloting of an aerial robot using monocular vision","authors":"Ting Sun, Shengyi Nie, Dit-Yan Yeung, Shaojie Shen","year":"2017","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sutton et al., 1999] Richard S Sutton, David A McAllester, Satin- der P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"1999","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2016] Tianhao Zhang, Gregory Kahn, Sergey Levine, and Pieter Abbeel. Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search. In ICRA, 2016.","title":"Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search","authors":"Tianhao Zhang, Gregory Kahn, Sergey Levine, Pieter Abbeel","year":"2016","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhu et al., 2017] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target- driven visual navigation in indoor scenes using deep reinforcement learning. In ICRA, 2017.","title":"Targetdriven visual navigation in indoor scenes using deep reinforcement learning","authors":"Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, Ali Farhadi","year":"2017","venue":"ICRA","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}