{"type":"ScholarlyArticle","title":"Keeping in Touch with Collaborative UAVs: A Deep Reinforcement Learning Approach","authors":[{"name":"Bo Yang","slug":"bo-yang","given":"Bo","family":"Yang","affiliation":"Institute of Computing Technology , Chinese Academy of Sciences , Beijing , China"},{"name":"Min Liu","slug":"min-liu","given":"Min","family":"Liu","affiliation":"Institute of Computing Technology , Chinese Academy of Sciences , Beijing , China"}],"abstract":"Effective collaborations among autonomous unmanned aerial vehicles (UAVs) rely on timely information sharing. However, the time-varying flight environment and the intermittent link connectivity pose great challenges to message delivery. In this paper, we leverage the deep reinforcement learning (DRL) technique to address the UAVs' optimal links discovery and selection problem in uncertain environments. As the multi-agent learning efficiency is constrained by the high-dimensional and continuous action spaces, we slice the whole action spaces into a number of tractable fractions to achieve efficient convergences of optimal policies in continuous domains. Moreover, for the nonstationarity issue that particularly challenges the multi-agent DRL with local perceptions, we present a multi-agent mutual sampling method that jointly interacts the intra-agent and inter-agent state-action information to stabilize and expedite the training procedure. We evaluate the proposed algorithm on the UAVs' continuous network connection task. Results show that the associated UAVs can quickly select the optimal connected links, which facilitate the UAVs' teamwork significantly.","doi":"10.24963/ijcai.2018/78","url":"https://ijcai.org/proceedings/2018/78","pdf":"https://ijcai.org/proceedings/2018/0078.pdf","markdown":"https://ijcai.org/proceedings/2018/78.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/78","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":78,"pages":{"first":562,"last":568},"keywords":["Agent-based and Multi-agent Systems: Multi-agent Learning","Machine Learning Applications: Applications of Reinforcement Learning","Machine Learning: Deep Learning","Agent-based and Multi-agent Systems: Agent Communication"],"videos":[],"references":[{"position":1,"raw":"[Anschel et al., 2017] Oron Anschel, Nir Baram and Nir Baram. Averaged-DQN: Variance reduction and stabi- lization for deep reinforcement learning. In Proceed- ings of the 34th International Conference on Machine Learning, Sydney, Australia, 2017.","title":"Averaged-DQN: Variance reduction and stabilization for deep reinforcement learning","authors":"Anschel","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barrett et al., 2017] Samuel Barrett, Avi Rosenfeld, Sarit Kraus and Peter Stone. Making friends on the fly: Co- operating with new teammates. Artificial Intelligence, 242: 132-171, 2017.","title":"Making friends on the fly: Cooperating with new teammates","authors":"Barrett","year":"2017","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2017] Yu Fan Chen, Michael Everett, Miao Li- u and Jonathan P. How. Socially aware motion plan- ning with deep reinforcement learning. In Proceedings of 2017 IEEE/RSJ International Conference on Intelli- gent Robots and Systems, Vancouver, Canada, 2017.","title":"How. Socially aware motion planning with deep reinforcement learning","authors":"Michael Yu Fan Chen, Miao Everett, Jonathan P Liu","year":"2017","venue":"Proceedings of 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chinchali et al., 2018] Sandeep Chinchali, Pan Hu, Tianshu Chu, Manu Sharma, Manu Bansal, Rakesh Misra, Mar- co Pavone and Sachin Katti. Cellular network traffic scheduling with deep reinforcement learning. In Pro- ceedings of the 32th AAAI Conference on Artificial In- telligence, New Orleans, USA, 2018.","title":"Cellular network traffic scheduling with deep reinforcement learning","authors":"Sandeep Chinchali, Pan Hu, Tianshu Chu, Manu Sharma, Manu Bansal, Rakesh Misra, Marco Pavone, Sachin Katti","year":"2018","venue":"Pro-ceedings of the 32th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Foerster et al., 2016] Jakob N. Foerster, Yannis M. Assael, Nando de Freitas and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcemen- t learning. In Proceedings of the 30th Conference on Neural Information Processing Systems, Barcelona, S- pain, 2016.","title":"Learning to communicate with deep multi-agent reinforcement learning","authors":"Jakob N Foerster, M Yannis, Nando Assael, Shimon De Freitas, Whiteson","year":"2016","venue":"Proceedings of the 30th Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gu et al., 2016] Shixiang Gu, Timothy Lillicrap, Ilya Sutskever and Sergey Levine. Continuous deep Q- learning with model-based acceleration. In Proceedings of the 33th International Conference on Machine Learn- ing, New York, USA, 2016.","title":"Continuous deep Qlearning with model-based acceleration","authors":"Gu","year":"2016","venue":"Proceedings of the 33th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hasselt et al., 2016] Hado van Hasselt, Arthur Guez and David Silver. Deep reinforcement learning with double Q-learning. In Proceedings of the 30th AAAI Confer- ence on Artificial Intelligence, Phoenix, USA, 2016.","title":"Deep reinforcement learning with double Q-learning","authors":"Hasselt","year":"2016","venue":"Proceedings of the 30th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] He He, Jordan Boyd-Graber, Kevin Kwok and Hal Daume III. Opponent modeling in deep rein- forcement learning. In Proceedings of the 33th Inter- national Conference on Machine Learning, New York, USA, 2016.","title":"Opponent modeling in deep reinforcement learning","authors":"He","year":"2016","venue":"Proceedings of the 33th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kulkarni et al., 2016] Tejas D. Kulkarni, Karthik R. Narasimhan and Ardavan Saeedi. Hierarchical deep re- inforcement learning: Integrating temporal abstraction and intrinsic motivation. In Proceedings of the 30th Conference on Neural Information Processing Systems, Barcelona, Spain, 2016.","title":"Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation","authors":"D Tejas, Karthik R Kulkarni, Ardavan Narasimhan, Saeedi","year":"2016","venue":"Proceedings of the 30th Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Leibo et al., 2017] Joel Z. Leibo, Vinicius Zambaldi, Mar- c Lanctot, Janusz Marecki and Thore Graepel. Multi- agent reinforcement learning in sequential social dilem- mas. In Proceedings of the 16th Conference on Au- tonomous Agents and MultiAgent Systems, São Paulo, Brazil, 2017.","title":"Multiagent reinforcement learning in sequential social dilemmas","authors":"Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, Thore Graepel","year":"2017","venue":"Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2017] Weirong Liu, Gaorong Qin, Yun He and Fei Jiang. Distributed cooperative reinforcemen- t learning-based traffic signal control that integrates V2X networks' dynamic clustering. IEEE Transactions on Vehicular Technology, 66(10): 8667-8681, 2017.","title":"Distributed cooperative reinforcement learning-based traffic signal control that integrates V2X networks' dynamic clustering","authors":"Liu","year":"2017","venue":"IEEE Transactions on Vehicular Technology","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, et al. Human-level control through deep reinforcement learning. Nature, 518(7540): 529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timo- thy P. Lillicrap, David Silver and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learn- ing. In Proceedings of the 33th International Confer- ence on Machine Learning, New York, USA, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy P Lillicrap, David Silver, Koray Kavukcuoglu","year":"2016","venue":"Proceedings of the 33th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Omidshafiei et al., 2017] Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 2017.","title":"Deep decentralized multi-task multi-agent reinforcement learning under partial observability","authors":"Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P How, John Vian","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sharma et al., 2017] Sahil Sharma, Aravind S. Lakshmi- narayanan and Balaraman Ravindran. Learning to re- peat: Fine grained action repetition. In Proceedings of the 5th International Conference on Learning Represen- tations, Toulon, France, 2017.","title":"Learning to repeat: Fine grained action repetition","authors":"Sharma","year":"2017","venue":"Proceedings of the 5th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tampuu et al., 2017] Ardi Tampuu, Tambet Matiisen, Dori- an Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PLoS ONE, 12(4): 1-15, 2017.","title":"Multiagent cooperation and competition with deep reinforcement learning","authors":"Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, Raul Vicente","year":"2017","venue":"PLoS ONE","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2017] Jingjing Wang, Chunxiao Jiang, Zhu Han, Yong Ren, Robert G. Maunder and Lajos Hanzo. Taking drones to the next level: Cooperative distributed unmanned-aerial-vehicular networks for small and mini drones. IEEE Vehicular Technology Magazine, 12(3): 73-82, 2017.","title":"Taking drones to the next level: Cooperative distributed unmanned-aerial-vehicular networks for small and mini drones","authors":"Jingjing Wang, Chunxiao Jiang, Zhu Han, Yong Ren, Robert G Maunder, Lajos Hanzo","year":"2017","venue":"IEEE Vehicular Technology Magazine","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2016] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot and Nando de Freitas. Dueling network architectures for deep reinforcement learning. In Proceedings of the 33th International Con- ference on Machine Learning, New York, USA, 2016.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Marc Hado Van Hasselt, Nando Lanctot, De Freitas","year":"2016","venue":"Proceedings of the 33th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2017] Di Wu, Dmitri I. Arkhipov, Minyoung Kim, Carolyn L. Talcott, Amelia C. Regan, Julie A. M- cCann and Nalini Venkatasubramanian. Addsen: Adap- tive data processing and dissemination for drone swarm- s in urban sensing. IEEE Transactions on Computers, 66(2): 183-197, 2017.","title":"Addsen: Adaptive data processing and dissemination for drone swarms in urban sensing","authors":"Di Wu, Dmitri I Arkhipov, Minyoung Kim, Carolyn L Talcott, Amelia C Regan, Julie A M-Ccann, Nalini Venkatasubramanian","year":"2017","venue":"IEEE Transactions on Computers","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xiao et al., 2018] Liang Xiao, Caixia Xie, Minghui Min and Weihua Zhuang. User-centric view of unmanned aerial vehicle transmission against smart attacks. IEEE Transactions on Vehicular Technology, 67(4): 3420- 3430, 2018.","title":"User-centric view of unmanned aerial vehicle transmission against smart attacks","authors":"Liang Xiao, Caixia Xie, Minghui Min, Weihua Zhuang","year":"2018","venue":"IEEE Transactions on Vehicular Technology","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2018] Zhiyuan Xu, Jian Tang, Jingsong Meng, Weiyi Zhang, Yanzhi Wang, Chi Harold Liu and De- jun Yang. Experience-driven networking: A deep rein- forcement learning based approach. In Proceedings of the 37th IEEE International Conference on Computer Communications, Honolulu, USA, 2018.","title":"Experience-driven networking: A deep reinforcement learning based approach","authors":"Zhiyuan Xu, Jian Tang, Jingsong Meng, Weiyi Zhang, Yanzhi Wang, Chi, Harold Liu, Dejun Yang","year":"2018","venue":"Proceedings of the 37th IEEE International Conference on Computer Communications","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ye et al., 2017] Dayong Ye, Minjie Zhang and Athanasios V. Vasilakos. A survey of self-organization mechanisms in multiagent systems. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 47(3): 441-461, 2017.","title":"A survey of self-organization mechanisms in multiagent systems","authors":"Minjie Dayong Ye, Athanasios V Zhang, Vasilakos","year":"2017","venue":"IEEE Transactions on Systems, Man, and Cybernetics: Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2018] Bo Zhang, Chi Harold Liu, Jian Tang, Zhiyuan Xu, Jian Ma and Wendong Wang. Learning- based energy-efficient data collection by unmanned ve- hicles in smart cities. IEEE Transactions on Industrial Informatics, 14(4): 1666-1676, 2018.","title":"Learningbased energy-efficient data collection by unmanned vehicles in smart cities","authors":"Bo Zhang, Chi Harold Liu, Jian Tang, Zhiyuan Xu, Jian Ma, Wendong Wang","year":"2018","venue":"IEEE Transactions on Industrial Informatics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2017] Zongzhang Zhang, Zhiyuan Pan and Mykel J. Kochenderfer. Weighted double Q-learning. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, Australia, 2017.","title":"Weighted double Q-learning","authors":"Zhang","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2017/483"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}