{"type":"ScholarlyArticle","title":"An Actor-Critic-Attention Mechanism for Deep Reinforcement Learning in Multi-view Environments","authors":[{"name":"Elaheh Barati","slug":"elaheh-barati","given":"Elaheh","family":"Barati","affiliation":"Department of Computer Science , Wayne State University , Detroit , MI , USA"},{"name":"Xuewen Chen","slug":"xuewen-chen","given":"Xuewen","family":"Chen","affiliation":"AIWAYS AUTO , Shanghai , China"}],"abstract":"In reinforcement learning algorithms, leveraging multiple views of the environment can improve the learning of complicated policies. In multi-view environments, due to the fact that the views may frequently suffer from partial observability, their level of importance are often different. In this paper, we propose a deep reinforcement learning method and an attention mechanism in a multi-view environment. Each view can provide various representative information about the environment. Through our attention mechanism, our method generates a single feature representation of environment given its multiple views. It learns a policy to dynamically attend to each view based on its importance in the decision-making process. Through experiments, we show that our method outperforms its state-of-the-art baselines on TORCS racing car simulator and three other complex 3D environments with obstacles. We also provide experimental results to evaluate the performance of our method on noisy conditions and partial observation settings.","doi":"10.24963/ijcai.2019/277","url":"https://ijcai.org/proceedings/2019/277","pdf":"https://ijcai.org/proceedings/2019/0277.pdf","markdown":"https://ijcai.org/proceedings/2019/277.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/277","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":277,"pages":{"first":2002,"last":2008},"keywords":["Machine Learning: Deep Learning","Machine Learning: Reinforcement Learning","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learn- ing on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.","title":"Tensorflow: Large-scale machine learning on heterogeneous distributed systems","authors":"Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bansal et al., 2018] Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent com- plexity via multi-agent competition. In ICLR, 2018.","title":"Emergent complexity via multi-agent competition","authors":"Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, Igor Mordatch","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Barati et al., 2019] Elaheh Barati, Xuewen Chen, and Zichun Zhong. Attention-based deep reinforcement learn- ing for multi-view environments. In AAMAS, pages 1805- 1807, 2019.","title":"Attention-based deep reinforcement learning for multi-view environments","authors":"Elaheh Barati, Xuewen Chen, Zichun Zhong","year":"2019","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Barbalios and Tzionas, 2014] Nikos Barbalios and Panagi- otis Tzionas. A robust approach for multi-agent natural resource allocation based on stochastic optimization algo- rithms. Applied Soft Computing, 18:12-24, 2014.","title":"Nikos Barbalios and Panagiotis Tzionas. A robust approach for multi-agent natural resource allocation based on stochastic optimization algorithms","authors":"Tzionas Barbalios","year":"2014","venue":"Applied Soft Computing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Barth-Maron et al., 2018] Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. In ICLR, 2018.","title":"Distributed distributional deterministic policy gradients","authors":"Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Alistair Muldal, Nicolas Heess, Timothy Lillicrap","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Heess et al., 2017] Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environ- ments. arXiv preprint arXiv:1707.02286, 2017.","title":"Emergence of locomotion behaviours in rich environments","authors":"Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Horgan et al., 2018] Dan Horgan, John Quan, David Bud- den, Gabriel Barth-Maron, Matteo Hessel, Hado Van Has- selt, and David Silver. Distributed prioritized experience replay. In ICLR, 2018.","title":"Distributed prioritized experience replay","authors":"Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, David Silver","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2015] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lauer and Riedmiller, 2000] Martin Lauer and Martin Ried- miller. An algorithm for distributed reinforcement learn- ing in cooperative multi-agent systems. In ICML, pages 535-542, 2000.","title":"An algorithm for distributed reinforcement learning in cooperative multi-agent systems","authors":"Riedmiller Lauer, Martin Lauer, Martin Riedmiller","year":"2000","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competitive envi- ronments. In NIPS, pages 6379-6390, 2017.","title":"Multiagent actor-critic for mixed cooperative-competitive environments","authors":"Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Openai, Igor Abbeel, Mordatch","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In ICML, pages 1928-1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nguyen et al., 2018] Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. Deep reinforcement learning for multi-agent systems: A review of chal- lenges, solutions and applications. arXiv preprint arXiv:1812.11794, 2018.","title":"Deep reinforcement learning for multi-agent systems: A review of challenges, solutions and applications","authors":"Thanh Thi Nguyen, Ngoc Duy Nguyen, Saeid Nahavandi","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Omidshafiei et al., 2017] Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P How, and John Vian. Deep decentralized multi-task multi-agent rein- forcement learning under partial observability. In ICML, pages 2681-2690, 2017.","title":"Deep decentralized multi-task multi-agent reinforcement learning under partial observability","authors":"Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P How, John Vian","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Palmer et al., 2018] Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient multi-agent deep reinforcement learning. In AAMAS, pages 443-451, 2018.","title":"Lenient multi-agent deep reinforcement learning","authors":"Gregory Palmer, Karl Tuyls, Daan Bloembergen, Rahul Savani","year":"2018","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peng et al., 2017] Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint arXiv:1703.10069, 2017.","title":"Multiagent bidirectionally-coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games","authors":"Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, Jun Wang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICML, pages 387-395, 2014.","title":"Deterministic policy gradient algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tassa et al., 2018] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.","title":"Deepmind control suite","authors":"Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego De Las, David Casas, Abbas Budden, Josh Abdolmaleki, Andrew Merel, Lefrancq","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yu- val Tassa. Mujoco: A physics engine for model-based con- trol. In IROS, pages 5026-5033. IEEE, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wiering and Van Hasselt, 2008] Marco A Wiering and Hado Van Hasselt. Ensemble algorithms in reinforcement learning. SMC, 38(4):930-936, 2008.","title":"Ensemble algorithms in reinforcement learning","authors":"A Marco, Hado Wiering, Van Hasselt","year":"2008","venue":"SMC","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wymann et al., 2000] Bernhard Wymann, Eric Espié, Christophe Guionneau, Christos Dimitrakakis, Rémi Coulom, and Andrew Sumner. Torcs, the open racing car simulator. Software available at http://torcs. sourceforge. net, 2000.","title":"Torcs, the open racing car simulator","authors":"Bernhard Wymann, Eric Espié, Christophe Guionneau, Christos Dimitrakakis, Rémi Coulom, Andrew Sumner","year":"2000","venue":null,"doi":null,"url":"http://torcs.sourceforge.net","internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}