{"type":"ScholarlyArticle","title":"Multi-Task Deep Reinforcement Learning for Continuous Action Control","authors":[{"name":"Zhaoyang Yang","slug":"zhaoyang-yang","given":"Zhaoyang","family":"Yang","affiliation":"School of Engineering and Information Technology , University of New South Wales , Australia; ,; School of Electronic and Information Engineering , South China University of Technology , China"},{"name":"Kathryn Merrick","slug":"kathryn-merrick","given":"Kathryn","family":"Merrick","affiliation":"School of Engineering and Information Technology , University of New South Wales , Australia; ,"},{"name":"Hussein Abbass","slug":"hussein-abbass","given":"Hussein","family":"Abbass","affiliation":"School of Engineering and Information Technology , University of New South Wales , Australia; ,"},{"name":"Lianwen Jin","slug":"lianwen-jin","given":"Lianwen","family":"Jin","affiliation":"School of Electronic and Information Engineering , South China University of Technology , China"}],"abstract":"In this paper, we propose a deep reinforcement learning algorithm to learn multiple tasks concurrently. A new network architecture is proposed in the algorithm which reduces the number of parameters needed by more than 75% per task compared to typical single-task deep reinforcement learning algorithms. The proposed algorithm and network fuse images with sensor data and were tested with up to 12 movement-based control tasks on a simulated Pioneer 3AT robot equipped with a camera and range sensors. Results show that the proposed algorithm and network can learn skills that are as good as the skills learned by a comparable single-task learning algorithm. Results also show that learning performance is consistent even when the number of tasks and the number of constraints on the tasks increased.","doi":"10.24963/ijcai.2017/461","url":"https://ijcai.org/proceedings/2017/461","pdf":"https://ijcai.org/proceedings/2017/0461.pdf","markdown":"https://ijcai.org/proceedings/2017/461.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/461","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra, IIIA-CSIC","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":461,"pages":{"first":3301,"last":3307},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Deep Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[],"references":[{"position":1,"raw":"[Abadi, et al., 2016] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. Figure 4: The performances of 12 highly constrained control tasks obtained from different experiments. The lines in the graphs show the average performances, while the corresponding shadow areas show standard deviations. arXiv preprint arXiv:1603.04467, 2016.","title":"TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. Figure 4: The performances of 12 highly constrained control tasks obtained from different experiments","authors":"Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo","year":"2016","venue":"The lines in the graphs show the average performances, while the corresponding shadow areas show standard deviations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bangaru, et al., 2016] Sai Praveen Bangaru, JS Suhas and Balaraman Ravindran. Exploration for Multi-task Reinforcement Learning with Deep Generative Models. In Neural Information Processing Systems Deep Reinforcement Learning Workshop, 2016.","title":"Exploration for Multi-task Reinforcement Learning with Deep Generative Models","authors":"J S Sai Praveen Bangaru, Balaraman Suhas, Ravindran","year":"2016","venue":"Neural Information Processing Systems Deep Reinforcement Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Borsa, et al., 2016] Diana Borsa, Thore Graepel and John Shawe-Taylor. Learning Shared Representations in Multi- task Reinforcement Learning. arXiv preprint arXiv:1603.02041 2016.","title":"Learning Shared Representations in Multitask Reinforcement Learning","authors":"Diana Borsa, Thore Graepel, John Shawe-Taylor","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dimitrakakis and Rothkopf, 2011] Christos Dimitrakakis and Constantin A. Rothkopf. Bayesian Multitask Inverse Reinforcement Learning. In European Workshop on Reinforcement Learning, pp.273-284, 2011.","title":"Bayesian Multitask Inverse Reinforcement Learning","authors":"Christos Dimitrakakis, Constantin A Rothkopf","year":"2011","venue":"European Workshop on Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Foerster, et al., 2016] Jakob N. Foerster, Yannis M. Assael, Nando de Freitas and Shimon Whiteson. Learning to Communicate with Deep Multi-Agent Reinforcement Learning. In Advances in Neural Information Processing Systems, pp.2137-2145, 2016.","title":"Learning to Communicate with Deep Multi-Agent Reinforcement Learning","authors":"Jakob N Foerster, M Yannis, Nando Assael, Shimon De Freitas, Whiteson","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Graves, et al., 2013] Alex Graves, Abdel-rahman Mohamed and Geoffrey Hinton. Speech Recognition with Deep Recurrent Neural Networks. In Proceedings of International Conference on Acoustics, Speech and Signal Processing, pp.6645-6649. IEEE, 2013.","title":"Speech Recognition with Deep Recurrent Neural Networks","authors":"Alex Graves, Abdel-Rahman Mohamed, Geoffrey Hinton","year":"2013","venue":"Proceedings of International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Grounds and Kudenko, 2008] Matthew Grounds and Daniel Kudenko. Parallel reinforcement learning with linear function approximation. In Adaptive Agents and Multi-Agent Systems III. Adaptation and Multi-Agent Learning, pp.60-74.","title":"Matthew Grounds and Daniel Kudenko. Parallel reinforcement learning with linear function approximation","authors":"Kudenko Grounds","year":"2008","venue":"Adaptive Agents and Multi-Agent Systems III. Adaptation and Multi-Agent Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Lei Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.","title":"Adam: A Method for Stochastic Optimization","authors":"P Diederik, Jimmy Lei Kingma, Ba","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Konidaris, et al., 2011] George Konidaris, Sarah Osentoski and Philip Thomas. Value Function Approximation in Reinforcement Learning using the Fourier Basis. In Proceedings of Proc. AAAI Conference on Artificial Intelligence, pp.380-385. AAAI, 2011.","title":"Value Function Approximation in Reinforcement Learning using the Fourier Basis","authors":"Konidaris","year":"2011","venue":"Proceedings of Proc. AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krishnamurthy, et al., 2016] Ramnandan Krishnamurthy, Aravind Lakshminarayanan, Peeyush Kumar and Balaraman Ravindran. Hierarchical Reinforcement Learning using Spatio-Temporal Abstractions and Deep Neural Networks. In International Conference on Machine Learning Abstraction in Reinforcement Learning Workshop, 2016.","title":"Hierarchical Reinforcement Learning using Spatio-Temporal Abstractions and Deep Neural Networks","authors":"Krishnamurthy","year":"2016","venue":"International Conference on Machine Learning Abstraction in Reinforcement Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krizhevsky, et al., 2012] Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Network. In Proceedings of Advances in Neural Information Processing Systems, pp.1097-1105.","title":"ImageNet Classification with Deep Convolutional Neural Network","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":null,"venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kulkarni, et al., 2016] Tejas D. Kulkarni, Karthik R. Narasimhan, Ardavan Saeedi and Joshua B. Tenenbaum. Hierarchical Deep Reinforcement Learning Integrating Temporal Abstraction and Intrinsic Motivation. In Proceedings of Advances in Neural Information Processing Systems, pp.3675-3683. 2016. [Lazaric, 2012] Alessandro Lazaric. Transfer in Reinforcement Learning: a Framework and a Survey. In Reinforcement Learning, Berlin, Heidelberg, 2012. [Lazaric and Ghavamzadeh, 2010] Alessandro Lazaric and Mohammad Ghavamzadeh. Bayesian Multi-Task Reinforcement Learning. In Proceedings of International Conference on Machine Learning, pp.599-606. ACM, 2010.","title":"Hierarchical Deep Reinforcement Learning Integrating Temporal Abstraction and Intrinsic Motivation","authors":"D Tejas, Karthik R Kulkarni, Ardavan Narasimhan, Joshua B Saeedi, Tenenbaum","year":"2010","venue":"Alessandro Lazaric. Transfer in Reinforcement Learning: a Framework and a Survey. In Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lillicrap, et al., 2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, et al. Continuous Control with Deep Reinforcement Learning. In Proceedings of International Conference on Learning Representations. 2016.","title":"Continuous Control with Deep Reinforcement Learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess","year":"2016","venue":"Proceedings of International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin, et al., 2013] Min Lin, Qiang Chen and Shuicheng Yan. Network in Network. arXiv preprint arXiv:1312.4400, 2013.","title":"Network in Network","authors":"Lin","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih, et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, et al. Asynchronous Methods for Deep Reinforcement Learning. In Proceedings of International Conference on Machine Learning. ACM, 2016.","title":"Asynchronous Methods for Deep Reinforcement Learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves","year":"2016","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih, et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, et al. Human-level Control through Deep reinforcement Learning. Nature, 518(7540):529-533. 2015.","title":"Human-level Control through Deep reinforcement Learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mohamed and Rezende, 2015] Shakir Mohamed and Danilo J. Rezende. Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning. In Proceedings of Advances in Neural Information Processing Systems, pp.2125-2133. MIT Press, 2015.","title":"Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning","authors":"Shakir Mohamed, Danilo J Rezende","year":"2015","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peters, et al., 2005] Jan Peters, Sethu Vijayakumar and Stefan Schaal. Natural Actor-Critic. In Proceedings of European Conference on Machine Learning, pp.280-291. 2005.","title":"Natural Actor-Critic","authors":"Peters","year":"2005","venue":"Proceedings of European Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Silver, et al., 2016] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, et al. Mastering the Game of Go with Deep Neural Networks and TreeSearch. Nature, 529(7587):484-489. 2016.","title":"Mastering the Game of Go with Deep Neural Networks and TreeSearch","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutton, 1988] Richard S. Sutton. Learning to Predict by the Methods of Temporal Differences. Machine Learning, 3(1):9-44. 1988.","title":"Learning to Predict by the Methods of Temporal Differences","authors":"Richard S Sutton","year":"1988","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tampuu, et al., 2015] Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, et al. Multiagent Cooperation and Competition with Deep Reinforcement Learning. arXiv preprint arXiv:1511.08779, 2015.","title":"Ardi Tampuu, Tambet Matiisen","authors":"Tampuu","year":"2015","venue":"Multiagent Cooperation and Competition with Deep Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Uhlenbeck and Ornstein, 1930] George E Uhlenbeck and Leonard S Ornstein. On the Theory of the Brownian Motion. Physical Review, 36(5):823. 1930.","title":"On the Theory of the Brownian Motion","authors":"Ornstein ; Uhlenbeck, E George, Leonard S Uhlenbeck, Ornstein","year":"1930","venue":"Physical Review","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Watkins and Dayan, 1992] Christopher J.C.H. Watkins and Peter Dayan. Q-Learning. Machine Learning, 8(3):279-292. 1992.","title":"Q-Learning","authors":"J C H Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang, et al., 2016] Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker and Wolfram Burgard. Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments. arXiv:1612.05533, 2016.","title":"Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments","authors":"Zhang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}