{"type":"ScholarlyArticle","title":"KoGuN: Accelerating Deep Reinforcement Learning via Integrating Human Suboptimal Knowledge","authors":[{"name":"Peng Zhang","slug":"peng-zhang","given":"Peng","family":"Zhang","affiliation":"College of Intelligence and Computing , Tianjin University; Noah's Ark Lab , Huawei"},{"name":"Jianye Hao","slug":"jianye-hao","given":"Jianye","family":"Hao","affiliation":"College of Intelligence and Computing , Tianjin University; Noah's Ark Lab , Huawei; Tianjin Key Lab of Machine Learning {zhangpeng123 ,"},{"name":"Weixun Wang","slug":"weixun-wang","given":"Weixun","family":"Wang","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Hongyao Tang","slug":"hongyao-tang","given":"Hongyao","family":"Tang","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Yi Ma","slug":"yi-ma","given":"Yi","family":"Ma","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Yihai Duan","slug":"yihai-duan","given":"Yihai","family":"Duan","affiliation":"College of Intelligence and Computing , Tianjin University"},{"name":"Yan Zheng","slug":"yan-zheng","given":"Yan","family":"Zheng","affiliation":"College of Intelligence and Computing , Tianjin University"}],"abstract":"Reinforcement learning agents usually learn from scratch, which requires a large number of interactions with the environment. This is quite different from the learning process of human. When faced with a new task, human naturally have the common sense and use the prior knowledge to derive an initial policy and guide the learning process afterwards. Although the prior knowledge may be not fully applicable to the new task, the learning process is significantly sped up since the initial policy ensures a quick-start of learning and intermediate guidance allows to avoid unnecessary exploration. Taking this inspiration, we propose knowledge guided policy network (KoGuN), a novel framework that combines human prior suboptimal knowledge with reinforcement learning. Our framework consists of a fuzzy rule controller to represent human knowledge and a refine module to finetune suboptimal prior knowledge. The proposed framework is end-to-end and can be combined with existing policy-based reinforcement learning algorithm. We conduct experiments on several control tasks. The empirical results show that our approach, which combines suboptimal human knowledge and RL, achieves significant improvement on learning efficiency of flat RL algorithms, even with very low-performance human prior knowledge.","doi":"10.24963/ijcai.2020/317","url":"https://ijcai.org/proceedings/2020/317","pdf":"https://ijcai.org/proceedings/2020/0317.pdf","markdown":"https://ijcai.org/proceedings/2020/317.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/317","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":317,"pages":{"first":2291,"last":2297},"keywords":["Machine Learning: Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24438/file_24438.m3u8","length_sec":281,"provider":"hls"}],"references":[{"position":1,"raw":"[Barto and Sutton, 1982] Andrew G Barto and Richard S Sutton. Simulation of anticipatory responses in classi- cal conditioning by a neuron-like adaptive element. Be- havioural Brain Research, 4(3):221-235, 1982.","title":"Simulation of anticipatory responses in classical conditioning by a neuron-like adaptive element","authors":"G Andrew, Richard S Barto, Sutton","year":"1982","venue":"Behavioural Brain Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berenji, 1992] Hamid R Berenji. A reinforcement learning- based architecture for fuzzy logic control. International Journal of Approximate Reasoning, 6(2):267-292, 1992.","title":"A reinforcement learningbased architecture for fuzzy logic control","authors":"Hamid R Berenji","year":"1992","venue":"International Journal of Approximate Reasoning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016. [Collobert et al., 2011]","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of machine learning research, 12(Aug):2493-2537, 2011.","title":"Natural language processing (almost) from scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa","year":"2011","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fischer et al., 2019] Marc Fischer, Mislav Balunovic, Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, and Martin Vechev. Dl2: Training and querying neural networks with logic. In Proceedings of International Conference on Ma- chine Learning, pages 1931-1941, 2019.","title":"Dl2: Training and querying neural networks with logic","authors":"Marc Fischer, Mislav Balunovic, Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, Martin Vechev","year":"2019","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Garcez et al., 2012] Artur S d'Avila Garcez, Krysia B Broda, and Dov M Gabbay. Neural-symbolic learning sys- tems: foundations and applications. Springer Science & Business Media, 2012.","title":"Neural-symbolic learning systems: foundations and applications","authors":"Artur S D'avila, Krysia B Garcez, Dov M Broda, Gabbay","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ha et al., 2016] David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv:1609.09106, 2016.","title":"Hypernetworks","authors":"David Ha, Andrew Dai, Quoc V Le","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hester et al., 2017] Todd Hester, Matej Vecerík, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Arnold, Ian Osband, John Aga- piou, Joel Z. Leibo, and Audrunas Gruslys. Learning from demonstrations for real world reinforcement learn- ing. CoRR, abs/1704.03732, 2017.","title":"Learning from demonstrations for real world reinforcement learning","authors":"Todd Hester, Matej Vecerík, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Arnold, Ian Osband, John Agapiou, Joel Z Leibo, Audrunas Gruslys","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pages 4565-4573, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu et al., 2016] Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural net- works with logic rules. arXiv:1603.06318, 2016.","title":"Harnessing deep neural networks with logic rules","authors":"Hu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] V Mnih, K Kavukcuoglu, D Silver, A. A. Rusu, J Veness, M. G. Bellemare, A Graves, M Ried- miller, A. K. Fidjeland, and G Ostrovski. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Mnih, Kavukcuoglu, A A Silver, Rusu, M G Veness, Bellemare, Graves, A K Riedmiller, G Fidjeland, Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Pomerleau, 1991] Dean Pomerleau. Efficient training of ar- tificial neural networks for autonomous navigation. Neural Computation, 3(1):88-97, 1991.","title":"Efficient training of artificial neural networks for autonomous navigation","authors":"Dean Pomerleau","year":"1991","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Schmidhuber, 1992] Jürgen Schmidhuber. Learning to con- trol fast-weight memories: An alternative to dynamic re- current networks. Neural Computation, 4(1):131-139, 1992.","title":"Learning to control fast-weight memories: An alternative to dynamic recurrent networks","authors":"Jürgen Schmidhuber","year":"1992","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Schulman et al., 2015a] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of Interna- tional Conference on Machine Learning, pages 1889- 1897, 2015.","title":"Trust region policy optimization","authors":"Schulman","year":"2015","venue":"Proceedings of International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Schulman et al., 2015b] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advan- tage estimation. arXiv:1506.02438, 2015.","title":"Highdimensional continuous control using generalized advantage estimation","authors":"Schulman","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Silver et al., 2016] D Silver, A. Huang, C. J. Maddison, A Guez, L Sifre, den Driessche G Van, J Schrittwieser, I Antonoglou, V Panneershelvam, and M Lanctot. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"A Silver, C J Huang, Maddison, L Guez, Sifre, G Den Driessche, J Van, I Schrittwieser, V Antonoglou, M Panneershelvam, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. [Tasfi, 2016] Norman Tasfi. Pygame learn- ing environment. https://github.com/ntasfi/ PyGame-Learning-Environment, 2016.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2016","venue":"Norman Tasfi. Pygame learning environment","doi":null,"url":"https://github.com/ntasfi/PyGame-Learning-Environment","internal_url":null},{"position":20,"raw":"[Wu et al., 2017] Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using kronecker- factored approximation. In Advances in neural informa- tion processing systems, pages 5279-5288, 2017.","title":"Scalable trust-region method for deep reinforcement learning using kroneckerfactored approximation","authors":"Wu","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yager and Zadeh, 2012] Ronald R Yager and Lotfi A Zadeh. An introduction to fuzzy logic applications in intelligent systems, volume 165. Springer Science & Business Media, 2012.","title":"An introduction to fuzzy logic applications in intelligent systems","authors":"R Ronald, Lotfi A Yager, Zadeh","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zadeh, 1965] L. A. Zadeh. Fuzzy sets. Information and Control, 8(3):338-353, 1965.","title":"Fuzzy sets","authors":"L A Zadeh","year":"1965","venue":"Information and Control","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zadeh, 1996] L. A. Zadeh. Knowledge representation in fuzzy logic. In Fuzzy Sets, Fuzzy Logic, And Fuzzy Sys- tems: Selected Papers, pages 764-774. World Scientific, 1996.","title":"Knowledge representation in fuzzy logic","authors":"L A Zadeh","year":"1996","venue":"Fuzzy Sets, Fuzzy Logic, And Fuzzy Systems: Selected Papers","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}