{"type":"ScholarlyArticle","title":"Multi-Constraint Deep Reinforcement Learning for Smooth Action Control","authors":[{"name":"Guangyuan Zou","slug":"guangyuan-zou","given":"Guangyuan","family":"Zou","affiliation":"College of Computer Science and Software Engineering , Shenzhen University , P.R. China; Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) , Shenzhen , China"},{"name":"Ying He","slug":"ying-he","given":"Ying","family":"He","affiliation":"College of Computer Science and Software Engineering , Shenzhen University , P.R. China; Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) , Shenzhen , China"},{"name":"F. Richard Yu","slug":"f-richard-yu","given":"F. Richard","family":"Yu","affiliation":"College of Computer Science and Software Engineering , Shenzhen University , P.R. China; Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) , Shenzhen , China"},{"name":"Longquan Chen","slug":"longquan-chen","given":"Longquan","family":"Chen","affiliation":"College of Computer Science and Software Engineering , Shenzhen University , P.R. China; Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) , Shenzhen , China"},{"name":"Weike Pan","slug":"weike-pan","given":"Weike","family":"Pan","affiliation":"College of Computer Science and Software Engineering , Shenzhen University , P.R. China"},{"name":"Zhong Ming","slug":"zhong-ming","given":"Zhong","family":"Ming","affiliation":"College of Computer Science and Software Engineering , Shenzhen University , P.R. China"}],"abstract":"Deep reinforcement learning (DRL) has been studied in a variety of challenging decision-making tasks, e.g., autonomous driving. \\textcolor{black}{However, DRL typically suffers from the action shaking problem, which means that agents can select actions with big difference even though states only slightly differ.} One of the crucial reasons for this issue is the inappropriate design of the reward in DRL. In this paper, to address this issue, we propose a novel way to incorporate the smoothness of actions in the reward. Specifically, we introduce sub-rewards and add multiple constraints related to these sub-rewards. In addition, we propose a multi-constraint proximal policy optimization (MCPPO) method to solve the multi-constraint DRL problem. Extensive simulation results show that the proposed MCPPO method has better action smoothness compared with the traditional proportional-integral-differential (PID) and mainstream DRL algorithms. The video is available at https://youtu.be/F2jpaSm7YOg.","doi":"10.24963/ijcai.2022/528","url":"https://ijcai.org/proceedings/2022/528","pdf":"https://ijcai.org/proceedings/2022/0528.pdf","markdown":"https://ijcai.org/proceedings/2022/528.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/528","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":528,"pages":{"first":3802,"last":3808},"keywords":["Robotics: Behavior and Control","Machine Learning: Deep Reinforcement Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986658","length_sec":78,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984801","length_sec":349,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Achiam et al., 2017] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimiza- tion. In ICML, 2017.","title":"Constrained policy optimization","authors":"Achiam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Altman, 1999] Eitan Altman. Constrained Markov decision processes. CRC Press, 1999.","title":"Constrained Markov decision processes","authors":"Eitan Altman","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Amodei et al., 2016] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. CoRR, abs/1606.06565, 2016.","title":"Concrete problems in AI safety","authors":"Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F Christiano, John Schulman, Dan Mané","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Badia et al., 2020] Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. In ICLR, 2020.","title":"Never give up: Learning directed exploration strategies","authors":"Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andrew Bolt, Charles Blundell","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bellman, 2015] Richard E. Bellman. Adaptive control pro- cesses: A guided tour. Princeton University Press, 2015.","title":"Adaptive control processes: A guided tour","authors":"Richard E Bellman","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Boyd and Vandenberghe, 2014] Stephen P. Boyd and Lieven Vandenberghe. Convex optimization. Cambridge University Press, 2014.","title":"Convex optimization","authors":"Stephen P Boyd, Lieven Vandenberghe","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2021] Chen Chen, Hongyao Tang, Jianye Hao, Wulong Liu, and Zhaopeng Meng. Addressing action os- cillations through learning policy inertia. In AAAI, 2021.","title":"Addressing action oscillations through learning policy inertia","authors":"Chen Chen, Hongyao Tang, Jianye Hao, Wulong Liu, Zhaopeng Meng","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chow et al., 2017] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. In J. Mach. Learn. Res., 18:167:1-167:51, 2017.","title":"Risk-constrained reinforcement learning with percentile risk criteria","authors":"Chow","year":"2017","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dosovitskiy et al., 2017] Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladlen Koltun. CARLA: An open urban driving simulator. In CoRL, 2017.","title":"CARLA: An open urban driving simulator","authors":"Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M López, Vladlen Koltun","year":"2017","venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Engstrom et al., 2020] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on PPO and TRPO. CoRR, 2020.","title":"Implementation matters in deep policy gradients: A case study on PPO and TRPO","authors":"Engstrom","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Girshick, 2015] Ross B. Girshick. Fast R-CNN. In ICCV, 2015.","title":"Fast R-CNN","authors":"Ross B Girshick","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, abs/1812.05905, 2018.","title":"Soft actor-critic algorithms and applications","authors":"Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lillicrap et al., 2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yu- val Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021] Xiao-Yang Liu, Zechu Li, Zhuoran Yang, Jiahao Zheng, Zhaoran Wang, Anwar Walid, Jian Guo, and Michael I. Jordan. Elegantrl-podracer: Scalable and elastic library for cloud-native deep reinforcement learn- ing. CoRR, abs/2112.05923, 2021.","title":"Elegantrl-podracer: Scalable and elastic library for cloud-native deep reinforcement learning","authors":"Xiao-Yang Liu, Zechu Li, Zhuoran Yang, Jiahao Zheng, Zhaoran Wang, Anwar Walid, Jian Guo, Michael I Jordan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin A Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Neunert et al., 2019] Michael Neunert, Abbas Abdol- maleki, Markus Wulfmeier, Thomas Lampe, Jost Tobias Springenberg, Roland Hafner, Francesco Romano, Jonas Buchli, Nicolas Heess, and Martin A. Riedmiller. Continuous-discrete reinforcement learning for hybrid control in robotics. In CoRL, 2019.","title":"Continuous-discrete reinforcement learning for hybrid control in robotics","authors":"Michael Neunert, Abbas Abdolmaleki, Markus Wulfmeier, Thomas Lampe, Jost Tobias Springenberg, Roland Hafner, Francesco Romano, Jonas Buchli, Nicolas Heess, Martin A Riedmiller","year":"2019","venue":"CoRL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Paternain et al., 2019] Santiago Paternain, Luiz F. O. Cha- mon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Con- strained reinforcement learning has zero duality gap. In NeurIPS, 2019.","title":"Constrained reinforcement learning has zero duality gap","authors":"Santiago Paternain, F O Luiz, Miguel Chamon, Alejandro Calvo-Fullana, Ribeiro","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schulman et al., 2016] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In ICLR, 2016.","title":"High-dimensional continuous control using generalized advantage estimation","authors":"John Schulman, Philipp Moritz, Sergey Levine, Michael I Jordan, Pieter Abbeel","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Siekmann et al., 2021] Jonah Siekmann, Kevin Green, John Warila, Alan Fern, and Jonathan W. Hurst. Blind bipedal stair traversal via sim-to-real reinforcement learning. In RSS, 2021.","title":"Blind bipedal stair traversal via sim-to-real reinforcement learning","authors":"Siekmann","year":"2021","venue":"RSS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Song et al., 2020] Wenjie Song, Shixian Liu, Yujun Li, Yi Yang, and Changle Xiang. Smooth actor-critic algo- rithm for end-to-end autonomous driving. In ACC, 2020.","title":"Smooth actor-critic algorithm for end-to-end autonomous driving","authors":"Wenjie Song, Shixian Liu, Yujun Li, Yi Yang, Changle Xiang","year":"2020","venue":"ACC","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Stooke et al., 2020] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learn- ing by PID Lagrangian methods. In ICML, 2020.","title":"Responsive safety in reinforcement learning by PID Lagrangian methods","authors":"Stooke","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutton and Barto, 1998] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, 1998.","title":"Reinforcement learning: An introduction","authors":"Richard S Sutton, Andrew G Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tessler et al., 2019] Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In ICLR, 2019.","title":"Reward constrained policy optimization","authors":"Chen Tessler, Daniel J Mankowitz, Shie Mannor","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2016] Ziyu Wang, Tom Schaul, Matteo Hes- sel, Hado van Hasselt, Marc Lanctot, and Nando de Fre- itas. Dueling network architectures for deep reinforcement learning. In ICML, 2016.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul, Matteo Hessel, Marc Hado Van Hasselt, Nando Lanctot, De Freitas","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2020] Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection- based constrained policy optimization. In ICLR, 2020.","title":"Projectionbased constrained policy optimization","authors":"Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, Peter J Ramadge","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}