{"type":"ScholarlyArticle","title":"Global Optimality of Single-Timescale Actor-Critic under Continuous State-Action Space: A Study on Linear Quadratic Regulator","authors":[{"name":"Xuyang Chen","slug":"xuyang-chen","given":"Xuyang","family":"Chen","affiliation":"National University of Singapore"},{"name":"Jingliang Duan","slug":"jingliang-duan","given":"Jingliang","family":"Duan","affiliation":"University of Science and Technology Beijing"},{"name":"Lin Zhao","slug":"lin-zhao","given":"Lin","family":"Zhao","affiliation":"National University of Singapore"}],"abstract":"Actor-critic methods have achieved state-of-the-art performance in various challenging tasks. However, theoretical understandings of their performance remain elusive and challenging. Existing studies mostly focus on practically uncommon variants such as double-loop or two-timescale stepsize actor-critic algorithms for simplicity. These results certify local convergence on finite state- or action- space only. We push the boundary to investigate the classic single-sample single-timescale actor-critic on continuous (infinite) state-action space, where we employ the canonical linear quadratic regulator (LQR) problem as a case study. We show that the popular single-timescale actor-critic can attain an epsilon-optimal solution with an order of epsilon to -2 sample complexity for solving LQR on the demanding continuous state-action space. Our work provides new insights into the performance of single-timescale actor-critic, which further bridges the gap between theory and practice.","doi":"10.24963/ijcai.2024/422","url":"https://ijcai.org/proceedings/2024/422","pdf":"https://ijcai.org/proceedings/2024/0422.pdf","markdown":"https://ijcai.org/proceedings/2024/422.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/422","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":422,"pages":{"first":3816,"last":3824},"keywords":["Machine Learning: ML: Reinforcement learning","Machine Learning: ML: Learning theory"],"videos":[],"references":[{"position":1,"raw":"[Anderson and Moore, 2007] Brian DO Anderson and John B Moore. Optimal control: linear quadratic methods. Courier Corporation, 2007.","title":"Optimal control: linear quadratic methods","authors":"D O Brian, John B Anderson, Moore","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barakat et al., 2022] Anas Barakat, Pascal Bianchi, and Julien Lehmann. Analysis of a target-based actor-critic algorithm with linear function approximation. In Interna- tional Conference on Artificial Intelligence and Statistics, pages 991-1040. PMLR, 2022.","title":"Analysis of a target-based actor-critic algorithm with linear function approximation","authors":"Barakat","year":"2022","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bhatnagar et al., 2009] Shalabh Bhatnagar, Richard S Sut- ton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Automatica, 45(11):2471-2482, 2009.","title":"Natural actor-critic algorithms","authors":"Shalabh Bhatnagar, Mohammad Richard S Sutton, Mark Ghavamzadeh, Lee","year":"2009","venue":"Automatica","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bradtke et al., 1994] Steven J Bradtke, B Erik Ydstie, and Andrew G Barto. Adaptive linear quadratic control using policy iteration. In Proceedings of 1994 American Control Conference-ACC'94, volume 3, pages 3475-3479. IEEE, 1994.","title":"Adaptive linear quadratic control using policy iteration","authors":"B Steven J Bradtke, Andrew G Erik Ydstie, Barto","year":"1994","venue":"Proceedings of 1994 American Control Conference-ACC'94","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Castro and Meir, 2010] Dotan Di Castro and Ron Meir. A convergent online single time scale actor critic algorithm. The Journal of Machine Learning Research, 11:367-410, 2010.","title":"A convergent online single time scale actor critic algorithm","authors":"Di Dotan, Ron Castro, Meir","year":"2010","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and Zhao, 2022] Xuyang Chen and Lin Zhao. Finite- time analysis of single-timescale actor-critic. arXiv preprint arXiv:2210.09921, 2022.","title":"Finitetime analysis of single-timescale actor-critic","authors":"Chen, Zhao, Xuyang Chen, Lin Zhao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2021] Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternat- ing stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems, 34:25294-25307, 2021.","title":"Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems","authors":"Chen","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cohen et al., 2019] Alon Cohen, Tomer Koren, and Yishay Mansour. Learning linear-quadratic regulators efficiently with only √ T regret. In International Conference on Ma- chine Learning, pages 1300-1309. PMLR, 2019.","title":"Learning linear-quadratic regulators efficiently with only √ T regret","authors":"Alon Cohen, Tomer Koren, Yishay Mansour","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dean et al., 2018] Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. Regret bounds for robust adaptive control of the linear quadratic regulator. Advances in Neural Information Processing Systems, 31, 2018.","title":"Regret bounds for robust adaptive control of the linear quadratic regulator","authors":"Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, Stephen Tu","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dean et al., 2020] Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. On the sample com- plexity of the linear quadratic regulator. Foundations of Computational Mathematics, 20(4):633-679, 2020.","title":"On the sample complexity of the linear quadratic regulator","authors":"Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, Stephen Tu","year":"2020","venue":"Foundations of Computational Mathematics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Duan et al., 2023] Jingliang Duan, Jie Li, Xuyang Chen, Kai Zhao, Shengbo Eben Li, and Lin Zhao. Optimiza- tion landscape of policy gradient methods for discrete-time static output feedback. IEEE Transactions on Cybernetics, 2023.","title":"Optimization landscape of policy gradient methods for discrete-time static output feedback","authors":"Jingliang Duan, Jie Li, Xuyang Chen, Kai Zhao, Shengbo, Eben Li, Lin Zhao","year":"2023","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fazel et al., 2018] Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gra- dient methods for the linear quadratic regulator. In Inter- national Conference on Machine Learning, pages 1467- 1476. PMLR, 2018.","title":"Global convergence of policy gradient methods for the linear quadratic regulator","authors":"Maryam Fazel, Rong Ge, Sham Kakade, Mehran Mesbahi","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Joarder and Omar, 2011] Anwar H Joarder and M Hafidz Omar. On statistical characteristics of the product of two correlated chi-square variables. Journal of Applied Statis- tical Science, 19(4):89-101, 2011.","title":"On statistical characteristics of the product of two correlated chi-square variables","authors":"H Anwar, M Joarder, Omar Hafidz","year":"2011","venue":"Journal of Applied Statistical Science","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kakade, 2001] Sham M Kakade. A natural policy gradi- ent. Advances in neural information processing systems, 14, 2001.","title":"A natural policy gradient","authors":"M Sham, Kakade","year":"2001","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Karmakar and Bhatnagar, 2018] Prasenjit Karmakar and Shalabh Bhatnagar. Two time-scale stochastic approxima- tion with controlled markov noise and off-policy temporal- difference learning. Mathematics of Operations Research, 43(1):130-151, 2018.","title":"Two time-scale stochastic approximation with controlled markov noise and off-policy temporaldifference learning","authors":"Prasenjit Karmakar, Shalabh Bhatnagar","year":"2018","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Konda and Tsitsiklis, 1999] Vijay Konda and John Tsitsik- lis. Actor-critic algorithms. Advances in neural informa- tion processing systems, 12, 1999.","title":"Actor-critic algorithms","authors":"Vijay Konda, John Tsitsiklis","year":"1999","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Krauth et al., 2019] Karl Krauth, Stephen Tu, and Benjamin Recht. Finite-time analysis of approximate policy iteration for the linear quadratic regulator. Advances in Neural In- formation Processing Systems, 32, 2019.","title":"Finite-time analysis of approximate policy iteration for the linear quadratic regulator","authors":"Karl Krauth, Stephen Tu, Benjamin Recht","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kumar et al., 2019] Harshat Kumar, Alec Koppel, and Ale- jandro Ribeiro. On the sample complexity of actor-critic method for reinforcement learning with function approxi- mation. arXiv preprint arXiv:1910.08412, 2019.","title":"On the sample complexity of actor-critic method for reinforcement learning with function approximation","authors":"Kumar","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey Hinton. Deep learning. nature, 521(7553):436-444, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Magnus, 1978] Jan R Magnus. The moments of products of quadratic forms in normal variables. Statistica Neer- landica, 32(4):201-210, 1978.","title":"The moments of products of quadratic forms in normal variables","authors":"Jan R, Magnus","year":"1978","venue":"Statistica Neerlandica","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Malik et al., 2019] Dhruv Malik, Ashwin Pananjady, Kush Bhatia, Koulik Khamaru, Peter Bartlett, and Martin Wain- wright. Derivative-free methods for policy optimization: Guarantees for linear quadratic systems. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2916-2925. PMLR, 2019.","title":"Derivative-free methods for policy optimization: Guarantees for linear quadratic systems","authors":"Dhruv Malik, Ashwin Pananjady, Kush Bhatia, Koulik Khamaru, Peter Bartlett, Martin Wainwright","year":"2019","venue":"The 22nd International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mania et al., 2019] Horia Mania, Stephen Tu, and Ben- jamin Recht. Certainty equivalence is efficient for linear quadratic control. Advances in Neural Information Pro- cessing Systems, 32, 2019.","title":"Certainty equivalence is efficient for linear quadratic control","authors":"Horia Mania, Stephen Tu, Benjamin Recht","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational conference on machine learning, pages 1928- 1937. PMLR, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nagar, 1959] Anirudh L Nagar. The bias and moment ma- trix of the general k-class estimators of the parameters in simultaneous equations. Econometrica: Journal of the Econometric Society, pages 575-595, 1959.","title":"The bias and moment matrix of the general k-class estimators of the parameters in simultaneous equations","authors":"L Anirudh, Nagar","year":"1959","venue":"Econometrica: Journal of the Econometric Society","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Olshevsky and Gharesifard, 2023] Alex Olshevsky and Bahman Gharesifard. A small gain analysis of single timescale actor critic. SIAM Journal on Control and Optimization, 61(2):980-1007, 2023.","title":"A small gain analysis of single timescale actor critic","authors":"Alex Olshevsky, Bahman Gharesifard","year":"2023","venue":"SIAM Journal on Control and Optimization","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rencher and Schaalje, 2008] Alvin C Rencher and G Bruce Schaalje. Linear models in statistics. John Wiley & Sons, 2008.","title":"Linear models in statistics","authors":"C Alvin, G Rencher, Bruce Schaalje","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Schacke, 2004] Kathrin Schacke. On the kronecker product. Master's thesis, University of Waterloo, 2004.","title":"On the kronecker product","authors":"Kathrin Schacke","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sutton et al., 1999] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approx- imation. Advances in neural information processing sys- tems, 12, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David Richard S Sutton, Satinder Mcallester, Yishay Singh, Mansour","year":"1999","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tu and Recht, 2018] Stephen Tu and Benjamin Recht. Least-squares temporal difference learning for the linear quadratic regulator. In International Conference on Ma- chine Learning, pages 5005-5014. PMLR, 2018.","title":"Least-squares temporal difference learning for the linear quadratic regulator","authors":"Stephen Tu, Benjamin Recht","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2019] Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv:1909.01150, 2019.","title":"Neural policy gradient methods: Global optimality and rates of convergence","authors":"Wang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wu et al., 2020] Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Pro- cessing Systems, 33:17617-17628, 2020.","title":"A finite-time analysis of two time-scale actor-critic methods","authors":"Frank Yue, Weitong Wu, Pan Zhang, Quanquan Xu, Gu","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Xu et al., 2020] Tengyu Xu, Zhe Wang, and Yingbin Liang. Non-asymptotic convergence analysis of two time- scale (natural) actor-critic algorithms. arXiv preprint arXiv:2005.03557, 2020.","title":"Non-asymptotic convergence analysis of two timescale (natural) actor-critic algorithms","authors":"Xu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yang et al., 2019] Zhuoran Yang, Yongxin Chen, Mingyi Hong, and Zhaoran Wang. Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost. Advances in neural information processing systems, 32, 2019.","title":"Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost","authors":"Yang","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang et al., 2020] Shangtong Zhang, Bo Liu, Hengshuai Yao, and Shimon Whiteson. Provably convergent two- timescale off-policy actor-critic with function approxima- tion. In International Conference on Machine Learning, pages 11204-11213. PMLR, 2020.","title":"Provably convergent twotimescale off-policy actor-critic with function approximation","authors":"Zhang","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhou and Lu, 2023] Mo Zhou and Jianfeng Lu. Single timescale actor-critic method to solve the linear quadratic regulator with convergence guarantees. Journal of Ma- chine Learning Research, 24(222):1-34, 2023.","title":"Single timescale actor-critic method to solve the linear quadratic regulator with convergence guarantees","authors":"Lu ; ] Mo Zhou, Jianfeng Zhou, Lu","year":"2023","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}