{"type":"ScholarlyArticle","title":"Function Approximation for Reinforcement Learning Controller for Energy from Spread Waves","authors":[{"name":"Soumyendu Sarkar","slug":"soumyendu-sarkar","given":"Soumyendu","family":"Sarkar","affiliation":"Hewlett Packard Enterprise , USA"},{"name":"Vineet Gundecha","slug":"vineet-gundecha","given":"Vineet","family":"Gundecha","affiliation":"Hewlett Packard Enterprise , USA"},{"name":"Sahand Ghorbanpour","slug":"sahand-ghorbanpour","given":"Sahand","family":"Ghorbanpour","affiliation":"Hewlett Packard Enterprise , USA"},{"name":"Alexander Shmakov","slug":"alexander-shmakov","given":"Alexander","family":"Shmakov","affiliation":"Hewlett Packard Enterprise , USA"},{"name":"Ashwin Ramesh Babu","slug":"ashwin-ramesh-babu","given":"Ashwin","family":"Ramesh Babu","affiliation":"Hewlett Packard Enterprise , USA"},{"name":"Avisek Naug","slug":"avisek-naug","given":"Avisek","family":"Naug","affiliation":"Hewlett Packard Enterprise , USA"},{"name":"Alexandre Pichard","slug":"alexandre-pichard","given":"Alexandre","family":"Pichard","affiliation":"Carnegie Clean Energy , Australia"},{"name":"Mathieu Cocho","slug":"mathieu-cocho","given":"Mathieu","family":"Cocho","affiliation":"Carnegie Clean Energy , Australia"}],"abstract":"The industrial multi-generator Wave Energy Converters (WEC) must handle multiple simultaneous waves coming from different directions called spread waves. These complex devices in challenging circumstances need controllers with multiple objectives of energy capture efficiency, reduction of structural stress to limit maintenance, and proactive protection against high waves. The Multi-Agent Reinforcement Learning (MARL) controller trained with Proximal Policy Optimization (PPO) algorithm can handle these complexities. In this paper, we explore different function approximations for the policy and critic networks in modeling the sequential nature of the system dynamics and find that they are key to better performance. We investigated the performance of a fully connected neural network (FCN), LSTM, and Transformer model variants with varying depths and gated residual connections. Our results show that the transformer model of moderate depth with gated residual connections around the multi-head attention, multi-layer perceptron, and the transformer block (STrXL) proposed in this paper is optimal and boosts energy efficiency by an average of 22.1% for these complex spread waves over the existing spring damper (SD) controller. Furthermore, unlike the default SD controller, the transformer controller almost eliminated the mechanical stress from the rotational yaw motion for angled waves. Demo: https://tinyurl.com/yueda3jh","doi":"10.24963/ijcai.2023/688","url":"https://ijcai.org/proceedings/2023/688","pdf":"https://ijcai.org/proceedings/2023/0688.pdf","markdown":"https://ijcai.org/proceedings/2023/688.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/688","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"AI for Good","track":"AI for Good","number":688,"pages":{"first":6201,"last":6209},"keywords":["AI for Good: Multidisciplinary Topics and Applications","AI for Good: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Anderlini et al., 2016] Enrico Anderlini, David IM Fore- hand, Paul Stansell, Qing Xiao, and Mohammad Abusara. Control of a point absorber using reinforcement learning. IEEE Transactions on Sustainable Energy, 7(4):1681- 1690, 2016.","title":"Control of a point absorber using reinforcement learning","authors":"Enrico Anderlini, I M David, Paul Forehand, Qing Stansell, Mohammad Xiao, Abusara","year":"2016","venue":"IEEE Transactions on Sustainable Energy","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderlini et al., 2017a] Enrico Anderlini, David IM Fore- hand, Elva Bannon, and Mohammad Abusara. Control of a realistic wave energy converter model using least-squares policy iteration. IEEE Transactions on Sustainable En- ergy, 8(4):1618-1628, 2017.","title":"Control of a realistic wave energy converter model using least-squares policy iteration","authors":"Anderlini","year":"2017","venue":"IEEE Transactions on Sustainable Energy","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Anderlini et al., 2017b] Enrico Anderlini, DIM Forehand, Elva Bannon, and Mohammad Abusara. Reactive control of a wave energy converter using artificial neural networks. International journal of marine energy, 19:207-220, 2017.","title":"Reactive control of a wave energy converter using artificial neural networks","authors":"Anderlini","year":"2017","venue":"International journal of marine energy","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Anderlini et al., 2018] E Anderlini, DIM Forehand, E Ban- non, Q Xiao, and M Abusara. Reactive control of a two- body point absorber using reinforcement learning. Ocean Engineering, 148:650-658, 2018.","title":"Reactive control of a twobody point absorber using reinforcement learning","authors":"Anderlini","year":"2018","venue":"Ocean Engineering","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Anderlini et al., 2020] Enrico Anderlini, Salman Husain, Gordon G Parker, Mohammad Abusara, and Giles Thomas. Towards real-time reinforcement learning con- trol of a wave energy converter. Journal of Marine Science and Engineering, 8(11):845, 2020.","title":"Towards real-time reinforcement learning control of a wave energy converter","authors":"Enrico Anderlini, Salman Husain, Gordon G Parker, Mohammad Abusara, Giles Thomas","year":"2020","venue":"Journal of Marine Science and Engineering","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duan et al., 2016] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Interna- tional conference on machine learning, pages 1329-1338.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel","year":null,"venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"PMLR, 2016.","title":null,"authors":"Pmlr","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gambarelli, 2022] Leonardo Gambarelli. On the viability and effectiveness of Reinforcement Learning techniques for Wave-Energy converter control. PhD thesis, Politec- nico di Torino, 2022.","title":"Leonardo Gambarelli. On the viability and effectiveness of Reinforcement Learning techniques for Wave-Energy converter control","authors":"Gambarelli","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on ma- chine learning, pages 1861-1870. PMLR, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2022] Tuo Li, Ming Zhang, Shuang-Rui Yu, and Zhi-Ming Yuan. Real-time control of wecs based on nar, narx and lstm artificial neural network. In The 32nd International Ocean and Polar Engineering Conference. OnePetro, 2022.","title":"Real-time control of wecs based on nar, narx and lstm artificial neural network","authors":"Tuo Li, Ming Zhang, Shuang-Rui Yu, Zhi-Ming Yuan","year":"2022","venue":"The 32nd International Ocean and Polar Engineering Conference","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lim et al., 2021] Bryan Lim, Sercan Ö Arık, Nicolas Lo- eff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. Inter- national Journal of Forecasting, 2021.","title":"Temporal fusion transformers for interpretable multi-horizon time series forecasting","authors":"Bryan Lim, Ö Sercan, Nicolas Arık, Tomas Loeff, Pfister","year":"2021","venue":"International Journal of Forecasting","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Mnih","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational conference on machine learning, pages 1928- 1937. PMLR, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Parisotto et al., 2020] Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayaku- mar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for re- inforcement learning. In International Conference on Ma- chine Learning, pages 7487-7498. PMLR, 2020.","title":"Stabilizing transformers for reinforcement learning","authors":"Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Plappert et al., 2017] Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. CoRR, abs/1706.01905, 2017.","title":"Parameter space noise for exploration","authors":"Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, Marcin Andrychowicz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rijnsdorp et al., 2018] Dirk P Rijnsdorp, Jeff E Hansen, and Ryan J Lowe. Simulating the wave-induced response of a submerged wave-energy converter using a non-hydrostatic wave-flow model. Coastal Engineering, 140:189-204, 2018.","title":"Simulating the wave-induced response of a submerged wave-energy converter using a non-hydrostatic wave-flow model","authors":"Jeff E Dirk P Rijnsdorp, Ryan J Hansen, Lowe","year":"2018","venue":"Coastal Engineering","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sarkar et al., 2021] Soumyendu Sarkar, Vineet Gun- decha, Alexander Shmakov, Sahand Ghorbanpour, Ashwin Ramesh Babu, Paolo Faraboschi, Mathieu Cocho, Alexandre Pichard, and Jonathan Fievez. Multi-objective reinforcement learning controller for multi-generator industrial wave energy converter. In NeurIPs Tackling Climate Change with Machine Learning Workshop, 2021.","title":"Multi-objective reinforcement learning controller for multi-generator industrial wave energy converter","authors":"Sarkar","year":"2021","venue":"NeurIPs Tackling Climate Change with Machine Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sarkar et al., 2022a] Soumyendu Sarkar, Vineet Gun- decha, Sahand Ghorbanpour, Alexander Shmakov, Ashwin Ramesh Babu, Alexandre Pichard, and Mathieu Cocho. Skip training for multi-agent reinforcement learn- ing controller for industrial wave energy converters. In 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE), pages 212-219. IEEE, 2022.","title":"Skip training for multi-agent reinforcement learning controller for industrial wave energy converters","authors":"Sarkar","year":"2022","venue":"IEEE 18th International Conference on Automation Science and Engineering (CASE)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sarkar et al., 2022b] Soumyendu Sarkar, Vineet Gun- decha, Alexander Shmakov, Sahand Ghorbanpour, Ashwin Ramesh Babu, Paolo Faraboschi, Mathieu Cocho, Alexandre Pichard, and Jonathan Fievez. Multi-agent reinforcement learning controller to maximize energy efficiency for multi-generator industrial wave energy converter. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12135-12144, 2022.","title":"Multi-agent reinforcement learning controller to maximize energy efficiency for multi-generator industrial wave energy converter","authors":"Sarkar","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Seo et al., 2021] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient explo- ration. In International Conference on Machine Learning, 2021.","title":"State entropy maximization with random encoders for efficient exploration","authors":"Seo","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Trigueiro et al., 2022] José Trigueiro, Miguel Ayala Botto, Susana Vieira, and João Henriques. Control of a wave energy converter using reinforcement learning. In Luís Brito Palma, Rui Neves-Silva, and Luís Gomes, editors, CONTROLO 2022, pages 567-576, Cham, 2022. Springer International Publishing.","title":"Control of a wave energy converter using reinforcement learning","authors":"Trigueiro","year":"2022","venue":"CONTROLO 2022","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2022] Zheng Wu, Yunfei Lu, Qi Xu, Weixing Chen, Weidong Zhang, and Feng Gao. Load optimization control of sjtu-wec based on machine learning. Ocean En- gineering, 249:110851, 2022.","title":"Load optimization control of sjtu-wec based on machine learning","authors":"Zheng Wu, Yunfei Lu, Qi Xu, Weixing Chen, Weidong Zhang, Feng Gao","year":"2022","venue":"Ocean Engineering","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yin and Jiang, 2023] Xiuxing Yin and Zhansi Jiang. Wave condition preview assisted real-time nonlinear predictive control of point-absorbing wave energy converter based on long short-term memory recurrent neural identification. Mechanical Systems and Signal Processing, 188:109669, 2023.","title":"Xiuxing Yin and Zhansi Jiang. Wave condition preview assisted real-time nonlinear predictive control of point-absorbing wave energy converter based on long short-term memory recurrent neural identification","authors":"Jiang Yin","year":"2023","venue":"Mechanical Systems and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zou et al., 2022] Shangyan Zou, Xiang Zhou, Irfan Khan, Wayne W. Weaver, and Syed Rahman. Optimization of the electricity generation of a wave energy converter us- ing deep reinforcement learning. Ocean Engineering, 244:110363, 2022.","title":"Optimization of the electricity generation of a wave energy converter using deep reinforcement learning","authors":"Zou","year":"2022","venue":"Ocean Engineering","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}