{"type":"ScholarlyArticle","title":"Recurrent Deep Multiagent Q-Learning for Autonomous Brokers in Smart Grid","authors":[{"name":"Yaodong Yang","slug":"yaodong-yang","given":"Yaodong","family":"Yang","affiliation":"School of Computer Software , Tianjin University"},{"name":"Jianye Hao","slug":"jianye-hao","given":"Jianye","family":"Hao","affiliation":"School of Computer Software , Tianjin University"},{"name":"Mingyang Sun","slug":"mingyang-sun","given":"Mingyang","family":"Sun","affiliation":"Imperial College London"},{"name":"Zan Wang","slug":"zan-wang","given":"Zan","family":"Wang","affiliation":"School of Computer Software , Tianjin University"},{"name":"Changjie Fan","slug":"changjie-fan","given":"Changjie","family":"Fan","affiliation":"NetEase, Inc."},{"name":"Goran Strbac","slug":"goran-strbac","given":"Goran","family":"Strbac","affiliation":"Imperial College London"}],"abstract":"The broker mechanism is widely applied to serve for interested parties to derive long-term policies in order to reduce costs or gain profits in smart grid. However, a broker is faced with a number of challenging problems such as balancing demand and supply from customers and competing with other coexisting brokers to maximize its profit. In this paper, we develop an effective pricing strategy for brokers in local electricity retail market based on recurrent deep multiagent reinforcement learning and sequential clustering. We use real household electricity consumption data to simulate the retail market for evaluating our strategy. The experiments demonstrate the superior performance of the proposed pricing strategy and highlight the effectiveness of our reward shaping mechanism.","doi":"10.24963/ijcai.2018/79","url":"https://ijcai.org/proceedings/2018/79","pdf":"https://ijcai.org/proceedings/2018/0079.pdf","markdown":"https://ijcai.org/proceedings/2018/79.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/79","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":79,"pages":{"first":569,"last":575},"keywords":["Agent-based and Multi-agent Systems: Multi-agent Learning","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Angelidakis and Chalkiadakis, 2015] Angelos Angelidakis and Georgios Chalkiadakis. Factored mdps for opti- mal prosumer decision-making in continuous state spaces. In the Proceedings of the 13th European Conference on Multi-Agent Systems, pages 91-107, 2015.","title":"Factored mdps for optimal prosumer decision-making in continuous state spaces","authors":"Angelos Angelidakis, Georgios Chalkiadakis","year":"2015","venue":"the Proceedings of the 13th European Conference on Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2017] Qingpeng Cai, Aris Filos-Ratsikas, Pingzhong Tang, and Yiwei Zhang. Reinforcement mechanism design for e-commerce. CoRR, 2017.","title":"Reinforcement mechanism design for e-commerce","authors":"Cai","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chang et al., 2003] Yu Han Chang, Tracey Ho, and Leslie Pack Kaelbling. All learning is local: Multi-agent learning in global reward games. In Proceedings of the 16th NIPS, pages 807-814, 2003.","title":"All learning is local: Multi-agent learning in global reward games","authors":"Yu Han, Chang, Tracey Ho, Leslie Pack, Kaelbling","year":"2003","venue":"Proceedings of the 16th NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chowdhury et al., 2015] Moinul Morshed Porag Chowd- hury, Russell Y. Folk, Ferdinando Fioretto, Christopher Kiekintveld, and William Yeoh. Investigation of learning strategies for the spot broker in power tac. In Proceedings of the 17th AMEC/TADA, 2015. [Detailed State Data, 2010] Average price by state by provider, 2010. https://www.eia.gov/electricity/data/state/.","title":"Investigation of learning strategies for the spot broker in power tac","authors":"Moinul Morshed, Porag Chowdhury, Russell Y Folk, Ferdinando Fioretto, Christopher Kiekintveld, William Yeoh","year":"2010","venue":"Proceedings of the 17th AMEC/TADA","doi":null,"url":"https://www.eia.gov/electricity/data/state/","internal_url":null},{"position":5,"raw":"[Energy Consumption Data, 2015] Electricity consump- tion in a sample of london households, 2015. https://data.london.gov.uk/dataset/smartmeter-energy- use-data-in-london-households.","title":"Electricity consumption in a sample of london households","authors":null,"year":"2015","venue":null,"doi":null,"url":"https://data.london.gov.uk/dataset/smartmeter-energy-use-data-in-london-households","internal_url":null},{"position":6,"raw":"[Fang et al., 2012] Xi Fang, Satyajayant Misra, Guoliang Xue, and Dejun Yang. Smart grid -the new and improved power grid: A survey. IEEE Communications Surveys and Tutorials, 14(4):944-980, 2012.","title":"Smart grid -the new and improved power grid: A survey","authors":"Xi Fang, Satyajayant Misra, Guoliang Xue, Dejun Yang","year":"2012","venue":"IEEE Communications Surveys and Tutorials","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gu et al., 2017] Shixiang Gu, Ethan Holly, Timothy Lilli- crap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy up- dates. In Proceedings of IEEE International Conference on Robotics and Automation, pages 3389-3396, 2017.","title":"Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates","authors":"Gu","year":"2017","venue":"Proceedings of IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hochreiter and Schmidhuber, 1997] S Hochreiter and J Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber ; S Hochreiter, J Hochreiter, Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Keogh and Ratanamahatana, 2005] Eamonn Keogh and Chotirat Ann Ratanamahatana. Exact indexing of dy- namic time warping. Knowledge and Information Systems, 7(3):358-386, 2005.","title":"Exact indexing of dynamic time warping","authors":"Eamonn Keogh, Ann Chotirat, Ratanamahatana","year":"2005","venue":"Knowledge and Information Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ketter et al., 2013] Wolfgang Ketter, Markus Peters, and John Collins. Autonomous agents in future energy mar- kets: the 2012 power trading agent competition. In Pro- ceedings of the 27th AAAI, pages 1298-1304, 2013.","title":"Autonomous agents in future energy markets: the 2012 power trading agent competition","authors":"Ketter","year":"2013","venue":"Proceedings of the 27th AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liefers et al., 2014] Bart Liefers, Jasper Hoogland, and La Poutré Han. A successful broker agent for power tac. Lecture Notes in Business Information Processing, pages 99-113, 2014.","title":"A successful broker agent for power tac","authors":"Liefers","year":"2014","venue":"Lecture Notes in Business Information Processing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518 7540:529-33, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin A Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Peters et al., 2013] Markus Peters, Wolfgang Ketter, May- tal Saar-Tsechansky, and John Collins. A reinforce- ment learning approach to autonomous decision-making in smart electricity markets. Machine Learning, 92, 2013.","title":"A reinforcement learning approach to autonomous decision-making in smart electricity markets","authors":"Markus Peters, Wolfgang Ketter, Maytal Saar-Tsechansky, John Collins","year":"2013","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Reddy and Veloso, 2011] Prashant P. Reddy and Manuela M. Veloso. Strategy learning for autonomous agents in smart grid markets. In Proceedings of the 20th IJCAI, pages 1446-1451, 2011.","title":"Strategy learning for autonomous agents in smart grid markets","authors":"P Prashant, Manuela M Reddy, Veloso","year":"2011","venue":"Proceedings of the 20th IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2011/244"},{"position":15,"raw":"[Reddy and Veloso, 2013] Prashant P. Reddy and Manuela M. Veloso. Negotiated learning for smart grid agents: entity selection based on dynamic partially observable features. In the Proceedings of 27th AAAI, pages 1313-1319, 2013.","title":"Negotiated learning for smart grid agents: entity selection based on dynamic partially observable features","authors":"P Prashant, Manuela M Reddy, Veloso","year":"2013","venue":"the Proceedings of 27th AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Robu et al., 2014] Valentin Robu, Meritxell Vinyals, Alex Rogers, and Nicholas R. Jennings. Efficient buyer groups for prediction-of-use electricity tariffs. In the Proceedings of the 28th AAAI, pages 451-457, 2014.","title":"Efficient buyer groups for prediction-of-use electricity tariffs","authors":"Valentin Robu, Meritxell Vinyals, Alex Rogers, Nicholas R Jennings","year":"2014","venue":"the Proceedings of the 28th AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sun et al., 2017] Mingyang Sun, Ioannis Konstantelos, and Goran Strbac. C-vine copula mixture model for clustering of residential electrical load pattern data. IEEE Transac- tions on Power Systems, 32(3):2382-2393, 2017.","title":"C-vine copula mixture model for clustering of residential electrical load pattern data","authors":"Mingyang Sun, Ioannis Konstantelos, Goran Strbac","year":"2017","venue":"IEEE Transactions on Power Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sutton and Barto, 2005] R. S Sutton and A. G Barto. Rein- forcement learning : an introduction. IEEE Transactions on Neural Networks, 16(1):285-286, 2005.","title":"Reinforcement learning : an introduction","authors":"R Sutton, A G Barto","year":"2005","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tumer and Agogino, 2007] Kagan Tumer and Adrian Agogino. Distributed agent-based air traffic flow man- agement. In Proceedings of the 6th AAMAS, page 255, 2007.","title":"Distributed agent-based air traffic flow management","authors":"Agogino Tumer, Kagan Tumer, Adrian Agogino","year":"2007","venue":"Proceedings of the 6th AAMAS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Urieli and Stone, 2014] Daniel Urieli and Peter Stone. Tac- tex'13: a champion adaptive power trading agent. In Pro- ceedings of the 13th AAMAS, pages 1447-1448, 2014.","title":"Daniel Urieli and Peter Stone. Tactex'13: a champion adaptive power trading agent","authors":"Stone Urieli","year":"2014","venue":"Proceedings of the 13th AAMAS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Urieli and Stone, 2016] Daniel Urieli and Peter Stone. An mdp-based winning approach to autonomous power trad- ing: Formalization and empirical analysis. In Proceedings of the 15th AAMAS, pages 827-835, 2016.","title":"Daniel Urieli and Peter Stone. An mdp-based winning approach to autonomous power trading: Formalization and empirical analysis","authors":"Stone Urieli","year":"2016","venue":"Proceedings of the 15th AAMAS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2016] Xishun Wang, Minjie Zhang, and Fenghui Ren. A hybrid-learning based broker model for strategic power trading in smart grid markets. Knowledge- Based Systems, 119:142-151, 2016.","title":"A hybrid-learning based broker model for strategic power trading in smart grid markets","authors":"Wang","year":"2016","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Watkins and Dayan, 1992] Christopher J. C. H. Watkins and Peter Dayan. Q-learning. In Machine Learning, pages 279-292, 1992.","title":"Q-learning","authors":"J C H Christopher, Peter Watkins, Dayan","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wen et al., 2015] Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei hao Su, David Vandyke, and Steve J. Young. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, page 1711-1721, 2015.","title":"Semantically conditioned lstm-based natural language generation for spoken dialogue systems","authors":"Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei Hao Su, David Vandyke, Steve J Young","year":"2015","venue":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zare et al., 2011] Kazem Zare, Mohsen Parsa Moghaddam, and Mohammad Kazem Sheikh-El-Eslami. Risk-based electricity procurement for large consumers. IEEE Trans- actions on Power Systems, 26(4):1826-1835, 2011.","title":"Risk-based electricity procurement for large consumers","authors":"Kazem Zare, Mohsen Parsa Moghaddam, Mohammad Kazem Sheikh-El-Eslami","year":"2011","venue":"IEEE Transactions on Power Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}