{"type":"ScholarlyArticle","title":"A Deep Reinforcement Learning Approach to Concurrent Bilateral Negotiation","authors":[{"name":"Pallavi Bagga","slug":"pallavi-bagga","given":"Pallavi","family":"Bagga","affiliation":"Royal Holloway , University of London , UK"},{"name":"Nicola Paoletti","slug":"nicola-paoletti","given":"Nicola","family":"Paoletti","affiliation":"Royal Holloway , University of London , UK"},{"name":"Bedour Alrayes","slug":"bedour-alrayes","given":"Bedour","family":"Alrayes","affiliation":"King Saud University , Saudi Arabia"},{"name":"Kostas Stathis","slug":"kostas-stathis","given":"Kostas","family":"Stathis","affiliation":"Royal Holloway , University of London , UK"}],"abstract":"We present a novel negotiation model that allows an agent to learn how to negotiate during concurrent bilateral negotiations in unknown and dynamic e-markets. The agent uses an actor-critic architecture with model-free reinforcement learning to learn a strategy expressed as a deep neural network. We pre-train the strategy by supervision from synthetic market data, thereby decreasing the exploration time required for learning during negotiation. As a result, we can build automated agents for concurrent negotiations that can adapt to different e-market settings without the need to be pre-programmed. Our experimental evaluation shows that our deep reinforcement learning based agents outperform two existing well-known negotiation strategies in one-to-many concurrent bilateral negotiations for a range of e-market settings.","doi":"10.24963/ijcai.2020/42","url":"https://ijcai.org/proceedings/2020/42","pdf":"https://ijcai.org/proceedings/2020/0042.pdf","markdown":"https://ijcai.org/proceedings/2020/42.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/42","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Agent-based and Multi-agent Systems","track":"Main track","number":42,"pages":{"first":297,"last":303},"keywords":["Agent-based and Multi-agent Systems: Agreement Technologies: Negotiation and Contract-Based Systems","Machine Learning Applications: Applications of Supervised Learning","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24879/file_24879.m3u8","length_sec":286,"provider":"hls"}],"references":[{"position":1,"raw":"[Alrayes and Stathis, 2013] Bedour Alrayes and Kostas Stathis. An agent architecture for concurrent bilateral negotiations. In Decision Support Systems III-Impact of Decision Support Systems for Global Environments, pages 79-89. Springer, 2013.","title":"Bedour Alrayes and Kostas Stathis. An agent architecture for concurrent bilateral negotiations","authors":"Stathis Alrayes","year":"2013","venue":"Decision Support Systems III-Impact of Decision Support Systems for Global Environments","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alrayes et al., 2016] Bedour Alrayes, Özgür Kafalı, and Kostas Stathis. RECON: a robust multi-agent environ- ment for simulating concurrent negotiations. In Recent advances in agent-based complex automated negotiation, pages 157-174. Springer, 2016.","title":"RECON: a robust multi-agent environment for simulating concurrent negotiations","authors":"Alrayes","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alrayes et al., 2018] Bedour Alrayes, Özgür Kafalı, and Kostas Stathis. Concurrent bilateral negotiation for open e-markets: the CONAN strategy. Knowledge and Informa- tion Systems, 56(2):463-501, 2018.","title":"Concurrent bilateral negotiation for open e-markets: the CONAN strategy","authors":"Alrayes","year":"2018","venue":"Knowledge and Information Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[An et al., 2006] Bo An, Kwang Mong Sim, Liang Gui Tang, Shuang Qing Li, and Dai Jie Cheng. Continuous-time ne- gotiation mechanism for software agents. IEEE Transac- tions on Systems, Man, and Cybernetics, Part B (Cyber- netics), 36(6):1261-1272, 2006.","title":"Continuous-time negotiation mechanism for software agents","authors":"Bo An, Kwang Mong Sim, Gui Liang, Shuang Qing Tang, Dai Jie Li, Cheng","year":"2006","venue":"IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bagga et al., 2020] Pallavi Bagga, Nicola Paoletti, Bedour Alrayes, and Kostas Stathis. A deep reinforcement learn- ing approach to concurrent bilateral negotiation. arXiv preprint arXiv:2001.11785, 2020.","title":"A deep reinforcement learning approach to concurrent bilateral negotiation","authors":"Pallavi Bagga, Nicola Paoletti, Bedour Alrayes, Kostas Stathis","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bakker et al., 2019] Jasper Bakker, Aron Hammond, Daan Bloembergen, and Tim Baarslag. RLBOA: A modular re- inforcement learning framework for autonomous negotiat- ing agents. In Proceedings of the 18th International Con- ference on Autonomous Agents and MultiAgent Systems, pages 260-268. International Foundation for Autonomous Agents and Multiagent Systems, 2019.","title":"RLBOA: A modular reinforcement learning framework for autonomous negotiating agents","authors":"Bakker","year":"2019","venue":"Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Faratin et al., 1998] Peyman Faratin, Carles Sierra, and Nick R Jennings. Negotiation decision functions for au- tonomous agents. Robotics and Autonomous Systems, 24(3-4):159-182, 1998.","title":"Negotiation decision functions for autonomous agents","authors":"Peyman Faratin, Carles Sierra, Nick R Jennings","year":"1998","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.","title":"Deep learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lee Rodgers and Nicewander, 1988] Joseph Lee Rodgers and W Alan Nicewander. Thirteen ways to look at the cor- relation coefficient. The American Statistician, 42(1):59- 66, 1988.","title":"Thirteen ways to look at the correlation coefficient","authors":"Joseph, Lee Rodgers, Alan Nicewander","year":"1988","venue":"The American Statistician","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lewis et al., 2017] Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? end-to-end learning for negotiation dialogues. arXiv preprint arXiv:1706.05125, 2017.","title":"Deal or no deal? end-to-end learning for negotiation dialogues","authors":"Mike Lewis, Denis Yarats, Devi Yann N Dauphin, Dhruv Parikh, Batra","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lillicrap et al., 2016] Timothy Paul Lillicrap, Jonathan James Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proceedings of the 4th International Conference on Learning Representations (ICLR 2016), 2016.","title":"Continuous control with deep reinforcement learning","authors":"Timothy Paul Lillicrap, Jonathan James Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2016","venue":"Proceedings of the 4th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mansour and Kowalczyk, 2014] Khalid Mansour and Ryszard Kowalczyk. Coordinating the bidding strategy in multiissue multiobject negotiation with single and multiple providers. IEEE transactions on cybernetics, 45(10):2261-2272, 2014.","title":"Coordinating the bidding strategy in multiissue multiobject negotiation with single and multiple providers","authors":"Khalid Mansour, Ryszard Kowalczyk","year":"2014","venue":"IEEE transactions on cybernetics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nguyen and Jennings, 2004] Thuc Duong Nguyen and Nicholas R Jennings. Coordinating multiple concurrent negotiations. In Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems-Volume 3, pages 1064-1071. IEEE Computer Society, 2004.","title":"Coordinating multiple concurrent negotiations","authors":"Thuc Duong, Nguyen, Nicholas R Jennings","year":"2004","venue":"Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Oliver, 1996] Jim R Oliver. A machine-learning approach to automated negotiation and prospects for electronic com- merce. Journal of management information systems, 13(3):83-112, 1996.","title":"A machine-learning approach to automated negotiation and prospects for electronic commerce","authors":"Jim R Oliver","year":"1996","venue":"Journal of management information systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Papangelis and Georgila, 2015] Alexandros Papangelis and Kallirroi Georgila. Reinforcement learning of multi-issue negotiation dialogue policies. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Dis- course and Dialogue, pages 154-158, 2015.","title":"Alexandros Papangelis and Kallirroi Georgila. Reinforcement learning of multi-issue negotiation dialogue policies","authors":"Georgila Papangelis","year":"2015","venue":"Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rahwan et al., 2002] Iyad Rahwan, Ryszard Kowalczyk, and Ha Hai Pham. Intelligent agents for automated one- to-many e-commerce negotiation. In Australian Com- puter Science Communications, volume 24, pages 197- 204. Australian Computer Society, Inc., 2002.","title":"Intelligent agents for automated oneto-many e-commerce negotiation","authors":"Rahwan","year":"2002","venue":"Australian Computer Science Communications","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rodriguez-Fernandez et al., 2019] J Rodriguez-Fernandez, T Pinto, F Silva, I Prac ¸a, Z Vale, and JM Corchado. Con- text aware Q-learning-based model for decision support in the negotiation of energy contracts. International Jour- nal of Electrical Power & Energy Systems, 104:489-501, 2019.","title":"Context aware Q-learning-based model for decision support in the negotiation of energy contracts","authors":"Rodriguez-Fernandez","year":"2019","venue":"International Journal of Electrical Power & Energy Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rubinstein, 1982] Ariel Rubinstein. Perfect equilibrium in a bargaining model. Econometrica: Journal of the Econo- metric Society, pages 97-109, 1982.","title":"Ariel Rubinstein. Perfect equilibrium in a bargaining model","authors":"Rubinstein","year":"1982","venue":"Econometrica: Journal of the Econometric Society","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Williams et al., 2012] Colin R Williams, Valentin Robu, Enrico H Gerding, and Nicholas R Jennings. Negotiating concurrently with unknown opponents in complex, real- time domains. In Proceedings of the 20th European Con- ference on Artificial Intelligence, pages 834-839, 2012.","title":"Negotiating concurrently with unknown opponents in complex, realtime domains","authors":"Colin R Williams, Valentin Robu, Enrico H Gerding, Nicholas R Jennings","year":"2012","venue":"Proceedings of the 20th European Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zou et al., 2014] Yi Zou, Wenjie Zhan, and Yuan Shao. Evolution with reinforcement learning in negotiation. PLOS one, 9(7):e102840, 2014.","title":"Evolution with reinforcement learning in negotiation","authors":"Yi Zou, Wenjie Zhan, Yuan Shao","year":"2014","venue":"PLOS one","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}