{"type":"ScholarlyArticle","title":"Competitive-Cooperative Multi-Agent Reinforcement Learning for Auction-based Federated Learning","authors":[{"name":"Xiaoli Tang","slug":"xiaoli-tang","given":"Xiaoli","family":"Tang","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"},{"name":"Han Yu","slug":"han-yu","given":"Han","family":"Yu","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"}],"abstract":"Auction-based Federated Learning (AFL) enables open collaboration among self-interested data consumers and data owners. Existing AFL approaches cannot manage the mutual influence among multiple data consumers competing to enlist data owners. Moreover, they cannot support a single data owner to join multiple data consumers simultaneously. To bridge these gaps, we propose the Multi-Agent Reinforcement Learning for AFL (MARL-AFL) approach to steer data consumers to bid strategically\r\ntowards an equilibrium with desirable overall system characteristics. We design a temperature-based reward reassignment scheme to make tradeoffs between cooperation and competition among AFL data consumers. In this way, it can reach an equilibrium state that ensures individual data consumers can achieve good utility, while preserving system-level social welfare. To circumvent potential collusion behaviors among data consumers, we introduce a bar agent to set a personalized bidding\r\nlower bound for each data consumer. Extensive experiments on six commonly adopted benchmark datasets show that MARL-AFL is significantly more advantageous compared to six state-of-the-art approaches, outperforming the best by 12.2%, 1.9% and 3.4% in terms of social welfare, revenue and accuracy, respectively.","doi":"10.24963/ijcai.2023/474","url":"https://ijcai.org/proceedings/2023/474","pdf":"https://ijcai.org/proceedings/2023/0474.pdf","markdown":"https://ijcai.org/proceedings/2023/474.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/474","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":474,"pages":{"first":4262,"last":4270},"keywords":["Machine Learning: ML: Federated learning","Agent-based and Multi-agent Systems: MAS: Multi-agent learning","AI Ethics, Trust, Fairness: ETF: Trustworthy AI","Machine Learning: ML: Reinforcement learning"],"videos":[],"references":[{"position":1,"raw":"[Bahreini et al., 2018] Tayebeh Bahreini, Hossein Badri, and Daniel Grosu. An envy-free auction mechanism for re- source allocation in edge computing systems. In SEC, pages 313-322, 2018.","title":"An envy-free auction mechanism for resource allocation in edge computing systems","authors":"Bahreini","year":"2018","venue":"SEC","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2017] Han Cai, Kan Ren, Weinan Zhang, Klean- this Malialis, Jun Wang, Yong Yu, and Defeng Guo. Real- time bidding by reinforcement learning in display adver- tising. In WSDM, pages 661-670, 2017.","title":"Realtime bidding by reinforcement learning in display advertising","authors":"Han Cai, Kan Ren, Weinan Zhang, Kleanthis Malialis, Jun Wang, Yong Yu, Defeng Guo","year":"2017","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] Yiqiang Chen, Xiaodong Yang, Xin Qin, Han Yu, Piu Chan, and Zhiqi Shen. Dealing with la- bel quality disparity in federated learning. In Federated Learning: Privacy and Incentive, pages 108-121. 2020.","title":"Dealing with label quality disparity in federated learning","authors":"Yiqiang Chen, Xiaodong Yang, Xin Qin, Han Yu, Piu Chan, Zhiqi Shen","year":"2020","venue":"Federated Learning: Privacy and Incentive","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Clanuwat et al., 2018] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint, page 1812.01718, 2018.","title":"Deep learning for classical japanese literature","authors":"Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, David Ha","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cohen et al., 2017] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. EMNIST: Extending M- NIST to handwritten letters. In IJCNN, pages 2921-2926, 2017.","title":"EMNIST: Extending M-NIST to handwritten letters","authors":"Gregory Cohen, Saeed Afshar, Jonathan Tapson, Andre Van Schaik","year":"2017","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2021] Yongheng Deng, Feng Lyu, Ju Ren, Yi- Chao Chen, Peng Yang, Yuezhi Zhou, and Yaoxue Zhang. Fair: Quality-aware federated learning with precise user incentive and model aggregation. In INFOCOM, 2021.","title":"Fair: Quality-aware federated learning with precise user incentive and model aggregation","authors":"Yongheng Deng, Feng Lyu, Ju Ren, Yi-Chao Chen, Peng Yang, Yuezhi Zhou, Yaoxue Zhang","year":"2021","venue":"INFOCOM","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2019] Guoju Gao, Mingjun Xiao, Jie Wu, He Huang, Shengqi Wang, and Guoliang Chen. Auction- based vm allocation for deadline-sensitive tasks in dis- tributed edge cloud. IEEE Transactions on Services Com- puting, 2019.","title":"Auctionbased vm allocation for deadline-sensitive tasks in distributed edge cloud","authors":"Gao","year":"2019","venue":"IEEE Transactions on Services Computing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hong et al., 2020] Hsiang-Jen Hong, Wenjun Fan, C Ed- ward Chow, Xiaobo Zhou, and Sang-Yoon Chang. Op- timizing social welfare for task offloading in mobile edge computing. In Networking, pages 524-528, 2020.","title":"Optimizing social welfare for task offloading in mobile edge computing","authors":"Hsiang-Jen Hong, Wenjun Fan, Edward Chow, Xiaobo Zhou, Sang-Yoon Chang","year":"2020","venue":"Networking","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiao et al., 2018] Yutao Jiao, Ping Wang, Dusit Niyato, and Zehui Xiong. Social welfare maximization auction in edge computing resource allocation for mobile blockchain. In ICC, pages 1-6. IEEE, 2018.","title":"Social welfare maximization auction in edge computing resource allocation for mobile blockchain","authors":"Jiao","year":"2018","venue":"ICC","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiao et al., 2019] Yutao Jiao, Ping Wang, Dusit Niyato, and Kongrath Suankaewmanee. Auction mechanism- s in cloud/fog computing resource allocation for public blockchain networks. IEEE Transactions on Parallel and Distributed Systems, 30(9):1975-1989, 2019.","title":"Auction mechanisms in cloud/fog computing resource allocation for public blockchain networks","authors":"Jiao","year":"2019","venue":"IEEE Transactions on Parallel and Distributed Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jiao et al., 2020] Yutao Jiao, Ping Wang, Dusit Niyato, Bin Lin, and Dong In Kim. Toward an automated auction framework for wireless federated learning services market. IEEE Transactions on Mobile Computing, 20(10):3034- 3048, 2020.","title":"Toward an automated auction framework for wireless federated learning services market","authors":"Jiao","year":"2020","venue":"IEEE Transactions on Mobile Computing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Khan et al., 2020] Latif U. Khan, Shashi Raj Pandey, N- guyen H. Tran, Walid Saad, Zhu Han, Minh N. H. Nguyen, and Choong Seon Hong. Federated learning for edge net- works: Resource optimization and incentive mechanism. IEEE Communications Magazine, 58(10):88-93, 2020.","title":"Federated learning for edge networks: Resource optimization and incentive mechanism","authors":"U Latif, Khan, Raj Shashi, Pandey, H Nguyen, Walid Tran, Zhu Saad, Minh N H Han, Choong Nguyen, Hong Seon","year":"2020","venue":"IEEE Communications Magazine","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krishnaraj et al., 2022] Nagappan Krishnaraj, Kiranmai Bellam, B Sivakumar, and Arockiam Daniel. The fu- ture of cloud computing: Blockchain-based decentralized cloud/fog solutions-challenges, opportunities, and stan- dards. Blockchain Security in Cloud Computing, pages 207-226, 2022.","title":"The future of cloud computing: Blockchain-based decentralized cloud/fog solutions-challenges, opportunities, and standards","authors":"Nagappan Krishnaraj, Kiranmai Bellam, B Sivakumar, Arockiam Daniel","year":"2022","venue":"Blockchain Security in Cloud Computing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Le et al., 2020] Tra Huong Thi Le, Nguyen H. Tran, Yan Kyaw Tun, Zhu Han, and Choong Seon Hong. Auc- tion based incentive design for efficient federated learning in cellular wireless networks. In WCNC, pages 1-6, 2020.","title":"Auction based incentive design for efficient federated learning in cellular wireless networks","authors":"Tra Huong, Thi Le, H Nguyen, Yan Tran, Zhu Kyaw Tun, Choong Han, Hong Seon","year":"2020","venue":"WCNC","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2012] Kuang-chih Lee, Burkay Orten, Ali Das- dan, and Wentong Li. Estimating conversion rate in dis- play advertising from past erformance data. In KDD, pages 768-776, 2012.","title":"Estimating conversion rate in display advertising from past erformance data","authors":"Kuang-Chih Lee, Burkay Orten, Ali Dasdan, Wentong Li","year":"2012","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Littman, 1994] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In ICML, pages 157-163, 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"Michael L Littman","year":"1994","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2020] Yang Liu, Anbu Huang, Yun Luo, He Huang, Youzhi Liu, Yuanyuan Chen, Lican Feng, Tian- jian Chen, Han Yu, and Qiang Yang. FedVision: An on- line visual object detection platform powered by federated learning. In IAAI, pages 13172-13179, 2020.","title":"FedVision: An online visual object detection platform powered by federated learning","authors":"Yang Liu, Anbu Huang, Yun Luo, He Huang, Youzhi Liu, Yuanyuan Chen, Lican Feng, Tianjian Chen, Han Yu, Qiang Yang","year":"2020","venue":"IAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2022] Zelei Liu, Yuanyuan Chen, Yansong Zhao, Han Yu, Yang Liu, Renyi Bao, Jinpeng Jiang, Zaiqing Nie, Qian Xu, and Qiang Yang. Contribution-aware federat- ed learning for smart healthcare. In IAAI, pages 12396- 12404, 2022.","title":"Contribution-aware federated learning for smart healthcare","authors":"Zelei Liu, Yuanyuan Chen, Yansong Zhao, Han Yu, Yang Liu, Renyi Bao, Jinpeng Jiang, Zaiqing Nie, Qian Xu, Qiang Yang","year":"2022","venue":"IAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lyu et al., 2022] Lingjuan Lyu, Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang, and Philip S. Yu. Privacy and robustness in federated learning: Attacks and defense. IEEE Transactions on Neural Networks and Learning Systems, 2022.","title":"Privacy and robustness in federated learning: Attacks and defense","authors":"Lyu","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Perlich et al., 2012] Claudia Perlich, Brian Dalessandro, Rod Hook, Ori Stitelman, Troy Raeder, and Foster Provost. Bid optimizing and inventory scoring in target- ed online advertising. In KDD, pages 804-812, 2012.","title":"Bid optimizing and inventory scoring in targeted online advertising","authors":"Claudia Perlich, Brian Dalessandro, Rod Hook, Ori Stitelman, Troy Raeder, Foster Provost","year":"2012","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ren et al., 2017] Kan Ren, Weinan Zhang, Ke Chang, Yifei Rong, Yong Yu, and Jun Wang. Bidding machine: Learn- ing to bid for directly optimizing profits in display adver- tising. IEEE Transactions on Knowledge and Data Engi- neering, 30(4):645-659, 2017.","title":"Bidding machine: Learning to bid for directly optimizing profits in display advertising","authors":"Weinan Kan Ren, Ke Zhang, Yifei Chang, Yong Rong, Jun Yu, Wang","year":"2017","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Roy et al., 2021] Palash Roy, Sujan Sarker, Md Abdur Raz- zaque, Md Mamun-or Rashid, Mohmmad Mehedi Has- san, and Giancarlo Fortino. Distributed task alloca- tion in mobile device cloud exploiting federated learn- ing and subjective logic. Journal of Systems Architecture, 113(2):doi:10.1016/j.sysarc.2020.101972, 2021.","title":"Distributed task allocation in mobile device cloud exploiting federated learning and subjective logic","authors":"Palash Roy, Sujan Sarker, Md Abdur Razzaque, Md Mamun-Or Rashid, Mohmmad Mehedi Hassan, Giancarlo Fortino","year":"2021","venue":"Journal of Systems Architecture","doi":"10.1016/j.sysarc.2020.101972","url":null,"internal_url":null},{"position":23,"raw":"[Shi et al., 2023] Yuxin Shi, Han Yu, and Cyril Leung. To- wards fairness-aware federated learning. IEEE Transac- tions on Neural Networks and Learning Systems, 2023.","title":"Towards fairness-aware federated learning","authors":"Yuxin Shi, Han Yu, Cyril Leung","year":"2023","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition","authors":"Zisserman Simonyan","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tan et al., 2022] Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learn- ing. IEEE Transactions on Neural Networks and Learning Systems, 2022.","title":"Towards personalized federated learning","authors":"Alysa Ziying Tan, Han Yu, Lizhen Cui, Qiang Yang","year":"2022","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Thi Le et al., 2021] Tra Huong Thi Le, Nguyen H. Tran, Yan Kyaw Tun, Minh N. H. Nguyen, Shashi Raj Pandey, Zhu Han, and Choong Seon Hong. An incentive mechanis- m for federated learning in wireless cellular networks: An auction approach. IEEE Transactions on Wireless Com- munications, 20(8):4874-4887, 2021.","title":"An incentive mechanism for federated learning in wireless cellular networks: An auction approach","authors":"Tra Huong, Thi Le, H Nguyen, Yan Tran, Minh N H Kyaw Tun, Nguyen, Raj Shashi, Zhu Pandey, Choong Han, Hong Seon","year":"2021","venue":"IEEE Transactions on Wireless Communications","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint, page 1708.07747, 2017.","title":"Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2019] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology, 10(2):12:1-12:19, 2019.","title":"Federated machine learning: Concept and applications","authors":"Qiang Yang, Yang Liu, Tianjian Chen, Yongxin Tong","year":"2019","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang, 2020] Shi Yang. A task offloading solution for inter- net of vehicles using combination auction matching model based on mobile edge computing. IEEE Access, 8:53261- 53273, 2020.","title":"A task offloading solution for internet of vehicles using combination auction matching model based on mobile edge computing","authors":"Yang Shi","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ying et al., 2020] Chenhao Ying, Haiming Jin, Xudong Wang, and Yuan Luo. Double insurance: Incentivized fed- erated learning with differential privacy in mobile crowd- sensing. In SRDS, pages 81-90, 2020.","title":"Double insurance: Incentivized federated learning with differential privacy in mobile crowdsensing","authors":"Chenhao Ying, Haiming Jin, Xudong Wang, Yuan Luo","year":"2020","venue":"SRDS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2017] Han Yu, Chunyan Miao, Yiqiang Chen, Si- mon Fauvel, Xiaoming Li, and Victor R. Lesser. Algorith- mic management for improving collective productivity in crowdsourcing. Scientific Reports, 7(12541), 2017.","title":"Algorithmic management for improving collective productivity in crowdsourcing","authors":"Han Yu, Chunyan Miao, Yiqiang Chen, Simon Fauvel, Xiaoming Li, Victor R Lesser","year":"2017","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zavodovski et al., 2019] Aleksandr Zavodovski, Suzan Bayhan, Nitinder Mohan, Pengyuan Zhou, Walter Wong, and Jussi Kangasharju. Decloud: Truthful decentral- ized double auction for edge clouds. In ICDCS, pages 2157-2167. IEEE, 2019.","title":"Decloud: Truthful decentralized double auction for edge clouds","authors":"Aleksandr Zavodovski, Suzan Bayhan, Nitinder Mohan, Pengyuan Zhou, Walter Wong, Jussi Kangasharju","year":"2019","venue":"ICDCS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zeng et al., 2020] Rongfei Zeng, Shixun Zhang, Jiaqi Wang, and Xiaowen Chu. Fmore: An incentive scheme of multi-dimensional auction for federated learning in MEC. In ICDCS, pages 278-288, 2020.","title":"Fmore: An incentive scheme of multi-dimensional auction for federated learning in MEC","authors":"Zeng","year":"2020","venue":"ICDCS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhan et al., 2020] Yufeng Zhan, Peng Li, Zhihao Qu, Deze Zeng, and Song Guo. A learning-based incentive mecha- nism for federated learning. IEEE Internet of Things Jour- nal, 7(7):6360-6368, 2020.","title":"A learning-based incentive mechanism for federated learning","authors":"Yufeng Zhan, Peng Li, Zhihao Qu, Deze Zeng, Song Guo","year":"2020","venue":"IEEE Internet of Things Journal","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhan et al., 2021] Yufeng Zhan, Jie Zhang, Zicong Hong, Leijie Wu, Peng Li, and Song Guo. A survey of incentive mechanism design for federated learning. IEEE Trans- actions on Emerging Topics in Computing, 10(2):1035- 1044, 2021.","title":"A survey of incentive mechanism design for federated learning","authors":"Yufeng Zhan, Jie Zhang, Zicong Hong, Leijie Wu, Peng Li, Song Guo","year":"2021","venue":"IEEE Transactions on Emerging Topics in Computing","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2014] Weinan Zhang, Shuai Yuan, and Jun Wang. Optimal real-time bidding for display advertising. In KDD, pages 1077-1086, 2014.","title":"Optimal real-time bidding for display advertising","authors":"Weinan Zhang, Shuai Yuan, Jun Wang","year":"2014","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang et al., 2021] Jingwen Zhang, Yuezhou Wu, and Rong Pan. Incentive mechanism for horizontal federated learn- ing based on reputation and reverse auction. In WWW, page 947-956, 2021.","title":"Incentive mechanism for horizontal federated learning based on reputation and reverse auction","authors":"Jingwen Zhang, Yuezhou Wu, Rong Pan","year":"2021","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2022a] Jingwen Zhang, Yuezhou Wu, and Rong Pan. Auction-based ex-post-payment incentive mechanism design for horizontal federated learning with reputation and contribution measurement. arXiv preprint, page 2201.02410, 2022.","title":"Auction-based ex-post-payment incentive mechanism design for horizontal federated learning with reputation and contribution measurement","authors":"Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhang et al., 2022b] Jingwen Zhang, Yuezhou Wu, and Rong Pan. Online auction-based incentive mechanism de- sign for horizontal federated learning with budget con- straint. arXiv preprint, page 2201.09047, 2022.","title":"Online auction-based incentive mechanism design for horizontal federated learning with budget constraint","authors":"Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zou et al., 2020] Liang Zou, Xinhui Yu, Ming Li, Meng Lei, and Han Yu. Nondestructive identification of coal and gangue via near-infrared spectroscopy based on improved broad learning. IEEE Transactions on Instrumentation and Measurement, 69(10):8043-8052, 2020.","title":"Nondestructive identification of coal and gangue via near-infrared spectroscopy based on improved broad learning","authors":"Liang Zou, Xinhui Yu, Ming Li, Meng Lei, Han Yu","year":"2020","venue":"IEEE Transactions on Instrumentation and Measurement","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}