{"type":"ScholarlyArticle","title":"Reinforcement mechanism design","authors":[{"name":"Pingzhong Tang","slug":"pingzhong-tang","given":"Pingzhong","family":"Tang","affiliation":"IIIS , Tsinghua University , China"}],"abstract":"We put forward a modeling and algorithmic framework to design and optimize mechanisms in dynamic industrial environments where a designer can make use of the data generated in the process to automatically improve future design. Our solution, coined reinforcement mechanism design, is rooted in game theory but incorporates recent AI techniques to get rid of nonrealistic modeling assumptions and to make automated optimization feasible. We instantiate our framework on the key application scenarios of Baidu and Taobao, two of the largest mobile app companies in China. For the Taobao case, our framework automatically designs mechanisms that allocate buyer impressions for the e-commerce website; for the Baidu case, our framework automatically designs dynamic reserve pricing schemes of advertisement auctions of the search engine. Experiments show that our solutions outperform the state-of-the-art alternatives and those currently deployed, under both scenarios.","doi":"10.24963/ijcai.2017/739","url":"https://ijcai.org/proceedings/2017/739","pdf":"https://ijcai.org/proceedings/2017/0739.pdf","markdown":"https://ijcai.org/proceedings/2017/739.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/739","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Early Career","track":"Early Career","number":739,"pages":{"first":5146,"last":5150},"keywords":["Agent-based and Multi-agent Systems: Economic paradigms, auctions and market-based systems","Machine Learning: Reinforcement Learning","Agent-based and Multi-agent Systems: Engineering methods, platforms, languages and tools"],"videos":[],"references":[{"position":1,"raw":"[Cai et al., 2012] Yang Cai, Constantinos Daskalakis, and S. Matthew Weinberg. Optimal multi-dimensional mecha- nism design: Reducing revenue to welfare maximization. In FOCS, pages 130-139, 2012.","title":"Optimal multi-dimensional mechanism design: Reducing revenue to welfare maximization","authors":"Yang Cai, Constantinos Daskalakis, S Matthew Weinberg","year":"2012","venue":"FOCS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2016] Qingpeng Cai, Aris Filos-Ratsikas, Chang Liu, and Pingzhong Tang. Mechanism design for personalized rec- ommender systems. In Proceedings of the 10th ACM Conference on Recommender Systems, Boston, MA, USA, September 15-19, 2016, pages 159-166, 2016.","title":"Mechanism design for personalized recommender systems","authors":"Cai","year":"2016","venue":"Proceedings of the 10th ACM Conference on Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cai et al., 2017] Qingpeng Cai, Aris Filos-Ratsikas, Pingzhong Tang, and Yiwei Zhang. A deep reinforcement learning frame- work for allocating buyer impressions in e-commerce websites. Working paper, 2017.","title":"A deep reinforcement learning framework for allocating buyer impressions in e-commerce websites","authors":"Cai","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Edelman et al., 2007] Benjamin Edelman, Michael Ostrovsky, and Michael Schwarz. Internet advertising and the generalized sec- ond price auction: Selling billions of dollars worth of keywords. American economic review, 97(1), 2007.","title":"Internet advertising and the generalized second price auction: Selling billions of dollars worth of keywords","authors":"Benjamin Edelman, Michael Ostrovsky, Michael Schwarz","year":"2007","venue":"American economic review","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hart and Nisan, 2012] Sergiu Hart and Noam Nisan. Approximate revenue maximization with multiple items. In ACM Conference on Electronic Commerce, page 656, 2012.","title":"Approximate revenue maximization with multiple items","authors":"Sergiu Hart, Noam Nisan","year":"2012","venue":"ACM Conference on Electronic Commerce","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2013] Di He, Wei Chen, Liwei Wang, and Tie-Yan Liu. A game-theoretic machine learning approach for revenue max- imization in sponsored search. In Twenty-Third International Joint Conference on Artificial Intelligence, 2013.","title":"A game-theoretic machine learning approach for revenue maximization in sponsored search","authors":"Di He, Wei Chen, Liwei Wang, Tie-Yan Liu","year":"2013","venue":"Twenty-Third International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2013/40"},{"position":7,"raw":"[Lahaie and Pennock, 2007] Sébastien Lahaie and David M. Pen- nock. Revenue analysis of a family of ranking rules for key- word auctions. In Proceedings 8th ACM Conference on Elec- tronic Commerce (EC-2007), San Diego, California, USA, June 11-15, 2007, pages 50-56, 2007.","title":"Revenue analysis of a family of ranking rules for keyword auctions","authors":"Sébastien Lahaie, David M Pennock","year":"2007","venue":"Proceedings 8th ACM Conference on Electronic Commerce (EC-2007)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Leibo et al., 2017] Joel Z. Leibo, Vinícius Flores Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent rein- forcement learning in sequential social dilemmas. In Proceed- ings of the 16th Conference on Autonomous Agents and MultiA- gent Systems, AAMAS 2017, São Paulo, Brazil, May 8-12, 2017, pages 464-473, 2017.","title":"Multi-agent reinforcement learning in sequential social dilemmas","authors":"Joel Z Leibo, Vinícius Flores Zambaldi, Marc Lanctot, Janusz Marecki, Thore Graepel","year":"2017","venue":"Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2017","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep rein- forcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mas-Colell et al., 1995] Andreu Mas-Colell, Michael D. Whin- ston, and Jerry R. Green. Microeconomic Theory. Oxford Uni- versity Press, June 1995.","title":"Microeconomic Theory","authors":"Andreu Mas-Colell, Michael D Whinston, Jerry R Green","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Milgrom, 2004] Paul Milgrom. Putting Auction Theory to Work. Cambridge University Press, 2004.","title":"Putting Auction Theory to Work","authors":"Paul Milgrom","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mirrokni et al., 2016] Vahab S. Mirrokni, Renato Paes Leme, Pingzhong Tang, and Song Zuo. Dynamic auctions with bank accounts. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 387-393, 2016.","title":"Dynamic auctions with bank accounts","authors":"S Vahab, Renato Mirrokni, Pingzhong Paes Leme, Song Tang, Zuo","year":"2016","venue":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016","doi":null,"url":null,"internal_url":"/proceedings/2016/62"},{"position":13,"raw":"[Mohri and Medina, 2015] Mehryar Mohri and Andres Muñoz Medina. Non-parametric revenue optimization for generalized second price auctions. In Proceedings of the Thirty-First Confer- ence on Uncertainty in Artificial Intelligence, UAI 2015, July 12- 16, 2015, Amsterdam, The Netherlands, pages 612-621, 2015.","title":"Non-parametric revenue optimization for generalized second price auctions","authors":"Mehryar Mohri, Andres Muñoz Medina","year":"2015","venue":"Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, UAI 2015","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mohri and Medina, 2016] Mehryar Mohri and Andrés Munoz Medina. Learning algorithms for second-price auctions with reserve. Journal of Machine Learning Research, 17(74):1-25, 2016.","title":"Learning algorithms for second-price auctions with reserve","authors":"Medina Mohri, Mehryar Mohri, Andrés Munoz Medina","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mohri and Munoz, 2014] Mehryar Mohri and Andres Munoz. Op- timal regret minimization in posted-price auctions with strategic buyers. In Advances in Neural Information Processing Systems, pages 1871-1879, 2014.","title":"Optimal regret minimization in posted-price auctions with strategic buyers","authors":"Munoz Mohri, Mehryar Mohri, Andres Munoz","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mohri and Munoz, 2015] Mehryar Mohri and Andres Munoz. Revenue optimization against strategic buyers. In Advances in Neural Information Processing Systems, pages 2530-2538, 2015.","title":"Revenue optimization against strategic buyers","authors":"Munoz Mohri, Mehryar Mohri, Andres Munoz","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Myerson, 1981] Roger B. Myerson. Optimal auction design. Mathematics of Operations Research, 6(1):58-73, 1981.","title":"Optimal auction design","authors":"B Roger, Myerson","year":"1981","venue":"Mathematics of Operations Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nekipelov et al., 2015] Denis Nekipelov, Vasilis Syrgkanis, and Éva Tardos. Econometrics for learning agents. In Proceedings of the Sixteenth ACM Conference on Economics and Computa- tion, EC '15, Portland, OR, USA, June 15-19, 2015, pages 1-18, 2015.","title":"Econometrics for learning agents","authors":"Denis Nekipelov, Vasilis Syrgkanis, Éva Tardos","year":"2015","venue":"Proceedings of the Sixteenth ACM Conference on Economics and Computation, EC '15","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ortega and Stocker, 2016] Pedro A Ortega and Alan A Stocker. Human decision-making under limited time. In Advances in Neu- ral Information Processing Systems, pages 100-108, 2016.","title":"Human decision-making under limited time","authors":"Pedro A Ortega, Alan A Stocker","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ostrovsky and Schwarz, 2011] Michael Ostrovsky and Michael Schwarz. Reserve prices in internet advertising auctions: a field experiment. In Proceedings 12th ACM Conference on Electronic Commerce (EC-2011), San Jose, CA, USA, June 5-9, 2011, pages 59-60, 2011.","title":"Reserve prices in internet advertising auctions: a field experiment","authors":"Schwarz Ostrovsky, Michael Ostrovsky, Michael Schwarz","year":"2011","venue":"Proceedings 12th ACM Conference on Electronic Commerce (EC-2011)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shen and Tang, 2017] Weiran Shen and Pingzhong Tang. Practical versus optimal mechanisms. In Proceedings of the 16th Confer- ence on Autonomous Agents and MultiAgent Systems, pages 78- 86. International Foundation for Autonomous Agents and Multi- agent Systems, 2017.","title":"Practical versus optimal mechanisms","authors":"Tang Shen, Weiran Shen, Pingzhong Tang","year":"2017","venue":"Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shen et al., 2017] Weiran Shen, Binghui Peng, Hanpeng Liu, Michael Zhang, Ruohan Qian, Yan Hong, Zhi Guo, Zongyao Ding, Pengjun Lu, and Pingzhong Tang. Reinforcement mech- anism design, with applications to dynamic pricing in sponsored search auctions. Working paper, 2017.","title":"Reinforcement mechanism design, with applications to dynamic pricing in sponsored search auctions","authors":"Weiran Shen, Binghui Peng, Hanpeng Liu, Michael Zhang, Ruohan Qian, Yan Hong, Zhi Guo, Zongyao Ding, Pengjun Lu, Pingzhong Tang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shoham and Leyton-Brown, 2009] Yoav Shoham and Kevin Leyton-Brown. Multiagent Systems: Algorithmic, Game theoretic and Logical Fundations. Cambridge Uni. Press, 2009.","title":"Multiagent Systems: Algorithmic, Game theoretic and Logical Fundations","authors":"Yoav Shoham, Kevin Leyton-Brown","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang and Sandholm, 2012] Pingzhong Tang and Tuomas Sand- holm. Mixed bundling auctions with reserve prices. In AAMAS, 2012.","title":"Pingzhong Tang and Tuomas Sandholm. Mixed bundling auctions with reserve prices","authors":"Sandholm Tang","year":"2012","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tang and Wang, 2016] Pingzhong Tang and Zihe Wang. Optimal auctions for negatively correlated items. In Proceedings of the 2016 ACM Conference on Economics and Computation, EC '16, Maastricht, The Netherlands, July 24-28, 2016, pages 103-120, 2016.","title":"Optimal auctions for negatively correlated items","authors":"Wang ; Tang, Pingzhong Tang, Zihe Wang","year":"2016","venue":"Proceedings of the 2016 ACM Conference on Economics and Computation, EC '16","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tian et al., 2014] Fei Tian, Haifang Li, Wei Chen, Tao Qin, En- hong Chen, and Tie-Yan Liu. Agent behavior prediction and its generalization analysis. arXiv preprint arXiv:1404.4960, 2014.","title":"Agent behavior prediction and its generalization analysis","authors":"Haifang Fei Tian, Wei Li, Tao Chen, Enhong Qin, Tie-Yan Chen, Liu","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yao, 2015] Andrew Chi-Chih Yao. An n-to-1 bidder reduction for multi-item auctions and its applications. In Proceedings of the Twenty-Sixth Annual ACM-SIAM Symposium on Discrete Algo- rithms, SODA 2015, San Diego, CA, USA, January 4-6, 2015, pages 92-109, 2015.","title":"An n-to-1 bidder reduction for multi-item auctions and its applications","authors":"Chi-Chih Andrew, Yao","year":"2015","venue":"Proceedings of the Twenty-Sixth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2015","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}