{"type":"ScholarlyArticle","title":"Environment Design for Biased Decision Makers","authors":[{"name":"Guanghui Yu","slug":"guanghui-yu","given":"Guanghui","family":"Yu","affiliation":"Washington University in St. Louis"},{"name":"Chien-Ju Ho","slug":"chien-ju-ho","given":"Chien-Ju","family":"Ho","affiliation":"Washington University in St. Louis"}],"abstract":"We study the environment design problem for biased decision makers. In an environment design problem, an informed principal aims to update the decision making environment to influence the decisions made by the agent. This problem is ubiquitous in various domains, e.g., a social networking platform might want to update its website to encourage more user engagement. In this work, we focus on the scenario in which the agent might exhibit biases in decision making. We relax the common assumption that the agent is rational and aim to incorporate models of biased agents in environment design. We formulate the environment design problem under the Markov decision process (MDP) and incorporate common models of biased agents through introducing general time-discounting functions. We then formalize the environment design problem as constrained optimization problems and propose corresponding algorithms. We conduct both simulations and real human-subject experiments with workers recruited from Amazon Mechanical Turk to evaluate our proposed algorithms.","doi":"10.24963/ijcai.2022/84","url":"https://ijcai.org/proceedings/2022/84","pdf":"https://ijcai.org/proceedings/2022/0084.pdf","markdown":"https://ijcai.org/proceedings/2022/84.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/84","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":84,"pages":{"first":592,"last":598},"keywords":["Humans and AI: Applications","Humans and AI: Human-AI Collaboration","Agent-based and Multi-agent Systems: Human-Agent Interaction","Planning and Scheduling: Markov Decisions Processes"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986777","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984920","length_sec":368,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y Ng. Appren- ticeship learning via inverse reinforcement learning. In Interna- tional Conference on Machine Learning, 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Abbeel, ; Ng, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Duan et al., 2020] Xiaoni Duan, Chien-Ju Ho, and Ming Yin. Does exposure to diverse perspectives mitigate biases in crowd- work? an explorative study. In AAAI Conference on Human Com- putation and Crowdsourcing, 2020.","title":"Does exposure to diverse perspectives mitigate biases in crowdwork? an explorative study","authors":"Duan","year":"2020","venue":"AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Duan et al., 2022] Xiaoni Duan, Chien-Ju Ho, and Ming Yin. The influences of task design on crowdsourced judgement: A case study of recidivism risk evaluation. In The Web Conference (WWW), 2022.","title":"The influences of task design on crowdsourced judgement: A case study of recidivism risk evaluation","authors":"Duan","year":"2022","venue":"The Web Conference (WWW)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Evans and Goodman, 2015] Owain Evans and Noah D Goodman. Learning the preferences of bounded agents. In NIPS Workshop on Bounded Optimality, 2015.","title":"Learning the preferences of bounded agents","authors":"Owain Evans, Noah D Goodman","year":"2015","venue":"NIPS Workshop on Bounded Optimality","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Evans et al., 2016] Owain Evans, Andreas Stuhlmüller, and Noah Goodman. Learning the preferences of ignorant, inconsistent agents. In AAAI Conference on Artificial Intelligence, 2016.","title":"Learning the preferences of ignorant, inconsistent agents","authors":"Owain Evans, Andreas Stuhlmüller, Noah Goodman","year":"2016","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Frazier et al., 2014] Peter Frazier, David Kempe, Jon Kleinberg, and Robert Kleinberg. Incentivizing exploration. In ACM Con- ference on Economics and Computation, 2014.","title":"Incentivizing exploration","authors":"Peter Frazier, David Kempe, Jon Kleinberg, Robert Kleinberg","year":"2014","venue":"ACM Conference on Economics and Computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gottwald and Braun, 2019] Sebastian Gottwald and Daniel A Braun. Bounded rational decision-making from elementary com- putations that reduce uncertainty. Entropy, 2019.","title":"Bounded rational decision-making from elementary computations that reduce uncertainty","authors":"Sebastian Gottwald, Daniel A Braun","year":"2019","venue":"Entropy","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hughes et al., 2020] Dana Hughes, Akshat Agarwal, Yue Guo, and Katia Sycara. Inferring non-stationary human preferences for human-agent teams. In IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 2020.","title":"Inferring non-stationary human preferences for human-agent teams","authors":"Dana Hughes, Akshat Agarwal, Yue Guo, Katia Sycara","year":"2020","venue":"IEEE International Conference on Robot and Human Interactive Communication","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kahneman, 2003] Daniel Kahneman. A perspective on judgment and choice: mapping bounded rationality. American Psycholo- gist, 58(9):697, 2003.","title":"A perspective on judgment and choice: mapping bounded rationality","authors":"Daniel Kahneman","year":"2003","venue":"American Psychologist","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kleinberg and Oren, 2014] Jon Kleinberg and Sigal Oren. Time- inconsistent planning: a computational problem in behavioral economics. In ACM Conference on Economics and Computa- tion, 2014.","title":"Timeinconsistent planning: a computational problem in behavioral economics","authors":"Oren ; Kleinberg, Jon Kleinberg, Sigal Oren","year":"2014","venue":"ACM Conference on Economics and Computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kleinberg et al., 2017] Jon Kleinberg, Sigal Oren, and Manish Raghavan. Planning with multiple biases. In ACM Conference on Economics and Computation, 2017.","title":"Planning with multiple biases","authors":"Jon Kleinberg, Oren Sigal, Manish Raghavan","year":"2017","venue":"ACM Conference on Economics and Computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mansour et al., 2015] Yishay Mansour, Aleksandrs Slivkins, and Vasilis Syrgkanis. Bayesian incentive-compatible bandit explo- ration. In ACM Conference on Economics and Computation, 2015.","title":"Bayesian incentive-compatible bandit exploration","authors":"Yishay Mansour, Aleksandrs Slivkins, Vasilis Syrgkanis","year":"2015","venue":"ACM Conference on Economics and Computation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Masters et al., 2021a] Peta Masters, Michael Kirley, and Wally Smith. Extended goal recognition: a planning-based model for strategic deception. In International Conference on Autonomous Agents and MultiAgent Systems, 2021.","title":"Extended goal recognition: a planning-based model for strategic deception","authors":"Masters","year":"2021","venue":"International Conference on Autonomous Agents and MultiAgent Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Masters et al., 2021b] Peta Masters, Wally Smith, and Michael Kirley. Extended goal recognition: Lessons from magic. Fron- tiers in Artificial Intelligence, 4, 2021.","title":"Extended goal recognition: Lessons from magic","authors":"Masters","year":"2021","venue":"Frontiers in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ng et al., 2000] Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Y Andrew, Stuart J Ng, Russell","year":"2000","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[O'Donoghue and Rabin, 1999] Ted O'Donoghue and Matthew Rabin. Doing it now or later. American Economic Review, 89(1):103-124, 1999.","title":"Ted O'Donoghue and Matthew Rabin. Doing it now or later","authors":"Rabin O'donoghue","year":"1999","venue":"American Economic Review","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rakhsha et al., 2020] Amin Rakhsha, Goran Radanovic, Rati De- vidze, Xiaojin Zhu, and Adish Singla. Policy teaching via en- vironment poisoning: Training-time adversarial attacks against reinforcement learning. In International Conference on Machine Learning, 2020.","title":"Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning","authors":"Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, Adish Singla","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ramachandran and Amir, 2007] Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In International Joint Conference on Artificial Intelligence, 2007.","title":"Bayesian inverse reinforcement learning","authors":"Amir Ramachandran, Deepak Ramachandran, Eyal Amir","year":"2007","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2007/416"},{"position":19,"raw":"[Schach et al., 2018] Sonja Schach, Sebastian Gottwald, and Daniel A Braun. Quantifying motor task performance by bounded rational decision theory. Frontiers in neuroscience, 2018.","title":"Quantifying motor task performance by bounded rational decision theory","authors":"Sonja Schach, Sebastian Gottwald, Daniel A Braun","year":"2018","venue":"Frontiers in neuroscience","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shah et al., 2019] Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca Dragan. On the feasibility of learning, rather than assum- ing, human biases for reward inference. In International Confer- ence on Machine Learning, 2019.","title":"On the feasibility of learning, rather than assuming, human biases for reward inference","authors":"Rohin Shah, Noah Gundotra, Pieter Abbeel, Anca Dragan","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Slivkins, 2019] Aleksandrs Slivkins. Introduction to multi-armed bandits. Found. Trends Mach. Learn., 2019.","title":"Introduction to multi-armed bandits","authors":"Aleksandrs Slivkins","year":"2019","venue":"Found. Trends Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Song et al., 2019] Zhao Song, Ron Parr, and Lawrence Carin. Re- visiting the softmax bellman operator: New benefits and new perspective. In International Conference on Machine Learning, 2019.","title":"Revisiting the softmax bellman operator: New benefits and new perspective","authors":"Zhao Song, Ron Parr, Lawrence Carin","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tang and Ho, 2019] Wei Tang and Chien-Ju Ho. Bandit learning with biased human feedback. In International Conference on Au- tonomous Agents and Multiagent Systems, 2019.","title":"Bandit learning with biased human feedback","authors":"Wei Tang, Chien-Ju Ho","year":"2019","venue":"International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang and Ho, 2021] Wei Tang and Chien-Ju Ho. On the bayesian rational assumption in information design. In AAAI Conference on Human Computation and Crowdsourcing, 2021.","title":"On the bayesian rational assumption in information design","authors":"Wei Tang, Chien-Ju Ho","year":"2021","venue":"AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tang et al., 2019] Wei Tang, Chien-Ju Ho, and Ming Yin. Lever- aging peer communication to enhance crowdsourcing. In The Web Conference (WWW), 2019.","title":"Leveraging peer communication to enhance crowdsourcing","authors":"Wei Tang, Chien-Ju Ho, Ming Yin","year":"2019","venue":"The Web Conference (WWW)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Parkes, 2008] Haoqi Zhang and David C Parkes. Value- based policy teaching with active indirect elicitation. In AAAI Conference on Artificial Intelligence, 2008.","title":"Valuebased policy teaching with active indirect elicitation","authors":"Parkes ; ] Haoqi Zhang, David C Zhang, Parkes","year":"2008","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang and Yu, 2013] Shunan Zhang and Angela J. Yu. Forget- ful bayes and myopic planning: Human learning and decision- making in a bandit setting. In Advances in Neural Information Processing Systems, 2013.","title":"Forgetful bayes and myopic planning: Human learning and decisionmaking in a bandit setting","authors":"Shunan Zhang, Angela J Yu","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2009] Haoqi Zhang, Yiling Chen, and David C Parkes. A general approach to environment design with one agent. In International Joint Conference on Artificial Intelli- gence, 2009.","title":"A general approach to environment design with one agent","authors":"Zhang","year":"2009","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2009/330"},{"position":29,"raw":"[Zhang et al., 2018] Haifeng Zhang, Jun Wang, Zhiming Zhou, Weinan Zhang, Ying Wen, Yong Yu, and Wenxin Li. Learning to design games: Strategic environments in reinforcement learn- ing. In International Joint Conference on Artificial Intelligence, 2018.","title":"Learning to design games: Strategic environments in reinforcement learning","authors":"Haifeng Zhang, Jun Wang, Zhiming Zhou, Weinan Zhang, Ying Wen, Yong Yu, Wenxin Li","year":"2018","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/426"},{"position":30,"raw":"[Zhang et al., 2020] Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against rein- forcement learning. In International Conference on Machine Learning, 2020.","title":"Adaptive reward-poisoning attacks against reinforcement learning","authors":"Zhang","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhi-Xuan et al., 2020] Tan Zhi-Xuan, Jordyn Mann, Tom Silver, Josh Tenenbaum, and Vikash Mansinghka. Online bayesian goal inference for boundedly rational planning agents. In Advances in Neural Information Processing Systems, 2020.","title":"Online bayesian goal inference for boundedly rational planning agents","authors":"Tan Zhi-Xuan, Jordyn Mann, Tom Silver, Josh Tenenbaum, Vikash Mansinghka","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}