{"type":"ScholarlyArticle","title":"IMO^3: Interactive Multi-Objective Off-Policy Optimization","authors":[{"name":"Nan Wang","slug":"nan-wang","given":"Nan","family":"Wang","affiliation":"University of Virginia"},{"name":"Hongning Wang","slug":"hongning-wang","given":"Hongning","family":"Wang","affiliation":"University of Virginia"},{"name":"Maryam Karimzadehgan","slug":"maryam-karimzadehgan","given":"Maryam","family":"Karimzadehgan","affiliation":"Google Research"},{"name":"Branislav Kveton","slug":"branislav-kveton","given":"Branislav","family":"Kveton","affiliation":"Amazon"},{"name":"Craig Boutilier","slug":"craig-boutilier","given":"Craig","family":"Boutilier","affiliation":"Google Research"}],"abstract":"Most real-world optimization problems have multiple objectives. A system designer needs to find a policy that trades off these objectives to reach a desired operating point. This problem has been studied extensively in the setting of known objective functions. However, we consider a more practical but challenging setting of unknown objective functions. In industry, optimization under this setting is mostly approached with online A/B testing, which is often costly and inefficient. As an alternative, we propose Interactive Multi-Objective Off-policy Optimization (IMO^3). The key idea of IMO^3 is to interact with a system designer using policies evaluated in an off-policy fashion to uncover which policy maximizes her unknown utility function. We theoretically show that IMO^3 identifies a near-optimal policy with high probability, depending on the amount of designer's feedback and training data for off-policy estimation. We demonstrate its effectiveness empirically on several multi-objective optimization problems.","doi":"10.24963/ijcai.2022/489","url":"https://ijcai.org/proceedings/2022/489","pdf":"https://ijcai.org/proceedings/2022/0489.pdf","markdown":"https://ijcai.org/proceedings/2022/489.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/489","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":489,"pages":{"first":3523,"last":3529},"keywords":["Knowledge Representation and Reasoning: Preference Modelling and Preference-Based Reasoning","Machine Learning: Experimental Methodology","Humans and AI: Human-AI Collaboration","Machine Learning: Optimisation","Uncertainty in AI: Decision and Utility Theory"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986586","length_sec":57,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984729","length_sec":349,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abeille and Lazaric, 2017] Marc Abeille and Alessandro Lazaric. Linear thompson sampling revisited. In AISTATS, 2017.","title":"Linear thompson sampling revisited","authors":"Abeille, Lazaric, Marc Abeille, Alessandro Lazaric","year":"2017","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 2016] Peter Auer, Chao-Kai Chiang, Ronald Or- tner, and Madalina M. Drugan. Pareto front identification from stochastic bandit feedback. In AISTATS, 2016.","title":"Pareto front identification from stochastic bandit feedback","authors":"Peter Auer, Chao-Kai Chiang, Ronald Ortner, Madalina M Drugan","year":"2016","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boutilier, 2002] Craig Boutilier. A pomdp formulation of preference elicitation problems. In Eighteenth National Conference on Artificial Intelligence, 2002.","title":"A pomdp formulation of preference elicitation problems","authors":"Craig Boutilier","year":"2002","venue":"Eighteenth National Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boutilier, 2013] Craig Boutilier. Computational decision support: Regret-based models for optimization and prefer- ence elicitation. In Comparative Decision Making: Analy- sis and Support Across Disciplines and Applications. 2013.","title":"Computational decision support: Regret-based models for optimization and preference elicitation","authors":"Craig Boutilier","year":"2013","venue":"Comparative Decision Making: Analysis and Support Across Disciplines and Applications","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Branke et al., 2008] J. Branke, K. Deb, Kaisa Miettinen, and R. Slowinski. Multiobjective Optimization: Interactive and Evolutionary Approaches. Springer-Verlag, 2008.","title":"Multiobjective Optimization: Interactive and Evolutionary Approaches","authors":"J Branke, K Deb, Kaisa Miettinen, R Slowinski","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Camerer, 2004] Colin F. Camerer. Advances in Behavioral Economics. 2004.","title":"Advances in Behavioral Economics","authors":"Colin F Camerer","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[de Carvalho et al., 2018] Vinicius Renan de Carvalho, Jaime Simão, and Sichman. Solving real-world multi-objective engineering optimization problems with an election-based hyper-heuristic. 2018.","title":"Solving real-world multi-objective engineering optimization problems with an election-based hyper-heuristic","authors":"Jaime Vinicius Renan De Carvalho, Sichman Simão","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deaton and Cartwright, 2018] Angus Deaton and Nancy Cartwright. Understanding and misunderstanding random- ized controlled trials. Social Science & Medicine, 2018.","title":"Understanding and misunderstanding randomized controlled trials","authors":"Cartwright Deaton, Angus Deaton, Nancy Cartwright","year":"2018","venue":"Social Science & Medicine","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Drugan and Nowé, 2013] Madalina M. Drugan and Ann Nowé. Designing multi-objective multi-armed bandits al- gorithms: A study. In IJCNN, 2013.","title":"Designing multi-objective multi-armed bandits algorithms: A study","authors":"M Madalina, Ann Drugan, Nowé","year":"2013","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dudik et al., 2011] Miroslav Dudik, John Langford, and Li- hong Li. Doubly robust policy evaluation and learning. ICML, 2011.","title":"Doubly robust policy evaluation and learning","authors":"Dudik","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dudík et al., 2015] Miroslav Dudík, Katja Hofmann, Robert E. Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In COLT, 2015.","title":"Contextual dueling bandits","authors":"Dudík","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jamieson and Talwalkar, 2016] Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hy- perparameter optimization. In AISTATS. PMLR, 2016.","title":"Non-stochastic best arm identification and hyperparameter optimization","authors":"Talwalkar Jamieson, Kevin Jamieson, Ameet Talwalkar","year":"2016","venue":"AISTATS. PMLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Karnin et al., 2013] Zohar Karnin, Tomer Koren, and Oren Somekh. Almost optimal exploration in multi-armed ban- dits. In ICML, 2013.","title":"Almost optimal exploration in multi-armed bandits","authors":"Karnin","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Keeney and Raiffa, 1976] Ralph L. Keeney and Howard Raiffa. Decisions with Multiple Objectives: Preferences and Value Trade-offs. Wiley, 1976.","title":"Decisions with Multiple Objectives: Preferences and Value Trade-offs","authors":"Ralph L Keeney, Howard Raiffa","year":"1976","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kohavi and Longbotham, 2011] Ron Kohavi and Roger Longbotham. Unexpected results in online controlled ex- periments. SIGKDD Explor. Newsl., 2011.","title":"Unexpected results in online controlled experiments","authors":"Longbotham Kohavi, Ron Kohavi, Roger Longbotham","year":"2011","venue":"SIGKDD Explor. Newsl","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kohavi et al., 2009] Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. Controlled experi- ments on the web: Survey and practical guide. KDD, 2009.","title":"Controlled experiments on the web: Survey and practical guide","authors":"Ron Kohavi, Roger Longbotham, Dan Sommerfield, Randal M Henne","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kveton et al., 2020] Branislav Kveton, Manzil Zaheer, Csaba Szepesvari, Lihong Li, Mohammad Ghavamzadeh, and Craig Boutilier. Randomized exploration in generalized linear bandits. In AISTATS, 2020. [Lambert and Pregibon, 2007] Diane Lambert and Daryl Pregibon. More bang for their bucks: assessing new fea- tures for online advertisers. SIGKDD Explor., 2007.","title":"Randomized exploration in generalized linear bandits","authors":"Manzil Branislav Kveton, Csaba Zaheer, Lihong Szepesvari, Mohammad Li, Craig Ghavamzadeh, Boutilier","year":"2007","venue":"More bang for their bucks: assessing new features for online advertisers","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lattimore and Szepesvári, 2020] Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2020.","title":"Tor Lattimore and Csaba Szepesvári. Bandit Algorithms","authors":"Szepesvári Lattimore","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liang and Qu, 2013] J. J. Liang and B. Y. Qu. Large-scale portfolio optimization using multiobjective dynamic mutli- swarm particle swarm optimizer. In IEEE-SIS, 2013.","title":"Large-scale portfolio optimization using multiobjective dynamic mutliswarm particle swarm optimizer","authors":"J J Liang, B Y Qu","year":"2013","venue":"IEEE-SIS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mas-Colell et al., 1995] Andreu Mas-Colell, Micheal D. Whinston, and Jerry R. Green. Microeconomic Theory. Oxford University Press, 1995.","title":"Microeconomic Theory","authors":"Andreu Mas-Colell, Micheal D Whinston, Jerry R Green","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[McFadden, 1974] Daniel McFadden. Conditional logit anal- ysis of qualitative choice behavior. In Frontiers in Econo- metrics. 1974.","title":"Conditional logit analysis of qualitative choice behavior","authors":"Daniel Mcfadden","year":"1974","venue":"Frontiers in Econometrics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Paria et al., 2019] Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. A flexible framework for multi- objective bayesian optimization using random scalariza- tions. In UAI, 2019.","title":"A flexible framework for multiobjective bayesian optimization using random scalarizations","authors":"Biswajit Paria, Kirthevasan Kandasamy, Barnabás Póczos","year":"2019","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Roijers et al., 2017] Diederik M. Roijers, Luisa M. Zintgraf, and Ann Nowé. Interactive thompson sampling for multi- objective multi-armed bandits. In ADT, 2017.","title":"Interactive thompson sampling for multiobjective multi-armed bandits","authors":"M Diederik, Luisa M Roijers, Ann Zintgraf, Nowé","year":"2017","venue":"ADT","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rosenbaum and Rubin, 1983] Paul R. Rosenbaum and Don- ald B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 1983.","title":"The central role of the propensity score in observational studies for causal effects","authors":"Paul R Rosenbaum, Donald B Rubin","year":"1983","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Strehl et al., 2010] Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. Learning from logged implicit explo- ration data. In NeurIPS, 2010.","title":"Learning from logged implicit exploration data","authors":"Alex Strehl, John Langford, Lihong Li, M Sham, Kakade","year":"2010","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Swaminathan and Joachims, 2015] Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. ICML, 2015.","title":"Counterfactual risk minimization: Learning from logged bandit feedback","authors":"Joachims Swaminathan, Adith Swaminathan, Thorsten Joachims","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tversky and Kahneman, 1974] Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases. Science, 1974.","title":"Judgment under uncertainty: Heuristics and biases","authors":"Kahneman Tversky, Amos Tversky, Daniel Kahneman","year":"1974","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Viappiani and Boutilier, 2010] Paolo Viappiani and Craig Boutilier. Optimal Bayesian recommendation sets and my- opically optimal choice query sets. In NeurIPS, 2010.","title":"Paolo Viappiani and Craig Boutilier. Optimal Bayesian recommendation sets and myopically optimal choice query sets","authors":"Boutilier Viappiani","year":"2010","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2021] Nan Wang, Branislav Kveton, and Maryam Karimzadehgan. Core: Capitalizing on rewards in bandit exploration. In UAI, 2021.","title":"Core: Capitalizing on rewards in bandit exploration","authors":"Nan Wang, Branislav Kveton, Maryam Karimzadehgan","year":"2021","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wong, 1994] Weng Kee Wong. Comparing robust properties of a, d, e and g-optimal designs. Computational Statistics & Data Analysis, 1994.","title":"Comparing robust properties of a, d, e and g-optimal designs","authors":"Weng Kee, Wong","year":"1994","venue":"Computational Statistics & Data Analysis","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yahyaa and Manderick, 2015] Saba Q. Yahyaa and Bernard Manderick. Thompson sampling for multi-objective multi- armed bandits problem. In ESANN, 2015.","title":"Thompson sampling for multi-objective multiarmed bandits problem","authors":"Saba Q Yahyaa, Bernard Manderick","year":"2015","venue":"ESANN","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang and Golovin, 2020] Richard Zhang and Daniel Golovin. Random hypervolume scalarizations for provable multi-objective black box optimization. In ICML, 2020.","title":"Random hypervolume scalarizations for provable multi-objective black box optimization","authors":"Richard Zhang, Daniel Golovin","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zitzler et al., 2000] Eckart Zitzler, Kalyanmoy Deb, and Lothar Thiele. Comparison of multiobjective evolution- ary algorithms: Empirical results. Evol. Comput., 2000.","title":"Comparison of multiobjective evolutionary algorithms: Empirical results","authors":"Eckart Zitzler, Kalyanmoy Deb, Lothar Thiele","year":"2000","venue":"Evol. Comput","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}