{"type":"ScholarlyArticle","title":"Using Contextual Bandits with Behavioral Constraints for Constrained Online Movie Recommendation","authors":[{"name":"Avinash Balakrishnan","slug":"avinash-balakrishnan","given":"Avinash","family":"Balakrishnan","affiliation":"IBM Research Yorktown Heights , NY 10598"},{"name":"Djallel Bouneffouf","slug":"djallel-bouneffouf","given":"Djallel","family":"Bouneffouf","affiliation":"IBM Research Yorktown Heights , NY 10598"},{"name":"Nicholas Mattei","slug":"nicholas-mattei","given":"Nicholas","family":"Mattei","affiliation":"IBM Research Yorktown Heights , NY 10598"},{"name":"Francesca Rossi","slug":"francesca-rossi","given":"Francesca","family":"Rossi","affiliation":"IBM Research Yorktown Heights , NY 10598"}],"abstract":"AI systems that learn through reward feedback about the actions they take are increasingly deployed in domains that have significant impact on our daily life. In many cases the rewards should not be the only guiding criteria, as there are additional constraints and/or priorities imposed by regulations, values, preferences, or ethical principles. We detail a novel online system, based on an extension of the contextual bandits framework, that learns a set of behavioral constraints by observation and uses these constraints as a guide when making decisions in an online setting while still being reactive to reward feedback. In addition, our system can highlight features of the context which are more predicted to be more rewarding and/or are in line with the behavioral constraints.  We demonstrate the system by building an interactive interface for an online movie recommendation agent and show that our system is able to act within a set of behavior constraints without significantly degrading overall performance.","doi":"10.24963/ijcai.2018/843","url":"https://ijcai.org/proceedings/2018/843","pdf":"https://ijcai.org/proceedings/2018/0843.pdf","markdown":"https://ijcai.org/proceedings/2018/843.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/843","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Demos","track":"Demos","number":843,"pages":{"first":5802,"last":5804},"keywords":["Machine Learning: Reinforcement Learning","Uncertainty in AI: Sequential Decision Making","Machine Learning: Online Learning","Multidisciplinary Topics and Applications: Philosophical and Ethical Issues"],"videos":[],"references":[{"position":1,"raw":"[Agrawal and Goyal, 2013] Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear pay- offs. In ICML (3), pages 127-135, 2013.","title":"Thompson sampling for contextual bandits with linear payoffs","authors":"Shipra Agrawal, Navin Goyal","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal and Goyal, 2016] Shipra Agrawal and Navin Goyal. Linear contextual bandits with knapsacks. In Proceedings of Advances in Neural Information Processing Systems (NIPS 2016), pages 3450-3458, 2016.","title":"Linear contextual bandits with knapsacks","authors":"Shipra Agrawal, Navin Goyal","year":"2016","venue":"Proceedings of Advances in Neural Information Processing Systems (NIPS 2016)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Armstrong, 2015] Stuart Armstrong. Motivated value selec- tion for artificial agents. In Workshops of the 29th AAAI: AI, Ethics, and Society, 2015.","title":"Motivated value selection for artificial agents","authors":"Stuart Armstrong","year":"2015","venue":"Workshops of the 29th AAAI: AI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Auer et al., 2002] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit prob- lem. Machine Learning, 47(2-3):235-256, 2002.","title":"Finite-time analysis of the multiarmed bandit problem","authors":"Peter Auer, Nicolò Cesa-Bianchi, Paul Fischer","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Briggs and Scheutz, 2015] Gordon Briggs and Matthias Scheutz. \"Sorry, I can't do that\": Developing mechanisms to appropriately reject directives in human-robot interac- tions. In AAAI Fall Symposium Series: AI for Human-Robot Interaction, 2015.","title":"Sorry, I can't do that\": Developing mechanisms to appropriately reject directives in human-robot interactions","authors":"Gordon Briggs, Matthias Scheutz","year":"2015","venue":"AAAI Fall Symposium Series: AI for Human-Robot Interaction","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chu et al., 2011] Wei Chu, Lihong Li, Lev Reyzin, and Robert E. Schapire. Contextual bandits with linear pay- off functions. In Proc. AISTATS, pages 208-214, 2011.","title":"Contextual bandits with linear payoff functions","authors":"Wei Chu, Lihong Li, Lev Reyzin, Robert E Schapire","year":"2011","venue":"Proc. AISTATS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Harper and Konstan, 2016] F Maxwell Harper and Joseph A Konstan. The Movielens datasets: History and context. ACM Transactions on Interactive Intelligent Systems (TiiS), 5(4):19, 2016.","title":"The Movielens datasets: History and context","authors":"Maxwell Harper, Joseph A Konstan","year":"2016","venue":"ACM Transactions on Interactive Intelligent Systems (TiiS)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jobber and Ellis-Chadwick, 2012] David Jobber and Fiona Ellis-Chadwick. Principles and Practice of Marketing. McGraw-Hill Higher Education, 2012.","title":"David Jobber and Fiona Ellis-Chadwick","authors":"Ellis-Chadwick Jobber","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lai and Robbins, 1985] Tallai Andherbertrobbins Lai and Herbert Robbins. Asymptotically efficient adaptive alloca- tion rules. Advances in Applied Mathematics, 6(1):4-22, 1985.","title":"Tallai Andherbertrobbins Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules","authors":"Robbins Lai","year":"1985","venue":"Advances in Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Langford and Zhang, 2008] John Langford and Tong Zhang. The Epoch-Greedy Algorithm for Contextual Multi-armed Bandits. In Proc. 21st NIPS, 2008.","title":"The Epoch-Greedy Algorithm for Contextual Multi-armed Bandits","authors":"Zhang ; ] Langford, John Langford, Tong Zhang","year":"2008","venue":"Proc. 21st NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2010] Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to per- sonalized news article recommendation. In Proc. 19th WWW, pages 661-670, USA, 2010.","title":"A contextual-bandit approach to personalized news article recommendation","authors":"Lihong Li, Wei Chu, John Langford, Robert E Schapire","year":"2010","venue":"Proc. 19th WWW","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mary et al., 2015] Jérémie Mary, Romaric Gaudel, and Philippe Preux. Bandits and recommender systems. In Machine Learning, Optimization, and Big Data, pages 325- 336, 2015.","title":"Bandits and recommender systems","authors":"Jérémie Mary, Romaric Gaudel, Philippe Preux","year":"2015","venue":"Machine Learning, Optimization, and Big Data","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep rein- forcement learning. CoRR, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sen, 1974] Amartya Sen. Choice, Ordering and Morality. Blackwell, Oxford, 1974.","title":"Choice, Ordering and Morality","authors":"Amartya Sen","year":"1974","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sutton and Barto, 2017] Richard S. Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2017.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew Barto","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Villar et al., 2015] Sofia S. Villar, Jack Bowden, and James Wason. Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges. Statistical Science, 30(2):199, 2015.","title":"Multi-armed bandit models for the optimal design of clinical trials: benefits and challenges","authors":"Villar","year":"2015","venue":"Statistical Science","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wu et al., 2015] Huasen Wu, R. Srikant, Xin Liu, and Chong Jiang. Algorithms with logarithmic or sublinear regret for constrained contextual bandits. In Advances in Neural Information Processing Systems (NIPS), pages 433-441, 2015.","title":"Algorithms with logarithmic or sublinear regret for constrained contextual bandits","authors":"Huasen Wu, R Srikant, Xin Liu, Chong Jiang","year":"2015","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}