{"type":"ScholarlyArticle","title":"SlateQ: A Tractable Decomposition for Reinforcement Learning with Recommendation Sets","authors":[{"name":"Eugene Ie","slug":"eugene-ie","given":"Eugene","family":"Ie","affiliation":"Google Research"},{"name":"Vihan Jain","slug":"vihan-jain","given":"Vihan","family":"Jain","affiliation":"Google Research"},{"name":"Jing Wang","slug":"jing-wang","given":"Jing","family":"Wang","affiliation":"Google Research"},{"name":"Sanmit Narvekar","slug":"sanmit-narvekar","given":"Sanmit","family":"Narvekar","affiliation":"Department of Computer Science , University of Texas at Austin"},{"name":"Ritesh Agarwal","slug":"ritesh-agarwal","given":"Ritesh","family":"Agarwal","affiliation":"Google Research"},{"name":"Rui Wu","slug":"rui-wu","given":"Rui","family":"Wu","affiliation":"Google Research"},{"name":"Heng-Tze Cheng","slug":"heng-tze-cheng","given":"Heng-Tze","family":"Cheng","affiliation":"Google Research"},{"name":"Tushar Chandra","slug":"tushar-chandra","given":"Tushar","family":"Chandra","affiliation":"Google Research"},{"name":"Craig Boutilier","slug":"craig-boutilier","given":"Craig","family":"Boutilier","affiliation":"Google Research"}],"abstract":"Reinforcement learning methods for recommender systems optimize recommendations for long-term user engagement. However, since users are often presented with slates of multiple items---which may have interacting effects on user choice---methods are required to deal with the combinatorics of the RL action space. We develop SlateQ, a decomposition of value-based temporal-difference and Q-learning that renders RL tractable with slates. Under mild assumptions on user choice behavior, we show that the long-term value (LTV) of a slate can be decomposed into a tractable function of its component item-wise LTVs. We demonstrate our methods in simulation, and validate the scalability and effectiveness of decomposed TD-learning on YouTube.","doi":"10.24963/ijcai.2019/360","url":"https://ijcai.org/proceedings/2019/360","pdf":"https://ijcai.org/proceedings/2019/0360.pdf","markdown":"https://ijcai.org/proceedings/2019/360.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/360","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":360,"pages":{"first":2592,"last":2599},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Machine Learning: Recommender Systems","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2015] M. Abadi, A. Agarwal, et al. Tensor- Flow: Large-scale machine learning on heterogeneous sys- tems, 2015. Software available from tensorflow.org.","title":"Tensor-Flow: Large-scale machine learning on heterogeneous systems","authors":"M Abadi, A Agarwal","year":"2015","venue":"Software available from tensorflow","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Breese et al., 1998] J. Breese, D. Heckerman, and C. Kadie. Empirical analysis of predictive algorithms for collabora- tive filtering. UAI-98, 43-52, Madison, WI, 1998.","title":"Empirical analysis of predictive algorithms for collaborative filtering","authors":"J Breese, D Heckerman, C Kadie","year":"1998","venue":"UAI-98","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Charnes and Cooper, 1962] A. Charnes and W. Cooper. Programming with linear fractional functionals. Naval Res. Log. Qrt., 9(3-4):181-186, 1962.","title":"Programming with linear fractional functionals","authors":"A Charnes, W Cooper","year":"1962","venue":"Naval Res. Log. Qrt","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and Hausman, 2000] K. Chen and W. Hausman. Mathematical properties of the optimal product line selec- tion problem using choice-based conjoint analysis. Mgmt. Sci., 46(2):327-332, 2000.","title":"Mathematical properties of the optimal product line selection problem using choice-based conjoint analysis","authors":"K Chen, W Hausman","year":"2000","venue":"Mgmt. Sci","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] M. Chen, A. Beutel, P. Covington, S. Jain, F. Belletti, E. Chi. Top-k off-policy correction for a REINFORCE recommender system. WSDM-19, 456-464.","title":"Top-k off-policy correction for a REINFORCE recommender system","authors":"M Chen, A Beutel, P Covington, S Jain, F Belletti, E Chi","year":null,"venue":"WSDM-19","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cheng et al., 2016] H. Cheng, L. Koc, et al. Wide & deep learning for recommender systems. Deep Learn. for Rec. Sys. Workshop, 7-10, Boston, 2016.","title":"Wide & deep learning for recommender systems","authors":"H Cheng, L Koc","year":"2016","venue":"Deep Learn. for Rec. Sys. Workshop","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Choi et al., 2018] S. Choi, H. Ha, U. Hwang, C. Kim, J. Ha, and S. Yoon. Reinforcement learning-based recommender system using biclustering technique. arXiv:1801.05532 [cs.IR], 2018.","title":"Reinforcement learning-based recommender system using biclustering technique","authors":"S Choi, H Ha, U Hwang, C Kim, J Ha, S Yoon","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Covington et al., 2016] P. Covington, J. Adams, and E. Sar- gin. Deep neural networks for YouTube recommendations. RecSys-16, 191-198, Boston, 2016.","title":"Deep neural networks for YouTube recommendations","authors":"P Covington, J Adams, E Sargin","year":"2016","venue":"RecSys-16","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Deshpande and Karypis, 2004] M. Deshpande and G. Karypis. Item-based top-n recommendation algorithms. ACM TOIS, 22(1):143-177, 2004.","title":"Item-based top-n recommendation algorithms","authors":"M Deshpande, G Karypis","year":"2004","venue":"ACM TOIS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gauci et al., 2018] J. Gauci, E. Conti, et al. Horizon: Face- book's open source applied reinforcement learning plat- form. arXiv:1811.00260 [cs.LG], 2018.","title":"Horizon: Facebook's open source applied reinforcement learning platform","authors":"J Gauci, E Conti","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ie et al., 2019] E. Ie, V. Jain, J. Wang, S. Navrekar, R. Agar- wal, R. Wu, H.-Tze Cheng, M. Lustman, V. Gatto, P. Cov- ington, J. McFadden, T. Chandra, and C. Boutilier. Re- inforcement learning for slate-based recommender sys- tems: A tractable decomposition and practical methodol- ogy. arXiv:1905.12767, 2019.","title":"Reinforcement learning for slate-based recommender systems: A tractable decomposition and practical methodology","authors":"E Ie, V Jain, J Wang, S Navrekar, R Agarwal, R Wu, H.-Tze Cheng, M Lustman, V Gatto, P Covington, J Mcfadden, T Chandra, C Boutilier","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2019] R. Jiang, S. Gowal, T. Mann, and D. Rezende. Beyond greedy ranking: Slate optimization via list-CVAE. ICLR-19, New Orleans, 2019.","title":"Beyond greedy ranking: Slate optimization via list-CVAE","authors":"R Jiang, S Gowal, T Mann, D Rezende","year":"2019","venue":"ICLR-19","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Joachims, 2002] T. Joachims. Optimizing search engines using clickthrough data. KDD-02, 133-142, 2002.","title":"Optimizing search engines using clickthrough data","authors":"T Joachims","year":"2002","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Konstan et al., 1997] J. Konstan, B. Miller, D. Maltz, J. Her- locker, L. Gordon, J. Riedl. GroupLens: Applying collab- orative filtering to Usenet news. CACM, 40:77-87, 1997.","title":"GroupLens: Applying collaborative filtering to Usenet news","authors":"J Konstan, B Miller, D Maltz, J Herlocker, L Gordon, J Riedl","year":"1997","venue":"CACM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Le and Lauw, 2017] D. Le and H. Lauw. Indexable Bayesian personalized ranking for efficient top-k recom- mendation. CIKM-17, 1389-1398, 2017.","title":"Indexable Bayesian personalized ranking for efficient top-k recommendation","authors":"D Le, H Lauw","year":"2017","venue":"CIKM-17","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Louviere et al., 2000] J. Louviere, D. Hensher, and J. Swait. Stated Choice Methods: Analysis and Application. Cam- bridge Univ. Press, 2000.","title":"Stated Choice Methods: Analysis and Application","authors":"J Louviere, D Hensher, J Swait","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Metz et al., 2017] L. Metz, J. Ibarz, N. Jaitly, and J. David- son. Discrete sequential prediction of continuous actions for deep RL. arXiv:1705.05035 [cs.LG], 2017.","title":"Discrete sequential prediction of continuous actions for deep RL","authors":"L Metz, J Ibarz, N Jaitly, J Davidson","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nemhauser et al., 1978] G. Nemhauser, L. Wolsey, and M. Fisher. An analysis of approximations for maximizing submodular set functions. Math. Prog., 14:265-294, 1978.","title":"An analysis of approximations for maximizing submodular set functions","authors":"G Nemhauser, L Wolsey, M Fisher","year":"1978","venue":"Math. Prog","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Puterman, 1994] M. Puterman. Markov Decision Pro- cesses: Discrete Stochastic Dynamic Programming. 1994.","title":"Markov Decision Processes: Discrete Stochastic Dynamic Programming","authors":"M Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rendle et al., 2010] S. Rendle, C. Freudenthaler, L. Schmidt-Thieme. Factorizing personalized Markov chains for next-basket recommendation. WWW10, 811-820.","title":"Factorizing personalized Markov chains for next-basket recommendation","authors":"S Rendle, C Freudenthaler, L Schmidt-Thieme","year":null,"venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Salakhutdinov and Mnih, 2007] R. Salakhutdinov and A. Mnih. Probabilistic matrix factorization. NIPS-07, 1257- 1264, Vancouver, 2007.","title":"Probabilistic matrix factorization","authors":"R Salakhutdinov, A Mnih","year":"2007","venue":"NIPS-07","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schön, 2010] C. Schön. On the optimal product line se- lection problem with price discrimination. Mgmt. Sci., 56(5):896-902, 2010.","title":"On the optimal product line selection problem with price discrimination","authors":"C Schön","year":"2010","venue":"Mgmt. Sci","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shani et al., 2005] G. Shani, D. Heckerman, and R. Braf- man. An MDP-based recommender system. JMLR, 6:1265-1295, 2005.","title":"An MDP-based recommender system","authors":"G Shani, D Heckerman, R Brafman","year":"2005","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sunehag et al., 2015] P. Sunehag, R. Evans, G. Dulac- Arnold, Y. Zwols, D. Visentin, and B. Coppin. Deep re- inforcement learning with attention for slate Markov de- cision processes with high-dimensional states and actions. arXiv:1512.01124 [cs.AI], 2015.","title":"Deep reinforcement learning with attention for slate Markov decision processes with high-dimensional states and actions","authors":"P Sunehag, R Evans, G Dulac-Arnold, Y Zwols, D Visentin, B Coppin","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sutton and Barto, 1998] R. Sutton, A. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.","title":"Reinforcement Learning: An Introduction","authors":"R Sutton, A Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Swaminathan et al., 2017] A. Swaminathan, A. Krishna- murthy, A. Agarwal, M. Dudik, J. Langford, D. Jose, and I. Zitouni. Off-policy evaluation for slate recommendation. NIPS-17, 3632-3642, Long Beach, CA, 2017.","title":"Off-policy evaluation for slate recommendation","authors":"A Swaminathan, A Krishnamurthy, A Agarwal, M Dudik, J Langford, D Jose, I Zitouni","year":"2017","venue":"NIPS-17","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Taghipour et al., 2007] N. Taghipour, A. Kardan, S. Ghidary. Usage-based web recommendations: A reinforcement learning approach. RecSys-07, 113-120.","title":"Usage-based web recommendations: A reinforcement learning approach","authors":"N Taghipour, A Kardan, S Ghidary","year":null,"venue":"RecSys-07","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[van den Oord et al., 2013] A. van den Oord, S. Dieleman, and B. Schrauwen. Deep content-based music recommen- dation. NIPS-13, 2643-2651, Lake Tahoe, NV, 2013.","title":"Deep content-based music recommendation","authors":"Oord Van Den","year":"2013","venue":"NIPS-13","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Van Seijen et al., 2009] H. Van Seijen, H. Van Hasselt, S. Whiteson, and M. Wiering. A theoretical and empirical analysis of expected SARSA. IEEE Symp. Adap. Dyn. Prog. and RL, 177-184, 2009.","title":"A theoretical and empirical analysis of expected SARSA","authors":"H Van Seijen, H Van Hasselt, S Whiteson, M Wiering","year":"2009","venue":"IEEE Symp. Adap. Dyn. Prog. and RL","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Viappiani & Boutilier, 2010] P. Viappiani, C. Boutilier. Op- timal Bayesian recommendation sets and myopically opti- mal choice query sets. NIPS, 2352-2360, 2010.","title":"Optimal Bayesian recommendation sets and myopically optimal choice query sets","authors":"P Viappiani, C Boutilier","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wu et al., 2017] C. Wu, A. Ahmed, A. Beutel, A. Smola, and H. Jing. Recurrent recommender networks. WSDM- 17, 495-503, Cambridge, UK, 2017.","title":"Recurrent recommender networks","authors":"C Wu, A Ahmed, A Beutel, A Smola, H Jing","year":"2017","venue":"WSDM-17","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhao et al., 2018] X. Zhao, L. Xia, L. Zhang, Z. Ding, D. Yin, J. Tang. Deep reinforcement learning for page-wise recommendations. RecSys-18, 95-103, Vancouver, 2018.","title":"Deep reinforcement learning for page-wise recommendations","authors":"Zhao","year":"2018","venue":"RecSys-18","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}