{"type":"ScholarlyArticle","title":"Advantage Amplification in Slowly Evolving Latent-State Environments","authors":[{"name":"Martin Mladenov","slug":"martin-mladenov","given":"Martin","family":"Mladenov","affiliation":"Google AI , Mountain View , CA , USA"},{"name":"Ofer Meshi","slug":"ofer-meshi","given":"Ofer","family":"Meshi","affiliation":"Google AI , Mountain View , CA , USA"},{"name":"Jayden Ooi","slug":"jayden-ooi","given":"Jayden","family":"Ooi","affiliation":"Google AI , Mountain View , CA , USA"},{"name":"Dale Schuurmans","slug":"dale-schuurmans","given":"Dale","family":"Schuurmans","affiliation":"Google AI , Mountain View , CA , USA; Department of Computer Science , University of Alberta , Edmonton , AB , Canada"},{"name":"Craig Boutilier","slug":"craig-boutilier","given":"Craig","family":"Boutilier","affiliation":"Google AI , Mountain View , CA , USA"}],"abstract":"Latent-state environments with long horizons, such as those faced by recommender systems, pose significant challenges for reinforcement learning (RL). In this work, we identify and analyze several key hurdles for RL in such environments, including belief state error and small action advantage. We develop a general principle called advantage amplification that an overcome these hurdles through the use of temporal abstraction. We propose several aggregation methods and prove they induce amplification in certain settings. We also bound the loss in optimality incurred by our methods in environments where latent state evolves slowly and demonstrate their performance empirically in a stylized user-modeling task.","doi":"10.24963/ijcai.2019/439","url":"https://ijcai.org/proceedings/2019/439","pdf":"https://ijcai.org/proceedings/2019/0439.pdf","markdown":"https://ijcai.org/proceedings/2019/439.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/439","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":439,"pages":{"first":3165,"last":3172},"keywords":["Machine Learning: Time-series;Data Streams","Machine Learning: Reinforcement Learning","Humans and AI: Personalization and User Modeling","Machine Learning: Recommender Systems"],"videos":[],"references":[{"position":1,"raw":"[Archak et al., 2012] N. Archak, V. Mirrokni, and S. Muthukrishnan. Budget optimization for online campaigns with positive carryover effects. WINE-12, pp.86-99, Liverpool, 2012.","title":"Budget optimization for online campaigns with positive carryover effects","authors":"N Archak, V Mirrokni, S Muthukrishnan","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baird III, 1999] L. C. Baird III. Reinforcement Learning Through Gradient Descent. PhD thesis, US Air Force Academy, 1999.","title":"Reinforcement Learning Through Gradient Descent","authors":"L C Baird, Iii","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barto and Mahadevan, 2003] A. G. Barto and S. Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13(1-2):41-77, 2003.","title":"Recent advances in hierarchical reinforcement learning","authors":"A G Barto, S Mahadevan","year":"2003","venue":"Discrete Event Dynamic Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bellemare et al., 2016] M. G. Bellemare, G. Ostrovski, A. Guez, P. S. Thomas, and R. Munos. Increasing the action gap: New operators for reinforcement learning. AAAI-16, pp.1476-1483, 2016.","title":"Increasing the action gap: New operators for reinforcement learning","authors":"M G Bellemare, G Ostrovski, A Guez, P S Thomas, R Munos","year":"2016","venue":"AAAI-16","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boyen and Koller, 1998] X. Boyen and D. Koller. Tractable inference for complex stochastic processes. UAI-98, pp.33- 42, 1998.","title":"Tractable inference for complex stochastic processes","authors":"Koller Boyen, X Boyen, D Koller","year":"1998","venue":"UAI-98","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Choi et al., 2018] S. Choi, H. Ha, U. Hwang, C. Kim, J. Ha, S. Yoon. Reinforcement learning-based recommender sys- tem using biclustering technique. arXiv:1801.05532, 2018.","title":"Reinforcement learning-based recommender system using biclustering technique","authors":"S Choi, H Ha, U Hwang, C Kim, J Ha, S Yoon","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Corless et al., 1996] R. M. Corless, G. H. Gonnet, D. E. G. Hare, D. J. Jeffrey, and D. E. Knuth. On the Lambert W function. Adv. in Comp. Math., 5(1):329-359, 1996.","title":"On the Lambert W function","authors":"R M Corless, G H Gonnet, D E G Hare, D J Jeffrey, D E Knuth","year":"1996","venue":"Adv. in Comp. Math","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Daniel et al., 2016] C. Daniel, G. Neumann, O. Kroemer, and J. Peters. Hierarchical relative entropy policy search. JMLR, 17(93):1-50, 2016.","title":"Hierarchical relative entropy policy search","authors":"C Daniel, G Neumann, O Kroemer, J Peters","year":"2016","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Downey et al., 2017] C. Downey, A. Hefny, B. Boots, G. J. Gordon, and B. Li. Predictive state recurrent neural net- works. NIPS-17, pp.6053-6064. 2017.","title":"Predictive state recurrent neural networks","authors":"C Downey, A Hefny, B Boots, G J Gordon, B Li","year":"2017","venue":"NIPS-17","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Everitt et al., 2017] T. Everitt, V. Krakovna, L. Orseau, and S. Legg. Reinforcement learning with a corrupted reward channel. IJCAI-17, pp.4705-4713, Melbourne, 2017.","title":"Reinforcement learning with a corrupted reward channel","authors":"T Everitt, V Krakovna, L Orseau, S Legg","year":"2017","venue":"IJCAI-17","doi":null,"url":null,"internal_url":"/proceedings/2017/656"},{"position":11,"raw":"[Farahmand, 2011] A. Farahmand. Action-gap phenomenon in reinforcement learning. NIPS-11, pp.172-180.","title":"Action-gap phenomenon in reinforcement learning","authors":"A Farahmand","year":null,"venue":"NIPS-11","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fox et al., 2016] R. Fox, A. Pakman, and N. Tishby. Taming the noise in reinforcement learning via soft updates. UAI- 16, pp.1889-1897, New York, 2016.","title":"Taming the noise in reinforcement learning via soft updates","authors":"R Fox, A Pakman, N Tishby","year":"2016","venue":"UAI-16","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Francois-Lavet et al., 2017] V. Francois-Lavet, G. Rabusseau, J. Pineau, D. Ernst, and R. Fonteneau. On overfitting and asymptotic bias in batch reinforcement learning with partial observability. JAIR, 65:1-30, 2019.","title":"On overfitting and asymptotic bias in batch reinforcement learning with partial observability","authors":"V Francois-Lavet, G Rabusseau, J Pineau, D Ernst, R Fonteneau","year":"2019","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gauci et al., 2018] J. Gauci, E. Conti, Y. Liang, K. Vi- rochsiri, Y. He, Z. Kaden, V. Narayanan, and X. Ye. Hori- zon: Facebook's open source applied reinforcement learn- ing platform. arXiv:1312.5602, 2018.","title":"Facebook's open source applied reinforcement learning platform","authors":"J Gauci, E Conti, Y Liang, K Virochsiri, Y He, Z Kaden, V Narayanan, X Ye, Horizon","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hauskrecht et al., 1998] M. Hauskrecht, N. Meuleau, L. P. Kaelbling, T. Dean, and C. Boutilier. Hierarchical solution of Markov decision processes using macro-actions. UAI-98, pp.220-229, Madison, WI, 1998.","title":"Hierarchical solution of Markov decision processes using macro-actions","authors":"M Hauskrecht, N Meuleau, L P Kaelbling, T Dean, C Boutilier","year":"1998","venue":"UAI-98","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hohnhold et al., 2015] H. Hohnhold, D. O'Brien, and D. Tang. Focusing on the long-term: It's good for users and business. KDD-15, pp.1849-1858, Sydney, 2015.","title":"Focusing on the long-term: It's good for users and business","authors":"H Hohnhold, D O'brien, D Tang","year":"2015","venue":"KDD-15","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ie et al., 2019] E. Ie, V. Jain, J. Wang, S. Navrekar, R. Agar- wal, R. Wu, H. Cheng, T. Chandra, and C. Boutilier. SlateQ: A tractable decomposition for reinforcement learning with recommendation sets. IJCAI-19, 2019. This volume.","title":"SlateQ: A tractable decomposition for reinforcement learning with recommendation sets","authors":"E Ie, V Jain, J Wang, S Navrekar, R Agarwal, R Wu, H Cheng, T Chandra, C Boutilier","year":"2019","venue":"IJCAI-19","doi":null,"url":null,"internal_url":"/proceedings/2019/360"},{"position":18,"raw":"[Jaber, 2006] M. Y Jaber. Learning and forgetting models and their applications. Handbook of Industrial and Systems Engineering, 30(1):30-127, 2006.","title":"Learning and forgetting models and their applications","authors":"M Y Jaber","year":"2006","venue":"Handbook of Industrial and Systems Engineering","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Littman and Sutton, 2002] M. L. Littman and R. S. Sutton. Predictive representations of state. NIPS-02, pp.1555-1561, Vancouver, 2002.","title":"Predictive representations of state","authors":"M L Littman, R S Sutton","year":"2002","venue":"NIPS-02","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mladenov et al., 2017] M. Mladenov, C. Boutilier, D. Schu- urmans, O. Meshi, G. Elidan, T. Lu. Logistic Markov decision processes. IJCAI-17, pp.2486-2493, 2017.","title":"Logistic Markov decision processes","authors":"M Mladenov, C Boutilier, D Schuurmans, O Meshi, G Elidan, T Lu","year":"2017","venue":"IJCAI-17","doi":null,"url":null,"internal_url":"/proceedings/2017/346"},{"position":21,"raw":"[Mladenov et al., 2019] M. Mladenov, O. Meshi, J. Ooi, D. Schuurmans, and C. Boutilier. Advantage ampli- fication in slowly evolving latent-state environments. arXiv:1905.13559 , 2019.","title":"Advantage amplification in slowly evolving latent-state environments","authors":"M Mladenov, O Meshi, J Ooi, D Schuurmans, C Boutilier","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nachum et al., 2017] O. Nachum, M. Norouzi, K. Xu, and D. Schuurmans. Bridging the gap between value and policy based reinforcement learning. NIPS-17, pp.1476-1483, Long Beach, CA, 2017.","title":"Bridging the gap between value and policy based reinforcement learning","authors":"O Nachum, M Norouzi, K Xu, D Schuurmans","year":"2017","venue":"NIPS-17","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Parr, 1998] R. Parr. Flexible decomposition algorithms for weakly coupled Markov decision processes. UAI-98, pp.422-430, Madison, WI, 1998.","title":"Flexible decomposition algorithms for weakly coupled Markov decision processes","authors":"R Parr","year":"1998","venue":"UAI-98","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Roberts and Tedrake, 2009] J. W. Roberts and R. Tedrake. Signal-to-noise ratio analysis of policy gradient algorithms. NIPS-09, pp.1361-1368, Vancouver, 2009.","title":"Signal-to-noise ratio analysis of policy gradient algorithms","authors":"J W Roberts, R Tedrake","year":"2009","venue":"NIPS-09","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schulman et al., 2015] J. Schulman, S. L., P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. ICML-15, pp.1889-1897, Sydney, 2015.","title":"Trust region policy optimization","authors":"J Schulman, S L, P Abbeel, M I Jordan, P Moritz","year":"2015","venue":"ICML-15","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shani et al., 2005] G. Shani, D. Heckerman, and R. Brafman. An MDP-based recommender system. JMLR, 6:1265-1295, 2005.","title":"An MDP-based recommender system","authors":"G Shani, D Heckerman, R Brafman","year":"2005","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sharma et al., 2017] S. Sharma, A. S. Lakshminarayanan, and B. Ravindran. Learning to repeat: Fine-grained ac- tion repetition for deep reinforcement learning. ICLR-17, Toulon, France, 2017.","title":"Learning to repeat: Fine-grained action repetition for deep reinforcement learning","authors":"S Sharma, A S Lakshminarayanan, B Ravindran","year":"2017","venue":"ICLR-17","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Silvestrini and Veredas, 2008] A. Silvestrini, D. Veredas. Temporal aggregation of univariate and multivariate time series models: A survey. J. Econ. Surv., 22:458-497, 2008.","title":"Temporal aggregation of univariate and multivariate time series models: A survey","authors":"A Silvestrini, D Veredas","year":"2008","venue":"J. Econ. Surv","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Singh et al., 1994] S. Singh, T. Jaakkola, and M. Jordan. Learning without state-estimation in partially observable Markovian decision processes. ICML94, pp.284-292, 1994.","title":"Learning without state-estimation in partially observable Markovian decision processes","authors":"S Singh, T Jaakkola, M Jordan","year":"1994","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Smallwood and Sondik, 1973] R. Smallwood, E. Sondik. The optimal control of partially observable Markov pro- cesses over a finite horizon. Op. Res., 21:1071-1088, 1973.","title":"The optimal control of partially observable Markov processes over a finite horizon","authors":"R Smallwood, E Sondik","year":"1973","venue":"Op. Res","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sutton et al., 1999] R. S. Sutton, D. Precup, and S. P. Singh. Between MDPs and Semi-MDPs: Learning, planning, and representing knowledge at multiple temporal scales. Artif. Intel., 112:181-211, 1999.","title":"Between MDPs and Semi-MDPs: Learning, planning, and representing knowledge at multiple temporal scales","authors":"R S Sutton, D Precup, S P Singh","year":"1999","venue":"Artif. Intel","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Taghipour et al., 2007] N. Taghipour, A. Kardan, and S. S. Ghidary. Usage-based web recommendations: A reinforce- ment learning approach. RecSys07, pp.113-120, 2007.","title":"Usage-based web recommendations: A reinforcement learning approach","authors":"N Taghipour, A Kardan, S S Ghidary","year":"2007","venue":"RecSys","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Theocharous et al., 2015] G. Theocharous, P. S. Thomas, and M. Ghavamzadeh. Personalized ad recommendation systems for life-time value optimization with guarantees. IJCAI-15, pp.1806-1812, 2015.","title":"Personalized ad recommendation systems for life-time value optimization with guarantees","authors":"G Theocharous, P S Thomas, M Ghavamzadeh","year":"2015","venue":"IJCAI-15","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Thodoroff et al., 2018] P. Thodoroff, A. Durand, J. Pineau, and D. Precup. Temporal regularization for Markov deci- sion process. NeurIPS-18, pp.1784-1794, Montreal, 2018.","title":"Temporal regularization for Markov decision process","authors":"P Thodoroff, A Durand, J Pineau, D Precup","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Thurstone, 1919] L. L. Thurstone. The learning curve equa- tion. Psychological Monographs, 26(3):i, 1919.","title":"The learning curve equation","authors":"L L Thurstone","year":"1919","venue":"Psychological Monographs","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wilhelm et al., 2018] M. Wilhelm, A. Ramanathan, A. Bonomo, S. Jain, E. H. Chi, and J. Gillenwater. Practical di- versified recommendations on youtube with determinantal point processes. CIKM18, pp.2165-2173, Torino, 2018.","title":"Practical diversified recommendations on youtube with determinantal point processes","authors":"M Wilhelm, A Ramanathan, A Bonomo, S Jain, E H Chi, J Gillenwater","year":"2018","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhao et al., 2018] X. Zhao, L. Xia, L. Zhang, Z. Ding, D. Yin, and J. Tang. Deep reinforcement learning for page- wise recommendations. RecSys-18, pp.95-103, 2018.","title":"Deep reinforcement learning for pagewise recommendations","authors":"Zhao","year":"2018","venue":"RecSys-18","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}