{"type":"ScholarlyArticle","title":"Logistic Markov Decision Processes","authors":[{"name":"Martin Mladenov","slug":"martin-mladenov","given":"Martin","family":"Mladenov","affiliation":"TU-Dortmund"},{"name":"Craig Boutilier","slug":"craig-boutilier","given":"Craig","family":"Boutilier","affiliation":"Google Research"},{"name":"Dale Schuurmans","slug":"dale-schuurmans","given":"Dale","family":"Schuurmans","affiliation":"University of Alberta"},{"name":"Ofer Meshi","slug":"ofer-meshi","given":"Ofer","family":"Meshi","affiliation":"Google Research"},{"name":"Gal Elidan","slug":"gal-elidan","given":"Gal","family":"Elidan","affiliation":"Google Research"},{"name":"Tyler Lu","slug":"tyler-lu","given":"Tyler","family":"Lu","affiliation":"Google Research"}],"abstract":"User modeling in advertising and recommendation has typically focused on myopic predictors of user responses. In this work, we consider the long-term decision problem associated with user interaction. We propose a concise specification of long-term interaction dynamics by combining factored dynamic Bayesian networks with logistic predictors of user responses, allowing state-of-the-art prediction models to be seamlessly extended. We show how to solve such models at scale by providing a constraint generation approach for approximate linear programming that overcomes the variable coupling and non-linearity induced by the logistic regression predictor. The efficacy of the approach is demonstrated on advertising domains with up to 2^54 states and 2^39 actions.","doi":"10.24963/ijcai.2017/346","url":"https://ijcai.org/proceedings/2017/346","pdf":"https://ijcai.org/proceedings/2017/0346.pdf","markdown":"https://ijcai.org/proceedings/2017/346.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/346","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":346,"pages":{"first":2486,"last":2493},"keywords":["Planning and Scheduling: Markov Decisions Processes","Machine Learning: Reinforcement Learning","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Amin et al., 2012] K. Amin, M. Kearns, P. Key, A. Schwaighofer. Budget optimization for sponsored search: Censored learning in MDPs. UAI-12, 543-553, 2012.","title":"Budget optimization for sponsored search: Censored learning in MDPs","authors":"K Amin, M Kearns, P Key, A Schwaighofer","year":"2012","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Archak et al., 2010] N. Archak, V. Mirrokni, and S. Muthukrishnan. Mining advertiser-specific user behavior using adfactors. WWW-10, 31-40, 2010.","title":"Mining advertiser-specific user behavior using adfactors","authors":"N Archak, V Mirrokni, S Muthukrishnan","year":"2010","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Archak et al., 2012] N. Archak, V. Mirrokni, and S. Muthukrishnan. Budget optimization for online campaigns with positive carryover effects. WINE-12, 86-99, 2012.","title":"Budget optimization for online campaigns with positive carryover effects","authors":"N Archak, V Mirrokni, S Muthukrishnan","year":"2012","venue":"WINE","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Boutilier and Lu, 2016] C. Boutilier and T. Lu. Budget allo- cation using weakly coupled, constrained Markov decision processes. UAI-16, 52-61, 2016.","title":"Budget allocation using weakly coupled, constrained Markov decision processes","authors":"C Boutilier, T Lu","year":"2016","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boutilier et al., 1995] C. Boutilier, R. Dearden, and M. Goldszmidt. Exploiting structure in policy construc- tion. IJCAI-95, 1104-1111, 1995.","title":"Exploiting structure in policy construction","authors":"C Boutilier, R Dearden, M Goldszmidt","year":"1995","venue":"IJCAI-95","doi":null,"url":null,"internal_url":"/proceedings/1995/143"},{"position":6,"raw":"[Boutilier et al., 1999] C. Boutilier, T. Dean, and S. Hanks. Decision theoretic planning: Structural assumptions and computational leverage. JAIR, 11:1-94, 1999.","title":"Decision theoretic planning: Structural assumptions and computational leverage","authors":"C Boutilier, T Dean, S Hanks","year":"1999","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Boyen and Koller, 1998] X. Boyen and D. Koller. Tractable inference for complex stochastic processes. UAI-98, 33- 42, 1998.","title":"Tractable inference for complex stochastic processes","authors":"Koller Boyen, X Boyen, D Koller","year":"1998","venue":"UAI-98","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Charikar et al., 1999] M. Charikar, R. Kumar, P. Raghavan, S. Rajagopalan, and A. Tomkins. On targeting Markov segments. STOC-99, 99-108, 1999.","title":"On targeting Markov segments","authors":"M Charikar, R Kumar, P Raghavan, S Rajagopalan, A Tomkins","year":"1999","venue":"STOC-99","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cheng et al., 2016] H. Cheng, L. Koc, J. Harmsen, T. Shaked, T. Chandra, H. Aradhye, G. Anderson, G. Cor- rado, W. Chai, M. Ispir, R. Anil, Z. Haque, L. Hong, V. Jain, X. Liu, H. Shah. Wide & deep learning for rec- ommender systems. Deep Learning for Rec. Sys., 7-10, 2016.","title":"Wide & deep learning for recommender systems","authors":"H Cheng, L Koc, J Harmsen, T Shaked, T Chandra, H Aradhye, G Anderson, G Corrado, W Chai, M Ispir, R Anil, Z Haque, L Hong, V Jain, X Liu, H Shah","year":"2016","venue":"Deep Learning for Rec. Sys","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Covington et al., 2016] P. Covington, J. Adams, and E. Sar- gin. Deep neural networks for YouTube recommendations. RecSys-16, 191-198, 2016.","title":"Deep neural networks for YouTube recommendations","authors":"P Covington, J Adams, E Sargin","year":"2016","venue":"RecSys-16","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dean and Kanazawa, 1989] T. Dean and K. Kanazawa. A model for reasoning about persistence and causation. Comput. Intell., 5(3):142-150, 1989.","title":"A model for reasoning about persistence and causation","authors":"T Dean, K Kanazawa","year":"1989","venue":"Comput. Intell","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Graepel et al., 2010] T. Graepel, J. Candela, T. Borchert, and R. Herbrich. Web-scale Bayesian click-through rate prediction for sponsored search advertising in Microsoft's Bing search engine. ICML-10, 13-20, 2010.","title":"Web-scale Bayesian click-through rate prediction for sponsored search advertising in Microsoft's Bing search engine","authors":"T Graepel, J Candela, T Borchert, R Herbrich","year":"2010","venue":"ICML-10","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Guestrin et al., 2003] C. Guestrin, D. Koller, R. Parr, and S. Venkataraman. Efficient solution algorithms for fac- tored MDPs. JAIR, 19:399-468, 2003.","title":"Efficient solution algorithms for factored MDPs","authors":"C Guestrin, D Koller, R Parr, S Venkataraman","year":"2003","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He and McAuley, 2016] R. He and J. McAuley. Fusing sim- ilarity models with Markov chains for sparse sequential recommendation. ICDM-16, 2016.","title":"Fusing similarity models with Markov chains for sparse sequential recommendation","authors":"R He, J Mcauley","year":"2016","venue":"ICDM-16","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hoey et al., 1999] J. Hoey, R. St-Aubin, A. Hu, and C. Boutilier. SPUDD: Stochastic planning using decision diagrams. UAI-99, 279-288, 1999.","title":"SPUDD: Stochastic planning using decision diagrams","authors":"J Hoey, R St-Aubin, A Hu, C Boutilier","year":"1999","venue":"UAI-99","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hohnhold et al., 2015] H. Hohnhold, D. O'Brien, and D. Tang. Focusing on the long-term: It's good for users and business. KDD-15, 1849-1858, 2015.","title":"Focusing on the long-term: It's good for users and business","authors":"H Hohnhold, D O'brien, D Tang","year":"2015","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2009] T. Li, N. Liu, J. Yan, G. Wang, F. Bai, and Z. Chen. A Markov chain model for integrating behav- ioral targeting into contextual advertising. Data Mining and Audience Intel. for Advert., 1-9, 2009.","title":"A Markov chain model for integrating behavioral targeting into contextual advertising","authors":"T Li, N Liu, J Yan, G Wang, F Bai, Z Chen","year":"2009","venue":"Data Mining and Audience Intel. for Advert","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[McMahan et al., 2013] H. McMahan, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady, L. Nie, T. Phillips, E. Davy- dov, D. Golovin, S. Chikkerur, D. Liu, A. Wattenberg, M. Hrafnkelsson, T. Boulos, J. Kubica. Ad click predic- tion: A view from the trenches. KDD, 1222-1230, 2013.","title":"Ad click prediction: A view from the trenches","authors":"H Mcmahan, G Holt, D Sculley, M Young, D Ebner, J Grady, L Nie, T Phillips, E Davydov, D Golovin, S Chikkerur, D Liu, A Wattenberg, M Hrafnkelsson, T Boulos, J Kubica","year":"2013","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[de Farias and Van Roy, 2003] D. P. de Farias, B. Van Roy. The linear programming approach to approximate dy- namic programming. Oper. Res., 51(6):850-865, 2003.","title":"The linear programming approach to approximate dynamic programming","authors":"Van De Farias, ; Roy, D P De Farias, B Van Roy","year":"2003","venue":"Oper. Res","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Puterman, 1994] M. Puterman. Markov Decision Pro- cesses: Discrete Stochastic Dyn. Prog. Wiley, 1994.","title":"Markov Decision Processes: Discrete Stochastic Dyn","authors":"M Puterman","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rendle et al., 2010] S. Rendle, C. Freudenthaler, and L. Schmidt-Thieme. Factorizing personalized Markov chains for next-basket recommendation. WWW-10, 811- 820, 2010.","title":"Factorizing personalized Markov chains for next-basket recommendation","authors":"S Rendle, C Freudenthaler, L Schmidt-Thieme","year":"2010","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Richardson et al., 2007] M. Richardson, E. Dominowska, and R. Ragno. Predicting clicks: Estimating the click- through rate for new ads. WWW-07, 521-530, 2007.","title":"Predicting clicks: Estimating the clickthrough rate for new ads","authors":"M Richardson, E Dominowska, R Ragno","year":"2007","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Robbel et al., 2016] P. Robbel, F. Oliehoek, and M. Kochenderfer. Exploiting anonymity in approxi- mate linear programming: Scaling to large multiagent mdps. AAAI-16, 2537-2543, 2016.","title":"Exploiting anonymity in approximate linear programming: Scaling to large multiagent mdps","authors":"P Robbel, F Oliehoek, M Kochenderfer","year":"2016","venue":"AAAI-16","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sahoo et al., 2012] N. Sahoo, P. Singh, and T. Mukhopad- hyay. A hidden Markov model for collaborative filtering. Mgmt. Info. Sys. Qrtly., 36(4), 2012.","title":"A hidden Markov model for collaborative filtering","authors":"N Sahoo, P Singh, T Mukhopadhyay","year":"2012","venue":"Mgmt. Info. Sys. Qrtly","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schuurmans and Patrascu, 2001] D. Schuurmans and R. Pa- trascu. Direct value approximation for factored MDPs. NIPS-01, 1579-1586, 2001.","title":"Direct value approximation for factored MDPs","authors":"D Schuurmans, R Patrascu","year":"2001","venue":"NIPS-01","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shani et al., 2005] G. Shani, D. Heckerman, and R. Braf- man. An MDP-based recommender system. JMLR, 6:1265-1295, 2005.","title":"An MDP-based recommender system","authors":"G Shani, D Heckerman, R Brafman","year":"2005","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Silver et al., 2013] D. Silver, L. Newnham, D. Barker, S. Weller, and J. McFall. Concurrent reinforcement learn- ing from customer interactions. ICML-13, 924-932, 2013.","title":"Concurrent reinforcement learning from customer interactions","authors":"D Silver, L Newnham, D Barker, S Weller, J Mcfall","year":"2013","venue":"ICML-13","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[St-Aubin et al., 2000] R. St-Aubin, J. Hoey, C. Boutilier. APRICODD: Approximate policy construction using de- cision diagrams. NIPS-00, 1089-1095, 2000.","title":"APRICODD: Approximate policy construction using decision diagrams","authors":"R St-Aubin, J Hoey, C Boutilier","year":"2000","venue":"NIPS-00","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Taghipour et al., 2007] N. Taghipour, A. Kardan, and S. Ghidary. Usage-based web recommendations: A rein- forcement learning approach. RecSys-07, 113-120, 2007.","title":"Usage-based web recommendations: A reinforcement learning approach","authors":"N Taghipour, A Kardan, S Ghidary","year":"2007","venue":"RecSys-07","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tan et al., 2016] Y. Tan, X. Xu, and Y. Liu. Improved recur- rent neural networks for session-based recommendations. Deep Learning for Rec. Sys., 17-22, 2016.","title":"Improved recurrent neural networks for session-based recommendations","authors":"Y Tan, X Xu, Y Liu","year":"2016","venue":"Deep Learning for Rec. Sys","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Theocharous et al., 2015] G. Theocharous, P. Thomas, and M. Ghavamzadeh. Personalized ad recommendation sys- tems for life-time value optimization with guarantees. IJCAI-15, 1806-1812, 2015.","title":"Personalized ad recommendation systems for life-time value optimization with guarantees","authors":"G Theocharous, P Thomas, M Ghavamzadeh","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/257"},{"position":32,"raw":"[Wu et al., 2017] C. Wu, A. Ahmed, A. Beutel, A. Smola, and H. Jing. Recurrent recommender networks. WSDM- 17, 2017.","title":"Recurrent recommender networks","authors":"C Wu, A Ahmed, A Beutel, A Smola, H Jing","year":"2017","venue":"WSDM-17","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}