{"type":"ScholarlyArticle","title":"Sample Efficient Policy Search for Optimal Stopping Domains","authors":[{"name":"Karan Goel","slug":"karan-goel","given":"Karan","family":"Goel","affiliation":"Carnegie Mellon University"},{"name":"Christoph Dann","slug":"christoph-dann","given":"Christoph","family":"Dann","affiliation":"Carnegie Mellon University"},{"name":"Emma Brunskill","slug":"emma-brunskill","given":"Emma","family":"Brunskill","affiliation":"Stanford University"}],"abstract":"Optimal stopping problems consider the question of deciding when to stop an observation-generating process in order to maximize a return. We examine the problem of simultaneously learning and planning in such domains, when data is collected directly from the environment. We propose GFSE, a simple and flexible model-free policy search method that reuses data for sample efficiency by leveraging problem structure. We bound the sample complexity of our approach to guarantee uniform convergence of policy value estimates, tightening existing PAC bounds to achieve logarithmic dependence on horizon length for our setting. We also examine the benefit of our method against prevalent model-based and model-free approaches on 3 domains taken from diverse fields.","doi":"10.24963/ijcai.2017/237","url":"https://ijcai.org/proceedings/2017/237","pdf":"https://ijcai.org/proceedings/2017/0237.pdf","markdown":"https://ijcai.org/proceedings/2017/237.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/237","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra, IIIA-CSIC","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":237,"pages":{"first":1711,"last":1717},"keywords":["Machine Learning: Reinforcement Learning","Uncertainty in AI: Markov Decision Processes","Uncertainty in AI: Sequential Decision Making"],"videos":[],"references":[{"position":1,"raw":"[Best et al., 2015] Graeme Best, Wolfram Martens, and Robert Fitch. A spatiotemporal optimal stopping prob- lem for mission monitoring with stationary viewpoints. In Robotics: Science and Systems, 2015.","title":"A spatiotemporal optimal stopping problem for mission monitoring with stationary viewpoints","authors":"Graeme Best, Wolfram Martens, Robert Fitch","year":"2015","venue":"Robotics: Science and Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Corbett and Anderson, 1995] Albert T. Corbett and John R. Anderson. Knowledge tracing: Modelling the acquisition of procedural knowledge. User Model. User-Adapt. Inter- act., 4(4):253-278, 1995.","title":"Knowledge tracing: Modelling the acquisition of procedural knowledge","authors":"Albert T Corbett, John R Anderson","year":"1995","venue":"User Model. User-Adapt. Interact","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deisenroth and Rasmussen, 2011] Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data- efficient approach to policy search. In ICML, pages 465-472, 2011.","title":"Pilco: A model-based and dataefficient approach to policy search","authors":"Rasmussen Deisenroth, Marc Deisenroth, Carl E Rasmussen","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Draney et al., 1995] Karen L Draney, Peter Pirolli, and Mark Wilson. A measurement model for a complex cogni- tive skill. Cognitively diagnostic assessment, pages 103- 125, 1995.","title":"A measurement model for a complex cognitive skill","authors":"Karen L Draney, Peter Pirolli, Mark Wilson","year":"1995","venue":"Cognitively diagnostic assessment","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Etzioni et al., 2003] Oren Etzioni, Rattapoom Tuchinda, Craig A. Knoblock, and Alexander Yates. To buy or not to buy: mining airfare data to minimize ticket purchase price. In KDD, pages 119-128, 2003.","title":"To buy or not to buy: mining airfare data to minimize ticket purchase price","authors":"Oren Etzioni, Rattapoom Tuchinda, Craig A Knoblock, Alexander Yates","year":"2003","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Feldstein and Rothschild, 1974] Martin S Feldstein and Michael Rothschild. Towards an economic theory of re- placement investment. Econometrica: Journal of the Econometric Society, pages 393-423, 1974.","title":"Towards an economic theory of replacement investment","authors":"S Martin, Michael Feldstein, Rothschild","year":"1974","venue":"Econometrica: Journal of the Econometric Society","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feng and Gallego, 1995] Youyi Feng and Guillermo Gal- lego. Optimal starting times for end-of-season sales and optimal stopping times for promotional fares. Manage- ment Science, 41(8):1371-1391, 1995.","title":"Optimal starting times for end-of-season sales and optimal stopping times for promotional fares","authors":"Gallego Feng","year":"1995","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ferguson, 1989] Thomas S Ferguson. Who solved the sec- retary problem? Statistical science, pages 282-289, 1989.","title":"Who solved the secretary problem? Statistical science","authors":"Thomas S Ferguson","year":"1989","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Glower et al., 1998] Michel Glower, Donald R Haurin, and Patric H Hendershott. Selling time and selling price: The influence of seller motivation. Real estate economics, 26(4):719-740, 1998.","title":"Selling time and selling price: The influence of seller motivation","authors":"Michel Glower, Patric H Donald R Haurin, Hendershott","year":"1998","venue":"Real estate economics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Groves and Gini, 2015] William Groves and Maria L. Gini. On optimizing airline ticket purchase timing. ACM TIST, 7(1):3:1-3:28, 2015.","title":"On optimizing airline ticket purchase timing","authors":"William Groves, Maria L Gini","year":"2015","venue":"ACM TIST","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jacka, 1991] SD","title":null,"authors":"Jacka","year":"1991","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Jacka. Optimal stopping and the ameri- can put. Mathematical Finance, 1(2):1-14, 1991.","title":"Optimal stopping and the american put","authors":"Jacka","year":"1991","venue":"Mathematical Finance","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jiang and Powell, 2015] Daniel R. Jiang and Warren B. Powell. An approximate dynamic programming algo- rithm for monotone value functions. Operations Research, 63(6):1489-1511, 2015.","title":"An approximate dynamic programming algorithm for monotone value functions","authors":"R Daniel, Warren B Jiang, Powell","year":"2015","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kearns et al., 1999] Michael J. Kearns, Yishay Mansour, and Andrew Y. Ng. Approximate planning in large pomdps via reusable trajectories. In NIPS, pages 1001-1007, 1999.","title":"Approximate planning in large pomdps via reusable trajectories","authors":"J Michael, Yishay Kearns, Andrew Y Mansour, Ng","year":"1999","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Koedinger et al., 2013] Kenneth R Koedinger, Emma Brun- skill, Ryan SJd Baker, Elizabeth A McLaughlin, and John Stamper. New potentials for data-driven intelligent tutor- ing system development and optimization. AI Magazine, 34(3):27-41, 2013.","title":"New potentials for data-driven intelligent tutoring system development and optimization","authors":"Emma Kenneth R Koedinger, Ryan Sjd Brunskill, Elizabeth A Baker, John Mclaughlin, Stamper","year":"2013","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Levine and Abbeel, 2014] Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In NIPS, pages 1071-1079, 2014.","title":"Learning neural network policies with guided policy search under unknown dynamics","authors":"Abbeel Levine, Sergey Levine, Pieter Abbeel","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lippman and McCall, 1976] Steven A Lippman and John McCall. The economics of job search: A survey. Eco- nomic inquiry, 14(2):155-189, 1976.","title":"The economics of job search: A survey","authors":"A Steven, John Lippman, Mccall","year":"1976","venue":"Economic inquiry","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mandel et al., 2014] Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline pol- icy evaluation across representations with applications to educational games. In AAMAS, pages 1077-1084. Interna- tional Foundation for Autonomous Agents and Multiagent Systems, 2014.","title":"Offline policy evaluation across representations with applications to educational games","authors":"Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, Zoran Popovic","year":"2014","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mordecki, 2002] Ernesto Mordecki. Optimal stopping and perpetual options for lévy processes. Finance and Stochas- tics, 6(4):473-493, 2002.","title":"Optimal stopping and perpetual options for lévy processes","authors":"Ernesto Mordecki","year":"2002","venue":"Finance and Stochastics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ng and Jordan, 2000] Andrew Y. Ng and Michael I. Jordan. PEGASUS: A policy search method for large mdps and pomdps. In UAI, 2000.","title":"PEGASUS: A policy search method for large mdps and pomdps","authors":"Andrew Y Ng, Michael I Jordan","year":"2000","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peskir and Shiryaev, 2006] Goran Peskir and Albert Shiryaev. Optimal stopping and free-boundary problems. Springer, 2006.","title":"Optimal stopping and free-boundary problems","authors":"Shiryaev Peskir, Goran Peskir, Albert Shiryaev","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Precup, 2000] Doina Precup. Eligibility traces for off- policy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.","title":"Eligibility traces for offpolicy policy evaluation","authors":"Doina Precup","year":"2000","venue":"Computer Science Department Faculty Publication Series","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ritter et al., 2009] Steven Ritter, Thomas K. Harris, Tristan Nixon, Daniel Dickison, R. Charles Murray, and Brendon Towle. Reducing the knowledge tracing space. In Educa- tional Data Mining, pages 151-160, 2009.","title":"Reducing the knowledge tracing space","authors":"Steven Ritter, Thomas K Harris, Tristan Nixon, R Charles Daniel Dickison, Brendon Murray, Towle","year":"2009","venue":"Educational Data Mining","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rust, 1987] John Rust. Optimal replacement of gmc bus engines: An empirical model of harold zurcher. Econo- metrica: Journal of the Econometric Society, pages 999- 1033, 1987.","title":"Optimal replacement of gmc bus engines: An empirical model of harold zurcher","authors":"John Rust","year":"1987","venue":"Econometrica: Journal of the Econometric Society","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sutton et al., 1999] Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS, pages 1057-1063, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"Richard S Sutton, David A Mcallester, P Satinder, Yishay Singh, Mansour","year":"1999","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tsitsiklis and Van Roy, 1999] John N Tsitsiklis and Ben- jamin Van Roy. Optimal stopping of markov processes: Hilbert space theory, approximation algorithms, and an ap- plication to pricing high-dimensional financial derivatives. IEEE Transactions on Automatic Control, 44(10):1840- 1851, 1999.","title":"Optimal stopping of markov processes: Hilbert space theory, approximation algorithms, and an application to pricing high-dimensional financial derivatives","authors":"Van Tsitsiklis, ; Roy, N John, Benjamin Tsitsiklis, Roy Van","year":"1999","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vapnik and Kotz, 1982] Vladimir Naumovich Vapnik and Samuel Kotz. Estimation of dependences based on empir- ical data, volume 40. Springer-Verlag New York, 1982.","title":"Estimation of dependences based on empirical data","authors":"Kotz ; Vapnik, Samuel Vapnik, Kotz","year":"1982","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wilson et al., 2014] Aaron Wilson, Alan Fern, and Prasad Tadepalli. Using trajectory data to improve bayesian opti- mization for reinforcement learning. Journal of Machine Learning Research, 15(1):253-282, 2014.","title":"Using trajectory data to improve bayesian optimization for reinforcement learning","authors":"Aaron Wilson, Alan Fern, Prasad Tadepalli","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yelp, 2016] Yelp. Metric optimization engine. https:// github.com/Yelp/MOE, 2016.","title":"Yelp. Metric optimization engine","authors":"Yelp","year":"2016","venue":null,"doi":null,"url":"https://github.com/Yelp/MOE","internal_url":null},{"position":30,"raw":"[Zilberstein, 1995] Shlomo Zilberstein. Operational ratio- nality through compilation of anytime algorithms. AI Mag- azine, 16(2):79, 1995.","title":"Shlomo Zilberstein. Operational rationality through compilation of anytime algorithms","authors":"Zilberstein","year":"1995","venue":"AI Magazine","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}