{"type":"ScholarlyArticle","title":"Unobserved Is Not Equal to Non-existent: Using Gaussian Processes to Infer Immediate Rewards Across Contexts","authors":[{"name":"Hamoon Azizsoltani","slug":"hamoon-azizsoltani","given":"Hamoon","family":"Azizsoltani","affiliation":"North Carolina State University"},{"name":"Yeo Jin Kim","slug":"yeo-jin-kim","given":"Yeo Jin","family":"Kim","affiliation":"North Carolina State University"},{"name":"Markel Sanz Ausin","slug":"markel-sanz-ausin","given":"Markel Sanz","family":"Ausin","affiliation":"North Carolina State University"},{"name":"Tiffany Barnes","slug":"tiffany-barnes","given":"Tiffany","family":"Barnes","affiliation":"North Carolina State University"},{"name":"Min Chi","slug":"min-chi","given":"Min","family":"Chi","affiliation":"North Carolina State University"}],"abstract":"Learning optimal policies in real-world domains with delayed rewards is a major challenge in Reinforcement Learning. We address the credit assignment problem by proposing a Gaussian Process (GP)-based immediate reward approximation algorithm and evaluate its effectiveness in 4 contexts where rewards can be delayed for long trajectories. In one GridWorld game and 8 Atari games, where immediate rewards are available, our results showed that on 7 out 9 games, the proposed GP-inferred reward policy performed at least as well as the immediate reward policy and significantly outperformed the corresponding delayed reward policy. In e-learning and healthcare applications, we combined GP-inferred immediate rewards with offline Deep Q-Network (DQN) policy induction and showed that the GP-inferred reward policies outperformed the policies induced using delayed rewards in both real-world contexts.","doi":"10.24963/ijcai.2019/273","url":"https://ijcai.org/proceedings/2019/273","pdf":"https://ijcai.org/proceedings/2019/0273.pdf","markdown":"https://ijcai.org/proceedings/2019/273.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/273","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":273,"pages":{"first":1974,"last":1980},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Abbeel and Ng, 2004] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learn- ing. In Proceedings of the twenty-first international con- ference on Machine learning, page 1. ACM, 2004.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Abbeel, ; Ng, Pieter Abbeel, Andrew Y Ng","year":"2004","venue":"Proceedings of the twenty-first international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Azizsoltani and Haldar, 2018] Hamoon Azizsoltani and Achintya Haldar. Reliability analysis of lead-free solders in electronic packaging using a novel surrogate model and kriging concept. Journal of Electronic Packaging, 140(4):041003, 2018.","title":"Reliability analysis of lead-free solders in electronic packaging using a novel surrogate model and kriging concept","authors":"Haldar Azizsoltani, Hamoon Azizsoltani, Achintya Haldar","year":"2018","venue":"Journal of Electronic Packaging","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Azizsoltani and Sadeghi, 2018] Hamoon Azizsoltani and Elham Sadeghi. Adaptive sequential strategy for risk esti- mation of engineering systems using gaussian process re- gression active learning. Engineering Applications of Ar- tificial Intelligence, 74:146-165, 2018.","title":"Adaptive sequential strategy for risk estimation of engineering systems using gaussian process regression active learning","authors":"Sadeghi Azizsoltani, Hamoon Azizsoltani, Elham Sadeghi","year":"2018","venue":"Engineering Applications of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Barto, 2013] Andrew G Barto. Intrinsic motivation and re- inforcement learning. In Intrinsically motivated learning in natural and artificial systems, pages 17-47. Springer, 2013.","title":"Intrinsic motivation and reinforcement learning","authors":"G Andrew, Barto","year":"2013","venue":"Intrinsically motivated learning in natural and artificial systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cressie, 1992] Noel Cressie. Statistics for spatial data. Terra Nova, 4(5):613-617, 1992.","title":"Noel Cressie. Statistics for spatial data","authors":"Cressie","year":"1992","venue":"Terra Nova","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cronbach and Snow, 1977] Lee J Cronbach and Richard E Snow. Aptitudes and instructional methods: A handbook for research on interactions. Irvington, 1977.","title":"Aptitudes and instructional methods: A handbook for research on interactions","authors":"J Lee, Richard E Cronbach, Snow","year":"1977","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Engel et al., 2003] Yaakov Engel, Shie Mannor, and Ron Meir. Bayes meets bellman: The gaussian process ap- proach to temporal difference learning. In ICML, 2003.","title":"Bayes meets bellman: The gaussian process approach to temporal difference learning","authors":"Yaakov Engel, Shie Mannor, Ron Meir","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Engel et al., 2005] Y. Engel, S. Mannor, and R. Meir. Re- inforcement learning with gaussian processes. In ICML, pages 201-208. ACM, 2005.","title":"Reinforcement learning with gaussian processes","authors":"Y Engel, S Mannor, R Meir","year":"2005","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gao and Toni, 2015] Yang Gao and Francesca Toni. Poten- tial based reward shaping for hierarchical reinforcement learning. In IJCAI, 2015.","title":"Potential based reward shaping for hierarchical reinforcement learning","authors":"Toni ; Gao, Yang Gao, Francesca Toni","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gimelfarb et al., 2018] Michael Gimelfarb, Scott Sanner, and Chi-Guhn Lee. Reinforcement learning with multi- ple experts: A bayesian model combination approach. In NeurIPS, 2018.","title":"Reinforcement learning with multiple experts: A bayesian model combination approach","authors":"Michael Gimelfarb, Scott Sanner, Chi-Guhn Lee","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Grześ and Kudenko, 2010] Marek Grześ and Daniel Ku- denko. Online learning of shaping rewards in reinforce- ment learning. Neural Networks, 23(4):541-550, 2010.","title":"Online learning of shaping rewards in reinforcement learning","authors":"Kudenko Grześ, Marek Grześ, Daniel Kudenko","year":"2010","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kim and Chi, 2018] Yeo Jin Kim and Min Chi. Temporal belief memory: Imputing missing data during RNN train- ing. In IJCAI, 2018.","title":"Temporal belief memory: Imputing missing data during RNN training","authors":"Jin Yeo, Min Kim, Chi","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/322"},{"position":15,"raw":"[Kulkarni et al., 2016] Tejas Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In NeurIPS, 2016.","title":"Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation","authors":"Tejas Kulkarni, Karthik Narasimhan, Ardavan Saeedi, Josh Tenenbaum","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kuss and Rasmussen, 2004] Malte Kuss and Carl E Ras- mussen. Gaussian processes in reinforcement learning. In NeurIPS, 2004.","title":"Gaussian processes in reinforcement learning","authors":"Rasmussen Kuss, Malte Kuss, Carl E Rasmussen","year":"2004","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 518(7540):529, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ng et al., 1999] Andrew Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, 1999.","title":"Policy invariance under reward transformations: Theory and application to reward shaping","authors":"Andrew Ng, Daishi Harada, Stuart Russell","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ng et al., 2000] Andrew Ng, Stuart J Russell, et al. Algo- rithms for inverse reinforcement learning. In ICML, 2000.","title":"Algorithms for inverse reinforcement learning","authors":"Andrew Ng, Stuart J Russell","year":"2000","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nychka et al., 2015] Douglas Nychka, Soutir Bandyopad- hyay, Dorit Hammerling, Finn Lindgren, and Stephan Sain. A multiresolution gaussian process model for the analysis of large spatial datasets. Journal of Computa- tional and Graphical Statistics, 24(2):579-599, 2015.","title":"A multiresolution gaussian process model for the analysis of large spatial datasets","authors":"Douglas Nychka, Soutir Bandyopadhyay, Dorit Hammerling, Finn Lindgren, Stephan Sain","year":"2015","venue":"Journal of Computational and Graphical Statistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Raghu et al., 2017] Aniruddh Raghu, Matthieu Ko- morowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:1711.09602, 2017.","title":"Deep reinforcement learning for sepsis treatment","authors":"Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, Marzyeh Ghassemi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rasmussen, 2003] Carl Edward Rasmussen. Gaussian pro- cesses in machine learning. In Summer School on Machine Learning, pages 63-71. Springer, 2003.","title":"Gaussian processes in machine learning","authors":"Carl Edward, Rasmussen","year":"2003","venue":"Summer School on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Renkl et al., 2002] Alexander Renkl, Robert K Atkinson, Uwe H Maier, and Richard Staley. From example study to problem solving: Smooth transitions help learning. The Journal of Experimental Education, 70(4):293-315, 2002.","title":"From example study to problem solving: Smooth transitions help learning","authors":"Alexander Renkl, Uwe H Robert K Atkinson, Richard Maier, Staley","year":"2002","venue":"The Journal of Experimental Education","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Russell, 1998] Stuart J Russell. Learning agents for uncer- tain environments. In COLT, pages 101-103, 1998.","title":"Learning agents for uncertain environments","authors":"Russell Stuart","year":"1998","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Schaul et al., 2015] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience re- play. arXiv preprint arXiv:1511.05952, 2015.","title":"Prioritized experience replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Schwonke et al., 2009] Rolf Schwonke, Alexander Renkl, Carmen Krieg, Jörg Wittwer, Vincent Aleven, and Ron Salden. The worked-example effect: Not an artefact of lousy control conditions. Computers in Human Behavior, 25(2):258-266, 2009.","title":"The worked-example effect: Not an artefact of lousy control conditions","authors":"Rolf Schwonke, Alexander Renkl, Carmen Krieg, Jörg Wittwer, Ron Vincent Aleven, Salden","year":"2009","venue":"Computers in Human Behavior","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Snow, 1991] Richard E Snow. Aptitude-treatment interac- tion as a framework for research on individual differences in psychotherapy. Journal of consulting and clinical psy- chology, 59(2):205, 1991.","title":"Aptitude-treatment interaction as a framework for research on individual differences in psychotherapy","authors":"Richard E Snow","year":"1991","venue":"Journal of consulting and clinical psychology","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with dou- ble q-learning. In AAAI, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wiewiora et al., 2003] Eric Wiewiora, Garrison W Cottrell, and Charles Elkan. Principled methods for advising rein- forcement learning agents. In ICML, 2003.","title":"Principled methods for advising reinforcement learning agents","authors":"Eric Wiewiora, W Garrison, Charles Cottrell, Elkan","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}