{"type":"ScholarlyArticle","title":"Learning Policies for Embodied Virtual Agents through Demonstration","authors":[{"name":"Jonathan Dinerstein","slug":"jonathan-dinerstein","given":"Jonathan","family":"Dinerstein","affiliation":"CS Department , Brigham Young University"},{"name":"Parris K. Egbert","slug":"parris-k-egbert","given":"Parris K.","family":"Egbert","affiliation":"CS Department , Brigham Young University"},{"name":"Dan Ventura","slug":"dan-ventura","given":"Dan","family":"Ventura","affiliation":"CS Department , Brigham Young University"}],"abstract":"Although many powerful AI and machine learning techniques exist, it remains difficult to quickly create AI for embodied virtual agents that produces visually lifelike behavior. This is important for applications (e.g., games, simulators, interactive displays) where an agent must behave in a manner that appears human-like. We present a novel technique for learning reactive policies that mimic demonstrated human behavior. The user demonstrates the desired behavior by dictating the agent's actions during an interactive animation. Later, when the agent is to behave autonomously, the recorded data is generalized to form a continuous state-to-action mapping. Combined with an appropriate animation algorithm (e.g., motion capture), the learned policies realize stylized and natural-looking agent behavior. We empirically demonstrate the efficacy of our technique for quickly producing policies which result in lifelike virtual agent behavior.","doi":null,"url":"https://ijcai.org/proceedings/2007/203","pdf":"https://ijcai.org/proceedings/2007/0203.pdf","markdown":"https://ijcai.org/proceedings/2007/203.md","bibtex":"https://ijcai.org/proceedings/2007/bibtex/203","volume":{"slug":"2007","title":"Proceedings of the Twentieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2007","url":"https://ijcai.org/proceedings/2007/","year":2007,"location":"Hyderabad, India","dates":"","editors":"Ramon López de Mántaras","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":null,"track":"","number":203,"pages":{"first":1257,"last":1261},"keywords":[],"videos":[],"references":[{"position":1,"raw":"[Angros et al., 2002] R Angros, L Johnson, J Rickel, and A Scholer. Learning domain knowledge for teaching pro- cedural skills. In Proceedings of AAMAS'02, 2002.","title":"Learning domain knowledge for teaching procedural skills","authors":"Angros, J Johnson, Rickel, Scholer","year":"2002","venue":"Proceedings of AAMAS'02","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Badler et al., 1999] N Badler, M Palmer, and R Bindi- ganavale. Animation control for real-time virtual humans. Communications of the ACM, 42(8):65-73, 1999.","title":"Animation control for real-time virtual humans","authors":"M Badler, Palmer, Bindiganavale","year":"1999","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cole et al., 2003] R Cole, S van Vuuren, B Pellom, K Ha- cioglu, J Ma, and J Movellan. Perceptive animated in- terfaces: first step toward a new paradigm for human- computer interaction. Proceedings of the IEEE, 91(9), 2003.","title":"Perceptive animated interfaces: first step toward a new paradigm for humancomputer interaction","authors":"S Cole, B Van Vuuren, K Pellom, J Hacioglu, Ma, Movellan","year":"2003","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dinerstein and Egbert, 2005] J Dinerstein and P K Egbert. Fast multi-level adaptation for interactive autonomous characters. ACM Transactions on Graphics, 24(2):262- 288, 2005.","title":"Fast multi-level adaptation for interactive autonomous characters","authors":"Egbert ; ] J Dinerstein, P K Dinerstein, Egbert","year":"2005","venue":"ACM Transactions on Graphics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dinerstein et al., 2005] J Dinerstein, D Ventura, and P K Egbert. Fast and robust incremental action prediction for interactive agents. Computational Intelligence, 21(1):90- 110, 2005.","title":"Fast and robust incremental action prediction for interactive agents","authors":"Dinerstein","year":"2005","venue":"Computational Intelligence","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dinerstein et al., 2006] J Dinerstein, P K Egbert, and David Cline. Enhancing computer graphics through machine learning: A survey. The Visual Computer, 22(12), 2006.","title":"Enhancing computer graphics through machine learning: A survey","authors":"P K Dinerstein, David Egbert, Cline","year":"2006","venue":"The Visual Computer","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Duncan, 2002] J Duncan. Ring masters. Cinefex, 89:64- 131, 2002.","title":"Ring masters","authors":"Duncan","year":"2002","venue":"Cinefex","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Friedrich et al., 1996] H. Friedrich, S. Munch, R. Dillmann, S. Bocionek, and M. Sassin. Robot programming by demonstration (RPD): supporting the induction by human interaction. Machine Learning, 23(2-3):163-189, 1996.","title":"Robot programming by demonstration (RPD): supporting the induction by human interaction","authors":"H Friedrich, S Munch, R Dillmann, S Bocionek, M Sassin","year":"1996","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gratch and Marsella, 2005] J Gratch and S Marsella. Eval- uating a computational model of emotion. Autonomous Agents and Multi-Agent Systems, 11(1):23-43, 2005.","title":"Evaluating a computational model of emotion","authors":"Marsella ; J Gratch, S Gratch, Marsella","year":"2005","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gratch et al., 2002] J Gratch, J Rickel, E Andre, J Cassell, E Petajan, and N Badler. Creating interactive virtual hu- mans: some assembly required. IEEE Intelligent Systems, pages 54-61, 2002.","title":"Creating interactive virtual humans: some assembly required","authors":"Gratch, E Rickel, J Andre, E Cassell, Petajan, Badler","year":"2002","venue":"IEEE Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Haykin, 1999","title":null,"authors":"Haykin","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"S Haykin. Neural Networks: A Comprehen- sive Foundation. Prentice Hall, Upper Saddle River, NJ., 2nd edition, 1999.","title":"Neural Networks: A Comprehensive Foundation","authors":"Haykin","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Isbell et al., 2001] C Isbell, C Shelton, M Kearns, S Singh, and P Stone. A social reinforcement learning agent. In Proceedings of Fifth International Conference on Au- tonomous Agents, pages 377-384, 2001.","title":"A social reinforcement learning agent","authors":"C Isbell, M Shelton, S Kearns, Singh, Stone","year":"2001","venue":"Proceedings of Fifth International Conference on Autonomous Agents","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kasper et al., 2001] M Kasper, G Fricke, K Steuernagel, and E von Puttkamer. A behavior-based mobile robot ar- chitecture for learning from demonstration. Robotics and Autonomous Systems, 34:153-164, 2001.","title":"A behavior-based mobile robot architecture for learning from demonstration","authors":"G Kasper, K Fricke, Steuernagel, Puttkamer","year":"2001","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Koschan and Abidi, 2001] A Koschan and M Abidi. A com- parison of median filter techniques for noise removal in color images. In 7th Workshop on Color Image Process- ing, pages 69-79, 2001.","title":"A comparison of median filter techniques for noise removal in color images","authors":"Abidi ; A Koschan, M Koschan, Abidi","year":"2001","venue":"7th Workshop on Color Image Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Laird, 2001] J Laird. It knows what you're going to do: Adding anticipation to the quakebot. In Proceedings of the Fifth International Conference on Autonomous Agents, pages 385-392, 2001.","title":"It knows what you're going to do: Adding anticipation to the quakebot","authors":"Laird","year":"2001","venue":"Proceedings of the Fifth International Conference on Autonomous Agents","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mataric, 2000] M Mataric. Getting humanoids to move and imitate. IEEE Intelligent Systems, pages 18-24, July 2000.","title":"Getting humanoids to move and imitate","authors":"Mataric","year":"2000","venue":"IEEE Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mitchell, 1997] T Mitchell. Machine Learning. WCB/McGraw-Hill, 1997.","title":"Machine Learning","authors":"Mitchell","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Monzani et al., 2001] J Monzani, A Caicedo, and D Thal- mann. Integrating behavioural animation techniques. Computer Graphics Forum, 20(3), 2001.","title":"Integrating behavioural animation techniques","authors":"Monzani, Caicedo, Thalmann","year":"2001","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ng and Jordan, 2000] A Y Ng and M Jordan. PEGASUS: A policy search method for large MDPs and POMDPs. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence, 2000.","title":"PEGASUS: A policy search method for large MDPs and POMDPs","authors":"A Y Ng, M Jordan","year":"2000","venue":"Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ng et al., 2004] A Y Ng, A Coates, M Diel, V Ganapathi, J Schulte, B Tse, E Berger, and E Liang. Inverted au- tonomous helicopter flight via reinforcement learning. In International Symposium on Experimental Robotics, 2004.","title":"Inverted autonomous helicopter flight via reinforcement learning","authors":"A Y Ng, A Coates, M Diel, V Ganapathi, J Schulte, B Tse, E Berger, Liang","year":"2004","venue":"International Symposium on Experimental Robotics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nicolescu, 2003] M Nicolescu. A Framework for Learn- ing From Demonstration, Generalization and Practice in Human-Robot Domains. PhD thesis, University of South- ern California, 2003.","title":"A Framework for Learning From Demonstration, Generalization and Practice in Human-Robot Domains","authors":"Nicolescu","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pomerleau, 1996] D Pomerleau. Neural network vision for robot driving. In S Nayar and T Poggio, editors, Early visual learning, pages 161-181. Oxford University Press, New York, NY, 1996.","title":"Neural network vision for robot driving","authors":"Pomerleau","year":"1996","venue":"Early visual learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Reynolds, 1987] C Reynolds. Flocks, herds, and schools: A distributed behavioral model. In Proceedings of SIG- GRAPH 1987, pages 25-34. ACM Press / ACM SIG- GRAPH, 1987.","title":"Flocks, herds, and schools: A distributed behavioral model","authors":"Reynolds","year":"1987","venue":"Proceedings of SIG-GRAPH 1987","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sammut et al., 1992] C Sammut, S Hurst, D Kedzier, and D Michie. Learning to fly. In Proceedings of Machine Learning, pages 385-393, 1992.","title":"Learning to fly","authors":"Sammut, D Hurst, Kedzier, Michie","year":"1992","venue":"Proceedings of Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton and Barto, 1998] R Sutton and A Barto. Reinforce- ment Learning: An Introduction. MIT Press, Cambridge, Massachusetts, 1998.","title":"Reinforcement Learning: An Introduction","authors":"R Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tesauro, 1995] G Tesauro. Temporal difference learning in TD-Gammon. Communications of the ACM, 38(3):58-68, 1995.","title":"Temporal difference learning in TD-Gammon","authors":"Tesauro","year":"1995","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[van Lent and Laird, 2001] M van Lent and J Laird. Learn- ing procedural knowledge through observation. In Pro- ceedings of International Conference On Knowledge Cap- ture, pages 179-186. ACM Press, 2001.","title":"Learning procedural knowledge through observation","authors":"Van Lent, ; ] M Laird, J Van Lent, Laird","year":"2001","venue":"Proceedings of International Conference On Knowledge Capture","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wingate and Seppi, 2005] D Wingate and K D Seppi. Pri- oritization methods for accelerating MDP solvers. Journal of Machine Learning Research, 6:851-881, 2005.","title":"Prioritization methods for accelerating MDP solvers","authors":"D Wingate, K D Seppi","year":"2005","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}