{"type":"ScholarlyArticle","title":"Embodied Conversational AI Agents in a Multi-modal Multi-agent Competitive Dialogue","authors":[{"name":"Rahul R. Divekar","slug":"rahul-r-divekar","given":"Rahul R.","family":"Divekar","affiliation":"Rensselaer Polytechnic Institute"},{"name":"Xiangyang Mou","slug":"xiangyang-mou","given":"Xiangyang","family":"Mou","affiliation":"Rensselaer Polytechnic Institute"},{"name":"Lisha Chen","slug":"lisha-chen","given":"Lisha","family":"Chen","affiliation":"Rensselaer Polytechnic Institute"},{"name":"Maíra Gatti de Bayser","slug":"maira-gatti-de-bayser","given":"Maíra","family":"Gatti de Bayser","affiliation":"IBM Research"},{"name":"Melina Alberio Guerra","slug":"melina-alberio-guerra","given":"Melina Alberio","family":"Guerra","affiliation":"IBM Research"},{"name":"Hui Su","slug":"hui-su","given":"Hui","family":"Su","affiliation":"Rensselaer Polytechnic Institute; IBM Research"}],"abstract":"In a setting where two AI agents embodied as animated humanoid avatars are engaged in a conversation with one human and each other, we see two challenges. One, determination by the AI agents about which one of them is being addressed. Two, determination by the AI agents if they may/could/should speak at the end of a turn. In this work we bring these two challenges together and explore the participation of AI agents in multi-party conversations. Particularly, we show two embodied AI shopkeeper agents who sell similar items aiming to get the business of a user by competing with each other on the price. In this scenario, we solve the first challenge by using headpose (estimated by deep learning techniques) to determine who the user is talking to. For the second challenge we use deontic logic to model rules of a negotiation conversation.","doi":"10.24963/ijcai.2019/940","url":"https://ijcai.org/proceedings/2019/940","pdf":"https://ijcai.org/proceedings/2019/0940.pdf","markdown":"https://ijcai.org/proceedings/2019/940.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/940","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Demos","track":"Demos","number":940,"pages":{"first":6512,"last":6514},"keywords":["AI: Others","AI: Human-Computer Interactive Systems","AI: Multiagent Systems"],"videos":[],"references":[{"position":1,"raw":"[Akhtiamov et al., 2017] Oleg Akhtiamov, Maxim Sidorov, Alexey A Karpov, and Wolfgang Minker. Speech and text analysis for multimodal addressee detection in human- human-computer interaction. In INTERSPEECH, pages 2521-2525, 2017.","title":"Speech and text analysis for multimodal addressee detection in humanhuman-computer interaction","authors":"Oleg Akhtiamov, Maxim Sidorov, Alexey A Karpov, Wolfgang Minker","year":"2017","venue":"INTERSPEECH","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Allen et al., 2019] David Allen, Rahul R Divekar, Jaimie Drozdal, and Lilit Balagyozyan. The Rensselaer Mandarin Project-a Cognitive and Immersive Language Learning Environment. In Thirty-third AAAI Conference on Artifi- cial Intelligence, 2019.","title":"The Rensselaer Mandarin Project-a Cognitive and Immersive Language Learning Environment","authors":"David Allen, Jaimie Rahul R Divekar, Lilit Drozdal, Balagyozyan","year":"2019","venue":"Thirty-third AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Andrist et al., 2016] Sean Andrist, Dan Bohus, Bilge Mutlu, and David Schlangen. Turn-taking and coordina- tion in human-machine interaction. AI Magazine, 37(4):5- 6, 2016.","title":"Turn-taking and coordination in human-machine interaction","authors":"Sean Andrist, Dan Bohus, Bilge Mutlu, David Schlangen","year":"2016","venue":"AI Magazine","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bulat and Tzimiropoulos, 2017] Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3d fa- cial landmarks). In International Conference on Computer Vision, 2017.","title":"How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3d facial landmarks)","authors":"Adrian Bulat, Georgios Tzimiropoulos","year":"2017","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen and Ji, 2018] Lisha Chen and Qiang Ji. Kernel den- sity network for quantifying uncertainty in face alignment. In 3rd Bayesian Deep Learning Workshop of Advances in Neural Information Processing Systems (NeurIPS), 2018.","title":"Kernel density network for quantifying uncertainty in face alignment","authors":"Lisha Chen, Qiang Ji","year":"2018","venue":"3rd Bayesian Deep Learning Workshop of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[de Bayser et al., 2018a] Maira Gatti de Bayser, Melina Al- berio Guerra, Paulo Cavalin, and Claudio Pinhanez. Spec- ifying and implementing multi-party conversation rules with finite-state-automata. In Workshops at the Thirty- Second AAAI Conference on Artificial Intelligence, 2018.","title":"Specifying and implementing multi-party conversation rules with finite-state-automata","authors":"De Bayser","year":"2018","venue":"Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[de Bayser et al., 2018b] Maira Gatti de Bayser, Clau- dio Pinhanez, Heloisa Candello, Marisa Affonso, Mauro Pichiliani Vasconcelos, Melina Alberio Guerra, Paulo Cavalin, and Renan Souza. Ravel: A MAS or- chestration platform for human-chatbots conversations. 2018.","title":"Ravel: A MAS orchestration platform for human-chatbots conversations","authors":"De Bayser","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Divekar et al., 2018a] Rahul R Divekar, Jaimie Drozdal, Yalun Zhou, Ziyi Song, David Allen, Robert Rouhani, Rui Zhao, Shuyue Zheng, Lilit Balagyozyan, and Hui Su. Interaction challenges in AI equipped environments built to teach foreign languages through dialogue and task- completion. In Proceedings of the 2018 on Designing In- teractive Systems Conference 2018, pages 597-609. ACM, 2018.","title":"Interaction challenges in AI equipped environments built to teach foreign languages through dialogue and taskcompletion","authors":"Divekar","year":"2018","venue":"Proceedings of the 2018 on Designing Interactive Systems Conference 2018","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Divekar et al., 2018b] Rahul R Divekar, Matthew Peveler, Robert Rouhani, Rui Zhao, Jeffrey O Kephart, David Allen, Kang Wang, Qiang Ji, and Hui Su. CIRA: An Architecture for Building Configurable Immersive Smart- Rooms. In Proceedings of SAI Intelligent Systems Confer- ence, pages 76-95. Springer, 2018.","title":"CIRA: An Architecture for Building Configurable Immersive Smart-Rooms","authors":"Divekar","year":"2018","venue":"Proceedings of SAI Intelligent Systems Conference","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Divekar et al., 2018c] Rahul R Divekar, Yalun Zhou, David Allen, Jaimie Drozdal, and Hui Su. Building human-scale intelligent immersive spaces for foreign language learning. iLRN 2018 Montana, page 94, 2018.","title":"Building human-scale intelligent immersive spaces for foreign language learning","authors":"Divekar","year":"2018","venue":"iLRN 2018 Montana","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Divekar et al., 2019] Rahul R Divekar, Jeffrey O Kephart, Xiangyang Mou, Lisha Chen, and Hui Su. You talkin' to me? -a practical attention-aware embodied agent. In Human-Computer Interaction -INTERACT 2019, 2019.","title":"You talkin' to me? -a practical attention-aware embodied agent","authors":"Jeffrey O Rahul R Divekar, Xiangyang Kephart, Lisha Mou, Hui Chen, Su","year":"2019","venue":"Human-Computer Interaction -INTERACT 2019","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Khouzaimi et al., 2016] Hatim Khouzaimi, Romain Laroche, and Fabrice Lefèvre. Reinforcement learning for turn-taking management in incremental spoken dialogue systems. In IJCAI, pages 2831-2837, 2016.","title":"Reinforcement learning for turn-taking management in incremental spoken dialogue systems","authors":"Hatim Khouzaimi, Romain Laroche, Fabrice Lefèvre","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/402"},{"position":13,"raw":"[Le Minh et al., 2018] Thao Le Minh, Nobuyuki Shimizu, Takashi Miyazaki, and Koichi Shinoda. Deep Learn- ing Based Multi-modal Addressee Recognition in Visual Scenes with Utterances. IJCAI 2018, pages 1546-1553, 2018.","title":"Deep Learning Based Multi-modal Addressee Recognition in Visual Scenes with Utterances","authors":"Thao Le, Minh, Nobuyuki Shimizu, Takashi Miyazaki, Koichi Shinoda","year":"2018","venue":"IJCAI 2018","doi":null,"url":null,"internal_url":"/proceedings/2018/214"},{"position":14,"raw":"[Norouzian et al., 2019] Atta Norouzian, Bogdan Mazoure, Dermot Connolly, and Daniel Willett. Exploring atten- tion mechanism for acoustic-based classification of speech utterances into system-directed and non-system-directed. arXiv preprint arXiv:1902.00570, 2019.","title":"Exploring attention mechanism for acoustic-based classification of speech utterances into system-directed and non-system-directed","authors":"Atta Norouzian, Bogdan Mazoure, Dermot Connolly, Daniel Willett","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ravuri and Stolcke, 2015] Suman Ravuri and Andreas Stol- cke. Recurrent neural network and LSTM models for lex- ical utterance classification. In Sixteenth Annual Confer- ence of the International Speech Communication Associa- tion, 2015.","title":"Recurrent neural network and LSTM models for lexical utterance classification","authors":"Suman Ravuri, Andreas Stolcke","year":"2015","venue":"Sixteenth Annual Conference of the International Speech Communication Association","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sheikhi and Odobez, 2015] Samira Sheikhi and Jean Marc Odobez. Combining dynamic head pose-gaze mapping with the robot conversational state for attention recogni- tion in human-robot interactions. Pattern Recognition Let- ters, 66:81-90, 2015.","title":"Combining dynamic head pose-gaze mapping with the robot conversational state for attention recognition in human-robot interactions","authors":"Samira Sheikhi, Jean Marc, Odobez","year":"2015","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tsai et al., 2015] TJ Tsai, Andreas Stolcke, and Malcolm Slaney. A study of multimodal addressee detection in human-human-computer interaction. IEEE Transactions on Multimedia, 17(9):1550-1561, 2015.","title":"A study of multimodal addressee detection in human-human-computer interaction","authors":"Andreas Tsai, Malcolm Stolcke, Slaney","year":"2015","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhu et al., 2019] Xiangyu Zhu, Xiaoming Liu, Zhen Lei, and Stan Z Li. Face alignment in full pose range: A 3D total solution. IEEE transactions on pattern analysis and machine intelligence, 41(1):78-92, 2019.","title":"Face alignment in full pose range: A 3D total solution","authors":"Xiangyu Zhu, Xiaoming Liu, Zhen Lei, Stan Z Li","year":"2019","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}