{"type":"ScholarlyArticle","title":"AVIN-Chat: An Audio-Visual Interactive Chatbot System with Emotional State Tuning","authors":[{"name":"Chanhyuk Park","slug":"chanhyuk-park","given":"Chanhyuk","family":"Park","affiliation":"Yonsei University {hanul500 ,"},{"name":"Jungbin Cho","slug":"jungbin-cho","given":"Jungbin","family":"Cho","affiliation":"Yonsei University {hanul500 ,"},{"name":"Junwan Kim","slug":"junwan-kim","given":"Junwan","family":"Kim","affiliation":"Yonsei University {hanul500 ,"},{"name":"Seongmin Lee","slug":"seongmin-lee","given":"Seongmin","family":"Lee","affiliation":"Yonsei University {hanul500 ,"},{"name":"Jungsu Kim","slug":"jungsu-kim","given":"Jungsu","family":"Kim","affiliation":"Yonsei University {hanul500 ,"},{"name":"Sanghoon Lee","slug":"sanghoon-lee","given":"Sanghoon","family":"Lee","affiliation":"Yonsei University {hanul500 ,"}],"abstract":"This work presents an audio-visual interactive chatbot (AVIN-Chat) system that allows users to have face-to-face conversations with 3D avatars in real-time. Compared to the previous chatbot services, which provide text-only or speech-only communications, the proposed AVIN-Chat can offer audio-visual communications providing users with a superior experience quality. In addition, the proposed AVIN-Chat emotionally speaks and expresses according to the user's emotional state. Thus, it enables users to establish a strong bond with the chatbot system, increasing the user's immersion. Through user subjective tests, it is demonstrated that the proposed system provides users with a higher sense of immersion than previous chatbot systems. The demonstration video is available at https://www.youtube.com/watch?v=Z74uIV9k7_k.","doi":"10.24963/ijcai.2024/1027","url":"https://ijcai.org/proceedings/2024/1027","pdf":"https://ijcai.org/proceedings/2024/1027.pdf","markdown":"https://ijcai.org/proceedings/2024/1027.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1027","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1027,"pages":{"first":8763,"last":8766},"keywords":["Natural Language Processing: NLP: Applications","Computer Vision: CV: Applications"],"videos":[],"references":[{"position":1,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing sys- tems, 33:1877-1901, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Guo et al., 2023] Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan. Prompttts: Controllable text-to- speech with text descriptions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.","title":"Prompttts: Controllable text-tospeech with text descriptions","authors":"Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, Xu Tan","year":"2023","venue":"ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Kang et al., 2020] Jiwoo Kang, Seongmin Lee, and Sanghoon Lee. Uv completion with self-referenced discrimination. In Eurographics (Short Papers), pages 61-64, 2020.","title":"Uv completion with self-referenced discrimination","authors":"Kang","year":"2020","venue":"Eurographics (Short Papers)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Kang et al., 2021] Jiwoo Kang, Seongmin Lee, and Sanghoon Lee. Competitive learning of facial fitting and synthesis using uv energy. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 52(5):2858-2873, 2021.","title":"Competitive learning of facial fitting and synthesis using uv energy","authors":"Kang","year":"2021","venue":"IEEE Transactions on Systems, Man, and Cybernetics: Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Lee et al., 2023a] Kyungjune Lee, Jeonghaeng Lee, Hyuck- sang Lee, Mingyu Jang, Seongmin Lee, and Sanghoon Lee. Faceclone: Interactive facial shape and motion cloning system using multi-view images. In 2023 IEEE International Conference on Multimedia and Expo Work- shops (ICMEW), pages 512-513. IEEE, 2023.","title":"Faceclone: Interactive facial shape and motion cloning system using multi-view images","authors":"Lee","year":"2023","venue":"IEEE International Conference on Multimedia and Expo Workshops (ICMEW)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Lee et al., 2023b] Seongmin Lee, Hyunse Yoon, Jiwoo Kang, Jungsu Kim, Jiwan Son, Jungwoo Huh, and Sanghoon Lee. Video-based stabilized 3d face alignment using temporal multi-discrimination. In 2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP), pages 1-6. IEEE, 2023.","title":"Video-based stabilized 3d face alignment using temporal multi-discrimination","authors":"Lee","year":"2023","venue":"IEEE 25th International Workshop on Multimedia Signal Processing (MMSP)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lee et al., 2023c] Seongmin Lee, Hyunse Yoon, Sohyun Park, Sanghoon Lee, and Jiwoo Kang. Stabilized temporal 3d face alignment using landmark displacement learning. Electronics, 12(17):3735, 2023.","title":"Stabilized temporal 3d face alignment using landmark displacement learning","authors":"Lee","year":"2023","venue":"Electronics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lei et al., 2023] Biwen Lei, Jianqiang Ren, Mengyang Feng, Miaomiao Cui, and Xuansong Xie. A hierarchical representation network for accurate and detailed face re- construction from in-the-wild images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 394-403, 2023.","title":"A hierarchical representation network for accurate and detailed face reconstruction from in-the-wild images","authors":"Lei","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lester et al., 2021] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, 2021.","title":"The power of scale for parameter-efficient prompt tuning","authors":"Lester","year":"2021","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Onizuka et al., 2019] Hayato Onizuka, Diego Thomas, Hideaki Uchiyama, and Rin-ichiro Taniguchi. Landmark- guided deformation transfer of template facial expressions for automatic generation of avatar blendshapes. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.","title":"Landmarkguided deformation transfer of template facial expressions for automatic generation of avatar blendshapes","authors":"Hayato Onizuka, Diego Thomas, Hideaki Uchiyama, Rin-Ichiro Taniguchi","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Peng et al., 2023] Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun He, Hongyan Liu, and Zhaoxin Fan. Emotalk: Speech-driven emotional disentanglement for 3d face animation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 20687- 20697, 2023.","title":"Emotalk: Speech-driven emotional disentanglement for 3d face animation","authors":"Peng","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Radford et al., 2023] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492-28518. PMLR, 2023.","title":"Robust speech recognition via large-scale weak supervision","authors":"Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, Ilya Sutskever","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Schick and Schütze, 2021] Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255-269, 2021.","title":"Exploiting cloze-questions for few-shot text classification and natural language inference","authors":"Schütze Schick, Timo Schick, Hinrich Schütze","year":"2021","venue":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Shin et al., 2020] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automati- cally generated prompts. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 4222-4235, 2020.","title":"Autoprompt: Eliciting knowledge from language models with automatically generated prompts","authors":"Taylor Shin, Yasaman Razeghi, Robert L Logan, I V, Eric Wallace, Sameer Singh","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Zheng et al., 2023] Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J. Black, and Otmar Hilliges. Pointa- vatar: Deformable point-based head avatars from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.","title":"Pointavatar: Deformable point-based head avatars from videos","authors":"Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J Black, Otmar Hilliges","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Zielonka et al., 2022] Wojciech Zielonka, Timo Bolkart, and Justus Thies. Instant volumetric head avatars. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4574-4584, 2022.","title":"Instant volumetric head avatars","authors":"Wojciech Zielonka, Timo Bolkart, Justus Thies","year":"2022","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}