{"type":"ScholarlyArticle","title":"Digital Avatars: Framework Development and Their Evaluation","authors":[{"name":"Timothy Rupprecht","slug":"timothy-rupprecht","given":"Timothy","family":"Rupprecht","affiliation":"Electrical and Computer Engineering , Northeastern University"},{"name":"Sung-En Chang","slug":"sung-en-chang","given":"Sung-En","family":"Chang","affiliation":"Electrical and Computer Engineering , Northeastern University"},{"name":"Yushu Wu","slug":"yushu-wu","given":"Yushu","family":"Wu","affiliation":"Electrical and Computer Engineering , Northeastern University"},{"name":"Lei Lu","slug":"lei-lu","given":"Lei","family":"Lu","affiliation":"Electrical and Computer Engineering , Northeastern University"},{"name":"Enfu Nan","slug":"enfu-nan","given":"Enfu","family":"Nan","affiliation":"Electrical and Computer Engineering , Northeastern University"},{"name":"Chih-hsiang Li","slug":"chih-hsiang-li","given":"Chih-hsiang","family":"Li","affiliation":"Electrical and Computer Engineering , Northeastern University"},{"name":"Caiyue Lai","slug":"caiyue-lai","given":"Caiyue","family":"Lai","affiliation":"Electrical and Computer Engineering , Northeastern University"},{"name":"Zhimin Li","slug":"zhimin-li","given":"Zhimin","family":"Li","affiliation":"Electrical and Computer Engineering , Northeastern University"},{"name":"Zhijun Hu","slug":"zhijun-hu","given":"Zhijun","family":"Hu","affiliation":"Electrical and Computer Engineering , Northeastern University"},{"name":"Yumei He","slug":"yumei-he","given":"Yumei","family":"He","affiliation":"Tulane University"},{"name":"David Kaeli","slug":"david-kaeli","given":"David","family":"Kaeli","affiliation":"Electrical and Computer Engineering , Northeastern University"},{"name":"Yanzhi Wang","slug":"yanzhi-wang","given":"Yanzhi","family":"Wang","affiliation":"Electrical and Computer Engineering , Northeastern University"}],"abstract":"We present a novel prompting strategy for artificial intelligence driven digital avatars. To better quantify how our prompting strategy affects anthropomorphic features like humor, authenticity, and favorability we present Crowd Vote - an adaptation of Crowd Score that allows for judges to elect a large language model (LLM) candidate over competitors answering the same or similar prompts. To visualize the responses of our LLM, and the effectiveness of our prompting strategy we propose an end-to-end framework for creating high-fidelity artificial intelligence (AI) driven digital avatars. This pipeline effectively captures an individual's essence for interaction and our streaming algorithm delivers a high-quality digital avatar with real-time audio-video streaming from server to mobile device. Both our visualization tool, and our Crowd Vote metrics demonstrate our AI driven digital avatars have state-of-the-art humor, authenticity, and favorability outperforming all competitors and baselines. In the case of our Donald Trump and Joe Biden avatars, their authenticity and favorability are rated higher than even their real-world equivalents.","doi":"10.24963/ijcai.2024/1031","url":"https://ijcai.org/proceedings/2024/1031","pdf":"https://ijcai.org/proceedings/2024/1031.pdf","markdown":"https://ijcai.org/proceedings/2024/1031.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1031","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1031,"pages":{"first":8780,"last":8783},"keywords":["Humans and AI: HAI: Human-AI collaboration","Humans and AI: HAI: Human-computer interaction","Humans and AI: HAI: Applications"],"videos":[],"references":[{"position":1,"raw":"[Amodei et al., 2016] Dario Amodei, Sundaram Anantha- narayanan, Rishita Anubhai, Jingliang Bai, Eric Batten- berg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to- end speech recognition in english and mandarin. In Inter- national conference on machine learning, pages 173-182.","title":"Deep speech 2: End-toend speech recognition in english and mandarin","authors":"Dario Amodei, Rishita Sundaram Ananthanarayanan, Jingliang Anubhai, Eric Bai, Carl Battenberg, Jared Case, Bryan Casper, Qiang Catanzaro, Guoliang Cheng, Chen","year":null,"venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"PMLR, 2016.","title":null,"authors":"Pmlr","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cheng et al., 2022] Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, and Nannan Wang. Videoretalking: Audio- based lip synchronization for talking head video editing in the wild. In SIGGRAPH Asia 2022 Conference Papers, SA '22, New York, NY, USA, 2022. Association for Com- puting Machinery.","title":"Videoretalking: Audiobased lip synchronization for talking head video editing in the wild","authors":"Kun Cheng, Xiaodong Cun, Yong Zhang, Menghan Xia, Fei Yin, Mingrui Zhu, Xuan Wang, Jue Wang, Nannan Wang","year":"2022","venue":"SIGGRAPH Asia 2022 Conference Papers, SA '22","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goes et al., 2022] Fabricio Goes, Zisen Zhou, Piotr Saw- icki, Marek Grzes, and Daniel G Brown. Crowd score: A method for the evaluation of jokes using large language model ai voters as judges. arXiv preprint arXiv:2212.11214, 2022.","title":"Crowd score: A method for the evaluation of jokes using large language model ai voters as judges","authors":"Fabricio Goes, Zisen Zhou, Piotr Sawicki, Marek Grzes, Daniel G Brown","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Oord et al., 2016] Aaron van den Oord, Sander Diele- man, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw au- dio. arXiv preprint arXiv:1609.03499, 2016. [","title":"Wavenet: A generative model for raw audio","authors":"Oord","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[OpenAI, 2023] OpenAI. GPT-3.5-turbo-instruct. https:// openai.com/, 2023. Accessed: February 13 -19, 2024.","title":"OpenAI. GPT-3.5-turbo-instruct","authors":"Openai","year":"2023","venue":null,"doi":null,"url":"https://openai.com/,2023","internal_url":null},{"position":7,"raw":"[Prajwal et al., 2020] K R Prajwal, Rudrabha Mukhopad- hyay, Vinay P. Namboodiri, and C.V. Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM International Conference on Multimedia, MM '20, page 484-492, New York, NY, USA, 2020. Association for Computing Ma- chinery.","title":"A lip sync expert is all you need for speech to lip generation in the wild","authors":"Rudrabha K R Prajwal, Mukhopadhyay, P Vinay, C V Namboodiri, Jawahar","year":"2020","venue":"Proceedings of the 28th ACM International Conference on Multimedia, MM '20","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Radford et al., 2023] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492-28518. PMLR, 2023.","title":"Robust speech recognition via large-scale weak supervision","authors":"Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, Ilya Sutskever","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ren et al., 2019] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. Advances in neural information processing systems, 32, 2019.","title":"Fastspeech: Fast, robust and controllable text to speech","authors":"Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.","title":"Llama: Open and efficient foundation language models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Wei et al., 2022] Jason Wei, Xuezhi Wang, Dale Schuur- mans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903, 2022.","title":"Chain of thought prompting elicits reasoning in large language models","authors":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H Chi, Quoc Le, Denny Zhou","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}