{"type":"ScholarlyArticle","title":"InViTe: Individual Virtual Transfer for Personalized 3D Face Generation System","authors":[{"name":"Mingyu Jang","slug":"mingyu-jang","given":"Mingyu","family":"Jang","affiliation":"Yonsei University {jmg1002 ,"},{"name":"Kyungjune Lee","slug":"kyungjune-lee","given":"Kyungjune","family":"Lee","affiliation":"Yonsei University {jmg1002 ,"},{"name":"Seongmin Lee","slug":"seongmin-lee","given":"Seongmin","family":"Lee","affiliation":"Yonsei University {jmg1002 ,"},{"name":"Hoseok Tong","slug":"hoseok-tong","given":"Hoseok","family":"Tong","affiliation":"Yonsei University {jmg1002 ,"},{"name":"Juwan Chung","slug":"juwan-chung","given":"Juwan","family":"Chung","affiliation":"Yonsei University {jmg1002 ,"},{"name":"Yusung Ro","slug":"yusung-ro","given":"Yusung","family":"Ro","affiliation":"Yonsei University {jmg1002 ,"},{"name":"Sanghoon Lee","slug":"sanghoon-lee","given":"Sanghoon","family":"Lee","affiliation":"Yonsei University {jmg1002 ,"}],"abstract":"With the expansion of the virtual communication industry using VR/AR, it has attracted increasing attention to enable users to represent their personalities in a 3D avatar. As the face of 3D avatars plays a crucial role in conveying human personality, a system that generates and manipulates 3D faces is desired. However, establishing the system is challenging due to the need for human effort and specialized knowledge. To fill this void, we present the Individual Virtual Transfer (InViTe), which enables the creation and customization of a 3D face according to the user's preference. Our proposed system is featured for 1) 3D face reconstruction with high fidelity texture map, 2) 3D face personalization, 3) realistic rendering results, and 4) real-time mobile virtual applications. We conduct an experiment to demonstrate that the proposed system can achieve sufficient individual personalization of 3D faces. Furthermore, we evaluate the system's data transmission protocol and demonstrate its efficiency. The demonstration video is available at https://www.youtube.com/watch?v=D 4pXZvGUWU.","doi":"10.24963/ijcai.2024/1008","url":"https://ijcai.org/proceedings/2024/1008","pdf":"https://ijcai.org/proceedings/2024/1008.pdf","markdown":"https://ijcai.org/proceedings/2024/1008.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1008","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1008,"pages":{"first":8683,"last":8686},"keywords":["Computer Vision: CV: Applications","Computer Vision: CV: 3D computer vision"],"videos":[],"references":[{"position":1,"raw":"[Abdal et al., 2021] Rameen Abdal, Peihao Zhu, Niloy J Mi- tra, and Peter Wonka. Styleflow: Attribute-conditioned ex- ploration of stylegan-generated images using conditional continuous normalizing flows. ACM Transactions on Graphics (ToG), 40(3):1-21, 2021.","title":"Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows","authors":"Rameen Abdal, Peihao Zhu, J Niloy, Peter Mitra, Wonka","year":"2021","venue":"ACM Transactions on Graphics (ToG)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ahn et al., 2018] Sewoong Ahn, Woojae Kim, Jinwoo Kim, Jaekyung Kim, and Sanghoon Lee. Visual preference pre- diction for enhanced images on ultra-high-definition dis- play. In 2018 25th IEEE International Conference on Im- age Processing (ICIP), pages 3548-3552. IEEE, 2018.","title":"Visual preference prediction for enhanced images on ultra-high-definition display","authors":"Ahn","year":"2018","venue":"25th IEEE International Conference on Image Processing (ICIP)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bai et al., 2023] Haoran Bai, Di Kang, Haoxian Zhang, Jin- shan Pan, and Linchao Bao. Ffhq-uv: Normalized facial uv-texture dataset for 3d face reconstruction. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 362-371, 2023.","title":"Ffhq-uv: Normalized facial uv-texture dataset for 3d face reconstruction","authors":"Haoran Bai, Di Kang, Haoxian Zhang, Jinshan Pan, Linchao Bao","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Daněček et al., 2022] Radek Daněček, Michael J Black, and Timo Bolkart. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20311-20322, 2022.","title":"Emoca: Emotion driven monocular face capture and animation","authors":"Daněček","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dastjerdi et al., 2023] Mohammad Reza Karimi Dastjerdi, Jonathan Eisenmann, Yannick Hold-Geoffroy, and Jean- Franc ¸ois Lalonde. Everlight: Indoor-outdoor editable hdr lighting estimation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 7420- 7429, 2023.","title":"Everlight: Indoor-outdoor editable hdr lighting estimation","authors":"Mohammad Reza, Karimi Dastjerdi, Jonathan Eisenmann, Yannick Hold-Geoffroy, Jean-Franc ¸ois Lalonde","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2019] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular mar- gin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690-4699, 2019.","title":"Arcface: Additive angular margin loss for deep face recognition","authors":"Deng","year":"2019","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jang et al., 2022] Mingyu Jang, Seongmin Lee, Jiwoo Kang, and Sanghoon Lee. Technical consideration towards robust 3d reconstruction with multi-view active stereo sen- sors. Sensors, 22(11):4142, 2022.","title":"Technical consideration towards robust 3d reconstruction with multi-view active stereo sensors","authors":"Mingyu Jang, Seongmin Lee, Jiwoo Kang, Sanghoon Lee","year":"2022","venue":"Sensors","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kachur et al., 2020] Alexander Kachur, Evgeny Osin, De- nis Davydov, Konstantin Shutilov, and Alexey Novok- shonov. Assessing the big five personality traits using real- life static facial images. Scientific Reports, 10(1):8487, 2020.","title":"Assessing the big five personality traits using reallife static facial images","authors":"Alexander Kachur, Evgeny Osin, Denis Davydov, Konstantin Shutilov, Alexey Novokshonov","year":"2020","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kartynnik et al., 2019] Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, and Matthias Grund- mann. Real-time facial surface geometry from monocular video on mobile gpus. In Third Workshop on Computer Vision for AR/VR, 2019.","title":"Real-time facial surface geometry from monocular video on mobile gpus","authors":"Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, Matthias Grundmann","year":"2019","venue":"Third Workshop on Computer Vision for AR/VR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2016] Haksub Kim, Sewoong Ahn, Woojae Kim, and Sanghoon Lee. Visual preference assessment on ultra-high-definition images. IEEE Transactions on Broadcasting, 62(4):757-769, 2016.","title":"Visual preference assessment on ultra-high-definition images","authors":"Kim","year":"2016","venue":"IEEE Transactions on Broadcasting","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim et al., 2021] Woojae Kim, Anh-Duc Nguyen, Jinwoo Kim, Jongyoo Kim, Heeseok Oh, and Sanghoon Lee. Di- verse and adjustable versatile image enhancer. IEEE Ac- cess, 9:80883-80896, 2021.","title":"Diverse and adjustable versatile image enhancer","authors":"Kim","year":"2021","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2023] Kyungjune Lee, Jeonghaeng Lee, Hyuck- sang Lee, Mingyu Jang, Seongmin Lee, and Sanghoon Lee. Faceclone: Interactive facial shape and motion cloning system using multi-view images. In 2023 IEEE International Conference on Multimedia and Expo Work- shops (ICMEW), pages 512-513. IEEE, 2023.","title":"Faceclone: Interactive facial shape and motion cloning system using multi-view images","authors":"Lee","year":"2023","venue":"IEEE International Conference on Multimedia and Expo Workshops (ICMEW)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2017] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph., 36(6):194-1, 2017.","title":"Learning a model of facial shape and expression from 4d scans","authors":"Li","year":"2017","venue":"ACM Trans. Graph","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2018] Tingting Li, Ruihe Qian, Chao Dong, Si Liu, Qiong Yan, Wenwu Zhu, and Liang Lin. Beautygan: Instance-level facial makeup transfer with deep genera- tive adversarial network. In Proceedings of the 26th ACM international conference on Multimedia, pages 645-653, 2018.","title":"Beautygan: Instance-level facial makeup transfer with deep generative adversarial network","authors":"Tingting Li, Ruihe Qian, Chao Dong, Si Liu, Qiong Yan, Wenwu Zhu, Liang Lin","year":"2018","venue":"Proceedings of the 26th ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2021] Tianye Li, Shichen Liu, Timo Bolkart, Ji- ayi Liu, Hao Li, and Yajie Zhao. Topologically consis- tent multi-view face inference using volumetric sampling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3824-3834, 2021.","title":"Topologically consistent multi-view face inference using volumetric sampling","authors":"Li","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Paysan et al., 2009] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. In 2009 sixth IEEE international conference on advanced video and signal based surveillance, pages 296-301. Ieee, 2009.","title":"A 3d face model for pose and illumination invariant face recognition","authors":"Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, Thomas Vetter","year":"2009","venue":"2009 sixth IEEE international conference on advanced video and signal based surveillance","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2020] Wencheng Wang, Xiaojin Wu, Xiaohui Yuan, and Zairui Gao. An experiment-based review of low-light image enhancement methods. Ieee Access, 8:87884-87917, 2020.","title":"An experiment-based review of low-light image enhancement methods","authors":"Wencheng Wang, Xiaojin Wu, Xiaohui Yuan, Zairui Gao","year":"2020","venue":"Ieee Access","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yang et al., 2023] Xingchao Yang, Takafumi Taketomi, and Yoshihiro Kanamori. Makeup extraction of 3d representa- tion via illumination-aware image decomposition. In Com- puter Graphics Forum, volume 42, pages 293-307. Wiley Online Library, 2023.","title":"Makeup extraction of 3d representation via illumination-aware image decomposition","authors":"Yang","year":"2023","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zheng et al., 2023] Yujian Zheng, Zirong Jin, Moran Li, Haibin Huang, Chongyang Ma, Shuguang Cui, and Xi- aoguang Han. Hairstep: Transfer synthetic to real us- ing strand and depth maps for single-view 3d hair mod- eling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12726- 12735, 2023.","title":"Hairstep: Transfer synthetic to real using strand and depth maps for single-view 3d hair modeling","authors":"Zheng","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}