{"type":"ScholarlyArticle","title":"GigaPevt: Multimodal Medical Assistant","authors":[{"name":"Pavel Blinov","slug":"pavel-blinov","given":"Pavel","family":"Blinov","affiliation":"Sber AI Lab , Moscow , Russia"},{"name":"Konstantin Egorov","slug":"konstantin-egorov","given":"Konstantin","family":"Egorov","affiliation":"Sber AI Lab , Moscow , Russia"},{"name":"Ivan Sviridov","slug":"ivan-sviridov","given":"Ivan","family":"Sviridov","affiliation":"Sber AI Lab , Moscow , Russia"},{"name":"Nikolay Ivanov","slug":"nikolay-ivanov","given":"Nikolay","family":"Ivanov","affiliation":"Sber AI Lab , Moscow , Russia"},{"name":"Stepan Botman","slug":"stepan-botman","given":"Stepan","family":"Botman","affiliation":"Sber AI Lab , Moscow , Russia"},{"name":"Evgeniy Tagin","slug":"evgeniy-tagin","given":"Evgeniy","family":"Tagin","affiliation":"Sber AI Lab , Moscow , Russia"},{"name":"Stepan Kudin","slug":"stepan-kudin","given":"Stepan","family":"Kudin","affiliation":"Sber AI Lab , Moscow , Russia"},{"name":"Galina Zubkova","slug":"galina-zubkova","given":"Galina","family":"Zubkova","affiliation":"Sber AI Lab , Moscow , Russia"},{"name":"Andrey V. Savchenko","slug":"andrey-v-savchenko","given":"Andrey V.","family":"Savchenko","affiliation":"Sber AI Lab , Moscow , Russia; HSE University , Nizhny Novgorod , Russia"}],"abstract":"Building an intelligent and efficient medical assistant is still a challenging AI problem. The major limitation comes from the data modality scarceness, which reduces comprehensive patient perception. This demo paper presents GigaPevt, the first multimodal medical assistant that combines the dialog capabilities of large language models with specialized medical models. Such an approach shows immediate advantages in dialog quality and metric performance, with a 1.18% accuracy improvement in the question-answering task.","doi":"10.24963/ijcai.2024/992","url":"https://ijcai.org/proceedings/2024/992","pdf":"https://ijcai.org/proceedings/2024/0992.pdf","markdown":"https://ijcai.org/proceedings/2024/992.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/992","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":992,"pages":{"first":8614,"last":8618},"keywords":["Computer Vision: CV: Applications","Computer Vision: CV: Vision and language","Natural Language Processing: NLP: Dialogue and interactive systems","Multidisciplinary Topics and Applications: MDA: Health and medicine"],"videos":[],"references":[{"position":1,"raw":"[Blinov et al., 2022] Pavel Blinov, Arina Reshetnikova, Aleksandr Nesterov, Galina Zubkova, and Vladimir Kokh. Rumedbench: a Russian medical language understanding benchmark. In International Conference on Artificial In- telligence in Medicine, pages 383-392. Springer, 2022.","title":"Rumedbench: a Russian medical language understanding benchmark","authors":"Pavel Blinov, Arina Reshetnikova, Aleksandr Nesterov, Galina Zubkova, Vladimir Kokh","year":"2022","venue":"International Conference on Artificial Intelligence in Medicine","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2018] Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and Andrew Zisserman. VGGFace2: A dataset for recognising faces across pose and age. In International Conference on Automatic Face and Gesture Recognition, 2018.","title":"VGGFace2: A dataset for recognising faces across pose and age","authors":"Qiong Cao, Li Shen, Weidi Xie, M Omkar, Andrew Parkhi, Zisserman","year":"2018","venue":"International Conference on Automatic Face and Gesture Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Eidinger et al., 2014] Eran Eidinger, Roee Enbar, and Tal Hassner. Age and gender estimation of unfiltered faces. IEEE Transactions on Information Forensics and Security, 9(12):2170-2179, 2014.","title":"Age and gender estimation of unfiltered faces","authors":"Eran Eidinger, Roee Enbar, Tal Hassner","year":"2014","venue":"IEEE Transactions on Information Forensics and Security","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gao et al., 2023] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023.","title":"Retrieval-augmented generation for large language models: A survey","authors":"Gao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778. IEEE, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jiang et al., 2019] Min Jiang, Yuanyuan Shang, and Guodong Guo. On visual BMI analysis from facial images. Image and Vision Computing, 89:183-196, 2019.","title":"On visual BMI analysis from facial images","authors":"Jiang","year":"2019","venue":"Image and Vision Computing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lewis et al., 2020] Patrick Lewis, Ethan Perez, Aleksan- dra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459-9474.","title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","authors":"Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-Tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"Curran Associates, Inc., 2020.","title":null,"authors":"Curran Associates, Inc","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2023] Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language model-augmented code emulator. arXiv preprint arXiv:2312.04474, 2023.","title":"Chain of code: Reasoning with a language model-augmented code emulator","authors":"Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, Brian Ichter","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2024] Xin Liu, Girish Narayanswamy, Akshay Paruchuri, Xiaoyu Zhang, Jiankai Tang, Yuzhe Zhang, Roni Sengupta, Shwetak Patel, Yuntao Wang, and Daniel McDuff. rPPG-Toolbox: Deep remote PPG toolbox. Ad- vances in Neural Information Processing Systems, 36, 2024.","title":"rPPG-Toolbox: Deep remote PPG toolbox","authors":"Xin Liu, Girish Narayanswamy, Akshay Paruchuri, Xiaoyu Zhang, Jiankai Tang, Yuzhe Zhang, Roni Sengupta, Shwetak Patel, Yuntao Wang, Daniel Mcduff","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lugaresi et al., 2019] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Yong, Juhyun Lee, et al. Mediapipe: A framework for perceiving and processing reality. In Proceedings of the Third workshop on computer vision for AR/VR at IEEE Computer Vision and Pattern Recognition (CVPR), 2019.","title":"Mediapipe: A framework for perceiving and processing reality","authors":"Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris Mcclanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Yong, Juhyun Lee","year":"2019","venue":"Proceedings of the Third workshop on computer vision for AR/VR at IEEE Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Makarov et al., 2023] Ilya Makarov, Andrey V. Savchenko, and Lyudmila V. Savchenko. Fast search of face recogni- tion model for a mobile device based on neural architec- ture comparator. IEEE Access, 11:65977-65990, 2023.","title":"Fast search of face recognition model for a mobile device based on neural architecture comparator","authors":"Makarov","year":"2023","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mollahosseini et al., 2017] Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. AffectNet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1):18-31, 2017.","title":"AffectNet: A database for facial expression, valence, and arousal computing in the wild","authors":"Ali Mollahosseini, Behzad Hasani, Mohammad H Mahoor","year":"2017","venue":"IEEE Transactions on Affective Computing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pourmirzaei et al., 2021] Mahdi Pourmirzaei, Gholam Ali Montazer, and Farzaneh Esmaili. Using self-supervised auxiliary tasks to improve fine-grained facial representa- tion. arXiv preprint arXiv:2105.06421, 2021.","title":"Using self-supervised auxiliary tasks to improve fine-grained facial representation","authors":"Pourmirzaei","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rothe et al., 2015] Rasmus Rothe, Radu Timofte, and Luc Van Gool. DEX: Deep expectation of apparent age from a single image. In Proceedings of the International Confer- ence on Computer Vision (CVPR) Workshops, pages 10- 15. IEEE, 2015.","title":"DEX: Deep expectation of apparent age from a single image","authors":"Radu Rasmus Rothe, Luc Timofte, Van Gool","year":"2015","venue":"Proceedings of the International Conference on Computer Vision (CVPR) Workshops","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Savchenko, 2021] Andrey V Savchenko. Facial expression and attributes recognition based on multi-task learning of lightweight neural networks. In Proceedings of the 19th International Symposium on Intelligent Systems and Infor- matics (SISY), pages 119-124. IEEE, 2021.","title":"Facial expression and attributes recognition based on multi-task learning of lightweight neural networks","authors":"Andrey V Savchenko","year":"2021","venue":"Proceedings of the 19th International Symposium on Intelligent Systems and Informatics (SISY)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Savchenko, 2022] Andrey V Savchenko. MT-EmotiEffNet for multi-task human affective behavior analysis and learn- ing from synthetic data. In Proceedings of European Con- ference on Computer Vision (ECCV) Workshops, pages 45-59. Springer, 2022.","title":"MT-EmotiEffNet for multi-task human affective behavior analysis and learning from synthetic data","authors":"Andrey V Savchenko","year":"2022","venue":"Proceedings of European Conference on Computer Vision (ECCV) Workshops","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv.2302.13971, 2023.","title":"LLaMA: Open and efficient foundation language models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wallace et al., 2022] William Wallace, Calvin Chan, Swathikan Chidambaram, Lydia Hanna, Fahad Mu- jtaba Iqbal, Amish Acharya, Pasha Normahani, Hutan Ashrafian, Sheraz R Markar, Viknesh Sounderajah, et al. The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review. NPJ Digital Medicine, 5(1):118, 2022.","title":"The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review","authors":"William Wallace, Calvin Chan, Swathikan Chidambaram, Lydia Hanna, Fahad Mujtaba Iqbal, Amish Acharya, Pasha Normahani, Hutan Ashrafian, Viknesh Sheraz R Markar, Sounderajah","year":"2022","venue":"NPJ Digital Medicine","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2017] Wenjin Wang, Albertus C. den Brinker, Sander Stuijk, and Gerard de Haan. Algorithmic princi- ples of remote PPG. IEEE Transactions on Biomedical Engineering, 64(7):1479-1491, 2017.","title":"Algorithmic principles of remote PPG","authors":"Wenjin Wang, C Albertus, Sander Brinker, Gerard Stuijk, De Haan","year":"2017","venue":"IEEE Transactions on Biomedical Engineering","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2018] Chen Wang, Thierry Pun, and Guillaume Chanel. A comparative survey of methods for remote heart rate detection from frontal face videos. Frontiers in Bio- engineering and Biotechnology, 6:33, 05 2018.","title":"A comparative survey of methods for remote heart rate detection from frontal face videos","authors":"Chen Wang, Thierry Pun, Guillaume Chanel","year":null,"venue":"Frontiers in Bioengineering and Biotechnology","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wei et al., 2022] Jason Wei, Xuezhi Wang, Dale Schuur- mans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.","title":"Chain-of-thought prompting elicits reasoning in large language models","authors":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, V Quoc, Denny Le, Zhou","year":"2022","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2017] Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/regression by conditional adversar- ial autoencoder. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 5810-5818. IEEE, 2017.","title":"Age progression/regression by conditional adversarial autoencoder","authors":"Zhang","year":"2017","venue":"Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}