{"type":"ScholarlyArticle","title":"ScreenAI: A Vision-Language Model for UI and Infographics Understanding","authors":[{"name":"Gilles Baechler","slug":"gilles-baechler","given":"Gilles","family":"Baechler","affiliation":""},{"name":"Srinivas Sunkara","slug":"srinivas-sunkara","given":"Srinivas","family":"Sunkara","affiliation":""},{"name":"Maria Wang","slug":"maria-wang","given":"Maria","family":"Wang","affiliation":""},{"name":"Fedir Zubach","slug":"fedir-zubach","given":"Fedir","family":"Zubach","affiliation":""},{"name":"Hassan Mansoor","slug":"hassan-mansoor","given":"Hassan","family":"Mansoor","affiliation":""},{"name":"Vincent Etter","slug":"vincent-etter","given":"Vincent","family":"Etter","affiliation":""},{"name":"Victor Carbune","slug":"victor-carbune","given":"Victor","family":"Carbune","affiliation":""},{"name":"Jason Lin","slug":"jason-lin","given":"Jason","family":"Lin","affiliation":""},{"name":"Jindong Chen","slug":"jindong-chen","given":"Jindong","family":"Chen","affiliation":""},{"name":"Abhanshu Sharma","slug":"abhanshu-sharma","given":"Abhanshu","family":"Sharma","affiliation":""}],"abstract":"Screen user interfaces (UIs) and infographics, sharing similar visual language and design principles, play important roles in human communication and human-machine interaction.\r\nWe introduce ScreenAI, a vision-language model that specializes in UI and infographics understanding.\r\nOur model improves upon the PaLI architecture with the flexible patching strategy of pix2struct and is trained on a unique mixture of datasets.\r\nAt the heart of this mixture is a novel screen annotation task in which the model has to identify the type and location of UI elements.\r\nWe use these text annotations to describe screens to Large Language Models and automatically generate question-answering (QA), UI navigation, and summarization training datasets at scale.\r\nWe run ablation studies to demonstrate the impact of these design choices.\r\nAt only 5B parameters, ScreenAI achieves new state-of-the-art results\r\non UI- and infographics-based tasks (Multipage DocVQA, WebSRC, and MoTIF), and new best-in-class performance on others (ChartQA, DocVQA, and InfographicVQA) compared to models of similar size.\r\nFinally, we release three new datasets: one focused on the screen annotation task and two others focused on question answering.","doi":"10.24963/ijcai.2024/339","url":"https://ijcai.org/proceedings/2024/339","pdf":"https://ijcai.org/proceedings/2024/0339.pdf","markdown":"https://ijcai.org/proceedings/2024/339.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/339","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Humans and AI","track":"Main Track","number":339,"pages":{"first":3058,"last":3068},"keywords":["Machine Learning: ML: Multi-modal learning","Machine Learning: ML: Multi-task and transfer learning","Computer Vision: CV: Vision, language and reasoning","Humans and AI: HAI: Human-computer interaction"],"videos":[],"references":[{"position":1,"raw":"[Aggarwal et al., 2023] Kriti Aggarwal, Aditi Khandelwal, Kumar Tanmay, Owais Mohammed Khan, Qiang Liu, Monojit Choud- hury, Subhojit Som, Vishrav Chaudhary, and Saurabh Tiwary. DUBLIN-document understanding by language-image network. arXiv preprint arXiv:2305.14218, 2023.","title":"DUBLIN-document understanding by language-image network","authors":"Kriti Aggarwal, Aditi Khandelwal, Kumar Tanmay, Owais Mohammed Khan, Qiang Liu, Monojit Choudhury, Subhojit Som, Vishrav Chaudhary, Saurabh Tiwary","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aghajanyan et al., 2021] Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. HTLM: Hyper-text pre-training and prompting of language models, 2021.","title":"HTLM: Hyper-text pre-training and prompting of language models","authors":"Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, Luke Zettlemoyer","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Alayrac et al., 2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neu- ral Information Processing Systems, 35:23716-23736, 2022.","title":"Flamingo: a visual language model for few-shot learning","authors":"Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Anil et al., 2023a] Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalk- wyk, Andrew M Dai, Anja Hauth, et al. Gemini: a fam- ily of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.","title":"Gemini: a family of highly capable multimodal models","authors":"Anil","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Anil et al., 2023b] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. PaLM 2 technical report. arXiv preprint arXiv:2305.10403, 2023.","title":"PaLM 2 technical report","authors":"Anil","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bai et al., 2021] Chongyang Bai, Xiaoxue Zang, Ying Xu, Srini- vas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Aguera y Arcas. UIBert: Learning generic multimodal representations for UI understanding, 2021.","title":"UIBert: Learning generic multimodal representations for UI understanding","authors":"Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, Blaise Aguera Y Arcas","year":"2021","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2021/235"},{"position":7,"raw":"[Burns et al., 2022] Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. A dataset for interactive vision language navigation with unknown com- mand feasibility. In European Conference on Computer Vision (ECCV), 2022.","title":"A dataset for interactive vision language navigation with unknown command feasibility","authors":"Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, Bryan A Plummer","year":"2022","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Carbune et al., 2024] Victor Carbune, Hassan Mansoor, Fangyu Liu, Rahul Aralikatte, Gilles Baechler, Jindong Chen, and Ab- hanshu Sharma. Chart-based reasoning: Transferring capabilities from llms to vlms, 2024.","title":"Chart-based reasoning: Transferring capabilities from llms to vlms","authors":"Hassan Victor Carbune, Fangyu Mansoor, Rahul Liu, Gilles Aralikatte, Jindong Baechler, Abhanshu Chen, Sharma","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":null,"venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2021a] Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton. Pix2seq: A language mod- eling framework for object detection. arXiv preprint arXiv:2109.10852, 2021.","title":"Pix2seq: A language modeling framework for object detection","authors":"Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chen et al., 2021b] Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. WebSRC: A dataset for web-based structural reading comprehension, 2021.","title":"WebSRC: A dataset for web-based structural reading comprehension","authors":"Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chen et al., 2022] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Good- man, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. PaLi: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022.","title":"PaLi: A jointly-scaled multilingual language-image model","authors":"Xi Chen, Xiao Wang, Soravit Changpinyo, Piotr Piergiovanni, Daniel Padlewski, Sebastian Salz, Adam Goodman, Basil Grycner, Lucas Mustafa, Beyer","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chen et al., 2023a] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. PaLI-X: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565, 2023.","title":"PaLI-X: On scaling up a multilingual vision and language model","authors":"Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Chen et al., 2023b] Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebas- tian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. PaLI-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199, 2023.","title":"PaLI-3 vision language models: Smaller, faster, stronger","authors":"Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Deka et al., 2017] Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th an- nual ACM symposium on user interface software and technology, pages 845-854, 2017.","title":"Rico: A mobile app dataset for building data-driven design applications","authors":"Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, Ranjitha Kumar","year":"2017","venue":"Proceedings of the 30th annual ACM symposium on user interface software and technology","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Deng et al., 2023] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. arXiv preprint arXiv:2306.06070, 2023.","title":"Mind2web: Towards a generalist agent for the web","authors":"Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, Yu Su","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Gehrmann et al., 2022] Sebastian Gehrmann, Sebastian Ruder, Vi- taly Nikolaev, Jan A. Botha, Michael Chavinda, Ankur Parikh, and Clara Rivera. Tata: A multilingual table-to-text dataset for african languages, 2022.","title":"Tata: A multilingual table-to-text dataset for african languages","authors":"Sebastian Gehrmann, Sebastian Ruder, Vitaly Nikolaev, Jan A Botha, Michael Chavinda, Ankur Parikh, Clara Rivera","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Gur et al., 2022] Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sha- ran Narang, Noah Fiedel, and Aleksandra Faust. Under- standing HTML with large language models. arXiv preprint arXiv:2210.03945, 2022.","title":"Understanding HTML with large language models","authors":"Gur","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[He et al., 2021] Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schubiner, Ruby Lee, Jindong Chen, and Blaise Agüera y Arcas. ActionBert: Leveraging user actions for semantic understanding of user in- terfaces, 2021.","title":"ActionBert: Leveraging user actions for semantic understanding of user interfaces","authors":"He","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Hsiao et al., 2022] Yu-Chung Hsiao, Fedir Zubach, Maria Wang, et al. ScreenQA: Large-scale question-answer pairs over mobile app screenshots. arXiv preprint arXiv:2209.08199, 2022.","title":"ScreenQA: Large-scale question-answer pairs over mobile app screenshots","authors":"Yu-Chung Hsiao, Fedir Zubach, Maria Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Huang et al., 2022] Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. LayoutLMv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4083- 4091, 2022.","title":"LayoutLMv3: Pre-training for document ai with unified text and image masking","authors":"Huang","year":"2022","venue":"Proceedings of the 30th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kafle et al., 2018] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering, 2018.","title":"Dvqa: Understanding data visualizations via question answering","authors":"Kushal Kafle, Brian Price, Scott Cohen, Christopher Kanan","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kil et al., 2023] Jihyung Kil, Soravit Changpinyo, Xi Chen, Hexi- ang Hu, Sebastian Goodman, Wei-Lun Chao, and Radu Soricut. PreSTU: Pre-training for scene-text understanding. In Proceed- ings of the IEEE/CVF International Conference on Computer Vi- sion, pages 15270-15280, 2023.","title":"PreSTU: Pre-training for scene-text understanding","authors":"Kil","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Kim et al., 2021] Geewook Kim, Teakgyu Hong, Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Donut: Docu- ment understanding transformer without OCR. arXiv preprint arXiv:2111.15664, 7:15, 2021.","title":"Donut: Document understanding transformer without OCR","authors":"Kim","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Kuo et al., 2023] Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew Dai, Zhifeng Chen, et al. MaMMUT: A simple archi- tecture for joint learning for multimodal tasks. arXiv preprint arXiv:2303.16839, 2023.","title":"MaMMUT: A simple architecture for joint learning for multimodal tasks","authors":"Weicheng Kuo, Dahun Piergiovanni, Xiyang Kim, Ben Luo, Wei Caine, Abhijit Li, Luowei Ogale, Andrew Zhou, Zhifeng Dai, Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Lee et al., 2023] Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Ur- vashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for vi- sual language understanding. In International Conference on Ma- chine Learning, pages 18893-18912. PMLR, 2023.","title":"Pix2struct: Screenshot parsing as pretraining for visual language understanding","authors":"Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Li and Li, 2022] Gang Li and Yang Li. Spotlight: Mobile UI un- derstanding using vision-language models with a focus. arXiv preprint arXiv:2209.14927, 2022.","title":"Spotlight: Mobile UI understanding using vision-language models with a focus","authors":"Li ; Li, Gang Li, Yang Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Li et al., 2020] Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natu- ral language description for mobile user interface elements, 2020.","title":"Widget captioning: Generating natural language description for mobile user interface elements","authors":"Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, Zhiwei Guan","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Li et al., 2021] Yang Li, Gang Li, Xin Zhou, Mostafa Dehghani, and Alexey Gritsenko. VUT: Versatile ui transformer for multi-modal multi-task user interface modeling. arXiv preprint arXiv:2112.05692, 2021.","title":"VUT: Versatile ui transformer for multi-modal multi-task user interface modeling","authors":"Yang Li, Gang Li, Xin Zhou, Mostafa Dehghani, Alexey Gritsenko","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Li et al., 2022a] Gang Li, Gilles Baechler, Manuel Tragut, and Yang Li. Learning to denoise raw mobile UI layouts for improv- ing datasets at scale. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1-13, 2022.","title":"Learning to denoise raw mobile UI layouts for improving datasets at scale","authors":"Li","year":"2022","venue":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Li et al., 2022b] Tao Li, Gang Li, Jingjie Zheng, Purple Wang, and Yang Li. MUG: Interactive multimodal grounding on user inter- faces, 2022.","title":"MUG: Interactive multimodal grounding on user interfaces","authors":"Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Liu et al., 2022] Fangyu Liu, Francesco Piccinno, Syrine Krich- ene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. MatCha: Enhanc- ing visual language pretraining with math reasoning and chart derendering. arXiv preprint arXiv:2212.09662, 2022.","title":"MatCha: Enhancing visual language pretraining with math reasoning and chart derendering","authors":"Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, Julian Martin, Eisenschlos","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Liu et al., 2023] Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. DePlot: One-shot visual language reasoning by plot-to-table translation, 2023.","title":"DePlot: One-shot visual language reasoning by plot-to-table translation","authors":"Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, Yasemin Altun","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Masry et al., 2022] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reason- ing. arXiv preprint arXiv:2203.10244, 2022.","title":"ChartQA: A benchmark for question answering about charts with visual and logical reasoning","authors":"Ahmed Masry, Xuan Do, Jia Long, Shafiq Qing Tan, Enamul Joty, Hoque","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Masry et al., 2023] Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning, 2023.","title":"Unichart: A universal vision-language pretrained model for chart comprehension and reasoning","authors":"Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, Shafiq Joty","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Mathew et al., 2021] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. DocVQA: A dataset for VQA on document images. In Proceedings of the IEEE/CVF winter conference on applica- tions of computer vision, pages 2200-2209, 2021.","title":"DocVQA: A dataset for VQA on document images","authors":"Minesh Mathew, Dimosthenis Karatzas, Jawahar","year":"2021","venue":"Proceedings of the IEEE/CVF winter conference on applications of computer vision","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Mathew et al., 2022] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. In- fographicVQA. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision, pages 1697-1706, 2022.","title":"In-fographicVQA","authors":"Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, Jawahar","year":"2022","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Mishra et al., 2019] Anand Mishra, Shashank Shekhar, Ajeet Ku- mar Singh, and Anirban Chakraborty. OCR-VQA: Visual ques- tion answering by reading text in images. In ICDAR, 2019.","title":"OCR-VQA: Visual question answering by reading text in images","authors":"Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, Anirban Chakraborty","year":"2019","venue":"ICDAR","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Nakano et al., 2021] Reiichiro Nakano, Jacob Hilton, Suchir Bal- aji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. We- bGPT: Browser-assisted question-answering with human feed- back. arXiv preprint arXiv:2112.09332, 2021.","title":"We-bGPT: Browser-assisted question-answering with human feedback","authors":"Nakano","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Powalski et al., 2021] Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, and Gabriela Pałka. Going full-tilt boogie on document understanding with text-image-layout transformer, 2021.","title":"Going full-tilt boogie on document understanding with text-image-layout transformer","authors":"Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, Gabriela Pałka","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551, 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2020","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Rawles et al., 2023] Christopher Rawles, Alice Li, Daniel Ro- driguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control. arXiv preprint arXiv:2307.10088, 2023.","title":"Android in the wild: A large-scale dataset for android device control","authors":"Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, Timothy Lillicrap","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Sharma et al., 2018] Piyush Sharma, Nan Ding, Sebastian Good- man, and Radu Soricut. Conceptual captions: A cleaned, hy- pernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, 2018.","title":"Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning","authors":"Piyush Sharma, Nan Ding, Sebastian Goodman, Radu Soricut","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Sunkara et al., 2022] Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Abhanshu Sharma, James Stout, et al. Towards better semantic understanding of mobile interfaces. arXiv preprint arXiv:2210.02663, 2022.","title":"Towards better semantic understanding of mobile interfaces","authors":"Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Abhanshu Sharma, James Stout","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Tang et al., 2023] Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. Unifying vision, text, and layout for universal document processing. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 19254- 19264, 2023.","title":"Unifying vision, text, and layout for universal document processing","authors":"Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, Mohit Bansal","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Tay et al., 2022] Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. UL2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations, 2022.","title":"UL2: Unifying language learning paradigms","authors":"Yi Tay, Mostafa Dehghani, Xavier Vinh Q Tran, Jason Garcia, Xuezhi Wei, Hyung Won Wang, Dara Chung, Tal Bahri, Steven Schuster, Zheng","year":"2022","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Tito et al., 2023] Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Hierarchical multimodal transformers for multipage DocVQA. Pattern Recognition, 144:109834, 2023.","title":"Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Hierarchical multimodal transformers for multipage DocVQA","authors":"Tito","year":"2023","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Par- mar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Wang et al., 2021] Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile ui summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Tech- nology, pages 498-510, 2021.","title":"Screen2words: Automatic mobile ui summarization with multimodal learning","authors":"Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, Yang Li","year":"2021","venue":"The 34th Annual ACM Symposium on User Interface Software and Technology","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Wang et al., 2022] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning, pages 23318-23340. PMLR, 2022.","title":"OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework","authors":"Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yang","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Wang et al., 2023] Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, and Xiaomo Liu. DocLLM: A layout- aware generative language model for multimodal document un- derstanding. arXiv preprint arXiv:2401.00908, 2023.","title":"DocLLM: A layoutaware generative language model for multimodal document understanding","authors":"Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, Xiaomo Liu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Wichers et al., 2018] Nevan Wichers, Dilek Hakkani-Tür, and Jin- dong Chen. Resolving referring expressions in images with la- beled elements. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 800-806. IEEE, 2018.","title":"Resolving referring expressions in images with labeled elements","authors":"Nevan Wichers, Dilek Hakkani-Tür, Jindong Chen","year":"2018","venue":"IEEE Spoken Language Technology Workshop (SLT)","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Wu et al., 2021] Jason Wu, Xiaoyi Zhang, Jeff Nichols, and Jef- frey P Bigham. Screen parsing: Towards reverse engineering of ui models from screenshots. In The 34th Annual ACM Sympo- sium on User Interface Software and Technology, pages 470-483, 2021.","title":"Screen parsing: Towards reverse engineering of ui models from screenshots","authors":"Jason Wu, Xiaoyi Zhang, Jeff Nichols, Jeffrey P Bigham","year":"2021","venue":"The 34th Annual ACM Symposium on User Interface Software and Technology","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Wu et al., 2023a] Jialin Wu, Xia Hu, Yaqing Wang, Bo Pang, and Radu Soricut. Omni-SMoLA: Boosting generalist multimodal models with soft mixture of low-rank experts, 2023.","title":"Omni-SMoLA: Boosting generalist multimodal models with soft mixture of low-rank experts","authors":"Wu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Wu et al., 2023b] Sijin Wu, Dan Zhang, Teng Hu, and Shikun Feng. DocPrompt: Large-scale continue pretrain for zero-shot and few-shot document question answering, 2023.","title":"DocPrompt: Large-scale continue pretrain for zero-shot and few-shot document question answering","authors":"Wu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Xue et al., 2020] Linting Xue, Noah Constant, Adam Roberts, Mi- hir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text- to-text transformer. arXiv preprint arXiv:2010.11934, 2020.","title":"mT5: A massively multilingual pre-trained textto-text transformer","authors":"Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Yang et al., 2022] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xi- aowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. UniTAB: Unifying text and box outputs for grounded vision-language modeling. In European Conference on Com- puter Vision, pages 521-539. Springer, 2022.","title":"UniTAB: Unifying text and box outputs for grounded vision-language modeling","authors":"Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, Lijuan Wang","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Zang et al., 2021] Xiaoxue Zang, Ying Xu, and Jindong Chen. Multimodal icon annotation for mobile applications. In Pro- ceedings of the 23rd International Conference on Mobile Human- Computer Interaction, pages 1-11, 2021.","title":"Multimodal icon annotation for mobile applications","authors":"Zang","year":"2021","venue":"Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction","doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Zhang et al., 2021] Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jef- frey Nichols, Jason Wu, Chris Fleizach, et al. Screen recognition: Creating accessibility metadata for mobile applications from pix- els. In Proceedings of the 2021 CHI Conference on Human Fac- tors in Computing Systems, pages 1-15, 2021.","title":"Screen recognition: Creating accessibility metadata for mobile applications from pixels","authors":"Xiaoyi Zhang, Lilian De Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach","year":"2021","venue":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}