UIBert: Learning Generic Multimodal Representations for UI Understanding

Bai, Chongyang; Zang, Xiaoxue; Xu, Ying; Sunkara, Srinivas; Rastogi, Abhinav; Chen, Jindong; Agüera y Arcas, Blaise

doi:10.24963/ijcai.2021/235

UIBert: Learning Generic Multimodal Representations for UI Understanding

Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, Blaise Agüera y Arcas

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Main Track. Pages 1705-1712.

https://doi.org/10.24963/ijcai.2021/235

Keywords

Humans and AI: Human-Computer Interaction

Abstract

To improve the accessibility of smart devices and to simplify their usage, building models which understand user interfaces (UIs) and assist users to complete their tasks is critical. However, unique challenges are proposed by UI-specific characteristics, such as how to effectively leverage multimodal UI features that involve image, text, and structural metadata and how to achieve good performance when high-quality labeled data is unavailable. To address such challenges we introduce UIBert, a transformer-based joint image-text model trained through novel pre-training tasks on large-scale unlabeled UI data to learn generic feature representations for a UI and its components. Our key intuition is that the heterogeneous features in a UI are self-aligned, i.e., the image and text features of UI components, are predictive of each other. We propose five pretraining tasks utilizing this self-alignment among different features of a UI component and across various components in the same UI. We evaluate our method on nine real-world downstream UI tasks where UIBert outperforms strong multimodal baselines by up to 9.26% accuracy.

Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Agüera y Arcas. 2021. UIBert: Learning Generic Multimodal Representations for UI Understanding. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 1705–1712. https://doi.org/10.24963/ijcai.2021/235

@inproceedings{ijcai2021p235,
  title     = {UIBert: Learning Generic Multimodal Representations for UI Understanding},
  author    = {Bai, Chongyang and Zang, Xiaoxue and Xu, Ying and Sunkara, Srinivas and Rastogi, Abhinav and Chen, Jindong and Agüera y Arcas, Blaise},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Prof. Zhi-Hua Zhou},
  pages     = {1705--1712},
  year      = {2021},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2021/235},
  url       = {https://doi.org/10.24963/ijcai.2021/235},
}

TY  - CPAPER
TI  - UIBert: Learning Generic Multimodal Representations for UI Understanding
AU  - Bai, Chongyang
AU  - Zang, Xiaoxue
AU  - Xu, Ying
AU  - Sunkara, Srinivas
AU  - Rastogi, Abhinav
AU  - Chen, Jindong
AU  - Agüera y Arcas, Blaise
T2  - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2021
SP  - 1705
EP  - 1712
DO  - 10.24963/ijcai.2021/235
UR  - https://doi.org/10.24963/ijcai.2021/235
ER  -