MatchVIE: Exploiting Match Relevancy between Entities for Visual Information Extraction

Tang, Guozhi; Xie, Lele; Jin, Lianwen; Wang, Jiapeng; Chen, Jingdong; Xu, Zhen; Wang, Qianying; Wu, Yaqiang; Li, Hui

doi:10.24963/ijcai.2021/144

MatchVIE: Exploiting Match Relevancy between Entities for Visual Information Extraction

Guozhi Tang, Lele Xie, Lianwen Jin, Jiapeng Wang, Jingdong Chen, Zhen Xu, Qianying Wang, Yaqiang Wu, Hui Li

Download BibTeX Cite

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. Main Track. Pages 1039-1045.

https://doi.org/10.24963/ijcai.2021/144

Keywords

Natural Language Processing: Information Extraction
Computer Vision: Structural and Model-Based Approaches, Knowledge Representation and Reasoning
Computer Vision: Language and Vision

Abstract

Visual Information Extraction (VIE) task aims to extract key information from multifarious document images (e.g., invoices and purchase receipts). Most previous methods treat the VIE task simply as a sequence labeling problem or classification problem, which requires models to carefully identify each kind of semantics by introducing multimodal features, such as font, color, layout. But simply introducing multimodal features can't work well when faced with numeric semantic categories or some ambiguous texts. To address this issue, in this paper we propose a novel key-value matching model based on a graph neural network for VIE (MatchVIE). Through key-value matching based on relevancy evaluation, the proposed MatchVIE can bypass the recognitions to various semantics, and simply focuses on the strong relevancy between entities. Besides, we introduce a simple but effective operation, Num2Vec, to tackle the instability of encoded values, which helps model converge more smoothly. Comprehensive experiments demonstrate that the proposed MatchVIE can significantly outperform previous methods. Notably, to the best of our knowledge, MatchVIE may be the first attempt to tackle the VIE task by modeling the relevancy between keys and values and it is a good complement to the existing methods.

Guozhi Tang, Lele Xie, Lianwen Jin, Jiapeng Wang, Jingdong Chen, Zhen Xu, Qianying Wang, Yaqiang Wu, and Hui Li. 2021. MatchVIE: Exploiting Match Relevancy between Entities for Visual Information Extraction. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. IJCAI, 1039–1045. https://doi.org/10.24963/ijcai.2021/144

@inproceedings{ijcai2021p144,
  title     = {MatchVIE: Exploiting Match Relevancy between Entities for Visual Information Extraction},
  author    = {Tang, Guozhi and Xie, Lele and Jin, Lianwen and Wang, Jiapeng and Chen, Jingdong and Xu, Zhen and Wang, Qianying and Wu, Yaqiang and Li, Hui},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Prof. Zhi-Hua Zhou},
  pages     = {1039--1045},
  year      = {2021},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2021/144},
  url       = {https://doi.org/10.24963/ijcai.2021/144},
}

TY  - CPAPER
TI  - MatchVIE: Exploiting Match Relevancy between Entities for Visual Information Extraction
AU  - Tang, Guozhi
AU  - Xie, Lele
AU  - Jin, Lianwen
AU  - Wang, Jiapeng
AU  - Chen, Jingdong
AU  - Xu, Zhen
AU  - Wang, Qianying
AU  - Wu, Yaqiang
AU  - Li, Hui
T2  - Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2021
SP  - 1039
EP  - 1045
DO  - 10.24963/ijcai.2021/144
UR  - https://doi.org/10.24963/ijcai.2021/144
ER  -