{"type":"ScholarlyArticle","title":"Learning Label Dependencies for Visual Information Extraction","authors":[{"name":"Minghong Yao","slug":"minghong-yao","given":"Minghong","family":"Yao","affiliation":"University of Science and Technology of China"},{"name":"Liansheng Zhuang","slug":"liansheng-zhuang","given":"Liansheng","family":"Zhuang","affiliation":"University of Science and Technology of China"},{"name":"Houqiang Li","slug":"houqiang-li","given":"Houqiang","family":"Li","affiliation":"University of Science and Technology of China"},{"name":"Jiuchang Wei","slug":"jiuchang-wei","given":"Jiuchang","family":"Wei","affiliation":"University of Science and Technology of China"}],"abstract":"Visual Information Extraction (VIE), which aims to extract structured information from visually rich document images, has drawn much attention due to its wide applications in document understanding. However, previous methods often treat the VIE task as a sequence labeling problem and ignore the label correlations in the sequence, which may significantly degrade their performance. To address this issue, this paper proposes a novel framework to exploit the potential of label correlations to improve the VIE models' performance. Its key idea is to learn the label dependency of entities, and use it to regularize the label sequence. Specifically, to capture the label dependency of entities, a label transformer is pre-trained to assign a higher likelihood to the label sequence that respects the label patterns of document layouts. During testing stages, an inference transformer is used to predict the label sequence by considering not only the features of each entity but also the likelihood of the label sequence evaluated by the label transformer. Our framework can be combined with existing popular VIE models such as LayoutLM and GeoLayoutLM. Extensive experiments on public datasets have demonstrated the effectiveness of our framework.","doi":"10.24963/ijcai.2024/731","url":"https://ijcai.org/proceedings/2024/731","pdf":"https://ijcai.org/proceedings/2024/0731.pdf","markdown":"https://ijcai.org/proceedings/2024/731.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/731","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":731,"pages":{"first":6615,"last":6623},"keywords":["Natural Language Processing: NLP: Applications","Natural Language Processing: NLP: Information extraction"],"videos":[],"references":[{"position":1,"raw":"[Appalaraju et al., 2021] Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. Docformer: End-to-end transformer for document under- standing. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.","title":"Docformer: End-to-end transformer for document understanding","authors":"Srikar Appalaraju, Bhavan Jasani, Urala Bhargava, Yusheng Kota, Xie, Manmatha","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bai et al., 2019] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Advances in Neu- ral Information Processing Systems, 2019.","title":"Deep equilibrium models","authors":"J Zico Shaojie Bai, Vladlen Kolter, Koltun","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bai et al., 2020] Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Multiscale deep equilibrium models. Advances in Neural Information Processing Systems, 2020.","title":"Advances in Neural Information Processing Systems","authors":"Bai","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Belanger and McCallum, 2016] David Belanger and An- drew McCallum. Structured prediction energy networks. In Proceedings of The 33rd International Conference on Machine Learning, 2016.","title":"Structured prediction energy networks","authors":"Mccallum Belanger","year":"2016","venue":"Proceedings of The 33rd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Broyden, 1965] C. G. Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of Computation, 1965.","title":"A class of methods for solving nonlinear simultaneous equations","authors":"C G Broyden","year":"1965","venue":"Mathematics of Computation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cheng et al., 2020] Mengli Cheng, Minghui Qiu, Xing Shi, Jun Huang, and Wei Lin. One-shot text field labeling using attention and belief propagation for structure information extraction. Proceedings of the 28th ACM International Conference on Multimedia, 2020.","title":"One-shot text field labeling using attention and belief propagation for structure information extraction","authors":"Cheng","year":"2020","venue":"Proceedings of the 28th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cui, 2021] Lei Cui. Document ai: Benchmarks, models and applications (presentation@ icdar 2021). In DIL workshop in ICDAR, 2021.","title":"Document ai: Benchmarks, models and applications (presentation@ icdar 2021)","authors":"Lei Cui","year":"2021","venue":"DIL workshop in ICDAR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dupont et al., 2018] Yoann Dupont, Marco Dinarelli, and Isabelle Tellier. Label-dependencies aware recurrent neu- ral networks. In Computational Linguistics and Intelligent Text Processing: 18th International Conference, CICLing 2017, Budapest, Hungary, April 17-23, 2017, Revised Se- lected Papers, Part I 18, 2018.","title":"Label-dependencies aware recurrent neural networks","authors":"Yoann Dupont, Marco Dinarelli, Isabelle Tellier","year":"2017","venue":"Computational Linguistics and Intelligent Text Processing: 18th International Conference, CICLing 2017","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2022] Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, and Liqing Zhang. Xylayoutlm: Towards layout-aware multimodal networks for visually-rich document under- standing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.","title":"Xylayoutlm: Towards layout-aware multimodal networks for visually-rich document understanding","authors":"Gu","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Huang et al., 2015] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991, 2015.","title":"Bidirectional lstm-crf models for sequence tagging","authors":"Zhiheng Huang, Wei Xu, Kai Yu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2019] Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. Icdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Con- ference on Document Analysis and Recognition, 2019.","title":"Icdar2019 competition on scanned receipt ocr and information extraction","authors":"Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, Jawahar","year":"2019","venue":"2019 International Conference on Document Analysis and Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2022] Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Pro- ceedings of the 30th ACM International Conference on Multimedia, 2022.","title":"Layoutlmv3: Pre-training for document ai with unified text and image masking","authors":"Huang","year":"2022","venue":"Proceedings of the 30th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jaume et al., 2019] Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), 2019.","title":"Funsd: A dataset for form understanding in noisy scanned documents","authors":"Guillaume Jaume, Hazim Kemal Ekenel, Jean-Philippe Thiran","year":"2019","venue":"2019 International Conference on Document Analysis and Recognition Workshops (ICDARW)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2021a] Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. Struc- turalLM: Structural pre-training for form understanding. In Proceedings of the 59th Annual Meeting of the Associ- ation for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021.","title":"Struc-turalLM: Structural pre-training for form understanding","authors":"Li","year":"2021","venue":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2021b] Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Varun Man- junatha, and Hongfu Liu. Selfdoc: Self-supervised doc- ument representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.","title":"Selfdoc: Self-supervised document representation learning","authors":"Li","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2021c] Yulin Li, Yuxi Qian, Yuechen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, and Errui Ding. Structured text understanding with multi-modal transformers. In MM '21: ACM Multi- media Conference, 2021.","title":"Structured text understanding with multi-modal transformers","authors":"Li","year":"2021","venue":"MM '21: ACM Multimedia Conference","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2019] Xiaojing Liu, Feiyu Gao, Qiong Zhang, and Huasha Zhao. Graph convolution for multimodal in- formation extraction from visually rich documents. In Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Pa- pers), 2019.","title":"Graph convolution for multimodal information extraction from visually rich documents","authors":"Xiaojing Liu, Feiyu Gao, Qiong Zhang, Huasha Zhao","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Luo et al., 2023] Chuwei Luo, Changxu Cheng, Qi Zheng, and Cong Yao. Geolayoutlm: Geometric pre-training for visual information extraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.","title":"Geolayoutlm: Geometric pre-training for visual information extraction","authors":"Luo","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nowozin et al., 2011] Sebastian Nowozin, Christoph H Lampert, et al. Structured learning and prediction in computer vision. Foundations and Trends® in Computer Graphics and Vision, 2011.","title":"Structured learning and prediction in computer vision","authors":"Sebastian Nowozin, Christoph H Lampert","year":"2011","venue":"Foundations and Trends® in Computer Graphics and Vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Park et al., 2019] Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwal- suk Lee. CORD: a consolidated receipt dataset for post- ocr parsing. In Workshop on Document Intelligence at NeurIPS 2019, 2019.","title":"CORD: a consolidated receipt dataset for postocr parsing","authors":"Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, Hwalsuk Lee","year":"2019","venue":"Workshop on Document Intelligence at NeurIPS 2019","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Qian et al., 2019] Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, and Regina Barzilay. GraphIE: A graph-based framework for information extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Papers), 2019.","title":"GraphIE: A graph-based framework for information extraction","authors":"Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, Regina Barzilay","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rrubaa and Amaresan, 2018] Panchendrarajan Rrubaa and Aravindh Amaresan. Bidirectional LSTM-CRF for named entity recognition. In Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation, 2018.","title":"Panchendrarajan Rrubaa and Aravindh Amaresan. Bidirectional LSTM-CRF for named entity recognition","authors":"Amaresan Rrubaa","year":"2018","venue":"Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang et al., 2021] Guozhi Tang, Lele Xie, Lianwen Jin, Ji- apeng Wang, Jingdong Chen, Zhen Xu, Qianying Wang, Yaqiang Wu, and Hui Li. Matchvie: Exploiting match rel- evancy between entities for visual information extraction. In Proceedings of the Thirtieth International Joint Confer- ence on Artificial Intelligence, 2021.","title":"Matchvie: Exploiting match relevancy between entities for visual information extraction","authors":"Guozhi Tang, Lele Xie, Lianwen Jin, Jiapeng Wang, Jingdong Chen, Zhen Xu, Qianying Wang, Yaqiang Wu, Hui Li","year":"2021","venue":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2021/144"},{"position":25,"raw":"[Tu and Gimpel, 2018] Lifu Tu and Kevin Gimpel. Learning approximate inference networks for structured prediction. In International Conference on Learning Representations, 2018.","title":"Learning approximate inference networks for structured prediction","authors":"Gimpel ; Tu, Lifu Tu, Kevin Gimpel","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems (NeurIPS), 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2021] Yiheng Wang, Zilong nd Xu, Lei Cui, Jingbo Shang, and Furu Wei. LayoutReader: Pre-training of text and layout for reading order detection. In Proceed- ings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, 2021.","title":"LayoutReader: Pre-training of text and layout for reading order detection","authors":"Wang","year":"2021","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2022] Jiapeng Wang, Lianwen Jin, and Kai Ding. Lilt: A simple yet effective language-independent layout transformer for structured document understanding. arXiv preprint arXiv:2202.13669, 2022.","title":"Lilt: A simple yet effective language-independent layout transformer for structured document understanding","authors":"Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xu et al., 2020] Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining, 2020.","title":"Layoutlm: Pre-training of text and layout for document image understanding","authors":"Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou","year":"2020","venue":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2021] Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. LayoutLMv2: Multi-modal pre-training for visually-rich document understanding. In Proceedings of the 59th An- nual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021.","title":"LayoutLMv2: Multi-modal pre-training for visually-rich document understanding","authors":"Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou","year":"2021","venue":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yao et al., 2021] Minghong Yao, Zhiguang Liu, Liangwei Wang, Houqiang Li, and Liansheng Zhuang. One-shot key information extraction from document with deep partial graph matching. arXiv preprint arXiv:2109.13967, 2021.","title":"One-shot key information extraction from document with deep partial graph matching","authors":"Yao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yu et al., 2020] Wenwen Yu, Ning Lu, Xianbiao Qi, Ping Gong, and Rong Xiao. PICK: Processing key infor- mation extraction from documents using improved graph learning-convolutional networks. In 2020 25th Interna- tional Conference on Pattern Recognition (ICPR), 2020.","title":"PICK: Processing key information extraction from documents using improved graph learning-convolutional networks","authors":"Wenwen Yu, Ning Lu, Xianbiao Qi, Ping Gong, Rong Xiao","year":"2020","venue":"2020 25th International Conference on Pattern Recognition (ICPR)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2021] Yue Zhang, Zhang Bo, Rui Wang, Jun- jie Cao, Chen Li, and Zuyi Bao. Entity relation extraction as dependency parsing in visually rich documents. In Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.","title":"Entity relation extraction as dependency parsing in visually rich documents","authors":"Yue Zhang, Zhang Bo, Rui Wang, Junjie Cao, Chen Li, Zuyi Bao","year":"2021","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}