{"type":"ScholarlyArticle","title":"TFLOP: Table Structure Recognition Framework with Layout Pointer Mechanism","authors":[{"name":"Minsoo Khang","slug":"minsoo-khang","given":"Minsoo","family":"Khang","affiliation":"Upstage AI , South Korea"},{"name":"Teakgyu Hong","slug":"teakgyu-hong","given":"Teakgyu","family":"Hong","affiliation":"Upstage AI , South Korea"}],"abstract":"Table Structure Recognition (TSR) is a task aimed at converting table images into a machine-readable format (e.g. HTML), to facilitate other applications such as information retrieval.\r\n    Recent works tackle this problem by identifying the HTML tags and text regions, where the latter is used for text extraction from the table document. These works however, suffer from misalignment issues when mapping text into the identified text regions. In this paper, we introduce a new TSR framework, called TFLOP (TSR Framework with LayOut Pointer mechanism), which reformulates the conventional text region prediction and matching into a direct text region pointing problem. Specifically, TFLOP utilizes text region information to identify both the table's structure tags and its aligned text regions, simultaneously. Without the need for region prediction and alignment, TFLOP circumvents the additional text region matching stage, which requires finely-calibrated post-processing. TFLOP also employs span-aware contrastive supervision to enhance the pointing mechanism in tables with complex structure. As a result, TFLOP achieves the state-of-the-art performance across multiple benchmarks such as PubTabNet, FinTabNet, and SynthTabNet. In our extensive experiments, TFLOP not only exhibits competitive performance but also shows promising results on industrial document TSR scenarios such as documents with watermarks or in non-English domain. Source code of our work is publicly available at: https://github.com/UpstageAI/TFLOP.","doi":"10.24963/ijcai.2024/105","url":"https://ijcai.org/proceedings/2024/105","pdf":"https://ijcai.org/proceedings/2024/0105.pdf","markdown":"https://ijcai.org/proceedings/2024/105.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/105","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":105,"pages":{"first":947,"last":955},"keywords":["Computer Vision: CV: Applications","Natural Language Processing: NLP: Applications","Computer Vision: CV: Recognition (object detection, categorization)"],"videos":[],"references":[{"position":1,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.","title":"End-to-end object detection with transformers","authors":"Carion","year":"2020","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deng et al., 2019] Yuntian Deng, David Rosenberg, and Gideon Mann. Challenges in end-to-end neural scientific table recognition. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 894- 901. IEEE, 2019.","title":"Challenges in end-to-end neural scientific table recognition","authors":"Deng","year":"2019","venue":"2019 International Conference on Document Analysis and Recognition (ICDAR)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Guo et al., 2022] Zengyuan Guo, Yuechen Yu, Pengyuan Lv, Chengquan Zhang, Haojie Li, Zhihui Wang, Kun Yao, Jingtuo Liu, and Jingdong Wang. Trust: An accurate and end-to-end table structure recognizer using splitting-based transformers. arXiv preprint arXiv:2208.14687, 2022.","title":"Trust: An accurate and end-to-end table structure recognizer using splitting-based transformers","authors":"Guo","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.","title":"Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn","authors":"He","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Huang et al., 2023] Yongshuai Huang, Ning Lu, Dapeng Chen, Yibo Li, Zecheng Xie, Shenggao Zhu, Liangcai Gao, and Wei Peng. Improving table structure recog- nition with visual-alignment sequential coordinate mod- eling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11134- 11143, 2023.","title":"Improving table structure recognition with visual-alignment sequential coordinate modeling","authors":"Yongshuai Huang, Ning Lu, Dapeng Chen, Yibo Li, Zecheng Xie, Shenggao Zhu, Liangcai Gao, Wei Peng","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Khosla et al., 2020] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised con- trastive learning. Advances in neural information process- ing systems, 33:18661-18673, 2020.","title":"Supervised contrastive learning","authors":"Khosla","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kim et al., 2022] Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Se- unghyun Park. Ocr-free document understanding trans- former. In European Conference on Computer Vision, pages 498-517. Springer, 2022.","title":"Ocr-free document understanding transformer","authors":"Kim","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Lewis et al., 2019] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: De- noising sequence-to-sequence pre-training for natural lan- guage generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.","title":"Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin et al., 2022] Weihong Lin, Zheng Sun, Chixiang Ma, Mingze Li, Jiawei Wang, Lei Sun, and Qiang Huo. Tsr- former: Table structure recognition with transformers. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6473-6482, 2022.","title":"Tsrformer: Table structure recognition with transformers","authors":"Weihong Lin, Zheng Sun, Chixiang Ma, Mingze Li, Jiawei Wang, Lei Sun, Qiang Huo","year":"2022","venue":"Proceedings of the 30th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2021a] Hao Liu, Xin Li, Bing Liu, Deqiang Jiang, Yinsong Liu, Bo Ren, and Rongrong Ji. Show, read and reason: Table structure recognition with flexible con- text aggregator. In Proceedings of the 29th ACM Interna- tional Conference on Multimedia, pages 1084-1092, 2021.","title":"Show, read and reason: Table structure recognition with flexible context aggregator","authors":"Liu","year":"2021","venue":"Proceedings of the 29th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2021b] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF in- ternational conference on computer vision, pages 10012- 10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Liu","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2022] Hao Liu, Xin Li, Bing Liu, Deqiang Jiang, Yinsong Liu, and Bo Ren. Neural collaborative graph ma- chines for table structure recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 4533-4542, 2022.","title":"Neural collaborative graph machines for table structure recognition","authors":"Hao Liu, Xin Li, Bing Liu, Deqiang Jiang, Yinsong Liu, Bo Ren","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lu et al., 2021] Ning Lu, Wenwen Yu, Xianbiao Qi, Yihao Chen, Ping Gong, Rong Xiao, and Xiang Bai. Master: Multi-aspect non-local network for scene text recognition. Pattern Recognition, 117:107980, 2021.","title":"Master: Multi-aspect non-local network for scene text recognition","authors":"Ning Lu, Wenwen Yu, Xianbiao Qi, Yihao Chen, Ping Gong, Rong Xiao, Xiang Bai","year":"2021","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lysak et al., 2023] Maksym Lysak, Ahmed Nassar, Niko- laos Livathinos, Christoph Auer, and Peter Staar. Opti- mized table tokenization for table structure recognition. arXiv preprint arXiv:2305.03393, 2023.","title":"Optimized table tokenization for table structure recognition","authors":"Maksym Lysak, Ahmed Nassar, Nikolaos Livathinos, Christoph Auer, Peter Staar","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lyu et al., 2023] Pengyuan Lyu, Weihong Ma, Hongyi Wang, Yuechen Yu, Chengquan Zhang, Kun Yao, Yang Xue, and Jingdong Wang. Gridformer: Towards accurate table structure recognition via grid prediction. In Proceed- ings of the 31st ACM International Conference on Multi- media, pages 7747-7757, 2023.","title":"Gridformer: Towards accurate table structure recognition via grid prediction","authors":"Lyu","year":"2023","venue":"Proceedings of the 31st ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2023] Chixiang Ma, Weihong Lin, Lei Sun, and Qiang Huo. Robust table detection and structure recogni- tion from heterogeneous document images. Pattern Recog- nition, 133:109006, 2023.","title":"Robust table detection and structure recognition from heterogeneous document images","authors":"Ma","year":"2023","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nassar et al., 2022] Ahmed Nassar, Nikolaos Livathinos, Maksym Lysak, and Peter Staar. Tableformer: Table struc- ture understanding with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 4614-4623, 2022.","title":"Tableformer: Table structure understanding with transformers","authors":"Ahmed Nassar, Nikolaos Livathinos, Maksym Lysak, Peter Staar","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[OpenAI, 2023] OpenAI. Gpt-4v(ision) system card. 2023.","title":"OpenAI. Gpt-4v(ision) system card","authors":"Openai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Paliwal et al., 2019] Shubham Singh Paliwal, D Vish- wanath, Rohit Rahul, Monika Sharma, and Lovekesh Vig. Tablenet: Deep learning model for end-to-end table detec- tion and tabular data extraction from scanned document images. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 128-133. IEEE, 2019.","title":"Tablenet: Deep learning model for end-to-end table detection and tabular data extraction from scanned document images","authors":"Shubham Singh Paliwal, D Vishwanath, Rohit Rahul, Monika Sharma, Lovekesh Vig","year":"2019","venue":"2019 International Conference on Document Analysis and Recognition (ICDAR)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qiao et al., 2021] Liang Qiao, Zaisheng Li, Zhanzhan Cheng, Peng Zhang, Shiliang Pu, Yi Niu, Wenqi Ren, Wenming Tan, and Fei Wu. Lgpma: Complicated table structure recognition with local and global pyramid mask alignment. In International conference on document anal- ysis and recognition, pages 99-114. Springer, 2021.","title":"Lgpma: Complicated table structure recognition with local and global pyramid mask alignment","authors":"Liang Qiao, Zaisheng Li, Zhanzhan Cheng, Peng Zhang, Shiliang Pu, Yi Niu, Wenqi Ren, Wenming Tan, Fei Wu","year":"2021","venue":"International conference on document analysis and recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Raja et al., 2020] Sachin Raja, Ajoy Mondal, and CV Jawa- har. Table structure recognition using top-down and bottom-up cues. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII 16, pages 70-86. Springer, 2020.","title":"Table structure recognition using top-down and bottom-up cues","authors":"Sachin Raja, Ajoy Mondal, Jawahar","year":"2020","venue":"Computer Vision-ECCV 2020: 16th European Conference","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schreiber et al., 2017] Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Dengel, and Sheraz Ahmed. Deepdesrt: Deep learning for detection and structure recognition of tables in document images. In 2017 14th IAPR interna- tional conference on document analysis and recognition (ICDAR), volume 1, pages 1162-1167. IEEE, 2017.","title":"Deepdesrt: Deep learning for detection and structure recognition of tables in document images","authors":"Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Dengel, Sheraz Ahmed","year":"2017","venue":"14th IAPR international conference on document analysis and recognition (ICDAR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shen et al., 2023] Huawen Shen, Xiang Gao, Jin Wei, Liang Qiao, Yu Zhou, Qiang Li, and Zhanzhan Cheng. Divide rows and conquer cells: Towards structure recognition for large tables. In Proceedings of the Thirty-Second Inter- national Joint Conference on Artificial Intelligence, pages 1369-1377, 2023.","title":"Divide rows and conquer cells: Towards structure recognition for large tables","authors":"Huawen Shen, Xiang Gao, Jin Wei, Liang Qiao, Yu Zhou, Qiang Li, Zhanzhan Cheng","year":"2023","venue":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2023/152"},{"position":24,"raw":"[Tensmeyer et al., 2019] Chris Tensmeyer, Vlad I Morariu, Brian Price, Scott Cohen, and Tony Martinez. Deep split- ting and merging for table structure decomposition. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 114-121. IEEE, 2019.","title":"Deep splitting and merging for table structure decomposition","authors":"Chris Tensmeyer, Brian Vlad I Morariu, Scott Price, Tony Cohen, Martinez","year":"2019","venue":"2019 International Conference on Document Analysis and Recognition (ICDAR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2019a] Wenhai Wang, Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Gang Yu, and Shuai Shao. Shape robust text detection with progressive scale expansion net- work. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9336- 9345, 2019.","title":"Shape robust text detection with progressive scale expansion network","authors":"Wang","year":"2019","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2019b] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog), 38(5):1-12, 2019.","title":"Dynamic graph cnn for learning on point clouds","authors":"Wang","year":"2019","venue":"ACM Transactions on Graphics (tog)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ye et al., 2021] Jiaquan Ye, Xianbiao Qi, Yelin He, Yihao Chen, Dengyi Gu, Peng Gao, and Rong Xiao. Pingan- vcgroup's solution for icdar 2021 competition on scientific literature parsing task b: table recognition to html. arXiv preprint arXiv:2105.01848, 2021.","title":"Pinganvcgroup's solution for icdar 2021 competition on scientific literature parsing task b: table recognition to html","authors":"Ye","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2022] Zhenrong Zhang, Jianshu Zhang, Jun Du, and Fengren Wang. Split, embed and merge: An accurate table structure recognizer. Pattern Recognition, 126:108565, 2022.","title":"Split, embed and merge: An accurate table structure recognizer","authors":"Zhenrong Zhang, Jianshu Zhang, Jun Du, Fengren Wang","year":"2022","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zheng et al., 2021] Xinyi Zheng, Douglas Burdick, Lucian Popa, Xu Zhong, and Nancy Xin Ru Wang. Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context. In Pro- ceedings of the IEEE/CVF winter conference on applica- tions of computer vision, pages 697-706, 2021.","title":"Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context","authors":"Xinyi Zheng, Douglas Burdick, Lucian Popa, Xu Zhong, Nancy Xin, Ru Wang","year":"2021","venue":"Proceedings of the IEEE/CVF winter conference on applications of computer vision","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhong et al., 2020] Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. Image-based table recognition: data, model, and evaluation. In European conference on computer vision, pages 564-580. Springer, 2020.","title":"Image-based table recognition: data, model, and evaluation","authors":"Xu Zhong, Elaheh Shafieibavani, Antonio Jimeno, Yepes","year":"2020","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}