{"type":"ScholarlyArticle","title":"Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks","authors":[{"name":"Haoyu Dong","slug":"haoyu-dong","given":"Haoyu","family":"Dong","affiliation":"Microsoft Research"},{"name":"Zhoujun Cheng","slug":"zhoujun-cheng","given":"Zhoujun","family":"Cheng","affiliation":"Shanghai Jiao Tong University"},{"name":"Xinyi He","slug":"xinyi-he","given":"Xinyi","family":"He","affiliation":"Xi'an Jiaotong University"},{"name":"Mengyu Zhou","slug":"mengyu-zhou","given":"Mengyu","family":"Zhou","affiliation":"Microsoft Research"},{"name":"Anda Zhou","slug":"anda-zhou","given":"Anda","family":"Zhou","affiliation":"University of Edinburgh"},{"name":"Fan Zhou","slug":"fan-zhou","given":"Fan","family":"Zhou","affiliation":"Shanghai Jiao Tong University"},{"name":"Ao Liu","slug":"ao-liu","given":"Ao","family":"Liu","affiliation":"Tokyo Institute of Technology"},{"name":"Shi Han","slug":"shi-han","given":"Shi","family":"Han","affiliation":"Microsoft Research"},{"name":"Dongmei Zhang","slug":"dongmei-zhang","given":"Dongmei","family":"Zhang","affiliation":"Microsoft Research"}],"abstract":"Following the success of pre-training techniques in the natural language domain, a flurry of table pre-training frameworks have been proposed and have achieved new state-of-the-arts on various downstream tasks such as table question answering, table type recognition, column relation classification, table search, and formula prediction. Various model architectures have been explored to best capture the characteristics of (semi-)structured tables, especially specially-designed attention mechanisms. Moreover, to fully leverage the supervision signals in unlabeled tables, diverse pre-training objectives have been designed and evaluated, for example, denoising cell values, predicting numerical relationships, and learning a neural SQL executor. This survey aims to provide a comprehensive review of model designs, pre-training objectives, and downstream tasks for table pre-training, and we further share our thoughts on existing challenges and future opportunities.","doi":"10.24963/ijcai.2022/761","url":"https://ijcai.org/proceedings/2022/761","pdf":"https://ijcai.org/proceedings/2022/0761.pdf","markdown":"https://ijcai.org/proceedings/2022/761.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/761","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Survey Track","track":"Survey Track","number":761,"pages":{"first":5426,"last":5435},"keywords":["Survey Track: Knowledge Representation and Reasoning","Survey Track: Data Mining","Survey Track: Natural Language Processing"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987303","length_sec":61,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985446","length_sec":720,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Adi et al., 2016] Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sen- tence embeddings using auxiliary prediction tasks. arXiv preprint arXiv:1608.04207, 2016.","title":"Fine-grained analysis of sentence embeddings using auxiliary prediction tasks","authors":"Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, Yoav Goldberg","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ainslie et al., 2020] Joshua Ainslie, Santiago Ontanon, Chris Al- berti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encod- ing long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 268-284, 2020.","title":"Etc: Encoding long and structured inputs in transformers","authors":"Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, Li Yang","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arik and Pfister, 2021] Sercan Ö Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6679-6687, 2021.","title":"Tabnet: Attentive interpretable tabular learning","authors":"Ö Sercan, Tomas Arik, Pfister","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Authors, 2021] Anonymous Authors. Flap: Table-to-text genera- tion with feature indication and numerical reasoning pretraining. ACL Rolling Review Nov., 2021.","title":"Anonymous Authors. Flap: Table-to-text generation with feature indication and numerical reasoning pretraining","authors":"Authors","year":"2021","venue":"ACL Rolling Review","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Barik et al., 2015] Titus Barik, Kevin Lubick, Justin Smith, John Slankas, and Emerson Murphy-Hill. Fuse: a reproducible, extendable, internet-scale corpus of spreadsheets. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repos- itories, pages 486-489. IEEE, 2015.","title":"Fuse: a reproducible, extendable, internet-scale corpus of spreadsheets","authors":"Titus Barik, Kevin Lubick, Justin Smith, John Slankas, Emerson Murphy-Hill","year":"2015","venue":"IEEE/ACM 12th Working Conference on Mining Software Repositories","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Belinkov et al., 2017] Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. What do neural ma- chine translation models learn about morphology? arXiv preprint arXiv:1704.03471, 2017.","title":"What do neural machine translation models learn about morphology","authors":"Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, James Glass","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cafarella et al., 2008] Michael J Cafarella, Alon Y Halevy, Yang Zhang, Daisy Zhe Wang, and Eugene Wu. Uncovering the rela- tional web. In WebDB, 2008.","title":"Uncovering the relational web","authors":"Alon Y Michael J Cafarella, Yang Halevy, Daisy Zhe Zhang, Eugene Wang, Wu","year":"2008","venue":"WebDB","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen and Cafarella, 2013] Zhe Chen and Michael Cafarella. Au- tomatic web spreadsheet data extraction. In Proceedings of the 3rd International Workshop on Semantic Search over the Web, 2013.","title":"Automatic web spreadsheet data extraction","authors":"Zhe Chen, Michael Cafarella","year":"2013","venue":"Proceedings of the 3rd International Workshop on Semantic Search over the Web","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2018] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. {TVM}: An auto- mated {End-to-End} optimizing compiler for deep learning. In 13th USENIX Symposium on Operating Systems Design and Im- plementation (OSDI 18), pages 578-594, 2018.","title":"{TVM}: An automated {End-to-End} optimizing compiler for deep learning","authors":"Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze","year":"2018","venue":"13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chen et al., 2019] Jiaoyan Chen, Ernesto Jiménez-Ruiz, Ian Hor- rocks, and Charles Sutton. Colnet: Embedding the semantics of web tables for column type prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 29-36, 2019.","title":"Colnet: Embedding the semantics of web tables for column type prediction","authors":"Chen","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chen et al., 2020] Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang Wang. Logic2text: High-fidelity natural language generation from logical forms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, 2020.","title":"Logic2text: High-fidelity natural language generation from logical forms","authors":"Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, William Yang, Wang","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chen et al., 2021] Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, and Denny Zhou. Spread- sheetcoder: Formula prediction from semi-structured context. In International Conference on Machine Learning. PMLR, 2021.","title":"Spreadsheetcoder: Formula prediction from semi-structured context","authors":"Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, Denny Zhou","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Cheng et al., 2021] Zhoujun Cheng, Haoyu Dong, Fan Cheng, Ran Jia, Pengfei Wu, Shi Han, and Dongmei Zhang. Fortap: Us- ing formulae for numerical-reasoning-aware table pretraining. preprint arXiv:2109.07323, 2021.","title":"Fortap: Using formulae for numerical-reasoning-aware table pretraining","authors":"Cheng","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Clark et al., 2019] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04341, 2019.","title":"What does bert look at? an analysis of bert's attention","authors":"Kevin Clark, Urvashi Khandelwal, Omer Levy, Christopher D Manning","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Deng et al., 2020] Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. Turl: table understanding through representation learning. Proceedings of the VLDB Endowment, 2020.","title":"Turl: table understanding through representation learning","authors":"Xiang Deng, Huan Sun, Alyssa Lees, You Wu, Cong Yu","year":"2020","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Deng et al., 2021] Xiang Deng, Ahmed Hassan, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. Structure-grounded pretraining for text-to-sql. In Proceedings of the 2021 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics, 2021.","title":"Structure-grounded pretraining for text-to-sql","authors":"Xiang Deng, Ahmed Hassan, Christopher Meek, Oleksandr Polozov, Huan Sun, Matthew Richardson","year":"2021","venue":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Dong et al., 2019] Haoyu Dong, Shijie Liu, Zhouyu Fu, Shi Han, and Dongmei Zhang. Semantic structure extraction for spread- sheet tables with a multi-task learning architecture. In Workshop on Document Intelligence at NeurIPS 2019, 2019.","title":"Semantic structure extraction for spreadsheet tables with a multi-task learning architecture","authors":"Haoyu Dong, Shijie Liu, Zhouyu Fu, Shi Han, Dongmei Zhang","year":"2019","venue":"Workshop on Document Intelligence at NeurIPS 2019","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Dong et al., 2020] Haoyu Dong Dong, Jinyu Wang, Zhouyu Fu, Shi Han, and Dongmei Zhang. Neural formatting for spreadsheet tables. In Proceedings of the 29th ACM International Confer- ence on Information & Knowledge Management, pages 305-314, 2020.","title":"Neural formatting for spreadsheet tables","authors":"Dong Haoyu, Jinyu Dong, Zhouyu Wang, Shi Fu, Dongmei Han, Zhang","year":"2020","venue":"Proceedings of the 29th ACM International Conference on Information & Knowledge Management","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Du et al., 2021] Lun Du, Fei Gao, Xu Chen, Ran Jia, Junshan Wang, Jiang Zhang, Shi Han, and Dongmei Zhang. Tabularnet: A neural network architecture for understanding semantic struc- tures of tabular data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 322- 331, 2021.","title":"Tabularnet: A neural network architecture for understanding semantic structures of tabular data","authors":"Lun Du, Fei Gao, Xu Chen, Ran Jia, Junshan Wang, Jiang Zhang, Shi Han, Dongmei Zhang","year":"2021","venue":"Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Eberius et al., 2015] Julian Eberius, Katrin Braunschweig, and Others. Building the dresden web table corpus: A classification approach. In 2015 IEEE/ACM 2nd International Symposium on Big Data Computing (BDC), pages 41-50. IEEE, 2015.","title":"Building the dresden web table corpus: A classification approach","authors":"Julian Eberius, Katrin Braunschweig, Others","year":"2015","venue":"IEEE/ACM 2nd International Symposium on Big Data Computing (BDC)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Eisenschlos et al., 2020] Julian Eisenschlos, Syrine Krichene, and Thomas Mueller. Understanding tables with intermediate pre- training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, 2020.","title":"Understanding tables with intermediate pretraining","authors":"Julian Eisenschlos, Syrine Krichene, Thomas Mueller","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Eisenschlos et al., 2021] Julian Eisenschlos, Maharshi Gor, Thomas Mueller, and William Cohen. Mate: Multi-view attention for table transformer efficiency. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021. Fetahu et al., 2019]","title":"Mate: Multi-view attention for table transformer efficiency","authors":"Julian Eisenschlos, Maharshi Gor, Thomas Mueller, William Cohen","year":"2019","venue":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"Besnik Fetahu, Avishek Anand, and Maria Koutraki. Tablenet: An approach for determining fine-grained relations for wikipedia tables. In The World Wide Web Confer- ence, pages 2736-2742, 2019.","title":"Tablenet: An approach for determining fine-grained relations for wikipedia tables","authors":"Besnik Fetahu, Avishek Anand, Maria Koutraki","year":"2019","venue":"The World Wide Web Conference","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Gol et al., 2019] Majid Ghasemi Gol, Jay Pujara, and Pedro Szekely. Tabular cell classification using pre-trained cell embed- dings. In 2019 IEEE International Conference on Data Mining (ICDM), pages 230-239. IEEE, 2019.","title":"Tabular cell classification using pre-trained cell embeddings","authors":"Jay Majid Ghasemi Gol, Pedro Pujara, Szekely","year":"2019","venue":"IEEE International Conference on Data Mining (ICDM)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Gong et al., 2020] Heng Gong, Yawei Sun, Xiaocheng Feng, Bing Qin, Wei Bi, Xiaojiang Liu, and Ting Liu. Tablegpt: Few-shot table-to-text generation with table structure reconstruction and content matching. In Proceedings of the 28th International Con- ference on Computational Linguistics, pages 1978-1988, 2020.","title":"Tablegpt: Few-shot table-to-text generation with table structure reconstruction and content matching","authors":"Heng Gong, Yawei Sun, Xiaocheng Feng, Bing Qin, Wei Bi, Xiaojiang Liu, Ting Liu","year":"2020","venue":"Proceedings of the 28th International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Guo et al., 2020] Tong Guo, Derong Shen, Tiezheng Nie, and Yue Kou. Web table column type detection using deep learning and probability graph model. In International Conference on Web Information Systems and Applications, pages 401-414. Springer, 2020.","title":"Web table column type detection using deep learning and probability graph model","authors":"Tong Guo, Derong Shen, Tiezheng Nie, Yue Kou","year":"2020","venue":"International Conference on Web Information Systems and Applications","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Herzig et al., 2020] Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Mueller, Francesco Piccinno, and Julian Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. In Pro- ceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pages 4320-4333, 2020.","title":"Tapas: Weakly supervised table parsing via pre-training","authors":"Jonathan Herzig, Krzysztof Pawel, Thomas Nowak, Francesco Mueller, Julian Piccinno, Eisenschlos","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Huang and He, 2018] Zhipeng Huang and Yeye He. Auto-detect: Data-driven error detection in tables. In Proceedings of the 2018 International Conference on Management of Data, 2018.","title":"Auto-detect: Data-driven error detection in tables","authors":"He Huang, Zhipeng Huang, Yeye He","year":"2018","venue":"Proceedings of the 2018 International Conference on Management of Data","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Huang et al., 2020] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint:2012.06678, 2020.","title":"Tabtransformer: Tabular data modeling using contextual embeddings","authors":"Xin Huang, Ashish Khetan, Milan Cvitkovic, Zohar Karnin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Hulsebos et al., 2021] Madelon Hulsebos, C ¸agatay Demiralp, and Paul Groth. Gittables: A large-scale corpus of relational tables. arXiv preprint arXiv:2106.07258, 2021.","title":"Gittables: A large-scale corpus of relational tables","authors":"Madelon Hulsebos, C ¸agatay Demiralp, Paul Groth","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Iida et al., 2021] Hiroshi Iida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. Tabbie: Pretrained representations of tabular data. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 3446-3456, 2021.","title":"Tabbie: Pretrained representations of tabular data","authors":"Hiroshi Iida, Dung Thai, Varun Manjunatha, Mohit Iyyer","year":"2021","venue":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Jiao et al., 2020] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tiny- BERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP, 2020.","title":"Tiny-BERT: Distilling BERT for natural language understanding","authors":"Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu","year":"2020","venue":"Findings of the Association for Computational Linguistics: EMNLP","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Kardas et al., 2020] Marcin Kardas, Piotr Czapla, Pontus Stene- torp, Sebastian Ruder, Sebastian Riedel, Ross Taylor, and Robert Stojnic. Axcell: Automatic extraction of results from machine learning papers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.","title":"Axcell: Automatic extraction of results from machine learning papers","authors":"Marcin Kardas, Piotr Czapla, Pontus Stenetorp, Sebastian Ruder, Sebastian Riedel, Ross Taylor, Robert Stojnic","year":"2020","venue":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Koci et al., 2018] Elvis Koci, Maik Thiele, Wolfgang Lehner, and Oscar Romero. Table recognition in spreadsheets via a graph representation. In 2018 13th IAPR International Workshop on Document Analysis Systems (DAS), pages 139-144. IEEE, 2018.","title":"Table recognition in spreadsheets via a graph representation","authors":"Koci","year":"2018","venue":"13th IAPR International Workshop on Document Analysis Systems (DAS)","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Krichene et al., 2021] Syrine Krichene, Thomas Müller, and Ju- lian Martin Eisenschlos. Dot: An efficient double transformer for nlp tasks with tables. arXiv preprint arXiv:2106.00479, 2021.","title":"Dot: An efficient double transformer for nlp tasks with tables","authors":"Krichene","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Lewis et al., 2020] Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence- to-sequence pre-training for natural language generation, trans- lation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.","title":"Bart: Denoising sequenceto-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Li et al., 2021] Junlong Li, Yiheng Xu, Lei Cui, and Furu Wei. Markuplm: Pre-training of text and markup language for visually-rich document understanding. arXiv preprint arXiv:2110.08518, 2021.","title":"Markuplm: Pre-training of text and markup language for visually-rich document understanding","authors":"Junlong Li, Yiheng Xu, Lei Cui, Furu Wei","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Lin et al., 2020] Xi Victoria Lin, Richard Socher, and Caiming Xiong. Bridging textual and tabular data for cross-domain text- to-sql semantic parsing. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 4870-4888, 2020.","title":"Bridging textual and tabular data for cross-domain textto-sql semantic parsing","authors":"Lin","year":"2020","venue":"Findings of the Association for Computational Linguistics: EMNLP 2020","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Linzen et al., 2016] Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535, 2016.","title":"Assessing the ability of lstms to learn syntax-sensitive dependencies","authors":"Linzen","year":"2016","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly opti- mized bert pretraining approach, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Liu et al., 2021] Qian Liu, Bei Chen, Jiaqi Guo, Zeqi Lin, and Jian-guang Lou. Tapex: Table pre-training via learning a neu- ral sql executor. arXiv preprint arXiv:2107.07653, 2021.","title":"Tapex: Table pre-training via learning a neural sql executor","authors":"Qian Liu, Bei Chen, Jiaqi Guo, Zeqi Lin, Jian-Guang Lou","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Nishida et al., 2017] Kyosuke Nishida, Kugatsu Sadamitsu, Ryuichiro Higashinaka, and Yoshihiro Matsuo. Understanding the semantic structures of tables with a hybrid deep neural network architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.","title":"Understanding the semantic structures of tables with a hybrid deep neural network architecture","authors":"Kyosuke Nishida, Kugatsu Sadamitsu, Ryuichiro Higashinaka, Yoshihiro Matsuo","year":"2017","venue":"Thirty-First AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Pasupat and Liang, 2015] Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Con- ference on Natural Language Processing, 2015.","title":"Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables","authors":"Liang Pasupat","year":"2015","venue":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learn- ing with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140), 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2020","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Rogers et al., 2020] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-866, 2020.","title":"A primer in bertology: What we know about how bert works","authors":"Anna Rogers, Olga Kovaleva, Anna Rumshisky","year":"2020","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Shi et al., 2021] Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Cicero Nogueira dos San- tos, and Bing Xiang. Learning contextual representations for se- mantic parsing with generation-augmented pre-training. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, 2021.","title":"Learning contextual representations for semantic parsing with generation-augmented pre-training","authors":"Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Cicero Nogueira Dos Santos, Bing Xiang","year":"2021","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Tang et al., 2021] Nan Tang, Ju Fan, Fangyi Li, Jianhong Tu, Xi- aoyong Du, Guoliang Li, Sam Madden, and Mourad Ouzzani. Rpt: relational pre-trained transformer is almost all you need to- wards democratizing data preparation. Proceedings of the VLDB Endowment, 14(8), 2021.","title":"Rpt: relational pre-trained transformer is almost all you need towards democratizing data preparation","authors":"Nan Tang, Ju Fan, Fangyi Li, Jianhong Tu, Xiaoyong Du, Guoliang Li, Sam Madden, Mourad Ouzzani","year":"2021","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Tay et al., 2020] Yi Tay, Mostafa Dehghani, Dara Bahri, and Don- ald Metzler. Efficient transformers: A survey. arXiv preprint arXiv:2009.06732, 2020.","title":"Efficient transformers: A survey","authors":"Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Thawani et al., 2021] Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. Representing numbers in nlp: a survey and a vision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, pages 644-656, 2021.","title":"Representing numbers in nlp: a survey and a vision","authors":"Avijit Thawani, Jay Pujara, Filip Ilievski, Pedro Szekely","year":"2021","venue":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Par- mar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Ad- vances in neural information processing systems, pages 5998- 6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Vig, 2019] Jesse Vig. A multiscale visualization of attention in the transformer model. arXiv preprint arXiv:1906.05714, 2019.","title":"A multiscale visualization of attention in the transformer model","authors":"Jesse Vig","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Wang et al., 2020] Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. Tuta: Tree-based trans- formers for generally structured table pre-training. arXiv preprint arXiv:2010.12537, 2020.","title":"Tuta: Tree-based transformers for generally structured table pre-training","authors":"Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, Dongmei Zhang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Wang et al., 2021a] Daheng Wang, Prashant Shiralkar, Colin Lockard, Binxuan Huang, Xin Luna Dong, and Meng Jiang. Tcn: Table convolutional network for web table interpretation. In Pro- ceedings of the Web Conference 2021, pages 4020-4032, 2021.","title":"Tcn: Table convolutional network for web table interpretation","authors":"Wang","year":"2021","venue":"Proceedings of the Web Conference 2021","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Wang et al., 2021b] Fei Wang, Kexuan Sun, Muhao Chen, Jay Pujara, and Pedro Szekely. Retrieving complex tables with multi-granular graph representation learning. arXiv preprint arXiv:2105.01736, 2021.","title":"Retrieving complex tables with multi-granular graph representation learning","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Wang et al., 2021c] Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. Tuta: Tree-based trans- formers for generally structured table pre-training. In Proceed- ings of the 27th ACM SIGKDD Conference on Knowledge Dis- covery & Data Mining, pages 1780-1790, 2021.","title":"Tuta: Tree-based transformers for generally structured table pre-training","authors":"Wang","year":"2021","venue":"Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Xie et al., 2022] Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv preprint arXiv:2201.05966, 2022.","title":"Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models","authors":"Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, I Sida, Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Xing and Wan, 2021] Xinyu Xing and Xiaojun Wan. Structure- aware pre-training for table-to-text generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2273-2278, 2021.","title":"Structureaware pre-training for table-to-text generation","authors":"Xinyu Xing, Xiaojun Wan","year":"2021","venue":"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021","doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Yang et al., 2022] Jingfeng Yang, Aditya Gupta, Shyam Upad- hyay, Luheng He, Rahul Goel, and Shachi Paul. Tableformer: Robust transformer modeling for table-text encoding. arXiv preprint arXiv:2203.00274, 2022.","title":"Tableformer: Robust transformer modeling for table-text encoding","authors":"Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, Shachi Paul","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Yin et al., 2020] Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.","title":"Tabert: Pretraining for joint understanding of textual and tabular data","authors":"Pengcheng Yin, Graham Neubig, Wen-Tau Yih, Sebastian Riedel","year":"2020","venue":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[Yoon et al., 2020] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 2020.","title":"Vime: Extending the success of self-and semi-supervised learning to tabular domain","authors":"Yoon","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Yoran et al., 2021] Ori Yoran, Alon Talmor, and Jonathan Berant. Turning tables: Generating examples from semi-structured ta- bles for endowing language models with reasoning skills. arXiv preprint arXiv:2107.07261, 2021.","title":"Turning tables: Generating examples from semi-structured tables for endowing language models with reasoning skills","authors":"Alon Ori Yoran, Jonathan Talmor, Berant","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":65,"raw":"[Yu et al., 2018] Tao Yu, Rui Zhang, Kai Yang, Michihiro Ya- sunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text- to-sql task. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 3911-3921, 2018.","title":"Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and textto-sql task","authors":"Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman","year":"2018","venue":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":66,"raw":"[Yu et al., 2020] Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Caiming Xiong, et al. Grappa: Grammar-augmented pre-training for table seman- tic parsing. In International Conference on Learning Represen- tations, 2020.","title":"Grappa: Grammar-augmented pre-training for table semantic parsing","authors":"Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Caiming Xiong","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":67,"raw":"[Zayats et al., 2021] Vicky Zayats, Kristina Toutanova, and Mari Ostendorf. Representations for question answering from docu- ments with tables and text. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2895-2906, 2021.","title":"Representations for question answering from documents with tables and text","authors":"Vicky Zayats, Kristina Toutanova, Mari Ostendorf","year":"2021","venue":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main","doi":null,"url":null,"internal_url":null},{"position":68,"raw":"[Zhang and Balog, 2020] Shuo Zhang and Krisztian Balog. Web table extraction, retrieval, and augmentation: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 2020.","title":"Web table extraction, retrieval, and augmentation: A survey","authors":"Shuo Zhang, Krisztian Balog","year":"2020","venue":"ACM Transactions on Intelligent Systems and Technology (TIST)","doi":null,"url":null,"internal_url":null},{"position":69,"raw":"[Zhang et al., 2019] Li Zhang, Shuo Zhang, and Krisztian Balog. Table2vec: Neural word and entity embeddings for table pop- ulation and retrieval. In Proceedings of International ACM SI- GIR Conference on Research and Development in Information Retrieval, 2019.","title":"Table2vec: Neural word and entity embeddings for table population and retrieval","authors":"Li Zhang, Shuo Zhang, Krisztian Balog","year":"2019","venue":"Proceedings of International ACM SI-GIR Conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":70,"raw":"[Zhang et al., 2020] Xingyao Zhang, Linjun Shou, Jian Pei, Ming Gong, Lijie Wen, and Daxin Jiang. A graph representation of semi-structured data for web question answering. In Proceedings of the 28th International Conference on Computational Linguis- tics, pages 51-61, 2020.","title":"A graph representation of semi-structured data for web question answering","authors":"Zhang","year":"2020","venue":"Proceedings of the 28th International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":71,"raw":"[Zhang et al., 2021] Yakun Zhang, Xiao Lv, Haoyu Dong, Wen- sheng Dou, Shi Han, Dongmei Zhang, Jun Wei, and Dan Ye. Semantic table structure identification in spreadsheets. In Pro- ceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 283-295, 2021.","title":"Semantic table structure identification in spreadsheets","authors":"Yakun Zhang, Xiao Lv, Haoyu Dong, Wensheng Dou, Shi Han, Dongmei Zhang, Jun Wei, Dan Ye","year":"2021","venue":"Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis","doi":null,"url":null,"internal_url":null},{"position":72,"raw":"[Zhou et al., 2020] Mengyu Zhou, Wang Tao, Ji Pengxin, Han Shi, and Zhang Dongmei. Table2analysis: Modeling and recommen- dation of common analysis patterns for multi-dimensional data. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):320-328, Apr. 2020.","title":"Table2analysis: Modeling and recommendation of common analysis patterns for multi-dimensional data","authors":"Mengyu Zhou, Wang Tao, Ji Pengxin, Han Shi, Zhang Dongmei","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}