{"type":"ScholarlyArticle","title":"Towards Incremental NER Data Augmentation via Syntactic-aware Insertion Transformer","authors":[{"name":"Wenjun Ke","slug":"wenjun-ke","given":"Wenjun","family":"Ke","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Zongkai Tian","slug":"zongkai-tian","given":"Zongkai","family":"Tian","affiliation":"Beijing Institute of Computer Technology and Application"},{"name":"Qi Liu","slug":"qi-liu","given":"Qi","family":"Liu","affiliation":"Beijing Institute of Computer Technology and Application"},{"name":"Peng Wang","slug":"peng-wang","given":"Peng","family":"Wang","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Jinhua Gao","slug":"jinhua-gao","given":"Jinhua","family":"Gao","affiliation":"Institute of Computing Technology , Chinese Academy of Sciences"},{"name":"Rui Qi","slug":"rui-qi","given":"Rui","family":"Qi","affiliation":"China Life Property Casualty Insurance Company Limited"}],"abstract":"Named entity recognition (NER) aims to locate and classify named entities in natural language texts. Most existing high-performance NER models employ a supervised paradigm, which requires a large quantity of high-quality annotated data during training. In order to help NER models perform well in few-shot scenarios, data augmentation approaches attempt to build extra data by means of random editing or by using end-to-end generation with PLMs. \r\nHowever, these methods focus on only the fluency of generated sentences, ignoring the syntactic correlation between the new and raw sentences. Such uncorrelation also brings low diversity and inconsistent labeling of synthetic samples. To fill this gap, we present SAINT (Syntactic-Aware InsertioN Transformer), a hard-constraint controlled text generation model that incorporates syntactic information. The proposed method operates by inserting new tokens between existing entities in a parallel manner. During insertion procedure, new tokens will be added taking both semantic and syntactic factors into account. Hence the resulting sentence can retain the syntactic correctness with respect to the raw data. Experimental results on two benchmark datasets, i.e., Ontonotes and Wikiann, demonstrate the comparable performance of SAINT over the state-of-the-art baselines.","doi":"10.24963/ijcai.2023/567","url":"https://ijcai.org/proceedings/2023/567","pdf":"https://ijcai.org/proceedings/2023/0567.pdf","markdown":"https://ijcai.org/proceedings/2023/567.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/567","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Natural Language Processing","track":"Main Track","number":567,"pages":{"first":5104,"last":5112},"keywords":["Natural Language Processing: NLP: Named entities","Natural Language Processing: NLP: Language generation"],"videos":[],"references":[{"position":1,"raw":"[Banerjee et al., 2019] Partha Sarathy Banerjee, Baisakhi Chakraborty, Deepak Tripathi, Hardik Gupta, and Sourabh S Kumar. A information retrieval based on ques- tion and answering and ner for unstructured information without using sql. Wireless Personal Communications, 2019.","title":"A information retrieval based on question and answering and ner for unstructured information without using sql","authors":"Partha Sarathy Banerjee, Baisakhi Chakraborty, Deepak Tripathi, Hardik Gupta, Sourabh S Kumar","year":"2019","venue":"Wireless Personal Communications","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2020] Hengyi Cai, Hongshen Chen, Yonghao Song, Cheng Zhang, Xiaofang Zhao, and Dawei Yin. Data manipulation: Towards effective instance learning for neural dialogue generation via learning to augment and reweight. In Proc. of ACL, 2020.","title":"Data manipulation: Towards effective instance learning for neural dialogue generation via learning to augment and reweight","authors":"Hengyi Cai, Hongshen Chen, Yonghao Song, Cheng Zhang, Xiaofang Zhao, Dawei Yin","year":"2020","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Campos et al., 2020] Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Mário Jorge, Celia Nunes, and Adam Jatowt. Yake! keyword extraction from single doc- uments using multiple local features. Inf. Sci., 2020.","title":"Yake! keyword extraction from single documents using multiple local features","authors":"Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Mário Jorge, Celia Nunes, Adam Jatowt","year":"2020","venue":"Inf. Sci","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Carlsson et al., 2022] Fredrik Carlsson, Joey Öhman, Fangyu Liu, Severine Verlinden, Joakim Nivre, and Mag- nus Sahlgren. Fine-grained controllable text generation using non-residual prompting. In Proc. of ACL, 2022.","title":"Fine-grained controllable text generation using non-residual prompting","authors":"Fredrik Carlsson, Joey Öhman, Fangyu Liu, Severine Verlinden, Joakim Nivre, Magnus Sahlgren","year":"2022","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021] Shuguang Chen, Gustavo Aguilar, Leonardo Neves, and Thamar Solorio. Data augmentation for cross-domain named entity recognition. In Proc. of EMNLP, 2021.","title":"Data augmentation for cross-domain named entity recognition","authors":"Shuguang Chen, Gustavo Aguilar, Leonardo Neves, Thamar Solorio","year":"2021","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2022] Xiang Chen, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, Huajun Chen, and Ningyu Zhang. LightNER: A lightweight tun- ing paradigm for low-resource NER via pluggable prompt- ing. In Proc. of COLING, 2022.","title":"LightNER: A lightweight tuning paradigm for low-resource NER via pluggable prompting","authors":"Xiang Chen, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, Huajun Chen, Ningyu Zhang","year":"2022","venue":"Proc. of COLING","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Conneau et al., 2020] Alexis Conneau, Kartikay Khandel- wal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross- lingual representation learning at scale. In Proc. of ACL, 2020.","title":"Unsupervised crosslingual representation learning at scale","authors":"Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov","year":"2020","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dai and Adel, 2020] Xiang Dai and Heike Adel. An analy- sis of simple data augmentation for named entity recogni- tion. In Proc. of COLING, 2020.","title":"An analysis of simple data augmentation for named entity recognition","authors":"Xiang Dai, Heike Adel","year":"2020","venue":"Proc. of COLING","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[de Lichy et al., 2021] Cyprien de Lichy, Hadrien Glaude, and William Campbell. Meta-learning for few-shot named entity recognition. In Proceedings of the 1st Workshop on Meta Learning and Its Applications to Natural Language Processing, 2021.","title":"Meta-learning for few-shot named entity recognition","authors":"De Lichy","year":"2021","venue":"Proceedings of the 1st Workshop on Meta Learning and Its Applications to Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ding et al., 2020] Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kruengkrai, Thien Hai Nguyen, Shafiq Joty, Luo Si, and Chunyan Miao. DAGA: Data augmentation with a generation approach for low-resource tagging tasks. In Proc. of EMNLP, 2020.","title":"DAGA: Data augmentation with a generation approach for low-resource tagging tasks","authors":"Ding","year":"2020","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gries, 1982] David Gries. A note on a standard strategy for developing loop invariants and loops. Science of Computer Programming, 1982.","title":"A note on a standard strategy for developing loop invariants and loops","authors":"David Gries","year":"1982","venue":"Science of Computer Programming","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hovy et al., 2006] Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. OntoNotes: The 90% solution. In Proc. of AACL, 2006.","title":"OntoNotes: The 90% solution","authors":"Hovy","year":"2006","venue":"Proc. of AACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Iso, 2022] Hayate Iso. Autotemplate: A simple recipe for lexically constrained text generation. arXiv preprint arXiv:2211.08387, 2022.","title":"Hayate Iso. Autotemplate: A simple recipe for lexically constrained text generation","authors":"Iso","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[KERAGHEL et al., 2020] Abdenacer KERAGHEL, Khalid BENABDESLEM, and Bruno CANITIA. Data augmen- tation process to improve deep learning-based ner task in the automotive industry field. In Proc. of IJCNN, 2020.","title":"Data augmentation process to improve deep learning-based ner task in the automotive industry field","authors":"Keraghel","year":"2020","venue":"Proc. of IJCNN","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kumar and Byrne, 2004] Shankar Kumar and William Byrne. Minimum Bayes-risk decoding for statistical machine translation. In Proc. of NAACL, 2004.","title":"Minimum Bayes-risk decoding for statistical machine translation","authors":"Shankar Kumar, William Byrne","year":"2004","venue":"Proc. of NAACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lample et al., 2016] Guillaume Lample, Miguel Balles- teros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recog- nition. In Proc. of NAACL, 2016.","title":"Neural architectures for named entity recognition","authors":"Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, Chris Dyer","year":"2016","venue":"Proc. of NAACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2019] Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. In Proc. of CVPR, 2019.","title":"End-to-end multi-task learning with attention","authors":"Shikun Liu, Edward Johns, Andrew J Davison","year":"2019","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2021] Linlin Liu, Bosheng Ding, Lidong Bing, Shafiq Joty, Luo Si, and Chunyan Miao. MulDA: A mul- tilingual data augmentation framework for low-resource cross-lingual NER. In Proc. of ACL, 2021.","title":"MulDA: A multilingual data augmentation framework for low-resource cross-lingual NER","authors":"Linlin Liu, Bosheng Ding, Lidong Bing, Shafiq Joty, Luo Si, Chunyan Miao","year":"2021","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2022a] Andy T. Liu, Wei Xiao, Henghui Zhu, De- jiao Zhang, Shang-Wen Li, and Andrew Arnold. Qaner: Prompting question answering models for few-shot named entity recognition. arXiv preprint arXiv:2203.01543, 2022.","title":"Qaner: Prompting question answering models for few-shot named entity recognition","authors":"Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2022b] Jian Liu, Yufeng Chen, and Jinan Xu. Low-resource ner by data augmentation with prompting. In Proc. of IJCAI, 2022.","title":"Low-resource ner by data augmentation with prompting","authors":"Liu","year":"2022","venue":"Proc. of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/590"},{"position":21,"raw":"[Ma et al., 2022a] Chang Ma, Song Zhang, Gehui Shen, and Zhihong Deng. Switch-gpt: An effective method for con- strained text generation under few-shot settings (student abstract). In Proc. of AAAI, 2022.","title":"Switch-gpt: An effective method for constrained text generation under few-shot settings","authors":"Ma","year":"2022","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ma et al., 2022b] Ruotian Ma, Xin Zhou, Tao Gui, Yid- ing Tan, Linyang Li, Qi Zhang, and Xuanjing Huang. Template-free prompt tuning for few-shot NER. In Proc. of NAACL, 2022.","title":"Template-free prompt tuning for few-shot NER","authors":"Ma","year":"2022","venue":"Proc. of NAACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ma et al., 2022c] Tingting Ma, Huiqiang Jiang, Qianhui Wu, Tiejun Zhao, and Chin-Yew Lin. Decomposed meta- learning for few-shot named entity recognition. In Proc. of ACL Findings, 2022.","title":"Decomposed metalearning for few-shot named entity recognition","authors":"Ma","year":"2022","venue":"Proc. of ACL Findings","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Min et al., 2020] Junghyun Min, R. Thomas McCoy, Di- panjan Das, Emily Pitler, and Tal Linzen. Syntactic data augmentation increases robustness to inference heuristics. In Proc. of ACL, 2020.","title":"Syntactic data augmentation increases robustness to inference heuristics","authors":"Min","year":"2020","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nallapati et al., 2016] Ramesh Nallapati, Bowen Zhou, Ci- cero dos Santos, C ¸aglar Gulc ¸ehre, and Bing Xiang. Ab- stractive text summarization using sequence-to-sequence RNNs and beyond. In Proc. of CoNLL, 2016.","title":"Abstractive text summarization using sequence-to-sequence RNNs and beyond","authors":"Ramesh Nallapati, Bowen Zhou, C Cicero Dos Santos, Bing ¸aglar Gulc ¸ehre, Xiang","year":"2016","venue":"Proc. of CoNLL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pan et al., 2017] Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross- lingual name tagging and linking for 282 languages. In Proc. of ACL, 2017.","title":"Crosslingual name tagging and linking for 282 languages","authors":"Pan","year":"2017","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Peters et al., 2017] Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. Semi- supervised sequence tagging with bidirectional language models. In Proc. of ACL, 2017.","title":"Semisupervised sequence tagging with bidirectional language models","authors":"Matthew E Peters, Waleed Ammar, Chandra Bhagavatula, Russell Power","year":"2017","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Phan and Nguyen, 2022] Uyen Phan and Nhung Nguyen. Simple semantic-based data augmentation for named en- tity recognition in biomedical texts. In Proceedings of the 21st Workshop on Biomedical Language Processing, 2022.","title":"Simple semantic-based data augmentation for named entity recognition in biomedical texts","authors":"Nguyen Phan, Uyen Phan, Nhung Nguyen","year":"2022","venue":"Proceedings of the 21st Workshop on Biomedical Language Processing","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Radford et al., 2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sabty et al., 2021] Caroline Sabty, Islam Omar, Fady Was- falla, Mohamed Islam, and Slim Abdennadher. Data augmentation techniques on arabic data for named entity recognition. Procedia Computer Science, 2021.","title":"Data augmentation techniques on arabic data for named entity recognition","authors":"Caroline Sabty, Islam Omar, Fady Wasfalla, Mohamed Islam, Slim Abdennadher","year":"2021","venue":"Procedia Computer Science","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Souza et al., 2020] Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. Portuguese named entity recognition us- ing bert-crf. arXiv preprint arXiv:1909.10649, 2020.","title":"Portuguese named entity recognition using bert-crf","authors":"Fábio Souza, Rodrigo Nogueira, Roberto Lotufo","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Sun and He, 2020] Xiao Sun and Jiajin He. A novel ap- proach to generate a large scale of supervised data for short text sentiment analysis. Multimedia Tools and Applica- tions, 2020.","title":"A novel approach to generate a large scale of supervised data for short text sentiment analysis","authors":"Xiao Sun, Jiajin He","year":"2020","venue":"Multimedia Tools and Applications","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2022] Yufei Wang, Can Xu, Qingfeng Sun, Huang Hu, Chongyang Tao, Xiubo Geng, and Daxin Jiang. PromDA: Prompt-based data augmentation for low- resource NLU tasks. In Proc. of ACL, 2022.","title":"PromDA: Prompt-based data augmentation for lowresource NLU tasks","authors":"Yufei Wang, Can Xu, Qingfeng Sun, Huang Hu, Chongyang Tao, Xiubo Geng, Daxin Jiang","year":"2022","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wei and Zou, 2019] Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proc. of EMNLP, 2019.","title":"EDA: Easy data augmentation techniques for boosting performance on text classification tasks","authors":"Zou ; ] Wei, Jason Wei, Kai Zou","year":"2019","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wu et al., 2020] Qianhui Wu, Zijia Lin, Guoxin Wang, Hui Chen, Börje F. Karlsson, Biqing Huang, and Chin-Yew Lin. Enhanced meta-learning for cross-lingual named en- tity recognition with minimal resources. In Proc. of AAAI, 2020.","title":"Enhanced meta-learning for cross-lingual named entity recognition with minimal resources","authors":"Qianhui Wu, Zijia Lin, Guoxin Wang, Hui Chen, Börje F Karlsson, Biqing Huang, Chin-Yew Lin","year":"2020","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yoo et al., 2019] Kang Min Yoo, Youhyun Shin, and Sang- goo Lee. Data augmentation for spoken language under- standing via joint variational generation. In Proc. of AAAI, 2019.","title":"Data augmentation for spoken language understanding via joint variational generation","authors":"Min Kang, Youhyun Yoo, Sanggoo Shin, Lee","year":"2019","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yu et al., 2018] Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. Qanet: Combining local convolution with global self-attention for reading comprehension. In Proc. of ICLR, 2018.","title":"Qanet: Combining local convolution with global self-attention for reading comprehension","authors":"Adams Wei, Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, V Quoc, Le","year":"2018","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang and Song, 2022] Hanqing Zhang and Dawei Song. DisCup: Discriminator cooperative unlikelihood prompt- tuning for controllable text generation. In Proc. of EMNLP, 2022.","title":"DisCup: Discriminator cooperative unlikelihood prompttuning for controllable text generation","authors":"Hanqing Zhang, Dawei Song","year":"2022","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhang et al., 2015] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Proc. of NeurIPS, 2015.","title":"Character-level convolutional networks for text classification","authors":"Xiang Zhang, Junbo, Jake Zhao, Yann Lecun","year":"2015","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhang et al., 2020] Yizhe Zhang, Guoyin Wang, Chunyuan Li, Zhe Gan, Chris Brockett, and Bill Dolan. POINTER: Constrained progressive text generation via insertion- based generative pre-training. In Proc. of EMNLP, 2020.","title":"POINTER: Constrained progressive text generation via insertionbased generative pre-training","authors":"Zhang","year":"2020","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhou et al., 2022] Jing Zhou, Yanan Zheng, Jie Tang, Li Jian, and Zhilin Yang. FlipDA: Effective and robust data augmentation for few-shot learning. In Proc. of ACL, 2022.","title":"FlipDA: Effective and robust data augmentation for few-shot learning","authors":"Jing Zhou, Yanan Zheng, Jie Tang, Li Jian, Zhilin Yang","year":"2022","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}