{"type":"ScholarlyArticle","title":"Recall, Retrieve and Reason: Towards Better In-Context Relation Extraction","authors":[{"name":"Guozheng Li","slug":"guozheng-li","given":"Guozheng","family":"Li","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Peng Wang","slug":"peng-wang","given":"Peng","family":"Wang","affiliation":"School of Computer Science and Engineering , Southeast University; Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University) , Ministry of Education"},{"name":"Wenjun Ke","slug":"wenjun-ke","given":"Wenjun","family":"Ke","affiliation":"School of Computer Science and Engineering , Southeast University; Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University) , Ministry of Education"},{"name":"Yikai Guo","slug":"yikai-guo","given":"Yikai","family":"Guo","affiliation":"Beijing Institute of Computer Technology and Application"},{"name":"Ke Ji","slug":"ke-ji","given":"Ke","family":"Ji","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Ziyu Shang","slug":"ziyu-shang","given":"Ziyu","family":"Shang","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Jiajun Liu","slug":"jiajun-liu","given":"Jiajun","family":"Liu","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Zijie Xu","slug":"zijie-xu","given":"Zijie","family":"Xu","affiliation":"School of Computer Science and Engineering , Southeast University"}],"abstract":"Relation extraction (RE) aims to identify relations between entities mentioned in texts. Although large language models (LLMs) have demonstrated impressive in-context learning (ICL) abilities in various tasks, they still suffer from poor performances compared to most supervised fine-tuned RE methods. Utilizing ICL for RE with LLMs encounters two challenges: (1) retrieving good demonstrations from training examples, and (2) enabling LLMs exhibit strong ICL abilities in RE. On the one hand, retrieving good demonstrations is a non-trivial process in RE, which easily results in low relevance regarding entities and relations. On the other hand, ICL with an LLM achieves poor performance in RE while RE is different from language modeling in nature or the LLM is not large enough. In this work, we propose a novel recall-retrieve-reason RE framework that synergizes LLMs with retrieval corpora (training examples) to enable relevant retrieving and reliable in-context reasoning. Specifically, we distill the consistently ontological knowledge from training datasets to let LLMs generate relevant entity pairs grounded by retrieval corpora as valid queries. These entity pairs are then used to retrieve relevant training examples from the retrieval corpora as demonstrations for LLMs to conduct better ICL via instruction tuning. Extensive experiments on different LLMs and RE datasets demonstrate that our method generates relevant and valid entity pairs and boosts ICL abilities of LLMs, achieving competitive or new state-of-the-art performance on sentence-level RE compared to previous supervised fine-tuning methods and ICL-based methods.","doi":"10.24963/ijcai.2024/704","url":"https://ijcai.org/proceedings/2024/704","pdf":"https://ijcai.org/proceedings/2024/0704.pdf","markdown":"https://ijcai.org/proceedings/2024/704.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/704","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":704,"pages":{"first":6368,"last":6376},"keywords":["Natural Language Processing: NLP: Information extraction"],"videos":[],"references":[{"position":1,"raw":"[Agrawal et al., 2022] Monica Agrawal, Stefan Hegsel- mann, Hunter Lang, et al. Large language models are few- shot clinical information extractors. In EMNLP, 2022.","title":"Large language models are fewshot clinical information extractors","authors":"Monica Agrawal, Stefan Hegselmann, Hunter Lang","year":"2022","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arora et al., 2023] Simran Arora, Avanika Narayan, Mayee F Chen, et al. Ask me anything: A simple strategy for prompting language models. In ICLR, 2023.","title":"Ask me anything: A simple strategy for prompting language models","authors":"Simran Arora, Avanika Narayan, Mayee F Chen","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baldini Soares et al., 2019] Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, et al. Matching the blanks: Dis- tributional similarity for relation learning. In ACL, 2019.","title":"Matching the blanks: Distributional similarity for relation learning","authors":"Baldini Livio, Nicholas Soares, Jeffrey Fitzgerald, Ling","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, et al. Language models are few-shot learners. In NeurIPS, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cabot and Navigli, 2021] Pere-Lluís Huguet Cabot and Roberto Navigli. Rebel: Relation extraction by end-to-end language generation. In Findings of EMNLP, 2021.","title":"Rebel: Relation extraction by end-to-end language generation","authors":"Pere-Lluís Huguet, Cabot, Roberto Navigli","year":"2021","venue":"Findings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2022a] Xiang Chen, Ningyu Zhang, Xin Xie, et al. Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In WWW, 2022.","title":"Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction","authors":"Chen","year":"2022","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2022b] Yanda Chen, Ruiqi Zhong, Sheng Zha, et al. Meta-learning via language model in-context tuning. In ACL, 2022.","title":"Meta-learning via language model in-context tuning","authors":"Chen","year":"2022","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2023] Zeming Chen, Gail Weiss, Eric Mitchell, et al. Reckoning: Reasoning through dynamic knowledge encoding. In NeurIPS, 2023.","title":"Reckoning: Reasoning through dynamic knowledge encoding","authors":"Chen","year":"2023","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Coda-Forno et al., 2023] Julian Coda-Forno, Marcel Binz, Zeynep Akata, et al. Meta-in-context learning in large lan- guage models. In NeurIPS, 2023.","title":"Meta-in-context learning in large language models","authors":"Julian Coda-Forno, Marcel Binz, Zeynep Akata","year":"2023","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Du et al., 2022] Zhengxiao Du, Yujie Qian, Xiao Liu, et al. GLM: General language model pretraining with autore- gressive blank infilling. In ACL, 2022.","title":"GLM: General language model pretraining with autoregressive blank infilling","authors":"Du","year":"2022","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gao et al., 2021] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In EMNLP, 2021.","title":"Simcse: Simple contrastive learning of sentence embeddings","authors":"Tianyu Gao, Xingcheng Yao, Danqi Chen","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hendrickx et al., 2019] Iris Hendrickx, Su Nam Kim, Zor- nitsa Kozareva, et al. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nomi- nals. arXiv preprint arXiv:1911.10422, 2019.","title":"Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals","authors":"Hendrickx","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Holtzman et al., 2021] Ari Holtzman, Peter West, Vered Shwartz, et al. Surface form competition: Why the highest probability answer isn't always right. In EMNLP, 2021.","title":"Surface form competition: Why the highest probability answer isn't always right","authors":"Holtzman","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hu et al., 2022] Edward J Hu, Yelong Shen, Phillip Wallis, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.","title":"Lora: Low-rank adaptation of large language models","authors":"J Edward, Yelong Hu, Phillip Shen, Wallis","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ji et al., 2023] Ke Ji, Yixin Lian, Jingsheng Gao, and Baoyuan Wang. Hierarchical verbalizer for few-shot hi- erarchical text classification. In ACL, 2023.","title":"Hierarchical verbalizer for few-shot hierarchical text classification","authors":"Ke Ji, Yixin Lian, Jingsheng Gao, Baoyuan Wang","year":"2023","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jimenez Gutierrez et al., 2022] Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, et al. Thinking about GPT-3 in-context learning for biomedical IE? think again. In Findings of EMNLP, 2022.","title":"Thinking about GPT-3 in-context learning for biomedical IE? think again","authors":"Jimenez Gutierrez","year":"2022","venue":"Findings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lewis et al., 2020] Mike Lewis, Yinhan Liu, Naman Goyal, et al. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and compre- hension. In ACL, 2020.","title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu, Naman Goyal","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2022] Guozheng Li, Xu Chen, Peng Wang, et al. Fastre: Towards fast relation extraction with convolutional encoder and improved cascade binary tagging framework. In IJCAI, 2022.","title":"Fastre: Towards fast relation extraction with convolutional encoder and improved cascade binary tagging framework","authors":"Guozheng Li, Xu Chen, Peng Wang","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/583"},{"position":20,"raw":"[Li et al., 2023a] Bo Li, Wei Ye, Jinglei Zhang, et al. Re- viewing labels: Label graph network with top-k prediction set for relation extraction. In AAAI, 2023.","title":"Reviewing labels: Label graph network with top-k prediction set for relation extraction","authors":"Li","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2023b] Bo Li, Dingyao Yu, Wei Ye, et al. Se- quence generation with label augmentation for relation ex- traction. In AAAI, 2023.","title":"Sequence generation with label augmentation for relation extraction","authors":"Li","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2023c] Guozheng Li, Peng Wang, and Wenjun Ke. Revisiting large language models as zero-shot relation ex- tractors. In Findings of EMNLP, 2023.","title":"Revisiting large language models as zero-shot relation extractors","authors":"Li","year":"2023","venue":"Findings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2024] Guozheng Li, Wenjun Ke, Peng Wang, Zi- jie Xu, Ke Ji, Jiajun Liu, Ziyu Shang, and Qiqing Luo. Unlocking instructive in-context learning with tabular prompting for relational triple extraction. arXiv preprint arXiv:2402.13741, 2024.","title":"Unlocking instructive in-context learning with tabular prompting for relational triple extraction","authors":"Guozheng Li, Wenjun Ke, Peng Wang, Zijie Xu, Ke Ji, Jiajun Liu, Ziyu Shang, Qiqing Luo","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, et al. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Liu et al., 2022] Jiachang Liu, Dinghan Shen, Yizhe Zhang, et al. What makes good in-context examples for GPT-3? In DeeLIO, 2022.","title":"What makes good in-context examples for GPT-3?","authors":"Liu","year":"2022","venue":"DeeLIO","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Liu et al., 2024] Jiajun Liu, Wenjun Ke, Peng Wang, Ziyu Shang, Jinhua Gao, Guozheng Li, Ke Ji, and Yanhe Liu. Towards continual knowledge graph embedding via incre- mental distillation. In AAAI, 2024.","title":"Towards continual knowledge graph embedding via incremental distillation","authors":"Jiajun Liu, Wenjun Ke, Peng Wang, Ziyu Shang, Jinhua Gao, Guozheng Li, Ke Ji, Yanhe Liu","year":"2024","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Luan et al., 2018] Yi Luan, Luheng He, Mari Ostendorf, et al. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. arXiv preprint arXiv:1808.09602, 2018.","title":"Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction","authors":"Yi Luan, Luheng He, Mari Ostendorf","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Luo et al., 2023] Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, et al. Reasoning on graphs: Faithful and inter- pretable large language model reasoning. arXiv preprint arXiv:2310.01061, 2023.","title":"Reasoning on graphs: Faithful and interpretable large language model reasoning","authors":"Luo","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ma et al., 2023] Yubo Ma, Yixin Cao, YongChing Hong, et al. Large language model is not a good few-shot in- formation extractor, but a good reranker for hard samples! In Findings of EMNLP, 2023.","title":"Large language model is not a good few-shot information extractor, but a good reranker for hard samples! In Findings of EMNLP","authors":"Yubo Ma, Yixin Cao, Yongching Hong","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Min et al., 2022a] Sewon Min, Mike Lewis, Hannaneh Ha- jishirzi, et al. Noisy channel language model prompting for few-shot text classification. In ACL, 2022.","title":"Noisy channel language model prompting for few-shot text classification","authors":"Min","year":"2022","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Min et al., 2022b] Sewon Min, Mike Lewis, Luke Zettle- moyer, et al. Metaicl: Learning to learn in context. In NAACL-HLT, 2022.","title":"Metaicl: Learning to learn in context","authors":"Min","year":"2022","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[OpenAI, 2022] OpenAI. Introducing chatgpt, 2022.","title":"OpenAI. Introducing chatgpt","authors":"Openai","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Paolini et al., 2021] Giovanni Paolini, Ben Athiwaratkun, Jason Krone, et al. Structured prediction as translation be- tween augmented natural languages. In ICLR, 2021.","title":"Structured prediction as translation between augmented natural languages","authors":"Giovanni Paolini, Ben Athiwaratkun, Jason Krone","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(1):5485- 5551, 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts","year":"2020","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP, 2019.","title":"Sentence-bert: Sentence embeddings using siamese bert-networks","authors":"Gurevych Reimers, Nils Reimers, Iryna Gurevych","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Roche, 2003] Christophe Roche. Ontology: a survey. IFAC Proceedings Volumes, 36(22):187-192, 2003.","title":"Christophe Roche. Ontology: a survey","authors":"Roche","year":"2003","venue":"IFAC Proceedings Volumes","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Sainz et al., 2021] Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, et al. Label verbalization and entailment for effective zero-and few-shot relation extraction. In EMNLP, 2021.","title":"Label verbalization and entailment for effective zero-and few-shot relation extraction","authors":"Sainz","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Shang et al., 2024] Ziyu Shang, Wenjun Ke, Nana Xiu, Peng Wang, Jiajun Liu, Yanhui Li, Zhizhao Luo, and Ke Ji. Ontofact: Unveiling fantastic fact-skeleton of llms via ontology-driven reinforcement learning. In AAAI, 2024.","title":"Ontofact: Unveiling fantastic fact-skeleton of llms via ontology-driven reinforcement learning","authors":"Ziyu Shang, Wenjun Ke, Nana Xiu, Peng Wang, Jiajun Liu, Yanhui Li, Zhizhao Luo, Ke Ji","year":"2024","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Shi et al., 2023] Freda Shi, Xinyun Chen, Kanishka Misra, et al. Large language models can be easily distracted by irrelevant context. In ICML, 2023.","title":"Large language models can be easily distracted by irrelevant context","authors":"Freda Shi, Xinyun Chen, Kanishka Misra","year":"2023","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.","title":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. Llama: Open and efficient foundation language models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. Attention is all you need. In NIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wan et al., 2023] Zhen Wan, Fei Cheng, Zhuoyuan Mao, et al. GPT-RE: In-context learning for relation extraction using large language models. In EMNLP, 2023.","title":"GPT-RE: In-context learning for relation extraction using large language models","authors":"Zhen Wan, Fei Cheng, Zhuoyuan Mao","year":"2023","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wang et al., 2022] Chenguang Wang, Xiao Liu, Zui Chen, et al. Deepstruct: Pretraining of language models for struc- ture prediction. In Findings of ACL, 2022.","title":"Deepstruct: Pretraining of language models for structure prediction","authors":"Chenguang Wang, Xiao Liu, Zui Chen","year":"2022","venue":"Findings of ACL","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Wang et al., 2023a] Peng Wang, Tong Shao, Ke Ji, Guozheng Li, and Wenjun Ke. fmlre: a low-resource rela- tion extraction model based on feature mapping similarity calculation. In AAAI, 2023.","title":"fmlre: a low-resource relation extraction model based on feature mapping similarity calculation","authors":"Wang","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Wang et al., 2023b] Peng Wang, Jiafeng Xie, Xiye Chen, et al. Pascore: a chinese overlapping relation extraction model based on global pointer annotation strategy. In IJ- CAI, 2023.","title":"Pascore: a chinese overlapping relation extraction model based on global pointer annotation strategy","authors":"Wang","year":"2023","venue":"IJ-CAI","doi":null,"url":null,"internal_url":"/proceedings/2023/579"},{"position":47,"raw":"[Wei et al., 2022a] Jason Wei, Maarten Bosma, Vincent Y Zhao, et al. Finetuned language models are zero-shot learners. In ICLR, 2022.","title":"Finetuned language models are zero-shot learners","authors":"Wei","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Wei et al., 2022b] Jason Wei, Xuezhi Wang, Dale Schuur- mans, et al. Chain of thought prompting elicits reasoning in large language models. In NeurIPS, 2022.","title":"Chain of thought prompting elicits reasoning in large language models","authors":"Wei","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Yamada et al., 2020] Ikuya Yamada, Akari Asai, Hiroyuki Shindo, et al. Luke: deep contextualized entity represen- tations with entity-aware self-attention. In EMNLP, 2020.","title":"Luke: deep contextualized entity representations with entity-aware self-attention","authors":"Yamada","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zhang et al., 2017] Yuhao Zhang, Victor Zhong, Danqi Chen, et al. Position-aware attention and supervised data improve slot filling. In EMNLP, 2017.","title":"Position-aware attention and supervised data improve slot filling","authors":"Yuhao Zhang, Victor Zhong, Danqi Chen","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Zhao et al., 2021] Zihao Zhao, Eric Wallace, Shi Feng, et al. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021.","title":"Calibrate before use: Improving few-shot performance of language models","authors":"Zhao","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Zhong and Chen, 2021] Zexuan Zhong and Danqi Chen. A frustratingly easy approach for entity and relation extrac- tion. In NAACL-HLT, 2021.","title":"A frustratingly easy approach for entity and relation extraction","authors":"Chen ; Zhong, Zexuan Zhong, Danqi Chen","year":"2021","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Zhou and Chen, 2022] Wenxuan Zhou and Muhao Chen. An improved baseline for sentence-level relation extrac- tion. In AACL, 2022.","title":"An improved baseline for sentence-level relation extraction","authors":"Chen ; ] Wenxuan Zhou, Muhao Zhou, Chen","year":"2022","venue":"AACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}