{"type":"ScholarlyArticle","title":"Empirical Analysis of Dialogue Relation Extraction with Large Language Models","authors":[{"name":"Guozheng Li","slug":"guozheng-li","given":"Guozheng","family":"Li","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Zijie Xu","slug":"zijie-xu","given":"Zijie","family":"Xu","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Ziyu Shang","slug":"ziyu-shang","given":"Ziyu","family":"Shang","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Jiajun Liu","slug":"jiajun-liu","given":"Jiajun","family":"Liu","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Ke Ji","slug":"ke-ji","given":"Ke","family":"Ji","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Yikai Guo","slug":"yikai-guo","given":"Yikai","family":"Guo","affiliation":"Beijing Institute of Computer Technology and Application"}],"abstract":"Dialogue relation extraction (DRE) aims to extract relations between two arguments within a dialogue, which is more challenging than standard RE due to the higher person pronoun frequency and lower information density in dialogues. However, existing DRE methods still suffer from two serious issues: (1) hard to capture long and sparse multi-turn information, and (2) struggle to extract golden relations based on partial dialogues, which motivates us to discover more effective methods that can alleviate the above issues. We notice that the rise of large language models (LLMs) has sparked considerable interest in evaluating their performance across diverse tasks. To this end, we initially investigate the capabilities of different LLMs in DRE, considering both proprietary models and open-source models. Interestingly, we discover that LLMs significantly alleviate two issues in existing DRE methods. Generally, we have following findings: (1) scaling up model size substantially boosts the overall DRE performance and achieves exceptional results, tackling the difficulty of capturing long and sparse multi-turn information; (2) LLMs encounter with much smaller performance drop from entire dialogue setting to partial dialogue setting compared to existing methods; (3) LLMs deliver competitive or superior performances under both full-shot and few-shot settings compared to current state-of-the-art; (4) LLMs show modest performances on inverse relations but much stronger improvements on general relations, and they can handle dialogues of various lengths especially for longer sequences.","doi":"10.24963/ijcai.2024/703","url":"https://ijcai.org/proceedings/2024/703","pdf":"https://ijcai.org/proceedings/2024/0703.pdf","markdown":"https://ijcai.org/proceedings/2024/703.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/703","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":703,"pages":{"first":6359,"last":6367},"keywords":["Natural Language Processing: NLP: Information extraction"],"videos":[],"references":[{"position":1,"raw":"[Arora et al., 2023] S Arora, A Narayan, M Chen, L Orr, N Guha, K Bhatia, I Chami, F Sala, and C Ré. Ask me anything: A simple strategy for prompting language mod- els. In ICLR, 2023.","title":"Ask me anything: A simple strategy for prompting language models","authors":"Arora, M Narayan, L Chen, N Orr, K Guha, I Bhatia, F Chami, C Sala, Ré","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cabot and Navigli, 2021] Pere-Lluís Huguet Cabot and Roberto Navigli. Rebel: Relation extraction by end-to-end language generation. In Findings of EMNLP, 2021.","title":"Rebel: Relation extraction by end-to-end language generation","authors":"Pere-Lluís Huguet, Cabot, Roberto Navigli","year":"2021","venue":"Findings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Giorgi et al., 2022] John Giorgi, Gary D Bader, and Bo Wang. A sequence-to-sequence approach for document-level relation extraction. In BioNLP, 2022.","title":"A sequence-to-sequence approach for document-level relation extraction","authors":"John Giorgi, Gary D Bader, Bo Wang","year":"2022","venue":"BioNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goyal et al., 2017] Priya Goyal, Piotr Dollár, Ross Gir- shick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Ky- rola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Ac- curate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.","title":"Accurate, large minibatch sgd: Training imagenet in 1 hour","authors":"Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2022] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.","title":"Lora: Low-rank adaptation of large language models","authors":"J Edward, Yelong Hu, Phillip Shen, Zeyuan Wallis, Yuanzhi Allen-Zhu, Shean Li, Lu Wang, Weizhu Wang, Chen","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kojima et al., 2022] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, 2022.","title":"Large language models are zero-shot reasoners","authors":"Takeshi Kojima, Shane Shixiang, Machel Gu, Yutaka Reid, Yusuke Matsuo, Iwasawa","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lee and Choi, 2021] Bongseok Lee and Yong Suk Choi. Graph based network with contextualized representations of turns in dialogue. In EMNLP, 2021.","title":"Graph based network with contextualized representations of turns in dialogue","authors":"Lee, Choi ; ] Bongseok Lee, Yong Suk Choi","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lewis et al., 2020] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL, 2020.","title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2022] Guozheng Li, Xu Chen, Peng Wang, Jiafeng Xie, and Qiqing Luo. Fastre: Towards fast relation ex- traction with convolutional encoder and improved cascade binary tagging framework. In IJCAI, 2022.","title":"Fastre: Towards fast relation extraction with convolutional encoder and improved cascade binary tagging framework","authors":"Guozheng Li, Xu Chen, Peng Wang, Jiafeng Xie, Qiqing Luo","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/583"},{"position":12,"raw":"[Li et al., 2023] Guozheng Li, Peng Wang, and Wenjun Ke. Revisiting large language models as zero-shot relation ex- tractors. In Findings of EMNLP, 2023.","title":"Revisiting large language models as zero-shot relation extractors","authors":"Li","year":"2023","venue":"Findings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2024] Guozheng Li, Wenjun Ke, Peng Wang, Zi- jie Xu, Ke Ji, Jiajun Liu, Ziyu Shang, and Qiqing Luo. Unlocking instructive in-context learning with tabular prompting for relational triple extraction. arXiv preprint arXiv:2402.13741, 2024.","title":"Unlocking instructive in-context learning with tabular prompting for relational triple extraction","authors":"Guozheng Li, Wenjun Ke, Peng Wang, Zijie Xu, Ke Ji, Jiajun Liu, Ziyu Shang, Qiqing Luo","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2022] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In ACL, 2022.","title":"P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks","authors":"Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, Jie Tang","year":"2022","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2023] Xiao Liu, Jian Zhang, Heng Zhang, Fuzhao Xue, and Yang You. Hierarchical dialogue under- standing with special tokens and turn-level attention. In ICLR, 2023.","title":"Hierarchical dialogue understanding with special tokens and turn-level attention","authors":"Xiao Liu, Jian Zhang, Heng Zhang, Fuzhao Xue, Yang You","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2024] Jiajun Liu, Wenjun Ke, Peng Wang, Ziyu Shang, Jinhua Gao, Guozheng Li, Ke Ji, and Yanhe Liu. Towards continual knowledge graph embedding via incre- mental distillation. In AAAI, 2024.","title":"Towards continual knowledge graph embedding via incremental distillation","authors":"Jiajun Liu, Wenjun Ke, Peng Wang, Ziyu Shang, Jinhua Gao, Guozheng Li, Ke Ji, Yanhe Liu","year":"2024","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Loshchilov and Hutter, 2019] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ma et al., 2023] Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. Large language model is not a good few- shot information extractor, but a good reranker for hard samples! In Findings of EMNLP, 2023.","title":"Large language model is not a good fewshot information extractor, but a good reranker for hard samples! In Findings of EMNLP","authors":"Yubo Ma, Yixin Cao, Yongching Hong, Aixin Sun","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[OpenAI, 2022] OpenAI. Introducing chatgpt, 2022.","title":"OpenAI. Introducing chatgpt","authors":"Openai","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Paolini et al., 2021] Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anub- hai, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. Structured prediction as translation between aug- mented natural languages. In ICLR, 2021.","title":"Structured prediction as translation between augmented natural languages","authors":"Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira Dos Santos, Bing Xiang, Stefano Soatto","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Poria et al., 2019] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In ACL, 2019.","title":"MELD: A multimodal multi-party dataset for emotion recognition in conversations","authors":"Poria","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(1):5485-5551, 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2020","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sanh et al., 2022] Victor Sanh, Albert Webson, Colin Raf- fel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In ICLR, 2022.","title":"Multitask prompted training enables zero-shot task generalization","authors":"Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Scao et al., 2022] Teven Le Scao, Angela Fan, Christo- pher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, Franc ¸ois Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.","title":"Teven Le Scao, Angela Fan","authors":"Scao","year":"2022","venue":"Bloom: A 176b-parameter open-access multilingual language model","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shang et al., 2024] Ziyu Shang, Wenjun Ke, Nana Xiu, Peng Wang, Jiajun Liu, Yanhui Li, Zhizhao Luo, and Ke Ji. Ontofact: Unveiling fantastic fact-skeleton of llms via ontology-driven reinforcement learning. In AAAI, 2024.","title":"Ontofact: Unveiling fantastic fact-skeleton of llms via ontology-driven reinforcement learning","authors":"Ziyu Shang, Wenjun Ke, Nana Xiu, Peng Wang, Jiajun Liu, Yanhui Li, Zhizhao Luo, Ke Ji","year":"2024","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Son et al., 2022] Junyoung Son, Jinsung Kim, Jungwoo Lim, and Heuiseok Lim. GRASP: Guiding model with RelAtional semantics using prompt for dialogue relation extraction. In COLING, 2022.","title":"GRASP: Guiding model with RelAtional semantics using prompt for dialogue relation extraction","authors":"Son","year":"2022","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. Llama: Open and efficient founda- tion language models. arXiv preprint arXiv:2302.13971, 2023.","title":"Hugo Touvron, Thibaut Lavril, Gautier Izacard","authors":"Touvron","year":"2023","venue":"Llama: Open and efficient foundation language models","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wadhwa et al., 2023] Somin Wadhwa, Silvio Amir, and By- ron C Wallace. Revisiting relation extraction in the era of large language models. In ACL, 2023.","title":"Revisiting relation extraction in the era of large language models","authors":"Somin Wadhwa, Silvio Amir, Byron C Wallace","year":"2023","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wan et al., 2023] Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kuro- hashi. Gpt-re: In-context learning for relation extraction using large language models. In EMNLP, 2023.","title":"Gpt-re: In-context learning for relation extraction using large language models","authors":"Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, Sadao Kurohashi","year":"2023","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2023a] Peng Wang, Tong Shao, Ke Ji, Guozheng Li, and Wenjun Ke. fmlre: a low-resource rela- tion extraction model based on feature mapping similarity calculation. In AAAI, 2023.","title":"fmlre: a low-resource relation extraction model based on feature mapping similarity calculation","authors":"Wang","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2023b] Peng Wang, Jiafeng Xie, Xiye Chen, Guozheng Li, and Wei Li. Pascore: a chinese overlapping relation extraction model based on global pointer annota- tion strategy. In IJCAI, 2023.","title":"Pascore: a chinese overlapping relation extraction model based on global pointer annotation strategy","authors":"Wang","year":"2023","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2023/579"},{"position":34,"raw":"[Xue et al., 2021] Fuzhao Xue, Aixin Sun, Hao Zhang, and Eng Siong Chng. Gdpnet: Refining latent multi-view graph for relation extraction. In AAAI, 2021.","title":"Gdpnet: Refining latent multi-view graph for relation extraction","authors":"Xue","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yao et al., 2019] Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. DocRED: A large-scale document-level relation extraction dataset. In ACL, 2019.","title":"DocRED: A large-scale document-level relation extraction dataset","authors":"Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, Maosong Sun","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yu et al., 2020] Dian Yu, Kai Sun, Claire Cardie, and Dong Yu. Dialogue-based relation extraction. In ACL, 2020.","title":"Dialogue-based relation extraction","authors":"Dian Yu, Kai Sun, Claire Cardie, Dong Yu","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zahiri and Choi, 2018] Sayyed M Zahiri and Jinho D Choi. Emotion detection on tv show transcripts with sequence- based convolutional neural networks. In AAAI, 2018.","title":"Emotion detection on tv show transcripts with sequencebased convolutional neural networks","authors":"Choi ; ] Zahiri, M Sayyed, Jinho D Zahiri, Choi","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2017] Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position-aware attention and supervised data improve slot filling. In EMNLP, 2017.","title":"Position-aware attention and supervised data improve slot filling","authors":"Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, Christopher D Manning","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhang et al., 2023] Kai Zhang, Bernal Jiménez Gutiérrez, and Yu Su. Aligning instruction tasks unlocks large lan- guage models as zero-shot relation extractors. In Findings of ACL, 2023.","title":"Aligning instruction tasks unlocks large language models as zero-shot relation extractors","authors":"Zhang","year":"2023","venue":"Findings of ACL","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhao et al., 2021] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021.","title":"Calibrate before use: Improving few-shot performance of language models","authors":"Zhao","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}