Making LLMs as Fine-Grained Relation Extraction Data Augmentor

1 Introduction

Relation Extraction (RE) is pivotal in Information Extraction (IE), seeking to identify relations between entities within textual data. Its significance resonates in downstream applications like event extraction [Xiang and Wang, 2019], knowledge graph [Luan et al., 2018], and intelligent question answering [Sun et al., 2021]. Despite the commendable success of current methodologies, which predominantly follow a supervised paradigm, a notable reliance exists on extensive datasets with high-quality annotations. In practical scenarios, the primary hurdles confronting RE revolve around lowresource challenges. These include the relatively modest size of available datasets, restricted application field scopes, and the complexities associated with labeling special domains.

* Corresponding authors.

Numerous approaches have been proposed to address the challenges mentioned above, including meta-learning [Hu et al., 2021; Liu et al., 2022; Pouran Ben Veyseh et al., 2023], transfer learning [Sarhan and Spruit, 2020; Gururaja et al., 2023], data augmentation [Hu et al., 2023; Zhao et al., 2023; Xu et al., 2023] and instruction prompting [Li et al., 2023]. Among these, Data Augmentation (DA) stands out as a plugand-play technology, offering direct applicability as a preprocessing method for a broad spectrum of tasks. While DA techniques have found success in tasks like Text Classification (TC) [Hsu et al., 2021] and Named Entity Recognition (NER) [Ke et al., 2023], their exploration in RE remains somewhat limited. This disparity arises due to the inherent fine-grained nature of RE compared to TC and NER. Modeling the intricate dependencies within RE proves challenging. As illustrated in Figure 1(a), the presence of the same entity pair in a sentence may result in entirely different relation types due to variations in context.

Dominant methods frame fine-grained DA into controlled text generation paradigm [Ke et al., 2023; Hu et al., 2023]. Fine-grained DA is broadly categorized into two paradigms: editing and generative methods. Editing methods involve simple transform operations like random exchange, insertion, and deletion. However, the imposition of rule restrictions limits the diversity of samples, consequently diminishing the generalization capacity of the RE models. For instance, as illustrated in the first instance in Figure 1(b), merely substituting the said for say and adding the lung fails to introduce substantial contextual diversity. Generative methods offer the advantage of producing more fluid and diverse samples. However, current generative approaches exhibit two notable shortcomings. Firstly, compared to the original sentence, the generated counterpart may deviate semantically, failing to preserve the relation dependency between the original entity pairs. As exemplified in Figure 1(b), owing to variations in contextual semantics, the relation type between entities Forsberg and Oct.19 transitions from date of death to date of birth. Secondly, existing methods lack specific hard constraints to ensure the inclusion of entity pairs during sentence generation. This oversight may introduce new entity pairs with unknown labels, leading to the generation of uncontrollable data. Consequently, when employing controlled text generation for RE DA, it becomes imperative to address the challenge of enhancing context diversity beyond entity pairs while preserving relation dependencies.

We argue that the crux of RE DA lies in preserving relation dependencies between pairs of entities through semantic consistency within the context. At the same time, to enhance the generalization ability of RE models, it is also necessary to ensure the diversity of contextual expressions during the generation process. Compared with existing pre-trained language models (PLMs) such as T5 [Raffel et al., 2020] and BART [Lewis et al., 2020], large language models (LLMs) such as GPT-3 [Brown et al., 2020], LLaMA [Touvron et al., 2023] and GPT-4 [OpenAI, 2024] show strong potential in generating diverse and contextually relevant texts, bringing new possibilities to RE DA. This paper proposes ConsistRE, an innovative RE DA method that maintains context consistency in RE. This method adds context constraints of keyword hints in the sentence synthesis process to ensure that the generated sentences maintain relation dependencies and semantic consistency while increasing the diversity of synthesized sentences with LLMs. Specifically, first, we apply statistical algorithms and semantic similarity to find the keyword hints most closely related to the relation instances based on a large amount of textual data. Following this, triples and keyword hints are included as controlled text as part of the prompt. During the sentence generation process, we filtered similar instances from both original and synthetic samples as demonstrations to enhance the performance of the LLMs. Finally, we select sentences that align more consistently with grammatical rules through syntactic dependency parsing to ensure that the generated sentences are more grammatically sound.

We assess the performance of our RE DA method on two RE models, ReDMP and SuRE, using three datasets:

SemEval, TACRED, and TCAREV. The experimental results underscore the remarkable effectiveness of our approach in enhancing the diversity of generated sentences while preserving relation dependencies. When applied to ReDMP, ConsistRE exhibits superior performance, achieving F1 values of 1.48%, 5.48%, and 3.16% higher than other optimal methods on SemEval, TACRED, and TACREV, respectively. Similarly, under SuRE, ConsistRE outperforms other methods, yielding F1 values higher by 2.03%, 2.35%, and 1.9%. To sum up, the contributions of this paper are three-fold:

• We argue that the cornerstone of RE DA lies in maintaining the relation dependency of synthetic sentences through semantic consistency with context.
• We introduce ConsistRE, a framework that aims to simultaneously maintain the consistency of dependencies and diversity of synthetic sentences with LLMs.
• We conduct extensive experiments on three public datasets, demonstrating the importance of maintaining relation dependencies through contextual constraints.

2 Methods

Assuming that a relation instance (s, h, r, t) is given from the original annotated dataset X, where s, h, r, t represent the source sentence, head entity, relation type, and tail entity, respectively. ConsistRE aims to derive a substantially larger augmented dataset Y that maintains high consistencywithX. For each instance (˜ s, h, r, t) ∈ Y , ˜ s is newly generated from s, while maintaining the original (h, r, t) unchanged.

The workflow of ConsistRE is illustrated in Figure 2. In the first stage, ConsistRE gathers a substantial amount of sentences related to triplet (h, r, t) from the Internet and acquires the keyword hints k most intricately associated with (s, h, r, t) utilizing statistical algorithms and semantic similarity. Moving on to the second stage, ConsistRE employs langchain 1 to select the most semantically similar instance as demonstrations d from the constructed example selector. Subsequently, d, (s, h, r, t), and k are integrated into a prompt template to generate prompts, and an LLM is employed to generate a set of sentence instances. Finally, in the third stage, syntactic dependency parsing is employed to select instances ˜ s with superior syntax, forming the augmented dataset Y .

2.1 Keyword Hints Retrieval

The initial stage of our approach involves acquiring the most pertinent keyword hints k. Here, k represents the context most closely related to the relation instance (s, h, r, t) and will later be used as a hard constraint during the sentence generation, aiming to maintain the dependency consistency of the relation in the generated sentences.

Related Sentences Retrieval

Given the intricate nature of RE that demands fine-grained modeling, the identification of relations between specified entities necessitates comprehensive and contextually rich support. Relying solely on contextual information derived from the original sentence s might prove insufficient in capturing

1 https://www.langchain.com/

Stage 1: Keyword Hints Retrieval

Stage 2: Sentence Generation

Stage 3: Syntactic Selection

the intricacies of the relations. Therefore, it becomes imperative to seek additional sentences with more extensive content to address this limitation. To mitigate this challenge, we augment our dataset by collecting substantial textual data from the Internet. Specifically, we utilize the search interface provided by Google 2 to gather a substantial set of sentences relevant to the triplet (h, r, t). The acquired sentences undergo preprocessing to extract pure text, forming the sentence set C for subsequent utilization in obtaining k.

Keyword Hints Obtain

We formulate a relevance score to discern keyword hints k that most effectively encapsulate entity relations within the retrieved sentence set C. Specifically, the relevance score q assigned to each occurring word w is defined as follows:

qpmi represents the score computed by the Pointwise Mutual Information (PMI) [Church and Hanks, 1989], which is a widely used linguistic statistical method to gauge word correlation. P(w), P(h), and P(t) respectively represent the probability of the calculated word w, head entity h, and tail entity t appearing in the sentences set C. P (w, h, t) represents the probability of all three appearing simultaneously.

qtf-idf represents the score calculated by the TF-IDF. The integration of TF-IDF aims to eliminate frequently occurring but semantically insignificant words. TF (w, c) represents the frequency of w in sentence c ∈ C, while IDF (w, c, C) represents the rarity of w in the sentence set C.

qsem represents the semantic similarity between s and w, which is employed to ensure that the k aligns closely with the semantics of the original sentence. EMB(s) and EMB(w) are encoded by Sentence-BERT [Reimers and Gurevych,

2 https://developers.google.com/custom-search

2019]. qpmi , qtf-idf and qsem are adjusted to range 0 to 1. By computing the relevance score q for each word, we select the w with the highest score as keyword hints k.

2.2 Sentence Generation

LLMs exhibit robust contextual learning capabilities and can be significantly augmented through few-shot in-context demonstrations. In the second stage, we aim to generate a set of high-quality sentences ˆ s. We break down prompt acquisition into the following two steps: demonstration selection and prompt formulation.

Demonstration Selection

To better stimulate and leverage the In-Context Learning (ICL) capabilities of LLMs, choosing similar relation instances from the example database to form the demonstration d in the few-shot prompt is essential. We employ the example selector in langchain to execute these steps, utilizing Sentence-BERT as the encoding model and FAISS 3 as the embedding database. Example database is initialized with original dataset X, and subsequent augmented data (˜ s, h, r, t) is added during the execution process. Iteratively increasing the number of examples in the example database can expand the optional range of demonstrations. We select three examples from the example database that are semantically closest to (s, h, r, t) as demonstrations. The format of the demonstration is as follows:

Knowledge: The relation between [head entity] and [tail entity] is [relation type]

Objective: Make sentences with given entities [head entity], [tail entity] and keyword [keyword hint]

Output: [source sentence]

Deserving a special mention, [keyword hint] in the demonstration is extracted from the source sentence using TopicRank [Bougouin et al., 2013].

Prompt Formulation

To enhance the context-learning accuracy of LLMs, we incorporate semi-formatted structural constraints into our prompt.

3 https://github.com/facebookresearch/faiss

Specifically, we input the relation instance (h, r, t) and keyword hints k into the task-prompt p. We combine the demonstration d selected in the preceding step and p sequentially in two steps to construct the prompt provided to the LLM to obtain the desired sentences ˆ s for each instance. ICL can be conceptualized as LLMs implicitly conducting Bayesian inference [Xie et al., 2022]:

Given the prompt p and multiple demonstrations, LLMs learn via marginalization by "selecting" the demonstration.

Additionally, we do not include the original sentence s in the prompt to maintain the diversity of synthetic sentences. Task-prompt p is defined as follows:

Knowledge: The relation between [h] and [t] is [r]

Objective: Make sentences with given entities [h], [t] and keyword [k]

Output:

2.3 Syntactic Selection

For the sentence ˆ s generated in the preceding stage, we posit that when the syntactic structure of the generated sentence closely aligns with the sentence s, the generated result is more consistent with the original one. In pursuit of this, we introduce a similarity calculation method based on syntactic dependency structure to aid in selecting instances with superior syntax for the final augmented sentences.

In particular, for the original sentence s and each sentence ˆ s within the corresponding candidate set, we utilize Stanford Parse 4 to conduct syntactic analysis, resulting in the generation of the respective syntactic dependency trees, denoted as T1 and T2. The structure of syntactic dependency trees can encapsulate the inter-word dependency relations and convey syntactic structural information.

Following this, we employ the Tree Edit Distance (TDS) to gauge the similarity between two syntactic dependency trees. TDS is a method employed for measuring the similarity between two tree structures, quantifying the disparity between one tree and another by calculating the minimum number of edit operations necessary to transform one tree into the other. These edit operations encompass inserting, deleting, and replacing nodes. The formula for calculating TDS can be expressed as follows:

Among them, T1 ′ and T2 ′ represent the subtrees of T1 and T2, respectively, after the removal of the root node. n1 and n2 denote the root nodes of T1 and T2. In this case, the cost function δ(·) for the three operations is uniformly defined as 1. The outlined issues can be efficiently addressed using dynamic programming [Zhang and Shasha, 1989]. Through the computation of TDS, we choose several sentences with the most favorable syntactic structure as the final augmented ˜ s, ensuring that the generated sentences exhibit sound syntactic structure and grammatical legitimacy.

4 https://stanfordnlp.github.io/CoreNLP

3 Experiments

In this section, we describe the datasets used, outline the experimental settings, present the baselines, and provide the results of the experiments.

3.1 Datasets and Experimental Settings

We conduct our experiments on three public RE datasets: SemEval 2010 Task 8 (SemEval) [Hendrickx et al., 2009], the TAC Relation Extraction Dataset (TACRED) [Zhang et al., 2017], and the revisited TAC Relation Extraction Dataset (TACREV) [Alt et al., 2020]. The statistics of datasets are presented in Table 1. SemEval is a traditional dataset widely employed in RE. It undergoes manual precision labeling and is devoid of noise. The SemEval dataset encompasses 19 relation types: Cause-Effect, Component-Whole, and others. TACRED is a more extensive dataset designed for RE. Its content primarily originates from news and online texts within the TAC KBP newswire and web forum corpus. Annotated through crowdsourcing, TACRED comprises 42 relation types. TACREV is a dataset derived from the original TACRED dataset. It addresses and rectifies some errors found in the annotated data within TACRED.

In our experimental setup, we sample 5, 10, 20, 50, and 100 instances for each relation type to simulate low-resource scenarios. Both ConsistRE and other baseline models augment the sampled data 3x to ensure a fair experiment comparison. The augmented data, along with the initial sampled data, is then fed into the RE model for training. The remainder of the data remains unseen by all DA methods and RE models. In this study, Micro-F1 is chosen as a critical metric to assess and compare all DA methods. We adopt gpt-3.5-turbo 5 as the backbone model of ConsistRE, and each result is averaged over three runs for reporting.

3.2 Baselines

We choose the following two types of DA methods as baselines for comparison:

Editing methods: WordNet Synonym Substitution (WSS) [Mueller and Thyagarajan, 2016] introduces lexical variations by replacing selected tokens with synonyms from WordNet [Fellbaum, 1998]. EDA [Wei and Zou, 2019] proposes a set of token-level word operations. Word Embedding Substitution (WES) [Jiao et al., 2020] enhances data diversity by substituting tokens with contextual word embeddings from BERT [Devlin et al., 2019].

5 https://openai.com/product

Generative methods: REMix [Teru, 2023] applies lexically constrained decoding to back-translation. LAMBADA [Anaby-Tavor et al., 2020] fine-tune GPT-2 and generate candidate examples. GDA [Hu et al., 2023] employs two modules for model training: one ensures semantic coherence through reordering, while the other maintains grammatical structure with a unified pattern.

To ensure a fair comparison of each DA method's performance, we employ the following two RE models as evaluation benchmarks: ReDMP [Tian et al., 2022] enhances performance by incorporating syntactic information through a syntax-induced encoder trained on auto-parsed data with dependency masking. SuRE [Lu et al., 2022] transforms relation extraction into a summarization task, improving precision and efficiency through indirect supervision, sentence and relation conversion techniques, and constraint decoding for robust inference.

3.3 Main Results

The experimental results on the three datasets are presented in Table 2. Base uses only the sampled original data from the training dataset without additional operations.

In general, most baselines outperform the nonaugmentation method (Base), highlighting the effectiveness of DA methods. With fewer sampled data (Shot-5, 10, and 20), DA methods consistently exhibit more significant performance improvements. However, under the experimental settings of Shot-50 and 100, the performance improvement is limited, and there is even a decline in performance.

Intuitively, generative methods are expected to outperform editing methods. However, in the context of our experimental setup, generative methods (LAMBADA on SemEval/TACREV, GDA on TACRED) exhibit noticeably poorer performance, even falling below Base. This can be attributed to generative methods needing to be adequately trained when the sample size is minimal (<15% on SemEval and <5% on TACRED/TACREV). In contrast, editing methods, being more straightforward and not reliant on extensive training data, achieve more promising results.

Across the three datasets and two evaluation models, our method consistently outperforms all other baseline methods on average without negative improvement in all sampling settings. Specifically, when tested with ReDMP, ConsistRE demonstrates F1 values that are 1.48%, 5.48%, and 3.16% higher than those of other optimal methods on SemEval, TACRED, and TACREV, respectively. Testing with ReDMP, F1 values of ConsistRE are higher by 2.03%, 2.35%, and 1.9% in three datasets, respectively. These results unequivocally showcase the superior adaptability of our method in generating a more significant number of new samples. This underscores the importance of emphasizing consistency and diversity of expression in the context.

3.4 Ablation Study

Our approach aims to generate augmented samples with consistent relation dependencies and diverse expressions by utilizing keyword hints. To assess the effectiveness of the components, we conduct ablation experiments on SemEval focusing on three aspects. Table 3 presents the results, where w/o keywords signifies that no keyword hints are added as restricted context during the sentence generation, w/o langchain refers to using a fixed example for demonstration, and w/o syntactic indicates the absence of syntactic selection.

The results reveal the positive significance of all three components for performance. Specifically, removing keyword hints leads to a significant performance decline on both ReDMP and SuRE, reaching 5.07% and 6.14%, respectively. Similarly, the removal of langchain and syntactic selection also caused a notable decline, with drops of 3.92% and 2.24% on ReDMP and 4.53% and 3.99% on SuRE. Notably, keyword hints have a pronounced impact on performance loss. This is because, without keyword hints, LLMs are prone to synthesizing sentences that deviate from semantics or fail to convey relation dependencies explicitly.

3.5 Analysis Experiments

In this section, we perform experiments to assess the influence of keyword hints and the size of the generated data on the performance. Additionally, we evaluate the diversity of the generated samples.

Keyword Hints Selection Strategy

The ablation experiment effectively demonstrates the impact of adding keyword hints closely related to the relation instance during sentence generation. Separate experiments are conducted on the SemEval dataset to assess the contributions of three keyword hints selection strategies, with results presented in Table 4. Firstly, it can be observed that using each of the three strategies individually yields better results than not using keyword hints. When using PMI alone, there is a performance decrease of 1.68% and 2.93%, respectively. This is due to the introduction of partially semantically irrelevant keyword hints leading to a deviation in relation dependencies. Using TF-IDF and semantic similarity alone resulted in performance drops of 4.33% and 4.27% on ReDMP and 3.25% and 4.08% on SuRE. This is because these two strategies cannot identify the most representative keyword hints. In comparison, PMI contributes the most to our method.

Number of Keyword Hints

In this experiment, we investigate how the quantity of keyword hints influences on SemEval. The results, depicted in Figure 3, reveal surprisingly consistent trends across all sampling settings on both ReDMP and SuRE. Using only one keyword hints suffices to achieve optimal results in all cases. Increasing the number of keyword hints does not lead to performance improvement; instead, there is a varying degree of decline across all sampling settings, sometimes even lower than when no keyword hints are used. This is because in SemEval, short sentences are predominant, and an excessive number of keyword hints as hard constraints can limit the diversity of expressions.

Generated Data Size

In this experiment, we report the performance of the RE model by combining the sampled original sentences and generated sentences. How to determine the optimal expansion ratio of generated sentences is of great significance in data augmentation. Less generated sentences may not fulfill the purpose of data augmentation, while too many sentences can alter the distribution of the original sentences, resulting in performance degradation. So we investigate the impact of different sizes of generated data on model performance. We conduct experiments on two RE models with expansion ratios ranging from 1 to 6 under the Shot-20 sampling setting on the SemEval dataset. The results are presented in Figure 4.

Most data augmentation methods exhibit considerable performance improvements as the expansion ratio increases from 1 to 4. However, as the expansion ratio increases, the improvements gradually become smaller and level off. REMix and GDA experienced significant performance drops, indicating that an excess of enhanced data changes data distribution. Meanwhile, EDA shows more minor performance improvements when increasing the expansion ratio in most cases, possibly due to poorer diversity in data generation. Additionally, LAMBADA performs lower than Base on ReDMP, likely due to insufficient training data. Our method consistently performs best under all ratio settings, illustrating that our approach can maintain the distribution of sampled original sentences unchanged under keyword hints constraints while increasing generated sentence diversity.

Diversity Evaluation

To assess the diversity of synthetic sentences, we introduce the Distinct [Li et al., 2016], which quantifies the number of distinct unigrams and bigrams divided by the total number of generated words. The calculation formula is as follows:

We set N as 1 and 2, representing the proportion of unique words and bigrams, respectively. The scores under all sampling settings on the SemEval dataset are presented in Figure 5. Overall, generative methods (LAMBADA, GDA) exhibit better diversity than editing methods (WSS, EDA). Notably, our method consistently outperforms others in diversity across almost all settings, providing further evidence of the effectiveness of our approach in enhancing the diversity of synthetic sentences.

3.6 Case Study

To provide further insight, we illustrate a case in Table 5. It can be observed that editing methods effectively maintain semantic consistency between sentences and their originals. However, simple replacement may struggle to enhance the diversity of samples, posing challenges to the generalization of RE models. Generative methods can synthesize sentences with significant contextual differences from the original ones, but they alter the distribution, leading to a biased dependency toward relation types like org:member of. When the keyword hints student is not used, the relation between Jane Bolin and Yale Law School in the generated sentence is not explicitly stated, potentially introducing semantic bias. Adding keyword hints student to the prompt helps the LLM better focus on the relation between the two entities, ensuring that the generated sentence maintains the dependency between them.

4 RelatedWork

4.1 Data Augmentation

(1) For editing methods, Mueller and Thyagarajan [2016] generate additional training data by replacing selected tokens with synonyms from WordNet. Wei and Zou [2019] propose a set of token-level word operations for data augmentation, encompassing synonym replacement, random insertion, swap, and deletion. Jiao et al. [2020] utilize word embeddings to obtain augmented data. (2) For generative methods, Xie et al. [2020] and Fabbri et al. [2021] utilize back-translation on each sentence. Lowell et al. [2021] adopt a strategy of masking multiple words in a sentence and generating new sentences by filling these masks. Anaby-Tavor et al. [2020] fine-tune GPT-2 and generate candidate examples for each class. Hu et al. [2023] employ two complementary modules to train a model, one maintaining semantics through reordering and the other preserving grammatical structure through a unified pattern. However, editing methods cannot satisfy diversity, and generative methods cannot maintain relation consistency. Our method applies semantically consistent contextual constraints and leverages LLMs to generate sentences simultaneously satisfying relation dependency consistency and diversity.

4.2 LLMs for Low-resource RE

The rise of LLMs demonstrates the advance in low-resource RE. Li et al. [2023] propose the summarize-and-ask prompting, exploring the possibilities of LLMs in zero-shot RE. Wan et al. [2023] add task-aware representation to demonstration retrieval and enrich the demonstrations with gold labelinduced reasoning logic. Wang et al. [2023] unify modeling of various IE tasks based on instruction tuning tasks and capturing inter-task dependencies. However, the efficiency of mapping inputs and labels with demonstrations needs to be improved to thoroughly express complex RE tasks [Deng et al., 2023]; computing resources will also limit prompt-tuning LLMs. Therefore, it is more practical to use LLMs for data generation and then transfer it to the RE model.

5 Conclusion

This paper posits that the primary challenge in low-resource RE DA is ensuring the semantic consistency and contextual diversity of generated sentences. To address this, we propose a novel method named ConsistRE. ConsistRE incorporates keyword hints closely related to the relation instances as contextual constraints in sentence generation with LLMs and complements it with syntactic dependency selection. Experiments conducted on three public datasets under low-resource settings substantiate the effectiveness of our approach.

Acknowledgments

We thank the reviewers for their insightful comments. This work was supported by National Science Foundation of China (Grant Nos.62376057) and the Start-up Research Fund of Southeast University (RF1028623234). All opinions are of the authors and do not reflect the view of sponsors.

Contribution Statement

The contributions of Yifan Zheng and Wenjun Ke to this paper were equal.