Resolving Word Vagueness with Scenario-guided Adapter for Natural Language Inference

1 Introduction

"Humans rely less on words than on visual images, auditory images and propositions or rules of logic in order to think. Even commonly used metaphors and analogies employ visual and spatial attributes to provide us with a quick and easy context in which to communicate and build thoughts [Pinker,

∗Corresponding Author

2003]," claimed a world-renowned linguist and evolutionary psychologist, Steven Pinker. This statement highlights the potential significance of visual information in human cognitive processing of real-world challenges, suggesting that it may, in fact, outweigh the importance of symbolic natural language information. This proposition warrants further investigation and consideration within the context of cognitive science and human-computer interaction research.

Natural language inference (NLI), as a vital task in NLP, has received widespread attention since it is closely related to human language comprehension and reasoning abilities. NLI involves knowledge from multiple fields, such as natural language understanding [Storks et al., 2019; Liu et al., 2024b] and logical reasoning [Li et al., 2024; Liu et al., 2024a], and is a challenging task. It has been widely used in various practical application scenarios, such as question answering [Liu et al., 2023a; Liu et al., 2023b], text summarization [Falke et al., 2019; Guan et al., 2021; Liu et al., 2021], and information retrieval [Murty et al., 2021; Liu et al., 2022]. Generally, NLI consists of two sentences, referred to as the premise and the hypothesis. The premise is an assertive sentence that provides descriptive information, while the hypothesis is a sentence that requires inference based on the premise. The goal of NLI is to determine whether the hypothesis can be inferred from the premise (i.e., entailment), or contradicts the premise (i.e., contradiction), or has no relationship with it (i.e., neutral).

Recently, modern NLI models [Chen et al., 2017; Conneau et al., 2017a] have made significant strides by leveraging the powerful feature extraction capabilities of deep learning to enhance their ability to understand and represent the semantic information of sentences. Successfully determining the semantic relationship in NLI tasks requires a deep comprehension of the semantics between sentences. Moreover, to further promote the development of NLI, researchers use human-labeling approaches to obtain large-scale high-quality annotated datasets, such as SNLI [Bowman et al., 2015] and MultiNLI [Williams et al., 2018], to evaluate the effectiveness of the proposed models. Despite the fruitful success achieved on these datasets, most previous models are still lim-

Premise: A group of women are playing volleyball.

Hypothesis: People are outside tossing a ball.

(B)

Figure 1: The two sentences have an entailment relationship when viewed in the context of image (A), but a contradiction relationship when viewed in the context of image (B).

ited in performance due to their focus on extracting textual information while ignoring other useful information, such as visuals, which are crucial for proper sentence semantics understanding that can be used to discern the semantic relations between sentences.

Previous studies [Gibson et al., 2019] have highlighted the limitations of language due to its inherent ambiguity and vagueness. This often leads to challenges in interpretation, as evident in the SNLI dataset, where different annotators may perceive the same pair of sentences as either an entailment or a contradiction. In contrast, images are more concrete and can convey complementary information [Pan et al., 2016] that is often overlooked by sentences such as facial expressions and motions, which can effectively solve textual ambiguity [Tong et al., 2020]. Thus, exploring the complementary information conveyed by images can help gain better insights into actual sentence meanings. As shown in Fig. 1, both the premise and hypothesis describe the scenario of people playing with balls. However, while the hypothesis specifies an outdoor setting, the location in the premise remains ambiguous. This ambiguity indicates a neutral relationship between the premise and hypothesis. Once the image related to the premise is provided, the relationship between the premise and hypothesis can be determined with certainty. In Fig. 1 (A), the location information suggests that people are outdoors, creating an entailment relationship between the premise and hypothesis. Conversely, in Fig. 1 (B), the location information indicates people are indoors, resulting in a contradictory relationship between the two. From this example, we can conclude that scenario images further enrich the semantics of sentences and provide necessary supplementary information. Although many images are gathered alongside the sentences during dataset collection, they are not given much attention. Hence, it is necessary to leverage relevant scenario images to fully understand the sentence semantics for NLI tasks.

Our objective is to incorporate scenario information into our sentence-level representations in a precise manner, which poses a considerable challenge. Naive integration of the two modalities can introduce noise that hinder semantic understanding, so finding an effective way to merge images with sentences remains an open problem. Although extensive research on multimodal tasks, there is currently no widely accepted method for combining image modality with sentences.

To address the aforementioned challenges, we propose a novel scenario-guided adapter, coined ScenaFuse, for NLI tasks. Our approach involves designing an image-sentence interaction module that generates scenario-guided sentence semantic representations by deeply interacting images and sentences in the attention module of the pre-trained model. This module benefits the semantic understanding by explicitly considering the scenario images. Consequently, the text representations obtained from our approach incorporate valuable scenario information, are more comprehensive, and possess stronger representational power compared to those generated using a single modality. Moreover, we introduce an attention-based image-sentence fusion module with the gate mechanism to perform multimodal feature fusion related to sentence semantic and scenario-guided representations. This module selects informative features from both modalities while minimizing the noise impact, which facilitates the evaluation of sentence relationships. In summary, our contributions are listed as follows:

• We introduce a novel scenario-guided adapter named ScenaFuse, which explicitly incorporates visual information from the relevant scenarios. This is a valuable attempt at integrating scene information into large pre-trained language models for NLI tasks.
• We propose two modules: the image-sentence interaction module and the image-sentence fusion module. The former enables deep interaction between images and sentences, and the latter integrates two different modalities of features through specifically designed mechanisms. Moreover, we further demonstrate the effectiveness of the proposed approach on recent popular large language models.
• We conduct extensive experiments on benchmarks to evaluate the effectiveness of our approach. Our empirical results demonstrate the superiority of ScenaFuse compared to other competitive baselines.

2 RelatedWork

2.1 Natural Language Inference

NLI models can be mainly divided into three categories: symbolic, statistical, and neural network-based models [Storks et al., 2019]. Symbolic models use logical forms to perform inference tasks and such models are heavily used in early NLI problems known as recognizing textual entailment (RTE) [MacCartney, 2009]. As data-driven feature training becomes more prevalent in NLP, statistical models based on feature engineering such as bag-of-words [Zhang et al., 2010], become the mainstream approach. Some work [Haim et al., 2006; Dagan et al., 2006] also attempts to incorporate external knowledge as a supplement to training data features and achieve favorable results in NLI. The recent advancement in deep learning, along with the availability of larger annotated datasets, has led to the success of more complex neural network models in NLI tasks. These models can be categorized into two types: sentence-encoding-based and inter-sentenceattention-based. Sentence-encoding-based models [Conneau et al., 2017b] utilize the same neural network architecture, such as LSTM [Bowman et al., 2015] and GRU [Vendrov et al., 2016] and their variants, to encode both the premise and hypothesis. A neural network classifier is then utilized to predict the relationship between the two sentences. On the other hand, models based on inter-sentence attention [Rockt¨aschel et al., 2016; Wang and Jiang, 2016] introduce attention mechanisms between the premise and hypothesis. This not only alleviates the vanishing gradient problem but also provides alignment between inputs and outputs, which allows for a better understanding of their relationships.

2.2 Multimodal Learning

Multimodal data has garnered considerable attention from both academia and industry due to its potential applications in learning, analysis, and research [Ramachandram and Taylor, 2017]. Multimodal models aim to extract information from varying modalities to learn expressive features that can be used for downstream tasks. The core of multimodal learning is to map unimodal data into a multimodal space, and subsequently acquiring the joint representation. Classic methods involve concatenating features from each modality and training the model, while more advanced approaches utilize neural networks to input each modality into several separate neural layers in an end-to-end manner, and then project them into hidden layers in the joint space [Antol et al., 2015]. This joint multimodal representation can then be processed through multiple hidden layers or directly used for prediction. Due to the incorporation of large quantities of data during neural network training, the resulting joint representation typically performs well on various tasks. Recently, many studies have introduced multimodality into NLP tasks, such as named entity recognition [Moon et al., 2018] and machine translation [Nishihara et al., 2020]. Generally, these methods enhance rough text understanding from non-textual perspectives by employing modality attention mechanisms to integrate information from heterogeneous sources. For example, MNMT [Nishihara et al., 2020] incorporates a supervised visual attention mechanism, which is trained with constraints between manually aligned words in the sentence and corresponding regions in the image. This mechanism can more accurately capture the relationship between words and image regions. MNER [Moon et al., 2018], on the other hand, proposes a universal modality-attention module that learns to reduce irrelevant modalities while amplifying the most informative ones for named entity recognition of data composed of text and images.

3 ProblemDefinition

In this section, we define the NLI problem in detail. Consider a scenario where we have been provided with a premise consisting of m words, represented by P = (wp,1,wp,2, · · · ,wp,m), as well as a hypothesis with nwords, depicted by H = (wh,1,wh,2, · · · ,wh,n). Additionally, an associated image I is also available. The token embeddings Xtex in P and H are the sum of word, segment and position embeddings for each token, where the word embeddings can be initialized randomly or using word2vec or GloVe, i.e., Xtex,i ∈ Rd . In general, we follow the experimental settings of large-scale pre-trained models such as BERT [Devlin et al., 2019] or RoBERTa [Liu et al., 2019b]. To formalize two sentences into this format, we use the sentence pair

"[CLS] P [SEP] H [SEP]" as model inputs. Here, [CLS] is a special symbol added to the beginning of each input example, and [SEP] is used as a special separator to separate sentences. We extract the [CLS] token as the output of each example. Because it lacks clear semantic information and can better integrate the semantic information of each word in the sentence. As a result, it represents the sentence's semantics better. Subsequently, we feed it into a fully connected layer for classification tasks.

For the associated image I, we reshape it 224x224 pixels and input it into the pre-trained image encoder Enc(·), such as VGG [Simonyan and Zisserman, 2014] or ResNet [He et al., 2016], which have already demonstrated the ability to extract meaningful representations from input images in deep layers. Subsequently, we obtain its visual representations by retaining the output of the last convolutional layer. This process essentially segments each input image into visual blocks of the same size represented by d′ -dimensional vectors, denoted as Xvis = Enc(I) = (Xvis,1,Xvis,2, · · · ,Xvis,k) ∈ Rk×d′ , where Xvis,i denotes the feature representation of the i-th visual block.

In summary, we expect to train a model F that can precisely determine the relationship between the premise and the hypothesis when relevant data is provided, i.e., y = F(P,H, I) ∈ {0, 1, 2}, where y=0 implies an entailment relationship, y=1 represents a neutral relationship, and y=2 represents a contradiction relationship.

4 Method

This section provides an in-depth explanation of the two main modules that constitute the scenario-guided adapter, namely the image-sentence interaction and the image-sentence fusion. To facilitate a better understanding of our model, we have depicted its overall framework in Fig. 2.

4.1 Image-sentence Interaction

As stated in the abstract, compared to language, images provide a more precise means of conveying information and may include important details that are difficult to express or neglected by language due to its inherent ambiguity and vagueness. Moreover, multimodal information has the potential to provide more discriminative input than a single modality, which can help address the issue of semantic ambiguity in text-only modality. Therefore, we explicitly incorporate visual contextual information to significantly enrich the semantic content of the sentence. To this end, we design an imagesentence interaction module. Specifically, we first adopt a linear transformation to project both modalities into the same representation space to achieve embedding space consistency and cross-modal semantic alignment between text representation modality Xtex and visual representation modality Xvis. This can be expressed as follows:

where φ(·) and ψ(·) are linear projection functions that project the text representation and visual representation to the same generated space, i.e., φ : Rd → Rf and ψ : Rd′ → Rf . ˜Xtex and ˜Xvis are generated sentence representations and visual representations, respectively.

Visual-enhanced Sentence Representation. Subsequently, we consider that the textual content in NLI tasks is limited. To address this limitation and enable the utilization of visual information to enhance word representation learning, we employ a multi-head attention mechanism akin to the Transformer architecture. Concretely, for each head, visual representations ˜Xvis are treated as queries, while the textual representations ˜Xtex are treated as keys and values, which can be formulated as:

where Qtex,Ktex, and Vtex are query vectors, key vectors, and value vectors obtained by applying the corresponding linear transformations. Ztex ∈ Rk×t is the derived visualenhanced sentence representation, where k is the length of all visual blocks. After performing Eq.2, the sentence explicitly interacts with the associated image.

Sentence-rectified Visual Representation. Furthermore, it is important to note that each visual block may have a close association with multiple input words. To gain a deeper understanding of the relationship between images and sentences, it is necessary to align the semantic information of each visual block with its relevant words. Additionally, we aim to impose certain constraints on the image through corresponding sentences to alleviate irrelevant associations. To achieve this, we use the following Eq.3 to obtain sentence-rectified visual representations, which can complement the previously obtained visual-enhanced sentence representations. Specifically, employing the multi-head attention mechanism, the sentence representation ˜Xtex acts as queries, while the visual representation ˜Xvis serves as keys and values. The procedure can be formulated as follows:

where Qvis,Kvis, and Vvis are also query vectors, key vectors, and value vectors derived by performing the corresponding linear transformations. Zvis ∈ Rl×t is the obtained sentence-rectified representation, which can be interpreted as constraining visual semantics from a textual perspective, where l is the length of all the tokens involved in the process.

Afterwards, we merge Ztex and Zvis by concatenation and apply a fully connected layer to serve as the output of the image-sentence interaction module. This process can be denoted as:

where ϕ(·) is another linear projector that projects visualenhanced sentence representation and sentence-rectified visual representation into an interaction space, i.e., ϕ : Rk+l → Rl . Zint ∈ Rl×t denotes the generated final embeddings of the full interaction of the two modalities. By incorporating information from both modalities, the resulting embeddings offer a more comprehensive representation compared to singlemodal input.

4.2 Image-sentence Fusion

Directly using the generated interactive embeddings may introduce potential bias and lead to suboptimal results. To alleviate this, we further design an image-sentence fusion module that applies an adaptive fusion strategy based on a gating mechanism to obtain more precise sentence-level representations and deduce the spread of errors caused by visual bias. This strategy requires simultaneously leveraging pure textual semantic representations ˜Xtex and interaction representations Zint obtained from the image-sentence interaction module. The specific architecture of the proposed module is illustrated in Fig. 2.

We first compute the attention coefficients α using pure semantic representation to update the interaction representations. Then, we obtain the attention coefficients β based on the updated interaction representations ˆZint and utilize them to obtain the updated semantic representation ˆXtex. This process can be formulated as follows:

where {Wα, Wβ} ∈ R2t×1 and {Wz, Wx} ∈ R1×t are weight matrices. {bα, bz, bβ, bx} are the corresponding biases. || and ⊙ denote the concatenation operation and element-wise multiplication, respectively.

Next, based on the gate mechanism, we design an approach that adaptively captures and fuses valuable and informative features from updated semantic and interaction representations, which is defined as follows:

where g is the calculated gating coefficient that dynamically controls the contribution of different representations. Wg ∈ R2t×1 denotes the trainable parameter. bg is the bias. σ(·) represents the sigmoid function. With Eq.6, we can obtain the fusion feature U ∈ Rl×t , which adaptively integrates semantic and interaction representations.

Finally, as we mentioned before, considering the potential noise in an image, it is unreasonable to align such noise with visual blocks and sentences. To address this issue, a filtering mechanism is required to selectively leverage the fusion feature U. If incorporating U enhances the model's ability to understand a sentence, then both the fusion and original semantic features are absorbed by the filter; otherwise, it filters out fusion information. Concretely, this filtering procedure can be expressed as follows:

where Wh ∈ R2t×1 and Wr ∈ Rt×t are weight parameters. bh and bu are the biases. h ∈ Rl determines which fusion information needs to be retained. R ∈ Rl×t denotes the filtered representations obtained by the filtering mechanism for NLI tasks.

The classic pre-trained model based on the Transformer architecture adopts a multi-head attention mechanism. Each attention head eventually acquires an attention output ˆA. However, in our designed scenario-guided adapter, we replace the filtered features A with the features ˆA obtained by the attention mechanism in pre-trained models. This ensures that the model can explicitly incorporate scenario information, thereby more comprehensively understanding sentences and alleviating word vagueness problems.

4.3 Model Training

After obtaining the filtered representation R, we input it into the remaining parts of the pre-trained model based on Transformer architecture, such as layer regularization and feedforward neural network layers, to obtain the final sentence semantic vector F ∈ Rℓ×t for NLI tasks, where ℓ represents the number of training examples.

Additionally, the objective function of this task is the crossentropy function, denoted as:

where ˆ Y is the output probability derived by a linear classifier parameterized by Wf ∈ Rt×3 .

5 Experiment

Dataset. We employ three datasets widely used by previous methods to validate the effectiveness of our proposed scenario-guided adapter. The statistical information of these datasets can be found in Table 1. The brief introduction to the datasets is provided below. (I) SNLI [Bowman et al., 2015] is a large-scale dataset designed for NLI tasks, where premises are summarized from photo captions in the Flickr30k and hypotheses generated by humans. Each instance in the dataset corresponds to an image, which provides scenario information. (II) SNLI-hard [Gururangan et al., 2018] is built upon the SNLI dataset, but it excludes examples from the original test set that have annotation artifacts. (III) SNLI-lexical [Glockner et al., 2018] is also based on the SNLI. However, it creates a new test set that requires simple lexical inference. Note that SNLI-hard and SNLI-lexical have the same training and validation sets as SNLI.

Baselines. We introduce four selected categories of baseline models for comparison. (I) Classical sentence-based neural models include LSTM [Hochreiter and Schmidhuber, 1997], ESIM [Chen et al., 2017], CAFE [Tay et al., 2018b], and CSRAN [Tay et al., 2018a]. (II) Pre-trained languagebased models contain BERT [Devlin et al., 2019], RoBERTa [Liu et al., 2019b], UERBERT [Xia et al., 2021], SemBERT [Zhang et al., 2020], and MT-DNN [Liu et al., 2019a]. Here, we use the base and large versions of BERT and RoBERTa models, and for the remaining models, we adopt BERT-base as the encoder. (III) Large language models include Bloom7.1B [Muennighoff et al., 2022] and Llama2-7B [Touvron et al., 2023]. Due to our limited computational resources, we choose the approximately 7B version to the best of our ability. (IV) Multimodal models consist of NIC [Vinyals et al., 2015], m-RNN [Mao et al., 2015], IEMLRN [Zhang et al., 2018], MIESR [Zhang et al., 2019], VisualBERT [Li et al., 2019], and CLIP [Radford et al., 2021].

Implementation Details. In the default experimental setting, we use a pre-trained ResNet-50 as the image encoder to initialize the visual representation Xvis, which is later frozen during training. The obtained image embeddings Xvis and sentence token embeddings Xtex are input into the bottom Transformer block of the pre-trained BERT-base model after linear projection. The BERT-base model is fine-tuned during training. Moreover, we use the AdamW optimizer with learning rate values of {1e-5, 2e-5, 3e-5, 5e-5}. The warm-up and weight decay are set as 0.1 and 1e-8, respectively. The batch size is determined by grid search in {16, 32, 64}. Additionally, the dropout is within the range of {0.1, 0.2, 0.3}. Meanwhile, we apply gradient clipping within {7.0, 10.0, 15.0} to prevent gradient explosion. We adopt three widely used evaluation metrics: accuracy (abbr. Acc), micro-precision (abbr. P), and micro-recall (abbr. R).

6 Result

Performance Comparison. We present the experimental results in Table 2. In-depth understanding based on the quantitative results are provided as follows.

• We observe that ScenaFuse outperforms all other models on all datasets, providing substantial evidence of its effectiveness in handling NLI tasks. This outcome could be attributed to the designed scene-guided adapter consisting of an image-sentence interaction module and an image-sentence fusion module. The model incorporates scene information, which previous models did not fully utilize, into the analysis, thereby alleviating the inherent vagueness of sentences and enabling a more comprehensive understanding of their semantic information. Furthermore, the introduction of the image-sentence interaction module allows visual representations from images to enrich sentence semantics, while textual representations from sentences can rectify visual information, benefiting each other. Next, the image-sentence fusion module deeply integrates the information from both modalities to obtain more precise sentence representations.
• We observe that ScenaFuse can effectively integrate with recent popular large language models (LLMs), such as Bloom and Llama2. This integration significantly improves the model performance compared to those based on other pretrained language models. This phenomenon is expected, as LLMs are trained on massive high-quality data and possess excellent text understanding capabilities. Furthermore, we find that our model with LLMs still exhibits certain improvements compared to using LLMs alone. Because they ignore scenario information, which can guide words to produce more informative representations.
• We find that sentence-based neural and multimodal models show varying degrees of decline in performance on the SNLI-hard and SNLI-lexical datasets. This finding indicates that the previously proposed NLI models' success is predominantly limited to simple examples, and their ability to recognize text entailment may not be as proficient as anticipated. When the dataset becomes challenging, i.e., removing annotation artifacts in the hypothesis (SNLI-hard) and requiring

identifying specific words in the sentences (SNLI-lexical), it poses difficulties for all models.

Further Discussion. It is possible that someone may question whether scenario could directly replace the premise. If so, we only need to leverage appropriate scenarios and hypotheses to assess inferential relations. To address this doubt, we conduct an experiment where we remove all premise data from the dataset and denote the model variant as ScenaFuserep. From Table 3, we find that the model performance on the removed premises shows a serious degradation. Although the premises summarize the content of the scenario, there are still significant differences between them. Scenarios may serve as reference information to enhance semantic understanding of sentences, but cannot replace premises. Because it may contain noise irrelevant to the core semantics of the sentences. Our work primarily focuses on inference relations between sentences and differs from the work on image and sentence retrieval.

Moreover, we further explore whether the scenario be substituted with the corresponding sentences. We note that the premises are extracted from photo captions in the Flickr30k corpus, indicating that the premise is a summary of the photo. Therefore, this is equivalent to generating a synonymous sentence of the original premise. We use the synonym substitution to generate semantically identical sentences for each premise. Then, we combine the original and the replaced premises into an extended premise and feed it into BERT for training. This model variant is denoted as ScenaFusesub. According to Table 3, when the corresponding sentences replace the scenario, there is some performance improvement compared to BERT, but it is far behind our model.

Ablation Study. We conduct a series of ablation studies on the evaluation dataset by designing different model variants to investigate the impact of various model components on experimental results. (I) w/o ISI: We remove the imagesentence interaction (ISI) module and replace it with the classic multi-head attention mechanism of Transformers, which is equivalent to a complete lack of image information. (II) w/o VESR: We eliminate the visual-enhanced sentence representation (VESR) and only retain the sentence-rectified representation in the ISI module. (III) w/o SRVR: We eliminate the sentence-rectified representation (SRVR) and only retain the visual-enhanced sentence representation in the ISI module. (IV) w/o ISF: We remove the image-sentence fusion (ISF) module and replace it with a simple concatenation. (V) w/o GM: We remove the gate mechanism (GM) of the ISF module and replace it with an average of updated semantic and interaction representations. (VI) w/o FM: We remove the filtering mechanism (FM) of the ISF module and replace it with the average of fusion features and sentence representations.

We reach the following conclusions based on the results presented in Table 4. Firstly, all the designed components make important contributions to the model, and removing any of them results in varying degrees of performance degradation. Secondly, removing the entire ISI leads to the most significant decline in performance, indicating that scenario information can assist in sentence semantic understanding. Thirdly, removing either VESR or SRVR leads to a performance decline, indicating that the two modules benefit each other and only appear together to learn better word embeddings. Fourth, when the whole ISF module is deleted, performance drops due to the gap between interaction vectors and sentence vectors. Besides, removing the GM or FM of the ISF module also results in performance degradation due to potential noise interference in the image.

7 Conclusion

We present ScenaFuse, which is a novel model that addresses the word ambiguity and vagueness problem in NLI tasks by incorporating scenario information explicitly through visual inputs. We utilize pre-trained language models and introduce an image-sentence interaction module to extract features from both modalities. These features are then made to interact using Transformer layers. Additionally, we incorporate an image-sentence fusion module that is specially designed to adaptively fuse the learned features from images and corresponding sentences. Extensive experiments demonstrate that our proposed ScenaFuse outperforms previous competitive methods on three datasets.

Acknowledgments

This work is supported in part by funds from the National Key Research and Development Program of China (No. 2021YFF1201200), the National Natural Science Foundation of China (No. 62172187 and No. 62372209). Fausto Giunchiglia's work is funded by European Union's Horizon 2020 FET Proactive project (No. 823783).

Contribution Statement

Y.H.L. and M.Y.L. designed and developed the method and analysed the data. Y.H.L., M.Y.L., and X.Y.F. drafted the paper. D.L., F.G., X.M.L., and L.H. revised the paper. X.Y.F., and R.C.G. supervised the project and contributed to the conception of the project.