AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Loss

1 Introduction

Visual Question Answering (VQA) targets at accurately answering natural language questions regarding a visual scene. Perceived as a more 'AI-complete' task, VQA has received critical attention from both computer vision and natural language processing communities. Recently, several large-scale benchmarks [Antol et al., 2015; Goyal et al., 2017] have been constructed to facilitate the advancement of VQA, followed by a number of elaborately designed visual reasoning models [Kazemi and Elqursh, 2017; Anderson et al., 2018; Zhang et al., 2018].

∗Corresponding Author: Liqiang Nie and Zhiyong Cheng.

Despite its flourishing development, there still exits a major issue in VQA, that is, most VQA models tend to make predictions by taking the language shortcut between questions (or more specifically, question types 1 ) and answers. For example, 2 takes large proportions of the answers to questions initiated with how many in the train set. In such case, the model will take the shortcut to leverage these language priors and directly yields answer 2 to how many questions when testing, regardless of both the remaining question words and the given image. As a result, the visual signal is largely under-explored, making the visual reasoning in VQA deteriorate into the a pure language matching problem without the consideration of 'V'. Obviously, it is a contradiction to the initial motivation of this vision-and-language task.

Many research efforts have been devoted to overcoming this arduous problem, which can be roughly projected into two groups of methods: 1) balancing the biased dataset; and 2) correcting VQA models for alleviating the effects of language priors. In detail, the first category of methods contribute to either enlarging [Goyal et al., 2017] or resplitting [Agrawal et al., 2018] the existing biased datasets. Notably, [Agrawal et al., 2018] curated a diagnostic dataset - VQA-CP, wherein the answer distributions of per question type are significantly distinct between the train and test sets. It thus provides a favorable test-bed for estimating the language prior effect the VQA model encounters. Based on this dataset, some pioneering work has been conducted to correct the advanced VQA models for alleviating language prior problem [Ramakrishnan et al., 2018; Chen et al., 2020; Guo et al., 2020]. A typical design of this category of methods is to attach an additional question-only branch, whereby the language prior is intentionally captured and further suppressed by the orthodox question-image branch.

Due to the fact that the answers in most VQA datasets are composed of a few words, existing VQA models often cast VQA as a classification task, i.e., classifying the answer from candidates rather than progressively generating answer words to achieve better visual reasoning capability. However, what attracts our attention is that, there has been no research diving into investigating the learned feature space that each answer/class engages. We argue that distinguishing answers in the learned feature space is the key to alleviate the language prior effect but existing models cannot well address this in the biased dataset. Inspired by this thought, we empirically visualize the 2-D embedding features of the answers to the question type how many in Fig 1b. It is obvious that different answers of the baseline are actually intertwined with each other in the two kinds of feature spaces. On the basis of this phenomenon, we make a further assumption: is it beneficial for overcoming the language prior problem if we properly separate the answers via manipulating their learned feature space?

1 We refer question type in VQA to the first few words in the given question.

Figure 1: Answer manifold embedding with respect to questions types in (a) and answers under how many question type in (b) and (c). Note that the numbers of each answer class are identical for better visualization. For (b) and (c), the answer distribution in the train set is demonstrated on the rightmost.

To address this question, we propose to revisit the language prior problem from the viewpoint of the answer feature space learning. The primary goal of this work is to introduce an adapted margin for different answers in the angular space under the corresponding question type, which can effectively separate the answer embeddings. Towards this end, we firstly reformulate the softmax loss function as a cosine loss by L2 normalizing both answer features x and weight vectors Wi [Liu et al., 2017; Wang et al., 2018a]. Consequently, the decision boundary can be computed through the cosine function of the angle θi between x and Wi, i.e., WT i x = ||Wi||2 · ||x||2 · cosθi = cosθi, s.t. ||Wi||2 = 1 and ||x||2 = 1. Thereafter, an adapted margin mi is employed to separate the answer features, i.e., cosθi -mi, where mi is computed based on the train set statistics of answer ai under the corresponding question type. In the ideal scenario, for each given question and its corresponding question type, frequent answers would span broader in the angular space with a smaller margin, while sparse answers span tighter in the angular space with a larger margin (see Fig. 1c). The intuition behind this is that frequent answers under certain question types often go with more training samples, and thus a smaller margin is required to control a broader space to sufficiently include these answers. In contrast, the number of sparse answers is relatively small, requiring a more tighter feature space. Moreover, we find that the answer embeddings in the Euclidean space become more discriminative, and further alleviating the language prior effect.

In a nutshell, the proposed adapted margin cosine loss is beneficial for overcoming the language prior problem in VQA, making itself being model-agnostic and easy to implement. Besides, it introduces NO incremental parameters or computational cost when testing. To the best of our knowledge, this is the first attempt to directly optimize the feature space learning to attack this problem. Additionally, we apply this loss function over three baseline methods, verifying its effectiveness on two VQA-CP benchmarks. The experimental results demonstrate that the baselines equipped with our loss achieve an absolute gain of over 15% averagely. The code is released for the re-implementation of this work 2 .

2 RelatedWork

In recent years, a considerable literature has thrived around the task of VQA [Antol et al., 2015; Malinowski et al., 2018; Goyal et al., 2017; Wang et al., 2018b; Zhang et al., 2018]. From the very beginning, researchers proposed to encode the question and image into a common latent space, based on which the accurate answers can be identified through classification [Antol et al., 2015; Malinowski et al., 2015]. With the prevailing advent of the attention mechanism, different contributions of each image region [Yang et al., 2016;

2 https://github.com/guoyang9/AdaVQA.

Fukui et al., 2016; Kazemi and Elqursh, 2017; Anderson et al., 2018] or each question word [Lu et al., 2016; Yu et al., 2017] for the answer prediction have been taken into consideration. Some studies have also explored the modular structure of questions for more explicit reasoning [Andreas et al., 2016]. Nevertheless, questions in real world often require reasoning beyond what is in the image. To better handle such situation, FVQA [Wang et al., 2018b] and OK-VQA [Marino et al., 2019] develop new benchmarks as well as insightful approaches to incorporate structured and unstructured knowledge in VQA, respectively.

It is noted that many VQA models are influenced by the language prior problem [Agrawal et al., 2018; Guo et al., 2019]. Therefore, some efforts have been devoted to the biased dataset balancing. For instance, VQA v2 [Goyal et al., 2017] adds complementary samples to the original VQA v1 dataset [Antol et al., 2015] such that for each sample <v,q,a> , another one with the same q, similar v yet a different a is generated. VQA-CP [Agrawal et al., 2018] re-splits the VQA v1 and v2 datasets that the answer distribution of per question type is distinct between the train and test sets. In addition to the studies on the dataset side, methods devised for tackling this problem mostly try to diminish the answer prediction influence from the question encoding layer [Selvaraju et al., 2019; Wu and Mooney, 2019; KV and Mittal, 2020; Jing et al., 2020]. One popular solution essentially involves a new question-only training branch, wherein the answers are predicted based solely on the question input [Ramakrishnan et al., 2018; Cadene et al., 2019; Clark et al., 2019; Chen et al., 2020]. The answer prediction from this branch is further suppressed by the original question-image one. In addition, [Guo et al., 2019] designed an evaluation metric to quantify the language prior effect in VQA models.

3 ProposedMethod

3.1 ProblemDefinition

Following the prevalent formulation, we consider the VQA task as a multi-label classification problem. That is, for a textual question Q regarding an image I, the objective function is given by:

where Ω and Θ denote the answer set and the model parameters, respectively. Note that there can be multiple correct answers for each instance. The negative log likelihood loss function is then formulated as 3 :

where W and x denote the weight matrix and feature vector directly adjacent to the answer prediction, respectively;

3 The individual loss is considered instead of the total instances throughout this paper due to the space limitation.

ai ∈ [0, 1] represents the corresponding answer label. Besides, we remove the bias vector for simplicity as we also found it contributes little to the final model performance.

3.2 Method Intuition

To date, most VQA methods tackling the language prior problem present to confront the prediction from a question-only branch. Little attention has been paid to explore the answer/class embedding in the learned feature space. In this work, we adopt a new viewpoint to overcome this problem, and validate its effectiveness in what follows. Specifically, we propose to place an adapted margin to separate the answer embeddings in the angular space properly based upon the statistics in the train set. Compared to the Euclidean space, the angular space is relatively easy to manipulate, as the margin range reduces from (-∞,+∞) to [-1, 1]. In the light of this, we firstly leverage the L2 normalization on weight vector Wi and feature vector x to ensure the posterior probability to be determined by the angle θi betweenWi and x (the answer feature space is converted from the Euclidean space to the angular space), i.e., ||Wi||2 = 1 and ||x||2 = 1. Accordingly, a modified normalized SoftMax loss (NSL) [Wang et al., 2018a] can be obtained by,

where s is a scale factor for more stable computation. In order to achieve a more discriminative classification boundary, LMLC [Wang et al., 2018a] introduces a fixed cosine margin to NSL,

where m implies the fixed cosine margin.

In implementation, we found that applying a fixed cosine margin cannot obtain satisfactory results. The key reason is that the answer distribution of each question type is highly biased, resulting in the incapability of learning a sufficient representation with a fixed margin in the angular space. In the next subsection, we will introduce a more sophisticated adapted margin cosine loss to overcome this issue.

3.3 AdaVQA Formulation

The results in our experiments (see Sect. 4.2) explicitly demonstrate that a fixed cosine margin yields limited improvements or even deteriorates the model performance. Based upon this observation, we argue that an adapted cosine margin is more favorable for overcoming the language priors in VQA. In view of this, a new loss function is elaborately defined as:

where mi is the adapted margin for answer i based on the given question type, nki denotes the number of answer i under the question type qtk in the train set, ϵ = 1e - 6 is a hyperparameter for avoiding computational overflow. The intuition behind this is that, for the current given question and its corresponding question type, the frequent answers span broader in the angular space (smaller margin), while sparse answers span tighter (larger margin). In other words, frequent answers imply more training samples, which require a broader feature space to sufficiently include these answers. On the contrast, a tighter feature space is acceptable for sparse answers as the number of training samples is much smaller. This setting enables the VQA models to place a more proper cosine margin in the angular answer feature space. Taking Fig. 1c as an example, answer 2 is more frequent than the other two answers (1 and 100) in the train set. As a result, it spans a larger feature space as we expect. In particular, Fig. 2 provides a general training pipeline of AdaVQA, where the feature space for the three answers (1, 2, 100) is learned based on the proposed loss function.

However, it will be a concern that such a margin restriction might damage the ability of the inter-type discrimination, i.e., the answer decision boundary among different question types becomes less discriminative. We therefore obtain the answer embeddings with respect to its question type and visualize them in Fig. 1a. From this figure, an obvious decision boundary pertaining to question types can be observed. One possible reason to this situation is that the cosine margin of each answer is computed according to the train set statistics under the given question type, while other answers outside this question type are kept untouched. This empirical setting inherently equip the model with the discrimination capability among different question types.

Entropy threshold. One major problem have yet to be addressed, i.e., not all questions within their corresponding question type would cause the language prior problem. Different from previous methods handling all questions without differentiation, we propose to employ an entropy threshold over each question type. Concretely, only the questions whose question type entropy is greater than the threshold will be considered for regularization. Specifically, the entropy for each question type is defined as:

where ¯mki is actually the occurrence probability of answer i under the question type qtk in the train set.

13 Partial derivatives. We further provide the partial derivatives of the weight vector Wi and feature vector x from our loss function. Let pi = s(cos θi -mi) = s( WT i ||Wi||2 · x ||x||2 - mi), and ˆ pi = exppi ∑|Ω| j=1 exp pj . The partial derivatives are obtained via,

and

Lower bound for s. The scale factor s is critical for the final answer feature learning. A too small s leads to an insufficient convergence as it limits the feature space span (as we found in our experiments that the loss goes 'nan' with a small s). In view of this, a lower bound for s should be prescribed. Without loss of generality, let Pi denote the expected minimum of the answer class i, the lower bound is defined as,

3.4 Comparison with Different Loss Functions

We consider the binary-class scenario for intuitively illustrating the decision boundary from different loss functions. As can be seen from Fig. 3, the decision boundary of the plain SoftMax one can be negative, which is enlarged to be equal to zero of the NSL loss [Wang et al., 2018a]. The LMLC [Wang et al., 2018a] defines a fixed margin for different classes, which is not suitable in our case for overcoming the language prior problem. Regarding our AdaVQA, the sparser answer (green one) is mapped to a smaller feature space while the more frequent answer (orange one) engages a larger feature space.

4 Experiments

4.1 Experimental Setup

Datasets. To validate the effectiveness of the proposed loss function, we conducted extensive experiments on the two

17

VQA-CP datastes: VQA-CP v2 and VQA-CP v1 [Agrawal et al., 2018], which are two public benchmarks for estimating the models' capability of overcoming the language prior problem in VQA.

Evaluationmetric. The standardmetric inVQA is adopted for evaluation [Antol et al., 2015]. That is, for each predicted answer a, the accuracy is computed as,

In addition, the answers are divided into three categories: yes/no, number and other.

Implementation details. We empirically applied our AdaVQA to three baselines: Strong-BL [Kazemi and Elqursh, 2017], Counter [Zhang et al., 2018] and UpDn [Anderson et al., 2018]. Different from most prior methods overcoming the language prior problem in VQA, for all the three baselines, we simply replaced the original loss with our AdaVQA and did NOT change any other settings, such as embedding size, learning rate, optimizer, and batch size.

4.2 Experimental Results

Overall Accuracy

The experimental results on VQA-CP v2 and VQA-CP v1 are illustrated in Tab. 1 and Tab. 2, respectively. The main observations from these two tables are as follows:

• Our method achieves the best on the two benchmark datasets except for the Other answer category (SCR [Chen et al., 2020] and LMH [Clark et al., 2019] outperform ours on VQA-CP v2 and VQA-CP v1, respectively.). One possible reason for this is that most questions under this answer category are starting with 'what ...', and the answers are therefore much more diverse than other categories. Accordingly, less language prior will be introduced under this category since the answers are less biased.
• For all the three baselines, i.e., Strong-BL, Counter and UpDn, when applying our loss function, a significant performance improvement (15% on average) can be observed. For example, on the VQA-CP v2 dataset, Counter+Ours achieves an absolute performance gain of 12.23% compared with Counter; and on the VQA-CP v1 dataset, UpDn+Ours outperforms the baseline UpDn with 23.18%.
• Compared with other methods whose backbone model is also UpDn on the VQA-CP v2 dataset, our method (UpDn+Ours) surpasses them with a large margin, especially for the three newly accepted approaches VGQE, CSS and Decomp-LR.

Ablation Study

For a deeper understanding of our AdaVQA, we further provided detailed ablation studies on the VQA-CP v2 dataset. From the results in Tab. 3 and Fig. 4, we can see that:

• As shown in Tab. 3, when using the L2 normalization on both the weight vector and the feature vector, the de facto NSL, the results are inconsistent amongst different methods. For example, the Counter with NSL surpasses the baseline with 11.41%, while Strong-BL with NSL even leads to a little bit deterioration of the performance.
• We then leveraged the fixed margin to yield effective feature discrimination, where the margins are tuned from 0.1 to 0.9 with a step size of 0.2. However, the results are not favorable (the improvement is limited), which validates the evidence that a fixed margin is not suitable for overcoming the language prior problem. In contrast, when replacing the fixed margin with our adapted one, the model can substantially outperform the one equipped with fixed margins, which additionally proves the superiority of AdaVQA.
• We investigated the different influence of the question type entropy and scales in Fig. 4. From this figure, we found that 1) an appropriate entropy should be considered, since a very large one will not bring more gains or even cause performance degradation; and 2) a too small scale is insufficient for learning the feature space, while too large ones will also lead to unsatisfactory results.

Case Study

Finally, we visualized two successful cases from our method and illustrated them in Fig. 5. Regarding the first case, as the answer 2 takes large proportion under the question type how many in the train set, the baseline model thus yields an answer 2 to this question. In contrast, our AdaVQA corrects this mistake, producing a more reasonable attention map (focusing on the elephant object). As for the second case, the baseline model wrongly predicting the answer white mainly attribute to white being more frequent under the question type what color in the train set. Besides, too much attention has been paid to the cat region, which contributes another reason to the incorrect answer, as the fur of the cat is white and black. On the contrary, our AdaVQA can guide the model to pay more attention to the target object - luggage, leading to the correct answer red.

5 Conclusion

We propose to tackle the language prior problem in VQA from a novel view of the answer feature space learning, which has not been explored in previous studies. To achieve this goal, we design an adapted margin cosine loss to discriminate the answers via properly characterizing the answer feature space. Concretely, for the given question, the frequent and sparse answers with respect to the corresponding question type are learned to span broader and tighter on the angular space, respectively. Extensive experiments over two benchmarks validate the effectiveness of the proposed loss function on three baselines.

It is worth mentioning that this work does not present an advanced model for pursuing the SOTA results in VQA. We instead expect that, with this adapted margin cosine loss, future research can focus more on the enhancement of the visual reasoning capability without too much influence of the language prior problem.

Acknowledgments

This work is supported by the National Key Research and Development Project of New Generation Artificial Intelligence, No.:2018AAA0102502.