Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation

1 Introduction

Scene graph generation is a fundamental task in computer vision that has been successfully applied to many other tasks, including image captioning [Yang et al., 2019], image retrieval [Johnson et al., 2015] and commonsense reasoning [Zellers et al., 2019]. Given an image, a relationship between objects in the image is typically denoted as a triple: (subject, predicate, object), where the predicate can also be denoted as relation. To detect such relationships requires the understanding of the image content globally. In scene graph generation, the representation of a relationship needs to preserve semantic information of the triple as well as the inherent attributes of the objects and the relations between them. It is a challenging task due to the distributional biases present in the datasets. For example, the benchmark dataset Visual Genome [Krishna et al., 2017] contains 150 distinct objects, producing possible unique relationships of approx. 22K. Such a large number of relationships are too arduous to train a model as it is impossible to cover each relationship with sufficient samples [Zellers and Yatskar, 2018]. This challenge is further complicated by the highly imbalanced distribution in the relations. It has been observed [Zellers and Yatskar, 2018; Zhang et al., 2019b; Dornadula et al., 2019;

∗Corresponding author

Chen et al., 2019] that the distribution of relations of Visual Genome is highly long-tail and biased: the head relations can have 10k instances whereas the tail relations have less than 10 each. Thus, a model can readily learn the representation of head relations but struggles to learn that of tail relations.

Many previous methods focus on the union region of a pair of objects [Deng et al., 2014; Dai et al., 2017], where only visual features are considered, but not distributional bias of relations. However, as mentioned before, due to the highly imbalanced nature of relations, a relation classifier is hardly well optimized by such uneven data. Xu et al [2017] developed a message passing strategy to aid relation recognition where how to refine the object and relation feature becomes the central goal. However, its performance still suffers from the lack of sufficient data required for learning. By counting the frequency of various relations, Neural Motifs [Zellers and Yatskar, 2018] discovers that some relations are highly correlated with the objects. For instance, the possession relation "has" always exists between some specific pairs of objects, such as subject "man" and object "eye". Similarly based on the statistic results from a dataset, KER [Chen et al., 2019] developed a knowledge routing network to preserve the relation bias into their model. Additionally, other work [Lu et al., 2016] utilized natural language information as an auxiliary tool to boost relation classification by mapping the language prior knowledge to relation phrases. One limitation of these methods is their reliance on the statistic bias knowledge, without which their results would decline significantly. Similarly, Gu et al [2019] leveraged ConceptNet [Speer et al., 2017], a commonsense knowledge graph, to bridge the gap between visual features and external knowledge by a recurrent neural network.

Moreover, many recent works discover that a wellrepresented contextual feature can significantly benefit relation recognition. Specifically, Graph R-CNN [Yang et al., 2018] develops an attentional Graph Convolutional Network (aGCN), focusing on learning the contextual information between two objects that are filtered by a Relation Proposal Network (RePN). Qi et al [2019] proposed two interacting modules to inject contextual clue to relation feature: a semantic transformer module concentrating on preserving semantic embedded relation features via projecting visual features and textual features to a common semantic space; and a graph self-attention module embedding a joint graph representation by aggregating neighboring nodes' information. Shi et al [2019] utilized the attention mechanism to enhance node and relationship representation and trace the reasoning-flow in complex scene scenarios.

In this paper we address two critical challenges in scene graph generation: (1) how to effectively encode contextual clue into its corresponding object representation; and (2) how to balance the severely skewed predicate distribution to improve model performance. Specifically, for the first challenge, we propose a scene-object interaction module aiming at learning the interplay coefficient between individual objects and their specific scene context. For instance, the relation triple "man riding bike" is usually associated with the outdoor scene instead of indoor. Therefore, the outdoor scene is a key contextual clue to aid us to confidently predict the "riding" relation once given the objects "man" and "bike" and the outdoor information. To this end, we treat annotated objects of each image as the scene label of the image and deploy a weighted multi-label classifier to learn the contextual scene clue. At the same time, we employ an additive attention technique to effectively fuse the clue and the objects' visual features. For the second challenge, we introduce a knowledge transfer module to enhance the representation of tail (datastarved) relations, by transferring the knowledge learned in head relations to tail relations. In addition, we also introduce a calibration operation, inspired by the notion of reachability in reinforcement learning [Savinov et al., 2018], to resize the head and tail features to enhance their features' discriminative ability. In summary, our contributions are threefold:

• We introduce a scene-object interaction module to fuse objects' visual feature and the scene contextual clue by an additive attention mechanism.
• To alleviate the imbalanced distribution of relations, we propose a head-to-tail knowledge transfer module to preserve rich knowledge learned from the head into the tail. Moreover, our calibration operation further enhances the discriminative ability of learned visual features.
• We evaluate our method on the standard scene graph generation dataset Visual Genome [Krishna et al., 2017] on three tasks: predicate classification, scene graph classification and scene graph detection, on which our model outperforms current state-of-the-art methods.

2 Method

Our overall framework, shown in Figure 1, consists of three main modules: (1) feature extraction, (2) scene-object interaction, and (3) knowledge transfer. Specifically, the sceneobject interaction module aims to combine scene context features into object features via an additive attention mechanism, while the knowledge transfer module focuses on fusing the knowledge learned in head and tail relations to enhance their representation.

2.1 Notations

A scene graph is a directed relation network extracted from a multi-object image. Each edge in a scene graph is represented by a triple (oi,rij,oj), consisting of two objects oi, oj and the relationship predicate rij between them. Additionally, a scene graph requires to localize each object in the referring image and we denote the localization of object oi as bi. Thus, given a set of object labels C and a set of relationship types R (including the none relation), a complete scene graph for an image consists of:

• A set of bounding boxes B = {b1, b2, . . . , bn}, where bi ∈ R4 denotes the coordinates of the top-left corner and the bottom-right corner, respectively.
• A set of objects O = {o1, o2, . . . , on}, assigning a class label oi ∈ C to each bi.
• A set of triples T = {(oi, rij, oj)}, where each oi, oj ∈ O, and rij ∈ R, and that i ̸= j.

2.2 Visual and Spatial Feature Extraction

The first step in scene graph generation is to detect objects in an image. Numerous object detection methods have been proposed, e.g., Faster R-CNN [Girshick, 2015]. To fairly compare to other baseline methods, we adopt Faster R-CNN trained on VGG-16 [Simonyan and Zisserman, 2014] as our object detection and localization backbone network.

For each detected object oi, we extract two types of features: visual features f o i ∈ R4096 and spatial features li ∈ R5 . Specifically, the visual feature extraction f o i follows that of Neural Motifs [Zellers and Yatskar, 2018]. The spatial features li is a 5-dimensional vector that encodes top-left and bottom-right coordinate and the size of object: li = [xti , yti , xbi , ybi , wi ∗ hi], where wi and hi are the width and height of the object respectively. Recent works [Zhuang et al., 2017; Woo et al., 2018] have demonstrated that the relative position of two objects in an image can significantly enhance relation recognition. Thus, we also encode the relative position into their relation representation as sij ∈ R5 . Concretely, we first convert li to the centralized coordinate as [xci , yci , wi, hi] and then calculate the relative spatial feature as sij = [ xtj -xci wi , ytj -yci hi , xbj -xci wi , ybj -yci hi , wj·hj wi ·hi ]. It is worth noting that sij is different from sji. To enrich the representation of sij , we feed the above raw 5-dimensional vector into a non-linear layer and convert it to a 256-dimension vector sij ∈ R256 .

As for the union region features f u ij of subject si and object oi, we first generate their union bounding box and follow the extraction of an object's visual feature to obtain f u ij .

2.3 The Scene-object InteractionModule

For scene graph generation, the correct recognition of relations not only depends on object features, but also takes important cues from the scene. For example, the scene of "outdoor" should contribute more to the relation "riding" while less to "holding", as riding mostly takes place in the outdoor, which is not the case for holding.

Many works, such as IMP [Xu et al., 2017] and Neural Motifs [Zellers and Yatskar, 2018], demonstrate the contextual representation has a conspicuous effect on the relation recognition. In this work, we propose a scene-object interaction module to encode the global scene contextual information into the object representation, which is implemented via an additive attention module widely used in machine translation models [Bahdanau et al., 2014; Britz et al., 2017]:

where f o i is the feature of object oi, f s is the global scene feature of an image, · denotes pointwise product, and wg computes a coefficient of interaction between the object and its contextualized scene. It is worth mentioning that all objects' features f o i in the same image share a common scene feature f s . ai is pruned to the interval [0,+∞], and a greater value of ai corresponds to more interaction with the scene, that is, the scene feature should contribute more to the object feature (see (3)). Note that LinkNet [2018] has also proposed to incorporate scene features, while we consider the contribution of a scene to relations via an attention mechanism instead of a simple concatenation as in LinkNet. wg is implemented by a fully-connected layer activated via a ReLu function. The global feature f s is learned by a weighted multi-label classification loss:

where Wc is a weight for each class and pre-calculated by counting the proportion of each object class in the training set, pc is the probability of each class output from a sigmoid function, lc is the true target label, and BCE(.) is a binary cross-entropy function aiming at classifying multi-label images. With the scene-object interaction, the object feature is then refined as:

From the refined object feature ˜f o i , the union region feature f u ij and the transformed relative spatial feature sij, we construct the final representation of each triple (si, rij, oj) as:

where × is the element-wise multiplication following [Zellers and Yatskar, 2018; Woo et al., 2018], [; ] is the vector concatenation operation, and f t ij ∈ R4096+256 .

2.4 Long-tail Knowledge Transfer

Many previous works [Zellers and Yatskar, 2018; Chen et al., 2019] have observed that the distribution of relations is significantly unbalanced and long-tail, that very few relations (the head) have orders of magnitude more data than the majority of the relations (the tail). Intuitively, the head relations can be accurately classified while the less frequent relations are much more challenging. Therefore, how to transfer knowledge learned in the head relations to the tail is a key point in our model.

Knowledge Codewords Construction. Inspired by the great success of knowledge transfer in domain adaptive learning [Hsu et al., 2017; Xie et al., 2018], our model adopts semantic codewords as the knowledge representation for each relation class. Our model first learns |R| codewords denoted as D = {dr}|R| r=1 , where |R| is the number of unique relation types. The codewords should possess two properties: discriminative and semantic. To this end, we add two constraints to learn D: a near-zero margin for intra-relation groups and a large margin for inter-relation groups, as follows:

(5) where M is a constant margin for inter-relation groups; Y = 1 if the relation of f t ij is r, otherwise Y = 0; dr is the learnable codewords; and dis(, ) is a metric function to calculate two features' distance, for which we choose L1 metric. Intuitively, Ld forces the same relation group to cluster together while pushes the inter-relation groups away.

Knowledge Transfer. Relations at the tail of the distribution are hard to be trained, as there is an insufficient amount of samples for training. Simply put, the challenge lays on the fact that feature f t ij learned of the tail relationships is not representative. Therefore, transferring knowledge learned from the head of the distribution to the tail is critical for the recognition of those data-starved relationships.

Inspired by the hallucination strategy used in metalearning [Zhang et al., 2019c; Zhang et al., 2019a], we propose a knowledge transfer method by hallucinating the learned features. Specifically, we first build a coarse classifier on f t ij , that is,

where p is a probability distribution over relation types R implemented by a softmax classification layer. Then, the hallucinated feature is calculated by:

where dr is the informative knowledge codewords learned by Equation 5. Similarly, we also apply an additive attention to combine the original feature f t ij with the hallucinated ˜f t ij :

where wf is the parameters of a nonlinear layer to calculate a coefficient of two features. Finally, we obtain the new relation features as the below:˜fij = f t ij + atij˜f t ij (9)

Long-tail Features Calibration. Ideally, ˜fij should be close to f t ij so that the fused feature does not change f t ij too much, because the head relations already have sufficient samples to be trained and the codewords of head relations should be close to f t ij . On the contrary, for the tail relations, the modification can be significant and arbitrary, consequentially leading to the confusion with the head relations.

Many previous works have demonstrated that the discrimination of the head and tail class representation plays an essential role in imbalanced data learning [Zhu et al., 2014]. To avoid this confusion, we calibrate fij to different scales for different frequency relationships by:

where p is the probability vector from Equation 6. Generally, as for the data-rich relations, max(p) should be a large value, possibly close to 1 whereas much smaller for the rare relations, because the frequent relations are trained by more data and their predicate prediction should be more confident. Thus, max(p) can be seen as a discriminative calibrating metric to separate the head and tail features. α is a constant scalar to resize them. Finally, we deploy a relation classifier on fij, on which a cross-entropy loss Lrel is imposed.

2.5 Learning

The overall loss function is as follows:

where Ls is a multi-label classification loss defined in Equation 2 to learn the scene feature, Ldet is the object detection loss of Faster-RCNN, Ld is the knowledge codewords learning loss defined in Equation 5, Lp is the coarse relation classification loss in Equation 6, and Lrel is the final relation classification loss defined above. ϵ = 0.01 serves to balance the term of the codewords loss. Note that the reason why ϵ is set to a small number is that Ld is a distance metric usually much greater than the other terms, but not that Ld is not important. All parameters in our model are differentiable, so the model is trained in an end-to-end fashion.

3 Experiment

We evaluate our method on three standard scene graph generation tasks: predicate classification (PredCls), scene graph classification (SGCls) and scene graph detection (SGDet). In PredCls, given ground-truth bounding boxes and objects, the task is to predict scene graph triples on these objects. In SGCls, given the ground-truth bounding boxes only, the task is to predict object labels and triples. In SGDet, the task is to localize bounding boxes, predict object labels and triples.

Specifically, the experiments are conducted to answer the following research questions:

• RQ1: How does our method compare with state-of-the-art scene graph generation methods?
• RQ2: How does each part of our model contribute to the relation recognition performance on three tasks?

RQ3: How well does our method perform in qualitative analysis?

3.1 Dataset and Implementation Details

Dataset. We conduct our method on the challenging and most widely used benchmark, Visual Genome (VG) [Krishna et al., 2017], which consists of 108,077 images with average annotations of 38 objects and 22 relations per image. The experimental settings follow the previous works [Zellers and Yatskar, 2018; Chen et al., 2019], where we use 150 object classes for C and 50 relations for R. Similar to Neural Motifs [Zellers and Yatskar, 2018], we utilize the statistical bias information as the extra knowledge to boost the relation recognition performance and we also report the results without this information.

Implementation Details. α is set as 10, ϵ at 0.01, and learning rate starts from 0.001 and decays with the training processing. Codewords D = {dr}|R| r=1 is initialized by pre-calculated clusters implemented by K-means. We apply the Faster R-CNN [Girshick, 2015] based on VGG-16 as the backbone object detection and localization network. The number of object proposals is 256, each of which is processed by RoIAalign [He et al., 2017] pooling to extract object and union region features. We adopt the Top-K Recall (denoted as R@K) following previous work [Zellers and Yatskar, 2018; Chen et al., 2019] as the evaluation metric and report R@20, R@50 and R@100 on the three tasks.

3.2 Comparison with State-of-the-art Methods (RQ1)

We compare our method to the following recent state-of-theart methods: KRE [Chen et al., 2019], GSA [Qi et al., 2019], Mem [Wang et al., 2019], IMP [Xu et al., 2017], and Neural Motifs [Zellers and Yatskar, 2018]. In addition, we also compare to Graph-RCNN [Yang et al., 2018], since it also develops an attention mechanism to learn contextual information. As the source code of LinkNet [Woo et al., 2018] is unavailable and we are unable to reproduce its results, we do not compare with LinkNet. It is worth noting that Neural Motifs and KRE use the relation bias as the additional prior to guide the recognition and we report their results with or without the bias. Also, we report two sets of results under different conditions, constraint and unconstraint, to calculate R@K, following IMP [Xu et al., 2017]

Table 1 shows the results on the three tasks. As some methods did not report their results on the R@20, the mean result is calculated according to their reported results. From Table 1, we can make the following observations.

(1) Our method is superior to other methods in the majority of cases even irrespective of the use of the bias information. Specifically, in terms of mean recall in the constraint setting, our method surpasses KRE, the best method among the baselines, by about 1.1 percentage points when the statistical bias information is used. A larger improvement of about 2 percentage points is achieved when that information is not used. Also, the similar comparison pattern can be found in Neural Motifs. Compared with KRE and Neural Motifs, the performance difference between with and without statistical bias information is less in our methods (2.8 percentage points vs 3.7 and 3.4), indicating that our method does not heavily rely on this bias, and that our model can essentially learn this bias from the raw data.

(2) GSM shows a great advantage in SGCls task but performs poorly in the task of the predicate classification. As GSM does not report the results on the scene graph detection task, we also do not report their mean recall.

(3) Similarly, our method achieves the best performance in the unconstraint setting. Due to the space limitation, we do not report the result when the bias information is discarded. However similar observations can be made.

3.3 Effectiveness of EachModule (RQ2)

We split our model into three modules: scene-object interaction (SO), knowledge transfer (KT) and feature calibration (FC). The baseline model (BL) denotes the simple model that only uses the feature generated by Faster-RCNN to recognize relations. The ablation study results are shown in Table 2, where we test the performance on the three tasks by adding each module one at a time. For a fair comparison, all ablated models are trained by the same number of epochs, set as 40.

We can observe that under both experimental conditions, constraint and unconstraint, the performance of the baseline is the worst. The addition of the scene-object interaction module SO improves the average performance by 2-3 percentage points, which confirms the crucial role the global contextual information plays in relation recognition. When we deploy the knowledge transfer module KT, a further 1 percentage point of improvements is gained. Finally, though the achievements from adding the feature calibration module FC is not as significant as the other two modules, it still obtains a noticeable lift of about 0.5 percentage point.

Our knowledge transfer module (KT) is specifically designed to solve the problem of data imbalance. To evaluate its effectiveness, Table 3 shows the predicate classification (PredCls) results of bottom-10 tail relations whose frequencies are substantially lower than the average frequency of all relations. The columns "w/o KT" (respectively "w KT") denote the model without (respectively with) knowledge transfer and feature calibration. The superiority of the knowledge transfer module can be clearly observed. It is worth noting that since the relation flying in has only five samples in the entire dataset, all its results are zero. More generally, the knowledge transfer module on average improves performance for each relation by 2-3 percentage points.

Briefly, we can draw two conclusions from the ablation study. (1) The three modules all positively contribute to the relation recognition performance, and their combination achieves the best results. (2) The scene-object interaction module is the most effective of the three, as it offers more contextual clues and knowledge, and the other two modules rely on the knowledge learned from the scene context.

3.4 Qualitative Results (RQ3)

Figure 2 visualizes some scene graph generation results of two models: the baseline model and the full model. We can observe that though the baseline model is able to capture many relations, it does get confused on some cases. Taking the second image as an example, the baseline model predicts that the keyboard is under the screen but in fact is under the table. The possible reason is that the baseline model only considers the visual and spatial feature of the screen and keyboard objects but does not consider the global scene feature.

4 Conclusion

In this work, we investigate the long-tail problem existing in scene graph generation. To address this issue, we propose an end-to-end framework consisting of three modules: sceneobject interaction, knowledge transfer and feature calibration, each of which has its specific function. The extensive experimental results show that our method significantly outperforms other state-of-the-art methods on all standard evaluation metrics. We observe that there still exists a large performance gap between the scene graph detection task and the predicate classification task. In future, we will focus on object label refinement, which is a promising way to improve scene graph generation performance.