Improving Information Extraction from Images with Learned Semantic Models

1 Introduction

The extraction of semantic information from unstructured data is a key challenge in artificial intelligence. Object detection in images has improved enormously within the last years, due to novel deep learning methods. However, the semantic expressiveness of image descriptions that consist simply of a set of objects is rather limited. Semantics is captured in more meaningful ways by the relationships between objects. In particular, visual relationships can be represented by triples, where two entities appearing in an image are linked through a relation (e.g. man-riding-elephant, man-wearing-hat). Due to the cubic combinatorial complexity of possible triples, it is likely that not all relevant triples do appear in the training data, which makes training a predictive model difficult. In this paper, we review our previously proposed approach published in [Baier et al., 2017], which uses a Bayesian fusion approach for combining visual object detection methods with a separately trained probabilistic semantic prior. Incorporating a probabilistic semantic prior especially helps in cases where the prediction of the classifier is not very certain, and for the generalization to unobserved triples in the training set. Further, we propose a new conditional multi-way model which is inspired by statistical link prediction methods. This model does not include an explicitly trained prior of the semantic triples, and is trained in a purely feedforward manner. The prior is implicitly learned in the latent representations of the entities. We conduct experiments on the Stanford Visual Relationship dataset recently published by [Lu et al., 2016]. For the Bayesian fusion model we evaluate different model variants on the task of predicting semantic triples and the corresponding bounding boxes of the subject and object entities detected in the image. Our experiments show that including the semantic model improves on the state-of-the-art result in the task of mapping images to their associated triples. The experiments further show that the conditional multi-way model proposed in this paper, especially in the task of predicting unobserved triples, achieves performance that is comparable to the Bayesian fusion model.

∗This paper is an abridged version, with some additional content, of a paper titled "Improving Visual Relationship Detection using Semantic Modeling of Scene Descriptions" published at the ISWC2017.

2 Background and RelatedWork

In this section, we discuss the most important background and related work for visual relationship detection.

2.1 Visual Relationship Detection

Visual relationship detection is concerned with the problem of detecting objects and their relationships in images. Extracting triples, i.e. visual relationships, from raw images is a challenging task, which has been a focus in the Semantic Web community for some time, e.g. [Bloehdorn et al., 2005; Serafini et al., 2017] and recently also gained substantial attention in mainstream computer vision [Sadeghi and Farhadi, 2011; Lu et al., 2016; Zhang et al., 2017]. In the approach from Lu et al. [Lu et al., 2016], a Region Convolutional Neural Network (RCNN) for detecting and classifying objects in the image was used. Given a pair of objects, their relationship is predicted using another Convolutional Neural Network (CNN). The prediction is combined with a prior based on the word embeddings of the objects and the relationships. Baier et al. [Baier et al., 2017] have used the same RCNN, but have replaced the word embedding model with a learned probabilistic semantic model. This model will be reviewed in Section 3.1. More recently, Zhang et al. [Zhang et al., 2017] have applied the translational embedding model [Bordes et al., 2013] for the task of visual relationship detection.

2.2 Semantic Tensor Models

A number of statistical models have been proposed for modeling graph-structured knowledge bases, often referred to as knowledge graphs. A knowledge graph G consists of a set of triples G = {(s, p, o)i}Ni=1 ⊆ E×R×E. The entities s, o ∈ E are referred to as subject and object of the triple, and the relation between the entities p ∈ R is referred to as predicate of the triple.

Link prediction methods can be described by a function θ : E × R × E → R, which maps a triple (s, p, o) to a real-valued score, which is a measure for the likelihood of the triple being true. Most recently developed link prediction models learn a latent representation, also called embedding, for the entities and the relations. In the following we describe the link prediction methods, which are used in this paper.

DistMult: DistMult [Yang et al., 2014] scores a triple by building the tri-linear dot product of the embeddings, such that

where as, rp, ao ∈ Rd are latent vector representations for subject, predicate, and object, and 〈·, ·, ·〉 denotes the dot product of multiple vectors. The dimensionality d of the embeddings, also called rank, is a hyperparameter of the model.

ComplEx: ComplEx [Trouillon et al., 2016] extends DistMult to complex-valued vectors for the embeddings of both, relations and entities. The score function is

where as, rp, ao ∈ Cd are complex-valued vector representations of subject, predicate and object. Re(·) denotes the real part of a complex number and · denotes the complex conjugate.

Multiway NN: The multiway neural network [Dong et al., 2014; Nickel et al., 2016] concatenates all embeddings and feeds them to a neural network of the form

θ(s, p, o) = βT tanh (W [as, rp, ao] + b1) + b2 (3) where [·, ·, ·] denotes the concatenation of the embeddings as, rp, ao ∈ Rd . The prediction is derived using a Mulilayer Perceptron with the weight matrix W ∈ R3d×z , the weight vector β ∈ Rz , and the biases b1 ∈ Rz , b2 ∈ R.

RESCAL: The tensor decomposition RESCAL [Nickel et al., 2011] learns vector embeddings for entities and matrix embeddings for relations. The score function is

with · denoting the dot product, as, ao ∈ Rd and Rp ∈ Rd×d .

2.3 Image Classification and Object Detection

The Region Convolutional Neural Network (RCNN) [Girshick et al., 2014] is a standard approach for detecting objects in images. It uses a selective search algorithm for getting candidate regions in an image. The RCNN algorithm then rejects most of the regions based on a classification score. As a result, a small set of region proposals is derived. Convolutional Neural Networks (CNNs) have become the standard approach for classifying images. CNNs apply convolutional filters in a hierarchical manner to an image. In this work, we use a specific CNN network architecture, which is called VGG-16 [Simonyan and Zisserman, 2014]. It consists of 16 convolutional layers and two dense output layers. The output of the second last layer of the network can be considered as a latent representation of the input image.

3 Modelling Visual Relationships

In this section, we present two different models for the task of visual relationship detection, both combining semantic tensor models and object detection in different ways. Figure 1 shows the processing pipeline for both models. Both assume that object candidate boxes are provided by an RCNN model. The goal is to predict the most likely triple (s, p, o) for each pair of subject/object candidate boxes (is, io). We define the union of the regions is and io as ip. The extracted triples then consist of two visual concepts s, o and their relationship p. This is different to knowledge graphs where the relations are typically not modelled on the concept level, but on the instance level. Nevertheless, the link prediction methods described in Section 2.2 can be applied to visual concepts, as well.

3.1 Bayesian FusionModel

In this model, we derive predictions from two different CNNs, one modelling p(s|is) and p(o|io), and the other one modelling p(p|ip). We combine these visual models with a tensor model serving as a semantic prior p(s, p, o) in a Bayesian way. We assume the joint distribution of all involved variables to factor as

with ˜ p denoting unnormalized probabilities. We can divide the joint probability of Equation 5 into two parts. The first part is ˜ p(s, p, o), which models semantic triples. The second part is ˜ p(is|s) · ˜ p(ip|p) · ˜ p(io|o), which models the visual part given the semantics. The semantic prior is modelled as

where θ is a semantic tensor model as described in Section 2.2 and exp is the activation function for Poisson regression. An advantage is that the semantic model can be trained separately from the visual model using only the absolute frequencies of triples in the training data. As we are predicting count data, we train the model using a Poisson cost function. 1 The semantic model and the visual models are combined by applying Bayes rule to Equation 5, such that

The additional terms of the denominator ˜ p(s), ˜ p(p), ˜ p(o) are derived through the marginalization of ˜ p(s, p, o) and a Laplacian smoothing. For each pair of bounding boxes, we pick the triple with the highest probability.

3.2 Conditional MultiwayModel

In this model, we derive for each pair of bounding boxes the subject s and the object o by applying a VGG classifier to the regions is and io. In a second step we then build a conditional tensor model for p(p|s, o, ip). We parameterize the model as

with as, ao ∈ Rd being latent vector representations for the visual concepts, rip ∈ Rd being a latent representation vector for the image patch ip, and [·, ·, ·] denoting the concatenation operation. For as and ao the representations are optimized in the learning procedure and stored in a lookup table. For deriving a representation of the predicate region ip, we model rip = Mhip , where hip is the activation of the second last layer of a VGG network with the image region ip as input. The matrix M maps the latent representation of the VGG network to a vector with the rank of the multi-way model. The probabilities for predicate p are derived by applying a Multilayer Perceptron with the additional parameters W1 ∈ R3d×z ,W2 ∈ Rz×|R| , b1 ∈ Rz , b2 ∈ R|R| .

1 Another sampling model would lead to a multinomial model, which would only result in a different normalization of the distribution.

To derive a single prediction for each pair of bounding boxs, we pick the subject ˆ s = argmaxp(s|is), object ˆ o = argmaxp(o|io), and predicate ˆ p = argmaxp(p|ip, ˆ s, ˆ o) with the highest probabilities. The confidence score for the triple (ˆ s, ˆ p, ˆ o), given an input region (is, ip, io) is calculated as

4 Experiments

We evaluate our proposed method on the recently published Stanford Visual Relationship dataset.

Setting We compare the twomodels presented in this paper and their variations, with the results from [Lu et al., 2016]. The settings are the same as in [Lu et al., 2016] and [Baier et al., 2017]. In all settings a single triple is derived for each pair of bounding boxes. In the first setting, which in [Lu et al., 2016] is referred to as Phrase Detection, a triple with its corresponding bounding boxes is considered correctly detected, if the triple is similar to the ground truth, and if the union of the bounding boxes has at least 50 percent overlap with the union of the ground truth bounding boxes. In Relationship Detection, both the bounding box of the subject and the bounding box of the object need at least 50 percent of overlap with their ground truth. In Predicate Detection, it is assumed that subject and object are given, and only the correct predicate linking both needs to be predicted. In Triple Detection, a triple is considered correct if it corresponds to the ground truth, independent of the predicted bounding boxes.

Results Table 1 shows the results for visual relationship detection. The first row shows the results, when only the visual part of the model is applied. This model performs poorly in all four settings. The full model in the second row adds the language prior to it, which drastically improves the results. The Conditional Multiway Model outperforms the language prior model in some settings and achieves very similar results in the others. In the last four rows we report the results of the Bayesian fusion model, with different link prediction methods. We see that the model performs consistently better than the state-of-the-art method proposed by [Lu et al., 2016]. Only DistMult is slightly worse, which might be due to the fact that it assumes symmetric scores when subject and object are exchanged.

Table 2 shows the results, when only evaluating triples which have not been observed in the training data. This task is much more difficult, as it requires the models to generalize to these triples. Also in this experiment, including the semantic model significantly improves the prediction. For the first three settings, the best performing method, which is the Multiway Neural Network, almost retrieves twice as many correct triples, as the state-of-the-art model of [Lu et al., 2016]. These results clearly show that our model is able to infer also new likely triples, which have not been observed in the training data. The Conditional Multiway Model achieves a performance close to the Bayesian fusion models, although it does not include a separately learned prior for the semantic triples.

5 Relationships to Perception andMemory

At a higher cognitive level, information which is ceaselessly acquired by the visual system of the brain, needs to be interpreted correctly. Goethe's proverb might be quite fitting for our proposed approach: "You only see what you know" since one can only perceive things for which we have an internal brain representation. This internal representation is used by the perceptual system and by the main declarative memory systems, i.e., the episodic memory (about events we remember) and the semantic memory system (about facts we know) [Tulving, 2002]. Perception even needs more: it needs to generalize to novel entities on the class level, since a subject constantly encounters novel entities: The perceptual system needs to be able to generalize to new scenes and needs to generalize on the associations of new perceptual components. According to the complementary learning system [McClelland et al., 1995], the memory systems are the basis for the brain to learn to generalize to new situations, a consolidation process which might happen largely during sleep and might involve the neocortex. Our work focuses on the generalization after training the perceptual system for scene comprehension. Our approaches can be related to some of the main current hypothesis about perception and memory. Many groups favor the Bayesian brain hypothesis, which assumes that the brain uses inherited and learned prior hypothesis to understand and reason about the world ([Knill and Pouget, 2004] and [Griffiths et al., 2008] are two examples). Our Bayesian fusion model fits precisely into this category. Its tensor model provides a prior model which is quite powerful in supporting the perceptual pipeline. It is a very rich prior compared to simple smoothness priors used in other approaches. In contrast, in the conditional multiway model the prior is represented in the latent representations of entities and weights in the feedforward neural network. The conceptual memory is formed implicitly in the end-to-end training of the model. Cognitive models which are pursuing this approach have been proposed and studied in [Tresp et al., 2015; 2017; Tresp and Ma, 2016]. This approach is more in favor of a theory which assumes that the brain, at least conceptionally, is trained end-to-end with few clearly interpretable functional modules.

6 Conclusion

We presented two approaches for visual relationship detection, which both include statistical semantic models. The first approach, which originally has been published in [Baier et al., 2017], combines standard computer vision methods with latent variable models for link prediction. We proposed a probabilistic framework, in form of a Bayesian fusion model, for integrating both the semantic prior and the computer vision algorithms into a joint model. The second approach uses a conditional multi-way model, which is inspired by link prediction methods. For the prediction of triples, which have not been observed in the training data, the performance of the second approach is on par with the first approach, as its structure helps to generalize to unobserved triples, without including a separately trained prior for the semantic triples. Both approaches form statistical models on the class level, and can thus generalize to new images. This is in contrast to typical knowledge graph models, where nodes correspond to specific instances. In cognitive terms, the Bayesian fusion model can more directly be related to the Bayesian brain hypothesis, as being pursued by many research teams, whereas the conditional multiway model is more closely related to the tensor memory hypothesis [Tresp and Ma, 2016].