A Graph-based Interactive Reasoning for Human-Object Interaction Detection

1 Introduction

The task of Human-Object Interaction (HOI) detection aims to localize and classify triplets of 〈 human, verb, object 〉 from a still image. Beyond detecting and comprehending instances, e.g., object detection [Ren et al., 2017; He et al., 2017], segmentation [Yang et al., 2018] and human pose estimation [Fang et al., 2017], detecting HOIs requires a deeper understanding of visual semantics to depict complex relationships between human-object pairs. HOI detection is related to action recognition [Sharma et al., 2015] but presents different challenges, e.g., an individual can simultaneously take multiple interactions with surrounding objects. Besides, associating ever-changing roles with various objects leads to finergrained and diverse samples of interactions.

∗Contact Author

Most existing HOI detection approaches infer HOIs by directly employing the appearance features of a person and an object extracted from Convolutional Neural Networks (CNNs) [He et al., 2016], which may neglect the detailed and high-level interactive semantics implied between the related targets. To remedy the limitation above, a number of algorithms exploit additional contextual cues from the image such as human intention [Xu et al., 2019] and attention boxes [Kolesnikov et al., 2018]. More recently, several works have taken advantage of additional annotations, e.g., human pose [Xu et al., 2019; Li et al., 2019] and body parts [Zhou and Chi, 2019; Wan et al., 2019]. Although incorporating contextual cues and annotations generally benefits feature expression, it brings several drawbacks.

Firstly, stack of convolutions and contextual cues are deficient in modelling HOIs since recognizing HOIs requires reasoning beyond feature extraction. However, dominant methods are limited by treating each visual component separately without considering crucial semantic dependencies among related targets (i.e., scene, human and object). Secondly, employing additional human pose and body parts in HOI detection algorithms brings a large computational burden.

To address these issues, we propose a novel graph-based model called Interactive Graph (abbr. in-Graph) to infer HOIs by reasoning and integrating strong interactive semantics among scene, human and object. As illustrated in Figure 1, our model goes beyond current approaches lacking the capability to reason interactive semantics. In particular, inGraph model contains three core procedures, i.e., a project function, a message passing process and an update function. Here, the project function generates a unified space to make two related targets syncretic and interoperable. The message passing process further integrates semantic information by propagating messages among nodes. Finally, the update function transforms the reasoned nodes to convolution space, providing enhanced representation for HOI-specific modeling.

Based on the proposed in-Graph model, we then offer a general framework referred to as in-GraphNet to implicitly parse scene-wide interactive semantics and instance-wide interactive semantics for inferring HOIs rather than treat each visual target separately. Concretely, the proposed inGraphNet is a multi-stream network assembling two-level inGraph models (i.e., scene-wide in-Graph and instance-wide in-Graph). The final HOI predictions are made by combining all exploited semantics. Moreover, our framework is free from additional annotations such as human pose.

We perform extensive experiments on two public benchmarks, i.e., V-COCO [Yatskar et al., 2016] dataset and HICODET [Chao et al., 2018] dataset. Our method provides obvious performance gain compared with the baseline and outperforms the state-of-the-art methods (both pose-free and posebased methods) by a sizable margin. We also provide detailed ablation studies of our method to facilitate the future research.

2 RelatedWork

2.1 Contextual Cues in HOI Detection

The early human activity recognition [Idrees et al., 2017] task is confined to scenes containing single human-centric action and ignores spatial localization of the person and related object. Therefore, Gupta [Yatskar et al., 2016] introduced visual semantic role labeling to learn interactions between human and object. HO-RCNN [Chao et al., 2018] introduced a three-branch architecture with one branch each for a human candidate, an object candidate, and an interaction pattern encoding the spatial position of the human and object. Recently, several works have taken advantage of contextual cues and detailed annotations to improve HOI detection. Auxiliary boxes [Gkioxari et al., 2015] were employed to encode context regions from the human bounding boxes. InteractNet [Gkioxari et al., 2018] extended the object detector Faster RCNN [Ren et al., 2017] with an additional branch and estimated an action-specific density map to identify the locations of interacted objects. iHOI [Xu et al., 2019] utilized human gaze to guide the attended contextual regions in a weaklysupervised setting for learning HOIs.

Very recently, human pose [Fang et al., 2017] has been widely adopted as an additional cue to tackle HOI detection. Pair-wise human parts correlation [Fang et al., 2018] was exploited to learn HOIs. TIN [Li et al., 2019] combined human pose and spatial configuration to encode pose configuration maps. PMFNet [Wan et al., 2019] developed a multibranch network to learn a pose-augmented relation representation to incorporate interaction context, object features and detailed human parts. RPNN [Zhou and Chi, 2019] introduced an object-bodypart graph and a human-bodypart graph to capture relationships between body parts and surrounding instances (i.e., human and object).

Although extracting contextual evidence benefits feature expression, it is not favored since additional annotations and computation are indispensable. For example, pose-based approaches are inseparable from pre-trained human pose estimators like Mask R-CNN [He et al., 2017] and AlphaPose [Fang et al., 2017], which bring large workload and computational burden.

2.2 Semantic Reasoning

Attention mechanism [Sharma et al., 2015] in action recognition helps to suppress irrelevant global information and highlight informative regions. Inspired by action recognition methods, iCAN [Gao et al., 2018] exploited an instancecentric attention mechanism to enhance the information from regions and facilitate HOI classification. Furthermore, Contextual Attention [Wang et al., 2019] proposed a deep contextual attention framework for HOI detection, in which contextaware appearance features for human and object were captured. PMFNet [Wan et al., 2019] focused on pose-aware attention for HOI detection by employing human parts. Overall, methods employing attention mechanisms learn informative regions but treat each visual target (i.e., scene, human and object) separately, which are still insufficient to exploit interactive semantics for inferring HOIs.

Graph Parsing Neural Network (GPNN) [Qi et al., 2018] introduced a learnable graph-based structure, in which HOIs were represented with a graph structure and parsed in an endto-end manner. The above structure was a generalization of Message Passing Neural Network [Gilmer et al., 2017] using a message function and a Gated Recurrent Unit (GRU) to iteratively update states. GPNN was innovative but showed some limitations. Firstly, it reasoned interactive features at the coarse instance-level (i.e., each instance was encoded as an infrangible node), which suffered from handling complex interactions. In addition, it required iteratively message passing and updating. Thirdly, it excluded semantic information from the scene in inferring HOIs. Lately, RPNN [Zhou and Chi, 2019] introduced a complicated structure with two graph-based modules incorporated together to infer HOIs, but in which fine-grained human pose was required as prior information.

In this paper, we aim to develop a novel graph-based model to provide interactive semantic reasoning between visual targets. Instead of coarse and iterative message passing between instances, our model captures pixel-level interactive semantics between targets all at once. Furthermore, our model is free from costly annotations like human pose.

3 ProposedMethod

3.1 Overview of in-Graph

The detailed design of proposed in-Graph model is provided in Figure 2. Scene, human and object are three semantic elements been considered as three visual targets in our model, referred to as Tscene, Thuman, and Tobject, respectively. The proposed in-Graph takes two targets once to conduct pixellevel interactive reasoning. Each of the targets takes convolutional feature X ∈ ℜH∗W∗D according to the corresponding boxes (i.e., the whole image, human candidate boxes and object candidate boxes) as input. Here H*W denotes locations and D denotes feature dimension.

We first propose a project function to map two feature tensors into a graph-based semantic space named interactive space, where a fully-connected graph structure can be built. Based on the graph structure, message passing process is then adopted as modeling the interaction among all nodes by propagating and aggregating interactive semantics. Finally, the update function provides a reversed projection over interactive space and output feature Y, enabling us to utilize the reasoned semantics in convolution space. We then describe its architecture in details and explain how we apply it into HOI detection task.

3.2 Project Function

Project function aims to provide an pattern P(∆) to fuse two targets together, after which message passing process can be efficiently computed. The calculation process of P(∆) can be divided into three parts: a feature conversion denoting as ϕ(X1,X2;Wϕ), a weights inference denoting as θ(X1, X2;Wθ) and a linear combination, where Wϕ and Wθ are learnable parameters. Finally, the function outputs a matrix V = P(X1, X2) ∈ ℜ2N∗2C , where X1 and X2 are input feature tensors, 2N denotes the number of nodes in interactive space and 2C refers to dimension.

Feature conversion. Given feature tensors of two targets X1,X2 ∈ ℜH∗W∗D , we first employ 1 ∗ 1 convolutions to reduce the dimensions of X1 and X2 to C, thus the computation of the block can be valid decreased. The obtained tensors are then reshaped from H ∗W ∗ C to planar L ∗ C, obtaining Xr 1 , Xr 2 ∈ ℜL∗C ,where its two-dimensional location pixels of H ∗ W are converted to one-dimensional vector L. After that, a concatenation operation is adopted to integrate Xr 1 and Xr 2 by dimensional C, obtaining Xr = [Xr 1 , Xr 2 ] ∈ ℜL∗2C .

Weights inference. Here, we infer learnable projection weights B so that semantic information from original features can be weighted aggregated. Instead of designing complicated calculations, we simply use convolution layers to generate the dynamically weights. In this step, X1 and X2 are feed into 1 ∗ 1 convolutions to obtain weight tensors with channel of N. Obtained feature tensors are then reshaped as planar B1,B2 ∈ ℜN∗L. Finally, integrated projection weights B = [B1,B2] = [b1, b2, . . . bi, . . . , b2N] ∈ ℜ2N∗L are obtained by a concatenation operation.

Linear combination. Since the project function involves two targets, linear combination is a necessary step to aggregate the semantic information and transform targets to the unified interactive space. In particular, node vi ∈ V in interactive space is calculated as follow. Here xrj ∈ ℜ1∗2C , vi ∈ ℜ1∗2C .

The proposed project function is simple and fast since all parameters are end-to-end learnable and come from 1*1 convolutions. Such a function achieves semantic fusion between two targets and maps them into an interactive space effectively.

3.3 Message Passing and Update Function

Message Passing

After projecting targets from convolution space to interactive space, we have a structured representation of a fullyconnected graph G = (V,E,A), where each node contains a feature tensor as its state and all nodes are considered as fully-connected with each other. Based on the graph structure, message passing process is adopted to broadcast and integrate semantic information from all nodes over the graph.

GPNN [Qi et al., 2018] applies an iterative process with GRU to enable nodes to communicate with each other, whereas it needs to run several times iteratively towards convergence. We reason interactive semantics over the graph structure by adopting a single-layer convolution to efficiently build communication among nodes. In our model, the message passing functionsM(∆) is computed by:

Here A denotes the adjacency matrix among nodes learned by gradient decent during training, reflecting the weights for edge E. ω(∆) denotes the state update of nodes. In our implementation, the operation of Conv1D(V ) is a channel-wise 1D convolution layer that performs Laplacian smoothing [Li et al., 2018] and propagates semantic information among all connected nodes. After information diffusion, the ⊕ implements addition point to point which updates the hidden node states according to the incoming messages.

Update Function

To apply above reasoning results into convolutional network, an update function U(∆) provides a reverse projection for reasoned nodes from interactive space to convolution space, which output Y as a new feature tensor. Given the reasoned nodes V ′ ∈ ℜ2N∗2C , update function first adopts a linear combination as follows:yi = ∑

Where the projection weights B is transposed and reused, here v′j ∈ ℜ1∗2C , yi ∈ ℜ1∗2C .

After the linear combination, we reshape the obtained tensor from planar L ∗ 2C to three-dimensional H ∗ W ∗ 2C. Finally, a 1 ∗ 1 convolution is attached to expand the feature dimensions from C to D to match the inputs. In this way, updated features in convolution space can play its due role in the following schedule.

3.4 in-GraphNet

Assembling in-Graph Model

Our in-Graph model improves the ability of modelling HOIs by employing interactive semantic reasoning beyond stack of convolutions. It is noted that the human visual system is able to progressively capture interactive semantics from the scene and related instances to recognize a HOI. Taking the HOI triplet 〈 human, surf, surfboard 〉 as an example, the scenewide interactive semantics connected with the scene (e.g., sea) and instances (e.g., human, surfboard) can be captured as prior knowledge and instance-wide interactive semantics between the person and surfboard are learned to further recognize the verb (i.e., surf) and disambiguate other candidates (e.g., carry). Inspired by this human perception, we assign in-Graphs in two levels to build in-GraphNet, which are scene-wide level and instance-wide level. The scene-wide inGraphs contain a human-scene in-Graph and an object-scene in-Graph, the instance-wide in-Graph refers to human-object in-Graph.

An overview of proposed in-GraphNet is shown in Figure 3. Since in-Graph models are light-weight, it can be easily incorporated into existing CNN architectures. ResNet-50

[He et al., 2016] is employed as the backbone network in our implementation. We denote the candidate bounding boxes of the human and object by bh and bo, respectively. We also express the box of the whole scene as bs. After the process of shared ResNet-50 C1-C4 and RoI pooling according to respective candidate boxes, ResNet-50 C5 generates input features for targets Tscene, Thuman, and Tobject, teamed as fs, fh and fo. These three targets are then assigned into three in-Graph models in pairs. While input features for in-Graph model have dimension D = 2048, we set reduced dimension C = 1024 , number of nodes N = 512.

Three Branches

The in-GraphNet contains a human-centric branch, an objectcentric branch and a spatial configuration branch, where fh, fo and the spatial configuration [Gao et al., 2018] based on two-channel interaction pattern [Chao et al., 2018] are adopted as basic feature representations, respectively. In scene-wide interactive reasoning, semantic features obtained from human-scene in-Graph and object-scene in-Graph are concatenated to fh and fo respectively to enrich presentations in human-centric and object-centric branches. In instance-wide interactive reasoning, semantic feature output from human-object in-Graph is concatenated into the objectcentric branch only, because appearance of an object is usually constant in different interactions and provides minor effect in human-centric representation. Finally, the enhanced features from three branches are fed into fully connected layers and perform classification operations. In this way, the entire framework is implemented to be fully differentiable and end-to-end trainable using gradient-based optimization. With the formulation above, rich interactive relations between visual targets can be explicitly utilized to infer HOIs.

Training

Since HOI detection is a multi-label classification problem where more than one HOI label might be assigned to a 〈 human, object 〉 candidate, our model is trained in a supervised fashion using the multi-label binary cross-entropy loss. All three branches are trained jointly, where the overall loss for each interaction category is the sum of three losses from three branches.

For each image, pairwise candidate boxes (Bh ∗ Bo) for each HOI category are assigned binary labels based on the prediction. Similar to general HOI detection frameworks [Gkioxari et al., 2018; Gao et al., 2018], we use a fusion of scores output from each branch to predict a final score for each HOI.

Here Sh, So and Ss denotes scores output from binary sigmoid classifiers in human-centric branch, object-centric branch and spatial configuration branch, respectively. Spre h and Spre o are object detection scores output from object detector for candidate boxes. Sh,o is the final score for each HOI.

4 Experiments and Evaluations

4.1 Experimental Setup

Datasets and evaluation metrics. We evaluate our model and compare it with the state-of-the-arts on two large-scale benchmarks, including V-COCO [Yatskar et al., 2016] and HICO-DET [Chao et al., 2018] datasets. V-COCO [Yatskar et al., 2016] includes 10,346 images, which is a subset of MS COCO dataset. It contains 16,199 human instances in total and provides 26 common HOI annotations. HICO-DET [Chao et al., 2018] contains about 48k images and 600 HOI categories over 80 object categories, which provides more than 150K annotated 〈 human, object 〉 pairs. We use role mean average precision (role mAP) [Yatskar et al., 2016] on both benchmarks.

Implementation details. Following the protocol in [Gao et al., 2018], human and object bounding boxes are generated using the ResNet-50 version of Faster R-CNN [Ren et al., 2017]. Human boxes with scores higher than 0.8 and object boxes with scores higher than 0.4 are kept for detecting HOIs. We train our model with Stochastic Gradient Descent (SGD), using a learning rate of 1e-4, a weight decay of 1e-4, and a momentum of 0.9. The strategy of interactiveness knowledge training [Li et al., 2019] is adopted in our training and the model is trained for 300K and 1800K iterations on V-COCO and HICO-DET, respectively. All our experiments are conducted by tensorflow on a GPU of GeForce GTX TITAN X.

4.2 Overall Performance

We compare our method with several state-of-the-arts in this subsection. Methods being compared are classified into two categories, i.e., methods that are free from human pose (top of tables) and methods relying on additional pose estimators (middle of tables). Meanwhile, we strip all modules related to in-Graphs from our proposed framework as the baseline. Comparison results on V-COCO and HICO-DET in terms of mAProle are shown in Table 1 and Table 2, respectively.

Firstly, our in-GraphNet obtains 48.9 mAP on V-COCO and 17.72 mAP on HICO-DET (Defualt full mode) and achieves absolute gains of 4.1 points and 2.3 points compared with the baseline, which are relative improvements of 9.4% and 15%. Besides, although pose-based methods usually perform better than pose-free methods, our method as a pose-free one outperforms all the others with the best performance, validating its efficacy for HOI detection task.

4.3 Ablation Studies

We adopt several ablation studies in this subsection. VCOCO serves as the primary testbeds on which we further analyze the individual effect of components in our method.

√

Comprehending in-Graph. We show two examples in Figure 4 to visualize the effects of three in-Graphs been adopted. The brightness of the pixel indicates how much the feature been noticed. Intuitively, the three in-Graphs learn different interactive semantics from pairwise reasoning. Humanscene in-Graph and Object-scene in-Graph exploit interactive semantics between scene and instances. Human-object in-Graph, on the other hand, mostly focuses on the regions roughly correspond to the on-going action between human and object. In addition, to further dissect in-Graph model, empirical tests have been conducted to study the effect of number of nodes in interactive spaces. As summarized in Table 3, we get the best result when N is set as the value of 512.

Effects of adopting different in-Graph models. As shown in Table 4, while directly concatenating [fh, fs] in humancentric branch and [fo, fs, fh] in object-centric branch, we can see that simply concatenating different targets without interactive reasoning barely improves the result. When we only adopt the scene-wide in-Graphs or the instance-wide in-Graph, the mAP are 48.3 and 47.7 respectively, indicating the respective effects of these two parts. Specifically, we can draw a relatively conclusion from detailed class-wise results that scene-wide in-Graphs are more adept in modelling interactions closely related to the environment, while instance-wide in-Graph performs better in depicting interactions closely related to human pose.

4.4 Quantitive Examples

For visualization, several examples of detection are given in Figure 5. We first compare our results with baseline to demonstrate our improvements. We can see from the first two rows that our method is capable of detecting various HOIs with higher scores. In addition, the third row shows that our method can adapt to complex environments to detect multiple people taking different interactions with diversified objects.

5 Conclusion

In this paper, we propose a graph-based model to address the problem of lack interactive reasoning in existing HOI detection methods. Beyond convolutions, our proposed in-Graph model efficiently reasons interactive semantics among visual targets by three procedures, i.e., a project function, a message passing process and an update function. We further construct an in-GraphNet assembling two-level in-Graph models in a multi-stream framework to parse scene-wide interactive semantics and instance-wide interactive semantics for inferring HOIs. The in-GraphNet is free from costly human pose and end-to-end trainable. Extensive experiments have been conducted to evaluate our method on two public benchmarks, including V-COCO and HICO-DET. Our method outperforms both existing human pose-free and human pose-based methods, validating its efficacy in detecting HOIs.

Acknowledgments

This paper was partially supported by National Engineering Laboratory for Video Technology - Shenzhen Division, and Shenzhen Municipal Development and Reform Commission (Disciplinary Development Program for Data Science and Intelligent Computing). Special acknowledgements are given to AOTO-PKUSZ Joint Lab for its support.