RR-Net: Injecting Interactive Semantics in Human-Object Interaction Detection

1 Introduction

Fine-grained understanding of visual contents is one of the fundamental problems in computer vision. The task of Human-Object Interaction (HOI) detection aims to localize and infer triplets of 〈 human, verb, object 〉 from a still image. Beyond comprehending instances, e.g., object detection [Ren et al., 2017; Yang et al., 2019], human pose estimation [Fang et al., 2017] and action recognition [Sharma et al., 2015], detecting HOIs requires a deeper understanding of visual semantics to depict complex and finer-grained relationships between 〈 human, object 〉 pairs. Therefore, HOI detection is challenging and usually needs to mine high-level and HOI-specific representations in solutions.

∗Contact Author

The conventional two-stage HOI detection methods [Gkioxari et al., 2018; Gao et al., 2018; Xu et al., 2019] first obtain human and object region proposals from a pretrained object detector [Ren et al., 2017] and then generate 〈 human, object 〉 pairs to make interaction classifications. The related techniques of HOI feature learning are all region-based where instance (e.g., human and object) attention features [Gkioxari et al., 2018; Gao et al., 2018; Xu et al., 2019], spatial configuration [Gao et al., 2018; Gupta et al., 2019; Ulutan et al., 2020] and human pose [Wan et al., 2019; Zhou and Chi, 2019] are exploited. Inspired by the point-based object detectors [Law and Deng, 2018; Zhou et al., 2019], as illustrated in Figure 1 (above), latest end-to-end HOI detectors pose HOI detection as a keypoint detection and grouping problem, which effectively improve the speed of HOI detection.

Previous region-based feature enhancements are hardly applicable to end-to-end detectors due to these end-to-end frameworks do not provide proposed instance regions during inference. Besides, although previous feature learning techniques generally benefit HOI representations, they bring several drawbacks: 1) Firstly, dominant methods focus on region-based feature enhancement but lack relation reasoning, however, comprehending HOI-specific interactive semantics from relation dependency among humans, objects and interactions is significant for HOI inference. 2) Secondly, region-based feature learning is complicated, and employing human pose or body-part features brings additional workload. Therefore, studying relation reasoning to integrate region-independent interactive semantics has become the key to improve end-to-end HOI detection.

In this work, as illustrated in Figure 1 (bottom), we propose relation reasoning for HOI detection, which captures and transmits HOI-specific interactive semantics among humans, objects and interactions where no human and object proposals are needed. The contributions are summarized as follows:

• A progressive Relation-aware Frame is first presented to build a coherent structure and parameter sharing pattern for interaction inference.
• Upon the frame, an Interaction Intensifier Module (abbr. IIM) is proposed, where interactive semantics from humans can be exploited and easily passed to objects.
• A Correlation Parsing Module (abbr. CPM) is designed subsequently, in which interactive correlations among visual targets can be exploited and integrated.
• We offer a framework named Relation Reasoning Network (RR-Net), which is the first to carry out regionindependent relation reasoning for HOI detection.

We perform extensive experiments on two public benchmarks (i.e., V-COCO [Yatskar et al., 2016] and HICO-DET [Chao et al., 2018] datasets). Our method with relation reasoning provides obvious performance gain compared with the baseline and outperforms the state-of-the-art methods by a sizable margin. Detailed ablation studies of our framework are also provided to facilitate future research.

2 RelatedWork

2.1 Frameworks in HOI Detection

Since human and object instances are basic elements in detecting HOIs, conventional HOI detectors are mostly based on object detection frameworks. InteractNet [Gkioxari et al., 2018] extends the object detector Faster R-CNN [Ren et al., 2017] with an additional branch for HOI detection. iCAN [Gao et al., 2018] then introduces a three-branch architecture with one branch each for a human candidate, an object candidate, and an interaction pattern, which has become the template framework for most later HOI detectors. TIN [Li et al., 2019] further proposes a multiple network to perform feature extraction, interactiveness discrimination and HOI classification, respectively. All above frameworks are two-stage and region-based, where region proposals obtained from pretrained object detectors are indispensable.

More recently, it has become popular to treat object detection as a keypoint estimation task [Law and Deng, 2018; Zhou et al., 2019] to improve detection efficiency. Following the same idea, IPNet [Wang et al., 2020] and PPDM [Liao et al., 2020] pose HOI detection as a keypoint detection and grouping problem, which propose to build end-toend HOI detectors. Specifically, PPDM [Liao et al., 2020] contains two parallel branches to predict three points (i.e., human, object and interaction points) and two displacements from the interaction point to its corresponding human and object points. Then, human with its corresponding object can be paired using the interaction point and displacements as bridge. However, without relation reasoning, these endto-end HOI detectors lack learning HOI-specific interactive semantics in their methods, thus may be deficient in modelling HOIs (e.g., bring problem of miss-pairing between human and object).

2.2 Feature Learning in HOI Detection

Contextual Cues. Several contextual cues have been explored to improve HOI detection. Given a 〈 human, object 〉 candidate with bounding boxes, the interaction pattern [Chao et al., 2018] or spatial configuration [Gao et al., 2018; Ulutan et al., 2020] is a coarse region layout encoded by a binary image with two channels. Tanmay [Gupta et al., 2019] encodes both absolute and relative position of the human and object boxes. Additionally, human pose obtained from pre-trained pose estimators [Fang et al., 2017] is unitized as a valid cue to support HOI detection. With human pose vector, human body-parts [Fang et al., 2018; Wan et al., 2019] are explored as a set of local regions centered at human keypoints with certain size. Although extracting region-based spatial encodings or body-parts improve detection results, these methods are not favored since additional computation and premade annotations are indispensable.

Appearance Attention and Reasoning. Contextual Attention [Wang et al., 2019] and GID-Net [Yang et al., 2020] propose to capture instance-centric attention features and context-aware appearance features for human and object. PMFNet [Wan et al., 2019] focuses on pose-aware attention by employing attentional human parts. VSGNet [Ulutan et al., 2020] proposes to refine visual features by the spatial configuration of human-object pair. Moreover, GPNN [Qi et al., 2018] and in-GraphNet [Yang and Zou, 2020] introduce learnable graph-based structures, in which HOIs are represented with graphs. RPNN [Zhou and Chi, 2019] introduces an object-body graph and a human-body graph to capture contexts between body-parts and surrounding instances, where fine-grained human body-part regions are required as prior information.

Overall, methods employing appearance attention mechanisms or graphs are enlightening but complicated, e.g., implementing attention mechanisms upon M human regions and N object regions needs M +N standalone calculations. Analogously, constructing graphs with M ∗ N possible 〈 human, object 〉 pairs needs to calculate M ∗ N candidate samples. More importantly, above methods neglect relation reasoning to explore implicit interactive semantics among humans, objects and interactions, whereas comprehending relation is essential to improve HOI detection. In this paper, unlike the previous techniques of HOI feature learning, we bring regionindependent relation reasoning to integrate and transmit HOIspecific interactive semantics for HOI inference.

3 ProposedMethod

3.1 Overview of RR-Net

An overview of our proposed RR-Net is shown in Figure 2. DLA-34 [Yu et al., 2018] is employed as the backbone network in our implementation to extract the global 128∗128∗64 DLA feature from an image. Our framework formulates HOI detection as a point detection and grouping task [Wang et al., 2020; Liao et al., 2020], where predicting HOI triplets consists of three steps: 1) predicting human points (H points), Kclass object points (O points) and N-class interaction points (I points). Here, I point is defined as the midpoint of a corresponding 〈 human, object 〉 pair. 2) predicting two displacements from the I point to the H point and O point respectively, where each displacement contains an abscissa value and an ordinate value. 3) grouping each I point with the H point and O point respectively to generate a set of triplets based on the predicted points and displacements.

We have noticed that most methods focus on HOI feature learning but none of them study the association between HOI cognitive processes or explore interactive semantics in these processes. In order to explore above information, a Relationaware Frame (See Section 3.2) is first proposed to provide a coherent structure and parameter sharing pattern for interaction inference. Upon the frame, an Interaction Intensifier Module (See Section 3.3) and a Correlation Parsing Module (See Section 3.4) are then developed to achieve regionindependent relation reasoning so that interactive semantics and correlations among visual targets can be integrated. Next, we will introduce the proposed framework in detail.

3.2 Relation-aware Frame

As illustrated in Figure 3 (a), to predict H points, O points, I points and two displacements, the baseline build different prediction feature heads independently by the function group G() = [Gho(),Gi(),Gdh(),Gdo()], and head parameters are optimized severally in backpropagation. It should be mentioned that we have merged Gdh() and Gdo() as Gd() in Figure 3 for simplicity. We argue that such a structure neglects the implicit relation among these predictions. It is noted that the human visual system usually detects instances and groups related ones coherently from the scene to recognize a HOI. Based on the above inspiration, we propose a correlated and coherent structure for interaction inference, which is termed as a Relation-aware Frame.

For predicting humans and objects, we first generate H and O head feature fho ∈ ℜH∗W∗(1+K) from global DLA head feature fdla ∈ ℜH∗W∗D by the function Gho(), where H ∗W denotes range, D denotes feature dimension and (1 +K) is the category synthesis of human and K-class objects. The two stages of our Relation-aware Frame are then established to break the independent predictions of baseline, which are: 1) Part A: building I head feature fi ∈ ℜH∗W∗N upon feature fho by the function Gi(); 2) Part B: building two D head features fdh, fdo ∈ ℜH∗W∗2 upon all point head features fp = concat(fho, fi) by the function Gd(). As illustrated in Figure 3 (b), in the fully Relation-aware Frame, output head features of each previous stage are used as input in the next stage. The function group G() are computed by:

where the first 3 ∗ 3 convolution transforms the features from input dimension to hidden dimension, and the second 1 ∗ 1 convolution acts as a fully connected layer to reduce the feature from hidden dimension to specified dimension to match the predictions. The next predictions of H points, O points and I points can be regarded as heatmap estimation processes by sigmoid functions. In this way, instance points prediction loss Lho, interaction point prediction loss Li and two displacements prediction loss Ld = Ldh + Ldo are correlated in backpropagation and parameters are optimized connectedly.

3.3 Interaction Intensifier Module

Human contexts often imply information of taking interactions with different objects. For example, we can clearly distinguish between kickers and riders by their postures and distribution. Meanwhile, kickers can easily remind us of football, but riders remind us of bicycles or motorcycles. To exploit interactive semantics above to intensify interactions, we propose an Interaction Intensifier Module (abbr. IIM) upon the Relation-aware Frame Part A.

After running Gho(), each obtained feature map of H and O heads contains characteristic information for prediction of corresponding instances. We innovatively employ these features to perform relation reasoning, where no human and object regions are needed. As mentioned above, the total number (1 +K) of categories contains person and K other object categories. As illustrated in Figure 4 (a), We can further cut fho ∈ ℜH∗W∗(1+K) into fh ∈ ℜH∗W∗1 and fo ∈ ℜH∗W∗K . Then, a multilayer perceptron (MLP) with two fully connected layers followed by a sigmoid operation is adopted on feature fh to generate the embedded interactive vector β = [β1, β2, ..., βK], denoting relevance between humans and different objects. Afterwards, O head features fo are weighted with vector β by a multiplication operation, generating the valued O head features f ′o .

where βk ∈ [0, 1] is the k-th interactive value for fo. Finally, IIM outputs f ′ho = concat(fh, f ′o ), which is in same form as inputs. Differing from treating human and object features separately, with the formulation above, interactive semantics from humans can be dynamically passed to objects to intensify interactions and promote I points prediction.

3.4 Correlation ParsingModule

Interactive correlations among humans, objects, and interactions provide crucial information for HOI inference. We propose Correlation Parsing Module (abbr. CPM) upon the Relation-aware Frame Part B to break away from the previous idea of region-based feature enhancement and exploit the interactive correlations among these targets. The detailed design of CPM is provided in Figure 4 (b). The CPM takes all point head features fp ∈ ℜH∗W∗(1+K+N) as input, where the human class, K object classes and N interaction classes are considered as (1 +K +N) visual targets.

In particular, a project function first provides a pattern to convert head feature X = fp into a dynamic adjacency matrix A. Specifically, given the input X ∈ ℜH∗W∗(1+K+N) , we embed two feature tensors (Xθ ,Xφ ) by θ(X) and φ(X) modeled by two 1 ∗ 1 convolutions. The obtained tensors are then reshaped fromH ∗W ∗(1+K+N) to L∗(1+K+N) by the Planar operation. Finally, a linear combination transforms visual targets into the unified adjacency matrix A ∈ ℜ(1+K+N)∗(1+K+N) , where (1+K+N) denotes the length and width of the adjacency matrix. Each element aij ∈ A denotes the correlation between two visual targets.

Figure 4: The detailed design of proposed IIM and CPM. IIM takes H and O head features as inputs and passes interactive semantics from humans to objects to intensify interactions. CPM exploits and integrates interactive correlations among visual targets and helps group HOI triplets without region proposals.

Based on the adjacency matrix, a message spreading process is further adopted to broadcast and integrate interactive correlations from all elements, which can be computed by:

where two operations of Conv1D are single-layer 1D convolutions to build communication among all elements. Between the convolutions, the ⊕ implements addition point to point which updates the hidden element states according to incoming information. The A′ is the updated adjacency matrix.

The following step provides a reverse projection of A′ to convolution space. Firstly, a g(X) function and a reshape operation act as linear embedding to compute the representation of input signal as visual targets. Then a weighted linear combination is employed as the reverse projection:

In this way, A′ plays a role of dynamical weights so that interactive correlations among visual targets can be weighted aggregated. Finally, we reshape the planar tensor Y to threedimensional fad. For displacements prediction, fad as the correlation encoding is concatenated with appearance feature fp to provide correlative information among all visual targets.

3.5 Training and Inference

Training. In addition to predicting categories of instances and interactions, we also need to regress the size and offset of instance boxes by Gwh() and Goff(), which have the same computation with G() in Section 3.2. In summary, the final loss L = Lp + λLd + Lr can be calculated as the weighted sum of point prediction loss Lp, displacement prediction loss Ld and instance box regression loss Lr, where λ is set as 0.1 [Zhou et al., 2019]. As mentioned in Section 3.2, point detection can be transformed into a heatmap estimation problem, where Lp = Lho + Li are optimized by element-wise focal loss. The Ld = Ldh + Ldo is trained by L1 loss, where Ldh and Ldo are losses for predicted displacement dpih to H point and displacement dpio to O point respectively for an I point. The Lr = Lwh + Loff is optimized by L1 loss, which is in charge of regressing box size (h, w) and local offset d = (dx, dy) for location of ground truth instance points.

Inference. After points detection, we select top T human, object and interaction points according to the corresponding confidence scores respectively. Then, point grouping process will determine the optimal triplets. Specifically, for each detected I point (xi , yi ) and predicted displacements dpih and dpio , several candidate instance points will be ranked according to two criteria, which are 1) have high confidence score, 2) be close to the human/object location that generated by interaction point location plus the human/object displacement.

For a grouped regressive H point (xh , yh ) and an O point (xo , yo ), their bounding boxes are determined as Bh = (xh , yh , hh , wh ) and Bo = (xo , yo , ho , wo ), where (hh , wh ) and (ho , wo ) denote the width and height of the boxes. With a set of detected HOI triplets, we use a fusion of scores to predict the final score Shoi for each of them:

where Sh (xh ,yh ) , Si (xi ,yi ) and So (xo ,yo ) are confidence scores of candidate H point, I point and O point.

4 Experiments and Evaluations

4.1 Experimental Setup

Datasets and Evaluation Metrics. We evaluate our method on two large-scale benchmarks, including V-COCO [Yatskar et al., 2016] and HICO-DET [Chao et al., 2018] datasets. VCOCO includes 10,346 images, which contains 16,199 human instances in total and provides 26 common verb categories. HICO-DET contains 47,776 images, where 80 object categories and 117 verb categories compose of 600 HOI categories. There are three different HOI category sets in HICODET, which are: (a) all 600 HOI categories (Full), (b) 138 HOI categories with less than 10 training instances (Rare), and (c) 462 HOI categories with 10 or more training instances (Non-Rare). A detected 〈 human, verb, object 〉 triplet is considered as a true positive if: 1) it has correct interaction label; 2) both the predicted human and object bounding boxes have IoU of 0.5 or higher with the ground-truth boxes. Following the standard protocols, we use role mean average precision (mAProle) [Yatskar et al., 2016] to report evaluation results.

Implementation details. During training, input images have the resolution of 512 ∗ 512, yielding a resolution of 128 ∗ 128 for all output head features. We employ standard data augmentation following [Zhou et al., 2019]. Our model is optimized with Adam. The batch-size is set as 15 for VCOCO and 20 for HICO-DET. We train the model for 140 epochs, with the initial learning rate of 5e-4 which drops 10x at 90 and 120 epochs respectively. The top predictions T is set as 100. Our experiments are conducted by Pytorch on a single GPU of NVIDIA Tesla P100.

4.2 Overall Performance

We compare our method with state-of-the-arts in this subsection. Meanwhile, we strip all components related to the Relation-aware Frame structure, IIM and CPM as baseline.

Performance on V-COCO. Comparison results in terms of mAProle are shown in Table 1. It can be seen that our proposed RR-Net has a mAP(%) of 54.2, obtaining the best performance among all methods. Although we do not adopt previous region-based feature learning (e.g., RPNN [Zhou and Chi, 2019], Contextual Att [Wang et al., 2019]), or employ human pose (e.g., PMFNet [Wan et al., 2019], TIN [Li et al., 2019]), our method outperforms these methods with sizable gains. Besides, our method achieves an absolute gain of 2.8 points, which is a relative improvement of 5.5% compared with the baseline, validating its efficacy in HOI detection task.

Performance on HICO-DET. Table 2 shows the comparison results on HICO-DET. Firstly, the detection results of our RR-Net are the best among all methods under the Full and Non-Rare settings, demonstrating that our method is more competitive than the others in detecting most common HOIs. It is noted that RR-Net is not preeminent in detecting rare HOIs (HOI categories with less than 10 training instances), because our relation reasoning modules need enough training samples to exert their effects. Besides, our RR-Net obtains 20.72 mAP on HICO-DET (Defualt Full), which achieves relative gain of 9.8% compared with the baseline. These results quantitatively show the efficacy of our method.

4.3 Ablation Studies

Performance Impact of Proposed Components. V-COCO [Yatskar et al., 2016] serves as the testbed in this subsection. We first study the performance impact of our proposed progressive Relation-aware Frame, Interaction Intensifier Module (IIM) and Correlation Parsing Module (CPM). As shown in Table 3, when we adopt the Relation-aware Frame Part A or Part B (See Figure 3) severally, the mAP are 52.4 and 52.2, indicating the respective effects of the two parts. Adopting the fully Relation-aware Frame obtains the mAP of 52.8, which boosts the performance by 1.4 points compared with the baseline. Meanwhile, we investigate the impact of employing IIM and CPM in Table 4. Compared with the baseline, employing IIM (upon Relation-aware Frame Part A) and CPM (upon Relation-aware Frame Part B) brings gains of 1.5 mAP and 1.4 mAP, respectively. Furthermore, improvements of 2.2 mAP and 2.0 mAP can be obtained while employing IIM and CPM respectively upon our fully Relation-aware Frame. Finally, RR-Net achieves an absolute gain of 2.8 mAP compared with the baseline. These evidences show that each proposed component of our region-independent relation reasoning indeed contributes to the final performance.

Effects visualization of IIM and CPM. Moreover, we take several examples of interaction point and displacements predictions before point grouping in Figure 5 to visualize the effects of IIM and CPM. Intuitively, with IIM and CPM, our framework makes better predictions of interaction point location and displacements compared with baseline. It is worth noting that due to the inaccurate prediction of interaction point location and displacements, the baseline framework in the second row may lead to miss-pairing between the human and horse. However, our framework provides predictions similar to the ground-truth annotation, which helps produce correct HOI detection in point grouping procedure.

4.4 Qualitative Examples

In Figure 6, we first compare our results with baseline in the first two rows to demonstrate our improvements. Each subplot displays one detected 〈 human, verb, object 〉 triplet for easy observation. We can see that our method is capable of detecting various HOIs with higher scores. In addition, the third and fourth rows show that our method is able to detect multiple people taking different interactions with diversified objects. Summarizing all above results validates that our method can learn HOI-specific semantics well from complex environments to provide high quality HOI detections.

5 Conclusion

Recent end-to-end HOI detectors are limited in modelling HOIs due to they are incapable to perform relation reasoning in their methods. Besides, previous region-based HOI feature learning is hardly applicable to end-to-end detectors. Therefore, we make a first try to develop a RR-Net to explore novel techniques of region-independent relation reasoning, which opens up a new direction to improve HOI detection. Specifically, a Relation-aware Frame is proposed, upon which an Interaction Intensifier Module and a Correlation Parsing Module are designed successively to reason HOI-specific interactive semantics without instance regions. Extensive experiments show that our RR-Net outperforms existing methods with sizable gaps, validating its efficacy in detecting HOIs.

Acknowledgments

Thanks to IER foundation (No. HT-JD-CXY-201904).