---
title: "Dual Relation Knowledge Distillation for Object Detection"
authors:
  - "Zhen-Liang Ni"
  - "Fukui Yang"
  - "Shengzhao Wen"
  - "Gang Zhang"
doi: "10.24963/ijcai.2023/142"
url: "https://ijcai.org/proceedings/2023/142"
pdf: "https://ijcai.org/proceedings/2023/0142.pdf"
venue: "Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence"
year: 2023
pages: "1276-1284"
track: "Main Track"
keywords:
  - "Computer Vision: CV: Recognition (object detection, categorization)"
---

# Dual Relation Knowledge Distillation for Object Detection

Zhen-Liang Ni, Fukui Yang, Shengzhao Wen, Gang Zhang

*Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence*, pages 1276-1284. DOI: [10.24963/ijcai.2023/142](https://doi.org/10.24963/ijcai.2023/142)

## Abstract

Knowledge distillation is an effective method for model compression. However, it is still a challenging topic to apply knowledge distillation to detection tasks. There are two key points resulting in poor distillation performance for detection tasks. One is the serious imbalance between foreground and background features, another one is that small object lacks enough feature representation. To solve the above issues, we propose a new distillation method named dual relation knowledge distillation (DRKD), including pixel-wise relation distillation and instance-wise relation distillation. The pixel-wise relation distillation embeds pixel-wise features in the graph space and applies graph convolution to capture the global pixel relation. By distilling the global pixel relation, the student detector can learn the relation between foreground and background features, and avoid the difficulty of distilling features directly for the feature imbalance issue. Besides, we find that instance-wise relation supplements valuable knowledge beyond independent features for small objects. Thus, the instance-wise relation distillation is designed, which calculates the similarity of different instances to obtain a relation matrix. More importantly, a relation filter module is designed to highlight valuable instance relations. The proposed dual relation knowledge distillation is general and can be easily applied for both one-stage and two-stage detectors. Our method achieves state-of-the-art performance, which improves Faster R-CNN based on ResNet50 from 38.4% to 41.6% mAP and improves RetinaNet based on ResNet50 from 37.4% to 40.3% mAP on COCO 2017.

## Full text

## 1 Introduction

In recent years, with the development of deep learning technology, object detection has made great progress [Duan et al., 2019; Yang et al., 2019; Law and Deng, 2018; Tian et al., 2019; Ren et al., 2015; Cai and Vasconcelos, 2018]. The detection framework can be roughly divided into two types, one-stage detector [Tian et al., 2019;

Figure 1: Visualization of pixel-wise relation features. The second row and the third row show common features and pixel-wise relation features, respectively. In pixel-wise relation features, the foreground features are highlighted, proving that pixel-wise relation distillation can make the detector focus on the foreground.

![Image](https://ijcai.org/proceedings/2023/fulltext/142/image_000000_663da722630da99414f88d7e8340161214a15500d639eb35d5ce09b3207121f8.png)

Lin et al., 2017] and two-stage detector [Ren et al., 2015; Cai and Vasconcelos, 2018; Law and Deng, 2018; Duan et al., 2019]. These deep learning methods achieve excellent performance and far surpass traditional detection methods [Ren et al., 2015; Lin et al., 2017; Duan et al., 2019]. However, these deep learning methods need high computational costs, limiting their deployment on mobile devices such as robots and mobile phones. How to balance the computational cost and detection performance is still a challenging topic. Knowledge distillation is an effective method to solve the above problem [Guo et al., 2021; Zhang and Ma, 2021; Wang et al., 2019; Heo et al., 2019; Chen et al., 2017]. It adopts the formof teacher-student learning to transfer knowledge from a large model to a small model. Usually, the student model can be deployed on mobile devices directly. Since the principle of knowledge distillation is simple and effective, it is widely used in computer vision tasks, such as classification, segmentation [Wang et al., 2019; Guo et al., 2021] and object detection.

However, knowledge distillation still faces many challenges in object detection [Dai et al., 2021; Zhang and Ma, 2021; Guo et al., 2021; Wang et al., 2019]. The imbalance between foreground and background is an important issue [Zhang and Ma, 2021; Guo et al., 2021]. Usually, the foreground pixels are far fewer than the background pixels. In the existing knowledge distillation methods [Wang et al., 2019; Heo et al., 2019], the student model learns all pixel features with the same priority from the teacher model. So more attention will be paid to the background feature, limiting the learning of the foreground feature. Since foreground features are critical for detection, distilling all features directly results in poor performance. Some work attempts to solve this issue. NLD [Zhang and Ma, 2021] extracts attention features for distillation to make the detector focus on the target area. The distillation method called DeFeat [Guo et al., 2021] attempts to distill foreground and background features separately. These methods achieve certain results, but do not consider the relationship between instances. Besides, it is difficult to distill effective knowledge from small instance features, resulting in poor performance of small object detection. Current distillation methods [Zhang and Ma, 2021; Dai et al., 2021] rarely take this issue into account, which limits their performance.

Figure 2: Visualization of instance-wise relation matrix. We select 3 images that contain 7 instances, 13 instances, and 13 instances respectively. The numbers on the coordinate axis indicate the index of instances in the same image. The index is sorted according to the size of the instance in the same image. For example, index 0 refers to the smallest instance, the index 6 or index 12 refers to the largest instance. We find that there are more mosaics between small instances which means that instance-wise relation supplements valuable knowledge beyond independent features for small objects. The small instance has richer relations with other size instances.

![Image](https://ijcai.org/proceedings/2023/fulltext/142/image_000001_387f456832e8269735338b14d4795798332fa1ae5d7443402291b725fa45a1f5.png)

To address the above problems, we propose dual relation knowledge distillation (DRKD) to enable the student model to learn the relation between pixels and instances from the teacher model. We observe that pixel-wise relation is not sensitive to the imbalance between foreground and background features and can make the detector focus on the foreground, which is shown in Figure 1. Moreover, instance-wise relations can provide valuable knowledge beyond independent features, especially for small objects. As illustrated in Figure 2, there are 7 objects in the first image. The numbers on the coordinate axis indicate the index of instances in the same image. The index is sorted according to the size of instances in the same image. We find that there are more mosaics between small instances which means that instance-wise relation supplements valuable knowledge beyond independent features for small objects. Thus, we design two relation distillation called pixel-wise relation distillation and instance-wise relation distillation.

The pixel-wise relation distillation is proposed to make the detector focus on the learning of foreground features. We adopt graph convolution to capture global pixel relations. The graph convolution [Chen et al., 2019] captures better feature representations than the attention module to improve model performance. First, the features from the coordinate space are embedded into the graph space. Then, the graph convolution is applied to capture relation features in graph space. Finally, the relation feature is projected back to the original coordinate space. The output feature is called the pixel-wise relation feature. By distilling the pixel-wise relation feature, the detector can pay more attention to the learning of the foreground features, as shown in Figure 1, addressing the imbalance issue.

The instance-wise relation distillation is designed to get richer representation for small instances, based on the fact that the small instance has richer relations with other size instances. First, the Embedded Gaussian function is applied to evaluate the relation of different instances, which is widely used in the attention mechanism [Wang et al., 2018; Fu et al., 2019]. The similarity between different size instances is calculated to get the relation matrix. Besides, we observe that different relations have different contributions to distillation in the experiment. Thus, a relation filter module is designed to emphasize valuable relations. The filtered relation matrix is distilled to transfer instance-wise relation from the teacher detector to the student detector. Meanwhile, the cropped foreground feature is used for distillation to further improve detection accuracy in our framework. The experiment proves that the detection accuracy of small objects can be improved by instance-wise relation distillation. The contributions of this work can be concluded as follows:

- We propose pixel-wise relation distillation based on graph convolution. The graph convolution can capture global context more efficiently than the attention mechanism, achieving a better distillation effect.
- We propose instance-wise relation distillation to get richer representation for small instances. The experiment proves that the detection accuracy of small objects can be improved by instance-wise relation distillation.
- Our dual relation knowledge distillation achieves stateof-the-art performance, which improves Faster R-CNN based on ResNet50 from 38.4% to 41.6% mAP and improves RetinaNet based on ResNet50 from 37.4% to 40.3% mAP on COCO2017. The proposed dual relation knowledge distillation is general and can be easily applied for both one-stage and two-stage detectors.

## 2 RelatedWork

## 2.1 Object Detection

Recently, object detection[Duan et al., 2019; Yang et al., 2019; Law and Deng, 2018] is widely studied. The researchers first propose two-stage detectors [Ren et al., 2015; Cai and Vasconcelos, 2018], generating candidate regions for classification and location. Two-stage detectors usually have higher accuracy but lower speed. The representative methods are Faster R-CNN [Ren et al., 2015] and Cascade R-CNN [Cai and Vasconcelos, 2018]. To solve the problem of two-stage detectors, one-stage detectors are proposed furtherly such as SSD [Liu et al., 2016], YOLOv3 [Redmon and Farhadi, 2018], RetinaNet [Lin et al., 2017], and Fcos [Tian et al., 2019] which do not require region proposals and can directly obtain the final detection results.

Figure 3: (a) The overview of the proposed dual relation knowledge distillation (DRKD). DRKD includes pixel-wise relation distillation, instance-wise relation distillation and instance distillation. The pixel-wise relation is captured by the graph convolution module GloRe. The instance-wise relation is modeled by calculating the similarity between instance features; (b) shows the architecture of GloRe used in pixel-wise relation distillation; (c) shows multi-scale DRKD.

![Image](https://ijcai.org/proceedings/2023/fulltext/142/image_000002_be6c39360dec83045cdf845988470e0df10fa20d8dd4770bdff30be85f24d30a.png)

Thus, one-stage detectors have higher speed but poor performance. To balance speed and performance, model compression is used in detection tasks, such as pruning [Han et al., 2015; Lin et al., 2020], quantification [Zhang et al., 2021; Gong et al., 2019], and distillation [Zhang and Ma, 2021; Dai et al., 2021]. Among them, distillation is the most widely used. However, the distillation still faces many challenges in the detection task. The main challenge is to solve the imbalance issue and improve the small object detection performance. Thus, we propose dual relation distillation.

## 2.2 Knowledge Distillation

Knowledge distillation is a common method of model compression. It can transfer the knowledge from the teacher detector to improve the performance of the student detector. However, it is still a challenging topic to apply knowledge distillation to object detection directly due to some problems such as the imbalance of foreground and background, and the small object detection problem. Some researchers attempt to apply distillation to object detection. Sun et al [Sun et al., 2020] not only distilled hidden features but also distilled classification output and bounding box regression output to improve detection accuracy. FGFI [Wang et al., 2019] proposed a fine-grained feature imitation for anchor-based detectors. It calculated the regions of the near object by the intersection between target boxes and anchors. DeFeat [Guo et al., 2021] decoupled features into foreground and background and distilled them separately to get better distillation performance. General instance distillation (GID) [Dai et al., 2021] proposed the general instance selection module and modeled the relational knowledge between instances for distillation. Non-local distillation [Zhang and Ma, 2021] applied non-local modules to capture global pixel relation and enables the student to learn the pixel relation from the teacher. Different from them, we not only distilled the pixel-wise relations and instance-wise relations but also designed more effective modules to extract them.

## 3 Method

The dual relation knowledge distillation (DRKD) is proposed to enable the student detector to learn the pixel-wise and instance-wise relations from the teacher detector, which is shown in Figure 3.

## 3.1 Pixel-wise Relation Distillation

Pixel-wise relation distillation helps student detectors learn the relation between foreground and background features, addressing the feature imbalance issue. The graph convolution module, named GloRe [Chen et al., 2019], is adopted to capture the global pixel relation. It can capture global context more efficiently than the attention mechanism [Chen et al., 2019], achieving a better distillation effect.

Specifically, we extract multi-scale features from the backbone of teachers and students respectively, and feed them to different GloRe modules for capturing the global pixel relation. After that, the pixel-wise relation features are distilled to transfer the global relation from the teacher to the student. The distillation loss is shown in Equ. (1). Besides, to minimize the feature difference between the student and teacher models, an adaptive convolution is added on the side of the student model.

<!-- formula-not-decoded -->

where k is the number of features. ti and si refer to the feature from the teacher and student respectively. φ represents the GloRe module. f represents adaptive convolution.

As shown in Figure 3 (b), GloRe contains three parts: graph embedding, graph convolution, and reprojection. The coordinate feature is first projected to a low-dimensional graph feature space. For the input feature X ∈ RC×W×H , we first project and transform it to X ∈ RC1×HW by linear layer. After that, the graph node features V ∈ RC1×C2 can be obtained by projecting X. The projection matrix is B ∈ RC2×HW . The projection method is a linear combination of original features, which is shown in Equ. (2). The graph node features can aggregate information from multiple regions.

<!-- formula-not-decoded -->

where B ∈ RC2×HW is learnable projection matrix. V ∈ RC1×C2 is the graph node features.

Based on graph node features, a graph convolution is used to capture the relation between nodes, which is defined by Equ. (3). A denotes the adjacency matrix, which is initialized randomly and updated with training. In the training process, the adjacency matrix learns the weights of the edges between nodes. The weight of edges reflects the relation of nodes. Based on the adjacency matrix and the state update matrix, the node features are updated to obtain the relation-aware features.

<!-- formula-not-decoded -->

where I ∈ RC1×C1 is an identity matrix. A ∈ RC1×C1 represents nodes adjacency matrix. V ∈ RC1×C2 is the graph node features. W ∈ RC2×C2 represents the state update matrix. Z ∈ RC1×C2 represents the relation-aware features in graph space.

Finally, the relation-aware features are projected back to the coordinate feature space, as shown in Equ. (4).

<!-- formula-not-decoded -->

where F ∈ RC1×HW is pixel-wise relation feature, B ∈ RC2×HW is learnable projection matrix, the same as Equ. (2). Z ∈ RC1×C2 represents the relation-aware features in graph space.

## 3.2 Instance-wise Relation Distillation

The instance-wise relation distillation is designed to get richer representation for small instances, based on the fact that the small instance has richer relations with other size instances. The Embedded Gaussian function is applied to model the similarity of instance features. Moreover, we design a relation filter module to emphasize valuable relations. The instancewise relation module is shown as Figure 4.

Figure 4: The architecture of instance-wise relation module. The similarity of different instances is calculated to obtain the relation matrix. A relation filter module is designed to emphasize valuable relation.

![Image](https://ijcai.org/proceedings/2023/fulltext/142/image_000003_4fa750122b2bf7a24a0404d36df5de636857f8ffa448746b6bb76fa13f404b0f.png)

## Instance Feature Extraction

To capture instance-wise relation, we need to extract the instance features. The ground truth of object coordinates is used to extract instance features from the feature map, according to the ratio of the input and the feature map. The extracted instance features are resized to the same size, which is shown in Equ. (5).

<!-- formula-not-decoded -->

where ξ represents ROI Align. x is input feature map. c represents coordinates of ground truth. o indicates the size of the output feature.

## Instance-wise Relation Module and Distillation

As shown in Figure 4, the input feature is X ∈ RN×DKK . N refers to the number of instances in the same image, D refers to the channel of the instance feature, K refers to the size of the instance feature map. The instance-wise relation module can be represented by Equ. (6).

<!-- formula-not-decoded -->

where si and sj refer to the instance feature. g1 and g2 refer to fully connected layer. wij is the weight from relation matrix W ∈ RN×N as shown in relation filter of Figure 4. ψ (si, sj) refers to instance-wise relation feature between instance si and sj . We define instance-wise relation distillation function as Equ. (7).

<!-- formula-not-decoded -->

where si and sj refer to the instance feature from student model and ti and tj refer to the instance feature from teacher model. ψ (si, sj) refers to instance-wise relation feature between instance si and sj. ψ (ti, tj) refers to instance-wise relation feature between instance ti and tj . f represents adaptive convolution.

Furthermore, instance features also help address the imbalance issue between foreground and background [Guo et al., 2021]. Instance distillation can make the detector pay attention to the learning of foreground features and accelerate the convergence of the student detector [Dai et al., 2021]. Therefore, we directly distill the instance features to further improve the distillation performance. L2 norm loss is adopted

Table 1: Distillation results of both one-stage detector RetinaNet and two-stage detector Faster RCNN on COCO2017. PR refers to Pixelwise Relation distillation loss, IR refers to Instance-wise Relation distillation loss, INS refers to Instance distillation loss. RX101 refers to ResNeXt101, R50 refers to ResNet50. One-stage framework refers to RetinaNet. Two-stage framework refers to Faster RCNN.

|   Epochs | Framewark   | Model          | PR   | IR   | INS   |   mAP |   AP50 |   AP75 |   APS |   APM |   APL |
|----------|-------------|----------------|------|------|-------|-------|--------|--------|-------|-------|-------|
|       12 | One-stage   | Teacher(RX101) | -    | -    | -     |  41.0 |   60.9 |   44.0 |  23.9 |  45.2 |  54.0 |
|       12 | One-stage   | Student(R50)   | ×    | ×    | ×     |  36.5 |   55.4 |   39.1 |  20.4 |  40.3 |  48.1 |
|       12 | One-stage   | Student(R50)   | ✓    | ×    | ×     |  38.0 |   56.8 |   40.7 |  22.0 |  41.8 |  50.9 |
|       12 | One-stage   | Student(R50)   | ×    | ✓    | ×     |  36.9 |   55.9 |   39.4 |  21.2 |  40.7 |  47.8 |
|       12 | One-stage   | Student(R50)   | ✓    | ✓    | ×     |  38.4 |   57.6 |   40.9 |  22.6 |  42.1 |  51.0 |
|       12 | One-stage   | Student(R50)   | ✓    | ✓    | ✓     |  38.5 |   57.6 |   41.1 |  21.6 |  42.6 |  50.8 |
|       24 | One-stage   | Student(R50)   | ×    | ×    | ×     |  37.4 |   56.7 |   39.6 |  20.0 |  40.7 |  49.7 |
|       24 | One-stage   | Student(R50)   | ✓    | ×    | ×     |  39.9 |   59.2 |   42.8 |  22.3 |  43.9 |  53.7 |
|       24 | One-stage   | Student(R50)   | ×    | ✓    | ×     |  38.3 |   57.5 |   40.8 |  21.2 |  41.9 |  50.6 |
|       24 | One-stage   | Student(R50)   | ✓    | ✓    | ×     |  40.1 |   59.4 |   42.9 |  23.8 |  43.9 |  53.6 |
|       24 | One-stage   | Student(R50)   | ✓    | ✓    | ✓     |  40.3 |   59.7 |   42.9 |  23.4 |  44.2 |  53.4 |
|       12 | Two-stage   | Teacher(RX101) | -    | -    | -     |  42.1 |   63.0 |   46.3 |  24.8 |  46.2 |  55.3 |
|       12 | Two-stage   | Student(R50)   | ×    | ×    | ×     |  37.4 |   58.1 |   40.4 |  21.2 |  41.0 |  48.1 |
|       12 | Two-stage   | Student(R50)   | ✓    | ×    | ×     |  39.4 |   60.2 |   43.1 |  22.3 |  43.4 |  51.8 |
|       12 | Two-stage   | Student(R50)   | ×    | ✓    | ×     |  38.2 |   59.1 |   41.6 |  22.3 |  41.8 |  49.4 |
|       12 | Two-stage   | Student(R50)   | ✓    | ✓    | ×     |  39.6 |   60.6 |   42.9 |  22.6 |  43.7 |  51.7 |
|       12 | Two-stage   | Student(R50)   | ✓    | ✓    | ✓     |  39.7 |   60.5 |   43.1 |  23.5 |  43.4 |  52.3 |

## Algorithm 1 Dual Relation Knowledge Distillation

- 1: Initialize LPR, LIR, and LINS to 0.
- 2: Calculate the loss LPR based on Equation (1).
- 3: Extract teacher instances t based on Equation (5).
- 4: Extract student instances s based on Equation (5).
- 5: for i in 0:length(t)
- 6: for j in 0:length(t)
- 7: Calculate Rt = ψ (ti, tj) based on Equation (6).
- 8: Calculate Rs = ψ (si, sj) based on Equation (6).
- 9: Calculate the loss LIR based on Equation (7).
- 10: Calculate the loss LINS based on Equation (8).
- 11: Calculate overall loss L based on Equation (9).

for distillation, which is shown in Equ. (8). The adaptive convolution is applied on the side of the student detector to minimize the feature difference between the student and teacher.

<!-- formula-not-decoded -->

where n refers to the number of foreground features. ti and si refer to the feature from the teacher model and student model, respectively. f represents adaptive convolution.

## 3.3 Overall Loss Function

According to the above analysis, the overall loss function contains four parts. Ldet is the task loss used to train the detection model. LPR is the pixel-wise relation distillation loss. LIR is the instance-wise relation distillation loss. LINS is the instance distillation loss. The overall loss function is shown in Equ. (9)

<!-- formula-not-decoded -->

where λ1, λ2, λ3 are hyperparameters used to balance different losses. The pseudo code is shown in Algorithm 1.

## 4 Experimental and Results

## 4.1 Implementation Details

COCO2017 [Lin et al., 2014] is used to evaluate our method, which is a challenging dataset in object detection. It contains 120k images and 80 object classes. All experiments are performed on 8 Tesla P40 GPUs. The optimizer we use is SGD. The batch size is set to 16. The initial learning rate is 0.02. The momentum is set to 0.9 and the weight decay is 0.0001. Unless specified, the ablation experiment usually adopts 1× learning schedule and the comparison experiment with other methods adopts 2× learning schedule. We consider Average Precision as evaluation metric, i.e., mAP , AP50, AP75, APS , APM and APL.

## 4.2 Ablation Study

To verify the effectiveness of the proposed distillation method, a series of ablation experiments are performed, which is shown in Table 1. Our method is evaluated on both the one-stage detector and the two-stage detector.

For the one-stage detector, RetinaNet based on ResNeXt101 [Xie et al., 2017] is chosen as the teacher detector and RetinaNet based on ResNet50 [He et al., 2016] is selected as the student detector. Under the training strategy of 24 epochs, the student detector achieves a 37.4% mAP. Applying pixel-wise relation distillation brings an increase of 2.5% mAP. Adopting instance-wise relation distillation achieves an increase of 0.9% mAP. Compared to the student detector without distillation, employing the proposed DRKD increases mAP by 2.9%. For small object detection, applying pixel-wise relation distillation and instance-wise relation distillation simultaneously achieves an increase of 3.8% APS from 20.0%APS to 23.8%APS.

Table 2: Comparison of different pixel relation modeling methods. Our pixel relation modeling module has better distillation performance than non-local.

| Model               | Distillation   |   mAP |
|---------------------|----------------|-------|
| RX101-RetinaNet     | Teacher        |  41.0 |
| R50-RetinaNet       | Student        |  36.5 |
| RX101-R50-RetinaNet | Non-Local      |  37.6 |
| RX101-R50-RetinaNet | Ours           |  38.0 |

Table 3: Distillation performance comparison of different instance feature size settings. RX101-R50-RetinaNet indicates that the teacher detector is ResNeXt101 based RetinaNet, while the student detector is ResNet50 based RetinaNet.PR refers to Pixel-wise Relation distillation loss, IR refers to Instance-wise Relation distillation loss, INS refers to Instance distillation loss.

| Model               | Distillation   | Size   |   mAP |
|---------------------|----------------|--------|-------|
| R50-RetinaNet       | -              | -      |  36.5 |
| RX101-R50-RetinaNet | IR             | 1×1    |  36.6 |
| RX101-R50-RetinaNet | IR             | 3×3    |  36.9 |
| RX101-R50-RetinaNet | IR             | 5×5    |  36.6 |
| RX101-R50-RetinaNet | PR+IR+INS      | 1×1    |  37.9 |
| RX101-R50-RetinaNet | PR+IR+INS      | 3×3    |  38.5 |
| RX101-R50-RetinaNet | PR+IR+INS      | 5×5    |  38.3 |

For the two-stage detector, Faster RCNN based on ResNeXt101 is chosen as the teacher detector and Faster RCNN based on ResNet50 is selected as the student detector. Under the training strategy of 12 epochs, the student detector achieves a 37.4% mAP. Applying pixel-wise relation distillation brings an increase of 2.0% mAP. Adopting instance-wise relation distillation achieves an increase of 0.8% mAP. Compared to the student detector without distillation, the application of our DRKD increases mAP by 2.3%. For small object detection, applying DRKD achieves an increase of 2.3% APS from 21.2%APS to 23.5%APS. Besides, experiments prove that our pixel relation modeling method based on graph convolution is better than non-local, which is shown in Table 2. The above ablation experiments prove that our method is effective on both one-stage and two-stage detectors.

Besides, we analyze the mean AP of objects with different sizes to determine the effects of different distillation methods. As shown in Table 1, pixel-wise relation distillation brings growth in APS, APM, and APL, indicating that it can improve the detection accuracy of objects with different sizes. This result verifies that the pixel-wise relation distillation helps to address the feature imbalance issue. Moreover, based on pixel-wise relation distillation, employing instancewise relation distillation in 24 epochs can further increase APS of RetinaNet by 1.5% APS from 22.3% APS to 23.8% APS. For small object detection, applying pixel-wise relation distillation and instance-wise relation distillation simultaneously achieves an increase of 3.8% APS from 20.0% APS to 23.8%APS. Our experiment proves that the detection accuracy of small objects can be improved significantly when using both two relation distillations.

Table 4: Validity verification of relation filtering module in instancewise relation distillation. PR refers to Pixel-wise Relation distillation loss, IR refers to Instance-wise Relation distillation loss, INS refers to Instance distillation loss.

| Model               | Distillation   | Filter   |   mAP |
|---------------------|----------------|----------|-------|
| RX101-RetinaNet     | Teacher        | -        |  41.0 |
| R50-RetinaNet       | Student        | -        |  36.5 |
| RX101-R50-RetinaNet | IR             | ×        |  36.6 |
| RX101-R50-RetinaNet | IR             | ✓        |  36.9 |
| RX101-R50-RetinaNet | PR+IR+INS      | ×        |  38.3 |
| RX101-R50-RetinaNet | PR+IR+INS      | ✓        |  38.5 |

Table 5: Hyper-parameter sensitivity study of λ1, λ2, and λ3, with RetinaNet on COCO 2017.

|    λ1 |   mAP |    λ2 |   mAP |    λ3 |   mAP |
|-------|-------|-------|-------|-------|-------|
| 0.006 |  37.9 | 0.001 |  36.7 | 0.005 |  38.4 |
| 0.004 |  38.0 | 0.002 |  36.8 | 0.006 |  38.5 |
| 0.002 |  37.7 | 0.004 |  36.9 | 0.008 |  38.4 |
| 0.001 |  37.2 | 0.006 |  36.8 |  0.01 |  38.3 |

## 4.3 Instance Feature Size Selection

Since the instance-wise relation is modeled based on instance features, the size of instance features may affect the performance of distillation. Therefore, experiments are performed to select the appropriate foreground feature size, which is shown in Table 3. Three different feature sizes, including 1 × 1, 3 × 3, and 5 × 5, are set up in the experiment. When only using instance-wise relation distillation, the 3 × 3 feature size achieves the best result 36.9%mAP, which exceeds baseline by 0.4% mAP. When using three distillation methods, the 3×3 feature size also gets the best result. Therefore, the 3 × 3 size is selected in all experiments.

## 4.4 Relation Filter Experiment

The relation filter module is set to highlight important instance relations. Related experiments are performed to verify its effectiveness, which is shown in Table 4. When only using instance-wise relation distillation, applying relation filter results in a 0.3% mAP increase. Based on our DRKD without relation filter, employing the relation filter brings a 0.2% mAP increase. The above results prove the effectiveness of the relation filter module.

## 4.5 Hyperparameter Selection

To obtain the best distillation performance, we analyze the sensitivity of hyperparameters. A series of experiments are set up to determine the value of λ1, λ2, and λ3 in Equ. (9). RetinaNet is chosen to select parameters for the one-stage detector. As shown in Table 5, when λ1 is 0.004, pixel-wise relation distillation gets the best performance. When λ2 is 0.004, instance-wise relation distillation gets the best performance. When λ3 is 0.006, instance distillation achieves the best performance. For two-stage detectors, λ1 and λ3 are multiplied by 0.25 and 0.3 respectively on the basis of the above values to obtain the best results.

Table 6: Comparison between our methods and other state-of-the-art distillation methods. RX101 refers to ResNeXt101, R50 refers to ResNet50.

| Framewark         | Method                            | Backbone   |   mAP | AP50   | AP75   | APS   |   APM |   APL |
|-------------------|-----------------------------------|------------|-------|--------|--------|-------|-------|-------|
| Cascade Mask RCNN | Teacher                           | RX101      |  47.3 | 66.3   | 51.7   | 28.2  |  51.7 |  62.7 |
| Faster RCNN       | Baseline                          | R50        |  38.4 | 59.0   | 42.0   | 21.5  |  42.1 |  50.3 |
| Faster RCNN       | EDKD [Chen et al., 2017]          | R50        |  38.7 | 59.0   | 42.1   | 22.0  |  41.9 |  51.0 |
| Faster RCNN       | FGFI [Wang et al., 2019]          | R50        |  39.1 | 59.8   | 42.8   | 22.2  |  42.9 |  51.1 |
| Faster RCNN       | Overhaul [Heo et al., 2019]       | R50        |  38.9 | 60.1   | 42.6   | 21.8  |  42.7 |  50.7 |
| Faster RCNN       | NLD(ICLR-21) [Zhang and Ma, 2021] | R50        |  41.5 | 62.2   | 45.1   | 23.5  |  45.0 |  55.3 |
| Faster RCNN       | GKD(ECCV-22) [Tang et al., 2022]  | R50        |  41.5 | 61.9   | 45.1   | 23.5  |  45.1 |  55.4 |
| Faster RCNN       | DRKD (Ours)                       | R50        |  41.6 | 62.4   | 45.3   | 24.2  |  45.3 |  55.3 |
| Cascade Mask RCNN | Teacher                           | RX101      |  47.3 | 66.3   | 51.7   | 28.2  |  51.7 |  62.7 |
| Grid RCNN         | Baseline                          | R50        |  40.4 | 58.4   | 43.6   | 22.8  |  43.9 |  53.3 |
| Grid RCNN         | NLD(ICLR-21) [Zhang and Ma, 2021] | R50        |  42.6 | 61.1   | 46.1   | 24.2  |  46.6 |  55.8 |
| Grid RCNN         | DRKD (Ours)                       | R50        |  43.0 | 61.8   | 46.5   | 25.1  |  46.5 |  56.3 |
| Cascade Mask RCNN | Teacher                           | RX101      |  47.3 | 66.3   | 51.7   | 28.2  |  51.7 |  62.7 |
| Dynamic RCNN      | Baseline                          | R50        |  39.8 | 58.3   | 43.2   | 23.0  |  42.8 |  52.4 |
| Dynamic RCNN      | NLD(ICLR-21) [Zhang and Ma, 2021] | R50        |  42.8 | 61.2   | 47.0   | 23.9  |  46.2 |  57.7 |
| Dynamic RCNN      | DRKD (Ours)                       | R50        |  43.0 | 61.6   | 47.3   | 24.1  |  46.3 |  57.6 |
| RetinaNet         | Teacher                           | RX101      |  41.0 | 60.9   | 44.0   | 23.9  |  45.2 |  54.0 |
| RetinaNet         | Baseline                          | R50        |  37.4 | 56.7   | 39.6   | 20.0  |  40.7 |  49.7 |
| RetinaNet         | Overhaul [Heo et al., 2019]       | R50        |  37.8 | 58.3   | 41.1   | 21.6  |  41.2 |  48.3 |
| RetinaNet         | NLD(ICLR-21) [Zhang and Ma, 2021] | R50        |  39.6 | 58.8   | 42.1   | 22.7  |  43.3 |  52.5 |
| RetinaNet         | FRS(NIPS-21) [Du et al., 2021]    | R50        |  40.1 | 59.5   | 42.5   | 21.9  |  43.7 |  54.3 |
| RetinaNet         | DRKD (Ours)                       | R50        |  40.3 | 59.7   | 42.9   | 23.4  |  44.2 |  53.4 |
| RetinaNet         | Teacher                           | RX101      |  41.0 | 60.9   | 44.0   | 23.9  |  45.2 |  54.0 |
| RepPoints         | Baseline                          | R50        |  38.6 | 59.6   | 41.6   | 22.5  |  42.2 |  50.4 |
| RepPoints         | NLD(ICLR-21) [Zhang and Ma, 2021] | R50        |  40.6 | 61.7   | 43.8   | 23.4  |  44.6 |  53.0 |
| RepPoints         | FGD(CVPR-22) [Yang et al., 2022]  | R50        |  41.3 | -      | -      | -     |  45.2 |  54.0 |
| RepPoints         | DRKD (Ours)                       | R50        |  41.7 | 62.5   | 44.9   | 24.3  |  45.6 |  55.0 |

## 4.6 Comparison with State-of-the-art Methods

To verify the excellent performance of the proposed method, our DRKD is compared with other state-of-the-art distillation methods on COCO2017. The experimental results are shown in Table 6. Since our DRKD can be applied to both onestage and two-stage detectors, we verify it on these two types of detectors. The experimental results show that our DRKD exceeds the current most advanced methods, achieving the state-of-the-art performance.

For the two-stage detector, the Cascaded Mask RCNN with ResNeXt 101 is selected as the teacher detector. Faster RCNN [Ren et al., 2015], Dynamic RCNN [Zhang et al., 2020], and Grid RCNN [Lu et al., 2019] are chosen as the student detector. The backbone of the student detector is ResNet50. Without distillation, Faster RCNN achieves 38.4% mAP. The application of our DRKD brings an increase of 3.2% mAP. FGFI [Wang et al., 2019] achieves 39.1% mAP, which is 2.5% mAP worse than our method. DRKD also exceeds GKD [Tang et al., 2022], NLD [Zhang and Ma, 2021]. Besides, Grid RCNN without distillation achieves 40.4% mAP. NLD [Zhang and Ma, 2021] is a state-of-the-art distillation method, which gets 42.6% mAP. Our DRKD obtains 43.0% mAP, exceeding NLD 0.4% mAP. For Dynamic RCNN, DKRD achieves 43.0% mAP which is higher than all other methods.

For the one-stage detector, RetinaNet with ResNeXt101 is selected as the teacher detector. RetinaNet [Lin et al., 2017] with ResNet50 is chosen as the student detector. Without distillation, RetinaNet achieves 37.4% mAP. Applying our DRKD brings an increase of 2.9% mAP, achieving 40.3% mAP. The mAP of DRKD is higher than that of FRS and NLD. Besides, RepPoints [Yang et al., 2019] without distillation achieves 38.6% mAP. Applying our DRKD brings an increase of 3.1% mAP. The accuracy of the student detector even exceeds the teacher detector by 0.7% mAP. It also exceeds FGD by 0.4% mAP. The above results show that our distillation method achieves state-of-the-art performance on both one-stage and two-stage detectors.

## 5 Conclusion

In this paper, we propose dual relation knowledge distillation. In it, the pixel-wise relation distillation helps to address the feature imbalance issue. Also, the instance-wise relation distillation is proposed to get a richer representation for small instances. We observe that the small instance has richer relations with other size instances. The experiment proves that the detection accuracy of small objects can be improved significantly by using our DRKD. The proposed distillation method is general and can be easily applied for both one-stage and two-stage detectors.

## Contribution Statement

Zhenliang Ni and Fukui Yang contribute equally to this paper, which is identified by * in the title.

## References

1. [Cai and Vasconcelos, 2018] Zhaowei Cai and Nuno Vas- concelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 6154-6162, 2018.
2. [Chen et al., 2017] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning effi- cient object detection models with knowledge distillation. Advances in neural information processing systems, 30, 2017.
3. [Chen et al., 2019] Yunpeng Chen, Marcus Rohrbach, Zhicheng Yan, Yan Shuicheng, Jiashi Feng, and Yannis Kalantidis. Graph-based global reasoning networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 433-442, 2019.
4. [Dai et al., 2021] Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, and Erjin Zhou. General in- stance distillation for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 7842-7851, 2021.
5. [Du et al., 2021] Zhixing Du, Rui Zhang, Ming-Fang Chang, Xishan Zhang, Shaoli Liu, Tianshi Chen, and Yunji Chen. Distilling object detectors with feature richness. In Neural Information Processing Systems, 2021.
6. [Duan et al., 2019] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 6569-6578, 2019.
7. [Fu et al., 2019] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Y- ongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual atten- tion network for scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), June 2019.
8. [Gong et al., 2019] Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4852-4861, 2019.
9. [Guo et al., 2021] Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors via decoupled features. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2154-2164, 2021.
10. [Han et al., 2015] Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural network. In NIPS, 2015.
11. [He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Comput- er Vision and Pattern Recognition (CVPR), June 2016.
12. [Heo et al., 2019] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 1921-1930, 2019.
13. [Law and Deng, 2018] Hei Law and Jia Deng. Cornernet: Detecting objects as paired keypoints. In Proceedings of the European conference on computer vision (ECCV), pages 734-750, 2018.
14. [Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In European conference on com- puter vision, pages 740-755. Springer, 2014.
15. [Lin et al., 2017] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense ob- ject detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
16. [Lin et al., 2020] Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank fea- ture map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1529- 1538, 2020.
17. [Liu et al., 2016] Wei Liu, Dragomir Anguelov, Dumitru Er- han, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pages 21-37. Springer, 2016.
18. [Lu et al., 2019] Xin Lu, Buyu Li, Yuxin Yue, Quanquan Li, and Junjie Yan. Grid r-cnn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7363-7372, 2019.
19. [Redmon and Farhadi, 2018] Joseph Redmon and Ali Farha- di. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
20. [Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. Advances in neural information processing systems, 28:91-99, 2015.
21. [Sun et al., 2020] Ruoyu Sun, Fuhui Tang, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian. Distilling object detectors with task adaptive regularization. arXiv preprint arX- iv:2006.13108, 2020.
22. [Tang et al., 2022] Sanli Tang, Zhongyu Zhang, Zhanzhan Cheng, Jing Lu, Yunlu Xu, Yi Niu, and Fan He. Distill- ing object detectors with global knowledge. In European Conference on Computer Vision, 2022.
23. [Tian et al., 2019] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object de- tection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9627-9636, 2019.
24. [Wang et al., 2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794-7803, 2018.
25. [Wang et al., 2019] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine- grained feature imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4933-4942, 2019.
26. [Xie et al., 2017] Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), July 2017.
27. [Yang et al., 2019] Ze Yang, Shaohui Liu, Han Hu, Liwei Wang, and Stephen Lin. Reppoints: Point set repre- sentation for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9657-9666, 2019.
28. [Yang et al., 2022] Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4643-4652, 2022.
29. [Zhang and Ma, 2021] Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In International Conference on Learning Representations, 2021.
30. [Zhang et al., 2020] Hongkai Zhang, Hong Chang, Bing- peng Ma, Naiyan Wang, and Xilin Chen. Dynamic r-cnn: Towards high quality object detection via dynamic train- ing. In European Conference on Computer Vision, pages 260-275. Springer, 2020.
31. [Zhang et al., 2021] Xiangguo Zhang, Haotong Qin, Yifu D- ing, Ruihao Gong, Qinghua Yan, Renshuai Tao, Yuhang Li, Fengwei Yu, and Xianglong Liu. Diversifying sample generation for accurate data-free quantization. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15658-15667, 2021.