Multi-Granularity Graph-Convolution-Based Method for Weakly Supervised Person Search
Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. Main Track. Pages 1326-1334.
https://doi.org/10.24963/ijcai.2024/147
Keywords
- Computer Vision: CV: Representation learning
- Computer Vision: CV: Recognition (object detection, categorization)
- Computer Vision: CV: Image and video retrieval
Abstract
One-step Weakly Supervised Person Search (WSPS) jointly performs pedestrian detection and person Re-IDentification (ReID) only with bounding box annotations, which makes the traditional person ReID problem more suitable and efficient for real-world applications. However, this task is very challenging due to the following reasons: 1) large feature gap between person ReID and general object detection tasks when learning shared representations; 2) difficult pseudo identity estimation for each person image with unrefined raw detection and dramatic scale changes. To address above issues, we propose a multi-granularity graph convolution framework to jointly optimize the aligned task features, as well as to assist the pseudo label estimation. Specifically, the multi-granularity feature alignment module (MFA) in the designed two-branch framework, employs cluster-level bi-directional interaction of various granularity information to narrow down the large feature gap. Further, upon the MFA module, we introduce the multi-granularity graph-convolution-based pseudo-label estimation module, to enhance feature representations for distinguishing diverse identities. Extensive experimental results demonstrate the effectiveness of the proposed method, and show superior performances to state-of-the art methods by a large margin on CUHK-SYSU and PRW datasets.
- [Cao et al., 2022] Jiale Cao, Yanwei Pang, Rao Muhammad Anwer, Hisham Cholakkal, Jin Xie, Mubarak Shah, and Fahad Shahbaz Khan. Pstr: End-to-end one-step person search with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9458-9467, 2022.
- [Chen et al., 2018] Di Chen, Shanshan Zhang, Wanli Ouyang, Jian Yang, and Ying Tai. Person search via a mask-guided two-stream cnn model. In Proceedings of the european conference on computer vision (ECCV), pages 734-750, 2018.
- [Chen et al., 2019] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.
- [Chen et al., 2020] Di Chen, Shanshan Zhang, Jian Yang, and Bernt Schiele. Norm-aware embedding for efficient person search. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 12615-12624, 2020.
- [Dong et al., 2020a] Wenkai Dong, Zhaoxiang Zhang, Chunfeng Song, and Tieniu Tan. Bi-directional interac- tion network for person search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2839-2848, 2020.
- [Dong et al., 2020b] Wenkai Dong, Zhaoxiang Zhang, Chunfeng Song, and Tieniu Tan. Instance guided pro- posal network for person search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2585-2594, 2020.
- [Feng et al., 2023] Cheng Feng, Dezhi Han, and Chongqing Chen. Dthn: Dual-transformer head end-to-end person search network. Computers, Materials & Continua, 77(1), 2023.
- [Han et al., 2019] Chuchu Han, Jiacheng Ye, Yunshan Zhong, Xin Tan, Chi Zhang, Changxin Gao, and Nong Sang. Re-id driven localization refinement for person search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9814-9823, 2019.
- [Han et al., 2021a] Byeong-Ju Han, Kuhyeun Ko, and Jae- Young Sim. End-to-end trainable trident person search network using adaptive gradient propagation. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 925-933, 2021.
- [Han et al., 2021b] Chuchu Han, Kai Su, Dongdong Yu, Zehuan Yuan, Changxin Gao, Nong Sang, Yi Yang, and Changhu Wang. Weakly supervised person search with region siamese networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12006-12015, 2021.
- [Hansun, 2013] Seng Hansun. A new approach of moving average method in time series analysis. In 2013 confer- ence on new media studies (CoNMedia), pages 1-4. IEEE, 2013.
- [He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
- [Jia et al., 2022] Chengyou Jia, Minnan Luo, Caixia Yan, Xiaojun Chang, and Qinghua Zheng. Cgua: Context- guided and unpaired-assisted weakly supervised person search. arXiv preprint arXiv:2203.14307, 2022.
- [Ke et al., 2022] Xiao Ke, Hao Liu, Wenzhong Guo, Baitao Chen, Yuhang Cai, and Weibin Chen. Joint sample en- hancement and instance-sensitive feature learning for ef- ficient person search. IEEE Transactions on Circuits and Systems for Video Technology, 32(11):7924-7937, 2022.
- [Kim et al., 2021] Hanjae Kim, Sunghun Joung, Ig-Jae Kim, and Kwanghoon Sohn. Prototype-guided saliency fea- ture learning for person search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4865-4874, 2021.
- [Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
- [Lan et al., 2018] Xu Lan, Xiatian Zhu, and Shaogang Gong. Person search by multi-scale matching. In Proceedings of the European conference on computer vision (ECCV), pages 536-552, 2018.
- [Lee et al., 2022] Sanghoon Lee, Youngmin Oh, Donghyeon Baek, Junghyup Lee, and Bumsub Ham. Oimnet++: Prototypical normalization and localization-aware learn- ing for person search. In European Conference on Com- puter Vision, pages 621-637. Springer, 2022.
- [Li and Miao, 2021] Zhengjia Li and Duoqian Miao. Se- quential end-to-end network for efficient person search. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pages 2011-2019, 2021.
- [Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748, 2018.
- [Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
- [Tang et al., 2024] Zongheng Tang, Yulu Gao, Tianrui Hui, Fengguang Peng, and Si Liu. Mi3c: Mining intra-and inter-image context for person search. Pattern Recogni- tion, 148:110169, 2024.
- [Wang et al., 2020] Cheng Wang, Bingpeng Ma, Hong Chang, Shiguang Shan, and Xilin Chen. Tcts: A task- consistent two-stage framework for person search. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11952-11961, 2020.
- [Wang et al., 2023] Benzhi Wang, Yang Yang, Jinlin Wu, Guo-jun Qi, and Zhen Lei. Self-similarity driven scale- invariant learning for weakly supervised person search. arXiv preprint arXiv:2302.12986, 2023.
- [Wang et al., 2024] Jiabei Wang, Yanwei Pang, Jiale Cao, Hanqing Sun, Zhuang Shao, and Xuelong Li. Deep intra- image contrastive learning for weakly supervised one-step person search. Pattern Recognition, 147:110047, 2024.
- [Xiao et al., 2017] Tong Xiao, Shuang Li, Bochao Wang, Liang Lin, and Xiaogang Wang. Joint detection and identi- fication feature learning for person search. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3415-3424, 2017.
- [Xin et al., 2023] Xiaoqi Xin, Dezhi Han, and Mingming Cui. Daaps: A deformable-attention-based anchor-free person search model. Computers, Materials & Continua, 77(2), 2023.
- [Yan et al., 2019] Yichao Yan, Qiang Zhang, Bingbing Ni, Wendong Zhang, Minghao Xu, and Xiaokang Yang. Learning context graph for person search. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, pages 2158-2167, 2019.
- [Yan et al., 2021] Yichao Yan, Jinpeng Li, Jie Qin, Song Bai, Shengcai Liao, Li Liu, Fan Zhu, and Ling Shao. Anchor- free person search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7690-7699, 2021.
- [Yan et al., 2022a] Tianyi Yan, Kuan Zhu, Guibo Zhu, Ming Tang, Jinqiao Wang, et al. Plug-and-play pseudo label cor- rection network for unsupervised person re-identification. arXiv preprint arXiv:2206.06607, 2022.
- [Yan et al., 2022b] Yichao Yan, Jinpeng Li, Shengcai Liao, Jie Qin, Bingbing Ni, Ke Lu, and Xiaokang Yang. Explor- ing visual context for weakly supervised person search. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pages 3027-3035, 2022.
- [Yang et al., 2021] Wenjie Yang, Houjing Huang, Xiaotang Chen, and Kaiqi Huang. Bottom-up foreground-aware fea- ture fusion for practical person search. IEEE Transactions on Circuits and Systems for Video Technology, 32(1):262- 274, 2021.
- [Yu et al., 2022] Rui Yu, Dawei Du, Rodney LaLonde, Daniel Davila, Christopher Funk, Anthony Hoogs, and Brian Clipp. Cascade transformers for end-to-end person search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7267- 7276, 2022.
- [Zhang et al., 2020] Yaobin Zhang, Weihong Deng, Mei Wang, Jiani Hu, Xian Li, Dongyue Zhao, and Dongchao Wen. Global-local gcn: Large-scale label noise cleans- ing for face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7731-7740, 2020.
- [Zhang et al., 2023a] Shizhou Zhang, Qingchun Yang, De Cheng, Yinghui Xing, Guoqiang Liang, Peng Wang, and Yanning Zhang. Ground-to-aerial person search: Benchmark dataset and approach. In Proceedings of the 31st ACM International Conference on Multimedia, pages 789-799, 2023.
- [Zhang et al., 2023b] Yuqi Zhang, Qi Qian, Hongsong Wang, Chong Liu, Weihua Chen, and Fan Wan. Graph convolution based efficient re-ranking for visual retrieval. IEEE Transactions on Multimedia, 2023.
- [Zhang et al., 2024] Pengcheng Zhang, Xiaohan Yu, Xiao Bai, Chen Wang, Jin Zheng, and Xin Ning. Joint dis- criminative representation learning for end-to-end person search. Pattern Recognition, 147:110053, 2024.
- [Zheng et al., 2017] Liang Zheng, Hengheng Zhang, Shaoyan Sun, Manmohan Chandraker, Yi Yang, and Qi Tian. Person re-identification in the wild. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 1367-1376, 2017.
- [Zhong et al., 2019] Jia-Xing Zhong, Nannan Li, Weijie Kong, Shan Liu, Thomas H Li, and Ge Li. Graph con- volutional label noise cleaner: Train a plug-and-play ac- tion classifier for anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1237-1246, 2019.
References were extracted automatically from the PDF and may contain errors.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
1 Introduction
Person Search (PS) aims to localize a given query person in a gallery of unconstrained scene images, which is a joint task of pedestrian detection and ReID [Dong et al., 2020a; Li and Miao, 2021; Yu et al., 2022]. It makes the traditional person ReID problem more practical and efficient for real-world applications, such as video surveillance, retail analytics, smart cities, etc. However, PS is more challenging due to the unrefined raw detection, dramatic scale and camera view changes, etc. Most existing PS methods adopt fully-supervised settings, termed SPS [Cao et al., 2022; Li and Miao, 2021], where both of the bounding boxes and identity labels are required for model training. Although such SPS methods have achieved high performance, it is very expensive and time-consuming to annotate all bounding boxes and identities, especially for obtaining a large number of different identity annotations across different scenes. Considering that it is much easier to annotate bounding boxes than person identities, our work focuses on WSPS, where only the bounding-box annotations are required for model training.
* corresponding authors.
Existing WSPS methods can be roughly divided into two categories: the two-step [Jia et al., 2022] and one-step [Wang et al., 2023; Wang et al., 2024] WSPS methods. The two-step WSPS methods first train a detector to crop person images and then apply an unsupervised learning (USL) ReID method for person retrieval. Though achieving higher performance, they suffer low efficiency, requiring high computation costs with two network parameters during training and being inconvenient for testing. In contrast, the one-step WSPS method jointly learns the person detector and identifier, which is more efficient and practical to be seamlessly integrated into realworld applications, with improved speed and resource utilization. Therefore, our paper focuses on the one-step WSPS.
For the one-step WSPS, it faces two main challenges: The first is the feature gap between person ReID and general object detection tasks [Han et al., 2021b; Dong et al., 2020a], where PS is a fine-grained task requiring accurate human appearance information, which is more likely to be affected by some redundant context information outside the bounding box. On the contrary, the general object detector usually needs to be robust to some subtle appearance changes. Such a feature gap is also the reason for the great performance gap between one-step and two-step SPS tasks. Secondly, how to accurately estimate the pseudo-identity for each person image is another challenge for WSPS. Compared to traditional USL ReID, the person features in WSPS come from the lowquality proposals generated by unrefined raw detection, resulting in inferior discriminative representation. In addition, the person images in various scenes exhibit dramatic size variations, making person feature matching difficult.
To address the above issues in one-step WSPS, we propose a multi-granularity graph convolution framework to jointly optimize the aligned task features, as well as to assist the pseudo label estimation for each person image. Specifically, for the Multi-granularity Feature Alignment (MFA) module, we aim to reduce the feature gap through task-aware information interaction, encouraging a closer feature representation between two tasks. We maintain two branches: Search branch and Instance branch. The search branch takes the whole scene image as input, and a detector is applied to obtain the detection features. The instance branch utilizes the ground truth bounding box as its input. Given diverse inputs, two branches contain different granularity information: the search branch incorporates broader scene context information, while the instance branch is expected to capture more detailed and finegrained appearance information. Consequently, we build a multi-granularity memory bank to store the granularity information contained in these two branch features. To narrow down the feature gap, we employ Cluster-level Multigranularity Alignment (CMA) to facilitate the cluster-level bi-directional interaction of various granularity information within the same pseudo identity. Besides, we add noise to the initial bounding boxes, obtaining additional augmented drifted boxes for feature generation. Then, we use Boxdrift Invariance Learning (BIL) to enhance the search features against low-quality detection bounding boxes.
Building upon the multi-granularity feature alignment module, we further introduce the Multi-granularity Neighborguided Graph Convolution Based Pseudo-label Estimation (MGPE) module, to enhance feature representations for distinguishing diverse identities. We argue that the effective utilization of multi-granularity features is crucial for representing a person image. Single-granularity features lack sufficient information, while simply aligning multi-granularity features within a person image usually struggles to represent the person image effectively. To address this issue, we employ an Offline Graph Convolution (OGC) formulation for information propagation, progressively gathering favorable granularity information from adjacent nodes and then enriching the feature representation for distinguishing diverse identities. Specifically, we propose the Neighbor Guided Sparse Graph Construction (NGSGC) strategy, ensuring that reliable and valuable neighbors are used for effective granularity information propagation. Finally, following representative one-step WSPS methods [Yan et al., 2022b; Wang et al., 2024], our method also alternates between feature representation learning and pseudo-label estimation.
Our contributions are summarized as follows:
- We propose the multi-granularity feature alignment module to facilitate bi-interaction across different granularities, narrowing down the feature gap that exists in the joint training of pedestrian detection and ReID tasks for one-step WSPS.
- We introduce the multi-granularity graph-convolutionbased pseudo-label estimation module, to enhance feature representations for distinguishing diverse identities, and finally obtain accurate pseudo labels for WSPS.
- Extensive experimental results demonstrate the effectiveness of the proposed method, and show superior performances to state-of-the-art methods by a large margin on the CUHK-SYSU and PRW datasets.
2 RelatedWork
2.1 Supervised Person Search
SPS is a task that combines pedestrian detection and person ReID with bounding boxes and person identity annotations for model training [Ke et al., 2022; Yang et al., 2021]. It can be divided into two-step and one-step SPS methods. Two-step SPS methods perform detection and Re-ID tasks, respectively [Chen et al., 2018; Dong et al., 2020b]. One representative work [Wang et al., 2020] solves the consistency problem existed between two tasks by introducing an identity-guided query detector and detection results adapted ReID model. One-step SPS methods jointly perform detection and ReID tasks with shared feature representations [Cao et al., 2022; Xiao et al., 2017; Yu et al., 2022]. [Xiao et al., 2017] first combines the two-step SPS method into a unified framework and proposes online instance matching to effectively train the ReID task. [Li and Miao, 2021] employs the sequential detection method to solve the problem of inferior features caused by low-quality proposals. Benefiting fromDETR, [Cao et al., 2022] first proposes the transformerbased person search framework, achieving better re-id performance with a multi-level supervision scheme. Two-step SPS methods can achieve better performance, but they suffer from low efficiency and high computation costs. Meanwhile, the feature gap in general detection and ReID tasks poses a new challenge for the joint training of one-stage SPS method.
2.2 Weakly Supervised Person Search
WSPS only relies on easily obtained bounding boxes as annotations for the joint training of detection and ReID tasks. Similar to SPS, WSPS can also be divided into two-step and one-step methods. For two-step WSPS methods, [Jia et al., 2022] first performs pedestrian detection and then alternate between context-informed clustering and unpaired-assisted memory training. For one-step WSPS methods, [Yan et al., 2022b] and [Han et al., 2021b] solve the task by clustering the features of proposals to generate pseudo-labels and then guiding the training process. [Wang et al., 2023] proposes scale-invariant learning to solve the scale variation obstacle, and a dynamic multi-label prediction is used for progressive true label seekness. InWSPS, the generation of pseudo-labels is essential to guide the training process, and the quality of these pseudo-labels significantly affects performances. Consequently, generating accurate pseudo-labels is another crucial challenge for WSPS.
2.3 Graph Convolutional Networks.
The Graph Convolutional Network (GCN) is widely used to propagate information and measure similarity relationships among samples [Kipf and Welling, 2016]. [Zhang et al., 2020; Zhong et al., 2019] use GCN to solve the problem of noisy labels. [Yan et al., 2022a] uses GCN to correct the initial noisy pseudo-labels. However, these GCN-based methods require training. [Zhang et al., 2023b] proposes a graph convolution re-ranking method based on vector generation formulation, which allows graph information to propagate in an offline manner. Motivated by this, we introduce a neighbor-guided graph convolution strategy to enhance feature representation and finally achieve better performance.

3 ProposedMethod
3.1 Overview
One-step WSPS jointly performs pedestrian detection and person ReID, with the bounding boxes as the only available annotations. As illustrated in Figure 1, our framework consists of two components: 1) Multi-Granularity Feature Alignment (MFA) representation learning module, which employs cluster-level bi-directional interaction of variant granularity information, to narrow down the feature gap between person ReID and object detection tasks; 2) Multi-Granularity Graph-Convolution-Based Pseudo-Label Estimation module, aiming to obtain enhanced feature representations facilitating accurate pseudo-label estimation for each person image. Intuitively, our method alternates between feature learning and pseudo-label estimation.
In MFA learning, we first build a multi-granularity memory bank to store the two branch features. For each branch, we employ intra-InfoNCE loss to assist the learning of branchspecific granularity information [Oord et al., 2018]. To facilitate interaction between distinct granularities, we introduce CMA with inter-InfoNCE loss to learn shared feature representations. Moreover, BIL enhances search branch features robust to low-quality proposals. For MGPE, we first utilize multi-granularity memory banks to construct the neighborguided graph. Then, the OGC is applied to progressively utilize neighborhood granularity information, enhancing feature representation for more accurate pseudo-label estimation. Focusing on one-step WSPS, our method combines ReID loss and detection loss, like Faster R-CNN [Ren et al., 2015] for joint task training.
3.2 Multi-granularity Feature Alignment Representation Learning
We adopt a two-stream network consisting of a search branch and an instance branch with shared network architecture and parameters. Specifically, the shared architecture includes the backbone network, region proposal network (RPN) and RoIAlign layers, where RPN is only used for detection while RoI-Align layers are used for both detection and ReID feature generation. Following DICL [Wang et al., 2024], we utilize the fused feature map from the last two layers of the backbone as input for both the RPN and RoI-Align layers. For a given scene image I, the search branch first uses RPN to detect the proposal bounding boxes corresponding to the i-th person image. RoI-Align layers are then employed to perform classification, regression, and feature extraction for search features f S i ∈ Rd based on proposals, where d is the dimension of features. While for instance branch, the shared network takes the i-th person patch cropped with the ground truth bounding box bi to extract the instance features f I i ∈ Rd .
Multi-granularity Memory Banks. To obtain the estimation of pseudo-labels and supervised information for each person image, we maintain two memory banks: search memory bank MS ∈ RN×d and instance memory bank MI ∈ RN×d , where N is the number of person images annotated with ground truth bounding boxes. Two memory banks, termed as multi-granularity memory banks, store the features extracted from their respective branches, and the i-th feature in the memory banks is updated as follows:
where ∗ ∈ {S, I} denotes the item belongs to the search branch or instance branch, and m is the momentum updating factor. The Exponential Moving Average (EMA) updating strategy improves the robustness of features stored in memory banks [Hansun, 2013].
During training, each feature f ∗ i will obtain a pseudo-label yi through clustering. Then, the centroid C∗k ∈ Rd for the cluster k can be computed as follows:
where Nk is the number of examples contained in cluster k. Basically, we apply the memory-based intra-InfoNCE loss to help each feature learn the intra-branch granularity information. The intra-InfoNCE loss is as follows:
where ⟨ · , ·⟩ means the cosine similarity, and P ∗ is the number of person features in the current mini-batch. P S is the number of proposals, while P I is the number of annotated boxes. K is the total number of clusters, C∗+ is the centroid of cluster yi, τ1 is the temperature hyper-parameter. Our twobranch intra-InfoNCE loss can be formulated as follows:
Cluster-level Multi-granularity Alignment. To reduce the discrepancy between two-stream features for different tasks, we design the cluster level multi-granularity alignment module to facilitate shared feature learning, assisted by the centroids stored in the memory banks. Specifically, we design the inter-InfoNCE loss to achieve feature alignment, which can be formulated as follows:
The corresponding overall cluster-level multi-granularity alignment inter-InfoNCE loss is as follows:
Box-drift Invariance Learning. As search features suffer from low-quality proposals and insufficient fine-grained information, we propose BIL to address the problems. Specifically, we duplicate the ground truth bounding boxes to generate Z additional boxes and then introduce random noise to them as follows:
where (a1, b1) and (a2, b2) represent the coordinates of the upper left corner and lower right corner of the ground truth bounding box, U(-ϵ, ϵ) denotes a random number obtained from a uniform distribution on the interval [-ϵ, ϵ], (a′1 , b′1 ) and (a′2 , b′2 ) are the new augmented box-drift bounding box coordinates. Before they crop the person patch, we first mask the background pixels outside the ground truth bounding box bi, to avoid instance feature capturing redundant background information. Then, we extract the augmented instance features for the i-th person image f IA i = {f Io i , f Id1 i , f Id2 i } corresponding to no-drift-box, drift-box-1, and drift-box-2. The box-drift invariance loss is as follows:
where [ · ]+ is max( · , 0), d(·, ·) is the Euclidean distance between examples, and δ is a margin parameter. f IA + = ∪yi=yj f IA j indicates the positive examples that have the same pseudo labels with i-th person image, and f IA - = ∪yi̸=yj f IA j indicates the negative examples that have the different pseudo-labels. d(f S i , f Io ) is a regularization term, avoiding missing fine-grained information caused by box drift. Lbox i searches the hard positive and hard negative examples to assist search features in learning robust features against low-quality proposals and fine-grained information aligned with instance features. Furthermore, f IA i is utilized to update the i-th featureMIi in the instancememory for finegrained information ensemble, as follows:
where wij is a weight factor representing the contribution of the box-drift features fj ∈ f IA i to the no-drift feature f Io i :
where τ2 is a temperature hyper-parameter. Masking the background pixel information allows box-drift features find more appearance details, leading to diverse fine-grained features in f IA i . The updating strategy successfully integrates various fine-grained information into a unified feature representation, enhancing the discriminative ability of features stored in the instance memory bank.
3.3 Multi-granularity Graph-convolution-based Pseudo-label Estimation.
Multi-granularity Graph Construction. To thoroughly investigate the relationships among various granularity features, we construct a multi-granularity graph containing 2N nodes based on our multi-granularity memory banks. The definition of the multi-granularity graph is as follows:
where [· ; ·] means feature concatenation in the column dimension, and M ∈ R2N×d . The initial edges of our graph are constructed by a dense affinity matrix as follows:
where A ∈ R2N×2N is the cosine similarity matrix, and Aij = Mi M⊤j , Mi = M[i, :], representing the relationship between node i and node j. Such a dense affinity matrix suffers from low efficiency and high computation costs.
Neighbor-Guided Sparse Graph Construction. To mitigate the problem caused by the dense affinity matrix, we introduce the NGSGC strategy. It dynamically searches neighbors for each example, aiming to create a sparse affinity matrix from the dense matrix. To select valuable neighbors, we introduce our multi-granularity neighbor consistency hypothesis, which supposes that reliable neighbors should be simultaneously close to both granularity features within the same person image. The neighbors for the i-th example can be obtained as follows:
with
where Γ(· , k1) is the top-k1 neighbors, '∧' represents logical-and operator. Indexes 1 to N represent the search feature nodes, whereas N + 1 to 2N correspond to the instance feature nodes. Obviously, i andˆi represent the indexes of two different granularity features under the same person image. Aiˆi represents the similarity relationship between two granularity features. Aji and Ajˆi denote the similarity of the j-th feature to the different granularity features within the ith person image. β is a relaxation coefficient used to control the selection of more reliable samples.
We use the Aiˆi to select neighbors, and we treat it as a discriminative ability score for i-th example. A high score of Aiˆi indicates that the person patch can yield similar feature representations under detection and ReID tasks in WSPS. We believe that such examples already have sufficient discriminative ability to generate correct pseudo-label estimation, and there is no need for additional excessive neighbors to enrich their own feature representation. On the contrary, we consider the examples with a low score require more valuable granularity information to enhance their representations. Consequently, Aji ≥ Aiˆi denotes that we limit the number of neighbors with different Aiˆi values corresponding to different discriminative abilities. Ajˆi ≥ Aiˆi - β denotes that we further select the reliable neighbors for information propagation based on our hypothesis.
Consequently, the new sparse affinity matrix used for our OCN information propagation formulation is as follows:
where γ is the temperature parameter. Considering that the similarity relationships between examples are pairwise, the affinity matrix should be symmetric. To achieve this, we adopt a simple averaging strategy as follows:
Offline Graph Convolution Feature Propagation. The standard GCN layer can be formulated as follows:
where A is the affinity matrix, and D is the diagonal degree matrix computed with Dii = ∑j Aij. X is the input features, and W is the parameter of GCN that is training required. When W is an identity matrix, the GCN is a feature propagation on the graph characterized by an affinity matrix.
Inspired by Eq. 19, our Offline Graph Convolution formulation can be rewritten as:
where M ∈ R2N×d is the input features (worked as graph nodes). ˆ A ∈ R2N×2N is the sparse affinity matrix obtained by Eq. 15 and 18. The degree matrix can be recomputed with Dii = ∑2N j=1 Aij. ˜ M is the enhanced output features, which has the same dimension as M. We perform such feature propagation several times, empirically setting it to 3 in our experiments. According to the experimental results, we use the features ˜ MS for clustering and then reassign the pseudo-labels to the features stored in memory banks and training examples.
The similarity matrix A may become very large due to the necessity of considering relationships among 2N nodes, requiring high computational costs. To enhance computational efficiency, we could divide the similarity matrix A into subgraphs and then utilize the divided subgraphs and their relationships for propagating information between nodes.
3.4 Training and Inference
On the whole, our model is trained in an end-to-end manner, and the overall loss arrives at:
where Ldet is the detection loss used in Faster R-CNN [Ren et al., 2015], and λ is a hyper-parameter to balance the ReID loss and detection loss.
During inference, we only use the search branch to detect the person and generate the ReID features for evaluation.
4 Experiments
4.1 Datasets and Settings
CUHK-SYSU[Xiao et al., 2017] is a large-scale public dataset for person search, consisting of 12,490 street images shotted by cameras and 5,694 movie snapshots. It contains 18,184 images, 8,432 annotated identities, and 96,143 annotated bounding boxes in total. The training set has 11,206 images with 5,532 unique identities, while the test set consists of 6,978 images and 2,900 query persons. In the testing set, there is a set of evaluation protocols for different query persons, with the gallery size varying from 50 to 4,000. For our results, the gallery size is set to 100 by default.
PRW[Zheng et al., 2017] is recorded by six separate cameras placed around the university. It consists of 11,816 images, 932 annotated identities, and 43,110 annotated bounding boxes. The training set has 5,704 images, 18,048 pedestrian bounding boxes, and 482 unique person identities. The test set consists of 6,112 images, 25,062 pedestrian bounding boxes, and 450 person identities.
Evaluation Protocol. Following the evaluation settings widely used in PS, we use the Cumulative Matching Characteristic (CMC) and the mean Averaged Precision (mAP) to evaluate the performance of our methods.
| Methods | PRW | CUHK-SYSU | ||
|---|---|---|---|---|
| mAP | top-1 | mAP | top-1 | |
| Fully supervised: | ||||
| MGTS [Chen et al., 2018] | 32.6 | 72.1 | 83.0 | 83.7 |
| CLSA [Lan et al., 2018] | 38.7 | 65.0 | 87.2 | 88.5 |
| Two-step RDLR [Han et al., 2019] | 42.9 | 70.2 | 93.0 | 94.2 |
| TCTS [Wang et al., 2020] | 46.8 | 87.5 | 93.9 | 95.1 |
| IGPN [Zhang et al., 2023a] | 47.2 | 87.0 | 90.3 | 91.4 |
| Weakly supervised: | ||||
| CGUA [Jia et al., 2022] | 42.7 | 86.9 | 91.0 | 92.2 |
| Fully supervised: | ||||
| CTXG [Yan et al., 2019] | 33.4 | 73.6 | 84.1 | 86.5 |
| NAE [Chen et al., 2020] | 44.0 | 81.1 | 92.1 | 92.9 |
| PGS [Kim et al., 2021] | 44.2 | 85.2 | 92.3 | 94.7 |
| BINet [Dong et al., 2020a] | 45.3 | 81.7 | 90.0 | 90.7 |
| AlignPS [Yan et al., 2021] | 45.9 | 81.9 | 93.1 | 93.4 |
| SeqNet [Li and Miao, 2021] | 46.7 | 83.4 | 93.8 | 94.6 |
| OIMNet++ [Lee et al., 2022] | 47.7 | 84.8 | 93.1 | 94.1 |
| PSTR [Cao et al., 2022] | 49.5 | 87.8 | 93.5 | 95.0 |
| AGWF [Han et al., 2021a] | 53.3 | 87.7 | 93.3 | 94.2 |
| One-step COAT [Yu et al., 2022] | 53.3 | 87.4 | 94.2 | 94.7 |
| DAAPS [Xin et al., 2023] | 48.6 | 84.7 | 95.0 | 95.6 |
| DTHN [Feng et al., 2023] | 50.7 | 85.1 | 93.9 | 94.3 |
| JDRL [Zhang et al., 2024] | 50.9 | 86.2 | 92.3 | 93.5 |
| MI3C [Tang et al., 2024] | 55.6 | 90.8 | 96.7 | 97.1 |
| Weakly supervised: | ||||
| CGPS [Yan et al., 2022b] | 16.2 | 68.0 | 80.0 | 82.3 |
| R-SiamNet [Han et al., 2021b] | 21.2 | 73.4 | 86.0 | 87.1 |
| SSL [Wang et al., 2023] | 30.7 | 80.6 | 87.4 | 88.5 |
| DICL* [Wang et al., 2024] | 33.4 | 80.5 | 87.5 | 88.9 |
| Ours (batch size = 2) | 36.5 | 81.3 | 87.7 | 89.5 |
| Ours | 37.6 | 82.0 | 88.4 | 89.9 |
4.2 Implementation Details
Our method is implemented on the open-source library mmdetection [Chen et al., 2019], where the backbone is built using ResNet50, pre-trained on ImageNet[He et al., 2016]. Following DICL [Wang et al., 2024], we employ the fused feature map of last two layers of backbone for RPN and deformable layers for RoI-Align layers. For the instance branch, we crop and resize the person patch into a fixed size of 229×96. We adopt the SGD optimizer with the batch size set to 4, the epochs set to 26, the momentum set to 0.9, and the weight decay set to 0.0005. The initial learning rate is 0.003 and is reduced by a factor of 10 at 22th and 24th epochs. For hyperparameters, we set m = 0.1, τ1 = 0.05, ϵ = 0.125, δ = 0.3, τ2 = 0.01, β = 0.1, γ = 0.3, λ = 0.25 for all datasets, while k1 = 10 for PRW and k1 = 5 for CUHKSYSU. Moreover, we conduct all the experiments on a single NVIDIA GeForce RTX 3090 GPU.
4.3 Comparision with the State-of-the-arts
In this section, we compare our method with current stateof-the-art methods, including fully supervised methods and weakly supervised methods. All methods can be divided into one-step and two-step methods.
Performance on PRW. In Table 1, we compare our method with existing WSPS methods. PRW poses a greater challenge with a larger gallery size (6112) compared to CUHK-SYSU (default 100). For the one-step WSPS, our method significantly outperforms all existing methods, surpassing the state-of-the-art method DICL mAP/top-1 by 4.2%/1.5%. Though a performance gap still exists for twostep WSPS, our method provides a more powerful performance approximation to the two-stage method. In comparison to some SPS methods, our approach surprisingly outperforms them.

Performance on CUHK-SYSU. Table 1 displays the mAP/top-1 results on the CUHK-SYSU dataset using the default gallery size of 100. Unlike PRW, which has a constant gallery size, CUHK-SYSU provides multiple gallery sizes ranging from 50 to 4,000. To assess the scalability of our method, we compare it with existing one-step WSPS methods across different gallery sizes, as shown in Fig. 2. Across all gallery sizes, our method consistently outperforms existing one-step WSPS methods, demonstrating its superior capabilities in handling large and challenging datasets.
4.4 Ablation Study
In our proposed method, the framework is built on a twostream network, containing a search branch and an instance branch. In Table 2, according to Eq. 21, the "base" denotes our baseline method, which contains the items λLintra+Ldet for model training, where Lintra is the ReID intra-InfoNCE and Ldet is the detection loss. On top of the baseline, there are three components in our proposed method: 1) the Inter- InfoNCE objective Linter, as mentioned in section 3.2; 2) the box-drift invariance learning Lbox with fine-grained feature ensemble updating strategy, as mentioned in section 3.2; 3) the multi-granularity graph-based pseudo-label estimation method "MGPE", as mentioned in section 3.3.
| base | Linter | Lbox | MGPE | PRW | CUHK-SYSU | |||
|---|---|---|---|---|---|---|---|---|
| mAP | top-1 | mAP | top-1 | |||||
| 1 | ✓ | 28.0 | 76.6 | 84.1 | 85.9 | |||
| 2 | ✓ ✓ | 32.0 | 79.3 | 87.0 | 88.7 | |||
| 3 | ✓ | ✓ | 30.5 | 78.5 | 86.3 | 88.1 | ||
| 4 | ✓ | ✓ | 34.2 | 80.6 | 87.2 | 88.9 | ||
| 5 | ✓ | ✓ | ✓ | 33.8 | 80.6 | 87.4 | 89.1 | |
| 6 | ✓ ✓ | ✓ | 36.5 | 81.3 | 88.2 | 89.7 | ||
| 7 | ✓ | ✓ | ✓ | 36.0 | 80.9 | 88.1 | 89.4 | |
| 8 | ✓ | ✓ | ✓ | ✓ | 37.6 | 82.0 | 88.4 | 89.9 |
| PRW | ||
|---|---|---|
| Method | mAP | top-1 |
| λLintra + Ldet | 28.0 | 76.6 |
| + λLinter (Batch instance) | 29.7 | 78.9 |
| + λLinter (Memory instance) | 30.9 | 78.3 |
| + λLSinter | 31.5 | 79.1 |
| + λLinter (LSinter + LIinter ) | 32.0 | 79.3 |
To reveal how each component contributes to the performance improvement, we implement eight variants of our method as shown in Table 2, where the check mark indicates the application of the component. Each row in Table 2 corresponds to one variant of our method, where we have completed the ablation experiment on both PRW and CUHK-SYSU datasets. For Linter, a large enhancement in mAP/top-1 performance is observed by comparing rows 8, 6, 5, and 2 with rows 7, 4, 3, and 1, respectively, with an average improvement of 2.8%/1.7% on PRW and 1.6%/1.0% on CUHK-SYSU. Using a similar comparison method, additional mAP/top-1 improvements of 1.8%/0.8% and 0.9%/0.8% are observed for Lbox. As for MGFE, a more substantial mAP/top-1 improvement averaging 4.9%/2.5% and 1.6%/1.5% is achieved.
Further Illustration of Cluster-level Multi-granularity Alignment. To further explain the effectiveness of component Linter, we investigate instance-level multi-granularity feature alignment and unidirectional information interaction. In Table 3, λLintra + Ldet serves as our baseline. +Linter (Batch instance) indicates that we replace the K clusterlevel centroids in Eq. 7 with P ∗ instance-level person features in the current mini-batch, where P ∗ person features are assigned labels with unique batch feature indexes. Taking Eq. 5 as an example, we replace CI+ and CIk with f I+ and f I j , respectively, where f I+ = f I i and j ∈ 1, ..., P S . While +Linter (Memory instance) indicates we replace the centroids in Eq. 7 with N instance-level person features in the memory bank. Similarly in Eq. 5, CI+ and CIk are replaced with MI+ and MIj , where MI+ = MIi and j ∈ 1, ...,N. The results demonstrate that our cluster-level feature alignment method +λLinter(LSinter +LIinter ) outperforms instance-level feature alignment and unidirectional information interaction, indicating the superior effectiveness of cluster-level and biinteraction strategies in our method.
Further Illustration of Multi-granularity Graph-based Pseudo-label Estimation. To investigate the effectiveness of enhanced features for pseudo-label estimation, we conduct comprehensive experiments on the dataset PRW with various features shown in Table 4. MS and MI mean directly using the features stored in two memory banks for the generation of pseudo-labels. ˜ MS and ˜ MI denote using our OGC to enhance the feature representations and Mean(f1, f2) =
| Clustering Features | PRW | |
|---|---|---|
| mAP | top-1 | |
| search features MS | 33.8 | 80.6 |
| instance featuresMI | 35.4 | 80.5 |
| Mean(MS , MI ) | 36.1 | 80.7 |
| enhanced search features ˜ MS | 37.6 | 82.0 |
| enhanced instance features ˜ MI | 36.4 | 81.4 |
| Mean( ˜ MS , ˜ MI ) | 36.8 | 80.6 |
| ˜ MS w/ top-k1 neighbors OGC | 35.0 | 81.7 |
(f1 + f2)/2. As shown in Table 4, enhanced search features ˜ MS achieve the best performance, which suggests that context information within search features is more critical than fine-grained appearance information contained in instance branch for pseudo-label estimation. We infer that during the inference stage, the features used for evaluation are obtained from the search branch. Therefore, emphasizing context information in pseudo label estimation is more beneficial for enhancing model performance. Furthermore, ˜ MS w/ top-k1 neighbors OGC means directly utilizing the raw top-k1 neighbors to propagate information. The inferior performance illustrates the effectiveness of our neighbor selection method.
Visualization Analysis. Fig. 3 gives some qualitative results of our method on PRW and CUHK-SYSU test sets.

5 Conclusion
This paper proposes a multi-granularity graph-convolutionbased framework for one-step weakly supervised person search. Facing the challenges of a large feature gap and difficult pseudo-identity estimation, we introduce the multigranularity feature alignment and multi-granularity graphconvolution-based pseudo-label estimation modules, to narrow down the feature gap and enhance feature representation for distinguishing diverse identities, respectively. Extensive experiments on the CUHK-SYSU and PRW datasets demonstrate the superiority of our method.
Acknowledgments
This work was supported in part by the National Key R&D Program of China under Grant No.2023YFA1008600, in part by NSFC under Grant NO.62176198, U22A2096 and 62106184, in part by the Key R&D Program of Shaanxi Province under Grant 2024GX-YBXM135, in part by the Key Laboratory of Big Data Intelligent Computing under Grant BDIC-2023-A-004.
@inproceedings{ijcai2024p147,
title = {Multi-Granularity Graph-Convolution-Based Method for Weakly Supervised Person Search},
author = {Tai, Haichun and Cheng, De and Li, Jie and Wang, Nannan and Gao, Xinbo},
booktitle = {Proceedings of the Thirty-Third International Joint Conference on
Artificial Intelligence, {IJCAI-24}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Kate Larson},
pages = {1326--1334},
year = {2024},
month = {8},
note = {Main Track},
doi = {10.24963/ijcai.2024/147},
url = {https://doi.org/10.24963/ijcai.2024/147},
}
TY - CPAPER TI - Multi-Granularity Graph-Convolution-Based Method for Weakly Supervised Person Search AU - Tai, Haichun AU - Cheng, De AU - Li, Jie AU - Wang, Nannan AU - Gao, Xinbo T2 - Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24 PB - International Joint Conferences on Artificial Intelligence Organization PY - 2024 SP - 1326 EP - 1334 DO - 10.24963/ijcai.2024/147 UR - https://doi.org/10.24963/ijcai.2024/147 ER -Copied
Machine-readable versions of this paper, for tools and AI agents: