{"type":"ScholarlyArticle","title":"HorNet: A Hierarchical Offshoot Recurrent Network for Improving Person Re-ID via Image Captioning","authors":[{"name":"Shiyang Yan","slug":"shiyang-yan","given":"Shiyang","family":"Yan","affiliation":"Nanjing Institute of Advanced Artificial Intelligence; Horizon Robotics"},{"name":"Jun Xu","slug":"jun-xu","given":"Jun","family":"Xu","affiliation":"Nanjing Institute of Advanced Artificial Intelligence; Horizon Robotics"},{"name":"Yuai Liu","slug":"yuai-liu","given":"Yuai","family":"Liu","affiliation":"Nanjing Institute of Advanced Artificial Intelligence; Horizon Robotics"},{"name":"Lin Xu","slug":"lin-xu","given":"Lin","family":"Xu","affiliation":"Nanjing Institute of Advanced Artificial Intelligence; Horizon Robotics"}],"abstract":"Person re-identification (re-ID)  aims to recognize a person-of-interest across different cameras with notable appearance variance. Existing research works focused on the capability and robustness of visual representation. In this paper, instead, we propose a novel hierarchical offshoot recurrent network (HorNet) for improving person re-ID via image captioning. Image captions are semantically richer and more consistent than visual attributes, which could significantly alleviate the variance. We use the similarity preserving generative adversarial network (SPGAN) and an image captioner to fulfill domain transfer and language descriptions generation. Then the proposed HorNet can learn the visual and language representation from both the images and captions jointly, and thus enhance the performance of person re-ID. Extensive experiments are conducted on several benchmark datasets with or without image captions, i.e., CUHK03, Market-1501, and Duke-MTMC, demonstrating the superiority of the proposed method. Our method can generate and extract meaningful image captions while achieving state-of-the-art performance.","doi":"10.24963/ijcai.2019/742","url":"https://ijcai.org/proceedings/2019/742","pdf":"https://ijcai.org/proceedings/2019/0742.pdf","markdown":"https://ijcai.org/proceedings/2019/742.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/742","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":742,"pages":{"first":5342,"last":5348},"keywords":["Natural Language Processing: Embeddings","Computer Vision: Language and Vision","Machine Learning: Deep Learning","Computer Vision: Computer Vision","Natural Language Processing: Natural Language Generation"],"videos":[],"references":[{"position":1,"raw":"[Almazan et al., 2018] Jon Almazan, Bojana Gajic, et al. Re-id done right: towards good practices for person re- identification. arXiv:1801.05339, 2018.","title":"Re-id done right: towards good practices for person reidentification","authors":"Almazan","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aneja et al., 2018] Jyoti Aneja, Aditya Deshpande, et al. Convolutional image captioning. In CVPR, pages 5561- 5570, 2018.","title":"Convolutional image captioning","authors":"Jyoti Aneja, Aditya Deshpande","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, et al. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bai et al., 2017a] Song Bai, Xiang Bai, et al. Scalable per- son re-identification on supervised smoothed manifold. In CVPR, pages 2530-2539, 2017.","title":"Scalable person re-identification on supervised smoothed manifold","authors":"Bai","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bai et al., 2017b] Xiang Bai, Mingkun Yang, et al. Deep- person: Learning discriminative deep features for person re-identification. arXiv:1711.10658, 2017.","title":"Deepperson: Learning discriminative deep features for person re-identification","authors":"Bai","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chang et al., 2018] Yan-Shuo Chang, Ming-Yu Wang, et al. Joint deep semantic embedding and metric learning for person re-identification. PRL, 2018.","title":"Joint deep semantic embedding and metric learning for person re-identification","authors":"Yan-Shuo Chang, Ming-Yu Wang","year":"2018","venue":"PRL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2018] Dapeng Chen, Hongsheng Li, et al. Improving deep visual representation for person re- identification by global and local image-language associa- tion. In ECCV, pages 54-70, 2018.","title":"Improving deep visual representation for person reidentification by global and local image-language association","authors":"Dapeng Chen, Hongsheng Li","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng et al., 2018] Weijian Deng, Liang Zheng, et al. Image-image domain adaptation with preserved self- similarity and domain-dissimilarity for person reidentifi- cation. In CVPR, pages 994-1003, 2018.","title":"Image-image domain adaptation with preserved selfsimilarity and domain-dissimilarity for person reidentification","authors":"Weijian Deng, Liang Zheng","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, et al. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ding et al., 2017] Guodong Ding, Salman Khan, et al. Let features decide for themselves: Feature mask network for person re-identification. arXiv:1711.07155, 2017.","title":"Let features decide for themselves: Feature mask network for person re-identification","authors":"Guodong Ding, Salman Khan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jang et al., 2016] Eric Jang, Shixiang Gu, et al. Cat- egorical reparameterization with gumbel-softmax. arXiv:1611.01144, 2016.","title":"Categorical reparameterization with gumbel-softmax","authors":"Eric Jang, Shixiang Gu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ke et al., 2018] Nan Rosemary Ke, Konrad Zolna, et al. Fo- cused hierarchical rnns for conditional sequence process- ing. arXiv:1806.04342, 2018.","title":"Focused hierarchical rnns for conditional sequence processing","authors":"Nan Rosemary Ke, Konrad Zolna","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2012] Wei Li, Rui Zhao, et al. Human reidentifi- cation with transferred metric learning. In ACCV, pages 31-44, 2012.","title":"Human reidentification with transferred metric learning","authors":"Wei Li, Rui Zhao","year":"2012","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2014] Wei Li, Rui Zhao, et al. Deepreid: Deep filter pairing neural network for person re-identification. In CVPR, pages 152-159, 2014.","title":"Deepreid: Deep filter pairing neural network for person re-identification","authors":"Wei Li, Rui Zhao","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2017a] Dangwei Li, Xiaotang Chen, et al. Learn- ing deep context-aware features over body and latent parts for person re-id. In CVPR, pages 384-393, 2017.","title":"Learning deep context-aware features over body and latent parts for person re-id","authors":"Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2017b] Wei Li, Xiatian Zhu, et al. Person re- identification by deep joint learning of multi-loss classi- fication. arXiv:1705.04724, 2017.","title":"Person reidentification by deep joint learning of multi-loss classification","authors":"Li","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liao et al., 2015] Shengcai Liao, Yang Hu, et al. Person re- id by local maximal occurrence representation and metric learning. In CVPR, pages 2197-2206, 2015.","title":"Person reid by local maximal occurrence representation and metric learning","authors":"Liao","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2017] Yutian Lin, Liang Zheng, et al. Improving person re-identification by attribute and identity learning. arXiv:1703.07220, 2017.","title":"Improving person re-identification by attribute and identity learning","authors":"Lin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2016] Jiawei Liu, Zheng-Jun Zha, et al. Multi- scale triplet cnn for person re-identification. In ACMMM, pages 192-196, 2016.","title":"Multiscale triplet cnn for person re-identification","authors":"Jiawei Liu, Zheng-Jun Zha","year":"2016","venue":"ACMMM","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qian et al., 2017] Xuelin Qian, Yanwei Fu, et al. Multi-scale deep learning architectures for person re-identification. pages 5399-5408. ICCV, 2017.","title":"Multi-scale deep learning architectures for person re-identification","authors":"Qian","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ristani et al., 2016] Ergys Ristani, Francesco Solera, et al. Performance measures and a data set for multi-target, multi-camera tracking. In ECCV, pages 17-35, 2016.","title":"Ergys Ristani, Francesco Solera, et al. Performance measures and a data set for multi-target, multi-camera tracking","authors":"Ristani","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schumann, 2017] Arne Schumann. Person re-identification by deep learning attribute-complementary information. In CVPR, pages 1435-1443, 2017.","title":"Person re-identification by deep learning attribute-complementary information","authors":"Arne Schumann","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Su et al., 2017] Chi Su, Jianing Li, et al. Pose-driven deep convolutional model for person re-identification. In ICCV, pages 3960-3969, 2017.","title":"Pose-driven deep convolutional model for person re-identification","authors":"Su","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sun et al., 2017] Yifan Sun, Liang Zheng, et al. Svdnet for pedestrian retrieval. In ICCV, pages 3800-3808, 2017.","title":"Svdnet for pedestrian retrieval","authors":"Yifan Sun, Liang Zheng","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yan et al., 2018a] Fei Yan, Josef Kittler, et al. Person re- identification with vision and language. In ICPR, pages 2136-2141, 2018.","title":"Person reidentification with vision and language","authors":"Yan","year":"2018","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yan et al., 2018b] Shiyang Yan, Fangyu Wu, et al. Image captioning using adversarial networks and reinforcement learning. In ICPR, pages 248-253, 2018.","title":"Image captioning using adversarial networks and reinforcement learning","authors":"Yan","year":"2018","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yi et al., 2014] Dong Yi, Zhen Lei, et al. Deep metric learn- ing for person re-identification. In ICPR, pages 34-39, 2014.","title":"Deep metric learning for person re-identification","authors":"Dong Yi, Zhen Lei","year":"2014","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao et al., 2017] Liming Zhao, Xi Li, et al. Deeply- learned part-aligned representations for person re- identification. In ICCV, pages 3219-3228, 2017.","title":"Deeplylearned part-aligned representations for person reidentification","authors":"Liming Zhao, Xi Li","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zheng et al., 2015a] Liang Zheng, Liyue Shen, et al. Person re-identification meets image search. arXiv:1502.02171, 2015.","title":"Person re-identification meets image search","authors":"Zheng","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zheng et al., 2015b] Liang Zheng, Liyue Shen, et al. Scal- able person re-identification: A benchmark. In ICCV, pages 1116-1124, 2015.","title":"Scalable person re-identification: A benchmark","authors":"Zheng","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zheng et al., 2016] Liang Zheng, Yi Yang, et al. Person re- identification: Past, present and future. arXiv:1610.02984, 2016.","title":"Person reidentification: Past, present and future","authors":"Liang Zheng, Yi Yang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zheng et al., 2017a] Zhedong Zheng, Liang Zheng, et al. A discriminatively learned cnn embedding for person reiden- tification. TOMM, 14(1):13, 2017.","title":"A discriminatively learned cnn embedding for person reidentification","authors":"Zheng","year":"2017","venue":"TOMM","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zheng et al., 2017b] Zhedong Zheng, Liang Zheng, et al. Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In CVPR, pages 3754- 3762, 2017.","title":"Unlabeled samples generated by gan improve the person re-identification baseline in vitro","authors":"Zheng","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zheng et al., 2018] Zhedong Zheng, Liang Zheng, et al. Pedestrian alignment network for large-scale person re- identification. TCSVT, 2018.","title":"Pedestrian alignment network for large-scale person reidentification","authors":"Zheng","year":"2018","venue":"TCSVT","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhong et al., 2017] Zhun Zhong, Xi Li, et al. Re-ranking person re-identification with k-reciprocal encoding. In CVPR, pages 1318-1327, 2017.","title":"Re-ranking person re-identification with k-reciprocal encoding","authors":"Zhun Zhong, Xi Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}