{"type":"ScholarlyArticle","title":"Cross-Modality Person Re-Identification with Generative Adversarial Training","authors":[{"name":"Pingyang Dai","slug":"pingyang-dai","given":"Pingyang","family":"Dai","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart City , Xiamen University , China; School of Information Science and Engineering , Xiamen University , China"},{"name":"Rongrong Ji","slug":"rongrong-ji","given":"Rongrong","family":"Ji","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart City , Xiamen University , China; School of Information Science and Engineering , Xiamen University , China"},{"name":"Haibin Wang","slug":"haibin-wang","given":"Haibin","family":"Wang","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart City , Xiamen University , China; School of Information Science and Engineering , Xiamen University , China"},{"name":"Qiong Wu","slug":"qiong-wu","given":"Qiong","family":"Wu","affiliation":"School of Information Science and Engineering , Xiamen University , China"},{"name":"Yuyu Huang","slug":"yuyu-huang","given":"Yuyu","family":"Huang","affiliation":"Fujian Key Laboratory of Sensing and Computing for Smart City , Xiamen University , China; School of Information Science and Engineering , Xiamen University , China"}],"abstract":"Person re-identification (Re-ID) is an important task in video surveillance which automatically searches and identifies people across different cameras. Despite the extensive Re-ID progress in RGB cameras, few works have studied the Re-ID between infrared and RGB images, which is essentially a cross-modality problem and widely encountered in real-world scenarios. The key challenge lies in two folds, i.e., the lack of discriminative information to re-identify the same person between RGB and infrared modalities, and the difficulty to learn a robust metric towards such a large-scale cross-modality retrieval. In this paper, we tackle the above two challenges by proposing a novel cross-modality generative adversarial network (termed cmGAN). To handle the issue of insufficient discriminative information, we leverage the cutting-edge generative adversarial training to design our own discriminator to learn discriminative feature representation from different modalities. To handle the issue of large-scale cross-modality metric learning, we integrates both identification loss and cross-modality triplet loss, which minimize inter-class ambiguity while maximizing cross-modality similarity among instances. The entire cmGAN can be trained in an end-to-end manner by using standard deep neural network framework. We have quantized the performance of our work in the newly-released SYSU RGB-IR Re-ID benchmark, and have reported superior performance, i.e., Cumulative Match Characteristic curve (CMC) and Mean Average Precision (MAP), over the state-of-the-art works [Wu et al., 2017], respectively.","doi":"10.24963/ijcai.2018/94","url":"https://ijcai.org/proceedings/2018/94","pdf":"https://ijcai.org/proceedings/2018/0094.pdf","markdown":"https://ijcai.org/proceedings/2018/94.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/94","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":94,"pages":{"first":677,"last":683},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Ahmed et al., 2015] Ejaz Ahmed, Michael J. Jones, and Tim K. Marks. An improved deep learning architecture for person re- identification. In CVPR, pages 3908-3916, 2015.","title":"An improved deep learning architecture for person reidentification","authors":"Ejaz Ahmed, Michael J Jones, Tim K Marks","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arjovsky et al., 2017] Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, pages 214-223, 2017.","title":"Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks","authors":"Arjovsky","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dalal and Triggs, 2005] Navneet Dalal and Bill Triggs. His- tograms of oriented gradients for human detection. In CVPR, pages 886-893, 2005.","title":"Histograms of oriented gradients for human detection","authors":"Triggs Dalal, Navneet Dalal, Bill Triggs","year":"2005","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ding et al., 2015] Shengyong Ding, Liang Lin, Guangrun Wang, and Hongyang Chao. Deep feature learning with relative dis- tance comparison for person re-identification. Pattern Recogni- tion, 48(10):2993-3003, 2015.","title":"Deep feature learning with relative distance comparison for person re-identification","authors":"Ding","year":"2015","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Farenzena et al., 2010] Michela Farenzena, Loris Bazzani, A- lessandro Perina, Vittorio Murino, and Marco Cristani. Person re-identification by symmetry-driven accumulation of local fea- tures. In CVPR, pages 2360-2367, 2010.","title":"Person re-identification by symmetry-driven accumulation of local features","authors":"Michela Farenzena, Loris Bazzani, Alessandro Perina, Vittorio Murino, Marco Cristani","year":"2010","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2014] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversari- al nets. In NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional ad- versarial networks. In CVPR, pages 5967-5976, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Köstinger et al., 2012] Martin Köstinger, Martin Hirzer, Paul Wohlhart, Peter M. Roth, and Horst Bischof. Large scale met- ric learning from equivalence constraints. In CVPR, pages 2288- 2295, 2012.","title":"Large scale metric learning from equivalence constraints","authors":"Martin Köstinger, Martin Hirzer, Paul Wohlhart, Peter M Roth, Horst Bischof","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kviatkovsky et al., 2013] Igor Kviatkovsky, Amit Adam, and E- hud Rivlin. Color invariants for person reidentification. PAMI, 35(7):1622-1634, 2013.","title":"Color invariants for person reidentification","authors":"Igor Kviatkovsky, Amit Adam, Ehud Rivlin","year":"2013","venue":"PAMI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ledig et al., 2017] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, An- drew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR, pages 105- 114, 2017.","title":"Photo-realistic single image super-resolution using a generative adversarial network","authors":"Ledig","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2013] Zhen Li, Shiyu Chang, Feng Liang, Thomas S. Huang, Liangliang Cao, and John R. Smith. Learning locally- adaptive decision functions for person verification. In CVPR, pages 3610-3617, 2013.","title":"Learning locallyadaptive decision functions for person verification","authors":"Zhen Li, Shiyu Chang, Feng Liang, Thomas S Huang, Liangliang Cao, John R Smith","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liao and Li, 2015] Shengcai Liao and Stan Z. Li. Efficien- t PSD constrained asymmetric metric learning for person re- identification. In ICCV, pages 3685-3693, 2015.","title":"Efficient PSD constrained asymmetric metric learning for person reidentification","authors":"Li ; Liao, Shengcai Liao, Stan Z Li","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liao et al., 2015] Shengcai Liao, Yang Hu, Xiangyu Zhu, and S- tan Z. Li. Person re-identification by local maximal occurrence representation and metric learning. In CVPR, pages 2197-2206, 2015.","title":"Person re-identification by local maximal occurrence representation and metric learning","authors":"Shengcai Liao, Yang Hu, Xiangyu Zhu, Stan Z Li","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ma et al., 2017] Xiaolong Ma, Xiatian Zhu, Shaogang Gong, X- udong Xie, Jianming Hu, Kin-Man Lam, and Yisheng Zhong. Person re-identification by unsupervised video matching. Pattern Recognition, 65:197-210, 2017.","title":"Person re-identification by unsupervised video matching","authors":"Xiaolong Ma, Xiatian Zhu, Shaogang Gong, Xudong Xie, Jianming Hu, Kin-Man Lam, Yisheng Zhong","year":"2017","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Matsukawa et al., 2016] Tetsu Matsukawa, Takahiro Okabe, Einoshin Suzuki, and Yoichi Sato. Hierarchical gaussian de- scriptor for person re-identification. In CVPR, pages 1363-1372, 2016.","title":"Hierarchical gaussian descriptor for person re-identification","authors":"Tetsu Matsukawa, Takahiro Okabe, Einoshin Suzuki, Yoichi Sato","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Paisitkriangkrai et al., 2015] Sakrapee Paisitkriangkrai, Chunhua Shen, and Anton van den Hengel. Learning to rank in person re-identification with metric ensembles. In CVPR, pages 1846- 1855, 2015.","title":"Learning to rank in person re-identification with metric ensembles","authors":"Paisitkriangkrai","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Peng et al., 2017a] Yuxin Peng, Xin Huang, and Yunzhen Zhao. An overview of cross-media retrieval: Concepts, methodologies, benchmarks and challenges. CoRR, abs/1704.02223, 2017.","title":"An overview of cross-media retrieval: Concepts, methodologies, benchmarks and challenges","authors":"Peng","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peng et al., 2017b] Yuxin Peng, Jinwei Qi, and Yuxin Yuan. Cm- gans: Cross-modal generative adversarial networks for common representation learning. CoRR, abs/1710.05106, 2017.","title":"Cmgans: Cross-modal generative adversarial networks for common representation learning","authors":"Peng","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qiu et al., 2017] Zhaofan Qiu, Yingwei Pan, Ting Yao, and Tao Mei. Deep semantic hashing with generative adversarial net- works. In ACM SIR, pages 225-234, 2017.","title":"Deep semantic hashing with generative adversarial networks","authors":"Qiu","year":"2017","venue":"ACM SIR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Srivastava and Salakhutdinov, 2014] Nitish Srivastava and Ruslan Salakhutdinov. Multimodal learning with deep boltzmann ma- chines. Journal of Machine Learning Research, 15(1):2949- 2980, 2014.","title":"Multimodal learning with deep boltzmann machines","authors":"Nitish Srivastava, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Subramaniam et al., 2016] Arulkumar Subramaniam, Moitreya Chatterjee, and Anurag Mittal. Deep neural networks with inex- act matching for person re-identification. In NIPS, pages 2667- 2675, 2016.","title":"Deep neural networks with inexact matching for person re-identification","authors":"Subramaniam","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2013] Hua Wang, Feiping Nie, and Heng Huang. Multi-view clustering and feature learning via structured sparsity. In ICML, pages 352-360, 2013.","title":"Multi-view clustering and feature learning via structured sparsity","authors":"Hua Wang, Feiping Nie, Heng Huang","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2016] Taiqing Wang, Shaogang Gong, Xiatian Zhu, and Shengjin Wang. Person re-identification by discriminative selection in video ranking. IEEE TPAMI, 38(12):2501-2514, 2016.","title":"Person re-identification by discriminative selection in video ranking","authors":"Wang","year":"2016","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2017] Bokun Wang, Yang Yang, Xing Xu, Alan Han- jalic, and Heng Tao Shen. Adversarial cross-modal retrieval. In ACM MM, pages 154-162, 2017.","title":"Adversarial cross-modal retrieval","authors":"Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, Heng Tao Shen","year":"2017","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2017] Ancong Wu, Wei-Shi Zheng, Hong-Xing Yu, Shaogang Gong, and Jianhuang Lai. Rgb-infrared cross-modality person re-identification. In ICCV, pages 5390-5399, 2017.","title":"Rgb-infrared cross-modality person re-identification","authors":"Ancong Wu, Wei-Shi Zheng, Hong-Xing Yu, Shaogang Gong, Jianhuang Lai","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xiong et al., 2014] Fei Xiong, Mengran Gou, Octavia I. Camps, and Mario Sznaier. Person re-identification using kernel-based metric learning methods. In ECCV, pages 1-16, 2014.","title":"Person re-identification using kernel-based metric learning methods","authors":"Fei Xiong, Mengran Gou, Octavia I Camps, Mario Sznaier","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yang et al., 2014] Yang Yang, Jimei Yang, Junjie Yan, Shengcai Liao, Dong Yi, and Stan Z. Li. Salient color names for person re-identification. In ECCV, pages 536-551, 2014.","title":"Salient color names for person re-identification","authors":"Yang Yang, Jimei Yang, Junjie Yan, Shengcai Liao, Dong Yi, Stan Z Li","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yi et al., 2014] Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z. Li. Deep metric learning for person re-identification. In ICPR, pages 34-39, 2014.","title":"Deep metric learning for person re-identification","authors":"Dong Yi, Zhen Lei, Shengcai Liao, Stan Z Li","year":"2014","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2014] Hanwang Zhang, Yang Yang, Huan-Bo Luan, Shuicheng Yan, and Tat-Seng Chua. Start from scratch: To- wards automatically identifying, modeling, and naming visual at- tributes. In ACM MM, pages 187-196, 2014.","title":"Start from scratch: Towards automatically identifying, modeling, and naming visual attributes","authors":"Hanwang Zhang, Yang Yang, Huan-Bo Luan, Shuicheng Yan, Tat-Seng Chua","year":"2014","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhao et al., 2013] Rui Zhao, Wanli Ouyang, and Xiaogang Wang. Unsupervised salience learning for person re-identification. In CVPR, pages 3586-3593, 2013.","title":"Unsupervised salience learning for person re-identification","authors":"Rui Zhao, Wanli Ouyang, Xiaogang Wang","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhao et al., 2017] Xin Zhao, Guiguang Ding, Yuchen Guo, Jun- gong Han, and Yue Gao. TUCH: turning cross-view hashing into single-view hashing via generative adversarial nets. In IJCAI, pages 3511-3517, 2017.","title":"TUCH: turning cross-view hashing into single-view hashing via generative adversarial nets","authors":"Xin Zhao, Guiguang Ding, Yuchen Guo, Jungong Han, Yue Gao","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/491"},{"position":32,"raw":"[Zheng et al., 2011] Wei-Shi Zheng, Shaogang Gong, and Tao X- iang. Person re-identification by probabilistic relative distance comparison. In CVPR, pages 649-656, 2011.","title":"Person re-identification by probabilistic relative distance comparison","authors":"Wei-Shi Zheng, Shaogang Gong, Tao Xiang","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}