{"type":"ScholarlyArticle","title":"ViT-P3DE∗: Vision Transformer Based Multi-Camera Instance Association with Pseudo 3D Position Embeddings","authors":[{"name":"Minseok Seo","slug":"minseok-seo","given":"Minseok","family":"Seo","affiliation":"Inter-University Semiconductor Research Center (ISRC ) Department of Electrical and Computer Engineering , Seoul National University"},{"name":"Hyuk-Jae Lee","slug":"hyuk-jae-lee","given":"Hyuk-Jae","family":"Lee","affiliation":"Inter-University Semiconductor Research Center (ISRC ) Department of Electrical and Computer Engineering , Seoul National University"},{"name":"Xuan Truong Nguyen","slug":"xuan-truong-nguyen","given":"Xuan Truong","family":"Nguyen","affiliation":"Inter-University Semiconductor Research Center (ISRC ) Department of Electrical and Computer Engineering , Seoul National University"}],"abstract":"Multi-camera instance association, which identifies identical objects among multiple objects in multi-view images, is challenging due to several harsh constraints. To tackle this problem, most studies have employed CNNs as feature extractors but often fail under such harsh constraints. Inspired by Vision Transformer (ViT), we first develop a pure ViT-based framework for robust feature extraction through self-attention and residual connection. We then propose two novel methods to achieve robust feature learning. First, we introduce learnable pseudo 3D position embeddings (P3DEs) that represent the 3D location of an object in the world coordinate system, which is independent of the harsh constraints. To generate P3DEs, we encode the camera ID and the object's 2D position in the image using embedding tables. We then build a framework that trains P3DEs to represent an object's 3D position in a weakly supervised manner. Second, we also utilize joint patch generation (JPG). During patch generation, JPG considers an object and its surroundings as a single input patch to reinforce the relationship information between two features. Ultimately, experimental results demonstrate that both ViT-P3DE and ViT-P3DE with JPG achieve state-of-the-art performance and significantly outperform existing works, especially when dealing with extremely harsh constraints.","doi":"10.24963/ijcai.2023/149","url":"https://ijcai.org/proceedings/2023/149","pdf":"https://ijcai.org/proceedings/2023/0149.pdf","markdown":"https://ijcai.org/proceedings/2023/149.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/149","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":149,"pages":{"first":1340,"last":1350},"keywords":["Computer Vision: CV: Recognition (object detection, categorization)","Computer Vision: CV: Applications"],"videos":[],"references":[{"position":1,"raw":"[Bogdan et al., 2018] Oleksandr Bogdan, Viktor Eckstein, Francois Rameau, and Jean-Charles Bazin. Deepcalib: A deep learning approach for automatic intrinsic calibration of wide field-of-view cameras. In Proceedings of the 15th ACM SIGGRAPH European Conference on Visual Media Production, pages 1-10, 2018.","title":"Deepcalib: A deep learning approach for automatic intrinsic calibration of wide field-of-view cameras","authors":"Oleksandr Bogdan, Viktor Eckstein, Francois Rameau, Jean-Charles Bazin","year":"2018","venue":"Proceedings of the 15th ACM SIGGRAPH European Conference on Visual Media Production","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2020] Zhongang Cai, Junzhe Zhang, Daxuan Ren, Cunjun Yu, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, and Chen Change Loy. Messytable: Instance association in multiple camera views. In European Conference on Computer Vision, pages 1-16. Springer, 2020.","title":"Messytable: Instance association in multiple camera views","authors":"Cai","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chavdarova et al., 2018] Tatjana Chavdarova, Pierre Baqué, Stéphane Bouquet, Andrii Maksai, Cijo Jose, Timur Bagautdinov, Louis Lettry, Pascal Fua, Luc Van Gool, and Franc ¸ois Fleuret. Wildtrack: A multi- camera hd dataset for dense unscripted pedestrian detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5030-5039, 2018.","title":"Wildtrack: A multicamera hd dataset for dense unscripted pedestrian detection","authors":"Tatjana Chavdarova, Pierre Baqué, Stéphane Bouquet, Andrii Maksai, Cijo Jose, Timur Bagautdinov, Louis Lettry, Pascal Fua, Luc Van Gool, Franc ¸ois, Fleuret","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2022] Cuiqun Chen, Mang Ye, Meibin Qi, Jingjing Wu, Jianguo Jiang, and Chia-Wen Lin. Structure- aware positional transformer for visible-infrared person re- identification. IEEE Transactions on Image Processing, 31:2352-2364, 2022.","title":"Structureaware positional transformer for visible-infrared person reidentification","authors":"Chen","year":"2022","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Choi et al., 2019] Jinwoo Choi, Chen Gao, Joseph CE Mes- sou, and Jia-Bin Huang. Why can't i dance in the mall? learning to mitigate scene bias in action recognition. Ad- vances in Neural Information Processing Systems, 32, 2019.","title":"Why can't i dance in the mall? learning to mitigate scene bias in action recognition","authors":"Jinwoo Choi, Chen Gao, Joseph Ce Messou, Jia-Bin Huang","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In 9th International Conference on Learning Representations, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"9th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Han et al., 2015] Xufeng Han, Thomas Leung, Yangqing Jia, Rahul Sukthankar, and Alexander C. Berg. Match- net: Unifying feature and metric learning for patch-based matching. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 3279-3286, 2015.","title":"Matchnet: Unifying feature and metric learning for patch-based matching","authors":"Han","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2021] Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang. Transreid: Transformer- based object re-identification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15013-15022, 2021.","title":"Transreid: Transformerbased object re-identification","authors":"Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, Wei Jiang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2022] Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, and Changhu Wang. Transfg: A transformer architecture for fine- grained recognition. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 36, pages 852-860, 2022.","title":"Transfg: A transformer architecture for finegrained recognition","authors":"Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"3rd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Koch et al., 2015] Gregory Koch, Richard Zemel, and Rus- lan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, vol- ume 2, page 0. Lille, 2015.","title":"Siamese neural networks for one-shot image recognition","authors":"Gregory Koch, Richard Zemel, Ruslan Salakhutdinov","year":"2015","venue":"ICML deep learning workshop","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 10012- 10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luo et al., 2021] Wenhan Luo, Junliang Xing, Anton Mi- lan, Xiaoqin Zhang, Wei Liu, and Tae-Kyun Kim. Multiple object tracking: A literature review. Artificial Intelligence, 293:103448, 2021.","title":"Multiple object tracking: A literature review","authors":"Luo","year":"2021","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Naseer et al., 2021] Muzammal Naseer, Kanchana Ranas- inghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vi- sion transformers. In Advances in Neural Information Pro- cessing Systems 34, pages 23296-23308, 2021.","title":"Intriguing properties of vision transformers","authors":"Naseer","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Raghu et al., 2021] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovit- skiy. Do vision transformers see like convolutional neu- ral networks? Advances in Neural Information Processing Systems, 34, 2021.","title":"Do vision transformers see like convolutional neural networks","authors":"Raghu","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Roig et al., 2011] Gemma Roig, Xavier Boix, Horesh Ben Shitrit, and Pascal Fua. Conditional random fields for multi-camera object detection. In 2011 International Con- ference on Computer Vision, pages 563-570. IEEE, 2011.","title":"Conditional random fields for multi-camera object detection","authors":"Gemma Roig, Xavier Boix, Horesh Ben Shitrit, Pascal Fua","year":"2011","venue":"2011 International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Schroff et al., 2015] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015.","title":"Facenet: A unified embedding for face recognition and clustering","authors":"Schroff","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Selvaraju et al., 2017] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on com- puter vision, pages 618-626, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"R Ramprasaath, Michael Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simo-Serra et al., 2015] Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, and Francesc Moreno-Noguer. Discriminative learning of deep convo- lutional feature point descriptors. In Proceedings of the IEEE international conference on computer vision, pages 118-126, 2015.","title":"Discriminative learning of deep convolutional feature point descriptors","authors":"Simo-Serra","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Susanto et al., 2012] Wandi Susanto, Marcus Rohrbach, and Bernt Schiele. 3d object detection with multiple kinects. In European Conference on Computer Vision, pages 93-102. Springer, 2012.","title":"3d object detection with multiple kinects","authors":"Wandi Susanto, Marcus Rohrbach, Bernt Schiele","year":"2012","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablay- rolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In In- ternational Conference on Machine Learning, pages 10347-10357. PMLR, 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"ternational Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2022] Tao Wang, Hong Liu, Pinhao Song, Tianyu Guo, and Wei Shi. Pose-guided feature disentan- gling for occluded person re-identification based on trans- former. In Thirty-Sixth AAAI Conference on Artificial In- telligence, pages 2540-2549, 2022.","title":"Pose-guided feature disentangling for occluded person re-identification based on transformer","authors":"Tao Wang, Hong Liu, Pinhao Song, Tianyu Guo, Wei Shi","year":"2022","venue":"Thirty-Sixth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yun et al., 2022] Ilwi Yun, Hyuk-Jae Lee, and Chae Eun Rhee. Improving 360 monocular depth estimation via non- local dense prediction transformer and joint supervised and self-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3224-3233, 2022.","title":"Improving 360 monocular depth estimation via nonlocal dense prediction transformer and joint supervised and self-supervised learning","authors":"Yun","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zagoruyko and Komodakis, 2015] Sergey Zagoruyko and Nikos Komodakis. Learning to compare image patches via convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4353-4361, 2015.","title":"Learning to compare image patches via convolutional neural networks","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou, 2018] Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National science review, 5(1):44-53, 2018.","title":"A brief introduction to weakly supervised learning","authors":"Zhi-Hua Zhou","year":"2018","venue":"National science review","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}