{"type":"ScholarlyArticle","title":"Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds","authors":[{"name":"Heng Wang","slug":"heng-wang","given":"Heng","family":"Wang","affiliation":"School of Computer Science , University of Sydney , Australia"},{"name":"Chaoyi Zhang","slug":"chaoyi-zhang","given":"Chaoyi","family":"Zhang","affiliation":"School of Computer Science , University of Sydney , Australia"},{"name":"Jianhui Yu","slug":"jianhui-yu","given":"Jianhui","family":"Yu","affiliation":"School of Computer Science , University of Sydney , Australia"},{"name":"Weidong Cai","slug":"weidong-cai","given":"Weidong","family":"Cai","affiliation":"School of Computer Science , University of Sydney , Australia"}],"abstract":"Dense captioning in 3D point clouds is an emerging vision-and-language task involving object-level 3D scene understanding. Apart from coarse semantic class prediction and bounding box regression as in traditional 3D object detection, 3D dense captioning aims at producing a further and finer instance-level label of natural language description on visual appearance and spatial relations for each scene object of interest. To detect and describe objects in a scene, following the spirit of neural machine translation, we propose a transformer-based encoder-decoder architecture, namely SpaCap3D, to transform objects into descriptions, where we especially investigate the relative spatiality of objects in 3D scenes and design a spatiality-guided encoder via a token-to-token spatial relation learning objective and an object-centric decoder for precise and spatiality-enhanced object caption generation. Evaluated on two benchmark datasets, ScanRefer and ReferIt3D, our proposed SpaCap3D outperforms the baseline method Scan2Cap by 4.94% and 9.61% in CIDEr@0.5IoU, respectively. Our project page with source code and supplementary files is available at https://SpaCap3D.github.io/.","doi":"10.24963/ijcai.2022/194","url":"https://ijcai.org/proceedings/2022/194","pdf":"https://ijcai.org/proceedings/2022/0194.pdf","markdown":"https://ijcai.org/proceedings/2022/194.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/194","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":194,"pages":{"first":1393,"last":1400},"keywords":["Computer Vision: Representation Learning","Computer Vision: Scene analysis and understanding","Computer Vision: Machine Learning for Vision","Computer Vision: 3D Computer Vision","Computer Vision: Vision and language"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986643","length_sec":82,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984786","length_sec":334,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Achlioptas et al., 2020] Panos Achlioptas, Ahmed Abdelre- heem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3D: Neural listeners for fine-grained 3D object identification in real-world scenes. In ECCV, pages 422- 440. Springer, 2020.","title":"Referit3D: Neural listeners for fine-grained 3D object identification in real-world scenes","authors":"Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, Leonidas Guibas","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Avetisyan et al., 2019] Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X Chang, and Matthias Nießner. Scan2CAD: Learning CAD model alignment in RGB-D scans. In CVPR, pages 2614-2623, 2019.","title":"Scan2CAD: Learning CAD model alignment in RGB-D scans","authors":"Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X Chang, Matthias Nießner","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In ACL Workshop: Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65- 72, 2005.","title":"METEOR: An automatic metric for MT evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"ACL Workshop: Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229. Springer, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2020] Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. ScanRefer: 3D object localization in RGB-D scans using natural language. In ECCV, pages 202-221. Springer, 2020.","title":"ScanRefer: 3D object localization in RGB-D scans using natural language","authors":"Dave Zhenyu, Chen, Angel X Chang, Matthias Nießner","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2021] Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2Cap: Context-aware dense captioning in RGB-D scans. In CVPR, pages 3193- 3203, 2021.","title":"Scan2Cap: Context-aware dense captioning in RGB-D scans","authors":"Zhenyu Chen, Ali Gholami, Matthias Nießner, Angel X Chang","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cornia et al., 2020] Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image captioning. In CVPR, pages 10578- 10587, 2020.","title":"Meshed-memory transformer for image captioning","authors":"Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, Rita Cucchiara","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dai et al., 2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In CVPR, pages 5828-5839, 2017.","title":"ScanNet: Richly-annotated 3D reconstructions of indoor scenes","authors":"Angela Dai, X Angel, Manolis Chang, Maciej Savva, Thomas Halber, Matthias Funkhouser, Nießner","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Herdade et al., 2019] Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. Image captioning: trans- forming objects into words. In NeurIPS, pages 11137- 11147, 2019.","title":"Image captioning: transforming objects into words","authors":"Simao Herdade, Armin Kappeler, Kofi Boakye, Joao Soares","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ji et al., 2021] Jiayi Ji, Yunpeng Luo, Xiaoshuai Sun, Fuhai Chen, Gen Luo, Yongjian Wu, Yue Gao, and Rongrong Ji. Improving image captioning by leveraging intra-and inter-layer global representation in transformer network. In AAAI, volume 35, pages 1655-1663, 2021.","title":"Improving image captioning by leveraging intra-and inter-layer global representation in transformer network","authors":"Jiayi Ji, Yunpeng Luo, Xiaoshuai Sun, Fuhai Chen, Gen Luo, Yongjian Wu, Yue Gao, Rongrong Ji","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson et al., 2016] Justin Johnson, Andrej Karpathy, and Li Fei-Fei. DenseCap: Fully convolutional localization networks for dense captioning. In CVPR, pages 4565- 4574, 2016.","title":"DenseCap: Fully convolutional localization networks for dense captioning","authors":"Justin Johnson, Andrej Karpathy, Li Fei-Fei","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2019] Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, and In So Kweon. Dense relational captioning: Triple- stream networks for relationship-based captioning. In CVPR, pages 6271-6280, 2019.","title":"Dense relational captioning: Triplestream networks for relationship-based captioning","authors":"Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, In So Kweon","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Landau and Jackendoff, 1993] Barbara Landau and Ray Jackendoff. \"What\" and \"where\" in spatial language and spatial cognition. Behavioral and Brain Sciences, 16(2):255-265, 1993.","title":"What\" and \"where\" in spatial language and spatial cognition","authors":"Jackendoff Landau, Barbara Landau, Ray Jackendoff","year":"1993","venue":"Behavioral and Brain Sciences","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2019a] Guang Li, Linchao Zhu, Ping Liu, and Yi Yang. Entangled transformer for image captioning. In ICCV, pages 8928-8937, 2019.","title":"Entangled transformer for image captioning","authors":"Li","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2019b] Xiangyang Li, Shuqiang Jiang, and Jun- gong Han. Learning object context for dense captioning. In AAAI, volume 33, pages 8650-8657, 2019.","title":"Learning object context for dense captioning","authors":"Li","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In ACL Workshop: Text Summa- rization Branches Out, pages 74-81, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"ACL Workshop: Text Summarization Branches Out","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2021] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3D object detection via trans- formers. In ICCV, pages 2949-2958, 2021.","title":"Group-free 3D object detection via transformers","authors":"Ze Liu, Zheng Zhang, Yue Cao, Han Hu, Xin Tong","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Luo et al., 2021] Yunpeng Luo, Jiayi Ji, Xiaoshuai Sun, Li- ujuan Cao, Yongjian Wu, Feiyue Huang, Chia-Wen Lin, and Rongrong Ji. Dual-level collaborative transformer for image captioning. In AAAI, volume 35, pages 2286-2293, 2021.","title":"Dual-level collaborative transformer for image captioning","authors":"Yunpeng Luo, Jiayi Ji, Xiaoshuai Sun, Liujuan Cao, Yongjian Wu, Feiyue Huang, Chia-Wen Lin, Rongrong Ji","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, pages 311-318, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, pages 8026-8037, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Qi et al., 2017] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NeurIPS, pages 5099-5108, 2017.","title":"Pointnet++: Deep hierarchical feature learning on point sets in a metric space","authors":"Li Charles R Qi, Hao Yi, Leonidas J Su, Guibas","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Qi et al., 2018] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3D ob- ject detection from RGB-D data. In CVPR, pages 918- 927, 2018.","title":"Frustum pointnets for 3D object detection from RGB-D data","authors":"Wei Charles R Qi, Chenxia Liu, Hao Wu, Leonidas J Su, Guibas","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Qi et al., 2019] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3D object de- tection in point clouds. In ICCV, pages 9277-9286, 2019.","title":"Deep hough voting for 3D object detection in point clouds","authors":"Or Charles R Qi, Kaiming Litany, Leonidas J He, Guibas","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Skubic et al., 2004] Marjorie Skubic, Dennis Perzanowski, Samuel Blisard, Alan Schultz, William Adams, Magda Bugajska, and Derek Brock. Spatial language for human- robot dialogs. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 34(2):154-167, 2004.","title":"Spatial language for humanrobot dialogs","authors":"Marjorie Skubic, Dennis Perzanowski, Samuel Blisard, Alan Schultz, William Adams, Magda Bugajska, Derek Brock","year":"2004","venue":"IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Uppal et al., 2022] Shagun Uppal, Sarthak Bhagat, Deva- manyu Hazarika, Navonil Majumder, Soujanya Poria, Roger Zimmermann, and Amir Zadeh. Multimodal re- search in vision and language: A review of current and emerging trends. Information Fusion, 77:149-171, 2022.","title":"Multimodal research in vision and language: A review of current and emerging trends","authors":"Shagun Uppal, Sarthak Bhagat, Devamanyu Hazarika, Navonil Majumder, Soujanya Poria, Roger Zimmermann, Amir Zadeh","year":"2022","venue":"Information Fusion","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In CVPR, pages 4566-4575, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"Ramakrishna Vedantam, Lawrence Zitnick, Devi Parikh","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Vig, 2019] Jesse Vig. A multiscale visualization of attention in the transformer model. In ACL: System Demonstrations, pages 37-42, 2019.","title":"A multiscale visualization of attention in the transformer model","authors":"Jesse Vig","year":"2019","venue":"ACL: System Demonstrations","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wald et al., 2020] Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3D semantic scene graphs from 3D indoor reconstructions. In CVPR, pages 3961-3970, 2020.","title":"Learning 3D semantic scene graphs from 3D indoor reconstructions","authors":"Johanna Wald, Helisa Dhamo, Nassir Navab, Federico Tombari","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2017] Linjie Yang, Kevin Tang, Jianchao Yang, and Li-Jia Li. Dense captioning with joint inference and visual context. In CVPR, pages 2193-2202, 2017.","title":"Dense captioning with joint inference and visual context","authors":"Linjie Yang, Kevin Tang, Jianchao Yang, Li-Jia Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yin et al., 2019] Guojun Yin, Lu Sheng, Bin Liu, Nenghai Yu, Xiaogang Wang, and Jing Shao. Context and attribute grounded dense captioning. In CVPR, pages 6241-6250, 2019.","title":"Context and attribute grounded dense captioning","authors":"Guojun Yin, Lu Sheng, Bin Liu, Nenghai Yu, Xiaogang Wang, Jing Shao","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2021] Chaoyi Zhang, Jianhui Yu, Yang Song, and Weidong Cai. Exploiting edge-oriented reasoning for 3D point-based scene graph analysis. In CVPR, pages 9705-9715, 2021.","title":"Exploiting edge-oriented reasoning for 3D point-based scene graph analysis","authors":"Chaoyi Zhang, Jianhui Yu, Yang Song, Weidong Cai","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhao et al., 2021] Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3DVG-Transformer: Relation modeling for visual grounding on point clouds. In ICCV, pages 2928- 2937, 2021.","title":"3DVG-Transformer: Relation modeling for visual grounding on point clouds","authors":"Zhao","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}