{"type":"ScholarlyArticle","title":"Joint Multi-view 2D Convolutional Neural Networks for 3D Object Classification","authors":[{"name":"Jinglin Xu","slug":"jinglin-xu","given":"Jinglin","family":"Xu","affiliation":"Northwestern Polytechnical University , Xi'an , China"},{"name":"Xiangsen Zhang","slug":"xiangsen-zhang","given":"Xiangsen","family":"Zhang","affiliation":"Northwestern Polytechnical University , Xi'an , China"},{"name":"Wenbin Li","slug":"wenbin-li","given":"Wenbin","family":"Li","affiliation":"Nanjing University , Nanjing , China"},{"name":"Xinwang Liu","slug":"xinwang-liu","given":"Xinwang","family":"Liu","affiliation":"National University of Defense Technology , Changsha , China"},{"name":"Junwei Han","slug":"junwei-han","given":"Junwei","family":"Han","affiliation":"Northwestern Polytechnical University , Xi'an , China"}],"abstract":"Three-dimensional (3D) object classification is widely involved in various computer vision applications, e.g., autonomous driving, simultaneous localization and mapping, which has attracted lots of attention in the committee. However, solving 3D object classification by directly employing the 3D convolutional neural networks (CNNs) generally suffers from high computational cost. Besides, existing view-based methods cannot better explore the content relationships between views. To this end, this work proposes a novel multi-view framework by jointly using multiple 2D-CNNs to capture discriminative information with relationships as well as a new multi-view loss fusion strategy, in an end-to-end manner. Specifically, we utilize multiple 2D views of a 3D object as input and integrate the intra-view and inter-view information of each view through the view-specific 2D-CNN and a series of modules (outer product, view pair pooling, 1D convolution, and fully connected transformation). Furthermore, we design a novel view ensemble mechanism that selects several discriminative and informative views to jointly infer the category of a 3D object. Extensive experiments demonstrate that the proposed method is able to outperform current state-of-the-art methods on 3D object classification. More importantly, this work provides a new way to improve 3D object classification from the perspective of fully utilizing well-established 2D-CNNs.","doi":"10.24963/ijcai.2020/443","url":"https://ijcai.org/proceedings/2020/443","pdf":"https://ijcai.org/proceedings/2020/0443.pdf","markdown":"https://ijcai.org/proceedings/2020/443.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/443","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":443,"pages":{"first":3202,"last":3208},"keywords":["Machine Learning: Multi-instance;Multi-label;Multi-view learning","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Brock et al., 2016] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint arXiv:1608.04236, 2016.","title":"Generative and discriminative voxel modeling with convolutional neural networks","authors":"Andrew Brock, Theodore Lim, James M Ritchie, Nick Weston","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chatfield et al., 2014] Ken Chatfield, Karen Simonyan, An- drea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. arXiv preprint arXiv:1405.3531, 2014.","title":"Return of the devil in the details: Delving deep into convolutional nets","authors":"Ken Chatfield, Karen Simonyan, Andrea Vedaldi, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Feng et al., 2018] Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao. Gvcnn: Group-view convolu- tional neural networks for 3d shape recognition. In CVPR, pages 264-272, 2018.","title":"Gvcnn: Group-view convolutional neural networks for 3d shape recognition","authors":"Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, Yue Gao","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Guo et al., 2013] Yulan Guo, Ferdous Sohel, Mohammed Bennamoun, Min Lu, and Jianwei Wan. Rotational pro- jection statistics for 3d local surface description and object recognition. IJCV, 105(1):63-86, 2013.","title":"Rotational projection statistics for 3d local surface description and object recognition","authors":"Yulan Guo, Ferdous Sohel, Mohammed Bennamoun, Min Lu, Jianwei Wan","year":"2013","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Guo et al., 2016] Yulan Guo, Mohammed Bennamoun, Fer- dous Sohel, Min Lu, Jianwei Wan, and Ngai Ming Kwok. A comprehensive performance evaluation of 3d local fea- ture descriptors. IJCV, 116(1):66-89, 2016.","title":"A comprehensive performance evaluation of 3d local feature descriptors","authors":"Yulan Guo, Mohammed Bennamoun, Ferdous Sohel, Min Lu, Jianwei Wan, Ming Ngai, Kwok","year":"2016","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hegde and Zadeh, 2016] Vishakh Hegde and Reza Zadeh. Fusionnet: 3d object classification using multiple data rep- resentations. arXiv preprint arXiv:1607.05695, 2016.","title":"Fusionnet: 3d object classification using multiple data representations","authors":"Zadeh Hegde, Vishakh Hegde, Reza Zadeh","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Johns et al., 2016] Edward Johns, Stefan Leutenegger, and Andrew J Davison. Pairwise decomposition of image se- quences for active multi-view recognition. In CVPR, pages 3813-3822, 2016.","title":"Pairwise decomposition of image sequences for active multi-view recognition","authors":"Edward Johns, Stefan Leutenegger, Andrew J Davison","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kanezaki et al., 2018] Asako Kanezaki, Yasuyuki Mat- sushita, and Yoshifumi Nishida. Rotationnet: Joint ob- ject categorization and pose estimation using multiviews from unsupervised viewpoints. In CVPR, pages 5010- 5019, 2018.","title":"Rotationnet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints","authors":"Asako Kanezaki, Yasuyuki Matsushita, Yoshifumi Nishida","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kazhdan et al., 2003] Michael Kazhdan, Thomas Funkhouser, and Szymon Rusinkiewicz. Rotation invariant spherical harmonic representation of 3 d shape descriptors. In SGP, pages 156-164, 2003.","title":"Rotation invariant spherical harmonic representation of 3 d shape descriptors","authors":"Thomas Michael Kazhdan, Szymon Funkhouser, Rusinkiewicz","year":"2003","venue":"SGP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Klokov and Lempitsky, 2017] Roman Klokov and Victor Lempitsky. Escape from cells: Deep kd-networks for the recognition of 3d point cloud models. In ICCV, pages 863- 872, 2017.","title":"Roman Klokov and Victor Lempitsky. Escape from cells: Deep kd-networks for the recognition of 3d point cloud models","authors":"Lempitsky Klokov","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2018] Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In CVPR, pages 9397-9406, 2018.","title":"So-net: Self-organizing network for point cloud analysis","authors":"Jiaxin Li, Ben M Chen, Gim Hee, Lee","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maturana and Scherer, 2015] Daniel Maturana and Sebas- tian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In IROS, pages 922-928, 2015.","title":"Voxnet: A 3d convolutional neural network for real-time object recognition","authors":"Scherer Maturana, Daniel Maturana, Sebastian Scherer","year":"2015","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Minto et al., 2018] Ludovico Minto, Pietro Zanuttigh, and Giampaolo Pagnutti. Deep learning for 3d shape classi- fication based on volumetric density and surface approxi- mation clues. In VISIGRAPP, pages 317-324, 2018.","title":"Deep learning for 3d shape classification based on volumetric density and surface approximation clues","authors":"Minto","year":"2018","venue":"VISIGRAPP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Paletta and Pinz, 2000] Lucas Paletta and Axel Pinz. Active object recognition by view integration and reinforcement learning. RAS, 31(1-2):71-86, 2000.","title":"Active object recognition by view integration and reinforcement learning","authors":"Pinz Paletta, Lucas Paletta, Axel Pinz","year":"2000","venue":"RAS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qi et al., 2016] Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. Vol- umetric and multi-view cnns for object classification on 3d data. In CVPR, pages 5648-5656, 2016.","title":"Volumetric and multi-view cnns for object classification on 3d data","authors":"Hao Charles R Qi, Matthias Su, Angela Nießner, Mengyuan Dai, Leonidas J Yan, Guibas","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qi et al., 2017] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pages 652-660, 2017.","title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","authors":"Hao Charles R Qi, Kaichun Su, Leonidas J Mo, Guibas","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ren et al., 2017] Mengwei Ren, Liang Niu, and Yi Fang. 3d-a-nets: 3d deep dense descriptor for volumet- ric shapes with adversarial networks. arXiv preprint arXiv:1711.10108, 2017.","title":"3d-a-nets: 3d deep dense descriptor for volumetric shapes with adversarial networks","authors":"Ren","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Su et al., 2015] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view con- volutional neural networks for 3d shape recognition. In ICCV, pages 945-953, 2015.","title":"Multi-view convolutional neural networks for 3d shape recognition","authors":"Su","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2019] Chu Wang, Marcello Pelillo, and Kaleem Siddiqi. Dominant set clustering and pooling for multi-view 3d object recognition. arXiv preprint arXiv:1906.01592, 2019.","title":"Dominant set clustering and pooling for multi-view 3d object recognition","authors":"Chu Wang, Marcello Pelillo, Kaleem Siddiqi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2014] Zhirong Wu, Shuran Song, Aditya Khosla, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets for 2.5 d object recognition and next-best-view prediction. arXiv preprint arXiv:1406.5670, 2014.","title":"3d shapenets for 2.5 d object recognition and next-best-view prediction","authors":"Zhirong Wu, Shuran Song, Aditya Khosla, Xiaoou Tang, Jianxiong Xiao","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2015] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR, pages 1912-1920, 2015.","title":"3d shapenets: A deep representation for volumetric shapes","authors":"Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, Jianxiong Xiao","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yavartanoo et al., 2018] Mohsen Yavartanoo, Eu Young Kim, and Kyoung Mu Lee. Spnet: Deep 3d object clas- sification and retrieval using stereographic projection. In ACCV, pages 691-706, 2018.","title":"Spnet: Deep 3d object classification and retrieval using stereographic projection","authors":"Mohsen Yavartanoo, Young Eu, Kyoung Kim, Lee Mu","year":"2018","venue":"ACCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}