{"type":"ScholarlyArticle","title":"Parts4Feature: Learning 3D Global Features from Generally Semantic Parts in Multiple Views","authors":[{"name":"Zhizhong Han","slug":"zhizhong-han","given":"Zhizhong","family":"Han","affiliation":"School of Software , Tsinghua University , Beijing , China; Department of Computer Science , University of Maryland , College Park , USA"},{"name":"Xinhai Liu","slug":"xinhai-liu","given":"Xinhai","family":"Liu","affiliation":"School of Software , Tsinghua University , Beijing , China; Beijing National Research Center for Information Science and Technology (BNRist )"},{"name":"Yu-Shen Liu","slug":"yu-shen-liu","given":"Yu-Shen","family":"Liu","affiliation":"School of Software , Tsinghua University , Beijing , China; Beijing National Research Center for Information Science and Technology (BNRist )"},{"name":"Matthias Zwicker","slug":"matthias-zwicker","given":"Matthias","family":"Zwicker","affiliation":"Department of Computer Science , University of Maryland , College Park , USA"}],"abstract":"Deep learning has achieved remarkable results in 3D shape analysis by learning global shape features from the pixel-level over multiple views. Previous methods, however, compute low-level features for entire views without considering part-level information. In contrast, we propose a deep neural network, called Parts4Feature, to learn 3D global features from part-level information in multiple views. We introduce a novel definition of generally semantic parts, which Parts4Feature learns to detect in multiple views from different 3D shape segmentation benchmarks. A key idea of our architecture is that it transfers the ability to detect semantically meaningful parts in multiple views to learn 3D global features. Parts4Feature achieves this by combining a local part detection branch and a global feature learning branch with a shared region proposal module. The global feature learning branch aggregates the detected parts in terms of learned part patterns with a novel multi-attention mechanism, while the region proposal module enables locally and globally discriminative information to be promoted by each other. We demonstrate that Parts4Feature outperforms the state-of-the-art under three large-scale 3D shape benchmarks.","doi":"10.24963/ijcai.2019/108","url":"https://ijcai.org/proceedings/2019/108","pdf":"https://ijcai.org/proceedings/2019/0108.pdf","markdown":"https://ijcai.org/proceedings/2019/108.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/108","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Computer Vision","track":"Main track","number":108,"pages":{"first":766,"last":773},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: 2D and 3D Computer Vision"],"videos":[],"references":[{"position":1,"raw":"Panos Achlioptas et al. Learn- ing representations and generative models for 3D point clouds. In The International Conference on Machine Learning, pages 40-49, 2018.","title":"Learning representations and generative models for 3D point clouds","authors":"Panos Achlioptas","year":"2018","venue":"The International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bai and others, 2017] Song Bai et al. GIFT: Towards scal- able 3D shape retrieval. IEEE Transaction on Multimedia, 19(6):1257-1271, 2017.","title":"GIFT: Towards scalable 3D shape retrieval","authors":"Song Bai, Bai","year":"2017","venue":"IEEE Transaction on Multimedia","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brock et al., 2016] Andrew Brock, Theodore Lim, J.M. Ritchie, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. In 3D deep learning workshop (NIPS), 2016.","title":"Generative and discriminative voxel modeling with convolutional neural networks","authors":"Andrew Brock, Theodore Lim, J M Ritchie, Nick Weston","year":"2016","venue":"3D deep learning workshop (NIPS)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cao et al., 2017] Zhangjie Cao, Qixing Huang, and Karthik Ramani. 3D object classification via spherical projections. In International Conference on 3D Vision. 2017.","title":"3D object classification via spherical projections","authors":"Cao","year":"2017","venue":"International Conference on 3D Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chatfield et al., 2014] Ken Chatfield, Karen Simonyan, An- drea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets. In British Machine Vision Conference, 2014.","title":"Return of the devil in the details: Delving deep into convolutional nets","authors":"Ken Chatfield, Karen Simonyan, Andrea Vedaldi, Andrew Zisserman","year":"2014","venue":"British Machine Vision Conference","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cohen et al., 2018] Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. In Interna- tional Conference on Learning Representations, 2018.","title":"Spherical CNNs","authors":"S Taco, Mario Cohen, Jonas Geiger, Max Köhler, Welling","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Girshick, 2015] Ross Girshick. Fast R-CNN. In IEEE In- ternational Conference on Computer Vision, pages 1440- 1448, 2015.","title":"Fast R-CNN","authors":"Ross Girshick","year":"2015","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Han and others, 2016] Zhizhong Han et al. Unsupervised 3D local feature learning by circle convolutional restricted boltzmann machine. IEEE Transactions on Image Pro- cessing, 25(11):5331-5344, 2016.","title":"Unsupervised 3D local feature learning by circle convolutional restricted boltzmann machine","authors":"Zhizhong Han","year":"2016","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han and others, 2017] Zhizhong Han et al. Mesh convo- lutional restricted boltzmann machines for unsupervised learning of features with structure preservation on 3D meshes. IEEE Transactions on Neural Network and Learning Systems, 28(10):2268 -2281, 2017.","title":"Mesh convolutional restricted boltzmann machines for unsupervised learning of features with structure preservation on 3D meshes","authors":"Others ; Zhizhong Han, Han","year":"2017","venue":"IEEE Transactions on Neural Network and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Han and others, 2019a] Zhizhong Han et al. Seqviews 2se- qlabels: Learning 3D global features via aggregating se- quential views by rnn with attention. IEEE Transactions on Image Processing, 28(2):1941-0042, 2019.","title":"Seqviews 2seqlabels: Learning 3D global features via aggregating sequential views by rnn with attention","authors":"Others ; Zhizhong Han, Han","year":"2019","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han and others, 2019b] Zhizhong Han et al. View inter- prediction gan: Unsupervised representation learning for 3D shapes by learning global shape memories to support local view predictions. In AAAI, 2019.","title":"View interprediction gan: Unsupervised representation learning for 3D shapes by learning global shape memories to support local view predictions","authors":"Others ; Zhizhong Han, Han","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Han et al., 2019] Zhizhong Han, Mingyang Shang, Xiyang Wang, Yu-Shen Liu, and Matthias Zwicker. Y2seq2seq: Cross-modal representation learning for 3D shape and text by joint reconstruction and prediction of view and word sequences. In AAAI, 2019.","title":"Y2seq2seq: Cross-modal representation learning for 3D shape and text by joint reconstruction and prediction of view and word sequences","authors":"Zhizhong Han, Mingyang Shang, Xiyang Wang, Yu-Shen Liu, Matthias Zwicker","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2018] Xinwei He, Yang Zhou, Zhichao Zhou, Song Bai, and Xiang Bai. Triplet-center loss for multi- view 3D object retrieval. In The IEEE Conference on Com- puter Vision and Pattern Recognition, 2018.","title":"Triplet-center loss for multiview 3D object retrieval","authors":"Xinwei He, Yang Zhou, Zhichao Zhou, Song Bai, Xiang Bai","year":"2018","venue":"The IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Johns et al., 2016] Edward Johns, Stefan Leutenegger, and Andrew J. Davison. Pairwise decomposition of image sequences for active multi-view recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3813-3822, 2016.","title":"Pairwise decomposition of image sequences for active multi-view recognition","authors":"Edward Johns, Stefan Leutenegger, Andrew J Davison","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kalogerakis and others, 2017] Evangelos Kalogerakis et al. 3D shape segmentation with projective convolutional net- works. In IEEE Conference on Computer Vision and Pat- tern Recognition, pages 6630-6639, 2017.","title":"3D shape segmentation with projective convolutional networks","authors":"Evangelos Kalogerakis","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kanezaki et al., 2018] Asako Kanezaki, Yasuyuki Mat- sushita, and Yoshifumi Nishida. Rotationnet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints. In IEEE Conference on Com- puter Vision and Pattern Recognition, 2018.","title":"Rotationnet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints","authors":"Asako Kanezaki, Yasuyuki Matsushita, Yoshifumi Nishida","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Muralikrishnan et al., 2018] Sanjeev Muralikrishnan, Vladimir G. Kim, and Siddhartha Chaudhuri. Tags2parts: Discovering semantic regions from shape tags. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2926-2935, 2018.","title":"Tags2parts: Discovering semantic regions from shape tags","authors":"Vladimir G Sanjeev Muralikrishnan, Siddhartha Kim, Chaudhuri","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qi and others, 2017] Charles Qi et al. Pointnet++: Deep hi- erarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems, pages 5105-5114, 2017.","title":"Pointnet++: Deep hierarchical feature learning on point sets in a metric space","authors":"Charles Qi","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qi et al., 2016] C R Qi, H Su, and M Niebner. Volumetric and multi-view cnns for object classification on 3D data. In IEEE Conference on Computer Vision and Pattern Recog- nition, pages 5648-5656, 2016.","title":"Volumetric and multi-view cnns for object classification on 3D data","authors":"H C R Qi, M Su, Niebner","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ren and others, 2015] Shaoqing Ren et al. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, pages 91-99, 2015.","title":"Faster R-CNN: Towards real-time object detection with region proposal networks","authors":"Ren, Shaoqing Ren","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Savva and others, 2017] Manolis Savva et al. SHREC'17 Large-Scale 3D Shape Retrieval from ShapeNet Core55. In Eurographics Workshop on 3D Object Retrieval, 2017.","title":"SHREC'17 Large-Scale 3D Shape Retrieval from ShapeNet Core55","authors":"Manolis Savva","year":"2017","venue":"Eurographics Workshop on 3D Object Retrieval","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sfikas and others, 2017] Konstantinos Sfikas et al. Exploit- ing the PANORAMA Representation for Convolutional Neural Network Classification and Retrieval. In EG Work- shop on 3D Object Retrieval, pages 1-7, 2017.","title":"Exploiting the PANORAMA Representation for Convolutional Neural Network Classification and Retrieval","authors":"Konstantinos Sfikas","year":"2017","venue":"EG Workshop on 3D Object Retrieval","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sharma et al., 2016] Abhishek Sharma, Oliver Grau, and Mario Fritz. VConv-DAE: Deep volumetric shape learning without object labels. In Proceedings of European Confer- ence on Computer Vision, pages 236-250, 2016.","title":"VConv-DAE: Deep volumetric shape learning without object labels","authors":"Abhishek Sharma, Oliver Grau, Mario Fritz","year":"2016","venue":"Proceedings of European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shi and others, 2015] B. Shi et al. Deeppano: Deep panoramic representation for 3D shape recognition. IEEE Signal Processing Letters, 22(12):2339-2343, 2015.","title":"Deeppano: Deep panoramic representation for 3D shape recognition","authors":"B Shi","year":"2015","venue":"IEEE Signal Processing Letters","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Su and others, 2015] Hang Su et al. Multi-view convolu- tional neural networks for 3D shape recognition. In Inter- national Conference on Computer Vision, pages 945-953, 2015.","title":"Multi-view convolutional neural networks for 3D shape recognition","authors":"Hang Su, Su","year":"2015","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2017] Chu Wang, Marcello Pelillo, and Kaleem Siddiqi. Dominant set clustering and pooling for multi-view 3D object recognition. In Proceedings of British Machine Vision Conference, 2017.","title":"Dominant set clustering and pooling for multi-view 3D object recognition","authors":"Chu Wang, Marcello Pelillo, Kaleem Siddiqi","year":"2017","venue":"Proceedings of British Machine Vision Conference","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wu and others, 2015] Zhirong Wu et al. 3D ShapeNets: A deep representation for volumetric shapes. In Proceed- ings of IEEE Conference on Computer Vision and Pattern Recognition, pages 1912-1920, 2015.","title":"3D ShapeNets: A deep representation for volumetric shapes","authors":"Zhirong Wu","year":"2015","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wu and others, 2016] Jiajun Wu et al. Learning a proba- bilistic latent space of object shapes via 3D generative- adversarial modeling. In Advances in Neural Information Processing Systems, pages 82-90, 2016.","title":"Learning a probabilistic latent space of object shapes via 3D generativeadversarial modeling","authors":"Jiajun Wu","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xie and others, 2018] Jianwen Xie et al. Learning descrip- tor networks for 3D shape synthesis and analysis. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.","title":"Learning descriptor networks for 3D shape synthesis and analysis","authors":"Jianwen Xie","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang et al., 2018] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.","title":"Foldingnet: Point cloud auto-encoder via deep grid deformation","authors":"Yaoqing Yang, Chen Feng, Yiru Shen, Dong Tian","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}