{"type":"ScholarlyArticle","title":"Hierarchical Instance Feature Alignment for 2D Image-Based 3D Shape Retrieval","authors":[{"name":"Heyu Zhou","slug":"heyu-zhou","given":"Heyu","family":"Zhou","affiliation":"School of Electrical and Information Engineering , Tianjin University , China zhy"},{"name":"Weizhi Nie","slug":"weizhi-nie","given":"Weizhi","family":"Nie","affiliation":"School of Electrical and Information Engineering , Tianjin University , China zhy"},{"name":"Wenhui Li","slug":"wenhui-li","given":"Wenhui","family":"Li","affiliation":"School of Electrical and Information Engineering , Tianjin University , China zhy"},{"name":"Dan Song","slug":"dan-song","given":"Dan","family":"Song","affiliation":"School of Electrical and Information Engineering , Tianjin University , China zhy"},{"name":"An-An Liu","slug":"an-an-liu","given":"An-An","family":"Liu","affiliation":"School of Electrical and Information Engineering , Tianjin University , China zhy"}],"abstract":"2D image-based 3D shape retrieval has become a hot research topic since its wide industrial applications and academic significance. However, existing view-based 3D shape retrieval methods are restricted by two settings,  1) learn the common-class features  while neglecting the instance visual characteristics,  2) narrow the global domain variations while ignoring the local semantic variations in each category. To overcome these problems, we propose a novel hierarchical instance feature alignment (HIFA) method for this task. HIFA consists of two modules, cross-modal instance feature learning and hierarchical instance feature alignment. Specifically,  we first use CNN to extract both 2D image and multi-view features. Then, we maximize the mutual information between the input data and the high-level feature to preserve as much as visual characteristics of an individual instance. To mix up the features in two domains, we enforce feature alignment considering both global domain and local semantic levels. By narrowing the global domain variations we impose the identical large norm restriction on both 2D and 3D feature-norm expectations to facilitate more transferable possibility. By narrowing the local variations we propose to minimize the distance between two centroids of the same class from different domains to obtain semantic consistency. Extensive experiments on two popular and novel datasets, MI3DOR and MI3DOR-2, validate the superiority of HIFA for 2D image-based 3D shape retrieval task.","doi":"10.24963/ijcai.2020/117","url":"https://ijcai.org/proceedings/2020/117","pdf":"https://ijcai.org/proceedings/2020/0117.pdf","markdown":"https://ijcai.org/proceedings/2020/117.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/117","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":117,"pages":{"first":839,"last":845},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Computer Vision: 2D and 3D Computer Vision","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24526/file_24526.m3u8","length_sec":666,"provider":"hls"}],"references":[{"position":1,"raw":"[Belghazi et al., 2018] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R. Devon Hjelm, and Aaron C. Courville. Mutual infor- mation neural estimation. In ICML, pages 530-539, 2018.","title":"Mutual information neural estimation","authors":"Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R Devon Hjelm, Aaron C Courville","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Borgwardt et al., 2006] Karsten M. Borgwardt, Arthur Gretton, Malte J. Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alexander J. Smola. Integrating structured biological data by kernel maximum mean discrepancy. In ISMB, pages 49-57, 2006.","title":"Integrating structured biological data by kernel maximum mean discrepancy","authors":"Karsten M Borgwardt, Arthur Gretton, Malte J Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, Alexander J Smola","year":"2006","venue":"ISMB","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cheng et al., 2018] Zhiyong Cheng, Ying Ding, Xiangnan He, Lei Zhu, Xuemeng Song, and Mohan S. Kankanhalli. A 3ncf: An adaptive aspect attention model for rating pre- diction. In IJCAI, pages 3748-3754, 2018.","title":"A 3ncf: An adaptive aspect attention model for rating prediction","authors":"Zhiyong Cheng, Ying Ding, Xiangnan He, Lei Zhu, Xuemeng Song, Mohan S Kankanhalli","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/521"},{"position":4,"raw":"[Donsker and Varadhan, 1975] Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, i. Math. Pures. Appl., 28:1-47, 1975.","title":"Asymptotic evaluation of certain markov process expectations for large time","authors":"D Monroe, Donsker, Sr Srinivasa Varadhan","year":"1975","venue":"i. Math. Pures. Appl","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Feng et al., 2018] Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao. GVCNN: group-view con- volutional neural networks for 3d shape recognition. In CVPR, pages 264-272, 2018.","title":"GVCNN: group-view convolutional neural networks for 3d shape recognition","authors":"Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, Yue Gao","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ganin and Lempitsky, 2015] Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backprop- agation. In ICML, pages 1180-1189, 2015.","title":"Unsupervised domain adaptation by backpropagation","authors":"Yaroslav Ganin, Victor S Lempitsky","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hjelm et al., 2019] R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep rep- resentations by mutual information estimation and maxi- mization. In ICLR, 2019.","title":"Learning deep representations by mutual information estimation and maximization","authors":"R, Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, Yoshua Bengio","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hong et al., 2016] Richang Hong, Zhenzhen Hu, Ruxin Wang, Meng Wang, and Dacheng Tao. Multi-view object retrieval via multi-scale topic models. IEEE Trans. Image Processing, 25(12):5814-5827, 2016.","title":"Multi-view object retrieval via multi-scale topic models","authors":"Hong","year":"2016","venue":"IEEE Trans. Image Processing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1106- 1114, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and et al., 2012] Bo Li and Tobias Schreck et al. Shrec'12 track: Sketch-based 3d shape retrieval. In 3DOR, pages 109-118, 2012.","title":"Shrec'12 track: Sketch-based 3d shape retrieval","authors":"Li","year":"2012","venue":"3DOR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2018] Anan Liu, Shu Xiang, Wenhui Li, Weizhi Nie, and Yuting Su. Cross-domain 3d model retrieval via visual domain adaption. In IJCAI, pages 828-834, 2018.","title":"Cross-domain 3d model retrieval via visual domain adaption","authors":"Anan Liu, Shu Xiang, Wenhui Li, Weizhi Nie, Yuting Su","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/115"},{"position":12,"raw":"[Long et al., 2013] Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S. Yu. Transfer feature learning with joint distribution adaptation. In ICCV, pages 2200-2207, 2013.","title":"Transfer feature learning with joint distribution adaptation","authors":"Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, Philip S Yu","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Long et al., 2017] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adaptation networks. In ICML, pages 2208-2217, 2017.","title":"Deep transfer learning with joint adaptation networks","authors":"Mingsheng Long, Han Zhu, Jianmin Wang, Michael I Jordan","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lu et al., 2019a] Xu Lu, Lei Zhu, Zhiyong Cheng, Jingjing Li, Xiushan Nie, and Huaxiang Zhang. Flexible online multi-modal hashing for large-scale multimedia retrieval. In MM, pages 1129-1137, 2019.","title":"Flexible online multi-modal hashing for large-scale multimedia retrieval","authors":"Lu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lu et al., 2019b] Xu Lu, Lei Zhu, Zhiyong Cheng, Liqiang Nie, and Huaxiang Zhang. Online multi-modal hashing with dynamic query-adaption. In SIGIR, pages 715-724, 2019.","title":"Online multi-modal hashing with dynamic query-adaption","authors":"Lu","year":"2019","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605, 2008.","title":"Laurens van der Maaten and Geoffrey Hinton","authors":"Hinton Maaten","year":"2008","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Maturana and Scherer, 2015] Daniel Maturana and Sebas- tian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In IROS, pages 922-928, 2015.","title":"Voxnet: A 3d convolutional neural network for real-time object recognition","authors":"Scherer Maturana, Daniel Maturana, Sebastian Scherer","year":"2015","venue":"IROS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Phong, 1975] Bui Tuong Phong. Illumination for computer generated pictures. Commun. ACM, 18(6):311-317, 1975.","title":"Illumination for computer generated pictures","authors":"Phong Bui Tuong","year":"1975","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qi et al., 2017a] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pages 77-85, 2017.","title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","authors":"Qi","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qi et al., 2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NIPS, pages 5105-5114, 2017.","title":"Pointnet++: Deep hierarchical feature learning on point sets in a metric space","authors":"Qi","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Su et al., 2015] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller. Multi-view con- volutional neural networks for 3d shape recognition. In ICCV, pages 945-953, 2015.","title":"Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition","authors":"Su","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2018] Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S. Yu. Visual domain adaptation with manifold embedded distribution alignment. In ACM MM, pages 402-410, 2018.","title":"Visual domain adaptation with manifold embedded distribution alignment","authors":"Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, Philip S Yu","year":"2018","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xie et al., 2018] Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. Learning semantic representations for unsu- pervised domain adaptation. In ICML, pages 5423-5432, 2018.","title":"Learning semantic representations for unsupervised domain adaptation","authors":"Shaoan Xie, Zibin Zheng, Liang Chen, Chuan Chen","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2019] Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adapta- tion. In ICCV, 2019.","title":"Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation","authors":"Xu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2017] Jing Zhang, Wanqing Li, and Philip Ogunbona. Joint geometrical and statistical alignment for visual domain adaptation. In CVPR, pages 5150-5158, 2017.","title":"Joint geometrical and statistical alignment for visual domain adaptation","authors":"Jing Zhang, Wanqing Li, Philip Ogunbona","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2019a] Heyu Zhou, An-An Liu, and Weizhi Nie. Dual-level embedding alignment network for 2d image-based 3d object retrieval. In ACM MM, pages 1667- 1675, 2019.","title":"An-An Liu, and Weizhi Nie. Dual-level embedding alignment network for 2d image-based 3d object retrieval","authors":"Zhou","year":"2019","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2019b] Heyu Zhou, An-An Liu, Weizhi Nie, and Jie Nie. Multi-view saliency guided deep neural net- work for 3d object retrieval and classification. IEEE Trans. Multimedia, 2019.","title":"Multi-view saliency guided deep neural network for 3d object retrieval and classification","authors":"Zhou","year":"2019","venue":"IEEE Trans. Multimedia","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}