{"type":"ScholarlyArticle","title":"Siamese CNN-BiLSTM Architecture for 3D Shape Representation Learning","authors":[{"name":"Guoxian Dai","slug":"guoxian-dai","given":"Guoxian","family":"Dai","affiliation":"NYU Multimedia and Visual Computing Lab; Dept. of ECE , NYU Abu Dhabi , UAE; Dept. of CSE , NYU Tandon School of Engineering , USA"},{"name":"Jin Xie","slug":"jin-xie","given":"Jin","family":"Xie","affiliation":"NYU Multimedia and Visual Computing Lab; Dept. of ECE , NYU Abu Dhabi , UAE"},{"name":"Yi Fang","slug":"yi-fang","given":"Yi","family":"Fang","affiliation":"NYU Multimedia and Visual Computing Lab; Dept. of ECE , NYU Abu Dhabi , UAE; Dept. of ECE , NYU Tandon School of Engineering , USA"}],"abstract":"Learning a 3D shape representation from a collection of its rendered 2D images has been extensively studied. However, existing view-based techniques have not yet fully exploited the information among all the views of projections. In this paper, by employing recurrent neural network to efficiently capture features across different views, we propose a siamese CNN-BiLSTM network for 3D shape representation learning. The proposed method minimizes a discriminative loss function to learn a deep nonlinear transformation, mapping 3D shapes from the original space into a nonlinear feature space. In the transformed space, the distance of 3D shapes with the same label is minimized, otherwise the distance is maximized to a large margin. Specifically, the 3D shapes are first projected into a group of 2D images from different views. Then convolutional neural network (CNN) is adopted to extract features from different view images, followed by a bidirectional long short-term memory (LSTM) to aggregate information across different views. Finally, we construct the whole CNN-BiLSTM network into a siamese structure with contrastive loss function. Our proposed method is evaluated on two benchmarks, ModelNet40 and SHREC 2014, demonstrating superiority over the state-of-the-art methods.","doi":"10.24963/ijcai.2018/93","url":"https://ijcai.org/proceedings/2018/93","pdf":"https://ijcai.org/proceedings/2018/0093.pdf","markdown":"https://ijcai.org/proceedings/2018/93.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/93","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":93,"pages":{"first":670,"last":676},"keywords":["Computer Vision: 2D and 3D Computer Vision","Computer Vision: Computer Vision"],"videos":[],"references":[{"position":1,"raw":"[Bai et al., 2016] Song Bai, Xiang Bai, Zhichao Zhou, Zhaoxiang Zhang, and Longin Jan Latecki. Gift: A real- time and scalable 3d shape search engine. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.","title":"Gift: A realtime and scalable 3d shape search engine","authors":"Song Bai, Xiang Bai, Zhichao Zhou, Zhaoxiang Zhang, Longin, Jan Latecki","year":"2016","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bronstein and Kokkinos, 2010] Michael M Bronstein and Iasonas Kokkinos. Scale-invariant heat kernel signatures for non-rigid shape recognition. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 1704-1711. IEEE, 2010.","title":"Scale-invariant heat kernel signatures for non-rigid shape recognition","authors":"M Michael, Iasonas Bronstein, Kokkinos","year":"2010","venue":"Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bu et al., 2014] Shuhui Bu, Zhenbao Liu, Junwei Han, Jun Wu, and Rongrong Ji. Learning high-level feature by deep belief networks for 3-d model retrieval and recogni- tion. IEEE Transactions on Multimedia, 16(8):2154-2167, 2014.","title":"Learning high-level feature by deep belief networks for 3-d model retrieval and recognition","authors":"Shuhui Bu, Zhenbao Liu, Junwei Han, Jun Wu, Rongrong Ji","year":"2014","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2003] Ding-Yun Chen, Xiao-Pei Tian, Yu-Te Shen, and Ming Ouhyoung. On visual similarity based 3d model retrieval. In Computer graphics forum, volume 22, pages 223-232. Wiley Online Library, 2003.","title":"On visual similarity based 3d model retrieval","authors":"Ding-Yun Chen, Xiao-Pei Tian, Yu-Te Shen, Ming Ouhyoung","year":"2003","venue":"Computer graphics forum","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chopra et al., 2005] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, volume 1, pages 539-546. IEEE, 2005.","title":"Learning a similarity metric discriminatively, with application to face verification","authors":"Sumit Chopra, Raia Hadsell, Yann Lecun","year":"2005","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv preprint arXiv:1512.03385, 2015.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hoffer and Ailon, 2015] Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In International Workshop on Similarity-Based Pattern Recognition, pages 84-92. Springer, 2015.","title":"Deep metric learning using triplet network","authors":"Elad Hoffer, Nir Ailon","year":"2015","venue":"International Workshop on Similarity-Based Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2014] Junlin Hu, Jiwen Lu, and Yap-Peng Tan. Discriminative deep metric learning for face verification in the wild. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pages 1875-1882, 2014.","title":"Discriminative deep metric learning for face verification in the wild","authors":"Hu","year":"2014","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kazhdan et al., 2003] Michael Kazhdan, Thomas Funkhouser, and Szymon Rusinkiewicz. Rotation invariant spherical harmonic representation of 3 d shape descriptors. In Symposium on geometry processing, volume 6, pages 156-164, 2003.","title":"Rotation invariant spherical harmonic representation of 3 d shape descriptors","authors":"Thomas Michael Kazhdan, Szymon Funkhouser, Rusinkiewicz","year":"2003","venue":"Symposium on geometry processing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li and Lu, et al., 2014] Bo Li and Yijuan Lu, et al. SHREC'14 track: Large scale comprehensive retrieval track benchmark. In Eurographics Workshop on 3D Object Retrieval, Strasbourg, France, 2014.","title":"SHREC'14 track: Large scale comprehensive retrieval track benchmark","authors":"Lu Li","year":"2014","venue":"Eurographics Workshop on 3D Object Retrieval","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li and Tang, 2015] Zechao Li and Jinhui Tang. Weakly su- pervised deep metric learning for community-contributed image retrieval. IEEE Transactions on Multimedia, 17(11):1989-1999, 2015.","title":"Weakly supervised deep metric learning for community-contributed image retrieval","authors":"Li, Tang ; ] Zechao Li, Jinhui Tang","year":"2015","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Litman et al., 2014] Roee Litman, Alex Bronstein, Michael Bronstein, and Umberto Castellani. Supervised learning of bag-of-features shape descriptors using sparse coding. In Computer Graphics Forum, volume 33, pages 127-136. Wiley Online Library, 2014.","title":"Supervised learning of bag-of-features shape descriptors using sparse coding","authors":"Litman","year":"2014","venue":"Computer Graphics Forum","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maturana and Scherer, 2015] Daniel Maturana and Sebas- tian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on, pages 922-928. IEEE, 2015.","title":"Voxnet: A 3d convolutional neural network for real-time object recognition","authors":"Scherer Maturana, Daniel Maturana, Sebastian Scherer","year":"2015","venue":"Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Oh Song et al., 2016] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 4004-4012, 2016.","title":"Deep metric learning via lifted structured feature embedding","authors":"Hyun Oh Song, Yu Xiang, Stefanie Jegelka, Silvio Savarese","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Qi et al., 2016] Charles R Qi, Hao Su, Matthias Niessner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. Vol- umetric and multi-view cnns for object classification on 3d data. arXiv preprint arXiv:1604.03265, 2016.","title":"Volumetric and multi-view cnns for object classification on 3d data","authors":"Hao Charles R Qi, Matthias Su, Angela Niessner, Mengyuan Dai, Leonidas J Yan, Guibas","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Ima- geNet Large Scale Visual Recognition Challenge. Inter- national Journal of Computer Vision (IJCV), 115(3):211- 252, 2015.","title":"Ima-geNet Large Scale Visual Recognition Challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, Li Fei-Fei","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shilane et al., 2004] Philip Shilane, Patrick Min, Michael Kazhdan, and Thomas Funkhouser. The princeton shape benchmark. In Shape modeling applications, 2004. Pro- ceedings, pages 167-178. IEEE, 2004.","title":"The princeton shape benchmark","authors":"Philip Shilane, Patrick Min, Michael Kazhdan, Thomas Funkhouser","year":"2004","venue":"Shape modeling applications","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Su et al., 2015] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convo- lutional neural networks for 3d shape recognition. In Pro- ceedings of the IEEE International Conference on Com- puter Vision, pages 945-953, 2015.","title":"Multi-view convolutional neural networks for 3d shape recognition","authors":"Su","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sun et al., 2009] Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In Computer graph- ics forum, volume 28, pages 1383-1392. Wiley Online Li- brary, 2009.","title":"A concise and provably informative multi-scale signature based on heat diffusion","authors":"Jian Sun, Maks Ovsjanikov, Leonidas Guibas","year":"2009","venue":"Computer graphics forum","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tatsuma et al., 2012] Atsushi Tatsuma, Hitoshi Koyanagi, and Masaki Aono. A large-scale shape benchmark for 3d object retrieval: Toyohashi shape benchmark. In Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific, pages 1-10. IEEE, 2012.","title":"A large-scale shape benchmark for 3d object retrieval: Toyohashi shape benchmark","authors":"Atsushi Tatsuma, Hitoshi Koyanagi, Masaki Aono","year":"2012","venue":"Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2015] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.","title":"3d shapenets: A deep representation for volumetric shapes","authors":"Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, Jianxiong Xiao","year":"2015","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xie et al., 2016] J. Xie, G. Dai, F. Zhu, E. Wong, and Y. Fang. Deepshape: Deep-learned shape descriptor for 3d shape retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP(99):1-1, 2016.","title":"Deepshape: Deep-learned shape descriptor for 3d shape retrieval","authors":"J Xie, G Dai, F Zhu, E Wong, Y Fang","year":"2016","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}