{"type":"ScholarlyArticle","title":"An Appearance-and-Structure Fusion Network for Object Viewpoint Estimation","authors":[{"name":"Yueying Kao","slug":"yueying-kao","given":"Yueying","family":"Kao","affiliation":"SAIT -China Lab , Samsung Research Institute China -Beijing (SRC-B)"},{"name":"Weiming Li","slug":"weiming-li","given":"Weiming","family":"Li","affiliation":"SAIT -China Lab , Samsung Research Institute China -Beijing (SRC-B)"},{"name":"Zairan Wang","slug":"zairan-wang","given":"Zairan","family":"Wang","affiliation":"SAIT -China Lab , Samsung Research Institute China -Beijing (SRC-B)"},{"name":"Dongqing Zou","slug":"dongqing-zou","given":"Dongqing","family":"Zou","affiliation":"SAIT -China Lab , Samsung Research Institute China -Beijing (SRC-B)"},{"name":"Ran He","slug":"ran-he","given":"Ran","family":"He","affiliation":"NLPR & CRIPAC , Institute of Automation , Chinese Academy of Sciences"},{"name":"Qiang Wang","slug":"qiang-wang","given":"Qiang","family":"Wang","affiliation":"SAIT -China Lab , Samsung Research Institute China -Beijing (SRC-B)"},{"name":"Minsu Ahn","slug":"minsu-ahn","given":"Minsu","family":"Ahn","affiliation":"Samsung Advanced Institute of Technology (SAIT)"},{"name":"Sunghoon Hong","slug":"sunghoon-hong","given":"Sunghoon","family":"Hong","affiliation":"Samsung Advanced Institute of Technology (SAIT)"}],"abstract":"Automatic object viewpoint estimation from a single image is an important but challenging problem in machine intelligence community. Although impressive performance has been achieved, current state-of-the-art methods still have difficulty to deal with the visual ambiguity and structure ambiguity in real world images. To tackle these problems, a novel Appearance-and-Structure Fusion network, which we call it ASFnet that estimates viewpoint by fusing both appearance and structure information, is proposed in this paper. The structure information is encoded by precise semantic keypoints and can help address the visual ambiguity. Meanwhile, distinguishable appearance features contribute to overcoming the structure ambiguity. Our ASFnet integrates an appearance path and a structure path to an end-to-end network and allows deep features effectively share supervision from both the two complementary aspects. A convolutional layer is learned to fuse the two path results adaptively. To balance the influence from the two supervision sources, a piecewise loss weight strategy is employed during training. Experimentally, our proposed network outperforms state-of-the-art methods on a public PASCAL 3D+ dataset, which verifies the effectiveness of our method and further corroborates the above proposition.","doi":"10.24963/ijcai.2018/684","url":"https://ijcai.org/proceedings/2018/684","pdf":"https://ijcai.org/proceedings/2018/0684.pdf","markdown":"https://ijcai.org/proceedings/2018/684.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/684","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Robotics","track":"Main track","number":684,"pages":{"first":4929,"last":4935},"keywords":["Robotics: Vision and Perception","Robotics: Robotics and Vision"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2017] Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. arXiv preprint arXiv:1711.07319, 2017.","title":"Cascaded pyramid network for multi-person pose estimation","authors":"Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, Jian Sun","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Elhoseiny et al., 2016] Mohamed Elhoseiny, Tarek El- Gaaly, Amr Bakry, and Ahmed Elgammal. A comparative analysis and study of multiview cnn models for joint object categorization and pose estimation. In International Conference on Machine Learning, pages 888-897, 2016.","title":"A comparative analysis and study of multiview cnn models for joint object categorization and pose estimation","authors":"Mohamed Elhoseiny, Tarek El-Gaaly, Amr Bakry, Ahmed Elgammal","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Felzenszwalb et al., 2010] Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and machine intelligence, 32(9):1627-1645, 2010.","title":"Object detection with discriminatively trained part-based models","authors":"Pedro F Felzenszwalb, Ross B Girshick, David Mcallester, Deva Ramanan","year":"2010","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Li et al., 2017] Chi Li, M Zeeshan Zia, Quoc-Huy Tran, Xiang Yu, Gregory D Hager, and Manmohan Chandraker. Deep supervision with shape concepts for occlusion-aware 3d object parsing. In IEEE Conference on Computer Vision and Pattern Recognition, pages 388-397, July 2017.","title":"Deep supervision with shape concepts for occlusion-aware 3d object parsing","authors":"Chi Li, M Zeeshan Zia, Quoc-Huy Tran, Xiang Yu, Gregory D Hager, Manmohan Chandraker","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Lin et al., 2017] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In IEEE Conference on Computer Vision and Pattern Recognition, pages 936 -944, 2017.","title":"Feature pyramid networks for object detection","authors":"Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Mahendran et al., 2017] Siddharth Mahendran, Haider Ali, and René Vidal. 3d pose regression using convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 494- 495, 2017.","title":"3d pose regression using convolutional neural networks","authors":"Haider Siddharth Mahendran, René Ali, Vidal","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Newell et al., 2016] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In European Conference on Computer Vision, pages 483-499, 2016.","title":"Stacked hourglass networks for human pose estimation","authors":"Alejandro Newell, Kaiyu Yang, Jia Deng","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Pavlakos et al., 2017] Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Konstantinos G Derpanis, and Kostas Daniilidis. 6-dof object pose from semantic keypoints. In International Conference on Robotics and Automation, pages 2011 -2018, 2017.","title":"6-dof object pose from semantic keypoints","authors":"Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Konstantinos G Derpanis, Kostas Daniilidis","year":"2017","venue":"International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Pepik et al., 2012] Bojan Pepik, Michael Stark, Peter Gehler, and Bernt Schiele. Teaching 3d geometry to deformable part models. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3362-3369, 2012.","title":"Teaching 3d geometry to deformable part models","authors":"Bojan Pepik, Michael Stark, Peter Gehler, Bernt Schiele","year":"2012","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Poirson et al., 2016] Patrick Poirson, Phil Ammirato, Cheng-Yang Fu, Wei Liu, Jana Kosecka, and Alexander C Berg. Fast single shot detection and pose estimation. In International Conference on 3D Vision, pages 676-684, 2016.","title":"Fast single shot detection and pose estimation","authors":"Patrick Poirson, Phil Ammirato, Cheng-Yang Fu, Wei Liu, Jana Kosecka, Alexander C Berg","year":"2016","venue":"International Conference on 3D Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Su et al., 2015] Hao Su, Charles R. Qi, Yangyan Li, and Leonidas J. Guibas. Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In IEEE International Conference on Computer Vision, pages 2686-2694, 2015.","title":"Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views","authors":"Hao Su, Charles R Qi, Yangyan Li, Leonidas J Guibas","year":"2015","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Szeto and Corso, 2017] Ryan Szeto and Jason J. Corso. Click here: Human-localized keypoints as guidance for viewpoint estimation. In IEEE International Conference on Computer Vision, pages 1604-1613, 2017.","title":"Click here: Human-localized keypoints as guidance for viewpoint estimation","authors":"Ryan Szeto, Jason J Corso","year":"2017","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Tekin et al., 2017] Bugra Tekin, Sudipta N Sinha, and Pascal Fua. Real-time seamless single shot 6d object pose prediction. arXiv preprint arXiv:1711.08848, 2017.","title":"Real-time seamless single shot 6d object pose prediction","authors":"Bugra Tekin, N Sudipta, Pascal Sinha, Fua","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tulsiani and Malik, 2015] Shubham Tulsiani and Jitendra Malik. Viewpoints and keypoints. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1510- 1519, 2015.","title":"Viewpoints and keypoints","authors":"Shubham Tulsiani, Jitendra Malik","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wu et al., 2016] Jiajun Wu, Tianfan Xue, Joseph J Lim, Yuandong Tian, Joshua B Tenenbaum, Antonio Torralba, and William T Freeman. Single image 3d interpreter network. In European Conference on Computer Vision, pages 365-382, 2016.","title":"Single image 3d interpreter network","authors":"Jiajun Wu, Tianfan Xue, Joseph J Lim, Yuandong Tian, Joshua B Tenenbaum, Antonio Torralba, William T Freeman","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xiang et al., 2014] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection in the wild. In IEEE Winter Conference on Applications of Computer Vision, pages 75-82, 2014.","title":"Beyond pascal: A benchmark for 3d object detection in the wild","authors":"Yu Xiang, Roozbeh Mottaghi, Silvio Savarese","year":"2014","venue":"IEEE Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yang et al., 2014] Linjie Yang, Jianzhuang Liu, and Xiaoou Tang. Object detection and viewpoint estimation with auto-masking neural network. In European conference on computer vision, pages 441-455, 2014.","title":"Object detection and viewpoint estimation with auto-masking neural network","authors":"Yang","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhang et al., 2013] Haopeng Zhang, Tarek El-Gaaly, Ahmed Elgammal, and Zhiguo Jiang. Joint object and pose recognition using homeomorphic manifold analysis. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence, pages 1012-1019, 2013.","title":"Joint object and pose recognition using homeomorphic manifold analysis","authors":"Zhang","year":"2013","venue":"Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang et al., 2014] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In European Conference on Computer Vision, pages 94-108, 2014.","title":"Facial landmark detection by deep multi-task learning","authors":"Zhanpeng Zhang, Ping Luo, Chen Change Loy, Xiaoou Tang","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}