{"type":"ScholarlyArticle","title":"Training Group Orthogonal Neural Networks with Privileged Information","authors":[{"name":"Yunpeng Chen","slug":"yunpeng-chen","given":"Yunpeng","family":"Chen","affiliation":"National University of Singapore"},{"name":"Xiaojie Jin","slug":"xiaojie-jin","given":"Xiaojie","family":"Jin","affiliation":"National University of Singapore"},{"name":"Jiashi Feng","slug":"jiashi-feng","given":"Jiashi","family":"Feng","affiliation":"National University of Singapore; AI Institute"},{"name":"Shuicheng Yan","slug":"shuicheng-yan","given":"Shuicheng","family":"Yan","affiliation":"Qihoo 360; National University of Singapore"}],"abstract":"Learning rich and diverse representations is critical for the performance of deep convolutional neural networks (CNNs). In this paper, we consider how to use privileged information to promote inherent diversity of a single CNN model such that the model can learn better representations and offer stronger generalization ability. To this end, we propose a novel group orthogonal convolutional neural network (GoCNN) that learns untangled representations within each layer by exploiting provided privileged information and enhances representation diversity effectively. We take image classification as an example where image segmentation annotations are used as privileged information during the training process. Experiments on two benchmark datasets – ImageNet and PASCAL VOC – clearly demonstrate the strong generalization ability of our proposed GoCNN model. On the ImageNet dataset, GoCNN improves the performance of state-of-the-art ResNet-152 model by absolute value of 1.2% while only uses privileged information of 10% of the training images, confirming effectiveness of GoCNN on utilizing available privileged knowledge to train better CNNs.","doi":"10.24963/ijcai.2017/212","url":"https://ijcai.org/proceedings/2017/212","pdf":"https://ijcai.org/proceedings/2017/0212.pdf","markdown":"https://ijcai.org/proceedings/2017/212.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/212","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":212,"pages":{"first":1532,"last":1538},"keywords":["Machine Learning: Deep Learning","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2015] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous dis- tributed systems. arXiv preprint arXiv:1512.01274, 2015.","title":"Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems","authors":"Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, Zheng Zhang","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cogswell et al., 2016] Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting in deep networks by decorrelating representa- tions. ICLR, 2016.","title":"Reducing overfitting in deep networks by decorrelating representations","authors":"Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, Dhruv Batra","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai et al., 2015] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convo- lutional networks for semantic segmentation. In Proceed- ings of the IEEE International Conference on Computer Vision, pages 1635-1643, 2015.","title":"Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation","authors":"Jifeng Dai, Kaiming He, Jian Sun","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pat- tern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Everingham et al., 2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zis- serman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.","title":"The pascal visual object classes (voc) challenge","authors":"Mark Everingham, Luc Van Gool, K I Christopher, John Williams, Andrew Winn, Zisserman","year":"2010","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Feyereisl et al., 2014] Jan Feyereisl, Suha Kwak, Jeany Son, and Bohyung Han. Object localization based on structural svm using privileged information. In Advances in Neural Information Processing Systems, pages 208- 216, 2014.","title":"Object localization based on structural svm using privileged information","authors":"Feyereisl","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goodfellow et al., 2013] Ian J Goodfellow, David Warde- Farley, Mehdi Mirza, Aaron C Courville, and Yoshua Ben- gio. Maxout networks. ICML, pages 1319-1327, 2013.","title":"Maxout networks","authors":"Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron C Courville, Yoshua Bengio","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hariharan et al., 2014] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In Computer vision-ECCV 2014, pages 297-312. Springer, 2014.","title":"Simultaneous detection and segmentation","authors":"Bharath Hariharan, Pablo Arbeláez, Ross Girshick, Jitendra Malik","year":"2014","venue":"Computer vision-ECCV 2014","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv preprint arXiv:1512.03385, 2015.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jin et al., 2015] Xiaojie Jin, Chunyan Xu, Jiashi Feng, Yun- chao Wei, Junjun Xiong, and Shuicheng Yan. Deep learn- ing with s-shaped rectified linear activation units. arXiv preprint arXiv:1512.07030, 2015.","title":"Deep learning with s-shaped rectified linear activation units","authors":"Xiaojie Jin, Chunyan Xu, Jiashi Feng, Yunchao Wei, Junjun Xiong, Shuicheng Yan","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lapin et al., 2014] Maksim Lapin, Matthias Hein, and Bernt Schiele. Learning using privileged information: Svm+ and weighted svm. Neural Networks, 53:95-108, 2014.","title":"Learning using privileged information: Svm+ and weighted svm","authors":"Maksim Lapin, Matthias Hein, Bernt Schiele","year":"2014","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lopez-Paz et al., 2016] David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying dis- tillation and privileged information. stat, 1050:26, 2016.","title":"Unifying distillation and privileged information","authors":"David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, Vladimir Vapnik","year":"2016","venue":"stat","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pechyony and Vapnik, 2011] Dmitry Pechyony and Vladimir Vapnik. Fast optimization algorithms for solving svm+. Stat. Learning and Data Science, 2011.","title":"Dmitry Pechyony and Vladimir Vapnik. Fast optimization algorithms for solving svm+","authors":"Vapnik Pechyony","year":"2011","venue":"Stat. Learning and Data Science","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Russakovsky et al., 2012] Olga Russakovsky, Yuanqing Lin, Kai Yu, and Li Fei-Fei. Object-centric spatial pooling for image classification. In Computer Vision-ECCV 2012, pages 1-15. Springer, 2012.","title":"Object-centric spatial pooling for image classification","authors":"Olga Russakovsky, Yuanqing Lin, Kai Yu, Li Fei-Fei","year":"2012","venue":"Computer Vision-ECCV 2012","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sharmanska et al., 2014] Viktoriia Sharmanska, Novi Quadrianto, and Christoph H Lampert. Learning to transfer privileged information. arXiv preprint arXiv:1410.0389, 2014.","title":"Viktoriia Sharmanska, Novi Quadrianto, and Christoph H Lampert. Learning to transfer privileged information","authors":"Sharmanska","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Song et al., 2011] Zheng Song, Qiang Chen, Zhongyang Huang, Yang Hua, and Shuicheng Yan. Contextualizing object detection and classification. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1585-1592. IEEE, 2011.","title":"Contextualizing object detection and classification","authors":"Zheng Song, Qiang Chen, Zhongyang Huang, Yang Hua, Shuicheng Yan","year":"2011","venue":"Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sra and Hosseini, 2016] Suvrit Sra and Reshad Hosseini. Geometric Optimization in Machine Learning, pages 73- 91. Springer International Publishing, Cham, 2016.","title":"Geometric Optimization in Machine Learning","authors":"Hosseini Sra","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Szegedy et al., 2016] Christian Szegedy, Sergey Ioffe, Vin- cent Vanhoucke, and Alex Alemi. Inception-v4, inception- resnet and the impact of residual connections on learning. arXiv preprint arXiv:1602.07261, 2016.","title":"Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex Alemi. Inception-v4, inceptionresnet and the impact of residual connections on learning","authors":"Szegedy","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tang, 2013] Yichuan Tang. Deep learning using linear sup- port vector machines. arXiv preprint arXiv:1306.0239, 2013.","title":"Deep learning using linear support vector machines","authors":"Yichuan Tang","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vapnik and Izmailov, 2015] Vladimir Vapnik and Rauf Iz- mailov. Learning using privileged information: similar- ity control and knowledge transfer. Journal of Machine Learning Research, 16:2023-2049, 2015.","title":"Vladimir Vapnik and Rauf Izmailov. Learning using privileged information: similarity control and knowledge transfer","authors":"Izmailov Vapnik","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vapnik and Vashist, 2009] Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privi- leged information. Neural networks, 22(5):544-557, 2009.","title":"Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information","authors":"Vashist Vapnik","year":"2009","venue":"Neural networks","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2016] Xingcheng Zhang, Zhizhong Li, Chen Change Loy, and Dahua Lin. Polynet: A pursuit of structural diversity in very deep networks. arXiv preprint arXiv:1611.05725, 2016.","title":"Polynet: A pursuit of structural diversity in very deep networks","authors":"Xingcheng Zhang, Zhizhong Li, Chen Change Loy, Dahua Lin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}