{"type":"ScholarlyArticle","title":"Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning","authors":[{"name":"Sihui Luo","slug":"sihui-luo","given":"Sihui","family":"Luo","affiliation":"Zhejiang University"},{"name":"Xinchao Wang","slug":"xinchao-wang","given":"Xinchao","family":"Wang","affiliation":"Stevens Institute of Technology"},{"name":"Gongfan Fang","slug":"gongfan-fang","given":"Gongfan","family":"Fang","affiliation":"Zhejiang University"},{"name":"Yao Hu","slug":"yao-hu","given":"Yao","family":"Hu","affiliation":"Alibaba Group"},{"name":"Dapeng Tao","slug":"dapeng-tao","given":"Dapeng","family":"Tao","affiliation":"Yunnan University"},{"name":"Mingli Song","slug":"mingli-song","given":"Mingli","family":"Song","affiliation":"Zhejiang University"}],"abstract":"An increasing number of well-trained deep networks have been released online by researchers and developers, enabling the community to reuse them in a plug-and-play way without accessing the training annotations. However, due to the large number of network variants, such public-available trained models are often of different architectures, each of which being tailored for a specific task or dataset. In this paper, we study a deep-model reusing task, where we are given as input pre-trained networks of heterogeneous architectures specializing in distinct tasks, as teacher models. We aim to learn a multitalented and light-weight student model that is able to grasp the integrated knowledge from all such heterogeneous-structure teachers, again without accessing any human annotation. To this end, we propose a common feature learning scheme, in which the features of all teachers are transformed into a common space and the student is enforced to imitate them all so as to amalgamate the intact knowledge. We test the proposed approach on a list of benchmarks and demonstrate that the learned student is able to achieve very promising performance, superior to those of the teachers in their specialized tasks.","doi":"10.24963/ijcai.2019/428","url":"https://ijcai.org/proceedings/2019/428","pdf":"https://ijcai.org/proceedings/2019/0428.pdf","markdown":"https://ijcai.org/proceedings/2019/428.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/428","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":428,"pages":{"first":3087,"last":3093},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Deep Learning","Machine Learning: Feature Selection ; Learning Sparse Models","Machine Learning: Multi-instance;Multi-label;Multi-view learning","Machine Learning: Developmental Learning"],"videos":[],"references":[{"position":1,"raw":"[Ben-David et al., 2010] Shai Ben-David, John Blitzer, Ko- by Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from differ- ent domains. Machine learning, 79(1-2):151-175, 2010.","title":"A theory of learning from different domains","authors":"Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, Jennifer Wortman Vaughan","year":"2010","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deng et al., 2018] Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. arXiv:1801.07698, 2018.","title":"Arcface: Additive angular margin loss for deep face recognition","authors":"Deng","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dietterich, 2000] Thomas G. Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems, pages 1-15, Berlin, Heidelberg, 2000. Springer.","title":"Ensemble methods in machine learning","authors":"G Thomas, Dietterich","year":"2000","venue":"Multiple Classifier Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gong et al., 2016] Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transfer- able components. In IEEE Conference on Machine Learn- ing, 2016.","title":"Domain adaptation with conditional transferable components","authors":"Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, Bernhard Schölkopf","year":"2016","venue":"IEEE Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gretton et al., 2012] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smo- la. A kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723-773, 2012.","title":"A kernel two-sample test","authors":"Arthur Gretton, Karsten M Borgwardt, J Malte, Bernhard Rasch, Alexander Schölkopf, Smola","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hansen and Peter, 1990] Lars Kai Hansen and Salamon Pe- ter. Neural network ensembles. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 12(10):993-1001, October 1990.","title":"Neural network ensembles","authors":"Peter ; Hansen, Lars, Kai Hansen, Salamon Peter","year":"1990","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pat- tern Recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and J- eff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2008] Gary B. Huang, Marwan Mattar, Tama- ra Berg, and Eric Learned-Miller. Labeled faces in the wild: A database forstudying face recognition in uncon- strained environments. In Workshop on Faces in 'Real- Life' Images: Detection, Alignment, and Recognition, Oc- tober 2008.","title":"Labeled faces in the wild: A database forstudying face recognition in unconstrained environments","authors":"B Gary, Marwan Huang, Tamara Mattar, Eric Berg, Learned-Miller","year":"2008","venue":"Workshop on Faces in 'Real-Life' Images: Detection, Alignment, and Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2016] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In Proceedings of the 14th European conference on computer vision, pages 646-661, 2016.","title":"Deep networks with stochastic depth","authors":"Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, Kilian Q Weinberger","year":"2016","venue":"Proceedings of the 14th European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Moschoglou et al., 2017] Stylianos Moschoglou, Athana- sios Papaioannou, Christos Sagonas, Jiankang Deng, and Stefanos Zafeiriou. Agedb: The first manually collected, in-the-wild age database. In IEEE Conference on Comput- er Vision and Pattern Recognition Workshops, 2017.","title":"Agedb: The first manually collected, in-the-wild age database","authors":"Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Stefanos Zafeiriou","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Romero et al., 2015] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gat- ta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In The International Conference on Learning Representa- tions, 2015.","title":"Fitnets: Hints for thin deep nets","authors":"Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio","year":"2015","venue":"The International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sengupta et al., 2016] Soumyadip Sengupta, Jun-Cheng. Chen, Carlos Castillo, Vishal M. Patel, Rama Chellappa, and David. W. Jacobs. Frontal to profile face verification in the wild. In IEEE Winter Conference on Applications of Computer Vision, pages 1-9, March 2016.","title":"Frontal to profile face verification in the wild","authors":"Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, M Vishal, Rama Patel, David W Chellappa, Jacobs","year":"2016","venue":"IEEE Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Shen et al., 2019] Chengchao Shen, Xinchao Wang, Jie Song, Li Sun, and Mingli Song. Amalgamating knowledge towards comprehensive classification. In Proceedings of the 33th AAAI Conference on Artificial Intelligence, 2019.","title":"Amalgamating knowledge towards comprehensive classification","authors":"Chengchao Shen, Xinchao Wang, Jie Song, Li Sun, Mingli Song","year":"2019","venue":"Proceedings of the 33th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Singh et al., 2016] Saurabh Singh, Derek Hoiem, and David Forsyth. Swapout: Learning an ensemble of deep archi- tectures. In Proceedings of the 29th Advances in Neural Information Processing Systems, pages 28-36. 2016.","title":"Swapout: Learning an ensemble of deep architectures","authors":"Saurabh Singh, Derek Hoiem, David Forsyth","year":"2016","venue":"Proceedings of the 29th Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E. Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In IEEE Con- ference on computer vision and pattern recognition, pages 1-9, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"IEEE Conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wan et al., 2013] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neu- ral networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning, vol- ume 28, pages 1058-1066, 2013.","title":"Regularization of neural networks using dropconnect","authors":"Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, Rob Fergus","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2011] Xinchao Wang, Zhu Li, and Dacheng Tao. Subspaces indexing model on Grassmann manifold for image search. IEEE Transactions on Image Process- ing, 20(9):2627-2635, 2011.","title":"Subspaces indexing model on Grassmann manifold for image search","authors":"Wang","year":"2011","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2018] Hui Wang, Hanbin Zhao, Xi Li, and X- u Tan. Progressive blockwise knowledge distillation for neural network acceleration. In Proceedings of the 27th International Joint Conference on Artifical Intelligence, pages 2769-2775, 2018.","title":"Progressive blockwise knowledge distillation for neural network acceleration","authors":"Hui Wang, Hanbin Zhao, Xi Li, Xu Tan","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artifical Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/384"},{"position":22,"raw":"[Ye et al., 2019] Jingwen Ye, Yixin Ji, Xinchao Wang, Kairi Ou, Dapeng Tao, and Mingli Song. Student becoming the master: Knowledge amalgamation for joint scene parsing, depth estimation, and more. In IEEE Conference on Com- puter Vision and Pattern Recognition, 2019.","title":"Student becoming the master: Knowledge amalgamation for joint scene parsing, depth estimation, and more","authors":"Jingwen Ye, Yixin Ji, Xinchao Wang, Kairi Ou, Dapeng Tao, Mingli Song","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yi et al., 2014] Dong Yi, Zhen Lei, Shengcai Liao, and S- tan Z. Li. Learning face representation from scratch. arX- iv:1411.7923, 2014.","title":"Learning face representation from scratch","authors":"Dong Yi, Zhen Lei, Shengcai Liao, Stan Z Li","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2017] Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 67-76, 2017.","title":"On compressing deep models by low rank and sparse decomposition","authors":"Xiyu Yu, Tongliang Liu, Xinchao Wang, Dacheng Tao","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zamir et al., 2018] Amir Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learn- ing. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3712-3722, June 2018.","title":"Taskonomy: Disentangling task transfer learning","authors":"Amir Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, Silvio Savarese","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":"/proceedings/2019/871"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}