{"type":"ScholarlyArticle","title":"Better and Faster: Knowledge Transfer from Multiple Self-supervised Learning Tasks via Graph Distillation for Video Classification","authors":[{"name":"Chenrui Zhang","slug":"chenrui-zhang","given":"Chenrui","family":"Zhang","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing 100871 , China"},{"name":"Yuxin Peng","slug":"yuxin-peng","given":"Yuxin","family":"Peng","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing 100871 , China"}],"abstract":"Video representation learning is a vital problem for\r\nclassification task. Recently, a promising unsupervised\r\nparadigm termed self-supervised learning\r\nhas emerged, which explores inherent supervisory\r\nsignals implied in massive data for feature learning\r\nvia solving auxiliary tasks. However, existing\r\nmethods in this regard suffer from two limitations\r\nwhen extended to video classification. First, they\r\nfocus only on a single task, whereas ignoring complementarity\r\namong different task-specific features\r\nand thus resulting in suboptimal video representation.\r\nSecond, high computational and memory cost\r\nhinders their application in real-world scenarios. In\r\nthis paper, we propose a graph-based distillation\r\nframework to address these problems: (1) We propose\r\nlogits graph and representation graph to transfer\r\nknowledge from multiple self-supervised tasks,\r\nwhere the former distills classifier-level knowledge\r\nby solving a multi-distribution joint matching problem,\r\nand the latter distills internal feature knowledge\r\nfrom pairwise ensembled representations with\r\ntackling the challenge of heterogeneity among different\r\nfeatures; (2) The proposal that adopts a\r\nteacher-student framework can reduce the redundancy\r\nof knowledge learned from teachers dramatically,\r\nleading to a lighter student model that solves\r\nclassification task more efficiently. Experimental\r\nresults on 3 video datasets validate that our proposal\r\nnot only helps learn better video representation\r\nbut also compress model for faster inference.","doi":"10.24963/ijcai.2018/158","url":"https://ijcai.org/proceedings/2018/158","pdf":"https://ijcai.org/proceedings/2018/0158.pdf","markdown":"https://ijcai.org/proceedings/2018/158.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/158","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":158,"pages":{"first":1135,"last":1141},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Structural and Model-Based Approaches, Knowledge Representation and Reasoning"],"videos":[],"references":[{"position":1,"raw":"[Agrawal et al., 2015] Pulkit Agrawal, Joao Carreira, and Ji- tendra Malik. Learning to see by moving. In ICCV, pages 37-45, 2015.","title":"Learning to see by moving","authors":"Pulkit Agrawal, Joao Carreira, Jitendra Malik","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Arjovsky et al., 2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017.","title":"Wasserstein gan","authors":"Martin Arjovsky, Soumith Chintala, Léon Bottou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Charikar et al., 2002] Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding frequent items in data streams. ALP, pages 784-784, 2002.","title":"Finding frequent items in data streams","authors":"Charikar","year":"2002","venue":"ALP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2017] Yuntao Chen, Naiyan Wang, and Zhaox- iang Zhang. Darkrank: Accelerating deep metric learn- ing via cross sample similarities transfer. arXiv preprint arXiv:1707.01220, 2017.","title":"Darkrank: Accelerating deep metric learning via cross sample similarities transfer","authors":"Yuntao Chen, Naiyan Wang, Zhaoxiang Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fernando et al., 2017] Basura Fernando, Hakan Bilen, Efs- tratios Gavves, and Stephen Gould. Self-supervised video representation learning with odd-one-out networks. In CVPR, pages 5729-5738, 2017.","title":"Self-supervised video representation learning with odd-one-out networks","authors":"Basura Fernando, Hakan Bilen, Efstratios Gavves, Stephen Gould","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2016] Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. Compact bilinear pooling. In CVPR, pages 317-326, 2016.","title":"Compact bilinear pooling","authors":"Yang Gao, Oscar Beijbom, Ning Zhang, Trevor Darrell","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NIPS Workshop, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":"NIPS Workshop","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jayaraman and Grauman, 2016] Dinesh Jayaraman and Kristen Grauman. Slow and steady feature analysis: higher order temporal coherence in video. In CVPR, pages 3852-3861, 2016.","title":"Slow and steady feature analysis: higher order temporal coherence in video","authors":"Dinesh Jayaraman, Kristen Grauman","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2011] Yu-Gang Jiang, Guangnan Ye, Shih-Fu Chang, Daniel Ellis, and Alexander C Loui. Consumer video understanding: A benchmark database and an eval- uation of human and machine performance. In ICMR, page 29, 2011.","title":"Consumer video understanding: A benchmark database and an evaluation of human and machine performance","authors":"Yu-Gang Jiang, Guangnan Ye, Shih-Fu Chang, Daniel Ellis, Alexander C Loui","year":"2011","venue":"ICMR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kuehne et al., 2013] Hilde Kuehne, Hueihan Jhuang, Rainer Stiefelhagen, and Thomas Serre. Hmdb51: A large video database for human motion recognition. In HPCSE, pages 571-582. 2013.","title":"Hmdb51: A large video database for human motion recognition","authors":"Kuehne","year":"2013","venue":"HPCSE","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2017] Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised representa- tion learning by sorting sequences. In ICCV, pages 667- 676, 2017.","title":"Unsupervised representation learning by sorting sequences","authors":"Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, Ming-Hsuan Yang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lopez-Paz et al., 2016] David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying dis- tillation and privileged information. In ICLR, 2016.","title":"Unifying distillation and privileged information","authors":"Lopez-Paz","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lotter et al., 2017] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. In ICLR, 2017.","title":"Deep predictive coding networks for video prediction and unsupervised learning","authors":"William Lotter, David Gabriel Kreiman, Cox","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luo et al., 2017] Zelun Luo, Lu Jiang, Jun-Ting Hsieh, Juan Carlos Niebles, and Li Fei-Fei. Graph distillation for action detection with privileged information. arXiv preprint arXiv:1712.00108, 2017.","title":"Graph distillation for action detection with privileged information","authors":"Luo","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Misra et al., 2016] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV, pages 527- 544, 2016.","title":"Shuffle and learn: unsupervised learning using temporal order verification","authors":"Ishan Misra, Lawrence Zitnick, Martial Hebert","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Noroozi and Favaro, 2016] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, pages 69-84, 2016.","title":"Unsupervised learning of visual representations by solving jigsaw puzzles","authors":"Favaro Noroozi, Mehdi Noroozi, Paolo Favaro","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pathak et al., 2016] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Con- text encoders: Feature learning by inpainting. In CVPR, pages 2536-2544, 2016.","title":"Context encoders: Feature learning by inpainting","authors":"Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A Efros","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Purushwalkam and Gupta, 2016] Senthil Purushwalkam and Abhinav Gupta. Pose from action: Unsupervised learning of pose features based on motion. In ECCV, 2016.","title":"Pose from action: Unsupervised learning of pose features based on motion","authors":"Senthil Purushwalkam, Abhinav Gupta","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Romero et al., 2015] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.","title":"Fitnets: Hints for thin deep nets","authors":"Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Soomro et al., 2012] Khurram Soomro, Amir Roshan Za- mir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.","title":"Ucf101: A dataset of 101 human actions classes from videos in the wild","authors":"Khurram Soomro, Mubarak Amir Roshan Zamir, Shah","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vapnik and Izmailov, 2015] Vladimir Vapnik and Rauf Iz- mailov. Learning using privileged information: similarity control and knowledge transfer. 16(20232049):55, 2015.","title":"Vladimir Vapnik and Rauf Izmailov","authors":"Izmailov Vapnik","year":"2015","venue":"Learning using privileged information: similarity control and knowledge transfer","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang and Gupta, 2015] Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, pages 2794-2802, 2015.","title":"Unsupervised learning of visual representations using videos","authors":"Wang, Gupta, Xiaolong Wang, Abhinav Gupta","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yosinski et al., 2014] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NIPS, pages 3320-3328, 2014.","title":"How transferable are features in deep neural networks?","authors":"Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zagoruyko and Komodakis, 2017] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Im- proving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.","title":"Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2017] Richard Zhang, Phillip Isola, and Alexei A. Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In CVPR, pages 1057-1068, 2017.","title":"Split-brain autoencoders: Unsupervised learning by cross-channel prediction","authors":"Richard Zhang, Phillip Isola, Alexei A Efros","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}