{"type":"ScholarlyArticle","title":"Evaluation Methods for Representation Learning: A Survey","authors":[{"name":"Kento Nozawa","slug":"kento-nozawa","given":"Kento","family":"Nozawa","affiliation":"The University of Tokyo; RIKEN AIP"},{"name":"Issei Sato","slug":"issei-sato","given":"Issei","family":"Sato","affiliation":"The University of Tokyo"}],"abstract":"Representation learning enables us to automatically extract generic feature representations from a dataset to solve another machine learning task. Recently, extracted feature representations by a representation learning algorithm and a simple predictor have exhibited state-of-the-art performance on several machine learning tasks. Despite its remarkable progress, there exist various ways to evaluate representation learning algorithms depending on the application because of the flexibility of representation learning. To understand the current applications of representation learning, we review evaluation methods of representation learning algorithms. On the basis of our evaluation survey, we also discuss the future direction of representation learning. The extended version, https://arxiv.org/abs/2204.08226, gives more detailed discussions and a survey on theoretical analyses.","doi":"10.24963/ijcai.2022/776","url":"https://ijcai.org/proceedings/2022/776","pdf":"https://ijcai.org/proceedings/2022/0776.pdf","markdown":"https://ijcai.org/proceedings/2022/776.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/776","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Survey Track","track":"Survey Track","number":776,"pages":{"first":5556,"last":5563},"keywords":["Survey Track: Machine Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987281","length_sec":96,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985424","length_sec":536,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Abnar et al., 2022] Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the Limits of Large Scale Pre-training. In ICLR, 2022.","title":"Exploring the Limits of Large Scale Pre-training","authors":"Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, Hanie Sedghi","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Adi et al., 2017] Yossi Adi, Einat Kermany, Yonatan Be- linkov, Ofer Lavi, and Yoav Goldberg. Fine-grained Anal- ysis of Sentence Embeddings Using Auxiliary Prediction Tasks. In ICLR, 2017.","title":"Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks","authors":"Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, Yoav Goldberg","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arora et al., 2018] Sanjeev Arora, Wei Hu, and Pravesh K. Kothari. An Analysis of the t-SNE Algorithm for Data Visualization. In COLT, 2018.","title":"An Analysis of the t-SNE Algorithm for Data Visualization","authors":"Sanjeev Arora, Wei Hu, Pravesh K Kothari","year":"2018","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Arora et al., 2019] Sanjeev Arora, Hrishikesh Khande- parkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A Theoretical Analysis of Contrastive Unsuper- vised Representation Learning. In ICML, 2019.","title":"A Theoretical Analysis of Contrastive Unsupervised Representation Learning","authors":"Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, Nikunj Saunshi","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bachman et al., 2019] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning Representations by Maximizing Mutual Information Across Views. In NeurIPS, 2019.","title":"Learning Representations by Maximizing Mutual Information Across Views","authors":"Philip Bachman, Devon Hjelm, William Buchwalter","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Baroni et al., 2014] Marco Baroni, Georgiana Dinu, and Germán Kruszewski. Don't Count, Predict! A Systematic Comparison of Context-counting vs. Context-predicting Semantic Vectors. In ACL, 2014.","title":"Don't Count, Predict! A Systematic Comparison of Context-counting vs. Context-predicting Semantic Vectors","authors":"Marco Baroni, Georgiana Dinu, Germán Kruszewski","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Bengio et al., 2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation Learning: A Review and New Perspectives. TPAMI, 2013.","title":"Representation Learning: A Review and New Perspectives","authors":"Yoshua Bengio, Aaron Courville, Pascal Vincent","year":"2013","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Beyer et al., 2020] Lucas Beyer, Olivier J. Hénaff, Alexan- der Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are We Done with ImageNet? arXiv:2006.07159v1 [cs.CV], 2020.","title":"Are We Done with ImageNet?","authors":"Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, Aäron Van Den Oord","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chapelle et al., 2006] Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-supervised Learning. MIT Press, 2006.","title":"Semi-supervised Learning","authors":"Olivier Chapelle, Bernhard Schölkopf, Alexander Zien","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Under- standing. In NAACL-HLT, 2019.","title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Donahue et al., 2014] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. In ICML, 2014.","title":"DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition","authors":"Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, Trevor Darrell","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Erhan et al., 2010] Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why Does Unsupervised Pre-training Help Deep Learning? JMLR, 2010.","title":"Why Does Unsupervised Pre-training Help Deep Learning? JMLR","authors":"Yoshua Dumitru Erhan, Aaron Bengio, Pierre-Antoine Courville, Pascal Manzagol, Samy Vincent, Bengio","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ericsson et al., 2021] Linus Ericsson, Henry Gouk, and Timothy M. Hospedales. How Well Do Self-Supervised Models Transfer? In CVPR, 2021.","title":"How Well Do Self-Supervised Models Transfer? In CVPR","authors":"Ericsson","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Espadoto et al., 2021] Mateus Espadoto, Rafael M. Mar- tins, Andreas Kerren, Nina S.T. Hirata, and Alexandru C. Telea. Toward a Quantitative Survey of Dimension Reduc- tion Techniques. TVCG, 2021.","title":"Toward a Quantitative Survey of Dimension Reduction Techniques","authors":"Mateus Espadoto, Rafael M Martins, Andreas Kerren, Nina S T Hirata, Alexandru C Telea","year":"2021","venue":"TVCG","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Garg and Liang, 2020] Siddhant Garg and Yingyu Liang. Functional Regularization for Representation Learning: A Unified Theoretical Perspective. In NeurIPS, 2020.","title":"Functional Regularization for Representation Learning: A Unified Theoretical Perspective","authors":"Liang Garg, Siddhant Garg, Yingyu Liang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In CVPR, 2014.","title":"Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.","title":"Deep Learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Gou et al., 2021] Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge Distillation: A Survey. IJCV, 2021.","title":"Knowledge Distillation: A Survey","authors":"Jianping Gou, Baosheng Yu, Stephen J Maybank, Dacheng Tao","year":"2021","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In CVPR, 2020.","title":"Momentum Contrast for Unsupervised Visual Representation Learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Hénaff et al., 2020] Olivier J. Hénaff, Aravind Srinivas, Jef- frey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aaron van den Oord. Data-Efficient Image Recogni- tion with Contrastive Predictive Coding. In ICML, 2020.","title":"Data-Efficient Image Recognition with Contrastive Predictive Coding","authors":"J Olivier, Aravind Hénaff, Jeffrey Srinivas, Ali De Fauw, Carl Razavi, S M Doersch, Aaron Ali Eslami, Van Den Oord","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Hinton et al., 2006] Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh. A Fast Learning Algorithm for Deep Belief Nets. Neural Computation, 2006.","title":"A Fast Learning Algorithm for Deep Belief Nets","authors":"Geoffrey E Hinton, Simon Osindero, Yee-Whye Teh","year":"2006","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Islam et al., 2021] Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Richard Radke, and Rogerio Feris. A Broad Study on the Transferability of Vi- sual Representations with Contrastive Learning. In ICCV, 2021.","title":"A Broad Study on the Transferability of Visual Representations with Contrastive Learning","authors":"Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Richard Radke, Rogerio Feris","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Jing and Tian, 2019] Longlong Jing and Yingli Tian. Self- supervised Visual Feature Learning with Deep Neural Net- works: A Survey. TPAMI, 2019.","title":"Longlong Jing and Yingli Tian. Selfsupervised Visual Feature Learning with Deep Neural Networks: A Survey","authors":"Tian Jing","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kaplan et al., 2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv:2001.08361v1 [cs.LG], 2020.","title":"Scaling Laws for Neural Language Models","authors":"Jared Kaplan, Sam Mccandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Kolesnikov et al., 2019] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting Self-Supervised Visual Representation Learning. In CVPR, 2019.","title":"Revisiting Self-Supervised Visual Representation Learning","authors":"Alexander Kolesnikov, Xiaohua Zhai, Lucas Beyer","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Kolesnikov et al., 2020] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General Visual Representation Learning. In ECCV, 2020.","title":"Big Transfer (BiT): General Visual Representation Learning","authors":"Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Kong et al., 2020] Lingpeng Kong, Cyprien de Masson D'Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yo- gatama. A Mutual Information Maximization Perspective of Language Representation Learning. In ICLR, 2020.","title":"A Mutual Information Maximization Perspective of Language Representation Learning","authors":"Kong","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Kornblith et al., 2019] Simon Kornblith, Jonathon Shlens, and Quoc V. Le. Do Better ImageNet Models Transfer Better? In CVPR, 2019.","title":"Do Better ImageNet Models Transfer Better?","authors":"Simon Kornblith, Jonathon Shlens, V Quoc, Le","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Le et al., 2018] Lei Le, Andrew Patterson, and Martha White. Supervised Autoencoders: Improving General- ization Performance with Unsupervised Regularizers. In NeurIPS, 2018.","title":"Supervised Autoencoders: Improving Generalization Performance with Unsupervised Regularizers","authors":"Lei Le, Andrew Patterson, Martha White","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey Hinton. Deep learning. Nature, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Levy and Goldberg, 2014] Omer Levy and Yoav Goldberg. Neural Word Embedding as Implicit Matrix Factorization. In NeurIPS, 2014.","title":"Neural Word Embedding as Implicit Matrix Factorization","authors":"Goldberg Levy, Omer Levy, Yoav Goldberg","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Li et al., 2018] Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit Inductive Bias for Transfer Learning with Convolutional Networks. In ICML, 2018.","title":"Explicit Inductive Bias for Transfer Learning with Convolutional Networks","authors":"Li","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Liu et al., 2022] Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised Learning is More Robust to Dataset Imbalance. In ICLR, 2022.","title":"Self-supervised Learning is More Robust to Dataset Imbalance","authors":"Liu","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Luong et al., 2016] Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task Sequence to Sequence Learning. In ICLR, 2016.","title":"Multi-task Sequence to Sequence Learning","authors":"Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, Lukasz Kaiser","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Mahajan et al., 2018] Dhruv Mahajan, Ross Girshick, Vig- nesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Ex- ploring the Limits of Weakly Supervised Pretraining. In ECCV, 2018.","title":"Exploring the Limits of Weakly Supervised Pretraining","authors":"Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, Laurens Van Der Maaten","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Maurer et al., 2016] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The Benefit of Multitask Representation Learning. JMLR, 2016.","title":"The Benefit of Multitask Representation Learning","authors":"Andreas Maurer, Massimiliano Pontil, Bernardino Romera-Paredes","year":"2016","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[McNamara and Balcan, 2017] Daniel McNamara and Maria-Florina Balcan. Risk Bounds for Transferring Representations With and Without Fine-Tuning. In ICML, 2017.","title":"Risk Bounds for Transferring Representations With and Without Fine-Tuning","authors":"Balcan Mcnamara, Daniel Mcnamara, Maria-Florina Balcan","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Mikolov et al., 2018] Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. Ad- vances in Pre-Training Distributed Word Representations. In LREC, 2018.","title":"Advances in Pre-Training Distributed Word Representations","authors":"Mikolov","year":"2018","venue":"LREC","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Nozawa and Sato, 2021] Kento Nozawa and Issei Sato. Un- derstanding Negative Samples in Instance Discriminative Self-supervised Representation Learning. In NeurIPS, 2021.","title":"Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning","authors":"Sato ; ] Kento Nozawa, Issei Nozawa, Sato","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Oliver et al., 2018] Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realis- tic Evaluation of Deep Semi-Supervised Learning Algo- rithms. In NeurIPS, 2018.","title":"Realistic Evaluation of Deep Semi-Supervised Learning Algorithms","authors":"Avital Oliver, Augustus Odena, Colin Raffel, Ekin D Cubuk, Ian J Goodfellow","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predic- tive Coding. arXiv:1807.03748v2 [cs.LG], 2018.","title":"Representation Learning with Contrastive Predictive Coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Pan and Yang, 2010] Sinno Jialin Pan and Qiang Yang. A Survey on Transfer Learning. TKDE, 2010.","title":"A Survey on Transfer Learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2010","venue":"TKDE","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Perozzi et al., 2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. DeepWalk: Online Learning of Social Rep- resentations. In KDD, 2014.","title":"DeepWalk: Online Learning of Social Representations","authors":"Bryan Perozzi, Rami Al-Rfou, Steven Skiena","year":"2014","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Su- pervision. In ICML, 2021.","title":"Learning Transferable Visual Models From Natural Language Supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Rumelhart et al., 1986] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning Internal Rep- resentations by Error Propagation. MIT Press, 1986.","title":"Learning Internal Representations by Error Propagation","authors":"David E Rumelhart, Geoffrey E Hinton, Ronald J Williams","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Sariyildiz et al., 2021] Mert Bulent Sariyildiz, Yannis Kalantidis, Diane Larlus, and Karteek Alahari. Concept Generalization in Visual Representation Learning. In ICCV, 2021.","title":"Concept Generalization in Visual Representation Learning","authors":"Mert Bulent Sariyildiz, Yannis Kalantidis, Diane Larlus, Karteek Alahari","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Saunshi et al., 2021] Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. A Mathematical Exploration of Why Lan- guage Models Help Solve Downstream Tasks. In ICLR, 2021.","title":"A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks","authors":"Nikunj Saunshi, Sadhika Malladi, Sanjeev Arora","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In ICLR, 2015.","title":"Very Deep Convolutional Networks for Large-Scale Image Recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Sun et al., 2017] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting Unreasonable Ef- fectiveness of Data in Deep Learning Era. In ICCV, 2017.","title":"Revisiting Unreasonable Effectiveness of Data in Deep Learning Era","authors":"Chen Sun, Abhinav Shrivastava, Saurabh Singh, Abhinav Gupta","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Sutskever et al., 2013] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the Importance of Initialization and Momentum in Deep Learning. In ICML, 2013.","title":"On the Importance of Initialization and Momentum in Deep Learning","authors":"Sutskever","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Tamkin et al., 2021] Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel Fein, Colin Schultz, and Noah Good- man. DABS: a Domain-Agnostic Benchmark for Self- Supervised Learning. In NeurIPS Datasets and Bench- marks Track, 2021.","title":"DABS: a Domain-Agnostic Benchmark for Self-Supervised Learning","authors":"Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel Fein, Colin Schultz, Noah Goodman","year":"2021","venue":"NeurIPS Datasets and Benchmarks Track","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Vapnik, 2000] Vladimir Naumovich Vapnik. The Nature of Statistical Learning Theory, Second Edition. Statistics for Engineering and Information Science. Springer, 2000.","title":"The Nature of Statistical Learning Theory, Second Edition","authors":"Vladimir Naumovich, Vapnik","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Wang et al., 2019] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A Stickier Benchmark for General-Purpose Language Un- derstanding Systems. In NeurIPS, 2019.","title":"SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems","authors":"Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R Bowman","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Wattenberg et al., 2016] Martin Wattenberg, Fernanda Viégas, and Ian Johnson. How to Use t-SNE Effectively. Distill, 2016.","title":"How to Use t-SNE Effectively","authors":"Martin Wattenberg, Fernanda Viégas, Ian Johnson","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Weston et al., 2008] Jason Weston, Frédéric Ratle, and Ro- nan Collobert. Deep Learning via Semi-Supervised Em- bedding. In ICML, 2008.","title":"Deep Learning via Semi-Supervised Embedding","authors":"Jason Weston, Frédéric Ratle, Ronan Collobert","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Yosinski et al., 2014] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How Transferable Are Features in Deep Neural Networks? In NeurIPS, 2014.","title":"How Transferable Are Features in Deep Neural Networks?","authors":"Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Zhai et al., 2019] Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S 4 L: Self-Supervised Semi- Supervised Learning. In ICCV, 2019.","title":"S 4 L: Self-Supervised Semi-Supervised Learning","authors":"Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, Lucas Beyer","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Zhai et al., 2020] Xiaohua Zhai, Joan Puigcerver, Alexan- der Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neu- mann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark. arXiv:1910.04867v2 [cs.CV], 2020.","title":"A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark","authors":"Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, Neil Houlsby","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}