{"type":"ScholarlyArticle","title":"Deep Neural Networks for High Dimension, Low Sample Size Data","authors":[{"name":"Bo Liu","slug":"bo-liu","given":"Bo","family":"Liu","affiliation":"University of Science and Technology , Hong Kong"},{"name":"Ying Wei","slug":"ying-wei","given":"Ying","family":"Wei","affiliation":"University of Science and Technology , Hong Kong"},{"name":"Yu Zhang","slug":"yu-zhang","given":"Yu","family":"Zhang","affiliation":"University of Science and Technology , Hong Kong"},{"name":"Qiang Yang","slug":"qiang-yang","given":"Qiang","family":"Yang","affiliation":"University of Science and Technology , Hong Kong"}],"abstract":"Deep neural networks (DNN) have achieved breakthroughs in applications with large sample size. However, when facing high dimension, low sample size (HDLSS) data, such as the phenotype prediction problem using genetic data in bioinformatics, DNN suffers from overfitting and high-variance gradients. In this paper, we propose a DNN model tailored for the HDLSS data, named Deep Neural Pursuit (DNP). DNP selects a subset of high dimensional features for the alleviation of overfitting and takes the average over multiple dropouts to calculate gradients with low variance. As the first DNN method applied on the HDLSS data, DNP enjoys the advantages of the high nonlinearity, the robustness to high dimensionality, the capability of learning from a small number of samples, the stability in feature selection, and the end-to-end training. We demonstrate these advantages of DNP via empirical results on both synthetic and real-world biological datasets.","doi":"10.24963/ijcai.2017/318","url":"https://ijcai.org/proceedings/2017/318","pdf":"https://ijcai.org/proceedings/2017/0318.pdf","markdown":"https://ijcai.org/proceedings/2017/318.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/318","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":318,"pages":{"first":2287,"last":2293},"keywords":["Multidisciplinary Topics and Applications: Computational Biology and e-Health","Machine Learning: Deep Learning","Machine Learning: Feature Selection/Construction"],"videos":[],"references":[{"position":1,"raw":"[Alipanahi et al., 2015] Babak Alipanahi, Andrew Delong, Matthew T Weirauch, and Brendan J Frey. Predicting the sequence specificities of DNA-and RNA-binding proteins by deep learning. Nature Biotechnology, 2015.","title":"Predicting the sequence specificities of DNA-and RNA-binding proteins by deep learning","authors":"Babak Alipanahi, Andrew Delong, Matthew T Weirauch, Brendan J Frey","year":"2015","venue":"Nature Biotechnology","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amaldi and Kann, 1998] Edoardo Amaldi and Viggo Kann. On the approximability of minimizing nonzero variables or unsatis- fied relations in linear systems. Theoretical Computer Science, 209(1):237-260, 1998.","title":"On the approximability of minimizing nonzero variables or unsatisfied relations in linear systems","authors":"Kann Amaldi, Edoardo Amaldi, Viggo Kann","year":"1998","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bach, 2014] Francis Bach. Breaking the curse of dimensionality with convex neural networks. arXiv:1412.8690, 2014.","title":"Breaking the curse of dimensionality with convex neural networks","authors":"Francis Bach","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bengio et al., 2005] Yoshua Bengio, Nicolas L Roux, Pascal Vin- cent, Olivier Delalleau, and Patrice Marcotte. Convex neural net- works. In Advances in Neural Information Processing Systems, pages 123-130, 2005.","title":"Convex neural networks","authors":"Yoshua Bengio, Nicolas L Roux, Pascal Vincent, Olivier Delalleau, Patrice Marcotte","year":"2005","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Breiman, 2001] Leo Breiman. Random forests. Machine Learn- ing, 45(1):5-32, 2001.","title":"Leo Breiman. Random forests","authors":"Breiman","year":"2001","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Consortium, 2015] 1000 Genomes Project Consortium. A global reference for human genetic variation. Nature, 526(7571):68-74, 2015.","title":"Genomes Project Consortium. A global reference for human genetic variation","authors":null,"year":"1000","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Duchi et al., 2011] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121- 2159, 2011.","title":"Adaptive subgradient methods for online learning and stochastic optimization","authors":"John Duchi, Elad Hazan, Yoram Singer","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Evgeniou and Pontil, 2007] A Evgeniou and Massimiliano Pontil. Multi-task feature learning. Advances in Neural Information Pro- cessing Systems, 19:41, 2007.","title":"A Evgeniou and Massimiliano Pontil. Multi-task feature learning","authors":"Pontil Evgeniou","year":"2007","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fan and Li, 2006] Jianqing Fan and Runze Li. Statistical chal- lenges with high dimensionality: feature selection in knowledge discovery. arXiv:math/0602133, 2006.","title":"Jianqing Fan and Runze Li. Statistical challenges with high dimensionality: feature selection in knowledge discovery","authors":"Fan, Li","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Friedman et al., 2000] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The Elements of Statistical Learning. first edition, 2000.","title":"The Elements of Statistical Learning","authors":"Jerome Friedman, Trevor Hastie, Robert Tibshirani","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Friedman, 2001] Jerome Friedman. Greedy function approxima- tion: a gradient boosting machine. Annals of Statistics, pages 1189-1232, 2001.","title":"Greedy function approximation: a gradient boosting machine","authors":"Jerome Friedman, Friedman","year":"2001","venue":"Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Bengio. Un- derstanding the difficulty of training deep feedforward neural net- works. In Proceedings of International Conference on Artificial Intelligence and Statistics, pages 249-256, 2010.","title":"Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot","year":"2010","venue":"Proceedings of International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of International Conference on Artificial Intelligence and Statis- tics, pages 315-323, 2011.","title":"Deep sparse rectifier neural networks","authors":"Xavier Glorot, Antoine Bordes, Yoshua Bengio","year":"2011","venue":"Proceedings of International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Guyon and Elisseeff, 2003] Isabelle Guyon and André Elisseeff. An introduction to variable and feature selection. Journal of Ma- chine Learning Research, 3:1157-1182, 2003.","title":"An introduction to variable and feature selection","authors":"Elisseeff Guyon, Isabelle Guyon, André Elisseeff","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv:1512.03385, 2015.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hinton et al., 2012] Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82-97, 2012.","title":"Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups","authors":"Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-Rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, Brian Kingsbury","year":"2012","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kalousis et al., 2007] Alexandros Kalousis, Julien Prados, and Melanie Hilario. Stability of feature selection algorithms: a study on high-dimensional spaces. Knowledge and Information Sys- tems, 12(1):95-116, 2007.","title":"Stability of feature selection algorithms: a study on high-dimensional spaces","authors":"Kalousis","year":"2007","venue":"Knowledge and Information Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Leung et al., 2016] Michael KK Leung, Andrew Delong, Babak Alipanahi, and Brendan J Frey. Machine learning in genomic medicine: a review of computational problems and data sets. Pro- ceedings of the IEEE, 104(1):176-197, Jan 2016.","title":"Machine learning in genomic medicine: a review of computational problems and data sets","authors":"Andrew Michael Kk Leung, Babak Delong, Brendan J Alipanahi, Frey","year":"2016","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2005] Fan Li, Yiming Yang, and Eric P Xing. From lasso regression to feature vector machine. In Advances in Neural In- formation Processing Systems, pages 779-786, 2005.","title":"From lasso regression to feature vector machine","authors":"Fan Li, Yiming Yang, Eric P Xing","year":"2005","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li et al., 2015] Yifeng Li, Chih-Yu Chen, and Wyeth W Wasser- man. Deep feature selection: Theory and application to iden- tify enhancers and promoters. In International Conference on Research in Computational Molecular Biology, pages 205-217. Springer, 2015.","title":"Deep feature selection: Theory and application to identify enhancers and promoters","authors":"Yifeng Li, Chih-Yu Chen, Wyeth W Wasserman","year":"2015","venue":"International Conference on Research in Computational Molecular Biology","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pati et al., 1993] Yagyensh Chandra Pati, Ramin Rezaiifar, and PS Krishnaprasad. Orthogonal matching pursuit: recursive func- tion approximation with applications to wavelet decomposition. In Proceedings of 27th Asilomar Conference on Signals, Systems and Computers, pages 40-44, 1993.","title":"Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition","authors":"Pati","year":"1993","venue":"Proceedings of 27th Asilomar Conference on Signals, Systems and Computers","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Perkins et al., 2003] Simon Perkins, Kevin Lacker, and James Theiler. Grafting: Fast, incremental feature selection by gradi- ent descent in function space. Journal of Machine Learning Re- search, 3:1333-1356, 2003.","title":"Grafting: Fast, incremental feature selection by gradient descent in function space","authors":"Simon Perkins, Kevin Lacker, James Theiler","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ravikumar et al., 2007] Pradeep Ravikumar, Han Liu, John Laf- ferty, and Larry Wasserman. Spam: Sparse additive models. In Advances in Neural Information Processing Systems, 2007.","title":"Spam: Sparse additive models","authors":"Pradeep Ravikumar, Han Liu, John Lafferty, Larry Wasserman","year":"2007","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tewari et al., 2011] Ambuj Tewari, Pradeep K Ravikumar, and In- derjit S Dhillon. Greedy algorithms for structurally constrained high dimensional problems. In Advances in Neural Information Processing Systems, pages 882-890, 2011.","title":"Greedy algorithms for structurally constrained high dimensional problems","authors":"Ambuj Tewari, Inderjit S Pradeep K Ravikumar, Dhillon","year":"2011","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tibshirani, 1996] Robert Tibshirani. Regression shrinkage and se- lection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), pages 267-288, 1996.","title":"Regression shrinkage and selection via the lasso","authors":"Robert Tibshirani","year":"1996","venue":"Journal of the Royal Statistical Society. Series B (Methodological)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xiong et al., 2015] Hui Y Xiong, Babak Alipanahi, Leo J Lee, Hannes Bretschneider, Daniele Merico, Ryan KC Yuen, Yimin Hua, Serge Gueroussov, Hamed S Najafabadi, Timothy R Hughes, et al. The human splicing code reveals new in- sights into the genetic determinants of disease. Science, 347(6218):1254806, 2015.","title":"The human splicing code reveals new insights into the genetic determinants of disease","authors":"Babak Hui Y Xiong, Leo J Alipanahi, Hannes Lee, Daniele Bretschneider, Merico, Yimin Ryan Kc Yuen, Serge Hua, Gueroussov, S Hamed, Timothy R Najafabadi, Hughes","year":"2015","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xu et al., 2014] Zhixiang Xu, Gao Huang, Kilian Q Weinberger, and Alice X Zheng. Gradient boosted feature selection. In Pro- ceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 522-531. ACM, 2014.","title":"Gradient boosted feature selection","authors":"Zhixiang Xu, Gao Huang, Kilian Q Weinberger, Alice X Zheng","year":"2014","venue":"Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yamada et al., 2014] Makoto Yamada, Wittawat Jitkrittum, Leonid Sigal, Eric P Xing, and Masashi Sugiyama. High- dimensional feature selection by feature-wise kernelized lasso. Neural Computation, 26(1):185-207, 2014.","title":"Highdimensional feature selection by feature-wise kernelized lasso","authors":"Makoto Yamada, Wittawat Jitkrittum, Leonid Sigal, Eric P Xing, Masashi Sugiyama","year":"2014","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yamada et al., 2016] Makoto Yamada, Jiliang Tang, Jose Lugo- Martinez, Ermin Hodzic, Raunak Shrestha, Avishek Saha, Hua Ouyang, Dawei Yin, Hiroshi Mamitsuka, Cenk Sahinalp, Pre- drag Radivojac, Filippo Menczer, and Yi Chang. Ultra high- dimensional nonlinear feature selection for big biological data. arXiv:1608.04048, 2016.","title":"Ultra highdimensional nonlinear feature selection for big biological data","authors":"Makoto Yamada, Jiliang Tang, Jose Lugo-Martinez, Ermin Hodzic, Raunak Shrestha, Avishek Saha, Hua Ouyang, Dawei Yin, Hiroshi Mamitsuka, Cenk Sahinalp, Predrag Radivojac, Filippo Menczer, Yi Chang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}