{"type":"ScholarlyArticle","title":"Learning Feature Engineering for Classification","authors":[{"name":"Fatemeh Nargesian","slug":"fatemeh-nargesian","given":"Fatemeh","family":"Nargesian","affiliation":"University of Toronto ,"},{"name":"Horst Samulowitz","slug":"horst-samulowitz","given":"Horst","family":"Samulowitz","affiliation":"IBM Research ,"},{"name":"Udayan Khurana","slug":"udayan-khurana","given":"Udayan","family":"Khurana","affiliation":"IBM Research ,"},{"name":"Elias B. Khalil","slug":"elias-b-khalil","given":"Elias B.","family":"Khalil","affiliation":"Georgia Institute of Technology"},{"name":"Deepak Turaga","slug":"deepak-turaga","given":"Deepak","family":"Turaga","affiliation":"IBM Research ,"}],"abstract":"Feature engineering is the task of improving predictive modelling performance on a dataset by transforming its feature space. Existing approaches to automate this process rely on either transformed feature space exploration through evaluation-guided search, or explicit expansion of datasets with all transformed features followed by feature selection. Such approaches incur high computational costs in runtime and/or memory. We present a novel technique, called Learning Feature Engineering (LFE), for automating feature engineering in classification tasks. LFE is based on learning the effectiveness of applying a transformation (e.g., arithmetic or aggregate operators) on numerical features, from past feature engineering experiences. Given a new dataset, LFE recommends a set of useful transformations to be applied on features without relying on model evaluation or explicit feature expansion and selection. Using a collection of datasets, we train a set of neural networks, which aim at predicting the transformation that impacts classification performance positively. Our empirical results show that LFE outperforms other feature engineering approaches for an overwhelming majority (89%) of the datasets from various sources while incurring a substantially lower computational cost.","doi":"10.24963/ijcai.2017/352","url":"https://ijcai.org/proceedings/2017/352","pdf":"https://ijcai.org/proceedings/2017/0352.pdf","markdown":"https://ijcai.org/proceedings/2017/352.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/352","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":352,"pages":{"first":2529,"last":2535},"keywords":["Machine Learning: Classification","Machine Learning: Feature Selection/Construction"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE TPAMI, 35(8), 2013.","title":"Representation learning: A review and new perspectives","authors":"Yoshua Bengio, Aaron Courville, Pascal Vincent","year":"2013","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bingham and Mannila, 2001] Ella Bingham and Heikki Mannila. Random projection in dimensionality reduction: Applications to image and text data. KDD, pages 245-250, 2001.","title":"Random projection in dimensionality reduction: Applications to image and text data","authors":"Mannila Bingham, Ella Bingham, Heikki Mannila","year":"2001","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Che et al., 2015] Zhengping Che, Sanjay Purushotham, Robinder Khemani, and Yan Liu. Distilling knowledge from deep networks with applications to healthcare do- main. arXiv preprint arXiv:1512.03542, 2015.","title":"Distilling knowledge from deep networks with applications to healthcare domain","authors":"Zhengping Che, Sanjay Purushotham, Robinder Khemani, Yan Liu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dor and Reich, 2012] Ofer Dor and Yoram Reich. Strength- ening learning algorithms by feature discovery. Informa- tion Sciences, 2012.","title":"Strengthening learning algorithms by feature discovery","authors":"Reich Dor","year":"2012","venue":"Ofer Dor and Yoram Reich","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fan et al., 2010] Wei Fan, Erheng Zhong, Jing Peng, Olivier Verscheure, Kun Zhang, Jiangtao Ren, Rong Yan, and Qiang Yang. Generalized and heuristic-free feature con- struction for improved accuracy. SDM, 2010.","title":"Generalized and heuristic-free feature construction for improved accuracy","authors":"Wei Fan, Erheng Zhong, Jing Peng, Olivier Verscheure, Kun Zhang, Jiangtao Ren, Rong Yan, Qiang Yang","year":"2010","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Feurer et al., 2015] Matthias Feurer, Aaron Klein, Katha- rina Eggensperger, Jost Tobias Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. NIPS, 2015.","title":"Efficient and robust automated machine learning","authors":"Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, Frank Hutter","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Garrido and Juste, 1998] Lluis Garrido and Aurelio Juste. On the determination of probability density functions by using neural networks. Computer Physics Communica- tions, 115(1):25 -31, 1998.","title":"On the determination of probability density functions by using neural networks","authors":"Juste Garrido, Lluis Garrido, Aurelio Juste","year":"1998","venue":"Computer Physics Communications","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Greenwald and Khanna, 2001] Michael Greenwald and Sanjeev Khanna. Space-efficient online computation of quantile summaries. SIGMOD, pages 58-66, 2001.","title":"Space-efficient online computation of quantile summaries","authors":"Khanna Greenwald, Michael Greenwald, Sanjeev Khanna","year":"2001","venue":"SIGMOD","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kalousis, 2002] Alexandros Kalousis. Algorithm selection via meta-learning. PhD thesis, Universite de Geneve, 2002.","title":"Algorithm selection via meta-learning","authors":"Alexandros Kalousis","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kanter and Veeramachaneni, 2015] James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automating data science endeavors. DSAA, 2015.","title":"Deep feature synthesis: Towards automating data science endeavors","authors":"Veeramachaneni ; Kanter, Max James, Kalyan Kanter, Veeramachaneni","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Katz et al., 2016] Gilad Katz, Eui Chul Richard Shin, and Dawn Song. Explorekit: Automatic feature generation and selection. ICDM, pages 979-984, 2016.","title":"Explorekit: Automatic feature generation and selection","authors":"Gilad Katz, Eui Chul, Richard Shin, Dawn Song","year":"2016","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Khurana et al., 2016] Udayan Khurana, Deepak Turaga, Horst Samulowitz, and Srinivasan Parthasarathy. Cognito: Automated Feature Engineering for Supervised Learning. ICDM, 2016.","title":"Cognito: Automated Feature Engineering for Supervised Learning","authors":"Udayan Khurana, Deepak Turaga, Horst Samulowitz, Srinivasan Parthasarathy","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lichman, 2013] M. Lichman. UCI machine learning repos- itory, 2013.","title":"UCI machine learning repository","authors":"M Lichman","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Markovitch and Rosenstein, 2002] Shaul Markovitch and Dan Rosenstein. Feature generation using general con- structor functions. Machine Learning, 2002.","title":"Feature generation using general constructor functions","authors":"Shaul Markovitch, Dan Rosenstein","year":"2002","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Michie et al., 1994] Donald Michie, D. J. Spiegelhalter, C. C. Taylor, and John Campbell, editors. Machine Learn- ing, Neural and Statistical Classification. 1994.","title":"Machine Learning","authors":"D J Donald Michie, C C Spiegelhalter, John Taylor, Campbell","year":"1994","venue":"Neural and Statistical Classification","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nair and Hinton, 2010] Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann ma- chines. ICML, pages 807-814, 2010.","title":"Rectified linear units improve restricted boltzmann machines","authors":"Vinod Nair, Geoffrey E Hinton","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Naumann et al., 2002] F. Naumann, C.-T. Ho, X. Tian, L. Haas, and N Megiddo. Attribute classification using feature analysis. ICDE, page 271, 2002.","title":"Attribute classification using feature analysis","authors":"F Naumann, C.-T Ho, X Tian, L Haas, Megiddo","year":"2002","venue":"ICDE","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rifkin and Klautau, 2004] Ryan Rifkin and Aldebaro Klau- tau. In defense of one-vs-all classification. JMLR, 5:101- 141, 2004.","title":"In defense of one-vs-all classification","authors":"Ryan Rifkin, Aldebaro Klautau","year":"2004","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shawe-Taylor and Cristianini, 2004] John Shawe-Taylor and Nello Cristianini. Kernel methods for pattern analysis. Cambridge university press, 2004.","title":"Kernel methods for pattern analysis","authors":"John Shawe, - Taylor, Nello Cristianini","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Skinner et al., 1994] C.J. Skinner, D.J. Holmes, and D. Holt. Multiple frame sampling for multivariate stratification. International Statistical Review, 62(3), 1994.","title":"Multiple frame sampling for multivariate stratification","authors":"C J Skinner, D J Holmes, D Holt","year":"1994","venue":"International Statistical Review","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov. Dropout: A simple way to prevent neural net- works from overfitting. Journal of Machine Learning, 15(1):1929-1958, 2014.","title":"Dropout: A simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Storcheus et al., 2015] Dmitry Storcheus, Afshin Ros- tamizadeh, and Sanjiv Kumar. A survey of modern ques- tions and challenges in feature extraction. Proceedings of The 1st International Workshop on Feature Extraction, NIPS, 2015.","title":"A survey of modern questions and challenges in feature extraction","authors":"Dmitry Storcheus, Afshin Rostamizadeh, Sanjiv Kumar","year":"2015","venue":"Proceedings of The 1st International Workshop on Feature Extraction","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vanschoren et al., 2014] Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. Openml: Networked science in machine learning. SIGKDD Explor. Newsl., 15(2):49-60, June 2014.","title":"Openml: Networked science in machine learning","authors":"Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, Luis Torgo","year":"2014","venue":"SIGKDD Explor. Newsl","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2013] Lu Wang, Ge Luo, Ke Yi, and Graham Cormode. Quantiles over data streams: An experimental study. SIGMOD, pages 737-748, 2013.","title":"Quantiles over data streams: An experimental study","authors":"Lu Wang, Ge Luo, Ke Yi, Graham Cormode","year":"2013","venue":"SIGMOD","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Weinberger et al., 2009] Kilian Weinberger, Anirban Das- gupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. ICML, 2009.","title":"Feature hashing for large scale multitask learning","authors":"Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, Josh Attenberg","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}