{"type":"ScholarlyArticle","title":"Hierarchical Feature Selection with Recursive Regularization","authors":[{"name":"Hong Zhao","slug":"hong-zhao","given":"Hong","family":"Zhao","affiliation":"Tianjin University , China; Lab of Granular Computing , Minnan Normal University , China"},{"name":"Pengfei Zhu","slug":"pengfei-zhu","given":"Pengfei","family":"Zhu","affiliation":"Tianjin University , China"},{"name":"Ping Wang","slug":"ping-wang","given":"Ping","family":"Wang","affiliation":"Tianjin University , China"},{"name":"Qinghua Hu","slug":"qinghua-hu","given":"Qinghua","family":"Hu","affiliation":"Tianjin University , China"}],"abstract":"In the big data era, the sizes of datasets have increased dramatically in terms of the number of samples, features, and classes. In particular, there exists usually a hierarchical structure among the classes. This kind of task is called hierarchical classification. Various algorithms have been developed to select informative features for flat classification. However, these algorithms ignore the semantic hyponymy in the directory of hierarchical classes, and select a uniform subset of the features for all classes. In this paper, we propose a new technique for hierarchical feature selection based on recursive regularization. This algorithm takes the hierarchical information of the class structure into account. As opposed to flat feature selection, we select different feature subsets for each node in a hierarchical tree structure using the parent-children relationships and the sibling relationships for hierarchical regularization. By imposing $\\ell_{2,1}$-norm regularization to different parts of the hierarchical classes, we can learn a sparse matrix for the feature ranking of each node. Extensive experiments on public datasets demonstrate the effectiveness of the proposed algorithm.","doi":"10.24963/ijcai.2017/487","url":"https://ijcai.org/proceedings/2017/487","pdf":"https://ijcai.org/proceedings/2017/0487.pdf","markdown":"https://ijcai.org/proceedings/2017/487.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/487","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":487,"pages":{"first":3483,"last":3489},"keywords":["Machine Learning: Classification","Machine Learning: Feature Selection/Construction"],"videos":[],"references":[{"position":1,"raw":"[Bach and Jordan, 2002] Francis R Bach and Michael I Jor- dan. Kernel independent component analysis. Journal of Machine Learning Research, 3(Jul):1-48, 2002.","title":"Kernel independent component analysis","authors":"R Francis, Michael I Bach, Jordan","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2010] Samy Bengio, Jason Weston, and David Grangier. Label embedding trees for large multi- class tasks. In Advances in Neural Information Processing Systems, pages 163-171, 2010.","title":"Label embedding trees for large multiclass tasks","authors":"Samy Bengio, Jason Weston, David Grangier","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li Jia Li, Kai Li, and Li Fei Fei. Imagenet: A large- scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. IEEE, 2009.","title":"Imagenet: A largescale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li Jia Li, Kai Li, Li Fei, Fei","year":"2009","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dimitrovski et al., 2011] Ivica Dimitrovski, Dragi Kocev, Suzana Loskovska, and Sašo Džeroski. Hierarchical annotation of medical images. Pattern Recognition, 44(10):2436-2449, 2011.","title":"Ivica Dimitrovski, Dragi Kocev, Suzana Loskovska, and Sašo Džeroski. Hierarchical annotation of medical images","authors":"Dimitrovski","year":"2011","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ding and Dubchak, 2001] Chris HQ Ding and Inna Dubchak. Multi-class protein fold recognition using sup- port vector machines and neural networks. Bioinformatics, 17(4):349-358, 2001.","title":"Multi-class protein fold recognition using support vector machines and neural networks","authors":"H Q Chris, Inna Ding, Dubchak","year":"2001","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duda et al., 2012] Richard O Duda, Peter E Hart, and David G Stork. Pattern classification. John Wiley & Sons, 2012. [","title":"Pattern classification","authors":"Peter E Richard O Duda, David G Hart, Stork","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Everingham et al., 2010] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisser- man. The pascal visual object classes (voc) challenge. In- ternational Journal of Computer Vision, 88(2):303-338, 2010.","title":"The pascal visual object classes (voc) challenge","authors":"Mark Everingham, Luc Van Gool, K I Christopher, John Williams, Andrew Winn, Zisserman","year":"2010","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Freeman et al., 2011] Cecille Freeman, Dana Kulić, and Ot- man Basir. Joint feature selection and hierarchical classi- fier design. In 2011 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 1728-1734. IEEE, 2011.","title":"Joint feature selection and hierarchical classifier design","authors":"Cecille Freeman, Dana Kulić, Otman Basir","year":"2011","venue":"IEEE International Conference on Systems, Man, and Cybernetics (SMC)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Freeman et al., 2013] C Freeman, D Kuli, and O Basir. Feature-selected tree-based classification. IEEE Transac- tions on Cybernetics, 43(6):1990-2004, 2013.","title":"Feature-selected tree-based classification","authors":"D Freeman, Kuli, Basir","year":"2013","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gopal and Yang, 2015] Siddharth Gopal and Yi Ming Yang. Hierarchical bayesian inference and recursive regulariza- tion for large-scale classification. ACM Transactions on Knowledge Discovery from Data, 9(3):18, 2015.","title":"Hierarchical bayesian inference and recursive regularization for large-scale classification","authors":"Siddharth Gopal, Yi Ming, Yang","year":"2015","venue":"ACM Transactions on Knowledge Discovery from Data","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gretton et al., 2005] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statis- tical dependence with hilbert-schmidt norms. In Interna- tional Conference on Algorithmic Learning Theory, pages 63-77. Springer, 2005.","title":"Measuring statistical dependence with hilbert-schmidt norms","authors":"Arthur Gretton, Olivier Bousquet, Alex Smola, Bernhard Schölkopf","year":"2005","venue":"International Conference on Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kira and Rendell, 1992] Kenji Kira and Larry A Rendell. A practical approach to feature selection. In Proceedings of the Ninth International Workshop on Machine Learning, pages 249-256, 1992.","title":"A practical approach to feature selection","authors":"Kenji Kira, Larry A Rendell","year":"1992","venue":"Proceedings of the Ninth International Workshop on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kosmopoulos et al., 2015] Aris Kosmopoulos, Ioannis Par- talas, Eric Gaussier, Georgios Paliouras, and Ion Androut- sopoulos. Evaluation measures for hierarchical classifica- tion: a unified view and novel approaches. Data Mining and Knowledge Discovery, 29(3):820-865, 2015.","title":"Evaluation measures for hierarchical classification: a unified view and novel approaches","authors":"Aris Kosmopoulos, Ioannis Partalas, Eric Gaussier, Georgios Paliouras","year":"2015","venue":"Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. 2009.","title":"Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images","authors":"Hinton Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Miller, 1995] George A Miller. Wordnet: a lexical database for english. Communications of the Acm, 38(11):39-41, 1995.","title":"Wordnet: a lexical database for english","authors":"George A Miller","year":"1995","venue":"Communications of the Acm","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nie et al., 2010] Fei Ping Nie, Heng Huang, Xiao Cai, and Chris H Ding. Efficient and robust feature selection via joint ℓ 2,1 -norms minimization. In Advances in Neural In- formation Processing Systems, pages 1813-1821, 2010.","title":"Efficient and robust feature selection via joint ℓ 2,1 -norms minimization","authors":"Ping Fei, Heng Nie, Xiao Huang, Chris H Cai, Ding","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Peng et al., 2005] Han Chuan Peng, Fu Hui Long, and Chris Ding. Feature selection based on mutual informa- tion criteria of max-dependency, max-relevance, and min- redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8):1226-1238, 2005.","title":"Feature selection based on mutual information criteria of max-dependency, max-relevance, and minredundancy","authors":"Fu Han Chuan Peng, Chris Hui Long, Ding","year":"2005","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Silla and Freitas, 2011] Carlos N. Silla and Alex A. Freitas. A survey of hierarchical classification across different ap- plication domains. Data Mining and Knowledge Discov- ery, 22(1-2):31-72, 2011.","title":"A survey of hierarchical classification across different application domains","authors":"Carlos N Silla, Alex A Freitas","year":"2011","venue":"Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Song et al., 2015] Jia Song, Pengzhou Zhang, Sijun Qin, and Junpeng Gong. A method of the feature selection in hierarchical text classification based on the category discrimination and position information. In International Conference on Industrial Informatics -Computing Tech- nology, Intelligent Technology, Industrial Information In- tegration, pages 132-135, 2015.","title":"A method of the feature selection in hierarchical text classification based on the category discrimination and position information","authors":"Jia Song, Pengzhou Zhang, Sijun Qin, Junpeng Gong","year":"2015","venue":"International Conference on Industrial Informatics -Computing Technology, Intelligent Technology, Industrial Information Integration","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tang and Liu, 2014] Jiliang Tang and Huan Liu. An unsu- pervised feature selection framework for social media da- ta. IEEE Transactions on Knowledge and Data Engineer- ing, 26(12):2914-2927, 2014.","title":"An unsupervised feature selection framework for social media data","authors":"Liu ; ] Jiliang Tang, Huan Tang, Liu","year":"2014","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Villela et al., 2015] Saulo Moraes Villela, De Castro Leite Saul, and Raul Fonseca Neto. Feature selection from mi- croarray data via an ordered search with projected margin. In Proceedings of the Twenty-Fourth International Join- t Conference on Artificial Intelligence, pages 3874-3881, 2015.","title":"Feature selection from microarray data via an ordered search with projected margin","authors":"Moraes Saulo, De Villela, Castro Leite, Raul Fonseca Saul, Neto","year":"2015","venue":"Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2015/544"},{"position":22,"raw":"[Wang and Guo, 2017] Shi Ping Wang and Wen Zhong Guo. Sparse multi-graph embedding for multimodal feature rep- resentation. IEEE Transactions on Multimedia, 2017.","title":"Sparse multi-graph embedding for multimodal feature representation","authors":"Ping Shi, Wen Wang, Guo Zhong","year":"2017","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wei et al., 2015] Leyi Wei, Minghong Liao, Xing Gao, and Quan Zou. An improved protein structural classes pre- diction method by incorporating both sequence and struc- ture information. IEEE Transactions on Nanobioscience, 14(4):339-349, 2015.","title":"An improved protein structural classes prediction method by incorporating both sequence and structure information","authors":"Wei","year":"2015","venue":"IEEE Transactions on Nanobioscience","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xiao et al., 2010] Jianxiong Xiao, James Hays, Krista A E- hinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 3485-3492. IEEE, 2010.","title":"Sun database: Large-scale scene recognition from abbey to zoo","authors":"Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, Antonio Torralba","year":"2010","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2011] Yi Yang, Heng Tao Shen, Zhi Gang Ma, Zi Huang, and Xiao Fang Zhou. ℓ 2,1 -norm regularized discriminative feature selection for unsupervised learning. In Proceedings of the Twenty-Second International Join- t Conference on Artificial Intelligence, volume 22, pages 1589-1594, 2011.","title":"ℓ 2,1 -norm regularized discriminative feature selection for unsupervised learning","authors":"Yi Yang, Heng Tao Shen, Zhi Gang Ma, Zi Huang, Xiao Fang Zhou","year":"2011","venue":"Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}