{"type":"ScholarlyArticle","title":"Linear Dimensionality Reduction for Multi-label Classification","authors":[{"name":"Shuiwang Ji","slug":"shuiwang-ji","given":"Shuiwang","family":"Ji","affiliation":"Arizona State University"},{"name":"Jieping Ye","slug":"jieping-ye","given":"Jieping","family":"Ye","affiliation":"Arizona State University"}],"abstract":"Dimensionality reduction is an essential step in high-dimensional data analysis. Many dimensionality reduction algorithms have been applied successfully to multi-class and multi-label problems. They are commonly applied as a separate data preprocessing step before classification algorithms. In this paper, we study a joint learning framework in which we perform dimensionality reduction and multi-label classification simultaneously. We show that when the least squares loss is used in classification, this joint learning decouples into two separate components, i.e., dimensionality reduction followed by multi-label classification. This analysis partially justifies the current practice of a separate application of dimensionality reduction for classification problems. We extend our analysis using other loss functions, including the hinge loss and the squared hinge loss. We further extend the formulation to the more general case where the input data for different class labels may differ, overcoming the limitation of traditional dimensionality reduction algorithms. Experiments on benchmark data sets have been conducted to evaluate the proposed joint formulations.Shuiwang Ji, Jieping Ye","doi":null,"url":"https://ijcai.org/proceedings/2009/182","pdf":"https://ijcai.org/proceedings/2009/0182.pdf","markdown":"https://ijcai.org/proceedings/2009/182.md","bibtex":"https://ijcai.org/proceedings/2009/bibtex/182","volume":{"slug":"2009","title":"Proceedings of the Twenty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2009","url":"https://ijcai.org/proceedings/2009/","year":2009,"location":"Pasadena, California, USA","dates":"","editors":"Hiroaki Kitano","isbn":"","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":null,"track":"","number":182,"pages":{"first":1077,"last":1081},"keywords":[],"videos":[],"references":[{"position":1,"raw":"[Amit et al., 2007] Y. Amit, M. Fink, N. Srebro, and S. Ull- man. Uncovering shared structures in multiclass classifi- cation. In ICML, pages 17-24, 2007.","title":"Uncovering shared structures in multiclass classification","authors":"Y Amit, M Fink, N Srebro, S Ullman","year":"2007","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ando and Zhang, 2005] R. K. Ando and T. Zhang. A frame- work for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Re- search, 6:1817-1853, 2005.","title":"A framework for learning predictive structures from multiple tasks and unlabeled data","authors":"R K Ando, T Zhang","year":"2005","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Arenas-García et al., 2007] J. Arenas-García, K. B. Pe- tersen, and L. K. Hansen. Sparse kernel orthonormalized PLS for feature extraction in large data sets. In NIPS, pages 33-40. 2007.","title":"Sparse kernel orthonormalized PLS for feature extraction in large data sets","authors":"J Arenas-García, K B Petersen, L K Hansen","year":"2007","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Argyriou et al., 2007] Andreas Argyriou, Theodoros Evge- niou, and Massimiliano Pontil. Convex multi-task feature learning. Machine Learning, 2007.","title":"Convex multi-task feature learning","authors":"Andreas Argyriou, Theodoros Evgeniou, Massimiliano Pontil","year":"2007","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fukunaga, 1990] K. Fukunaga. Introduction to statistical pattern recognition. Academic Press Professional, 2nd edition, 1990.","title":"Introduction to statistical pattern recognition","authors":"K Fukunaga","year":"1990","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Golub and Van Loan, 1996] G. H. Golub and C F. Van Loan. Matrix Computations. The Johns Hopkins Univer- sity Press, 3rd edition, 1996.","title":"Matrix Computations","authors":"G H Golub, C F Van Loan","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hotelling, 1936] H. Hotelling. Relations between two sets of variates. Biometrika, 28(3-4):321-377, 1936.","title":"Relations between two sets of variates","authors":"H Hotelling","year":"1936","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jolliffe, 2002] I. T. Jolliffe. Principal Component Analysis. Springer-Verlag, New York, 2nd edition, 2002.","title":"Principal Component Analysis","authors":"I T Jolliffe","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lanckriet et al., 2004] G. R. G. Lanckriet, N. Cristianini, P. Bartlett, L. E. Ghaoui, and M. I. Jordan. Learning the kernel matrix with semidefinite programming. Journal of Machine Learning Research, 5:27-72, 2004.","title":"Learning the kernel matrix with semidefinite programming","authors":"G R G Lanckriet, N Cristianini, P Bartlett, L E Ghaoui, M I Jordan","year":"2004","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[McCallum, 1999] A. McCallum. Multi-label text classifica- tion with a mixture model trained by EM. In AAAI Work- shop on Text Learning, 1999.","title":"Multi-label text classification with a mixture model trained by EM","authors":"A Mccallum","year":"1999","venue":"AAAI Workshop on Text Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Nemirovski, 1994] A. Nemirovski. Efficient methods in convex programming, 1994. Lecture Notes.","title":"Efficient methods in convex programming","authors":"A Nemirovski","year":"1994","venue":"Lecture Notes","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Overton, 1993] M. L. Overton. Optimality conditions and duality theory for minimizing sums of the largest eigen- values of symmetric matrices. Math. Programming, 62(1):321-357, 1993.","title":"Optimality conditions and duality theory for minimizing sums of the largest eigenvalues of symmetric matrices","authors":"M L Overton","year":"1993","venue":"Math. Programming","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Rifkin and Klautau, 2004] Ryan Rifkin and Aldebaro Klau- tau. In defense of one-vs-all classification. Journal of Ma- chine Learning Research, 5:101-141, 2004.","title":"In defense of one-vs-all classification","authors":"Ryan Rifkin, Aldebaro Klautau","year":"2004","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sun et al., 2008] L. Sun, S. Ji, and J. Ye. A least squares formulation for canonical correlation analysis. In ICML, pages 1024-1031, 2008.","title":"A least squares formulation for canonical correlation analysis","authors":"L Sun, S Ji, J Ye","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ueda and Saito, 2003] N. Ueda and K. Saito. Parametric mixture models for multi-labeled text. In NIPS, pages 721-728. 2003.","title":"Parametric mixture models for multi-labeled text","authors":"N Ueda, K Saito","year":"2003","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wu et al., 2006] M. Wu, B. Schölkopf, and G. Bakir. A di- rect method for building sparse kernel learning algorithms. Journal of Machine Learning Research, 7:603-624, 2006.","title":"A direct method for building sparse kernel learning algorithms","authors":"M Wu, B Schölkopf, G Bakir","year":"2006","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xue et al., 2007] Y. Xue, X. Liao, L. Carin, and B. Krishna- puram. Multi-task learning for classification with Dirich- let process priors. Journal of Machine Learning Research, 8:35-63, 2007.","title":"Multi-task learning for classification with Dirichlet process priors","authors":"Y Xue, X Liao, L Carin, B","year":"2007","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhang and Zhou, 2008] Y. Zhang and Z.-H. Zhou. Multi- label dimensionality reduction via dependency maximiza- tion. In AAAI, pages 1503-1505, 2008.","title":"Multilabel dimensionality reduction via dependency maximization","authors":"Y Zhang, Z.-H Zhou","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}