{"type":"ScholarlyArticle","title":"A Fast Adaptive Randomized PCA Algorithm","authors":[{"name":"Xu Feng","slug":"xu-feng","given":"Xu","family":"Feng","affiliation":"Department of Computer Science and Technology , BNRist , Tsinghua University , Beijing , China"},{"name":"Wenjian Yu","slug":"wenjian-yu","given":"Wenjian","family":"Yu","affiliation":"Department of Computer Science and Technology , BNRist , Tsinghua University , Beijing , China"}],"abstract":"It is desirable to adaptively determine the number of dimensions (rank) for PCA according to a given tolerance of low-rank approximation error. In this work, we aim to develop a fast algorithm solving this adaptive PCA problem. We propose to replace the QR factorization in randQB_EI algorithm with matrix multiplication and inversion of small matrices, and propose a new error indicator to incrementally evaluate approximation error in Frobenius norm. Combining the shifted power iteration technique for better accuracy, we finally build up an algorithm named farPCA. Experimental results show that farPCA is much faster than the baseline methods (randQB_EI, randUBV and svds) in practical setting of multi-thread computing, while producing nearly optimal results of adpative PCA.","doi":"10.24963/ijcai.2023/411","url":"https://ijcai.org/proceedings/2023/411","pdf":"https://ijcai.org/proceedings/2023/0411.pdf","markdown":"https://ijcai.org/proceedings/2023/411.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/411","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":411,"pages":{"first":3695,"last":3704},"keywords":["Machine Learning: ML: Feature extraction, selection and dimensionality reduction","Data Mining: DM: Big data and scalability","Data Mining: DM: Theoretical foundations of data mining"],"videos":[],"references":[{"position":1,"raw":"[Ailon and Chazelle, 2006] Nir Ailon and Bernard Chazelle. Approximate nearest neighbors and the fast johnson- lindenstrauss transform. In Proceedings of the thirty- eighth annual ACM symposium on Theory of computing, pages 557-563, 2006.","title":"Approximate nearest neighbors and the fast johnsonlindenstrauss transform","authors":"Ailon, Chazelle, Nir Ailon, Bernard Chazelle","year":"2006","venue":"Proceedings of the thirtyeighth annual ACM symposium on Theory of computing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Allen-Zhu and Li, 2016] Zeyuan Allen-Zhu and Yuanzhi Li. LazySVD: Even faster SVD decomposition yet with- out agonizing pain. In Advances in Neural Information Processing Systems, pages 974-982, 2016.","title":"LazySVD: Even faster SVD decomposition yet without agonizing pain","authors":"Allen-Zhu And Li, ; ] Zeyuan Allen-Zhu, Yuanzhi Li","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Benjamin Erichson et al., 2017] N. Benjamin Erichson, Steven L. Brunton, and J. Nathan Kutz. Compressed singular value decomposition for image and video pro- cessing. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 1880-1888, Oct. 2017.","title":"Compressed singular value decomposition for image and video processing","authors":"N Benjamin Erichson, Steven L Brunton, J Nathan Kutz","year":"2017","venue":"Proc. IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bjarkason, 2019] Elvar K Bjarkason. Pass-efficient ran- domized algorithms for low-rank matrix approximation using any number of views. SIAM Journal on Scientific Computing, 41(4):A2355-A2383, 2019.","title":"Pass-efficient randomized algorithms for low-rank matrix approximation using any number of views","authors":"Elvar K Bjarkason","year":"2019","venue":"SIAM Journal on Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Demmel et al., 2012] James Demmel, Laura Grigori, Mark Hoemmen, and Julien Langou. Communication-optimal parallel and sequential qr and lu factorizations. SIAM Jour- nal on Scientific Computing, 34(1):A206-A239, 2012.","title":"Communication-optimal parallel and sequential qr and lu factorizations","authors":"James Demmel, Laura Grigori, Mark Hoemmen, Julien Langou","year":"2012","venue":"SIAM Journal on Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ding et al., 2020] Xiangyun Ding, Wenjian Yu, Yuyang Xie, and Shenghua Liu. Efficient model-based collabo- rative filtering with fast adaptive PCA. In 2020 IEEE 32nd International Conference on Tools with Artificial Intelli- gence (ICTAI), pages 955-960. IEEE, 2020.","title":"Efficient model-based collaborative filtering with fast adaptive PCA","authors":"Xiangyun Ding, Wenjian Yu, Yuyang Xie, Shenghua Liu","year":"2020","venue":"IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Drineas and Mahoney, 2016] Petros Drineas and Michael W Mahoney. Randnla: Randomized nu- merical linear algebra. Communications of the ACM, 59(6):80-90, 2016.","title":"Randnla: Randomized numerical linear algebra","authors":"Petros Drineas, Michael W Mahoney","year":"2016","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Eckart and Young, 1936] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211-218, 1936.","title":"The approximation of one matrix by another of lower rank","authors":"Young ; Eckart, Carl Eckart, Gale Young","year":"1936","venue":"Psychometrika","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Feng et al., 2018a] Xu Feng, Yuyang Xie, Mingye Song, Wenjian Yu, and Jie Tang. Fast randomized PCA for sparse data. In Proc. the 10th Asian Conference on Machine Learning (ACML), pages 710-725, 14-16 Nov 2018.","title":"Fast randomized PCA for sparse data","authors":"Feng","year":"2018","venue":"Proc. the 10th Asian Conference on Machine Learning (ACML)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Feng et al., 2018b] Xu Feng, Wenjian Yu, and Yaohang Li. Faster matrix completion using randomized svd. In Proc. the 30th IEEE International Conference on Tools with Ar- tificial Intelligence (ICTAI), pages 608-615, 2018.","title":"Faster matrix completion using randomized svd","authors":"Feng","year":"2018","venue":"Proc. the 30th IEEE International Conference on Tools with Artificial Intelligence (ICTAI)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Feng et al., 2022] Xu Feng, Wenjian Yu, and Yuyang Xie. Pass-efficient randomized SVD with boosted accuracy. In Proc. the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD), 19-23 Sep. 2022.","title":"Pass-efficient randomized SVD with boosted accuracy","authors":"Wenjian Xu Feng, Yuyang Yu, Xie","year":"2022","venue":"Proc. the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Golub et al., 1981] Gene H Golub, Franklin T Luk, and Michael L Overton. A block lanczos method for comput- ing the singular values and corresponding singular vectors of a matrix. ACM Transactions on Mathematical Software (TOMS), 7(2):149-169, 1981.","title":"A block lanczos method for computing the singular values and corresponding singular vectors of a matrix","authors":"Golub","year":"1981","venue":"ACM Transactions on Mathematical Software (TOMS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Halko et al., 2011] N. Halko, P. G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompo- sitions. SIAM Review, 53(2):217-288, 2011.","title":"Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions","authors":"N Halko, P G Martinsson, J A Tropp","year":"2011","venue":"SIAM Review","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hallman, 2022] Eric Hallman. A block bidiagonalization method for fixed-accuracy low-rank matrix approxima- tion. SIAM Journal on Matrix Analysis and Applications, 43(2):661-680, 2022.","title":"A block bidiagonalization method for fixed-accuracy low-rank matrix approximation","authors":"Eric Hallman","year":"2022","venue":"SIAM Journal on Matrix Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Harper and Konstan, 2016] F. Maxwell Harper and Joseph A. Konstan. The Movielens datasets: His- tory and context. ACM Transactions on Interactive Intelligent Systems (TiiS), 5(4):19, 2016.","title":"The Movielens datasets: History and context","authors":"F, Maxwell Harper, Joseph A Konstan","year":"2016","venue":"ACM Transactions on Interactive Intelligent Systems (TiiS)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Leskovec and Krevl, 2014] Jure Leskovec and Andrej Krevl. SNAP datasets: Stanford large network dataset collection. http://snap.stanford.edu/data, June 2014.","title":"SNAP datasets: Stanford large network dataset collection","authors":"Jure Leskovec, Andrej Krevl","year":"2014","venue":null,"doi":null,"url":"http://snap.stanford.edu/data","internal_url":null},{"position":17,"raw":"[Li et al., 2017] H. Li, G. C. Linderman, A. Szlam, K. P. Stanton, Y. Kluger, and M. Tygert. Algorithm 971: An implementation of a randomized algorithm for principal component analysis. ACM Transactions on Mathematical Software, 43(3):1-14, 2017.","title":"Algorithm 971: An implementation of a randomized algorithm for principal component analysis","authors":"H Li, G C Linderman, A Szlam, K P Stanton, Y Kluger, M Tygert","year":"2017","venue":"ACM Transactions on Mathematical Software","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Martinsson and Tropp, 2020] Per-Gunnar Martinsson and Joel A. Tropp. Randomized numerical linear algebra: Foundations and algorithms. Acta Numerica, 29:403-572, 2020.","title":"Per-Gunnar Martinsson and Joel A. Tropp. Randomized numerical linear algebra: Foundations and algorithms","authors":"Tropp Martinsson","year":"2020","venue":"Acta Numerica","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Martinsson and Voronin, 2016] P. G. Martinsson and S. Voronin. A randomized blocked algorithm for ef- ficiently computing rank-revealing factorizations of matrices. SIAM J. Sci. Comput, 38:S485--S507, 2016.","title":"A randomized blocked algorithm for efficiently computing rank-revealing factorizations of matrices","authors":"P G Martinsson, S Voronin","year":"2016","venue":"SIAM J. Sci. Comput","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Musco and Musco, 2015] Cameron Musco and Christopher Musco. Randomized block Krylov methods for stronger and faster approximate singular value decomposition. In Advances in Neural Information Processing Systems, pages 1396-1404, 2015.","title":"Randomized block Krylov methods for stronger and faster approximate singular value decomposition","authors":"Musco Musco, Cameron Musco, Christopher Musco","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qiu et al., 2021] Jiezhong Qiu, Laxman Dhulipala, Jie Tang, Richard Peng, and Chi Wang. Lightne: A lightweight graph processing system for network embed- ding. In Proc. the 2021 ACM SIGMOD international con- ference on Management of data, pages 2281-2289, 2021.","title":"Lightne: A lightweight graph processing system for network embedding","authors":"Qiu","year":"2021","venue":"Proc. the 2021 ACM SIGMOD international conference on Management of data","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Voronin and Martinsson, 2015] Sergey Voronin and Per- Gunnar Martinsson. RSVDPACK: An implementation of randomized algorithms for computing the singular value, interpolative, and CUR decompositions of matri- ces on multi-core and GPU architectures. arXiv preprint arXiv:1502.05366, 2015.","title":"RSVDPACK: An implementation of randomized algorithms for computing the singular value, interpolative, and CUR decompositions of matrices on multi-core and GPU architectures","authors":"Martinsson ; Voronin, Per-Gunnar Martinsson","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Woolfe et al., 2008] Franco Woolfe, Edo Liberty, Vladimir Rokhlin, and Mark Tygert. A fast randomized algorithm for the approximation of matrices. Applied and Computa- tional Harmonic Analysis, 25(3):335-366, 2008.","title":"A fast randomized algorithm for the approximation of matrices","authors":"Edo Franco Woolfe, Vladimir Liberty, Mark Rokhlin, Tygert","year":"2008","venue":"Applied and Computational Harmonic Analysis","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2017] Wenjian Yu, Yu Gu, Jian Li, Shenghua Liu, and Yaohang Li. Single-pass PCA of large high- dimensional data. In Proc. International Joint Conference on Artificial Intelligence (IJCAI), pages 3350-3356, Aug. 2017.","title":"Single-pass PCA of large highdimensional data","authors":"Yu","year":"2017","venue":"Proc. International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2018] Wenjian Yu, Yu Gu, and Yaohang Li. Ef- ficient randomized algorithms for the fixed-precision low- rank matrix approximation. SIAM Journal on Matrix Anal- ysis and Applications, 39(3):1339-1359, 2018.","title":"Efficient randomized algorithms for the fixed-precision lowrank matrix approximation","authors":"Yu","year":"2018","venue":"SIAM Journal on Matrix Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2019a] Jiabao Zhang, Shenghua Liu, Wenjian Yu, Wenjie Feng, and Xueqi Cheng. Eigenpulse: detecting surges in large streaming graphs with row augmentation. In Proc. the 23rd Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 501-513, 2019.","title":"Eigenpulse: detecting surges in large streaming graphs with row augmentation","authors":"Zhang","year":"2019","venue":"Proc. the 23rd Pacific-Asia Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2019b] Jie Zhang, Yuxiao Dong, Yan Wang, Jie Tang, and Ming Ding. Prone: Fast and scalable net- work representation learning. In Proc. the 28th Inter- national Joint Conference on Artificial Intelligence, vol- ume 19, pages 4278-4284, 2019.","title":"Prone: Fast and scalable network representation learning","authors":"Zhang","year":"2019","venue":"Proc. the 28th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2019/594"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}