{"type":"ScholarlyArticle","title":"Efficient Adaptive Online Learning via Frequent Directions","authors":[{"name":"Yuanyu Wan","slug":"yuanyu-wan","given":"Yuanyu","family":"Wan","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Nan Wei","slug":"nan-wei","given":"Nan","family":"Wei","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Lijun Zhang","slug":"lijun-zhang","given":"Lijun","family":"Zhang","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"}],"abstract":"By employing time-varying proximal functions, adaptive subgradient methods (ADAGRAD) have improved the regret bound and been widely used in online learning and optimization. However, ADAGRAD with full matrix proximal functions (ADA-FULL) cannot deal with large-scale problems due to the impractical time and space complexities, though it has better performance when gradients are correlated. In this paper, we propose ADA-FD, an efficient variant of ADA-FULL based on a deterministic matrix sketching technique called frequent directions. Following ADA-FULL, we incorporate our ADA-FD into both primal-dual subgradient method and composite mirror descent method to develop two efficient methods. By maintaining and manipulating low-rank matrices, at each iteration, the space complexity is reduced from $O(d^2)$ to $O(\\tau d)$ and the time complexity is reduced from $O(d^3)$ to $O(\\tau^2d)$, where $d$ is the dimensionality of the data and $\\tau \\ll d$ is the sketching size. Theoretical analysis reveals that the regret of our methods is close to that of ADA-FULL as long as the outer product matrix of gradients is approximately low-rank. Experimental results show that our ADA-FD is comparable to ADA-FULL and outperforms other state-of-the-art algorithms in online convex optimization as well as in training convolutional neural networks (CNN).","doi":"10.24963/ijcai.2018/381","url":"https://ijcai.org/proceedings/2018/381","pdf":"https://ijcai.org/proceedings/2018/0381.pdf","markdown":"https://ijcai.org/proceedings/2018/381.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/381","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":381,"pages":{"first":2748,"last":2754},"keywords":["Machine Learning: Machine Learning","Machine Learning Applications: Big data ; Scalability","Machine Learning: Online Learning"],"videos":[],"references":[{"position":1,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih-Jen Lin. Libsvm : a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):1-27, 2011.","title":"Libsvm : a library for support vector machines","authors":"Chang, Lin ; ] Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Charikar et al., 2004] Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding frequent items in da- ta streams. Theoretical Computer Science, 312(1):3-15, 2004.","title":"Finding frequent items in data streams","authors":"Charikar","year":"2004","venue":"Theoretical Computer Science","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chechik et al., 2010] Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 11:1109-1135, 2010.","title":"Large scale online learning of image similarity through ranking","authors":"Chechik","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Drineas et al., 2006] Petros Drineas, Ravi Kannan, and Michael W. Mahoney. Fast monte carlo algorithms for matrices i: Approximating matrix multiplication. SIAM Journal on Computing, 36(1):132-157, 2006.","title":"Fast monte carlo algorithms for matrices i: Approximating matrix multiplication","authors":"Petros Drineas, Ravi Kannan, Michael W Mahoney","year":"2006","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duchi et al., 2010] John Duchi, Shai Shalev-Shwartz, Y- oram Singer, and Ambuj Tewari. Composite objective mir- ror descent. In Proceedings of the 23rd Annual Conference on Learning Theory, pages 14-26, 2010.","title":"Composite objective mirror descent","authors":"John Duchi, Shai Shalev-Shwartz, Yoram Singer, Ambuj Tewari","year":"2010","venue":"Proceedings of the 23rd Annual Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Duchi et al., 2011] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, 2011.","title":"Adaptive subgradient methods for online learning and stochastic optimization","authors":"John Duchi, Elad Hazan, Yoram Singer","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Freund and Schapire, 1999] Yoav Freund and Robert E. Schapire. Large margin classification using the perceptron algorithm. Machine Learning, 37(3):277-296, 1999.","title":"Large margin classification using the perceptron algorithm","authors":"Yoav Freund, Robert E Schapire","year":"1999","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ghashami and Phillips, 2014] Mina Ghashami and Jeff M. Phillips. Relative errors for deterministic low-rank matrix approximations. In Proceedings of the twenty-fifth annu- al ACM-SIAM symposium on Discrete algorithms, pages 707-717, 2014.","title":"Relative errors for deterministic low-rank matrix approximations","authors":"Phillips Ghashami, Mina Ghashami, Jeff M Phillips","year":"2014","venue":"Proceedings of the twenty-fifth annual ACM-SIAM symposium on Discrete algorithms","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ghashami et al., 2016] Mina Ghashami, Jeff M. Phillips E- do Liberty, and David P. Woodruff. Frequent directions: Simple and deterministic matrix sketching. SIAM Journal on Computing, 45(5):1762-1792, 2016.","title":"Frequent directions: Simple and deterministic matrix sketching","authors":"Mina Ghashami, Jeff M Phillips Edo, Liberty, David P Woodruff","year":"2016","venue":"SIAM Journal on Computing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hager, 1989] William W. Hager. Updating the inverse of a matrix. SIAM review, 31(2):221-239, 1989.","title":"Updating the inverse of a matrix","authors":"William W Hager","year":"1989","venue":"SIAM review","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hazan et al., 2007] Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex op- timization. Machine Learning, 69(2):169-192, 2007.","title":"Logarithmic regret algorithms for online convex optimization","authors":"Elad Hazan, Amit Agarwal, Satyen Kale","year":"2007","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jain et al., 2008] Prateek Jain, Brian Kulis, Inderjit S. D- hillon, and Kristen Grauman. Online metric learning and fast similarity search. In Advances in Neural Information Processing Systems 21, pages 761-768, 2008.","title":"Online metric learning and fast similarity search","authors":"Prateek Jain, Brian Kulis, Inderjit S Dhillon, Kristen Grauman","year":"2008","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kakade et al., 2008] Sham M. Kakade, Shai Shalev- Shwartz, and Ambuj Tewari. Efficient bandit algorithms for online multiclass prediction. In Proceedings of the 25th International Conference on Machine Learning, pages 440-447, 2008.","title":"Efficient bandit algorithms for online multiclass prediction","authors":"M Sham, Shai Kakade, Ambuj Shalev-Shwartz, Tewari","year":"2008","venue":"Proceedings of the 25th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kaski, 1998] Samuel Kaski. Dimensionality reduction by random mapping: Fast similarity computation for cluster- ing. In Proceedings of the 1998 IEEE International Joint Conference on Neural Networks, pages 413-418, 1998.","title":"Dimensionality reduction by random mapping: Fast similarity computation for clustering","authors":"Samuel Kaski, Kaski","year":"1998","venue":"Proceedings of the 1998 IEEE International Joint Conference on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. Technical report, Univer- sity of Toronto, 2009.","title":"Alex Krizhevsky. Learning multiple layers of features from tiny images","authors":"Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krummenacher et al., 2016] Gabriel Krummenacher, Brian McWilliams, Yannic Kilcher, Joachim M. Buhmann, and Nicolai Meinshausen. Scalable adaptive stochastic opti- mization using random projections. In Advances in Neu- ral Information Processing Systems 29, pages 1750-1758, 2016.","title":"Scalable adaptive stochastic optimization using random projections","authors":"Gabriel Krummenacher, Brian Mcwilliams, Yannic Kilcher, Joachim M Buhmann, Nicolai Meinshausen","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. In Proceedings of the IEEE, volume 86, pages 2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liberty, 2013] Edo Liberty. Simple and deterministic ma- trix sketching. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and da- ta mining, pages 581-588, 2013.","title":"Edo Liberty. Simple and deterministic matrix sketching","authors":"Liberty","year":"2013","venue":"Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Luo et al., 2016] Haipeng Luo, Alekh Agarwal, Nicolo Cesa-Bianchi, and John Langford. Efficient second order online learning by sketching. In Advances in Neural Infor- mation Processing Systems 29, pages 902-910, 2016.","title":"Efficient second order online learning by sketching","authors":"Haipeng Luo, Alekh Agarwal, Nicolo Cesa-Bianchi, John Langford","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mairal et al., 2010] Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online learning for ma- trix factorization and sparse coding. Journal of Machine Learning Research, 11:19-60, 2010.","title":"Online learning for matrix factorization and sparse coding","authors":"Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Misra and Gries, 1982] J. Misra and David Gries. Finding repeated elements. Science of Computer Programming, 2(2):143-152, 1982.","title":"Finding repeated elements","authors":"Gries Misra, ] J Misra, David Gries","year":"1982","venue":"Science of Computer Programming","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nalko et al., 2011] N. Nalko, P. G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompo- sitions. SIAM review, 53(2):217-288, 2011.","title":"Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions","authors":"N Nalko, P G Martinsson, J A Tropp","year":"2011","venue":"SIAM review","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Netzer et al., 2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learn- ing. In NIPS Workshop on Deep Learning and Unsuper- vised Feature Learning 2011, 2011.","title":"Reading digits in natural images with unsupervised feature learning","authors":"Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y Ng","year":"2011","venue":"NIPS Workshop on Deep Learning and Unsupervised Feature Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shalev-Shwartz, 2011] Shai Shalev-Shwartz. Online learn- ing and online convex optimization. Foundations and Trends in Machine Learning, 4(2):107-194, 2011.","title":"Online learning and online convex optimization","authors":"Shai Shalev-Shwartz","year":"2011","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wan and Zhang, 2018] Yuanyu Wan and Lijun Zhang. Ac- celerating adaptive online learning by matrix approxima- tion. In Proceedings of the 22nd Pacific-Asia Conference on Knowledge Discovery and Data Mining, 2018.","title":"Accelerating adaptive online learning by matrix approximation","authors":"Wan, Zhang, Yuanyu Wan, Lijun Zhang","year":"2018","venue":"Proceedings of the 22nd Pacific-Asia Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Woodruff, 2014] David P. Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends in Ma- chine Learning, 10(1-2):1-157, 2014.","title":"Sketching as a tool for numerical linear algebra","authors":"P David, Woodruff","year":"2014","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xiao, 2009] Lin Xiao. Dual averaging method for regu- larized stochastic learning and online optimization. In Advances in Neural Information Processing Systems 22, pages 2116-2124, 2009.","title":"Dual averaging method for regularized stochastic learning and online optimization","authors":"Lin Xiao","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yuan et al., 2015] Yuan Yuan, Jianwu Fang, and Qi Wang. Online anomaly detection in crowd scenes via structure analysis. IEEE Transactions on Cybernetics, 43(3):548- 561, 2015.","title":"Online anomaly detection in crowd scenes via structure analysis","authors":"Yuan Yuan, Jianwu Fang, Qi Wang","year":"2015","venue":"IEEE Transactions on Cybernetics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}