{"type":"ScholarlyArticle","title":"Learning Low-Rank Tensor Cores with Probabilistic ℓ0-Regularized Rank Selection for Model Compression","authors":[{"name":"Tianxiao Cao","slug":"tianxiao-cao","given":"Tianxiao","family":"Cao","affiliation":"ShanghaiTech University"},{"name":"Lu Sun","slug":"lu-sun","given":"Lu","family":"Sun","affiliation":"ShanghaiTech University"},{"name":"Canh Hao Nguyen","slug":"canh-hao-nguyen","given":"Canh Hao","family":"Nguyen","affiliation":"Kyoto University"},{"name":"Hiroshi Mamitsuka","slug":"hiroshi-mamitsuka","given":"Hiroshi","family":"Mamitsuka","affiliation":"Kyoto University"}],"abstract":"Compressing deep neural networks is of great importance for real-world applications on resource-constrained devices. Tensor decomposition is one promising answer that retains the functionality and most of the expressive power of the original deep models by replacing the weights with their decomposed cores. Decomposition with optimal ranks can achieve a good compression-accuracy trade-off, but it is expensive to optimize due to its discrete and combinatorial nature. A common practice is to set all ranks equal and tune one hyperparameter, but it may significantly harm the flexibility and generalization. In this paper, we propose a novel automatic rank selection method for deep model compression that allows learning model weights and decomposition ranks simultaneously. We propose to penalize the ℓ0 (quasi-)norm of the slices of decomposed tensor cores during model training. To avoid combinatorial optimization, we develop a probabilistic formulation and apply an approximate Bernoulli gate to each of the slices of tensor cores, which can be implemented in an end-to-end and scalable framework via gradient descent. It enables the automatic rank selection to be incorporated with arbitrary tensor decompositions and neural network layers such as linear layers, convolutional layers, and embedding layers. Comprehensive experiments on various tasks, including image classification, text sentiment classification, and neural machine translation, demonstrate the superior effectiveness of the proposed method over baselines.","doi":"10.24963/ijcai.2024/418","url":"https://ijcai.org/proceedings/2024/418","pdf":"https://ijcai.org/proceedings/2024/0418.pdf","markdown":"https://ijcai.org/proceedings/2024/418.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/418","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":418,"pages":{"first":3780,"last":3788},"keywords":["Machine Learning: ML: Matrix/tensor methods","Machine Learning: ML: Learning sparse models"],"videos":[],"references":[{"position":1,"raw":"[Arora et al., 2018] Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning, pages 244-253. PMLR, 2018.","title":"On the optimization of deep networks: Implicit acceleration by overparameterization","authors":"Sanjeev Arora, Nadav Cohen, Elad Hazan","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ba and Caruana, 2014] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? Advances in neural information processing systems, 27, 2014.","title":"Do deep nets really need to be deep?","authors":"Caruana Ba, Jimmy Ba, Rich Caruana","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cettolo et al., 2014] Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign. In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, pages 2-17, 2014.","title":"Report on the 11th iwslt evaluation campaign","authors":"Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, Marcello Federico","year":"2014","venue":"Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2020] Zhiyu Cheng, Baopu Li, Yanwen Fan, and Yingze Bao. A novel rank selection scheme in ten- sor ring decomposition based on reinforcement learning for deep neural networks. In ICASSP 2020-2020 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3292-3296. IEEE, 2020.","title":"A novel rank selection scheme in tensor ring decomposition based on reinforcement learning for deep neural networks","authors":"Zhiyu Cheng, Baopu Li, Yanwen Fan, Yingze Bao","year":"2020","venue":"ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dai et al., 2023] Wei Dai, Jicong Fan, Yiming Miao, and Kai Hwang. Deep learning model compression with rank reduction in tensor decomposition. IEEE Transactions on Neural Networks and Learning Systems, 2023.","title":"Deep learning model compression with rank reduction in tensor decomposition","authors":"Wei Dai, Jicong Fan, Yiming Miao, Kai Hwang","year":"2023","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dolgov et al., 2021] Sergey Dolgov, Dante Kalise, and Karl K Kunisch. Tensor decomposition methods for high- dimensional hamilton-jacobi-bellman equations. SIAM Journal on Scientific Computing, 43(3):A1625-A1650, 2021.","title":"Tensor decomposition methods for highdimensional hamilton-jacobi-bellman equations","authors":"Sergey Dolgov, Dante Kalise, Karl K Kunisch","year":"2021","venue":"SIAM Journal on Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2020] Ze-Feng Gao, Song Cheng, Rong-Qiang He, Zhi-Yuan Xie, Hui-Hai Zhao, Zhong-Yi Lu, and Tao Xiang. Compressing deep neural networks by matrix prod- uct operators. Physical Review Research, 2(2):023300, 2020.","title":"Compressing deep neural networks by matrix product operators","authors":"Ze-Feng Gao, Song Cheng, Rong-Qiang He, Zhi-Yuan Xie, Hui-Hai Zhao, Zhong-Yi Lu, Tao Xiang","year":"2020","venue":"Physical Review Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Garipov et al., 2016] Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov. Ultimate ten- sorization: compressing convolutional and fc layers alike. arXiv preprint arXiv:1611.03214, 2016.","title":"Ultimate tensorization: compressing convolutional and fc layers alike","authors":"Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, Dmitry Vetrov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu et al., 2022] Jiaqi Gu, Ben Keller, Jean Kossaifi, An- ima Anandkumar, Brucek Khailany, and David Z Pan. Heat: Hardware-efficient automatic tensor decompo- sition for transformer compression. arXiv preprint arXiv:2211.16749, 2022.","title":"Heat: Hardware-efficient automatic tensor decomposition for transformer compression","authors":"Jiaqi Gu, Ben Keller, Jean Kossaifi, Anima Anandkumar, Brucek Khailany, David Z Pan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gusak et al., 2019] Julia Gusak, Maksym Kholi- avchenko, Evgeny Ponomarev, Larisa Markeeva, Philip Blagoveschensky, Andrzej Cichocki, and Ivan Oseledets. Automated multi-stage compression of neural networks. In 2019 IEEE/CVF International Conference on Com- puter Vision Workshop (ICCVW), pages 2501-2508. IEEE Computer Society, 2019.","title":"Automated multi-stage compression of neural networks","authors":"Julia Gusak, Maksym Kholiavchenko, Evgeny Ponomarev, Larisa Markeeva, Philip Blagoveschensky, Andrzej Cichocki, Ivan Oseledets","year":"2019","venue":"IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han et al., 2015] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman cod- ing. arXiv preprint arXiv:1510.00149, 2015.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hashemizadeh et al., 2020] Meraj Hashemizadeh, Michelle Liu, Jacob Miller, and Guillaume Rabusseau. Adaptive tensor learning with tensor networks. 2020.","title":"Adaptive tensor learning with tensor networks","authors":"Meraj Hashemizadeh, Michelle Liu, Jacob Miller, Guillaume Rabusseau","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hawkins and Zhang, 2021] Cole Hawkins and Zheng Zhang. Bayesian tensorized neural networks with au- tomatic rank selection. Neurocomputing, 453:172-180, 2021.","title":"Bayesian tensorized neural networks with automatic rank selection","authors":"Zhang Hawkins, Cole Hawkins, Zheng Zhang","year":"2021","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hayashi et al., 2019] Kohei Hayashi, Taiki Yamaguchi, Yohei Sugawara, and Shin-ichi Maeda. Exploring unex- plored tensor network decompositions for convolutional neural networks. Advances in Neural Information Process- ing Systems, 32, 2019.","title":"Exploring unexplored tensor network decompositions for convolutional neural networks","authors":"Kohei Hayashi, Taiki Yamaguchi, Yohei Sugawara, Shin-Ichi Maeda","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hillar and Lim, 2013] Christopher J Hillar and Lek-Heng Lim. Most tensor problems are np-hard. Journal of the ACM (JACM), 60(6):1-39, 2013.","title":"Most tensor problems are np-hard","authors":"J Christopher, Lek-Heng Hillar, Lim","year":"2013","venue":"Journal of the ACM (JACM)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hrinchuk et al., 2020] Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova, and Ivan Oseledets. Tensorized embedding layers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4847-4860, 2020.","title":"Tensorized embedding layers","authors":"Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova, Ivan Oseledets","year":"2020","venue":"Findings of the Association for Computational Linguistics: EMNLP 2020","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kodryan et al., 2023] Maxim Kodryan, Dmitry Kropotov, and Dmitry Vetrov. Mars: Masked automatic ranks selec- tion in tensor decompositions. In International Conference on Artificial Intelligence and Statistics, pages 3718-3732.","title":"Maxim Kodryan, Dmitry Kropotov, and Dmitry Vetrov. Mars: Masked automatic ranks selection in tensor decompositions","authors":"Kodryan","year":"2023","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"PMLR, 2023.","title":null,"authors":"Pmlr","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kwon et al., 2023] Taehyung Kwon, Jihoon Ko, Jinhong Jung, and Kijung Shin. Tensorcodec: Compact lossy com- pression of tensors without strong data assumptions. arXiv preprint arXiv:2309.10310, 2023.","title":"Tensorcodec: Compact lossy compression of tensors without strong data assumptions","authors":"Kwon","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lebedev et al., 2014] Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine- tuned cp-decomposition. arXiv preprint arXiv:1412.6553, 2014.","title":"Speeding-up convolutional neural networks using finetuned cp-decomposition","authors":"Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, Victor Lempitsky","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Li et al., 2021] Nannan Li, Yu Pan, Yaran Chen, Zixiang Ding, Dongbin Zhao, and Zenglin Xu. Heuristic rank se- lection with progressively searching tensor ring network. Complex & Intelligent Systems, pages 1-15, 2021.","title":"Heuristic rank selection with progressively searching tensor ring network","authors":"Nannan Li, Yu Pan, Yaran Chen, Zixiang Ding, Dongbin Zhao, Zenglin Xu","year":"2021","venue":"Complex & Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Li et al., 2022] Sunzhu Li, Peng Zhang, Guobing Gan, Xi- uqing Lv, Benyou Wang, Junqiu Wei, and Xin Jiang. Hy- poformer: Hybrid decomposition transformer for edge- friendly neural machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pages 7056-7068, 2022.","title":"Hypoformer: Hybrid decomposition transformer for edgefriendly neural machine translation","authors":"Sunzhu Li, Peng Zhang, Guobing Gan, Xiuqing Lv, Benyou Wang, Junqiu Wei, Xin Jiang","year":"2022","venue":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Louizos et al., 2018] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through l 0 regularization. In International Conference on Learning Representations, 2018.","title":"Learning sparse neural networks through l 0 regularization","authors":"Christos Louizos, Max Welling, Diederik P Kingma","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Ma et al., 2019] Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. A tensorized transformer for language modeling. Advances in neural information processing systems, 32, 2019.","title":"A tensorized transformer for language modeling","authors":"Ma","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Maddison et al., 2016] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations, 2016.","title":"The concrete distribution: A continuous relaxation of discrete random variables","authors":"Andriy Chris J Maddison, Yee Whye Mnih, Teh","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Nakajima et al., 2010] Shinichi Nakajima, Masashi Sugiyama, and Ryota Tomioka. Global analytic solution for variational bayesian matrix factorization. Advances in Neural Information Processing Systems, 23, 2010.","title":"Global analytic solution for variational bayesian matrix factorization","authors":"Shinichi Nakajima, Masashi Sugiyama, Ryota Tomioka","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Novikov et al., 2015] Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in neural information processing systems, 28, 2015.","title":"Tensorizing neural networks","authors":"Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, Dmitry P Vetrov","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Orús, 2019] Román Orús. Tensor networks for complex quantum systems. Nature Reviews Physics, 1(9):538-550, 2019.","title":"Tensor networks for complex quantum systems","authors":"Román Orús","year":"2019","venue":"Nature Reviews Physics","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Oseledets, 2011] Ivan V Oseledets. Tensor-train de- composition. SIAM Journal on Scientific Computing, 33(5):2295-2317, 2011.","title":"Tensor-train decomposition","authors":"V Ivan, Oseledets","year":"2011","venue":"SIAM Journal on Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Pan et al., 2022] Yu Pan, Zeyong Su, Ao Liu, Wang Jingquan, Nannan Li, and Zenglin Xu. A unified weight initialization paradigm for tensorial convolutional neural networks. In International Conference on Machine Learn- ing, pages 17238-17257. PMLR, 2022.","title":"A unified weight initialization paradigm for tensorial convolutional neural networks","authors":"Yu Pan, Zeyong Su, Ao Liu, Wang Jingquan, Nannan Li, Zenglin Xu","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Post, 2018] Matt Post. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Ma- chine Translation: Research Papers, page 186. Associa- tion for Computational Linguistics, 2018.","title":"A call for clarity in reporting bleu scores","authors":"Matt Post","year":"2018","venue":"Proceedings of the Third Conference on Machine Translation: Research Papers","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Rouard et al., 2023] Simon Rouard, Francisco Massa, and Alexandre Défossez. Hybrid transformers for music source separation. In ICASSP 2023-2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), pages 1-5. IEEE, 2023.","title":"Hybrid transformers for music source separation","authors":"Simon Rouard, Francisco Massa, Alexandre Défossez","year":"2023","venue":"ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Tibshirani, 1996] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-288, 1996.","title":"Regression shrinkage and selection via the lasso","authors":"Robert Tibshirani","year":"1996","venue":"Journal of the Royal Statistical Society Series B: Statistical Methodology","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Tomioka et al., 2010] Ryota Tomioka, Kohei Hayashi, and Hisashi Kashima. Estimation of low-rank tensors via con- vex optimization. arXiv preprint arXiv:1010.0789, 2010.","title":"Estimation of low-rank tensors via convex optimization","authors":"Tomioka","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wang et al., 2018] Wenqi Wang, Yifan Sun, Brian Eriksson, Wenlin Wang, and Vaneet Aggarwal. Wide compression: Tensor ring nets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9329- 9338, 2018.","title":"Wide compression: Tensor ring nets","authors":"Wenqi Wang, Yifan Sun, Brian Eriksson, Wenlin Wang, Vaneet Aggarwal","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Wang et al., 2021] Benyou Wang, Yuxin Ren, Lifeng Shang, Xin Jiang, and Qun Liu. Exploring extreme pa- rameter compression for pre-trained language models. In International Conference on Learning Representations, 2021.","title":"Exploring extreme parameter compression for pre-trained language models","authors":"Wang","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wen et al., 2016] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29, 2016.","title":"Learning structured sparsity in deep neural networks","authors":"Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, Hai Li","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Yaguchi et al., 2019] Atsushi Yaguchi, Taiji Suzuki, Shuhei Nitta, Yukinobu Sakata, and Akiyuki Tanizawa. Decomposable-net: Scalable low-rank compression for neural networks. arXiv preprint arXiv:1910.13141, 2019.","title":"Decomposable-net: Scalable low-rank compression for neural networks","authors":"Atsushi Yaguchi, Taiji Suzuki, Shuhei Nitta, Yukinobu Sakata, Akiyuki Tanizawa","year":"2019","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2021/447"},{"position":45,"raw":"[Yamada et al., 2020] Yutaro Yamada, Ofir Lindenbaum, Sa- hand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In International Conference on Machine Learning, pages 10648-10659. PMLR, 2020.","title":"Feature selection using stochastic gates","authors":"Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, Yuval Kluger","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Yang et al., 2017] Yinchong Yang, Denis Krompass, and Volker Tresp. Tensor-train recurrent neural networks for video classification. In International Conference on Ma- chine Learning, pages 3891-3900. PMLR, 2017.","title":"Tensor-train recurrent neural networks for video classification","authors":"Yang","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Yin et al., 2022] Miao Yin, Yang Sui, Wanzhao Yang, Xiao Zang, Yu Gong, and Bo Yuan. Hodec: Towards ef- ficient high-order decomposed convolutional neural net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12299- 12308, 2022.","title":"Hodec: Towards efficient high-order decomposed convolutional neural networks","authors":"Yang Miao Yin, Wanzhao Sui, Xiao Yang, Yu Zang, Bo Gong, Yuan","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zhao et al., 2016] Qibin Zhao, Guoxu Zhou, Shengli Xie, Liqing Zhang, and Andrzej Cichocki. Tensor ring decom- position. arXiv preprint arXiv:1606.05535, 2016.","title":"Tensor ring decomposition","authors":"Zhao","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zhen et al., 2022] Peining Zhen, Ziyang Gao, Tianshu Hou, Yuan Cheng, and Hai-Bao Chen. Deeply tensor com- pressed transformers for end-to-end object detection. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pages 4716-4724, 2022.","title":"Deeply tensor compressed transformers for end-to-end object detection","authors":"Zhen","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Ziyin and Wang, 2023] Liu Ziyin and Zihao Wang. spred: Solving l1 penalty with sgd. In International Conference on Machine Learning, pages 43407-43422. PMLR, 2023.","title":"spred: Solving l1 penalty with sgd","authors":"Wang ; Ziyin, Liu Ziyin, Zihao Wang","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}