{"type":"ScholarlyArticle","title":"Automatically Gating Multi-Frequency Patterns through Rectified Continuous Bernoulli Units with Theoretical Principles","authors":[{"name":"Zheng-Fan Wu","slug":"zheng-fan-wu","given":"Zheng-Fan","family":"Wu","affiliation":"School of Computer Science and Engineering , Southeast University , Nanjing , 210096 , China; MOE Key Laboratory of Computer Network and Information Integration ( Southeast University) , China"},{"name":"Yi-Nan Feng","slug":"yi-nan-feng","given":"Yi-Nan","family":"Feng","affiliation":"School of Computer Science and Engineering , Southeast University , Nanjing , 210096 , China; MOE Key Laboratory of Computer Network and Information Integration ( Southeast University) , China"},{"name":"Hui Xue","slug":"hui-xue","given":"Hui","family":"Xue","affiliation":"School of Computer Science and Engineering , Southeast University , Nanjing , 210096 , China; MOE Key Laboratory of Computer Network and Information Integration ( Southeast University) , China"}],"abstract":"Different nonlinearities are only suitable for responding to different frequency signals. The locally-responding ReLU is incapable of modeling high-frequency features due to the spectral bias, whereas the globally-responding sinusoidal function is intractable to represent low-frequency concepts cheaply owing to the optimization dilemma. Moreover, nearly all the practical tasks are composed of complex multi-frequency patterns, whereas there is little prospect of designing or searching a heterogeneous network containing various types of neurons matching the frequencies, because of their exponentially-increasing combinatorial states. In this paper, our contributions are three-fold: 1) we propose a general Rectified Continuous Bernoulli (ReCB) unit paired with an efficient variational Bayesian learning paradigm, to automatically detect/gate/represent different frequency responses; 2) our numerically-tight theoretical framework proves that ReCB-based networks can achieve the optimal representation ability, which is O(m^{η/(d^2)}) times better than that of popular neural networks, for a hidden dimension of m, an input dimension of d, and a Lipschitz constant of η; 3) we provide comprehensive empirical evidence showing that ReCB-based networks can keenly learn multi-frequency patterns and push the state-of-the-art performance.","doi":"10.24963/ijcai.2022/499","url":"https://ijcai.org/proceedings/2022/499","pdf":"https://ijcai.org/proceedings/2022/0499.pdf","markdown":"https://ijcai.org/proceedings/2022/499.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/499","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":499,"pages":{"first":3594,"last":3600},"keywords":["Machine Learning: Classification","Machine Learning: Optimisation","Machine Learning: Kernel Methods","Machine Learning: Automated Machine Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986731","length_sec":88,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984874","length_sec":300,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bond-Taylor and Willcocks, 2020] Sam Bond-Taylor and Chris G Willcocks. Gradient origin networks. arXiv preprint arXiv:2007.02798, 2020.","title":"Gradient origin networks","authors":"Willcocks Bond-Taylor, Sam Bond-Taylor, Chris G Willcocks","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Feinerman and Newman, 1975] Robert P. Feinerman and Donald J. Newman. Polynomial approximation. Bull. Amer. Math. Soc., 81:28-30, 1975.","title":"Polynomial approximation","authors":"Robert P Feinerman, Donald J Newman","year":"1975","venue":"Bull. Amer. Math. Soc","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT press, 2016.","title":"Deep Learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jacot et al., 2018] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. arXiv preprint arXiv:1806.07572, 2018.","title":"Neural tangent kernel: Convergence and generalization in neural networks","authors":"Arthur Jacot, Franck Gabriel, Clément Hongler","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kazemi et al., 2019] Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Mar- cus Brubaker. Time2vec: Learning a vector representation of time. arXiv preprint arXiv:1907.05321, 2019.","title":"Time2vec: Learning a vector representation of time","authors":"Rishab Seyed Mehran Kazemi, Sepehr Goel, Janahan Eghbali, Jaspreet Ramanan, Sanjay Sahota, Stella Thakur, Cathal Wu, Pascal Smyth, Marcus Poupart, Brubaker","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Loshchilov and Hutter, 2016] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.","title":"Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts","authors":"Hutter Loshchilov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mildenhall et al., 2020] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv preprint arXiv:2003.08934, 2020.","title":"Nerf: Representing scenes as neural radiance fields for view synthesis","authors":"Ben Mildenhall, P Pratul, Matthew Srinivasan, Jonathan T Tancik, Ravi Barron, Ren Ramamoorthi, Ng","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Parascandolo et al., 2017] Giambattista Parascandolo, Heikki Huttunen, and Tuomas Virtanen. Taming the waves: Sine as activation function in deep neural networks. ICLR 2017 conference submission, 2017.","title":"Taming the waves: Sine as activation function in deep neural networks","authors":"Giambattista Parascandolo, Heikki Huttunen, Tuomas Virtanen","year":"2017","venue":"ICLR 2017 conference submission","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. NIPS 2017 Autodiff Workshop, 2017.","title":"Automatic differentiation in pytorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Devito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":"NIPS 2017 Autodiff Workshop","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Poggio et al., 2017] Tomaso Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco, Brando Miranda, and Qianli Liao. Why and when can deep-but not shallow-networks avoid the curse of dimensionality: a review. International Journal of Automation and Computing, 14(5):503-519, 2017.","title":"Why and when can deep-but not shallow-networks avoid the curse of dimensionality: a review","authors":"Tomaso Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco, Brando Miranda, Qianli Liao","year":"2017","venue":"International Journal of Automation and Computing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ramachandran et al., 2017] Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.","title":"Searching for activation functions","authors":"Ramachandran","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Sitzmann et al., 2020a] Vincent Sitzmann, Eric R Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. arXiv preprint arXiv:2006.09662, 2020.","title":"Metasdf: Meta-learning signed distance functions","authors":"Sitzmann","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sitzmann et al., 2020b] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.","title":"Implicit neural representations with periodic activation functions","authors":"Sitzmann","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutskever et al., 2013] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In Interna- tional conference on machine learning, pages 1139-1147, 2013.","title":"On the importance of initialization and momentum in deep learning","authors":"Sutskever","year":"2013","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tancik et al., 2020] Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006.10739, 2020.","title":"Fourier features let networks learn high frequency functions in low dimensional domains","authors":"Matthew Tancik, P Pratul, Ben Srinivasan, Sara Mildenhall, Nithin Fridovich-Keil, Utkarsh Raghavan, Ravi Singhal, Jonathan T Ramamoorthi, Ren Barron, Ng","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xu et al., 2019] Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Self-attention with functional time representation learning. In Advances in Neu- ral Information Processing Systems, pages 15915-15925, 2019.","title":"Self-attention with functional time representation learning","authors":"Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, Kannan Achan","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xue and Wu, 2020] Hui Xue and Zheng-Fan Wu. Baker- nets: Bayesian random kernel mapping networks. In Pro- ceedings of the Twenty-Ninth International Joint Confer- ence on Artificial Intelligence, IJCAI-20, pages 3073-3079. International Joint Conferences on Artificial Intelligence Organization, 7 2020.","title":"Bakernets: Bayesian random kernel mapping networks","authors":"Hui Xue, Zheng-Fan Wu","year":"2020","venue":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xue et al., 2019] Hui Xue, Zheng-Fan Wu, and Wei-Xiang Sun. Deep spectral kernel learning. In Proceedings of the Twenty-Eighth International Joint Conference on Artifi- cial Intelligence, IJCAI-19, pages 4019-4025. International Joint Conferences on Artificial Intelligence Organization, 7 2019.","title":"Deep spectral kernel learning","authors":"Hui Xue, Zheng-Fan Wu, Wei-Xiang Sun","year":"2019","venue":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19","doi":null,"url":null,"internal_url":"/proceedings/2019/558"},{"position":22,"raw":"[Zhong et al., 2019] Ellen D Zhong, Tristan Bepler, Joseph H Davis, and Bonnie Berger. Reconstructing continuous dis- tributions of 3d protein structure from cryo-em images. In International Conference on Learning Representations, 2019.","title":"Reconstructing continuous distributions of 3d protein structure from cryo-em images","authors":"Ellen D Zhong, Tristan Bepler, Joseph H Davis, Bonnie Berger","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}