{"type":"ScholarlyArticle","title":"Convexity Certificates for Symbolic Tensor Expressions","authors":[{"name":"Paul G. Rump","slug":"paul-g-rump","given":"Paul G.","family":"Rump","affiliation":"University Jena"},{"name":"Niklas Merk","slug":"niklas-merk","given":"Niklas","family":"Merk","affiliation":"University Jena"},{"name":"Julien Klaus","slug":"julien-klaus","given":"Julien","family":"Klaus","affiliation":"University Jena"},{"name":"Maurice Wenig","slug":"maurice-wenig","given":"Maurice","family":"Wenig","affiliation":"University Jena"},{"name":"Joachim Giesen","slug":"joachim-giesen","given":"Joachim","family":"Giesen","affiliation":"University Jena"}],"abstract":"Knowing that a function is convex ensures that any local minimum is also a global minimum. Here, we implement an approach to certify the convexity of twice-differentiable functions by certifying that their second-order derivative is positive semidefinite. Both the computation of the second-order derivative and the certification of positive semidefiniteness are done symbolically. Previous implementations of this approach assume that the function to be minimized takes scalar or vector inputs, meaning that the second-order derivative is at most a matrix. However, the input of many machine learning problems is naturally given in the form of matrices or higher order tensors, in which case the second-order derivative becomes a tensor of at least fourth order. The familiar linear algebra notations and known rules for determining whether a matrix is positive semidefinite are not sufficient to deal with these higher order expressions. Here, we present a formal language for tensor expressions that allows us to generalize semidefiniteness to higher-order tensors and thereby certify the convexity of a broader set of functions.","doi":"10.24963/ijcai.2024/216","url":"https://ijcai.org/proceedings/2024/216","pdf":"https://ijcai.org/proceedings/2024/0216.pdf","markdown":"https://ijcai.org/proceedings/2024/216.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/216","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Constraint Satisfaction and Optimization","track":"Main Track","number":216,"pages":{"first":1953,"last":1960},"keywords":["Machine Learning: ML: Symbolic methods","Constraint Satisfaction and Optimization: CSO: Solvers and tools","Machine Learning: ML: Optimization"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: A Sys- tem for Large-Scale Machine Learning. In Kimberly Kee- ton and Timothy Roscoe, editors, USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 265-283. USENIX Association, 2016.","title":"TensorFlow: A System for Large-Scale Machine Learning","authors":"Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, Xiaoqiang Zheng","year":"2016","venue":"USENIX Symposium on Operating Systems Design and Implementation (OSDI)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ahmadi and Parrilo, 2013] Amir Ali Ahmadi and Pablo A. Parrilo. A complete characterization of the gap between convexity and sos-convexity. SIAM Journal on Optimiza- tion, 23(2):811-833, 2013.","title":"A complete characterization of the gap between convexity and sos-convexity","authors":"Ali Amir, Pablo A Ahmadi, Parrilo","year":"2013","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Aho et al., 1986] Alfred V. Aho, Ravi Sethi, and Jeffrey D. Ullman. Compilers: Principles, Techniques, and Tools. Addison-Wesley, 1986.","title":"Compilers: Principles, Techniques, and Tools","authors":"Alfred V Aho, Ravi Sethi, Jeffrey D Ullman","year":"1986","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bach et al., 2023] Francis R. Bach, Elisabetta Cornacchia, Luca Pesce, and Giovanni Piccioli. Theory and ap- plications of the Sum-Of-Squares technique. CoRR, abs/2306.16255, 2023.","title":"Theory and applications of the Sum-Of-Squares technique","authors":"R Francis, Elisabetta Bach, Luca Cornacchia, Giovanni Pesce, Piccioli","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blanchard et al., 2020] Pierre Blanchard, Desmond J. Higham, and Nicholas J. Higham. Accurate Computation of the Log-Sum-Exp and Softmax Functions. IMA Journal of Numerical Analysis, 41(4):2311-2330, 08 2020.","title":"Accurate Computation of the Log-Sum-Exp and Softmax Functions","authors":"Pierre Blanchard, Desmond J Higham, Nicholas J Higham","year":"2020","venue":"IMA Journal of Numerical Analysis","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cocke, 1970] John Cocke. Global common subexpression elimination. In Robert S. Northcote, editor, Proceedings of a Symposium on Compiler Optimization, pages 20-24. ACM, 1970.","title":"Global common subexpression elimination","authors":"John Cocke","year":"1970","venue":"Proceedings of a Symposium on Compiler Optimization","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Diamond and Boyd, 2016] Steven Diamond and Stephen P. Boyd. CVXPY: A Python-Embedded Modeling Language for Convex Optimization. Journal of Machine Learning Research, 17:83:1-83:5, 2016.","title":"CVXPY: A Python-Embedded Modeling Language for Convex Optimization","authors":"Steven Diamond, Stephen P Boyd","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dressler et al., 2019] Mareike Dressler, Sadik Iliman, and Timo de Wolff. An approach to constrained polynomial optimization via nonnegative circuit polynomials and ge- ometric programming. Journal of Symbolic Computation, 91:149-172, 2019.","title":"An approach to constrained polynomial optimization via nonnegative circuit polynomials and geometric programming","authors":"Mareike Dressler, Sadik Iliman, Timo De Wolff","year":"2019","venue":"Journal of Symbolic Computation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dressler et al., 2023] Mareike Dressler, Salma Kuhlmann, and Moritz Schick. Geometrical study of the cone of sums of squares plus sums of nonnegative circuits. CoRR, 2305.14848, 2023.","title":"Geometrical study of the cone of sums of squares plus sums of nonnegative circuits","authors":"Mareike Dressler, Salma Kuhlmann, Moritz Schick","year":"2023","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grant and Boyd, 2008] Michael C. Grant and Stephen P. Boyd. Graph implementations for nonsmooth convex pro- grams. In Vincent D. Blondel, Stephen P. Boyd, and Hi- denori Kimura, editors, Recent Advances in Learning and Control, volume 371 of Lecture Notes in Control and In- formation Sciences, pages 95-110. Springer, 2008.","title":"Graph implementations for nonsmooth convex programs","authors":"C Michael, Stephen P Grant, Boyd","year":"2008","venue":"Recent Advances in Learning and Control","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Harris et al., 2020] Charles R. Harris, K. Jarrod Millman, Stéfan van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585:357-362, 2020.","title":"Array programming with NumPy","authors":"Charles R Harris, K Jarrod Millman, Stéfan Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H Van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández Del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, Travis E Oliphant","year":"2020","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Helton and Nie, 2010] J. William Helton and Jiawang Nie. Semidefinite representation of convex sets. Mathematical Programming, 122(1):21-64, 2010.","title":"Semidefinite representation of convex sets","authors":"Nie Helton, ] J William Helton, Jiawang Nie","year":"2010","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hickey et al., 2001] Timothy J. Hickey, Qun Ju, and Maarten H. van Emden. Interval arithmetic: From princi- ples to implementation. Journal of the ACM, 48(5):1038- 1068, 2001.","title":"Interval arithmetic: From principles to implementation","authors":"Timothy J Hickey, Qun Ju, Maarten H Van Emden","year":"2001","venue":"Journal of the ACM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Klaus et al., 2022] Julien Klaus, Niklas Merk, Konstantin Wiedom, Sören Laue, and Joachim Giesen. Convexity Certificates from Hessians. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2022.","title":"Convexity Certificates from Hessians","authors":"Julien Klaus, Niklas Merk, Konstantin Wiedom, Sören Laue, Joachim Giesen","year":"2022","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Klaus et al., 2023] Julien Klaus, Mark Blacher, and Joachim Giesen. Compiling Tensor Expressions into Einsum. In International Conference on Computational Science (ICCS), volume 14074 of Lecture Notes in Computer Science, pages 129-136. Springer, 2023.","title":"Compiling Tensor Expressions into Einsum","authors":"Julien Klaus, Mark Blacher, Joachim Giesen","year":"2023","venue":"International Conference on Computational Science (ICCS)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Laue et al., 2019] Sören Laue, Matthias Mitterreiter, and Joachim Giesen. GENO -GENeric Optimization for Clas- sical Machine Learning. In Advances in Neural Infor- mation Processing Systems (NeurIPS), pages 2187-2198, 2019.","title":"GENO -GENeric Optimization for Classical Machine Learning","authors":"Sören Laue, Matthias Mitterreiter, Joachim Giesen","year":"2019","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Laue et al., 2020] Sören Laue, Matthias Mitterreiter, and Joachim Giesen. A Simple and Efficient Tensor Calcu- lus. In AAAI Conference on Artificial Intelligence (AAAI), pages 4527-4534, 2020.","title":"A Simple and Efficient Tensor Calculus","authors":"Sören Laue, Matthias Mitterreiter, Joachim Giesen","year":"2020","venue":"AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Parrilo, 2000] Pablo Parrilo. Structured semidefinite pro- grams and semialgebraic geometry methods in robustness and optimization. PhD thesis, California Institute of Tech- nology, 2000.","title":"Pablo Parrilo. Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization","authors":"Parrilo","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, edi- tors, Advances in Neural Information Processing Systems (NeurIPS), pages 8024-8035, 2019.","title":"PyTorch: An Imperative Style, High-Performance Deep Learning Library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z Yang, Zachary Devito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, Soumith Chintala","year":"2019","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}