{"type":"ScholarlyArticle","title":"Backpropagation of Unrolled Solvers with Folded Optimization","authors":[{"name":"James Kotary","slug":"james-kotary","given":"James","family":"Kotary","affiliation":"University of Virginia"},{"name":"My H Dinh","slug":"my-h-dinh","given":"My H","family":"Dinh","affiliation":"University of Virginia"},{"name":"Ferdinando Fioretto","slug":"ferdinando-fioretto","given":"Ferdinando","family":"Fioretto","affiliation":"University of Virginia"}],"abstract":"The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks. \r\nA central challenge in this setting is backpropagation through the solution of an optimization problem, which typically lacks a closed form. One typical strategy is algorithm unrolling, which relies on automatic differentiation through the operations of an iterative solver. While flexible and general, unrolling can encounter accuracy and efficiency issues in practice. These issues can be avoided by analytical differentiation of the optimization, but current frameworks impose rigid requirements on the optimization problem's form. This paper provides theoretical insights into the backward pass of unrolled optimization, leading to a system for generating  efficiently solvable analytical models of backpropagation. Additionally, it proposes a unifying view of unrolling and analytical differentiation through optimization mappings. Experiments over various model-based learning tasks demonstrate the advantages of the approach both computationally and in terms of enhanced expressiveness.","doi":"10.24963/ijcai.2023/218","url":"https://ijcai.org/proceedings/2023/218","pdf":"https://ijcai.org/proceedings/2023/0218.pdf","markdown":"https://ijcai.org/proceedings/2023/218.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/218","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Constraint Satisfaction and Optimization","track":"Main Track","number":218,"pages":{"first":1963,"last":1970},"keywords":["Constraint Satisfaction and Optimization: CSO: Constraint optimization","Machine Learning: ML: Optimization","Machine Learning: ML: Applications"],"videos":[],"references":[{"position":1,"raw":"[Adams and Zemel, 2011] Ryan Prescott Adams and Richard S Zemel. Ranking via sinkhorn propagation. arXiv preprint arXiv:1106.1925, 2011.","title":"Ranking via sinkhorn propagation","authors":"Adams, Zemel, Ryan Prescott, Adams, Richard S Zemel","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agrawal et al., 2019] Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter. Differentiable convex optimization layers. Ad- vances in neural information processing systems, 32, 2019.","title":"Differentiable convex optimization layers","authors":"Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, J Zico Kolter","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Amos and Kolter, 2017] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learn- ing, pages 136-145. PMLR, 2017.","title":"Optnet: Differentiable optimization as a layer in neural networks","authors":"Amos, Kolter, Brandon Amos, J Zico, Kolter","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Attouch et al., 2013] Hedy Attouch, Jérôme Bolte, and Be- nar Fux Svaiter. Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized gauss-seidel methods. Mathematical Programming, 137(1):91-129, 2013.","title":"Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized gauss-seidel methods","authors":"Hedy Attouch, Jérôme Bolte, Benar Fux Svaiter","year":"2013","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Audet et al., 2004] Charles Audet, Jack Brimberg, Pierre Hansen, Sébastien Le Digabel, and Nenad Mladenović. Pooling problem: Alternate formulations and solution methods. Management science, 50(6):761-776, 2004.","title":"Pooling problem: Alternate formulations and solution methods","authors":"Charles Audet, Jack Brimberg, Pierre Hansen, Sébastien, Le Digabel, Nenad Mladenović","year":"2004","venue":"Management science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Beck, 2017] Amir Beck. First-order methods in optimiza- tion. SIAM, 2017.","title":"First-order methods in optimization","authors":"Amir Beck","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Berrada et al., 2018] Leonard Berrada, Andrew Zisserman, and M. Pawan Kumar. Smooth loss functions for deep top- k classification. ArXiv, abs/1802.07595, 2018.","title":"Smooth loss functions for deep topk classification","authors":"Leonard Berrada, Andrew Zisserman, M Pawan Kumar","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Boyd et al., 2011] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed op- timization and statistical learning via the alternating direc- tion method of multipliers. Foundations and Trends R in Machine learning, 3(1):1-122, 2011.","title":"Distributed optimization and statistical learning via the alternating direction method of multipliers","authors":"Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein","year":"2011","venue":"Foundations and Trends R in Machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Elmachtoub and Grigas, 2021] Adam N Elmachtoub and Paul Grigas. Smart \"predict, then optimize\". Management Science, 2021.","title":"Smart \"predict, then optimize","authors":"N Adam, Paul Elmachtoub, Grigas","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gurobi Optimization, LLC, 2023] Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual. url- https://www.gurobi.com, 2023.","title":"Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual","authors":null,"year":"2023","venue":null,"doi":null,"url":"https://www.gurobi.com,2023","internal_url":null},{"position":11,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 4700-4708, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kotary et al., 2021] James Kotary, Ferdinando Fioretto, Pascal Van Hentenryck, and Bryan Wilder. End-to-end constrained optimization learning: A survey. In Proceed- ings of the Thirtieth International Joint Conference on Ar- tificial Intelligence, IJCAI-21, pages 4475-4482, 2021.","title":"End-to-end constrained optimization learning: A survey","authors":"James Kotary, Ferdinando Fioretto, Pascal Van Hentenryck, Bryan Wilder","year":"2021","venue":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21","doi":null,"url":null,"internal_url":"/proceedings/2021/610"},{"position":13,"raw":"[Kotary et al., 2023a] James Kotary, Francesco Di Vito, and Ferdinando Fioretto. Differentiable model selection for ensemble learning. In Proceedings of the Fifteen Interna- tional Joint Conference on Artificial Intelligence, IJCAI- 23, 2023.","title":"Differentiable model selection for ensemble learning","authors":"Kotary","year":"2023","venue":"Proceedings of the Fifteen International Joint Conference on Artificial Intelligence, IJCAI-23","doi":null,"url":null,"internal_url":"/proceedings/2023/217"},{"position":14,"raw":"[Kotary et al., 2023b] James Kotary, My H. Dinh, and Ferdi- nando Fioretto. Backpropagation of unrolled solvers with folded optimization. arXiv preprint arXiv:2301.12047, 2023.","title":"Backpropagation of unrolled solvers with folded optimization","authors":"Kotary","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Martins and Astudillo, 2016] Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Interna- tional conference on machine learning, pages 1614-1623.","title":"From softmax to sparsemax: A sparse model of attention and multi-label classification","authors":"Astudillo Martins, Andre Martins, Ramon Astudillo","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"PMLR, 2016.","title":null,"authors":"Pmlr","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Monga et al., 2021] Vishal Monga, Yuelong Li, and Yon- ina C Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Sig- nal Processing Magazine, 38(2):18-44, 2021.","title":"Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing","authors":"Yuelong Vishal Monga, Yonina C Li, Eldar","year":"2021","venue":"IEEE Signal Processing Magazine","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Munkres, 2018] James R Munkres. Analysis on manifolds. CRC Press, 2018.","title":"Analysis on manifolds","authors":"James R Munkres","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Quarteroni et al., 2010] Alfio Quarteroni, Riccardo Sacco, and Fausto Saleri. Numerical mathematics, volume 37. Springer Science & Business Media, 2010.","title":"Numerical mathematics","authors":"Alfio Quarteroni, Riccardo Sacco, Fausto Saleri","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wilder et al., 2019] Bryan Wilder, Bistra Dilkina, and Milind Tambe. Melding the data-decisions pipeline: Decision-focused learning for combinatorial optimization. In AAAI, volume 33, pages 1658-1665, 2019.","title":"Melding the data-decisions pipeline: Decision-focused learning for combinatorial optimization","authors":"Bryan Wilder, Bistra Dilkina, Milind Tambe","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}