{"type":"ScholarlyArticle","title":"Combining ADMM and the Augmented Lagrangian Method for Efficiently Handling Many Constraints","authors":[{"name":"Joachim Giesen","slug":"joachim-giesen","given":"Joachim","family":"Giesen","affiliation":"Friedrich-Schiller-Universität Jena"},{"name":"Soeren Laue","slug":"soeren-laue","given":"Soeren","family":"Laue","affiliation":"Friedrich-Schiller-Universität Jena; Data Assessment Solutions"}],"abstract":"Many machine learning methods entail minimizing a loss-function that is the sum of the losses for each data point. The form of the loss function is exploited algorithmically, for instance in stochastic gradient descent (SGD) and in the alternating direction method of multipliers (ADMM).  However, there are also machine learning methods where the entailed optimization problem features the data points not in the objective function but in the form of constraints, typically one constraint per data point. Here, we address the problem of solving convex optimization problems with many convex constraints. Our approach is an extension of ADMM. The straightforward implementation of ADMM for solving constrained optimization problems in a distributed fashion solves constrained subproblems on different compute nodes that are aggregated until a consensus solution is reached. Hence, the straightforward approach has three nested loops: one for reaching consensus, one for the constraints, and one for the unconstrained problems. Here, we show that solving the costly constrained subproblems can be avoided. In our approach, we combine the ability of ADMM to solve convex optimization problems in a distributed setting with the ability of the augmented Lagrangian method to solve constrained optimization problems. Consequently, our algorithm only needs two nested loops. We prove that it inherits the convergence guarantees of both ADMM and the augmented Lagrangian method. Experimental results corroborate our theoretical findings.","doi":"10.24963/ijcai.2019/629","url":"https://ijcai.org/proceedings/2019/629","pdf":"https://ijcai.org/proceedings/2019/0629.pdf","markdown":"https://ijcai.org/proceedings/2019/629.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/629","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning Applications","track":"Main track","number":629,"pages":{"first":4525,"last":4531},"keywords":["Agent-based and Multi-agent Systems: Multi-agent Learning","Machine Learning Applications: Big data ; Scalability"],"videos":[],"references":[{"position":1,"raw":"[Andersen et al., 2012] Martin Andersen, Joachim Dahl, Zhang Liu, and Lieven Vandenberghe. Interior-Point Methods for Large-Scale Cone Programming, pages 55- 84. MIT Press, 2012.","title":"Interior-Point Methods for Large-Scale Cone Programming","authors":"Martin Andersen, Joachim Dahl, Zhang Liu, Lieven Vandenberghe","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boyd et al., 2011] Stephen P. Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed opti- mization and statistical learning via the alternating direc- tion method of multipliers. Foundations and Trends in Ma- chine Learning, 3(1):1-122, 2011.","title":"Distributed optimization and statistical learning via the alternating direction method of multipliers","authors":"Stephen P Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein","year":"2011","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gabay and Mercier, 1976] Daniel Gabay and Bertrand Mercier. A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Computers & Mathematics with Applications, 2(1):17 - 40, 1976.","title":"A dual algorithm for the solution of nonlinear variational problems via finite element approximation","authors":"Daniel Gabay, Bertrand Mercier","year":"1976","venue":"Computers & Mathematics with Applications","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ghadimi et al., 2015] Euhanna Ghadimi, André Teixeira, Iman Shames, and Mikael Johansson. Optimal parameter selection for the alternating direction method of multipli- ers (admm): Quadratic problems. IEEE Trans. Automat. Contr., 60(3):644-658, 2015.","title":"Optimal parameter selection for the alternating direction method of multipliers (admm): Quadratic problems","authors":"Ghadimi","year":"2015","venue":"IEEE Trans. Automat. Contr","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Giesen and Laue, 2016] Joachim Giesen and Sören Laue. Distributed convex optimization with many convex con- straints. CoRR, abs/1610.02967, 2016.","title":"Distributed convex optimization with many convex constraints","authors":"Laue Giesen, Joachim Giesen, Sören Laue","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Glowinski and Marroco, 1975] R. Glowinski and A. Mar- roco. Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de dirichlet non linéaires. ESAIM: Mathe- matical Modelling and Numerical Analysis -Modélisation Mathématique et Analyse Numérique, 9(R2):41-76, 1975.","title":"Marroco. Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de dirichlet non linéaires","authors":"R Glowinski, A","year":"1975","venue":"ESAIM: Mathematical Modelling and Numerical Analysis -Modélisation Mathématique et Analyse Numérique","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hestenes, 1969] Magnus R. Hestenes. Multiplier and gradi- ent methods. Journal of Optimization Theory and Appli- cations, 4(5):303-320, 1969.","title":"Multiplier and gradient methods","authors":"Magnus R Hestenes","year":"1969","venue":"Journal of Optimization Theory and Applications","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang and Sidiropoulos, 2016] Kejun Huang and Nicholas D Sidiropoulos. Consensus-admm for general quadratically constrained quadratic programming. IEEE Transactions on Signal Processing, 64(20):5297-5310, 2016.","title":"Consensus-admm for general quadratically constrained quadratic programming","authors":"Kejun Huang, Nicholas D Sidiropoulos","year":"2016","venue":"IEEE Transactions on Signal Processing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Morales and Nocedal, 2011] José Luis Morales and Jorge Nocedal. Remark on \"algorithm 778: L-BFGS-B: fortran subroutines for large-scale bound constrained optimiza- tion\". ACM Trans. Math. Softw., 38(1):7:1-7:4, 2011.","title":"José Luis Morales and Jorge Nocedal. Remark on \"algorithm 778: L-BFGS-B: fortran subroutines for large-scale bound constrained optimization","authors":"Nocedal Morales","year":"2011","venue":"ACM Trans. Math. Softw","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Mosk-Aoyama et al., 2010] Damon Mosk-Aoyama, Tim Roughgarden, and Devavrat Shah. Fully distributed algo- rithms for convex optimization problems. SIAM Journal on Optimization, 20(6):3260-3279, 2010.","title":"Fully distributed algorithms for convex optimization problems","authors":"Mosk-Aoyama","year":"2010","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Powell, 1969] M. J. D. Powell. Algorithms for nonlinear constraints that use lagrangian functions. Mathematical Programming, 14(1):224-248, 1969.","title":"Algorithms for nonlinear constraints that use lagrangian functions","authors":"M J D Powell","year":"1969","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Shalev-Shwartz and Srebro, 2008] Shai Shalev-Shwartz and Nathan Srebro. SVM optimization: inverse depen- dence on training set size. In International Conference on Machine Learning (ICML), pages 928-935, 2008.","title":"SVM optimization: inverse dependence on training set size","authors":"Shalev-Shwartz, Srebro ; Shai Shalev-Shwartz, Nathan Srebro","year":"2008","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Shivaswamy et al., 2006] Pannagadatta K. Shivaswamy, Chiranjib Bhattacharyya, and Alexander J. Smola. Sec- ond order cone programming approaches for handling missing and uncertain data. Journal of Machine Learning Research, 7:1283-1314, 2006.","title":"Second order cone programming approaches for handling missing and uncertain data","authors":"K Pannagadatta, Chiranjib Shivaswamy, Alexander J Bhattacharyya, Smola","year":"2006","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Slater, 1950] Morton Slater. Lagrange multipliers revisited. Cowles Foundation Discussion Papers 80, Cowles Foun- dation for Research in Economics, Yale University, 1950.","title":"Lagrange multipliers revisited","authors":"Morton Slater","year":"1950","venue":"Cowles Foundation Discussion Papers","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Tsang et al., 2007] Ivor W. Tsang, András Kocsor, and James T. Kwok. Simpler core vector machines with en- closing balls. In International Conference on Machine Learning (ICML), pages 911-918, 2007.","title":"Simpler core vector machines with enclosing balls","authors":"Ivor W Tsang, András Kocsor, James T Kwok","year":"2007","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Zhang and Kwok, 2014] Ruiliang Zhang and James T Kwok. Asynchronous distributed admm for consensus optimization. In International Conference on Machine Learning (ICML), pages 1701-1709, 2014.","title":"Asynchronous distributed admm for consensus optimization","authors":"Kwok ; ] Ruiliang Zhang, James T Zhang, Kwok","year":"2014","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Zhu and Martínez, 2012] Minghui Zhu and Sonia Martínez. On distributed convex optimization under inequality and equality constraints. IEEE Trans. Automat. Contr., 57(1):151-164, 2012.","title":"On distributed convex optimization under inequality and equality constraints","authors":"Martínez Zhu, Minghui Zhu, Sonia Martínez","year":"2012","venue":"IEEE Trans. Automat. Contr","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhu et al., 1997] Ciyou Zhu, Richard H. Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: L-BFGS-B: for- tran subroutines for large-scale bound-constrained opti- mization. ACM Trans. Math. Softw., 23(4):550-560, 1997.","title":"Algorithm 778: L-BFGS-B: fortran subroutines for large-scale bound-constrained optimization","authors":"Ciyou Zhu, Richard H Byrd, Peihuang Lu, Jorge Nocedal","year":"1997","venue":"ACM Trans. Math. Softw","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}