{"type":"ScholarlyArticle","title":"Toward Designing Convergent Deep Operator Splitting Methods for Task-specific Nonconvex Optimization","authors":[{"name":"Risheng Liu","slug":"risheng-liu","given":"Risheng","family":"Liu","affiliation":"International School of Information Science & Engineering , Dalian University of Technology; Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province"},{"name":"Shichao Cheng","slug":"shichao-cheng","given":"Shichao","family":"Cheng","affiliation":"Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province; School of Mathematical Science , Dalian University of Technology"},{"name":"Yi He","slug":"yi-he","given":"Yi","family":"He","affiliation":"International School of Information Science & Engineering , Dalian University of Technology; Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province"},{"name":"Xin Fan","slug":"xin-fan","given":"Xin","family":"Fan","affiliation":"International School of Information Science & Engineering , Dalian University of Technology; Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province"},{"name":"Zhongxuan Luo","slug":"zhongxuan-luo","given":"Zhongxuan","family":"Luo","affiliation":"International School of Information Science & Engineering , Dalian University of Technology; Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province; School of Mathematical Science , Dalian University of Technology"}],"abstract":"Operator splitting methods have been successfully used in computational sciences, statistics, learning and vision areas to reduce complex problems into a series of simpler subproblems. However, prevalent splitting schemes are mostly established only based on the mathematical properties of some general optimization models. So it is a laborious process and often requires many iterations of ideation and validation to obtain practical and task-specific optimal solutions, especially for nonconvex problems in real-world scenarios. To break through the above limits, we introduce a new algorithmic framework, called Learnable Bregman Splitting (LBS), to perform deep-architecture-based operator splitting for nonconvex optimization based on specific task model. Thanks to the data-dependent (i.e., learnable) nature, our LBS can not only speed up the convergence, but also avoid unwanted trivial solutions for real-world tasks. Though with inexact deep iterations, we can still establish the global convergence and estimate the asymptotic convergence rate of LBS only by enforcing some fairly loose assumptions. Extensive experiments on different applications (e.g., image completion and deblurring) verify our theoretical results and show the superiority of LBS against existing methods.","doi":"10.24963/ijcai.2018/342","url":"https://ijcai.org/proceedings/2018/342","pdf":"https://ijcai.org/proceedings/2018/0342.pdf","markdown":"https://ijcai.org/proceedings/2018/342.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/342","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":342,"pages":{"first":2468,"last":2474},"keywords":["Computer Vision: Statistical Methods and Machine Learning","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Arias et al., 2011] Pablo Arias, Gabriele Facciolo, Vicent Caselles, and Guillermo Sapiro. A variational framework for exemplar-based image inpainting. IJCV, 93(3):319-347, 2011.","title":"A variational framework for exemplar-based image inpainting","authors":"Arias","year":"2011","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bauschke et al., 1997] Heinz H Bauschke, Jonathan M Bor- wein, et al. Legendre functions and the method of random bregman projections. Journal of Convex Analysis, 4(1):27- 67, 1997.","title":"Legendre functions and the method of random bregman projections","authors":"Jonathan M Heinz H Bauschke, Borwein","year":"1997","venue":"Journal of Convex Analysis","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Beck and Teboulle, 2009] Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183-202, 2009.","title":"A fast iterative shrinkage-thresholding algorithm for linear inverse problems","authors":"Amir Beck, Marc Teboulle","year":"2009","venue":"SIAM Journal on Imaging Sciences","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2015] Yunjin Chen, Wei Yu, and Thomas Pock. On learning optimized reaction diffusion processes for ef- fective image restoration. In CVPR, 2015.","title":"On learning optimized reaction diffusion processes for effective image restoration","authors":"Yunjin Chen, Wei Yu, Thomas Pock","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Danielyan et al., 2012] Aram Danielyan, Vladimir Katkovnik, and Karen Egiazarian. Bm3d frames and variational image deblurring. IEEE TIP, 21(4):1715-1728, 2012.","title":"Bm3d frames and variational image deblurring","authors":"Aram Danielyan, Vladimir Katkovnik, Karen Egiazarian","year":"2012","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Davis and Yin, 2016] Damek Davis and Wotao Yin. Conver- gence rate analysis of several splitting schemes. In Splitting Methods in Communication, Imaging, Science, and Engi- neering, pages 115-163. 2016.","title":"Convergence rate analysis of several splitting schemes","authors":"Yin ; Davis, Damek Davis, Wotao Yin","year":"2016","venue":"Splitting Methods in Communication, Imaging, Science, and Engineering","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He and Wang, 2014] Liangtian He and Yilun Wang. Iterative support detection-based split bregman method for wavelet frame-based image inpainting. IEEE TIP, 23(12):5470- 5485, 2014.","title":"Liangtian He and Yilun Wang. Iterative support detection-based split bregman method for wavelet frame-based image inpainting","authors":"Wang He","year":"2014","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krishnan and Fergus, 2009] Dilip Krishnan and Rob Fergus. Fast image deconvolution using hyper-laplacian priors. In NIPS, 2009.","title":"Fast image deconvolution using hyper-laplacian priors","authors":"Dilip Krishnan, Rob Fergus","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kruse et al., 2017] Jakob Kruse, Carsten Rother, and Uwe Schmidt. Learning to push the limits of efficient fft-based image deconvolution. In ICCV, 2017.","title":"Learning to push the limits of efficient fft-based image deconvolution","authors":"Kruse","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2011] Zhouchen Lin, Risheng Liu, and Zhixun Su. Linearized alternating direction method with adaptive penalty for low-rank representation. In NIPS, 2011.","title":"Linearized alternating direction method with adaptive penalty for low-rank representation","authors":"Lin","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2015] Zhouchen Lin, Risheng Liu, and Huan Li. Linearized alternating direction method with parallel split- ting and adaptive penalty for separable convex programs in machine learning. Machine Learning, 99(2):287, 2015.","title":"Linearized alternating direction method with parallel splitting and adaptive penalty for separable convex programs in machine learning","authors":"Lin","year":"2015","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2016] Risheng Liu, Guangyu Zhong, Junjie Cao, Zhouchen Lin, Shiguang Shan, and Zhongxuan Luo. Learn- ing to diffuse: A new perspective to design pdes for visual analysis. IEEE TPAMI, 38(12):2457-2471, 2016.","title":"Learning to diffuse: A new perspective to design pdes for visual analysis","authors":"Risheng Liu, Guangyu Zhong, Junjie Cao, Zhouchen Lin, Shiguang Shan, Zhongxuan Luo","year":"2016","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2018] Risheng Liu, Xin Fan, Shichao Cheng, X- iangyu Wang, and Zhongxuan Luo. Proximal alternating direction network: A globally converged deep unrolling framework. In AAAI, 2018.","title":"Proximal alternating direction network: A globally converged deep unrolling framework","authors":"Risheng Liu, Xin Fan, Shichao Cheng, Xiangyu Wang, Zhongxuan Luo","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Passty, 1979] Gregory B Passty. Ergodic convergence to a ze- ro of the sum of monotone operators in hilbert space. Jour- nal of Mathematical Analysis and Applications, 72(2):383- 390, 1979.","title":"Ergodic convergence to a zero of the sum of monotone operators in hilbert space","authors":"Gregory B Passty","year":"1979","venue":"Journal of Mathematical Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Roth and Black, 2009] Stefan Roth and Michael J. Black. Fields of experts. IJCV, 82(2):205-229, 2009.","title":"Fields of experts","authors":"Stefan Roth, Michael J Black","year":"2009","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schmidt et al., 2016] Uwe Schmidt, Jeremy Jancsary, Sebas- tian Nowozin, Stefan Roth, and Carsten Rother. Cascades of regression tree fields for image restoration. IEEE TPAMI, 38(4):677-689, 2016.","title":"Cascades of regression tree fields for image restoration","authors":"Schmidt","year":"2016","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schuler et al., 2013] Christian J Schuler, Harold Christo- pher Burger, Stefan Harmeling, and Bernhard Scholkopf. A machine learning approach for non-blind image decon- volution. In CVPR, 2013.","title":"A machine learning approach for non-blind image deconvolution","authors":"Schuler","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shi et al., 2016] Hao-Jun Michael Shi, Shenyinying Tu, Yangyang Xu, and Wotao Yin. A primer on coordinate descent algorithms. arXiv:1610.00040, 2016.","title":"A primer on coordinate descent algorithms","authors":"Michael Hao-Jun, Shenyinying Shi, Yangyang Tu, Wotao Xu, Yin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sun et al., 2013] Libin Sun, Sunghyun Cho, Jue Wang, and James Hays. Edge-based blur kernel estimation using patch priors. In ICCP, 2013.","title":"Edge-based blur kernel estimation using patch priors","authors":"Libin Sun, Sunghyun Cho, Jue Wang, James Hays","year":"2013","venue":"ICCP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Uwe and Stefan, 2014] Schmidt Uwe and Roth Stefan. Shrinkage fields for effective image restoration. In CVPR, 2014.","title":"Shrinkage fields for effective image restoration","authors":"Stefan ; Uwe, Schmidt Uwe, Roth Stefan","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2008] Yilun Wang, Junfeng Yang, Wotao Yin, and Yin Zhang. A new alternating minimization algorithm for total variation image reconstruction. SIAM Journal on Imaging Sciences, 1(3):248-272, 2008.","title":"A new alternating minimization algorithm for total variation image reconstruction","authors":"Yilun Wang, Junfeng Yang, Wotao Yin, Yin Zhang","year":"2008","venue":"SIAM Journal on Imaging Sciences","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu and Yin, 2017] Yangyang Xu and Wotao Yin. A globally convergent algorithm for nonconvex optimization based on block coordinate update. Journal of Scientific Computing, pages 1-35, 2017.","title":"A globally convergent algorithm for nonconvex optimization based on block coordinate update","authors":"Yin ; ] Yangyang Xu, Wotao Xu, Yin","year":"2017","venue":"Journal of Scientific Computing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2014] Jian Zhang, Debin Zhao, Ruiqin Xiong, Siwei Ma, and Wen Gao. Image restoration using joint statistical modeling in a space-transform domain. IEEE TCSVT, 24(6):915-928, 2014.","title":"Image restoration using joint statistical modeling in a space-transform domain","authors":"Jian Zhang, Debin Zhao, Ruiqin Xiong, Siwei Ma, Wen Gao","year":"2014","venue":"IEEE TCSVT","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2017] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep cnn denoiser prior for image restoration. In CVPR, 2017.","title":"Learning deep cnn denoiser prior for image restoration","authors":"Kai Zhang, Wangmeng Zuo, Shuhang Gu, Lei Zhang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zoran and Weiss, 2011] Daniel Zoran and Yair Weiss. From learning models of natural image patches to whole image restoration. In ICCV, 2011.","title":"From learning models of natural image patches to whole image restoration","authors":"Weiss ; ] Zoran, Yair Daniel Zoran, Weiss","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}