{"type":"ScholarlyArticle","title":"Projected Gradient Descent Algorithms for Solving Nonlinear Inverse Problems with Generative Priors","authors":[{"name":"Zhaoqiang Liu","slug":"zhaoqiang-liu","given":"Zhaoqiang","family":"Liu","affiliation":"National University of Singapore"},{"name":"Jun Han","slug":"jun-han","given":"Jun","family":"Han","affiliation":"PCG , Tencent"}],"abstract":"In this paper, we propose projected gradient descent (PGD) algorithms for signal estimation from noisy nonlinear measurements. We assume that the unknown signal lies near the range of a Lipschitz continuous generative model with bounded inputs. In particular, we consider two cases when the nonlinear link function is either unknown or known. For unknown nonlinearity, we make the assumption of sub-Gaussian observations and propose a linear least-squares estimator. We show that when there is no representation error, the sensing vectors are Gaussian, and the number of samples is sufficiently large, with high probability, a PGD algorithm converges linearly to a point achieving the optimal statistical rate using arbitrary initialization. For known nonlinearity, we assume monotonicity, and make much weaker assumptions on the sensing vectors and allow for representation error. We propose a nonlinear least-squares estimator that is guaranteed to enjoy an optimal statistical rate. A corresponding PGD algorithm is provided and is shown to also converge linearly to the estimator using arbitrary initialization. In addition, we present experimental results on image datasets to demonstrate the performance of our PGD algorithms.","doi":"10.24963/ijcai.2022/454","url":"https://ijcai.org/proceedings/2022/454","pdf":"https://ijcai.org/proceedings/2022/0454.pdf","markdown":"https://ijcai.org/proceedings/2022/454.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/454","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":454,"pages":{"first":3271,"last":3277},"keywords":["Machine Learning: Learning Theory","Machine Learning: Probabilistic Machine Learning","Machine Learning: Unsupervised Learning","Machine Learning: Theory of Deep Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986573","length_sec":85,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Bora et al., 2017] A. Bora, A. Jalal, E. Price, and A. Di- makis. Compressed sensing using generative models. In ICML, 2017.","title":"Compressed sensing using generative models","authors":"A Bora, A Jalal, E Price, A Dimakis","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Foucart and Rauhut, 2013] S. Foucart and H. Rauhut. A mathematical introduction to compressive sensing. Springer New York, 2013.","title":"A mathematical introduction to compressive sensing","authors":"S Foucart, H Rauhut","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Genzel, 2016] M. Genzel. High-dimensional estimation of structured signals from non-linear observations with gen- eral convex loss functions. IEEE Trans. Inf. Theory, 63(3):1601-1619, 2016.","title":"High-dimensional estimation of structured signals from non-linear observations with general convex loss functions","authors":"M Genzel","year":"2016","venue":"IEEE Trans. Inf. Theory","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Han, 1987] A. Han. Non-parametric analysis of a gener- alized regression model: The maximum rank correlation estimator. J. Econom., 35(2-3):303-316, 1987.","title":"Non-parametric analysis of a generalized regression model: The maximum rank correlation estimator","authors":"A Han","year":"1987","venue":"J. Econom","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hand et al., 2018] P. Hand, O. Leong, and V. Voroninski. Phase retrieval under a generative prior. In NeurIPS, 2018.","title":"Phase retrieval under a generative prior","authors":"P Hand, O Leong, V Voroninski","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Heckel and Hand, 2019] R. Heckel and P. Hand. Deep de- coder: Concise image representations from untrained non- convolutional networks. In ICLR, 2019.","title":"Deep decoder: Concise image representations from untrained nonconvolutional networks","authors":"R Heckel, P Hand","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hyder et al., 2019] R. Hyder, V. Shah, C. Hegde, and S. Asif. Alternating phase projected gradient descent with generative priors for solving compressive phase retrieval. In ICASSP, 2019.","title":"Alternating phase projected gradient descent with generative priors for solving compressive phase retrieval","authors":"R Hyder, V Shah, C Hegde, S Asif","year":"2019","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jagatap and Hegde, 2019] G. Jagatap and C. Hegde. Algo- rithmic guarantees for inverse imaging with untrained net- work priors. In NeurIPS, 2019.","title":"Algorithmic guarantees for inverse imaging with untrained network priors","authors":"G Jagatap, C Hegde","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jalal et al., 2021] A. Jalal, S. Karmalkar, A. Dimakis, and E. Price. Instance-optimal compressed sensing via poste- rior sampling. In ICML, 2021.","title":"Instance-optimal compressed sensing via posterior sampling","authors":"A Jalal, S Karmalkar, A Dimakis, E Price","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kamath et al., 2020] A. Kamath, S. Karmalkar, and E. Price. On the power of compressed sensing with generative models. In ICML, 2020.","title":"On the power of compressed sensing with generative models","authors":"A Kamath, S Karmalkar, E Price","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krahmer and Ward, 2011] F. Krahmer and R. Ward. New and improved Johnson-Lindenstrauss embeddings via the restricted isometry property. SIAM J. Math. Anal., 43(3):1269-1281, 2011.","title":"New and improved Johnson-Lindenstrauss embeddings via the restricted isometry property","authors":"F Krahmer, R Ward","year":"2011","venue":"SIAM J. Math. Anal","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[LeCun et al., 1998] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11), 1998.","title":"Gradient-based learning applied to document recognition","authors":"Y Lecun, L Bottou, Y Bengio, P Haffner","year":"1998","venue":"Proc. IEEE","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu and Scarlett, 2020a] Z. Liu and J. Scarlett. The gener- alized Lasso with nonlinear observations and generative priors. In NeurIPS, 2020.","title":"The generalized Lasso with nonlinear observations and generative priors","authors":"Scarlett ; Liu, ] Z Liu, J Scarlett","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu and Scarlett, 2020b] Z. Liu and J. Scarlett. Information- theoretic lower bounds for compressive sensing with gen- erative models. IEEE J. Sel. Areas Inf. Theory, 1(1):292- 303, 2020.","title":"Informationtheoretic lower bounds for compressive sensing with generative models","authors":"Scarlett ; Liu, ] Z Liu, J Scarlett","year":"2020","venue":"IEEE J. Sel. Areas Inf. Theory","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2015] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In ICCV, 2015.","title":"Deep learning face attributes in the wild","authors":"Z Liu, P Luo, X Wang, X Tang","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2020] Z. Liu, S. Gomes, A. Tiwari, and J. Scar- lett. Sample complexity bounds for 1-bit compressive sensing and binary stable embeddings with generative pri- ors. In ICML, 2020.","title":"Sample complexity bounds for 1-bit compressive sensing and binary stable embeddings with generative priors","authors":"Z Liu, S Gomes, A Tiwari, J Scarlett","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2021a] Z. Liu, S. Ghosh, J. Han, and J. Scar- lett. Robust 1-bit compressive sensing with par- tial Gaussian circulant matrices and generative priors. https://arxiv.org/2108.03570, 2021.","title":"Scarlett. Robust 1-bit compressive sensing with partial Gaussian circulant matrices and generative priors","authors":"Liu","year":"2021","venue":null,"doi":null,"url":"https://arxiv.org/2108.03570","internal_url":null},{"position":18,"raw":"[Liu et al., 2021b] Z. Liu, S. Ghosh, and J. Scarlett. Towards sample-optimal compressive phase retrieval with sparse and generative priors. In NeurIPS, 2021.","title":"Towards sample-optimal compressive phase retrieval with sparse and generative priors","authors":"Liu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2022] Z. Liu, J. Liu, S. Ghosh, J. Han, and J. Scarlett. Generative principal component analysis. In ICLR, 2022.","title":"Generative principal component analysis","authors":"Z Liu, J Liu, S Ghosh, J Han, J Scarlett","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ongie et al., 2020] G. Ongie, A. Jalal, C. Metzler, R. Bara- niuk, A. Dimakis, and R. Willett. Deep learning tech- niques for inverse problems in imaging. IEEE J. Sel. Areas Inf. Theory, 1(1):39-56, 2020.","title":"Deep learning techniques for inverse problems in imaging","authors":"G Ongie, A Jalal, C Metzler, R Baraniuk, A Dimakis, R Willett","year":"2020","venue":"IEEE J. Sel. Areas Inf. Theory","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Oymak and Soltanolkotabi, 2017] S. Oymak and M. Soltanolkotabi. Fast and reliable parameter esti- mation from nonlinear observations. SIAM J. Optim., 27(4):2276-2300, 2017.","title":"Fast and reliable parameter estimation from nonlinear observations","authors":"S Oymak, M Soltanolkotabi","year":"2017","venue":"SIAM J. Optim","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Peng et al., 2020] P. Peng, S. Jalali, and X. Yuan. Solving inverse problems via auto-encoders. IEEE J. Sel. Areas Inf. Theory, 1(1):312-323, 2020.","title":"Solving inverse problems via auto-encoders","authors":"P Peng, S Jalali, X Yuan","year":"2020","venue":"IEEE J. Sel. Areas Inf. Theory","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Plan and Vershynin, 2016] Y. Plan and R. Vershynin. The generalized Lasso with non-linear observations. IEEE Trans. Inf. Theory, 62(3):1528-1537, 2016.","title":"The generalized Lasso with non-linear observations","authors":"Y Plan, R Vershynin","year":"2016","venue":"IEEE Trans. Inf. Theory","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Plan et al., 2017] Y. Plan, R. Vershynin, and E. Yudovina. High-dimensional estimation with geometric constraints. Inf. Inference, 6(1):1-40, 2017.","title":"High-dimensional estimation with geometric constraints","authors":"Y Plan, R Vershynin, E Yudovina","year":"2017","venue":"Inf. Inference","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Qiu et al., 2020] S. Qiu, X. Wei, and Z. Yang. Robust one- bit recovery via ReLU generative networks: Improved sta- tistical rates and global landscape analysis. In ICML, 2020.","title":"Robust onebit recovery via ReLU generative networks: Improved statistical rates and global landscape analysis","authors":"S Qiu, X Wei, Z Yang","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Raj et al., 2019] A. Raj, Y. Li, and Y. Bresler. GAN-based projector for faster recovery with convergence guarantees in linear inverse problems. In ICCV, 2019.","title":"GAN-based projector for faster recovery with convergence guarantees in linear inverse problems","authors":"A Raj, Y Li, Y Bresler","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shah and Hegde, 2018] V. Shah and C. Hegde. Solving lin- ear inverse problems using GAN priors: An algorithm with provable guarantees. In ICASSP, 2018.","title":"Solving linear inverse problems using GAN priors: An algorithm with provable guarantees","authors":"V Shah, C Hegde","year":"2018","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Soltani and Hegde, 2017] M. Soltani and C. Hegde. Fast al- gorithms for demixing sparse signals from nonlinear ob- servations. IEEE Trans. Sig. Proc., 65(16):4209-4222, 2017.","title":"Fast algorithms for demixing sparse signals from nonlinear observations","authors":"M Soltani, C Hegde","year":"2017","venue":"IEEE Trans. Sig. Proc","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tibshirani, 1996] R Tibshirani. Regression shrinkage and selection via the Lasso. JRSSB, pages 267-288, 1996.","title":"Regression shrinkage and selection via the Lasso","authors":"Tibshirani","year":"1996","venue":"JRSSB","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Vershynin, 2010] R. Vershynin. Introduction to the non-asymptotic analysis of random matrices. https://arxiv.org/abs/1011.3027, 2010.","title":"Introduction to the non-asymptotic analysis of random matrices","authors":"R Vershynin","year":"2010","venue":null,"doi":null,"url":"https://arxiv.org/abs/1011.3027","internal_url":null},{"position":31,"raw":"[Wei et al., 2019] X. Wei, Z. Yang, and Z. Wang. On the statistical rate of nonlinear recovery in generative models with heavy-tailed data. In ICML, 2019.","title":"On the statistical rate of nonlinear recovery in generative models with heavy-tailed data","authors":"X Wei, Z Yang, Z Wang","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2016] Z. Yang, Z. Wang, H. Liu, Y. Eldar, and T. Zhang. Sparse nonlinear regression: Parameter estima- tion under nonconvexity. In ICML, 2016.","title":"Sparse nonlinear regression: Parameter estimation under nonconvexity","authors":"Z Yang, Z Wang, H Liu, Y Eldar, T Zhang","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2018] K. Zhang, Z. Yang, and Z. Wang. Non- linear structured signal estimation in high dimensions via iterative hard thresholding. In AISTATS, 2018.","title":"Nonlinear structured signal estimation in high dimensions via iterative hard thresholding","authors":"K Zhang, Z Yang, Z Wang","year":"2018","venue":"AISTATS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}