Universality Laws and Performance Analysis of the Generalized Linear Models Citation Abbasi, Ehsan (2020) Universality Laws and Performance Analysis of the Generalized Linear Models. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/873c-ej41. https://resolver.caltech.edu/CaltechTHESIS:06092020-005908250 Abstract In the past couple of decades, non-smooth convex optimization has emerged as a powerful tool for the recovery of structured signals (sparse, low rank, etc.) from noisy linear or non-linear measurements in a variety of applications in genomics, signal processing, wireless communications, machine learning, etc.. Taking advantage of the particular structure of the unknown signal of interest is critical since in most of these applications, the dimension p of the signal to be estimated is comparable, or even larger than the number of observations n . With the advent of Compressive Sensing there has been a very large number of theoretical results that study the estimation performance of non-smooth convex optimization in such a high-dimensional setting . A popular approach for estimating an unknown signal β₀ ϵ ℝ ᵖ in a generalized linear model , with observations y = g( X β₀) ϵ ℝ ⁿ , is via solving the estimator β̂ = arg min β L ( y , X β + λf ( β ). Here, L (•,•) is a loss function which is convex with respect to its second argument, and f (•) is a regularizer that enforces the structure of the unknown β₀. We first analyze the generalization error performance of this estimator, for the case where the entries of X are drawn independently from real standard Gaussian distribution. The precise nature of our analysis permits an accurate performance comparison between different instances of these estimators, and allows to optimally tune the hyperparameters based on the model parameters. We apply our result to some of the most popular cases of generalized linear models, such as M-estimators in linear regression, logistic regression and generalized margin maximizers in binary classification problems, and Poisson regression in count data models. The key ingredient of our proof is the Convex Gaussian Min-max Theorem (CGMT) , which is a tight version of the Gaussian comparison inequality proved by Gordon in 1988. Unfortunately, having real iid entries in the features matrix X is crucial in this theorem, and it cannot be naturally extended to other cases. But for some special cases, we prove some universality properties and indirectly extend these results to more general designs of the features matrix X , where the entries are not necessarily real, independent, or identically distributed. This extension, enables us to analyze problems that CGMT was incapable of, such as models with quadratic measurements, phase-lift in phase retrieval, and data recovery in massive MIMO, and help us settle a few long standing open problems in these areas. Item Type: Thesis (Dissertation (Ph.D.)) Subject Keywords: Generalized Linear Models, Convex Optimization, Performance Analysis, Machine Learning, Statistical Learning Degree Grantor: California Institute of Technology Division: Engineering and Applied Science Major Option: Electrical Engineering Thesis Availability: Public (worldwide access) Research Advisor(s): Hassibi, Babak Thesis Committee: Vaidyanathan, P. P. (chair) Tropp, Joel A. Chandrasekaran, Venkat Hassibi, Babak Defense Date: 18 May 2020 Record Number: CaltechTHESIS:06092020-005908250 Persistent URL: https://resolver.caltech.edu/CaltechTHESIS:06092020-005908250 DOI: 10.7907/873c-ej41 Related URLs: URL URL Type Description http://papers.nips.cc/paper/5739-lasso-with-non-linear-measurements-is-equivalent-to-one-with-linear-measurements Publisher Article adapted for Chapter 2. https://arxiv.org/pdf/1601.06233 arXiv Article adapted for Chapter 2. https://arxiv.org/pdf/1510.01413 arXiv Article adapted for Chapter 2. https://arxiv.org/pdf/1801.06609 arXiv Article adapted for Chapter 9. http://papers.nips.cc/paper/9369-the-impact-of-regularization-on-high-dimensional-logistic-regression Publisher Article adapted for Chapter 2. https://doi.org/10.1109/ITW.2016.7606820 DOI Article adapted for Chapter 3. https://doi.org/10.1109/ICASSP.2019.8683890 DOI Article adapted for Chapter 7. http://papers.nips.cc/paper/9404-universality-in-learning-from-linear-measurements Publisher Article adapted for Chapter 6. https://doi.org/10.1109/ISIT.2019.8849405 DOI Article adapted for Chapter 5. ORCID: Author ORCID Abbasi, Ehsan 0000-0002-0185-7933 Default Usage Policy: No commercial reproduction, distribution, display or performance rights in this work are provided. ID Code: 13804 Collection: CaltechTHESIS Deposited By: Ehsan Abbasi Deposited On: 09 Jun 2020 22:15 Last Modified: 19 Jun 2020 18:44 Thesis Files Preview PDF - Final Version See Usage Policy. 2MB Repository Staff Only: item control page

Universality Laws and Performance Analysis of the Generalized Linear Models Thesis by Ehsan Abbasi In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy CALIFORNIA INSTITUTE OF TECHNOLOGY Pasadena, California 2020 Defended May 18, 2020 ii © 2020 Ehsan Abbasi ORCID: 0000-0002-0185-7933 All rights reserved except where otherwise noted. iii ACKNOWLEDGEMENTS It feels good to have an end to this chapter of my life, but it is the journey that matters in the end. Fortunately, this journey has been incredible for me since its beginning. Of course as life happens, there has been tons of ups and downs, there has been moments on the mountaintops and moments in deeps valleys of despair. But as I look back to my years at Caltech, all I remember is the incredible moments I had among colleagues and friends, whom I consider my second family now. First of all, I would like to express my deepest gratitude to my advisor, Prof. Babak Hassibi. He is the main reason behind my spectacular stay at Caltech, by creating a trusting, friendly, and unstressful workplace. In fact, he was the best advisor, mentor, teacher, researcher and role model I could ever think of. His unique perspective taught me how to also see the big picture in every problem, how to think creatively and ask fundamental questions, how to present my ideas and results, how to work with others and more importantly, how to do independent research. His words have always been encouraging, his trust made me confident in my abilities and his patience and kindness has always been inspiring. After years of working with him, his brightness, even when encountering a problem for the first time, never seizes to impress me. Academic aspects aside, his unique ways of operating his group promotes a heartwarming sense of trust, togetherness, and friendship among the members. Not only is he the person I look up to in my academic career, he is also the person who has helped me a great deal on a personal level, with invaluable advice, discussions, and debates on a wide variety of topics such as ethics, culture, books, movies, and philosophies. He is in every aspect a great teacher and advisor, and I feel really grateful and fortunate to be one of his students. I would also like to thank the members of my defense and candidacy committees, Professor Joel Tropp, Professor P. P. Vaidyanathan, Professor Venkat Chandrasekaran, Professor Thomas Vidik, and Professor Shuki Bruck, for their time, interest, and useful inputs. Their knowledge and expertise have been instrumental to my study at Caltech. One of the greatest things at Caltech was the opportunity to work with some of the smartest people in the world. My deepest appreciation goes to my collaborators, Christos Thrampoulidis and Fariborz Salehi, for their great help as well as their friendship. I truly enjoyed working with my former and current labmates who made my experience at Caltech a sheer pleasure. Thank you Wael, Kishore, Ramya, Matt, iv Wei, Anatoly, Hikmet, Sahin, Ahmed, Navid, Taylan, Oren, and Gautam, for maintaining a friendly environment in Moore 155. I would like to specifically thank my first collaborator, my mentor and my friend, Christos Thrampoulidis. I was fortunate and lucky to work under his supervision during my first years at Caltech, that taught me a great deal of my knowledge as well as my academic perspective. Thank you Christos, for being so patient with me, for the great help and true friendship, as well as the invaluable lessons I learned from you through countless technical discussions we had. No matter how far you are, these thoughts will always rekindle a great deal of amazing memories we shared in old town Pasadena, our trips to conferences, our workouts in Caltech’s gym and all the stories we shared with each other about our dramas! I will be forever in your debt. I am also greatly thankful to Fariborz, with whom I shared many sleepless deadline nights. Thank you, Fariborz, for keeping me motivated to write down my proofs, for our amazing trips to Illinois, Michigan, our recurrent drives to San Diego, for our amazing academic and philosophical discussions, for being understanding and patient with me in my ups and downs, and for being a great friend as well as a smart perfectionist collaborator. During my years at Caltech, I was not able to visit home, and unfortunately, due to the travel ban, my family could not visit me here either. So my journey wouldn’t have been possible without the countless help and company of many friends I have in Caltech and Los Angeles. One of life’s greatest joys is to meet someone who understands you in every way. In that aspect, I was lucky to meet my wife Hanieh during my PhD. Thank you, Hanieh, for being so patient with me during my long working night, for being so supportive of my works, for being my partner in crime and best friend and for being down for every crazy idea I come up with. Especially, the last few month in quarantine would have been intolerable without you being by my side. Thanks to you, I cannot image my life being any better. Among my friends whom helped me the most, I would like to especially thank my closest friend Pooya Vahidi, who became the brother I never had. Thank you Pooya for making my experience at Caltech so amazing, for the amazing surprise trips we planed (or at least wanted to plan), parties we attended together, dramas we went through together, and for making every day of my stay at Caltech full of unforgettable memories. I was greatly blessed to have many other amazing friends in southern California. I am going to miss Parham and our intellectual conversations, Aryan and our shared interest in tech gadgets, Peyman and his funny jokes and pranks, Pourya and his obsession with black holes, Omid and our etical conversations, Shaghayegh’s company in every plan we made, Homa’s funny moments, Ehsan Emamjome and our v hangouts in LA, Nazanin and Cameron, a couple I have always enjoyed hanging out with, Roohy and our endless nights, Hossein and his amazing plans, and all others, for their friendship and support during my graduate studies. Last but not least, I would like to thank my parents, Abolfazl Abbasi and Parvin Naghash, as well as my little sister Mahsan Abbasi, for their unconditional love and unceasing support throughout my entire life. One of my greatest lucks was being raised in such an understanding family. Their continuous encouragement and prayer is what has motivated me in my life, and I am forever indebted to them for their sacrifices. You are the best family I could ever dream of. I’m especially grateful that my lovely sister has joined me here in US. Because of her, I feel more complete, as a big part of me is now closer to me. In the end, I would like to take this remarkable moment to say to my my parents: "I love you, Dad and Mom. Thank you for everything you have done for me! To my beloved parents, Parvin, and Abolfazl. vii ABSTRACT In the past couple of decades, non-smooth convex optimization has emerged as a powerful tool for the recovery of structured signals (sparse, low rank, etc.) from noisy linear or non-linear measurements in a variety of applications in genomics, signal processing, wireless communications, machine learning, etc.. Taking advantage of the particular structure of the unknown signal of interest is critical since in most of these applications, the dimension 𝑝 of the signal to be estimated is comparable, or even larger than the number of observations 𝑛. With the advent of Compressive Sensing there has been a very large number of theoretical results that study the estimation performance of non-smooth convex optimization in such a high-dimensional setting. A popular approach for estimating an unknown signal 𝛽0 ∈ R 𝑝 in a generalized linear model, with observations y = 𝑔(X𝛽0 ) ∈ R𝑛 , is via solving the estimator 𝛽ˆ = arg min 𝛽 L (y, X𝛽) + 𝜆 𝑓 (𝛽). Here, L (·, ·) is a loss function which is convex with respect to its second argument, and 𝑓 (·) is a regularizer that enforces the structure of the unknown 𝛽0 . We first analyze the generalization error performance of this estimator, for the case where the entries of X are drawn independently from real standard Gaussian distribution. The precise nature of our analysis permits an accurate performance comparison between different instances of these estimators, and allows to optimally tune the hyperparameters based on the model parameters. We apply our result to some of the most popular cases of generalized linear models, such as M-estimators in linear regression, logistic regression and generalized margin maximizers in binary classification problems, and Poisson regression in count data models. The key ingredient of our proof is the Convex Gaussian Min-max Theorem (CGMT), which is a tight version of the Gaussian comparison inequality proved by Gordon in 1988. Unfortunately, having real iid entries in the features matrix X is crucial in this theorem, and it cannot be naturally extended to other cases. But for some special cases, we prove some universality properties and indirectly extend these results to more general designs of the features matrix X, where the entries are not necessarily real, independent, or identically distributed. This extension, enables us to analyze problems that CGMT was incapable of, such as models with quadratic measurements, phase-lift in phase retrieval, and data recovery in massive MIMO, and help us settle a few long standing open problems in these areas. viii TABLE OF CONTENTS Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii List of Illustrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvi Chapter I: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Generalized Linear Models . . . . . . . . . . . . . . . . . . . . . . 2 1.2 The Link Function in the Generalized Linear Models . . . . . . . . . 3 1.3 High Dimensional Regime . . . . . . . . . . . . . . . . . . . . . . . 5 1.4 Convex Recovery Method and Performance Measure . . . . . . . . . 6 1.5 Design of the Features Matrix, X . . . . . . . . . . . . . . . . . . . 7 1.6 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . . 9 Chapter II: Precise Performance Analysis of Generalized Linear Models . . . 11 2.1 Linear Models and M-Estimators . . . . . . . . . . . . . . . . . . . 13 2.2 BER Analysis of the Box Relaxation for BPSK Signal Recovery . . . 35 2.3 Binary Classification . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.4 Generalized Margin Maximizers . . . . . . . . . . . . . . . . . . . . 51 2.5 Highlight of the Proof and CGMT Framework . . . . . . . . . . . . 66 2.6 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Chapter III: General Performance Metrics for the LASSO . . . . . . . . . . . 75 3.1 Problem Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 3.3 Proof Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Chapter IV: Sparse Covariance Estimation from Quadratic Measurements: A Precise Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 4.1 Problem Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 4.2 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 4.3 Proof Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Chapter V: Scalable covariance estimation in graphical models with provable guarantees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.2 The Algorithm: Computational & Statistical Guarantees . . . . . . . 101 5.3 Discussion and Numerical Experiments . . . . . . . . . . . . . . . . 105 5.4 Proof of the Main Results . . . . . . . . . . . . . . . . . . . . . . . 109 Chapter VI: Universality in Learning from Linear Measurements . . . . . . . 118 6.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 6.2 Main Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 6.3 Applications: Quadratic Measurements . . . . . . . . . . . . . . . . 127 6.4 Simultaneously Sparse and Low-rank Matrices . . . . . . . . . . . . 132 ix Chapter VII: Performance Analysis of Convex Data Detection In MIMO . . . 145 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 7.2 Problem Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 7.3 Main Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 7.4 Proof Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Chapter VIII: Achieving near Maximum-Likelihood Performance in Massive MIMO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 8.2 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . 158 Chapter IX: A Precise Analysis of PhaseMax in Phase Retrieval . . . . . . . 167 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 9.2 Problem Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 9.3 Main Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 9.4 Proof Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Chapter X: Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . 178 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 .1 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . 194 .2 Proofs for Section .1 . . . . . . . . . . . . . . . . . . . . . . . . . . 201 .3 Proof of Auxiliary Lemmas . . . . . . . . . . . . . . . . . . . . . . 217 x LIST OF ILLUSTRATIONS Number Page 2.1 Squared error of the 𝑙1 -Regularized LAD with Gaussian (◦) and Bernoulli () measurements as a function of the regularizer parameter 𝜆 for two different values of the normalized number of measurements, namely 𝛿 = 0.7 and 𝛿 = 1.2. Also, √ iid iid 𝛽0,𝑖 ∼ 𝑝 𝑥 (𝑥) = 0.9𝛿0 (𝑥) + 0.1𝜙(𝑥)/ 0.1 and z 𝑗 ∼ 𝑝 𝑧 (𝑧) = 0.7𝛿0 (𝑧) + 0.3𝜙(𝑧) for 𝜙(𝑥) = √1 𝑒 −𝑥 /2 . For the simulations, we used 𝑝 = 768 and the data were 2 2𝜋 averaged over 5 independent realizations. . . . . . . . . . . . . . . . . . . 33 2.2 Comparing the squared error of the ℓ1 -Regularized LAD with the corresponding error of the LASSO. Both are plotted as functions of the regularizer parameter 𝜆, for two different values of the normalized measurements, namely 𝛿 = 0.7 iid and 𝛿 = 1.2. The noise and signal are iid sparse-Gaussian as follows: 𝛽0,𝑖 ∼ √ 𝑝 𝑥 (𝑥) = 0.9𝛿0 (𝑥) + 0.1𝜙(𝑥)/ 0.1 and z 𝑗 ∼ 𝑝 𝑧 (𝑧) = 0.9𝛿0 (𝑧) + 0.1𝜙(𝑧) with 𝜙(𝑥) = √1 𝑒 −𝑥 /2 . For the simulations, we used 𝑝 = 768 and the data were 2 2𝜋 averaged over 5 independent realizations. . . . . . . . . . . . . . . . . . . 33 2.3 Squared error of the ℓ1 -Regularized M-Estimator with Huber-loss as a function of √ iid the regularizer parameter 𝜆. Here, 𝛿 = 0.7, 𝛽0 ∼ 𝑝 𝑥 (𝑥) = 0.9𝛿0 (𝑥) +0.1𝜙(𝑥)/ 0.1 1 and 𝑝 𝑧 (𝑧) = 0.9𝛿(𝑧) + 0.1𝜂(𝑧) with 𝜙(𝑥) = √1 𝑒 −𝑥 /2 and 𝜂(𝑧) = 𝜋 (1+𝑧 2 ) . For 2 2𝜋 the simulations, we used 𝑝 = 1024 and the data are averaged over 5 independent realizations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.4 Squared error of the ℓ1,2 -Regularized Lasso for group sparse signal composed of 512 blocks of size 3 each, as a function of the regularizer parameter 𝜆. Here, 𝛿 = 0.75, each block is zero with probability 0.95, otherwise its entries are i.i.d. N (0, 1) and z 𝑗 ∼ 𝑝 𝑧 (𝑧) = 0.3𝜙(𝑧) with 𝜙(𝑥) = √1 𝑒 −𝑥 /2 . The simulations are 2 iid 2𝜋 averaged over 10 independent realizations. . . . . . . . . . . . . . . . . . . 34 2.5 BER Performance of the Boxed Relaxation: 𝑃𝑒 as a function of SNR for different values of the ration 𝛿 = d𝑚/𝑛e. The theoretical prediction follows from Theorem 3. For the simulations, we used 𝑛 = 512. The data are averages over 20 independent realizations of the channel matrix and of the noise vector for each value of the SNR. 38 2.6 Bit error probability of the Box Relaxation Optimization (BRO) in (2.56) in comparison to the Matched Filter Bound (MFB) for 𝛿 = 0.7 (dashed lines) and 𝛿 = 1 (solid lines). The red curves follow the formula of Thm. 3, the green ones correspond to (2.59), and, 𝑃𝑒𝑀 𝐹 𝐵 of (2.60) is in blue. . . . . . . . . . . . . . . . . . . . 39 xi 2.7 The performance of the regularized logistic regression under ℓ22 penalty (a) the correlation factor 𝛼¯ (b) the variance 𝜎 ¯ 2 , and (c) the mean-squared 2 error 1 || 𝛽ˆ − 𝛽∗ || . The dashed lines depict the theoretical result derived 𝑝 from Theorem 5, and the dots are the result of empirical simulations. The empirical results is the average over 100 independent trials with 𝑝 = 250 and 𝜅 = 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 2.8 The performance of the regularized logistic regression under ℓ1 penalty (a) the correlation factor 𝛼¯ (b) the variance 𝜎 ¯ 2 , and (c) the mean-squared 2 error 1 || 𝛽ˆ − 𝛽∗ || . The dashed lines are the theoretical result derived from 𝑝 Theorem 4, and the dots are the result of empirical simulations. For the numerical simulations, the result is the average over 100 independent trials with 𝑝 = 250 and 𝜅 = 1 . . . . . . . . . . . . . . . . . . . . . . . . . . 49 2.9 The support recovery in the regularized logistic regression with ℓ1 penalty for (a) 𝐸 1 : the probability of false detection, (b) 𝐸 2 : the probability of missing an entry of the support. The dashed lines are the theoretical results derived from Lemma 5, and the dots are the result of empirical simulations. For the numerical simulations, the result is the average over 100 independent trials with 𝑝 = 250 and 𝜅 = 1 and 𝜖 = 0.001 . . . . . . . . . . . . . . . . 51 2.10 2.11 The phase transition, 𝛿∗ , for the separability of the dataset, where the feature vector, x𝑖 is drawn from the Gaussian distribution, N (0, 1𝑝 I 𝑝 ),
𝑒𝑡 and the labels are 𝑦𝑖 ∼ Rad 𝜌(x𝑇𝑖 w★) , for 𝜌(𝑧) = 𝑒𝑡 +𝑒 The −𝑡 . empirical result is the average over 20 trials with 𝑝 = 150, and the theoretical results are from Theorem 6. . . . . . . . . . . . . . . . . 56 Generalization error of the general max margin classifier under three penalty functions, ℓ1 norm with the red line (ℓ1 -GMM), ℓ2 norm with the blue line (ℓ2 -GMM), and ℓ∞ norm with the black line (ℓ∞ GMM). In this figure, the entries of w★ are drawn independently from N (0, 𝜅 2 ) Gaussian distribution. Solid lines correspond to the theoretical results derived from Theorem 7, while the circles are the result of empirical simulations. For the numerical simulations, the result is the average over 100 independent trials with 𝑝 = 200 and 𝜅 = 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 xii 2.12 Generalization error of the general max margin classifier under three penalty functions, ℓ1 norm with the red line (ℓ1 -GMM), ℓ2 norm with the blue line (ℓ2 -GMM), and ℓ∞ norm with the black line (ℓ∞ GMM). In this figure, the underlying vector w★ is 𝑠-sparse, where the non-zero entries are drawn independently from N (0, 𝜅 2 /𝑠) Gaussian distribution. Solid lines correspond to the theoretical results derived from Theorem 7, and the circles are the result of empirical simulations. For the numerical simulations, the result is computed by taking the average over 100 independent trials with 𝑝 = 200, 𝑠 = .1 and 𝜅 = 2. . . . . . . . . . . . . . . . . . . . . . . . 64 2.13 Generalization error of the general max margin classifier under three penalty functions, ℓ1 norm with the red line (ℓ1 -GMM), ℓ2 norm with the blue line (ℓ2 -GMM), and ℓ∞ norm with the black line (ℓ∞ -GMM). In this figure, the entries of w★ are drawn independently from 𝜅 ∗ Rad(0.5) Rademacher distribution. Solid lines correspond to the theoretical results derived from Theorem 7, and the circles are the result of empirical simulations. For the numerical simulations, the result is the average over 100 independend trials with 𝑝 = 200 and 𝜅 = 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.1 Performance of Square root Lasso with respect to Ψ(x) = √1𝑝 kxk2 (Red Line) and Ψ(x) = 𝑝1 kxk1 (Blue line) as a function of 𝜆. The theoretical prediction comes from Theorem 9. For the simulations, we used 𝑝 = 256, 𝛿 = 0.8, 𝜌 = 0.1, SNR=0.5 and the data are averaged over 5 independent realization of the Problem. . . . . . 80 3.2 Probability of successful detection of support and off support entries as a function of 𝜆 for two different problem setup. The theoretical prediction (Solid and dashed lines) comes from Theorem10. For the simulations (Squares and Circles), we used 𝑝 = 256, SNR= 0.5, 𝜖 = 10−3 , 𝜌 = 0.1 and the data are averaged over 5 independent realizations of problem. For solid lines and squares and circles, we used 𝛿 = 0.8, while for dashed lines and empty squares and circles 𝛿 = 1.2. . . . . 4.1 82 Performance of the optimization (4.2) with respect to 𝜙(W) = kWk𝐹 /𝑝, as a function of 𝜆. Circles represent numerical simulations, and solid lines are theoretical predictions from Theorem 12. For simulations, we used 𝑝 = 120, 𝛿 = .8, E[𝑧𝑖2 ] = 1, and three choices of sparsity factor; 𝜅 = .05 in red, 𝜅 = .1 in blue, and 𝜅 = .2 in black. The results are averaged over 80 random realizations of data. For 𝜆 > 𝜆+ , the output of (4.2) will be positive definite. . . . . . . . . . . . . . . . . 91 xiii 4.2 Phse transition regimes for the optimization (4.2), in terms of the oversampling ratio 𝛿 = 𝑝( 2𝑛 𝑝+1) , and the sparsity factor 𝜅. Solid line comes from (4.8). For the empirical results, we used 𝑝 = 40. The results are averaged over 20 independent realization of measurement vectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 5.1 Plots of the probability of signed support recovery of the precision matrix corresponding to a chain graph, and for different values of 𝑝 as a function of (a) the number of observations 𝑛, and, (b) of the scaled sample size 𝑛/log 𝑝. The different curves pile up in (b) as predicted by Corollary 5. Each simulation point corresponds to an average over 𝑁 = 40 points. . . . . 110 5.2 Plots of the probability of signed support recovery of the precision matrix corresponding to a star graph with parameter 𝑑 = 𝑝/10, and for two different values of 𝑝 as a function of (a) the number of observations 𝑛, and, (b) of the scaled sample size 𝑛/log 𝑝/𝑑. The different curves pile up in (b) as predicted by Corollary 5. Each simulation point corresponds to an average over 𝑁 = 40 points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 6.1 Phase transition regimes for the estimator E{x0 , A, R 𝑝 , k · kℓ1 }, in terms of the oversampling ratio 𝛿 = 𝑛𝑝 and 𝑠 = kx𝑝0 k0 , for the cases of (a) Gaussian measurements and (b) Bernoulli measurements and (c) 𝜒2 measurements. The blue lines indicate the theoretical estimate for the phase transition derived from Corollary 7. In the simulations we used vectors of size 𝑝 = 256. The data is averaged over 10 independent realization of the measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 6.2 Phase transition regimes for both estimators 6.7 and (6.8), with 𝑓 (X) = Tr(X), in terms of the oversampling ratio 𝛿 = 𝑛𝑝 and 𝑟 = Rank(X0 ), for the cases of (a) estimator (6.7) with quadratic measurements and (b) estimator (6.8) with Gaussian measurements. In the simulations we used matrices of size 𝑝 = 40. The data is averaged over 20 independent realization of the measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 6.3 Phase transition regimes for both estimators (6.7) and (6.8), with 𝑓 (X) = kXkℓ1 , in terms of the oversampling ratio 𝛿 = 𝑝𝑛2 and 𝑠 = kX𝑝02k0 , for the cases of (a) estimator (6.7) with quadratic measurements and (b) estimator (6.8) with Gaussian measurements. The blue lines indicate the theoretical estimate for the phase transition derived from equation (6.9). In the simulations we used matrices of size 𝑝 = 40. The data is averaged over 20 independent realization of the measurements. . . . . . . . . . . . . . . . 132 xiv 7.1 SER Performance of the Circular Relaxation (CR) for 16-PSK: 𝑃𝑒 as a function of SNR for the two cases where 𝛿 = .8 and 𝛿 = 1. The theoretical prediction follows from Theorem 18 and the high-SNR analysis comes from Section 7.3. For the simulation, we used signals of size 𝑝 = 128 randomly uniform from the n 𝑗 𝜋with each entry chosen o set S𝑃𝑆𝐾 = 𝑒 8 𝑖 : 𝑖 = 0, . . . , 15 . The data are averages over 30 independent realizations of the channel matrix and the noise vector. . 150 7.2 SER Performance of the Box Relaxation for 16-QAM: 𝑃𝑒 as a function of SNR for the two cases where 𝛿 = .8 and 𝛿 = 1. The theoretical prediction follows from Theorem 18 and the high-SNR analysis comes from Section 7.3. For the simulation, we used signals of size 𝑝 = 128 with each entry chosen uniformly at random in the set S𝑄 𝐴𝑀 = {±1, ±3}2 . The data are averages over 30 independent realizations of the channel matrix and the noise vector. . . . . . . . . 152 8.1 The proposed two-pronged algorithm. . . . . . . . . . . . . . . . . . 159 8.2 SER Performance of the convex relaxation (Blue line), After doing the local search (Green line) and the Matched Filter Bound (Red line) for 8-PSK: SER as a function of SNR for the two cases. For the simulation, we used signals of size n = 128 with each entry chosen randomly uniform from 8PSK constellation. The data are averages over 100 independent realizations of the channel matrix and the noise vector. The left figure corresponds to 𝛿 = .9 and for the right figure 𝛿 = 1.1 . . . . . . . . . . . . . . . . . . . 160 8.3 SER performance of our two-step algorithm with respect to SNR, for three choises of convex relaxation; Zero-Forcing (Red lines), MMSE (Black lines) and Convex Hull relaxation (Blue lines). The green curve corresponds to the performance of the Matched Filter Bound. Solid lines represent the performance of the first steps only, and the dashed lines are the final performance of the two-step algorithm. For our simulations we used 𝑝 = 128 with 𝑛𝑝 = 1.1. The results is averaged over 200 random independent realization of the channel matrix and noise vector. . . . . . . . . . . . . . . . . . . . . . . . . 166 xv 9.1 Phase transition regimes for the PhaseMax problem in terms of the oversampling ratio 𝛿 = 𝑚/𝑛 and 𝜃, the angle between x0 and xinit . For the empirical results, we used signals of size 𝑛 = 128. The data is averaged over 10 independent realization of the measurement vectors. The blue line indicates the sharp phase transition bounds derived in Theorem 20 and the red line comes from the results of [78], which is referred to as the GS Bound. . . . . . . . . . . . . . . . . . . . . . . 172 xvi LIST OF TABLES Number Page 6.1 Summary of the parameters that are discussed in this section. The last row is for a 𝑝 × 𝑝 rank-𝑟 matrix whose smallest sub-matrix with non-zero entries is 𝑘 by 𝑘. The third column shows the number of required quadratic measurements for perfect recovery. . . . . . . . . 132 1 Chapter 1 INTRODUCTION Data in today’s technology and industry work is indispensable. Most organizations now understand that if they gather all the data that is available to them, they can analyze and get significant value from it. As a result, the last decade has seen a sustained exponential growth rate in data stored and used. This has led to an immense interest in the buzz words such as "Big Data" and "Statistical Inference", where the question is how to efficiently deduce the most information about the unknown variables of interest. Classical estimation theory has extensively investigated this question under various models, when the number of unknown variables is small compared to collected data. But in many modern applications (e.g. financial data, machine learning, wireless communications, sensor networks, genome signal processing, image processing, DNA sequencing, etc.), the number of unknown variable of interest has become larger and larger. Therefore, a lot of classical tools in estimation theory fail to address the same questions, with the dimensionality explosion that we experience in today’s applications. More importantly, in many of these applications, the number of unknown variables is even larger than the number of observations (or measurements) we have (e.g. consider the DNA sequencing where the unknown data is the human genome, or in image processing where the unknown is a large scale image, or in finance). Consider the following simple but fundamental example to make our idea concrete. We would like to recover an unknown vector 𝛽 ∈ R 𝑝 from the system of linear equations below, with 𝑛 equations, y = X𝛽 ∈ R𝑛 , (1.1) where y ∈ R𝑛 and X ∈ R𝑛×𝑝 are given. Obviously, when we do not have any other information about the unknown 𝛽, it is necessary to have more measurements than unknowns for a consistent recovery (𝑛 should be greater than or equal to 𝑝). But in many applications, the unknown 𝛽 is constrained by some structure (e.g. sparsity, where only a limited unknown number entries of 𝛽 are non-zero, or the case where the entries of 𝛽 are chosen from a finite alphabet like ±1.) In these examples, although the unknown data is 𝑝-dimensional with 𝑝 ≥ 𝑛, it lies on a lower dimensional manifold with a lower degree of freedom, which may make recovery of the 2 unknown data feasible. These examples, give rise to problems like, how to exploit the given structure to efficiently recover the unknown?, or under what conditions are our estimators consistent? Among different estimation methods, convex estimators are popular as they exhibit numerical stability, and more flexibility. Besides, they are more tractable when it comes to computational analysis. In the past couple of decades, non-smooth convex optimization has emerged as a powerful tool for the recovery of structured signals (sparse, low rank, etc.) from generalized linear measurements in a variety of applications in genomics, signal processing, wireless communications, machine learning, etc.. How to take advantage of the particular structure of the unknown signal of interest is critical since as explained, in most of these applications, the dimension 𝑝 of the signal to be estimated is comparable, or even larger than the number of observations 𝑛 ([40] and references therein). With the advent of Compressive Sensing there has been a very large number of theoretical results that study the estimation performance of non-smooth convex optimization in such a high-dimensional setting. 1.1 Generalized Linear Models Linear models describe a continuous output variable as a function of the predictors, and are widely used in statistical data analysis. But in practice, the underlying models can be much more complicated than a simple linear model. In this thesis, we focus on a special class of models, known as the generalized linear models. In this section, we define the set up for these models, mention some of their applications and state of the art, and finally explain the thesis organization. Mathematical Formulation Consider the problem of recovering a 𝑝-dimensional signal 𝛽0 ∈ R 𝑝 , from 𝑛 measurements of the form 𝑦𝑖 = 𝑔𝑖 (x𝑖T 𝛽0 ), 𝑖 = 1, . . . , 𝑛 . (1.2) Here 𝑔𝑖 (.) is a known or unknown link function, and may include a random component such as noise. For instance, in the case of linear measurements we may have, 𝑦𝑖 = x𝑖T 𝛽0 + 𝑧𝑖 , (1.3) where 𝑧𝑖 ’s are the unknown noise entries. Our goal is to recover the vector 𝛽0 , given the measurements 𝑦𝑖 ’s, the feature vectors x𝑖 ’s and depending on the problem, some 3 information about the link function 𝑔𝑖 (.) or the structure of the unknown vector 𝛽0 . Henceforth, let 𝑦   1  .  y =  ..  ∈ R𝑛 ,   𝑦𝑚     xT   1  .  X =  ..  ∈ R𝑛×𝑝 ,  T x 𝑚     𝑔 (𝑥 )   1 1   .  𝑔(x) =  ..  ∈ R𝑛 .   𝑔𝑚 (𝑥 𝑚 )    (1.4) Here, y denotes the vector of 𝑛 measurements, X is the features matrix, and 𝑔 : R𝑛 → R𝑛 is the link function for which y = 𝑔(X𝛽0 ) (1.5) Our model so far is very general as we did not impose any assumptions on any of the parameters. Thus, to answer "how to recover the unknown vector?", we have to make the model more clear, define the tools we would like to work with, and our performance measures. 1.2 The Link Function in the Generalized Linear Models The link function 𝑔(·), plays a major role in how we approach this problem. We will explain a few important and popular cases of the link function. Linear Inverse Problems in Compressed Sensing The idea in compressed sensing is to recover a signal with low-dimensional structures from high dimensional measurements. The structured signal can be a sparse vector [38], a low rank matrix [135], a vector chosen from a finite alphabet[166], etc. There has been tremendous research under the name of compressed sensing in the last decades [6, 40, 42, 61, 73, 129, 165, 179]. The classical setting of linear inverse problems considers recovering an unknown 𝛽0 ∈ R 𝑝 , given linear noisey measurements of the form y = X𝛽0 + z ∈ R𝑛 , (1.6) where the measurements y ∈ R𝑛 and the features matrix X ∈ R𝑛×𝑝 are given. Convex estimators are a popular method of recovering the unknown in these problems. Especially, for the case of structured 𝛽0 , there are principled ways to recover the unknown using a convex estimator, based on the idea of atomic decomposition [40] and also other methods in [10, 15], as well as non-convex methods such as [122]. 4 Classification Problems Logistic regression is the most commonly used statistical model for predicting dichotomous outcomes [85]. It has been extensively employed in many areas of engineering and applied sciences, such as in the medical [27, 181] and social sciences [96]. As an example, in medical studies logistic regression can be used to predict the risk of developing a certain disease (e.g. diabetes) based on a set of observed characteristics from the patient (age, gender, weight, etc.) Linear regression is a very useful tool for predicting a quantitive response. However, in many situations the response variable is qualitative (or categorical) and linear regression is no longer appropriate [89]. This is mainly due to the fact that least-squares often succeeds under the assumption that the error components are independent with normal distribution. In categorical predictions, however, the error components are neither independent nor normally distributed [124]. In logistic regression we model the probability that the label, 𝑌 , belongs to a certain category. When no prior knowledge is available regarding the structure of the parameters, maximum likelihood is often used for fitting the model. Maximum likelihood estimation (MLE) is a special case of maximum a posteriori estimation (MAP) that assumes a uniform prior distribution on the parameters. In this problem, we assume the the measurements are given by the following model, 𝑒𝑥 , 𝜖𝑖 ∼ Bernouli(𝑃) . 𝑒 𝑥 + 𝑒 −𝑥 In this case, each measurement, 𝑦𝑖 , will be +1 or −1 with probability 𝜌(𝑥𝑖 ) or 1 − 𝜌(𝑥𝑖 ), respectively. y = 𝑔(x) = Sign(𝜌(x) − 𝜖) , 𝜌(𝑥) = Phase Retrieval Problem The fundamental problem of recovering a signal from magnitude-only measurements is known as phase retrieval. This problem has a rich history and occurs in many areas in engineering and applied physics such as astronomical imaging [69], X-ray crystallography [116], medical imaging [55], and optics [191]. In most of these cases, measuring the phase is either expensive or even infeasible. For instance, in some optical settings, detection devices like CCD cameras and photosensitive films cannot measure the phase of a light wave and instead measure the photon flux. The goal is to recover an unknown data 𝛽0 from the magnitude only measurements of the form, y = |X𝛽0 | ∈ C𝑛 , where y and X are given and | · | is the element-wise absolute value operator. (1.7) 5 Poisson Regression in Count Data Models Poisson regression assumes that the observations 𝑦𝑖 take a Poisson distribution with mean xT 𝛽0 ,   𝑦𝑖 = Pois x𝑖T 𝛽0 , 𝑖 = 1, . . . , 𝑛 . (1.8) Poisson distribution has applications in many areas such as telecommunications (number of arriving calls in a system), Biology (number of mutations on a DNA), Finance and insurance (number of losses or claims in a period of time), etc.. 1.3 High Dimensional Regime In the classical regime, the common modeling assumption was that the number of unknown variables, 𝑝, is fixed, while the number of measurements, 𝑛, grow large. This problem is well studied for the cases of linear regression, classification problems and some other cases. But with the rise of big data, the number of unknowns in modern applications of statistical inference could grow as large as the number of observations. This new assumption, requires a difference performance analysis for the previous problems. There has been a lot of effort in the last decade, to answer the same questions in the classical regime, under this new assumption, which we will mention in Section 2.1, for different applications. Throughout this thesis, we are especially interested in the over-parameterized regime, where the number of measurements, 𝑛, is less than the number of unknowns, 𝑝. This problem is usually ill-posed unless some prior information about the structure of the unknown data in given. For instance, sometimes the true unknown data, lies on a low-dimensional manifold in its original 𝑝-dimensional space. One of the most famous structures is data sparsity. In this case, most of the entries of the unknown data is zero, but of course the indices of the zero entries are unknown. Other popular structures includes, signals that are block sparse [152, 160], signals with entries drawn from a finite alphabet [166, 174], low rank matrices [135], or sometime vectors or matrices that exhibit a few simultaneous structures [128]. Recently there has been a unifying framework that can extend the analysis techniques that were initially developed for the sparse signal recovery, to other structures [18, 56, 63, 165]. Formally, most of the results of this Thesis will be applied to a sequence of problem instance {𝛽0 , X, 𝑔(·), L (·, ·), 𝑓 (·)} 𝑝∈N , indexed by 𝑝, such that the properties we mentioned and our assumptions hold for all members of this sequence. We will not write the subscript 𝑛 for arguments to avoid overloading notations. Our results are asymptotic and hold as 𝑛 → ∞. 6 1.4 Convex Recovery Method and Performance Measure Among various recovery methods, we focus on convex optimization based estimations. Convex methods are often preferred as they exhibit numerical stability, and more flexibility. Besides, they are more tractable when it comes to computational or performance analysis. These convex methods estimate the unknown vector 𝛽0 , by solving the following convex optimization problem, 𝛽ˆ = arg min L (X𝛽, y) + 𝜆 𝑓 (𝛽) . 𝛽 (1.9) Here, L (X𝛽, y) is a loss function that penalizes the residual, and is convex with respect to its first argument. The function 𝑓 (·) (which we call the regularizer) is a convex function that enforces the structure of the unknown vector 𝛽0 , to the final ˆ The positive parameter 𝜆, is a regularization parameter that balances estimation 𝛽. the cost function and the regularizer. This form includes a lot of famous estimators including ℓ1 -regularized least squares (aka the LASSO), penalized least absolute devations estimator (aka LAD), ridge regression, maximum-likelihood estimators, Support vector machine or perceptron or logistic regression in classification problems, etc. There are standard solvers for each one of these examples, all of which benefit from the convex nature of this estimator. In this thesis, our focus is on the recovery performance of such estimators, rather than algorithmic issues. So our main question will be, how well the optimization (1.9) can estimate the unknown data 𝛽0 ? Performance Measures If we want to measure the recovery performance of the estimator (1.9), we need to define our performance measure first. For now, we keep our performance measure general in the form of ˆ 𝛽0 ) , 𝜓( 𝛽, (1.10) where 𝜓(·) is a function that is supposed to measure the deviance of 𝛽ˆ from 𝛽0 , in our desired way depending on our application. For instance, in the classification problems, the performance measure could be the generalization error, which is      ˆ 𝛽0 ) = Prob Sign 𝜌(xT 𝛽) ˆ − 𝜖 ≠ Sign 𝜌(xT 𝛽0 ) − 𝜖 , 𝜓( 𝛽, 𝜌(𝑡) = 𝑒𝑡 , 𝑒 𝑡 + 𝑒 −𝑡 𝜖 ∼ Unif(0, 1), x ∼ N (0, I) . (1.11) 7 We will impose some natural assumptions on the function 𝜓(·, ·) later for our analysis. 1.5 Design of the Features Matrix, X Performance of the estimator (1.9) depends on all problem parameters, including properties of the features matrix X. As the most basic example, consider the problem of recovering 𝛽0 ∈ R 𝑝 from the system of linear equations y = X𝛽0 ∈ R 𝑝 . (1.12) We need the features matrix X to be full rank for a consistent recovery of 𝛽0 . So in a worst case scenario analysis of the recovery performance for this problem, we should make sure X is full rank. But in practice, if we assume a random ensemble for the entries of X, the probability of X being singular may be very small. For example, in the case that the entries of X are independently drawn from a continuous probability distribution, the probability of X being singular world be zero. Besides, it has been observed that randomly generated features matrices can yield good estimates in the high dimensional regime [26, 68, 72]. Besides, assuming a random ensemble for the features matrix enables us to analyze the average case performance, or high probability performance in high dimensional regime. In particular, matrices sampled from Gaussian distribution has been traditionally useful in the performance analysis of estimators in compressed sensing [38]. Recently, with the development of two frameworks known as AMP [56] and CGMT [165], researchers have been able to answer most of the previously unknown questions in compressed sensing and classification. For these frameworks, the it is essential to assume that the entries of the features matrix X, are independently drawn from standard real Gaussian distribution. Although their analysis answered a lot of open questions in compressed sensing and classification problems, the Gaussian assumption is very restrictive when it comes to practical problems. Universality As discussed, assuming a randomly drawn features matrix X with iid Gaussian entries, has several benefits. First, it enables us to utilize a wide set of tools in probability. And second, we will not have to consider the worst case scenarios in the design of the features matrix. But even more importantly, this can be a good start in the analysis of such complex problems. Once we have a clear understanding of how the estimator behaves with an iid Gaussian features matrix, we can move forward and investigate how general these results are and how can one extend them 8 to other cases. As a matter of fact, some of the results that hold under this Gaussian assumption, enjoy a remarkable universality property in high dimensions, that extend the same result to a wide variety of other distributions. Donoho and Tanner in [58] and Bayati et al, [17] were first to observe and show some universality in phase transition of an special case of estimator (1.9). Later, Oymak and Tropp [130] showed this universality for a wider range of problems and distributions. But in most of these cases, having real independent entries in the features matrix was essential. Although these works are of great interest, the independence assumption on the entries of the measurement vectors can be restrictive. In certain applications in communications, phase retrieval, covariance estimation, the entries of the measurement matrix have correlations. In this thesis, we show a much stronger universality result which holds for a broader class of measurement distributions. Here is some of the applications in which at least one of the key assumptions on the features matrix (real and iid entries) does not hold. Quadratic Measurements Consider the case, where we wish to recover an unknown matrix 𝚺0 , from measurements of the form, 𝑦𝑖 = a𝑖T 𝚺0 a𝑖 + 𝑧𝑖 , 𝑖 = 1, . . . , 𝑛 , (1.13) where the features vectors a𝑖 ’ and measurements 𝑦𝑖 ’s are given. In this example, the measurements are still linear with respect to the unknown matrix 𝚺0 , but quadratic with respect to the features vectors a𝑖 ’s. We define 𝛽0 = 𝚺®0 and x𝑖 = a𝑖®a𝑖T , where ®· is the vectorized version of a matrix. Then we have 𝑦𝑖 = x𝑖T 𝛽0 + 𝑧𝑖 , 𝑖 = 1, . . . , 𝑛 , (1.14) Obviously, even by imposing a generic iid distribution on the entries of a𝑖 ’s, the entries of x𝑖 will be highly dependent. Therefore, one cannot simply apply the classical result on this new problem. This problem shows up in many applications such as Covariance sketching for data streams [44, 120], non-coherent energy measurements in communications [180], phase retrieval problem, [36, 86, 148, 190] etc. 9 Data Recovery in Massive MIMO Here, the goal is to recover a 𝑝-dimensional vector 𝛽0 ∈ C 𝑝 where the entries of 𝛽0 are independently drawn from the discrete set S ⊂ C with distribution 𝛽0,𝑖 ∼ 𝑝 𝛽 . The set S defines the modulation used for data transmission (e.g. QAM, PSK, etc.). For this purpose, we are given the noisy multiple-input multiple-output (MIMO) relation of the form y = X𝛽0 + z ∈ C𝑛 , (1.15) where X ∈ C𝑛×𝑝 is the known MIMO channel matrix with i.i.d. entries drawn from NC (0, 1𝑝 ) and z ∈ C𝑛 is the unknown noise vector with i.i.d. random complex Gaussian NC (0, 𝜎 2 ) entries. The important question here would be does the same performance analysis techniques in real case hold for the case of complex features matrix as well? Interestingly, we show that the same results and techniques are not necessarily applicable to the case of a complex features matrix, as we will have examples of both scenarios in future chapters. 1.6 Organization of the Thesis In this thesis, we investigate various scenarios for the generalized linear model in (1.9). In Chapter 2, we impose an iid standard Gaussian distribution on the entries of the features matrix X, and analyze the performance of the general estimator in (1.9). Then we will apply our analysis of some interesting examples of generalized linear models such as M-estimators (Linear Regression models), binary classification problem (such as logistic regression and generalized margin maximizers), and also in data recover in massive MIMO with a real channel. In Chapter 3, we apply our result to the square-root LASSO problem with a general performance function 𝜓(·). Later in Chapter 4, we investigate the problem of covariance estimation with quadratic measurements of the form (1.13). We show some universality properties that enables us to use the same methods as in Chapter 2 to analyze the performance of convex estimators in such scenario. In Chapter 5, we propose a fast algorithm for covariance estimation in graphical models with theoretical guarantees. In Chapter 6, we prove a universality result for the phase transition of the linear inverse problems with a wide range of distribution for the features matrix X. As a result, we show that the phase transition in successful recovery of the unknown data, depends only on the first and second order statistics of the rows of the features ma- 10 trix X. As an application, we show that the minimum number of random quadratic measurements (also known as rank-one projections) required to recover a low rank positive semi-definite matrix is 3𝑛𝑟, where 𝑛 is the dimension of the matrix and 𝑟 is its rank. As a consequence, we settle the long standing open question of determining the minimum number of measurements required for perfect signal recovery in phase retrieval using the celebrated PhaseLift algorithm, and show it to be 3𝑛. Chapter 7, investigates the problem of data recovery in massive MIMO, and shows that under specific conditions on the constellation (among other conditions) and after some modifications of the original problem, we can come up with an equivalent real estimation problem that can be analyzed by the tools introduced in Chapter 2. We use these results in Chapter 8, and propose a two-step algorithm for a near-maximum likelihood data recovery in massive MIMO. As Chapter 9 derives a phase transition in perfect recovery in complex phase-max problem, it concludes that universality does not always holds from problems with complex features matrices to their corresponding real problems. Finally, we gather some of the proofs of previous sections in the Appendix. 11 Chapter 2 PRECISE PERFORMANCE ANALYSIS OF GENERALIZED LINEAR MODELS In this chapter1, we first summarize the problem setup and introduce our assumptions, and then state the main theorem. We would like to recover the 𝑝 dimensional vector 𝛽 from measurements of the form y = 𝑔(X𝛽0 ) ∈ R𝑛 . (2.1) We are given the observation vector y and the features matrix X and sometimes some information about the link function g(·). We do so by solving the following convex optimization, 𝛽ˆ = arg min L (X𝛽, y) + 𝜆 𝑓 (𝛽) . 𝛽 (2.2) Here the loss function L (·, ·) is convex with respect to its first argument, and the regularizer 𝑓 (·) is also convex. In this section, our goal is to analyze performance of the optimization (2.2), in terms of ˆ 𝛽0 ) , 𝜓( 𝛽, (2.3) ˆ As already where 𝜓(·, ·) is a function that measures the distance between 𝛽0 and 𝛽. noted in the introduction, this performance measure depends on all the problem parameters including design properties of the features matrix X, distribution of the noise and the underlying vector 𝛽0 , properties of the loss function and the regularizer function, etc. In this section, we focus on the case that the features matrix X has iid standard Gaussian entries. We will generalize the model in the next chapters. The performance of this optimization, depends on the Loss function, the regularizer, distribution of the link function g(·), and distribution (or properties) of the unknown data 𝛽0 through the following Moreau envelope transformation. The Moreau envelopes of the loss function L (·, ·) and the regularizer function 𝑓 (·) are respectively defined as 1 kv − xk 2 + L (v, y) , eL (x, y, 𝜏) := min v 2𝜏 1 e 𝑓 (x, 𝜏) := min kv − xk 2 + 𝑓 (x) . (2.4) v 2𝜏 1 Some of the materials of this chapter are based on the works in [143, 165, 166] 12 Similarly, the proximal operator is defined as the minimizers of the above optimizations, ProxL (x, y, 𝜏) := arg min v Prox 𝑓 (x, 𝜏) := arg min v 1 kv − xk 2 + L (v, y) , 2𝜏 1 kv − xk 2 + 𝑓 (x) . 2𝜏 (2.5) Note that since the loss function takes two input variable, its corresponding Moreau envelope takes three input, whearas the corresponding Moreau envelope for the regularizer takes only two inputs. Assumption 1, introduces an essential functional, through which the performance of the convex optimization depends on the loss function, the regularizer, and distribution and properties of the link function and unknown data. Assumption 1 (Functionals 𝐿 and 𝐹) We say that Assumption 1 holds for the functions L and 𝑓 , and for the distributions 𝑝 g(·) and 𝑝 𝛽0 , if for all 𝑐 1 , 𝑐 2 ∈ R and 𝜏 > 0, there exist continuous functions 𝐿 : R × R × R>0 → R and 𝐹 : R × R × R>0 → R such that 𝑃 → 𝐿 (𝑐 1 , 𝑐 2 , 𝜏) eL (𝑐 1 h1 + 𝑐 2 X𝛽0 , g(X𝛽0 ) , 𝜏) − and 𝑃 e 𝑓 (𝑐 1 h2 + 𝑐 2 𝛽0 , 𝜏) − → 𝐹 (𝑐 1 , 𝑐 2 , 𝜏) . (2.6) where, the convergence is in probability over distribution of the function g(·), distribution (or properties) of the unknown data 𝛽0 , the random matrix X ∈ R𝑛×𝑝 with iid standard Gaussian entries, and the random vectors h1 ∈ R𝑛 and h2 ∈ R 𝑝 with iid standard Gaussian random entries. Assumption 1 holds naturally for a wide range of norms, loss functions, and regularizers, due to the law of large numbers. In Section 2.1, we will derive the funcionals 𝐿 and 𝐹 for some interesting examples. The second assumption we introduce is related to the performance function 𝜓(·, ·), which measures how well the optimization works. Assumption 2 (Performance Function 𝜓(·, ·)) We say that Assumption 2 holds for the function 𝜓(·, ·) and for the distributions 𝑝 𝛽0 , if for all 𝑐 1 , 𝑐 2 ∈ R and 𝜏 > 0, there exist continuous function Ψ : R × R × R>0 → R such that 𝑃 𝜓(Prox 𝑓 (𝑐 1 𝛽0 + 𝑐 2 h2 , 𝜏) , 𝛽0 ) − → Ψ(𝑐 1 , 𝑐 2 , 𝜏) . (2.7) 13 where, the convergence is in probability over the distribution (or properties) of the unknown data 𝛽0 , and the random vector h2 ∈ R 𝑝 with iid standard Gaussian random entries. Assumption 2 is valid for a wide range of function 𝜓. Simple instances include 𝑝-norms, for which the assumption holds by the law of large numbers when the entries 𝛽0 are drawn independently from the same distribution. Out main theorem, analyzes the convex estimator (2.2), in its most general form. The proof of Theorem 1 is deferred to Section 2.6, as well as explanation about the CGMT framework, which is the main tool that we use to prove this theorem. Theorem 1 Let 𝛽ˆ be the solution the convex estimator (2.2), which is an estimation of the unknown data 𝛽0 , where the entries of the features matrix X ∈ R𝑛×𝑝 are drawn independently from standard Gaussian distribution. Also Assumptions 1 and 2 hold √ with functionals 𝐿, 𝐹 and Ψ, 𝛿 = 𝑛/𝑝 and 𝜅 = k 𝛽0 k/ 𝑝. Consider the following min-max optimization over 6 scalars (𝛼, 𝜎, 𝜏1 , 𝜏2 , 𝑡, 𝛾), min 𝛼∈R 𝜎,𝜏1 ≥0 1 𝜎 𝜎𝜏2 𝑡 2 𝜎𝜏2 𝛾 2 𝜅 2 𝜎𝜏2 𝑡 𝑡 + + 𝐿 (𝜎, 𝛼, − − ) + 𝐹 (− √ , 𝛼 − 𝜎𝜏2 𝛾, 𝜎𝜏2 ) . 𝑡,𝜏2 ≥0 2𝜏1 2𝜏2 2𝛿 2 𝑡𝜏1 𝛿 max 𝛾∈R (2.8) If this min-max has a unique solution, ( 𝛼, ˆ 𝜎, ˆ 𝜏ˆ1 , 𝜏ˆ2 , 𝑡ˆ, 𝛾), ˆ then as 𝑝 and 𝑛 grow to infinity with 𝛿 = 𝑝/𝑛, we have 𝜎 ˆ 𝜏ˆ2 𝑡ˆ ˆ 𝛽0 ) = Ψ( 𝛼ˆ − 𝜎 lim 𝜓( 𝛽, ˆ 𝜏ˆ2 𝛾, ˆ √ ,𝜎 ˆ 𝜏ˆ2 ) . 𝑝,𝑛→∞ 𝛿 (2.9) Theorem 1 derives a precise analysis for the performance of the convex estimator (2.2). A few remarks are in place. As discussed earlier, the result of this theorem applies to a sequence of problem instances of the convex estimator, with growing dimensions 𝑛 and 𝑝, such that 𝑛/𝑝 = 𝛿. Then, the convergence in theorem is in probability over the randomness of the features matrix X, distribution of the link function g and unknown data 𝛽0 (if there’s any). We now investigate two classes of popular models with this theorem. First, we consider the case when the link function g, simply adds a random noise to its input, as in linear models. Second, we analyze the case of binary classification. 2.1 Linear Models and M-Estimators We consider the standard problem of recovering an unknown signal 𝛽0 ∈ R 𝑝 from a vector y ∈ R𝑛 of 𝑛 noisy, linear observations given by y = X𝛽0 + z ∈ R𝑛 . 14 Here, X ∈ R𝑛×𝑝 is the (known) measurement matrix, and, z ∈ R𝑛 is the noise vector; the latter is generated from some distribution density in R𝑛 , say 𝑝 z . Our focus is on the high-dimensional regime where both the dimensions of the ambient space 𝑛 and the number of measurements 𝑚 are large [60, 146]. This is different than the classical one, where 𝑝 is small and fixed and only 𝑛 is assumed large. Of special interest is the scenario of compressed measurements, in which 𝑝 < 𝑛. In principle, such inverse problems are ill-posed, unless the unknown vector is somehow structurally constrained to only have very few degrees of freedom relative to its ambient space. Such signals are called structured signals; popular examples of such structure include sparsity, block-sparsity, low-rankness, etc. [10, 40]. We model such structural information on 𝛽0 by assuming that it is sampled from an 𝑛-dimensional probability density 𝑝 𝛽0 . Regularized M-estimators. The most widely used approach to obtain an estimate x̂ of the unknown 𝛽0 from the vector y of observations is via solving the convex program 𝛽ˆ := arg min L (y − X𝛽) + 𝜆 𝑓 (𝛽). 𝛽 (2.10) The loss function L : R𝑛 → R measures the deviation of X 𝛽ˆ from the observations y, the regularizer 𝑓 : R 𝑝 → R aims to promote the particular structure of 𝛽0 , and, the regularizer parameter 𝜆 > 0 balances between the two. Henceforth, both L and 𝑓 are assumed to be convex. Also, 𝑓 will typically be non-smooth. We refer to the minimization problems of the form in (2.10) as regularized M-estimators. Different choices of the loss function and of the regularizer give rise to a number of well-known estimators. A few concrete examples might suffice: (i) the LASSO [176] corresponds to (2.10) with L (v) = 21 kvk22 and 𝑓 (x) = kxk1 . General choices of the regularizer for the same loss function lead to the Generalized LASSO [129, 133] (ii) The regularized-LAD [192] minimizes an ℓ1 -loss function. (iii) The (generalized) square-root LASSO [22] solves (2.10) for L (v) = kvk2 . In the first two examples Í the loss function is separable over its entries, i.e. L (v) = 𝑛𝑗=1 ℓ(v 𝑗 ) for convex ℓ : R → R; in contrast, the square-root LASSO does not belong to this category. Accordingly, the regularizer function might be separable (e.g. ℓ1 -norm) or not (e.g. nuclear-norm). Prior Work With the advent of Compressed Sensing there is a very large number of theoretical results that have appeared in recent years in place for various types of regularized 15 M-estimators. The vast majority of those results hold under standard incoherence or restricted eigenvalue conditions on the measurement matrix X 2, but they are order-wise in nature, i.e., they characterize the error performance only up to loose constants. While this line of work includes unifying frameworks for the analysis of general instances of (2.10), the loose constants involved in the error bounds do not permit any accurate comparisons among the different instances (e.g. [14, 106, 123, 189] and references therein); therefore, they cannot be used to answer optimality questions of the nature discussed in this Section. This section derives precise characterizations of the error behavior (ones that do not involve unknown constants). Results of this nature have appeared in the literature under the additional assumption of an iid Gaussian distribution imposed on the entries of the matrix X. The inspiration behind these studies can be traced back to the seminal work of Donoho [57, 59] on the phase-transition of ℓ1 -minimization in the Compressed Sensing problem. This and the extensive follow-up literature mostly focused on the noiseless signal recovery problem. More recently, researchers have initiated the study of the exact reconstruction error of instances of (2.10) in the presence of noise. Unfortunately, no unifying treatment that holds for general instances has hitherto been available. To the best of our knowledge, our work is the first to obtain precise characterizations of the error performance of (2.10) for general convex loss functions, convex regularizers, and noise and signal distributions under a Gaussian assumption on the random measurement matrix X. In the rest of this section, we briefly outline the relevant literature. The first precise results on the performance of non-smooth convex optimization methods appear in the literature in the context of noiseless linear inverse problems that arise in Compressed Sensing. Here, the vector of measurements of the unknown structured signal 𝛽0 takes the form y = X𝛽0 ∈ R𝑛 and recovery is attempted via solving miny=X𝛽 𝑓 (𝛽), for an appropriately chosen convex regularizer 𝑓 . In the absence of noise, the standard measure of performance becomes that of the minimum number of measurements required for exact recovery of 𝛽0 . By now, there is an elegant and complete theory that precisely characterizes this number when X has entries iid Gaussian. The theory was built in a series of recent papers [6, 17, 40, 57, 129, 153, 156]. Our work extends the analysis to the noisy setting. In the presence of noise, the analysis is inherently more challenging since: (a) 2 Such conditions have been shown to be satisfied by a wide class of randomly designed measure- ment matrices, (e.g. [50, 68, 72] and references therein). A more recent line of works obtains similar order-wise bounds under even weaker assumptions on the randomness properties of X [101, 150, 178]. 16 one needs to characterize the precise value of the estimation error, rather than just discriminating between exact recovery or not; (b) the performance depends not only on the number of measurements but also on the noise and signal statistics. Also, it naturally includes the results of the noiseless case as special instances. However, many of the ideas, analytical tools and concepts developed in the works [6, 40, 153, 156] have proved to be useful in extending the results to the noisy setting. In the noisy setting, the first precise results analyzed the error performance of regularized least-squared (a.k.a. generalized-LASSO) under an iid gaussianity assumption on the noise distribution [18, 63, 129, 154, 168, 172, 173]. It has been only very recently, that El Karoui [65, 95], and, Donoho and Montanari [56, 64] were able to rigorously3 predict the error performance of M-estimators under more general assumptions on the loss function and on the noise distribution. However, the papers by Donoho and Montanari assume no regularization and El Karoui considers the special case of ridge regularization. Finally, the very recent paper [29] builds upon [56] and extends the study to the case of ℓ1 -regularization. In short, our work achieves by several means a more complete and transparent treatment of the subject, overcoming the limitations of previous endeavors as follows: (i) We consider arbitrary convex regularizers, (ii) We identify minimal and generic assumptions under which the general result holds, (iii) We remove any smoothness and strong-convexity assumptions on the loss function, which are required in all previous works. Also, the loss function (and regularizer) need not be separable (e.g., we allow L (v) = kvk2 or kvk∞ ), and, the distributions need not be iid. (iv) We remove boundedness assumptions on the moments of the noise distribution. Notably, our proof technique is fundamentally different than that of [65] and [56], and, it appears to be more direct and insightful in several ways. Applying Theorem 1 to M-Estimators In order to apply Theorem 1, we first need to translate Assumption 1 and 2 to this special case. Note that the Moreau envelope for this subtractive case of loss function in (2.10) becomes 1 1 kv − xk 2 + L (v, y) = min kv − xk 2 + L̄ (v − y) v v 2𝜏 2𝜏 1 2 = min kv − (x − y)k + L̄ (v) (2.11) v 2𝜏 eL (x, y, 𝜏) = min 3 The study of high-dimensional M-estimators has been previously considered in [19, 66]. How- ever, those results are only based on heuristic arguments and simulations. 17 Recall from Assumption 1, that the link function and the loss function affect the estimation performance through the link function 𝐿 that depends on the Moreau envelope as follows, 𝐿 (𝑐 1 , 𝑐 2 , 𝜏) = lim eL (𝑐 1 h1 + 𝑐 2 X𝛽0 , g(X𝛽0 ) , 𝜏) 𝑛,𝑝→∞ = lim min 𝑛,𝑝→∞ v 1 kv − (𝑐 1 h1 + (𝑐 2 − 1) X𝛽0 − z)k 2 + L̄ (v) 2𝜏 (2.12) Note that X is a random matrix with iid standard Gaussian entries. Thus, X𝛽0 can be replaces with h̃ k 𝛽0 k, where h̃ is a random vector withqiid standard Gaussian entries. Finally, 𝑐 1 h1 + (𝑐 2 − 1)X𝛽0 can be replaces by 𝑐21 + (𝑐 2 − 1) 2 k 𝛽0 k 2 h, where h is a random standard Gaussian vector. Therefore, the functional 𝐿(𝑐 1 , 𝑐 2 , 𝜏) q 2 is simply a function of 𝜏 and 𝑐 1 + (𝑐 2 − 1) 2 k 𝛽0 k 2 . If we use this new function in the equations of Theorem 1, we can derive the result of [165], after a few changes of variables. We specialize the general result of Theorem 1 to the popular case where the loss function L and the regularizer 𝑓 are both separable, and the noise vector and signal 𝛽0 both have entries iid. To make things concrete, assume4 L (v) = 𝑛 Õ iid ℓ(v 𝑗 ) and 𝑧 𝑗 ∼ 𝑝 𝑍 , 𝑗 = 1, . . . , 𝑛. 𝑓 (𝑥𝑖 ) and 𝛽0𝑖 ∼ 𝑝 𝑥 , 𝑖 = 1, . . . , 𝑛. 𝑗=1 𝑓 (x) = 𝑝 Õ iid 𝑖=1 Henceforth, both ℓ and 𝑓 are proper closed convex functions. Also, it is further assumed that ℓ(0) = 0 = min ℓ(𝑣) 𝑣 and 𝑓 (0) = 0. (2.13) Satisfying Assumption 1 To apply Theorem 1, we first need to verify that Assumption 1 holds for both the loss function and the noise distribution, and, for the regularizer and the signal distribution. 4 Note the slight abuse of notation here in using 𝑓 to denote both the vector-valued and scalar regularizer function. 18 Loss function and noise distribution In the separable case Assumption 1 essentially translate to the following requirement on ℓ and 𝑝 𝑍 :   E |ℓ+0 (𝑐𝐺 + 𝑍)| 2 < ∞, for all 𝑐 ∈ R. (2.14) where the expectation is over 𝑍 ∼ 𝑝 𝑍 and 𝐺 ∼ N (0, 1). This is shown in Lemma 1 below. Lemma 1 (Expected Moreau envelope–Loss fcn) If ℓ and 𝑝 𝑍 satisfy (2.14), then, Assumption 1 hold with 𝐿(𝑐, 𝜏) = E [eℓ (𝑐𝐺 + 𝑍, y, 𝜏) − ℓ(𝑍)] . (2.15) This lemmas is simply a result of the law of large numbers. Regularizer and Signal Distribution Not surprisingly, the required condition on 𝑓 and 𝑝 𝑥 becomes   E | 𝑓+0 (𝑐𝐻 + 𝛽0 )| 2 < ∞, for all 𝑐 ∈ R. (2.16) where the expectation is over 𝛽0 ∼ 𝑝 𝑥 and 𝐻 ∼ N (0, 1). Additionally, the following mild assumptions are required: ∃ 𝛽+ > 0, 𝛽− < 0 such that 0 ≤ 𝑓 (𝑥 ± ) < ∞ and E𝛽02 < ∞. (2.17) Lemma 2 (Expected Moreau Envelope–Regularizer fcn) If 𝑓 and 𝑝 𝑥 satisfy (2.16) and (2.17), then, Assumption 1 hold with   𝐹 (𝑐, 𝜏) = E e 𝑓 (𝑐𝐻 + 𝛽0 , 𝜏) − 𝑓 (𝑋0 ) . (2.18) The Expected Moreau Envelope If conditions (2.14), (2.16) and (2.17) are satisfied, then Theorem 1 is applicable with 𝐿 and 𝐹 given as in (2.15) and (2.18), respectively. We call those functions, the Expected Moreau Envelopes. It is apparent from Theorem 1 that they play a key role in determining the error performance of the corresponding M-estimators. Moreover, they possess two key features, namely, smoothness and strict convexity; we elaborate on these here. 19 Lemma 3 (Smoothness) Suppose ℓ is a closed proper convex function and 𝑝 𝑍 a noise density such that (2.14) holds. Then, the function 𝐿 (𝑐, 𝜏) := E [eℓ (𝑐𝐺 + 𝑍, 𝜏) − ℓ(𝑍)] is differentiable in R × R>0 with   𝜕𝐿 = E e0ℓ (𝑐𝐺 + 𝑍; 𝜏) 𝐺 𝜕𝑐 and
2i 𝜕𝐿 1 h = − E e0ℓ (𝑐𝐺 + 𝑍; 𝜏) . 𝜕𝜏 2 Note that 𝐿 is smooth, regardless of any non-smoothness of ℓ. This is a wellknown fact about Moreau envelope approximations, and also, one of the primal reasons behind the important role those functions play in convex analysis [136]. The property is naturally inherited to the Expected Moreau envelopes as revealed by the lemma above. Lemma 4 (Strict Convexity) Suppose ℓ is a closed proper convex function and 𝑝 𝑍 a noise density such that (2.14) holds and the following are satisfied: 1. Either there exists 𝑥 ∈ R at which ℓ is not differentiable, or, there exists interval I ⊂ R where ℓ is differentiable with a strictly increasing derivative, 2. Var(𝑍) ≠ 0 5, and, at each 𝑧 ∈ R, 𝑝 𝑍 (𝑧) is either a Dirac delta function or it is continuous. Then, 𝐿(𝑐, 𝜏) := E [eℓ (𝑐𝐺 + 𝑍, 𝜏) − ℓ(𝑍)] is jointly strictly convex in R>0 × R>0 . Remark 1 The function 𝐿 is strictly convex, without requiring any strong or strict convexity assumption on ℓ. Interestingly, this property is not in general true for Moreau envelope approximations, but, it turns out to be the case for the Expected Moreau envelope 𝐿. The fact that the latter further involves taking an expectation over 𝑐𝐺 + 𝑍, with 𝐺 having a nonzero density on the entire real line, turns out to be critical. We are now ready to state our main result of this section which characterizes the squared error of separable M-estimators. This is essentially a corollary of Theorem 1. 5 We require that there exist at least two values of 𝑧 ∈ R for which 𝑝 𝑍 (𝑧) > 0. In particular, there is no requirement that Var(𝑍) be defined, e.g. Cauchy distribution is allowed. 20 Theorem 2 Suppose ℓ and 𝑝 𝑍 satisfy (2.14), and, the two conditions of Lemma 4. Further assume that 𝑓 , 𝑝 𝑥 satisfy (2.16) and (2.17). Let 𝛽ˆ be any minimizer of the separable M-estimator, and consider the problem in (2.8) with 𝐿 and 𝐹 given as in (2.15) and (2.18), respectively. If the solution to the (2.8) is unique and bounded, then it holds in probability that 1 ˆ k 𝛽 − 𝛽0 k22 = 𝛼★2 . 𝑛→∞ 𝑝 lim where 𝛼★ is the solution to the system of equations in (2.19), in 4 unknowns 𝛼, 𝛾, 𝜈, 𝜅. As a system of nonlinear equations Theorem 2 predicts the error of the M-estimator as the optimizer 𝛼★ to a convexconcave optimization problem with four optimization variables. Equivalently, 𝛼★ can be expressed via the first-order optimality conditions (stationary equations) corresponding to this optimization. Recall from Lemma 3 that 𝐿 and 𝐹 are differentiable (irrespective of smoothness of ℓ and 𝑓 ). The solution to the (2.8) can be derived by taking derivative of the objective function of (2.8). This results in the following system of non-linear equations. " 2#     𝜆 𝜆 𝛾 𝛾   · e0𝑓 𝐻 + 𝑋0 ; , 𝛼2 = E − 𝐻    𝜈 𝜈 𝜈 𝜈    h i    𝛾 2 = 𝛿 · E e0 (𝛼𝐺 + 𝑍, 𝜅)
2 ,  ℓ (2.19)     0  𝜈𝛼 = 𝛿 · E eℓ (𝛼𝐺 + 𝑍, 𝜅) · 𝐺 ,          𝛾 𝜆 𝜆  0 𝛾    𝜅𝛾 = 𝜈 − 𝜈 · E e 𝑓 𝜈 𝐻 + 𝑋0 ; 𝜈 · 𝐻 .  Here, 𝑒0𝑓 and 𝑒0ℓ , denote the first derivatives of the Moureau envelopes with respect to their first argument. Remark 2 The system of equations in (2.19) can be easily reformulated in terms of the proximal operator of 𝑓 and ℓ, using 1 ( 𝜒 − proxℓ ( 𝜒; 𝜏)), 𝜏 and similar for 𝑓 (see Lemma 40(iii)). In the case of additional smoothness assumptions on the loss function and/or the regularizer, further reformulations are possible. For example, if ℓ is two times differentiable, then using Stein’s formula for Normal random variables we can make the following substitution in (2.19):     E e0ℓ (𝛼𝐺 + 𝑍, 𝜅) · 𝐺 = 𝛼 · E e00ℓ (𝛼𝐺 + 𝑍, 𝜅) , (2.20) e0ℓ ( 𝜒, 𝜏) = 21 where the double-prime superscript denotes the second derivative with respect to the first argument. Such reformulations, are often convenient for analysis purposes; see for example Remark 6. Remark 3 The system of equations in (2.19) comprises of four nonlinear equations in four unknowns. Setting t = (𝛼, 𝛽, 𝜈, 𝜅) for the vector of unknowns, the system of equations in (2.19) can be written as t = 𝑆(t), for appropriately defined 𝑆 : R4 → R4 . We have empirically observed that a simple recursion t 𝑘+1 = 𝑆(t 𝑘 ), 𝑘 = 0, 1, . . . converges to a solution t∗ satisfying t∗ = 𝑆(t∗ ). This observation is particularly useful since it allows for efficient numerical experimentations, cf. Section 2.1. It is certainly an interesting and practically useful subject of future work to identify analytic conditions under which such simple recursive schemes provide efficient means of solving (2.19). Remark 4 The results of this section extend naturally, and without any extra effort, to the case of “block-seperable" loss functions and/or regularizers. A popular example that falls in this category is ℓ1,2 -regularization, which is typically used for Í𝑏 the recovery of block-sparse signals. In such a case 𝑓 (x) = 𝑖=1 k [x]𝑖 k2 , where [x]𝑖 = [x (𝑖−1)𝑡+1 , x (𝑖−1)𝑡+2 , . . . , x (𝑖−1)𝑡+𝑡 ], 𝑖 = 1, . . . , 𝑏 is the 𝑖 th block of x. Here, 𝑏 is the number of blocks and 𝑡 is the length of each block. In the proportional highdimensional regime, one would assume 𝑏 growing linearly with 𝑝 with a constant ratio of 1/𝑡. Next, we explore some popular examples, where we can apply Theorem 2. No Regularization Consider an M-estimator without regularization, i.e., 𝛽ˆ := arg min 𝑛 Õ 𝛽 ℓ(y 𝑗 − x𝑇𝑗 𝛽 𝑗 ). (2.21) 𝑗=1 iid For simplicity, we consider z 𝑗 ∼ 𝑝 𝑍 and a separable loss function. Assuming that ℓ and 𝑝 𝑍 satisfy the assumptions of Theorem 2, and, noting that 𝑓 = 0 =⇒ 𝐹 (𝑐, 𝜏) = 0, the squared error of (2.21) is predicted by the minimizer 𝛼∗ of the following (SPO) problem 𝜏𝑔 𝛾𝜏𝑔 + 𝛿𝐿 (𝛼, ) − 𝛼𝛾, 𝛼≥0 𝛾≥0 2 𝛾 inf sup 𝜏𝑔 >0 (2.22) 22 where we have performed the (straightforward) optimization over 𝜏ℎ : inf 𝜏ℎ >0 𝜏2ℎ + 𝛾2 2𝜏ℎ = 𝛾. We may equivalently express 𝛼∗ as the solution to the first-order optimality conditions of (2.22). In particular, the stationary equations (see (2.19)) simplify in this case to the following system of two equations in two unknowns: h
2i    𝛼2 = 𝛿𝜅 2 E e0ℓ (𝛼𝐺 + 𝑍, 𝜅) ,  (2.23)     𝛼 = 𝛿𝜅 · E e0ℓ (𝛼𝐺 + 𝑍, 𝜅) · 𝐺 .  Starting from (2.23), some interesting conclusions can be drawn regarding the performance of M-estimators without regularization, which we gather in the following remarks. Remark 5 It follows from (2.23) that in the absence of regularization, it is required that the number of measurements 𝑛 is at least as large as the dimension of the ambient space 𝑝 (𝛿 ≥ 1), in order for the recovery to be stable, i.e. the error be finite. To see this, assume stable recovery, then there exists (𝛼∗ , 𝜅∗ ) satisfying (2.23). Starting from the second equation, applying the Cauchy-Schwarz inequality and substituting back the first equation we find: s   2   0  𝛼∗ 0 𝛼∗ = 𝛿𝜅 ∗ · E eℓ (𝛼∗ 𝐺 + 𝑍, 𝜅∗ ) · 𝐺 ≤ 𝛿𝜅 ∗ · E eℓ (𝛼∗ 𝐺 + 𝑍, 𝜅∗ ) ) = 𝛿𝜅 ∗ √ 𝛿𝜅 ∗ ⇒ 𝛿 ≥ 1. (2.24) Remark 6 Assume 𝑒 ℓ is two times differentiable (e.g., this is the case if ℓ is two times differentiable). Then, applying Stein’s formula (2.20), a simple rearrangement of (2.23) shows that h
i 0 (𝛼 𝐺 + 𝑍, 𝜅 ) 2 E e ∗ ℓ ∗ 1 𝛼∗2 =  . (2.25)  00  2 𝛿 E eℓ (𝛼∗ 𝐺 + 𝑍, 𝜅∗ ) The formula above coincides with the corresponding expression in [56], but the latter requires additional smoothness and strong-convexity assumptions on ℓ, which are not necessary for (2.23) to hold. The proof of [56] is based on the AMP framework [62]. Remark 7 The simplest instance of the general M-estimator is the Least-squares, i.e. 𝛽ˆ := min 𝛽 ky − X𝛽k22 . Of course, in this case, 𝛽ˆ has a closed form expression 23 which can be directly used to predict the error behavior [170]. However, for illustration purposes, we show how the same result can be also obtained from (2.23). This is also one of the few cases where 𝛼∗ can be expressed in closed form. iid Assume 𝛿 > 1 and z 𝑗 ∼ 𝑝 𝑍 with bounded second moment, i.e. 0 < E𝑍 2 = 𝜎 2 < ∞. Then, it can be readily checked that all assumptions hold for 12 (·) 2 , 𝑝 𝑧 . Also, 𝜒 1 and 𝑒001 2 ( 𝜒; 𝜏) = 1+𝜏 . Solving for the second equation in (2.23) 𝑒01 2 ( 𝜒; 𝜏) = 1+𝜏 2 (·) 2 (·) 1 gives 𝜅∗ = 𝛿−1 . Substituting this into the first, we recover the well-known formula 𝛼∗2 = 𝜎 2 1 . 𝛿−1 (2.26) Ridge Regularization A popular regularizer in the machine learning and statistics literature is the ridge regularizer (also known as Tikhonov regularizer), i.e. 𝛽ˆ := arg min 𝑛 Õ 𝛽 ℓ(y 𝑗 − x𝑇𝑗 𝛽 𝑗 ) + 𝜆 𝑗=1 k 𝛽k22 . 2 (2.27) We specialize Theorem 1 to that case. For simplicity, we assume a separable loss iid iid function, and, z 𝑗 ∼ 𝑝 𝑍 and 𝛽0𝑖 ∼ 𝑝 𝑋 . We will apply Theorem 2. Suppose that ℓ satisfies the assumptions. Also, assume E𝛽02 = 𝜎𝛽2 < ∞. Then, for 𝑓 = 21 (·) 2 , it is easily verified that E[( 𝑓 0 (𝑐𝐻 + 𝛽0 )) 2 ] = E[(𝑐𝐻 + 𝛽0 ) 2 ] < ∞. Hence, the squared-error of (2.27) is predicted by 𝛼∗ , the unique solution to (2.19) with 𝐹 (𝑐, 𝜏) = 𝑐2 + 𝜎𝛽2 2(𝜏 + 1) − 𝜎𝛽2 . The first-order optimality conditions (see (2.19)) of this problem simplify after some algebra to the following two equations in two unknowns: (   𝛼2 = 𝛿𝜅 2 · E e0ℓ (𝛼𝐺 + 𝑍, 𝜅) 2 + 𝜆2 𝜅 2 𝜎𝛽2 , (2.28)   𝛼 (1 − 𝜆𝜅) = 𝛿𝜅 · E e0ℓ (𝛼𝐺 + 𝑍, 𝜅) · 𝐺 . Remark 8 Assume proxℓ (𝑥; 𝜏) is two times differentiable with respect to 𝑐 (e.g., this is the case if ℓ is two times differentiable), and write prox0ℓ (𝑥, 𝜏) for the derivative with respect to 𝑥. Applying (2.20), a simple rearrangement of (2.28) yields the following equivalent system of equations    0   𝛿 − 1 + 𝜅𝜆 = 𝛿 · E proxℓ (𝛼𝐺 + 𝑍; 𝜅) ,  h (2.29)
2i  𝛼2 = 𝛿E 𝛼𝐺 + 𝑍 − proxℓ (𝛼𝐺 + 𝑍; 𝜅) + 𝜆2 𝜅 2 𝜎𝛽2 .   24 The formula above coincides with the corresponding expression in [95, Thm. 2.1]6. The result in [95] requires additional smoothness assumptions on ℓ. Our result holds under relaxed assumptions and has been derived as a corollary of Theorem 1. On the other hand, [95, Thm. 2.1] is shown to be true for design matrices X with iid entries beyond Gaussian, e.g. sub-Gaussian. Remark 9 Consider a least-squares loss function where ℓ(𝑥) = 21 𝑥 2 and a noise 𝑥 and prox0ℓ (𝑥; 𝜏) = distribution of variance E𝑍 2 = 𝜎𝑧2 < ∞. Then proxℓ (𝑥; 𝜏) = 1+𝜏 1 1+𝜏 . Substituting in (2.29) gives       1 − 𝜅𝜆 = 𝛿𝜅 , 1+𝜅 𝜅2 𝜅2  2   ) = 𝛿 · 𝜎 2 + 𝜆2 𝜅 2 𝜎𝛽2 . 𝛼 (1 − 𝛿 ·  2 2 𝑧 (1 + 𝜅) (1 + 𝜅)  (2.30) Now, we can solve these to get the following closed form expression for 𝛼∗ :  −1   𝜅2 𝜅2 2 2 2 2 𝛼 = 𝛿· , · 𝜎𝑧 + 𝜆 𝜎𝛽 𝜅 · 1 − 𝛿 · (1 + 𝜅) 2 (1 + 𝜅) 2  2 where 1−𝛿−𝜆+ (2.31) p (1 − 𝛿 − 𝜆) 2 + 4𝜆 . (2.32) 2𝜆 Observe that letting 𝜆 → 0 (which would correspond to ordinary least-squares) and assuming 𝛿 > 1, 𝜅 in (2.32) approaches 1/(𝛿 − 1) and the optimal 𝛼2 in (2.31) becomes 𝜎𝑧2 /(𝛿 − 1), which agrees with (2.26), as expected. 𝜅= iid Remark 10 Let a Gaussian input distribution 𝛽0,𝑖 ∼ N (0, 1) and any noise distribution of power E𝑍 2 = 𝜎𝑧2 < ∞. We show that a ridge-regularized M-estimator with a least-squares loss function and optimally tuned 𝜆 achieves asymptotically the Minimum Mean-Squared Error (MMSE) of estimating 𝛽0 from y = X𝛽0 + z. First, we use the results of Remark 9 to calculate the achieved error of the Mestimator optimized over the values of the regularizer parameter:  1 ˆ k 𝛽 − 𝛽0 k22 = inf 𝛼2 (𝜅(𝜆), 𝜆) as in (2.31) | 𝜅(𝜆) satisfies (2.32) . 𝜆>0 𝜆>0 𝑝→∞ 𝑝 (2.33) 𝑜 ∗ := inf lim 6 In comparing (2.29) to [95, Eqn. (4)], due to some differences in normalizations the following “dictionary" needs to be used to match the results: 𝛼 ↔ 𝑟 𝜌 (𝜅), 𝜅 ↔ 𝑐 𝜌 (𝜅), 𝛿−1 𝜆 ↔ 𝜏 and 𝛿−1 ↔ 𝜅. 25 The optimization over 𝜆 is possible as follows. From (2.30), we find 𝛿  𝜅  2 (1 − 𝜅𝜆) 2 . = 𝜅+1 𝛿 (2.34) Substituting this in (2.31), and denoting 𝛽 = 𝜅𝜆, gives 𝛿𝛽2 + 𝜎 2 (1 − 𝛽) 2 𝛼 = . 𝛿 − (1 − 𝛽) 2 2 (2.35) Minimizing 𝛼2 over 𝜆 > 0 in (2.33) is equivalent to minimizing the fraction above over 0 < 𝛽 < 1, since there always exist 𝜅, 𝜆 satisfying 𝛽 = 𝜅𝜆 and (2.34). Thus, performing the optimization over 0 < 𝛽 < 1 in (2.35) we find  p 1 2 2 2 4 1 − 𝜎 − 𝛿 + (1 − 𝛿) + 2𝜎 (𝛿 + 1) + 𝜎 . 𝑜∗ = 2 (2.36) To complete the proof of the claim, we combine this with [196, Thm. 8, Eqn. (56)], where it is shown that the MMSE is given by the same expression as in the right-hand side above. Cone-constrained M-estimators Constrained M-estimators solve 𝛽ˆ = arg min 𝑛 Õ 𝛽∈C ℓ(y 𝑗 − x𝑇𝑗 𝛽), (2.37) 𝑗=1 for some set 𝛽 ∈ C. The role of the regularizer in (2.10) is played here by the constraint 𝛽 ∈ C. In particular, it is common that C takes the form C = {𝛽 | 𝑔(𝛽) ≤ 𝑔(𝛽0 )}, i.e., it is chosen as the set of descent directions of some convex function 𝑔. In this setting, 𝛽0 is assumed to be a structured signal (e.g. sparse, low-rank) and 𝑔 is chosen to promote the particular structure (e.g. ℓ1 -norm, nuclear-norm) [40, 73, 129, 133]. Of course, such a formulation assumes prior knowledge of the value of 𝑔 at 𝛽0 . Also, in this case, there exists by Lagrangian duality a value of 𝜆 for which the regularized M-estimator with 𝑓 (𝑥) = 𝑔(𝑥) is equivalent to (2.37). A relaxation that is often undertaken to facilitate the analysis of (2.37) involves substituting C by its conic hull, which is also known as the tangent cone of 𝑔 at 𝛽0 (e.g. [40]). We call the resulting program, a cone-constrained M-estimator. For the special case of an ℓ2 -loss function, the squared error performance of constrained M-estimators has been previously considered in [129, 154] (also, see Remark 12 26 below). The analysis was performed in the high-SNR regime, where noise variance approaches zero. In this regime it was shown that the conic-relaxation above is exact. In this section, we analyze the error performance of cone-constrained Mestimators with general loss functions and derive some interesting conclusions. Before proceeding further, observe that (2.37) is an instance of (2.10) with a nonseparable regularizer; hence, it also serves to showcase the applicability of Theorem 1 to such settings. Consider solving (2.37) with C = K + 𝛽0 := {𝜆h | 𝜆 ≥ 0, 𝑔(𝛽0 + h) ≤ 𝑔(𝛽0 )} + 𝛽0 and 𝑔 a proper, closed, convex function. Here, K is the tangent cone of 𝑔 at 𝛽0 , and 𝛽0 is assumed fixed. The constrained minimization above can be written in the general form of regularized M-estimators in (2.10)LASSO by choosing the regularizer to be the indicator function for the cone, i.e. 𝑓 (𝛽) = 𝜹 {𝛽∈C} . Let DistC (v), denote the distance of a vector v to a set C. We have, e𝜹 {𝛽 ∈ C } (𝑐h + 𝛽0 ; 𝜏) = 1 𝑐2 1 min k𝑐h − vk22 = Dist2K (𝑐h) = Dist2K (h). 2𝜏 v∈C−𝛽0 2𝜏 2𝜏 In the last equality above we have used the homogeneity of the cone K. Let K ◦ denote the polar cone of K, and,     𝐷 K := E Dist2K ◦ (h) = E khk22 − Dist2K (h) . This quantity is known as the statistical dimension [6] of the cone K, or, as the Gaussian distance squared [129]. It can be though of as a measure of the size of the cone, and also, it is very closely related to the gaussian width of K[6]. We assume that 𝐷K → 𝐷 K ∈ (0, 1). 𝑝 (2.38) This translates to an assumption on the degrees of freedom of the structured signal 𝛽0 being proportional to its dimension. For example, for a 𝑘-sparse 𝛽0 and 𝑔(𝛽) = k 𝛽k1 , (2.38) is satisfied for 𝑘 = 𝜌 𝑝, 𝜌 ∈ (0, 1). 2 𝑐 With (2.38), Assumption 1(a) holds with 𝐹 (𝑐, 𝜏) = 2𝜏 (1 − 𝐷 K ). For this, it is straightforward to check that Assumption 1(b) is also satisfied. Overall, if ℓ, 𝑝 𝑍 satisfy the conditions of Theorem 2 and 𝑔, 𝛽0 are such that (2.38) holds, then 27 Theorem 1 applies, and the squared error of the cone-constrained M-estimator in (2.37) is predicted by the unique minimizer 𝛼∗ of the (SPO) problem below: q  
𝛾𝜏𝑔 inf sup + 𝛿 · E eℓ 𝛼𝐺 + 𝑍; 𝜏𝑔 /𝛾 − ℓ(𝑍) − 𝛼𝛾 𝐷 K . (2.39) 𝛼≥0 𝛾≥0 2 𝜏𝑔 >0 Compared to (2.8) (which involves six optimization variables), we have performed 2 the (straightforward) optimization over 𝜏ℎ : inf 𝜏ℎ >0 𝜏2ℎ + 𝛾2 𝐷 K 2𝜏ℎ = 𝛾𝐷 K . Remark 11 Starting from (2.39) we can conclude on the minimum number of measurements required for stable recovery. We show that the normalized number of measurements 𝛿 need to be at least as large as 𝐷 K , in order for the error to be finite. This is to be compared with the case where no regularization is used that required 𝛿 ≥ 1 > 𝐷 K (see Remark 5). To prove the claim, assume finite error, then the value where it converges is predicted by (2.39). Standard first-order optimality conditions give7
2i 𝛿 h ≥ 0, (2.40a) 𝛾 − E e0ℓ (𝛼𝐺 + 𝑍; 𝜏𝑔 /𝛾) 𝛾 q 0 𝛿E[eℓ (𝛼𝐺 + 𝑍; 𝜏𝑔 /𝛾) · 𝐺] − 𝛾 𝐷 K ≥ 0, (2.40b) q
2i 𝜏 𝛿𝜏 h 0 + 2 E eℓ (𝛼𝐺 + 𝑍; 𝜏𝑔 /𝛾) − 𝛼 𝐷 K ≤ 0. (2.40c) 2 2𝛾 Starting from the second equation, applying the Cauchy-Schwarz inequality and substituting back the first equation we conclude as follows: r q  2 𝛾 𝛾 𝐷 K ≤ 𝛿E[eℓ (𝛼𝐺 + 𝑍; 𝜏𝑔 /𝛾) · 𝐺] ≤ 𝛿 E[ e0ℓ (𝛼𝐺 + 𝑍; 𝜏𝑔 /𝛾) ] ≤ 𝛿 √ ⇒ 𝛿 ≥ 𝐷 K . 𝛿 (2.41) Remark 12 Consider a least-squares loss function and a noise distribution of variance E𝑍 2 = 𝜎 2 < ∞. Then, the solution to (2.39) admits an insightful closed form expression. First, in (2.39) perform the optimization over 𝜏𝑔 . Equating (2.40a) to 0, √ √ gives 𝜏𝑔 = 𝛿 𝛼2 + 𝜎 2 − 𝛾. Substituting this in (2.39), we are left to solve for  p  q √ 𝛾2 inf sup 𝛾 𝛿 𝛼2 + 𝜎 2 − 𝛼 𝐷 K − . 𝛼≥0 𝛾≥0 2 7 The three equations in (2.40) correspond to differentiation of the objective of (2.39) with respect to 𝜏, 𝛼 and 𝛾, respectively. If any of the variables is zero at the optimal, then, the corresponding equation holding with an inequality is necessary and sufficient. On the other hand, if the optimal is strictly positive, then the equation should hold with equality. 28 It can be easily checked that if 𝛿 > 𝐷 K , then the optimal 𝛼∗ is 𝛼∗2 = 𝜎 2 𝐷K . 𝛿 − 𝐷K (2.42) It is insightful to compare this with (2.26), the corresponding error formula for least-suares: the only difference is that 1 is substituted with the statistical dimension 𝐷 K . Also, verifying the conclusion of the previous remark, we now require 𝛿 > 𝐷 K instead of 𝛿 > 1, implying that recovery is in general possible with less measurements than the dimension of the signal. The result in (2.42) was first proved for ℓ1 -regularization in [154], and, was later generalized in [129, 169] (also, [170]). In contrast to the lengthy treatments in those references, the result was derived here as a simple corollary of Theorem 1. Remark 13 In (2.40b) apply Stein’s inequality and combine it with (2.40a) to yield h
2i 0 𝐷K 𝛾 2 /𝛿 𝐷 K E eℓ (𝛼𝐺 + 𝑍; 𝜏𝑔 /𝛾)  00  ≥   . 𝛼2 ≥ (2.43) 𝛿 E eℓ (𝛼𝐺 + 𝑍; 𝜏𝑔 /𝛾) 𝛿 E e00ℓ (𝛼𝐺 + 𝑍; 𝜏𝑔 /𝛾)   For the first inequality above, we have assumed that at the optimal, E e00ℓ (𝛼𝐺 + 𝑍; 𝜏𝑔 /𝛾) < ∞. When this holds, (see Remark 14 for an instance where this is not the case) we can use the above to lower bound the error performance in terms of the Fisher information of the noise. Based on a result of [109], Donoho and Montanari prove in [56, Lem. 3.4,3.5] that the right-hand side in (2.43) is further lower bounded by 2  𝜕 log 𝑝 𝑍 (𝑧) denotes the Fisher information 𝐼 (𝑍)/(1 + 𝛼2 𝐼 (𝑍)), where 𝐼 (𝑍) = E 𝜕𝑧 of the random variable 𝑍, which is assumed to have a differentiable density. Using this and solving for 𝛼2 , we conclude with 𝛼2 ≥ 1 . 𝛿 − 𝐷 K 𝐼 (𝑍) 𝐷K (2.44) For Gaussian noise of variance 𝜎 2 , we have 1/𝐼 (𝑍) = 𝜎 2 . In this case the lower bound in (2.44) coincides with the error formula of the least-squares loss function, thus proving optimality of the latter. Remark 14 The lower bound in (2.44) only holds if the optimal 𝛼∗ in (2.39) is strictly positive. This is not always the case: under circumstances, it is possible to choose the loss function such that the resulting cone-constrained M-estimator is 29 consistent, i.e. 𝛼∗ = 0. Theorem 1 is the starting point to identifying such interesting scenarios. Here, we illustrate this through an example: we assume a sparse gaussian-noise model and use a Least Absolute Deviations (LAD) loss function. More precisely, 𝑝 𝑍 (𝑍) = 𝑠¯𝛿0 (𝑍) + (1 − 𝑠¯) √1 exp(−𝑍 2 /2), 𝑠¯ ∈ (0, 1) and ℓ(𝑣) = |𝑣|. In Section .3 2𝜋 we prove that when 𝑠¯, 𝛿 and 𝐷 K are such that ) ( r ∫ ∞ 2 𝛿 ≥ 𝐷 K + min 𝑠¯ (1 + 𝜅 2 ) + (𝛿 − 𝑠¯) (𝐺 − 𝜅) 2 exp(−𝐺 2 /2)d𝐺 , (2.45) 𝜅>0 𝜋 𝜅 then the first-order optimality conditions in (2.40) are satisfied for 𝛼 → 0, 𝜏𝑔 → 0 and some 𝛾 > 0. Thus, when the number of measurements is large enough such that (2.45) holds, then 𝛼∗ = 0, and, 𝛽0 is perfectly recovered8. Generalized LASSO The generalized LASSO solves 1 𝛽ˆ := arg min ky − X𝛽k22 + 𝜆 𝑓 (𝛽). 𝛽 2 (2.46) For simplicity, suppose that 𝑓 is separable and satisfies the assumptions of Theorem iid 2. Also, assume z 𝑗 ∼ 𝑝 𝑍 such that 0 < E𝑍 2 =: 𝜎 2 < ∞. Then, for ℓ = 12 (·) 2 , it is easily verified that E[(ℓ0 (𝑐𝐺 + 𝑍)) 2 ] = E[(𝑐𝐺 + 𝑍) 2 ] < ∞. Hence, the squared-error of (2.46) is predicted by 𝛼∗ , the unique solution to the (2.19) with 𝑐2 +𝜎 2 − 𝜎2 . 𝐿 (𝑐, 𝜏) = 2(𝜏+1) Equivalently, the error is predicted by the solution to the stationary equations in 𝜒 (2.19) with 𝑒01 2 ( 𝜒; 𝜏) = 1+𝜏 . The second and third equations in (2.19) give 2 (·) 𝛾 2 (1 + 𝜅) 2 = 𝛿(𝛼2 + 𝜎 2 ), 𝜈(1 + 𝜅) = 𝛿. 8 In the context that it appears here, the perfect recovery condition in (2.45) has been shown previously in [167]. The problem is very closely related to the demixing problem in which one aims to extract two (or more) constituents from a mixture of structured vectors [112]. In that context, recovery conditions like the one in (2.45) have been generalized to other kinds of structures beyond sparsity [73, 112, 113]. Our purpose here has been to illustrate how Theorem 1 can be used to derive such results. Besides, the generality of the paper’s setup offers the potential to extend such consistency-type results beyond cone-constrained M-estimators and beyond fixed signals 𝛽0 . This is an interesting direction for future research. 30 Solving these for 𝜅 and 𝜈, and substituting them in the remaining two equations results in the following system of two nonlinear equations in two unknowns   2 √ 2 2 √  2 2 +𝜎 2  𝛼 𝜆 𝛼 +𝜎 𝛼 0  √ 𝐻 + 𝛽0 , 𝜆 √ −𝐻 ,  𝛿 𝛼2 +𝜎2 = E 𝛾 𝑒 𝑓  𝛿 𝛾 𝛿 (2.47) h √ 2 2  i √ √  2 +𝜎 2 𝛿 𝛼 +𝜎 𝛼 0 2  √  𝛾(1 − 𝛿) + 𝛾 √ 2 2 = 𝜆E 𝑒 𝑓 𝐻 + 𝛽0 , 𝜆 √ ·𝐻 . 𝛿 𝛾 𝛿 𝛼 +𝜎  For the special case of ℓ1 -regularization, the result above was proved by Bayati and Montanari [18] using the AMP framework. In the generality presented here, the result appears to be novel. Remark 15 An interesting observation from (2.47) is that the generalized LASSO cannot achieve perfect recovery, irrespective of the choice   of the regularizer function. 2 𝜆 0 √𝜎 𝜆𝜎 To see this, the first equation in (2.47) for 𝛼 = 0 gives E 𝛾 𝑒 𝑓 ( 𝐻 + 𝛽0 , √ ) − 𝐻 = 𝛿 𝛾 𝛿 0. Then, it must hold, almost surely, that the argument under the expectation sign be equal to zero. Evaluating the derivative  of the envelope function, this becomes  𝜆𝜎 𝜎 equivalent to 𝛽0 = prox 𝑓 √ 𝐻 + 𝛽0 ; √ . This, when combined with the optimality 𝛿 𝛾 𝛿 conditions for the Moreau envelope gives that almost surely √𝜎 𝐻 ∈ 𝜕 𝑓 (𝛽0 ). Thus, 𝛿 we have reached a contradiction because 𝐻 can take any real value as a Gaussian random variable. Square-root LASSO The (Generalized) Square-root LASSO (also known as ℓ2 -LASSO [129]) solves9 √ 𝛽ˆ := arg min 𝑝ky − X𝛽k2 + 𝜆 𝑓 (𝛽). 𝛽 (2.48) In contrast to the other examples in this section, the square-root LASSO is an instance of (2.10) with a non-separable loss function. Observe the normalization of √ the loss function with a 𝑝-factor. This is to make the loss function of the same order as the regularizer.   √ One can show that when L (v) = 𝑝kvk2 and z ∼ 𝑝 z with E kzk22 /𝑛 = 𝜎 2 ∈ (0, ∞), then Assumption 1(a) holds with 𝐿(𝛼, 𝜏) = √  𝜏   √1 ( 𝛼2 + 𝜎 2 − 𝜎) − 2𝛿  √ √ , if 𝛿 𝛼2 + 𝜎 2 ≥ 𝜏, 1   2𝜏 (𝛼2 + 𝜎 2 ) − √𝜎 𝛿  , otherwise. 𝛿 (2.49) 9 We refer the interested reader to [22, 129, 169] for a discussion on the similarities and differences between (2.48) and the Generalized LASSO in (2.46). 31 Also, Assumption 1(b) is trivially satisfied. Thus, considering any regularizer that satisfies Assumptions 1(b), Theorem 1 applies, and predicts the squared error of (2.48) as the unique minimizer 𝛼∗ to the following optimization: √ √     𝛾 𝛿 𝛼2 + 𝜎 2 𝛼𝜏ℎ 𝛼𝛾 𝛼𝜆 inf sup − − +𝜆·𝐹 , + √ √  𝛼≥0 𝛾≥0 2 2𝜏ℎ 𝜏ℎ 𝜏ℎ  𝛿 𝛼2 + 𝜎 2  𝜏ℎ >0 𝛼𝛾 2   , if 𝛾 ≤ 1 . , otherwise (2.50) To arrive to (2.50) starting from (2.8), we have replaced 𝐿 with (2.49) and have performed the minimization over 𝜏𝑔 as shown below: inf √ √  𝜏𝑔   𝛾𝜏2𝑔 − 2𝛾  + 𝛿 𝛼2 + 𝜎 2 𝛾𝜏 𝛾𝛿 𝜏𝑔 ≥0   2𝜏 (𝛼2 + 𝜎 2 ) + 2𝑔 𝑔  𝜏2 , if 𝛿(𝛼2 + 𝜎 2 ) ≥ 𝛾𝑔2 , otherwise. √ √    𝛽 𝛿 𝛼2 + 𝜎 2  = √ √   𝛿 𝛼2 + 𝜎 2  , if 𝛾 ≤ 1 . , otherwise (2.51) The optimization  in (2.51)  can be simplified one step further. One can easily show 𝛼𝛾 2 𝛼𝛾 𝛼𝜆 that − 2𝜏ℎ + 𝜆𝐹 𝜏ℎ , 𝜏ℎ is a non-increasing function of 𝛾 for 𝛾 > 0. Therefore, the (SPO) becomes equivalent to the following   √ p 𝛼𝛾 𝛼𝜆 𝛼𝜏ℎ 𝛼𝛾 2 2 2 − +𝜆·𝐹 , . (2.52) inf sup 𝛾 𝛿 𝛼 + 𝜎 − 𝛼≥0 0≤𝛾≤1 2 2𝜏ℎ 𝜏ℎ 𝜏ℎ 𝜏ℎ ≥0 The fact that the optimization in (2.52) predicts the squared error of (2.48), has been recently shown by the authors in [164]. That work only considers the square-root LASSO10, while here, we have (re)-derived the result as a corollary of the general Theorem 1. Heavy-tails In this section, we investigate instances where the noise distribution has unbounded moments. In the presence of (say) heavy-tailed noise, it is a common practice to use a loss function that grows to infinity no faster than linearly. This is also suggested by Assumption 1(a) (cf. (2.16) for the separable case), as has already been discussed. iid For a mere illustration, we assume z ∼ Cauchy(0, 1) and consider two examples of loss functions for which we show that Theorem 1 is applicable. 10 Note however, that [164] considers a more general measurement model than the one of the current paper, one that allows for nonlinearities. 32 LAD As a first example, consider the regularized-LAD estimator: 𝛽ˆ = arg min ky − X𝛽k1 + 𝜆 𝑓 (𝛽). 𝛽 (2.53) The loss function is separable, with ℓ(𝑣) = |𝑣|. Easily, for all 𝑐 ∈ R     E |ℓ+0 (𝑐𝐺 + 𝑍)| 2 = E |sign(𝑐𝐺 + 𝑍)| 2 = 1 < ∞, satisfying the assumption in (2.14). Also, E𝑍 2 is undefined, but, sup𝑣 ℓ(𝑣) |𝑣| = 1 < ∞, thus, (2.16) holds. Finally, | · | is not differentiable at zero satisfying the conditions of Lemma 4. With these, Theorem 2 is applicable. Huber-loss The Huber-loss function with parameter 𝜌 > 0 is defined as ℎ 𝜌 (𝑣) =  2   𝑣2  , |𝑣| ≤ 𝜌, 2   𝜌|𝑣| − 𝜌2  , otherwise. (2.54) Consider a regularized M-estimator with ℓ(𝑣) = ℎ 𝜌 (𝑣). We show here that this choice satisfies the Assumptions of Theorem 2. Indeed, for all 𝑐 ∈ R       E |ℓ+0 (𝑐𝐺 + 𝑍)| 2 ≤ E |𝑐𝐺 + 𝑍 | |𝑐𝐺 + 𝑍 | ≤ 𝜌 + E 𝜌 |𝑐𝐺 + 𝑍 | > 𝜌 < ∞, (2.55) satisfying the assumption in (2.14). Also, sup𝑣 ℓ(𝑣) |𝑣| = 𝜌 < ∞, thus, (2.16) holds. Finally, ℎ 𝜌 is differentiable with a strictly increasing derivative in the interval [−𝜌, 𝜌]. With these, Theorem 2 is applicable. Figure 2.3 illustrates the validity of the prediction via numerical simulations. Numerical Simulations We have performed a few numerical simulations on specific instances of M-estimators that were previously discussed in Section 2.1. Their purpose is to illustrate the validity of the prediction of Theorem 1, and of the remarks that followed as a consequence of it. Figure 2.1 . We consider the regularized LAD estimator of (2.53) under an iid sparse-Gaussian noise model. The unknown signal is also considered sparse, which leads to the natural choice of ℓ1 regularization, i.e. 𝑓 (𝛽) = k 𝛽k1 . Apart from the very close agreement of the theoretical prediction of Theorem 1 to the simulated data, the following facts are worth observing. 33 1 Gaussian Bernoulli Theory 0.9 0.8 δ = 0.7 1 kx̂ − x0 k22 n 0.7 0.6 δ = 1.2 0.5 0.4 0.3 0.2 0.1 0 0 0.5 1 1.5 2 2.5 3 λ Figure 2.1: Squared error of the 𝑙1 -Regularized LAD with Gaussian (◦) and Bernoulli () measurements as a function of the regularizer parameter 𝜆 for two different values of the normalized number √ iid of measurements, namely 𝛿 = 0.7 and 𝛿 = 1.2. Also, 𝛽0,𝑖 ∼ 𝑝 𝑥 (𝑥) = 0.9𝛿0 (𝑥) + 0.1𝜙(𝑥)/ 0.1 and 2 iid z 𝑗 ∼ 𝑝 𝑧 (𝑧) = 0.7𝛿0 (𝑧) + 0.3𝜙(𝑧) for 𝜙(𝑥) = √1 𝑒 −𝑥 /2 . For the simulations, we used 𝑝 = 768 and 2𝜋 the data were averaged over 5 independent realizations. 0.9 δ = 0.7 δ = 1.2 0.8 0.7 1 2 n kx̂ − x0 k2 0.6 LAD 0.5 0.4 LASSO 0.3 0.2 0.1 0 0 0.5 1 1.5 2 2.5 3 λ Figure 2.2: Comparing the squared error of the ℓ1 -Regularized LAD with the corresponding error of the LASSO. Both are plotted as functions of the regularizer parameter 𝜆, for two different values of the normalized measurements, namely 𝛿 = 0.7 and 𝛿 = 1.2. The noise and signal are iid sparse√ iid Gaussian as follows: 𝛽0,𝑖 ∼ 𝑝 𝑥 (𝑥) = 0.9𝛿0 (𝑥) + 0.1𝜙(𝑥)/ 0.1 and z 𝑗 ∼ 𝑝 𝑧 (𝑧) = 0.9𝛿0 (𝑧) + 0.1𝜙(𝑧) 2 with 𝜙(𝑥) = √1 𝑒 −𝑥 /2 . For the simulations, we used 𝑝 = 768 and the data were averaged over 5 2𝜋 independent realizations. 34 2.4 Simulation Theory 2.2 2 kx̂−x0 k22 n 1.8 1.6 1.4 1.2 1 0.8 0.6 1 2 3 4 5 6 7 8 9 10 λ Figure 2.3: Squared error of the ℓ1 -Regularized M-Estimator with Huber-loss as a function 1 kx̂ n x0 k22 √ iid of the regularizer parameter 𝜆. Here, 𝛿 = 0.7, 𝛽0 ∼ 𝑝 𝑥 (𝑥) = 0.9𝛿0 (𝑥) + 0.1𝜙(𝑥)/ 0.1 and 2 1 𝑝 𝑧 (𝑧) = 0.9𝛿(𝑧) + 0.1𝜂(𝑧) with 𝜙(𝑥) = √1 𝑒 −𝑥 /2 and 𝜂(𝑧) = 𝜋 (1+𝑧 2 ) . For the simulations, we used 2𝜋 𝑝 = 1024 and the data are averaged over 5 independent realizations. Theory Figure 2.4: Squared error of the ℓ1,2 -Regularized Lasso for group sparse signal composed of 512 blocks of size 3 each, as a function of the regularizer parameter 𝜆. Here, 𝛿 = 0.75, each block is iid zero with probability 0.95, otherwise its entries are i.i.d. N (0, 1) and z 𝑗 ∼ 𝑝 𝑧 (𝑧) = 0.3𝜙(𝑧) with 2 𝜙(𝑥) = √1 𝑒 −𝑥 /2 . The simulations are averaged over 10 independent realizations. 2𝜋 35 - When the number of measurements 𝑛 gets large enough, then, for an appropriate range of values of the regularizer parameter, the estimator is consistent, i.e. the unknown signal 𝛽0 is perfectly recovered. This is relevant to Remark 14 where we proved this to be the case for the closely related cone-constrained LAD estimator. For that, we were able to quantify how large 𝑛 should be as a function of the sparsities of the noise and of the signal, see (2.45). - The prediction of Theorem 1 remains accurate when the measurement matrix has entries iid Bernoulli ({±1}). This suggests that the error behavior (at least of this specific instant of M-estimator) undergoes some universality properties. Figure 2.2 . The model for both the noise and for the unknown signal here is the same as in Figure 2.1, i.e. both are iid sparse. We use ℓ1 -regularization, and, two different loss functions, namely, a least-absolute-deviations one and a least-squares one, corresponding to a LAD and a LASSO estimator, respectively. The figure aims to compare the performance of the two. Intuition suggests that the LAD is more appropriate for a sparse noise model, since ℓ1 promotes sparsity. This is indeed the case, in the sense that for good choices of the regularizer parameter 𝜆, the LAD outperforms by far the LASSO. However, it is worth observing that for a different and relatively big range of values of 𝜆, the LASSO performs better. This indicates the importance of the correct tuning of the regularizer parameter, to which the predictions of Theorem 1 can offer valuable guidelines and insights. Figure 2.3 . For this figure, we have assumed an ℓ1 -regularized estimator with Huber-loss ℓ(𝑣) = ℎ1 (𝑣) (see (2.54)). The noise is iid Cauchy(0, 1). In Section 2.1 it was shown that all the assumptions of Theorem 2 are satisfied in this setting. The figure, validates the prediction. To obtain the prediction we numerically solved the corresponding system of nonlinear equations (see (2.19)) using the efficient iterative scheme described in Remark 3. Figure 2.4 . We include this as an example of an M-estimator with non-separable loss function. For the plot, we use the square-root LASSO with ℓ1,2 -regularization. The analytical prediction was derived solving (2.52). 2.2 BER Analysis of the Box Relaxation for BPSK Signal Recovery The problem of recovering an unknown BPSK vector from a set of noise corrupted linearly related measurements arises in numerous applications, such as Massive MIMO [41, 121, 126, 193]. As a result, a large host of exact and heuristic optimiza- 36 tion algorithms have been proposed. Exact algorithms, such as sphere decoding and its variants, become computationally prohibitive as the problem dimension grows. Heuristic algorithms such as zero-forcing, MMSE, decision-feedback, etc., [71, 81, 83] have inferior performances that are often difficult to precisely characterize. One popular heuristic is the so called "Box Relaxation" which replaces the discrete set {±1}𝑛 with the convex set [−1, 1] 𝑛 [108, 162, 198]. This allows one to recover the signal via convex optimization followed by hard thresholding. Despite its popularity, very little is known about the performance of this method. In this section, we exactly characterize its bit-wise error probability in the regime of large dimensions and under Gaussian assumptions. Setup Our goal is to recover an 𝑛-dimensional BPSK vector x0 ∈ {±1}𝑛 from the noisy multiple-input multiple output (MIMO) relation y = Ax0 + z ∈ R𝑚 , where A ∈ R𝑚×𝑛 is the MIMO channel matrix (assumed to be known) and z ∈ R𝑚 is the noise vector. We assume that A has entries iid N (0, 1/𝑛) and z has entries iid N (0, 𝜎 2 ). The normalization is such that the reciprocal of the noise variance 𝜎 2 is equal to the Signal-to-Noise Ratio, i.e. SNR = 1/𝜎 2 . Our goal is to recover an 𝑛-dimensional BPSK vector x0 ∈ {±1}𝑛 11 from the noisy multiple-input multiple output (MIMO) relation y = Ax0 + z ∈ R𝑚 , where A ∈ R𝑚×𝑛 is the MIMO channel matrix (assumed to be known) and z ∈ R𝑚 is the noise vector. We assume that A has entries iid N (0, 1/𝑛) and z has entries iid N (0, 𝜎 2 ). The normalization is such that the reciprocal of the noise variance 𝜎 2 is equal to the Signal-to-Noise Ratio, i.e. SNR = 1/𝜎 2 . The Maximum-Likelihood (ML) decoder. The ML decoder which maximizes the probability of error (assuming the x0,𝑖 are equally likely) is given by minx∈{±1}𝑛 ky − Axk2 . Solving the above, is often computationally intractable, especially when 𝑛 is large, and therefore a variety of heuristics have been proposed (zero-forcing, mmse, decision-feedback, etc.) [185]. Box Relaxation Optimization. The heuristic we shall use, we refer to it as Box Relaxation Optimization (BRO). It consists of two steps. The first one involves solving a convex relaxation of the ML algorithm, where x ∈ {±1}𝑛 is relaxed to x ∈ [−1, 1] 𝑛 . The output of the optimization is hard-thresholded in the second step 11 For this sectoin, we are going to use the conventional notations in communications for consis- tency with related works. In these works, the dimension of the unknown signal is 𝑛 (instead of 𝑝), while the number of measuements is 𝑚 (instead of 𝑛) 37 to produce the final binary estimate. Formally, the algorithm outputs an estimate x∗ of x0 given as x̂ = arg min ky − Axk2 , −1≤x𝑖 ≤1 x∗ = sign( x̂), (2.56) where the sign function returns the sign of its input and acts element-wise on input vectors. Bit error probability. We evaluate the performance of the detection algorithm by the bit error probability 𝑃𝑒 , defined as the expectation of the Bit Error Rate 𝐵𝐸 𝑅 . Formally, 𝑛 1Õ 1{x∗𝑖 ≠x0,𝑖 } , 𝐵𝐸 𝑅 := 𝑛 𝑖=1 𝑛
1Õ 𝑃𝑒 := E [𝐵𝐸 𝑅 ] = Pr x𝑖∗ ≠ x0,𝑖 . 𝑛 𝑖=1 (2.57a) (2.57b) Our main result analyzes the 𝑃𝑒 of the BRO in (2.56). We assume a large-system limit where 𝑚, 𝑛 → ∞ at a proportional rate 𝛿. The SNR is assumed constant; in particular, it does not scale with 𝑛. Let 𝑄(·) denote the Q-function associated with 2 the standard normal density 𝑝(ℎ) = √1 e−ℎ /2 . 2𝜋 Theorem 3 (𝑃𝑒 of the BRO) Let 𝑃𝑒 denote the bit error probability of the detection scheme in (2.56) for some fixed but unknown BPSK signal x0 ∈ {±1}𝑛 . For constant SNR and 𝑚𝑛 → 𝛿 ∈ ( 21 , ∞), it holds: lim 𝑃𝑒 = 𝑄(1/𝜏∗ ), 𝑛→∞ where 𝜏∗ is the unique solution to   2 ∫  𝜏 1 1/SNR 𝜏 ∞ 2 min 𝛿− + − ℎ+ 𝑝(ℎ) 𝑑ℎ. 𝜏>0 2 2 2𝜏 2 𝜏2 𝜏 (2.58) Theorem 3 derives a precise formula for the bit error probability of the (BRO). The formula involves solving a convex and deterministic minimization problem in (2.58). 38 10 0 = 0.7 10 -1 10 -2 Pe =1 10 -3 10 -4 Simulation Theory 10 -5 -4 -2 0 2 4 6 8 10 12 14 10 log10 (SN R) Figure 2.5: BER Performance of the Boxed Relaxation: 𝑃𝑒 as a function of SNR for different values of the ration 𝛿 = d𝑚/𝑛e. The theoretical prediction follows from Theorem 3. For the simulations, we used 𝑛 = 512. The data are averages over 20 independent realizations of the channel matrix and of the noise vector for each value of the SNR. Computing 𝜏∗ . It can be shown that the objective function of (2.58) is strictly convex when 𝛿 > 21 . When 𝛿 < 21 , it is well known that even the noiseless box relaxation fails [40]. (In fact, 𝛿 = 12 is the recovery threshold for this convex relaxation.) Thus, (2.58) has a unique solution 𝜏∗ . Observe that the problem parameters 𝛿 and SNR appear explicitly in (2.58); naturally then 𝜏∗ is indeed a function of those. The minimization in (2.58) can be efficiently solved numerically. In addition, owing to the strict convexity of the objective function, 𝜏∗ can be equivalently expressed as the unique solution to the corresponding first order optimality conditions. Numerical illustration. Figure 2.5 illustrates the accuracy of the prediction of Theorem 3. Note that although the theorem requires 𝑛 → ∞, the prediction is already accurate for 𝑛 ranging on a few hundreds. p 𝑃𝑒 at high-SNR. It can be shown that when SNR  1, then 𝜏∗ = 1/ (𝛿 − 1/2)𝑆𝑁 𝑅. This can be intuitively understood as follows: at high-SNR, we expect 𝜏∗ to be going to zero (correspondingly 𝑃𝑒 to be small). When this is the case, the last term in (2.58) is negligible; then, 𝜏∗ is the solution to min𝜏>0 2𝜏 𝛿 − 21 + 1/SNR 2𝜏 which gives the derided result. Hence, for SNR  1, 39 10 0 = 0.7 10 -1 =1 10 -2 10 -3 3 dB Pe 10 -4 10 -5 10 -6 10 -7 10 BRO (Thm. 2.1) BRO (High-SNR) MFB -8 10 -9 5 6 7 8 9 10 11 12 13 14 15 10 log10 (SN R) Figure 2.6: Bit error probability of the Box Relaxation Optimization (BRO) in (2.56) in comparison to the Matched Filter Bound (MFB) for 𝛿 = 0.7 (dashed lines) and 𝛿 = 1 (solid lines). The red curves follow the formula of Thm. 3, the green ones correspond to (2.59), and, 𝑃𝑒𝑀 𝐹 𝐵 of (2.60) is in blue. p lim 𝑃𝑒 ≈ 𝑄( (𝛿 − 1/2) · SNR). 𝑛→∞ (2.59) In Figure 2.6 we have plotted this high-SNR expression for the log10 (𝑃𝑒 ) vs its exact value as predicted by Theorem 3. It is interesting to observe that the former is actually a very good approximation to the latter even for small practical values of SNR. The range of SNR values for which the approximation is valid becomes larger with increasing 𝛿. Heuristically, for 𝛿 > 0.7 the expression in (2.59) is a good proxy for the true probability of error at practical SNR values. Comparison to the matched filter bound. Theorem 3 gives us a handle on the 𝑃𝑒 of BRO in (2.56) and therefore allows to evaluate its practical performance. Here, we compare the performance to an idealistic case, where all 𝑛−1, but 1, bits of x0 are known to us. As is customary in the field, we refer to the bit error probability of this case as the matched filter bound MFB and denote it by 𝑃𝑒𝑀 𝐹 𝐵 . The MFB corresponds to the probability of error in detecting (say) x0,𝑛 ∈ {±1} from: ỹ = x0,𝑛 a𝑛 + z, where Í𝑛−1 ỹ = y − 𝑖=1 x0,𝑖 a𝑖 is assumed known, and, a𝑖 denotes the 𝑖 𝑡ℎ column of A. The ML estimate is just the sign of the projection of the vector ỹ to the direction of a𝑛 . Without loss of generality assume that x0,𝑛 = 1. Then, the output of the matched 40 ˜ where 𝑋˜ = ka𝑛 k 2 + 𝜎 2 𝜈, where 𝜈 ∼ N (0, 1). When 𝑛 → ∞, filter becomes sign( 𝑋), 𝑃 ka𝑛 k 2 − → 𝛿12 . Hence, with probability one, √ lim 𝑃𝑒𝑀 𝐹 𝐵 = lim P( 𝑋˜ < 0) = 𝑄( 𝛿 · SNR). (2.60) 𝑛→∞ 𝑛→∞ A direct comparison of (2.60) to (2.59) shows that at high-SNR, the performance of 𝛿 the BRO is 10 log10 𝛿−1/2 dB off that of the MFB. In particular, in the square case (𝛿 = 1), where the number of receive and transmit antennas are the same, the BRO is 3 dB off the MFB. When the number of receive antennas is much larger, i.e. when 𝛿 → ∞, then the performance of the BRO approaches the MFB. Next, we would like to apply our main Theorem 1 to binary classification problems. 2.3 Binary Classification Classical results in logistic regression mainly concern the regime where the sample size, 𝑛, is overwhelmingly larger than the feature dimension 𝑝. It can be shown that in the limit of large samples when 𝑝 is fixed and 𝑛 → ∞, the maximum likelihood estimator provides an efficient estimate of the underlying parameter, i.e., an unbiased estimate with covariance matrix approaching the inverse of the Fisher information [103, 183]. However, in most modern applications in data science, the datasets often have a huge number of features, and therefore, the assumption 𝑛𝑝  1 is not valid. Sur and Candes [37, 157, 159] have recently studied the performance of the maximum likeliood estimator for logistic regression in the regime where 𝑛 is proportional to 𝑝. Their findings challenge the conventional wisdom, as they have shown that in the linear asymptotic regime the maximum likelikehood estimate is not even unbiased. Their analysis provides the precise performance of the maximum likelihood estimator. There have been many studies in the literature on the performance of regularized (penalized) logistic regression, where a regularizer is added to the negative loglikelihood function (a partial list includes [30, 94, 182]). These studies often require the underlying parameter to be heavily structured. For example, if the parameters are sparse the sparsity is taken to be 𝑜( 𝑝). Furthermore, they provide orderwise bounds on the performance but do not give a precise characterization of the quality of the resulting estimate. A major advantage of adding a regularization term is that it allows for recovery of the parameter vector even in regimes where the maximum likelihood estimate does not exist (due to an insufficient number of observations.) 𝑃 12 We use − → to denote convergence in probability with 𝑛 → ∞. 41 In this section, we study regularized logistic regression (RLR) for parameter estimation in high-dimensional logistic models. Inspired by recent advances in the performance analysis of M-estimators for linear models [56, 66, 165], we precisely characterize the assymptotic performance of the RLR estimate. Our characterization is through a system of six nonlinear equations in six unknowns, through whose solution all locally-Lipschitz performance measures such as the mean, mean-squared error, probability of support recovery, etc., can be determined. In the special case when the regularization term is absent, our 6 nonlinear equations reduce to the 3 nonlinear equations reported in [157]. When the regularizer is quadratic in parameters, the 6 equations also simplifies to 3. When the regularizer is the ℓ1 norm, which corresponds to the popular sparse logistic regression [98, 99], our equations can be expressed in terms of 𝑞-functions, and quantities such as the probability of correct support recovery can be explicitly computed. Numerous numerical simulations validate the theoretical findings across a range of problem settings. To the extent of our knowledge, this is the first work that precisely characterizes the performance of the regularized logistic regression in high dimensions. Mathematical Setup Consider the scenario when the observations 𝑦𝑖 are binary in the following form, 𝑦𝑖 =    +1  w.p. 𝜌(x𝑖T 𝛽0 )   −1  w.p. 1 − 𝜌(x𝑖T 𝛽0 ) (2.61) where 𝜌 : R → [0, 1]. In practice, the function 𝜌 is often unknown, but choices like Hyperbolic tangent or tangent inverse functions are made for recovery of the separating hyper plane 𝛽0 . In this case, the link function g is 𝑔(x) = Sign(𝜌(x) − 𝜖) , , 𝜖𝑖 ∼ Unif(0, 1) . where 𝜖 has a uniform distribution between 0 and 1. It’s not hard to check that the output of the link function will be 1 with probability 𝜌(x), and −1 with probability 1 − 𝜌(x). The performance of the convex estimator (2.2) has been analyzed for a wide variety of loss functions and regularizers. Schur et al. [158] analyzed this optimization for the case of logistic loss function, 42 where there is no regularization function 𝑓 . The logistic loss function is defined as L (X𝛽, y) = 𝑛 Õ   −𝑙𝑜𝑔 𝑒 x𝑖 𝛽 + 𝑒 −x𝑖 𝛽 + 𝑦𝑖 x𝑖 𝛽 . (2.62) 𝑖=1 It’s not hard to see that this loss function is the maximum likelihood estimator of 𝛽0 in model (2.61), where the function 𝜌 is the Hyperbolic tangent function which is 𝜌(𝑥) = 𝑒𝑥 . 𝑒 𝑥 + 𝑒 −𝑥 (2.63) Salehi et al. [143], analyzed the same problem for the regularized case in their work using CGMT framework. One interesting result that is not derived in these series of works is the performance analysis of Support Vector Machine and the Perceptron method. In support vector machine, the loss function for classification is define as L 𝑆𝑉 𝑀 (X𝛽, y) = 𝑛 Õ   T max 0, 1 − 𝑦𝑖 x𝑖 𝛽 . (2.64) 𝑖=1 For the Perceptron method, the loss function is defined as L 𝑃𝑒𝑟𝑐𝑒 𝑝𝑡𝑟𝑜𝑛 (X𝛽, y) = 𝑛 Õ  T  max 0, −𝑦𝑖 x𝑖 𝛽 . (2.65) 𝑖=1 Performance Analysis of Logistic Regression Assume we have 𝑛 samples from a logistic model with parameter 𝛽∗ ∈ R 𝑝 . Let 𝑛 denote the set of samples (a.k.a. the training data), where for D = {(x𝑖 , 𝑦𝑖 )}𝑖=1 𝑖 = 1, 2, . . . , 𝑛, x𝑖 ∈ R 𝑝 is the feature vector and the label 𝑦𝑖 ∈ {0, 1} is a Bernouli random variable with, P[𝑦𝑖 = 1|x𝑖 ] = 𝜌0 (x𝑇𝑖 𝛽∗ ) , for 𝑖 = 1, 2, . . . , 𝑛 , (2.66) 𝑡 𝑒 where 𝜌0 (𝑡) := 1+𝑒 𝑡 is the standard logistic function. The goal is to compute an estimate for 𝛽∗ from the training data D. The maximum likelihood estimator, 𝛽ˆ 𝑀 𝐿 , is defined as, 𝛽ˆ 𝑀 𝐿 = arg max𝑝 𝑛 Ö 𝛽∈R P 𝛽 (𝑦𝑖 |x𝑖 ) = arg max𝑝 𝑇 𝑛 Ö 𝑒 𝑦𝑖 (x𝑖 𝛽) x𝑇 𝛽 𝑖=1 1 + 𝑒 𝑖 𝑛 Õ = arg min𝑝 𝜌(x𝑇𝑖 𝛽) − 𝑦𝑖 (x𝑇𝑖 𝛽) . 𝛽∈R 𝑖=1 𝛽∈R 𝑖=1 (2.67) 43 Where 𝜌(𝑡) := log(1+𝑒 𝑡 ) is the link function which has the standard logistic function as its derivative. The last optimization is simply minimization over the negative loglikelihood. This is a convex optimization program as the log-likelihood is concave with respect to 𝛽. In many interesting settings the underlying parameter possesses cerain structure(s) (sparse, low-rank, finite-alphabet, etc.). In order to exploit this structure we assume 𝑓 : R 𝑝 → R is a convex function that measures the (so-called) "complexity" of the structured solution. We fit this model by the regularized maximum (binomial) likelihood defined as follows, 1 𝛽ˆ = arg min𝑝 · 𝛽∈R 𝑛 𝑛 Õ  𝜆 𝜌(x𝑇𝑖 𝛽) − 𝑦𝑖 (x𝑇𝑖 𝛽) + 𝑓 (𝛽) . 𝑝 𝑖=1 (2.68) Here, 𝜆 ∈ R+ is the regularization parameter that must be tuned properly. In this section, we study the linear asymptotic regime in which the problem dimensions 𝑝, 𝑛 grow to infinity at a proportional rate, 𝛿 := 𝑛𝑝 > 0. Our main result characterizes ∗ || √ . For the performance of 𝛽ˆ in terms of the ratio, 𝛿, and the signal strength, 𝜅 = ||𝛽 𝑝 Í our analysis we assume that the regularizer 𝑓 (·) is separable, 𝑓 (w) = 𝑖 𝑓˜(𝑤 𝑖 ), and i.i.d. 𝑛 the data points are drawn independently from the Gaussian distribution, {x𝑖 }𝑖=1 ∼ 1 ∗ N (0, 𝑝 I 𝑝 ). We further assume that the entries of 𝛽 are drawn from a distribution Π. Our main result characterizes the performance of the resulting estimator through the solution of a system of six nonlinear equations with six unknowns. In particular, we use the solution to compute some common descriptive statistics of the estimate, such as the mean and the variance. As we will see in Theorem 4, given the signal strength 𝜅, and the ratio 𝛿, the asymptotic performance of RLR is characterized by the solution to the following system of nonlinear equations with six unknowns (𝛼, 𝜎, 𝛾, 𝜃, 𝜏, 𝑟). 44 
 𝑟   𝜅 2 𝛼 = E 𝛽 Prox𝜆𝜎𝜏 𝑓˜(·) 𝜎𝜏(𝜃 𝛽 + √ 𝑍) ,    𝛿    
  𝑟 1   E 𝑍 Prox 𝜎𝜏(𝜃 𝛽 + 𝑍) 𝛾 = , √ √ ˜  𝜆𝜎𝜏 𝑓 (·)   𝑟 𝛿 𝛿   
2 𝑟  2 2 2   𝑍) , 𝜅 𝛼 + 𝜎 = E Prox 𝜎𝜏(𝜃 𝛽 + √ ˜   𝜆𝜎𝜏 𝑓 (·) 𝛿 
2 2  0  2  𝛾 = E 𝜌 (−𝜅𝑍 ) 𝜅𝛼𝑍 + 𝜎𝑍 − Prox (𝜅𝛼𝑍 + 𝜎𝑍 ) ,  1 1 2 1 2 𝛾𝜌(·)   𝑟2    
  𝜃𝛾 = −2 E 𝜌00 (−𝜅𝑍1 )Prox𝛾𝜌(·) 𝜅𝛼𝑍1 + 𝜎𝑍2 ,        2𝜌0 (−𝜅𝑍1 ) 𝛾  
= E . 1 −   𝜎𝜏 1 + 𝛾𝜌00 Prox𝛾𝜌(·) (𝜅𝛼𝑍1 + 𝜎𝑍2 )  (2.69) Here 𝑍, 𝑍1 , 𝑍2 are standard normal variables, and 𝛽 ∼ Π, where Π denotes the distribution on the entries of 𝛽∗ . The following remarks provide some insights on solving the nonlinear system. Remark 16 (Proximal Operators) It is worth noting that the equations in (2.69) include the expectation of functionals of two proximal operators. The first three equations are in terms of Prox 𝑓˜(·) , which can be computed explicitly for most widely used regularizers. For instance, in ℓ1 -regularization, the proximal operator is the 𝑥 (|𝑥| − 𝑡)+ . The remaining well-known shrinkage function defined as 𝜂(𝑥, 𝑡) := |𝑥| equations depend on computing the proximal operator of the link function 𝜌(·). For 𝑥 ∈ R, Prox𝑡 𝜌(·) (𝑥) is the unique solution of 𝑧 + 𝑡 𝜌0 (𝑧) = 𝑥. Remark 17 (Numerical Evaluation) Define v := [𝛼, 𝜎, 𝛾, 𝜃, 𝜏, 𝑟] 𝑇 as the vector of unknonws. The nonlinear system (2.69) can be reformulated as v = 𝑆(v) for a properly defined 𝑆 : R6 → R6 . We have empirically observed in our numerical simulations that a fixed-point iterative method, v𝑡+1 = 𝑆(v𝑡 ), converges to v∗ , such that v∗ = 𝑆(v∗ ). We are now able to present our main result. Theorem 4 below describes the average ˆ the solution of the RLR. The derived expression is in behavior of the entries of 𝛽, ¯ 𝜏, terms of the solution of the nonlinear system (2.69), denoted by ( 𝛼, ¯ 𝜎, ¯ 𝛾, ¯ 𝜃, ¯ 𝑟). ¯ An informal statement of our result is that as 𝑛 → ∞, the entries of 𝛽ˆ converge as follows, 𝑑 𝛽ˆ 𝑗 → Γ(𝛽∗𝑗 , 𝑍) , for 𝑗 = 1, 2, . . . , 𝑝 , (2.70) 45 where 𝑍 is a standard normal random variable, and Γ : R2 → R is defined as,
¯ + √𝑟¯ 𝑑) . Γ(𝑐, 𝑑) := Prox𝜆𝜎¯ 𝜏¯ 𝑓˜(·) 𝜎 ¯ 𝜏( ¯ 𝜃𝑐 𝛿 (2.71) In other words, the RLR solution has the same behavior as applying the proximal operator on the "perturbed signal", i.e., the true signal added with a Gaussian noise. Theorem 4 Consider the optimization program (2.68), where for 𝑖 = 1, 2, . . . , 𝑛, x𝑖 has the multivariate Gaussian distribution N (0, 1𝑝 I 𝑝 ), and 𝑦𝑖 = 𝐵𝑒𝑟 (x𝑇𝑖 𝛽∗ ), and the entries of 𝛽∗ are drawn independently from a distribution Π. Assume the parameters 𝛿, 𝜅, and 𝜆 are such that the nonlinear system (2.69) has a unique ¯ 𝜏, solution ( 𝛼, ¯ 𝜎, ¯ 𝛾, ¯ 𝜃, ¯ 𝑟). ¯ Then, as 𝑝 → ∞, for any locally-Lipschitz13 function Ψ : R × R → R , we have, 𝑝 
 1Õ P Ψ( 𝛽ˆ 𝑗 , 𝛽∗𝑗 ) −→ E Ψ Γ(𝛽, 𝑍), 𝛽 , 𝑝 𝑗=1 (2.72) where 𝑍 ∼ N (0, 1), 𝛽 ∼ Π is independent of 𝑍, and the function Γ(·, ·) is defined in (2.71). Correlation and variance of the RLR estimate As the first application of Theorem 4 we compute common descriptive statistics of ˆ In the following corollaries, we establish that the parametrs 𝛼, the estimate 𝛽. ¯ and 𝜎 ¯ in (2.69) correspond to the correlation and the mean-squared error of the resulting estimate. 𝑃 Corollary 1 As 𝑝 → ∞, ||𝛽1∗ ||2 𝛽ˆ𝑇 𝛽∗ −→ 𝛼¯ . Proof 1 Recall that ||𝛽∗ || 2 = 𝑝𝜅 2 . Applying Theorem 4 with Ψ(𝑢, 𝑣) = 𝑢𝑣 gives, 𝑝
 1 Õ ˆ ∗ 𝑃 1  𝑟¯ 1 ˆ𝑇 ∗ ¯ 𝛽 𝛽 = 𝛽 𝛽 −→ E 𝛽 Prox 𝜎 ¯ 𝜏( ¯ 𝜃 𝛽 + 𝑍) = 𝛼¯ , √ ˜ 𝑗 𝑗 𝜆𝜎 ¯ 𝜏¯ 𝑓 (·) ||𝛽∗ || 2 𝜅 2 𝑝 𝑗=1 𝜅2 𝛿 (2.73) where the last equality is derived from the first equation in the nonlinear sys¯ 𝜏, tem (2.69), along with the fact that ( 𝛼, ¯ 𝜎, ¯ 𝛾, ¯ 𝜃, ¯ 𝑟) ¯ is a solution to this system. 13 A function Φ : R 𝑑 → R is said to be locally-Lipschitz if,  𝑑 ∀𝑀 > 0, ∃𝐿 𝑀 ≥ 0, such that ∀x, y ∈ − 𝑀, +𝑀 : |Φ(x) − Φ(y)| ≤ 𝐿 𝑀 ||x − y|| . 46 Corollary 1 states that upon centering 𝛽ˆ around 𝛼𝛽 ¯ ∗ , it becomes decorrelated from ˆ 𝛽∗ . Therefore, we define a new estimate 𝛽˜ := 𝛼𝛽¯ and compute its mean-squared error in the following corollary. 𝑃 Corollary 2 As 𝑝 → ∞, 1𝑝 || 𝛽˜ − 𝛽∗ || 2 −→ 𝜎𝛼¯¯ 2 . 2 Proof 2 We appeal to Theorem 4 with Ψ(𝑢, 𝑣) = (𝑢 − 𝛼𝑣) ¯ 2,

2 𝜎 1 1 ˆ 1  𝑟¯ ¯2 1 ˜ ∗ 2 ∗ 2
𝑃 ¯ || 𝛽−𝛽 || = 2 || 𝛽−𝛼𝛽 ¯ || −→ 2 E Prox𝜆𝜎¯ 𝜏¯ 𝑓˜(·) 𝜎 ¯ 𝜏( ¯ 𝜃 𝛽+ √ 𝑍) −𝛼𝛽 ¯ = 2, 𝑝 𝛼¯ 𝑝 𝛼¯ 𝛼¯ 𝛿 (2.74) where the last equality is derived from the third equation in the nonlinear system (2.69) together with the result of Corollary 1. In the next two sections, we investigate other properties of the estimate 𝛽ˆ under ℓ1 and ℓ2 regularization. RLR with ℓ22 -regularization The ℓ2 norm regularization is commonly used in machine learning applications to stabilize the model. Adding this regularization would simply shrink all the parameters toward the origin and hence decrease the variance of the resulting model. Here, we provide a precise performance analysis of the RLR with ℓ22 -regularization, i.e., 𝑝 𝑛  𝜆 Õ 1 Õ 𝑇 𝑇 ˆ (2.75) 𝜌(x𝑖 𝛽) − 𝑦𝑖 (x𝑖 𝛽) + 𝛽𝑖2 . 𝛽 = arg min𝑝 · 𝛽∈R 𝑛 2𝑝 𝑖=1 𝑖=1 To analyze (2.75), we use the result of Theorem 4. It can be shown that in the ¯ 𝜏, nonlinear system (2.69), 𝜃, ¯ 𝑟¯ can be derived explicitely from solving the first three equations. This is due to the fact that the proximal operator of 𝑓˜(·) = 21 (·) 2 can be expressed in the following closed-form, 1 1 𝑥 Prox𝑡 𝑓˜(·) (𝑥) = arg min (𝑦 − 𝑥) 2 + 𝑦 2 = . (2.76) 𝑦∈R 2𝑡 2 1+𝑡 This indicates that the proximal operator in this case is just a simple rescaling. Substituting (2.76) in the nonlinear system (2.69), we can rewrite the first three equations as follows, 𝛼    𝜃= ,   𝛾𝛿    𝛿𝛾  
, 𝜏= (2.77) 𝜎 1 − 𝜆𝛿𝛾    𝜎    𝑟= √ .   𝛾 𝛿  47 (a) (b) (c) Figure 2.7: The performance of the regularized logistic regression under ℓ22 penalty (a) the 2 correlation factor 𝛼¯ (b) the variance 𝜎 ¯ 2 , and (c) the mean-squared error 𝑝1 || 𝛽ˆ − 𝛽∗ || . The dashed lines depict the theoretical result derived from Theorem 5, and the dots are the result of empirical simulations. The empirical results is the average over 100 independent trials with 𝑝 = 250 and 𝜅 = 1 . Therefore we can state the following Theorem for ℓ22 -regularization: Theorem 5 Consider the optimization (2.75) with parameters 𝜅, 𝛿, and 𝛾, and the same assumptions as in Theorem 4. As 𝑝 → ∞, for any locally-Lipschitz function Ψ(·, ·), the following convergence holds, 𝑝 
 1Õ P Ψ( 𝛽ˆ 𝑗 − 𝛼𝛽 ¯ ∗𝑗 , 𝛽∗𝑗 ) −→ E Ψ 𝜎𝑍, ¯ 𝛽 , 𝑝 𝑗=1 (2.78) where 𝑍 is standard normal, 𝛽 ∼ Π, and 𝛼, ¯𝜎 ¯ are the unique solutions to the following nonlinear system of equations,                  
2 𝜎2 = E 𝜌0 (−𝜅𝑍1 ) 𝜅𝛼𝑍1 + 𝜎𝑍2 − Prox𝛾𝜌(·) (𝜅𝛼𝑍1 + 𝜎𝑍2 ) , 2𝛿 
 𝛼 − = E 𝜌00 (−𝜅𝑍1 )Prox𝛾𝜌(·) 𝜅𝛼𝑍1 + 𝜎𝑍2 , 2𝛿   1 2𝜌0 (−𝜅𝑍1 )
. 1 − + 𝜆𝛾 = E 00 𝛿 1 + 𝛾𝜌 Prox𝛾𝜌(·) (𝜅𝛼𝑍1 + 𝜎𝑍2 ) (2.79) 48 The proof is deferred to the Appendix. Theorem 5 states that upon centering the ˆ it becomes decorrelated from 𝛽∗ and the distribution of the entries estimate 𝛽, approach a zero-mean Gaussian distribution with variance 𝜎 ¯ 2. Figure 2.7 depicts the performance of the regularized estimate for different values of 𝜆. As observed in the figure, increasing the value of 𝜆 reduces the correlation factor 𝛼¯ (Figure 2.7a) and the variance 𝜎 ¯ 2 (Figure 2.7b). Figure 2.7c shows the meansquared-error of the estimate as a function of 𝜆 . It indicates that for different values of 𝛿 there exist an optimal value 𝜆 opt that achieves the minimum mean-squared error. Sparse Logistic Regression In this section we study the performance of our estimate when the regularizer is the ℓ1 norm. In modern machine learning applications the number of features, 𝑝, is often overwhelmingly large. Therefore, to avoid overfitting one typically needs to perform feature selection, that is, to exclude irrelevant variables from the regression model [89]. Adding an ℓ1 penalty to the loss function is the most popular approach for feature selection. As a natural consequence of the result of Theorem 4, we study the performance of RLR with ℓ1 regularizer (referred to as "sparse LR") and evaluate its success in recovery of the sparse signals. In Section 2.3, we extend our general analysis to the case of sparse LR. In other words, we will precisely analyze the performance of the solution of the following optimization, 1 𝛽ˆ = arg min𝑝 · 𝛽∈R 𝑛 𝑛 Õ  𝜆 𝜌(x𝑇𝑖 𝛽) − 𝑦𝑖 (x𝑇𝑖 𝛽) + ||𝛽||1 . 𝑝 𝑖=1 (2.80) In Section 2.3, we explicitly describe the expectations in the nonlinear system (2.69) using two 𝑞-functions14. In Section 2.3, we analyze the support recovery in the resulting estimate and show that the two 𝑞-functions represent the probability of on and off support recovery. Convergence behavior of sparse LR For our analysis in this section, we assume each entry 𝛽𝑖∗ , for 𝑖 = 1, . . . , 𝑝, is sampled i.i.d. from a distribution, Π(𝛽) = (1 − 𝑠) · 𝛿0 (𝛽) + 𝑠 · 𝜙( √𝛽𝜅 )
𝑠 √𝜅 𝑠 , 14 The 𝑞-function is the tail distribution of the standard normal r.v. ∫ ∞ 𝑒−𝑥2 /2 √ 𝑑𝑥 . 𝑡 2𝜋 (2.81) defined as, 𝑄(𝑡) := 49 (a) (b) (c) Figure 2.8: The performance of the regularized logistic regression under ℓ1 penalty (a) the 2 correlation factor 𝛼¯ (b) the variance 𝜎 ¯ 2 , and (c) the mean-squared error 𝑝1 || 𝛽ˆ − 𝛽∗ || . The dashed lines are the theoretical result derived from Theorem 4, and the dots are the result of empirical simulations. For the numerical simulations, the result is the average over 100 independent trials with 𝑝 = 250 and 𝜅 = 1 . −𝑡 2 /2 where 𝑠 ∈ (0, 1) is the sparsity factor, 𝜙(𝑡) := 𝑒√ is the density of the standard 2𝜋 normal distribution, and 𝛿0 (·) is the Dirac delta function. In other words, entries of 𝛽∗ are zero with probability 1 − 𝑠, and the non-zero entries have a Gaussian distribution with appropriately defined variance. Although our analysis can be extended further, here we only present the result for a Gaussian distribution on the non-zero entries. The proximal operator of 𝑓˜(·) = | · | is the soft-thresholding 𝑥 (𝑥−𝑡)+ . Therefore, we are able to explicitly compute operator defined as, 𝜂(𝑥, 𝑡) = |𝑥| the expectations with respect to 𝑓˜(·) in the nonlinear system (2.69). To streamline the representation, we define the following two proxies, 𝑡1 = q 𝜆 𝑟2 𝜃 2 𝜅2 𝛿 + 𝑠 , 𝑡2 = 𝜆 . √𝑟 𝛿 (2.82) In the next section, we provide an interpretation for 𝑡1 and 𝑡 2 . In particular, we will show that 𝑄( 𝑡¯1 ), and 𝑄( 𝑡¯2 ) are related to the probabilities of on and off support 50 recovery. We can rewrite the first three equations in (2.69) as follows, 𝛼 = 𝜃 · 𝑄(𝑡1 ) , 2𝜎𝜏

𝛿𝛾 = 𝑠 · 𝑄 𝑡1 + (1 − 𝑠) · 𝑄 𝑡2 , 2𝜎𝜏   2 𝛼2 + 𝜎 2 
 𝛿𝛾𝜆2 𝛾𝑟 2 𝜅 𝜙(𝑡1 ) 𝜙(𝑡2 )   = + + 𝜅 2 𝜃 2 · 𝑄 𝑡1 − 𝜆2 (𝑠 · + (1 − 𝑠) · ).  2 2 2𝜎𝜏 2𝜎𝜏 𝑡1 𝑡2 2𝜎 𝜏  (2.83) Appending the three equations in (2.83) to the last three equations in (2.69) gives the nonlinear system for sparse LR. Upon solving these equations, we can use the result of Theorem 4 to compute various performance measures on the estimate ˆ Figure 2.8 shows the performance of our estimate as a function of 𝜆. It can 𝛽. be seen that the bound derived from our theoretical result matches the empirical simulations. Also, it can be inferrred from Figure 2.8c that the optimal value of 𝜆 (𝜆 opt that achieves the minimum mean-squared error) is a decreasing function of 𝛿.          Support recovery In this section, we study the support recovery in sparse LR. As mentioned earlier, sparse LR is often used when the underlying paramter has few non-zero entries. We define the support of 𝛽∗ as Ω := { 𝑗 |1 ≤ 𝑗 ≤ 𝑝, 𝛽∗𝑗 ≠ 0}. Here, we would like to compute the probability of success in recovery of the support of 𝛽∗ . Let 𝛽ˆ denote the solution of the optimization (2.80). We fix the value 𝜖 > 0 as a hard-threshold based on which we decide whether an entry is on the support or not. ˆ In other words, we form the following set as our estimate of the support given 𝛽, Ω̂ = { 𝑗 |1 ≤ 𝑗 ≤ 𝑝, | 𝛽ˆ 𝑗 | > 𝜖 } (2.84) In order to evaluate the success in support recovery, we define the following two error measures, 𝐸 1 (𝜖) = Prob{ 𝑗 ∈ Ω̂| 𝑗 ∉ Ω} , 𝐸 2 (𝜖) = Prob{ 𝑗 ∉ Ω̂| 𝑗 ∈ Ω} . (2.85) In our estimation, 𝐸 1 represents the probability of false alarm, and 𝐸 2 is the probability of misdetection of an entry of the support. The following lemma indicates the asymptotic behavior of both errors as 𝜖 approaches zero . Lemma 5 (Support Recovery) Let 𝛽ˆ be the solution to the optimization (2.80), and the entries of 𝛽∗ have distribution Π defined in (2.81). Assume 𝜆 is chosen such that ¯ 𝜏, the nonlinear system (2.69) has a unique solution ( 𝛼, ¯ 𝜎, ¯ 𝛾, ¯ 𝜃, ¯ 𝑟). ¯ As 𝑝 → ∞ we 51 (a) (b) Figure 2.9: The support recovery in the regularized logistic regression with ℓ1 penalty for (a) 𝐸 1 : the probability of false detection, (b) 𝐸 2 : the probability of missing an entry of the support. The dashed lines are the theoretical results derived from Lemma 5, and the dots are the result of empirical simulations. For the numerical simulations, the result is the average over 100 independent trials with 𝑝 = 250 and 𝜅 = 1 and 𝜖 = 0.001 . have,
𝜆 𝑃 lim 𝐸 1 (𝜖) −→ 2 𝑄 𝑡¯1 where, 𝑡¯1 = 𝑟¯ , and, √ 𝜖↓0 𝛿 𝑃 lim 𝐸 2 (𝜖) −→ 1 − 2 𝑄 𝑡¯2 where, 𝑡¯2 = q 𝜖↓0 2.4
(2.86) 𝜆 . 𝑟¯2 𝜃¯2 𝜅 2 𝛿 + 𝑠 Generalized Margin Maximizers Machine learning models have been very successful in many applications, ranging from spam detection, face and pattern recognition, to the analysis of genome sequencing and financial markets. However, despite this indisputable success, our knowledge on why the various machine learning methods exhibit the performances they do is still at a very early stage. To make this gap between the theory and the practice narrower, researchers have recently begun to revisit simple machine learning models with the hope that understanding their performance will lead the way to understanding the performance of more complex machine learning methods. More specifically, studies on the performance of diffrent classifiers for binary classification dates back to the seminal work of Vapnik in the 1980’s [184]. In an effort to find the ”optimal” hyperplane that separates the data, he presented an upper bound on the test error which is inversely proportional to the margin (minimum distance of the datapoints to the separating hyperplane), and concluded that the max-margin classifier is indeed the desired classifier. It has also been observed that to construct such optimal hyperplanes one only has to take into account a small amount of the 52 training data, the so-called support vectors [46]. In this section, we challenge the conventional wisdom by showing that when the underlying parameter has certain structure one can come up with classifiers that outperform the max-margin classifier. We introduce the Generalized Margin Maximizer (GMM) which takes into account the structure of the underlying parameter as well as the minimimum distance of the datapoints to the separating hyperplane. We provide sharp asymptotic results on various performance measures (such as the generalization error) of GMM and show that an appropriate choice of the potential function can in fact improve the resulting estimator. Prior work There have been many recent attempts to understand the generalization behavior of simple machine learning models [16, 21, 84, 114, 197]. Most of these studies focus on the least-squares/ridge regression, where the loss function is the squared ℓ2 -norm, and derive sharp asymptotics on the performance of the estimator. In particular, in [84, 97] the authors have shown that the minimum-norm least square solution demonstrates the so-called "double-descent" behavior [20]. A more recent line of research studies the generalization performance of gradient descent (GD) for binary classification. It has been shown [151]) that for a separable dataset, GD (when applied on the logistic loss) converges in direction to the max-margin classifier (a.k.a. hard-margin SVM). The performance of max-margin classifier has been recently analyzed in two independent works [51, 118]. State of the Art We analyze the performance of GMM in the high-dimensional regime where both the number of parameters, 𝑝, and the number of samples 𝑛 grows, and analyze the asymptotic performance as a function of the overparameterization ratio 𝛿 := 𝑛𝑝 > 0. First, we provide the phase transition condition for the separability of data (i.e., derive the exact value of 𝛿∗ such that the data is separable for all 𝛿 > 𝛿∗ 15.) Consequently, we analyze the performance in the interpolating regime (𝛿 > 𝛿∗ ). To the best of our knowledge, this is the first theoretical result that provides sharp asymptotics on the performance of GMM classifiers on separable data. For our analysis, we exploit the Convex Gaussian Min-max Theorem (CGMT) [155, 171] which is a strengthened version of a classical Gaussian comparison inequality due 15 Concurrent to the submission of this paper, a similar phase transition has been demonstrated in [97] for a somewhat different model. 53 to Gordon [79]. This framework replaces the original optimization with another optimization problem that has a similar performance, yet is much simpler to analyze as it becomes nearly separable. Previously, the CGMT has been successfully applied to derive the precise performance in a number of applications such as regularized M-estimators [165], analysis of the generalized lasso [117, 171], data detection in massive MIMO [1, 9, 175], and PhaseMax in phase retrieval [54, 141, 142]. More recently, this framework has been employed in a series of works by multiple groups of researchers to characterize the performance of the logistic loss minimizer in binary classification [143, 161]. Furthermore, in an analogous avenue of research, the CGMT framework has been utilized to study the generalization behavior of the gradient descent algorithm in the interpolating regime, where there exists a (nonempty) set of parameters that perfectly fit the training data [51, 118]. Mathematical setup We consider the problem of binary classification, having a set of training data, 𝑛 , where each of the sample points consists of a 𝑝-dimensional D = {(x𝑖 , 𝑦𝑖 )}𝑖=1 feature vector, x𝑖 , and a binary label, 𝑦𝑖 ∈ {±1}. We assume that the dataset D is generated from a logistic-type model with the underlying parameter w★ ∈ R 𝑝 . This means that 𝑦𝑖 ∼ Rad(𝜌(x𝑇𝑖 w★)) , 𝑖 = 1, . . . , 𝑛 , (2.87) where 𝜌 : R → [0, 1] is a non-decreasing function and is often referred to as the link function. A commonly-used instance of the link function is the standard logistic function defined as 𝜌(𝑡) := 1+𝑒1 −𝑡 . When 𝑛/𝑝 is sufficiently large, i.e., when we have access to a sufficiently large number of samples, the maximum-likelihood estimator( ŵ 𝑀 𝐿 ) is well-defined. In such settings, the MLE is often the estimator of choice due to its desirable properties in the classical statistics. Sur and Candès [157] have recently studied the performance of the MLE in logistic regression in the high-dimensional regime, where the number of observations and parameters are comparable, and show, among other things, that the maximum likelihood estimator is biased. Their results have been extended to regularized logistic regression [143], assuming some prior knowledge on the structure of the data. In particular, it has been observed that, when the regularization parameter is tuned properly, the regularized logistic regression can outperform the MLE. Inspired by the recent results on analyzing the generalization error of machine learn- 54 ing models, in this section, we study the generalization error of binary classification, in a regime of parameters known as the interpolating regime. Here, the assumption is that there exists a parameter vector that can perfectly fit (interpolate) the data, i.e., ∃w0 s.t. Sign(w𝑇0 x𝑖 ) = 𝑦𝑖 , for 𝑖 = 1, 2, . . . , 𝑛. (2.88) Let W denote the set of all the parameters that interpolate the data. W = {w ∈ R 𝑝 : Sign(w𝑇 x𝑖 ) = 𝑦𝑖 , for 1 ≤ 𝑖 ≤ 𝑛.}. (2.89) It has been observed that in many machine learning tasks, the iterative solvers that minimize the loss function often converge to one of the points in the set W (the training error converges to zero). Therefore, one can (qualitatively) pose the following important (yet still mysterious) question: Which point(s) in W is (are) ”better” estimator(s) of the actual parameter, w★? In an attempt to find an answer to this question, we focus on the simple (yet fundamental) model of binary classification. We assume that the underlying parameter, w★ possesses certain structure (sparse, low-rank, block-sparse, etc.), and consider a locally-Lipschitz and convex function 𝜓 : R 𝑝 → R which encourages this structure. We introduce the Generalized Margin Maximizer (GMM) as the solution to the following optimization: min 𝜓(w) w∈R 𝑝 (2.90) s.t. 𝑦𝑖 (x𝑇𝑖 w) ≥ 1, for 1 ≤ 𝑖 ≤ 𝑛. It is worth noting that the condition on the separability of the dataset is crucial for the optimization program (2.90) to have a feasible point. Remark 18 It can be shown that when 𝜓(·) is absolutely scalable16, the GMM can be found by solving the following equivalent optimization program, kwk 𝜓(w) 𝜓(w) = max × . 𝑇 𝑇 kwk w∈R𝑑 min 𝑦 𝑖 (x𝑖 w) w∈R𝑑 min 𝑦 𝑖 (x𝑖 w) max 1≤𝑖≤𝑛 (2.91) 1≤𝑖≤𝑛 The first multiplicative term on the right indicates the margin associated with the separator w, and the second term, 𝜓(w) kwk takes into account the structure of the 16 A function 𝑓 : R 𝑑 → R is absolutely scalable when, ∀v ∈ R𝑑 , ∀𝛼 ∈ R, 𝑓 (𝛼v) = |𝛼| 𝑓 (v). All ℓ 𝑝 norms, for example, are absolutely scalable. 55 model. Hence, we refer to the objective function in the optimization (2.91) as the generalized margin, and the solution to this optimization is called the generalized margin maximizer (GMM). In this section, we study the linear asymptotic regime in which the problem dimensions 𝑝, 𝑛 grow to infinity at a proportional rate, 𝛿 := 𝑛𝑝 > 0. Our main result characterizes the performance of the solution of (2.90), ŵ, in terms of the ratio, k w★ k 𝑛 , are 𝛿, and the signal strength, 𝜅 := √ 𝑝 . We assume that the datapoints, {x𝑖 }𝑖=1 drawn independently from the Gaussian distribution. Our main result characterizes the performance of the resulting estimator through the solution of a system of five nonlinear equations with five unknowns. In particular, as an application of our main result, we can accurately predict the generalization error of the resulting estimator. Main Results In this section, we present the main results of the paper, that is the characterization of the performance of the generalized margin maximizers. Our results are represented in terms of a summary functional, 𝑐 𝑡 (·, ·), which incorporates the information about the underlying model. Definition 1 For the parameter 𝑡 > 0, the function 𝑐 𝑡 : R × R+ → R+ is defined as,   𝑐 𝑡 (𝑠, 𝑟) = E (1 − 𝑡𝑠𝑍1𝑌 − 𝑟 𝑍2 )+2 , (2.92) i.i.d. where 𝑍1 , 𝑍2 ∼ N (0, 1), and 𝑌 ∼ Rad(𝜌(𝑡𝑍1 )). Asymptotic phase transition Here, we provide the necessary and sufficient condition for the separability of the data. Theorem 6 (Phase transition) Consider the generalized max margin optimization defined in Section 2.4. As 𝑛, 𝑝 → ∞ at a fixed overparameterization ratio 𝛿 := 𝑛𝑝 ∈ (0, ∞), this optimization program (almost surely) has a solution (or equivalenty, the set W is nonempty) if and only if, 𝑐 𝜅 (𝑠, 𝑟) . 𝑠,𝑟 ≥0 𝑟2 𝛿 > 𝛿∗ = 𝛿∗ (𝜅) := inf (2.93) Remark 19 Theorem 6 indicates the necessary and sufficient condition for the existense of GMM. It is worth mentioning that this condition, which is simply the 56 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.5 1 1.5 2 2.5 3 Figure 2.10: The phase transition, 𝛿∗ , for the separability of the dataset, where the feature vector, x𝑖 is drawn from the Gaussian distribution, N (0, 1𝑝 I 𝑝 ), and the labels
𝑒𝑡 are 𝑦𝑖 ∼ Rad 𝜌(x𝑇𝑖 w★) , for 𝜌(𝑧) = 𝑒𝑡 +𝑒 −𝑡 . The empirical result is the average over 20 trials with 𝑝 = 150, and the theoretical results are from Theorem 6. condition on separability of the dataset D, does not depend on the choice of the potential function 𝜓(·). Remark 20 The phase transition (2.93), is valid for any link function 𝜌(·). This generalizes the former results in [37]. Note that the summary functional, 𝑐 𝜅 (·, ·), contains the choice of the link function and can be computed numerically. The following lemma explains the behavior of 𝛿∗ as 𝜅 varies. Lemma 6 𝛿∗ is a decreasing function of 𝜅, with 𝛿∗ (0) = 21 and lim𝜅→+∞ 𝛿∗ (𝜅) = 0. k √ The result of Lemma 6 can be intuitively verified. Recall that 𝜅 = kw 𝑝 and 𝑦𝑖 ∼ Rad(𝜌(x𝑇𝑖 w★)). Therefore, 𝜅 → ∞ translates to having 𝑦𝑖 = Sign(x𝑇𝑖 w★). In this case our training data is always separable for any number of observations 𝑛. Besides, the case of 𝜅 = 0 corresponds to having random labels assigned to feature vectors x𝑖 . [47] showed that in this case, as 𝑝 → ∞, 𝛿 > 0.5 is the necessary and ★ 57 sufficient condition for the separability of the data set. Figure 2.10 provides a comparison between the theoretical result in Theorem 6, and the empirical results derived from numerical simulations for 𝑝 = 150 and 20 trials. As seen in this plot, the theory matches well with the empirical simulations. A nonlinear system of equations Our main result in Section 2.4 precisely characterizes the performance of GMM in terms of a system of 5 nonlinear equations with 5 unknowns, (𝛼, 𝜎, 𝛽, 𝛾, 𝜏), defined as follows, √
  ★𝑇  1 ★  = 𝛼𝜅 2 ,  𝑝 E w Prox𝜎𝜏𝜓(·) (𝛼 − 𝜎𝜏𝛾)w + 𝛽𝜎𝜏 𝛿h    √
 q 𝑐 𝜅 (𝛼,𝜎)  𝑇  1 ★  , =   𝑝 E h Prox𝜎𝜏𝜓(·) (𝛼 − 𝜎𝜏𝛾)w + 𝛽𝜎𝜏 𝛿h 𝛿    2 √
1 ★ = 𝛼2 𝜅 2 + 𝜎 2 , 𝑝 E Prox𝜎𝜏𝜓(·) (𝛼 − 𝜎𝜏𝛾)w + 𝛽𝜎𝜏 𝛿h   p  2𝜅 2 𝛾 𝜕𝑐 𝜅 (𝛼,𝜎)   = 𝑐 𝜅 (𝛼, 𝜎),  𝜕𝛼 𝛽  √     𝜕𝑐 𝜅 (𝛼,𝜎) = 2 𝑐 𝜅 (𝛼,𝜎) .  𝜕𝜎 𝛽𝜏 (2.94) Remark 21 The first three equations in the nonlinear system (2.94) capture the role of the potential function, via its proximal operator. When 𝜓(·) is separable, these functions can further be reduced to the proximal operator of a real-valued function. For instance, when 𝜓(·) = k·k 1 , the proximal operator is simply equivalent to 𝑥 (|𝑥| − 𝑡)+ ) on each entry. applying the well known shrinkage (defined as 𝜂(𝑥, 𝑡) = |𝑥| For more information on the proximal operators, please refer to [132]. Asymptotic performance of GMM We are now ready to present the main result of the paper. Theorem 7 characterizes the asymptotic behavior of GMM, that is the solution to the optimization program (2.90). It connects the performance of GMM to the solution of the nonlinear system of equations (2.94), and informally states that, 𝐷 ŵ → Γ(w★, h), as 𝑝 → ∞, (2.95) where h ∈ R 𝑝 has standard normal entries, and Γ : R 𝑝 × R 𝑝 → R 𝑝 is defined as, √
Γ(v1 , v2 ) = Prox𝜎¯ 𝜏𝜓(·) ( 𝛼¯ − 𝜎 ¯ 𝜏¯ 𝛾)v ¯ 1 + 𝛽¯𝜎 ¯ 𝜏¯ 𝛿v2 , ¯ ¯ 𝛾, where ( 𝛼, ¯ 𝜎, ¯ 𝛽, ¯ 𝜏) ¯ is the solution to the nonlinear system (2.94). (2.96) 58 Theorem 7 Let ŵ be the solution of the GMM optimization (2.90), where for 𝑖 = 1, 2, . . . , 𝑛, x𝑖 has the multivariate Gaussian distribution N (0, 1𝑝 I 𝑝 ), and k w★ k 𝑦𝑖 ∼ Rad(𝜌(x𝑇𝑖 w★)), and w★ is drawn from a distribution Π with 𝜅 = √ 𝑝 . As 𝑛, 𝑝 → ∞ at a fixed overparameterization ratio 𝛿 = 𝑛𝑝 > 𝛿∗ (𝜅), the nonlinear ¯ 𝛾, system (2.94) has a unique solution ( 𝛼, ¯ 𝜎, ¯ 𝛽, ¯ 𝜏). ¯ Furthermore, for any locallyLipschitz function 𝐹 : R 𝑝 × R 𝑝 → R, we have, 𝑃 𝐹 ( ŵ, w★) → E[𝐹 (Γ(w, h), w)], (2.97) where h ∈ R 𝑝 has standard normal entries, w ∼ Π is independent of h, and the function Γ(·, ·) is defined in (2.96). In short, we introduce dual variables and write down the Lagrangian which contains a bilinear form with respect to a matrix with i.i.d. Gaussian entries. Exploiting the CGMT framework, we then analyze the nearly-separable auxiliary optimization to find its optimal value, and show that the nonlinear system (2.94) corresponds to its optimality condition. Remark 22 The result in Theorem 7 is stated for a general locally-Lipschitz function 𝐹 (·, ·). To evaluate a specific performance measure, one can appeal to this theorem with an appropriate choice of 𝐹. As an example, the function 𝐹 (u, v) = 1𝑝 ku − vk 2 gives the mean-squared error (MSE). Generalization error Theorem 7 can be utilized to derive useful information on the performance of the classifier. In fact, using this theorem one can show that the parameters 𝛼, ¯ and 𝜎 ¯ respectively correspond to the correlation (to the underlying parameter) and the mean-squared error of the resulting estimator. An important measure of performance is the generalization error, which indicates the success of the trained model on unseen data. Here, we compute the generalization error of the GMM classifier. We do so, by appealing to the result of Theorem 7. Definition 2 The generalization error for a binary classifier with parameter ŵ is defined as, 𝐺𝐸 ŵ = Px {Sign(x𝑇 ŵ) ≠ Sign(x𝑇 w★)}, (2.98) where the probability is computed with respect to the distribution of the test data. 59 It can be shown that when the distribution of the test data is rotationally invariant (e.g., Gaussian, uniform dist. on the unit-sphere), GE only depends on the angle between ŵ and w★. The following lemma provides sharp asymptotics on the generalization error of the GMM classifier. Lemma 7 (Generalization Error) Let ŵ be the GMM classifier defined in Section 2.4. Assume 𝛿 > 𝛿∗ , and the (test) data is distributed according to the multivariate Gaussian distribution N (0, 1𝑝 I 𝑝 ). Then, as 𝑝 → ∞, we have, 𝑃 𝐺𝐸 ŵ → 𝜅 𝛼¯ 1 acos( √ ), 𝜋 𝜅 2 𝛼¯ 2 + 𝜎 ¯2 (2.99) where 𝛼¯ and 𝜎 ¯ are derived by solving the nonlinear system (2.94). Proof 3 We first note that when the data is normally distributed, the generalization error for ŵ is defined as, 𝐺𝐸 ŵ = 1 ŵ𝑇 w★ acos( ★ ). kw k k ŵk 𝜋 (2.100) We appeal to the result of Theorem 7 with two different functions. Using 𝐹1 (u, v) = 1 𝑇 𝑝 v u in (2.97) will give, √
 1 𝑇 ★ 𝑃 1  ★𝑇 ( 𝛼¯ − 𝜎 ¯ 𝜏¯ 𝛾)w ¯ ★ + 𝛽¯𝜎 ¯ 𝜏¯ 𝛿h . ŵ w → E w Prox𝜎¯ 𝜏𝜓(·) ¯ 𝑝 𝑝 (2.101) ¯ 𝛾, Since ( 𝛼, ¯ 𝜎, ¯ 𝛽, ¯ 𝜏) ¯ is the solution to the nonlinear system, we can replace the expectation from the first equation in (2.94),which gives the following, 1 𝑇 ★ 𝑃 2 ŵ w → 𝜅 𝛼. ¯ 𝑝 (2.102) Similarly, using the result of Theorem 7 for the measure function 𝐹2 (u, v) = 1𝑝 kuk 2 , along with the third equation in (2.94) gives, p 1 𝑃 ¯2. (2.103) √ k ŵk → 𝜅 2 𝛼¯ 2 + 𝜎 𝑝 The proof is the consequence of (2.100), (2.102), and (2.103), along with the continuity of the function acos(·). GMM for Various Structures As explained earlier, the potential function 𝜓(·) is chosen to encourage the structure of the underlying parameter. In this section, we investigate the performance of the GMM classifier for some common structures and the corresponding choices of the potential function. 60 Max-margin classifier (ℓ2 -GMM) The ℓ2 -norm regularization is commonly used in machine learning applications to stabilize the model. Here, we study the performance of the GMM classifier when 𝜓(·) = 21 k·k 22 , i.e., the solution to the following optimization program, 1 kwk 22 w∈R 2 s.t. 𝑦𝑖 (x𝑇𝑖 w) ≥ 1, for 1 ≤ 𝑖 ≤ 𝑛. min𝑝 (2.104) The optimization program (2.104) is called the hard-margin SVM and the corresponding solution is the max-margin classifier, as it maximizes the minimum distance (margin) of the datapoints from the separating hyperplane. The conventional justification for using such a classifier is that the risk of a classifier is inversely proportional to its margin. The performance of ℓ2 -GMM (2.104), has been earlier analyzed in [51] and [118]. The form we present below in (2.106), differes in appearance to the results of [51], but can be shown to be equivalent. When 𝜓(·) = 12 k·k 22 , the proximal operator has the following closed-form, Prox 𝑡 k·k 2 (u) = 2 1 u. 1+𝑡 (2.105) By replacing the proximal operator in the nonlinear system (2.94), we can explicitly find two of the variables (𝛽, and 𝛾) and reduce it to the following system of three nonlinear equations in three unknowns, p √   𝑐 𝜅 (𝛼, 𝜎) = 𝜎 𝛿,      𝜕𝑐 𝜅 (𝛼, 𝜎) −2𝜅 2 𝛼𝜏𝜎𝛿  = , (2.106) 𝜕𝛼 1 + 𝜎𝜏    𝜕𝑐 𝜅 (𝛼, 𝜎) 2𝜎𝛿    = .  𝜕𝜎 1 + 𝜎𝜏 Sparse classifier (ℓ1 -GMM) In today’s machine learning applications, typically the number of available features, 𝑝, is overwhelmingly large. To reduce the risk of overfitting in such settings, feature selection methods are often performed to exclude irrelevant variables from the model [89]. Adding an ℓ1 penalty is the most popular approach for feature selection. As a natural consequence of our main result in Theorem 7, here we analyze the asymptotic performance of GMM when the potential function is the ℓ1 norm, and evaluate its success on the unseen data (i.e., the test error) when the underlying 61 parameter, w★, is sparse. min w∈R 𝑝 kwk 1 (2.107) s.t. 𝑦𝑖 (x𝑇𝑖 w) ≥ 1, for 1 ≤ 𝑖 ≤ 𝑛. In this case, the proximal operator of the potential function (k·k 1 ) is basically equivalent to applying the soft-thresholding operator, on each entry, i.e., Prox𝑡 k·k 1 (u) = 𝜂(u, 𝑡), (2.108) 𝑥 where 𝜂(𝑥, 𝑡) := |𝑥| (|𝑥| − 𝑡)+ is the soft-thresholding operator. Here, for a sparsity factor 𝑠 ∈ (0, 1], we assume the entries of w★ are sampled i.i.d. from the following distribution, 𝜙( √𝑤𝜅 )
𝑠 Π𝑠 (𝑤) = (1 − 𝑠) · 𝛿0 (𝑤) + 𝑠 · , (2.109) 𝜅 √ 𝑠 𝑡2 𝑒− 2 where 𝛿0 (·) is the Dirac delta function, and 𝜙(𝑡) := √ is the density of the standard 2𝜋 normal random variable. This means that each of the entries of w★ are zero with probability 1 − 𝑠, and the nonzero entries have independent Gaussian distribution 2 with variance 𝜅𝑠 . Having this assumption we can further simplify the first three equations in the nonlinear system (2.94), and present them in terms of q-functions. To streamline our representation, we introduce the following proxies, 𝑡1 = q 1 , 𝑡2 = √ . 𝛽 𝛿 𝜅2 (𝛼 − 𝜎𝜏𝛾) 2 + 𝛽2 𝜎 2 𝜏 2 𝛿 𝜎𝜏 (2.110) 𝑠 We also define the function 𝜒 : R → R+ as,   𝜒(𝑡) = E (𝑍 − 𝑡)+2 , 𝑍 ∼ N (0, 1) = 𝑄(𝑡)(1 + 𝑡 2 ) − 𝑡𝜙(𝑡), (2.111) ∫∞ Where 𝑄(𝑡) := 𝑡 𝜙(𝑥)𝑑𝑥 denotes the tail distribution of standard normal random variable. We are now able to simplify the first three equations in (2.94) and derive the following nonlinear system,   𝛼  𝑄(𝑡1 ) = 2(𝛼−𝜎𝜏𝛾) ,   √   𝑐 𝜅 (𝛼,𝜎)   𝑠 · 𝑄(𝑡1 ) + (1 − 𝑠) · 𝑄(𝑡2 ) = 2𝛽𝜎𝜏𝛿 ,      2 2 𝑠 𝜅 𝛼 1 · 𝜒(𝑡1 ) + (1−𝑠) · 𝜒(𝑡2 ) = 2𝜎 2 𝜏 2 + 2𝜏 2 , 𝑡12 𝑡22   p  2𝜅 2 𝛾 𝜕𝑐 𝜅 (𝛼,𝜎)   = 𝑐 𝜅 (𝛼, 𝜎),  𝜕𝛼 𝛽  √    2 𝑐 𝜅 (𝛼,𝜎) (𝛼,𝜎)   𝜕𝑐 𝜅𝜕𝜎 = . 𝛽𝜏  (2.112) 62 The nonlinear system (2.112) can be solved via numerical methods. For our numerical simulations in Section 2.4 we exploit accelerated fixed-point methods to solve the nonlinear system. Using the result of Lemma 7, we can compute the generalization error. Another important measure in this setting (when w★ is sparse) is the probability of error in support recovery. Let Ω ⊆ [ 𝑝] denote the support of w★ (i.e. Ω = { 𝑗 : w★𝑗 ≠ 0}.) For a pre-defined threshold 𝜖, we form the following estimate of the support, Ω̂𝜖 = { 𝑗 : 1 ≤ 𝑗 ≤ 𝑝, | ŵ 𝑗 | > 𝜖 }. (2.113) The following lemma establishes the success in the support recovery: Lemma 8 (Support Recovery) For a sparsity factor 𝑠 ∈ (0, 1], let the entries of w★ have distribution Π𝑠 defined in (2.109), and ŵ be the solution to the optimization (2.107). Then, as 𝑝 → ∞, we have,  𝑃 lim 𝑃1 (𝜖) := P 𝑗 ∉ Ω̂𝜖 | 𝑗 ∈ Ω → 1 − 2𝑄( 𝑡¯1 ) 𝜖↓0 (2.114)  𝑃 lim 𝑃2 (𝜖) := P 𝑗 ∈ Ω̂𝜖 | 𝑗 ∉ Ω → 2𝑄( 𝑡¯2 ) , 𝜖↓0 where 𝑡¯1 and 𝑡¯2 are defined as in (2.110), with variables derived from solving the nonlinear system (2.112). Binary classifier (ℓ∞ -GMM) As the last example of structured classifiers, here we study the case where w★ ∈ {±} 𝑝 . To encourage this structure, the potential function is chosen to be the ℓ∞ norm. In linear regression, k·k ∞ is used to recover the binary signals, i.e., when w★ ∈ {±1} 𝑝 [40]. This problem arises in integer programming and has some connections to the Knapsack problem [110]. Here, we consider analyzing the performance of the solution of the following optimization program, min w∈R 𝑝 kwk ∞ (2.115) s.t. 𝑦𝑖 (x𝑇𝑖 w) ≥ 1, for 1 ≤ 𝑖 ≤ 𝑛. It can be shown that the proximal operator of the ℓ∞ -norm can be derived by projecting the points onto the ℓ1 -ball. We use this connection to present the proximal operator in this case in terms of the soft-thresholding operator 𝜂(·, ·). For a vector w whose entries are drawn independently from a distribution Π, we can present the following formula for the proximal operator: Prox𝑡 𝑝k·k ∞ (w) = w − Prox𝜆k·k 1 (w), (2.116) 63 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0 1 2 3 4 5 6 7 8 9 10 Figure 2.11: Generalization error of the general max margin classifier under three penalty functions, ℓ1 norm with the red line (ℓ1 -GMM), ℓ2 norm with the blue line (ℓ2 -GMM), and ℓ∞ norm with the black line (ℓ∞ -GMM). In this figure, the entries of w★ are drawn independently from N (0, 𝜅 2 ) Gaussian distribution. Solid lines correspond to the theoretical results derived from Theorem 7, while the circles are the result of empirical simulations. For the numerical simulations, the result is the average over 100 independent trials with 𝑝 = 200 and 𝜅 = 2. where 𝜆 := 𝜆(𝑡) is the smallest nonnegative number that satisfies,     E |𝜂(𝑊, 𝜆)| = E (|𝑊 | − 𝜆)+ ≤ 𝑡. (2.117) Here, the expectation is with respect to 𝑊 ∼ Π. Note that 𝜆 is a non-increasing function of 𝑡, and 𝜆 = 0 whenever 𝑡 ≥ E|𝑊 |. Similar to the case of ℓ1 -GMM, here we can use the closed-form of the proximal operator to simplify the first three equations in the nonlinear system (2.94). For our numerical simulations in the next section, we have done the computations for three different distributions: (1) The i.i.d. Gaussian distribution, (2) the sparse distribution
defined in (2.109), and (3) the uniform binary distribution, Π = Unif {±1} 𝑝 . Numerical Simulations In this section, we investigate the validity of our theoretical results with multiple numerical simulations applied to the three different cases of GMM classifiers elaborated in Section 2.4. For each of the three potentials discussed in the paper (i.e., ℓ1 , ℓ2 , and ℓ∞ norms) we perform numerical simulations for three different models on 64 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0 1 2 3 4 5 6 7 8 9 10 Figure 2.12: Generalization error of the general max margin classifier under three penalty functions, ℓ1 norm with the red line (ℓ1 -GMM), ℓ2 norm with the blue line (ℓ2 GMM), and ℓ∞ norm with the black line (ℓ∞ -GMM). In this figure, the underlying vector w★ is 𝑠-sparse, where the non-zero entries are drawn independently from N (0, 𝜅 2 /𝑠) Gaussian distribution. Solid lines correspond to the theoretical results derived from Theorem 7, and the circles are the result of empirical simulations. For the numerical simulations, the result is computed by taking the average over 100 independent trials with 𝑝 = 200, 𝑠 = .1 and 𝜅 = 2. the distribution of w★. In other words, we change the distribution of the entries of w★ and evaluate the performance of the aforementioned classifiers on each model. As will observed in our numerical simulations, the appropriate choice of the potential function in the GMM optimization (2.90) has an impact on the generlization error of the resulting classifier. The three different distribution that we choose for the underlying parameter are as follows: Gaussian: in the first model, we assume that the entries of w★ are drawn from a zero-mean Gaussian distribution, N (0, 𝜅 2 ). In this model, the direction of w★ (which indicates the separating hyperplane) is distributed uniformly on the unit sphere. Figure 2.11 gives the generalization error when w★ has Gaussian distribution. The solid lines show the theoretical results derived from Theorem 7 and Lemma 7. The circles depict empirical results that are computed by taking the average over 100 trials with 𝑝 = 200 and 𝜅 = 2. Although our theory provides the generalization error in the asymptotic regime, it appropriately matches the result of empirical simulations in our simulations in finite dimensions. It can be observed 65 0.5 0.45 0.4 0.35 0.4 0.35 0.3 0.3 0.25 0.25 0.2 0.2 0.15 0.5 1 1.5 0.15 1 2 3 4 5 6 7 8 9 10 Figure 2.13: Generalization error of the general max margin classifier under three penalty functions, ℓ1 norm with the red line (ℓ1 -GMM), ℓ2 norm with the blue line (ℓ2 -GMM), and ℓ∞ norm with the black line (ℓ∞ -GMM). In this figure, the entries of w★ are drawn independently from 𝜅 ∗ Rad(0.5) Rademacher distribution. Solid lines correspond to the theoretical results derived from Theorem 7, and the circles are the result of empirical simulations. For the numerical simulations, the result is the average over 100 independend trials with 𝑝 = 200 and 𝜅 = 2. in this figure that the max-margin classifier (ℓ2 -GMM) outperforms the other two classifiers. We should also note that as the overparameterization ratio, 𝛿, grows the generalization error increases which indicates that the estimator is not reliable for large values of 𝛿. Sparse: here, we assume that the entries of w★ are drawn from the sparse distribution represented in (2.109), i.e., each entry is nonzero with probability 𝑠, and the nonzero entries have i.i.d. Gaussian distribution with appropriately-defined variance. Figure 2.12 demonstrates the result of the numerical simulations for this model for the three different classifiers of interest. The empirical result is the average over 100 trials with 𝑝 = 200, 𝑠 = 0.1, and 𝜅 = 2. Similar to the previous case, the empirical results match the theory. Also, it can be observed that the ℓ1 -GMM outperforms the two other classifiers in the regime of 𝛿 that the classifiers performs well (i.e. 𝛿 w 6.) Similarly, we can observe that for large values of 𝛿 all the classifiers perform poorly. Binary: in this model the entries of w★ are independently drawn from {+𝜅, −𝜅}, i.e., w★ is uniformly chosen on the discrete set {±𝜅} 𝑝 . Figure 2.13 shows the result of numerical simulations under this model. Similar to previous cases the empirical 66 results (𝜅 = 2, 𝑝 = 200) match the theory. Also, the ℓ∞ -GMM classifier outperforms the other two classifiers for 𝛿 < 1 (which corresponds to the underparameterized setting). However, the max-margin classifier performs better for larger values of 𝛿. 2.5 Highlight of the Proof and CGMT Framework Developed by Thrampoulidis et al., Convex Gaussian Min-Max Theorem (CGMT) is a noble framework that enables us to analyze a wide variety of problems, including the convex estimator (2.2), and is the main idea behind the proof of Theorem 1. The CGMT is an extension of a Gaussian comparison inequality proved by Gordon in 1988 [79, 80]. Starting with the works of Rudelson and Vershynin [139], Stojnic [153], Oymak [127], Chandrasekaran [40] and also Amelunxen, Maccoy [6] Gordon’s original theorem has played a key role in the analysis of (underdetermined) noiseless linear inverse problems. We refer the interested reader to [165, 171] for more details and a discussion on the relation of the CGMT to the result by Gordon. Before going through the technicalities of CGMT, we present a simple understanding of how CGMT framework can be applied in practice. Recall the convex estimator (2.2) √
√ 𝛽ˆ = arg min L X𝛽/ 𝑝, g(X𝛽0 / 𝑝) + 𝜆 𝑓 (𝛽) . 𝛽 (2.118) After a couple of steps that we will explain shortly, we can rewrite this optimization in the form of the following min-max optimizations, which we refer to as the Primary Optimization (PO), Φ(X) := min max uT X𝛽 + Ψ(𝛽, u) . (2.119) 𝛽∈S𝛽 u∈Su This is a more complicated form of our initial optimization, because of the extra maximization. Besides, as we go more through the technical details, we can see that one of the main restrictions in the analysis of this optimization, in how the variables u and 𝛽 are coupled through the features matrix X. If we were somehow able to decouple these two, the analysis would get much easier. This is what CGMT actually does. The Convex Gaussian Min-max Theorem associates with the primary optimization (PO) problem a simplified Auxiliary Optimization (AO) problem, from which we can tightly infer properties of the original (PO), such as the optimal cost, the optimal solution, etc.. In the other words, CGMT introduces another optimization that is much simpler to analyze, and its solution has the same properties as the solution to 67 the PO. Specifically, the Auxiliary optimization is given as follows 𝜙(h1 , h2 ) := min max k 𝛽k2 hT1 u + kuk2 hT2 w + Ψ(𝛽, u) . (2.120) 𝛽∈S𝛽 u∈Su As you see, the auxiliary optimization is the same as the primary optimization, except the bi-linear form uT X𝛽 which is replaces by two two terms k 𝛽k2 hT1 u and kuk2 hT2 w. This simple change will significantly help us later in the analysis. In the other words, we will be able to analyze the properties of the solution to the auxiliary optimization using techniques in convex analysis and high dimensional probability and random matrix theory (which are not applicable to the primary optimization). Afterwards, the CGMT tells us that the primary and auxiliary optimizations shares some properties, including those of our interest. Next, we will go through a precise analysis of our theorem, using the CGMT framework. 2.6 Proof of Theorem 1 In this section, we rigorously prove Theorem 1. Recall the convex estimator (2.2), √
√ 𝛽ˆ = arg min L X𝛽/ 𝑝, g(X𝛽0 / 𝑝) + 𝜆 𝑓 (𝛽) . 𝛽 (2.121) Note that the entries of X are independently drawn from N (0, 1). Now, we rewrite the optimization by introducing a new variable, min𝑝 𝛽∈R √ w=X𝛽0 / 𝑝 √
L w, g(X𝛽0 / 𝑝) + 𝜆 𝑓 (𝛽) . (2.122) In the next step, we bring the constraint over w in the objective function using Lagrange multiplier, 1 √
√ min𝑝 max𝑛 L w, g(X𝛽0 / 𝑝) + 𝜆 𝑓 (𝛽) + uT (X𝛽/ 𝑝 − w) . 𝛽∈R u∈R 𝑛 𝑛 (2.123) w∈R Note that the two terms X𝛽0 and uT X𝛽 are dependent. We would like to decompose the later term in such a way that one becomes independent of the other. This will help us use CGMT, as you will see later. To do so, we decompose the variable 𝛽 into its projection in the direction of 𝛽0 and the subspace orthogonal to 𝛽0 . In the ˜ where 𝛽⊥𝛽 ˜ 0 . The optimization becomes, other words, we rewrite 𝛽 = 𝛼𝛽0 + 𝛽, min ˜ 𝑝 𝛽∈R w∈R𝑛 1 √
˜ √ 𝑝 + 𝛼X𝛽0 /√ 𝑝 − w) . max𝑛 L w, g(X𝛽0 / 𝑝) + 𝜆 𝑓 ( 𝛽˜ + 𝛼𝛽0 ) + uT (X 𝛽/ u∈R 𝑛 𝛼∈R 1 ˜T 𝑝 𝛽 𝛽0 =0 (2.124) 68 Now, since the matrix X is Gaussian with iid entries, the variables X𝛽0 and X 𝛽˜ are independent. We reorder the term on optimization to get   1 1 T √ √
T ˜ ˜ min max𝑛 √ u X 𝛽 + L w, g(X𝛽0 / 𝑝) + 𝜆 𝑓 ( 𝛽 + 𝛼𝛽0 ) + u (𝛼X𝛽0 / 𝑝 − w) . ˜ 𝑝 u∈R 𝑛 𝑝 𝑛 𝛽∈R w∈R𝑛 𝛼∈R 1 ˜T 𝑝 𝛽 𝛽0 =0 (2.125) Now, the first bi-linear term uT X 𝛽˜ is independent of the rest of the objective function. Up to now, it seems like we’ve build a more difficult optimization to analyze by introducing new variables and a maximization. But the point of every step up to now was simply to utilize the CGMT framework. Note that what we have in (2.125) is in the form of the primary optimization (2.119), in the CGMT framework. Next, we give a precise statement of the CGMT framework and apply it to the optimization (2.125). The Convex Gaussian Min-max Theorem Consider the primary optimization (PO) and the Auxiliary optimization (AO) below, Φ(X) := min max uT X𝛽 + Ψ(𝛽, u) , (PO) (2.126a) 𝛽∈S𝛽 u∈Su 𝜙(h1 , h2 ) := min max k 𝛽k2 hT1 u + kuk2 hT2 w + Ψ(𝛽, u) , (AO) . (2.126b) 𝛽∈S𝛽 u∈Su where X ∈ R𝑛×𝑝 , h1 ∈ R𝑛 , h2 ∈ R 𝑝 , Su ⊂ R𝑛 , S𝛽 ⊂ R 𝑝 and Ψ : R 𝑝 × R𝑛 → R. We denote 𝛽Φ := 𝛽Φ (X) and 𝛽𝜙 := 𝛽𝜙 (h1 , h2 ) any optimal minimizers in (2.126a) and (2.126b), respectively. Then, we have the following result. Theorem 8 (CGMT) In (2.126), let S𝛽 , Su be compact sets, Ψ be continuous on S𝛽 × Su , and, X, h1 and h2 all have entries iid standard normal. The following statements are true: 1. For all 𝑐 ∈ R: P( Φ(X) < 𝑐 ) ≤ 2P( 𝜙(h1 , h2 ) ≤ 𝑐 ). 2. Further assume that S𝛽 , Su are convex sets and 𝜓 is convex-concave on S𝛽 × Su . Then, for all 𝑐 ∈ R, P( Φ(X) > 𝑐 ) ≤ 2P( 𝜙(h − 1, h2 ) ≥ 𝑐 ). 69 In particular, for all 𝜇 ∈ R, 𝑡 > 0, P( |Φ(X) − 𝜇| > 𝑡 ) ≤ 2P( |𝜙(h1 , h2 ) − 𝜇| ≥ 𝑡 ). 3. Let S be an arbitrary open subset of S𝛽 and S 𝑐 = S𝛽 /S . Denote 𝜙S 𝑐 (h1 , h2 ) the optimal cost of the optimization in (2.126b) when the minimization over w is now constrained over w ∈ S c . If there exist constants 𝜙, 𝜙S c and 𝜂 > 0 such that a) 𝜙S 𝑐 ≥ 𝜙 + 3𝜂, b) 𝜙(h1 , h2 ) < 𝜙 + 𝜂 with probability at least 1 − 𝑝, c) 𝜙S 𝑐 (h1 , h2 ) > 𝜙S 𝑐 − 𝜂 with probability at least 1 − 𝑝, then, P(𝛽Φ (X) ∈ S) ≥ 1 − 4𝑝. The first two statements of Theorem 8 are identical to [171, Thm. 3], and, a proof is included therein. The second statement of the theorem states that under the theorem assumption, the values of the optimal objective functions Φ(X) and 𝜙(h1 , h2 ) converge to the same values (if the later converges). Using the third statement, we show that if the optimal solution of the (AO) lies within some set S with probability approaching to 1 (as the dimensions 𝑝 and 𝑛 increase), and optimizing (AO) over the set S c will result in a strictly larger optimal objective value, then the optimal solution in (PO) will be also in the set S with probability approaching to 1. Using this, we would like to show that the optimal ˜ 2 will converge to the same value for (AO) and (PO). To do so, values for 𝛼, and k 𝛽k we show that these terms in (AO) converge to a unique value, and so will they in the (PO). Rigorously applying this theorem to the primary optimization is discussed in the proof of Theorem 2 in the Appendix. For now, let’s apply CGMT to the optimization (2.125), and get the auxiliary optimization from it. Clearly, the optimization (2.125) has the desired format of (PO) in the CGMT. Therefore, the corresponding Auxiliary Optimization will be min max𝑛 ˜ 𝑝 u∈R 𝛽∈R w∈R𝑛 𝛼∈R 1 ˜T 𝑝 𝛽 𝛽0 =0 1 √
T ˜ ˜ Tu + √ k 𝛽kh kukh 𝛽 + L w, g(X𝛽 / 𝑝) + 𝜆 𝑓 ( 𝛽˜ + 𝛼𝛽0 ) 0 1 2 𝑛 𝑝 𝑛 𝑝 1 √ 1 √ + uT (𝛼X𝛽0 / 𝑝 − w) . 𝑛 (2.127) 70 Corollary 3 (Asymptotic CGMT) Using the same notation as in Theorem 8, sup𝑃 𝑃 pose there exists constants 𝜙 < 𝜙S c such that 𝜙(g, h) − → 𝜙 and 𝜙S c (g, h) − → 𝜙S c . Then, lim P(wΦ (G) ∈ S) = 1. 𝑛→∞ Analysis of the AO Out next step would be to analyze the auxiliary optimization. The idea is to use dimension reduction techniques to reduce the optimization (2.127) to an optimization over scalars. To do so, for most of the high dimensional variables, we would like to do the optimization over their direction first. In the first step, we would like to optimize over the direction of u. In the other √ words, we rewrite u/ 𝑛 = 𝑡 · ũ, where k ũk = 1 and 𝑡 ≥ 0. Then, if we solve the optimization over ũ, we get min ˜ 𝑝 𝛽∈R w∈R𝑛 𝛼∈R 1 ˜T 𝑝 𝛽 𝛽0 =0 ˜ 𝑡 k 𝛽k 𝛼 𝑡 √
max √ k √ h1 + √ X𝛽0 − wk + √ hT2 𝛽˜ + L w, g(X𝛽0 / 𝑝) 𝑡≥0 𝑝 𝑝 𝑝𝑛 𝑛 + 𝜆 𝑓 ( 𝛽˜ + 𝛼𝛽0 ) . (2.128) ˜ but it also exists inside the function Next, we would like to do the same trick for 𝛽, 𝑓 . We can pull it out using the same trick that we used to pull out X𝛽 out of the loss function. By introducing new variable v to rewrite the optimization as min ˜ 𝑝 v, 𝛽∈R w∈R𝑛 𝛼∈R 1 ˜T 𝑝 𝛽 𝛽0 =0 ˜ v= 𝛽+𝛼𝛽 0 ˜ k 𝛽k 𝛼 𝑡 𝑡 √
max √ k √ h1 + √ X𝛽0 − wk + √ hT2 𝛽˜ + L w, g(X𝛽0 / 𝑝) 𝑡≥0 𝑝 𝑝 𝑝𝑛 𝑛 + 𝜆 𝑓 (v) . (2.129) And then, using Lagrange multiplier x and 𝛾 to bring the constraints over 𝛽˜ to the objective function. ˜ 𝑡 k 𝛽k 𝛼 𝑡 √
min max √ k √ h1 + √ X𝛽0 − wk + √ hT2 𝛽˜ + L w, g(X𝛽0 / 𝑝) ˜ 𝑝 𝑡≥0 𝑝 𝑝 𝑝𝑛 𝑛 v, 𝛽∈R w∈R𝑛 𝛼∈R 𝛾∈R x∈R 𝑝 + 𝜆 𝑓 (v) + 1 T ˜ 𝛾 x ( 𝛽 + 𝛼𝛽0 − v) + 𝛽˜T 𝛽0 . 𝑝 𝑝 (2.130) Now, we would like to solve the optimization over the direction of 𝛽˜ as we did so for u. But the optimization above doesn’t look convex concave with respect to its vari- 71 ables and we are not by default allowed to switch minimization and maximization. At this point, recall that the (PO) in (2.125) is itself convex. In fact, for it, all conditions of Sion’s min-max Theorem [149] are met, thus, the order of min-max operations can be flipped. According to the CGMT, the (PO) and the (AO) are tightly related in an asymptotic setting. We use this, to translate the convexity properties of the (PO) to the (AO). In essence, we show that when dimensions grow, the order of min-max operations in the (AO) can be flipped. Thus, we will instead consider the following problem as the (AO): ˜ 𝑡 𝛼 𝑡 k 𝛽k √
max min √ k √ h1 + √ X𝛽0 − wk + √ hT2 𝛽˜ + L w, g(X𝛽0 / 𝑝) min 𝑝 vR 𝑡≥0 𝜎= k√𝛽˜ k 𝑛 𝑝 𝑝 𝑝𝑛 𝑛 w∈R 𝛼∈R 𝜎≥0 𝛾∈R x∈R 𝑝 𝑝 + 𝜆 𝑓 (v) + 1 T ˜ 𝛾 x ( 𝛽 + 𝛼𝛽0 − v) + 𝛽˜T 𝛽0 . 𝑝 𝑝 (2.131) Observe that the objective function remains the same; it is only the order of min-max operations that is slightly modified. Since the objective function is not necessarily convex-concave in its arguments, there is no immediate guarantee that the two problems in (2.130) and (2.131) are equivalent for all realizations of h1 and h2 . However, the lemma below essentially shows that such a strong duality holds with high probability over h1 and h2 in high dimensions. Hence, the problem in (2.131) can be as well used, instead of the one in (2.130), in order to analyze the (PO). For this reason, henceforth, we refer to (2.131) as the (AO) problem. ˆ˜ Lemma 9 Let 𝛽(X) denote an optimal solution of (2.125). Consider the (AO) problem in (2.131). Let 𝜎∗ be the value that the optimal 𝜎 in (2.131) converges to. ˜ 2 − 𝜎∗ | < 𝜖 }, and, 𝜙S 𝑐 (h1 , h2 ) be the For any 𝜖 > 0 define the set S := { 𝛽˜ | |k 𝛽k optimal cost of the same optimization as in (2.131), only this time the minimization over 𝛽˜ is further constrained such that 𝛽˜ ∉ S. Then,   ˆ˜ − 𝜎 | < 𝜖 lim P |k 𝛽(X)k = 1. 2 ∗ 𝑝→∞ This lemma and its proof is similar to Lemma A.3 in [165]. Now we solve the 72 ˜ optimization over the direction of 𝛽˜ and let 𝜎 := k√𝛽k𝑝 . We get 𝑡 𝜎 𝑡 𝛼 max √ k𝜎h1 + √ X𝛽0 − wk − √ k𝛾𝛽0 + x + √ h2 k 𝑡≥0 𝑝 𝑝 𝑛 𝛿 min𝑝 v∈R w∈R𝑛 𝛾∈R 𝛼∈R,𝜎≥0 x∈R 𝑝 + 1 T √
x (𝛼𝛽0 − v) + L w, g(X𝛽0 / 𝑝) + 𝜆 𝑓 (v) . 𝑝 (2.132) Now, we have gotten closer to our goal which is to build an optimization over scalars. In the next step, we would like to do the optimization over x. What helps us next, is the following trick, 1 𝑥2𝜏 𝑥 = min + . (2.133) 𝜏≥0 2𝜏 2 Using this trick, we would like to square the k · k2 norms in the optimization and get 𝑡 𝑡𝜏1 𝛼 𝜎 𝜎𝜏2 𝑡 min𝑝 max + k𝜎h1 + √ X𝛽0 − wk 2 − − k𝛾𝛽0 + x + √ h2 k 2 v∈R 𝑡,𝜏2 ≥0 2𝜏1 2𝑛 𝑝 2𝜏2 2𝑝 𝛿 𝑛 w∈R 𝛾∈R 𝛼∈R x∈R 𝑝 𝜎,𝜏1 ≥0 + 1 T √
x (𝛼𝛽0 − v) + L w, g(X𝛽0 / 𝑝) + 𝜆 𝑓 (v) . 𝑝 (2.134) Now we can simply do the optimization over x and get min𝑝 v∈R w∈R𝑛 𝛼∈R 𝜎,𝜏1 ≥0 𝑡 𝑡𝜏1 𝛼 𝜎 1 𝑡𝜏2 𝜎 + k𝜎h1 + √ X𝛽0 − wk 2 − + k(𝛼 − 𝜎𝛾𝜏2 ) 𝛽0 + √ h2 − vk 2 𝑡,𝜏2 ≥0 2𝜏1 2𝑛 𝑝 2𝜏2 2𝜎𝜏2 𝑝 𝛿 max 𝛾∈R − 𝜎𝜏2 𝑡 √
k𝛾𝛽0 + √ h2 k 2 + L w, g(X𝛽0 / 𝑝) + 𝜆 𝑓 (v) . 2𝑝 𝛿 (2.135) Now, we use the Monreau envelope notation defined in (2.4), for the minimization over v and w. Recall that the Monreau envelope is defined as eL (x, y, 𝜏) := min v e 𝑓 (x, 𝜏) := min v 1 kv − xk 2 + L (v, y) , 2𝜏 1 kv − xk 2 + 𝑓 (x) . 2𝜏 (2.136) (2.137) √ √ Also let 𝜅 = k 𝛽0 k/ 𝑝. We can replace X𝛽0 / 𝑝 with 𝜅h3 , where h3 is a random vector with iid standard Gaussians. Replacing these in (2.135) yields, 𝑡 𝜎 𝜎𝜏2 𝑡 min max − − k𝛾𝛽0 + √ h2 k 2 𝛼∈R 𝑡,𝜏2 ≥0 2𝜏1 2𝜏2 2𝑝 𝛿 𝜎,𝜏1 ≥0 𝛾∈R     1 𝜎𝜏2 𝑡 + eL 𝜎h1 + 𝛼𝜅h3 , g(𝜅h3 ), + e 𝑓 − √ h2 + (𝛼 − 𝜎𝜏2 𝛾) 𝛽0 , 𝜎𝜏2 . 𝑡𝜏1 𝛿 (2.138) 73 Up to now, analysis of the Auxiliary optimization was precise without letting the problem dimensions grow or applying any convergence lemma. Now, we would like to replace the Monreau envelope functions and and the norm of the random vector above (third term) with their corresponding point-wise convergence functions. But since these functions are inside the min-max, we are typically not allowed to do so. But in this case, we would like to use the following lemma in convex analysis which will make this possible. Lemma 10 Consider a sequence of proper, convex stochastic functions 𝑀𝑛 : (0, ∞) → R, and, a deterministic function 𝑀 : (0, ∞) → R, such that: 𝑃 1. 𝑀𝑛 (𝑥) − → 𝑀 (𝑥), for all 𝑥 > 0, 2. there exists 𝑧 > 0 such that 𝑀 (𝑥) > inf 𝑥>0 𝑀 (𝑥) for all 𝑥 ≥ 𝑧. 𝑃 Then, inf 𝑥>0 𝑀𝑛 (𝑥) − → inf 𝑥>0 𝐹 (𝑥). It’s not hard to check that the objective function in (2.138) satisfies the assumptions of lemma 10. By applying this lemma consecutively on (2.138), and using Assumptions 1, we can do the following replacements in the objective function of (2.138), 2 𝜎𝜏2 𝑡 𝜎𝜏2 𝛾 2 𝜅 2 𝑃 𝜎𝜏2 𝑡 k𝛾𝛽0 + √ h2 k 2 − → + 2𝑝 2𝛿 2 𝛿   1 𝑃 1 ) eL 𝜎h1 + 𝛼𝜅h3 , g(𝜅h3 ), − → 𝐿 (𝜎, 𝛼, 𝑡𝜏1 𝑡𝜏1   𝜎𝜏2 𝑡 𝜎𝜏2 𝑡 𝑃 e 𝑓 − √ h2 + (𝛼 − 𝜎𝜏2 𝛾) 𝛽0 , 𝜎𝜏2 − → 𝐹 (− √ , 𝛼 − 𝜎𝜏2 𝛾, 𝜎𝜏2 ) , 𝛿 𝛿 (2.139) where the functionals 𝐿 and 𝐹 are defined in Assumption 1. Therefore, (2.138) becomes min 𝛼∈R 𝜎,𝜏1 ≥0 𝑡 𝜎 𝜎𝜏2 𝑡 2 𝜎𝜏2 𝛾 2 𝜅 2 1 𝜎𝜏2 𝑡 − − + + 𝐿 (𝜎, 𝛼, ) + 𝐹 (− √ , 𝛼 − 𝜎𝜏2 𝛾, 𝜎𝜏2 ) . 𝑡,𝜏2 ≥0 2𝜏1 2𝜏2 2𝛿 2 𝑡𝜏1 𝛿 max 𝛾∈R (2.140) This is the scalar optimization in Theorem 1. Let’s denote its solutions with the scalars ( 𝛼, ˆ 𝜎, ˆ 𝜏ˆ1 , 𝜏ˆ2 , 𝑡ˆ, 𝛾). ˆ Then, if one follows the steps of the analysis of the (AO), it can be observed that the final solution 𝛽ˆ in the Auxiliary optimization (2.127) in terms of the scalars (𝛼, 𝜎, 𝜏1 , 𝜏2 , 𝑡, 𝛾), is   ˆ 𝜎 ˆ 𝜏 ˆ 𝑡 2 𝛽ˆ(AO) = Prox 𝑓 ( 𝛼ˆ − 𝜎 ˆ 𝜏ˆ2 𝛾) ˆ 𝛽0 + ( √ )h2 , 𝜎 ˆ 𝜏ˆ2 (2.141) 𝛿 74 Therefore, using Assumption 2, for the auxiliary optimization, we have 𝜎 ˆ 𝜏ˆ2 𝑡ˆ lim 𝜓( 𝛽ˆ(AO) , 𝛽0 ) = Ψ( 𝛼ˆ − 𝜎 ˆ 𝜏ˆ2 𝛾, ˆ √ ,𝜎 ˆ 𝜏ˆ2 ) . 𝑝,𝑛→∞ 𝛿 (2.142) Now, let’s define the set S to be S = {(𝛼, 𝜎, 𝜏1 , 𝜏2 , 𝑡, 𝛾) 𝜎𝜏2 𝑡 𝜎 ˆ 𝜏ˆ2 𝑡ˆ s.t. |Ψ(𝛼 − 𝜎𝜏2 𝛾, √ , 𝜎𝜏2 ) − Ψ( 𝛼ˆ − 𝜎 ˆ 𝜏ˆ2 𝛾, ˆ √ ,𝜎 ˆ 𝜏ˆ2 )| < 𝜖 } 𝛿 𝛿 (2.143) Since the solution to the optimization (2.140) is unique, if we optimize the auxiliary optimization over S c , the value of optimal objective is going to increase. Thus, we can utilize the third part of Theorem 8 to show that the same convergence happens in Primary Optimization and we will have 𝜎 ˆ 𝜏ˆ2 𝑡ˆ ˆ 𝜏ˆ2 ) . lim 𝜓( 𝛽ˆ(PO) , 𝛽0 ) = Ψ( 𝛼ˆ − 𝜎 ˆ 𝜏ˆ2 𝛾, ˆ √ ,𝜎 𝑝,𝑛→∞ 𝛿 This concludes the proof. (2.144) 75 Chapter 3 GENERAL PERFORMANCE METRICS FOR THE LASSO In this chapter1, we extend the applicability of the CGMT framework and the precise results that it yields to more general performance metrics. For concreteness, we focus primarily on the problem of sparse recovery under ℓ1 -regularized leastsquares (a.k.a LASSO). We also discuss how the results extend to more general structured signal recovery problems and to a wide family of convex recovery methods known as regularized M-estimators. We establish accurate predictions of a wide range of performance metrics that have a Lipschitz property. For illustration,this result can be used to accurately predict the probability that the LASSO successfully identifies the non-zero entries of the unknown signal; specializing the result to the high-SNR regime yields bounds that are geometric in nature and admit insightful interpretations. There is an increasing line of work on the precise analysis of regularized Mestimators. Please see [165, Sec. 7] for an exhaustive review. We have already referred to the works that use the CGMT framework [129, 154, 165, 171]. The most general result is included in [165] which characterizes the ℓ2 -reconstruction error of general regularized M-estimators under very generic settings; yet, no other performance metrics have been considered thus far. A different line of works is based on a state evolution framework for an iterative Approximate Message Passing (AMP) algorithm inspired by statistical physics ([18, 63] and the references therein). To the best of our knowledge, the AMP framework has not been used to analyze general regularized M-estimators. Nonetheless, [18, 63, 119] have considered Lipschitz performance metrics for the LASSO; our result extends the formulae to general regularized M-estimators. Overall, the two methods of analysis are very different (and of their own value each); this is the first time that the CGMT framework is used for general performance metrics. 3.1 Problem Setup Consider the problem of recovering a sparse signal 𝛽0 ∈ R 𝑝 comprised of only 𝑘 non-zero measurements from 𝑛 noisy linear observations of the form y = X𝛽0 + z ∈ R𝑛 , where X is the measurement matrix and z is the noise vector. The typical 1 This chapter is mostly based on [4] 76 approach to produce an estimate 𝛽ˆ of 𝛽0 is by solving an ℓ1 -regularized least-squares minimization, as follows: 𝜆 𝛽ˆ = arg min ky − X𝛽k2 + √ k 𝛽k1 . (3.1) 𝛽 𝑝 √ Here, 𝜆 > 0 is a regularization parameter. (The normalization with 𝑝 is for convenience in the analysis). This method is known as row Square-root LASSO in the statistics literature [22], and is a slight variation of the popular LASSO; see [129] for a discussion. Our analysis applies to both instances, but we focus on the former for concreteness. Also, for convenience, we shall often refer to (3.1) simply as the LASSO. Measuring Performance A “good estimate" might translate to a variety of different desired attributes associˆ This translates to a variety of different performance metrics, which we ated with 𝛽. discuss here. ℓ2 -reconstruction error: A standard and somewhat generic measure of performance is the ℓ2 -reconstruction error, which measures the deviation of 𝛽ˆ from the true signal 𝛽0 in the ℓ2 -norm. Formally, the metric acts on the reconstruction error vector ŵ := 𝛽ˆ − 𝛽0 and returns its Euclidean norm, i.e., Ψℓ2 ( ŵ) := k ŵk2 = k 𝛽ˆ − 𝛽0 k2 . The ℓ2 -error in estimating the coefficients of 𝛽0 also controls the mean squared prediction error, i.e. the error in predicting a (future) response to a fresh (random) measurement (e.g. [130, Sec. 8.1]). Lipschitz Metrics: Beyond the ℓ2 -reconstruction error, we consider performance metrics Ψ : R 𝑝 → R that act on the error vector ŵ := 𝛽ˆ − 𝛽0 and which satisfy a Lipschitz property, i.e. |Ψ(x) − Ψ(y)| ≤ 𝐿 · kx − yk2 for all x, y ∈ R 𝑝 and some 𝐿. One common such metric is Ψ(w) = kwk1 , [123]. Support Recovery: In the problem of sparse recovery a natural performance metric that arises in a variety of contexts (e.g. subset selection in regression, structure estimation in graphical models, sparse approximation [188]) is that of support recovery, i.e. identifying whether an entry of the unknown signal 𝛽0 is on the support (aka is non-zero), or it is off the support (aka is zero). We take a decision based on the solution 𝛽ˆ of the LASSO: declare the 𝑖 𝑡ℎ entry to be on the support ˆ such a hardiff | 𝛽ˆ𝑖 | ≥ 𝜖. Here 𝜖 > 0 is a user-defined threshold imposed on 𝛽; thresholding operation is practical due to machine precision inaccuracies in solving (3.1). 77 In Theorem 10 we accurately predict the (per-entry) rate of successful on-support and off-support recovery. Formally, let ˆ = 1 Φ𝜖,on ( 𝛽) 𝑘 ˆ = Φ𝜖,off ( 𝛽) Õ 1{| 𝛽ˆ𝑖 |≥𝜖 } (3.2a) 𝑖∈𝑆(𝛽0 ) Õ 1 1{| 𝛽ˆ𝑖 |≤𝜖 } , 𝑛−𝑘 (3.2b) 𝑖∉𝑆(𝛽0 ) ˆ (resp. Φ𝜖,off ( 𝛽)) ˆ where 1S is the indicator function of a set S. The metric Φ𝜖,on ( 𝛽) measures the ratio of the non-zero (reps. zero) entries of 𝛽0 that are properly identified to be on (resp. off) the support. An equivalent way to interpret the metrics defined above is to consider their exˆ = (1/𝑘) Í𝑖∈S(𝛽 ) P(| 𝛽ˆ𝑖 | ≥ 𝜖) measures the pectation. For instance, E[Φ𝜖,on ( 𝛽)] 0 average probability that a single non-zero entry of 𝛽0 is correctly identified to be ˆ on the support. In particular, if the entries of 𝛽ˆ are iid, then in the limit Φ𝜖,on ( 𝛽) converges to the probability that a single on-support entry is correctly identified. Working Hypothesis The unknown signal 𝛽0 ∈ R 𝑝 is k-sparse: its first 𝑘 entries are sampled iid from a distribution 𝑝 𝛽0 of zero mean and of unit variance (E[(𝛽0 )𝑖2 ] = 1), and the rest of them are zero. The measurement matrix X ∈ R𝑛×𝑝 has entries iid zero mean Gaussian random variables with variance 1𝑝 (denote N (0, 1/𝑝)). The noise vector z ∈ R𝑛 has entries iid N (0, 𝜎 2 ). We study the linear asymptotic regime in which the problem dimensions 𝑝, 𝑛 and 𝑘 all grow to infinity at proportional rates 2: 𝑘/𝑝 → 𝜌 ∈ (0, 1) and 𝑛/𝑝 → 𝛿 ∈ (0, ∞). Also, the regularizer parameter 𝜆 in (3.1) is considered to be constant, in particular independent of 𝑝. Under the current setting, the Signal to Noise Ratio (SNR) becomes SNR := 𝜌/𝜎 2 . 3.2 Results We gather our main results in this section. For a sequence of random variables 𝑃 {X ( 𝑝) } and a constant 𝑐, {X (𝑛) } − → 𝑐 denotes convergence in probability as 𝑝 → ∞. We reserve the letters 𝐻 and 𝑋0 to denote (scalar) random variables with distributions N (0, 1) and 𝑝 𝛽0 , respectively. 2 The results of Section 3.2 apply on a sequence of problem instances {𝛽 , X, z, 𝑛, 𝑘 } indexed 0 𝑝 by 𝑝 ∈ N such that the properties mentioned hold for all members of the sequence for all 𝑝. To keep notation clear we do not explicitly use the subscript 𝑝 for symbols of the sequence. 78 ℓ2 -reconstruction Error The precise characterization of the ℓ2 -reconstruction error has been performed in [164, 172] via the CGMT framework (also, [18, 63] have analyzed the problem via an alternative framework called AMP). We include a statement of the result here since it helps us set up some necessary definitions for the presentation of the more general result that follows in the next section. 𝜓-distance functional: For a function 𝜓 : R → R, let Dist𝜓(.) (·, ·) : R × R>0 → R be defined as Dist𝜓(.) (𝜅, 𝜆) := 𝜌 · E[𝜓(𝑋0 − 𝜂(𝜅𝐻 + 𝑋0 , 𝜅𝜆))] (3.3) + (1 − 𝜌) · E[𝜓(𝜂(𝜅𝐻, 𝜅𝜆))], where the expectation is over both 𝑋0 ∼ 𝑝 𝛽0 and 𝐻 ∼ N (0, 1), and 𝜂(𝑋, 𝜏) = (𝑋/|𝑋 |) max{|𝑋 |−𝜏, 0} denotes the soft-thresholding operator. The function returns the distance, with respect to the function 𝜓(.), between a r.v. 𝑋0 and the soft threshold operator applied to the random variable itself after adding a Gaussian noise to it. This motivates the terminology used. Also, note the implicit dependence of the functional on the rest of the problem parameters, namely 𝜌, 𝛿 and 𝜎. noindent𝜆 crit : There exists a critical value of the regularizer parameter, namely 𝜆 crit , such that the error behavior is different when 𝜆 ≤ 𝜆 crit compared to 𝜆 > 𝜆 crit [129]. Define the pair (𝛼crit , 𝜆crit ) as the solution to the following system of equations:   2 = Dist  𝛼crit  (.) 2 (𝜅 crit , 𝜆 crit ), (3.4)   𝛿 = 𝜌 · P {|𝜅𝐻 + 𝑋0 | ≥ 𝜆 crit 𝜅 𝑐𝑟𝑖𝑡 } + 2(1 − 𝜌)𝑄(𝜆 crit ),  q 2 + 𝜎 2 )/𝛿 and 𝑄(·) is the standard Q-function. It is shown in where 𝜅 𝑐𝑟𝑖𝑡 = (𝛼crit [164, Sec. 2.C] that if 𝛿 ≤ 1, then (3.4) has a unique solution. Otherwise, define 𝜆crit = 0. With these we are ready to state the first lemma. Lemma 11 ([164]) Under the working hypothesis of Section 3.1 and for any fixed p 𝜆 > 0, define 𝛼 := 𝛼(𝜆) as the unique solution to the equation 𝛼2 = Dist (.) 2 ( (𝛼2 + 𝜎 2 )/𝛿, 𝜆), if 𝜆 ≥ 𝜆crit , and, as 𝛼 = 𝛼crit , otherwise. Then, it holds in probability that lim 𝑝→∞ 1𝑝 k 𝛽ˆ − 𝛽0 k22 = 𝛼2 . Extensive empirical evidence suggest that the system of the two nonlinear equations in two unknowns in (3.4) can be solved numerically very efficiently using a simple iterative fixed-point method (see also [165, Rem. 4.3.3]). Figure 3.1 below illustrates the accuracy of the lemma. 79 Lipschitz Performance Metrics Theorem 9 below generalizes Lemma 11 to metrics that attain a Lipschitz property. Assumption 3 below formally defines the required properties of such metrics. Assumption 3 (Lipschitz metrics) We say Assumption 3 holds for the Lipschitz function Ψ : R 𝑝 → R if • For all constants 𝑐 > 0, there exists a constant 𝐶 > 0 such that for all 𝛽 ∈ R 𝑝 that √ √ k 𝛽k ≤ 𝑐 𝑝, we have |Ψ(𝛽)| ≤ 𝐶 𝑝. • For all x, y ∈ R 𝑝 , |Ψ(x) − Ψ(y)| ≤ √𝐿𝑝 kx − yk2 , for a constant 𝐿 independent on 𝑝. • For all 𝛼, 𝜆 > 0 and h ∼ N (0, I 𝑝 ), there exists function Γ : R>0 × R>0 → R such that 𝑃 Ψ(𝛽0 − 𝜂(𝜅h + 𝛽0 , 𝜆𝜅)) − → Γ(𝜅, 𝜆). (3.5) Here, 𝜂 is the “vector" soft-threshold operator acting element-wise on the entries of its first argument. The first is a simple scaling requirement such that Ψ(x) = O (1). The second imposes a growth condition on the Lipschitz constant with respect to 𝑝 (this is necessary for the asymptotic analysis but can potentially be relaxed). The third requirement of Assumption 3 is easier to interpret in the “separable-case" in which Í Ψ(x) = (1/𝑏) 𝑖 𝜓(x𝑖 ) for some 𝐿-Lipschitz scalar function 𝜓. Then, condition (3.5) holds by the WLLN for Γ(𝜅, 𝜆) = Dist𝜓 (𝜅, 𝜆) (recall (3.3)). Theorem 9 (Lipschitz performance of LASSO) Under the working hypothesis of Section 3.1 and with 𝛼 and 𝜆 crit defined as in Lemma 11, fix 𝜆 > 0, let 𝜆ˆ = √ √ 𝑚𝑎𝑥{𝜆, 𝜆 crit } and 𝜅 = 𝛼2 + 𝜎 2 / 𝛿. Then, for any Lipschitz function Ψ(𝑥) that ˆ satisfies Assumption 3, it holds in probability that, lim 𝑝→∞ Ψ( 𝛽ˆ − 𝛽0 ) = Γ(𝜅, 𝜆). Evaluating the prediction only involves identifying the function Γ as per Assumption 3, and calculating the parameters 𝛼 and 𝜆 crit as per Lemma 11. Of course, Lemma 11 follows from Theorem 9 when applied for Ψ( 𝛽ˆ − 𝛽0 ) = √1𝑝 k 𝛽˜ − 𝛽0 k2 , since p the latter is easily shown to satisfy Assumption 3 for Γ(𝜅, 𝜆) = Dist (.) 2 (𝜅, 𝜆). A different Lipschitz performance metric that is often of interest in practice is the ℓ1 reconstruction errof Ψ( 𝛽ˆ − 𝛽0 ) = (1/𝑝)k 𝛽ˆ − 𝛽0 k1 . This is an example of a separable 80 0.4 0.35 ℓ2 -reconstruction error ℓ1 -reconstruction error 0.3 Ψ(x0 − x̂) 0.25 0.2 0.15 0.1 0.05 0 0 λ 0.5 Crit 1 1.5 2 2.5 3 λ Figure 3.1: Performance of Square root Lasso with respect to Ψ(x) = √1𝑝 kxk2 (Red Line) and Ψ(x) = 𝑝1 kxk1 (Blue line) as a function of 𝜆. The theoretical prediction comes from Theorem 9. For the simulations, we used 𝑝 = 256, 𝛿 = 0.8, 𝜌 = 0.1, SNR=0.5 and the data are averaged over 5 independent realization of the Problem. metric, thus it satisfies Assumption 3 for Γ(𝜅, 𝜆) = Dist|·| (𝜅, 𝜆). See Figure 3.1 for an illustration. Observe that the prediction of the theorem (although asymptotic) is accurate for problem dimensions of only a few hundreds. Also, the precise nature of the predictions allows optimal tuning of the regularizer parameter 𝜆, the number of measurements 𝛿, etc.. Support Recovery Theorem 10 below characterizes the support recovery metrics introduced in (3.2). Recall that 𝜖 > 0 is a fixed hard threshold imposed on the entries of the solution 𝛽ˆ to the LASSO in order to decide whether an entry is on or off the support. Theorem 10 (Probability of support recovery) Under the working hypothesis of Section 3.1 and with 𝛼 and 𝜆 crit defined as in Lemma 11, fix 𝜆 > 0, let 𝜅 = p (𝛼2 + 𝜎 2 )/𝛿 and 𝜆ˆ = max{𝜆 crit , 𝜆}. Then, for any 𝜖 > 0, it holds in probability ˆ = P{|𝜅𝐻 + 𝑋0 | ≥ 𝜖 + 𝜆𝜅} ˆ and that lim 𝑝→∞ Φ𝜖,on ( 𝛽) 81 ˆ = P{|𝜅𝐻| ≤ 𝜖 + 𝜆𝜅}. ˆ lim 𝑝→∞ Φ𝜖,off ( 𝛽) The metrics in (3.2) are not Lipschitz. Hence, they don’t satisfy all requirements of Assumption 3 of Section 3.2, and Theorem 9 is not directly applicable. Nonetheless, the core idea behind the proof of the theorem is similar to that of Theorem 9 and requires only a few extra arguments (see Section 3.3). Figure 3.2 illustrates the validity of the prediction. ˆ the formula of the theorem for Φ𝜖,off ( 𝛽) ˆ Remark 23 (Off-support) When 𝜖  𝜆𝜅, ˆ ∼ P{|𝐻| ≤ 𝜆}, ˆ which is independent of the problem reduces to P{|𝜅𝐻| ≤ 𝜖 + 𝜆𝜅} parameters 𝛿, 𝜌 and SNR. This simple observation is verified in Figure 3.2: the off-support recovery probability is the same for different values of under-sampling parameter 𝛿 as long as 𝜆 ≥ 𝜆 crit . Remark 24 (Large/Small 𝜆) It is easy to conclude from Theorem 10 that as 𝜆 becomes large Φ𝜖,off (reps. Φ𝜖,on ) converge to one (resp. zero). Of course, this behavior is expected since large values for the regularizer parameter put more emphasis on the ℓ1 -regularization term in (3.1), thus promoting sparser solutions. Reversed behavior is observed when 𝜆 takes values close to zero. Remark 25 (Optimal 𝜆) A natural question becomes that of determining the optimal value of the regularizer parameter. In order to balance between on- and off- support recovery probabilities a reasonable performance metric becomes Φ𝜖 = 𝜔Φ𝜖,on + (1 − 𝜔)Φ𝜖,off for 𝜔 ∈ [0, 1]. Theorem 10 precisely characterizes the behavior of this as a function of 𝜆; thus, it determines the optimal value of 𝜆 that minimizes Φ𝜖 . Remark 26 (High-SNR Regime) Here, we analyze the probability of support recovery at SNR  1 (eqv. 𝜎 2 → 0). In this regime, 𝜆 crit takes a simple form: if 𝛿−𝜌 ) where 𝑄 −1 is the inverse Q-function, otherwise, 𝛿 < 1, then 𝜆 crit = 𝑄 −1 ( 2(1−𝜌) 𝜆crit = 0 [129, Sec. 8]. Let us first examine the behavior of “off-support" recovery probability. When 𝜎 2  1, the formula of Theorem 10 reduces to the following simpler one:   q 𝜖 ˆ ∼ 1 − 2𝑄 𝜆ˆ + ˆ , lim Φ𝜖,off ( 𝛽) 𝛿 − 𝐷 (𝜆) (3.6) 𝑝→∞ 𝜎 82 Probability of Successful On and Off Support Recovery 1 Prob. of Successful Recovery 0.9 On Support Off Support 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 λ Crit 0.5 1 1.5 2 2.5 3 λ Figure 3.2: Probability of successful detection of support and off support entries as a function of 𝜆 for two different problem setup. The theoretical prediction (Solid and dashed lines) comes from Theorem10. For the simulations (Squares and Circles), we used 𝑝 = 256, SNR= 0.5, 𝜖 = 10−3 , 𝜌 = 0.1 and the data are averaged over 5 independent realizations of problem. For solid lines and squares and circles, we used 𝛿 = 0.8, while for dashed lines and empty squares and circles 𝛿 = 1.2. ˆ 𝜆ˆ = max{𝜆, 𝜆 crit }, and for 𝜆 such that 𝛿 > 𝐷 (𝜆), 𝐷 (𝜆) = 𝜌 · E[(𝐻 − 𝜆) 2 𝐻 > 0] + (1 − 𝜌) · E[𝜂2 (𝐻, 𝜆)]. Several remarks are in place here. First, when the threshold 𝜖 does not scale with 𝜎 and 𝜎 → 0, then naturally the probability converges to one. The same is true as 𝜆 grows large, which is again expected. Finally, the term 𝐷 (𝜆) is known as the “Gaussian squared distance" in the relevant literature of noiseless linear inverse problems and admits insightful geometric interpretations [6, 73, 129]. In particular, min𝜆>0 𝐷 (𝜆) is known to be an asymptotically tight approximation of the exact phase transition threshold of ℓ1 -minimization [6, 153]. This can also be seen in (3.6) which requires 𝛿 > min𝜆>0 𝐷 (𝜆). Also, the formula is valid for 𝜆 such that ˆ 𝛿 > 𝐷 (𝜆). ˆ ∼ For the on-support probability, we can show that it behaves as lim 𝑝→∞ Φ𝜖,On ( 𝛽) p  ˆ Similar remarks can be made. P | 𝜅𝐻 ˆ + 𝑋0 | ≥ 𝜖 + 𝜆ˆ 𝜅ˆ for 𝜅ˆ = 𝜎/ 𝛿 − 𝐷 (𝜆). 83 Generalizations The results of Section 3.2 on Lipschitz-like performance metrics for the LASSO extend to general regularized M-estimators. We outline the result here and defer a detailed treatment to the extended version of the paper. Regularized M-estimators solve 𝛽ˆ = arg min L (y − X𝛽) + 𝜆 𝑓 (𝛽), 𝛽 (3.7) where L : R𝑛 → R is a proper continuous convex loss function and 𝑓 a convex regularizer. Clearly, the LASSO is an instance of (3.7) for L (·) = k · k2 and 𝑓 (·) = k · k1 . Depending on the noise distribution and on the particular structure of 𝛽0 , different choices for the loss and regularizer function might be more appropriate [165]. Here, for simplicity we focus on separable functions of the form L (x) = Í𝑛 Í𝑝 (1/𝑝) 𝑖=1 ℓ(x𝑖 ) (wlog, ℓ(0) = 0) and 𝑓 (x) = (1/𝑝) 𝑖=1 𝑓˜(x𝑖 ) for non-negative convex functions 𝑓˜, ℓ : R × R. Also, extending on the assumption of Section 3.1 we assume that the entries of the noise vector z are sampled iid from a distribution 𝑝 𝑍 (not necessarily Gaussian). the (rather mild) assumptions i h We only require  0  2 2 0 E |ℓ (𝑐𝐻 + Z)| < ∞ and E 𝑓˜ (𝑐𝐻 + 𝑋0 ) < ∞ (see [165, Sec. 4] for details). where the expectations are taken over 𝑍 ∼ 𝑝 𝑍 , 𝑋0 ∼ 𝑝 𝑋 and 𝐻, 𝐺 ∼ N (0, 1). Here, 𝑓˜0 (𝑥) = sup𝑠∈𝜕 𝑓˜(𝑥) |𝑠| where 𝜕 𝑓˜(𝑥) is the subdifferential of 𝑓˜ at 𝑥, and similar for ℓ0. We also need to recall the notion of the Moreau Envelope function. For a convex function 𝜙 : R → R, the Moreau Envelope function of 𝜙 at 𝑥 with parameter 𝜏 > 0 is 1 k𝑥 −𝑦k 2 +𝜙(𝑦) We denote the optimal value of 𝑦 above defined as e 𝑓 (𝑥; 𝜏) := min𝑦 2𝜏 as prox𝜙 (𝑥, 𝜏). With these, [165, Thm. 4.1] characterizes the ℓ2 -reconstruction error of general regularized M-estimators as follows. There exists a unique 𝛼 for which it holds in probability that lim 𝑝→∞ 1𝑝 k 𝛽ˆ − 𝛽0 k 2 = 𝛼2 , where 𝛼 is the solution to the following system of four equations in four unknowns (if such a solution does not exist then 𝛼 = 0):  2   𝛾 𝜆 2  𝛼 = E 𝑋0 − prox 𝑓 ( 𝜈 𝐻 + 𝑋0 ; 𝜈 )     h i    𝛾 2 = 𝛿E e0 (𝛼𝐻 + Z; 𝜅)
2  ℓ     0 (𝛼𝐻 + Z; 𝜅)  𝜈𝛼 = 𝛿E 𝐻 · e  ℓ        𝜅𝛾 = E 𝐻 · prox 𝑓 ( 𝛾𝜈 𝐻 + 𝑋0 ; 𝜆𝜈 )  (3.8) 84 Imposing an extra requirement that the loss function be strongly convex and following similar steps as in the proof of Theorem 9 (see Section 3.3), we extend the result to Lipschitz performance metrics. In particular, for any Ψ satisfying Assumption 3 it holds in probability lim 𝑝→∞ Ψ( 𝛽ˆ − 𝛽0 ) = Γ 𝑓 (𝛾/𝜈, 𝜆/𝛾), where (𝛼, 𝛾, 𝜈, 𝜅) are as in (3.8) and the function Γ 𝑓 : R>0 × R>0 → R is defined as 𝑃 Ψ(𝑋0 − prox 𝑓 (𝜅𝐻 + 𝑋0 ; 𝜆) − → Γ 𝑓 (𝜅, 𝜆). 3.3 Proof Outline Our analysis is based on the recently developed Convex Gaussian Min-max Theorem (CGMT) framework [165, 171]. The following lemma specializes the general result of [165, Thm. 6.1] to the LASSO method in (3.1). Theorem 11 (CGMT for LASSO) Let X, z, 𝛽0 be as in Section 3.1, g ∈ R𝑛 , h ∈ R 𝑝 have entries iid N (0, 1) and all be independent of each other. Consider the optimizations: 𝜆 1 min √ kz − Xwk2 + k 𝛽0 + wk1 , w 𝑝 𝑝 s h𝑇 w 𝜆 kgk2 kwk22 + 𝜎 2 − 𝛾 √ + k 𝛽0 + wk1 . min max 𝛾 √ w 0<𝛾≤1 𝑝 𝑝 𝑝 𝑝 (3.9) (3.10) Denote 𝜙(g, h) the optimal cost of the latter. Further, for an open set S ⊂ R𝑛 denote 𝜙S c (g, h) its optimal cost when the minimization over w is now constrained 𝑃 over w ∈ S. Suppose there exists constants 𝜙 < 𝜙S such that 𝜙(g, h) − → 𝜙 𝑃 and 𝜙S c (g, h) − → 𝜙S c . Then, for any minimizers wΦ and w𝜙 of (3.9) and (3.10), respectively, the events {w𝜙 ∈ S} and {wΦ ∈ S} occur with probability 1 in the limit of 𝑝 → ∞. The minimization in (3.9) corresponds to the LASSO, only now the optimization variable is the error vector w := 𝛽 − 𝛽0 . To see how the theorem is applicable, 𝑃 𝑃 suppose we are interested in showing Ψ( 𝛽ˆ − 𝛽0 ) − → 𝛼∗ (eqv. Ψ(wΦ ) − → 𝑑∗ ) for some constant 𝑑∗ . Then, we need to apply Theorem for the set SΨ := {w |Ψ(w) −𝑑∗ | < 𝛿}, where 𝛿 > 0 is an arbitrarily small constant. The theorem suggests that 𝑑∗ is the converging limit of Ψ(w𝜙 ), the solution to the Auxiliary Optimization (AO) in (3.10). The strategy becomes now clear [165]. First, we need to analyze the (AO) problem in (3.10) and find the converging limit of Ψ(w𝜙 ), say 𝑑∗ . The second step consists of showing that the objective function of the (AO) strictly increases when 85 w is constrained such that Ψ(w𝜙 ) is far from 𝑑∗ . Of course, the premise of this machinery is that these two tasks are much simpler to complete for the (AO) rather than for the original LASSO minimization [165]. The basics of these steps are outlined in the next two sections; details are deferred to the extended version of the paper. Proving Theorem 9 It is shown in [164] that w𝜙,𝑖 = 𝛽0,𝑖 − 𝜂(𝜅(g, h)h𝑖 + 𝛽0,𝑖 , 𝜅(g, h) · 𝜆) where 𝜅(g, h) := p √ 𝛼2 (g, h) + 𝜎 2 / 𝛿 and 𝛼(g, h) can be expressed as the minimizer of a random optimization problem (e.g. [164, eqn. (46)]). Moreover, it is shown in [164] that 𝛼(g, h) converges to 𝛼 as this is defined in Lemma 11. Conditioning on the h.p. 𝑃 −𝜂 (𝜅h + 𝛽 , 𝜅 · 𝜆)). But event that 𝛼(g, h) → 𝛼, we show that Ψ(w𝜙,𝑖 ) − → Ψ(𝛽0 − → 0 the latter term converges to Γ(𝜅, 𝜆) by assumption, thus showing that the formula of Theorem 9 holds for the solution of the (AO) problem. It is also worth mentioning that the above argument shows the entries of w𝜙 to be asymptotically iid. Next, we need to verify that the objective function of the (AO) strictly increases when w is such that |Ψ(w) − Γ(𝜅, 𝜆)| > 2𝛿 > 0. First, if w is such, then using the result of the previous section it follows that |Ψ(w) − Ψ(w𝜙 )| > 𝛿 with probability √ approaching 1. Then, the Lipschitzness property of Ψ implies that kw − w𝜙 k2 / 𝑝 > 𝛿/𝐿. The desired conclusion then follows by showing that the objective function in (3.10) q is strongly convex in w and recalling optimality of w𝜙 . In particular, 𝛾kgk2 kwk22 /𝑛 + 𝜎 2 is strongly convex with coefficient 𝜏/𝑝 for some constant 𝜏 > 0 (independent of p). Thus for the objective function of the optimization in kw−w 𝜙 k22 (3.10) (say 𝐹 (·)) it holds 𝐹 (w) ≥ 𝐹 (w𝜙 ) + 𝐶 ≥ 𝐹 (w𝜙 ) + 𝜏𝛿 𝑝 𝐿. On the Support Recovery The two metric defined in (3.2) do not satisfy the Lipschitz property. Nevertheless the proof of Theorem 10 follows from Theorem 9 when combined with a weakconvergence argument. Let 𝜓 : R → R be arbitrary 𝐿-Lipschitz function. By Í 𝑃 Theorem 9, (1/𝑝) 𝑖 𝜓(wΦ,𝑖 ) − → Dist𝜓 (𝜅, 𝜆). Since this holds for all Lipschitz functions, the empirical probability measure of wΦ converges [25, Thm. 25.8]. 𝑃 Hence, it follows (almost identically as in [25, Thm. 19]) that Ψ𝜖,𝑜𝑛 − → Γ(𝜅, 𝜆), where Í𝑘 Γ as in Assumption 3 for the function Ψ(w) = 1/𝑘 𝑖=1 1{|w𝑖 −𝛽0,𝑖 |≥𝜖 } . Simplifying the “Γ(𝜅, 𝜆)-term" yields the statement of Theorem 10. 86 Chapter 4 SPARSE COVARIANCE ESTIMATION FROM QUADRATIC MEASUREMENTS: A PRECISE ANALYSIS The problem of covariance estimation1 arises in many areas of modern statistics and information processing systems such as finance [5], when the underlying signal is high-dimensional, and/or the memory or computation power is limited. Therefore, in the current big data era, finding an efficient algorithm (in terms of sample complexity and memory requirements) to accurately estimate the covariance matrix is of great importance. Recently, in [44, 48, 145], a framework for covariance estimation using phaseless, or energy measurements has been proposed. This type of measurements find applications in covariance estimation of data streams [44, 120], spectrum estimation of stochastic processes from energy measurements, noncoherent subspace detection from energy measurements [145], and others. Mathematically, given an unknown covariance matrix 𝚺0 ∈ R 𝑝×𝑝 , the problem reduces to estimating 𝚺0 from a number of 𝑚 quadratic samples of the form a𝑖T 𝚺0 a𝑖 +𝑧𝑖 , 𝑖 = 1, . . . , 𝑚. Here, the measurement vectors a𝑖 ’s are given and 𝑧𝑖 ’s represent noise. [44, 48, 145] provide different convex and non-convex optimization algorithms for the recovery of 𝚺0 . in this section, we provide a convex optimization formulation that is similar to that of [44], and precisely characterize its performance. Moreover, in the noiseless setting, our analysis framework provides the exact phase transition, i.e., the necessary and sufficient number of measurements for perfect recovery of the underlying covariance matrix. We are particularly interested in analyzing the underdetermined case, where we have measurements and 𝑛 denotes the number of variables. In such settings, 𝑛 < 𝑝( 𝑝+1) 2 the problem is ill-posed and the recovery is not possible unless the covariance matrix belongs to a low-dimensional set. In many practical settings, the covariance matrix possesses certain structures. For instance, one common assumption is that the pairwise correlation has small magnitude for many pairs of entries of the underlying random vector, and hence, the covariance matrix has many (near)zero entries. in this section, we focus on the problem of recovery of a sparse covariance matrix. The convex optimization formu1 This chapter in mainly based on the work in [2] 87 lation includes a regularization term that only enforces sparsity on the non-diagonal entries of the matrix. We then analyze the error performance of the solution. An order-wise analysis of a similar convex estimator for this problem exists in the works of [44, 48]. However, they provide upper bounds on the error of the estimated covariance matrix, with order-wise phase transitions. The key contribution of this section is to precisely characterize the error in the estimate and to present the necessary and sufficient number of measurements required for perfect recovery of the covariance matrix. In practice, having a precise theoretical understanding is extremely useful in designing the proper measurement settings. The organization of this chapter is as follows. In Section 4.1 we introduce the main notations and mathematically set up the problem. Section 4.2 includes the main result of the paper followed by discussions and numerical simulations. In this section, we also present a major outcome of our main theorem which is the the characterization of the phase transition in the noiseless setting. Finally, Section 4.3 concludes the paper by describing the key steps of the proof of our main theorem. 4.1 Problem Setup Notations We gather here the basic notations that are used throughout the paper. Bold lower letters are reserved for vectors and upper letters are used for matrices. For a vector v, 𝑣 𝑖 denotes its 𝑖 th entry and kvk is its ℓ2 -norm. (·) T is used to denote the transpose. 𝑋 ∼ 𝑝 𝑋 implies that the random variable 𝑋 has a density 𝑝 𝑋 , E[𝑋] denotes P its expected value. For a sequence of random variables {𝑋 (𝑖) }𝑖∈N , 𝑋 (𝑖) → 𝐶 indicates convergence in probability, i.e., lim P{|𝑋 (𝑖) − 𝐶 | > 𝜖 } = 0 . N (𝜇, Σ) 𝑖→∞ denotes the multivariate Gaussian distribution, with mean 𝜇 and covariance matrix Σ. I𝑑 represents the identity matrix in dimension 𝑑. S 𝑝 refers to the set of 𝑝 × 𝑝 symmetric matrices. For X ∈ S 𝑝 , kXk𝐹 , Tr(X) and kXk0 , respectively represent the Frobenius norm, the trace, and the number of non-zero entries. We also define Í kXk1− = 𝑖≠ 𝑗 |𝑋𝑖, 𝑗 | as the ℓ1 -norm of the non-diagonal entries of matrix X. The function 𝜓 : S 𝑝 → R is said to be Lipschitz if |𝜓(X) − 𝜓(Y)| ≤ 𝐿𝑝 kX − Yk𝐹 , for some constant 𝐿 > 0. For a function 𝑓 : S → R, we define its proximal operator as following, 1 kx − vk 2 + 𝑓 (x), ∀v ∈ S, 𝑡 ∈ R+ . x∈S 2𝑡 Prox 𝑓 (v, 𝑡) = arg min 88 Setup Following [44], we consider the problem of recovering an unknown symmetric matrix 𝚺0 ∈ S 𝑝 , from 𝑛 (noise-corrupted) quadratic measurements of the form, 𝑦𝑖 =
1 𝑇 1 a𝑖 𝚺0 a𝑖 + 𝑧𝑖 = Tr 𝚺0 a𝑖 a𝑖| + 𝑧𝑖 , 𝑖 = 1 . . . , 𝑛 , 𝑝 𝑝 (4.1) 𝑛 where {a𝑖 ∈ R 𝑝 }𝑖=1 , is the set of known measurement vectors, and z = [𝑧1 , 𝑧2 , . . . , 𝑧 𝑚 ] T ∈ R𝑛 is the noise vector. Throughout this section, for our analysis purposes, we assume that a𝑖 ’s are independently drawn from the Gaussian distribution with mean zero and covariance matrix I 𝑝 , the noise vector is independent of all the measurement vectors, and has independent zero mean entries with variance 𝜎 2 . The normalization 1𝑝 in (4.1) ensures that the measurement matrices a𝑖 a𝑖| are approximately unit-(Frobenius)norm. Our result is asymptotic which assumes a fixed oversampling ratio, 𝛿 := 𝑝( 2𝑛 𝑝+1) ∈ (0, ∞), while 𝑝 → ∞. Our interest is in studying the case where 𝛿 < 1, in which the problem is ill-posed. Therefore, one needs to efficiently exploit the low-dimensional structure of the underlying covariance matrix, 𝚺0 . Here, we focus on the setting where the covariance matrix is sparse, i.e., it has a few non-zero entries and define kS k 𝜅 = 𝑝𝑝2 0 as the sparsity factor. This happens in practical applications when a large number of entries have small pairwise correlations. We analyze the following convex optimization formulation for recovery of a sparse covariance matrix, 2 𝑛  Õ 𝜆 1 1 𝑇 𝚺ˆ = arg min 𝑦𝑖 − a𝑖 𝚺a𝑖 + 2 k𝚺k −1 . (4.2) 𝚺∈S 𝑝 2𝑛 𝑝 𝑝 𝑖=1 Recall from our notations in Section 4.1, the regularization term, k𝚺k −1 , enforces sparsity only on the non-diagonal entries of 𝚺, as the diagonals of a covariance matrix consist of positive entries. Due to the positive-definiteness of the covariance matrix, researchers often restrict the optimization program to the cone of positive semidefinite matrices [36, 44]. Here, we relax that constraint and let the feasible set contain all symmetric matrices. This relaxation not only simplifies the optimization program (by removing some constraints), but at the same time we will show that, when 𝜆 is tuned properly, it will provide a positive-definite estimate for 𝚺0 , which indeed is equal to the solution of the semidefinite program. Contribution The optimization (4.2) resembles the well-known LASSO problem in the literature [176]. When the measurement matrix in LASSO is Gaussian, there exist 89 powerful tools such as CGMT [165] and AMP [62] that could precisely analyze its performance. For example, the CGMT framework has been successfully applied to analyze the performance in a number of applications including analysis of regularized M-estimators [165], massive MIMO [1, 166, 174], and PhaseMax in phase retrieval[54, 142]. Roughly speaking, for a convex optimization problem over an instance of a Gaussian process, the CGMT associates a secondary optimization with similar performance yet often much simpler to analyze. Unfortunately, these frameworks do not apply to (4.2), because the measurement matrix is a𝑖 a𝑖| whose entries are neither Gaussian nor independent. To provide a precise analysis, we introduce a novel comparison lemma, that associates an equivalent optimization with our initial optimization problem (see Lemma 12). The equivalent optimization has i.i.d. Gaussian entries in its measurement, which makes it suitable to analyze via CGMT. Lemma 12 claims that under some conditions, the performance of these two optimizations is asymptotically the same. Therefore, analysis of the equivalent optimization via CGMT, characterizes the performance of our initial problem. To the best of our knowledge, this is the first work that introduces an equivalent optimization to analyze the performance of the problem of signal reconstruction from quadratic Gaussian measurements. 4.2 Main Results The goal is to analyze the performance of our estimate, 𝚺ˆ in (4.2). Let 𝜙(·) be a function which is Lipschitz and either convex or concave. Popular convex instances include 𝜙(X) = 𝑝12 kXk1 , and 𝜙(X) = kXk𝐹 /𝑝, and a concave one is 𝜙(X) = 𝜆 min (X). Our result is asymptotic in the sense that given a sequence of
( 𝑝) ( 𝑝) 𝑛 problem instances indexed by 𝑝, 𝚺0 , {a𝑖 }𝑖=1 , z ( 𝑝) 𝑝∈N , it characterizes the error ( 𝑝) performance as the limiting behavior of the sequence {𝜙( 𝚺ˆ ( 𝑝) − 𝚺 )} 𝑝∈N . To 0 streamline the notations, we often drop the superscript ( 𝑝) when understood from the context. Let Σ0 = [𝜎𝑖, 𝑗 ] ∼ 𝑝 𝚺0 , where 𝑝 𝚺0 denote the distribution of the underlying covariance matrix, which also incorporates the sparsity structure of 𝚺0 . For instance, one can assume that the off-diagonal entries of 𝚺0 are non-zero with probability 𝜅 ∈ [0, 1], so that the average sparsity of the resulting covariance is 𝜅 · 𝑝( 𝑝 − 1). It will be observed in Theorem 12, that 𝑝 𝚺0 plays a role in the performance of (4.2) via the summary functionals 𝐹𝜆 (·, ·), 𝐺 𝜆 (·, ·), and Φ𝜆 (·, ·), as follows. i.i.d. Assumption 4 Let 𝚺0 ∼ 𝑝 𝚺0 , and H = [ℎ𝑖, 𝑗 ] such that ℎ𝑖, 𝑗 = ℎ 𝑗,𝑖 ∼ N (0, 1), 90 for 1 ≤ 𝑖 ≤ 𝑗 ≤ 𝑛. We say that Assumption 4 holds, if there exist functions 𝐹𝜆 : R × R+ → R+ , 𝐺 𝜆 : R × R+ → R, and Φ𝜆 : R × R>0 → R such that for all 𝑠 ∈ R and 𝜏 > 0,
1 P kProx k·k −1 𝚺0 + 𝑠H, 𝜆𝜏 − 𝚺0 k𝐹2 → − 𝐹𝜆 (𝑠, 𝜏), 2 𝑝  
P 1 Tr H · Prox k·k −1 𝚺0 + 𝑠H, 𝜆𝜏 → − 𝐺 𝜆 (𝑠, 𝜏), 𝑝2  
P 𝜙 Prox k·k −1 𝚺0 + 𝑠H, 𝜆𝜏 − 𝚺0 → − Φ𝜆 (𝑠, 𝜏), and, (4.3) where the convergence is in probability, over the distributions of 𝚺0 and H. Later in this section, we argue that Assumption 4 holds under very generic settings, and the the functions, 𝐹𝜆 , 𝐺 𝜆 , and Φ𝜆 , capture the role of 𝜆 and 𝑝 𝚺0 in our analysis of error performance. We now present the main result of the paper which characterizes the limiting behavior of 𝜙( 𝚺ˆ − 𝚺0 ) in terms of 𝛿 and 𝜎. Theorem 12 Let Assumption 4 holds, and 𝚺ˆ ∈ S 𝑝 denote the solution to the convex optimization (4.2), given 𝑛 quadratic observations of the form (4.1). Then, as 𝑝 → ∞, s p   2 + 2𝛼★2 𝜎 𝜎 2 + 2𝛼★2 P ˆ , , (4.4) 𝜙( 𝚺 − 𝚺0 ) → − Φ𝜆 2𝛿 2𝛽★ where (𝛼★, 𝛽★) is the unique solution to the following system of non-linear equations with two unknowns, 𝛼 and 𝛽,  q √   𝜎 2 +2𝛼2 𝜎 2 +2𝛼2 2  ,   𝛼 = 𝐹𝜆  2𝛿 , 2𝛽   q (4.5) q √ √  2 𝜎 2 +2𝛼2 𝜎 2 +2𝛼2 2 2  𝜎 + 2𝛼 − · 𝐺 , , 𝛽 =  𝜆  𝛿 2𝛿 2𝛽  and the functions 𝐹𝜆 (·, ·) and 𝐺 𝜆 (·, ·) are defined in (4.3). A few remarks are in place regarding Theorem 12: [Frobenius Norm of the Error] Here, we show that the parameter 𝛼★ in the Theorem 12, represents the limiting value of the Frobenius of the error of (4.2), 91 1 ˆ 1 2 𝑝 k 𝚺 − S 𝑝 k𝐹 . Since, by choosing 𝜙(X) = 𝑝 2 kXk𝐹 , from the definitions of Φ𝜆 (𝑠, 𝜏) and 𝐹𝜆 (𝑠, 𝜏) in (4.3), we get Φ𝜆 (𝑠, 𝜏) = 𝐹𝜆 (𝑠, 𝜏). (4.6) Therefore, applying Theorem 12, the error performance can be computed as follows, s p   2 + 2𝛼★2 𝑝 𝜎 𝜎 2 + 2𝛼★2 1 ˆ 2 (4.7) k 𝚺 − 𝚺0 k 𝐹 → − Φ𝜆 , 2𝛿 2𝛽★ 𝑝2 s p   𝜎 2 + 2𝛼★2 𝜎 2 + 2𝛼★2 2 = 𝐹𝜆 = 𝛼★ , , 2𝛿 2𝛽★ 0.5 Simulations Theory 0.45 0.4 0.35 0.3 0.25 0.2 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Figure 4.1: Performance of the optimization (4.2) with respect to 𝜙(W) = kWk𝐹 /𝑝, as a function of 𝜆. Circles represent numerical simulations, and solid lines are theoretical predictions from Theorem 12. For simulations, we used 𝑝 = 120, 𝛿 = .8, E[𝑧𝑖2 ] = 1, and three choices of sparsity factor; 𝜅 = .05 in red, 𝜅 = .1 in blue, and 𝜅 = .2 in black. The results are averaged over 80 random realizations of data. For 𝜆 > 𝜆+ , the output of (4.2) will be positive definite. where the last two equalities come from (4.6) and (4.5). Figure 4.1 demonstrates that the empirical result well matches the theoretical result derived from Theorem 12 . For our numerical simulations, the underlying covariance matrix, 𝚺0 , is chosen to be a (uniformly) subsampled symmetric Gaussian matrix added by a multiple of the kS k identity matrix (such that 𝜆 min (Σ0 ) = 0.1 and 𝜅 = 𝑝𝑝2 0 ). 92 [Tuning 𝜆] In order to guarantee that the solution of (4.2) is positive definite, we need to tune the regularization parameter 𝜆. Here, we show that if 𝜆 is chosen such that the matrix M̂ := Prox k·k1− (𝚺0 + 𝑠★ · H, 𝜆𝜏★) is positive definite, then Ŝ will also √ q 2 ★2 𝜎 2 +2𝛼★2 ★ ★ , and 𝜏 = 𝜎2𝛽+2𝛼 , with (𝛼★, 𝛽★) being be positive definite. Here 𝑠 = ★ 2𝛿 the solution of (4.5). To show this, we define the performance measure of optimization (4.2) to be the concave Lipschitz function 𝜙(X) = √1𝑝 𝜆 min (X + 𝚺0 ). Applying Theorem 12 yields, 
 P 1 1 − 0, √ 𝜆 min ( Ŝ) − √ 𝜆 min Prox k·k1− 𝚺0 + 𝑠★ · H, 𝜆𝜏★ → 𝑝 𝑝 As a result, when 𝜆 is tuned properly such that M̂ := Prox k·k1− (𝚺0 + 𝑠★ · H, 𝜆𝜏★)  0, then Ŝ would be a positive definite matrix. Computing the appropriate range of 𝜆 for an arbitrary distribution of 𝑝 𝚺0 is beyond the scope of this thesis. For the case where 𝚺0 is chosen as the addition of a subsampled Gaussian plus a multiple of identity, using results from random matrix theory, we can show that 𝜆 > 𝐶 log( 𝜆min𝑝(𝚺0 ) ), where 𝐶 is a constant independent of 𝑝 and 𝚺0 , is sufficient for Ŝ being PD. Figure 4.1 also specifies the range of 𝜆 under which the estimate Ŝ is positive definite, for three choices of the sparsity factor, 𝜅. Phase Transition Using the result of Theorem 12, we are able to characterize the phase transition of our convex optimization program, which provides us with the necessary and sufficient number of measurements for the perfect recovery of the underlying covariance matrix. Here, we assume that the measurements are noiseless (z = 0). The goal rec is to identify 𝛿rec = 𝑝(2𝑛𝑝+1) that indicates the exact phase transition, such that , as 𝑝 → ∞, 𝛿 > 𝛿rec is the necessary and sufficient condition for perfect recovery of 𝚺0 . We state the following corollary (without proof) that characterizes the exact phase transition of the the optimization program (4.2): Corollary 4 Consider the optimization program (4.2), given 𝑛 noiseless measurements (z = 0) of the form (4.1). For a fixed oversampling ratio 𝛿 = 𝑝( 2𝑛 𝑝+1) and the sparsity factor 𝜅 = kS𝑝02k0 , the optimization program (4.2) perfectly recovers the true covariance matrix (in the sense that lim 𝑝→∞ P{|| 𝚺ˆ − 𝚺0 || > 𝜖 } = 0, for any fixed 𝜖 > 0), if and only if 𝛿 > 𝛿rec , where 𝛿rec is the unique solution of the following 93 0.1 0.2 0.3 0.4 0.5 0.6 Theory 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 Figure 4.2: Phse transition regimes for the optimization (4.2), in terms of the oversampling ratio 𝛿 = 𝑝( 2𝑛 𝑝+1) , and the sparsity factor 𝜅. Solid line comes from (4.8). For the empirical results, we used 𝑝 = 40. The results are averaged over 20 independent realization of measurement vectors. equation,     2𝛿 − 𝜅 −1 2𝛿 − 𝜅 𝛿𝑄 = (1 − 𝜅)𝜑 𝑄 . (4.8) 2 − 2𝜅 2 − 2𝜅 ∫∞ −𝑥 2 /2 Here, 𝜑(𝑥) = 𝑒√ , and 𝑄(𝑥) = 𝑥 𝜑(𝑧)𝑑𝑧, represent the probability density and 2𝜋 the tail distribution of the standard normal distribution, respectively. −1  Corollary 4 specifies that the optimization (4.2) achieves perfect recovery w.p.a. 1, if and only if 𝑛 > 𝛿rec · 𝑝( 𝑝 + 1)/2. Note that 𝛿rec is only a function of the sparsity factor 𝜅, and is independent of other statistics of S0 . Figure 4.2 illustrates validity of the Corollary 4. For numerical simulations, we used the same model to general 𝚺0 , as for the Figure 4.1. As observed in the figure, the error becomes zero only in the regime where 𝑛 > 𝛿rec · 𝑝( 𝑝 + 1)/2. 4.3 Proof Outline The complete proof of this theorem can be found in the (15) and (16). Here, we outline the fundamental ideas behind the proof of Theorem 12. Consider the optimization problem (4.2). To get a direct handle on the error term, it is 94 convenient to rewrite the optimization in terms of a new variable, W := Σ − 𝚺0 . Thus, (4.2) is equivalent to the following, 2 𝑛  1 1 Õ 𝜆 min (4.9) 𝑧𝑖 − Tr(WA𝑖 ) + 2 kW + 𝚺0 k1− , W∈S 𝑝 2𝑛 𝑝 𝑝 𝑖=1 where A𝑖 = a𝑖 a𝑇𝑖 , and the goal is to analyze 𝜙( Ŵ). To have a simpler notation, we rewrite the optimization in a vectorized format. Let,  Vec| (A )   1    .  , w = Vec(W), x0 = Vec(𝚺0 ) , .. A =  (4.10)    | Vec (A𝑚 )    where Vec(X) is the vectorization of X and Mat(x) is its inverse transform. For a vector x, we also define kxk1− = kMat(x)k1− . Using these notations, (4.9) can be rewritten as 1 1 𝜆 Ψ(A) = min 𝜓(A, w) = kz − Awk 2 + 2 kw + x0 k1− , Mat(w)∈S𝑛 2𝑛 𝑝 𝑝
Ŵ(A) = Mat arg min 𝜙(A, w) . (4.11) Mat(w)∈S 𝑝 We proceed onward by analyzing the optimization (4.11) which is similar to the popular LASSO problem. As stated before, the main bottleneck in analyzing this optimization is the fact that the entries of A are not i.i.d. Gaussian. Instead, we prove the desired indirectly, via the following two steps. First, we show that the properties of Ψ(·) are preserved asymptotically, as we replace A, with a carefullydesigned Gaussian matrix B with independent entries. In other words, Ŵ(A) and Ŵ(B) have the same asymptotic performance. Consequently, we utilize the CGMT framework to analyze Ŵ(B). Leaving some technical details for section (15) and (16), the mechanics are easy to explain and provide valuable intuition regarding our approach. Step 1. We start by introducing some new notations. Let G𝑖 , for 𝑖 = 1, 2, . . . , 𝑛, be a symmetric matrix whose diagonal and (upper) non-diagonal entries are drawn independently from distributions N (1, 2) and N (0, 1), respectively. Also, define, Vec| (G )   1    ..   ∈ R𝑛×𝑝2 . B=  (4.12) .    Vec| (G𝑛 )    The following comparison lemma forms the heart of our proof, allowing us to have the same performance replacing A with B in (4.11). 95 Lemma 12 Consider the functions Ψ(.) and Ŵ(.) defined in (4.11) and the random matrices A and B from (4.10) and (4.12). Let the assumptions in Theorem 12 hold and both parameters 𝜓(W(A)) and 𝜓(W(B)) converge in probability as 𝑝 → ∞.

P Then, 𝜓 W(A) − 𝜓 W(B) → − 0. Lemma 12 essentially states that replacing matrices a𝑖 a𝑖T , for 𝑖 = 1, 2, . . . , 𝑛, in the optimization (4.9) with Gaussian matrices G𝑖 does not alter the performance measure. It is worth noting that the stated result is only valid when all the stated conditions hold. We defer the technical details in the proof to (15) and (16), but once it is established we only need to analyze the performance of W(B). Step 2. We utilize the CGMT framework to analyze the performance of W(B). Lemma 13 Let (𝛼★, 𝛽★) be the unique solution to the system of equations (4.5), then as 𝑝 → ∞, we have s p   𝜎 2 + 2𝛼★2 𝜎 2 + 2𝛼★2 P , 𝜙(W(B)) → − Φ . (4.13) 2𝛿 2𝛽★ To show this lemma we need to apply the CGMT. We refer the interested reader to section V.D. of [165]. So in the next section, we will focus on the proof of the Lemma 12. Proof of Lemma 12 Proving that 𝜓( Ŵ(A)) and 𝜓( Ŵ(B)) converge to the same value is one step away from proving that for every 𝐶 > 0 𝑝 min Mat(w)∈S 𝑝 1 2 2 kwk ≤𝐶 2𝑝 𝜓(A, w) − min 𝜓(B, w) → − 0. (4.14) Mat(w)∈S 𝑝 1 2 2 kwk ≤𝐶 2𝑝 Once we have this lemma, if 2𝑝1 2 k Ŵ(A)k𝐹2 and 2𝑝1 2 k𝑊ˆ𝑏(B)k𝐹2 converge to different values 𝑠1 and 𝑠2 , choosing 𝐶 = (𝑠1 + 𝑠2 )/2 in (4.14) results in a contradiction. Thus 2𝑝1 2 kW(A)k𝐹2 and 2𝑝1 2 kW(B)k𝐹2 converge to the same value. Then, using Lipschitsness and convexity of 𝜙(.) and the same set of arguments as in the Section IV-B of [4] shows that 𝜓(W(A)) and 𝜓(W(B)) converge to the same value. It remains to prove (4.14). We define    |𝑥|  if |𝑥| > 1 𝜂(𝑥) = . (4.15)   38 + 68 𝑥 2 − 18 𝑥 4 o.w.  96 𝜂(.) can be also applied to matrices, where it acts element-wise. It is twice differentiable and convex. Besides, 𝑡 · 𝜂(X/𝑡) is jointly convex in X and 𝑡 > 0, since it is the perspective function of 𝜂(·). Next, we introduce a new optimization, 1 𝜆𝑡 1 w + x0
kz − Awk 2 + 2 · 𝜂 W∈S 𝑝 2𝑛 𝑝 𝑡 𝑝 𝜖 + 2 kwk 2 . 2𝑝 Ψ𝑡,𝜖 (A) = min (4.16) This function is convex in 𝑡 and concave in 𝜖. Furthermore, inf sup −𝐶𝜖 + Ψ𝑡,𝜖 (A) = 𝑡>0 𝜖 >0 min W∈S 𝑝 1 2 2 kwk ≤𝐶 𝜓(A, w) . (4.17) 2𝑝 Thus, if we show that 𝑝 Ψ𝑡,𝜖 (A) − Ψ𝑡,𝜖 (B) → − 0, (4.18) we can apply Lemma 14 on (4.17), twice, to prove (4.14). The key ingredient of proving (4.18) is the Lindeberg replacement principle and the strong convexity of the regularizer in (4.16). We omit the details due to the limited space, but such similar results has been shown in [130, 131]. We now present lemma bellow which is also known as the convexity lemma in the literature [136]. Lemma 14 Consider a series of convex functions 𝑓 𝑝 : R>0 → R that converges point-wise to the function 𝑓 : R>0 → R. Besides, there exists 𝑀 > 0 such that for all 𝑥 > 𝑀, we have 𝑓 (𝑥) > inf 𝑠>0 𝑓 (𝑠). Then 𝑓 (.) is also convex and 𝑝 inf 𝑠>0 𝑓 𝑝 (𝑠) → − inf 𝑠>0 𝑓 (𝑠). 97 Chapter 5 SCALABLE COVARIANCE ESTIMATION IN GRAPHICAL MODELS WITH PROVABLE GUARANTEES 5.1 Introduction variance matrices play a fundamental role in behavioral analysis of multivariate random variables in a variety of fields including finance and economics, engineering, and environmental and physical sciences. In modern such inference applications, one is faced with the problem of estimating covariance matrices associated with data, of a very large dimension 𝑝, from a few (and potentially less than 𝑝), number of observations 𝑛. Also, it is often the case that the true unknown matrix possesses some low-dimensional structure. This might be a property directly associated with the covariance matrix (examples including sparse or approximately low-rank covariance matrices), but it can also arise in an indirect form. For example, a very well-encountered structural model, which is relevant to graphical models for Gaussian random variables, is one in which the inverse of the covariance matrix (rather than itself), also termed the precision matrix, is sparse. Here, the sparsity pattern of the concentration matrix implies the structure of the associated graph in the Gaussian Markov Random Field (GMRF), and is thus critical to estimate. Overall, modern inference procedures for covariance estimation need to have the following favorable properties: • on the statistical/theoretical side, it is important that they provably reveal the underlying structures of the true desired matrices (such as the support pattern of the precision matrix) while having access to only few number of observations. • on the computational side, it is critical that their computational complexity scales with the increasing problem dimensions, thus allowing to solve practical instances in which 𝑛 and 𝑝 are on the range of (at least) a few hundrends. In a similar flavor, algorithms that allow performing operations in a parallel fashion on different machines thus speeding up the total performance are also desirable. 98 in this section, we consider the classical problem of estimating the sparse precision matrix of a multivariate zero-mean random variable, given 𝑛 iid observations {x𝑖 }𝑖=1,...,𝑛 ∈ R 𝑝 . While this problem has attracted lots of attention over the past decade or so, and several algorithms have been proposed and analyzed in the relevant literature, it appears that none of them enjoys both the computational and the theoretical features discussed above. We propose a novel algorithm that combines all these; specifically, it (i) is scalable, (ii) is parallelizable, and (iii) provably estimates the desired structure of the underlying graphical model from only a few number of observations. Background and Motivation We consider the problem of estimating the covariance matrix Σ of a multivariate zero-mean random variable, given 𝑛 i.i.d. observations {x𝑖 }𝑖=1,...,𝑛 ∈ R 𝑝 . We focus on the regime of high-dimensions in which both 𝑛 and 𝑝 are large. Of particular, interest is the case of limited number of observations 𝑛  𝑝. In this Í𝑛 high-dimensional setting the classical sample covariance estimator R = 𝑖=1 x𝑖 x𝑖T has been shown to perform poorly[92][93]. Besides, it cannot accommodate for any prior knowledge on the structure of Σ. Many different kinds of structures have been considered in the literature. For example, for the case of banded covariance (or concentration) matrices, where the entries decrease as a function of their distance from the diagonal, popular estimators include banded estimators [195][24][32], and shrinkage estimators [102][76]. in this section, we focus on a widely-studied model where the concentration matrix Σ−1 is sparse. This model shows up in graphical models for Gaussian random variables. It is classically known that in graphical models, the sparsity pattern of the concentration matrix implies the structure of the associated graph in the Gaussian Markov random field (GMRF). ℓ1 -penalized log det-optimization. A popular approach to estimate a sparse precision matrix is via solving the following convex optimization program: ˆ = arg min Tr(𝛀R) − log det(𝛀) + 𝜆k𝛀− k1 , 𝛀 𝛀0 (5.1) Í𝑛 x𝑖 x𝑖T is the sample correlation matrix, and k𝛀− k1 = where recall that R := 𝑛1 𝑖=1 Í 𝑖≠ 𝑗 |𝛀𝑖 𝑗 | is the ℓ1 -norm of the off-diagonal entries of its argument. When the observations are generated from a Gaussian distribution, then the objective function 99 in (5.1) is nothing but the ℓ1 -penalized (negative) log-likelihood function. The ℓ1 -regularization is known to promote sparse solutions under several settings [68]. On the analysis side, taking advantage of the convex nature of (5.1), the estimator ˆ has been well-analyzed in the literature and it has been proved to enjoy favorable 𝛀 properties. When the x𝑖 ’s are Gaussian, Rothman q et al. [138] showed, under ˆ − Σ−1 k 2 ≈ O ( (𝑠+𝑝) log 𝑝 ), where 𝑠 denotes the standard assumptions on Σ, that k 𝛀 𝐹 𝑛 number of non-zero non-diagonal entries in the concentration matrix; this implies consistency in a Frobenius-norm sense, as long as 𝑛 > O ((𝑠 + 𝑝) log 𝑝). Later in 2008, Lam and Fan [100] further showed that (5.1) additionally recovers the sparsity √ pattern of the concentration matrix as long as 𝑠 = O ( 𝑝) and 𝑛 = Ω((𝑠 + 𝑝) log 𝑝). Since then, Ravikumar et. al. [134] have extended these results beyond Gaussians to a general class of random variables with appropriate tail bounds (which includes sub-Gaussians and random variables with bounded moments), under additional coherence assumptions on Σ. Importantly, they proved consistency with respect to ˆ − Σ−1 kmax = max𝑖, 𝑗 |( 𝛀 ˆ − Σ−1 )𝑖, 𝑗 |. the stronger notion of the max-norm k 𝛀 ˆ of the convex These rich set of performance guarantees apply to the solution 𝛀 optimization (5.1). However, the computational task of obtaining (𝜖-close approxˆ via standard generic convex solvers, such as interior point methods imations of) 𝛀 and other second-order-methods [28], does not scale well with the problem dimensions, which makes solving (5.1) prohibitive in any practical scenario where 𝑝 is on the order of even a few hundreds! GLASSO. The computational challenge of scaling the minimization in (5.1), has led to significant research activity on deriving fast alternative algorithms [49][199][75]. A very popular method towards this direction is the Graphical LASSO (GLASSO) [75]. GLASSO is an iterative algorithm, which starts with an initialization (say) ¯ and at each step, it solves a specific LASSO problem (aka ℓ1 -regularized least𝚺, squares) with the goal of updating a specific row/column of this matrix. After 𝑝 iterations the entire matrix is updated once and the process continues until convergence (in the sense that the updated matrix remains in some 𝜖-neighborhood of the previous estimate). While the details of the updates are not important for our discussion, it should be emphasized that solving LASSO problems can be done very efficiently using off-the-shelf solvers (e.g. [74]). Hence, GLASSO is scalable and is used in practice, instead of (5.1). Unfortunately, despite its obvious algorithmic advantages when compared to (5.1), no matching theoretical guarantees for GLASSO exist in the literature. This is 100 despite the fact that GLASSO can be interpreted as a block-wise coordinate descent approach for solving (5.1). For example, the answers to the following questions are unknown: “What is the rate of convergence of GLASSO?", or “If you were to run the algorithm for (say) 𝑝 steps (during which the entire initial matrix 𝚺¯ is updated once), how good an estimate is obtained?". It should also be mentioned that more recent proposed extensions of GLASSO such as the P-GLASSO and DP-GLASSO [111], although apparently faster, still suffer by the lack of any analytical guarantees on their rate of convergence and on the performance after a fixed number of iterations. Adding to this, note that even though GLASSO is scalable, it is not parallelizable since solving the LASSO at each iteration depends on the previous one. Contribution In this chapter, we propose a new algorithm that combines the virtues of both GLASSO and of the convex estimator in (5.1), i.e., it is fast and scalable with problem dimensions, and it provably attains the same order-wise statistical guarantees as (5.1). Moreover, it involves solving only 𝑝 independent LASSO problems which can be performed on different machines; thus it is also parallelizable. The algorithm starts with a shrinkage estimator 𝚺¯ of Σ1. Then, it solves 𝑝 independent LASSO problems. Each LASSO problem uses 𝚺¯ to obtain an estimate of the 𝑖 th row/column of the concentration matrix. These 𝑝 estimates are combined in the ˆ of the concentration matrix 𝛀 = Σ−1 . We give last step to obtain a final estimate 𝛀 the details in Section 5.2. The scalability of the algorithm is obvious: its computational complexity is the same as that of solving 𝑝 LASSO problems. On top of that (something that is not true for GLASSO), the LASSO problems are independent of each other, thus they can be run over parallel machines to achieve further improved computational performance. Finally, when the random variables are Gaussians, we prove recovery bounds that coincide with state of the art corresponding results on the performance of the convex program in (5.1). In particular, we show that with the correct tuning of the input parameters, the algorithm exactly recovers the zero-pattern of the concentration matrix, which here corresponds to the structure of the underlying Gaussian graphical model. Furthermore, we provide consistency guarantees on the max-norm as in [134]. 1 Other appropriate initializations are also possible, we discuss these in later sections. 101 Paper Organization The rest of the paper is organized as follows. In Section 5.2 we describe how the proposed algorithm operates, and we present its computational features and accompanying theoretical results on its statistical performance. A more detailed discussion, a thorough comparison of the results to the relevant literature and numerical simulations are included in Section 5.3. The proofs, and also an illustration of the intuition behind the algorithm are deferred to the Appendix. 5.2 The Algorithm: Computational & Statistical Guarantees For convenience, denote the concentration matrix as 𝛀 := Σ−1 and [ 𝑝] = {1, . . . , 𝑝}. For a matrix M and a vector v we denote M 𝑘,ℓ and v 𝑘 the (𝑘, ℓ) 𝑡ℎ entry of M and the 𝑘 𝑡ℎ entry of v, respectively. FGL Algorithm We call our algorithm the Fast Graphical-LASSO (FGL) algorithm. In this section, we describe how the algorithm operates. Í𝑛 x𝑖 x𝑖T be the sample covariance matrix and DR be a Initialization. Let R = 𝑛1 𝑖=1 diagonal matrix with the same diagonal entries as R. For a constant 0 ≤ 𝜇 ≤ 1, the shrinkage estimator R 𝜇 of Σ is defined as R 𝜇 := 𝜇DR + (1 − 𝜇)R. (5.2) Of course, R0 = R. Further note that R 𝜇 is always positive definite for all 0 < 𝜇 ≤ 1, even in the regime of few observations 𝑛 < 𝑝. We will see that this property is critical for the satisfactory performance of the algorithm in the presence of few observations. FGL takes as input a parameter 𝜇 and utilizes 𝚺¯ = R 𝜇 as an estimate of Σ. Our main theorem specifies appropriate values for tuning 𝜇 that guarantee good statistical performance. (For example, we will see that naively setting 𝜇 = 0, eqv. initializing the FGL with just R, guarantees good performance only in the case 𝑛 > 𝑝). 𝑝 independent LASSOs. The main body of the algorithm is solving 𝑝 independent LASSO optimization problems. Each one of them uses the initialization R 𝜇 and outputs an estimate of a certain row/column of the precision matrix (a total 𝑝 of them). Each one of the 𝑝 main operations of the algorithm is independent of one other, but can all be described in a common language. For this, fix any 𝑖 = 1, . . . , 𝑝. 102 Construct matrix 𝚺¯ (𝑖) by permuting the 𝑖 th and 𝑝 th rows and columns of 𝚺¯ such that the 𝑖 th (resp. 𝑝 th ) row and column of 𝚺¯ (𝑖) is the 𝑝 th (resp. 𝑖 th ) row and column of 𝚺¯ 2 Consider the following partition of 𝚺¯ (𝑖) : " # ¯ 11 𝝈 ¯ 𝚺 12 𝚺¯ (𝑖) = , T ¯ 12 𝝈 𝜎 ¯ 22 (5.4) where we have suppressed the dependence of the elements of the partition on the index 𝑖 in order to keep the notation simple. Using this notation, and for an input parameter 𝜆 > 0, solve the following LASSO problem, 1 T ¯ 12 𝛽ˆ(i) := arg min 𝛽T 𝚺¯ 11 𝛽 + 𝝈 𝛽 + 𝜆k 𝛽k1 . 𝛽 2 Observe that 𝛽ˆ ∈ R 𝑝−1 . Use this to construct 𝝎 (𝑖) ∈ R 𝑝 as follows    1, 𝑘 = 𝑖,     (𝑖) ˆ −1 × 𝛽ˆ (𝑖) , 𝑘 = 𝑝, 𝝎 𝑘 := ( 𝜎 ¯ 22 + 𝛽ˆT 𝚺¯ 11 𝛽) 𝑘 = 1, . . . , 𝑝. 𝑖      𝛽ˆ𝑘(𝑖) , 𝑘 ∉ {𝑖, 𝑝},  (5.5) (5.6) ˆ of the precision matrix 𝛀, Output. The FGL algorithm outputs an estimate 𝛀 based on the previously computed vectors 𝝎 (𝑖) , 𝑖 = 1, . . . , 𝑝 as follows: ˆ 𝑘,ℓ := (𝝎 (𝑘) + 𝝎 (ℓ) )/2, 𝛀 ℓ 𝑘 𝑘, ℓ = 1, . . . , 𝑝. (5.7) All these are outlined in Algorithm 1 below. Algorithm 1 FGL Algorithm Input: Observations {x 𝑗 ∈ R 𝑝 } 𝑗=1,...,𝑛 . Parameters 0 ≤ 𝜇 ≤ 1, 𝜆 > 0. ˆ of the precision matrix 𝛀 = Σ−1 . Output: Estimate 𝛀 Set 𝚺¯ = R 𝜇 = 𝜇DR + (1 − 𝜇)R as in (5.2). for all 𝑖 = 1, . . . , 𝑝 do ¯ 12 are as in Solve the LASSO problem in (5.5) to find 𝛽ˆ (𝑖) , where 𝚺¯ 11 and 𝝈 (5.4). Use 𝛽ˆ (𝑖) to form 𝝎 (𝑖) as in (5.6). end for ˆ as in (5.7). Using the 𝝎 (𝑖) ’s, construct 𝛀 2 Formally, define permutation matrices P (𝑖) , 𝑖 = 1, . . . , 𝑝 such that   1,    (𝑖) P 𝑘,ℓ = 1,    0,  ¯ (𝑖) . Then, 𝚺¯ (𝑖) = P (𝑖) 𝚺P 𝑘 = ℓ ∉ {𝑖, 𝑝}, (𝑘, ℓ) ∈ {(𝑖, 𝑝), ( 𝑝, 𝑖)}, else. (5.3) 103 Computational Performance It is clear that the main computational burden of the FGL algorithm is that of solving 𝑝 LASSO problems. This makes the algorithm scalable. Moreover, observe that the LASSO problems are all independent of each other. The outputs 𝛽ˆ (𝑖) of each ˆ are combined only at the last step of the algorithm to form the final estimate 𝛀. Thus, each LASSO problem can be solved separately on a different machine. Each machine needs only have access to the matrix 𝚺¯ computed at the first stage of the algorithm. (See however Section 5.3 where it is shown that storing 𝚺¯ is not necessary and all operations can be performed via sole access to the observation vectors, when 𝑛 > 𝑝). Statistical Performance Aside from the obvious computational virtues discussed above, it is further shown in this section that the FGL algorithm enjoys provable performance guarantees. As is typical, the guarantees require some conditions on the true unknown covariance matrix Σ. We start with these and state the main result in Theorem 13. The theorem holds for the case of Gaussian random variables, but we expect the result to generalize to wider classes, such as sub-gaussians and variables with bounded moments. We leave these to a future long version of the paper. Some notation & Assumptions Let M ∈ R 𝑝×𝑝 . For sets S, R ⊆ [ 𝑝], MS,R denotes a sub-matrix of M with entries (M𝑖, 𝑗 )𝑖∈S, 𝑗 ∈R . Also, we write kMkmax = max𝑖, 𝑗 |𝑀𝑖, 𝑗 | Í𝑝 for the maximum element-wise norm of M and kMk∞ = max𝑖 𝑗=1 |M𝑖, 𝑗 | for the induced infinity norm. Our analysis keeps explicit track of the positive quantities 𝜅, 𝛾, 𝜆¯ and 𝜆 defined below, so that they can scale in a non-trivial manner with the problem dimension p. • First we define the parameter 𝜅 > 0 that corresponds to the diagonal dominance of the covariance matrix. We have, Õ ∀𝑖 ∈ [ 𝑝], |𝛀𝑖, 𝑗 | ≤ 𝜅𝛀𝑖,𝑖 . (5.8) 𝑗≠𝑖 • We also define 𝛾 > 0 such that ∀𝑖 ∈ [ 𝑝], k(ΣS𝑖 ,S𝑖 ) −1 k∞ ≤ 𝛾, corresponding to the ℓ∞ -norm of the inverse sub-covariance matrices. 104 • We use 0 < 𝜆 ≤ 𝜆¯ < ∞ to denote the minimum and maximum eigenvalues of Σ, 𝜆 𝑚𝑎𝑥 {Σ} ≤ 𝜆¯ 𝜆 𝑚𝑖𝑛 {Σ} ≥ 𝜆. We require the following assumption on the covariance matrix. See Section 5.3 for a discussion of its interpretation. Assumption 5 There exists a constant 0 < 𝛼 < 1 such that, ∀𝑖 ∈ [ 𝑝], kΣS̄𝑖 ,S𝑖 (ΣS𝑖 ,S𝑖 ) −1 k∞ ≤ 1 − 𝛼. Main Result. We are now ready to present our main theorem characterizing the statistical performance of the FGL algorithm in terms of both support recovery and max-norm consistency. Theorem 13 (Zero-pattern & max-norm Guarantees) Let the observations {x𝑖 }𝑖=1,...,𝑛 follow a zero mean Gaussian distribution of covariance Σ ∈ R 𝑝×𝑝 that further satˆ be the output of the FGL Algorithm 1 with isfies Assumption 5. Let 𝛀 = Σ−1 , and 𝛀 q 𝜏 log 𝑝 8−2𝛼 ★ ¯ (𝜅 + 1) 𝜆 . Then, for input parameters 𝜇 = 𝜇 defined in (5.11) and 𝜆 = 𝛼 𝑛 any 𝜏 > 2, the following statements hold with probability at least 1 − 5/𝑝 𝜏−2 . ˆ ⊆ Supp(𝛀). (i) Supp( 𝛀) (ii) Further suppose that 2 ¯ (4 − 𝛼) 𝜆𝛾(𝐶 1 + 𝜅) 𝑛>𝑀 𝑑 2 𝜏 log 𝑝, 𝛼𝜆  (5.9) for some constant 𝑀 and 𝐶1 = 1 + (8−2𝛼)𝛼 (𝜅+1) . Then, r ˆ − 𝛀kmax ≤ 𝐶2 k𝛀 𝜏 log 𝑝 , 𝑛 (5.10) ¯ ¯ for 𝐶2 = 2𝜆𝜆 𝛾(𝐶1 + 𝜅) + 𝜆𝜅(1 + 𝜅 + 𝜅 2 + 𝐶1 (2𝜅 + 1)). In the statement of the theorem, the bounds and the value of the regularizer parameter 𝜆 are both specified in terms of a parameter 𝜏 > 2. Larger values of 𝜏 lead to looser 105 bounds on the error performance, but yield faster rates of probability convergence. Further, we suggest the following tuning for 𝜇 3 ( ) r r √ ¯ 40 2 𝜆 𝜏 log 𝑝 𝑝𝜏 log 𝑝 𝜇★ = min 0.5, , 16 . (5.11) max𝑖≠ 𝑗 {|Σ𝑖, 𝑗 |} 𝑛 𝑛 Statement (i) of the theorem guarantees recovery of the zero pattern of the precision matrix 𝛀. Equivalently, for Gaussian graphical models, it guarantees recovery of the structure of the underlying graphical model. A further refinement of this result to perfect signed-support recovery is possible with an additional assumption on the minimum on-support entry of 𝛀, as stated in Corollary 5 in Section 5.3. The second statement of Theorem 13 proves consistency of the FGL estimate in the max-norm sense as long as 𝑛 = O (log 𝑝). This is the same order-wise result as the one obtained by Ravikumar et al. [134] regarding the log det-minimization in (5.1); see Section 5.3. Also, the bound of Theorem 13(ii) can be further used to conclude bounds on the Frobenius and on the Spectral norm of the error. We discuss these in Section 5.3. 5.3 Discussion and Numerical Experiments We start in Section 5.3 with a couple of extensions of Theorem 13 to signed-support recovery and bounds on the Frobenius and on the spectral norm. In Section 5.3, (i) we discuss the main differences of the proposed algorithm to the GLASSO, (ii) we compare Theorem 13 to corresponding results on the performance of the convex log det minimization (5.1) derived in[75, 102, 134] and show that the FGL algorithm enjoys the same order-wise guarantees as the latter and (iii) we provide an interpretation of Assumption 5. Finally, in Section 5.3 we present the results of numerical experiments. Extensions Signed-support Recovery. A further refinement of Theorem 13(i) to perfect signed-support recovery is possible with an additional assumption on the minimum on-support entry of 𝛀 as shown in the corollary 5 below. 3 It turns out from the analysis that the initial matrix 𝚺 ¯ of Algorithm 1 needs be positive definite. As discussed, the shrinkage estimators R 𝜇 enjoys that property even when 𝑛  𝑝. The larger the value of 𝜇 is the better the bound on the minimum eigenvalue of R 𝜇 , but at the same time as the parameter 𝜇 increases, we loose control of error of the estimator in terms of the operator norm and element-wise maximum norm. It turns out that the tuning suggested in (5.11) serves both of the aforementioned goals well enough. 106 Corollary 5 (Perfect signed-support recovery) In addition to the assumptions of q Theorem 13, suppose that 𝑛 and 𝑝 further satisfy 𝐶2 𝜏 log 𝑝 𝑛 ≤ min𝛀𝑖, 𝑗 ≠0 |𝛀𝑖, 𝑗 |. 𝑖≠ 𝑗 Then, with probability at least 1 − 𝑝 𝜏−2 , we achieve perfect signed-support recovery, ˆ 𝑖, 𝑗 ) = sign(𝛀𝑖, 𝑗 ), ∀(𝑖, 𝑗). i.e., sign( 𝛀 5 Rates in spectral and Frobenius norms. The bound on the max-norm of Theorem 13(ii) can be used to derive bounds on the Frobenius and spectral norms of the error, as well. Corollary 6 (Spectral and Frobenius norm bounds) Under the same setup and 5 , it holds assumptions as in Theorem 13(ii), with probability at least 1 − 𝑝 𝜏−2 r ˆ − 𝛀k𝐹 = O k𝛀 r ˆ − 𝛀k2 = O k𝛀 (𝑠 + 𝑝)𝜏 log 𝑝 𝑛 ! and ! min{(𝑠 + 𝑝), 𝑑 2 }𝜏 log 𝑝 . 𝑛 (5.12) Using bounds on spectral norm, we can also provide guarantees for positive definiteness of the matrix. Note that the covariance matrix 𝛀 is positive definite with ˆ for which k 𝛀 ˆ − 𝛀k2 < 𝜆¯ −1 holds, is also 𝜆 𝑚𝑖𝑛 {𝛀} ≥ 𝜆¯ −1 . Thus, any matrix 𝛀 positive definite. This in turn implies that, if 𝜆¯ is constant, then enough samples on ˆ is positive definite. In this the order of O (min{𝑠 + 𝑝, 𝑑 2 }𝜏 log 𝑝) guarantee that 𝛀 case, it can be inverted to further obtain an estimate of the covariance matrix. Further remarks ¯ Theorem 13 characterizes the performance of the FGL On Initializations of 𝚺. algorithm under the initialization 𝚺¯ = R 𝜇 for appropriate tuning of the parameter 𝜇 as in (5.11). It turns out from the analysis that the same desired performance is attained as long as the initialization 𝚺¯ is positive definite, and is appropriately close to the true Σ in both the spectral norm and the max-norm. For example, the sample covariance matrix R satisfies these conditions, but only when 𝑛 > 𝑝. This suggests, that R, which otherwise might have been a standard candidate for initialization of such an iterative covariance estimation algorithms, is a good initialization only when the number of observations is relatively large. However, when this is the case, initializing with R leads to reduced space complexity . To see this note that 107 calculating the LASSO objective in (5.5) does not require computing and storing R, but rather, it can be done directly solely via access on the x𝑖 ’s since for any two Í𝑛 Í𝑛 vectors u and v, it holds uT Rv = 𝑛1 𝑖=1 (uT x𝑖 )(vT x𝑖 ), and Rv = 𝑛1 𝑖=1 (vT x𝑖 )x𝑖 . Comparison to the GLASSO. The Graphical-LASSO algorithm [75] is an iterative algorithm. It starts with an initial estimate 𝚺¯ 0 (say) of Σ and it iteratively updates its rows and columns. At each iteration, it solves a LASSO problem (as the name of the algorithm suggests). This is similar to the LASSO problem in (5.5) of the FGL algorithm, but other than that there are important differentiating features between the two algorithms as discussed next. First, the GLASSO at each iteration updates ¯ Instead, the the last (after permutation) row/column of the covariance estimate 𝚺. ¯ but rather outputs an estimate of the FGL algorithm never operates directly on 𝚺, precision matrix. Each iteration of the GLASSO results in the last row/column of the 𝚺¯ −1 that is sparse (owing to the structure-promoting nature of the LASSO), but at the same time it ruins any structure of the rest of the blocks of 𝚺¯ −1 obtained in previous iterations. In contrast, since the FGL algorithm operates directly on the precision matrix, it does not suffer from this issue. Besides, updating 𝚺¯ at each iteration makes the analysis of the GLASSO hard. The fact that the solutions of the LASSO problems in our algorithm are independent allows us to obtain the theoretical guarantees in Theorem 13. To the best of our knowledge, analogous results for the GLASSO are not available in the literature Regarding time complexity, both the algorithms are scalable. Yet, FGL is a “oneshot" algorithm in the sense that it only requires solving 𝑝 LASSO problems. Algorithms like the GLasso [75], the P-GLasso and the DP-Glasso [111] require at least the same complexity. An additional feature of FGL, as mentioned before, is the fact that it can be parallelized over different machines for even faster performance. Comparison with existing guarantees on the log det-optimization. As discussed in the introduction, the convex nature of the estimator in (5.1) allows analyzing its statistical performance [134, 138]. In particular, Ravikumar et al.[134, Thm. 1] used a primal-dual witness approach to prove that, under an appropriate incoherence assumption on the Hessian of the log det term and under enough number of observations 𝑛 > 𝐶𝑑 2 𝜏 log 𝑝 (for some 𝐶 a constant that depends on the incoherence q 𝜏 log 𝑝 ˆ ˆ , parameter), the optimal solution 𝛀 of (5.1) satisfies k 𝛀 − 𝛀kmax ≤ 𝑀3 𝑛 1 with probability at least 1 − 𝑝 𝜏−2 . Here, 𝑀3 depends on the model parameters ˆ − 𝛀kmax = such as 𝛾, 𝜅, etc. (for example, when these are constants then k 𝛀 p O ( (𝜏 log 𝑝)/𝑛)). Of course, this bound coincides with the result of Theorem 108 ˆ ⊆ Supp(𝛀), which is also 13(ii). Moreover, it is shown in [134] that Supp( 𝛀) guaranteed by Theorem 13(i). Hence, for Gaussian random variables, Theorem 13 shows that the FGL algorithm attains the same order-wise performance bounds as state-of the-art corresponding results on the convex optimization in (5.1). It should be noted however that the results in [134] go beyond Gaussians; we defer such extensions for our setting to future work. Also, the two results hold under different incoherent conditions (compare Assumption 5 to [134, Ass. 1]), which are not directly comparable. Please refer to the new remark for a discussion on the interpretation of Assumption 5. It is worth mentioning that the error bound on the Frobenius norm derived in Corollary 6 also coincides with corresponding best known results in the literature. In particular, Rothman et al. [138] showed that under a mild restriction on the minimum eigenavalue of the covariance matrix it holds with high probability that q (𝑠+𝑝) log 𝑝 ˆ − 𝛀k𝐹 ≤ 𝑀1 , for 𝑀 depending on the parameters 𝜆¯ and 𝜆. k𝛀 𝑛 Overall, it has been shown that the algorithm proposed in this section enjoys performance guarantees (at least in the Gaussian case), which are as strong as the best known ones in the literature regarding the performance of the ℓ1 -regularized log det minimization. However, the former has superior computational performance and can scale with increasing high-dimensions of modern applications. On Assumption 5. The incoherence Assumption 5 is similar (but not the same) to standard assumptions imposed on Σ in the performance analysis of the LASSO [115, 177]. Intuitively, this assumption limits the influence between different random variables in the following form. Suppose x is a zero-mean Gaussian random vector with covariance Σ. Then Assumption 5 is equivalent to the following, ∀𝑖 ∈ [ 𝑝],   max kE xS̄𝑖 |xS𝑖 k∞ ≤ 1 − 𝛼. kx S𝑖 k∞ ≤1 (5.13) Since E[xS̄𝑖 ] = 0, this can be interpreted as a requirement that the influence of the variables in S𝑖 on the variables in S̄𝑖 is not large. Numerical Experiments In this section, we illustrate the validity of the predictions of Theorem 13 via numerical simulations. For two different structures of the underlying graphical model, namely a line-graph and a star-graph, and for varying parameters 𝑝 and 𝑛, we produce realizations of Gaussian observations and report the probability of successful signed-support recovery (see Corollary 6) and the max-norm of the 109 estimation error. The structure of the underlying graph determines the sparsity pattern of the concentration matrix. Chain graph. For the chain graph, the precision matrix 𝛀 has non-zero entries only on the diagonal and on the upper and lower diagonals, i.e., it is tri-diagonal. For the simulations, we set 𝛀𝑖, 𝑗 = 0.5 for |𝑖 − 𝑗 | = 1 and 𝛀𝑖,𝑖 = 2. Note that for the chain graph 𝑑 = 2, and for the specific values of the precision matrix, Assumption 5 is satisfied with parameter 𝛼 = 0.732. For each one of the simulated values of pairs (𝑛, 𝑝), we draw 𝑁 = 40 batches of 𝑛 Gaussian random observations with covarianceqΣ = Ω−1 . We run q the FGL algorithm log 𝑝 log 𝑝 on each data batch with input parameters 𝜆 = 𝑛 and 𝜇 = 𝑛 (in consistency with the scaling suggested by Theorem 13.). In Figure 5.1a we have plotted the probability of signed support recovery as a function of the number of observations. As expected, more samples are needed as the problem size 𝑝 increases. In Figure 5.1b the data are plotted against the rescaled sample size 𝑛/log 𝑝. As suggested by Corollary 5, the curves corresponding to different values of 𝑝 pile up, verifying that a sample size of O (𝑑 2 log 𝑝) is sufficient for successful signed support recovery. Star graph. We build the star graph by connecting its central node (first random variable) to 𝑝/10 other nodes and the rest of the nodes are disconnected. In fact, in the first row/column of the concentration matrix we set the first 𝑝/10 entries to be 0.5 and similarly, we add a scaled identity matrix to make the smallest eigen value to 1. Thus, for the star graph 𝑑 = 𝑝/10, and for the specific values of the precision matrix, Assumption 5 is satisfied with parameter 𝛼 = 0.8. 5.4 Proof of the Main Results Proof of Theorem 13 Proof 4 In this section we prove Theorem 13 through several steps and lemmas. Before anything, we mention this result which is a simple modification of the results in [187, Proposition 2.1] and [134, Lemma 1.] Lemma 15 Consider a zero-mean Gaussian random variable with covariance ma¯ Given observations {x𝑖 }𝑖=1,...,𝑛 of the random variable, trix Σ where 𝜆 max (Σ) = 𝜆. Í𝑛 x𝑖 x𝑇𝑖 . Then with probability at we construct the sample covariance as R = 𝑛1 𝑖=1 110 Prob. of Successful Support Recovery. 1 0.9 0.9 0.8 0.8 0.7 0.7 Prob. of Success Prob. of Success Prob. of Successful Support Recovery. 1 0.6 0.5 0.4 0.3 0.2 0 400 500 600 700 800 900 1000 1100 0.5 0.4 0.3 0.2 p=80 p=160 p=240 p=300 0.1 0.6 p=80 p=160 p=240 p=300 0.1 0 50 1200 100 150 200 n n/ log p (a) (b) 250 300 Figure 5.1: Plots of the probability of signed support recovery of the precision matrix corresponding to a chain graph, and for different values of 𝑝 as a function of (a) the number of observations 𝑛, and, (b) of the scaled sample size 𝑛/log 𝑝. The different curves pile up in (b) as predicted by Corollary 5. Each simulation point corresponds to an average over 𝑁 = 40 points. Prob. of Successful Support Recovery. Prob. of Successful Support Recovery. 1 1 p=80 p=150 0.9 0.8 0.8 0.7 0.7 Prob. of Success Prob. of Success 0.9 p=80 p=150 0.6 0.5 0.4 0.6 0.5 0.4 0.3 0.3 0.2 0.2 0.1 0.1 0 0 0 2 4 6 8 10 12 14 16 18 20 0 200 400 600 n/(d log p) n (a) (b) 800 1000 Figure 5.2: Plots of the probability of signed support recovery of the precision matrix corresponding to a star graph with parameter 𝑑 = 𝑝/10, and for two different values of 𝑝 as a function of (a) the number of observations 𝑛, and, (b) of the scaled sample size 𝑛/log 𝑝/𝑑. The different curves pile up in (b) as predicted by Corollary 5. Each simulation point corresponds to an average over 𝑁 = 40 points. 5 least 1 − 𝑝 𝜏−2 we have √ kR − Σkmax ≤ 80 2𝜆¯ r 𝜏 log 𝑝 , 𝑛 r |S𝑖 |𝜏 log 𝑝 kRS𝑖 ,S𝑖 − ΣS𝑖 ,S𝑖 k2 ≤ 32𝜆¯ , 𝑛 (5.14) 1200 111 if 𝑛 satisfies (5.9). In the next step, we desire to get similar bounds on the error of the shrinkage estimator with parameter chosen as in (5.11). We prove the following lemma, Lemma 16 Consider a zero-mean Gaussian random variable with covariance ma¯ Given observations {x𝑖 }𝑖=1,...,𝑛 of the random variable, trix Σ where 𝜆 max (Σ) = 𝜆. we construct the shrinkage estimator with parameter 𝜇 set to be as in (5.11). Then 5 with probability at least 1 − 𝑝 𝜏−2 we have r √ 𝜏 log 𝑝 kR 𝜇 − Σkmax ≤ 160 2𝜆¯ , 𝑛 r |S𝑖 |𝜏 log 𝑝 , kR 𝜇,(S𝑖 ,S𝑖 ) − ΣS𝑖 ,S𝑖 k2 ≤ 96𝜆¯ 𝑛 if 𝑛 satisfies (5.9). Proof 5 First, we desire to bound kR − Σkmax . Note that due to (5.2) we have, kR 𝜇 − Σkmax ≤ 𝜇kDR − Σkmax + (1 − 𝜇)kR − Σkmax   ≤ 𝜇 max kR 𝜇 − Σkmax , max |Σ𝑖, 𝑗 | 𝑖, 𝑗 + kR − Σkmax r √ 𝜏 log 𝑝 ≤ 160 2𝜆¯ 𝑛 (5.15) where the last inequality is due to lemma 15 and the way we chose 𝜇 in (5.11). Now for the spectral norm we have, kR 𝜇,(S𝑖 ,S𝑖 ) − ΣS𝑖 ,S𝑖 k2 ≤ 𝜇kDR,(S𝑖 ,S𝑖 ) − ΣS𝑖 ,S𝑖 k2 + (1 − 𝜇)kRS𝑖 ,S𝑖 − ΣS𝑖 ,S𝑖 k2 ≤ 𝜇 kΣS𝑖 ,S𝑖 k2 + kDR,(S𝑖 ,S𝑖 ) − DΣ,(S𝑖 ,S𝑖 ) k2
+ kDΣ,(S𝑖 ,S𝑖 ) k2 + kRS𝑖 ,S𝑖 − ΣS𝑖 ,S𝑖 k2
≤ 𝜇 2𝜆¯ + kRS𝑖 ,S𝑖 − ΣS𝑖 ,S𝑖 k2 + kRS𝑖 ,S𝑖 − ΣS𝑖 ,S𝑖 k2 r |S𝑖 |𝜏 log 𝑝 ≤ 96𝜆¯ 𝑛 where the last inequality is due to lemma 15 and the value of 𝜇 in (5.11). (5.16) 112 In the next step, we prove that the shrinkage estimator R 𝜇 has similar assumptions as Σ including the incoherence assumption. Lemma 17 Consider a zero-mean Gaussian random variable with covariance ma¯ Given observations {x𝑖 }𝑖=1,...,𝑛 of the random variable, we trix Σ where 𝜆 max (Σ) = 𝜆. construct the shrinkage estimator with parameter 𝜇 set to be as in (5.11). Suppose that Σ satisfies assumption [A1] with parameter 𝛼. Then with probability at least 5 1 − 𝑝 𝜏−2 the shrinkage estimator satisfies assumption [A1] with parameter 𝛼/2 and also inequality (5.8) with parameter 2𝛾 if 𝑛 satisfies (5.9). 5 Proof 6 According to lemma 16 with probability at least 1 − 𝑝 𝜏−2 we have r √ 𝜏 log 𝑝 , kR 𝜇 − Σkmax ≤ 160 2𝜆¯ 𝑛 r |S𝑖 |𝜏 log 𝑝 kR 𝜇,(S𝑖 ,S𝑖 ) − ΣS𝑖 ,S𝑖 k2 ≤ 96𝜆¯ . (5.17) 𝑛 Let us denote E := R 𝜇 − Σ and rename 𝚺¯ := R 𝜇 to avoid unnecessary notations. Then for all 𝑖 we have k 𝚺¯ S̄𝑖 ,S𝑖 ( 𝚺¯ S̄𝑖 ,S𝑖 ) −1 k∞ = k(ES̄𝑖 ,S𝑖 + ΣS̄𝑖 ,S𝑖 )(ES𝑖 ,S𝑖 + ΣS𝑖 ,S𝑖 ) −1 k∞ ≤ kΣS̄𝑖 ,S𝑖 (ΣS̄𝑖 ,S𝑖 ) −1 k∞ .k(I + (ΣS𝑖 ,S𝑖 ) −1 ES𝑖 ,S𝑖 ) −1 k∞ + kES̄𝑖 ,S𝑖 k∞ k( 𝚺¯ S𝑖 ,S𝑖 ) −1 k∞ 1 + 2𝛾.|S𝑖 |kES̄𝑖 ,S𝑖 kmax ≤ (1 − 𝛼). 1 − k(ΣS𝑖 ,S𝑖 ) −1 ES𝑖 ,S𝑖 k∞ (5.18) Now we bound each term above, (note that |S𝑖 | ≤ 𝑑) √ k(ΣS𝑖 ,S𝑖 ) −1 ES𝑖 ,S𝑖 k∞ ≤ 𝑑 k(ΣS𝑖 ,S𝑖 ) −1 ES𝑖 ,S𝑖 k2 ≤ k(ΣS𝑖 ,S𝑖 ) −1 k2 kES𝑖 ,S𝑖 k2 √ r 96𝜆¯ 𝑑 𝑑𝜏 log 𝑝 3𝛼 ≤ ≤ 𝜆 𝑛 4−𝛼 (5.19) where the last inequality is because of (5.9). On the other hand, r √ 𝜏 log 𝑝 𝛼 2𝛾.|S𝑖 |kES̄𝑖 ,S𝑖 kmax ≤ 320 2𝛾𝑑 𝜆¯ ≤ (5.20) 𝑛 4 And the last inequality is due to the choice of 𝑛 (5.9) for large enough constant 𝑀. Now, combining (5.18), (5.19) and (5.20) implies the following, k 𝚺¯ S̄𝑖 ,S𝑖 ( 𝚺¯ S̄𝑖 ,S𝑖 ) −1 k∞ ≤ 1 − 𝛼 2 (5.21) 113 as desired. Now we just need to bound the following k( 𝚺¯ S̄𝑖 ,S𝑖 ) −1 k∞ ≤ k(ΣS̄𝑖 ,S𝑖 ) −1 k∞ .k(I + (ΣS𝑖 ,S𝑖 ) −1 ES𝑖 ,S𝑖 ) −1 k∞ 1 ≤𝛾 1 − k(ΣS𝑖 ,S𝑖 ) −1 ES𝑖 ,S𝑖 k∞ (5.22) Now we need to bound the later term, k(ΣS𝑖 ,S𝑖 ) −1 ES𝑖 ,S𝑖 k∞ ≤ √ 𝑑 k(ΣS𝑖 ,S𝑖 ) −1 ES𝑖 ,S𝑖 k2 ≤ k(ΣS𝑖 ,S𝑖 ) −1 k2 kES𝑖 ,S𝑖 k2 √ r 96𝜆¯ 𝑑 𝑑𝜏 log 𝑝 1 ≤ ≤ 𝜆 𝑛 2 (5.23) and the last inequality is similarly due to the choice of 𝑛. Combining (5.22) and (5.23) leads to k( 𝚺¯ S̄𝑖 ,S𝑖 ) −1 k∞ ≤ 2𝛾 (5.24) 5 Now from now on suppose 𝚺¯ = R 𝜇 and with probability at least 1 − 𝑝 𝜏−2 , lemma 17 holds. Recall that the FGL algorithm performs 𝑝 steps of iterations 𝑖 = 1, . . . , 𝑝 and in the step 𝑖, it estimates the 𝑖 𝑡ℎ row/column of the concentration matrix 𝛀. All these steps are independent and can be performed in parallel. Thus, we just analyze how well the algorithm performs in recovering the 𝑝 𝑠𝑡 row/column of 𝛀 and then the same goes for the other steps as well. So for now assume that the we are in the 𝑝 𝑡ℎ step of the algorithm that recovers 𝑝 𝑡ℎ row/column of 𝛀. The algorithm first solves the following Lasso problem, 1 T ¯ 12 𝛽 + 𝜆k 𝛽k1 . 𝛽ˆ = arg min 𝛽T 𝚺¯ 11 𝛽 + 𝝈 𝛽 2 (5.25) Recall that the output of this problem was supposed to be an estimation 𝛽ˆ of the 𝜔12 quantity 𝛽0 := 𝜔 and now we analyze its performance in recovering the structure 22 of 𝛽0 and also in terms of the maximum norm of the error k 𝛽ˆ − 𝛽0 k𝑚𝑎𝑥 . Performance of the Lasso To analyze the Lasso problem (5.25), we utilize a well-known technique called primal-dual witness. Note that based of the assumption [B2], 𝚺¯ 11 is positive definite which implies strict convexity of the objective function in (5.25) and also uniqueness ˆ Besides, 𝛽ˆ satisfies the optimality condition of the optimizer 𝛽. ¯ 12 + 𝜆x̂ = 0, 𝚺¯ 11 𝛽ˆ + 𝝈 ˆ 1, x̂ ∈ 𝜕 k 𝛽k (5.26) 114 ˆ 1 is sub-differential of the ℓ1 -norm calculated at 𝛽. ˆ Now we proceed by where 𝜕 k 𝛽k ˜ 𝑥) constructing a pair ( 𝛽, ˜ that satisfies (5.26) and due to uniqueness of the optimizer ˆ We are going to do this in two steps, first constructing the pair ( 𝛽, ˜ 𝑥) 𝛽˜ = 𝛽. ˜ and then proving that this pair satisfies (5.26). ˜ 𝑥): 1. Constructing the pair ( 𝛽, ˜ We denote support of the vector 𝛽0 by S and its 𝑐 compliment by S and for a vector v ∈ R 𝑝−1 , vS ∈ R|S| is a sub-vector of v with entries {𝑣 𝑖 }𝑖∈S . ˜ 𝑥), In order to construct ( 𝛽, ˜ we set 𝛽˜S𝑐 = 0 and 1 T 𝛽˜S = arg min 𝛽ST 𝚺¯ S,S 𝛽S + 𝜎 ¯ 12 S 𝛽S + 𝜆k 𝛽S k1 . 𝛽S 2 Thus 𝛽˜S satisfies the following optimality condition, 𝚺¯ S,S 𝛽˜S + 𝜎 ¯ 12 S + 𝜆𝑥˜S = 0, 𝑥˜S ∈ 𝜕 k 𝛽˜S k1 . (5.27) Note that in (5.27) we also set 𝑥˜S . At last we choose 𝑥˜S𝑐 to be 1 ¯ 12 S𝑐 ). 𝑥˜S𝑐 = − ( 𝚺¯ S𝑐 ,S 𝛽˜S + 𝜎 𝜆 (5.28) ˜ 𝑥), Because of the way we constructed the pair ( 𝛽, ˜ we already have 𝚺¯ 11 𝛽˜ + 𝜎 ¯ 12 + 𝜆𝑥˜ = 0, 𝑥˜S ∈ 𝜕 k 𝛽˜S k1 . We just need to check if 𝑥˜S𝑐 ∈ 𝜕 k 𝛽˜S𝑐 k1 or equivalently if k 𝑥˜S𝑐 k∞ ≤ 1 because 𝛽˜S𝑐 = 0. 2. Verifying Optimality Conditions: First we define error of estimation in 𝚺¯ to be Z := 𝚺¯ − Σ and also Δ := 𝚺¯ 11 𝛽˜ − 𝚺11 𝛽0 . Thus    Δ S = 𝚺¯ S,S ( 𝛽˜S − 𝛽0 S ) + ZS,S 𝛽0 S    Δ S𝑐 = 𝚺¯ S𝑐 ,S ( 𝛽˜S − 𝛽0 S ) + ZS𝑐 ,S 𝛽0 S  ⇒ Δ S𝑐 = WΔ S − WZS,S 𝛽0 S + ZS𝑐 ,S 𝛽0 S , (5.29) where W := 𝚺¯ S𝑐 ,S ( 𝚺¯ S,S ) −1 . Now, combining (5.28), (5.29) and 𝜎 ¯ 12 S𝑐 = −ΣS𝑐 ,S 𝛽0 S + z12 S𝑐 implies the following, 𝑥˜S𝑐 = 1 1 1 1 WΔ S + WZS,S 𝛽0 S − ZS𝑐 ,S 𝛽0 S − z12 S𝑐 . 𝜆 𝜆 𝜆 𝜆 (5.30) 115 We also make use the equation (5.27) and 𝜎 ¯ 12 S = −ΣS,S 𝛽0 S + z2 12 S to rewrite (5.30) as 1 1 1 𝑥˜S𝑐 = Wz12 S + W𝑥˜S + WZS,S 𝛽0 S − ZS𝑐 ,S 𝛽0 S 𝜆 𝜆 𝜆 1 − z12 S𝑐 . (5.31) 𝜆 We desire to bound k 𝑥˜S𝑐 k∞ and so 1 k 𝑥˜S𝑐 k∞ ≤ kWk∞ kz12 S k∞ + kWk∞ k 𝑥˜S k∞ 𝜆 1 + kWk∞ kZS,S k𝑚𝑎𝑥 k 𝛽0 S k1 𝜆 1 1 (5.32) + kZS𝑐 ,S k𝑚𝑎𝑥 k 𝛽0 S k1 + kz12 S𝑐 k𝑚𝑎𝑥 . 𝜆 𝜆 From the assumptions we have a bound on all the variables above including 𝛼¯ kZk𝑚𝑎𝑥 ≤ 𝑓 (𝑛, 𝑝), kWk∞ ≤ 1 − 𝛼, ¯ k 𝛽0 k1 ≤ 𝜅 and 𝜆 = 4−2 𝛼¯ (𝜅 + 1) 𝑓 (𝑛, 𝑝) that we can use in (5.32) which results in k 𝑥˜S𝑐 k∞ ≤ 1. 3. Deriving Performance of the Lasso: The previous two steps showed that 𝛽ˆ = 𝛽˜ is the unique answer to the lasso problem (5.25) with the property that ˜ ⊂Support{𝛽0 }. This already proves the first part of the theorem Since Support{ 𝛽} ˜ ⊂Support{𝛽0 } in the way we estimate 𝝎ˆ 12 from 𝛽ˆ in the algorithm, Support{ 𝛽} implies Support{𝝎ˆ 12 } ⊂Support{𝜔12 }. On the other hand, due to equation (5.26) Δ + z12 + 𝜆𝑥ˆ = 0 ⇒ kΔ k∞ ≤ 𝜆 + kz12 k∞ ≤ 𝐶1 𝑓 (𝑛, 𝑝), (5.33) ¯ (𝜅+1) . We also can rewrite (5.27) as where 𝐶1 = 1 + (4−2𝛼) 𝛼¯ 𝚺¯ S,S 𝛽˜S − 𝚺¯ S,S 𝛽0 S + 𝚺¯ S,S 𝛽˜0 S + 𝜎 ¯ 12 S + 𝜆𝑥˜S = 0 ⇒ ¯ −1 ¯ −1 𝛽˜S − 𝛽0 S = 𝚺¯ −1 S,S ZS,S 𝛽0 S − 𝚺S,S z12 S − 𝜆 𝚺S,S 𝑥˜S . Thus, k 𝛽˜ − 𝛽0 k∞ = k 𝛽˜S − 𝛽0 S k∞ ≤ 𝛾(𝜅 + 1) 𝑓 (𝑛, 𝑝) (4 − 2𝛼) ¯ 𝑓 (𝑛, 𝑝) = 𝛾(𝐶1 + 𝜅) 𝑓 (𝑛, 𝑝), + 𝛾(𝜅 + 1) 𝛼¯ (5.34) ˜ where we used 𝛽˜S𝑐 = 𝛽0 S𝑐 = 0 and k 𝚺¯ −1 S,S k∞ ≤ 𝛾 from the assumptions. Besides, ( 𝛽− ˜ ⊂ S =Support{𝛽0 }, 𝛽0 ) has at most 𝑠 = |S| non-zeros entries because Support{ 𝛽} so k 𝛽˜ − 𝛽0 k1 ≤ 𝑠k 𝛽˜ − 𝛽0 k∞ ≤ 𝑠𝛾(𝐶1 + 𝜅) 𝑓 (𝑛, 𝑝) ≤ 1, where last inequality is because of the additional assumption in the theorem. (5.35) 116 12 Estimating 𝜔12 and 𝜔22 from 𝜔 𝜔22 We know from previous section that how well the Lasso problem (5.25) performs 12 in estimating 𝜔 𝜔22 . The FGL algorithm uses the following equations to estimate 𝜔12 and 𝜔22 using the answer 𝛽ˆ to (5.25), 1 , 𝜔ˆ 22 = 𝜎 ¯ 22 + 𝛽ˆT 𝚺¯ 11 𝛽ˆ 𝛽ˆ 𝝎ˆ 12 = . (5.36) 𝜎 ¯ 22 + 𝛽ˆT 𝚺¯ 11 𝛽ˆ We are interested in bounding the error terms | 𝜔ˆ 22 − 𝜔22 | and 𝝎ˆ 12 − 𝜔12 k∞ k. As the first step, ˆ = (𝜎22 − 𝜎 (𝜎22 + 𝛽0T 𝚺11 𝛽0 ) − ( 𝜎 ¯ 22 + 𝛽ˆT 𝚺¯ 11 𝛽) ¯ 22 ) + 2𝛽0T Δ + ( 𝛽ˆ − 𝛽0 ) T Δ + 𝛽0T ( 𝚺¯ 11 − 𝚺11 ) 𝛽0 + 𝛽0T ( 𝚺¯ 11 − 𝚺11 )( 𝛽ˆ − 𝛽0 ) Therefore, Cauchy–Schwarz and triangle inequality implies ˆ ≤ |𝜎22 − 𝜎 |(𝜎22 + 𝛽0T 𝚺11 𝛽0 ) − ( 𝜎 ¯ 22 + 𝛽ˆT 𝚺¯ 11 𝛽)| ¯ 22 | + 2k 𝛽0 k1 kΔ k∞ + k 𝛽ˆ − 𝛽0 k1 kΔ k + k 𝛽0 k12 k 𝚺¯ 11 − 𝚺11 k𝑚𝑎𝑥 + k 𝛽0 k1 k 𝚺¯ 11 − 𝚺11 k𝑚𝑎𝑥 k 𝛽ˆ − 𝛽0 k1 ≤ (1 + 2𝜅𝐶1 + 𝐶1 + 𝜅 2 + 𝜅) 𝑓 (𝑛, 𝑝) (5.37) On the other hand, because of the extra assumption in the theorem | 𝜎 ¯ 22 − 𝜎22 | ≤ 𝑓 (𝑛, 𝑝) ≤ 𝜆/2 and therefore ˆ ≥𝜎 (𝜎 ¯ 22 + 𝛽ˆT 𝚺¯ 11 𝛽) ¯ 22 ≥ 𝜎22 − | 𝜎 ¯ 22 − 𝜎22 | ≥ 𝜆 − 𝜆/2 = 𝜆/2 Now, combining (5.37) and (5.38) gives us the following 1 1 − 𝜔ˆ 22 − 𝜔22 = 𝜎 ¯ 22 + 𝛽ˆT 𝚺¯ 11 𝛽ˆ 𝜎22 + 𝛽0T 𝚺11 𝛽0 2 ≤ 2 (1 + 2𝜅𝐶1 + 𝐶1 + 𝜅 2 + 𝜅) 𝑓 (𝑛, 𝑝) 𝜆 (5.38) (5.39) Finally, k 𝝎ˆ 12 − 𝜔12 k∞ = k 𝛽ˆ𝜔ˆ 22 − 𝛽0 𝜔ˆ 22 + 𝛽0 𝜔ˆ 22 − 𝛽0 𝜔22 k∞ ≤ k 𝛽ˆ − 𝛽0 k∞ 𝜔ˆ 22 + | 𝜔ˆ 22 − 𝜔22 |k 𝛽0 k∞ 2 ≤ 𝛾(𝐶1 + 𝜅) 𝑓 (𝑛, 𝑝) + 𝜅(1 + 2𝜅𝐶1 + 𝐶1 + 𝜅 2 + 𝜅) 𝑓 (𝑛, 𝑝), 𝜆 (5.40) 117 where the last inequality is from (5.38), (5.34) and (5.39). This implies our final result in the theorem, k 𝝎ˆ 12 − 𝜔12 k∞   2 2 ≤ 𝛾(𝜅 + 𝐶1 ) + 𝜅(1 + 𝜅 + 𝜅 + 𝐶1 (2𝜅 + 1)) 𝑓 (𝑛, 𝑝). 𝜆 (5.41) 118 Chapter 6 UNIVERSALITY IN LEARNING FROM LINEAR MEASUREMENTS Recovering1 a structured signal from a set of linear observations appears in many applications in areas ranging from finance to biology, and from imaging to signal processing. More formally, the goal is to recover an unknown vector x0 ∈ R 𝑝 , from observations of the form 𝑦𝑖 = a𝑖T x0 , for 𝑖 = 1, . . . , 𝑛. In many modern applications, the ambient dimension of the signal, 𝑝, is often (overwhelmingly) larger than the number of observations, 𝑝. In such cases, there are infinitely many solutions that satisfy the linear equations arising from the observations, and therefore to obtain a unique solution one must assume some prior structure on the unknown vector. Common examples of structured signals are sparse and group-sparse vectors [35, 61], low-rank matrices [34, 135], and simultaneously-structured matrices [39, 128]. To this end, we use a convex penalty function 𝑓 : R 𝑝 → R, that captures the structure of the structured signal, in the sense that signals that do not adhere to the desired structure will have a higher cost. Therefore, the following estimator is used to recover x0 , x̂ = arg min 𝑓 (x) x subject to, 𝑦𝑖 = a𝑖T x, 𝑖 = 1, . . . , 𝑛 . (6.1) Popular choices of 𝑓 (·) include the ℓ1 -norm for sparse vectors [176], and the nuclear norm for low-rank matrices [135]. A canonical question in this area is “how many measurements are needed to recover x0 via this estimator?" This question has been extensively studied in the literature (see [6, 40, 156] and the references therein.) The answer depends on the a𝑖 and is very difficult to determine for any given set of measurement vectors. As a result, it is common to assume that the measurement vectors are drawn randomly from a given distribution and to ask whether the unknown vector can be recovered with high probability. In the special case where the entries of the measurement matrix are drawn iid from a Gaussian distribution, the minimum number of measurements for the recovery of x0 with high probability is known (and is related to the concept of the Gaussian width [6, 40, 156]). For instance, it has been shown that 2𝑘 log( 𝑝/𝑘) linear measurements 1 This chapter is mainly based on the work in [3] 119 is required to recover a 𝑘−sparse signal [58], and 3𝑟 𝑝 measurements suffice for the recovery of a symmetric 𝑝 × 𝑝 rank-𝑟 matrix [40, 127]. Recently, Oymak et al [130] showed that these thresholds remain unchanged, as long as the entries of each a𝑖 are i.i.d and drawn from a "well-behaved" distribution. It has also been shown that similar universality holds in the case of noisy measurements [131]. Although these works are of great interest, the independence assumption on the entries of the measurement vectors can be restrictive. In certain applications in communications, phase retrieval, covariance estimation, the entries of the measurement vectors a𝑖 have correlations. in this section, we show a much stronger universality result which holds for a broader class of measurement distributions. Here is an informal description of our result: Assume the measurement vectors a𝑖 are drawn iid from some given distribution. In other words, the measurement vectors are iid random, but their entries are not necessarily so. Then the minimum number of observations needed to recover x0 from (6.1) with high probability, depends only on the first two statistics of the a𝑖 , i.e., their mean vector 𝜇, and covariance matrix 𝚺. We anticipate that this universality result will have many practical ramifications. in this section we focus on the ramifications to the problem of recovering a structured matrix, X0 ∈ R 𝑝×𝑝 , from quadratic measurements (a.k.a. rank-one projections). In this problem, we are given observations of the form 𝑦𝑖 = a𝑖T X0 a𝑖 = Tr(X0 (a𝑖 a𝑖T )) = vec(𝑋) 𝑡 vec(a𝑖 a𝑖𝑡 ) for 𝑖 = 1, . . . , 𝑚.2 Such measurement schemes appear in a variety of problems [31, 44, 104, 105, 194]. An interesting application of learning from quadratic measurements is the PhaseLift algorithm [36] for phase retrieval. In phase retrieval, the goal is to recover the signal x0 from quadratic measurements of the form, 𝑦𝑖 = |a𝑖T x0 | 2 = a𝑖T (x0 xT0 )a𝑖 . Note that x0 x𝑡0 is a low-rank (in this case rank-1) matrix and PhaseLift relaxes this constraint to a non-negativity constraint and minimizes nuclear norm to encourage a low rank solution. Quadratic measurements also appears in non-coherent energy measurements in communications and signal processing [8, 180], sparse covariance estimation [44, 194], and sparse phase retrieval [104, 147]. Recently, Chen et al [44] proved sufficient bounds on the number of measurements for various structures on the matrix X0 . However, to the best of our knowledge, prior to this work, the precise number of required measurements for 2 The reader should pardon the abuse of notation as the measurement vectors are now vec(a a𝑡 ). 𝑖 𝑖 120 perfect recovery was unknown. For example, when the a𝑖 have iid Gaussian entries (note that the measurement vectors, which are now vec(a𝑖 a𝑖𝑡 ), are no longer iid Gaussian) we show that 3𝑝𝑟 measurement is necessary and sufficient for the perfect recovery of a rank-𝑟 matrix from quadratic measurements. In the special case of phase retrieval, we therefore demonstrate that 3𝑝 measurements is necessary and sufficient for perfect recovery of x0 , which settles the long standing open question of the recovery threshold for PhaseLift. In particular, this indicates that 2𝑝 extra phaseless measurements is all that is needed to compensate the missing phase information. The remainder of the paper is structured as follows. The problem setup and definitions are given in Section 6.1. In Section 6.2, we introduce our universality framework, which states that the number of required observations for the recovery of an unknown model depends only on the first two statistics of the measurement vectors. As an applications, in Section 6.3, we apply this universality theorem to derive tight bounds (i.e., necessary and sufficient conditions) on the required number of observations for matrix recovery via quadratic measurements. 6.1 Preliminaries Notations We start by introducing some notations that are used throughout the paper. Bold lower letters x, y, . . . are used to denote vectors, and bold upper letters X, Y, . . . are for matrices. For a matrix X ∈ R𝑛×𝑝 , Vec(X) ∈ R 𝑝𝑛 returns the vectorized form of the matrix. kXk2 , kXk𝐹 , kXk★ and Tr(X) represent the operator norm, the Frobenius norm, the nuclear norm and the trace of the matrix X, respectively. kxkℓ 𝑝 denotes the ℓ 𝑝 -norm of the vector x and for matrices, kXkℓ 𝑝 = kVec(X)kℓ 𝑝 . For both vectors and matrices, k · k0 indicates the number of non-zero entries. The set of 𝑝 × 𝑝 positive definite matrices and positive semi-definite matrices are denoted 𝑝 by S++ and S+𝑛 , respectively. The letters g and G are reserved for a Gaussian random vector and matrix with i.i.d. standard normal entries. The letter H is reserved for a random Gaussian Wigner matrix, that is a symmetric matrix whose upperdiagonal entries drawn independently from N (0, 1) whose its diagonals entries are drawn independently from N (0, 2). Finally, the letter I is reserved for the identity matrix. For a random vector a, E[a] and Cov[a] represent the expected value and the covariance matrix of a. 121 Problem Setup We consider the problem of recovering the unknown vector x0 ∈ S ⊆ R 𝑝 from 𝑛 observations of the form 𝑦𝑖 = a𝑖T x0 , 𝑖 = 1, . . . , 𝑛. Here, the known measurement vectors a𝑖 ∈ R 𝑝 ’s are drawn independently and identically from a random distribution. These observations can be reformulated as y = Ax0 , (6.2) where y = [𝑦 1 , . . . , 𝑦 𝑛 ] T ∈ R𝑛 and A = [a1 , . . . , a𝑛 ] T ∈ R𝑛×𝑝 . We focus on the high-dimensional setting where both 𝑛 and 𝑝 grow large. We use the notation 𝑛 = 𝜃 ( 𝑝), to fix the rate at which 𝑛 grows compared to 𝑝. Of special interest is the underdetermined case where the number of measurement is smaller than the ambient dimension. In this case, the problem of signal reconstruction is generally ill-posed unless some prior information is available regarding the structure of x0 . Some popular cases of structures include, sparse vectors, low-rank matrices, and simultaneously-structured matrices. Convex estimator: To recover the structured vector x0 , we minimize a convex function 𝑓 : R 𝑝 → R that enforces this structure. We do this minimization for all feasible points x ∈ S, that satisfy y = Ax. We formally define such estimators as follows, Definition 3 Let x0 ∈ S where S ⊆ R 𝑝 is a convex set. For a convex function 𝑓 : R 𝑝 → R and a measurement matrix A ∈ R𝑛×𝑝 , we define the convex estimator E{x0 , A, S, 𝑓 (·)} as following, x̂ = arg min 𝑓 (x) . x∈S Ax=Ax0 (6.3) We say E{x0 , A, S, 𝑓 (·)} has perfect recovery if and only if x̂ = x0 . Note that we are given the observation vector y = Ax0 in the constraint of (6.3). We aim to characterize the perfect recovery criteria for this estimator. Given a structured vector x0 , the perfect recovery of an estimator E{x0 , A, S, 𝑓 (·)} depends on three factors; the number of observations 𝑛 compared to the dimension of the ambient 𝑛 , and the penalty function, space 𝑝, properties of the measurement vectors {a𝑖 }𝑖=1 𝑓 (·). We briefly explain each factor, below. The rate function 𝜃 (·): We work in the high dimensional regime where both 𝑛 and 𝑝 grow to infinity with a fixed rate 𝑛 = 𝜃 ( 𝑝). Finding the minimum number of 122 measurements to recover x0 via (6.3), translates to finding the smallest rate function 𝜃★ (·), for which our estimator has perfect recovery. This optimal rate function depends on the problem settings and varies in different problems. For instance, in 𝑝 order to recover a rank-𝑟 matrix in S+ , we will need the measurements to be of order 𝑛 = O ( 𝑝), while in the case of 𝑘-sparse matrices, the measurements will be of order 𝑛 = O (𝑘 log( 𝑝 2 /𝑘)), where in many applications 𝑘 is a fraction of 𝑝 2 . The penalty function: We use a convex function 𝑓 (·) that promotes the particular structure of x0 . Exploiting a convex penalty for the recovery of structured signals has been studied extensively [6, 33, 40, 62, 156, 165]. Chandrasekaran et. al. [40] introduced the concept of the atomic norm, which is a convex surrogate defined based on a set of (so-called) "atoms". For instance, the corresponding atomic norm for sparse recovery is the ℓ1 -norm and for low-rank matrix recovery the nuclear norm. Another interesting scenario is when the underlying parameter x0 simultaneously exhibits multiple structures such as being low-rank and sparse. For simultaneously structured signals building the set of atoms is often intractable. Therefore, it has been proposed [43, 128] to use a weighted sum of corresponding atomic norms for each structure as the penalty. The measurement vectors: We consider a random ensemble, where the vectors 𝑛 are drawn independently and identically from a random distribution. Later {a𝑖 }𝑖=1 in Section 6.1, we formally present the required assumptions on this distribution. It has been observed that the estimator (6.3) exhibits a phase transition phenomenon, i.e., there exist a phase transition rate 𝜃★ ( 𝑝), such that when 𝑛 > 𝜃★ ( 𝑝) the optimization program (6.3) successfully recover x0 with high probability, otherwise, when 𝑛 < 𝜃★ ( 𝑝) it fails with high probability [6, 40]. The question is that how is this phase transition is related to the properties of the measurement vectors a𝑖 ’s? Universality in learning: Directly calculating the precise phase transition behavior of the estimator E (x0 , A, S, 𝑓 (·)), for a general random distribution on the measurement vectors is very challenging. Recently, as an extension of Gaussian comparison lemmas due to Gordon [79, 80] and earlier work in [6, 40, 153, 156], a new framework, known as CGMT [165, 171], has been developed which made this analysis 𝑛 , are independently drawn from the possible when the measurement vectors {a𝑖 }𝑖=1 Gaussian distribution, N (0, I 𝑝 ). Another parallel work that makes this analysis 123 possible under the same conditions is known as AMP [62]. However, the Gaussian assumption is critical in the analysis through these frameworks, which restricts us from investigating a vast variety of practical problems. As our main result, we show that, for a broad class of distributions, the phase transition of E (x0 , A, S, 𝑓 (·)) depends only on the first two statistics of the distribution 𝑛 . As a result, the phase transition of the estimator on the measurement vectors {a𝑖 }𝑖=1 remains unchanged when we replace the measurement vectors with the ones drawn from a Gaussian distribution with the same mean vector and covariance matrix. As the phase transition is the same as the one with Gaussian measurements, we can use the CGMT framework to analyze the latter and get the desired result. Equivalent Gaussian Problem: Let 𝜇 := E[a𝑖 ] and 𝚺¯ := Cov[a𝑖 ] for 𝑖 = 1, 2, . . . , 𝑛, and consider the following problem: 1. We are given 𝑛 observations of the form 𝑦˜ 𝑖 = g𝑖T x0 and the measurement 𝑛 . vectors {g𝑖 }𝑖=1 2. The rows of the measurement matrix G = [g1 , . . . , g𝑛 ] T ∈ R𝑛×𝑝 are indepen¯ dently drawn from the multivariate Gaussian distribution N (𝜇, 𝚺). 3. We use the estimator E (x0 , G, S, 𝑓 (·)), as in Definition 3, to recover x0 . In Theorem 14, we show that under certain conditions, the two estimators E (x0 , A, S, 𝑓 (·)) and E (x0 , G, S, 𝑓 (·)) asymptotically exhibit the same phase transition behavior. Before stating our main result in Section 6.2, we discuss the assumptions needed for our universality to hold. Assumptions We show universality for a wide range of distributions on the measurement vector as well as a broad class of convex penalties. Here, we give the conditions needed for the measurement matrix, Assumption 6 [The Measurement Vectors] We say the measurement matrix A = [a1 , . . . , a𝑛 ] T ∈ R𝑛×𝑝 satisfies Assumption 6 with parameters 𝜇 ∈ R 𝑝 and 𝚺¯ ∈ R 𝑝×𝑝 , if the followings hold true. 1. [Sub-Exponential Tails] The vectors a𝑖 ’s are independently drawn from a random sub-exponential distribution, with mean 𝜇 and covariance 𝚺¯  0. 124 k𝜇k 2 −𝜏1 , 2 2. [Bounded Mean] For some constants 𝑐 1 , 𝜏1 > 0, we have E[ka −𝜇k 2] ≤ 𝑐1 · 𝑝 𝑖 for all 𝑖. 𝑖k ) 3. [Bounded Power] For some constants 𝑐 2 , 𝜏2 > 0, we have EVar(ka 2 [ka −𝜇k 2 ] ≤ 𝑖 𝑐 2 · 𝑝 −𝜏2 for all 𝑖 . 2 Assumption 6 summarizes the technical conditions that are essential in the proof of our main theorem. The first assumption on the tail of the distribution enables us to exploit concentration inequalities for sub-exponential distributions. We allow the vector a𝑖 to have a non-zero mean in Assumption 1.2. Yet we require the power of its mean to be small compared to the power of the random part of the vector. Intuitively, one would like the measurement vectors to sample diversely from all the directions in R 𝑝 , and not be biased towards a specific direction. Finally, Assumption 1.3 is meant to control the dependencies among the entries of a𝑖 and is used to prove concentration of 1𝑝 a𝑖T Ma𝑖 around its mean, for a matrix M with bounded operator norm. For instance, for a Gaussian vector g ∼ N (0, I), we have Var[kgk 2 ] = 2𝑝 and E2 [kgk 2 ] = 𝑝 2 . So Assumption 1.3 is satisfied with 𝑐 2 = 2 and 𝜏2 = 1. We will examine these assumptions for the applications discussed in Section 6.3. In addition, we need to enforce a few conditions on the penalty function 𝑓 (·) as follows, Assumption 7 [The Penalty Function] We say the funtion 𝑓 (·) satisfies Assumption 2, if the following holds true. 1. [Separablity] 𝑓 (·) is continuous, convex and separable, where 𝑓 (x) = . Í𝑝 𝑖=1 𝑓𝑖 (𝑥𝑖 ) 2. [Smoothness] The functions { 𝑓𝑖 (·)} are three times differentiable everywhere, except for a finite number of points. 3. [Bounded Third Derivative] For any 𝐶 > 0, there exists a constant 𝑐 𝑓 > 0, 3 such that for all 𝑖, we have | 𝜕 𝜕𝑥𝑓𝑖 (𝑥) 3 | ≤ 𝑐 𝑓 , for all smooth points in the domain of 𝑓𝑖 (·) such that |𝑥| < 𝐶. As observed in the Assumption 2.1, we only consider the special (yet popular) case of separable penalty functions. Common choices include kxkℓ1 and kxkℓ22 for vectors, and kXkℓ1 , kXk𝐹 and Tr(X) (which is equivalent to the nuclear norm of X when X ∈ S+ ) for matrices. We can also apply our theorem for ℓ 𝑝 -norm. This is due to 125 𝑝 the fact that replacing k · kℓ 𝑝 with k · kℓ 𝑝 does not change our estimate, and the latter is a separable function. 6.2 Main Result In this section, we state our main theorem which shows that the performance of the convex estimator E (x0 , A, S, 𝑓 (·)), is independent of the distribution of the measurement vectors. So we can replace them with the Gaussian random vectors with the same mean and covariance. Next, using CGMT framework [165, 171], we analyze the phase transition in the case with Gaussian measurements, in Corollary 7. Later, we will apply this result to some well-known problems in Section 6.3. Universality Theorem Theorem 14 [non-Gaussian=Gaussian] Consider the problem of recovering x0 ∈ S ⊆ R 𝑝 from the measurements y = Ax0 ∈ R𝑛 , using a convex penalty function 𝑓 (·) in the estimator E{x0 , A, S, 𝑓 (·)} in (6.3). Assume S is a convex set and 𝑝 and 𝑛 are growing to infinity at a fixed rate 𝑛 = 𝜃 ( 𝑝). Also assume that 1. 𝑓 : R 𝑝 → R is a convex function that satisfies Assumption 7. 2. The measurement matrix A = [a1 , . . . , a𝑛 ] T satisfies Assumption 6, with 𝜇 := E[a𝑖 ] and 𝚺¯ := Cov[a𝑖 ] for all 𝑖 = 1, . . . , 𝑛 . 3. G = [g1 , . . . , g𝑛 ] T ∈ R𝑛×𝑝 is a random Gaussian matrix with independent ¯ . rows drawn from Gaussian distribution N (𝜇, 𝚺) Then the estimator E{x0 , A, S, 𝑓 (·)} (introduced in Definition 3) succeeds in recovering x0 with probability approaching one (as 𝑝 and 𝑛 grow large), if and only if the estimator E{x0 , G, S, 𝑓 (·)} succeeds with probability approaching one. Theorem 14 shows that only the mean and covariance of the measurement vectors a𝑖 affect the required number of measurements for perfect recovery in (6.3). Although Theorem 14 holds for 𝑛 and 𝑝 growing to infinity, the result of our numerical simulations in Section 6.2, indicates the validity of universality for values of 𝑝 and 𝑛 ranging in the order of hundreds. Analysis of the Gaussian Estimator Theorem 14 shows the equivalence of the convex estimator E{x0 , A, S, 𝑓 (·)} and the Gaussian estimator E{x0 , G, S, 𝑓 (·)}. We can utilize the CGMT framework to 126 analyze the perfect recovery conditions for E{x0 , G, S, 𝑓 (·)}. Before doing so, we need the definition of the descent cone, Definition 4 [Descent Cone] The descent cone of a convex function 𝑓 (·) at point x0 is defined as D 𝑓 (x0 ) = Cone ({y : 𝑓 (y) ≤ 𝑓 (x0 )}) , (6.4) which is a convex cone. Here, Cone(S) denotes the conic-hull of the set S. Corollary 7 Consider the problem of recovering the vector x0 ∈ S, given the observations y = Gx0 ∈ R𝑛 , via the estimator E{x0 , G, S, 𝑓 (·)} introduced earlier. Assume that the rows of G are independent Gaussian random vectors with mean 𝜇 and covariance 𝚺¯ = MMT . Let 𝛿 := 𝑛/𝑝 and the set S and the penalty function 𝑓 (·) be convex. E{x0 , G, S, 𝑓 (·)} succeed in recovering x0 with probability approaching one (as 𝑝 and 𝑛 grow to infinity), if and only if     T √ √   w g ★   𝛿> 𝛿 =E max q  w∈(S−x )∩𝐷 (x ) 1 𝑓 0 T 𝜇) 2   1 T0 𝑝 1 + (w 𝑝  √ 𝑝 M w∈𝑆 𝑝−1    (6.5) where 𝑆 𝑝−1 is the 𝑝-dimensional unit sphere, and the expected value is over the ¯ Gaussian vector g ∼ N (0, 𝚺). ["Pseudo Gaussian Width"] When 𝜇 = 0 and 𝚺¯ = I, the expected value in (6.5) resembles the definition of the Gaussian width [139]. It has been shown that when the measurements are i.i.d. Gaussian, the square of the Gaussian width indicates the phase transition for linear inverse problems [6, 40, 156]. The Gaussian width has been computed for several interesting examples, such as sparse recovery, and lowrank matrix recovery. Using our universality result in Theorem 14, we can state that the square of the Gaussian width indicates the phase transition in the non-Gaussian setting as well. Numerical Results To validate the result of Theorem 14, we performed numerical simulations under various distributions for the measurement vectors. For our simulations in Figure 6.1, we use the estimator E{x0 , A, R 𝑝 , k · kℓ1 } to recover a 𝑘-sparse signal x0 under three 𝑛 . In each of the three plots, random ensembles for the measurement vectors {a𝑖 }𝑖=1 127 we computed the norm of the estimation error E{x0 , A, R 𝑝 , k · kℓ1 }, for different over sampling ratios 𝛿 = 𝑛/𝑝 and multiple sparsity factors 𝑠 = 𝑘/𝑝. We generated the 𝑛 for each figure, as follows, measurement vectors {a𝑖 }𝑖=1 • For each trial, we generate a random matrix M ∈ R 𝑝×𝑝 , with i.i.d. standard Gaussian random variables. 𝚺¯ = MMT will play the role of the covariance matrix of the measurement vectors. 𝑛 are drawn independently from the Gaussian distribu• For Figure 6.1a, {a𝑖 }𝑖=1 ¯ tion N (0, 𝚺). • For the measurement vectors of the Figure 6.1b, we first generate i.i.d centered bernouli vectors Ber(.8), and multiply each vector by M. • For the measurement vectors of the Figure 6.1c, we first generate i.i.d centered 𝜒1 vectors, and multiply each vector by M. The blue line in the figures shows the theoretical phase transition derived as a result of Corollary 7. It can be observed that the phase transition for all the three random schemes is the same, as predicted by Theorem 14. It also matches the theoretical phase transition derived from Corollary 7. Next, to illustrate the applicability and the implications of the results, we present some examples where our universality theorem can be applied. 6.3 Applications: Quadratic Measurements In this section we consider the problem of recovering a matrix from (so-called) quadratic measurements. The goal is to reconstruct a symmetric matrix X0 ∈ R 𝑝×𝑝 in a convex set S, given 𝑛 measurements of the form,   𝑦𝑖 = a𝑖T X0 a𝑖 = Tr X0 · (a𝑖 a𝑖T ) , 𝑖 = 1, . . . , 𝑛 . (6.6) Depending on the application, the matrix X0 may exhibit various structures. Similar to (6.3), we use the convex penalty function 𝑓 : R 𝑝×𝑝 → R, to enforce this structure via the following convex estimator, X̂ = arg min 𝑓 (X) X∈S subject to: a𝑖T Xa𝑖 = a𝑖T X0 a𝑖 , 𝑖 = 1, . . . , 𝑛 . (6.7) 128 0.7 0.7 Theory Theory 0.6 0.6 0.5 0.5 0.4 0.4 0.3 0.3 0.2 0.2 0.1 0.1 0.05 0.1 0.15 0.2 0.25 0.3 0.05 0.1 (a) 0.15 0.2 0.25 (b) 0.7 Theory 0.6 0.5 0.4 0.3 0.2 0.1 0.05 0.1 0.15 0.2 0.25 0.3 (c) Figure 6.1: Phase transition regimes for the estimator E{x0 , A, R 𝑝 , k · kℓ1 }, in terms of the oversampling ratio 𝛿 = 𝑛𝑝 and 𝑠 = kx𝑝0 k0 , for the cases of (a) Gaussian measurements and (b) Bernoulli measurements and (c) 𝜒2 measurements. The blue lines indicate the theoretical estimate for the phase transition derived from Corollary 7. In the simulations we used vectors of size 𝑝 = 256. The data is averaged over 10 independent realization of the measurements. 0.3 129 Note that the measurements in (6.6) are linear with respect to the matrix X0 , yet quadratic with respect to the measurement vectors a𝑖 . We can define x̃0 := 2 2 Vec(X0 ) ∈ R 𝑝 and ã𝑖 := Vec(a𝑖 a𝑖T ) ∈ R 𝑝 , such that the measurements take the familiar form, 𝑦𝑖 = ã𝑖T x̃0 . In order to apply the result of Theorem 14, one should 𝑛 satisfy Assumption 6. check if the vectors {ã𝑖 }𝑖=1 𝑛 satisfy the following conditions, then It can be shown that if the vectors {a𝑖 }𝑖=1 𝑛 . Assumption 6 holds true for {ã𝑖 = Vec(a𝑖 a𝑖T )}𝑖=1 𝑛 satisfy Assumption 3, if Assumption 8 We say vectors {a𝑖 }𝑖=1 1. a𝑖 ’s are drawn independently from a sub-Gaussian distribution. 2. For each 𝑖, the entries of a𝑖 are independent, zero-mean and unit-variance. In particular, this assumption is valid when {a𝑖 }’s have i.i.d. standard normal entries. Therefore, when Assumption 8 holds, we can apply Theorem 14 to show that the required number of measurements for perfect recovery in (6.7) is equal to the required number of measurements for the success of the following estimator, X̂ = arg min 𝑓 (X) X∈S subject to: Tr ((H𝑖 + I)X) = Tr ((H𝑖 + I)X0 ) , 𝑖 = 1, . . . , 𝑛 , (6.8) where I is the 𝑝 × 𝑝 identity matrix and H𝑖 ’s are independent Gaussian Wigner matrices (defined in Section 6.1). Corollary 8 presents a formal statement. Corollary 8 Consider the problem of recovering the matrix X0 ∈ S ⊆ R 𝑝×𝑝 , from 𝑛 quadratic measurements of the form (6.6), using the estimator (6.7). Let S and 𝑓 (·) be convex set and function satisying Assumption 7. Assume, 𝑛 satisfy Assumption 8, and, • The measurement vectors {a𝑖 }𝑖=1 𝑛 is a set of independent Gaussian Wigner matrices. • {H𝑖 ∈ R 𝑝×𝑝 }𝑖=1 Then, as 𝑛 and 𝑝 grow to infinity at a fixed rate 𝑛 = 𝜃 ( 𝑝), the estimator (6.7) perfectly recovers X0 with probability approaching one if and only if the estimator (6.8) perfectly recovers X0 with probability approaching one. 130 Therefore, in order to find the phase transition, it is sufficient to analyze the equivalent optimization (6.8) which is possible via the CGMT framework. Proceeding onward, we exploit the CGMT framework along with Corollary 7 to find the required number of measurements for the recovery of X0 in two specific applications. Low-rank Matrix Recovery Assume the unknown matrix X0  0 has rank 𝑟, where 𝑟 is a constant ( i.e., 𝑟 does not grow with problem dimensions 𝑛, 𝑝.) Such matrices appear in many applications such as traffic data monitoring, array signal processing and phase retrieval. The nuclear norm, || · ||★, is often used as the convex surrogate for low-rank matrix recovery [135]. Hence, we are interested in analyzing the optimization (6.7), with the choice of 𝑓 (X) = kXk★, where the optimization is over the set of PSD matrices. Note that Tr(·) = || · ||★ within this set, which satisfies Assumption 7. According to Corollary 8, the perfect recovery in (6.7) is equivalent to perfect recovery in (6.8), where the same choice of 𝑓 (X) = Tr(X). The analysis of the later through CGMT yields the following corollary. Corollary 9 Consider the optimization program (6.7), where the matrix X0  0 has 𝑛 rank 𝑟, 𝑓 (X) = Tr(X), the set S is the PSD cone and the measurement vectors {a𝑖 }𝑖=1 satisfy Assumption 8. Assume 𝑝, 𝑛 → ∞ at the proportional rate 𝛿 := 𝑛𝑝 ∈ (0, +∞). The estimator perfectly recovers X0 if 𝛿 > 3𝑟. Corollary 9 indicates that 3𝑟 𝑝 measurements is needed to perfectly recover a rank-𝑟 PSD matrix X0 , from quadratic measurements. Although, the error of estimation gets extremely small, much before the threshold 𝑛 = 3𝑝𝑟. To the extent of our knowledge, this is the first work that precisely computes the phase transition of lowrank matrix recovery from quadratic measurements. Figure6.2 depicts the result of numerical simulations. For different values of 𝑟 and 𝛿, the Frobenius norm of the error of the estimators (6.7) and (6.8) has been computed, which shows the same phase transition in both cases. Phase Transition of PhaseLift in Phase Retrieval An important application for the result of Corollary 9, is when the underlying matrix X0 is of rank 1. This appears in the problem of phase retrieval, where X0 = x0 xT0 is the lifted version of the signal. The optimization program (6.7) with 𝑓 (X) = Tr(X) in this case, is known as PhaseLift [36]. Corollary 9 states that the phase transition of 131 12 12 11 11 10 10 9 9 8 8 7 7 6 6 5 5 4 4 3 3 2 2 1 2 3 4 5 6 (a) 7 8 9 10 1 2 3 4 5 6 7 8 9 (b) Figure 6.2: Phase transition regimes for both estimators 6.7 and (6.8), with 𝑓 (X) = Tr(X), in terms of the oversampling ratio 𝛿 = 𝑛𝑝 and 𝑟 = Rank(X0 ), for the cases of (a) estimator (6.7) with quadratic measurements and (b) estimator (6.8) with Gaussian measurements. In the simulations we used matrices of size 𝑝 = 40. The data is averaged over 20 independent realization of the measurements. the PhaseLift algorithm happens at 𝛿★ = 3, i.e., 𝑛 > 3𝑝 measurements is needed for the perfect signal reconstruction in PhaseLift. We should emphasize the significance of this result as establishing the exact phase transition of the PhaseLift algorithm was long an open problem. Sparse Matrix Recovery Let X0  0 represent the covariance matrix of a set of random variables. In certain applications, the covariance matrix has many near-zero entries as the correlations are small for many pairs of random variables. Such matrices arise in applications in spectrum estimation, biology and finance [44, 67]. We are interested in analyzing estimator (6.7), where 𝑓 (X) = kXkℓ1 promotes the sparsity in the optimization. As k · kℓ1 satisfies Assumption 7, applying the result of Corollary 8, the perfect recovery in (6.7) is equivalent to the perfect recovery in the estimator (6.8), with the same penalty function. Analyzing the optimization (6.8) via CGMT leads to the following result: Corollary 10 Let 𝛿 := 𝑝𝑛2 , 𝑠 := kX𝑝02k0 . As 𝑝 → ∞, the optimization program (6.7), with 𝑓 (X) = kXkℓ1 can successfully recover the signal iff 𝛿 > 𝛿★, where 𝛿★ is the unique solution to the following nonlinear equation,      −1 2𝑥 − 𝑠 −1 2𝑥 − 𝑠 𝑥·𝑄 = (1 − 𝑠)𝜙 𝑄 , (6.9) 2 − 2𝑠 2 − 2𝑠 10 132 Theory 0.9 Theory 0.9 0.8 0.8 0.7 0.7 0.6 0.6 0.5 0.5 0.4 0.4 0.3 0.3 0.2 0.2 0.1 0.2 0.3 0.4 0.5 0.6 0.1 0.2 0.3 0.4 0.5 (b) (a) Figure 6.3: Phase transition regimes for both estimators (6.7) and (6.8), with 𝑓 (X) = kXkℓ1 , in terms of the oversampling ratio 𝛿 = 𝑝𝑛2 and 𝑠 = kX𝑝02k0 , for the cases of (a) estimator (6.7) with quadratic measurements and (b) estimator (6.8) with Gaussian measurements. The blue lines indicate the theoretical estimate for the phase transition derived from equation (6.9). In the simulations we used matrices of size 𝑝 = 40. The data is averaged over 20 independent realization of the measurements. Model 𝑘 sparse matrix Rank-𝑟 PSD matrix S&L (𝑘, 𝑟) matrix Penalty function 𝑓 (·) k · kℓ1 Tr(·) Tr(·) + 𝜆k · k1 No. of required measurements 𝑝 2 𝛿★ defined in (6.9) 3pr O (min(𝑘 2 , 𝑟 𝑝)) Table 6.1: Summary of the parameters that are discussed in this section. The last row is for a 𝑝 × 𝑝 rank-𝑟 matrix whose smallest sub-matrix with non-zero entries is 𝑘 by 𝑘. The third column shows the number of required quadratic measurements for perfect recovery. √ where 𝜙(𝑥) = exp(−𝑥 2 /2)/ 2𝜋 and 𝑄 −1 (·) is inverse of the Q-function. Figure 6.3b compares the empirical result with the theoretical phase transition derived from Corollary 10 Each plot shows the norm of the error with respect to the sparsity of the matrix X0 and the ratio 𝛿 = 𝑝𝑛2 . A comparison between the two plots indicates that the phase transitions of the two estimators (6.7) and (6.8) with 𝑓 (X) = kXkℓ1 match. 6.4 Simultaneously Sparse and Low-rank Matrices Another interesting example is where the unknown matrix X0  0 is simultaneously sparse and low rank. To recover X0 , we would like to simultaneously minimize the 0.6 133 penalty functions 𝑓 (1) (X) = kXkℓ1 and 𝑓 (2) (X) = kXk★, for all feasible matrices X ∈ S that align our measurements in (6.6). Here, each function 𝑓 (𝑖) (·) enforces one of the structures on X. So, a natural choice for the regularizer function in (6.7) would be 𝑓 (X) = 𝑓 (1) (X) + 𝜆 𝑓 (2) (X), where 𝜆 is a regularizing parameter. Oymak et al [128] studied phase transition for perfect recovery of simultaneously structured matrices. Their results are based on Gordon’s comparison lemma which is only applicable to the cases of linear Gaussian measurements. We can use the result of Corollary 8 to extend their result to settings with quadratic measurements, as the phase transition regime is equivalent in both cases. Let X0 ∈ R 𝑝×𝑝 be a rank-𝑟 PSD matrix. Also assume that the largest sub-matrix in X0 that contains all non-zero entries is 𝑘 by 𝑘. If we choose 𝑓 (X) = kXkℓ1 + 𝜆Tr(X), they show that O (min(𝑘 2 , 𝑟 𝑝)) measurements is required for perfect recovery. Conclusion We have investigated an estimation problem under linear observations. We aimed to characterize the minimum number of observations that are needed for perfect recovery of the unknown model. Our main result indicated that this phase transition, only depends on the first two statistics of the measurement vector. Therefore, it remains unchanged as we replace these vectors with the Gaussian one, with the same mean vector and covariance matrix. The later can be analyzed through existing frameworks such as CGMT. As one of the applications of this universality, we investigated the case of matrix recovery via the so called quadratic measurements, and derived the minimum number of observations required for the recovery of a structured matrix. Due to the space constraint, we moved the discussions regarding the case of simultaneously structured matrices to the appendix. Table 6.1, summarizes these results for the cases of three structures. Proof of Theorem 14 Consider the following optimization Φ1 = min 𝑓 (x) , Ax0 =Ax (6.10) Without loss of generality, assume that 𝑓 (0) = 0. We change the variable to w = x − x0 , which gives the following Φ1 = min 𝑓 (w + x0 ) , Aw=0 (6.11) This optimization has perfect recovery, iff ŵ = 0, or equivalently iff Φ1 = 0. We would like to show that if Φ1 = 0 with probability converging to 1, then the same 134 holds if we replace the measurements vectors a𝑖 , with another set of measurement vectors with the same mean and covariance. We rewrite this optimization in the form of this min-max optimization, 𝜆 kAwk 2 + 𝑓 (w + x0 ) w 2 𝜆>0 𝜆 1 = sup min min kAwk 2 + 𝑓 (w + x0 ) + kwk 2 w 2 2𝜇 𝜆>0 𝜇>0 1 1 1 kwk 2 = sup 𝜆 · min min kAwk 2 + 𝑓 (w + x0 ) + w 2 𝜇>0 𝜆 2𝜆𝜇 𝜆>0 Φ1 = sup min (6.12) Informally, we first show that for fixed values of 𝜆 and 𝜇, the values of last minimization remains unchanged as we change the random measurement vectors inside it (as 𝑝 and 𝑛 grow to infinity). Next, we use Lemma 18 (See [165] Section A.4 and B.5) to switch the min-max over 𝜇 and 𝜆, with the limit over 𝑝 and 𝑛. By fixing the values of 𝜆 and 𝜇, from now on, we redefine the function 𝑓 (·) to 1 be 𝜆1 𝑓 (w + x0 ) + 2𝜆𝜇 kwk 2 , which is strongly convex. Note that we would like the following assumptions holds for these two set of random measurement vectors. Assumption 1: Assume A = [a1 , . . . , a𝑛 ] T ∈ R𝑛×𝑝 and B = [b1 , . . . , b𝑛 ] T ∈ R𝑛×𝑝 are two random matrices, such that e = E [a𝑖 ] = E [b𝑖 ] ∀𝑖     𝚺 = E a𝑖 a𝑖T = E b𝑖 b𝑖T ∀𝑖 kek 2 = 0, 𝑝→∞ 𝑝 2 lim (6.13) Besides, there exists 𝜏 > 0 such that for any matrix M ∈ R 𝑝×𝑝 such that kMk2 ≤ 𝜅, there exists some 𝑐 that only depends on 𝜅 that   1 T −𝜏 Var a Ma and, 𝑖 ≤ 𝑐· 𝑝 𝑖 2 𝑝   1 T −𝜏 Var b Mb . (6.14) 𝑖 ≤ 𝑐· 𝑝 𝑖 𝑝2 Now we want to investigate equivalence of the following two optimizations. Let A = [a1 , . . . , a𝑛 ] and B = [b1 , . . . , b𝑛 ] be 𝑛 by 𝑝 measurement matrices and 𝑛 2 1 Õ 𝑧𝑖 − wT a𝑖 + 𝑓 (w + x0 ) , w 2𝑛 𝑖=1 𝑛  2 Õ 1 ΦA = min 𝑧𝑖 − wT b𝑖 + 𝑓 (w + x0 ) . w 2𝑛 𝑖=1 ΦB = min (6.15) 135 Theorem 15 Consider the optimizations in (6.15). If lim |E [ΦB − ΦA ]| = 0 , 𝑛,𝑝→∞ (6.16) and if for constants 𝐶 and 𝛿 > 0, P Pr (|ΦA − 𝐶 | > 𝛿) → − 0, (6.17) as 𝑛, 𝑝 → ∞. Then, P Pr (|ΦB − 𝐶 | > 3𝛿) → − 0, (6.18) Proof 7 We first define the function 𝑔 : R → R as follows.    0       (|𝑥| − 1) 2  𝑔(𝑥) =   2 − (|𝑥| − 3) 2      2  if |𝑥| ≤ 1, if 1 < |𝑥| ≤ 2, if 2 < |𝑥| ≤ 3, if |𝑥| > 3 . (6.19) Note that 𝑔(.) is continuously differentiable with its first derivative bounded by 2. Now,        ΦB − 𝐶 1 ΦB − 𝐶 Pr {|ΦB − 𝐶 | > 3𝛿} = Pr 𝑔 >2 ≤ E 𝑔 𝛿 2 𝛿         ΦA − 𝐶 ΦA − 𝐶 1 1 ΦB − 𝐶 ≤ E 𝑔 + E 𝑔 −𝑔 2 𝛿 2 𝛿 𝛿    1 ΦA − 𝐶 ΦB − 𝐶 0 ≤ Pr {|ΦA − 𝐶 | > 𝛿} + E 𝑔 (𝜁) · − 2 𝛿 𝛿 𝑛,𝑝→∞ 1 ≤ Pr {|ΦA − 𝐶 | > 𝛿} + |E [ΦA − ΦB ]| −−−−−−→ 0 (6.20) 𝛿 Theorem 16 Consider the optimizations in (6.15). If A, B and 𝑓 (.) satisfy Assumption 6 and 7, respectively, then lim |E [ΦA ] − E [ΦB ]| → 0 . 𝑛,𝑝→∞ (6.21) Proof 8 For 𝑘 = 0, . . . , 𝑛, we define 𝑘 𝑛  2 1 Õ 2 1 Õ T T Φ 𝑘 := min 𝑧𝑖 − a𝑖 w + 𝑧𝑖 − b𝑖 w + 𝑓 (w + x0 ) . w 2𝑛 2𝑛 𝑖=𝑘+1 𝑖=1 (6.22) 136 We have |E [ΦA − ΦB ]| = |E [Φ𝑛 − Φ0 ]| ≤ 𝑛 Õ |E [Φ 𝑘 − Φ 𝑘−1 ]| . (6.23) 𝑘=1 Now it suffices to show that there exists a constant 𝑐, such that for any 𝑘, |E [Φ 𝑘 − Φ 𝑘−1 ]| ≤ 𝑐 𝑛−(1+𝜏/2) , (6.24) for some positive constant 𝜏. Since, then combining (6.24) and (6.23) yields, |E [ΦA − ΦB ]| ≤ 𝑛 Õ |E [Φ 𝑘 − Φ 𝑘−1 ]| ≤ 𝑐 𝑛−𝜏/2 → 0 . (6.25) 𝑘=1 Let M 𝑘 = [a1 , . . . , a 𝑘−1 , b 𝑘+1 , . . . , b𝑛 ] T ∈ R (𝑛−1)×𝑝 , and, z 𝑘 = [𝑧 1 , . . . , 𝑧 𝑘−1 , 𝑧 𝑘+1 , . . . , 𝑧 𝑛 ] T ∈ R𝑛−1 . (6.26) This helps us rewrite Φ 𝑘 and Φ 𝑘−1 as 2 1 1  2 T Φ 𝑘 = min kz 𝑘 − M 𝑘 wk + 𝑧 𝑘 − a 𝑘 w + 𝑓 (w + x0 ) , w 2𝑛 2𝑛 2 1 1  Φ 𝑘−1 = min kz 𝑘 − M 𝑘 wk 2 + 𝑧 𝑘 − bT𝑘 w + 𝑓 (w + x0 ) . w 2𝑛 2𝑛 (6.27) As of this point, we fix 𝑘 and drop the subscript 𝑘 from 𝑧 𝑘 , z 𝑘 , M 𝑘 , a 𝑘 and b 𝑘 for simplicity. The expectation in (6.24) is over the randomness in 𝑧, z, M, a and b, which can be written as    |E [Φ 𝑘 − Φ 𝑘−1 ]| = E{M,z} E{𝑧,a,b} Φ 𝑘 − Φ 𝑘−1 {M, z}   [Φ 𝑘 − Φ 𝑘−1 ] . ≤ E{M,z} E {𝑧,a,b} {M,z} (6.28) We first fix M and z, and bound the inner expectation in (6.28). Now let, 2 1 1  kz − Mwk 2 + 𝑧 − aT w + 𝑓 (w + x0 ) , 2𝑛 2𝑛 Φ(a, 𝑧) = min 𝜙(a, w) , 𝜙(a, 𝑧, w) = w Φ̄ = Φ(0, 0), and, w̄ = arg min 𝜙(0, 0, w) . (6.29) 137 With these new definitions, we have Φ 𝑘 = Φ(a, 𝑧) and Φ 𝑘−1 = Φ(b, 𝑧) and thus, E{𝑧,a,b} [Φ 𝑘 − Φ 𝑘−1 ] = E{𝑧,a,b} [Φ(a, 𝑧) − Φ(b, 𝑧)] 2     𝜎 2 + k w̄k 𝑛  ≤ E{𝑧,a} Φ(a, 𝑧) − Φ̄ − T 𝛀b])  2𝑛(1 + E[b     2     𝜎 2 + k w̄k 𝑛   + E{𝑧,b} Φ(b, 𝑧) − Φ̄ − 2𝑛(1 + E[bT 𝛀b])     (6.30) So since E[bT 𝛀b] = E[aT 𝛀a], it remains to show that for positive constants 𝑐 and 𝜏, 2     𝜎 2 + k w̄k 𝑛  ≤ 𝑐 𝑛−(1+𝜏/2) , and,  E{𝑧,a} Φ(a, 𝑧) − Φ̄ − T 𝛀a])  2𝑛(1 + E[a     2     𝜎 2 + k w̄k 𝑛  ≤ 𝑐 𝑛−(1+𝜏/2) . E{𝑧,b} Φ(b, 𝑧) − Φ̄ − T 2𝑛(1 + E[b 𝛀b])     (6.31) w̄+x0 ) We show the later, and the proof of the first is similar. Define v = 𝜕 𝑓 (𝜕w and 𝜕 2 𝑓 ( w̄+x0 ) V = 𝜕w2 and 2 1  1 kz − Mwk 2 + 𝑧 − bT w + 𝑓 ( w̄ + x0 ) + vT (w − w̄) 2𝑛 2𝑛 1 + (w − w̄) T V( w − w̄) , 2 Ψ(b, 𝑧) = min 𝜓(b, 𝑧, w) , and, w̃ = arg min 𝜓(b, 𝑧, w) . (6.32) 𝜓(b, 𝑧, w) = w Note that by writing the optimality conditions, it is easy to show that Ψ(0, 0) = Φ(0, 0) = Φ̄. Thus, 2     𝜎 2 + k w̄k 𝑛   ≤ E{𝑧,b} [|Φ(b, 𝑧) − Ψ(b, 𝑧)|] E{𝑧,b} Φ(b, 𝑧) − Φ̄ −  T 2𝑛(1 + E[b 𝛀b])     2     𝜎 2 + k w̄k 𝑛  .  + E{𝑧,b} Ψ(b, 𝑧) − Ψ(0, 0) − (6.33) T 𝛀b])  2𝑛(1 + E[b     So we have to bound the two terms on the right hand side of (6.33). We start with bounding E{𝑧,b} [|Φ(b, 𝑧) − Ψ(b, 𝑧)|]. Note that for any w we have |𝜓(b, 𝑧, w) − 𝜙(b, 𝑧, w)| ≤ 𝐶𝑓 kw − w̄k33 . 𝑛 (6.34) 138 Besides, due to strong convexity of 𝑓¯(.) we have, |𝜓(b, 𝑧, w) − Ψ(b, 𝑧)| ≥ 𝜖 kw − w̃k22 . 𝑛 (6.35) We have two cases. First if k w̃ − w̄k3 ≤ 9 𝐶𝜖 𝑓 . Consider the set S = {w : kw − w̃k3 = k w̃ − w̄k3 }. For any w in the set S we have  𝐶𝑓  𝜙(b, 𝑧, w) − 𝜙(b, 𝑧, w̃) ≥ 𝜓(b, 𝑧, w) − 𝜓(b, 𝑧, w̃) − kw − w̄k33 + k w̃ − w̄k33 𝑛   𝐶 𝜖 𝑓 2 3 3 kw − w̄k3 + k w̃ − w̄k3 ≥ kw − w̃k2 − 𝑛 𝑛  𝐶𝑓  𝜖 4 kw − w̄k33 + 5 k w̃ − w̄k33 ≥ kw − w̃k32 − 𝑛 𝑛  9 𝐶𝑓 𝜖 2 (6.36) k w̃ − w̄k3 − k w̃ − w̄k3 ≥ 0 . = 𝑛 9 𝐶𝑓 This means that the optimal value of 𝜙(b, 𝑧, w) lies within S. arg min 𝜙(b, 𝑧, w), Now if w𝜙 =

Ψ(b, 𝑧) − Φ(b, 𝑧) = 𝜓(b, 𝑧, w̃) − 𝜓(b, 𝑧, w𝜙 ) + 𝜓(b, 𝑧, w𝜙 ) − 𝜙(b, 𝑧, w𝜙 )
𝐶𝑓 ≤ 𝜓(b, 𝑧, w𝜙 ) − 𝜙(b, 𝑧, w𝜙 ) ≤ kw𝜙 − w̄k33 𝑛  8 𝐶𝑓 4 𝐶𝑓  kw𝜙 − w̃k33 + k w̃ − w̄k33 ≤ k w̃ − w̄k33 . (6.37) ≤ 𝑛 𝑛 And,
Φ(b, 𝑧) − Ψ(b, 𝑧) = 𝜙(b, 𝑧, w𝜙 ) − 𝜙(b, 𝑧, w̃) + (𝜙(b, 𝑧, w̃) − 𝜓(b, 𝑧, w̃)) 𝐶𝑓 ≤ (𝜙(b, 𝑧, w̃) − 𝜓(b, 𝑧, w̃)) ≤ (6.38) k w̃ − w̄k33 . 𝑛 Thus, (6.37) and (6.37) implies that |Φ(b, 𝑧) − Ψ(b, 𝑧)| ≤ 8 𝐶𝑓 k w̃ − w̄k33 . 𝑛 (6.39) Case 2 if k w̃ − w̄k3 ≥ 9 𝐶𝜖 𝑓 .
Φ(b, 𝑧) − Ψ(b, 𝑧) = 𝜙(b, 𝑧, w𝜙 ) − 𝜙(b, 𝑧, w̄) + (𝜙(b, 𝑧, w̄) − 𝜙(0, 0, w̄)) + (𝜓(0, 0, w̄) − 𝜓(b, 𝑧, w̄)) + (𝜓(b, 𝑧, w̄) − 𝜓(b, 𝑧, w̃)) 2 1  ≤ 𝜓(b, 𝑧, w̄) − 𝜓(b, 𝑧, w̃) ≤ 𝑧 − bT w̄ . (6.40) 2𝑛 139 Ψ(b, 𝑧) − Φ(b, 𝑧) ≤ (𝜓(b, 𝑧, w̃) − 𝜓(b, 𝑧, w̄)) + (𝜓(b, 𝑧, w̄) − 𝜓(0, 0, w̄)) + (𝜙(0, 0, w̄) − 𝜙(0, 0, w̄)) 2 1  ≤ 𝑧 − bT w̄ . 2𝑛 (6.41) So finally, |Ψ(b, 𝑧) − Φ(b, 𝑧)| ≤ 2 1  𝑧 − bT w̄ . 2𝑛 (6.42) So by combining the two cases, we get     2 8 𝐶𝑓 1 3 T k w̃ − w̄k3 + 1 k w̃−w̄k3 > 9 𝐶𝜖 𝑧 − b w̄ . |Φ(b, 𝑧) − Ψ(b, 𝑧)| ≤ 1 k w̃−w̄k3 ≤ 9 𝐶𝜖 𝑓 𝑓 𝑛 2𝑛 (6.43) Therefore, E [|Φ(b, 𝑧) − Ψ(b, 𝑧)|]  ≤ E 1 k w̃−w̄k3 ≤ 9 𝐶𝜖       2  8 𝐶𝑓 1 3 T k w̃ − w̄k3 + E 1 k w̃−w̄k3 > 9 𝐶𝜖 𝑧 − b w̄ 𝑓 𝑓 𝑛 2𝑛 s    
4 8𝐶 𝑓 1 𝜖 3 E k w̃ − w̄k3 + ≤ Pr k w̃ − w̄k3 ≥ E[ 𝑧 − bT w̄ ] 𝑛 2𝑛 9 𝐶𝑓 v u  t   1 E k w̃ − w̄k33
8𝐶 𝑓  3 T w̄ 4 ] E k w̃ − w̄k3 + ≤ E[ 𝑧 − b 𝑛 2𝑛 ( 9𝐶𝜖 𝑓 ) 3 𝐶 𝑛5/4 On the other hand, it is easy to see that ≤ Ψ(b, 𝑧) − Ψ(0, 0) = (6.44) (𝑧 − bT w̄) 2 , 2𝑛(1 + bT 𝛀−1 b) where 𝛀 = V + MT M. Note that 2     𝜎 2 + k w̄k 𝑛  E Ψ(b, 𝑧) − Ψ(0, 0) − 2𝑛(1 + E[bT 𝛀b])     2   k w̄k  (𝑧 − bT w̄) 2  𝜎2 + 𝑛   = E −  T T −1 2𝑛(1 + E[b 𝛀b]) 2𝑛(1 + b 𝛀 b)      1  T 2 T T E (𝑧 − b w̄) b 𝛀b − E[b 𝛀b] ≤ 2𝑛 r   h
2i 1 T 4 T T E (𝑧 − b w̄) E b 𝛀b − E[b 𝛀b] ≤ 2𝑛 𝐶 ≤ 1+𝜏/2 . 𝑛 (6.45) (6.46) 140 Now putting (6.44) and (6.46) in (6.33), results in 2     𝜎 2 + k w̄k 𝑛  E{𝑧,b} Φ(b, 𝑧) − Φ̄ − 2𝑛(1 + E[bT 𝛀b])     3/2 q    27𝐶 𝑓
8𝐶 𝑓  3 E[ 𝑧 − bT w̄ 4 ] ≤ E k w̃ − w̄k33 + E k w̃ − w̄k 3 3/2 𝑛 2𝑛𝜖 r   h
2i 1 (6.47) E (𝑧 − bT w̄) 4 E bT 𝛀b − E[bT 𝛀b] + 2𝑛 𝑐 𝑛−(1+𝜏/2) (6.48)     It remains to bound E (𝑧 − bT w̄) 4 and E k w̃ − w̄k33 . For the first one, let 1𝑝 e = E[b] and b̃ = b − 1𝑝 e. Then,   E (𝑧 − bT w̄) 4 = E[𝑧 4 ] + 6E[𝑧2 ] E[(bT w̄) 2 ] + E[(bT w̄) 4 ] = E[𝑧 4 ] + 6E[𝑧 2 ] (E[( b̃T w̄) 2 ] + (eT w̄) 2 ) + E[( b̃T w̄) 4 ] 𝑝 + 6E[( b̃T w̄) 2 ] (eT w̄) 2 + (eT w̄) 4 ≤ 𝐶1 + 𝐶2 k w̄k 2 + 𝐶3 k w̄k 4 . (6.49) On the other hand, let 𝛀−1 = [𝜔1 . . . , 𝜔 𝑝 ] T . Since 𝛀−1  1/𝜖, # " 3 i h T   (𝑧 − b w̄) −1 T −1 3 𝛀 b (𝑧 − b w̄) 𝛀 b E k w̃ − w̄k33 = E ≤ E 3 (1 + bT 𝛀−1 b) 3 i 4 h i h 3 3 ≤ 4 E (𝑧 − bT w̄) 𝛀−1 b̃ 3 + 3 E (𝑧 − bT w̄) 𝛀−1 e 3 𝑝 r q  i 4   h  6 −1 3 T 6 −1 ≤ 4 E (𝑧 − b w̄) E 𝛀 b̃ 3 + 3 𝛀 e 3 E (𝑧 − bT w̄) 6 𝑝 v u " #2 t q  Õ    4 3 ≤ 4 E (𝑧 − bT w̄) 6 E |𝜔T𝑘 b̃| 3 + 3 𝛀−1 e 2 E (𝑧 − bT w̄) 6 𝑝 𝑘 4kek 3 q   𝐶 ≤ ( 3 + 2 32 ) E (𝑧 − bT w̄) 6 (6.50) 𝑝𝜖 𝜖 𝑝 which concludes the proof. Theorem 17 let WA and WB bt the optimal solutions to (6.15). If for any function 𝑓 (.), that satisfies our conditions, ΦA − ΦB → 0 , (6.51) 141 then, 1 1 kWA k𝐹2 − 2 kWB k𝐹2 → 0 . 2 𝑝 𝑝 (6.52) Proof 9 Assume that 𝑝12 kWA k𝐹2 and 𝑝12 kWB k𝐹2 converge to difference values of 𝐶A and 𝐶B . Choose 𝐶 = (𝐶B + 𝐶A )/2 and consider the following optimization, 𝑛 1 Õ (𝑧𝑖 − Tr(A𝑖 · W)) 2 + 𝑓 (W) , Φ̄A = min 1 2 ≤𝐶 2𝑛 kWk 2 𝐹 𝑖=1 𝑝 W∈H 𝑝 𝑛 1 Õ (𝑧𝑖 − Tr(B𝑖 · W)) 2 + 𝑓 (W) . Φ̄B = min 1 2 ≤𝐶 2𝑛 kWk 2 𝐹 𝑖=1 (6.53) 𝑝 W∈H 𝑝 We show that the two should converge to the same value, which is a contradiction since 𝑓 (.) is strongly convex and one should converge to ΦA and the other should be larger that ΦB . Using min-max theorem, they can be rewritten as 𝑛 𝜆 1 Õ (𝑧𝑖 − Tr(A𝑖 · W)) 2 + 𝑓 (W) + 2 kWk𝐹2 , Φ̄A = sup − 𝜆 𝐶 + min𝑝 W∈H 2𝑛 𝑝 𝜆>0 𝑖=1 𝑛 1 Õ 𝜆 (𝑧𝑖 − Tr(B𝑖 · W)) 2 + 𝑓 (W) + 2 kWk𝐹2 . (6.54) Φ̄B = sup − 𝜆 𝐶 + min𝑝 W∈H 2𝑛 𝑝 𝜆>0 𝑖=1 Due to the assumption of the theorem, the two inside converge to the same value for any fixed 𝜆. So the concave version of Lemma 18 shows that Φ̄A and Φ̄B also converge to the same value which is a contradiction. Lemma 18 Consider a series of convex functions 𝑓 𝑝 : R>0 → R that converges point-wise to the function 𝑓 : R>0 → R. Besides, there exists 𝑀 > 0 such that for any 𝑥 > 𝑀, we have 𝑓 (𝑥) > inf 𝑠>0 𝑓 (𝑠). Then 𝑓 (.) is also convex and 𝑝 inf 𝑠>0 𝑓 𝑝 (𝑠) → − inf 𝑠>0 𝑓 (𝑠). Lemma 19 Let w̄ be the optimal solution to the optimization min w 1 kz − Awk 2 + 𝑓 (w + x0 ) , 2 (6.55) where 𝑓 (.) is strongly convex with constant 𝜖. Then k w̄k ≤ 2 (kAT zk + k∇ 𝑓 (x0 )k) 𝜖 (6.56) 142 Proof 10 let 𝜙(A, w) = 1 kz − Awk 2 + 𝑓 (w + x0 ) . 2 (6.57) We have   𝜖 0 > 𝜙(A, w̄) − 𝜙(A, 0) ≥ w̄T −AT z + ∇ 𝑓 (x0 ) + k w̄k 2 . 2 Therefore,     𝜖 k w̄k 2 ≤ w̄T −AT z + ∇ 𝑓 (x0 ) ≤ k w̄k kAT zk + k∇ 𝑓 (x0 )k , 2 (6.58) which concludes the proof. Now let w̄ be the optimizer of 𝜙(A, w) and E[A] = 1eT . Due to optimality we have, 0 = AT (Aw̄ − z) + ∇ 𝑓 (x0 + w̄) (6.59) Lemma 20 Let w̄ be the optimal solution to the optimization min w 1 kz − Awk 2 + 𝑓 (w + x0 ) , 2 (6.60) where 𝑓 (.) is strongly convex with constant 𝜖 and A ∈ R𝑛×𝑝 is a random value with E[A] = 1e𝑡 and B = A − 1e𝑡 . Then k w̄k ≤ 2 (kAT zk + k∇ 𝑓 (x0 )k) 𝜖 (6.61) Proof 11 We have, 1 kz − Bw̄ − 1eT w̄k 2 + 𝑓 ( w̄ + x0 ) 2 𝑛 ≥ (eT w̄) 2 + (eT w̄) · 1T (Bw̄ − z) 2 𝜙(A, 0) ≥ 𝜙(A, w̄) = (6.62) Therefore, 2 2 (eT w̄) 2 + (eT w̄) · 1T (Bw̄ − z) − 𝜙(A, 0) ≤ 0 𝑛 𝑝 (6.63) This results in 2 T 2 2 2 2 1 (Bw̄ − z) + 𝜙(A, 0) ≤ |1T z| + k w̄k · kBT 1k + kzk 2 + 𝑓 (x0 ) 𝑛 𝑛 𝑛 𝑛 𝑛  2 T 4  T 2 ≤ |1 z| + kA zk + k∇ 𝑓 (x0 )k · kBT 1k + kzk 2 + 𝑓 (x0 ) (6.64) 𝑛 𝑛𝜖 𝑛 |eT w̄| ≤ 143 Some notes on the Corollary 9 Out goal is to analyze the following optimization and see under what conditions it will succeed.  1 𝑝 Tr (A𝑖 Tr(X − X0 ) , min  √  √ 2+I)X0 = 𝑝1 Tr (A𝑖 2+I)X , ∀𝑖 s.t. X  0 . (6.65) Here, we initially assume that X0 = x0 xT0 is a rank-one matrix where 1𝑝 kx0 k 2 = 1. This optimization succeeds in recovering X0 , if and only if the optimal value of objective function is zero (and it fails if the optimal value is negative). Now, assuming an iid standard Gaussian distribution for the entries of the matrices A𝑖 , we can use the proof of Theorem 1 and CGMT framework, and get the following optimization. p √ p 1 𝛼 + 𝛾(𝑡 + 1) + −1 𝛽 𝛿 2𝛼2 + 𝑡 2 − 𝛽,𝜏≥0 2𝜏 2𝛼𝜏 min max 𝛼≥0 𝑡∈R 𝛾∈R 1 − 2𝛼𝜏   2 𝛼𝜏𝛽 1 T , x0 x0 + √ H + 𝛼𝜏(𝛾 − 1)I 𝑝 𝑝 + 𝐹 (6.66) where H is a Wigner random matrix, and (·)+ is the positive definite part of a matrix. We will have perfect recovery if and only if this optimization is non-negative. First step of understanding this optimization, would be to analyze the high dimensional behavior of the positive part of the matrix 1 𝛼𝜏𝛽 x0 xT0 + √ H + 𝛼𝜏(𝛾 − 1)I . 𝑝 𝑝 (6.67) We know that the eigenvalue distribution of matrix H follows the semi-circular law and has eigenvalues from −2 to 2. Besides, adding a rank-one matrix x0 xT0 /𝑝, affects this eigen-value distribution only if the coefficient of this rank one distortion is more √ than the coefficient of H/ 𝑝. Meaning that if 𝛼𝛽𝜏 < 1 we will have an extra √ H will have eigenvalue at 1 + 𝛼2 𝜏 2 𝛽2 . Otherwise, the eigenvalues of 1𝑝 x0 xT0 + 𝛼𝜏𝛽 𝑝 √ H which is semi-circular. Besides, the same distribution of the eigenvalues of 𝛼𝜏𝛽 𝑝 in optimization (6.66), we should always have 𝛼𝜏(𝛾 − 1 + 2𝛽) < 0. Otherwise, the positive part of the matrix in (6.67) will have an infinitely large Frobenius norm, which the minimization will avoid. Therefore, we sould always have 𝛾 − 1 − 2𝛽 < 0. Considering this constraint, if we solve this optimization over 𝛾 and the optimization 144 becomes, p √ √   2 + 𝑡 2 − 2(𝛼 2 − 𝑡 2 )(1 + 𝑡)) + 𝑡  𝛽( 𝛿 2𝛼    √ √    𝛽( 𝛿 2𝛼2 + 𝑡 2 − 𝛼𝜙1 (𝛼, 𝑡) − 2𝑡) + 𝑡  min max √ √ 𝛼≥0 𝛽≥0   𝛽( 𝛿 2𝛼2 + 𝑡 2 − 2𝑡 − 2) + 𝑡  𝑡∈R     ∞  √ if 𝑡 2 < 𝛼2 < 𝑡 2 + 2(𝑡 + 1)(1 − 1 + 𝑡) 2 , √ if 𝑡 2 + 2(𝑡 + 1)(1 − 1 + 𝑡) 2 < 𝛼2 < 1 , if 𝛼2 > 1 , if 𝛼2 ≤ 𝑡 2 . (6.68) where the function 𝜙(·, ·) is defined as 𝜙1 (𝛼, 𝑡) = 1 𝛼2 𝜏 3 + 3𝜏 − 2𝛼𝜏 2 + . 2 1− 1+𝑡 ≤𝜏≤ 1 2𝜏 √ min 𝛼 (6.69) 𝛼 So we would like to see under what conditions, the optimization (6.68) will be negative, which means the initial optimization (6.65) will fail in recovering X0 . The optimization (6.68) will be negative, if there exists a negative 𝑡 < 0 that makes the coefficient of 𝛽 negative. It’s not hard to see that the minimization over 𝛼 and 𝑡 in (6.68) happens when both 𝛼 and 𝑡 go to zero. Thus, it’s the ratio of 𝛼/𝑡 will define the result of this optimization. A closer look at this optimization shows that it will be minimized when 𝛼2 > 32 𝑡 2 , especially, when 𝑡/𝛼− → 0. In this scenario, the second case in the objective √ √ function of (6.68) holds, and the objective function will converge to 𝛽𝛼( 2𝛿 = 6) which is always positive if 𝛿 > 3. In the other words, this optimization is zero if 𝛿 ≥ 3, otherwise, there exists negative values of 𝑡 that makes it negative. Thus we have perfect recovery iff 𝛿 > 3. This proof was for the case that rank of the unknown matrix X0 was one. For the Í𝑟 case of a rank 𝑟 matrix, the proof will be the same, expect that X0 = 𝑖=1 x𝑖 x𝑖T . So in the matrix (6.67), we will have a rank 𝑟 distortion of a Wigner matrix, and following the same steps, we can show that if 𝛿 > 3𝑟, the objective function always remains non-negative. 145 Chapter 7 PERFORMANCE ANALYSIS OF CONVEX DATA DETECTION IN MIMO 7.1 Introduction We consider the problem of recovering a transmit signal1, 𝛽0 ∈ D 𝑝 , form 𝑛 (noisy) linear observations of the form y = X𝛽0 + z, where D ⊂ C denotes the discrete transmit constellation, and z ∈ C𝑛 is the noise vector. This problem has a pivotal role in signal detection in multiple-input, multiple-output (MIMO) communication systems [90, 91, 126], where X ∈ C𝑛×𝑝 , often referred to as the channel state information, is a known matrix. In such settings, 𝑛 and 𝑝 correspond to the number of transmit and receive antennas, respectively. The Maximum Likelihood(ML) estimator is the desirable theoretical solution for this problem. There has been numerous studies to investigate algorithms that can generate exact or approximate solutions for this problem. Due to the combinatorial nature [185] of the problem, exact algorithms (e.g. sphere decoding [83]) are computationally prohibitive, especially in a very large system (e.g. massive MIMO) [140]. Therefore, various heuristics have been proposed and used in practice [71, 81] to approximate the ML solution. Despite tractable computational complexity, the precise performance analysis of such methods are often challenging. Due to the practical advantages of convex algorithms, one conventional approach to solve this problem is to relax the discrete set 𝐷𝑐 to a continuous convex set S and utilize convex programming to search over S instead of D [108, 162, 198]. The performance of this method for data recovery has been investigated in the works of [9, 91, 166] for the real valued constellations, specifically, BPSK and PAM when the channel matrix is Gaussian. To do so, Thrampoulidis et. al. [166] utilized a framework that they had developed, known as the CGMT framework [4, 165]. The CGMT framework has been successfully applied to analyze the performance in a number of other applications including analysis of regularized M- estimators [165], and PhaseMax in phase retrieval [53, 141, 142]. Unfortunately, The CGMT framework can not be readily extended to the complex settings (which indeed is the desirable case in many practical applications). 1 This chapter is mainly based on the work in [1] 146 The major result of this section is to introduce a new comparison lemma for complex Gaussian processes to study the convex detection problem for complex constellations. In particular, we precisely characterize the symbol error rate performance of the convex method, for a general constellation D and a convex relaxation S. Our theorem also allows us to derive the necessary and sufficient number of antennas, 𝑛, required for data recovery in the high-SNR regime which enables us to precisely characterize the phase-transition regions. Through our analysis, we can further observe the relationship between the choice of the convex relaxation with its corresponding phase transition. As an example, we analyze the loss in performance when choosing a relaxation that is easier to implement in a convex program for the case of PSK modulation. 7.2 Problem Setup Notations We gather here the basic notations that are used throughout this section. We reserve the letter 𝑗 for the complex unit. For a complex scalar 𝑥 ∈ C,q 𝑥 Re and 𝑥 Im 2 + 𝑥2 . correspond to the real are imaginary parts of 𝑥, respectively and |𝑥| = 𝑥 Re Im 2 2 N (𝜇, 𝜎 ) denotes real Gaussian distribution with mean 𝜇 and variance 𝜎 . Similarly, NC (𝜇, 𝜎 2 ) refers to a complex Gaussian distribution with real and imaginary parts drawn independently from N (𝜇Re , 𝜎 2 /2) and N (𝜇Im , 𝜎 2 /2), respectively. 𝑋 ∼ 𝑝 𝑋 implies that the random variable 𝑋 has a density 𝑝 𝑋 . We reserve the letters 𝐺 and 𝐻 to denote (scalar) standard normal random variables. Similarly, 𝐻C is reserved to denote a complex NC (0, 2) random variable. The bold lower letters are reserved for vectors and for a vector v, v𝑖 denotes its 𝑖 th entry. Finally, for a convex set S ⊂ C, the projection and distance functions with respect to D are defined as PS (x) := arg min kx − yk y∈S PS (x) := min kx − yk. y∈S (7.1) Setup Our goal is to recover an 𝑝-dimensional vector 𝛽0 ∈ C 𝑝 where the entries of 𝛽0 are independenty drawn from the discrete set D ⊂ C with distribution x0,𝑖 ∼ 𝑝 𝑋 . The set D defines the modulation used for data transmission (e.g. QAM, PSK, etc.). For this purpose, we are given the noisy multiple-input multiple-output (MIMO) relation of the form y = X𝛽0 + z ∈ C𝑛 , (7.2) 147 where X ∈ C𝑛×𝑝 is the known MIMO channel matrix with i.i.d. entries drawn from NC (0, 1𝑝 ) and z ∈ C𝑛 is the unknown noise vector with i.i.d. random complex Gaussian NC (0, 𝜎 2 ) entries. Estimator The ML estimator of x0 in this scenario is 1 𝛽ˆ = arg min𝑝 ky − X𝛽k 2 . 𝛽∈D 2𝑛 (7.3) Since solving (7.3) is computationally intractable, a variety of heuristic methods, such as zero-forcing, MMSE, decision-feedback, have been proposed. In this section, we make use of convex programming to estimate 𝛽0 . In the first step, we relax D to a convex set S and minimize the objective function of (7.3) over this relaxed convex set, 1 𝛽˜ = arg min𝑝 ky − X𝛽k 2 . 𝛽∈S 2𝑛 (7.4) Next, we map each entry of 𝛽˜ to the closest point in D to build our final estimation of 𝛽0 , 𝛽ˆ𝑖 = arg min |𝛽 − 𝛽˜𝑖 |, 𝛽∈D 𝑖 = 1, . . . , 𝑝. (7.5) We refer to this method as the Convex Decoder Algorithm (CDA). In this section, we will precisely analyze the performance of the CDA as a function of the problem parameters such as 𝜎, 𝑛/𝑝, D and S. Note that the performance of CDA depends on the constellation D and the way we relax it to the convex set S. Later in Section 7.3, we observe the impact of choosing two different relaxations on the performance of CDA and its phase-transition regions with the help of our main theorem. Symbol error probability We characterize the performance of CDA in terms of the symbol error probability, defined as the expected value of the Symbol Error Rate (SER) where, 𝑝 1Õ 𝑆𝐸 𝑅 := 1ˆ , 𝑝 𝑖=1 𝛽𝑖 ≠𝛽0,𝑖 𝑝
1Õ 𝑃𝑒 := E[𝑆𝐸 𝑅] = P 𝛽ˆ𝑖 ≠ (𝛽0 ) 𝑖 . 𝑝 𝑖=1 (7.6) 148 Here 𝛽ˆ is the output of CDA in (7.5), 1E is the indicator of the event E and the probability P(·) is over the randomness of X, z and 𝛽0 . We introduce the notation S𝑥 for 𝑥 ∈ D, as the set of all points in S that will be mapped to 𝛽 in (7.5). Equivalently, S 𝛽 := {𝛽0 ∈ S : ∀y ∈ D, |𝑏𝑒𝑡𝑎0 − 𝑏𝑒𝑡𝑎| < |𝑏𝑒𝑡𝑎0 − 𝑦|}. (7.7) This notation helps us interpret our main theorem more clearly. Using this notation, we can rewrite the symbol error probability defined in (7.6) as 𝑝
1Õ 𝑃𝑒 = P 𝛽˜𝑖 ∉ S (𝛽0 ) 𝑖 , 𝑝 𝑖=1 (7.8) where 𝛽˜ is the minimizer of (7.4). Assumptions We impose two mild assumptions on the problem. First, we assume that the entries of 𝛽0 are i.i.d. random variables with 𝛽0,𝑖 ∼ 𝑝 𝛽 and also P 𝛽 (𝑟 1 + 𝑗𝑟 2 ) = P 𝛽 (𝑟 2 + 𝑗𝑟 1 ), ∀𝑟 1 , 𝑟 2 ∈ R. Second, we want the convex set to be symmetric in the sense that if (𝑟 1 + 𝑗𝑟 2 ) ∈ S, then also (𝑟 2 + 𝑗𝑟 1 ) ∈ S. Modulations Using our main theorem, we can precisely analyze the SER of the CDA in terms of the SER for a constellation D which we relax to an arbitrary convex set S. For a better understanding of the theorem and to show how to apply it to different schemes, we will work with two conventional modulations; Phase-Shift Keying (PSK) and Quadrature Amplitude Modulation (QAM). 𝑁-PSK Constellation: each entry of 𝛽0 is randomly o n 𝑗2 𝜋 In the 𝑁-PSK constellation, drawn from D = 𝑒 𝑁 𝑖 : 𝑖 = 0, . . . , 𝑁 − 1 . The entries of D are distributed over the unit circle in the complex space and therefore the Signal to Noise Ratio (SNR) will be 1/𝜎 2 . Next, we need an appropriate convex relaxation of D for CDA. We suggest two candidates for this purpose and compare their performances later in Section 7.3. In one which we will refer to as the Circular Relaxation (CR), we choose the set S (CR) = {𝛽 ∈ C : |𝛽| ≤ 1} as the convex set in (7.4). The simple structure of S (CR) makes its implementation easier in the convex program (7.4). In another scenario, we consider the convex hull of D as the relaxed set S and refer to it as Convex Hull Relaxation (CHR). Thus, S (CHR) = Conv(D) will be used in (7.4) which might be harder to implement compared to S (CR) . But we will show that since (CHR) is a tighter relaxation, its corresponding CDA performs better in 149 terms of SER. 𝑁 2 -QAM Constellation: We also briefly talk about the 𝑁 2 -QAM modulation where   𝑁 −1 D = (𝑟 + 𝑗 𝑠) − (1 + 𝑗) : 𝑟, 𝑠 ∈ {0, . . . , 𝑁 − 1} . 2 −1) Under this constellation the SNR will be 𝑁 (𝑁 . The relaxation that is often used 6𝜎 2 for this modulation is known as the Box Relaxation (BR) [166] which is 2  𝑁 −1 𝑁 −1 S = (𝑥 + 𝑗 𝑦) ∈ C : |𝑥| ≤ , |𝑦| ≤ 2 2  (7.9) Using our main theorem, we can calculate the SER of CDA under box relaxation and rederive the results of [91, 166]. 7.3 Main Result Our main result explicitly characterizes the limiting behavior of the symbol error rate of the convex decoder algorithm, under the high dimensional regime where 𝑛, 𝑝 → ∞ with a constant ratio 𝛿 := 𝑛/𝑝. Theorem 18 (SER analysis of CDA) Let SER denote the symbol error rate of the Convex Decoder Algorithm (CDA), for random signal 𝛽0 ∈ D with entries drawn independently from the distribution 𝑝 𝑋 . Let S be a convex relaxation of D and S and 𝑝 𝑋 satisfy the assumptions in Section 7.2. Fix 𝑆𝑁 𝑅 and 𝛿 = 𝑛/𝑝 and consider the optimization 𝐻C 𝛿 − 1 𝜎2 𝜏 𝜏 + + E[Dist2S (𝑋 + √ )]. 𝜏>0 2𝜏𝛿 4 4 𝜏 𝛿 min If (7.10) has a unique answer 𝜏 ∗ , then in the limit of 𝑝, 𝑛 → ∞   𝐻C lim 𝑃𝑒 = P PS (𝑋 + √ ) ∉ S 𝑋 . 𝑝,𝑛→∞ 𝜏∗ 𝛿 (7.10) (7.11) The expected value and probability in (7.10) and (7.11) are over 𝑋 ∼ 𝑝 𝑋 and 𝐻C ∼ NC (0, 2), respectively. 150 Figure 7.1: SER Performance of the Circular Relaxation (CR) for 16-PSK: 𝑃𝑒 as a function of SNR for the two cases where 𝛿 = .8 and 𝛿 = 1. The theoretical prediction follows from Theorem 18 and the high-SNR analysis comes from Section 7.3. For the simulation, we used signals size 𝑝 = 128 with o each entry chosen randomly n of 𝑗𝜋 uniform from the set S𝑃𝑆𝐾 = 𝑒 8 𝑖 : 𝑖 = 0, . . . , 15 . The data are averages over 30 independent realizations of the channel matrix and the noise vector. Theorem 18 provides a formula to calculate the SER of the convex decoder, under a general constellation in the high dimensional regime. Theorem 18 provides a formula to calculate the SER of the convex decoder, under a general constellation in the high dimensional regime. Remark 27 (Computing 𝜏 ∗ ) The objective function in (7.10) is convex and only involves one scalar variable. Thus, 𝜏 ∗ can, in principle, be efficiently numerically computed. It can be shown that 𝜏 ∗ is the minimizer of (7.10) if and only if it is the answer to the corresponding first-order optimality condition, " #! 1 2 1 𝐻C 2 = 𝜎 + E 𝑋 − PS (𝑋 + √ ) . (7.12) 𝜏∗2 2 𝜏∗ 𝛿 Although, this does not provide us with a closed form formula to calculate 𝜏 ∗ , in all our simulations a fixed-point iterative method converges to 𝜏 ∗ over a handful of 151 iterations. It can be also shown that (7.12) has a unique solution if 𝛿 > 𝛿∗ for some 𝛿∗ ∈ (0, 1) which depends on S. Next, we apply Theorem 18 to the 𝑁-PSK and 𝑁 2 -QAM modulations introduced in Section 7.2 to calculate their corresponding symbol error probabilities and phasetransition thresholds in the high-SNR regime. Figures 1 and 2 verify the accuracy of the prediction of Theorem 18 for 16-PSK and 16-QAM modulations, respectively. Note that although the theorem requires 𝑝 → ∞ , the prediction is already accurate for 𝑝 = 128. In these figures, we have also plotted the high-SNR expressions for SER that we derive in the next Section for both modulations. Interestingly, we observe that this high-SNR expression gives us a good enough approximation of the exact value of SER, even for small practical values of SNR. 𝑁-PSK Constellation Under the 𝑁-PSK setup, the set D is defined in Section 7.2. We investigate the error performance of the convex decoder algorithm for two different convex relaxations for the set D; Circular Relaxation (CR) and Convex-Hull Relaxation (CHR). The effect of using different relaxations shows up in the projection function in equations (7.12) and (7.11). Define S(SR) = {𝑐 ∈ C : |𝑐| ≤ 1} as the circular relaxation of D. The projection function on this set has the following form, PS(CR) (𝛽) =   𝛽  if |𝛽| ≤ 1   𝛽/|𝛽|  otherwise. (7.13) Therefore, 𝜏 ∗ can be efficiently calculated using a fixed-point iterative method to solve (7.12). Furthermore, due to the symmetric nature of the 𝑁-PSK constellation, the probability of error for each symbol in D can be derived in the following closed form,  √  𝜋 ∗ 𝑃𝑒 = P |𝐺 | > tan( )(𝐻 + 𝜏 𝛿) , (7.14) 𝑁 where 𝐺 and 𝐻 are i.i.d. N (0, 1). High-SNR Analysis Let S(CR) and S(CHR) denote the circular relaxation and convexhull relaxation of the set S. It can be shown that for SNR  1, 𝜏 ∗ grows large √ proportional to SNR. As a consequence, the last term in (7.10) can be approxi1 𝑁+4 mated by 8𝜏𝛿 and 8𝑁𝜏𝛿 for the cases of (CR) and (CHR), respectively. This results 152 Figure 7.2: SER Performance of the Box Relaxation for 16-QAM: 𝑃𝑒 as a function of SNR for the two cases where 𝛿 = .8 and 𝛿 = 1. The theoretical prediction follows from Theorem 18 and the high-SNR analysis comes from Section 7.3. For the simulation, we used signals of size 𝑝 = 128 with each entry chosen uniformly at random in the set S𝑄 𝐴𝑀 = {±1, ±3}2 . The data are averages over 30 independent realizations of the channel matrix and the noise vector. q q 2SNR(𝛿−3/4) for (CR) and 𝜏 ∗ = 2SNR(𝛿−3/4+1/𝑁) for (CHR). Putting these 𝛿 𝛿 ∗ values for 𝜏 in (7.14) yields their corresponding high-SNR symbol error probabil- in 𝜏 ∗ = ities,   p 𝜋 𝑃𝑒(CR) = P |𝐺 | > tan( )(𝐻 + 2SNR · (𝛿 − 3/4)) , 𝑁   p 𝜋 (CHR) 𝑃𝑒 = P |𝐺 | > tan( )(𝐻 + 2SNR · (𝛿 − 3/4 + 1/𝑁)) . 𝑁 (7.15) (7.16) The difference between phase-transitions of these two cases can be observed from equations (7.15) and (7.16). While for (CR) we need 𝛿 > 3/4 for consistent data recovery, this threshold changes to 𝛿 > (3/4 − 1/𝑁) for (CHR). This essentially means that 𝑝/𝑁 additional MIMO receivers is required at the expense of having a simpler convex set. This verifies the fact that while the optimization over S(CR) might be done faster over the Circular Relaxation due to its simple structure, we need more measurements (or higher SNR) to get the same performance for (CR) 153 compared to (CHR). In other words, the performance of (CR) is 10 log10 ( 𝛿−3/4+1/𝑝 𝛿−3/4 ) off that of (CHR). Comparison to the matched filter bound. The matched filter is the ideal impractical case where we assume to have the first 𝑝 − 1 entries of 𝛽0 and we want to recover the last entry. We compare the symbol error probability of this scenario, referred to as the Matched Filter Bound (MFB), with the 𝑃𝑒 of the convex decoder that can be derived from Theorem 18. The matched filter bound corresponds to the probability of error in detecting 𝑋 ∈ D from ỹ = 𝑋x + z, where x ∈ C 𝑝 with Gaussian entries drawn from NC (0, √1𝑝 ), and z is the noise vector with entries NC (0, 𝜎 2 ). Then, the probability of error of the ML estimator of 𝑋 in 𝑁-PSK will be   √ 𝜋 P |𝐺 | > tan( )(𝐻 + 2SNR · 𝛿) . 𝑁 (7.17) Comparison of (7.15) with (7.17) shows that in the high-SNR regime the perfor𝛿 )dB off from the (MFB). In particular, in the square mance of (CR) is 10 log10 ( 𝛿−3/4 case (𝛿 = 1), where the number of receive and transmit antennas are the same, the (CR) is 6dB off the (MFB). Besides, as 𝛿 → ∞ (meaning that the number of antennas grows largecompared to users), the performance of (CR) and (CHR) approaches (MFB). 𝑁 2 -QAM Constellation In 𝑁 2 -QAM, each entry of 𝛽0 is randomly chosen from the set D defined in Section 7.2 with distribution 𝑝 𝑋 . The conventional relaxation for this constellation is the Box Relaxation (BR) [108, 162, 198] defined in (7.9). Similar to the previous section, In order to use Theorem 18, we need to form the projection function to S in (7.1) which is straightforward for a box set. Once 𝜏 ∗ is obtained using equation (7.12) (or recruiting other methods to solve (7.10)), we shall use (7.11) to calculate 𝑃𝑒 of 𝑁 2 -QAM constellation. Here, unlike the 𝑁-PSK case, the probability of error in the recovery is not the same for different symbols in D. Using the same set of arguments in Section 7.3, it can be shown that in the high1 SNR regime, the last term in the objective function of (7.10) approaches 2𝜏𝛿𝑁 . q Therefore the answer to the minimization problem will be 𝜏 ∗ = 2SNR(𝛿−(𝑁−1)/𝑁) . 𝛿 𝑁−1 ★ This implies that 𝛿 = 𝑁 is the recovery threshold for the Box Relaxation of the set D. It can also be shown that for 𝛿 > 𝛿★, the problem (7.10) is strictly convex 154 and therefore has a unique solution. This is consistent with the result of [91] which proves the same phase-transition region for the Box Relaxation. 7.4 Proof Outline In this section we introduce the main ideas used in the proof of Theorem 18. The goal is to analyze the performance of the following optimization problem: 1 𝛽ˆ = arg min𝑝 ||y − X𝛽|| 2 𝛽∈S 2𝑛 (7.18) We rewrite (7.18) by changing variable to vector w = 𝛽 − 𝛽0 1 ||z − Xw|| w∈S −𝛽0 2𝑛 ŵ = arg min 𝑝 (7.19) Now let X̃ = [X 𝑅 , X 𝐼 ; −X 𝐼 , X 𝑅 ] ∈ R2𝑛×2𝑝 and z0 = [z 𝑅 ; z 𝐼 ] ∈ R2𝑛 , where X 𝑅 and z 𝑅 (X 𝐼 and z 𝐼 ) are the real (imaginary) parts of X and z, respectively. Now (7.19) can be written as w★ = arg min w∈R𝑏𝑏 2 𝑝 w𝑖 + 𝑗w𝑛+𝑖 ∈S−(𝛽0 )𝑖 1 0 1 ||z − p x̃ · w|| 2 . 4𝑛 2𝑝 (7.20) This optimization is difficult to analyze and current methods for asymptotic analysis of such optimizations fail here, because of the dependence between the entries of X̃. The main step of our proof is to show that in the asymptotic regime when 𝑝, 𝑛 =→ ∞ with 𝑛/𝑝 = 𝛿, the SER in the optimization (7.19) converges to the one in the following. w★ = arg min w∈R2 𝑝 w𝑖 + 𝑗w 𝑝+𝑖 ∈S−(𝛽0 )𝑖 1 0 1 ||z − p B · w|| 2 . 4𝑛 2𝑝 (7.21) Here, z0 ∈ R2𝑛 is a vector with i.i.d. N (0, 𝜎 2 /2) entries and B ∈ R (2𝑛)×(2𝑝) is a matrix whose entries are independently drawn from N (0, 1). To do so, we first show this in the case that both the objective functions have an extra strongly convex term 𝜖 kwk 2 /2. Under this scenario, we can utilize the Lindeberg method as in [3, 130]. In equations (6.15) followed by Theorems 15 and 16 we prove that these two optimizations converge to the same value (with the difference that for our optimizations we should change two dependent rows of B with two dependent rows of X̃). The idea is to replace the rows of X̃ to B in 𝑛 steps. In each step, we replace the 155 rows 𝑖 and 𝑝 + 𝑖 in the X̃ (that are independent from the rest of X̃) with the the rows 𝑖 and 𝑖 + 𝑝 in the B. We can show that each step changes the SER on the order of O ( 𝑝 5/4 ). So as 𝑛 → ∞, the SER doesn’t change. Next, we use the RIP condition for Gaussian matrices to show that removing the extra 𝜖 kwk 2 /2 term in the optimization does not affect the SER for small enough 𝜖 (See Sections 3.1 and 3.3.2 in the appendix of [131] for more details regarding these two steps). Then, we just need to analyze performance of (7.21) instead of (7.19). For the rest of the proof, we apply the CGMT framework and the same tools as in [165](Section 5.3). The idea is to rewrite (7.21) as the following min-max problem, 1 1 1 (7.22) max √ u𝑡 z0 − √ u𝑡 Bw − ||u|| 2 , 2𝑛 w𝑖 + 𝑗w 𝑝+𝑖 ∈S−(𝛽0 )𝑖 u∈R 2 𝑝𝑛 2 2𝑛 This enables us to apply the CGMT which associates with (7.22), the following simplified optimization whose analysis provides us with the desired properties of the initial optimization. min 1 1 max √ u𝑡 z0 − ||u|| 2 w𝑖 + 𝑗w 𝑝+𝑖 ∈S−(𝛽0 )𝑖 u 2 2𝑛 1 + √ (g𝑡 u||w|| + h𝑡 w0 ||u||) , 2 𝑝𝑛 min where g ∈ R2𝑛 and h ∈ R2𝑝 have i.i.d. standard Gaussian entries. It can be shown that the optimization over u results in 1 ||w|| 1 √ ||z0 + p g|| + √ h𝑡 w . w𝑖 + 𝑗w 𝑝+𝑖 ∈S−(𝛽0 )𝑖 2𝑛 2 𝑝𝑛 2𝑝 min (7.23) √ 1 Using 𝑥 = min𝜏>0 2𝜏 + 𝜏𝑥 2 , optimization (7.23) can be written as 1 𝜏kz0 k 2 𝜏||w|| 2 kgk 2 1 + + min + √ h𝑡 w. 𝜏>0 2𝜏 w𝑖 + 𝑗w 𝑝+𝑖 ∈S−(𝛽0 )𝑖 4𝑛 8𝑛𝑝 2 𝑝𝑛 min (7.24) Using dimension reduction techniques, we can show that from the following deterministic optimization, we can tightly infer the properties of (7.24). 1 𝜏𝜎 2 𝜏||w|| 2 1 + + min + √ h𝑡 w. 𝜏>0 2𝜏 w𝑖 + 𝑗w 𝑝+𝑖 ∈S−(𝛽0 )𝑖 4 4𝑝 2 𝑝𝑛 min (7.25) A completion of squares in the minimization over w, the weak law of large numbers and convex techniques (See Section A.3 in [165] to see how WLLN can be applied here) results in the final deterministic optimization 𝛿 − 1 𝜎2 𝜏 𝜏 𝐻𝑐 min + + E[Dist2S (𝑋 + √ )]. 𝜏>0 2𝜏𝛿 4 4 𝜏 𝛿 (7.26) 156 Besides, the optimal w can be obtained by putting the optimizer of (7.26) in the minimization over w in the last term of the (7.25). Similar to the proof of [166](section 3), SER of w∗ derived here is equal to the one from (7.19) which is   𝐻C 𝑃𝑒 → P PS (𝑋 + √ ) ∉ S 𝑋 (7.27) 𝜏∗ 𝛿 157 Chapter 8 ACHIEVING NEAR MAXIMUM-LIKELIHOOD PERFORMANCE IN MASSIVE MIMO 8.1 Introduction The last several decades has seen a sustained exponential growth rate in wireless traffic (known as Cooper’s law). Due to increasing applications of wireless networks, such as real-time video, wireless gaming, virtual reality, the emerging internet-of-things, etc., this exponential growth is forecast to continue unabatedly into the future. The goal of the upcoming 5G standard is to enable reliable communication at these higher data rates (often 100 Mbits/sec and beyond). One of the most promising technologies suggested for 5G is Massive MIMO, where each base station is equipped with a very large number of antennas (many tens to even hundreds). This increases the capacity of the network many-fold (ideally, by the number of base station antennas) without adding new base stations or increasing the frequency spectrum. in this section, we address the signal processing challenge in Massive MIMO [126, 185], where the base station is confronted with a deluge of data, coming from potentially several hundred independent streams, and is required to reliably and efficiently recover the data. The maximum-likelihood (ML) estimator, which is the desireable theoretical solution for this problem, can not be computed exactly due to its combinatorial nature [83, 185]. Various heuristics, with tractable computational complexities, have been proposed to approximate the ML solution often have performance quite distant from ML. Due to practical advantages, convex-optimization-based methods have gained significant attenstions in the recent years. The performance of convex relaxation for signal recovery in MIMO communication systems have been analyzed in [9, 166, 174] for real-valued constellations (e.g. BPSK, PAM), and very recently, in [1], it has been extended to the complex-valued constellations (e.g. PSK, QAM). Here, we develop an algorithm that employs a combination of convex and nonconvex techniques. Our proposed method essentially exploits the solution of a convex optimization as the initial point for a local search method. We provide the- 158 oretical bounds on the performance of symbol update as the post-processing local search method. Combining this with the theoretical guarantees on the performance of the convex optimization, we will find the regimes of SNR under which our proposed method has a performance very close to the ML estimate. Our numerical simulations validates this result by showing that the proposed method has a performance very close to the matched filter bound (which itself is a lowerbound on the performance of the ML estimate.) 8.2 Problem Formulation Notations We gather here the notations that are used throughout the paper. The bold lower letters are reserved for vectors and upper letters are used for matrices. For a vector v, v𝑖 represent its 𝑖 th entry, and for a matrix M, M 𝑘 represent its 𝑘 th column. We reserve the letter 𝑗 for the complex unit. For a complex scalar 𝑥 ∈ C, Re{𝑥}, and Im{𝑥} correspond to its real and imaginary part, and |𝑥| is its absolute value. NC (𝜇, 𝜎 2 ) denotes the complex Gaussian distribution, with real and imaginary parts drawn independently from N (𝜇Re , 𝜎 2 /2), and N (𝜇Img , 𝜎 2 /2). 𝑄(·) denotes the tail 𝑃 distribution of standrad normal distribution, and → indicates convergence in probability. 𝑋 ∼ 𝑝 𝑋 implies that the random variable 𝑋 has a density 𝑝 𝑋 . For a set S, conv(S) denotes its convex hull. Problem Setup Consider the problem of recovering a transmitted data vector 𝛽0 ∈ D 𝑝 , where 𝑝 is the number of data streams and D ⊂ C defines the transmit constellation (QAM, PSK, etc.) for the streams. In our setting, 𝑝 can be considered to be the number of mobile users operating in a cell at a certain frequency band. Assuming that the base station is employingq𝑛 receive antennas, we seek to recover 𝛽 ∈ D 𝑝 from the noisy linear relation y = SNR X𝛽 + z ∈ C𝑛 , where, X ∈ C𝑛×𝑝 0 𝑝 𝑛 is the known MIMO channel matrix and z ∈ C is the unknown noise vector. We 0 consider a random setup where the entries of X and z are both random. The noise vector z has i.i.d. zero-mean unit-variance circularly-symmetric complex Gaussian (NC (0, 1)) entries. In our analysis, we will also assume that the entries of X are i.i.d. NC (0, 1), in which case SNR is, in fact, the signal-to-noise-ratio. This is a reasonable assumption when the environment is rich-scattering and there is sufficient separation between the receive antennas. When the symbols of 𝛽0 are chosen uniformly at random, the estimator that minimizes the block probability of error is the maximum-likelihood (ML) estimator 159 Figure 8.1: The proposed two-pronged algorithm. which is given by 2 s 1 𝛽ˆML = arg min𝑝 y− 𝛽∈D 2𝑛 SNR X𝛽 𝑝 . (8.1) This problem is computationally intractable (in fact, known to be NP hard) when 𝑝 and 𝑛 are large. As a result, many computationally-efficient heuristics have been proposed to approximately solve (8.1), including zero-forcing, mmse, and decisionfeedback-equalization. Unfortunately, these methods underperform ML by quite a bit (especially when 𝑝 and 𝑛 are large). Furthermore, they often lack rigorous performance analysis, especially in the case of decision-feedback-equalization. An exact method for solving (8.1) is the sphere decoder [70]. While it has reasonable complexity when 𝑝 and 𝑛 are small, it cannot be scaled to the dimensions encountered in massive MIMO [83]. We propose a two-pronged attack to develop methods that can efficiently achieve near ML performance. However, before doing so, let us examine more closely the performance of (8.1). This will give us a benchmark against which we can compare our methods. As mentioned, computing the ML solution of (8.1) is practically impossible. Furthermore, a rigorous formula for the SER (symbol-errorrate) corresponding to (8.1) is not known. Using the heuristic replica method from statistical physics, Tanaka [163] gave a formula for the SER of (8.1) when the modulation is BPSK, i.e., D = {±1} and all signals are real. Since we cannot be certain of the validity of that formula, we use a rigorous lower bound on the SER of the ML estimator which is called the matched filter bound (MFB). This bound is obtained by assuming that a genie provides the receiver with the correct value of all symbols in 𝛽0 except for the first one, and we use the maximume-likelihood estimator for just the first symbol (instead of the whole block). This is a lower bound on the SER of any estimator and we shall use it for comparison purposes. 160 Figure 8.2: SER Performance of the convex relaxation (Blue line), After doing the local search (Green line) and the Matched Filter Bound (Red line) for 8-PSK: SER as a function of SNR for the two cases. For the simulation, we used signals of size n = 128 with each entry chosen randomly uniform from 8-PSK constellation. The data are averages over 100 independent realizations of the channel matrix and the noise vector. The left figure corresponds to 𝛿 = .9 and for the right figure 𝛿 = 1.1 . Two-step Algorithm Our two-step algorithm computes the solution to a convex optimization and then perform a local search to further improves the performance. We mathematically present our proposed method in Algorithm 2. The convex optimization (8.2) inimizes the loss function in (8.1) over the set C, where C is a convex set that contains 161 all the constellation points. After finding the solution to the optimization program, we map it to closest point in the constellation. This step can be done entrywise and therefore efficiently. In the second step of the algorithm, we perform a local search method. For our analysis, we use a simple symbol update algorithm, which iteratively checks if changing a certain entry can lower the cost function. Note that the optimization program (8.3) has only |D| feasible points. Figure 8.2 shows the performance of our two step algorithm, in terms of the symbolerror-rate (SER), with respect to SNR, for PSK and QAM constellations. As depicted in the figure, the second step makes considerable contribution to the performance, especially for larger values of SNR. Besides, we see that the performance of the two-step algorithm gets stupendously close to MFB for larger SNR values. Algorithm 2 Two-pronged Algorithm Step 1: Convex Relaxation  Choose a convex set C, such that D 𝑝 ⊆ C,  Solve the following convex optimization problem, s SNR 1 X𝛽|| 2 , 𝛽ˆ = arg min ||y − 𝛽∈C 2𝑛 𝑝 (8.2)  Map the result to the closest point in constellation, i.e., for 𝑖 = 1, 2, . . . , 𝑝, i.e. 𝛽˜𝑖 := arg min 𝛽∈D |𝛽 − 𝛽ˆ𝑖 |. Step 2: Local Search (Symbol Update)  For 𝑖 = 1, 2, . . . , 𝑝: s 𝛽∈D SNR X𝛽|| 𝑝 s.t. 𝛽 𝑗 = 𝛽˜ 𝑗 , ∀ 𝑗 ≠ 𝑖. 𝛽¯𝑖 = arg min𝑝 ||y − (8.3) Analysis of the convex relaxation The convex-optimization-based methods [108, 162, 198] have gained significant attentions in recent years mainly due to the availability of fast convex solvers as well as the theoretical guarantees that can be provided on their performance. The difficulty in solving (8.1) is the nonconvexity of the constraint set D 𝑝 (the set is, in 162 fact, discrete). One approach to approximating (8.1) is to relax the discrete set to a convex one. If we remove the constraint entirely and write 2 s 1 © 𝛽˜ZF = 𝑞 ­arg min y− 𝛽 2𝑛 « SNR ª X𝛽 ® , 𝑝 ¬ (8.4) where 𝑞 simply quantizes each entry of its argument to the closest constellation point in D, we obtain the zero-forcing equalizer. Similarly, if we relax D 𝑝 to the ball k 𝛽k ≤ 𝑐, for an appropriate constant 𝑐, then the MMSE equalizer can be written as s 2 1 SNR © ª 𝛽˜MMSE = 𝑞 ­arg min y− X𝛽 ® . (8.5) k 𝛽k≤𝑐 2𝑛 𝑝 « ¬ √ For example, for BPSK we should take 𝑐 = 𝑝. Clearly, neither of the above relaxations are the best convex relaxations that one can use. The best would be to relax D to its convex hull, S = conv(D), in which case the estimator becomes, s 1 © y− 𝛽˜conv = 𝑞 ­arg min𝑝 𝛽∈S 2𝑛 « 2 SNR ª X𝛽 ® . 𝑝 ¬ (8.6) Many efficient methods for solving the above convex optimization problem exist [23, 28]. While these relaxations have been extensively used, analyzing their performance was, for a long time, an open problem. In the past 4-5 years, the performance of the convex relaxation methods have been analyzed for real-valued constellation in the works of [9, 91, 166, 174]. More specifically, Thrampoulidis et. al. [174] exploited a framework known as the Convex Gaussian Min-max Theorem (CGMT) [165, 171] to compute the performance of the box relaxation method for BPSK and PAM modulations. More recently, in [1], the authors of this paper have extended the previous results and provided a precise performance analysis for a complex-valued constellation, D ⊂ C, a general convex set C such that D 𝑝 ⊂ C. In order to compare performance of different convex relaxations, let us consider 163 n o 2𝜋 8-PSK constellation as an example, where D8-PSK = 𝑒 𝑗 8 𝑖 |𝑖 = 0, . . . , 7 . Using Theorem 3.1 in [1], the symbol error rate for the aforementioned relaxations can be bounded as follows, p  SNR · (𝛿 − 1) . ! r 5 SERConv ≤ 16 · 𝑄 SNR · (𝛿 − ) 8 √  SERMFB ≤ 16 · 𝑄 SNR · 𝛿 . SERZF ≤ 16 · 𝑄 and, (8.7) Here, SERZF , SERConv and SERMFB correspond to the symbol error rate of zero-forcing, convex hull relaxation and matched filter bound, respectively. More importantly, this result allows one to say something about the distribution of errors in a single block of transmission. It turns out that the number of errors in each ˜ concentrates around 𝑝SER [165]. This block of data, i.e., the number of errors in 𝛽, is a remarkable result and tells us that we have a very good idea about the number of errors that appear in the output of the convex optimization step. This knowledge will be critical in developing the second step of our approach. Analysis of the post processing (symbol update) In this section we analyze the performance of the post-processing local search method. The goal of this step is to reduce the number of erronous symbols in the result of the first step. As depicted in Figure 8.1, the symbol update method receives an input x̃ ∈ D 𝑝 , which is the result of the first step of the algorithm. Using the measurement matrix, X, and the measurement vector y, it updates the symbols one-by-one as in (8.3), to reduce the the objective value, ||y − X𝛽||. We analyze this step to realize the number errors this step is able to correct, in terms of a few problem parameters that we will define next. For a constellation set D ⊂ C, we define 𝑑min = 𝑑min (D) := min𝑠1 ≠𝑠2 ∈D |𝑠1 − 𝑠2 |, and 𝑑max = 𝑑max (D) := max𝑠1 ,𝑠2 ∈D |𝑠1 − 𝑠2 |. We assume the entries of 𝛽˜ are independently distributed such that, (𝛽𝑖∗ , 𝛽˜𝑖 ) ∼ 𝑝 𝑑 , for 𝑖 = 1, 2, . . . , 𝑝. Here, 𝑝 𝑑 denotes a probability mass function (PMF) over D 2 . We use 𝑝 𝑑 to define the average error distance, as follows, Definition 5 Let 𝑃 : D 2 → [0, 1] be a probability mass function over D 2 . The 164 average error distance 𝐷¯ is defined as, q 𝐷¯ := 𝐷¯ (𝑃) := E (𝑥,𝑦)∼𝑃 [|𝑥 − 𝑦| 2 ] (8.8) Theorem 19 provides an upper bound on the SER of the output of the symbol update algorithm, in terms of the parameters 𝐷¯ and 𝑑min . Theorem 19 Consider the symbol update algorithm as in (8.3). Let X has i.i.d. i.i.d. complex normal entries, and for 𝑖 = 1, 2, . . . , 𝑝, (𝛽𝑖∗ , 𝛽˜𝑖 ) ∼ 𝑝 𝑑 , where 𝑝 𝑑 is a probability mass function over D 2 . Then, as 𝑝, 𝑛 → ∞ with the fixed ratio 𝛿 := 𝑛𝑝 , ¯ can be bounded almost surely as, the symbol-error-rate of the output, 𝛽, r 𝑑min 𝛿 · SNR
SER ≤ (|D| − 1)𝑄 √ . (8.9) 2 1 + SNR · 𝐷¯ 2 Before stating the proof, the following remark is in place. Remark 28 It is worth nothing that the probability of error in the input, 𝛽˜ , lies ¯ Parameter 𝐷¯ is only defined for a tighter whitin the PMF 𝑝 𝑑 and consequently 𝐷. analysis of this step and can be simply bounded as 𝐷¯ ≤ SER 𝛽˜ · 𝑑max , where SER 𝛽˜ is ˜ Therefore, we have the following, the symbol-error-rate of the input signal 𝛽. s
𝑑min 𝛿 · SNR SER ≤ (|D| − 1)𝑄 √ . (8.10) 2 1 + SNR · 𝑑max · SER 𝛽˜ Proof 12 (outline) As 𝑝 → ∞, the WLLN asserts, 𝑝 1Õ 𝑃 SER = 1 𝛽¯𝑖 ≠𝛽∗ → P( 𝛽¯1 ≠ 𝛽1∗ ). 𝑖 𝑝 𝑖=1 (8.11) Recall that 𝛽¯1 is the output of the symbol update which is computed via solving the following optimization, r ¯𝛽˜ = arg min ||y − SNR X 𝛽˜ + (𝑠 − 𝛽˜ )X
|| . (8.12) 1 1 1 𝑠∈D Replacing y = q SNR X𝛽∗ + z would result, 𝑝 s 𝛽¯1 = arg min || 𝑠∈D SNR (𝑠 − 𝛽1∗ )X1 + v||, 𝑝 (8.13) 165 where v ∈ C𝑛 is defined as, s v := SNR Õ ˜ ( 𝛽𝑖 − 𝛽𝑖∗ )X𝑖 − z . 𝑝 𝑖=2 𝑝 (8.14) Exploiting the assumption that the entries of X𝑖 ’s and z are independently drawn i.i.d. from NC (0, 1) distribution and (𝛽𝑖∗ , 𝛽˜𝑖 ) ∼ 𝑝 𝑑 , we have the followig via WLLN, ||v|| → q 𝑛(1 + SNR · 𝐷¯ 2 ) . (8.15) Next, choose 𝑠 ∈ D − {𝛽1∗ }. We have, s P( 𝛽¯1 = 𝑠) ≤ P( S𝑁 𝑅 |𝑠 − 𝛽1∗ | 2 ||X1 || 2 𝑝 (8.16) +2Re{(𝑠 − 𝛽1∗ )v★X1 } < 0) . It can be shown that the expression in the right-hand-side of (8.16) converges in probability to, r |𝑠 − 𝛽1∗ | 𝛿 · SNR
. (8.17) RHS → 𝑄 √ 1 + SNR · 𝐷¯ 2 2 Since 𝑑min ≤ |𝑠 − 𝛽1∗ |, we can get an upper bound by replacing |𝑠 − 𝛽1∗ | with 𝑑min . The result (upper bound on SER) is then derived by taking a union bound over all 𝑠 ∈ D − {𝛽1∗ }. The following proposition can be proved by combining the results of Theorem 2.1, with theoretical guarantees on the first step that are derived in [1]. Proposition 1 Consider the assumptions of Theorem 19. Also assume that SNR goes to infinity as 𝑝 → ∞ (with any order larger than constant). Then the relative gap between the performance of the MFB (SER 𝑀 𝐹 𝐵 ) and our two step algorithm (SER 𝐴𝑙𝑔 ) goes to zero, i.e. SER 𝐴𝑙𝑔 − SER 𝑀 𝐹 𝐵 →0. SER 𝑀 𝐹 𝐵 (8.18) This Proposition simply states that the proposed algorithm achives the matched filter bound (MFB) when the SNR grows at a rate greater than a constant. Due to lack of space, we defer the proof, which is a combination of Theorem 19 and performance 166 10 0 SER 10 -1 10 -2 Convex Hull Relaxation Convex Hull Relaxation + Local Search Zero-Forcing Zero-Forcing + Local Search MMSE MMSE + Local Search Matched Filter Bound 10 -3 10 -4 8 9 10 11 12 13 14 15 16 17 SNR (dB) Figure 8.3: SER performance of our two-step algorithm with respect to SNR, for three choises of convex relaxation; Zero-Forcing (Red lines), MMSE (Black lines) and Convex Hull relaxation (Blue lines). The green curve corresponds to the performance of the Matched Filter Bound. Solid lines represent the performance of the first steps only, and the dashed lines are the final performance of the two-step algorithm. For our simulations we used 𝑝 = 128 with 𝑛𝑝 = 1.1. The results is averaged over 200 random independent realization of the channel matrix and noise vector. analysis of the first step, to an extended version of this paper. We conclude the section by presenting some numerical simulations that verify the validity of our proposition. Figure 8.3 shows the SER of our two-step algorithm with respect to SNR, for various convex relaxations in the first step. We used zero-forcing, MMSE, and Convex Hull relaxation for recovering an 8-PSK signal as the first step. As seen in the figure, applying the symbol update considerably improves the SER of the convex relaxation in the large SNR regime. We also reach the MFB performance as the SNR gets reasonably large. Besides, the best performance corresponds to the tightest relaxation. The convex hull relaxation performs better than MMSE which itself outperforms the Zero-Forcing. 167 Chapter 9 A PRECISE ANALYSIS OF PHASEMAX IN PHASE RETRIEVAL 9.1 Introduction The1 fundamental problem of recovering a signal from magnitude-only measurements is known as phase retrieval. This problem has a rich history and occurs in many areas in engineering and applied physics such as astronomical imaging [69], Xray crystallography [116], medical imaging [55], and optics [191]. In most of these cases, measuring the phase is either expensive or even infeasible. For instance, in some optical settings, detection devices like CCD cameras and photosensitive films cannot measure the phase of a light wave and instead measure the photon flux. Reconstructing a signal from magnitude-only measurements is generally very difficult due to loss of important phase information. Therefore, phase retrieval faces fundamental theoretical and algorithmic challenges and a variety of methods were suggested [86]. Convex methods have recently gained significant attention to solve the phase retrieval problem. These methods are mainly based on semidefinite programming by linearizing the resulting quadratic constraints using the idea of lifting [11, 13, 36, 77, 87, 88, 128, 144, 190]. Due to the convex nature of their formulation, these algorithms usually have rigorous theoretical guarantees. However, semidefinite relaxation squares the number of unknowns which makes these algorithms computationally complex, especially in large systems. This caveat makes these approaches intractable in real-world applications. Introduced in two independent works [12, 78], PhaseMax is a recently proposed convex formulation for the phase retrieval problem in the original 𝑛−dimensional parameter space. This method maximizes a linear functional over a convex feasible set. The constrained set in this optimization is obtained by relaxing the non-convex equality constraints in the original phase retrieval problem to convex inequality constraints. To form the objective function, PhaseMax relies on an initial estimate of the true signal which must be externally provided. The simple formulation of the PhaseMax method makes it appealing for practical applications. In addition, existing theoretical analysis indicates this method achieves perfect recovery for a nearly optimal number of random measurements. The analysis 1 This chapter is mainly based on the work in [142] 168 in [12, 78, 82] suggests that 𝑚 > 𝐶𝑛, where 𝐶 is a constant that depends on the quality of initial estimate (xinit ), is the sufficient number of measurements for perfect signal reconstruction when the measurement vectors are drawn independently from the Gaussian distribution. The exact phase transition threshold, i.e. the exact value of the constant 𝐶, for the real PhaseMax has been recently derived in [52, 53]. However, for the practical case of complex signals, previous results could only provide an upper bound on 𝐶. In this paper, we characterize the phase transition regimes for the perfect signal recovery in the PhaseMax algorithm. Our result is asymptotic and assumes that the measurement vectors are derived independently from Gaussian distribution. To the extent of our knowledge, this is the first work that computes the exact phase transition bound of the (complex-valued) PhaseMax in phase retrieval. In our analysis, we utilize the recently developed Convex Gaussian Min-max Theorem (CGMT) [165] which uses Gaussian process methods. CGMT has been successfully applied in a number of different problems including the performance analysis of structured signal recovery in M-estimators [4, 165], massive MIMO [1, 166] and etc. CGMT has been also used by Dhifallah et. al. [53] to analyze the real version of the PhaseMax. But unfortunately, the complex case does not directly fit into the framework of CGMT. Therefore, in this paper we introduce a secondary optimization that provably has the same phase transition bounds as PhaseMax and that also can be analyzed by CGMT. The organization of the chapter is as follows. In section 9.2 we introduce the main notations and mathematically setup the problem. In section 9.3, we present our main result followed by discussions and the result of numerical simulations. Finally, section 9.4 includes an outline of the proof of the main theorem. 9.2 Problem Setup Notations We gather here the basic notations that are used throughout this paper. We reserve the letter 𝑗 for the complex unit. For a complex scalar 𝑥 ∈ C,q𝑥 Re and 𝑥 Im correspond to 2 + 𝑥 2 . N (𝜇, 𝜎 2 ) dethe real and imaginary parts of 𝑥, respectively, and |𝑥| = 𝑥 Re Im notes real Gaussian distribution with mean 𝜇 and variance 𝜎 2 . Similarly, NC (𝜇, 𝜎 2 ) refers to a complex Gaussian distribution with real and imaginary parts drawn independently from N (𝜇Re , 𝜎 2 /2) and N (𝜇Im , 𝜎 2 /2), respectively. R (2𝜎 2 ) denotes the Rayleigh distribution with second moment equal to 2𝜎 2 . 𝑋 ∼ 𝑝 𝑋 implies that the 169 random variable 𝑋 has a density 𝑝 𝑋 . Bold lower letters are reserved for vectors and upper letters are used for matrices. For a vector v, 𝑣 𝑖 denotes its 𝑖 th entry and ||v|| is its 𝑙2 norm. (·)★ is used to denote the conjugate transpose. For a complex vector v, vRe and vIm denotes its real and complex parts, respectively. Also, v(𝑘 : 𝑙) is a column vector consisting of entries with index from 𝑘 to 𝑙 of v. We use caligraphy letters for sets. For set S, cone(S) is the closed conical hull of S. Setup Let x0 ∈ C𝑛 denote the underlying signal. We consider the phase retrieval problem with the goal of recovering x0 from 𝑚 magnitude-only measurements of the form, 𝑏𝑖 = |a★𝑖x0 |, 𝑖 = 1, . . . , 𝑚. (9.1) 𝑚 is the set of known measureThroughout this paper we assume that {a𝑖 ∈ C𝑛 }𝑖=1 ment vectors where the a𝑖 ’s are independently drawn from the complex Gaussian distribution with mean zero and covariance matrix I. As mentioned earlier, the PhaseMax method relies on an initial estimate of the true signal. xinit ∈ C𝑛 is used to represent this initial guess. We assume both x0 and xinit are independent of all the measurement vectors. The PhaseMax algorithm provides a convex formulation of the phase retrieval problem by simply relaxing the equality constraints in (9.1) into convex inequality constraints. This results in the following convex optimization problem: 𝑥ˆ = arg max𝑛 Re{xinit★ x} x∈C subject to: (9.2) |a★𝑖x| ≤ 𝑏𝑖 , 1 ≤ 𝑖 ≤ 𝑚. This optimization searches for a feasible vector that posses the most real correlation with xinit . Note that because of the global phase ambiguity of the measurements in (9.1), we can estimate x0 up to a global phase. Therefore, we define the following performance measure for the PhaseMax method, k 𝑥𝑒 ˆ 𝑗 𝜙 − x0 k . 𝜙∈[−𝜋,𝜋] kx0 k Dist( 𝑥, ˆ x0 ) = min (9.3) Under this setting, a perfect recovery of x0 means Dist( 𝑥, ˆ x0 ) = 0. In this paper we investigate the necessary and sufficient conditions under which the optimization program (9.2) perfectly recovers the true signal. 170 9.3 Main Result In this section, we present the main result of the paper which provides us with the necessary and sufficient number of measurements for the perfect recovery of the PhaseMax method in (9.2) under different scenarios. Our result is asymptotic which assumes a fixed oversampling ratio 𝛿 := 𝑚𝑛 ∈ [0, ∞), while 𝑛 → ∞. In theorem 20, we introduce 𝛿rec which depends on the problem parameters and prove that the condition 𝛿 > 𝛿rec , is necessary and sufficient for perfect recovery. Our result reveals significant dependence between 𝛿rec and the quality of the initial guess. We use the following similarity measure to quantify the caliber of the initial estimate: Re(𝑒 𝑗 𝜙 x★init x0 ) . 𝜌init := max 0≤𝜙<2𝜋 ||x0 || ||xinit || (9.4) Note that the multiplication by a unit amplitude scalar in the above definition is due to the global phase ambiguity of the phase retrieval solution (the true phase of x0 is dissolved in the absolute value in (9.1)). Therefore, for convenience we assume both xinit and x0 are aligned unit norm vectors (||x0 || = ||xinit || = 1), which results in 𝜌init = x★init x0 . We also define 𝜃 as the angle between xinit and x0 , and therefore, 𝜌init = cos 𝜃. We now present the main result of the paper which characterizes the phase transition regimes of PhaseMax for perfect recovery, in terms of 𝛿 and 𝜌init . Theorem 20 Consider the PhaseMax problem defined in section 9.2. For a fixed oversampling ratio 𝛿 = 𝑚𝑛 > 4, the optimization program (9.2) perfectly recovers the true signal (in the sense that lim𝑛→∞ P(Dist( 𝑥, ˆ x0 ) > 𝜖) = 0, for any fixed 𝜖 > 0) if and only if, 𝛿 > 𝛿rec := where 𝜌init is defined in (9.4). 4 4 = , 2 cos2 𝜃 𝜌init (9.5) Theorem 20 establishes a sharp phase transition behavior for the performance of PhaseMax. The inequality (9.5) can also be rewritten in terms of 𝜃 (or 𝜌init ) when the oversampling ratio, 𝛿, is fixed, r 𝜌init = cos 𝜃 > 4 . 𝛿 (9.6) The proof of Theorem 20 consists of two main steps. First, we introduce a real optimization program with 2𝑛 − 1 variables and prove that it has the same phase 171 transition bounds as PhaseMax in (9.2). The point of this step is that this new real optimization is especially built in a way that its performance can be precisely analyzed using well known tools like CGMT. Therefore, the next step would be to apply the CGMT framework to the new real optimization and to derive its phase transition bounds. We postpone a detailed version of the proof to section 9.4. Remark 29 The condition 𝛿 > 4 is proven to be fundamentally necessary for the phase retrieval problem under generic measurements to have a unique solution [45]. This is consistent with Theorem 20 where you can observe that even in the best scenario where xinit is aligned with x0 , we still need 𝑚 > 4𝑛 measurements for PhaseMax to have x0 as the solution. On the other hand, in the case where xinit carries no information about x0 (xinit is orthogonal to x0 ), recovery of x0 by PhaseMax is not guaranteed regardless of the number of measurements. 4 is Remark 30 It is shown in the work of Goldstein et. al. [78] that 𝛿 > 1−2𝜃/𝜋 sufficient for perfect recovery of x0 . This bound is compared to our result in Fig. 9.1 which shows phase transition regions of PhaseMax derived from empirical results. Although the simulations are run on the signals of size 𝑛 = 128, one can see that the blue line that comes from Theorem 20, perfectly predicts phase transition boundary. 9.4 Proof Outline In this part we introduce the main ideas used in the proof of Theorem 20. As mentioned earlier in section 9.3, we assume x0 is a unit norm vector aligned with xinit . Due to rotational invariance of the Gaussian distribution, without loss of generality, we assume x0 = e1 , the first vector of the standard basis in C𝑛 . Furthermore, the optimization program (9.2) is scalar invariant. So, we can assume kxinit k = 1. The proof consists of two main steps: In the first step, we analyze the complex optimization problem (9.2) and find the necessary and sufficient condition under which x̂ = x0 . Consequently, we use this condition to build an equivalent real optimization problem. Lemma 24 introduces this equivalent real optimization ERO, in R2𝑛−1 , and states that the perfect recovery in the PhaseMax algorithm occurs if and only if zero is the unique minimizer of the ERO. In the second step, we adopt the CGMT framework to analyze the ERO and investigate the conditions on 𝜌init (or 𝜃) under which the unique answer to the ERO is 0. Therefore ,as a result of Lemma 24, these conditions will guarantee the perfect recovery in the initial PhaseMax optimization (9.2). 172 0 Theorem 1 GS Bound 0.1 0.2 (rad) 0.3 0.4 0.5 0.6 0.7 0.8 4 4.5 5 5.5 6 6.5 7 7.5 8 Figure 9.1: Phase transition regimes for the PhaseMax problem in terms of the oversampling ratio 𝛿 = 𝑚/𝑛 and 𝜃, the angle between x0 and xinit . For the empirical results, we used signals of size 𝑛 = 128. The data is averaged over 10 independent realization of the measurement vectors. The blue line indicates the sharp phase transition bounds derived in Theorem 20 and the red line comes from the results of [78], which is referred to as the GS Bound. Introducing the Real Optimization ERO We define the error vector w := x − x0 and rewrite (9.2) in terms of w, max w∈C𝑛 Re{xinit★ w} (9.7) subject to: |a★𝑖 (e1 + w)| ≤ 𝑏𝑖 , 1 ≤ 𝑖 ≤ 𝑚. For 𝑖 = 1, 2, . . . , 𝑚 , we use 𝜙𝑖 := phase(ai★x0 ) to define aligned measurement vectors ã𝑖 := 𝑒 𝑗 𝜙𝑖 a𝑖 . Therefore, we have, 𝑏𝑖 = ã★𝑖x0 = ( ã𝑖 )1 , for 𝑖 = 1, 2, . . . , 𝑚 , (9.8) where ( ã𝑖 )1 is the first entry of ã𝑖 . Let D := {w ∈ C𝑛 : Re{x★init w} ≥ 0} be the set of all vectors w with nonnegative objective value and F := {w ∈ C𝑛 : |a★𝑖 (e1 + w)| ≤ 𝑏𝑖 , for 𝑖 = 1, 2, . . . , 𝑚} be the feasible set of the optimization problem (9.7). The 173 following lemmas prove necessary and sufficient conditions for perfect recovery in PhaseMax, based on these notations. Lemma 21 x0 is the unique optimal solution of (9.2) if and only if D Ñ F = {0}. Ñ Proof 13 For w ∈ D F , x0 + w is a solution of (9.2) with an objective value Ñ greater than the value for x0 . Therefore, D F = {0} is equivalent to x0 be a local minimum of (9.2) which is also a global minimum due to convexity of (9.2). Lemma 22 D Ñ F = {0} if and only if D Ñ cone(F ) = {0}. Proof 14 Note that D ⊂ C𝑛 is a convex cone and F ⊂ C𝑛 is a convex set. The proof is the consequence of the following equality, Ù Ù D cone(F ) = cone(D F ). Lemma 23 cone(F ) = Ñ𝑚 𝑖=1 {w ∈ C 𝑛 : Re{ ã★ w} ≤ 0}. 𝑖 Proof 15 Let d ∈ F , |𝑏𝑖 + ã★𝑖d| ≤ 𝑏𝑖 , for 𝑖 = 1, 2, . . . , 𝑚. (9.9) Therefore, Re{ã★𝑖d} = Re{𝑏𝑖 + ã★𝑖d} − 𝑏𝑖 , ≤ |𝑏𝑖 + ã★𝑖d| − 𝑏𝑖 , (9.10) ≤0. Ñ𝑚 This shows that cone(F ) ⊆ 𝑖=1 {w ∈ C𝑛 : Re{ã★𝑖w} ≤ 0}. To show the other direction, choose d ∈ C𝑛 such that: Re{ã★𝑖d} < 0, for 𝑖 = 1, 2, . . . , 𝑚. One can show that there exists 𝑅 > 0, such that for all 𝑟 ≤ 𝑅, 𝑟d ∈ F . Therefore, d ∈ cone(F ). This concludes the proof. We have the following corollary as a result of Lemma 21, Lemma 22, and Lemma 23. Corollary 11 x0 is the unique optimal solution of (9.2) if and only if, {w : Re{x★init w} ≥ 0 Re{ã★𝑖w} ≤ 0, for 1 ≤ 𝑖 ≤ 𝑚} = {0}. (9.11) 174 We are now ready to establish the equivalent real optimization ERO. We will show that the ERO has the exact phase transition bounds as PhaseMax in (9.2). max w 0 ∈R2𝑛−1 subject to: 𝜂𝑇 w0 |a0𝑇𝑖 (e1 + w0)| ≤ 𝑏𝑖 , (9.12) 1 ≤ 𝑖 ≤ 𝑚, 𝑚 are (2𝑛 − 1) where e1 is the first vector of the standard basis in R2𝑛−1 , 𝜂 and {a0𝑖 }𝑖=1 dimensional real vectors defined as, " # " # Re{xinit } Re{ ã } 𝑖 𝜂 := and a0𝑖 := , ∀𝑖. (9.13) −Im{xinit (2 : 𝑛)} −Im{ã𝑖 (2 : 𝑛)} Here Im{ã𝑖 (2 : 𝑛)} is the imaginary part of the last 𝑛 − 1 entries of ã𝑖 . We conclude this step of the proof with the following lemma: Lemma 24 x0 is the unique optimal solution of the PhaseMax method if and only if w0 = 0 is the unique optimal solution of (9.12). The proof of Lemma 24 is straightforward by defining " # Re{w} w0 = ∈ R2𝑛−1 , Im{w(2 : 𝑛)} (9.14) and then showing that the optimality conditions for w0 = 0 in (9.12) is equivalent to (9.11). It is worth mentioning that the result of Lemma 24 is valid for any set of measurement vectors {a𝑖 }. In the next part, we use this result to compute the phase transition of PhaseMax when the measurement vectors are drawn independently from the Gaussian distribution. Convex Gaussian Min-Max Theorem Our analysis is based on the recently developed Convex Gaussian Min-max Theorem (CGMT) [165]. The CGMT associates with a Primary Optimization (PO) problem an Auxiliary Optimization (AO) problem from which we can investigate various properties of the primary optimization, such as phase transitions. In particular, the (PO) and the (AO) problems are defined respectively as follows: Φ(G) := min max u𝑇 Gw + 𝜓(u, w), (9.15a) 𝜙(g, h) := min max kwkg𝑇 u − kukh𝑇 w + 𝜓(u, w), (9.15b) w∈Sw u∈Su w∈Sw u∈Su 175 where G ∈ R𝑚×𝑛 , g ∈ R𝑚 , h ∈ R𝑛 , Sw ⊂ R𝑛 , Su ⊂ R𝑚 and 𝜓 : R𝑛 × R𝑚 → R. Denote wΦ := wΦ (G) and w𝜙 := w𝜙 (g, h) any optimal minimizers in (9.15a) and (9.15b), respectively. The following lemma is a result of CGMT [165]. Lemma 25 Consider the two optimizations (9.15a) and (9.15b). Let Sw , Su be convex and compact sets, 𝜓 be continuous and convex-concave on Sw × Su , and, G, g and h all have entries iid standard normal. Suppose there exist 𝛼 such that in the limit of 𝑛 → ∞ it holds in probability that kw𝜙 (g, h)k → 𝛼. Then, the same holds for wΦ (G) and we have kwΦ (G)k → 𝛼. In the next section, first we will rewrite the ERO in the form of the optimization (9.15a). This enables us to apply Lemma 25 to the ERO and derive an Auxiliary Optimization in the form of (9.15b). This lemma indicates that if kw𝜙 (g, h)k → 0 for the (AO), then kwΦ (G)k → 0 for the ERO and we have perfect recovery. (AO) can be analyzed using the conventional concentration results in high dimensions. Computing the Phase Transition for PhaseMax In this part we adopt the CGMT framework along with the result of Lemma 24 to compute the exact phase transition of the PhaseMax algorithm under the Gaussian measurement scheme. We start by calculating the distribution of the entries of a0𝑖 that are defined in (9.13). Recall that a𝑖 ’s are independently drawn from the complex Gaussian distribution with mean zero and covariance matrix I. Therefore, the distribution of the entries of ã𝑖 ’s that were defined in section 9.4, is as follows: 1. The first entry of ã𝑖 is the absolute value of the first entry of the a𝑖 . Therefore, it has a Rayleigh distribution, i.e., ( ã𝑖 )1 ∼ R (1), (9.16) 2. The remaining entries of ã𝑖 remain standard Gaussian random variables, ( ã𝑖 )𝑘 ∼ NC (0, 1), for 2 ≤ 𝑘 ≤ 𝑛 , (9.17) 3. The entries of ã𝑖 remain independent. This implies that all the entries of a0𝑖 are independent, the first entry of a0𝑖 has a R (1) distribution and the rest of the entries have Gaussian distribution N (0, 12 ). We form 176 the measurement matrix A ∈ R𝑚×(2𝑛−1) by stacking vectors {ai𝑇 , 1 ≤ 𝑖 ≤ 𝑚}. Let A1 ∈ R𝑚 be the first column of A, and à ∈ R𝑚×(2𝑛−1) be the remaining part (i.e., A = [A1 Ã]). x0 = e1 implies that A1 = [𝑏 1 , 𝑏 2 , . . . , 𝑏 𝑚 ] 𝑇 , where 𝑏𝑖 ’s are defined in (9.1). Using the Lagrange multipliers, we can reformulate (9.12) as the following minmax program, min max − 𝜂𝑇 w + (𝜆 − 𝜇)𝑇 Ãw̃ 𝑤 1 ∈R 𝜆,𝜇∈R+𝑚 w̃∈R2𝑛−2
− (𝜆 + 𝜇)𝑇 A1 + (𝜆 − 𝜇)𝑇 A1 (1 + 𝑤 1 ) , (9.18) where 𝑤 1 denotes the first entry of w and w̃ is the remaining part. Define v := 𝜆 − 𝜇 . It can be shown that optimal values of (9.18) satisfy 𝜆 + 𝜇 = |𝜆 − 𝜇|. Here, | · | denotes the component-wise absolute value. Therefore, (9.18) can be rewritten as an optimization over v ∈ R𝑚 and w ∈ R2𝑛−1 in the following form: min max − 𝜂𝑇 w + v𝑇 Ãw̃ + v𝑇 A1 (1 + 𝑤 1 ) − |v|𝑇 A1 . 𝑤 1 ∈R v∈R𝑚 w̃∈R2𝑛−2 (9.19) Note that à has i.i.d. standard normal entries. One can check that (9.19) satisfies the condition of Lemma 25. Hence, we can form the (AO) as follows, min max − 𝜂𝑇 w + v𝑇 g|| w̃|| + ||v||h𝑇 w̃ 𝑤 1 ∈R v∈R𝑚 w̃∈R2𝑛−2 (9.20) 𝑇 𝑇 + v A1 (1 + 𝑤 1 ) − |v| A1 , where g ∈ R𝑚 and h ∈ R2𝑛−2 with entries drawn independently from standard normal distribution. Analysis of (9.20) is similar to [53]. Due to lack of space, we defer technical details to the full version of the paper. We conclude the paper with a theorem that characterizes the performance of the ERO. Let w∗ be the optimizer of (9.20). Define 𝑠∗ := 1 + 𝑤 ∗1 and 𝑡 ∗ := || w̃∗ ||. Theorem 21 In the asymptotic regime where 𝑚, 𝑛 → ∞, and 𝛿 := 𝑚𝑛 , 𝑠∗ and 𝑡 ∗ converges to the solution of the following deterministic optimization, r q 𝛿 max 𝜌init 𝑠 + 1 − 𝜌init 2 𝑡 2 − 𝑝(𝑡, 𝑠) 2 𝑠∈[−1,1], 𝑡≥0 (9.21) 2 2𝑡 . subject to: 𝑝(𝑡, 𝑠) ≤ 𝛿 177 In the above optimization, 𝑝(𝑡, 𝑠) is define as, p 𝑝(𝑡, 𝑠) =𝑡 2 + (1 + 𝑠) [1 + 𝑠 − 𝑡 2 + (1 + 𝑠) 2 ] p + (1 − 𝑠) [1 − 𝑠 − 𝑡 2 + (1 − 𝑠) 2 ] (9.22) It can be shown that 𝜌init > √2 is the necessary and sufficient condition for (𝑡 ∗ , 𝑠∗ ) = 𝛿 (0, 1) to be the unique solution of (9.21) which is equivalent to the perfect recovery in the ERO. 178 C h a p t e r 10 CONCLUSION AND FUTURE WORK We will conclude with some brief remarks n the results of this thesis, and mentioning a few related directions that worth further exploration. The universality results we showed were for a few special, but useful and practical cases, and does not always hold. For instance, in the last chapter we observe that the phase transition we achieve for complex phase retrieval with phase-max, is not simply equivalent to the case where we replace the complex Gaussian matrix, with its corresponding real Gaussian matrix, as we did for massive MIMO. Also in Chapter 6, we have observed that the assumptions we mentioned for the regularizer are necessary. For instance, universality will not hold by choosing a nuclear norm as the regularizer, without being constraint to PSD matrices. Note that if we are constrained to PSD matrices, the nuclear norm simply becomes the trace function, which satisfies the assumptions of the Chapter 6. The precise analysis we proposed in Chapter 2, paved the way for a few possible future directions. Calculating the best regularizer parameter 𝜆 and choosing the right loss function and regularizer based on the problem parameters are two interesting directions that we would like to consider as future work. Besides, the universality results enables us to go further, and analyze the best designs for the feature matrices X. This can be helpful in the applications that we can manipulate the features matrix for the best performance. Another area that has not been investigated yet, is the analysis of the count data models, such as Poisson regression. These models are very popular with tens of applications in telecommunications (number of arriving calls in a system), Biology (number of mutations on a DNA), Finance and insurance (number of losses or claims in a period of time), etc.. Using the results of Chapter 2, we can easily analyze their performance and derive results on consistency of count data models under various conditions. 179 Finally, new applications of generalized linear models and also non-linear models, always leads to new research directions where CGMT and such universality results framework are applicable. We hope that this thesis was a practical guide in using CGMT framework and corresponding universality results to those new directions. 180 BIBLIOGRAPHY [1] Abbasi, E., Salehi, F., and Hassibi, B. (2019a). Performance analysis of convex data detection in mimo. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4554–4558. IEEE. [2] Abbasi, E., Salehi, F., and Hassibi, B. (2019b). Sparse covariance estimation from quadratic measurements: A precise analysis. In 2019 IEEE International Symposium on Information Theory (ISIT), pages 2074–2078. IEEE. [3] Abbasi, E., Salehi, F., and Hassibi, B. (2019c). Universality in learning from linear measurements. In Advances in Neural Information Processing Systems, pages 12372–12382. [4] Abbasi, E., Thrampoulidis, C., and Hassibi, B. (2016). General performance metrics for the lasso. In Information Theory Workshop (ITW), 2016 IEEE, pages 181–185. IEEE. [5] Ait-Sahalia, Y., Fan, J., and Xiu, D. (2010). High-frequency covariance estimates with noisy and asynchronous financial data. Journal of the American Statistical Association, 105(492):1504–1517. [6] Amelunxen, D., Lotz, M., McCoy, M. B., and Tropp, J. A. (2014). Living on the edge: Phase transitions in convex programs with random data. Information and Inference: A Journal of the IMA, 3(3):224–294. [7] Andersen, P. K. and Gill, R. D. (1982). Cox’s regression model for counting processes: a large sample study. The annals of statistics, pages 1100–1120. [8] Ariananda, D. D. and Leus, G. (2012). Compressive wideband power spectrum estimation. IEEE Transactions on signal processing, 60(9):4775–4789. [9] Atitallah, I. B., Thrampoulidis, C., Kammoun, A., Al-Naffouri, T. Y., Hassibi, B., and Alouini, M.-S. (2017). Ber analysis of regularized least squares for bpsk recovery. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pages 4262–4266. IEEE. [10] Bach, F. R. (2010). Structured sparsity-inducing norms through submodular functions. In Advances in Neural Information Processing Systems, pages 118– 126. [11] Bahmani, S. and Romberg, J. (2015). Efficient compressive phase retrieval with constrained sensing vectors. In Advances in Neural Information Processing Systems, pages 523–531. [12] Bahmani, S. and Romberg, J. (2016). Phase retrieval meets statistical learning theory: A flexible convex relaxation. arXiv preprint arXiv:1610.04210. 181 [13] Balan, R., Casazza, P., and Edidin, D. (2006). On signal reconstruction without phase. Applied and Computational Harmonic Analysis, 20(3):345–356. [14] Banerjee, A., Chen, S., Fazayeli, F., and Sivakumar, V. (2014). Estimation with norm regularization. In Advances in Neural Information Processing Systems, pages 1556–1564. [15] Baraniuk, R. G., Cevher, V., and Wakin, M. B. (2010). Low-dimensional models for dimensionality reduction and signal recovery: A geometric perspective. Proceedings of the IEEE, 98(6):959–971. [16] Bartlett, P. L., Long, P. M., Lugosi, G., and Tsigler, A. (2019). Benign overfitting in linear regression. arXiv preprint arXiv:1906.11300. [17] Bayati, M., Lelarge, M., Montanari, A., et al. (2015). Universality in polytope phase transitions and message passing algorithms. The Annals of Applied Probability, 25(2):753–822. [18] Bayati, M. and Montanari, A. (2012). The lasso risk for gaussian matrices. Information Theory, IEEE Transactions on, 58(4):1997–2017. [19] Bean, D., Bickel, P. J., El Karoui, N., and Yu, B. (2013). Optimal m-estimation in high-dimensional regression. Proceedings of the National Academy of Sciences, 110(36):14563–14568. [20] Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849–15854. [21] Belkin, M., Hsu, D. J., and Mitra, P. (2018). Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. In Advances in neural information processing systems, pages 2300–2311. [22] Belloni, A., Chernozhukov, V., and Wang, L. (2011). Square-root lasso: pivotal recovery of sparse signals via conic programming. Biometrika, 98(4):791–806. [23] Ben-Tal, A. and Nemirovski, A. (2001). Lectures on modern convex optimization: analysis, algorithms, and engineering applications, volume 2. Siam. [24] Bickel, P. J. and Levina, E. (2008). Regularized estimation of large covariance matrices. The Annals of Statistics, pages 199–227. [25] Billingsley, P. (1979). Probability and measure. N.Y.: Wiley. [26] Boche, H., Calderbank, R., Kutyniok, G., and Vybíral, J. (2015). Compressed sensing and its applications. Springer. [27] Boyd, C. R., Tolson, M. A., and Copes, W. S. (1987). Evaluating trauma care: the triss method. trauma score and the injury severity score. The Journal of trauma, 27(4):370–378. 182 [28] Boyd, S. and Vandenberghe, L. (2004). Convex optimization. Cambridge university press. [29] Bradic, J. and Chen, J. (2015). Robustness in sparse linear models: relative efficiency based on robust approximate message passing. arXiv preprint arXiv:1507.08726. [30] Bunea, F. et al. (2008). Honest variable selection in linear and logistic regression models via l1 and l1+ l2 penalization. Electronic Journal of Statistics, 2:1153–1194. [31] Cai, T. T., Zhang, A., et al. (2015). Rop: Matrix recovery via rank-one projections. The Annals of Statistics, 43(1):102–138. [32] Cai, T. T., Zhang, C.-H., Zhou, H. H., et al. (2010). Optimal rates of convergence for covariance matrix estimation. The Annals of Statistics, 38(4):2118– 2144. [33] Candes, E. J. (2008). The restricted isometry property and its implications for compressed sensing. Comptes rendus mathematique, 346(9-10):589–592. [34] Candès, E. J. and Recht, B. (2009). Exact matrix completion via convex optimization. Foundations of Computational mathematics, 9(6):717. [35] Candes, E. J., Romberg, J. K., and Tao, T. (2006). Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 59(8):1207–1223. [36] Candes, E. J., Strohmer, T., and Voroninski, V. (2013). Phaselift: Exact and stable signal recovery from magnitude measurements via convex programming. Communications on Pure and Applied Mathematics, 66(8):1241–1274. [37] Candès, E. J. and Sur, P. (2018). The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression. arXiv preprint arXiv:1804.09753. [38] Candes, E. J. and Tao, T. (2006). Near-optimal signal recovery from random projections: Universal encoding strategies? IEEE transactions on information theory, 52(12):5406–5425. [39] Chandrasekaran, V., Parrilo, P. A., and Willsky, A. S. (2010). Latent variable graphical model selection via convex optimization. In 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1610– 1613. IEEE. [40] Chandrasekaran, V., Recht, B., Parrilo, P. A., and Willsky, A. S. (2012). The convex geometry of linear inverse problems. Foundations of Computational mathematics, 12(6):805–849. 183 [41] Charles, J., Ramina, G., Arian, M., and Christoph, S. (2015). Optimality of large mimo detection via approximate message passing. In Information Theory (ISIT), 2015 IEEE International Symposium on. IEEE. [42] Chen, S. S., Donoho, D. L., and Saunders, M. A. (2001). Atomic decomposition by basis pursuit. SIAM review, 43(1):129–159. [43] Chen, Y., Chi, Y., and Goldsmith, A. J. (2014). Estimation of simultaneously structured covariance matrices from quadratic measurements. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7669–7673. IEEE. [44] Chen, Y., Chi, Y., and Goldsmith, A. J. (2015). Exact and stable covariance estimation from quadratic sampling via convex programming. IEEE Transactions on Information Theory, 61(7):4034–4059. [45] Conca, A., Edidin, D., Hering, M., and Vinzant, C. (2015). An algebraic characterization of injectivity in phase retrieval. Applied and Computational Harmonic Analysis, 38(2):346–356. [46] Cortes, C. and Vapnik, V. (1995). Support-vector networks. Machine learning, 20(3):273–297. [47] Cover, T. M. (1965). Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE transactions on electronic computers, pages 326–334. [48] Dasarathy, G., Shah, P., Bhaskar, B. N., and Nowak, R. D. (2015). Sketching sparse matrices, covariances, and graphs via tensor products. IEEE Transactions on Information Theory, 61(3):1373–1388. [49] d’Aspremont, A., Banerjee, O., and El Ghaoui, L. (2008). First-order methods for sparse covariance selection. SIAM Journal on Matrix Analysis and Applications, 30(1):56–66. [50] Davenport, M. A., Duarte, M. F., Eldar, Y. C., and Kutyniok, G. (2011). Introduction to compressed sensing. Preprint, 93:1–64. [51] Deng, Z., Kammoun, A., and Thrampoulidis, C. (2019). A model of double descent for high-dimensional binary linear classification. arXiv preprint arXiv:1911.05822. [52] Dhifallah, O. and Lu, Y. M. (2017). Fundamental limits of phasemax for phase retrieval: A replica analysis. arXiv preprint arXiv:1708.03355. [53] Dhifallah, O., Thrampoulidis, C., and Lu, Y. M. (2017). Phase retrieval via linear programming: Fundamental limits and algorithmic improvements. In 2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1071–1077. IEEE. 184 [54] Dhifallah, O., Thrampoulidis, C., and Lu, Y. M. (2018). Phase retrieval via polytope optimization: Geometry, phase transitions, and new algorithms. arXiv preprint arXiv:1805.09555. [55] Dierolf, M., Menzel, A., Thibault, P., Schneider, P., Kewish, C. M., Wepf, R., Bunk, O., and Pfeiffer, F. (2010). Ptychographic x-ray computed tomography at the nanoscale. Nature, 467(7314):436–439. [56] Donoho, D. and Montanari, A. (2016). High dimensional robust m-estimation: Asymptotic variance via approximate message passing. Probability Theory and Related Fields, 166(3-4):935–969. [57] Donoho, D. and Tanner, J. (2009a). Counting faces of randomly projected polytopes when the projection radically lowers dimension. Journal of the American Mathematical Society, 22(1):1–53. [58] Donoho, D. and Tanner, J. (2009b). Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 367(1906):4273–4293. [59] Donoho, D. L. (2006). High-dimensional centrally symmetric polytopes with neighborliness proportional to dimension. Discrete & Computational Geometry, 35(4):617–652. [60] Donoho, D. L. et al. (2000). High-dimensional data analysis: The curses and blessings of dimensionality. AMS Math Challenges Lecture, pages 1–32. [61] Donoho, D. L. et al. (2006). Compressed sensing. IEEE Transactions on information theory, 52(4):1289–1306. [62] Donoho, D. L., Maleki, A., and Montanari, A. (2009). Message-passing algorithms for compressed sensing. Proceedings of the National Academy of Sciences, 106(45):18914–18919. [63] Donoho, D. L., Maleki, A., and Montanari, A. (2011). The noise-sensitivity phase transition in compressed sensing. Information Theory, IEEE Transactions on, 57(10):6920–6941. [64] Donoho, D. L. and Montanari, A. (2015). Variance breakdown of huber (m)estimators: n/p \rightarrow m\in (1,\infty). arXiv preprint arXiv:1503.02106. [65] El Karoui, N. (2018). On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators. Probability Theory and Related Fields, 170(1-2):95–175. [66] El Karoui, N., Bean, D., Bickel, P. J., Lim, C., and Yu, B. (2013). On robust regression with high-dimensional predictors. Proceedings of the National Academy of Sciences, 110(36):14557–14562. 185 [67] El Karoui, N. et al. (2008). Operator norm consistent estimation of largedimensional sparse covariance matrices. The Annals of Statistics, 36(6):2717– 2756. [68] Eldar, Y. C. and Kutyniok, G. (2012). Compressed sensing: theory and applications. Cambridge University Press. [69] Fienup, C. and Dainty, J. (1987). Phase retrieval and image reconstruction for astronomy. Image Recovery: Theory and Application, pages 231–275. [70] Fincke, U. and Pohst, M. (1985). Improved methods for calculating vectors of short length in a lattice, including a complexity analysis. Mathematics of computation, 44(170):463–471. [71] Foschini, G. J. (1996). Layered space-time architecture for wireless communication in a fading environment when using multi-element antennas. Bell labs technical journal, 1(2):41–59. [72] Foucart, S. and Rauhut, H. (2017). A mathematical introduction to compressive sensing. Bull. Am. Math, 54:151–165. [73] Foygel, R. and Mackey, L. (2014). Corrupted sensing: Novel guarantees for separating structured signals. IEEE Transactions on Information Theory, 60(2):1223–1247. [74] Friedman, J., Hastie, T., Höfling, H., Tibshirani, R., et al. (2007). Pathwise coordinate optimization. The Annals of Applied Statistics, 1(2):302–332. [75] Friedman, J., Hastie, T., and Tibshirani, R. (2008). Sparse inverse covariance estimation with the graphical lasso. Biostatistics, 9(3):432–441. [76] Furrer, R. and Bengtsson, T. (2007). Estimation of high-dimensional prior and posterior covariance matrices in kalman filter variants. Journal of Multivariate Analysis, 98(2):227–255. [77] Goemans, M. X. and Williamson, D. P. (1995). Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM), 42(6):1115–1145. [78] Goldstein, T. and Studer, C. (2016). Phasemax: Convex phase retrieval via basis pursuit. arXiv preprint arXiv:1610.07531. [79] Gordon, Y. (1985). Some inequalities for gaussian processes and applications. Israel Journal of Mathematics, 50(4):265–289. [80] Gordon, Y. (1988). On milman’s inequality and random subspaces which escape through a mesh in r n. In Geometric Aspects of Functional Analysis, pages 84–106. Springer. 186 [81] Grötschel, M., Lovász, L., and Schrijver, A. (2012). Geometric algorithms and combinatorial optimization, volume 2. Springer Science & Business Media. [82] Hand, P. and Voroninski, V. (2016). An elementary proof of convex phase retrieval in the natural parameter space via the linear program phasemax. arXiv preprint arXiv:1611.03935. [83] Hassibi, B. and Vikalo, H. (2005). On the sphere-decoding algorithm i. expected complexity. Signal Processing, IEEE Transactions on, 53(8):2806– 2818. [84] Hastie, T., Montanari, A., Rosset, S., and Tibshirani, R. J. (2019). Surprises in high-dimensional ridgeless least squares interpolation. arXiv preprint arXiv:1903.08560. [85] Hosmer Jr, D. W., Lemeshow, S., and Sturdivant, R. X. (2013). Applied logistic regression, volume 398. John Wiley & Sons. [86] Jaganathan, K., Eldar, Y. C., and Hassibi, B. (2015). Phase retrieval: An overview of recent developments. arXiv preprint arXiv:1510.07713. [87] Jaganathan, K., Oymak, S., and Hassibi, B. (2012). Recovery of sparse 1-d signals from the magnitudes of their fourier transform. In Information Theory Proceedings (ISIT), 2012 IEEE International Symposium On, pages 1473–1477. IEEE. [88] Jaganathan, K., Oymak, S., and Hassibi, B. (2013). Sparse phase retrieval: Convex algorithms and limitations. In Information Theory Proceedings (ISIT), 2013 IEEE International Symposium on, pages 1022–1026. IEEE. [89] James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013). An introduction to statistical learning, volume 112. Springer. [90] Jeon, C., Ghods, R., Maleki, A., and Studer, C. (2015). Optimality of large mimo detection via approximate message passing. In Information Theory (ISIT), 2015 IEEE International Symposium on, pages 1227–1231. IEEE. [91] Jeon, C., Maleki, A., and Studer, C. (2016). On the performance of mismatched data detection in large mimo systems. In Information Theory (ISIT), 2016 IEEE International Symposium on, pages 180–184. IEEE. [92] Johnstone, I. M. (2001). On the distribution of the largest eigenvalue in principal components analysis. Annals of statistics, pages 295–327. [93] Johnstone, I. M. and Lu, A. Y. (2004). Sparse principal components analysis. Unpublished manuscript, 7. 187 [94] Kakade, S., Shamir, O., Sindharan, K., and Tewari, A. (2010). Learning exponential families in high-dimensions: Strong convexity and sparsity. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 381–388. [95] Karoui, N. E. (2013). Asymptotic behavior of unregularized and ridgeregularized high-dimensional robust regression estimators: rigorous results. arXiv preprint arXiv:1311.2445. [96] King, G. and Zeng, L. (2001). Logistic regression in rare events data. Political analysis, 9(2):137–163. [97] Kini, G. and Thrampoulidis, C. (2020). Analytic study of double descent in binary classification: The impact of loss. arXiv preprint arXiv:2001.11572. [98] Koh, K., Kim, S.-J., and Boyd, S. (2007). An interior-point method for largescale l1-regularized logistic regression. Journal of Machine learning research, 8(Jul):1519–1555. [99] Krishnapuram, B., Carin, L., Figueiredo, M. A., and Hartemink, A. J. (2005). Sparse multinomial logistic regression: Fast algorithms and generalization bounds. IEEE transactions on pattern analysis and machine intelligence, 27(6):957–968. [100] Lam, C. and Fan, J. (2009). Sparsistency and rates of convergence in large covariance matrix estimation. Annals of statistics, 37(6B):4254. [101] Lecué, G. and Mendelson, S. (2014). Sparse recovery under weak moment assumptions. arXiv preprint arXiv:1401.2188. [102] Ledoit, O. and Wolf, M. (2004). A well-conditioned estimator for largedimensional covariance matrices. Journal of multivariate analysis, 88(2):365– 411. [103] Lehmann, E. L. and Romano, J. P. (2006). Testing statistical hypotheses. Springer Science & Business Media. [104] Li, X. and Voroninski, V. (2013). Sparse signal recovery from quadratic measurements via convex programming. SIAM Journal on Mathematical Analysis, 45(5):3019–3033. [105] Li, Y., Sun, Y., and Chi, Y. (2016). Low-rank positive semidefinite matrix recovery from corrupted rank-one measurements. IEEE Transactions on Signal Processing, 65(2):397–408. [106] Li, Y.-H., Hsieh, Y.-P., Zerbib, N., and Cevher, V. (2015). A geometric view on constrained m-estimators. arXiv preprint arXiv:1506.08163. [107] Liese, F. and Miescke, K.-J. (2008). Statistical decision theory: estimation, testing, and selection. Springer Science & Business Media. 188 [108] Ma, W.-K., Davidson, T. N., Wong, K. M., Luo, Z.-Q., and Ching, P.C. (2002). Quasi-maximum-likelihood multiuser detection using semi-definite relaxation with application to synchronous cdma. IEEE transactions on signal processing, 50(4):912–922. [109] Madiman, M. and Barron, A. (2007). Generalized entropy power inequalities and monotonicity properties of information. Information Theory, IEEE Transactions on, 53(7):2317–2329. [110] Mangasarian, O. L. and Recht, B. (2011). Probability of unique integer solution to a system of linear equations. European Journal of Operational Research, 214(1):27–30. [111] Mazumder, R. and Hastie, T. (2012). The graphical lasso: New insights and alternatives. Electronic journal of statistics, 6:2125. [112] McCoy, M. B., Cevher, V., Dinh, Q. T., Asaei, A., and Baldassarre, L. (2014). Convexity in source separation: Models, geometry, and algorithms. Signal Processing Magazine, IEEE, 31(3):87–95. [113] McCoy, M. B. and Tropp, J. A. (2014). Sharp recovery bounds for convex demixing, with applications. Foundations of Computational Mathematics, 14(3):503–567. [114] Mei, S. and Montanari, A. (2019). The generalization error of random features regression: Precise asymptotics and double descent curve. arXiv preprint arXiv:1908.05355. [115] Meinshausen, N. and Bühlmann, P. (2006). High-dimensional graphs and variable selection with the lasso. The annals of statistics, pages 1436–1462. [116] Millane, R. P. (1990). Phase retrieval in crystallography and optics. JOSA A, 7(3):394–411. [117] Miolane, L. and Montanari, A. (2018). The distribution of the lasso: Uniform control over sparse balls and adaptive parameter tuning. arXiv preprint arXiv:1811.01212. [118] Montanari, A., Ruan, F., Sohn, Y., and Yan, J. (2019). The generalization error of max-margin linear classifiers: High-dimensional asymptotics in the overparametrized regime. arXiv preprint arXiv:1911.01544. [119] Mousavi, A., Maleki, A., and Baraniuk, R. G. (2013). Parameterless optimal approximate message passing. arXiv preprint arXiv:1311.0035. [120] Muthukrishnan, S. et al. (2005). Data streams: Algorithms and applications. Foundations and Trends® in Theoretical Computer Science, 1(2):117–236. 189 [121] Narasimhan, T. L. and Chockalingam, A. (2014). Channel hardeningexploiting message passing (chemp) receiver in large-scale mimo systems. Selected Topics in Signal Processing, IEEE Journal of, 8(5):847–860. [122] Needell, D. and Tropp, J. A. (2009). Cosamp: Iterative signal recovery from incomplete and inaccurate samples. Applied and computational harmonic analysis, 26(3):301–321. [123] Negahban, S. N., Ravikumar, P., Wainwright, M. J., and Yu, B. (2012). A unified framework for high-dimensional analysis of 𝑚-estimators with decomposable regularizers. Statistical Science, 27(4):538–557. [124] Nelder, J. A. and Wedderburn, R. W. (1972). Generalized linear models. Journal of the Royal Statistical Society: Series A (General), 135(3):370–384. [125] Newey, W. K. and McFadden, D. (1994). Large sample estimation and hypothesis testing. Handbook of econometrics, 4:2111–2245. [126] Ngo, H. Q., Larsson, E. G., and Marzetta, T. L. (2013). Energy and spectral efficiency of very large multiuser mimo systems. Communications, IEEE Transactions on, 61(4):1436–1449. [127] Oymak, S. and Hassibi, B. (2010). New null space results and recovery thresholds for matrix rank minimization. arXiv preprint arXiv:1011.6326. [128] Oymak, S., Jalali, A., Fazel, M., Eldar, Y. C., and Hassibi, B. (2015). Simultaneously structured models with application to sparse and low-rank matrices. IEEE Transactions on Information Theory, 61(5):2886–2908. [129] Oymak, S., Thrampoulidis, C., and Hassibi, B. (2013). The squared-error of generalized lasso: A precise analysis. arXiv preprint arXiv:1311.0830. [130] Oymak, S. and Tropp, J. A. (2017). Universality laws for randomized dimension reduction, with applications. Information and Inference: A Journal of the IMA, 7(3):337–446. [131] Panahi, A. and Hassibi, B. (2017). A universal analysis of large-scale regularized least squares solutions. In Advances in Neural Information Processing Systems, pages 3381–3390. [132] Parikh, N., Boyd, S., et al. (2014). Proximal algorithms. Foundations and Trends® in Optimization, 1(3):127–239. [133] Plan, Y. and Vershynin, R. (2015). The generalized lasso with non-linear observations. arXiv preprint arXiv:1502.04071. [134] Ravikumar, P., Wainwright, M. J., Raskutti, G., Yu, B., et al. (2011). Highdimensional covariance estimation by minimizing ℓ1 -penalized log-determinant divergence. Electronic Journal of Statistics, 5:935–980. 190 [135] Recht, B., Fazel, M., and Parrilo, P. A. (2010). Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review, 52(3):471–501. [136] Rockafellar, R. T. (2015). Convex analysis. Princeton university press. [137] Rockafellar, R. T. and Wets, R. J.-B. (2009). Variational analysis, volume 317. Springer Science & Business Media. [138] Rothman, A. J., Bickel, P. J., Levina, E., Zhu, J., et al. (2008). Sparse permutation invariant covariance estimation. Electronic Journal of Statistics, 2:494–515. [139] Rudelson, M. and Vershynin, R. (2006). Sparse reconstruction by convex relaxation: Fourier and gaussian measurements. In 2006 40th Annual Conference on Information Sciences and Systems, pages 207–212. IEEE. [140] Rusek, F., Persson, D., Lau, B. K., Larsson, E. G., Marzetta, T. L., Edfors, O., and Tufvesson, F. (2013). Scaling up mimo: Opportunities and challenges with very large arrays. IEEE signal processing magazine, 30(1):40–60. [141] Salehi, F., Abbasi, E., and Hassibi, B. (2018a). Learning without the phase: Regularized phasemax achieves optimal sample complexity. In Advances in Neural Information Processing Systems, pages 8655–8666. [142] Salehi, F., Abbasi, E., and Hassibi, B. (2018b). A precise analysis of phasemax in phase retrieval. In 2018 IEEE International Symposium on Information Theory (ISIT), pages 976–980. IEEE. [143] Salehi, F., Abbasi, E., and Hassibi, B. (2019). The impact of regularization on high-dimensional logistic regression. In Advances in Neural Information Processing Systems, pages 11982–11992. [144] Salehi, F., Jaganathan, K., and Hassibi, B. (2017). Multiple illumination phaseless super-resolution (mips) with applications to phaseless super doa estimation and diffraction imaging. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pages 3949–3953. IEEE. [145] Scharf, L. L. and Friedlander, B. (1994). Matched subspace detectors. IEEE Transactions on signal processing, 42(8):2146–2157. [146] Serdobolskii, V. (2013). Multivariate statistical analysis: A high-dimensional approach, volume 41. Springer Science & Business Media. [147] Shechtman, Y., Beck, A., and Eldar, Y. C. (2014). Gespar: Efficient phase retrieval of sparse signals. IEEE transactions on signal processing, 62(4):928– 938. 191 [148] Shechtman, Y., Eldar, Y. C., Szameit, A., and Segev, M. (2011). Sparsity based sub-wavelength imaging with partially incoherent light via quadratic compressed sensing. Optics express, 19(16):14807–14822. [149] Sion, M. et al. (1958). On general minimax theorems. Pacific Journal of mathematics, 8(1):171–176. [150] Sivakumar, V., Banerjee, A., and Ravikumar, P. K. (2015). Beyond subgaussian measurements: High-dimensional structured estimation with subexponential designs. In Advances in Neural Information Processing Systems, pages 2197–2205. [151] Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. (2018). The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822–2878. [152] Stojnic, M. (2009a). Block-length dependent thresholds in block-sparse compressed sensing. arXiv preprint arXiv:0907.3679. [153] Stojnic, M. (2009b). Various thresholds for ℓ1 -optimization in compressed sensing. arXiv preprint arXiv:0907.3666. [154] Stojnic, M. (2013a). A framework to characterize performance of lasso algorithms. arXiv preprint arXiv:1303.7291. [155] Stojnic, M. (2013b). A framework to characterize performance of lasso algorithms. arXiv preprint arXiv:1303.7291. [156] Stojnic, M. (2013c). Upper-bounding l1-optimization weak thresholds. arXiv preprint arXiv:1303.7289. [157] Sur, P. and Candès, E. J. (2018). A modern maximum-likelihood theory for high-dimensional logistic regression. arXiv preprint arXiv:1803.06964. [158] Sur, P. and Candès, E. J. (2019). A modern maximum-likelihood theory for high-dimensional logistic regression. Proceedings of the National Academy of Sciences, 116(29):14516–14525. [159] Sur, P., Chen, Y., and Candès, E. J. (2017). The likelihood ratio test in highdimensional logistic regression is asymptotically a rescaled chi-square. Probability Theory and Related Fields, pages 1–72. [160] Taeb, A., Maleki, A., Studer, C., and Baraniuk, R. (2013). Maximin analysis of message passing algorithms for recovering block sparse signals. arXiv preprint arXiv:1303.2389. [161] Taheri, H., Pedarsani, R., and Thrampoulidis, C. (2019). Sharp guarantees for solving random equations with one-bit information. In 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 765–772. IEEE. 192 [162] Tan, P. H., Rasmussen, L. K., and Lim, T. J. (2001). Constrained maximumlikelihood detection in cdma. IEEE Transactions on Communications, 49(1):142– 153. [163] Tanaka, T. (2002). A statistical-mechanics approach to large-system analysis of cdma multiuser detectors. IEEE Transactions on Information theory, 48(11):2888–2910. [164] Thrampoulidis, C., Abbasi, E., and Hassibi, B. (2015a). Lasso with nonlinear measurements is equivalent to one with linear measurements. In Advances in Neural Information Processing Systems, pages 3420–3428. [165] Thrampoulidis, C., Abbasi, E., and Hassibi, B. (2016a). Precise error analysis of regularized 𝑀-estimators in high-dimensions. arXiv preprint arXiV: 1601.06233. [166] Thrampoulidis, C., Abbasi, E., Xu, W., and Hassibi, B. (2016b). Ber analysis of the box relaxation for bpsk signal recovery. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pages 3776– 3780. IEEE. [167] Thrampoulidis, C. and Hassibi, B. (2014). Estimating structured signals in sparse noise: A precise noise sensitivity analysis. In Communication, Control, and Computing (Allerton), 2014 52nd Annual Allerton Conference on, pages 866–873. IEEE. [168] Thrampoulidis, C. and Hassibi, B. (2015). Isotropically random orthogonal matrices: Performance of lasso and minimum conic singular values. In Information Theory (ISIT), 2015 IEEE International Symposium on. IEEE. [169] Thrampoulidis, C., Oymak, S., and Hassibi, B. (2014). Simple error bounds for regularized noisy linear inverse problems. Information Theory, 2014. f 2014. Proceedings. International Symposium on, pages 3007–3011. [170] Thrampoulidis, C., Oymak, S., and Hassibi, B. (2015b). Recovering structured signals in noise: Least-squares meets compressed sensing. In Compressed Sensing and Its Applications, pages 97–141. Springer. [171] Thrampoulidis, C., Oymak, S., and Hassibi, B. (2015c). Regularized linear regression: A precise analysis of the estimation error. In Proceedings of The 28th Conference on Learning Theory, pages 1683–1709. [172] Thrampoulidis, C., Panahi, A., Guo, D., and Hassibi, B. (2015d). Precise error analysis of the lasso. In 40th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2015, arxivPreprint arXiv:1502.04977. [173] Thrampoulidis, C., Panahi, A., and Hassibi, B. (2015e). Asymptotically exact error analysis for the generalized ℓ22 -lasso. In Information Theory (ISIT), 2015 IEEE International Symposium on. IEEE. 193 [174] Thrampoulidis, C., Xu, W., and Hassibi, B. (2018). Symbol error rate performance of box-relaxation decoders in massive mimo. IEEE Transactions on Signal Processing, 66(13):3377–3392. [175] Thrampoulidis, C., Zadik, I., and Polyanskiy, Y. (2019). A simple bound on the ber of the map decoder for massive mimo systems. arXiv preprint arXiv:1903.03949. [176] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), pages 267– 288. [177] Tropp, J. A. (2006). Just relax: Convex programming methods for identifying sparse signals in noise. Information Theory, IEEE Transactions on, 52(3):1030– 1051. [178] Tropp, J. A. (2014). Convex recovery of a structured signal from independent random linear measurements. arXiv preprint arXiv:1405.1102. [179] Tropp, J. A. and Gilbert, A. C. (2007). Signal recovery from random measurements via orthogonal matching pursuit. IEEE Transactions on information theory, 53(12):4655–4666. [180] Tropp, J. A., Laska, J. N., Duarte, M. F., Romberg, J. K., and Baraniuk, R. G. (2009). Beyond nyquist: Efficient sampling of sparse bandlimited signals. arXiv preprint arXiv:0902.0026. [181] Tu, J. V. (1996). Advantages and disadvantages of using artificial neural networks versus logistic regression for predicting medical outcomes. Journal of clinical epidemiology, 49(11):1225–1231. [182] Van de Geer, S. A. et al. (2008). High-dimensional generalized linear models and the lasso. The Annals of Statistics, 36(2):614–645. [183] Van der Vaart, A. W. (2000). Asymptotic statistics, volume 3. Cambridge university press. [184] Vapnik, V. (1982). Estimation of dependences based on empirical data berlin. [185] Verdu, S. (1998). Multiuser detection. Cambridge university press. [186] Vershynin, R. (2010). Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027. [187] Vershynin, R. (2012). How close is the sample covariance matrix to the actual covariance matrix? Journal of Theoretical Probability, 25(3):655–686. [188] Wainwright, M. J. (2009). Sharp thresholds for high-dimensional and noisy sparsity recovery using-constrained quadratic programming (lasso). Information Theory, IEEE Transactions on, 55(5):2183–2202. 194 [189] Wainwright, M. J. (2014). Structured regularizers for high-dimensional problems: Statistical and computational issues. Annual Review of Statistics and Its Application, 1:233–253. [190] Waldspurger, I., d’Aspremont, A., and Mallat, S. (2015). Phase recovery, maxcut and complex semidefinite programming. Mathematical Programming, 149(1-2):47–81. [191] Walther, A. (1963). The question of phase retrieval in optics. Journal of Modern Optics, 10(1):41–49. [192] Wang, L. (2013). The l1 penalized lad estimator for high dimensional linear regression. Journal of Multivariate Analysis, 120:135–151. [193] Wen, C.-K., Chen, J.-C., Wong, K.-K., and Ting, P. (2014). Message passing algorithm for distributed downlink regularized zero-forcing beamforming with cooperative base stations. Wireless Communications, IEEE Transactions on, 13(5):2920–2930. [194] White, C. D., Sanghavi, S., and Ward, R. (2015). The local convexity of solving systems of quadratic equations. arXiv preprint arXiv:1506.07868. [195] Wu, W. B. and Pourahmadi, M. (2003). Nonparametric estimation of large covariance matrices of longitudinal data. Biometrika, 90(4):831–844. [196] Wu, Y. and Verdú, S. (2012). Optimal phase transitions in compressed sensing. IEEE Transactions on Information Theory, 58(10):6241–6263. [197] Xu, J. and Hsu, D. (2019). How many variables should be entered in a principal component regression equation? arXiv preprint arXiv:1906.01139. [198] Yener, A., Yates, R. D., and Ulukus, S. (2002). Cdma multiuser detection: A nonlinear programming approach. IEEE Transactions on Communications, 50(6):1016–1024. [199] Yuan, M. and Lin, Y. (2007). Model selection and estimation in the gaussian graphical model. Biometrika, 94(1):19–35. .1 Proof of Theorem 2 Here, we prove Theorem 2. The proof consists of several steps and intermediate results, that are stated as lemmas. The proofs of those are all deferred to Appendix .2. 195 Preliminaries 𝛽ˆ := arg min L (y − X𝛽) + 𝜆 𝑓 (𝛽). 𝛽 Recall that y = X𝛽0 + z. Our goal is to characterize the nontrivial limiting behavior √ √ of k 𝛽ˆ − 𝛽0 k2 / 𝑝. We start with a simple change of variables w := (𝛽 − 𝛽0 )/ 𝑝, to directly get a handle on the error vector w. Also, we normalize the objective by dividing with 𝑝 so that the optimal cost is of constant order. Then, ŵ := arg min w
1 √ √
L z − 𝑝Xw + 𝜆 𝑓 𝛽0 + 𝑝w . 𝑝 (1) Instead of the optimization problem above, we will analyze a simpler Auxiliary Optimization (AO) that is tightly related to the Primary Optimization (PO) in (1) via the CGMT. The CGMT for M-estimators In this section, we show how the CGMT Theorem 8 can be applied to predict the limiting behavior of the solution k ŵk2 to the minimization in (1). The main challenge here is to express (1) as a (convex-concave) minimax optimization in which the involved random matrix (here X) appears in a bilinear form, exactly as in (2.126a). Also, some side technical details need to be taken care of. For example, in (2.126a) the optimization constraints are required by Theorem 8 to be bounded, which is not the case with (1). We start with addressing this immediately next. Boundedness of the Error The constraint set over which w is optimized in (2.126a) is unbounded. We will introduce “artificial" boundedness constraints that allow applying Theorem 8, while they do not affect the optimization itself. For this purpose, recall our goal of proving that k ŵk2 converges to some (finite) 𝛼∗ defined in Theorem 2. Define the set Sw = {w | kwk2 ≤ 𝐾𝛼 }, where 𝐾𝛼 := 𝛼∗ + 𝜁 (2) for a constant 𝜁 > 0, and, consider the “bounded" version of (1):
1 √ √
L z − 𝑝Xw + 𝜆 𝑓 𝛽0 + 𝑝w . w∈Sw 𝑝 ŵ𝐵 := arg min (3) 196 We expect that the additional constraint w ∈ Sw in (3) will not affect the optimization with high probability when 𝑝 is large enough. The idea here is that the minimizer of the original unconstrained problem in (1) satisfies k ŵk2 ≈ 𝛼∗ < 𝐾𝛼 w.h.p.. Of course, this latter statement is yet to be proven; but onnce this is done, we can return and confirm that our initial expectation is met. Lemma 26 below shows that 𝑃 if k ŵ𝐵 k − → 𝛼∗ < 𝐾𝛼 , then, the same is true for the optimal of (1). Lemma 26 For the two optimizations in (1) and (3), let ŵ and ŵ𝐵 be optimal 𝑃 𝑃 solutions. Also, recall the definition of 𝐾𝛼 in (2). If k ŵ𝐵 k2 − → 𝛼∗ , then k ŵk2 − → 𝛼∗ . Owing to the result of the lemma, henceforth, we work with the bounded optimization in (3). Using some abuse of notation, we will refer to optimal solution of (3) as ŵ, rather than ŵ𝐵 . Identifying the (PO) Here, we bring the minimization in (3) it in the form of the (PO) in (2.126a). For this purpose, we will use Lagrange duality. Note that the former can be equivalently expressed as 1 √ √ L ( 𝑝v) + 𝜆 𝑓 (𝛽0 + 𝑝w) w∈Sw ,v 𝑝 ŵ = arg min √ subject to v = z − 𝑝Xw. Associating a dual variable u to the equality constraint above, we write it as 1  1 √ √ ŵ = arg min max √ −u𝑇 ( 𝑝X)w + u𝑇 z − u𝑇 v + L (v) + 𝜆 𝑓 (𝛽0 + 𝑝w) . u w∈Sw ,v 𝑝 𝑝 (4) It takes no much effort to check that the objective function above is in the desired format of (2.126a): the random matrix X appears in a bilinear term u𝑇 Xw, and, the rest of the terms form a convex-concave function in u, w. Furthermore, we can use Assumption 1 to show that the optimal u∗ is bounded, which is a requirement of Theorem 8. In the same lines as in Section .1, we henceforth work with the “bounded" version of (4), namely, 1  1 √ √ ŵ = arg min max √ −u𝑇 ( 𝑝X)w + u𝑇 z − u𝑇 v + L (v) + 𝜆 𝑓 (𝛽0 + 𝑝w) . w∈Sw ,v u∈Su 𝑝 𝑝 (5) for Su := {u | kuk2 ≤ 𝐾 𝛽 } and 𝐾 𝛽 > 0 a sufficiently large constant. 197 Lemma 27 If Assumption 1(b) holds, then there exists sufficiently large constant 𝐾 𝛽 , such that the optimization problem in (5) is equivalent to that in (3), with probability approaching 1 in the limit of 𝑝 → ∞. As a last step, before writing down the corresponding (AO) problem, it will be useful for the analysis of the latter, to express 𝑓 in a variational form through its Fenchel conjugate, which gives, 1  √ ŵ = arg min max √ −u𝑇 ( 𝑝X)w + u𝑇 z − u𝑇 v w∈Sw ,v u∈Su ,s 𝑝 1 √ + L (v) + 𝜆s𝑇 𝛽0 + 𝜆 𝑝s𝑇 w − 𝜆 𝑓 ∗ (s) . 𝑝 (6) The (AO) Having identified (6) as the (PO) in our application, it is straightforward to write the corresponding (AO) problem following (2.126b): 1  max √ kwk2 g𝑇 u − kuk2 h𝑇 w + u𝑇 z − u𝑇 v w∈Sw ,v u∈Su ,s 𝑝 1 √ + L (v) + 𝜆s𝑇 𝛽0 + 𝜆 𝑝s𝑇 w − 𝜆 𝑓 ∗ (s) . 𝑝 min (7) Once we have identified the (AO) problem, Corollary 3 suggests analyzing that one 𝑃 instead of the (PO). Our goal is showing that k ŵk2 − → 𝛼∗ . For this, we wish to apply the corollary to the following set S = {w | |kwk2 − 𝛼∗ | > 𝜖 }, for arbitrary 𝜖 > 0. Asymptotic min-max property of the (AO) It turns out that verifying the conditions of the corollary for the (AO) as it appears in (7) is not directly easy. In short, what makes the analysis cumbersome is the fact that the optimization in (7) is not convex (e.g. if g𝑇 u is negative, then kwk2 g𝑇 u is not convex). Thus, flipping the order of min-max operations that would simplify the analysis is not directly justified. At this point, recall that the (PO) in (6) is itself convex. In fact, for it, all conditions of Sion’s min-max Theorem [149] are met, thus, the order of min-max operations can be flipped. According to the CGMT, the (PO) and the (AO) are tightly related in 198 an asymptotic setting. We use this, to translate the convexity properties of the (PO) to the (AO). In essence, we show that when dimensions grow, the order of min-max operations in the (AO) can be flipped. Thus, we will instead consider the following problem as the (AO): 1 1 max √ (kwk2 g + z − v)𝑇 u − √ kuk2 h𝑇 w 0≤𝛽≤𝐾 𝛽 kwk2 ≤𝐾 𝛼 kuk2 =𝛽 𝑝 𝑝 𝜙(g, h) := max min s v + 1 𝜆 𝜆 𝜆 L (v) + s𝑇 𝛽0 + √ s𝑇 w − 𝑓 ∗ (s). 𝑝 𝑝 𝑝 𝑝 (8) Observe that the objective function remains the same; it is only the order of min-max operations that is slightly modified compared to (7). Since the objective function is not necessarily convex-concave in its arguments, there is no immediate guarantee that the two problems in (7) and (8) are equivalent for all realizations of g and h. However, the lemma below essentially shows that such a strong duality holds with high probability over g and h in high dimensions. Hence, the problem in (8) can be as well used, instead of the one in (7), in order to analyze the (PO). For this reason, henceforth, we refer to (8) as the (AO) problem. Lemma 28 Let ŵ(X) denote an optimal solution of (1). Consider the (AO) problem in (8). Let 𝛼∗ be as defined in Theorem 2. For any 𝜖 > 0 define the set S := {w | |kwk2 − 𝛼∗ | < 𝜖 }, and, 𝜙S 𝑐 (g, h) be the optimal cost of the same optimization as in (8), only this time the minimization over w is further constrained such that w ∉ S. Assume that for any 𝐾𝛼 > 𝛼∗ and for any sufficiently large 𝐾 𝛽 , there exist constants 𝜙 < 𝜙S c such that for all 𝜂 > 0, with probability approaching one in the limit of 𝑝 → ∞ the following hold: 1. 𝜙(g, h) < 𝜙 + 𝜂, 2. 𝜙S c (g, h) > 𝜙S c − 𝜂. Then, lim P ( |k ŵ(X)k2 − 𝛼∗ | < 𝜖 ) = 1. 𝑝→∞ In view of Lemma 28, what remains in order to prove Theorem 2 is satisfying the conditions of the lemma. This involves a thorough analysis of the (AO) problem in (8), which is the subject of the next few sections. 199 Scalarization Observe that the optimization in (8) is over vectors. The purpose of this section is to simplify the (AO) into an optimization involving only scalar variables. Of course, one of this has to play the role of the norm of w, which is the quantity of interest. The main idea behind the “scalarization" step of the (AO) is to perform the optimization over only the direction of the vector variables while keeping their magnitude constant. This is already hinted by the rearrangement of the order of min-max operations going from (7) to (8). Also, this process is facilitated by the following two: 1. The bilinear term u𝑇 Xw that appears in the (PO) conveniently “splits" into the two terms kwk2 g𝑇 u and kuk2 h𝑇 w in the (AO), 2. The term involving the regularizer, i.e. 𝑓 (𝛽0 + w) has been expressed in a variational form as sups s𝑇 𝛽0 + s𝑇 w − 𝑓 ∗ (s). The details of the reduction step are all summarized in Lemma 29 below which shows that the (AO) reduces to the following convex minimax problem on four scalar optimization variables:   𝜏𝑔 𝛽𝜏𝑔 1 + eL 𝛼g + z; inf sup 0≤𝛼≤𝐾 𝛼 0≤𝛽≤𝐾 𝛽 2 𝑝 𝛽 𝜏𝑔 >0 𝜏ℎ >0 −   2 2  𝛽𝛼 1 𝛼𝜆   𝛼𝜏2ℎ + 𝛽2𝜏𝛼ℎ khk  − 𝜆 · e h + 𝛽 ; 𝑓 0 𝑝 𝑝 𝜏ℎ 𝜏ℎ ,𝛼 > 0   𝜆𝑝 𝑓 (𝛽0 )  ,𝛼 = 0 , (9) where recall that 1 ku − vk22 + 𝜔(v)} v 2𝜏 denotes the (vector) 𝜏-Moreau envelope of a function 𝜔 : R𝑑 → R evaluated at u ∈ R𝑑 . e𝜔 (u; 𝜏) := min{ Lemma 29 (Scalarization of the (AO)) The following statements are true regarding the two minimax optimization problems in (8) and (9): 1. They have the same optimal cost. 2. The objective function in (9) is continuous on its domain, (jointly) convex in (𝛼, 𝜏𝑔 ) and (jointly) concave in (𝛽, 𝜏ℎ ). 3. The order of inf-sup in (9) can be flipped without changing the optimization. 200 Convergence Analysis The goal of this section is to show that the (AO) satisfies the conditions of Lemma 28. This requires a convergence analysis of its optimal cost. We work with the scalarized version of the (AO) that was derived in the previous section: 𝜙(g, h, z, 𝛽0 ) = sup R 𝑝 (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ; g, h, z, 𝛽0 ), inf 0≤𝛼≤𝐾 𝛼 0≤𝛽≤𝐾 𝛽 𝜏𝑔 >0 𝜏 >0    ℎ 𝜏𝑔 𝛽𝜏𝑔 1 + eL 𝛼g + z; − L (z) R𝑝 = 2 𝑝 𝛽 n   o 2 2  𝛽𝛼 𝜆 𝛼𝜆   𝛼𝜏2ℎ + 𝛽2𝜏𝛼ℎ khk  − e h + 𝛽 ; − 𝑓 (𝛽 ) 𝑓 0 0 𝑝 𝑝 𝜏ℎ 𝜏ℎ −  0  (10) ,𝛼 > 0 . ,𝛼 = 0 Here, when compared to (9), we have subtracted from the objective the terms L (z) and 𝑓 (𝛽0 ), which of course does not affect the optimization. The optimization is of course random over the realizations of g, h, z and 𝛽0 , and, by the WLLN, it is easy to identify the converging value of the objective function R 𝑝 for fixed parameter values 𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ . For our goals, we need to show that minimax of the converging sequence of objectives converges to the minimax of the objective of the (SOP). Convexity of R 𝑝 plays a crucial role here since is being use to conclude local uniform convergence from the pointwise convergence. Uniform convergence is a requirement to conclude the desired.1 Lemma 30 (Convergence properties of the (AO)) Let R 𝑝 (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ) := R 𝑝 (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ; g, h, z, 𝛽0 ) be defined as in (10), and, 𝜙 A := 𝜙 A (g, h, z, 𝛽0 ) := inf sup R 𝑝 (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ), (11) 𝛼∈A 0≤𝛽≤𝐾 𝛽 𝜏𝑔 >0 𝜏ℎ >0 for A ⊆ [0, ∞). Further consider the following deterministic convex program 𝜙 A := inf sup D (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ) := 𝛼∈A 𝛽≥0 𝜏𝑔 >0 𝜏 >0 ℎ    𝜏 𝛽𝜏   2𝑔 + 𝛿 · 𝐿 𝛼, 𝛽𝑔  ,𝛽 >0   −𝛿 · 𝐿 0  ,𝛽=0 −   2  𝛼𝛽 𝛼𝜆   𝛼𝜏2ℎ + 𝛼𝛽  − 𝜆 · 𝐹 , 2𝜏ℎ 𝜏ℎ 𝜏ℎ ,𝛼 > 0  0  ,𝛼 = 0 . (12) 1 Interestingly, some of the tools used for this part of the proof are similar to those classically used for the study of consistency of 𝑀-estimators in the classical regime where 𝑝 is fixed and 𝑛 goes to infinity, see for example the Arg-min theorem in [107, Thm. 7.70], [125, Thm. 2.7]. 201 where 𝐿 and 𝐹 as in Theorem 2. If Assumption 1 hold, then, 𝑃 1. R 𝑛 (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ) − → D (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ), for all (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ), and, D (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ) is convex in (𝛼, 𝜏𝑔 ) and concave in (𝛽, 𝜏ℎ ). 2. Assume 𝛼∗ is the unique minimizer in (12) with A := [0, ∞). For any 𝜖 > 0, define S𝜖 := {𝛼 | |𝛼 − 𝛼∗ | < 𝜖 }. Then, for any sufficiently large constants 𝐾𝛼 > 𝛼∗ and 𝐾 𝛽 > 0, and for all 𝜂 > 0, it holds with probability approaching 1 as 𝑝 → ∞: a) 𝜙 [0,𝐾 𝛼 ] < 𝜙 [0,∞) + 𝜂, b) 𝜙 [0,𝐾 𝛼 ]\S𝜖 ≥ 𝜙 [0,∞)\S𝜖 − 𝜂, c) 𝜙 [0,∞)\S𝜖 > 𝜙 [0,∞) . Putting all the Pieces Together We are now ready to conclude the proof of Theorem 2. Proof 16 (Proof of Theorem 2) Fix any 𝜖 > 0. Consider the set S𝜖 = {w | |kwk2 − 𝛼∗ k2 < 𝜖 as in Lemma 28. We use the same notation as in the lemma. Let 𝐾𝛼 > 𝛼∗ and arbitrarily large (but finite) 𝐾 𝛽 > 0. From Lemma 29(i) 𝜙(g, h) is equal to the optimal cost of the optimization in (9). But, from Lemma 30(b)(i), the latter converges in probability to some constant 𝜙 (see Lemma 30 for the exact value constant). The same line of arguments applies to 𝜙S𝜖𝑐 (g, h), showing that it converges to another constant 𝜙S𝜖𝑐 . Again from Lemma 30(iii): 𝜙S𝜖𝑐 > 𝜙. Thus, the conditions of Lemma 28 are satisfied, and, it implies that the magnitude of any optimal minimizer (say) ŵ (𝑃𝑂) of the (PO) problem in (6) satisfies ŵ (𝑃𝑂) ∈ S in probability, in the limit of 𝑝 → ∞. Once, we have a min-max optimization that consits of only scalars, we only have to investigate the optimality conditions to get to the system of non-linear equations in Theorem 2. .2 Proofs for Section .1 Proof of Theorem 8(iii) Consider the following event E = {ΦS 𝑐 (G) ≥ 𝜙S c − 𝜂 , Φ(G) ≤ 𝜙 + 𝜂}. 202 In this event, it is not hard to check using assumption (a) that ΦS c > Φ, or equivalently wΦ ∈ S. Thus, it suffices to show that E occurs with probability at least 1 − 4𝑝. Indeed, from statement (i) of the theorem and assumption (c), P(ΦS 𝑐 (G) < 𝜙S c − 𝜂) ≤ 2P(𝜙S 𝑐 (g, h) ≤ 𝜙S c − 𝜂) ≤ 2𝑝. Also, from statement (ii) of the theorem and assumption (b), P(Φ(G) > 𝜙 + 𝜂) ≤ 2P(𝜙(g, h) ≥ 𝜙 + 𝜂) ≤ 2𝑝. Combining the above displays the claim follows from a union bound. Proof of Corollary 3 Call 𝜂 := (𝜙S c − 𝜙)/3 > 0. By assumption, for any 𝑝 > 0 there exists 𝑁 := 𝑁 (𝜂, 𝑝) such that the events {𝜙 < 𝜙 + 𝜂} and {𝜙S c > 𝜙S c − 𝜂} occur with probability at least 1 − 𝑝 each, for all 𝑝 > 𝑁. Then, for all 𝑝 > 𝑁, we can apply Theorem 8(iii) to conclude that wΦ (G) ∈ S with probably at least 1 − 4𝑝. Since this holds for all 𝑝 > 0, the proof is complete. Proof of Lemma 26 For convenience, denote with 𝑀 (w) the objective function in (1). For some 𝜖 > 0 such that 𝛼 + 𝜖 < 𝐾𝛼 (e.g. 𝜖 = 𝜁/2 in (2)), denote D := {w | 𝛼 − 𝜖 ≤ kwk2 ≤ 𝛼 + 𝜖 }. By assumption, with probability approaching 1 (w.p.a. 1). ŵ𝐵 ∈ D. (13) For the shake of a contradiction, assume that there exists optimal solution ŵ of (1) such that ŵ ∉ D w.p.a. 1. Clearly, 𝑀 ( ŵ) ≤ 𝑀 ( ŵ𝐵 ). (14) Suppose ŵ ∈ Sw , then ŵ is optimal for (3) and satisfies (13), which contradicts our assumption. Thus, ŵ ∉ Sw . Next, let w𝜃 := 𝜃 ŵ + (1 − 𝜃) ŵ𝐵 for 𝜃 ∈ (0, 1) such that w𝜃 ∉ D and w𝜃 ∈ S𝑤 (always possible, by definition of D). By the convexity of 𝐹 and (14), it follows that 𝑀 ( ŵ𝜃 ) ≤ 𝑀 ( ŵ𝐵 ). Hence, ŵ𝜃 is optimal for (3) and satisfies (13), which, again, is a contradiction. This completes the proof. 203 Proof of Lemma 27 It suffices to prove the equivalence of the optimization (4) and (5). Let w∗ , v∗ , u∗ be
optimal in (4). To prove the claim, we show that u∗ ∈ Su ⇔ ku∗ k2 ≤ 𝐾 𝛽 w.p.a. 1. From the first order optimality conditions in (4), we find that 1 u∗ ∈ √ 𝜕L (v∗ ) 𝑝 √ v∗ = z − 𝑝Xw∗ . (15) (16) Recall Assumption 1 and consider two cases. First, if supv∈R𝑛 sups∈𝜕L (v) ksk2 < ∞, √ the claim follows directly by (15). Next, assume that w.h.p., kzk2 ≤ 𝐶1 𝑝 for constant 𝐶1 > 0. Also, a standard high probability bound on the spectral norm of Gaussian matrices gives kXk2 ≤ 𝐶2 , e.g. [186]. Using these, boundedness of w∗ √ and (16), we find that kv∗ k2 ≤ 𝐶3 𝑝 w.h.p.. Then, the normalization condition √ √1 sups∈𝜕L (v) ksk2 ≤ 𝐶 for all kvk2 ≤ 𝑐 𝑝 and all 𝑝 ∈ N, yields the desired, i.e. 𝑝 ku∗ k2 ≤ 𝐶 holds with probability approaching 1 as 𝑝 → ∞. Proof of Lemma 28 Let w∗ denote an optimal solution of the “bounded" optimization in (6). It will suffice to prove that w∗ ∈ S in probability. To see this, recall from Lemma 27 that (6) is asymptotically equivalent to (3). Then, Lemma 26 and the assumption 𝛼∗ < 𝐾𝛼 guarantee that ŵ(X) ∈ S in probability, as desired. Denote Φ := Φ(X) the optimal cost of the minimization in (6) and ΦS 𝑐 := ΦS 𝑐 (X) the optimal cost of the same problem when the minimization is further restricted to be over the set w ∈ S 𝑐 . Note that w∗ ∈ S iff ΦS 𝑐 (X) > Φ(X); hence, it will suffice to prove that the latter event occurs in probability. We do so by relating the (PO) in (6) to the Auxiliary Optimization (AO) in (8) using Theorem 8. For concreteness, denote the objective function in (8) with 𝐴(w, v, u, s), and, recall Sw := {w | kwk2 ≤ 𝐾𝛼 }, Su := {u | kuk2 ≤ 𝐾 𝛽 }. With these, define 𝜙 𝑃 := 𝜙 𝑃 (g, h) := min max 𝐴(w, v, u, s) w∈Sw ,v u∈Su ,s 𝜙 𝐷 := 𝜙 𝐷 (g, h) := max min 𝐴(w, v, u, s). u∈Su ,s w∈Sw ,v and (17) Observe here that the order of min-max in 𝜙 𝑃 is exactly as in the original formulation of the CGMT, cf. (2.126b); 𝜙 𝐷 is the dual of it, while 𝜙 in (8) involves yet another change in the order of the optimizations. The reason we prefer to work with the 204 later problem, is that this particular order allows for a number of simplifications performed in Section .1. As done before, denote with 𝜙 𝑃 S 𝑐 , 𝜙 𝐷 S 𝑐 the optimal costs of the optimization problems in (17) under the additional constraint w ∈ S 𝑐 . The two problems in (17) are related to the one in (8) as follows: 𝜙 𝑃 S 𝑐 = min max 𝐴(w, v, u, s) = min max max 𝐴(w, v, u, s) w∈Sw ,v 𝛽,s kuk2 =𝛽 w∈S 𝑐 w∈Sw ,v u∈Su ,s w∈S 𝑐 ≥ max min max 𝐴(w, v, u, s) = 𝜙S 𝑐 , 𝛽,s w∈Sw ,v kuk2 =𝛽 w∈S 𝑐 (18) where the inequality follows from the min-max inequality [136, Lem. 36.1]. Similarly, 𝜙 𝐷 = max min 𝐴(w, v, u, s) = max max min 𝐴(w, v, u, s) 𝛽,s kuk2 =𝛽 w∈Sw ,v u∈Su ,s w∈Sw ,v ≤ max min max 𝐴(w, v, u, s) = 𝜙. 𝛽,s w∈Sw ,v kuk2 =𝛽 (19) Furthermore, they are related to the (PO) via the CGMT. From Theorem 8(i), for all 𝑐 ∈ R: P(ΦS 𝑐 < 𝑐) ≤ 2P(𝜙 𝑃 S 𝑐 ≤ 𝑐). (20) Also, from Theorem 8(ii)2: P(Φ > 𝑐) ≤ 2P(𝜙 𝐷 ≥ 𝑐). (21) The remaining of the proof is in the same lines as the proof of 8(iii), but is included for clarity. Let 𝜂 := (𝜙S c − 𝜙)/3 > 0. We may apply (20) for 𝑐 = 𝜙S 𝑐 − 𝜂 and combine with (18) to find that P(ΦS 𝑐 < 𝜙S 𝑐 − 𝜂) ≤ 2P(𝜙 𝑃 S 𝑐 ≤ 𝜙S 𝑐 − 𝜂) ≤ 2P(𝜙S 𝑐 ≤ 𝜙S 𝑐 − 𝜂). (22) From assumption (b) the last term above tends to zero as 𝑝 → ∞. In a similar way, combining (21), (19) and assumption (a), we find that P(Φ > 𝜙 + 𝜂) ≤ 2P(𝜙 𝐷 ≥ 𝜙 + 𝜂) ≤ 2P(𝜙 ≥ 𝜙 + 𝜂), (23) goes to zero with 𝑝 → ∞. Denote the event E = {ΦS 𝑐 ≥ 𝜙S 𝑐 − 𝜂 and Φ ≤ 𝜙 + 𝜂}. From (22) and (23) the event occurs with probability approaching 1. Furthermore, in this event, after using assumption (a), we have Φ𝑏 S 𝑐 ≥ 𝜙S 𝑐 − 𝜂 > 𝜙 + 𝜂 ≥ Φ𝑏 ; equivalently, the optimal minimizer satisfies w∗ ∈ S, which completes the proof. 2 more precisely, please refer to equation (32) in [171]. 205 Proof of Lemma 29 (i) We start by showing how the vector optimization in (8) can be reduced to the scalar one that appears in (9). This requires the following steps. Optimizing over the direction of u: Performing the inner maximization is easy. In particular, using the fact that max kuk2 =𝛽 u𝑇 t = 𝛽ktk2 for all 𝛽 ≥ 0 the problem simplifies to a max-min one: 𝛽 𝛽 √ k kwk2 g + z − v k2 − √ h𝑇 w 0≤𝛽≤𝐾 𝛽 ,s kwk≤𝐾 𝛼 ,v 𝑝 𝑝 𝜆 𝜆 𝜆 1 + L (v) + s𝑇 x0 + √ s𝑇 w − 𝑓 ∗ (s), 𝑝 𝑝 𝑝 𝑝 max min (24) Optimizing over the direction of w: Next, we fix kwk2 = 𝛼, and, similar to what was done above, minimize over its direction: 𝛽 1 𝛼 𝜆 𝜆 max min √ k 𝛼g + z − v k2 + L (v) − √ k 𝛽h − 𝜆sk2 + s𝑇 x0 − 𝑓 ∗ (s). 0≤𝛽≤𝐾 𝛽 ,s 0≤𝛼≤𝐾 𝛼 ,v 𝑝 𝑝 𝑝 𝑝 𝑝 (25) Changing the orders of min-max: Denote with 𝑀 (𝛼, 𝛽, v, s) the objective function above. It can be checked that 𝑀 is jointly convex in (𝛼, v) and jointly concave in (𝛽, s) (cf. Lemma 34). Thus, minv 𝑀 is convex in 𝛼 and jointly concave in (𝛽, s). Furthermore, the constraint sets are all convex and the one over which minimization over 𝛼 occurs is bounded. Hence, as in [149, Cor. 3.3] we can flip the order of max 𝛽,s min𝛼 , to conclude with min max max min 𝑀 (𝛼, 𝛽, v, s). 0≤𝛼≤𝐾 𝛼 0≤𝛽≤𝐾 𝛽 s v Also, observe that the order of optimization among v and s does not affect the outcome. √ 𝜒 } to both the terms The square-root trick: We apply the fact that 𝜒 = inf 𝜏>0 { 2𝜏 + 2𝜏 1 1 √ k𝛼g + z − vk2 and √ k 𝛽h − 𝜆sk2 : 𝑝 𝑝   𝛽𝜏𝑔 1 𝛽 2 + min k 𝛼g + z − v k2 + L (v) min max inf sup 0≤𝛼≤𝐾 𝛼 0≤𝛽≤𝐾 𝛽 𝜏𝑔 >0 𝜏ℎ >0 2 𝑝 v 2𝜏𝑔   𝛼𝜏ℎ 1 𝛼 2 𝑇 ∗ − − min k 𝛽h − 𝜆sk2 − 𝜆s x0 + 𝜆 𝑓 (s) . 2 𝑝 s 2𝜏ℎ (26) Identifying the Moreau envelope: Arguing as before, we can change the order of optimization between 𝛽 and 𝜏𝑔 . Also, it takes only a few algebra steps and using 206 basic properties of Moreau envelope functions (in particular, Lemma 35(ii)) in order to rewrite the last summand in (26) as below. If 𝛼 > 0, then,   𝛼 2 𝑇 ∗ k 𝛽h − 𝜆sk2 − 𝜆s x0 + 𝜆 𝑓 (s) min s 2𝜏ℎ   𝜏ℎ 𝜏ℎ 𝜏ℎ 𝛽 2 𝑇 = − kx0 k2 − 𝛽h x0 + 𝜆 · e 𝑓 ∗ h+ x0 ; (27) 2𝛼 𝜆 𝛼𝜆 𝛼𝜆   𝛼𝜆 𝛽2 𝛼 𝛽𝛼 2 khk − 𝜆 · e 𝑓 h + x0 ; . (28) = 2𝜏ℎ 𝜏ℎ 𝜏ℎ Otherwise, if 𝛼 = 0, then the same term equals −𝜆 𝑓 (x0 ) since maxs s𝑇 x0 − 𝑓 ∗ (s) = 𝑓 (x0 ). (ii) The continuity of the objective function in (9) follows directly from the continuity of the Moreau envelope functions, cf. [137, Lem. 1.25, 2.26]. In particular, regarding the two branches of the objective: it can be checked, using the continuity of the Moreau envelope, that the limit of the RHS in (28) as 𝛼 → 0 evaluates to −𝜆 𝑓 (x0 ). (In fact, this is the unique extension of the upper branch to a continuous finite convex function on the whole 𝛼 ≥ 0, 𝜏 > 0, as per [136, Thm. 10.3]). Convexity of (9) can be checked from (26). By applying Lemma 34, after minimization over v the Moreau Envelope remains jointly convex with respect to 𝛼 and 𝜏𝑔 and concave in 𝛽. The same argument (and similar lemma) holds for the last term of (26) in which after minimization over s it remains jointly convex in 𝛽 and 𝜏ℎ and concave in 𝛼. Then the negative sign before this term makes it jointly concave in 𝛽 and 𝜏ℎ and convex over 𝛼. Proof of Lemma 30 (a) By Assumption 1 the normalized Moreau envelope functions in (10) converge 𝑃 in probability to 𝐿 and 𝐹, respectively. Also, khk22 /𝑝 − → 1 by the WLLN. This proves the convergence part. Lemma 29(i) showed R 𝑝 to be convex-concave. Then, the same holds for D by point-wise convergence and the fact that convexity is preserved by point wise limits. (b) Call 𝑀𝑛 (𝛼) = sup inf R 𝑛 (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ) 0≤𝛽≤𝐾 𝛽 𝜏𝑔 >0 𝜏ℎ >0 and 𝑀 (𝛼) = sup inf D𝑛 (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ). 0≤𝛽 𝜏𝑔 >0 𝜏ℎ >0 (29) 207 The bulk of the proof consists of showing that the following two statements hold ∀ compact susets A ⊂ (0, ∞) and sufficiently large 𝐾 𝛽 := 𝐾 𝛽 (A) > 0 𝑃 : inf 𝑀𝑛 (𝛼) − → inf 𝑀 (𝛼) 𝛼∈A 𝛼∈A (30) and, ∀𝜖 > 0, w.p.a.1 : 𝑀𝑛 (0) < 𝑀 (0) + 𝜖 . (31) Before proceeding with the proof of those, let us show how the conclusion of the lemma is reached once (30) and (31) are established. Using (30) and (31) to prove the lemma : Fix 𝐾𝛼 > 𝛼∗ , any 𝛿 > 0 such that A := [𝛼∗ − 2𝛿, 𝛼∗ + 2𝛿] ⊂ (0, 𝐾𝛼 ] and 𝐾 𝛽 > 0 large enough such that (30) and (31) both hold. Then, for all 𝜖 > 0, w.p.a.1: min 𝑀𝑛 (𝛼) ≤ min 𝑀𝑛 (𝛼) ≤ 𝑀𝑛 (𝛼∗ ) < 𝑀 (𝛼∗ ) + 𝜖 . 0≤𝛼≤𝐾 𝛼 𝛼∈A (32) For the last inequality above: if 𝛼∗ = 0, it follows from (31), or otherwise from (30). Next, consider the compact set A𝑙 = {𝛼 > 0 | 𝛼 ∈ [𝛼∗ − 2𝛿, 𝛼 − 𝛿] } and A𝑟 = {𝛼 > 0 | 𝛼 ∈ [𝛼∗ + 𝛿, 𝛼 + 2𝛿] }. (Note that if 𝛼∗ = 0, then A𝑙 is empty.) From (30), we know that for all 𝜖 > 0, w.p.a.1 min 𝑀𝑛 (𝛼) > min 𝑀 (𝛼) − 𝜖 𝛼∈A𝑙 𝛼∈A𝑙 and min 𝑀𝑛 (𝛼) > min 𝑀 (𝛼) − 𝜖 . 𝛼∈A𝑢 𝛼∈A𝑢 Let A𝑙𝑢 = A𝑙 ∪ A𝑢 and combine the above to find min 𝑀𝑛 (𝛼) > min 𝑀 (𝛼) − 𝜖 . 𝛼A𝑙𝑢 (33) 𝛼∈A𝑙𝑢 By assumption on uniqueness of 𝛼∗ and on convexity of 𝑀, we have 𝑀 (𝛼∗ ) < min 𝑀 (𝛼) (34) 𝛼∈A𝑙𝑢 and 𝑀 (𝛼∗ ) = min𝛼∈A 𝑀 (𝛼). Thus, Applying (32) and (33) for 𝜖 = (min𝛼∈A𝑙𝑢 𝑀 (𝛼)− 𝑀 (𝛼∗ ))/3 yields w.p.a.1 : min 𝑀𝑛 (𝛼) > min 𝑀 (𝛼) − 𝜖 > 𝑀 (𝛼∗ ) + 𝜖 > 𝛼∈A𝑙𝑢 𝛼∈A𝑙𝑢 min 𝛼∈[𝛼∗ −2𝛿,𝛼∗ +2𝛿] Thus, w.p.a.1, 𝛼ˆ 𝑛 := arg min 𝑀𝑛 (𝛼) ∈ (𝛼∗ − 𝛿, 𝛼∗ + 𝛿). 𝛼∈A 𝑀𝑛 (𝛼). (35) 208 In this event, for any 𝛼 ∉ A, there is a convex combination 𝛼𝜃 := 𝜃 𝛼ˆ 𝑛 + (1 − 𝜃)𝛼, (𝜃 < 1) that equals either 𝛼∗ − 2𝛿 or 𝛼∗ + 2𝛿. By convexity, 𝑀𝑛 (𝛼𝜃 ) ≤ 𝜃 𝑀𝑛 ( 𝛼ˆ 𝑛 ) + (1 − 𝜃) 𝑀𝑛 (𝛼) Also, from (35), 𝑀𝑛 ( 𝛼ˆ 𝑛 ) < 𝑀𝑛 (𝛼𝜃 ). Combining those, we find 𝑀𝑛 ( 𝛼ˆ 𝑛 ) < 𝑀𝑛 (𝛼), implying that 𝛼ˆ 𝑛 is the minimizer of 𝑀𝑛 over the entire [0, 𝐾𝛼 ] w.p.a.1. In other words, for all 𝜖 w.p.a. 1, min 𝛼∈[0,𝐾 𝛼 ]\(𝛼∗ −𝛿,𝛼∗ +𝛿) 𝑀𝑛 (𝛼) ≥ min 𝑀𝑛 (𝛼) > min 𝑀 (𝛼) − 𝜖 . 𝛼∈A𝑙𝑢 𝛼∈A𝑙𝑢 (36) To establish a connection with the three statements (i)-(iii) of the lemma, observe that 𝜙 [0,∞) = 𝑀 (𝛼∗ ). Also, 𝜙 [0,∞)\S 𝛿 = min𝛼∈A𝑙𝑢 𝑀 (𝛼) (by convexity). With these, (i) corresponds directly to (32), (ii) to (36), and, (iii) to (34). Proof of (30) and (31) : From the first statement of the lemma, the objective function R 𝑛 of the (AO) converges point-wise to D. We will use this to show that the minimax value of R 𝑝 converges to the corresponding minimax of D. The proof is based on a repeated use of Lemma 31 below, about convergence of the infimum of a sequence of convex converging stochastic processes. This fact is essentially a consequence of what is known in the literature as the convexity lemma, according to which point wise convergence of convex functions implies uniform convergence in compact subsets. Please refer to Section .2 for the proof. Lemma 31 (Min-convergence – Open Sets) Consider a sequence of proper, convex stochastic functions 𝑀𝑛 : (0, ∞) → R, and, a deterministic function 𝑀 : (0, ∞) → R, such that: 𝑃 1. 𝑀𝑛 (𝑥) − → 𝑀 (𝑥), for all 𝑥 > 0, 2. there exists 𝑧 > 0 such that 𝑀 (𝑥) > inf 𝑥>0 𝑀 (𝑥) for all 𝑥 ≥ 𝑧. 𝑃 Then, inf 𝑥>0 𝑀𝑛 (𝑥) − → inf 𝑥>0 𝐹 (𝑥). 1) Fix 𝛼 ≥ 0, 𝛽 > 0, and, 𝜏ℎ > 0. Consider 𝛼,𝛽,𝜏ℎ (𝜏𝑔 ) := 𝑅𝑛 (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ), (37) 𝑀 𝛼,𝛽,𝜏ℎ (𝜏𝑔 ) := D (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ). (38) 𝑀𝑛 209 𝛼,𝛽,𝜏 𝑃 The functions {𝑀𝑛 } are convex. Furthermore, 𝑀𝑛 ℎ (𝜏𝑔 ) − → 𝑀 𝛼,𝛽,𝜏ℎ (𝜏𝑔 ) point wise in 𝜏𝑔 . Next, we show that 𝑀 𝛼,𝛽,𝜏ℎ is level-bounded, i.e. it satisfies condition (b) of Lemma 31. In view of Lemma 32, it suffices to show that lim𝜏𝑔 →∞ 𝑀 𝛼,𝛽,𝜏ℎ (𝜏𝑔 ) =   𝐿(𝛼,𝜏𝑔 /𝛽) +∞, or lim𝜏𝑔 →∞ 2𝛽 + 𝛿 · > 0. By assumption 1, lim𝜏𝑔 →∞ 𝐿(𝛼, 𝜏𝑔 /𝛽) = 𝜏𝑔 −𝐿 0 . There is two cases to be considered. Either 𝐿 0 < ∞, or else, Assumption 1 holds. Either way, lim𝜏𝑔 →∞ 𝐿(𝛼, 𝜏𝑔 /𝛽)/𝜏𝑔 = 0 and we are done. Now, we can apply Lemma 31 to conclude that 𝛼,𝛽,𝜏ℎ inf 𝑀𝑛 𝜏𝑔 >0 𝑃 (𝜏𝑔 ) − → inf 𝑀 𝛼,𝛽,𝜏ℎ (𝜏𝑔 ). 𝜏𝑔 >0 (39) 2) Next, again for fixed 𝛼 ≥ 0, 𝜏ℎ > 0, consider (we use some abuse of notation here, with the purpose of not overloading notation) 𝛼,𝛽,𝜏ℎ 𝑀𝑛𝛼,𝜏ℎ (𝛽) := inf 𝑀𝑛 (𝜏𝑔 ) 𝜏𝑔 >0 𝑀 𝛼,𝜏ℎ (𝛽) := inf 𝑀 𝛼,𝛽,𝜏ℎ (𝜏𝑔 ) 𝜏𝑔 >0 The functions {𝑀𝑛𝛼,𝜏ℎ } are concave in 𝛽, as the point wise minima of concave 𝑃 functions. Furthermore, 𝑀𝑛𝛼,𝜏ℎ (𝛽) − → 𝑀 𝛼,𝜏ℎ (𝛽) point wise in 𝛽 > 0, by (39). 𝛼 > 0: For now and until further notice, restrict attention to the case 𝛼 > 0. Also, consider first 𝛽 > 0. We show that 𝑀 𝛼,𝜏ℎ is level-bounded, i.e. it satisfies condition (b) of Lemma 31. In view of Lemma 32, it suffices to show that lim 𝛽→+∞ 𝑀 𝛼,𝜏ℎ (𝛽) = −∞, or lim 𝛽→+∞ inf 𝜏𝑔 >0 𝑀 𝛼,𝛽,𝜏ℎ (𝜏𝑔 ) = −∞. This condition is equivalent to the following   (∀𝑀 > 0)(∃𝐵 > 0) 𝛽 > 𝐵 ⇒ (∃{𝜏𝑔 } 𝑘 ) [𝐷 (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ) < −𝑀] . (40) First, we show that   𝛼𝛽 𝛼𝜆 𝛼𝛽2 −𝜆·𝐹 , = +∞ lim 𝛽→+∞ 2𝜏ℎ 𝜏ℎ 𝜏ℎ (41) This follows by Assumption 1 when applied for 𝑐 = 𝛼𝛽/𝜏ℎ and 𝜏 = 𝛼𝜆/𝜏ℎ (recall here that 𝛼 > 0). 𝛽𝜏 Next, choose {𝜏𝑔 } 𝑘 → 0. For that choice, 2𝑔 + 𝐿 (𝛼, 𝜏𝑔 /𝛽) → lim𝜏→0 𝐿 (𝛼, 𝜏) < ∞, where boundedness follows by Assumption 1. Thus, (40) is correct and we may apply Lemma 31 to conclude that 𝑃 sup 𝑀𝑛𝛼,𝜏ℎ (𝛽) − → sup 𝑀 𝛼,𝜏ℎ (𝛽). 𝛽>0 𝛽>0 (42) 210 Now, we investigate the case 𝛽 = 0. We have, 𝑀𝑛𝛼,𝜏ℎ (0) = − 1𝑝 L (z) − 𝛼𝜏2ℎ +     𝛼,𝜏ℎ 𝛼𝜆 𝜆 (0) = −𝛿𝐿 0 − 𝛼𝜏2ℎ + 𝐹 (0, 𝛼𝜆 𝑝 e 𝑓 x0 ; 𝜏ℎ − 𝑓 (x0 ) and 𝑀𝑛 𝜏ℎ ). 𝑃 If 𝐿 0 < ∞, then by assumption, 𝑀𝑛𝛼,𝜏ℎ (0) − → 𝑀 𝛼,𝜏ℎ (0). Combined with (42), we find 𝑃 sup 𝑀𝑛𝛼,𝜏ℎ (𝛽) − → sup 𝑀 𝛼,𝜏ℎ (𝛽). 𝛽≥0 (43) 𝛽≥0 Now, consider the case 𝐿 0 = +∞. Clearly, the optimal 𝛽 for 𝑀 𝛼,𝜏ℎ is not at zero; thus, sup 𝛽≥0 𝑀 𝛼,𝜏ℎ (𝛽) = sup 𝛽>0 𝑀 𝛼,𝜏ℎ (𝛽). Also, by assumption, for all 𝑀,   1 lim 𝑝→∞ P 𝑝 L (z) > 𝑀 = 1. Letting, 𝜖 > 0 and 𝑀 := − sup 𝛽>0 𝑀 𝛼,𝜏ℎ (𝛽) + 𝜖 + 𝛼,𝜏ℎ 𝛼𝜏ℎ 𝛼𝜆 (0) < sup 𝛽>0 𝑀 𝛼,𝜏ℎ (𝛽) − 𝜖 ≤ sup 𝛽>0 𝑀𝑛𝛼,𝜏ℎ (𝛽), 2 − 𝐹 (0, 𝜏ℎ ), then w.p.a.1, 𝑀𝑛 where the last inequality follows because of (42). Again, this leads to (43). To sum up, (43) holds for all 𝛼 > 0. 𝛼 = 0: We show that for all 𝜖 > 0, the following holds w.p.a.1: sup 𝑀𝑛𝛼=0,𝜏ℎ (𝛽) < sup 𝑀 𝛼=0,𝜏ℎ (𝛽) + 𝜖 . 𝛽≥0 (44) 𝛽≥0 To begin with, note that for all 𝑝, 𝛽𝜏𝑔 1 + min sup 𝑀𝑛𝛼=0,𝜏ℎ (𝛽) ≤ sup lim 𝑝 v 𝛽≥0 𝛽>0 𝜏𝑔 →0 2   𝛽 2 kz − vk2 + L (v) − L (z) = 0, 2𝜏𝑔 (45) where we have used Lemma 40(ix). Next, we show that 𝑀 𝛼=0,𝜏ℎ (𝛽) = 0. (46) Using Assumption 1 on the non-negativity of 𝐿 0 and Assumption 1 that lim𝜏→0 𝐿 (𝑐, 𝜏) = 𝛽𝜏 0, it follows that 𝑀 𝛼=0,𝜏ℎ (𝛽) ≤ sup 𝛽>0 lim𝜏𝑔 →0 2𝑔 + 𝐿 (0, 𝜏𝑔 /𝛽) = 0. Thus, it will suffice for the claim if we prove 𝛽𝜏𝑔 + 𝐿(0, 𝜏𝑔 /𝛽) = 0, 𝛽→∞ 𝜏𝑔 >0 2 lim inf (47) or equivalently,   𝛽2 𝐿(0, 𝜅) lim inf 𝜅 + = 0. 𝛽→∞ 𝜅>0 2 𝜅 2 Fix some 𝛽 > 0. Note that lim𝜅→0 𝜅𝛽2 +𝐿 (0, 𝜅)  = 0, where  we have used Assumption 1 that lim𝜏→0 𝐿(0, 𝜏) = 0. Also, lim𝜅→∞ 𝜅 𝛽2 𝐿(0,𝜅) 2 + 𝜅 = +∞, using Assumption 211 p 1 this time. Now, consider only 𝛽 > −𝐿 2,+ (0, 0) (see Assumption 1). Then, the 2 function 𝜅𝛽2 + 𝐿 (0, 𝜅) has a positive derivative at 𝜅 → 0+ . From this and convexity, it follows that for all 𝜅 > 0, 𝜅𝛽2 𝜅𝛽2 + 𝐿 (0, 𝜅) ≥ lim + 𝐿 (0, 𝜅) = 0. 𝜅→0 2 2 This proves (47) as desired. To complete the argument, (44) follows by (45) and (46), and with this we have completed the proof of (31). 3) Keep 𝛼 > 0 fixed and consider 𝑀𝑛𝛼 (𝜏ℎ ) := sup 𝑀𝑛𝛼,𝜏ℎ (𝛽), 𝛽≥0 𝑀 𝛼 (𝜏ℎ ) := sup 𝑀 𝛼,𝜏ℎ (𝛽), 𝛽≥0 The functions {𝑀𝑛𝛼 } and 𝐹 are all concave in 𝜏ℎ , as the point wise maxima of 𝑃 jointly concave functions. Furthermore, 𝑀𝑛𝛼 (𝜏ℎ ) − → 𝑀 𝛼 (𝜏ℎ ) point wise in 𝜏ℎ , by (43). Next, we show that 𝑀 𝜏ℎ is level-bounded, i.e. it satisfies condition (b) of Lemma 31. In view of Lemma 32, it suffices to show that lim𝜏ℎ →∞ 𝑀 𝛼 (𝜏ℎ ) = +∞, or lim𝜏ℎ →∞ sup 𝛽>0 inf 𝜏𝑔 >0 D (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ) = −∞. This is equivalent to the following   (∀𝑀 > 0)(∃𝑇 > 0) 𝜏ℎ > 𝑇 ⇒ (∀{𝛽} 𝑘 )(∃{𝜏𝑔 } 𝑘 ) [𝐷 (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ) < −𝑀] . (48) Consider the function H (𝛽, 𝜏ℎ ) := 𝛼𝛽 𝛼𝜆 𝛼𝜏ℎ 𝛼𝛽2 + − 𝜆 · 𝐹( , ). 2 2𝜏ℎ 𝜏ℎ 𝜏ℎ We show that H (𝛽, 𝜏ℎ ) ≥ 𝛼𝜏ℎ . 2  2 khk 2 To see this note that e 𝑓 (𝑐h + x0 ; 𝜏) ≤ 𝑐 2𝜏 + 𝑓 (x0 ). Thus, 1𝑝 e 𝑓 (𝑐h + x0 ; 𝜏) − 𝑓 (x0 ) ≤ 𝑐2 khk 2 2𝜏 𝑝 . The LHS converges to 𝐹 (𝑐, 𝜏) by Assumption 1 and the RHS converges to 2 𝛼𝛽 𝑐 𝑐2 𝛼𝜆 2𝜏 . Therefore, 𝐹 (𝑐, 𝜏) ≤ 2𝜏 . Applying this for 𝑐 = 𝜏ℎ and 𝜏 = 𝜏ℎ , we have that 𝛼𝛽2 𝛼𝛽 𝛼𝜆 2𝜏ℎ − 𝜆 · 𝐹 ( 𝜏ℎ , 𝜏ℎ ) ≥ 0, as desired. Then,   𝛽𝜏𝑔 𝜏𝑔 𝛼𝜏ℎ D (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ) ≤ + 𝛿 · 𝐿 𝛼, − . 2 𝛽 2 212 𝜏 Also, note that for all 𝛽 > 0, we can choose (sequence) of 𝜏𝑔 , such that 𝛽𝜏𝑔 , 𝛽𝑔 → 0.   𝜏𝑔 𝛽𝜏𝑔 Then, 2 + 𝛿 · 𝐿 𝛼, 𝛽 → lim𝜏→0 𝐿(𝛼, 𝜏) =: 𝐴 < ∞. It can then be seen that (48) holds for (say) 𝑇 := 𝑇 (𝑀) = 4( 𝐴 + 𝑀)/𝛼. We can apply Lemma 31 to conclude that 𝑃 sup 𝑀𝑛𝛼 (𝜏ℎ ) − → sup 𝑀 𝛼 (𝜏ℎ ). 𝜏ℎ >0 (49) 𝜏ℎ >0 4) Finally, consider 𝑀𝑛 (𝛼) := sup 𝑀𝑛𝛼 (𝜏ℎ ), 𝜏ℎ >0 𝑀 (𝛼) := sup 𝑀 𝛼 (𝜏ℎ ). (50) 𝜏ℎ >0 The functions {𝑀𝑛 } and 𝐹 are all convex in 𝜏ℎ , as the point wise maxima of 𝑃 convex functions. Furthermore, 𝑀𝑛 (𝛼) − → 𝑀 (𝛼) point wise in 𝛼, by (49). By assumption of the lemma, 𝐹 has a unique minimizer 𝛼∗ , which of course implies level boundedness. Thus, we can apply Lemma 10 to conclude that 𝑃 inf 𝑀𝑛 (𝛼) − → inf 𝑀 (𝛼). 𝛼>0 𝛼>0 (51) 𝑃 Besides, pointwise convergence 𝑀 𝑝 (𝛼) − → 𝑀 (𝛼) translates to uniform convergence over any compact subset A ⊂ (0, ∞) by the Convexity lemma [7, Cor.. II.1] ,[107, Lem. 7.75]. Hence, 𝑃 inf 𝑀𝑛 (𝛼) − → inf 𝑀 (𝛼). 𝛼∈A 𝛼∈A This is of course same as the desired in (30). Recall, (31) was established in (44). The only thing remaining is showing that there exists an optimal 𝛽∗ in sup 𝛽≥0 𝑀 𝛼,𝜏ℎ (𝛽) that is bounded by some sufficiently large 𝐾 𝛽 (A). This follows from the level-boundedness arguments above as detailed immediately next. Boundedness of solutions : For a compact subset A ⊂ (0, ∞), we argue that there exists bounded 𝛽∗ and sequences {𝜏𝑔 ∗ } 𝑘 , {𝜏ℎ ∗ } 𝑘 such that (𝛼∗ , {𝜏𝑔 ∗ } 𝑘 , 𝛽∗ , {𝜏ℎ ∗ } 𝑘 ) approaches min𝛼∈A sup𝜏ℎ >0 inf 𝜏𝑔 >0 D (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ). This follows from the work above. In partic𝛽≥0 ular, at each step in the proof of (30) above, we showed level-boundedness of the corresponding functions. For example, (48) shows that there exists (sufficiently large) 𝑇ℎ (𝛼) > 0 such that sup𝜏ℎ >0 𝑀 𝛼 (𝜏ℎ ) is equal to sup𝑇ℎ (𝛼)≥𝜏ℎ >0 𝑀 𝛼 (𝜏ℎ ). This holds for all 𝛼; so, in particular, is true for 𝑇ℎ := max𝛼∈A 𝑇ℎ (𝛼). Next, from (41) there 213 exists 𝐾 𝛽 (𝛼, 𝑇ℎ ), such that sup 𝛽≥0 𝑀 𝛼,𝜏ℎ (𝛽) is equal to sup𝐾 𝛽 (𝛼,𝑇ℎ )≥𝛽≥0 𝑀 𝛼,𝜏ℎ (𝛽). Again, this holds for all 𝛼 ∈ A, thus there exists sufficiently large 𝐾 𝛽 > 0 such that (see also Lemma 33) min sup inf D (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ) = inf 𝛼∈A 𝜏ℎ >0 𝜏𝑔 >0 𝛽≥0 𝛼∈A sup inf D (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ) 𝜏ℎ >0 𝜏𝑔 >0 𝐾 𝛽 ≥𝛽≥0 The objective function D above is convex-concave. Also, the constraint sets over 𝛼 and 𝛽 are compact. Furthermore, the optimization of D over 𝜏𝑔 and 𝜏ℎ is separable. With these and an application of Sion’s minimax theorem, the order of inf–sup between the four optimization variables can be flipped arbitrarily without affecting the outcome. Thus, for example, inf sup inf D (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ) = inf sup D (𝛼, 𝜏𝑔 , 𝛽, 𝜏ℎ ) 𝛼∈A 𝜏ℎ >0 𝜏𝑔 >0 𝛽≥0 𝛼∈A 𝜏ℎ >0 𝜏𝑔 >0 ≥𝛽≥0 The same is of course true for the corresponding random optimizations (also, Lemma 29(iii)). Auxiliary Lemmas Proof 17 (Proof of Lemma 31) First, convexity is preserved by point wise limits, so that 𝐹 (𝑥) is also convex. Using this and level-boundedness condition (b) of the lemma, it is easy to show that inf 𝑥>0 𝐹 (𝑥) > −∞. Since 𝐹 is proper and (lower) level-bounded, the only way inf 𝑥>0 𝐹 (𝑥) = −∞ is if lim𝑥→0 𝐹 (𝑥) = −∞. But, this is not possible as follows: Fix 0 < 𝑥 1 < 𝑥 2 < 𝑥 3 . Then, for any 0 < 𝑥 < 𝑥 1 and 2 𝜃 := 𝑥𝑥33−𝑥 −𝑥 , convexity gives   1 1 𝑥3 − 𝑥1 𝑥2 − 𝑥1 𝐹 (𝑥) ≥ 𝐹 (𝑥 2 ) − 1 − 𝐹 (𝑥3 ) ≥ − |𝐹 (𝑥 2 )| − |𝐹 (𝑥 3 )|. 𝜃 𝜃 𝑥3 − 𝑥2 𝑥3 − 𝑥2 Next, we show that for sufficiently small 𝜖 > 0, there exist 𝑥 0 > 𝑥 𝜖 > 0: inf 𝐹 (𝑥) ≤ 𝐹 (𝑥 𝜖 ) ≤ inf 𝐹 (𝑥) + 𝜖 𝑥>0 𝑥>0 and 𝐹 (𝑥 𝜖 ) < 𝐹 (𝑥 0 ). (52) We show the claim for all 0 < 𝜖 < 𝜖 1 := (𝐹 (𝑧) − inf 𝑥>0 𝐹 (𝑥)). Since inf 𝑥>0 𝐹 (𝑥) is finite, there exists 𝑥 𝜖 > 0 such that 𝐹 (𝑥 𝜖 ) − 𝜖 ≤ inf 𝑥>0 𝐹 (𝑥). Without loss of generality, 𝑥 𝜖 < 𝑧. Pick any 𝑥 0 > 𝑧. For the shake of contradiction, assume 𝐹 (𝑥 0 ) ≤ 𝐹 (𝑥 𝜖 ). Then, by convexity, for some 𝜃 ∈ (0, 1) 𝐹 (𝑧) ≤ 𝜃𝐹 (𝑥 𝜖 ) + (1 − 𝜃)𝐹 (𝑥 0 ) ≤ 𝐹 (𝑥 𝜖 ) ≤ inf 𝐹 (𝑥) + 𝜖 < 𝐹 (𝑧). 𝑥>0 214 Thus, 𝐹 (𝑥 𝜖 ) < 𝐹 (𝑥 0 ). In order to establish the desired, it suffices that for all arbitrarily small 𝛿 > 0, w.p.a. 1, | inf 𝐹𝑛 (𝑥) − inf 𝐹 (𝑥)| < 𝛿. 𝑥>0 (53) 𝑥>0 Fix some 0 < 𝜖 < min{𝜖1 , 𝛿} such that (52) holds, and, also some 0 < 𝜖 0 < min{(𝐹 (𝑥0 ) − 𝐹 (𝑥 𝜖 ))/4, 𝛿/4, 𝛿 − 𝜖 }. (54) Let 𝐾 = [𝑎, 𝑏] ⊂ (0, ∞) be compact subset such that 𝑎 < 𝑥 𝜖 < 𝑥0 ≤ 𝑏 and 𝛿−2𝜖 0 𝑎 = 2𝛿−𝜖 0 𝑥 𝜖 . The functions {𝐹𝑝 } are convex and they converge point wise to 𝐹 in the open set (0, ∞). This implies uniform convergence in compact sets by the Convexity lemma [7, Cor.. II.1] ,[107, Lem. 7.75]. That is, there exists sufficiently large 𝑁1 such that the event sup |𝐹𝑛 (𝑥) − 𝐹 (𝑥)| < 𝜖 0 (55) 𝑥∈𝐾 occurs w.p.a. 1, for all 𝑝 > 𝑁1 . In this event, inf 𝐹𝑛 (𝑥) ≤ 𝐹𝑛 (𝑥 𝜖 ) < 𝐹 (𝑥 𝜖 ) + 𝜖 0 ≤ inf 𝐹 (𝑥) + 𝜖 + 𝜖 0 ≤ inf 𝐹 (𝑥) + 𝛿 𝑥>0 𝑥>0 𝑥>0 It remains to prove the other side of (53). In what follows, take 𝑝 ≥ 𝑁1 and condition on the high probability event in (55). Let us first show level-boundedness of 𝐹𝑛 . Consider the event inf 𝑥>𝑥0 𝐹𝑛 (𝑥) < inf 𝑥≤𝑥0 𝐹𝑛 (𝑥). If this happens, then, inf 𝑥>𝑥0 𝐹𝑛 (𝑥) < 𝐹𝑛 (𝑥 𝜖 ), in which case there exists (by continuity of 𝐹𝑛 ), 𝑥 𝑛 > 𝑥 0 such that 𝐹𝑛 (𝑥 𝑛 ) < 𝐹𝑛 (𝑥 𝜖 ). But then, convexity implies that for some 0 < 𝜃 𝑛 < 1, 𝐹𝑛 (𝑥 0 ) ≤ 𝜃 𝑛 𝐹𝑛 (𝑥 𝑛 ) + (1 − 𝜃 𝑛 )𝐹𝑛 (𝑥 𝜖 ) < 𝐹𝑛 (𝑥 𝜖 ) ≤ 𝐹 (𝑥 𝜖 ) + 𝜖 0 < 𝐹 (𝑥 0 ) − 𝜖 0, (56) where we also used (55) and (54). Of course, this contradicts (55). Thus, inf 𝐹𝑛 (𝑥) = inf 𝐹𝑛 (𝑥). 𝑥>0 𝑥≤𝑥 0 (57) Using (57), convexity and properness of {𝐹𝑛 }, it can be shown that inf 𝑥>0 𝐹𝑛 (𝑥) > −∞. The argument is the same as the one used in the beginning of the proof for 𝐹, thus is omitted for brevity. 215 Overall, for all 𝑝 > 𝑁1 , conditioned on (55), there is some 0 < 𝑥 𝑛 ≤ 𝑥 0 such that inf 𝐹𝑛 (𝑥) ≥ 𝐹𝑛 (𝑥 𝑛 ) − 𝜖 0 . (58) 𝑥>0 If 𝑎 ≤ 𝑥 𝑛 ≤ 𝑏, then a direct application of (55) gives the desired 𝐹𝑛 (𝑥 𝑛 ) ≥ 𝐹 (𝑥 𝑛 ) − 𝜖 0 ≥ inf 𝐹 (𝑥) − 𝜖 0 ⇒ inf 𝐹𝑛 (𝑥) ≥ inf 𝐹 (𝑥) − 2𝜖 0 ≥ inf 𝐹 (𝑥) − 𝛿. 𝑥>0 𝑥>0 𝑥>0 𝑥>0 Next, assume that 0 < 𝑥 𝑛 < 𝑎. There exists 𝜃 𝑛 ∈ (0, 1) such that 𝜃 𝑛 𝑥 𝑛 +(1−𝜃 𝑛 )𝑥 𝜖 = 𝑎. In fact, 𝜃𝑛 = 𝑥𝜖 − 𝑎 𝛿 − 2𝜖 0 ≥ (1 − 𝑎/𝑥 𝜖 ) = . 𝑥𝜖 − 𝑥𝑛 2𝛿 − 𝜖 0 (59) Then, by convexity and (55), 𝐹𝑛 (𝑎) ≤ 𝜃 𝑛 𝐹𝑛 (𝑥 𝑛 ) + (1 − 𝜃 𝑛 )𝐹𝑛 (𝑥 𝜖 ). Rearranging and using (55) 1 − 𝜃𝑛 1 𝐹𝑛 (𝑎) − 𝐹𝑛 (𝑥 𝜖 ) 𝜃𝑛 𝜃𝑛 1 1 − 𝜃𝑛 ≥ (𝐹 (𝑎) − 𝜖 0) − (𝐹 (𝑥 𝜖 ) + 𝜖 0) 𝜃𝑛 𝜃𝑛     1 1 − 𝜃𝑛 ≥ inf 𝐹 (𝑥) − 𝜖 − inf 𝐹 (𝑥) + 𝛿 𝑥>0 𝜃 𝑛 𝑥>0 𝜃𝑛 𝐹𝑛 (𝑥 𝑛 ) ≥ Combining this with (58) and (59), yields the desired inf 𝑥>0 𝐹𝑛 (𝑥 𝑛 ) ≥ inf 𝑥>0 𝐹 (𝑥)−𝛿. Lemma 32 (Level-bounded convex fcns) Let 𝐹 : (0, ∞) → R be convex. Then, the following two statements are equivalent: 1. There exists 𝑧 > 0 such that 𝐹 (𝑥) > inf 𝑥>0 𝐹 (𝑥) for all 𝑥 ≥ 𝑧. 2. lim𝑥→∞ 𝐹 (𝑥) = +∞. Proof 18 (a)⇒(b): Clearly, there exists 0 < 𝑥0 < 𝑧, such that 𝐹 (𝑧) > 𝐹 (𝑥 0 ). Then, by convexity, for all 𝑥 > 𝑧 it holds 𝐹 (𝑥) ≥ 𝐹 (𝑧) + 𝐹 (𝑧) − 𝐹 (𝑥 0 ) (𝑥 − 𝑧). 𝑧 − 𝑥0 | {z } >0 Taking limits of 𝑥 → ∞ on both sides above, proves the claim. (a)⇐(b): A a proper functions, 𝐹 has a nonempty domain in (0, ∞). Hence, inf 𝑥>0 𝐹 (𝑥) < ∞ and can choose some 𝑀 > inf 𝑥>0 𝐹 (𝑥). From (b), there exists 𝑧 > 0 such that 𝐹 (𝑥) ≥ 𝑀 for all 𝑥 ≥ 𝑧, as desired. 216 Lemma 33 (Saddle-points) For a convex-concave function 𝐹 : R × R → R, consider the minimax optimization inf 𝑥 sup𝑦 𝐹 (𝑥, 𝑦). Let 𝐶, 𝐷 be compact subsets such that there exists at least one saddle point (𝑥 ∗ , 𝑦 ∗ ) ∈ 𝐶 × 𝐷. Then, inf sup 𝐹 (𝑥, 𝑦) = inf sup 𝐹 (𝑥, 𝑦). 𝑥 𝑥∈𝐶 𝑦∈𝐷 𝑦 Proof 19 First observe that, inf sup 𝐹 (𝑥, 𝑦) = inf sup 𝐹 (𝑥, 𝑦) 𝑥 𝑥∈𝐶 𝑦 𝑦 Since 𝐹 has a saddle-point, the LHS above is equal to sup𝑦 inf 𝑥 𝐹 (𝑥, 𝑦) [136, Lem. 36.2]. Also, from Sion’s minimax theorem, the RHS is equal to sup𝑦 inf 𝑥∈𝐶 𝐹 (𝑥, 𝑦). Thus, it suffices to prove that sup inf 𝐹 (𝑥, 𝑦) = sup inf 𝐹 (𝑥, 𝑦). 𝑦 𝑥∈𝐶 𝑦∈𝐷 𝑥∈𝐶 Clearly, this holds with a “≥" sign. To prove equality, let (𝑥∗ , 𝑦 ∗ ) be a saddle point. Then, sup inf 𝐹 (𝑥, 𝑦) = inf sup 𝑓 (𝑥, 𝑦) ≤ sup 𝑓 (𝑥 ∗ , 𝑦) ≤ 𝑓 (𝑥∗ , 𝑦 ∗ ) = sup inf 𝐹 (𝑥, 𝑦). 𝑦 𝑥∈𝐶 𝑥∈𝐶 𝑦 𝑦 𝑦∈𝐷 𝑥∈𝐶 1 Lemma 34 The function ℎ(𝛼, 𝜏, v) = 2𝜏 k𝛼x + z − vk22 is jointly convex in its arguments. Proof 20 The function k𝛼x − vk22 is trivially jointly convex in 𝛼 and v. So its perspective function which is 1𝜏 k𝛼x − vk22 is also jointly convex in all its arguments, same as its shifted version which is ℎ(𝛼, 𝜏, v). Lemma 35 Let 𝑓 : R 𝑝 → R be convex. Then,
1. prox 𝑓 (x; 𝜏) + 𝜏 · prox 𝑓 ∗ x/𝜏; 𝜏 −1 = x, 2. e 𝑓 (x; 𝜏) + e 𝑓 ∗ (x/𝜏; 1/𝜏) = kxk 2𝜏 . 2 217 .3 Proof of Auxiliary Lemmas Proof of Lemma 10 First, convexity is preserved by point wise limits, so that 𝐹 (𝑥) is also convex. Using this and level-boundedness condition (b) of the lemma, it is easy to show that inf 𝑥>0 𝐹 (𝑥) > −∞. Since 𝐹 is proper and (lower) level-bounded, the only way inf 𝑥>0 𝐹 (𝑥) = −∞ is if lim𝑥→0 𝐹 (𝑥) = −∞. But, this is not possible as follows: Fix 2 0 < 𝑥1 < 𝑥2 < 𝑥3 . Then, for any 0 < 𝑥 < 𝑥1 and 𝜃 := 𝑥𝑥33−𝑥 −𝑥 , convexity gives   1 𝑥3 − 𝑥1 𝑥2 − 𝑥1 1 𝐹 (𝑥3 ) ≥ − |𝐹 (𝑥 2 )| − |𝐹 (𝑥 3 )|. 𝐹 (𝑥) ≥ 𝐹 (𝑥 2 ) − 1 − 𝜃 𝜃 𝑥3 − 𝑥2 𝑥3 − 𝑥2 Next, we show that for sufficiently small 𝜖 > 0, there exist 𝑥 0 > 𝑥 𝜖 > 0: inf 𝐹 (𝑥) ≤ 𝐹 (𝑥 𝜖 ) ≤ inf 𝐹 (𝑥) + 𝜖 𝑥>0 𝑥>0 𝐹 (𝑥 𝜖 ) < 𝐹 (𝑥0 ). and (60) We show the claim for all 0 < 𝜖 < 𝜖1 := (𝐹 (𝑧) − inf 𝑥>0 𝐹 (𝑥)). Since inf 𝑥>0 𝐹 (𝑥) is finite, there exists 𝑥 𝜖 > 0 such that 𝐹 (𝑥 𝜖 ) − 𝜖 ≤ inf 𝑥>0 𝐹 (𝑥). Without loss of generality, 𝑥 𝜖 < 𝑧. Pick any 𝑥 0 > 𝑧. For the shake of contradiction, assume 𝐹 (𝑥0 ) ≤ 𝐹 (𝑥 𝜖 ). Then, by convexity, for some 𝜃 ∈ (0, 1) 𝐹 (𝑧) ≤ 𝜃𝐹 (𝑥 𝜖 ) + (1 − 𝜃)𝐹 (𝑥0 ) ≤ 𝐹 (𝑥 𝜖 ) ≤ inf 𝐹 (𝑥) + 𝜖 < 𝐹 (𝑧). 𝑥>0 Thus, 𝐹 (𝑥 𝜖 ) < 𝐹 (𝑥 0 ). In order to establish the desired, it suffices that for all arbitrarily small 𝛿 > 0, w.p.a. 1, | inf 𝐹𝑛 (𝑥) − inf 𝐹 (𝑥)| < 𝛿. 𝑥>0 𝑥>0 (61) Fix some 0 < 𝜖 < min{𝜖1 , 𝛿} such that (60) holds, and, also some 0 < 𝜖 0 < min{(𝐹 (𝑥 0 ) − 𝐹 (𝑥 𝜖 ))/4, 𝛿/4, 𝛿 − 𝜖 }. (62) Let 𝐾 = [𝑎, 𝑏] ⊂ (0, ∞) be compact subset such that 𝑎 < 𝑥 𝜖 < 𝑥0 ≤ 𝑏 and 𝛿−2𝜖 0 𝑎 = 2𝛿−𝜖 0 𝑥 𝜖 . The functions {𝐹𝑛 } are convex and they converge point wise to 𝐹 in the open set (0, ∞). This implies uniform convergence in compact sets by the Convexity lemma ,[107, Lem. 7.75]. That is, there exists sufficiently large 𝑁1 such that the event sup |𝐹𝑛 (𝑥) − 𝐹 (𝑥)| < 𝜖 0 𝑥∈𝐾 (63) 218 occurs w.p.a. 1, for all 𝑛 > 𝑁1 . In this event, inf 𝐹𝑛 (𝑥) ≤ 𝐹𝑛 (𝑥 𝜖 ) < 𝐹 (𝑥 𝜖 ) + 𝜖 0 ≤ inf 𝐹 (𝑥) + 𝜖 + 𝜖 0 ≤ inf 𝐹 (𝑥) + 𝛿 𝑥>0 𝑥>0 𝑥>0 (64) It remains to prove the other side of (61). In what follows, take 𝑛 ≥ 𝑁1 and condition on the high probability event in (63). Let us first show level-boundedness of 𝐹𝑛 . Consider the event inf 𝑥>𝑥0 𝐹𝑛 (𝑥) < inf 𝑥≤𝑥0 𝐹𝑛 (𝑥). If this happens, then, inf 𝑥>𝑥0 𝐹𝑛 (𝑥) < 𝐹𝑛 (𝑥 𝜖 ), in which case there exists (by continuity of 𝐹𝑛 ), 𝑥 𝑛 > 𝑥0 such that 𝐹𝑛 (𝑥 𝑛 ) < 𝐹𝑛 (𝑥 𝜖 ). But then, convexity implies that for some 0 < 𝜃 𝑛 < 1, 𝐹𝑛 (𝑥 0 ) ≤ 𝜃 𝑛 𝐹𝑛 (𝑥 𝑛 ) + (1 − 𝜃 𝑛 )𝐹𝑛 (𝑥 𝜖 ) < 𝐹𝑛 (𝑥 𝜖 ) ≤ 𝐹 (𝑥 𝜖 ) + 𝜖 0 < 𝐹 (𝑥 0 ) − 𝜖 0, (65) where we also used (63) and (62). Of course, this contradicts (63). Thus, inf 𝐹𝑛 (𝑥) = inf 𝐹𝑛 (𝑥). 𝑥>0 (66) 𝑥≤𝑥 0 Using (66), convexity and properness of {𝐹𝑛 }, it can be shown that inf 𝑥>0 𝐹𝑛 (𝑥) > −∞. The argument is the same as the one used in the beginning of the proof for 𝐹, thus is omitted for brevity. Overall, for all 𝑛 > 𝑁1 , conditioned on (63), there is some 0 < 𝑥 𝑛 ≤ 𝑥0 such that inf 𝐹𝑛 (𝑥) ≥ 𝐹𝑛 (𝑥 𝑛 ) − 𝜖 0 . (67) 𝑥>0 If 𝑎 ≤ 𝑥 𝑛 ≤ 𝑏, then a direct application of (63) gives the desired 𝐹𝑛 (𝑥 𝑛 ) ≥ 𝐹 (𝑥 𝑛 ) − 𝜖 0 ≥ inf 𝐹 (𝑥) − 𝜖 0 ⇒ inf 𝐹𝑛 (𝑥) ≥ inf 𝐹 (𝑥) − 2𝜖 0 ≥ inf 𝐹 (𝑥) − 𝛿. 𝑥>0 𝑥>0 𝑥>0 𝑥>0 Next, assume that 0 < 𝑥 𝑛 < 𝑎. There exists 𝜃 𝑛 ∈ (0, 1) such that 𝜃 𝑛 𝑥 𝑛 +(1−𝜃 𝑛 )𝑥 𝜖 = 𝑎. In fact, 𝑥𝜖 − 𝑎 𝛿 − 2𝜖 0 𝜃𝑛 = ≥ (1 − 𝑎/𝑥 𝜖 ) = . (68) 𝑥𝜖 − 𝑥𝑛 2𝛿 − 𝜖 0 Then, by convexity and (63), 𝐹𝑛 (𝑎) ≤ 𝜃 𝑛 𝐹𝑛 (𝑥 𝑛 ) + (1 − 𝜃 𝑛 )𝐹𝑛 (𝑥 𝜖 ). Rearranging and using (63) 1 1 − 𝜃𝑛 𝐹𝑛 (𝑥 𝑛 ) ≥ 𝐹𝑛 (𝑎) − 𝐹𝑛 (𝑥 𝜖 ) 𝜃𝑛 𝜃𝑛 1 1 − 𝜃𝑛 ≥ (𝐹 (𝑎) − 𝜖 0) − (𝐹 (𝑥 𝜖 ) + 𝜖 0) 𝜃𝑛 𝜃𝑛     1 1 − 𝜃𝑛 ≥ inf 𝐹 (𝑥) − 𝜖 − inf 𝐹 (𝑥) + 𝛿 𝑥>0 𝜃 𝑛 𝑥>0 𝜃𝑛 Combining this with (67) and (68), yields the desired inf 𝑥>0 𝐹𝑛 (𝑥 𝑛 ) ≥ inf 𝑥>0 𝐹 (𝑥) − 𝛿. 219 Lemma 36 (Level-bounded convex fcns) Let 𝐹 : (0, ∞) → R be convex. Then, the following two statements are equivalent: 1. There exists 𝑧 > 0 such that 𝐹 (𝑥) > inf 𝑥>0 𝐹 (𝑥) for all 𝑥 ≥ 𝑧. 2. lim𝑥→∞ 𝐹 (𝑥) = +∞. Proof 21 (a)⇒(b): Clearly, there exists 0 < 𝑥0 < 𝑧, such that 𝐹 (𝑧) > 𝐹 (𝑥 0 ). Then, by convexity, for all 𝑥 > 𝑧 it holds 𝐹 (𝑥) ≥ 𝐹 (𝑧) + 𝐹 (𝑧) − 𝐹 (𝑥 0 ) (𝑥 − 𝑧). 𝑧 − 𝑥0 | {z } >0 Taking limits of 𝑥 → ∞ on both sides above, proves the claim. (a)⇐(b): A a proper functions, 𝐹 has a nonempty domain in (0, ∞). Hence, inf 𝑥>0 𝐹 (𝑥) < ∞ and can choose some 𝑀 > inf 𝑥>0 𝐹 (𝑥). From (b), there exists 𝑧 > 0 such that 𝐹 (𝑥) ≥ 𝑀 for all 𝑥 ≥ 𝑧, as desired. Lemma 37 (Saddle-points) For a convex-concave function 𝐹 : R × R → R, consider the minimax optimization inf 𝑥 sup𝑦 𝐹 (𝑥, 𝑦). Let 𝐶, 𝐷 be compact subsets such that there exists at least one saddle point (𝑥 ∗ , 𝑦 ∗ ) ∈ 𝐶 × 𝐷. Then, inf sup 𝐹 (𝑥, 𝑦) = inf sup 𝐹 (𝑥, 𝑦). 𝑥 𝑥∈𝐶 𝑦∈𝐷 𝑦 Proof 22 First observe that, inf sup 𝐹 (𝑥, 𝑦) = inf sup 𝐹 (𝑥, 𝑦) 𝑥 𝑥∈𝐶 𝑦 𝑦 Since 𝐹 has a saddle-point, the LHS above is equal to sup𝑦 inf 𝑥 𝐹 (𝑥, 𝑦) [136, Lem. 36.2]. Also, from Sion’s minimax theorem, the RHS is equal to sup𝑦 inf 𝑥∈𝐶 𝐹 (𝑥, 𝑦). Thus, it suffices to prove that sup inf 𝐹 (𝑥, 𝑦) = sup inf 𝐹 (𝑥, 𝑦). 𝑦 𝑥∈𝐶 𝑦∈𝐷 𝑥∈𝐶 Clearly, this holds with a “≥" sign. To prove equality, let (𝑥∗ , 𝑦 ∗ ) be a saddle point. Then, sup inf 𝐹 (𝑥, 𝑦) = inf sup 𝑓 (𝑥, 𝑦) ≤ sup 𝑓 (𝑥 ∗ , 𝑦) ≤ 𝑓 (𝑥∗ , 𝑦 ∗ ) = sup inf 𝐹 (𝑥, 𝑦). 𝑦 𝑥∈𝐶 𝑥∈𝐶 𝑦 𝑦 𝑦∈𝐷 𝑥∈𝐶 220 1 Lemma 38 The function ℎ(𝛼, 𝜏, v) = 2𝜏 k𝛼x + z − vk22 is jointly convex in its arguments. Proof 23 The function k𝛼x − vk22 is trivially jointly convex in 𝛼 and v. So its perspective function which is 1𝜏 k𝛼x − vk22 is also jointly convex in all its arguments, same as its shifted version which is ℎ(𝛼, 𝜏, v). Lemma 39 Let 𝑓 : R𝑛 → R be convex. Then,
1. prox 𝑓 (x; 𝜏) + 𝜏 · prox 𝑓 ∗ x/𝜏; 𝜏 −1 = x, 2. e 𝑓 (x; 𝜏) + e 𝑓 ∗ (x/𝜏; 1/𝜏) = kxk 2𝜏 . 2 Properties of the Moreau envelope: In this section we have gathered some very useful properties of Moreau envelopes of convex functions. We have made heavy use of those results for the proofs of Theorem (1) and (2). Some of the results are standard, while others are more tailored towards our interests. Lemma 40 (Properties of the Moreau envelope) Let ℓ : R → R be a proper, closed, convex function. For 𝜏 > 0, consider its Moreau envelope function and its proximal operator: 1 ( 𝜒 − 𝑣) 2 + ℓ(𝑣), 2𝜏 1 proxℓ ( 𝜒; 𝜏) := arg min ( 𝜒 − 𝑣) 2 + ℓ(𝑣) 2𝜏 The following statements are true: eℓ ( 𝜒; 𝜏) := min 𝑣 (69a) (69b) 1. proxℓ ( 𝜒; 𝜏) is single valued and continuous. Furthermore, ℓ0𝜒,𝜏 := 1 ( 𝜒 − proxℓ ( 𝜒; 𝜏)) ∈ 𝜕ℓ(proxℓ ( 𝜒; 𝜏)). 𝜏 (70) 2. eℓ ( 𝜒; 𝜏) is jointly convex in ( 𝜒, 𝜏). 3. eℓ ( 𝜒; 𝜏) is continuously differentiable with respect to both 𝑥 and 𝜏. The gradients are given by: 𝜕𝑒 ℓ 1 = ( 𝜒 − proxℓ ( 𝜒; 𝜏)) = ℓ0𝜒,𝜏 , 𝜕𝜒 𝜏 2 𝜕𝑒 ℓ 1 1 𝐸 2 ( 𝜒, 𝜏) := = − 2 ( 𝜒 − proxℓ ( 𝜒; 𝜏)) 2 = − ℓ0𝜒,𝜏 . 𝜕𝜏 2 2𝜏 𝐸 1 ( 𝜒, 𝜏) := (71) (72) 221 4. Fix 𝜒 and 𝜏 > 0. Consider the function Δ : R × (−𝜏, ∞) → R: Δ (𝑥, 𝑦) := (𝐸 1 ( 𝜒 + 𝑥, 𝜏 + 𝑦) − 𝐸 1 ( 𝜒, 𝜏))𝑥 + (𝐸 2 ( 𝜒 + 𝑥, 𝜏 + 𝑦) − 𝐸 2 ( 𝜒, 𝜏))𝑦 Then,  𝑦 0 Δ (𝑥, 𝑦) ≥ 𝜏 + (ℓ 𝜒+𝑥,𝜏+𝑦 − ℓ0𝜒,𝜏 ) 2 . 2 (73) 5. eℓ (𝑥; 𝜏) is non-increasing in 𝜏. 6. lim𝜏→∞ eℓ (𝑥; 𝜏) = min𝑣 ℓ(𝑣). 7. lim𝜏→∞ 1𝜏 |𝑥 − proxℓ (𝑥; 𝜏) | = 0. 0 8. If 0 ∈ arg min𝑣 ℓ(𝑣), then proxℓ (𝑥; 𝜏) 𝑥 ≥ 0, |proxℓ (𝑥; 𝜏) | ≤ |𝑥| and |ℓprox |≤ ℓ (𝑥;𝜏),𝜏 0 |. |ℓ𝑥,𝜏 9. eℓ (𝑥 𝑛 ; 𝜏𝑛 ) → ℓ(𝑥) whenever 𝑥 𝑛 → 𝑥 while 𝜏𝑛 → 0+ in such a way that the sequence {|𝑥 𝑛 − 𝑥|/𝜏𝑛 }𝑛∈N is bounded. Proof 24 (i) From [137, Thm. 2.26(a)], proxℓ ( 𝜒; 𝜏) is known to be continuous single valued mapping. Besides, from standard optimality conditions: 1 ( 𝜒 − proxℓ ( 𝜒; 𝜏)) ∈ 𝜕ℓ(proxℓ ( 𝜒; 𝜏)) 𝜏 (74) For convenience, we have define ℓ0𝜒,𝜏 := 1𝜏 ( 𝜒 − proxℓ ( 𝜒; 𝜏) ∈ 𝜕ℓ(proxℓ ( 𝜒; 𝜏)). Note that if ℓ is differentiable at proxℓ ( 𝜒; 𝜏), then ℓ0𝜒,𝜏 is the derivative of ℓ at that point. (ii) Trivially, ℎ( 𝜒, 𝑣) := ( 𝜒 − 𝑣) 2 is a jointly convex function of 𝑣 and 𝑥. Thus, its perspective function 𝜏ℎ( 𝜒𝜏 , 𝜏𝑣 ) = 1𝜏 ( 𝜒 − 𝑣) 2 is also jointly convex over 𝜏, 𝑥 and 𝑣 and so after minimization over 𝑣, the function remains jointly convex over 𝑥 and 𝜏 (cf. [137, Prop. 2.22]). (iii) See [137, Thm. 2.26(b)] for differentiability with respect to 𝑥. Next, we mimic the argument to conclude about differentiability with respect to 𝜏. It suffices to show that ℎ(𝑦) := eℓ ( 𝜒; 𝜏 + 𝑦) − eℓ ( 𝜒; 𝜏) + 2𝜏𝑦 2 ( 𝜒 − proxℓ ( 𝜒; 𝜏)) 2 is differentiable 1 2 at 𝑦 = 0 with 𝜕ℎ 𝜕𝑦 = 0. We know eℓ ( 𝜒; 𝜏) = 2𝜏 ( 𝜒 − proxℓ ( 𝜒; 𝜏)) + ℓ(proxℓ ( 𝜒; 𝜏)), 1 whereas eℓ ( 𝜒; 𝜏 + 𝑦) ≤ 2(𝜏+𝑦) ( 𝜒 − proxℓ ( 𝜒; 𝜏)) 2 + ℓ(proxℓ ( 𝜒; 𝜏)). Thus, 1 1 𝑦 ( 𝜒 − proxℓ ( 𝜒; 𝜏)) 2 − ( 𝜒 − proxℓ ( 𝜒; 𝜏)) 2 + 2 ( 𝜒 − proxℓ ( 𝜒; 𝜏)) 2 2(𝜏 + 𝑦) 2𝜏 2𝜏 2 𝑦 = 2 ( 𝜒 − proxℓ ( 𝜒; 𝜏)) 2 . (75) 2𝜏 (𝜏 + 𝑦) ℎ(𝑦) ≤ 222 Besides because of convexity of ℎ(𝑦), 0 = ℎ(0) ≤ 21 ℎ(𝑦) + 12 ℎ(−𝑦) or equivalently ℎ(𝑦) ≥ −ℎ(−𝑦). Thus, (75) gives: ℎ(𝑦) ≥ 𝑦2 ( 𝜒 − proxℓ ( 𝜒; 𝜏)) 2 . 2𝜏 2 (𝜏 − 𝑦) (76) Combining (75) and (76) leads to the following 𝑦2 𝑦2 2 ( ( 𝜒 − prox 𝜒; 𝜏)) ≤ ℎ(𝑦) ≤ ( 𝜒 − proxℓ ( 𝜒; 𝜏)) 2 ℓ 2𝜏 2 (𝜏 − 𝑦) 2𝜏 2 (𝜏 + 𝑦) (77) Here, ℎ(𝑦) is sandwiched between two continuously differentiable functions at 0 with zero derivatives. This completes the proof. (iv) From (71) and (72), we have Δ (𝑥, 𝑦) = (ℓ0𝜒+𝑥,𝜏+𝑦 − ℓ0𝜒,𝜏 )𝑥 −  02 02 𝑦 ℓ ( 𝜒 + 𝑥, 𝜏 + 𝑦) − ℓ ( 𝜒, 𝜏) 2  𝑦 0 0 0 0 = (ℓ 𝜒+𝑥,𝜏+𝑦 − ℓ 𝜒,𝜏 ) 𝑥 − ℓ + ℓ 𝜒,𝜏 . 2 𝜒+𝑥,𝜏+𝑦 (78)  (79) On the other hand, due to optimality conditions in (70), proxℓ ( 𝜒 + 𝑥; 𝜏 + 𝑦) − proxℓ ( 𝜒; 𝜏) = 𝑥 − (𝜏 + 𝑦)ℓ0𝜒+𝑥,𝜏+𝑦 + 𝜏ℓ0𝜒,𝜏   𝑦 𝑦 = 𝑥 − (ℓ0𝜒+𝑥,𝜏+𝑦 + ℓ0𝜒,𝜏 ) − (𝜏 + )(ℓ0𝜒+𝑥,𝜏+𝑦 − ℓ0𝜒,𝜏 ). 2 2 (80) Finally, from convexity of ℓ, it follows from the monotonicity property of the subdifferential that (ℓ0𝜒+𝑥,𝜏+𝑦 − ℓ0𝜒,𝜏 )(proxℓ ( 𝜒 + 𝑥; 𝜏 + 𝑦) − proxℓ ( 𝜒; 𝜏)) ≥ 0. Combining the three displays above gives the desired inequality. (v) This follows directly by non-positivity of the derivative as in (72). (vi) Using the decreasing nature of eℓ (𝑥; 𝜏) w.r.t. 𝜏, we have lim eℓ (𝑥; 𝜏) = inf𝜏>0 min 𝜏→∞ 𝑣 1 1 (𝑥−𝑣) 2 +ℓ(𝑣) = min inf𝜏>0 (𝑥−𝑣) 2 +ℓ(𝑣) = min ℓ(𝑣). 𝑣 𝑣 2𝜏 2𝜏 (81) (vii) Fix an 𝜖 > 0. Since lim𝜏→∞ eℓ (𝑥; 𝜏) = min𝑣 ℓ(𝑣), there exist 𝑇𝜖0 such that for all 𝜏 ≥ 𝑇𝜖 := max{2, 𝑇𝜖0 }, |eℓ (𝑥; 𝜏) − min ℓ(𝑣)| = 𝑣 1 (𝑥 − proxℓ (𝑥; 𝜏)) 2 + (ℓ(proxℓ (𝑥; 𝜏)) − min ℓ(𝑣)) < 𝜖 2 𝑣 2𝜏 (82) 223 1 Then, 2𝜏 (𝑥 − proxℓ (𝑥; 𝜏)) 2 < 𝜖 2 , which gives r r 2 2 1 |𝑥 − proxℓ (𝑥; 𝜏) | < 𝜖 ≤𝜖 ≤𝜖 𝜏 𝜏 𝑇𝜖 (83) Therefore, lim𝜏→∞ 1𝜏 |𝑥 − proxℓ (𝑥; 𝜏) | = 0. (viii) By (70) and the assumption 0 ∈ arg min𝑣 ℓ(𝑣), we find proxℓ (0; 𝜏) = 0. Monotonicity of the prox operator [137, Prop. 12.19], gives (proxℓ (𝑥; 𝜏)−proxℓ (0; 𝜏))𝑥 ≥ 0, which then shows proxℓ (𝑥; 𝜏) 𝑥 ≥ 0. Also, monotonicity of the subdifferential of 0 𝑥 ≥ 0. Those two, when combined with optimality conditions in (70) give ℓ gives ℓ𝑥,𝜏 0 𝑥 − proxℓ (𝑥; 𝜏) = 𝜏ℓ𝑥,𝜏 =⇒ 𝑥 2 ≥ proxℓ (𝑥; 𝜏) 𝑥 =⇒ |𝑥| ≥ |proxℓ (𝑥; 𝜏) 𝑥|. It remains to show that max𝑠∈𝜕ℓ(proxℓ (𝑥;𝜏)) |𝑠| ≤ max𝑠∈𝜕ℓ(𝑥) |𝑠|. Since 0 ∈ arg min𝑣 ℓ(𝑣), it follows by convexity that (0 ≤ 𝑥 1 ≤ 𝑥 2 or 𝑥2 ≤ 𝑥1 ≤ 0) =⇒ max |𝑠| ≤ max |𝑠| 𝑠∈𝜕ℓ(𝑥 1 ) 𝑠∈𝜕ℓ(𝑥 2 ) Observe that the LHS of the implication above is equivalent to (|𝑥 2 | ≥ |𝑥1 | and 𝑥 1 𝑥2 ≥ 0). Then apply it for 𝑥 1 = proxℓ (𝑥; 𝜏) and 𝑥 2 = 𝑥, to conclude. (ix) Please see [137, Thm. 1.25]. On Remark 14 Substituting the envelope function of | · | in (2.40) gives: 2    − 𝛽(𝛼𝐺+𝑍)  𝛽 2𝜏 2 + 𝑠¯E 1  2  − 2𝛽  , |𝛼𝐺 + 𝑍 | ≤ 𝜏𝛽 + (𝛿 − 𝑠¯)E , otherwise 2 2    − 𝛽𝛼 𝐺2  2𝜏 1   − 2𝛽  , |𝛼𝐺 | ≤ 𝜏𝛽 ≥ 0, , otherwise (84a) 𝑠¯E     𝛽𝐺 (𝛼𝐺+𝑍) 2 , |𝛼𝐺 + 𝑍 | ≤ 𝜏𝛽   𝐺 sign(𝛼𝐺 + 𝑍)  , otherwise + (𝛿 − 𝑠¯)E     𝛼𝐺 2 𝛽 𝜏   𝐺 sign(𝐺)  , |𝛼𝐺 k ≤ 𝜏𝛽 q − 𝛽 𝐷 K ≥ 0, , otherwise (84b) 2    (𝛼𝐺+𝑍)  𝜏 2𝜏 + 𝑠¯E  2  − 2𝜏  , |𝛼𝐺 + 𝑍 | ≤ 𝜏𝛽 , otherwise + (𝛿 − 𝑠¯)E  2 2   𝛼2𝜏𝐺  , |𝛼𝐺 | ≤ 𝜏𝛽   − 2𝜏  , otherwise q − 𝛼 𝐷 K ≤ 0. (84c) 𝜏 Define 𝜅 := 𝛽𝛼 and 𝜌 := 𝜏𝛽 . In order to find a sufficient condition for 𝛼 to be zero, we assume 𝛼 → 0, 𝜏 → 0, 𝜌 → 0 and 𝜅 ≥ 0 and look for conditions under which 224 the equations in (84) are consistent. Under these assumptions, one can check that (84c) is satisfied (the argument converges to zero), while, (84b) and (84a) become (𝛿 − 𝑠¯) 2 𝜅 ∫ ∞ ∫ 𝜅 q 2 𝐺 𝜙(𝐺)d𝐺 + (𝛿 − 𝑠¯) 𝐺𝜙(𝐺)d𝐺 ≥ 𝛽 𝐷 K , 𝜅 ∫ 𝜅 ∫ ∞ 𝛿 − 𝑠¯ 2 2 𝛽 ≥ 𝑠¯ + 2 2 𝐺 𝜙(𝐺)d𝐺 + 2(𝛿 − 𝑠¯) 𝜙(𝐺)d𝐺, 𝜅 0 𝜅 (85a) 0 (85b) √ 2 where 𝜙(𝐺) = 𝑒 −𝐺 /2 / 2𝜋 and we multiplied (84a) by 𝛽2 to get (85b). Observe that (85a) upper bounds 𝛽 while (85b) derives a lower bound on it. Thus, consistency of the set of equations (85) is achieved if the following holds: ∫ 𝜅 ∫ ∞ (𝛿 − 𝑠¯) 1 2 (2 𝐺 𝜙(𝐺)d𝐺 + (𝛿 − 𝑠¯) 𝐺𝜙(𝐺)d𝐺) 2 𝜅 𝐷K 0 𝜅 ∫ ∫ ∞ 𝛿 − 𝑠¯ 𝜅 2 𝐺 𝜙(𝐺)d𝐺 + 2(𝛿 − 𝑠¯) 𝜙(𝐺)d𝐺. ≥ 𝑠¯ + 2 𝜅2 0 𝜅 Or, equivalently, 𝐷K ≤ ∫∞ 𝐺 2 𝜙(𝐺)d𝐺 + (𝛿 − 𝑠¯) 𝜅 𝐺𝜙(𝐺)d𝐺) 2 0 ∫ ∫∞ . 𝑠¯ 𝜅 2 𝑠¯ + 2 𝛿− 𝐺 𝜙(𝐺)d𝐺 + 2(𝛿 − 𝑠 ¯ ) 𝜙(𝐺)d𝐺 2 𝜅 0 𝜅 (2 (𝛿−𝜅 𝑠¯) ∫𝜅 (86) Thus if maximum of the right side of (86) with respect to 𝜅 is greater than 𝐷 K , all our variables satisfy (2.40) and the optimal value in (2.39) occurs when 𝛼 → 0, 𝜏 𝜏 → 0 and 𝛼𝛽 → 𝜅 which means 𝛼∗ = 0. We will show that ∫ 2 𝜙(𝐺)d𝐺 + (𝛿 − 𝑠¯) ∞ 𝐺𝜙(𝐺)d𝐺) 2 𝐺 0 𝜅 ∫𝜅 ∫∞ max 𝛿− 𝑠 ¯ 2 𝜅>0 𝑠¯ + 2 𝐺 𝜙(𝐺)d𝐺 + 2(𝛿 − 𝑠¯) 𝜅 𝜙(𝐺)d𝐺 𝜅2 0 ∫ ∞ 2 ≥ 𝛿 − min 𝑠¯ (1 + 𝜅 ) + 2(𝛿 − 𝑠¯) (𝐺 − 𝜅) 2 𝜙(𝐺)d𝐺 (2 (𝛿−𝜅 𝑠¯) ∫𝜅 𝜅>0 (87) 𝜅 If both this and (2.45) are true then, there will be a 𝜅 for which (86) holds and as we discussed, this implies 𝛼∗ = 0. ∫∞ ∫∞ For convenience, we define 𝐴𝜅 = 𝜅 𝐺 2 𝜙(𝐺)d𝐺, 𝐵 𝜅 = 𝜅 𝐺𝜙(𝐺)d𝐺 and 𝐶𝜅 = ∫∞ 𝜙(𝐺)d𝐺. The optimal 𝜅 for the right side of (87) satisfies the following due to 𝜅 the first optimality condition 2(𝛿 − 𝑠¯) 𝜅𝐵 ˆ 𝜅ˆ − 2(𝛿 − 𝑠¯) 𝜅ˆ2𝐶𝜅ˆ = 𝜅ˆ2 𝑠¯ (88) 225 For this value of 𝜅, the left side of (87) becomes ∫ 𝜅ˆ ∫∞ (2 (𝛿−𝜅ˆ 𝑠¯) 0 𝐺 2 𝜙(𝐺) 𝑑𝐺 + (𝛿 − 𝑠¯) 𝜅ˆ 𝐺𝜙(𝐺) 𝑑𝐺) 2 = (𝛿 − 𝑠¯)(1 − 2𝐴𝜅ˆ + 2𝜅𝐵 ˆ 𝜅ˆ ) ∫ ∫∞ 𝑠¯ 𝜅ˆ 2 𝑠¯ + 2 𝛿− 𝐺 𝜙(𝐺) 𝑑𝐺 + 2(𝛿 − 𝑠 ¯ ) 𝜙(𝐺) 𝑑𝐺 𝜅ˆ2 0 𝜅ˆ = 𝛿 − 𝑠¯ − 2(𝛿 − 𝑠¯) 𝜅𝐵 ˆ 𝜅ˆ + 2(𝛿 − 𝑠¯) 𝜅ˆ2𝐶𝜅ˆ − 2(𝛿 − 𝑠¯) 𝐴𝜅ˆ + 4(𝛿 − 𝑠¯) 𝜅𝐵 ˆ 𝜅ˆ − 2(𝛿 − 𝑠¯) 𝜅ˆ2𝐶𝜅ˆ = 𝛿 − 𝑠¯ (1 + 𝜅ˆ2 ) − 2(𝛿 − 𝑠¯)( 𝐴𝜅ˆ − 2𝐵 𝜅ˆ + 𝜅ˆ2𝐶𝜅ˆ ) = 𝛿 − 𝑠¯ (1 + 𝜅ˆ2 ) ∫ ∞ + 2(𝛿 − 𝑠¯) (𝐺 − 𝜅) ˆ 2 𝜙(𝐺) 𝑑𝐺, 𝜅ˆ where the first and third equalities follow after substituting 𝑠¯ using (88). This proves (87) as desired to conclude the claim of the remark. Satisfying Assumption 1(a) on Section 2.1 It only takes a few calculations to show that √  𝜏   √1 k𝛼g + zk2 − 2𝛿  1 √ 𝑛𝛿 e 𝑝k·k2 (𝛼g + z; 𝜏) = k𝛼g+zk 2  1 𝑛  2𝜏 𝑛 2  , if 𝛿k𝛼g+zk2 √ ≥ 𝜏, 𝑛 (89) , otherwise. kzk 2 Assume that 0 < E 𝑛 2 =: 𝜎 2 < ∞. From (89), it can be seen that 𝑛1 e√ 𝑝k·k2 (𝛼g + z; 𝜏) √ √ is a Lipschitz convex function of k𝛼g+zk . Also, k𝛼g+zk converges in probability to 𝑛 𝑛 √ 𝛼2 + 𝜎 2 , thus √  2 2  √ −𝜎 − 𝜏  𝛼 +𝜎  2𝛿 1 √ √ 𝛿 (e 𝑝k·k2 (𝛼g + z; 𝜏) − kzk2 𝑝) → 𝐿 (𝛼, 𝜏) = 1  𝑛  2𝜏 (𝛼2 + 𝜎 2 ) − √𝜎 𝛿  , if 𝛿(𝛼2 + 𝜎 2 ) ≥ 𝜏 2 , , otherwise.