Optimization-Based Statistical Inference: Constrained Inverse Problems, Worst-Case Priors, and Kernel Regression Citation Batlle Franch, Pau (2025) Optimization-Based Statistical Inference: Constrained Inverse Problems, Worst-Case Priors, and Kernel Regression. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/1v1b-6612. https://resolver.caltech.edu/CaltechTHESIS:05302025-005324047 Abstract Optimization provides a worst-case framework for quantifying uncertainty in statistical inference, delivering robust and transparent performance guarantees. While this approach provides rigorous bounds, it cannot easily incorporate large-scale data or produce estimates at a prescribed confidence level. To bridge this gap, this thesis develops optimization-based methods that assimilate data while retaining worst-case robustness, exploring three different contexts: Ill-posed inverse problems, Bayesian inference with unknown priors, and Gaussian process regression. In the first, we introduce a new framework for frequentist, optimization-based intervals that provably achieves desired coverage. The framework unifies many previously proposed optimization-based intervals and disproves a conjecture dating back to 1965. In the second, we introduce data-likelihood constraints in Wald’s two-player zero-sum game, which renders the game computationally tractable and provides explicit certificates of minimax optimality. In the third, we develop new Gaussian process (GP) based methods for learning and solving partial differential equations and operator learning. In each setting, our GP algorithms achieve stronger convergence guarantees than existing machine-learning techniques without sacrificing predictive accuracy. Across these three settings, estimates for the unknown quantity (a finite-dimensional parameter, a prior distribution, or a function, respectively) are obtained as the solution to an optimization problem that characterizes either worst-case or minimax optimality, therefore contributing towards a single optimization-centric view of uncertainty quantification. Item Type: Thesis (Dissertation (Ph.D.)) Subject Keywords: Optimization-based inference, uncertainty quantification, inverse problems, constrained inference, Gaussian process regression, kernel methods Degree Grantor: California Institute of Technology Division: Engineering and Applied Science Major Option: Computing and Mathematical Sciences Minor Option: Applied And Computational Mathematics Awards: W. P. Carey & Co. Prize in Applied Mathematics, 2025. Thesis Availability: Public (worldwide access) Research Advisor(s): Owhadi, Houman Thesis Committee: Tropp, Joel A. (chair) Hoffmann, Franca Braverman, Amy Owhadi, Houman Defense Date: 27 May 2025 Funders: Funding Agency Grant Number National Science Foundation 2425909 Air Force Office of Scientific Research FA9550-18-1-0271 Air Force Office of Scientific Research FA9550-20-1-0358 Air Force Office of Scientific Research FA9550-24-1-0237 NASA Jet Propulsion Laboratory 1695777 Beyond Limits MODEL Center for Autonomous Systems and Technologies at Caltech 25550091 Department of Energy (DOE) DE-SC0023163 Vannevar Bush Faculty Fellowship N000-14-25-1-2035 Record Number: CaltechTHESIS:05302025-005324047 Persistent URL: https://resolver.caltech.edu/CaltechTHESIS:05302025-005324047 DOI: 10.7907/1v1b-6612 Related URLs: URL URL Type Description https://arxiv.org/abs/2310.02461 arXiv Article adapted for Chapter 2 https://arxiv.org/abs/2502.02674 arXiv Article adapted for Chapter 3 https://doi.org/10.1016/j.jcp.2022.111608 DOI Article adapted for Chapter 4 https://doi.org/10.1016/j.jcp.2023.112549 DOI Article adapted for Chapter 5 https://doi.org/10.1016/j.jcp.2024.113488 DOI Article adapted for Chapter 6 https://doi.org/10.1073/pnas.2403449121 DOI Article adapted for Chapter 7 ORCID: Author ORCID Batlle Franch, Pau 0000-0003-4886-058X Default Usage Policy: No commercial reproduction, distribution, display or performance rights in this work are provided. ID Code: 17304 Collection: CaltechTHESIS Deposited By: Pau Batlle Franch Deposited On: 01 Jun 2025 19:37 Last Modified: 09 Jun 2025 20:37 Thesis Files PDF - Final Version See Usage Policy. 20MB Repository Staff Only: item control page

Optimization-based Statistical Inference: constrained inverse problems, worst-case priors, and kernel regression Thesis by Pau Batlle Franch In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy CALIFORNIA INSTITUTE OF TECHNOLOGY Pasadena, California 2025 Defended May 27, 2025 ii © 2025 Pau Batlle Franch ORCID: 0000-0003-4886-058X All rights reserved except where otherwise noted iii To my parents and my sister iv “It’s only when you look at an ant through a magnifying glass on a sunny day that you realize how often they burst into flames.” —Harry Hill. v ACKNOWLEDGEMENTS First, I would like to express my gratitude to my advisor, Houman Owhadi, for his help and support, both academically and financially, through all these years and the different roadblocks and successes of the journey. Thanks as well to the professors serving on my candidacy and thesis committees, Andrew Stuart, Franca Hoffmann, Joel Tropp, and Amy Braverman, for their time and helpful feedback. Many thanks to all of my collaborators throughout the years; every project has been a learning experience, and I am grateful for the time and effort everyone has given to such projects. Special thanks to Mike Stanley, who first introduced me to the field of constrained inverse problems, a field I deeply enjoyed working in for the last three years. My journey would surely not have been the same had I never encountered such research problems, and I am quite proud of the joint effort we put in advancing that field. Thank you as well to Pratik Patil, Mikael Kuusela, and Javier Ruiz-Lupon for their help and collaboration in this project. Thanks to Bamdad Hosseini, Yifan Chen, and Matthieu Darcy for countless research and casual discussions about the intricacies of Gaussian Processes. Similarly, discussing UQ and Kernel Flows with Jouni Susiluoto, Otto Lamminpaeae, and Amy Braverman from the Uncertainty Quantification group at JPL has been extremely enjoyable and motivating to pursue my research in the area. Thank you to Michael Hecht for hosting me as a visiting researcher in CASUS, Germany, and to Boumediene Hamzi for the opportunity to collaborate with Alpha Lambda Iota. On a personal level, I would like to thank all of my family for their support, especially my parents, Pere and Anna, and my sister Carla, whose optimism and cheerful energy motivate me to be better and happier every day. I am extremely grateful to have met wonderful people during my PhD, whether at conferences or at Caltech, whom I am very happy to consider good friends. I would like to give special thanks to Matthieu Darcy, Eitan Levin, and Joe Slote for all the support and help through all the high highs and low lows of this period of my life. It has been an immense pleasure to share all the experiences we shared through the years. Many thanks as well to many other friends and colleagues who I have shared plenty of good times and experiences with: Theo Bourdais, Edoardo Calvello, Siki Wang, Dohyeon Kim, Chris Yeh, Roy Wang, Margaret Trautner, John Lathrop, Lauren vi Conger, Sebastian Lamas, Matt Levine, Francesca Li, Janina Schreiber, Jamiree Harrison, and Nina Fischer, thanks to every one of you for all the unforgettable moments. Thanks as well to my university colleagues Pol Mestres, Adam Teixidó, and Tomàs Ortega for the adventures around Southern California, and to my friends back home Andreu Vergés, Pau Castillo, and Xavier Rubiés, for sweetening my periodic visits through games of padel and late-night dinners. vii ABSTRACT Optimization provides a worst-case framework for quantifying uncertainty in statistical inference, delivering robust and transparent performance guarantees. While this approach provides rigorous bounds, it cannot easily incorporate large-scale data or produce estimates at a prescribed confidence level. To bridge this gap, this thesis develops optimization-based methods that assimilate data while retaining worst-case robustness, exploring three different contexts: Ill-posed inverse problems, Bayesian inference with unknown priors, and Gaussian process regression. In the first, we introduce a new framework for frequentist, optimization-based intervals that provably achieves desired coverage. The framework unifies many previously proposed optimization-based intervals and disproves a conjecture dating back to 1965. In the second, we introduce data-likelihood constraints in Wald’s two-player zero-sum game, which renders the game computationally tractable and provides explicit certificates of minimax optimality. In the third, we develop new Gaussian process (GP) based methods for learning and solving partial differential equations and operator learning. In each setting, our GP algorithms achieve stronger convergence guarantees than existing machine-learning techniques without sacrificing predictive accuracy. Across these three settings, estimates for the unknown quantity (a finite-dimensional parameter, a prior distribution, or a function, respectively) are obtained as the solution to an optimization problem that characterizes either worst-case or minimax optimality, therefore contributing towards a single optimization-centric view of uncertainty quantification. viii PUBLISHED CONTENT AND CONTRIBUTIONS Stanley, Michael, Pau Batlle, Pratik Patil, Houman Owhadi, and Mikael Kuusela (2025). “Confidence intervals for functionals in constrained inverse problems via data-adaptive sampling-based calibration”. In: arXiv preprint arXiv:2502.02674. P.B contributed to conceptualization, proof of the theoretical results, and numerical experiments. Batlle, Pau, Matthieu Darcy, Bamdad Hosseini, and Houman Owhadi (2024). “Kernel methods are competitive for operator learning”. In: Journal of Computational Physics 496. P.B and M.D share equal contribution as main contributors of this work, with contributions to conceptualization, writing, and numerical results., p. 112549. doi: https://doi.org/10.1016/j.jcp.2023.112549. Kaveh, Hojjat, Pau Batlle, Mateo Acosta, Pranav Kulkarni, Stephen J Bourne, and Jean Philippe Avouac (2024). “Induced seismicity forecasting with uncertainty quantification: Application to the Groningen gas field”. In: Seismological Research Letters 95.2A. P.B contributed to the conceptualization of the Uncertainty Quantification algorithm and writing of its corresponding section, pp. 773–790. doi: https : //doi.org/10.1785/0220230179. Pandey, Biraj, Bamdad Hosseini, Pau Batlle, and Houman Owhadi (2024). “Diffeomorphic Measure Matching with Kernels for Generative Modeling”. In: SIAM Journal on Mathematics of Data Science (to appear). P.B contributed to conceptualization, early prototyping, and editing. Schreiber, Janina, Pau Batlle, Damar Wicaksono, and Michael Hecht (2024). “PMBO: Enhancing Black-Box Optimization through Multivariate Polynomial Surrogates”. In: arXiv preprint arXiv:2403.07485. P.B contributed to the conceptualization, early prototyping, and editing. Batlle, Pau, Yifan Chen, Bamdad Hosseini, Houman Owhadi, and Andrew M Stuart (2023). “Error analysis of kernel/GP methods for nonlinear and parametric PDEs”. In: Journal of Computational Physics. P.B contributed to numerical results of Section 4, and writing. doi: https : //doi.org/10.1016/j.jcp.2024.113488. Batlle, Pau, Pratik Patil, Michael Stanley, Houman Owhadi, and Mikael Kuusela (2023). “Optimization-based frequentist confidence intervals for functionals in constrained inverse problems: Resolving the Burrus conjecture”. In: arXiv preprint arXiv:2310.02461. P.B is the main contributor of this work, and contributed to the conceptualization, investigation, proof of the main mathematical results, numerical experiments, and writing. ix Bourdais, Théo, Pau Batlle, Xianjin Yang, Ricardo Baptista, Nicolas Rouquette, and Houman Owhadi (2023). “Codiscovering graphical structure and functional relationships within data: A Gaussian Process framework for connecting the dots”. In: Proceedings of the National Academy of Sciences (PNAS). P.B contributed to conceptualization, early prototyping, and writing. doi: https: //doi.org/10.1073/pnas.2403449121. Dingle, Kamaludin, Pau Batlle, and Houman Owhadi (2023). “Multiclass classification utilising an estimated algorithmic probability prior”. In: Physica D: Nonlinear Phenomena 448. P.B contributed to the conceptualization of the Gaussian process-based algorithm and writing revision, p. 133713. doi: https://doi.org/10.1016/j.physd. 2023.133713. Bajgiran, Hamed Hamze, Pau Batlle, Houman Owhadi, Mostafa Samir, Clint Scovel, Mahdy Shirdel, Michael Stanley, and Peyman Tavallali (2022). “Uncertainty quantification of the 4th kind; optimal posterior accuracy-uncertainty tradeoff with the minimum enclosing ball”. In: Journal of Computational Physics 471. P.B contributed to the conceptualization, writing, and numerical experiments in Sections 1 and 6., p. 111608. doi: https://doi.org/10.1016/j.jcp.2022. 111608. x TABLE OF CONTENTS Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Published Content and Contributions . . . . . . . . . . . . . . . . . . . . . . viii Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix List of Illustrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii Chapter I: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Chapter II: Optimization-based frequentist confidence intervals for functionals in constrained inverse problems: Resolving the Burrus conjecture . . . 7 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Strict bounds intervals from test inversion . . . . . . . . . . . . . . . 18 2.3 General interval construction methodology . . . . . . . . . . . . . . 29 2.4 Refuting the Burrus conjecture . . . . . . . . . . . . . . . . . . . . 34 2.5 Numerical examples . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 2.7 Proofs in Section 2.2 . . . . . . . . . . . . . . . . . . . . . . . . . . 50 2.8 Proofs in Section 2.3 . . . . . . . . . . . . . . . . . . . . . . . . . . 56 2.9 Proofs in Section 2.4 . . . . . . . . . . . . . . . . . . . . . . . . . . 59 2.10 Additional numerical illustrations in Section 2.5 . . . . . . . . . . . 67 Chapter III: Confidence intervals for functionals in constrained inverse problems via data-adaptive sampling-based calibration . . . . . . . . . . . . . 69 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 3.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 3.3 Interval constructions . . . . . . . . . . . . . . . . . . . . . . . . . 80 3.4 Theoretical justification . . . . . . . . . . . . . . . . . . . . . . . . 88 3.5 Implementation methodology . . . . . . . . . . . . . . . . . . . . . 90 3.6 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . 100 3.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 3.8 Proofs in Section 3.3 . . . . . . . . . . . . . . . . . . . . . . . . . . 111 3.9 Proof of Theorem 3.4.1 . . . . . . . . . . . . . . . . . . . . . . . . 111 3.10 Additional details and illustrations in Section 3.6 . . . . . . . . . . . 115 Chapter IV: Uncertainty Quantification of the 4th kind; optimal posterior accuracy-uncertainty tradeoff with the minimum enclosing ball . . . . . . 118 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 4.2 Previous approaches to UQ . . . . . . . . . . . . . . . . . . . . . . 126 4.3 Uncertainty Quantification of the 4th Kind . . . . . . . . . . . . . . 130 4.4 Computational framework . . . . . . . . . . . . . . . . . . . . . . . 138 4.5 Minimum enclosing ball algorithm . . . . . . . . . . . . . . . . . . 147 4.6 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 xi 4.7 General loss functions and rarity assumptions . . . . . . . . . . . . . 156 4.8 Supporting theorems and proofs . . . . . . . . . . . . . . . . . . . . 160 Chapter V: Kernel methods are competitive for operator learning . . . . . . . 172 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 5.2 The RKHS/GP framework for operator learning . . . . . . . . . . . 183 5.3 Convergence and error analysis . . . . . . . . . . . . . . . . . . . . 189 5.4 Numerics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 5.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 5.6 Review of operator valued kernels and GPs . . . . . . . . . . . . . . 208 5.7 An alternative regularization of operator regression . . . . . . . . . . 212 5.8 Expressions for the kernels used in experiments . . . . . . . . . . . . 213 Chapter VI: Error Analysis of Kernel/GP Methods for Nonlinear and Parametric PDEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 6.2 Kernel Methods for Parametric PDEs . . . . . . . . . . . . . . . . . 224 6.3 Error Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 6.4 Numerical Experiments . . . . . . . . . . . . . . . . . . . . . . . . 246 6.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 6.6 Sobolev Sampling Inequalities on Manifolds . . . . . . . . . . . . . 253 6.7 Bounds on Fill Distances . . . . . . . . . . . . . . . . . . . . . . . . 254 6.8 The Choice of Nugget Terms . . . . . . . . . . . . . . . . . . . . . 255 Chapter VII: Discovering graphical structure and functional relationships within data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 7.1 Additional details on our proposed approach. . . . . . . . . . . . . . 273 7.2 Algorithm Overview for Type 3 problems: An Informal Summary . . 276 7.3 Type 2 problems: Formal description and GP-based Computational Graph Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 7.4 Hardness and well-posed formulation of Type 3 problems. . . . . . . 280 7.5 A Gaussian Process method for Type 3 problems . . . . . . . . . . . 284 7.6 Algorithm pseudocode. . . . . . . . . . . . . . . . . . . . . . . . . 295 7.7 Analysis of the signal-to-noise ratio test. . . . . . . . . . . . . . . . 298 7.8 Supplementary information on examples. . . . . . . . . . . . . . . . 303 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308 xii LIST OF ILLUSTRATIONS Number Page 1.1 Setup of the particle unfolding inference problem. In it, we are aiming to recover a function 𝜑 of a discrete distribution of particle counts 𝑥 ∗ , given a distribution 𝑦 of blurred particle counts obtained by a noisy measurement 𝑦 = 𝐹 (𝑥 ∗ ) + 𝜀. . . . . . . . . . . . . . . . . . . . . . . 3 1.2 In optimization-based methods, the observation of the data can be used to shrink the constraint set of the optimization programs considered 4 1.3 In Optimization-based methods, the observation of the data can be used to shrink the constraint set of the optimization programs considered 5 2.1 Illustration of the problem setup. We seek to construct confidence intervals [𝐼 − ( 𝒚), 𝐼 + ( 𝒚)] ⊆ R for 𝜑(𝒙 ∗ ) ∈ R from an observation 𝒚 ∈ R𝑚 sampled from 𝑃𝒙 ∗ that satisfies a frequentist coverage guarantee in finite sample while being as small (in length) as possible. . . . . . 9 2.2 Illustration of the simultaneous approach for confidence interval building, which works generically for any X, 𝜑, and 𝑃. The intersection of X and C( 𝒚) occurs in the original parameter space R 𝑝 , and is then projected via the functional of interest function into the real line. The confidence interval is then constructed using the minimum and maximum of the quantity of interest 𝜑 over the intersection X ∩ C( 𝒚). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.3 Illustration of the classical duality between hypothesis testing and confidence set building as seen in the product space of the data and parameter spaces. Pairs of the dual hypothesis test and the confidence set can be viewed as a set S in the product space (the compatibility region). For fixed data 𝒚, a confidence set is given by C(𝒚) = {𝒙 : (𝒚, 𝒙) ∈ S}, and for a fixed parameter 𝒙, the acceptance region is given by A (𝒙) = {𝒚 : ( 𝒚, 𝒙) ∈ S}. . . . . . . . . . . . . . . . . . . 19 xiii 2.4 Comparison of quantile functions and CDFs for LLRs with different true parameter values. The left panel provides the true values of the quantile function as a function of 𝑥 ∗ across different confidence levels. As proven in Example 2.3.3, the quantile of 𝜒12 is greater than the true quantile for all 𝑥 ∗ and all levels. The right panel shows the CDFs for LLRs under different values of the true parameters, 𝑥 ∗ . From Example 2.3.3, as the true parameter increases, the CDF is increasingly dominated by its 𝜒12 component, so it follows that as 𝑥 ∗ increases, the CDF approaches the 𝜒12 CDF. This figure also provides a visual explanation of why using the true quantile or the true quantile function to compute the interval in (2.24) produces shorter intervals compared to those computed with the 𝜒12 quantile. . . . . . . . . . . 32 2.5 Difference of cumulative distribution functions between the LLR test statistic and 𝜒12 distribution for the statistics defined in (2.44) (left) and (2.45) (right). Stochastic dominance, which is equivalent to the Burrus conjecture, is broken in the right example only. There is a direct correspondence between the points at which the CDF difference is negative and confidence levels 1 − 𝛼 that fail to hold (see Remark 3). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.6 The (left) figure shows estimated coverage for each 95% confidence interval and each true 𝑥 ∗ for the one-dimensional constrained Gaussian model. Both the Truncated Gaussian (SSB) and OSB intervals overcover when 𝑥 ∗ = 0 while the MQ𝜇 interval predictably achieves nominal coverage. All the coverage values converge as 𝑥 ∗ gets larger, as the problem moves toward the unconstrained problem where all the intervals are effectively the same. The intervals surrounding the estimated values are 95% Clopper–Pearson intervals, expressing the Monte Carlo uncertainty of each coverage estimate. The (right) figure shows an estimate of the expected interval length for each method (with 95% confidence intervals that are nearly length zero since the standard error of each estimate is nearly zero with 105 realizations each). Similarly to the coverage results in the left panel, as 𝑥 ∗ gets larger, the expected OSB and MQ𝜇 interval lengths converge while the SSB intervals remain slightly larger. . . . . . . . . . . . . . . . . 43 xiv 2.7 Estimated interval coverage (left) and expected lengths (right) for 95% intervals resulting from the SSB, OSB, MQ, and MQ𝜇 methods for the Gaussian linear model in (2.4) with 𝑲 = 𝑰2 , 𝜑(𝒙) = 𝒉⊤ 𝒙 = 𝑥 1 − 𝑥 2 , and X = {𝒙 ∈ R2 : 𝒙 ≥ 0}. . . . . . . . . . . . . . . . . . . 44 2.8 Estimated 95th quantiles from the LLR test statistic null distributions in the region [0, 1] 2 ⊂ R2+ where color shows the estimated quantiles. In contrast to the unconstrained case, the quantile is dependent on the true parameter, and the quantile surface is non-trivial. . . . . . . . . 44 2.9 Estimated interval coverage (left) and expected lengths (right) for 68% intervals resulting from the SSB, OSB, MQ, and MQ𝜇 methods for the Gaussian linear model in (2.4) with 𝑲 = 𝑰3 , 𝜑(𝒙) = 𝒉⊤ 𝒙 = 𝑥 1 + 𝑥2 − 𝑥 3 , and X = {𝒙 ∈ R3 : 𝒙 ≥ 0}. . . . . . . . . . . . . . . . 45 2.10 For the numerical example in Section 2.5, considering 𝒙 ∗ ∈ {𝒙 ∗ (𝑡) = (𝑡, 𝑡, 1) ⊤ : 0 < 𝑡 ≤ 𝑒} ⊂ R3 , we estimate the 68% LLR test statistic quantiles along with 95% nonparametric (NP) confidence intervals for percentiles (Hahn and 2 Meeker, 1991). The test statistic quantiles exceeding 𝜒1,0.32 correspond to the Burrus conjecture failing in this scenario. We note that for this example, the Burrus conjecture fails close to the constraint boundary while the 𝑄 𝜒2 (0.68) quantile becomes valid once suffi1 ciently far from the boundary. . . . . . . . . . . . . . . . . . . . . . 46 2.11 Estimated interval coverage (left) and expected lengths (right) for 95% intervals resulting from the SSB, OSB, MQ, and MQ𝜇 methods for the Gaussian linear model in (2.4) with 𝑲 = 𝑰2 , 𝜑(𝒙) = 𝒉⊤ 𝒙 = 𝑥 1 − 𝑥 2 , and X = [0, 1] 2 ⊂ R2 . . . . . . . . . . . . . . . . . . . . . 47 2.12 Estimated interval coverage (left) and expected lengths (right) for 95% intervals resulting from the SSB, OSB, MQ, and MQ𝜇 methods for the Gaussian linear model in (2.4) with 𝑲 = 𝑰3 , 𝜑(𝒙) = 𝒉⊤ 𝒙 = 𝑥 1 + 𝑥2 − 𝑥 3 , and X = {𝒙 ∈ R3 : 𝒙 ≥ 0}. . . . . . . . . . . . . . . . 68 xv 3.1 (Left) A particular quantile surface, 𝑄 𝒙 (1 − 𝛼), where 𝒙 ≥ 0 and 𝛼 = 0.32. This surface was obtained via Monte Carlo sampling the LLR test statistic over a grid of 𝒙’s defined by the two-dimensional constrained-Gaussian scenario similar to that in Section 3.6, but with  ⊤ 𝒉 := 1 1 . (Right-Top) An illustration of the Berger–Boos set and other, a 1 − 𝜂 confidence set for 𝒙 ∗ , which prevents having to contend with an unbounded parameter space. Additionally, it can eliminate “worst-case” parameter settings in X that are far from the data, thus potentially making the resulting intervals less conservative. (RightBottom) An illustration of an LLR test statistic curve, 𝜆(𝜇, 𝒚), over the QoI domain and the interval endpoints that result from using either Equation (3.5) for the “Sliced” interval or Equation (3.6) for the “Global” interval. . . . . . . . . . . . . . . . . . . . . . . . . . . 71 3.2 Numerical illustrations of the VGS Sampler’s infeasibility in high dimensional regimes. The left panel shows the computed acceptance probability of a point drawn by the VGS Sampler with data generated from a non-negatively constrained Gaussian noise model. Crucially, at only 30 dimensions, the acceptance probability is already less than 10−4 for this particular setup. The right panel shows the computed probability mass function for the number of non-negative constraint complying coordinates of a VGS sample with data generated from a non-negatively constrained linear Gaussian model in 40 dimensions with a non-identity forward model. Since this is an example using a forward model with a large condition number (≈ 1.6 × 104 ), we critically note that there is empirically zero probability of generating a sample within the non-negativity constraints. . . . . . . . . . . . . 93 3.3 Polytope sampler output for a realization of the 80-dimensional illposed inverse problem studied in Section 3.6. The left panel contains a histogram of sampled functional values which both span and cover well the range of 𝐼BB ( 𝒚) (shown by the dashed gray lines in both plots). The right panel contains trace plots of the 14 Vaidya walks (each indicated by a different color) which together constitute the full sample. Our heuristic for choosing starting points along the lines connecting the parameter settings generating the endpoints of 𝐼BB (𝒚) and the Chebyshev center of the polytope provides a good initial spread of starting functional values. . . . . . . . . . . . . . . . . . . 98 xvi 3.4 Estimated coverages and expected lengths across all four interval constructions and OSB for comparison at the 68% level for the twodimensional constrained Gaussian setting. All four of our interval constructions are comparable to OSB with respect to coverage, but OSB shows better-expected length performance, aside from our two sliced interval constructions. Although the OSB intervals are defined using the global max-quantile (𝑄 max 1−𝛼 ) and therefore can potentially be improved upon by limiting the considered parameter space via the Berger–Boos set, due to the rapidity with which the 𝛼-quantile 2 quantile (see Figure 5.3 in (Batlle, Stanley, surface meets the 𝜒1,𝛼 et al., 2023)), the OSB interval lengths are difficult to beat in practice. 103 3.5 (Left) Four realizations of the data-generating process where the observations are shown in red. For each realization, the blue points are uniformly distributed samples from its Berger–Boos set, sampled using the VGS sampler. (Center) For a realization of the data-generating process, we plot the distribution of 𝛾-quantiles for the points sampled by the VGS sampler. Notably, a non-trivial percent of these are 2 defining the OSB interval. (Right) For the same realizaabove 𝜒1,𝛼 tion, we plot the estimated sliced max-quantile function, 𝑚 b𝛾 (𝜇) in 2 in red. The blue points correspond to sampled orange alongside 𝜒1,𝛼 parameter values, each of which has a functional and quantile value, while the solid blue line shows the LLR over the functional varies. All intervals can be read immediately from this image by inspecting where the blue LLR curve intersects the sampled points. . . . . . . . 104 3.6 Estimated coverage and expected length across all four interval constructions and OSB for comparison at the 68% level for the threedimensional constrained Gaussian example. All four of our interval constructions achieve nominal coverage while the OSB interval does not. While the Global interval constructions pay a steep price in expected length compared to OSB, the Sliced constructions are only slightly longer than OSB. . . . . . . . . . . . . . . . . . . . . . . . . 104 xvii 3.7 Confidence interval lengths in the Berger–Boos setting, averaged over values of 𝒚, for varying 𝜂 and 𝒙 ∗ . The minimum average length occurs at a small 𝜂 > 0, showing that the construction is beneficial if 𝜂 is tuned correctly. This occurs because even for moderately small 𝜂, the Berger–Boos set, which is a three-dimensional sphere intersected with the non-negative orthant, avoids the point with the highest 1 − 𝛼 quantile. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 3.8 Parameter values for the smooth and adversarial settings for 𝒙 ∗ used to illustrate our interval construction versus the OSB interval. The adversarial setting is made more difficult by the sharp jumps in parameter values. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 3.9 3.10 3.11 Estimated coverage and expected length across all four interval constructions and OSB at the 68% level for the smooth wide-bin deconvolution experiment. While the Global interval constructions overcover like the OSB interval, the Sliced interval constructions reduce both over-coverage and expected interval length. . . . . . . . . . . . 108 Estimated coverage and expected length across all four interval constructions and OSB at the 68% level for the adversarial wide-bin deconvolution experiment. While the OSB interval fails to achieve nominal coverage, all four of our interval constructions do. Interestingly, the Sliced interval constructions are meaningfully shorter than the OSB interval while also providing coverage. . . . . . . . . . . . . 109 When sampling points using Algorithm 6, the vast majority of samples are found closer to the non-negativity constraint boundary. The true parameter setting is shown by the red point, while the parameter settings sampled by Algorithm 6 are shown by the blue points. This sampling prioritization helps adequately sample the regions of the 2 . FurBerger–Boos set where the quantile surface is larger than 𝜒1,𝛼 thermore, the vast majority of sampled points lie within the Berger– Boos set with some lying outside within the bounding polytope. . . . 116 xviii 3.12 4.1 4.2 4.3 4.4 4.5 4.6 The importance-like sampler described by Algorithm 6 is more effective than the Polytope sampler described by Algorithm 5 at sampling parameter settings with 𝛾-quantile greater than 1.1. Each parameter setting sampled by Algorithm 6 is shown by a blue point. This improved ability helps ensure the coverage guarantee shown in the left panel of Figure 3.6. . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 The Uncertainty Quantification (UQ) problem. Here, Θ is the space of parameters, 𝜃 † is the true unknown parameter, 𝜑 a quantity of interest, and 𝑃 is the physical model determining the distribution 𝑝 from which the data 𝑥 is observed . . . . . . . . . . . . . . . . . . . 119 Example of the minimum enclosing ball 𝐵 about the image 𝜑(Θ𝑥 (𝛼)) (in green) with radius 𝑅 and center 𝑑 = 𝑧∗ . An optimal discrete Í measure 𝜇 := 𝑤 𝑖 δ𝑧𝑖 (𝑧𝑖 = 𝜑(𝜃 𝑖 )) on the range of 𝜑 for the maximum variance problem is characterized by the fact that it is supported on Í the intersection of 𝜑(Θ𝑥 (𝛼)) and 𝜕𝐵 and 𝑑 = 𝑧 ∗ = 𝑤 𝑖 𝑧𝑖 is the center of mass of the measure 𝜇. The size of the solid red balls indicates the size of the corresponding weights 𝑤 𝑖 . . . . . . . . . . . . . . . . . . 122 𝛼 − 𝛽 relation, likelihood level sets, risk value and decision for different choices of 𝛼 (and consequently 𝛽) for the 1 coin problem after observing four heads and one tail. Three different values in the 𝛼 − 𝛽 curve are highlighted across the plots . . . . . . . . . . . . . . . . . 125 2D likelihood level sets and minimum enclosing balls for different values of 𝛼, visualized as level sets of the likelihood function (left) and projected onto a 2D plane (right) . . . . . . . . . . . . . . . . . 126 Curse of dimensionality in discretizing the prior. The data is of the form 𝑥 = 𝑚(𝜃) + 𝜖N (0, 1) where 𝑚 is deterministic and 𝜖N (0, 1) is small noise. (a) For the continuous prior, the posterior concentrates around M := {𝜃 ∈ Θ|𝑚(𝜃) = 𝑥}. (b) For the discretized prior, the posterior concentrates on the delta Dirac that is the closest to M. . . 129 𝛼 − 𝛽 relation, likelihood level sets, risk value and decision for different choices of 𝛼 (and consequently 𝛽) for the normal mean estimation problem with 𝜏 = 3 and observed value 𝑥 = 1.5. Three different values in the 𝛼 − 𝛽 curve are highlighted across the plots . . . . . . . . 150 xix 4.7 4.8 4.9 4.10 5.1 5.2 5.3 5.4 5.5 Quadratic model results: (left-top) 𝛽𝛼 vs 𝛼, (right-top) risk vs 𝛽𝛼 , (left-bottom) the supporting surfaces with the decision points at the center of each surface and (right-bottom) maximum-likelihood solution and supporting points of the minimum enclosing ball for 𝛽𝛼 = 𝛽∗ = .05 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 (Top image) Data D := 𝒙 (𝑡) , 𝑡 ∈ 𝑇, generated, according to the Gaussian noise model (4.83): solid red is the prey component and solid blue the predator component of the generated data 𝒙 (𝑡) , the dotted red and dotted blue are the prey-predator components of the LotkaVolterra solution 𝒎(𝑡, 𝜽 ∗ ) for 𝑡 ∈ 𝑇. (Bottom image) Uncertainty in the population dynamics corresponding to the worst-case measure: (1) red is prey and blue is predator, (2) solid line is the Lotka-Volterra evolution 𝒎(𝑡, 𝜽 ∗ ), 𝑡 ∈ 𝑇, fine dots 𝒎(𝑡, 𝜽 1 ), 𝑡 ∈ 𝑇, and coarse dots 𝒎(𝑡, 𝜽 2 ), where 𝑆 := {𝜽 1 , 𝜽 2 } is the set of support points of the worst case (posterior) measure (located on the boundary of the minimum enclosing ball). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Minimum enclosing ball for the Lotka-Volterra Model: (left) the dashed line indicates the boundary of the likelihood region ΘD (𝛼), the solid circle its minimum enclosing ball, the red point the data generating value 𝜽 ∗ and the blue point the center of the minimum enclosing ball and the optimal estimate of 𝜽 ∗ . The two yellow points comprise the set 𝑆 := {𝜽 1 , 𝜽 2 }. (right) a projected view with the yellow columns indicating the weights (.5, .5) of the set 𝑆 in their determination of a worst-case (posterior) measure. . . . . . . . . . . 155 Monte Carlo numerically confirms the result from Theorem 4.4.1 in both the quadratic function estimation (left image) and LotkaVolterra (right image) examples. . . . . . . . . . . . . . . . . . . . 155 Commutative diagram of our operator learning setup. . . . . . . . . . 175 Example of training data and test prediction and pointwise errors for the Darcy flow problem (5.3). . . . . . . . . . . . . . . . . . . . . . 179 Generalization of fig. 5.1 to the mesh invariant setting where the measurement functionals are different at test time. . . . . . . . . . . 189 Example of training data and test prediction and pointwise errors for the Burger’s equation (5.49). . . . . . . . . . . . . . . . . . . . . . . 201 Example of training data and test prediction and pointwise errors for the Advection problem (5.50)-I. . . . . . . . . . . . . . . . . . . . . 202 xx 5.6 5.7 5.8 5.9 5.10 6.1 6.2 6.3 6.4 7.1 7.2 7.3 Example of training data and test prediction and pointwise errors for the Advection problem (5.50)-II. . . . . . . . . . . . . . . . . . . . . 203 Example of training data and test prediction and pointwise errors for the Helmholtz problem (5.53). . . . . . . . . . . . . . . . . . . . . . 203 Example of training data and test prediction and pointwise errors for the Structural Mechanics problem (5.54). . . . . . . . . . . . . . . . 204 Example of training data and test prediction and pointwise errors for the Navier-Stokes problem (5.55). . . . . . . . . . . . . . . . . . . . 205 Accuracy complexity tradeoff achieved in the problems in (De Hoop et al., 2022). Data for NNs was obtained from the aforementioned article. Linear model refers to the linear kernel, vanilla GP is our implementation with the nonlinear kernels and minimal preprocessing, GP+PCA corresponds to preprocessing through PCA both the input and the output to reduce complexity. . . . . . . . . . . . . . . . . . . 207 A summary of the main steps in our proof of convergence rates outlined in Theorems 6.3.1, 6.3.2, 6.3.4 and 6.3.6. The 1–4 norms denote arbitrary norms on appropriate Banach spaces while the ∥ · ∥ U norm can be chosen as an RKHS norm or another desired norm with respect to which the numerical algorithm is stable. . . . . . . . . . . 220 𝐿 2 test errors of solutions to Problem (6.39) as a function of the number of collocation points. Left: 𝛽 = 1; right: 𝛽 = 4. In both cases, we choose Matérn kernel with 𝜈 = 7/2. Reported slopes in the legend denote empirical convergence rates. . . . . . . . . . . . . . . . . . . 247 𝐿 2 test errors of solutions to Problem (6.39) as a function of the number of collocation points with 𝛽 = 4. Left: Matérn kernel with 𝜈 = 5/2; right: Matérn kernel with 𝜈 = 9/2. Reported slopes in the legend denote empirical convergence rates. . . . . . . . . . . . . . . 248 𝐿 2 test error of solutions to Problem (6.41) as a function of the number of collocation points. Left: vanilla gaussian kernel; Right: Gaussian kernel adapted to the regularity of 𝐴. Reported slopes in the legend denote empirical convergence rates. . . . . . . . . . . . . . . . . . . 250 The three levels of complexity of function approximation. . . . . . . 257 Ancestors identification in Type 3 problem. . . . . . . . . . . . . . . 259 (a-d) The Fermi-Pasta-Ulam-Tsingou system. (e-k) The Google Covid 19 open data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 xxi 7.4 7.5 7.6 7.7 7.8 (a-c) Chemical reaction network. (d-g) Algebraic equations. (h-j) Cell signaling network. . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 (a) Cell signaling network comparisons. (b-h) The BCR reaction benchmark. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 Histogram of the eigenvalues of 𝐷 𝛾 =(7.11) for 𝛾 = 10−2 (good choice) and 𝛾 = 10−6 (bad choice). . . . . . . . . . . . . . . . . . . . 273 Formal description of Type 2 problems. . . . . . . . . . . . . . . . . 277 (a) Electric circuit. (b) Resistance, capacitance, and inductances are nonlinear functions of currents and voltages (c) Measurements. (d) Kirchhoff’s circuit laws. (e) The computational graph with unknown functions represented as red edges. (f) Recovered functions. . . . . . 279 Computational Hypergraph Discovery with three variables . . . . . . 280 7.9 7.10 The structure of the hypergraph is identifiable in (a), (b), and non-identifiable in (c). The relationship between variables is implicit in (d). . . . . . . . . 281 7.11 (a) CHD formulation as a manifold discovery problem and hypergraph representation, (b) The hypergraph representation of an affine manifold is equivalent to its Row Echelon Form Reduction. . . . . . . 283 7.12 Feature map generalization . . . . . . . . . . . . . . . . . . . . . . . 285 7.13 Iterating by removing the least active modes from the signal . . . . . 294 7.14 Computing the ancestors of the variable 𝑥¤0 in the Fermi-Pasta-UlamV (𝑛) Tsingou problem. (a) Noise-to-Signal Ratio, denoted as V (𝑠)+V (𝑛) (𝑞), with respect to the number of proposed ancestors, represented by 𝑞. Additionally, we include a visualization of the quantiles derived from the 𝑍-test, as described in Section 7.7. Notably, when there is no signal present, the noise-to-signal ratio is expected to fall within the shaded area with a probability of 0.9. (b) Increments in the Noise-to-Signal V (𝑛) V (𝑛) Ratio, defined as V (𝑠)+V (𝑛) (𝑞) − V (𝑠)+V (𝑛) (𝑞 − 1), as a function of the number of ancestors, denoted as 𝑞. The horizontal axis represents the number of proposed ancestors for 𝑥¤0 . Determining an appropriate stopping point based solely on absolute noise-to-signal ratio levels can be challenging. In contrast, the increments in the noise-to-signal ratio clearly exhibit a discernible maximum, offering a practical point for decision-making. . . . . . . . . . . . . . . . . . . . . . . . . . . 295 xxii LIST OF TABLES Number Page 3.1 Summary of methods based on global and sliced max quantile, and whether they are optimization-based or inversion-based. . . . . . . . 88 4.1 Comparison of the three previous approaches to uncertainty quantification with our proposed method in Section 4.3 . . . . . . . . . . . . 130 5.1 The 𝐿 2 relative test error of the Darcy flow problem in our running example. The kernel approach is compared with variations of DeepONet and FNO. Results of our kernel method are presented below the dashed line with the pertinent choice of the kernel 𝑆. . . . . . . . 179 5.2 Summary of datasets used for benchmarking. The first three examples were considered in (L. Lu, X. Meng, et al., 2022), and the last four were taken from (De Hoop et al., 2022). . . . . . . . . . . . . . . . . 199 5.3 Summary of numerical results: we report the 𝐿 2 relative test error of our numerical experiments and compare the kernel approach with variations of DeepONet , FNO, PCA-Net, and PARA-Net. We considered two choices of the kernel 𝑆, the rational quadratic and the Matérn, but we observed little difference between the two. . . . . . . 206 5.4 Comparison between Cholesky preconditioning and PCA dimensionality reduction on three examples for our vanilla kernel implementation with the Matérn kernel. . . . . . . . . . . . . . . . . . . . . . . 207 6.1 A qualitative comparison of the properties of traditional PDE solvers (such as FEM, FVM, FDM, spectral methods, etc.) against kernel methods and ANNs. . . . . . . . . . . . . . . . . . . . . . . . . . . 223 6.2 Numerical results for the HJB equation (6.42), computing the quantity 𝑉 (𝒙 0 , 0). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 1 Chapter 1 INTRODUCTION A crucial part of the scientific process is the combination of observations of realworld data with previously known knowledge about a given physical process, to obtain a theory that can generalize and predict future events. Given the current scale of available computing power and data, classical scientific discovery has been increasingly complemented with computational and data-driven statistical tools. In this context, statistical inference methodologies aim to provide uncertainty estimates in the process of recovering unknowns from a combination of data and previous information. This thesis explores theory and applications of optimization-based perspectives on statistical inference, extending its original formulation in the context of worst-case Uncertainty Quantification into a more modern setup in which it aims to produce statistical estimates while preserving the certification properties of the optimizationbased method. We consider different statistical inference problems and applications that can be described with the additive noise model 𝑦 = 𝐹 (𝑥 ∗ ) + 𝜀, in which 𝑦 is the observed experimental data, 𝐹 (the forward model) is a physical process, dependent on a state of the world or parameter 𝑥 ∗ , and 𝜀 ∼ 𝜇 is some experimental or measurement noise. The scientist’s knowledge about the different elements of the problem is applicationdependent. Statistical techniques vary depending on which elements we aim to infer from which others, and under which assumptions/knowledge of 𝐹, 𝑥 ∗ and 𝜇, the distribution of 𝜀, we aim to perform such inference. For example, distribution-free methods make no assumptions about the distribution of 𝜀, frequentist methods make distribution assumptions on 𝜀 but not on 𝑥 ∗ , constrained inference methods further assume 𝑥 ∗ ∈ X, and Bayesian methods assume 𝑥 ∗ ∼ 𝜈 for a given, known 𝜈. For all the elements present (including the prior distribution 𝜈, whenever included as part of the model), a range of assumptions is possible, including fully unknown, knowledge of a constraint set, and fully known. There is an inherent tradeoff between accuracy and robustness in Uncertainty Quantification methods, illustrated in this context by the strength of the assumptions any particular method uses: methods with few assumptions are robust, but may overcover 2 if they are unable to capture all that is known from a problem, while methods with many assumptions can be much more accurate, but might suffer from brittleness and undercover if any of the assumed knowledge is even slightly wrong (H. Owhadi, C. Scovel, and T. Sullivan, 2015b). In real applications, overcovering leads to the extra economic cost of preparing against scenarios that were not needed, and undercovering leads to taking unnecessary risks. Given these potential consequences of over-assuming and under-assuming in inference problems, and given that many real problems have partial knowledge about the unknowns, this thesis aims to build optimization-based methods that are able to include such partial knowledge, without relying on extra information that is not available to the decision problem. An example inference problem that will help illustrate the importance of assumptions is the particle unfolding problem (Spano, 2014), illustrated in Figure 1.1. The distribution of particle counts with respect to a quantity of interest (usually, energy) in certain particle physics cannot be observed directly, but is observed through a noisy measurement of a known “blurring” process 𝐹 that destroys some part of the signal. We refer to this as an ill-posed inverse problem because, even with a noiseless measurement, we would not be able to recover the totality of 𝑥 ∗ . It is nevertheless of scientific interest to estimate a function of 𝑥 ∗ , referred to as 𝜑(𝑥 ∗ ) (for example, the number of particles over a given energy threshold), given the data 𝑦. In this problem, we can model 𝐹 and the distribution of 𝜀 as known (we assume that the measurement device can be calibrated in-lab before the experiment), but a prior distribution on 𝑥 ∗ , which is a common regularization technique for ill-posedness in inverse problems, might not necessarily be available. Early works dating back to the 1960s proposed using the known constraint 𝑥 ∗ ≥ 0 to obtain meaningful inference without prior regularization (W. R. Burrus, 1965), a problem which we consider in more generality in Chapters 2 and 3. In this setup, not being able to include the constraint 𝑥 ∗ ≥ 0 (under-assuming) can make the given function of interest 𝜑(𝑥 ∗ ) hard or even impossible to infer, and adding a prior distribution (over-assuming) might lead to intervals without the proper coverage if such a prior is not correct, as observed in a related application (Patil, Kuusela, and Hobbs, 2022). It is hence crucial to develop algorithms that can make use of “what we know, and only what we know”. More precisely, in Chapters 2 and 3, we consider building frequentist confidence intervals for functionals of interest in constrained inverse problems, which corre- 3 Figure 1.1: Setup of the particle unfolding inference problem. In it, we are aiming to recover a function 𝜑 of a discrete distribution of particle counts 𝑥 ∗ , given a distribution 𝑦 of blurred particle counts obtained by a noisy measurement 𝑦 = 𝐹 (𝑥 ∗ ) + 𝜀. sponds to using the data y and knowledge of 𝐹, the distribution of 𝜀 and some constraints 𝑥 ∗ ∈ X to calibrate in a frequentist sense a confidence interval for a quantity of interest 𝜑(𝑥 ∗ ). In this problem, standard worst-case analysis shows that before observing any data, it holds that  ∗  𝜑(𝑥 ) ∈ inf 𝜑(𝑥), sup 𝜑(𝑥) 𝑥∈X ∀𝑥 ∗ ∈ X. (1.1) 𝑥∈X If we were to observe the data 𝑦 = 𝐹 (𝑥) without noise, defining C(𝑦) := {𝑥 : 𝐹 (𝑥) = 𝑦} we would be able to certify that ∗   𝜑(𝑥 ) ∈ inf 𝜑(𝑥), 𝑥∈X∩C(𝑦) sup 𝜑(𝑥) ∀𝑥 ∗ ∈ X. (1.2) 𝑥∈X∩C(𝑦) In the noisy case, in which we observe 𝑦 = 𝐹 (𝑥) + 𝜀, with 𝜀 sampled from a noise distribution, the main problem addressed in Chapters 2 and 3 is how to build an equivalent set C(𝑦; 𝛼) such that the property (1.2) holds with probability 1 − 𝛼 i.e. P𝑦∼𝑃 𝑥∗ 𝜑(𝑥 ∗ ) ∈ !  inf 𝑥∈X∩C(𝑦;𝛼) 𝜑(𝑥), sup 𝑥∈X∩C(𝑦;𝛼) 𝜑(𝑥) ≥ 1−𝛼 ∀𝑥 ∗ ∈ X. (1.3) 4 Figure 1.2: In optimization-based methods, the observation of the data can be used to shrink the constraint set of the optimization programs considered while having the resulting interval be as small as possible. We note that we treat 𝑥 ∗ as a fixed unknown and the probability is only taken with respect to the randomness induced by 𝜀. Figure 1.2 illustrates how the process of data observation leads to a shrinkage of the set we are optimizing over, from X to X ∩ C(𝑦; 𝛼), leading to potentially tighter Uncertainty Quantification. Chapter 2, published as (Batlle, Stanley, et al., 2023), focuses on the theoretical background of this problem, illustrating the connections with hypothesis test inversion of many different previously used techniques (W. R. Burrus, 1965; Philip B. Stark, 1992a; Philip B. Stark, 1994) and disproving a conjecture originally stated in (W. R. Burrus, 1965) claiming that a particular construction of C(𝑦; 𝛼) has correct 1 − 𝛼 frequentist coverage. Chapter 3, published as (Stanley, Batlle, et al., 2025), focuses on building a sampling algorithm that scales to higher-dimensional problems, by leveraging quantile regression techniques and maximization of p-values over confidence sets, a theoretical technique pioneered originally by (Roger L. Berger and Boos, 1994). In Chapter 4, published as (Bajgiran et al., 2022a), inverse problems are considered from a Bayesian and decision-theoretical perspective, with optimization and certification being performed with respect to an unknown prior distribution. This is, we consider that, before sampling our data, 𝑥 ∗ ∼ 𝜇 for some unknown distribution 𝜇, and we aim to find certificates similar to (1.3). As opposed to the worst-case approach with respect to measures developed in (H. Owhadi, C. Scovel, T. J. Sullivan, et al., 2013b), we aim to find minimax optimal certificates in the setting of the classical decision-theoretical framework of (A. Wald, 1945), in which a prior distribution is chosen by nature adversarially to the decision-maker as a randomized strategy for a zero-sum game. In such a game, we introduce likelihood constraints after seeing the data. This corresponds to a novel perspective in Uncertainty Quantification, which we name of the fourth kind (in 5 Figure 1.3: In Optimization-based methods, the observation of the data can be used to shrink the constraint set of the optimization programs considered reference to the three previous families: worst-case, Bayesian, and the aforementioned Wald’s decision-theoretical), which consists of formulating and solving a two-player zero-sum data-dependent game subject to likelihood constraints. Figure 1.3 illustrates the main differences between the classical decision-theoretical UQ and our new approach. This algorithm has been later used in (Kaveh et al., 2024) for the quantification of uncertainty in the number of earthquakes induced by reservoir operations for gas extraction in the Groningen region in the Netherlands. In Chapters 5, 6, and 7, published as (Batlle, Darcy, et al., 2023; Batlle, Yifan Chen, et al., 2023; Bourdais et al., 2023) respectively, three different functional regression settings are considered, in which we aim to recover the forward model 𝐹 based on pairs of {𝑥𝑖 , 𝑦𝑖 = 𝐹 (𝑥𝑖 ) + 𝜀𝑖 }𝑖∈I data. In all the situations, we consider kernel methods, which emerge in the context of optimization-based statistical inference as solutions of an optimal recovery game in a Banach space B. In particular, letting an unknown 𝑢 ∈ B we consider the problem of recovering it from linear measurements Φ(𝑢) := ([𝜙1 , 𝑢], . . . , [𝜙𝑚 , 𝑢]) ∈ R𝑚 , with an estimator Ψ(Φ(𝑢)) ∈ B. If we pose ∥𝑢 − Ψ(Φ(𝑢))∥ B (which the this as a zero-sum game with loss function E (𝑢, Ψ) = ∥𝑢∥ B decision maker aims to minimize by choosing Ψ, and nature aims to minimize by adversarially choosing 𝑢), Gaussian processes conditioned on the data, equivalently thought of as kernel methods by seeing kernels as Gaussian process covariance 6 functions, arise as minimax optimal solutions of the randomized version of the optimal recovery game (Houman Owhadi and Clint Scovel, 2019a). Chapter 5 considers using kernel methods for operator learning, the task of learning an operator between Banach spaces of functions. In Chapter 6, error estimates for the use of kernel methods to solve partial differential equations are developed, and in Chapter 7, we use Gaussian processes as a sensing tool to discover functional relationships between data. This task can be thought of as recovering particular properties about the unknown 𝑓 , in cases in which recovering the full 𝑓 is unfeasible due to the quantity or quality of the available data. In the sequel, each chapter defines distinct notation within its content. 7 Chapter 2 OPTIMIZATION-BASED FREQUENTIST CONFIDENCE INTERVALS FOR FUNCTIONALS IN CONSTRAINED INVERSE PROBLEMS: RESOLVING THE BURRUS CONJECTURE We present an optimization-based framework to construct confidence intervals for functionals in constrained inverse problems, ensuring valid one-at-a-time frequentist coverage guarantees. Our approach builds upon the now-called strict bounds intervals, originally pioneered by (W. R. Burrus, 1965; Burt W. Rust and Walter R. Burrus, 1972), which offer ways to directly incorporate any side information about the parameters during inference without introducing external biases. This family of methods allows for uncertainty quantification in ill-posed inverse problems without needing to select a regularizing prior. By tying optimization-based intervals to an inversion of a constrained likelihood ratio test, we translate interval coverage guarantees into type I error control and characterize the resulting interval via solutions to optimization problems. Along the way, we refute the Burrus conjecture, which posited that, for possibly rank-deficient linear Gaussian models with positivity constraints, a correction based on the quantile of the chi-squared distribution with one degree of freedom suffices to shorten intervals while maintaining frequentist coverage guarantees. Our framework provides a novel approach to analyzing the conjecture, and we construct a counterexample employing a stochastic dominance argument, which we also use to disprove a general form of the conjecture. We illustrate our framework with several numerical examples and provide directions for extensions beyond the Rust–Burrus method for nonlinear, non-Gaussian settings with general constraints. 2.1 Introduction Advances in data collection and computational power in recent years have led to an increase in the prevalence of high-dimensional, ill-posed inverse problems, especially within the physical sciences. These challenges are particularly evident in domains such as remote sensing and data assimilation, where uncertainty quantification (UQ) in inverse problems is of paramount importance. Many of these inverse problems also come with inherent physical constraints on their parameters. This paper focuses on constrained inverse problems for which the noise model is 8 known, and the forward model, defined on a finite-dimensional parameter space, can be computationally evaluated. Our primary objective is to construct a confidence interval for a functional of the forward model parameters. Formally, we consider statistical models of the form 𝒚 ∼ 𝑃𝒙 ∗ , where 𝒚 ∈ R𝑚 is sampled according to a parametric probability distribution. Here 𝒙 ∗ ∈ R 𝑝 is a fixed unknown parameter, which we know a priori lies within the set X; see Figure 2.1 for an illustration. Our goal is to construct confidence intervals for a known onedimensional functional 𝜑(𝒙 ∗ ) ∈ R. Ideally, we want the length of these intervals to be as small as possible, while still maintaining a nonasymptotic frequentist coverage guarantee. In other words, given a prescribed coverage level 1−𝛼 for some 𝛼 ∈ (0, 1), we want to construct functions of the data 𝐼 − ( 𝒚) and 𝐼 + ( 𝒚) such that the following coverage guarantee holds in finite sample1 :
inf P 𝒚∼𝑃 𝒙 𝜑(𝒙) ∈ [𝐼 − ( 𝒚), 𝐼 + ( 𝒚)] ≥ 1 − 𝛼. 𝒙∈X (2.1) While the requirement (2.1) requires that we maintain at least 1−𝛼 coverage, we also want it to be approximately accurate by minimizing the slack in the inequality. Ensuring such proper calibration, namely, confidence intervals that do not undercover (fail to meet the 1 − 𝛼 guarantee for some 𝒙 ∈ X) or overcover (are too large and therefore exceed the required coverage) is paramount in practical applications. This is especially true in contexts that require stringent safety and certification standards. Intervals that undercover yield unreliable inferences that may expose the system to unforeseen risks. Conversely, intervals that overcover might lead to excessive economic costs by needing to guard against scenarios that are unlikely to occur. In many applied contexts, Bayesian methods constitute a primary set of techniques for uncertainty quantification. These methods leverage a prior for regularization, derived either from the intrinsic details of the problem or introduced externally. A key advantage of this regularization approach is the natural UQ that emerges from the Bayesian statistical framework. Specifically, the combination of a predefined prior and data likelihood results in a posterior distribution via Bayes’ theorem. This distribution can subsequently be used to derive the intended posterior UQ. However, there is a caveat: Bayesian methods can offer marginal coverage (probability over 𝒙 and 𝒚) if the prior is correctly specified. They do not necessarily provide conditional coverage (probability over 𝒚 given 𝒙). The former notion of coverage is weaker (and, 1 This form of “simple” interval is only for expositional simplicity. One can consider more general forms of confidence sets I ( 𝒚) beyond simple intervals, which we will do when describing the general framework in Section 2.3. 9 Figure 2.1: Illustration of the problem setup. We seek to construct confidence intervals [𝐼 − (𝒚), 𝐼 + (𝒚)] ⊆ R for 𝜑(𝒙 ∗ ) ∈ R from an observation 𝒚 ∈ R𝑚 sampled from 𝑃𝒙 ∗ that satisfies a frequentist coverage guarantee in finite sample while being as small (in length) as possible. in particular, the latter implies the former, but the converse may not be true), as it replaces the infimum in the coverage requirement (2.1) with a probability distribution over 𝒙. Generally, Bayesian methods may not align with the analyst’s expectations due to inherent bias (Kuusela, 2016; Patil, Kuusela, and Hobbs, 2022). While, in theory, priors present an effective mechanism to incorporate scientific knowledge into UQ, they can inadvertently introduce extraneous information (Philip B. Stark, 2015) and a lack of robustness in the resulting estimates (Houman Owhadi, Clint Scovel, and Tim Sullivan, 2015a; Houman Owhadi, Clint Scovel, and Tim Sullivan, 2015b; H. Owhadi and C. Scovel, 2017b). On the other hand, we could consider a basic worst-case approach that is rooted in the simple observation that   ∗ 𝜑(𝒙 ) ∈ inf 𝜑(𝒙), sup 𝜑(𝒙) . (2.2) 𝒙∈X 𝒙∈X Of course, this method is inherently conservative given the absence of assumptions and any specific knowledge regarding data generation. More importantly, the method does not use observations 𝑦 in any way to calibrate the confidence set. This means that the sets cannot be fine-tuned to approximately achieve the desired 1 − 𝛼 coverage level. Nevertheless, they illustrate the essential idea of constructing a confidence interval based on the outcomes of two boundary optimization problems, an approach that the more sophisticated methods that we will study in this paper build on. We 10 shall henceforth refer to such intervals with the notation: inf /sup 𝒙 𝜑(𝒙) 𝒙 st  := 𝒙∈X  inf 𝜑(𝒙), sup 𝜑(𝒙) . 𝒙∈X 𝒙∈X An example of such a more sophisticated method is the so-called “simultaneous” approach (Philip B. Stark, 1992a; Philip B. Stark, 1994), which provides intervals with at least 1 − 𝛼 frequentist coverage for the functional of interest 𝜑(𝒙 ∗ ) from confidence sets for the parameter 𝒙 ∗ . The approach can be summarized in three steps (see Figure 2.2 for an illustration): Step 1. Construct a set C(𝒚) ∈ R 𝑝 that is a 1 − 𝛼 confidence set for 𝒙 ∗ . Step 2. Intersect this set C(𝒚) with the constraint set X. Step 3. Project this intersection through the functional of interest 𝜑. The term “simultaneous” refers to Steps 1 and 2 being independent of the quantity of interest 𝜑, so the resulting set from Step 2 can be simultaneously projected to different quantities of interest. Under mild assumptions, the resulting intervals can be equivalently written as  ISSB ( 𝒚) :=  inf 𝒙∈X∩C(𝒚) 𝜑(𝒙), sup 𝒙∈X∩C(𝒚) 𝜑(𝒙) = inf /sup 𝒙 𝜑(𝒙) 𝒙 st (2.3) 𝒙 ∈ X ∩ C( 𝒚). This illustrates how the simultaneous approach is a refinement of the basic worstcase method (2.2): the observation of the data 𝒚 shrinks the “pre-data set” X into a smaller “post-data set” X ∩ C( 𝒚), which is then projected through 𝜑 in a worst-case manner. Given that this simultaneous framework is broadly encapsulated in (Philip B. Stark, 1992a) as “strict bounds,” we label these intervals as “simultaneous strict bounds” or SSB intervals, for short. Unlike methods that rely on explicit regularization through a prior, the techniques outlined above leverage only the physical constraints and the functional of interest to address the underlying ill-posedness of the inverse problem. This approach allows for uncertainty quantification without the need to assume a prior distribution, circumventing potential biases and miscalibrated coverage issues previously mentioned. Although the interval (2.3) has guaranteed coverage for 𝜑(𝒙 ∗ ) inherited from the coverage of C(𝒚), this method generally suffers from overcoverage, especially when 11 Figure 2.2: Illustration of the simultaneous approach for confidence interval building, which works generically for any X, 𝜑, and 𝑃. The intersection of X and C(𝒚) occurs in the original parameter space R 𝑝 , and is then projected via the functional of interest function into the real line. The confidence interval is then constructed using the minimum and maximum of the quantity of interest 𝜑 over the intersection X ∩ C( 𝒚). the dimension of X is large (Patil, Kuusela, and Hobbs, 2022; Stanley, Patil, and Kuusela, 2022; Kuusela and Philip B. Stark, 2017a). This happens due to two main factors: (i) its generality cannot account for the specific structure of 𝑃, 𝜑, and X; and (ii) while the set C(𝒚) being a 1 − 𝛼 confidence set is a sufficient condition, it is not necessary for (2.3) to ensure accurate coverage, which implies that smaller sets might also produce valid confidence intervals. Consequently, an important research direction has been constructing confidence intervals that are shorter than the simultaneous approach, but still maintain nominal coverage for a given 𝜑. Sometimes, this is achieved by assuming that 𝑃, 𝜑, and X come from a particular class (Philip B. Stark, 1994; Burt W. Rust and Walter R. Burrus, 1972; Tenorio, Fleck, and Moses, 2007; Patil, Kuusela, and Hobbs, 2022; Stanley, Patil, and Kuusela, 2022). In the sequel, we discuss one such special class. The Burrus conjecture The Gaussian linear forward model with nonnegativity constraints and a linear functional of interest is a setting that has attracted significant attention, going back to the works of (W. R. Burrus, 1965; Burt W. Rust and Walter R. Burrus, 1972). These foundational studies consider the applied problem of unfolding gamma-ray and neutron spectra from pulse-height distributions under rank-deficient linear systems. They demonstrated that incorporating the nonnegativity physical constraint allowed for the computation of nontrivial (i.e., finite length) intervals for linear functionals of the parameters. In order to describe the construction of these intervals, consider the canonical form of the Gaussian linear model with nonnegativity constraints, along 12 with a linear functional of interest: 𝒚 = 𝑲𝒙 ∗ + 𝜺, 𝜺 ∼ N (0, 𝑰𝑚 ) , | {z } with 𝒙∗ ≥ 0 | {z } and constraints model 𝜑(𝒙 ∗ ) = 𝒉⊤ 𝒙 ∗ . (2.4) | {z } functional Here, 𝑲 ∈ R𝑚×𝑝 is the forward operator2 , 𝒙 ∗ ∈ R 𝑝 is the true parameter vector, and 𝒉 ∈ R 𝑝 contains weights for the functional of interest. In this setting, (W. R. Burrus, 1965; Burt W. Rust and Walter R. Burrus, 1972) posed that the following interval construction yields valid 1 − 𝛼 confidence intervals, a result now known as the Burrus conjecture (Bert W. Rust and O’Leary, 1994): min/max 𝒙 𝒙 IOSB ( 𝒚) := st 𝒉⊤ 𝒙 ∥ 𝒚 − 𝑲𝒙∥ 22 ≤ 𝜓𝛼2 (𝒚), (2.5) 𝒙 ≥ 0, where 𝜓𝛼2 = 𝑧2𝛼/2 + 𝑠2 (𝒚). Here 𝑧 𝛼 is the upper quantile of standard normal such that P(𝑍 > 𝑧 𝛼 ) = 𝛼 for 𝑍 ∼ N (0, 1), and 𝑠2 ( 𝒚) is defined through an optimization problem as follows: ( min ∥ 𝒚 − 𝑲𝒙∥ 22 2 𝒙 𝑠 ( 𝒚) := (2.6) st 𝒙 ≥ 0. Comparison of (2.5) with (2.3) shows that Rust and Burrus proposed a “simultaneouslike” construction. In this construction, the set {𝒙 : ∥ 𝒚 − 𝑲𝒙∥ 22 ≤ 𝜓𝛼2 (𝒚)} plays the role of C( 𝒚). It typically does not represent a 1−𝛼 confidence set for 𝒙 ∗ , thus relaxing the stringent assumption of the SSB interval construction. Furthermore, a possible simultaneous interval for this setting can be built by observing that ∥ 𝒚−𝑲𝒙 ∗ ∥ 22 ∼ 𝜒𝑚2 . This yields the following valid 1 − 𝛼 interval: min/max 𝒙 𝒙 st 𝒉⊤ 𝒙 ∥ 𝒚 − 𝑲𝒙∥ 22 ≤ 𝑄 𝜒𝑚2 (1 − 𝛼) (2.7) 𝒙 ≥ 0. Here, 𝑄 𝜒𝑚2 is the quantile function of a 𝜒𝑚2 distribution. It should be noted that the data-dependent term 𝜓𝛼2 ( 𝒚) in (2.5) could be considerably smaller than 𝑄 𝜒𝑚2 (1 − 𝛼), especially when 𝑚 is large and 𝛼 is small. So if the Burrus conjecture were true, 2 Note that the forward operator 𝑲 is allowed to be column rank deficient and the overparameter- ized setting when 𝑝 > 𝑚 is allowed. 13 it would provide a significant reduction in the length of the interval for problems in the class (2.4). For instance, assuming 𝛼 = 0.05 (so that we are after a 95% coverage level), (Stanley, Patil, and Kuusela, 2022) observe an expected length reduction of about a factor of two across a variety of functionals in a particle unfolding application. The gain in the interval length originates from the fact that these intervals take into account that we are only required to guarantee coverage for one specific functional. Given that intervals of the form (2.5) are designed to provide coverage for one functional at a time, following the nomenclature of (Stanley, Patil, and Kuusela, 2022), we refer to these intervals as “one-at-a-time strict bounds” or OSB intervals, for short3 . (Burt W. Rust and Walter R. Burrus, 1972) and subsequently (Bert W. Rust and O’Leary, 1994) investigated the conjecture posed in (W. R. Burrus, 1965), with (Bert W. Rust and O’Leary, 1994) purpoting to have found definitive proof for the conjecture’s validity. However, this claim was later refuted by (Tenorio, Fleck, and Moses, 2007) through a two-dimensional counterexample. In this work, we demonstrate that, in fact, this two-dimensional counterexample proposed in (Tenorio, Fleck, and Moses, 2007) is not a valid counterexample. However, we present and prove another counterexample that refutes the conjecture and we propose ways to fix the previous faulty results by reinterpreting the conjecture. We achieve this through a novel hypothesis test-based framework that not only revisits but also broadens the scope beyond the linear Gaussian setting paired with positivity constraints in which the conjecture was originally proposed. Summary and outline In this paper, we frame the problem of confidence interval construction for functionals in constrained, ill-posed problems through the inversion of a particular likelihood ratio test. This perspective allows us to reinterpret the interval coverage guarantee in terms of type-I error control associated with the test and, subsequently, the distribution of the log-likelihood ratio under the null hypothesis. We also establish connections between different fields of hypothesis testing with likelihood ratio tests, optimization-based confidence intervals, and chance-constrained optimization. A detailed summary of contributions in this paper along with an outline for the paper 3 (Patil, Kuusela, and Hobbs, 2022; Stanley, Patil, and Kuusela, 2022) also extend the setting and the conjecture to encompass linear constraints of the form 𝑨𝒙 ≤ 𝒃. Such constraints are of interest in practical applications such as XCO2 retrieval and particle unfolding. For simplicity, we present only the positivity constraint case here. However, our counterexample based on positivity constraints in Section 2.4 will also be sufficient to disprove the conjecture in this general case. 14 is given below. (1) Strict bounds intervals from test inversion. In Section 2.2, we present a general framework to construct strict bounds intervals through test inversion, resulting in two optimization problems for the interval endpoints. This approach generalizes the Rust–Burrus-type interval technique to potentially nonlinear and non-Gaussian settings. Our main result in Theorem 2.2.4 proves coverage of the test inversion construction and Proposition 2.2.5 provides sufficient conditions under which the coverage is tight. Examples in Section 2.2 provide straightforward but concrete analytical illustrations of our framework. (2) General interval construction methodology. In Section 2.3, we present a general methodology for computing confidence intervals that builds on the framework in Section 2.2. We outline the methodology in Algorithm 1 and discuss two key components: the chance-constrained optimization problem and the stochastic dominance argument in Section 2.3 and Section 2.3, respectively. The chance-constrained optimization problem allows us to obtain optimal decision values for the proposed framework, while the stochastic dominance argument provides a theoretical tool to find provable upper bounds. (3) Refuting the Burrus conjecture. In Section 2.4, we demonstrate that our method successfully recovers previously proposed OSB intervals for the linear Gaussian setting. In Theorem 2.4.1, we leverage this novel interpretation to disprove the Burrus conjecture (Burt W. Rust and Walter R. Burrus, 1972; Bert W. Rust and O’Leary, 1994) in the general case, by refuting a previously proposed counterexample and providing a new, provably correct counterexample in Lemma 2.4.5. Furthermore, we provide a negative result that disproves a natural generalization of the original conjecture in Proposition 2.4.6. Our proof technique provides a method to detect when the Rust–Burrus approach is effective and when it falls short and introduces a means to rectify the earlier erroneous examples. (4) Illustrative numerical examples. In Section 2.5, we elucidate our findings through a suite of numerical illustrations. These span various scenarios, including the counterexample to the Burrus conjecture. We show that test inversionbased intervals, which have provable guarantees, achieve better coverage calibration than previous approaches. 15 Other related work Given the effectiveness of the strict bounds methodology in high-dimensional illposed inverse problems, this paper seeks to deepen our understanding of these intervals and provide related perspectives by connecting them with the broader statistical literature. Specifically, we relate these intervals to the well-developed areas of likelihood ratio tests, test inversions, and constrained inference, enabling us to make rigorous statements about their properties and generalize the methodology beyond its earlier confines. We provide a brief overview of earlier work in this area below. Confidence intervals in penalized inverse problems Various optimization-based strategies exist for constructing confidence intervals for functionals in linear inverse problems with constraints. A first connection between optimization and inference in inverse problems is given by classical approaches seeking to optimize an objective function to balance data misfit with regularization, while adhering to prior constraints (Hansen, 1992; Hansen and O’Leary, 1993). It is then common to use the variability of the minimizer to quantify uncertainty. Another closely related strategy employs Bayesian methods to estimate the posterior distribution of model parameters given a regularizing prior and subsequently constructs credible intervals from marginal distributions (Tarantola, 2005; Andrew M. Stuart, 2010). Since these methods effectively quantify uncertainty around the expectation of the regularized estimator, their coverage is highly dependent on the precision of prior information and strength of regularization (Patil, Kuusela, and Hobbs, 2022; Kuusela, 2016; Kuusela and Victor M Panaretos, 2015a). A recent line of work starting with (Javanmard and Montanari, 2014) attempts to improve the coverage of confidence intervals derived from penalized estimators by “de-biasing” the regularized estimators; however, in practice, guarantees can only be obtained asymptotically and finite-sample performance depends on the choice of tuning parameters. An alternative line of work in optimization-based confidence intervals focuses on ensuring correct frequentist finite-sample coverage. This approach is more resistant to the aforementioned challenges associated with relying heavily on prior assumptions or de-biasing and offers a robust framework for uncertainty quantification. In the following section, we will describe these optimization-based methods in more detail. 16 Optimization-based confidence intervals and the Burrus conjecture This paper is largely motivated by the literature (Burt W. Rust and Walter R. Burrus, 1972; O’Leary and Bert W. Rust, 1986; Bert W. Rust and O’Leary, 1994; Tenorio, Fleck, and Moses, 2007; Patil, Kuusela, and Hobbs, 2022; Stanley, Patil, and Kuusela, 2022) surrounding the Burrus conjecture (see Section 2.4 for further discussion), which makes a claim about how to set a calibration parameter in an optimizationbased confidence interval construction so that the resulting interval has a desired level of coverage for a single functional. For intervals with a simultaneous coverage guarantee for an arbitrary collection of functionals, (Philip B. Stark, 1992b) provides the most general optimization-based confidence interval construction. While these intervals provide the desired coverage, they are overly conservative in terms of length when compared to intervals calibrated for a specific functional (Stanley, Patil, and Kuusela, 2022). These prior works consider only the Gaussian linear inverse problem and can thus be seen as a particular instance of the more general optimization-based confidence intervals treated in this paper. Inverting likelihood ratio tests and constrained inference Traditionally, optimizationbased confidence interval constructions in inverse problems have developed somewhat independently of the broader statistical literature, often overlooking the duality between confidence intervals and hypothesis testing (George Casella and Roger L. Berger, 2002; Wasserman, 2004; Lehmann and Romano, 2008). Our work reinterprets these optimization-based confidence intervals from the inverse problem literature as inverted hypothesis tests and situates them within the realm of constrained testing and inference; see, e.g., (Gouriéroux, Holly, and Monfort, 1982; Wolak, 1987; Robertson, F. T. Wright, and Dykstra, 1988; Alexander Shapiro, 1988; Wolak, 1989; Molenberghs and Verbeke, 2007), among others. The constrained inference literature often employs the 𝜒¯ 2 distribution, a convex combination of 𝜒2 distributions with different degrees of freedom, dictated by the problem constraints. Recent work in (M. Yu, Gupta, and Kolar, 2019) has extended these constrained testing frameworks to high-dimensional settings with linear inequality constraints, examining both sparse and non-sparse scenarios. Although such tests can be more powerful than their unconstrained counterparts, their definitions typically limit the null hypothesis to linear subspaces, complicating their use in test inversion scenarios (Silvapulle and Sen, 2011). Although there have been applications of constrained test inversion (Feldman and 17 Cousins, 1998), these are limited in scope due to grid-based inversion approaches. The statistics literature contains other approaches to inverting likelihood ratio tests (LRTs), which center around sampling procedures (Cash, 1979; Venzon and Moolgavkar, 1988; Garthwaite and Buckland, 1992; Murphy, 1995; Neale and Miller, 1997; Schweiger et al., 2018). Alternatively, one can sample from the parameter space and the forward model to generate training data for a quantile regression, which can then be used to invert an LRT (Niccolò Dalmasso, Izbicki, and A. B. Lee, 2020; Masserano, Dorigo, et al., 2023; Niccolò Dalmasso, Masserano, et al., 2023; Fisher, Schweiger, and Rosset, 2020). Since these latter approaches require sampling points in the parameter space, they are practically limited to compact parameter spaces and may encounter difficulties with high-dimensional parameters. In scenarios where the data can be split, approaches such as Universal Inference (Wasserman, Ramdas, and Balakrishnan, 2020) offer a way to obtain confidence sets for irregular likelihoods with nonasymptotic coverage. Worst-case and likelihood-free methods Most of the approaches and methods referenced and described above make strong assumptions about the underlying data-generating distribution (e.g., linear forward model and Gaussian noise). To generalize these assumptions, one can either take a worst-case approach within the model class (e.g., (Donoho, 1994) which looks at worst-case confidence intervals for linear inverse problems) or remove distribution assumptions altogether. For example, Optimal Uncertainty Quantification (OUQ) (see, e.g., (H. Owhadi, C. Scovel, T. J. Sullivan, et al., 2013a)) does not assume a particular likelihood function to perform statistical inference by relying instead on worst-case bounds. If one is willing to make boundedness assumptions on the parameter space, simulation-based inference approaches such as (Gutmann and Corander, 2015; O. Thomas et al., 2022; Niccolò Dalmasso, Izbicki, and A. B. Lee, 2020; Masserano, Dorigo, et al., 2023; Niccolò Dalmasso, Masserano, et al., 2023; Cranmer, Brehmer, and Louppe, 2020; Cranmer, Pavez, and Louppe, 2016) explore the use of sampling-only access to the likelihood, typically through a simulator, which has found particular relevance in the physical sciences. While these likelihood-free methods are advantageous in contexts where the likelihood is uncertain, unknown, or accessible only through a simulator, they tend to yield conservative estimates when a well-defined likelihood is available. 18 2.2 Strict bounds intervals from test inversion Suppose that we observe data 𝒚 ∈ R𝑚 according to a data-generating process 𝒚 ∼ 𝑃𝒙 ∗ . Here, 𝑃𝒙 ∗ is a distribution that depends on a fixed but unknown parameter 𝒙 ∗ ∈ R 𝑝 . Furthermore, suppose that we have prior knowledge that this parameter 𝒙 ∗ lies in a constraint set X ⊆ R 𝑝 , namely 𝒙 ∗ ∈ X. Given a nominal coverage level 1 − 𝛼, where 𝛼 ∈ (0, 1), this paper investigates methods for constructing a 1 − 𝛼 confidence interval for 𝜑(𝒙 ∗ ), where 𝜑 : R 𝑝 → R is a known one-dimensional quantity of interest4 (we will also refer to 𝜑 as a functional of interest). More precisely, we are interested in constructing an interval I𝛼 ( 𝒚) ⊆ R for 𝜑(𝒙 ∗ ) that satisfies the following coverage requirement:
P 𝒚∼𝑃 𝒙∗ 𝜑(𝒙 ∗ ) ∈ I𝛼 (𝒚) ≥ 1 − 𝛼, for all 𝒙 ∗ ∈ X. (2.8) Our primary focus lies in intervals that (i) effectively utilize the information that 𝒙 ∗ ∈ X, (ii) are valid (i.e., satisfying the coverage requirement in (2.8)) in the finite data and noisy regimes (rather than, e.g., in the large system or noiseless limits), (iii) do not make overly restrictive assumptions (e.g., identifiability) about the structure of the parametric model 𝑃𝒙 ∗ , and (iv) are short in length5 . We view the observation vector 𝒚 as a single observation in R𝑚 drawn from a multivariate distribution 𝑃𝒙 ∗ . This may include the case of repeated sampling (i.i.d. or not) from an experiment and aggregating the samples in a vector. In this case, 𝑃𝒙 ∗ is then defined as the measure that accounts for all the observations.6 Review: classical test inversion for simple null hypotheses We briefly review the concept of test inversion and the duality between hypothesis testing and confidence sets upon which the subsequent subsections will be built. After observing 𝒚 ∼ 𝑃𝒙 ∗ , two classical statistical tasks emerge: (i) determining whether 𝒙 ∗ = 𝒙 for a particular 𝒙 ∈ X at a significance level 𝛼 (hypothesis testing), and (ii) constructing a subset of X that contains 𝒙 ∗ with a coverage level 1 − 𝛼 (confidence set building). In hypothesis testing, for a given parameter 𝒙, one can 4 Confidence sets of several functionals of interest with guarantees can be constructed by using the proposed method with, e.g., Bonferroni correction, but studying the performance of that approach is beyond the scope of this work 5 Note that length will, in general, depend on the unknown parameter 𝒙 ∗ . There are several notions for the “optimality” of the method with respect to length, such as minimax length (Donoho, 1994; Schafer and Philip B. Stark, 2009) or expected length (Stanley, Patil, and Kuusela, 2022), among others. 6 For example, in the typical case where a 𝑑 dimensional vector is observed a total of 𝑛 times, we aggregate the results in an 𝑚 = 𝑛 × 𝑑 dimensional vector. Throughout, we use 𝑚 to denote the total dimensionality of the observation vector. 19 consider the hypothesis test: 𝐻0 : 𝒙 ∗ = 𝒙 versus 𝐻1 : 𝒙 ∗ ≠ 𝒙. (2.9) We then build an acceptance region 𝐴(𝒙) in the data space (the space in which the observations 𝒚 live) corresponding to the observations that would not reject 𝐻0 , with the condition that 𝐻0 is rejected with probability at most 𝛼 when it is true. In confidence set building, one builds a subset in parameter space (the space in which the parameters 𝒙 live) as a function of the data, C(𝒚), such that it contains 𝒙 ∗ with probability at least 1 − 𝛼 (over repeated samples of 𝒚 ∼ 𝑃𝒙 ∗ ). Lifting to the product space of the data and parameter spaces (see Figure 2.3 for an illustration), both tasks amount to the construction of a compatibility region S. For a fixed observation 𝒚, a confidence set is given by C( 𝒚) = {𝒙 : ( 𝒚, 𝒙) ∈ S}, and for a fixed parameter 𝒙, the acceptance region is given by A (𝒙) = {𝒚 : ( 𝒚, 𝒙) ∈ S}. Observe that P( 𝒚 ∈ A (𝒙)) = P(𝒙 ∈ C( 𝒚)). Therefore, a procedure that forms confidence sets with coverage 1 − 𝛼 for any possible data 𝒚 also creates a procedure that yields valid hypothesis tests at the level 𝛼 for any possible parameter value 𝒙, and vice versa. This observation can be used to create confidence sets as the set of parameter values that would not be rejected by a hypothesis test, a construction known as test inversion (see, e.g., Chapter 7 of (George Casella and Roger L. Berger, 2002) or Chapter 5 of (Victor M. Panaretos, 2016)). Figure 2.3: Illustration of the classical duality between hypothesis testing and confidence set building as seen in the product space of the data and parameter spaces. Pairs of the dual hypothesis test and the confidence set can be viewed as a set S in the product space (the compatibility region). For fixed data 𝒚, a confidence set is given by C(𝒚) = {𝒙 : (𝒚, 𝒙) ∈ S}, and for a fixed parameter 𝒙, the acceptance region is given by A (𝒙) = {𝒚 : ( 𝒚, 𝒙) ∈ S}. 20 Formulation and inversion of constrained likelihood ratio tests The starting point of this work is the inversion of specific hypothesis tests that can incorporate the constraint information X and the functional of interest 𝜑. We will establish that the test inversion can be achieved by solving two endpoint optimization problems. We note that unlike the simple null versus composite alternative tests (2.9) described in Section 2.2, the tests we will consider have composite nulls. We focus on the continuous case and assume that the Lebesgue measure dominates the set of distributions P := {𝑃𝒙 | 𝒙 ∈ X}. However, a discrete analog can also be constructed using a similar approach as in (Feldman and Cousins, 1998). Let 𝐿 𝒙 be the density of 𝑃𝒙 , and let ℓ𝒙 := log 𝐿 𝒙 . For any 𝜇 ∈ R, denote the level sets of the quantity of interest 𝜑 by Φ 𝜇 . These are defined as follows: Φ 𝜇 := {𝒙 : 𝜑(𝒙) = 𝜇} ⊆ R 𝑝 . (2.10) Subsequently, define a hypothesis test 𝑇𝜇 as follows: 𝐻0 : 𝒙 ∗ ∈ Φ 𝜇 ∩ X 𝐻1 : 𝒙 ∗ ∈ X \ Φ 𝜇 . versus (2.11) We can test hypothesis (2.11) (for a fixed 𝜇) with a Likelihood Ratio (LR) test statistic defined as the following function of the observed data 𝒚: 𝐿 𝒙 ( 𝒚) sup Λ(𝜇, 𝒚) := 𝒙∈Φ 𝜇 ∩X sup 𝐿 𝒙 ( 𝒚) (2.12) . 𝒙∈X The corresponding log-likelihood ratio (LLR) statistic 𝜆(𝜇, 𝒚) is given by   𝜆(𝜇, 𝒚) = −2 log Λ(𝜇, 𝒚) = −2 sup ℓ𝒙 (𝒚) − sup ℓ𝒙 ( 𝒚) 𝒙∈Φ 𝜇 ∩X = inf 𝒙∈Φ 𝜇 ∩X (2.13) 𝒙∈X − 2ℓ𝒙 ( 𝒚) − inf − 2ℓ𝒙 ( 𝒚). 𝒙∈X As is standard (see, e.g., (George Casella and Roger L. Berger, 2002; Wasserman, 2004)), we use the supremum over all X in the denominator of (2.12), instead of over X \ Φ 𝜇 7 . The factor of −2 helps connect with the standard likelihood ratio test in the context of Wilks’ theorem and is needed, together with the optimization being over the whole space, to reinterpret the previous constrained inference intervals as coming from the inversion of this test (see Section 2.4). 7 (Schervish, 1995) provides conditions for the equality of both test statistics. 21 Motivation behind the choice of test and test statistic In addition to the reinterpretation of the previous constrained inference intervals as a result of inverting this test, there are other theoretical and practical reasons that make it a reasonable choice for this work. Theoretically, the LR emerges as the optimal test statistic (resulting in the most powerful level-𝛼 test) in the simple versus simple hypothesis testing setting via the Neyman–Pearson Lemma (George Casella and Roger L. Berger, 2002; Lehmann and Romano, 2008). Although uniformly most powerful tests do not exist in general, LR tests have been effective in several contexts. For example, (Abraham Wald, 1943) provides some optimality properties for the likelihood ratio test in terms of its asymptotic average power. Although our test of interest does not fall under the simple versus simple paradigm and we are interested in nonasymptotic properties, these two properties support the sensibility of adopting the LR-based test. Furthermore, the literature on constrained inference (Robertson, F. T. Wright, and Dykstra, 1988; Silvapulle and Sen, 2011) extensively uses the LR, deriving both asymptotic and nonasymptotic log-likelihood ratio (LLR) distributions in various scenarios, often leading to the 𝜒¯ 2 distribution. These characterizations indicate that, in very specific situations, it is possible to obtain the distribution of the test statistic under the null hypothesis, either exactly or in an asymptotic sense. Our setting extends well beyond those situations because our setup is general, and we do not make particular assumptions on the likelihood model or the constraint set. The distribution of the LLR and test inversion In hypothesis testing, we reject the null hypothesis when the values of 𝜆(𝜇, 𝒚) exceed a threshold. This indicates that there is substantial evidence against the data being generated by a distribution in the composite null defined by 𝜇. To choose a rejection region, we next study the distribution of the LLR, denoted as 𝜆(𝜇, 𝒚), in the context where 𝜇 = 𝜑(𝒙) (pertaining to the null hypothesis) and 𝒚 ∼ 𝑃𝒙 , a data sampling model, across various values of 𝒙 ∈ X. Let 𝐹𝒙 denote the distribution of 𝜆(𝜑(𝒙), 𝒚) for any 𝒙 ∈ X, where 𝒚 ∼ 𝑃𝒙 . To simplify the notation, we will write 𝜆 ∼ 𝐹𝒙 to indicate that an LLR is sampled following the procedure described above. To ensure an 𝛼-level test for test inversion, we need to control the distribution of the test statistic under the null hypothesis. Since the null is composite, the false positive rate must hold for any parameter under the null hypothesis 𝐻0 . Suppose that we are conducting a test 𝑇𝜇 to determine whether 𝜇∗ = 𝜑(𝒙 ∗ ) equals a particular 𝜇 ∈ 𝜑(X) ⊆ R, that is, 𝒙 ∗ ∈ Φ 𝜇 ∩ X. We use 𝜆 > 𝑞 𝛼 as the rejection 22 region, where 𝑞 𝛼 is a predetermined decision threshold. Under the null hypothesis, if the decision threshold satisfies sup P𝜆∼𝐹𝒙 (𝜆 > 𝑞 𝛼 ) ≤ 𝛼 (2.14) 𝒙∈Φ 𝜇 ∩X for all 𝛼 ∈ (0, 1), then we say 𝑇𝜇 is a level-𝛼 test.8 Inverting the test with respect to 𝜇 will require choosing an appropriate 𝑞 𝛼 for all 𝜇; henceforth we will denote it as 𝑞 𝛼 (𝜇). We seek to invert this test using a methodology similar to that outlined in Section 2.2, but adapted to accommodate the composite null hypothesis. The acceptance region is formally defined as: A 𝛼 (𝜇) := {𝒚 : 𝜆(𝜇, 𝒚) ≤ 𝑞 𝛼 (𝜇)} . (2.15) Subsequently, we define the proposed confidence set for 𝜇∗ = 𝜑(𝒙 ∗ ) ∈ R through test inversion as follows: C𝛼 (𝒚) := {𝜇 : 𝜆(𝜇, 𝒚) ≤ 𝑞 𝛼 (𝜇)}. (2.16) We prove in Lemma 2.2.1 that if (2.14) is satisfied for 𝜇∗ := 𝜑(𝒙 ∗ ) (that is, 𝑇𝜇∗ is a level-𝛼 test), the resulting confidence set will have the desired 1 − 𝛼 coverage, thus extending the classical test inversion framework to our specific case. Lemma 2.2.1 (Coverage of the inverted test). Let 𝛼 ∈ (0, 1). Let 𝒙 ∗ be the true parameter value and 𝜇∗ its image under 𝜑. If 𝑇𝜇∗ is a level-𝛼 test, then P 𝒚∼𝑃 𝒙∗ (𝜇∗ ∈ C𝛼 (𝒚)) ≥ 1 − 𝛼. Proof sketch. The proof is based on a straightforward test inversion argument. For a detailed proof, see Section 2.7. □ To ensure that condition (2.14) holds in practice, when 𝒙 ∗ and therefore 𝜇∗ are both unknown, we need to satisfy this condition for all possible null hypotheses. 8 Here 𝑞 𝛼 is the decision value corresponding to intervals with a coverage probability of 1-𝛼, aligning with classical textbook notation (see, e.g., (George Casella and Roger L. Berger, 2002), (Wasserman, 2004)). For any random variable 𝑍, we will denote with the subscript 𝛼 the cutoff points that satisfy P(𝑍 > 𝑧 𝛼 ) = 𝛼. 23 Specifically, we choose an appropriate 𝑞 𝛼 (𝜇) for each 𝜇 to ensure that all hypothesis tests 𝑇𝜇 are level-𝛼. Formally, this is expressed as: sup sup P𝜆∼𝐹𝒙 (𝜆 > 𝑞 𝛼 (𝜇)) ≤ 𝛼. (2.17) 𝜇∈𝜑(X) 𝒙∈Φ 𝜇 ∩X This condition is equivalent9 to sup P𝜆∼𝐹𝒙 (𝜆 > 𝑞 𝛼 (𝜑(𝒙))) ≤ 𝛼. (2.18) 𝒙∈X Although (2.18) lacks the interpretation of (2.17) of having hypothesis tests for each different 𝜇 ∈ 𝜑(X), it simplifies the calculations. We refer to a set of values 𝑞 𝛼 (𝜇) that satisfy (2.18) (or equivalently (2.17)) as valid values. Since 𝜇 in (2.17) is equal to 𝜑(𝒙) as 𝒙 ∈ Φ 𝜇 , we can use 𝑞 𝛼 (𝜇) and 𝑞 𝛼 (𝜑(𝒙)) interchangeably. From Lemma 2.2.1, we know that valid values can be used in (2.16) to construct a confidence set for 𝜇∗ with the correct 1 − 𝛼 coverage. Moreover, as argued in the proof of Lemma 2.2.1, the probability that the set (2.16) covers the unknown 𝜇∗ is given by (2.19) P 𝒚∼𝑃 𝒙∗ (𝜇∗ ∈ C𝛼 (𝒚)) = 1 − P𝜆∼𝐹𝒙∗ (𝜆 > 𝑞 𝛼 (𝜇∗ )) , which is guaranteed to be at least 1 − 𝛼 by the condition (2.17). To obtain intervals with the smallest possible size while maintaining coverage, we aim to find the optimal decision values 𝑞 𝛼 (𝜇), which are solutions to optimization problems involving the quantiles 𝑄 𝐹𝒙 : [0, 1] → R of the distributions of the family {𝐹𝒙 , 𝒙 ∈ X}. Lemma 2.2.2 (Optimal decision values). The optimal (smallest valid) value of 𝑞 𝛼 (𝜇) is given by the maximum quantile (MQ) optimization problem: 𝑄 max 𝜇,1−𝛼 := sup 𝑄 𝐹𝒙 (1 − 𝛼). (2.20) 𝒙∈Φ 𝜇 ∩X Furthermore, if one wants to choose a single 𝑞 𝛼 for all 𝜇, then the optimal value is given by max 𝑄 max (2.21) 1−𝛼 := sup 𝑄 𝜇,1−𝛼 = sup 𝑄 𝐹𝒙 (1 − 𝛼). 𝜇∈𝜑(X) 𝒙∈X Proof sketch. It can be directly checked that the proposed quantities are valid and that using any smaller decision value leads to intervals with undercoverage for at least one point. See Section 2.7 for more details. □ 9 Note that every 𝒙 ∈ X is accounted for in (2.17) since 𝜇 = 𝜑(𝒙). 24 In the event that only an upper bound on the quantities defined in Lemma 2.2.2 can be obtained, those can also be used as valid decision values, as stated precisely in the following corollary. Corollary 2.2.3. For every 𝜇 ∈ R, 𝜈(𝜇) ≥ 𝑄 max 𝜇,1−𝛼 (as defined in Lemma 2.2.2) if and only if 𝜈(𝜇) is a valid decision value for 𝑇𝜇 for a particular 𝛼. In addition, 𝜈 ≥ 𝑄 max 1−𝛼 if and only if 𝜈 is a valid decision value for 𝑇𝜇 for all 𝜇 ∈ R for a particular 𝛼. The result follows immediately from substituting the proposed 𝜈 values into the probability statements in Lemma 2.2.2. Theoretical and computational methods for obtaining valid 𝑞 𝛼 (𝜇) are discussed in Section 2.3, and we investigate the computation of C𝛼 (𝒚) via optimization techniques in the next subsection, assuming valid 𝑞 𝛼 (𝜇) are known. Characterizing the inverted confidence set via optimization problems The set defined in (2.16) produces a random collection of real numbers that contains the true functional value with a probability of at least 1 − 𝛼. Although this set is not necessarily an interval, it is contained within an interval whose (possibly infinite) extremes are computable through optimization techniques. Given a valid 𝑞 𝛼 (𝜇), which satisfies either (2.17) or (2.18), let us define the following sets: D (𝒚) := {𝒙 : − 2ℓ𝒙 (𝒚) ≤ 𝑞 𝛼 (𝜑(𝒙)) + inf −2ℓ𝒙 ′ (𝒚)} ⊆ R 𝑝 , ′ (2.22) X̄𝛼 (𝒚) := X ∩ D ( 𝒚). (2.23) 𝒙 ∈X If X̄𝛼 ( 𝒚) ≠ ∅, we further define:  I𝛼 (𝒚) := inf  𝜑(𝒙), 𝒙∈X̄𝛼 (𝒚) sup 𝜑(𝒙) . (2.24) 𝒙∈X̄𝛼 (𝒚) If X̄𝛼 ( 𝒚) = ∅, let I𝛼 (𝒚) be the empty interval. Theorem 2.2.4 (From test inversion to optimization-based intervals). For any 𝛼 ∈ (0, 1), and for any 𝒙 ∈ X, let I𝛼 ( 𝒚) be the interval constructed according to (2.24). It holds that P 𝒚∼𝑃 𝒙 (𝜑(𝒙) ∈ I𝛼 ( 𝒚)) ≥ 1 − 𝛼. In other words, I𝛼 (𝒚) is a valid 1 − 𝛼 confidence interval for 𝜑(𝒙 ∗ ). 25 Proof sketch. We prove that  inf 𝜆(𝜇,𝒚)≤𝑞 𝛼 (𝜇)  𝜇, sup 𝜇 = I𝛼 (𝒚). (2.25) 𝜆(𝜇,𝒚)≤𝑞 𝛼 (𝜇) The object on the left-hand side is defined by enclosing C𝛼 (𝒚) within the smallest possible interval that contains it, and therefore, it has guaranteed coverage. The equality arises from the equivalence between the optimization problems under consideration. For a complete proof, see Section 2.7. □ Remark 1 (Comparison with the simultaneous strict bound intervals). Observe that the construction of I𝛼 ( 𝒚) follows the form outlined in (2.3) for the simultaneous strict bound intervals. However, a key distinction lies in not requiring that D ( 𝒚) ⊆ X serves as a 1 − 𝛼 confidence set for 𝒙. This relaxation will translate into shorter intervals when 𝑞 𝛼 is chosen appropriately. Remark 2 (Handling empty constrained sets). If 𝛼 is chosen such that 1−𝛼 becomes too small, the set X̄𝛼 (𝑦) can be empty. In that case, we default to the empty interval, under the interpretation that there are no parameter values that simultaneously agree with the constraint and the observed data (at a particular level 𝛼). However, the actual interval produced under this circumstance does not compromise the 1 − 𝛼 coverage level provided by the theorem. If a point estimate inside the constraint region is desired, an option is to choose the closest point from X to D. This point specifically ensures the continuity of the interval with respect to 𝛼 in many standard scenarios. Generally, an empty set X̄𝛼 ( 𝒚) should inform one of three possibilities: either (i) an outlier event has been observed, or (ii) the initial assumption that 𝒙 ∈ X is flawed, or (iii) the forward model 𝑃𝒙 is misspecified. Here, the definition of an “outlier” is intrinsically linked to the choice of 𝛼. A larger 𝛼 will make such events more frequent, as it broadens the range of data considered as outliers. We also present a partial converse result, stating that the interval coverage implies the validity of 𝑞 𝛼 , subject to appropriate assumptions on 𝜑, 𝑃, and X. This result will be instrumental in refuting the coverage claims of the Rust–Burrus intervals, and consequently, the Burrus conjecture, as discussed in Section 2.4. Proposition 2.2.5 (Coverage implies validity of quantile levels). Assume that X forms a convex cone, ℓ𝒙 ( 𝒚) is a concave function, and 𝜑(𝒙) is linear. Define I𝛼 ( 𝒚) as in Theorem 2.2.4, for a particular choice of 𝑞 𝛼 (𝜇). If I𝛼 (𝒚) is a valid 1 − 𝛼 confidence interval for all 𝒙, then the values of 𝑞 𝛼 (𝜇) are valid. 26 Proof sketch. Generally, the values of 𝑞 𝛼 (𝜇) are valid if and only if C𝛼 ( 𝒚) constitutes a 1−𝛼 set. Since I𝛼 (𝒚) is the smallest interval that contains C𝛼 ( 𝒚), if C𝛼 ( 𝒚) is already an interval, then the result holds. The assumptions on X, ℓ𝒙 (𝒚), and 𝜑 ensure that this is the case by the convexity of the function 𝜇 ↦→ inf −2ℓ𝒙 ( 𝒚) 𝜑(𝒙)=𝜇 𝒙∈X for any 𝒚. For a detailed proof, see Section 2.7. □ Finally, we remark that the construction presented in this paper provides an approach to uncertainty quantification that does not rely on a specific point estimator, distinguishing it from many other UQ procedures. However, if one wishes to obtain a point estimator, it is worth noting that the midpoint of the interval can be justified from a decision-theoretic perspective. This idea has been discussed in previous works by (Micchelli and Rivlin, 1977; Bajgiran et al., 2022b), among others. Illustrative examples To elucidate the general methodology outlined in Theorem 2.2.4, we offer two simple illustrative examples where the LLR and its distribution are explicitly computable: a one-dimensional constrained Gaussian scenario and an unconstrained linear Gaussian case. Constrained Gaussian in one dimension As a tangible example, consider the following one-dimensional model: 𝑦 = 𝑥 ∗ + 𝜀, 𝜀 ∼ N (0, 1) | {z } with 𝑥∗ ≥ 0 | {z } and constraints model 𝜑(𝑥 ∗ ) = 𝑥 ∗ . | {z } (2.26) functional In this case, the distribution of the LLR is precisely known. Hence, a confidence interval can be constructed without resorting to the techniques introduced in Section 2.3, which are otherwise necessary when such information is not available. The form of the hypothesis test 𝑇𝜇 , as given in (2.11), is as follows: 𝐻0 : 𝑥 ∗ = 𝜇 versus 𝐻1 : 𝑥 ∗ ≠ 𝜇 and 𝑥 ∗ ≥ 0. (2.27) The LLR as defined in (2.13) for the test (2.27) is given by 𝜆(𝜇, 𝑦) = inf (𝑦 − 𝑥) 2 − inf (𝑦 − 𝑥) 2 𝑥=𝜇,𝑥≥0    (𝑦 − 𝜇) 2 ,  =   (𝑦 − 𝜇) 2 − 𝑦 2 ,  𝑥≥0 𝑦 ≥ 0, 𝑦 < 0. (2.28) 27 We can also derive its distribution under the null hypothesis (i.e., when 𝑥 ∗ = 𝜇, leading to 𝑦 = 𝜇 + 𝜀) for any 𝜇 ∈ [0, ∞), as formalized below. Example 2.2.6 (Distribution of the LLR statistic for a constrained Gaussian in one dimension). For 𝜆(𝜇, 𝑦) as defined in (2.28) with 𝜇 ≥ 0, when 𝑦 ∼ N (𝜇, 1) (null hypothesis), for all 𝑐 > 0, we have    𝜒12 (𝑐) + 12 ,  P(𝜆(𝜇, 𝑦) ≤ 𝑐) = √   𝜒12 (𝑐) · 1{𝑐 < 𝜇2 } + {Φ( 𝑐) − Φ((−𝜇2 − 𝑐)/(2𝜇))} · 1{𝑐 ≥ 𝜇2 },  𝜇=0 𝜇 > 0, where 𝜒12 and Φ are the CDFs of a 𝜒12 and a standard Gaussian, respectively. Proof. See Section 2.7. □ The expression for 𝜆(𝜇, 𝑦), with the appropriately scaled log transformation, is equivalent to Equation (4.3) in (Feldman and Cousins, 1998) where the Neyman confidence interval construction for the same problem is considered. (Feldman and Cousins, 1998) characterizes this quantity as a likelihood ordering for determining an acceptance region. By virtue of the previous result and Lemma 2.2.2, we can take 𝑄 𝜇 (1 − 𝛼), where 𝑄 𝜇 is the quantile of the distribution of 𝜆(𝜇, 𝑦) when 𝜇 is fixed, as 𝑞 𝛼 (𝜇) satisfying (2.14). A direct computation shows    𝑄 𝜒2 (1 − 𝛼),  1 𝑞 𝛼 (𝜇) = 𝑄 𝜇 (1 − 𝛼) =  𝑟 𝜇,𝛼 ,  1 − 𝛼 < 𝜒12 (𝜇2 ), 1 − 𝛼 ≥ 𝜒12 (𝜇2 ), (2.29) √ where 𝑟 𝜇,𝛼 is the unique nonnegative root of the function 𝑥 ↦→ Φ( 𝑥)−Φ((−𝜇2 − 𝑥)/(2𝜇))− (1−𝛼), which can be found using numerical methods. Therefore, D = {𝑥 : (𝑦−𝑥) 2 ≤ 𝑞 𝛼 (𝑥) + min𝑥 ′ ≥0 (𝑦 − 𝑥 ′) 2 } and the final form of the confidence interval becomes   I𝛼 (𝑦) = min 𝑥, max 𝑥 . 𝑥∈D 𝑥≥0 𝑥∈D 𝑥≥0 For a numerical comparison of this interval with alternative methods, we refer the reader to Section 2.5. Unconstrained Gaussian linear model Consider the following problem setup: 𝒚 = 𝑲𝒙 ∗ + 𝜺, 𝜺 ∼ N (0, 𝑰𝑚 ) | {z } model and 𝜑(𝒙 ∗ ) = 𝒉⊤ 𝒙 ∗ . | {z } functional (2.30) 28 Assume 𝑲 ∈ R𝑚×𝑝 has full column rank. The assumption Cov( 𝒚) = 𝑰𝑚 is without loss of generality as it is equivalent to assuming a known positive definite covariance for 𝒚 and performing a basis change with the Cholesky factor. Note that this setup is the same as (2.4) but the parameter space is not constrained, that is, X = R 𝑝 , and the forward model 𝑲 is assumed to be full rank. Using the framework established in Section 2.2, our aim is to invert the following family of hypothesis tests: 𝐻0 : 𝒉 ⊤ 𝒙 ∗ = 𝜇 versus 𝐻1 : 𝒉⊤ 𝒙 ∗ ≠ 𝜇. (2.31) The LLR as defined in (2.13) for the test (2.31) takes the form 𝜆(𝜇, 𝒚) := min ∥ 𝒚 − 𝑲𝒙∥ 22 − min ∥ 𝒚 − 𝑲𝒙∥ 22 . ⊤ 𝒙 : 𝒉 𝒙=𝜇 𝒙 (2.32) In this particular scenario, the LLR admits a closed-form expression and has a straightforward distribution, as formalized below: Example 2.2.7 (Distribution of the LLR statistic for the unconstrained Gaussian linear model). 𝜆(𝜇, 𝒚) for the unconstrained full column rank Gaussian linear model (2.32) can be expressed in closed form as 𝜆(𝜇, 𝒚) = (𝒉⊤ (𝑲 ⊤ 𝑲) −1 𝑲 ⊤ 𝒚 − 𝜇) 2 . 𝒉⊤ (𝑲 ⊤ 𝑲) −1 𝒉 (2.33) Furthermore, for any 𝒙 ∗ , whenever 𝒚 ∼ N (𝑲𝒙 ∗ , 𝑰𝑚 ), 𝜆(𝒉⊤ 𝒙 ∗ , 𝒚) is distributed as a chi-squared distribution with 1 degree of freedom. Proof. See Section 2.7. □ Using the above results, we can set 𝑞 𝛼 (𝜇) = 𝑄 𝜒2 (1 − 𝛼) for all values of 𝜇. Here, 1 𝑄 𝜒2 represents the quantile function of a chi-squared distribution with 1 degree of 1 freedom. Consequently, we can express the interval in (2.24) as:   ⊤ ⊤ I𝛼 ( 𝒚) = min 𝒉 𝒙, max 𝒉 𝒙 , (2.34) 𝒙∈D (𝒚) 𝒙∈D ( 𝒚) where we define D (𝒚) := {𝒙 : ∥ 𝒚−𝑲𝒙∥ 22 ≤ 𝑄 𝜒2 (1−𝛼)+min𝒙 ′ ∥ 𝒚−𝑲𝒙 ′ ∥ 22 }. Similarly, 1 let us define 𝑧 𝛼 = Φ−1 (1 − 𝛼), where Φ is the cumulative distribution function of the standard normal distribution. Using the equivalence 𝑧 2𝛼/2 = 𝑄 𝜒2 (1 − 𝛼), we 1 can rewrite the expression in terms of the standard normal. Moreover, as shown 29 in Appendix A of (Patil, Kuusela, and Hobbs, 2022), the endpoints of the above interval can be calculated in closed form and are given by   √︃ √︃ −1 −1 ⊤ ⊤ ⊤ ⊤ ⊤ ⊤ I𝛼 ( 𝒚) = 𝒉 b 𝒙 − 𝑧 𝛼/2 𝒉 (𝑲 𝑲) 𝒉, 𝒉 b 𝒙 + 𝑧 𝛼/2 𝒉 (𝑲 𝑲) 𝒉 , (2.35) where we define the least-squares estimator b 𝒙 = (𝑲 ⊤ 𝑲) −1 𝑲 ⊤ 𝒚. This interval is equivalent to the one derived from observing that b 𝒙 ∼ N (𝒙 ∗ , (𝑲 ⊤ 𝑲) −1 ). Therefore, 𝒙 ∼ N (𝒉⊤ 𝒙 ∗ , 𝒉⊤ (𝑲 ⊤ 𝑲) −1 𝒉). The interval in (2.35) is thus a standard we have 𝒉⊤ b construction of a Gaussian 1 − 𝛼 confidence interval. Our construction therefore coincides with the classical interval in this case where a guaranteed-coverage interval can be obtained with standard manipulations; however, our framework remains valid in constrained, rank-deficient, non-Gaussian and/or nonlinear problems where few alternative approaches are available. 2.3 General interval construction methodology In this section, we outline the core practical methodology for constructing intervals derived from Theorem 2.2.4. To summarize the preceding, Lemma 2.2.1 asserts that if we know 𝑞 𝛼 (𝜇) satisfying (2.18), we can invert the hypothesis test with a composite null hypothesis defined in (2.11) to yield a valid 1−𝛼 confidence interval. Lemma 2.2.2 poses two optimization problems that, if solved, yield valid decision values. The optimization problems in Lemma 2.2.2 give rise to two approaches of increasing complexity: (𝑖) finding a single 𝑞 𝛼 that is valid for any 𝜇 and (𝑖𝑖) finding valid 𝑞 𝛼 (𝜇) dependent on 𝜇 ∈ 𝜑(X). While approach (𝑖𝑖) can lead to tighter intervals, it is usually at the cost of more complex theoretical analysis and computations, including the computational complexity of solving the optimization problems in (2.24). In particular, when we reinterpret previously proposed optimization-based methods in Section 2.4, we will observe that these previously proposed methods are of type (𝑖), which this work expands to accommodate type (𝑖𝑖) generalizations. In this section, we briefly analyze the hardness of the optimization problems (2.20) and (2.21) by connecting them to the chance-constrained optimization literature. In case solving these problems is impractical, in Section 2.3, we describe using stochastic dominance as a theoretical tool that can be used to create and analyze provable upper bounds to the optimization problems. Stochastic dominance will also be used in Section 2.4 as the main technique to disprove coverage of the previously proposed Rust–Burrus intervals. This section is summarized in a meta-algorithm, detailed in Section 2.3. 30 Maximum quantile problems as chance-constrained optimization Lemma 2.2.2 presents optimization problems for finding the maximum quantiles, which are crucial for the proposed hypothesis test inversion procedure. We show that these problems, of the form sup𝒙∈Φ 𝜇 ∩X 𝑄 𝐹𝒙 (1 − 𝛼) and sup𝒙∈X 𝑄 𝐹𝒙 (1 − 𝛼), can be formulated as chance-constrained optimization (CCO) problems (see (Geng and Xie, 2019a) for a review of theory and applications of CCO). Lemma 2.3.1 (Chance constrained characterization of the max quantile problems). Let S ⊆ X. Then the max quantile optimization problem sup𝒙∈S 𝑄 𝐹𝒙 (1 − 𝛼) can be equivalently written as the chance constrained optimization problem: sup 𝑞 𝑞,𝒙 st 𝒙∈S (2.36) 𝑞∈R P𝑢∼U ( [0,1]) (F (𝒙, 𝑢) ≤ 𝑞) ≤ 1 − 𝛼, where F (𝒙, 𝑢) = 𝐹𝒙−1 (𝑢), with 𝐹𝒙−1 being the (possibly generalized) inverse CDF of 𝐹𝒙 Proof. By using the definition of (1 − 𝛼)-quantile of 𝑋 as the maximum 𝑞 such that 𝑑 P(𝑋 ≤ 𝑞) ≤ 1 − 𝛼, and 𝜆 ∼ 𝐹𝒙 = F (𝒙, 𝑢 ∼ U ([0, 1]) □ Note that Lemma 2.3.1 applies to both (2.20) and (2.21) by choosing appropriate S. In general, CCO problems are known to be strongly NP-hard (Geng and Xie, 2019a), even with convexity assumptions for F . Although various algorithms exist for general chance-constrained optimization, we leave the development of an algorithm specific to this problem and comparison with the aforementioned algorithms for future work. In our numerical examples (see Section 2.5), we solve these problems using gradient-free optimizers that do not exploit the chance-constrained structure but instead see the quantile function as a noisy black-box function to be optimized, with evaluations performed by estimating quantiles from large amount of samples. In higher dimensional scenarios, more advanced techniques tailored to the chance-constrained structure might be required. One can also write optimization problem (2.36) and the interval optimization problem (2.24) jointly as one chance-constrained optimization problem; see Section 2.8. 31 Analytical ways to obtain quantile levels via stochastic dominance In this subsection, we develop an analytical tool to find valid 𝑞 𝛼 that allows for a straightforward evaluation for any confidence level 1 − 𝛼 ∈ (0, 1). We first consider the case where we aim to choose a valid 𝑞 𝛼 for all 𝜇. We propose taking 𝑞 𝛼 = 𝑄 𝑋 (1 − 𝛼), where 𝑄 𝑋 is the quantile function of a random variable 𝑋 with a known, easy-to-compute distribution. We establish that for the resulting confidence interval to maintain a 1 − 𝛼 coverage guarantee for any 𝛼, 𝑋 must stochastically dominate the random variable with distribution 𝐹𝒙 ∗ , i.e., 𝜆(𝜇∗ , 𝒚) where 𝒚 is a random variable with distribution 𝑃𝒙 ∗ . This is denoted as 𝑋 ⪰ 𝜆(𝜇∗ , 𝒚) or, with slight abuse of notation, as 𝑋 ⪰ 𝐹𝒙 ∗ . Following the classical definition of stochastic dominance for real-valued random variables (see, e.g., (Shaked and Shanthikumar, 2007)), we say that 𝑋 ⪰ 𝑌 if and only if P(𝑋 ≥ 𝑧) ≥ P(𝑌 ≥ 𝑧) 10 for all 𝑧 ∈ R Lemma 2.3.2 (Valid quantile level via stochastic dominance). 𝑄 𝑋 (1 − 𝛼) serves as a valid (in the sense of (2.18)) choice for 𝑞 𝛼 for all 𝛼 if and only if 𝑋 ⪰ 𝜆(𝜇∗ , 𝒚), where 𝒚 ∼ 𝑃𝒙 ∗ . Proof. See Section 2.8. □ Remark 3 (Partial validity of quantile levels). If 𝑋 does not stochastically dominate 𝜆(𝜇∗ , 𝒚), a valid 𝑞 𝛼 can still be identified for specific 𝛼 levels, provided that certain conditions are met. Specifically, 𝑧 can serve as a valid 𝑞 𝛼 where 𝛼 = 1−𝐹𝑋 (𝑧) and 𝐹𝑋 being the cumulative distribution function of 𝑋, if and only if P(𝑋 ≤ 𝑧) ≤ P(𝑌 ≤ 𝑧) for some value of 𝑧. Remark 4 (Support restriction). Candidates for 𝑋 can be restricted to the range [0, ∞) without loss of generality, as 𝜆(𝜇∗ , 𝒚) is supported on this range by moving the mass a candidate 𝑋 might have in (−∞, 0) to 0. An economic interpretation of our result is that agents with nondecreasing utility functions would prefer a reward drawn from 𝑋 over one from 𝜆(𝜇∗ , 𝒚). In practical scenarios where the true parameter 𝒙 ∗ is unknown, it is required to establish stochastic dominance for the entire family of distributions 𝐹𝒙 , where 𝒙 ∈ X. Although all stochastically dominant distributions provide correct coverage when used to obtain 𝑞 𝛼 , a larger stochastic dominance gap provides more conservative 10 One can equivalently define stochastic dominance with strict inequalities 𝑋 > 𝑧 and 𝑌 > 𝑧; see (Shaked and Shanthikumar, 2007) 32 (a) (b) Figure 2.4: Comparison of quantile functions and CDFs for LLRs with different true parameter values. The left panel provides the true values of the quantile function as a function of 𝑥 ∗ across different confidence levels. As proven in Example 2.3.3, the quantile of 𝜒12 is greater than the true quantile for all 𝑥 ∗ and all levels. The right panel shows the CDFs for LLRs under different values of the true parameters, 𝑥 ∗ . From Example 2.3.3, as the true parameter increases, the CDF is increasingly dominated by its 𝜒12 component, so it follows that as 𝑥 ∗ increases, the CDF approaches the 𝜒12 CDF. This figure also provides a visual explanation of why using the true quantile or the true quantile function to compute the interval in (2.24) produces shorter intervals compared to those computed with the 𝜒12 quantile. bounds. Furthermore, if 𝑋1 , 𝑋2 both stochastically dominate the family 𝐹𝒙 for all 𝒙 ∈ X, we can take the pointwise minimum 𝑞 𝛼 = min{𝑄 𝑋1 (1 − 𝛼), 𝑄 𝑋2 (1 − 𝛼)} which will be no worse than using either 𝑋1 or 𝑋2 . The perspective of stochastic dominance also enables the use of coupling arguments to identify stochastically dominating distributions. For instance, one approach to find stochastically dominating distributions to a given 𝐹𝒙 is finding a function 𝑔(𝜑(𝒙), 𝒚) such that for all 𝑧, P(𝑔(𝜑(𝒙), 𝒚) ≥ 𝑧) ≥ P(𝜆(𝜑(𝒙), 𝒚) ≥ 𝑧), where the randomness is from 𝒚 ∼ 𝑃𝒙 . A particular case is that of nonrandom bounds. If 𝑔(𝜑(𝒙), 𝒚) ≥ 𝜆(𝜑(𝒙), 𝒚) almost surely in 𝒚 (as opposed to when 𝒚 ∼ 𝑃𝒙 ), then this implies a coupling of random variables once 𝒚 is sampled that implies stochastic dominance (see e.g. Theorem 4.2.3 in (Roch, 2024)). This technique can be generalized to find 𝑞 𝛼 (𝜇). Instead of finding a stochastic dominant variable 𝑋 such that 𝑋 ⪰ 𝐹𝒙 for all 𝒙 ∈ X, we aim to find a distribution 𝑋 𝜇 for each 𝜇, such that 𝑋 𝜇 ⪰ 𝐹𝒙 for all 𝒙 ∈ Φ 𝜇 ∩X, and then set 𝑞 𝛼 (𝜇) = 𝑄 𝑋 𝜇 (1−𝛼). This ensures that 𝑄 𝑋 𝜇∗ ⪰ 𝐹𝒙 ∗ , providing the desired coverage guarantees. 33 As an illustration, we revisit the one-dimensional constrained example discussed in Section 2.2. We consider the model 𝑦 = 𝑥 ∗ + 𝜀, where 𝜀 ∼ N (0, 1), 𝑥 ∗ ≥ 0, and 𝜑(𝑥) = 𝑥. We recall that we have 𝜆(𝜇, 𝑦) = (𝑦 − 𝜇) 2 − 1(𝑦 < 0)𝑦 2 and an analytical solution for the quantile of the distribution of 𝜆(𝜇, 𝑦) for every 𝜇, which we can use as valid 𝑞 𝛼 . We extend the results in Example 2.2.6 below to prove that the distribution of 𝜆(𝜇, 𝑦) is stochastically dominated by a 𝜒12 . See Figure 2.4 for an illustration. Example 2.3.3 (Stochastic dominance for LLR for constrained Gaussian in one dimension). For the LLR 𝜆(𝜇, 𝑦), when 𝑦 ∼ N (𝜇, 1) under the null hypothesis, we have that, for 𝑍 ∼ 𝜒12 , 𝑍 ⪰ 𝜆(𝜇, 𝑦) for all 𝜇 ≥ 0. Proof. See Section 2.8. □ For this example, given the stochastic dominance result, we can define 1 − 𝛼 confi2 dence intervals using 𝜒1,1−𝛼 instead of using 𝑞 𝛼 (𝜇). This produces larger intervals than using the true quantile, but the true quantile in the closed form will generally be unavailable in more complex examples, while the presented stochastic dominance tools can still be used. The intervals using the 𝜒12 quantile are min/max 𝑥 𝑥 I𝛼 (𝑦) := 𝑥 st (2.37) 𝑥≥0 2 (𝑥 − 𝑦) 2 ≤ 𝜒1,1−𝛼 + min (𝑥 ′ − 𝑦) 2 . ′ 𝑥 ≥0 General confidence interval construction In this section, we present our meta-algorithm that uses the methodologies described in the preceding sections. The goal of this meta-algorithm is to construct a 1 − 𝛼 confidence interval for a given quantity of interest 𝜑(𝒙 ∗ ). The algorithmic steps are outlined in Algorithm 1. It is worth noting that the optimization problems defined in (2.39) and (2.40) may not always be convex or straightforward to solve. However, their dual formulations can be constructed, offering provably valid confidence intervals for any feasible dual solution (Philip B. Stark, 1992b). We defer the exploration of specialized optimization techniques specifically tailored to solve (2.39) and (2.40) to future work. 34 Algorithm 1 Meta-algorithm for confidence interval construction Input: Observed data 𝒚, log-likelihood model ℓ𝒙 ( 𝒚), quantity of interest functional 𝜑, constraint set X, miscoverage level 𝛼. 1: Test statistic: Write down the LLR test statistic 𝜆(𝜇, 𝒚) = inf 𝒙∈Φ 𝜇 ∩X − 2ℓ𝒙 ( 𝒚) − inf − 2ℓ𝒙 (𝒚). 𝒙∈X (2.38) 2: Distribution control: Control 𝐹𝒙 , the distribution of 𝜆(𝜑(𝒙), 𝒚) where 𝒚 ∼ 𝑃𝒙 , for all 𝒙 ∈ X, by either: A. Explicit solution: Obtain 𝐹𝒙 explicitly, and let 𝑞 𝛼 (𝜇) := sup𝒙∈Φ 𝜇 ∩X 𝑄 𝐹𝒙 (1 − 𝛼). B. Computational way to directly find valid 𝑞 𝛼 (Section 2.3): Solve sup𝒙∈Φ 𝜇 ∩X 𝑄 𝐹𝒙 (1 − 𝛼) (to set 𝑞 𝛼 (𝜇)) or sup𝒙∈X 𝑄 𝐹𝒙 (1 − 𝛼) (to set 𝑞 𝛼 ) numerically. C. Analytical way using stochastic dominance (Section 2.3): Construct a distribution 𝑋 that stochastically dominates 𝐹𝒙 for all 𝒙 ∈ X, and let 𝑞 𝛼 := 𝑄 𝑋 (1 − 𝛼), or construct distributions 𝑋 𝜇 that stochastically dominate 𝐹𝒙 for all 𝒙 ∈ Φ 𝜇 ∩ X and let 𝑞 𝛼 (𝜇) := 𝑄 𝑋 𝜇 (1 − 𝛼). 3: Confidence interval calculation: Obtain the confidence intervals by solving the pair of optimization problems that is easier in the particular case: I. Parameter space formulation: min/max 𝒙 𝜑(𝒙) 𝒙 𝒙∈X − 2ℓ𝒙 ( 𝒚) ≤ 𝑞 𝛼 (𝜑(𝒙)) + inf −2ℓ𝒙 ′ ( 𝒚). ′ st (2.39) 𝒙 ∈X II. Functional space formulation: min/max 𝜇 𝜇 𝜇 st 𝜇 ∈ 𝜑(X) ⊆ R inf − 2ℓ𝒙 ( 𝒚) − inf − 2ℓ𝒙 ( 𝒚) ≤ 𝑞 𝛼 (𝜇). 𝒙∈Φ 𝜇 ∩X (2.40) 𝒙∈X Output: Confidence interval with coverage 1 − 𝛼. 2.4 Refuting the Burrus conjecture As discussed in Section 2.1, the family of constrained problems that has received the most attention is the positivity-constrained version of the problem as described 35 in Section 2.2. To recap, the model is defined as follows: 𝒚 ∼ N (𝑲𝒙 ∗ , 𝑰𝑚 ) with X = {𝒙 : 𝒙 ≥ 0} and 𝜑(𝒙 ∗ ) = 𝒉⊤ 𝒙 ∗ . (2.41) Here 𝑲 ∈ R𝑚×𝑝 is the forward linear operator. We again emphasize here that 𝑲 need not have full column rank, so we can for example have 𝑝 > 𝑚. It was initially conjectured in (W. R. Burrus, 1965; Burt W. Rust and Walter R. Burrus, 1972) that a valid 1 − 𝛼 confidence interval could be obtained as min/max 𝒙 𝒙 st 𝒉⊤ 𝒙 ∥ 𝒚 − 𝑲𝒙∥ 22 ≤ 𝜓𝛼2 (2.42) 𝒙 ≥ 0. Here, 𝜓𝛼2 = 𝑧2𝛼/2 + 𝑠2 (𝒚), with 𝑧 𝛼/2 being the previously defined standard Gaussian quantile, and 𝑠2 ( 𝒚) is defined as the optimal value of 𝒙 ∥ 𝒚 − 𝑲𝒙∥ 22 st 𝒙 ≥ 0. min Although initially believed to be proved in (Bert W. Rust and O’Leary, 1994), an error in the proof was later identified in (Tenorio, Fleck, and Moses, 2007), along with a counterexample. However, we demonstrate that this counterexample actually satisfies the conjecture, leaving the conjecture unresolved until now prior to our work, to the best of our knowledge. The main result of this section is the construction of a new valid counterexample using the test inversion perspective developed in Section 2.2 and the stochastic dominance approach of Section 2.3, disproving the conjecture. Theorem 2.4.1 (Refutation of the Burrus conjecture). The Burrus conjecture is false in general. The two-dimensional example previously proposed of a particular instance of (2.41) in (Tenorio, Fleck, and Moses, 2007), 𝑲 = 𝑰2 and 𝒉 = (1, −1) ⊤ with 𝒙 ∗ = (𝑎, 𝑎) ⊤ such that 𝑎 ≥ 0, does not constitute a valid counterexample to the Burrus conjecture. However, the following constitutes a valid counterexample for the Burrus conjecture: 𝑲 = 𝑰3 and 𝒉 = (1, 1, −1) ⊤ with 𝒙 ∗ = (0, 0, 1) ⊤ . 36 The main idea of the proof is to first connect the conjecture to our framework, identifying the conjectured intervals as a particular case of our construction with a particular choice of 𝑞 𝛼 . We then apply Proposition 2.2.5 to show that coverage is equivalent to a valid choice of 𝑞 𝛼 . Finally, we present a counterexample to prove that the proposed 𝑞 𝛼 is not universally valid. The proof is divided into several lemmas for clarity. Our approach is novel in that it diverges from previous geometric perspectives on the Gaussian likelihood (Burt W. Rust and Walter R. Burrus, 1972; Bert W. Rust and O’Leary, 1994; O’Leary and Bert W. Rust, 1986), instead leveraging the test inversion and stochastic dominance perspectives developed in Section 2.2 and Section 2.3. Proof outline of Theorem 2.4.1 This subsection provides a structured outline of the proof for Theorem 2.4.1, which refutes the Burrus conjecture. We break down the proof into several key lemmas. Lemma 2.4.2 (Framing the Burrus conjecture as test inversion). The construction of intervals in (2.42) for a particular instance of the problem (𝒙 ∗ , 𝑲, 𝒉) is equivalent to the general construction in Theorem 2.2.4 for the model 𝒚 ∼ N (𝑲𝒙 ∗ , 𝑰𝑚 ), with 𝒙 ∗ ≥ 0 component wise, and 𝜑(𝒙) = 𝒉⊤ 𝒙, using the threshold 𝑞 𝛼 (𝜇) = 𝑧2𝛼/2 independent of 𝜇. Therefore, it is equivalent to inverting a hypothesis test 𝐻0 : 𝒉⊤ 𝒙 = 𝜇 versus 𝐻1 : 𝒉⊤ 𝒙 ≠ 𝜇 with LLR 𝜆(𝜇, 𝒚) := min ∥ 𝒚 − 𝑲𝒙∥ 22 − min ∥ 𝒚 − 𝑲𝒙∥ 22 . ⊤ 𝒉 𝒙=𝜇 𝒙≥0 𝒙≥0 (2.43) Furthermore, the interval has correct coverage if and only if 𝑞 𝛼 = 𝑧 2𝛼/2 is valid in the sense of satisfying the false positive guarantee (2.17). Proof. See Section 2.9. □ Lemma 2.4.3 (Reducing the Burrus conjecture to stochastic dominance). The construction of intervals in (2.42) has the right coverage for any 𝛼 (and hence the conjecture holds) for a particular instance of the problem (𝒙 ∗ , 𝑲, 𝒉) if and only if the log-likelihood ratio test statistic 𝜆(𝜇 = 𝒉⊤ 𝒙 ∗ , 𝒚) := min ∥ 𝒚 − 𝑲𝒙∥ 22 − min ∥ 𝒚 − 𝑲𝒙∥ 22 𝒉⊤ 𝒙=𝒉⊤ 𝒙 ∗ 𝒙≥0 𝒙≥0 is stochastically dominated by a 𝜒12 distribution whenever 𝒚 ∼ N (𝑲𝒙 ∗ , 𝑰𝑚 ). 37 Proof. See Section 2.9. □ As an example, the constrained one-dimensional example considered in Section 2.2 satisfies the stochastic dominance result and hence the conjecture. Furthermore, using Example 2.2.7, an alternative characterization of the conjecture is the stochastic dominance of the unconstrained LLR test statistic min 𝒉⊤ 𝒙=𝒉⊤ 𝒙 ∗ ∥ 𝒚 − 𝑲𝒙∥ 22 −min𝒙 ∥ 𝒚 − 𝑲𝒙∥ 22 over the constrained test statistic min 𝒉⊤ 𝒙=𝒉⊤ 𝒙 ∗ ∥ 𝒚 − 𝑲𝒙∥ 22 − min𝒙≥0 ∥ 𝒚 − 𝑲𝒙∥ 22 . 𝒙≥0 We use Lemma 2.4.3 to prove both that the example in (Tenorio, Fleck, and Moses, 2007) obeys the conjecture and that our new counterexample does not. Invalidity of a previous counterexample in two dimensions The previously proposed counterexample from (Tenorio, Fleck, and Moses, 2007) is a two-dimensional problem with 𝑲 = 𝑰2 , 𝒙 ∗ = (𝑎, 𝑎) ⊤ with 𝑎 ≥ 0, 𝒉 = (1, −1) ⊤ (and therefore 𝜇∗ = 𝒉⊤ 𝒙 ∗ = 0). The LLR test statistic is 𝜆(𝜇∗ = 0, 𝒚) = 𝑥min ∥𝒙 − 𝒚∥ 22 − min ∥𝒙 − 𝒚∥ 22 =𝑥 1 2 𝒙≥0 𝒙≥0 which, after solving the optimization problems, is equal to    𝑦 2 + 𝑦 22 − (𝑦 1 − max(𝑦 1 , 0)) 2 − (𝑦 2 − max(𝑦 2 , 0)) 2 ,  𝜆(𝜇 , 𝒚) = 1   12 (𝑦 1 − 𝑦 2 ) 2 − (𝑦 1 − max(𝑦 1 , 0)) 2 − (𝑦 2 − max(𝑦 2 , 0)) 2 ,  ∗ 𝑦 1 + 𝑦 2 < 0, 𝑦 1 + 𝑦 2 ≥ 0, which we can equivalently write as 1 𝜆(𝜇∗ , 𝒚) = (𝑦 21 + 𝑦 22 ) 1{𝑦 1 + 𝑦 2 < 0} + (𝑦 1 − 𝑦 2 ) 2 1{𝑦 1 + 𝑦 2 ≥ 0} 2 2 2 − 𝑦 1 1{𝑦 1 < 0} − 𝑦 2 1{𝑦 2 < 0}. (2.44) Lemma 2.4.4 (Invalidity of a previous counterexample). The LLR statistic 𝜆(𝜇∗ , 𝒚) in (2.44) is stochastically dominated by a 𝜒12 random variable whenever 𝒚 ∼ N (𝒙 ∗ , 𝑰2 ), 𝒙 ∗ = (𝑎, 𝑎) ⊤ for 𝑎 ≥ 0, and 𝒉 = (1, −1) ⊤ . Therefore, it does not constitute a valid counterexample to the conjecture. Proof sketch. The proof follows from a coupling argument between the LLR and a 𝜒12 random variable. See Section 2.9 for proof details. □ In summary, we used Lemma 2.4.3 to demonstrate that the previously proposed counterexample actually satisfies the conjecture. 38 A new provably valid counterexample in three dimensions We now present a new counterexample in R3 to refute the Burrus conjecture. Specifically, we consider 𝑲 = 𝑰3 , 𝒙 ∗ = (0, 0, 1) ⊤ , and 𝒉 = (1, 1, −1) ⊤ , yielding 𝜇∗ = −1. We prove that 𝜒12 does not stochastically dominate 𝜆(𝜇∗ , 𝒚), which in this case is 𝜆(𝜇∗ = −1, 𝒚) = min 𝑥 1 +𝑥 2 −𝑥3 =−1 𝒙≥0 ∥𝒙 − 𝒚∥ 22 − min ∥𝒙 − 𝒚∥ 22 . 𝒙≥0 (2.45) We prove that E[𝜆(𝜇∗ , 𝒚)] > E[ 𝜒12 ] = 1. Here, the expectation is taken with respect to 𝒚 ∼ N (𝒙 ∗ , 𝑰3 ), and the inequality is a general sufficient condition to refute stochastic dominance and hence for the conjecture to break. Lemma 2.4.5 (Validity of a new counterexample). 𝜆(𝜇∗ , 𝒚) in (2.45) is not stochastically dominated by a 𝜒12 random variable whenever 𝒚 ∼ N (𝒙 ∗ , 𝑰3 ) with 𝒙 ∗ = (0, 0, 1) ⊤ . Therefore, it constitutes a valid counterexample to the general conjecture. Proof sketch. We compute the expected value and show that it is greater than 1 (the expected value of a 𝜒12 ), therefore proving stochastic dominance. See Section 2.9 for the proof details. □ Remark 5 (A more general counterexample). The validity of the counterexample does not hinge on 𝒙 ∗ being on the boundary of the constraint set. In fact, the example remains valid for 𝒙 ∗ = (𝜀, 𝜀, 1) ⊤ with 𝜀 > 0 sufficiently small. We choose 𝜀 = 0 for the simplicity of the proof. See Figure 2.10 for numerical evidence, where the quantiles over the dashed line correspond to valid counterexamples. Figure 2.5 shows the difference between the two examples. By plotting the difference between the CDF of 𝜆 (obtained numerically with 𝑁 = 106 samples) and the CDF of a 𝜒12 distribution, we observe stochastic dominance for the two-dimensional example in Figure 2.5 (left panel) and no stochastic dominance (hence breaking of the conjecture) for the three-dimensional example in Figure 2.5 (right panel). Section 2.5 contains numerical coverage studies for both scenarios agreeing with the observation made here. A negative result in high dimensions After establishing that the 𝜒12 distribution fails to stochastically dominate the constrained log-likelihood ratio, a natural question arises: Is there another distribution, possibly within the 𝜒𝑘2 family, that can stochastically dominate the constrained LLR? If such a distribution exists, it would allow us to redefine 𝜓𝛼2 in (2.42) as 𝑠2 +𝑄 𝑋 (1−𝛼), 39 Difference of CDFs in previously proposed 2D counterexample 0.04 Difference of CDFs in our proposed 3D counterexample CDF λ - CDF χ21 CDF λ - CDF χ21 ↑ Stochastic dominance 0.00 0.03 ↓ Not stochastic dominance 0.02 −0.02 0.01 ↑ Stochastic dominance ↓ Not stochastic dominance 0.00 −0.01 0 2 4 6 8 10 −0.04 0 z 2 4 6 8 10 z Figure 2.5: Difference of cumulative distribution functions between the LLR test statistic and 𝜒12 distribution for the statistics defined in (2.44) (left) and (2.45) (right). Stochastic dominance, which is equivalent to the Burrus conjecture, is broken in the right example only. There is a direct correspondence between the points at which the CDF difference is negative and confidence levels 1 − 𝛼 that fail to hold (see Remark 3). making the 𝑄 𝑋 term in the optimization problem dimension independent, leading to intervals with shorter length in large dimensions. It is worth noting that in the unconstrained scenario, the LLR distribution is precisely 𝜒12 , regardless of the dimensionality of the problem. However, the following proposition shows that no such dimension-independent distribution exists for the constrained case. Proposition 2.4.6 (A negative result in high dimensions). The family of constrained LLRs for general 𝑲, 𝒉 in arbitrary dimensions, defined as 𝜆(𝜇 = 𝒉⊤ 𝒙 ∗ , 𝒚) = min ∥ 𝒚 − 𝑲𝒙∥ 22 − min ∥ 𝒚 − 𝑲𝒙∥ 22 , 𝒉⊤ 𝒙=𝒉⊤ 𝒙 ∗ 𝒙≥0 𝒙≥0 cannot be stochastically dominated a dimension-independent way by any finite-mean distribution (including all 𝜒𝑘2 for 𝑘 ≥ 1). Proof sketch. We construct a sequence of examples with increasing dimensions and demonstrate that the expected value of the constrained LLR grows unbounded as the dimension increases. This result negates the possibility of stochastic dominance by any finite-mean distribution. For a detailed proof, see Section 2.9. □ 2.5 Numerical examples In this section, we provide numerical illustrations for the procedures and theoretical results described above. In particular, we analyze coverage properties of four types of intervals. The first two intervals come from previous works: ISSB (2.3), which has provably correct coverage but is known to overcover when inferring a single functional, and IOSB (2.5), which comes from the Burrus conjecture and, as we 40 proved in this work, does not correctly cover in general. The last two are the interval constructions described in this work by solving the max quantile problems (2.20) and (2.21) to find 𝑞 𝛼 (𝜇) (Section 2.3): IMQ (where 𝑞 𝛼 does not depend on 𝜇) and the more refined IMQ𝜇 (where 𝑞 𝛼 depends on 𝜇). Both of these intervals have provable coverage. We analyze four settings in which the Burrus conjecture applies, including a onedimensional example in Section 2.5, the previously proposed invalid counterexample in Section 2.5 (for which we prove in Lemma 2.4.4 that IOSB correctly covers), our proposed counterexample in Section 2.5 (for which we prove in Lemma 2.4.5 that IOSB does not cover for at least some level 1 − 𝛼 and a certain 𝒙 ∗ ), and a setting with a bounded constraint set11 X in Section 2.5. Throughout the examples, the observed coverage (or lack thereof) agrees with the developed theoretical results, and we observe that our interval IMQ𝜇 consistently fixes the miscalibration of the other interval types: when IOSB undercovers, IMQ𝜇 is on average longer than IOSB to obtain coverage, and when IOSB overcovers, IMQ𝜇 is on average shorter than IOSB with coverage closer to the prescribed level 1 − 𝛼. Constrained Gaussian in one dimension We revisit the constrained Gaussian model in one dimension (2.26) described in Section 2.2, 𝑦 = 𝑥 ∗ + 𝜀, 𝜀 ∼ N (0, 1), 𝑥 ∗ ≥ 0, and 𝜑(𝑥) = 𝑥. We perform a simulation experiment using six true parameter settings of 𝑥 ∗ ∈ {0, 2−3 , 2−2 , 2−1 , 20 , 21 }. We focus on settings closer to the boundary since that is where the biggest differences between the considered intervals exist. For each of these settings, we simulate 105 observations according to the model (2.26) and compute three different 95% confidence intervals for each sample: interval (2.24) using the actual quantile function given in (2.29) (IMQ 𝜇 12 ), interval (2.24) using the stochastically dominating 𝑄 𝜒2 (1 − 𝛼) quantile (IOSB , which in this problem is equal to IMQ ), and the standard 1 Truncated Gaussian interval, which equals the SSB interval in this case (ISSB ). The 11 Strictly speaking, this setting is not included in the original Burrus conjecture but is later extended in (Patil, Kuusela, and Hobbs, 2022; Stanley, Patil, and Kuusela, 2022). 12 Since this is a one-dimensional problem with 𝜑(𝑥) = 𝑥, this is equivalent to being able to solve all the 𝜇−dependant quantile optimization problems 41 intervals computed with the true quantile function are characterized by: IMQ 𝜇 (𝑦) := min/max 𝑥 𝑥 𝑥 st (2.46) 𝑥≥0 (𝑥 − 𝑦) 2 ≤ 𝑞 𝛼 (𝑥) + min (𝑥 − 𝑦) 2 , 𝑥≥0 where 𝑞 𝛼 (𝑥) is given by (2.29). For the stochastically dominating 𝑄 𝜒2 (1 − 𝛼), the 1 interval in (2.24) becomes: IOSB (𝑦) = min/max 𝑥 𝑥 𝑥 st (2.47) 𝑥≥0 (𝑥 − 𝑦) 2 ≤ 𝑄 𝜒2 (1 − 𝛼) + min (𝑥 − 𝑦) 2 . 1 𝑥≥0 Finally, the truncated Gaussian interval, which is shown below to be equivalent to the SSB interval in this case, is defined as:   ISSB (𝑦) := 𝑦 − 𝑧 𝛼/2 , 𝑦 + 𝑧 𝛼/2 ∩ R≥0 . (2.48) Observe that (2.47) admits an explicit solution: √︃ √︃     [𝑦 − 𝑄 𝜒12 (1 − 𝛼), 𝑦 + 𝑄 𝜒12 (1 − 𝛼)] ∩ R≥0 ,  √︃ √︃ IOSB (𝑦) =  2 , 𝑦 + 𝑄 2 (1 − 𝛼) + 𝑦 2 ] ∩ R ,  (1 − 𝛼) + 𝑦 𝑄 [𝑦 − 2 ≥0  𝜒1 𝜒1  𝑦≥0 𝑦 < 0. (2.49) √︃ Furthermore, note that 𝑄 𝜒2 (1 − 𝛼) = 𝑧 𝛼/2 , so that (2.49) is always larger than or 1 equal to (2.48). Conversely, we can express (2.48) as the solution to optimization problems, illustrating that the truncated Gaussian interval is equivalent to the SSB interval for this case: ISSB (𝑦) = min/max 𝑥 𝑥 𝑥 st 𝑥≥0 (2.50) (𝑥 − 𝑦) 2 ≤ 𝑧2𝛼/2 . To empirically estimate coverage, for each 𝑥 ∗ setting and each interval type, we compute 105 intervals and keep track of their coverage of the true parameter. The left panel in Figure 2.6 shows how IOSB based on 𝑄 𝜒2 (1 − 𝛼) over-covers when 1 the true parameter is on the boundary, which makes sense as this setting of 𝑞 𝛼 holds for all 𝑥 ∗ , and therefore is a conservative quantile. As expected, the interval 42 computed with 𝑞 𝛼 (𝑥) maintains the nominal 95% coverage over all considered 𝑥 ∗ values. This shows that knowing the quantile function means that we can compute an interval with exact nominal coverage that is adaptive to the unknown true parameter value. Additionally, we note that as 𝑥 ∗ grows, the estimated coverage values across these methods converge, illustrating the intuition that when 𝑥 ∗ gets sufficiently far from the constraint boundary, the problem is essentially unconstrained, and all considered methods produce nearly identical results. The right panel in Figure 2.6 shows each interval’s expected length as a function of 𝑥 ∗ . Again, we observe the tightness of the interval (2.24) constructed with the true quantile function 𝑞 𝛼 (𝑥) compared to the interval constructed with the stochastically dominating quantile, 𝑄 𝜒2 (1− 𝛼). Similarly to coverage, as 𝑥 ∗ grows, the expected interval lengths of IMQ 𝜇 1 and IOSB converge, and the methods become indistinguishable. Also, observe that the truncated Gaussian intervals have a smaller expected length compared to the intervals computed with 𝑞 𝛼 (𝑥). We note that this length observation is particular to this one-dimensional example, as OSB intervals have been shown to be shorter than SSB intervals in higher dimensional problems (O’Leary and Bert W. Rust, 1986; Burt W. Rust and Walter R. Burrus, 1972; Stanley, Patil, and Kuusela, 2022). Constrained Gaussian in two dimensions We consider the Gaussian linear model in (2.4) with 𝑲 = 𝑰2 , 𝜑(𝒙) = 𝒉⊤ 𝒙 = 𝑥 1 − 𝑥 2 and X = {𝒙 ∈ R2 : 𝒙 ≥ 0}. The work (Tenorio, Fleck, and Moses, 2007) proposes this scenario as a counterexample to the Burrus conjecture, but as shown in Lemma 2.4.4, it is in fact a case where the 𝜒12 distribution stochastically dominates the LLR for all true 𝒙 ∗ ∈ {𝒙 : 𝒉⊤ 𝒙 = 0, 𝒙 ≥ 0}, so the OSB intervals proposed by the conjecture have provably correct coverage for 𝒙 ∗ in this set. We estimate interval coverage with 1 − 𝛼 = 0.95 for the four types of intervals (ISSB , IOSB , IMQ and IMQ𝜇 ) for three true parameter values, two of them inside the {𝒙 : 𝒉⊤ 𝒙 = 0, 𝒙 ≥ 0} region in which Lemma 2.4.4 applies, and one outside. In this and all the examples that follow, we solve the max quantile optimization problems (2.20), (2.21) using the Bayesian Optimization package BayesOpt (Martinez-Cantin, 2014) (see fig. 2.8 for an illustration of the quantile function being optimized in this particular example). We solve the outer optimization problems with the convex optimization package CVXPY (Diamond and Boyd, 2016; Agrawal et al., 2018) (for the convex problems in ISSB , IOSB and IMQ ) or root-finding numerical algorithms (in the case of IMQ𝜇 , where we use the functional space formulation in algorithm 1 and look for those 𝜇 satisfying the constraint as equality). 43 Figure 2.6: The (left) figure shows estimated coverage for each 95% confidence interval and each true 𝑥 ∗ for the one-dimensional constrained Gaussian model. Both the Truncated Gaussian (SSB) and OSB intervals overcover when 𝑥 ∗ = 0 while the MQ𝜇 interval predictably achieves nominal coverage. All the coverage values converge as 𝑥 ∗ gets larger, as the problem moves toward the unconstrained problem where all the intervals are effectively the same. The intervals surrounding the estimated values are 95% Clopper–Pearson intervals, expressing the Monte Carlo uncertainty of each coverage estimate. The (right) figure shows an estimate of the expected interval length for each method (with 95% confidence intervals that are nearly length zero since the standard error of each estimate is nearly zero with 105 realizations each). Similarly to the coverage results in the left panel, as 𝑥 ∗ gets larger, the expected OSB and MQ𝜇 interval lengths converge while the SSB intervals remain slightly larger. For each parameter value and all intervals, coverage and expected length are estimated by drawing 5 × 104 observations from the data generating process, computing all interval types for each generated observation, and then checking coverage and length. The coverage confidence intervals are 95% Clopper–Pearson intervals for a binomial parameter, whereas the length confidence intervals are standard asymptotic Gaussian intervals using sample means and standard errors. The results are shown in fig. 2.7. We observe correct coverage for the OSB intervals, in agreement with Lemma 2.4.4 (which applies for 𝒙 ∗ = (0, 0) ⊤ and 𝒙 ∗ = (0.33, 0.33) ⊤ ). We observe that the SSB intervals are the longest on average and tend to overcover, and that the MQ and MQ𝜇 intervals have nearly identical properties to the OSB intervals. This is because, for this problem setting, solving the optimization problems (2.20) and (2.21) recovers the 𝜒12 quantile (up to the numerical precision of the optimization solvers) of the Burrus conjecture as the maximum quantile (both for all X and for 𝒉⊤ 𝒙 = 𝜇 for any 𝜇), so both of those intervals actually recover the OSB intervals in this case. 44 Figure 2.7: Estimated interval coverage (left) and expected lengths (right) for 95% intervals resulting from the SSB, OSB, MQ, and MQ𝜇 methods for the Gaussian linear model in (2.4) with 𝑲 = 𝑰2 , 𝜑(𝒙) = 𝒉⊤ 𝒙 = 𝑥 1 −𝑥 2 , and X = {𝒙 ∈ R2 : 𝒙 ≥ 0}. Figure 2.8: Estimated 95th quantiles from the LLR test statistic null distributions in the region [0, 1] 2 ⊂ R2+ where color shows the estimated quantiles. In contrast to the unconstrained case, the quantile is dependent on the true parameter, and the quantile surface is non-trivial. Constrained Gaussian in three dimensions We use the three-dimensional counterexample of the Burrus conjecture from Section 2.4 to numerically show that the IOSB intervals undercover in this example and that the max-quantile intervals are able to fix the undercoverage. Concretely, we consider the Gaussian linear model in (2.4) with 𝑲 = 𝑰3 , 𝜑(𝒙) = 𝒉⊤ 𝒙 = 𝑥 1 + 𝑥 2 − 𝑥3 and X = {𝒙 ∈ R3 : 𝒙 ≥ 0}. We repeat the same experimental setup as in Section 2.5, comparing the four interval types for 𝒙 ∗ = (0, 0, 0) ⊤ and 𝒙 ∗ = (0, 0, 1) ⊤ ; this last parameter value is the one analyzed in Lemma 2.4.5 and for which we know the OSB interval can undercover for some 𝛼. Figure 2.9 shows the results for 1−𝛼 = 0.68 (one 45 Figure 2.9: Estimated interval coverage (left) and expected lengths (right) for 68% intervals resulting from the SSB, OSB, MQ, and MQ𝜇 methods for the Gaussian linear model in (2.4) with 𝑲 = 𝑰3 , 𝜑(𝒙) = 𝒉⊤ 𝒙 = 𝑥 1 + 𝑥2 − 𝑥 3 , and X = {𝒙 ∈ R3 : 𝒙 ≥ 0}. sigma interval coverage), and we include in Section 2.10 the results for 1 − 𝛼 = 0.95, which lead to the same conclusions. Furthermore, to illustrate that the conjecture breaks in an area around the studied point (0, 0, 1) ⊤ in Figure 2.10, we plot the numerically estimated 68% LLR test statistic quantiles for parameters of the form (𝑡, 𝑡, 1), showing that there is a range of points with a larger quantile than the 𝜒12 quantile, implying undercoverage. The results agree with our theoretical findings in Lemma 2.4.5, as we observe that the OSB intervals undercover both at 95% and 68% confidence levels, thus invalidating the Burrus conjecture. In contrast, the SSB, MQ and MQ𝜇 intervals all have provable coverage in this scenario, which is reflected in the estimated coverage values. Notably, the MQ𝜇 intervals are not much longer than the OSB intervals, but they obtain the required coverage, enlarging the conjectured intervals just enough. The simpler MQ intervals overcover a bit more, which illustrates the benefit of solving (2.21) over (2.20) when computationally feasible. Nevertheless, their length is not much larger than MQ𝜇 and significantly smaller than for SSB, the other simple method with coverage guarantees. Bounded constraint set in two dimensions As a last case study, we consider a modification of the example in Section 2.5 in which the constraint set is chosen to be the bounded set X = [0, 1] 2 . While not in the original scope of the Burrus conjecture, which only considers X = {𝒙 : 𝒙 ≥ 0}, 46 Figure 2.10: For the numerical example in Section 2.5, considering 𝒙 ∗ ∈ {𝒙 ∗ (𝑡) = (𝑡, 𝑡, 1) ⊤ : 0 < 𝑡 ≤ 𝑒} ⊂ R3 , we estimate the 68% LLR test statistic quantiles along with 95% nonparametric (NP) confidence intervals for percentiles (Hahn 2 and Meeker, 1991). The test statistic quantiles exceeding 𝜒1,0.32 correspond to the Burrus conjecture failing in this scenario. We note that for this example, the Burrus conjecture fails close to the constraint boundary while the 𝑄 𝜒2 (0.68) quantile be1 comes valid once sufficiently far from the boundary. as mentioned in Section 2.1, the IOSB interval construction has since then been used with constraints of the form A𝒙 ≤ b by replacing 𝒙 ≥ 0 with A𝒙 ≤ b in the optimization problems (2.5) and (2.6) (Patil, Kuusela, and Hobbs, 2022; Stanley, Patil, and Kuusela, 2022). We use the same experimental setup as in Section 2.5, taking into account that the change in X affects both the interval optimization problem and the optimizations required for 𝑞 𝛼 (𝜇) (since the LLR statistic changes as well). The results are shown in Figure 2.11. We observe that in this setting, as opposed to the previous three-dimensional problem, the OSB intervals overcover, because the maximum quantile of the LLR test statistic over the constraint set is smaller than the 𝜒12 quantile. Furthermore, our intervals MQ, and especially MQ𝜇, are able to exploit this fact to obtain shorter intervals than OSB with coverage closer to 1 − 𝛼 by using the actual max quantiles over X instead of the 𝜒12 quantile used by OSB. 2.6 Discussion This paper presents a framework for constructing confidence intervals with guaranteed frequentist coverage for a given functional of forward model parameters in the presence of constraints. For the specific case of the Gaussian linear forward model with nonnegativity constraints, we refute the Burrus conjecture (W. R. Burrus, 1965) by providing a counterexample and propose a more general approach for interval construction. Our approach hinges on the inversion of a specific like- 47 Figure 2.11: Estimated interval coverage (left) and expected lengths (right) for 95% intervals resulting from the SSB, OSB, MQ, and MQ𝜇 methods for the Gaussian linear model in (2.4) with 𝑲 = 𝑰2 , 𝜑(𝒙) = 𝒉⊤ 𝒙 = 𝑥 1 − 𝑥2 , and X = [0, 1] 2 ⊂ R2 . lihood ratio test, and we offer theoretical and practical insights into the properties of the constructed intervals via illustrative examples. Our framework is versatile, accommodating potentially nonlinear, non-Gaussian, and rank-deficient settings. At a high level, the practical effectiveness of UQ methods depends on the (sometimes implicit) assumptions of the method. Different methods come into play depending on what we assume or know, be it the likelihood, the constraints, or the prior in Bayesian settings. In classical statistics, confidence intervals serve as a valuable tool for UQ, especially for one-dimensional quantities of interest. These intervals are constructed to offer guaranteed coverage under repeated sampling, aligning with frequentist principles. While frequentist coverage guarantees are a useful criterion, especially in contexts where repeatability is essential, we acknowledge that the “best” UQ method is often context-dependent. For example, this frequentist approach is the most natural in applications like remote sensing (Patil, Kuusela, and Hobbs, 2022), where repeatability is a key requirement. Conversely, when it is natural to think of the parameter as arising from a prior distribution, Bayesian methods are well-motivated and have desirable properties. A key aim of this paper is to serve as a basis for the future development of these UQ procedures. We conclude this paper by discussing a few possible directions for future work: • Data-adaptive calibration procedure. We saw in Section 2.5 that MQ is valid where OSB is not and can leverage smaller quantiles to produce tighter intervals. 48 In the event that one does not have the assumed true parameter bounding box, it may be possible to create a data-generated one and adjust the error budget accordingly. Such a procedure would enable us to expand the use of the MQ intervals to scenarios with unbounded parameter constraints. We are currently investigating this approach, which will be the subject of a future follow-up paper. • Exploration of high-dimensional problems. A key benefit of the optimizationbased interval construction is that it promises to provide a solution to uncertainty quantification in high-dimensional and ultra-high-dimensional problems, where most alternative approaches (including sampling-based ones) are infeasible. For example, in (Stanley, Patil, and Kuusela, 2022), we applied a precursor of the methods presented here in a problem where 𝑝 = 80, and we are currently exploring the use of these methods in a data-assimilation setting where 𝑝 ≈ 104 . Indeed, optimization is one of the only computational techniques known to work in ultra-high-dimensional problems, such as those in 4DVar data assimilation (Kalmikov and Heimbach, 2014; Forget et al., 2015; J. Liu, Bowman, Meemong, et al., 2016). While optimization has been successfully used for point estimation in such problems, the approaches developed here may enable modifying the existing programs to obtain confidence intervals in addition to point estimates. • Joint confidence sets for multiple functionals. Since our framework is devised for UQ of a single functional, its application to collections of functionals (a higher-dimensional quantity of interest), would be a natural and desirable extension. Trivially, given a collection of 𝐾 functionals, one could apply this methodology 𝐾 times and use the Bonferroni correction to adjust the confidence levels so that they all cover at the desired coverage level. Although this approach might be practically reasonable when 𝐾 is small, it becomes markedly inefficient as 𝐾 becomes large. Furthermore, this approach would create a 𝐾-dimensional hyper-rectangle for the quantity of interest, which may not be the optimal geometry for bounding the quantity of interest. As such, extending the framework of Section 2.3 to simultaneously consider the 𝐾 functionals of interest would be the first step to creating a more nuanced approach. One way this can be achieved is by appropriately adjusting the definition of 𝐻0 in the hypothesis test in (2.11). • Choice of test statistics beyond LLR. The log-likelihood ratio test statistic considered in this work connects with the Rust–Burrus intervals and is observed to perform well in practice, but other choices can be explored in future work. While the LLR is a natural choice for the generic problem, im- 49 proving the interval length on particular families of problems with different test statistics might be possible. Since the main theoretical machinery comes from the test inversion framework, which is independent of the actual form of the test statistic, alternate versions of Theorem 2.2.4 can be constructed as long as the test statistic constructs valid level-𝛼 hypothesis tests; the resulting intervals of which could be explored theoretically and numerically. For instance, the construction of confidence intervals in Section 2.2, originally written for 𝜆(𝜇, 𝒚) = inf 𝒙∈Φ 𝜇 ∩X −2ℓ𝒙 (𝒚) − inf 𝒙∈X −2ℓ𝒙 (𝒚), readily generalizes to test statistics of the form 𝜆 𝑓 ,𝑔 (𝜇, 𝒚) = inf 𝒙∈Φ 𝜇 ∩X 𝑓 (𝒙, 𝒚) − 𝑔(𝒚). In that case, (2.22) becomes {𝒙 : 𝑓 (𝒙, 𝒚) ≤ 𝑞 𝛼 (𝜑(𝒙)) +𝑔(𝒚)}, where 𝑞 𝛼 must be valid for the particular choice of 𝑓 and 𝑔, and can be obtained by analyzing the distribution of 𝜆 𝑓 ,𝑔 . • Generalization to simulation-based problems. An extension of our methodology to settings in which the likelihood is not exactly known can be considered, ranging from only partial knowledge of the form of the likelihood to full simulation-based (likelihood-free) settings where the likelihood is not known explicitly but can be sampled from. A possible avenue is to develop robust worstcase approaches with respect to possible likelihoods. In a fully likelihood-free setting, approaches such as (Niccolò Dalmasso, Izbicki, and A. B. Lee, 2020; Heinrich, 2022; Masserano, Dorigo, et al., 2023; Niccolò Dalmasso, Masserano, et al., 2023) provide ways to invert hypothesis tests to obtain confidence sets in these scenarios in multidimensional parameter spaces. Projections of these sets could produce confidence intervals for a functional of the model parameters, as seen for the SSB intervals. However, as we have explored, orienting the hypothesis test to the given functional of interest can have dramatic length benefits for the resulting confidence interval (as seen for the OSB and MQ intervals). Since the log-likelihood plays a key role in the definition of our intervals, extensions providing ways to relax that dependence would be a necessary first step. 50 2.7 Proofs in Section 2.2 Proof of Lemma 2.2.1 To prove the lemma, we need to show that the probability of 𝜇∗ being in the confidence set C𝛼 (𝒚) is at least 1 − 𝛼. Towards this end, observe that P 𝒚∼𝑃 𝒙∗ (𝜇∗ ∈ C𝛼 ( 𝒚)) = P 𝒚∼𝑃 𝒙∗ (𝒚 ∈ 𝐴𝛼 (𝜇∗ )) = 1 − P 𝒚∼𝑃 𝒙∗ ( 𝒚 ∉ 𝐴𝛼 (𝜇∗ )) ≥ 1− sup 𝒙∈Φ 𝜇∗ ∩X P 𝒚∼𝑃 𝒙 ( 𝒚 ∉ 𝐴𝛼 (𝜇∗ )) ≥ 1 − 𝛼, as desired. This completes the proof. Proof of Lemma 2.2.2 The value 𝑄 max is a valid decision value for any given 𝜇, since for all 𝒙 ∈ Φ 𝜇 ∩ X it 𝜇 holds that !
P𝜆∼𝐹𝒙 𝜆 > sup 𝑄 𝐹𝒙′ (1 − 𝛼) ≤ P𝜆∼𝐹𝒙 𝜆 > 𝑄 𝐹𝒙 (1 − 𝛼) = 𝛼. (2.51) 𝒙 ′ ∈Φ 𝜇 ∩X For any 𝑣 < 𝑄 max that one could use as a decision value, there exists e 𝒙 ∈ Φ𝜇 ∩ X 𝜇 such that 𝑄 𝐹e𝒙 (1 − 𝛼) > 𝑣. Therefore, P𝜆∼𝐹e𝒙 (𝜆 > 𝑣) > 𝛼, and thus 𝑣 is not a valid decision value. 𝑄 max is clearly valid for all 𝜇, and a similar argument shows that choosing any smaller 𝑣 would make it not valid, as there exists a e 𝒙 ∈ X with a larger quantile than 𝑣, making 𝑣 invalid as a decision value for 𝜇 = 𝜑(e 𝒙 ). The equality between the two formulations comes from the same argument that shows (2.17) is equivalent to (2.18). Proof of Theorem 2.2.4 Assume X̄𝛼 ( 𝒚) is nonempty and write as shorthand inf 𝒙∈X /sup𝒙∈X 𝑓 (𝒙) for the interval   inf 𝑓 (𝒙), sup 𝑓 (𝒙) . 𝒙∈X 𝒙∈X Observe that C𝛼 (𝒚) ⊆ inf / sup 𝜇. 𝜇∈C𝛼 (𝒚) 𝜇∈C𝛼 (𝒚) (2.52) From Lemma 2.2.1, C𝛼 ( 𝒚) ⊆ I𝛼 (𝒚) implies that I𝛼 ( 𝒚) is also a 1 − 𝛼 confidence interval. We prove this interval exactly equals the defined I𝛼 ( 𝒚) in (2.24). Unpacking 51 the definition of C𝛼 (𝒚), we write the interval inf /sup 𝜇 𝜇 𝜇 st (2.53) 𝜇∈R − 2 log Λ(𝜇, 𝒚) ≤ 𝑞 𝛼 (𝜇). We can write different optimization problems which are equivalent to the optimization problem (2.53). First, we use the definition of Λ to write: inf /sup 𝜇 𝜇 𝜇 (2.54) 𝜇∈R st inf −2ℓ𝒙 ( 𝒚) − inf −2ℓ𝒙 ( 𝒚) ≤ 𝑞 𝛼 (𝜇). 𝜑(𝒙)=𝜇,𝒙∈X 𝒙∈X Notice that we can rewrite the feasibility condition of 𝜇 as follows: −2ℓ𝒙 ( 𝒚) ≤ 𝑞 𝛼 (𝜇) + inf −2ℓ𝒙 ( 𝒚) inf 𝒙∈X 𝜑(𝒙)=𝜇,𝒙∈X as there exists 𝒙 ∈ X such that 𝜑(𝒙) = 𝜇 and −2ℓ𝒙 ( 𝒚) ≤ 𝑞 𝛼 (𝜇) + inf −2ℓ𝒙 ( 𝒚). 𝒙∈X Therefore, the optimization problem can be rewritten with 𝒙 and 𝜇 as the optimization variables: inf /sup 𝜇 𝜇,𝒙 𝜇,𝒙 st 𝒙 ∈ X, 𝜇 ∈ R (2.55) 𝜑(𝒙) = 𝜇 − 2ℓ𝒙 ( 𝒚) ≤ 𝑞 𝛼 (𝜇) + inf −2ℓ𝒙 ( 𝒚). 𝒙∈X And 𝜇 can be eliminated using the constraint, yielding inf /sup 𝒙 𝜑(𝒙) 𝒙 st (2.56) 𝒙∈X − 2ℓ𝒙 (𝒚) ≤ 𝑞 𝛼 (𝜑(𝒙)) + inf −2ℓ𝒙 (𝒚), 𝒙∈X that is, inf 𝒙∈X̄𝛼 ( 𝒚) /sup𝒙∈X̄𝛼 (𝒚) 𝜑(𝒙). The choice when X̄𝛼 ( 𝒚) is empty does not affect coverage properties. An alternative proof comes by observing the set equality 𝜑({𝒙 ∈ X : −2ℓ𝒙 (𝒚) ≤ 𝑞 𝛼 (𝜑(𝒙))+inf 𝒙∈X −2ℓ𝒙 (𝒚)}) = {𝜇 ∈ 𝜑(X) : inf 𝜑(𝒙)=𝜇,𝒙∈X −2ℓ𝒙 ( 𝒚)− inf 𝒙∈X −2ℓ𝒙 ( 𝒚) ≤ 𝑞 𝛼 (𝜇)}, and the result follows. This finishes the proof. 52 Proof of Proposition 2.2.5 We have, by definition and test inversion, that 𝑞 𝛼 (𝜇) are valid if and only if C𝛼 ( 𝒚) := {𝜇 : 𝜆(𝜇, 𝒚) ≤ 𝑞 𝛼 (𝜇)} is a valid 1− 𝛼 confidence interval for any 𝒙 ∈ X. Since I𝛼 (𝒚) is the smallest interval that contains C𝛼 (𝒚), we aim to prove that C𝛼 ( 𝒚) is already an interval (including singletons or empty sets), so that C𝛼 (𝒚) = I𝛼 ( 𝒚) and the result holds. Define the function: 𝜇 ↦→ F (𝜇) = inf −2ℓ𝒙 ( 𝒚) (2.57) 𝜑(𝒙)=𝜇 𝒙∈X for a given 𝒚, supported in all 𝜇 such that Φ 𝜇 ∩ X ≠ ∅. Write C𝛼 ( 𝒚) explicitly using (2.13), we get ) ( C𝛼 ( 𝒚) := 𝜇 : inf − 2ℓ𝒙 ( 𝒚) − inf − 2ℓ𝒙 (𝒚) ≤ 𝑞 𝛼 . 𝒙∈X 𝜑(𝒙)=𝜇 𝒙∈X (2.58) The second term on the left-hand side does not depend on 𝜇, so it is enough to prove that any set of the form {𝜇 : F (𝜇) ≤ 𝑧} is an interval, which is implied by the function F (𝜇) being convex in 𝜇 (for a fixed 𝒚). Indeed, if the set is not an interval, we have 𝜇− < 𝜇 < 𝜇+ with 𝜇− , 𝜇+ ∈ C𝛼 ( 𝒚) and 𝜇 ∉ C𝛼 (𝒚) which contradicts convexity, since F (𝜇) ≥ 𝑧 > 𝛾F (𝜇− ) + (1 − 𝛾)F (𝜇+ ). To see convexity and finish the proof, let 𝜇1 ≠ 𝜇2 and let G(𝒙) := −2ℓ𝒙 ( 𝒚), a convex function by assumption. Write for 𝑖 = 1, 2: 𝑥𝑖 ∈ argmin𝜑(𝒙)=𝜇 − 2ℓ𝒙 ( 𝒚), 𝒙∈X with 𝑥𝑖 being any possible element in the set of minimizers, so that F (𝜇𝑖 ) = G(𝒙𝑖 ). For any 0 < 𝛾 < 1, 𝛾𝒙 1 + (1 − 𝛾)𝒙2 ∈ X since X is a convex cone and 𝜑(𝛾𝒙1 + (1 − 𝛾)𝒙2 ) = 𝛾𝜇1 + (1 − 𝛾)𝜇2 , since 𝜑 is linear, so 𝛾𝒙1 + (1 − 𝛾)𝒙2 is a feasible point of the optimization problem inf 𝜑(𝒙)=𝛾𝜇1 +(1−𝛾)𝜇2 𝒙∈X −2ℓ𝒙 ( 𝒚), that has optimal value F (𝛾𝜇1 + (1 − 𝛾)𝜇2 ). Therefore, by convexity of G and definition of the 𝒙𝑖 , we have that: F (𝛾𝜇1 +(1−𝛾)𝜇2 ) ≤ G(𝛾𝒙 1 +(1−𝛾)𝒙2 ) ≤ 𝛾G(𝒙1 )+(1−𝛾)G(𝒙2 ) = 𝛾F (𝜇1 )+(1−𝛾)F (𝜇2 ). (2.59) This completes the proof. 53 Proof of Example 2.2.6 Since the case when 𝜇∗ = 0 is of particular interest, we show the result in this specific case and then generalize to the case of 𝜇∗ > 0. Case of 𝜇∗ = 0 When 𝜇∗ = 0, we can argue from symmetry of the standard Gaussian about the origin to write down the CDF in closed form. For 𝑐 ≥ 0, we have P 𝜇0 (ℓ0 ≤ 𝑐) = P 𝜇0 (ℓ0 ≤ 𝑐, 𝑦 < 0) + P 𝜇0 (ℓ0 ≤ 𝑐, 𝑦 ≥ 0) = P 𝜇0 (ℓ0 ≤ 𝑐 | 𝑦 < 0) P 𝜇0 (𝑦 < 0) + P 𝜇0 (ℓ0 ≤ 𝑐 | 𝑦 ≥ 0) P 𝜇0 (𝑦 ≥ 0). (2.60) By definition, P 𝜇0 (𝑦 < 0) = P 𝜇0 (𝑦 ≥ 0) = 12 , so only the conditional probabilities remain. By (2.28), we have P 𝜇0 (ℓ0 ≤ 𝑐 | 𝑦 < 0) = P 𝜇0 (0 ≤ 𝑐 | 𝑦 < 0) = 1   P 𝜇0 (ℓ0 ≤ 𝑐 | 𝑦 ≥ 0) = P 𝜇0 𝑦 2 ≤ 𝑐 | 𝑦 ≥ 0 . (2.61) In (2.61), we immediately observe that     √ √
P 𝜇0 𝑦 2 ≤ 𝑐 | 𝑦 ≥ 0 = P 𝜇0 𝑦 2 ≤ 𝑐, 𝑦 ≥ 0 P 𝜇0 (𝑦 ≥ 0) −1 = 2P 𝜇0 0 ≤ 𝑦 ≤ 𝑐 = 2Φ( 𝑐)−1. (2.62) But we also have that   √ √
√ P 𝜇0 𝑦 2 ≤ 𝑐 = P 𝜇0 − 𝑐 ≤ 𝑦 ≤ 𝑐 = 2Φ( 𝑐) − 1. So we have     P 𝜇0 𝑦 2 ≤ 𝑐 | 𝑦 ≥ 0 = P 𝜇0 𝑦 2 ≤ 𝑐 . Hence, we obtain P 𝜇0 (ℓ0 ≤ 𝑐 | 𝑦 ≥ 0) = 𝜒12 (𝑐). Note this independence on the sign of 𝑦 means that the magnitude of 𝑦 is statistically independent of its direction. Thus, when 𝜇0 = 0, the log-likelihood ratio has the following distribution: 1 1 (2.63) ℓ0 ∼ + 𝜒12 . 2 2 This completes the case when 𝜇∗ = 0. 54 Case of 𝜇∗ > 0 When 𝜇 > 0, the closed-form solution to the CDF of ℓ0 becomes more complicated, as we can no longer use symmetry around the origin. Picking up at (2.60), we first note that when 𝑦 ∼ N (𝜇0 , 1), we have P 𝜇0 (𝑦 < 0) = Φ(−𝜇0 ) and P 𝜇0 (𝑦 ≥ 0) = Φ(𝜇0 ). Next, we must find the conditional probabilities. Starting with the case when {𝑦 < 0}, we obtain   P 𝜇0 (𝑦 − 𝜇0 ) 2 − 𝑦 2 ≤ 𝑐 | 𝑦 < 0 (2.64)   = P 𝜇0 −2𝑦𝜇0 + 𝜇02 ≤ 𝑐 | 𝑦 < 0 ! 𝜇02 − 𝑐 |𝑦<0 = P 𝜇0 𝑦 ≥ 2𝜇0 ! 𝜇02 − 𝑐 −1 = Φ(−𝜇0 ) P 𝜇0 𝑦 ≥ ,𝑦 < 0 2𝜇0 ( ! ) 𝜇02 − 𝑐 −1 2 2 = Φ(−𝜇0 ) 0 · 1{𝑐 ≤ 𝜇0 } + P 𝜇0 ≤ 𝑦 ≤ 0 1{𝑐 > 𝜇0 } 2𝜇0 ! −𝜇02 − 𝑐 −1 = Φ(−𝜇0 ) P 𝜇0 ≤ 𝑦 − 𝜇0 ≤ −𝜇0 1{𝑐 > 𝜇02 } 2𝜇0 !) ( −𝜇02 − 𝑐 −1 1{𝑐 > 𝜇02 }. (2.65) = Φ(−𝜇0 ) Φ(−𝜇0 ) − Φ 2𝜇0 Then, when {𝑦 ≥ 0}, we have   P 𝜇0 (𝑦 − 𝜇0 ) 2 ≤ 𝑐 | 𝑦 ≥ 0 (2.66) √ √
= P 𝜇0 − 𝑐 ≤ 𝑦 − 𝜇 0 ≤ 𝑐 | 𝑦 ≥ 0 √ √
= Φ(𝜇0 ) −1 P 𝜇0 − 𝑐 ≤ 𝑦 − 𝜇0 ≤ 𝑐, 𝑦 ≥ 0 √ √
= Φ(𝜇0 ) −1 P 𝜇0 0 ≤ 𝑦 ≤ 𝑐 + 𝜇0 1{− 𝑐 + 𝜇0 ≤ 0} √ √ √
+ Φ(𝜇0 ) −1 P 𝜇0 − 𝑐 + 𝜇0 ≤ 𝑦 ≤ 𝑐 + 𝜇0 1{− 𝑐 + 𝜇0 > 0}  √ √
= Φ(𝜇0 ) −1 Φ( 𝑐) − Φ(−𝜇0 ) 1{𝑐 ≥ 𝜇02 } + (2Φ( 𝑐) − 1)1{𝑐 < 𝜇02 } . (2.67) Putting together (2.65) and (2.67), we obtain the following CDF: ( !) 2−𝑐 −𝜇 √ 0 P 𝜇0 (ℓ0 ≤ 𝑐) = 𝜒12 (𝑐) · 1{𝑐 < 𝜇02 } + Φ( 𝑐) − Φ · 1{𝑐 ≥ 𝜇02 }. (2.68) 2𝜇0 This completes the case of 𝜇∗ > 0. 55 Proof of Example 2.2.7 We derive this result using a duality argument inspired by (Gouriéroux, Holly, and Monfort, 1982). By definition, we have 𝜆(𝜇∗ , 𝒚) = min ∥ 𝒚 − 𝑲𝒙∥ 22 − min ∥ 𝒚 − 𝑲𝒙∥ 22 . 𝒙:𝜑(𝒙)=𝜇∗ 𝒙 (2.69) For ease of notation, let b 𝒙 ∗ = argmin ∥ 𝒚 − 𝑲𝒙∥ 22 . Consider the Lagrangian for the 𝒙 : 𝒉⊤ 𝒙=𝜇∗ first optimization in (2.69): 𝐿 (𝒙, 𝜆) = ∥ 𝒚 − 𝑲𝒙∥ 22 + 𝜆(𝒉⊤ 𝒙 − 𝜇∗ ). (2.70) First-order optimality allows solving for b 𝒙 ∗ as a function of the dual variable 𝜆: ∇𝒙 𝐿 (𝒙, 𝜆) = −2𝑲 ⊤ ( 𝒚 − 𝑲𝒙) + 𝜆𝒉 = 0 =⇒ −2𝑲 ⊤ 𝒚 + 2𝑲 ⊤ 𝑲𝒙 + 𝜆𝒉 = 0 1 =⇒ b 𝒙 ∗ = (𝑲 ⊤ 𝑲) −1 𝑲 ⊤ 𝒚 − 𝜆(𝑲 ⊤ 𝑲) −1 𝒉 2 1 ⊤ −1 ∗ =⇒ b 𝒙 =b 𝒙 − 𝜆(𝑲 𝑲) 𝒉. 2 Substituting back into the LLR, we obtain 𝜆(𝜇∗ , 𝒚) = ∥ 𝒚 − 𝑲b 𝒙 ∗ ∥ 22 − ∥ 𝒚 − 𝑲b 𝒙 ∥ 22 1 = ∥ 𝒚 − 𝑲b 𝒙 + 𝜆𝑲 (𝑲 ⊤ 𝑲) −1 𝒉∥ 22 − ∥ 𝒚 − 𝑲b 𝒙 ∥ 22 . 2 (2.71) Performing some algebra, we note that 1 ∥ 𝒚 − 𝑲b 𝒙 + 𝜆𝑲 (𝑲 ⊤ 𝑲) −1 𝒉∥ 22 = ∥ 𝒚 − 𝑲b 𝒙 ∥ 22 2 1 + 𝜆( 𝒚 − 𝑲b 𝒙 ) ⊤ 𝑲 (𝑲 ⊤ 𝑲) −1 𝒉 + 𝜆2 𝒉⊤ (𝑲 ⊤ 𝑲) −1 𝒉. 4 Thus, we have 𝜆(𝒚 − 𝑲b 𝒙 ) ⊤ 𝑲 (𝑲 ⊤ 𝑲) −1 𝒉 = 𝜆𝒚 ⊤ 𝑲 (𝑲 ⊤ 𝑲) −1 𝒉 − 𝜆b 𝒙 ⊤ 𝑲 ⊤ 𝑲 (𝑲 ⊤ 𝑲) −1 𝒉 = 𝜆b 𝒙 ⊤ 𝒉 − 𝜆b 𝒙⊤ 𝒉 = 0. So the substitution in (2.71) can be further simplified such that: 1 𝜆(𝜇∗ , 𝒚) = 𝜆2 𝒉⊤ (𝑲 ⊤ 𝑲) −1 𝒉. 4 (2.72) 56 We now turn our attention to finding 𝜆. Note that this optimization defining the Lagrangian (2.70) is convex with an affine equality constraint. Therefore, strong duality holds. We then define the dual function as follows: 𝑔(𝜆) = min 𝐿 (𝒙, 𝜆) = 𝐿 (b 𝒙 ∗ , 𝜆) 𝒙 = ∥ 𝒚 − 𝑲b 𝒙 ∗ ∥ 22 + 𝜆(𝒉⊤ b 𝒙 ∗ − 𝜇∗ )   1 1 ⊤ ⊤ −1 ⊤ −1 2 ⊤ ∗ = ∥ 𝒚 − 𝑲b 𝒙 + 𝜆𝑲 (𝑲 𝑲) 𝒉∥ 2 + 𝜆 𝒉 b 𝒙 − 𝜆𝒉 (𝑲 𝑲) 𝒉 − 𝜇 . (2.73) 2 2 We note that we can make many of the same simplifications above to arrive at the simplified dual function: 1 𝑔(𝜆) = ∥ 𝒚 − 𝑲b 𝒙 ∥ 22 + 𝜆𝒉⊤ − 𝜆2 𝒉⊤ (𝑲 ⊤ 𝑲) −1 𝒉 + 𝜆𝒉⊤ b 𝒙 − 𝜆𝜇∗ . (2.74) 4 To maximize 𝑔(𝜆), we again use the following first order optimality condition: 𝑑𝑔 1 = − 𝜆𝒉⊤ (𝑲 ⊤ 𝑲) −1 𝒉 + 𝒉⊤ b 𝒙 − 𝜇∗ = 0 𝑑𝜆 2 2 (𝒉⊤ b 𝒙 − 𝜇∗ ) b (2.75) =⇒ 𝜆 = ⊤ ⊤ −1 . 𝒉 (𝑲 𝑲) 𝒉 Substituting (2.75) back into (2.72), we obtain 2  1 2 (𝒉⊤ b 𝒙 − 𝜇∗ ) ∗ 𝒉⊤ (𝑲 ⊤ 𝑲) −1 𝒉 𝜆(𝜇 , 𝒚) = 4 𝒉⊤ (𝑲 ⊤ 𝑲) −1 𝒉 (𝒉⊤ b 𝒙 − 𝜇∗ ) 2 = ⊤ ⊤ −1 . 𝒉 (𝑲 𝑲) 𝒉 For the second part, observe that when 𝒚 ∼ N (𝑲𝒙 ∗ , 𝑰𝑚 ), we have 𝒉⊤ (𝑲 ⊤ 𝑲) −1 𝑲 ⊤ 𝒚 ∼ N (𝒉⊤ 𝒙 ∗ , 𝒉⊤ (𝑲 ⊤ 𝑲) −1 𝒉), hence (2.33) is the square of a one-dimensional standard Gaussian distribution. This finishes the proof. 2.8 Proofs in Section 2.3 Proof of Lemma 2.3.2 Let 𝑌 := 𝜆(𝒚, 𝜇∗ ). Recall the validity of 𝑞 𝛼 can be written as P(𝑌 ≤ 𝑞 𝛼 ) ≥ 1 − 𝛼 from (2.14) as: 𝑋 ⪰ 𝑌 ⇐⇒ P(𝑋 ≥ 𝛾) ≥ P(𝑌 ≥ 𝛾), for all 𝛾 ⇐⇒ 𝛼 = P(𝑋 ≥ 𝑄 𝑋 (1 − 𝛼)) ≥ P(𝑌 ≥ 𝑄 𝑋 (1 − 𝛼)), for all 𝛼 ⇐⇒ 1 − 𝛼 = P(𝑋 ≤ 𝑄 𝑋 (1 − 𝛼)) ≤ P(𝑌 ≤ 𝑄 𝑋 (1 − 𝛼)), for all 𝛼 ⇐⇒ 𝑄 𝑋 (1 − 𝛼) is a valid 𝑞 𝛼 for all 𝛼. This finishes the proof. 57 Joint formulation of change-constrained optimization problem (2.36) In this section, we provide details on formulating the optimization problem (2.36) and the interval optimization problem (2.24) as a single chance-constrained optimization problem. By joining (2.36) and (2.24) as a single optimization problem, we can use a similar argument as in the proof of Theorem 2.2.4. Starting with problem (2.54): inf /sup 𝜇 𝜇 𝜇 (2.76) 𝜇∈R st −2ℓ𝒙 (𝒚) − inf −2ℓ𝒙 (𝒚) ≤ 𝑞 𝛼 (𝜇), inf 𝒙∈X 𝜑(𝒙)=𝜇,𝒙∈X and substituting 𝑞 𝛼 (𝜇) = sup𝒙∈Φ 𝜇 ∩X 𝑄 𝐹𝒙 (1 − 𝛼) = − inf 𝒙∈Φ 𝜇 ∩X −𝑄 𝐹𝒙 (1 − 𝛼), we obtain: inf /sup 𝜇 𝜇 𝜇 st 𝜇∈R (2.77) h inf 𝜑(𝒙1 )=𝜇,𝒙 1 ∈X, 𝜑(𝒙2 )=𝜇,𝒙 2 ∈X i −2ℓ𝒙1 ( 𝒚) − 𝑄 𝐹𝒙2 (1 − 𝛼) ≤ inf −2ℓ𝒙 (𝒚), 𝒙∈X which can be transformed to parameter space as: inf / sup 𝒙1 ,𝒙2 𝒙1 ,𝒙2 st 𝜑(𝒙1 ) 𝒙1 , 𝒙2 ∈ X (2.78) 𝜑(𝒙1 ) = 𝜑(𝒙2 ) − 2ℓ𝒙1 (𝒚) − 𝑄 𝐹𝒙2 (1 − 𝛼) ≤ inf −2ℓ𝒙 (𝒚). 𝒙∈X Further unpacking 𝑄 𝐹𝒙2 (1−𝛼) as in Lemma 2.3.1, we obtain the chance-constrained optimization problem: inf / sup 𝒙 1 ,𝒙 2 ,𝑞 𝒙 1 ,𝒙 2 ,𝑞 st 𝜑(𝒙1 ) 𝒙1 , 𝒙2 ∈ X 𝜑(𝒙1 ) = 𝜑(𝒙 2 ) − 2ℓ𝒙1 (𝒚) ≤ inf −2ℓ𝒙 (𝒚) + 𝑞 𝒙∈X P𝑢∼U ( [0,1]) (F (𝒙2 , 𝑢) ≤ 𝑞) ≤ 1 − 𝛼, where F (𝒙, 𝑢) = 𝐹𝒙−1 (𝑢). (2.79) 58 Proof of Example 2.3.3 Similar to Example 2.2.6 in Section 2.7, this proof is divided into two cases. Case of 𝜇∗ = 0 Since the case when 𝜇∗ = 0 is of particular interest, we show the result in this specific case and then generalize. Thus, when 𝜇0 = 0, the log-likelihood ratio has the following distribution: ℓ0 ∼ 1 1 2 + 𝜒 . 2 2 1 (2.80) Additionally, this distribution implies the following stochastic dominance: P 𝜇0 (ℓ0 ≤ 𝑐) =  1 1 + 𝜒12 (𝑐) ≥ 𝜒12 (𝑐), 2 (2.81) that is, the log-likelihood ratio CDF is stochastically dominated by the chi-squared with one degree of freedom distribution. This means that the type-I error of the test can be controlled at the 𝛼 level. When 𝜇 > 0, the closed-form solution to the CDF of ℓ0 becomes more complicated, as we can no longer use symmetry around the origin. From the result of Example 2.2.6, we have the following CDF: ( !) 2−𝑐 −𝜇 √ 0 · 1{𝑐 ≥ 𝜇02 }. (2.82) P 𝜇0 (ℓ0 ≤ 𝑐) = 𝜒12 (𝑐) · 1{𝑐 < 𝜇02 } + Φ( 𝑐) − Φ 2𝜇0 Note, a quick check of (2.82) when 𝜇0 = 0 reveals agreement with (2.81) such that √ √ √
1 1 1 1 1 1 P 𝜇0 (ℓ0 ≤ 𝑐) = Φ( 𝑐) = Φ( 𝑐) − + = 2Φ( 𝑐) − 1 + = 𝜒12 (𝑐) + . 2 2 2 2 2 2 (2.83) This completes the case of 𝜇∗ = 0. Case of 𝜇∗ > 0 We already demonstrated above the chi-squared with one degree of freedom dominates the log-likelihood ratio when 𝜇0 = 0. We now show that the dominance holds when 𝜇0 > 0. Clearly, when 𝑐 < 𝜇02 , P 𝜇0 (ℓ0 ≤ 0) = 𝜒12 (𝑐), making it in fact equal to the chi-squared with one degree of freedom. Suppose 𝑐 ≥ 𝜇02 . Define ! −𝜇02 − 𝑐 √ ℎ(𝑐) := Φ( 𝑐) − Φ − 𝜒12 (𝑐). 2𝜇0 The stochastic dominance occurs if and only if ℎ(𝑐) ≥ 0 for all 𝑐 ≥ 𝜇02 . 59   √ √ √ −𝜇2 −𝑐 Note first that 𝜒12 (𝑐) = Φ( 𝑐) −Φ(− 𝑐) and therefore ℎ(𝑐) = Φ(− 𝑐) −Φ 2𝜇0 0 . Since Φ(·) is a monotonically increasing function, it is sufficient to show that √ −𝜇2 −𝑐 − 𝑐 − 2𝜇0 0 ≥ 0 for all 𝑐 ≥ 𝜇02 . We do so below. Define a function 𝑓 as follows: −𝜇02 − 𝑐 √ 𝑓 (𝑐) = − 𝑐 − . 2𝜇0 Observe that when 𝑐 = 𝜇02 , 𝑓 (𝑐) = 0. Consider when 𝑐 > 𝜇02 . We obtain the following first and second derivatives: √ −𝜇0 + 𝑐 1 ′ 𝑓 (𝑐) = and 𝑓 ′′ (𝑐) = 𝑐−3/2 . √ 4 2𝜇0 𝑐 √ By the constraint 𝑐 > 𝜇02 , it follows that −𝜇0 + 𝑐 > 0, and therefore, 𝑓 ′ (𝑐) > 0 for all 𝑐 > 𝜇02 . Additionally, 𝑓 ′′ (𝑐) > 0 for all 𝑐 > 𝜇02 , so 𝑓 is convex. Hence, we conclude that 𝑓 is a monotonically increasing function for 𝑐 > 𝜇02 , which starts at 0 when 𝑐 = 𝜇02 , and thus 𝑓 (𝑐) ≥ 0 for all 𝑐 ≥ 𝜇02 . It therefore follows that −𝜇02 − 𝑐 √ Φ(− 𝑐) ≥ Φ 2𝜇0 ! , and hence ℎ(𝑐) ≥ 0 for all 𝑐 ≥ 𝜇02 . As such, we conclude that P 𝜇0 (ℓ0 ≤ 𝑐) ≥ 𝜒12 (𝑐) for all 𝑐 ≥ 0. In other words, that the sampling distribution for the log-likelihood ratio is stochastically dominated by a chi-squared distribution with one degree of freedom. This completes the case of 𝜇∗ > 0. 2.9 Proofs in Section 2.4 Proof of Theorem 2.4.1 The proof follows by combining Lemmas 2.4.2 and 2.4.3. Proof of Lemma 2.4.2 The proof follows by direct inspection and substitution of 𝑞 𝛼 and −2ℓ𝒙 ( 𝒚) = ∥ 𝒚 − 𝑲𝒙∥ 22 . The interval has the coverage if the 𝑞 𝛼 is valid by Theorem 2.2.4 and only if by Proposition 2.2.5. Proof of Lemma 2.4.3 The proof follows by observing 𝑧2𝛼/2 = 𝑄 𝜒2 (1 − 𝛼) and applying Lemma 2.3.2 1 60 Proof of Lemma 2.4.4 We argue by coupling. Note that 12 (𝑦 1 − 𝑦 2 ) 2 ∼ 𝜒12 , so that it suffices to show 𝜆 ≤ 12 (𝑦 1 − 𝑦 2 ) 2 for every 𝑦 to constitute a valid coupling that proves stochastic dominance. This is clearly true when 𝑦 1 + 𝑦 2 ≥ 0, since 𝜆 − 12 (𝑦 1 − 𝑦 2 ) 2 is equal to non-positive terms only. When 𝑦 1 + 𝑦 2 < 0, if both are strictly negative, then 𝜆 = 0 ≤ 21 (𝑦 1 − 𝑦 2 ) 2 . Then assume without loss of generality that 𝑦 1 is non-negative, then 𝑦 2 has to be negative. Then 𝜆 = 𝑦 21 , but 𝑦 1 ≥ 0, 𝑦 2 < 0 and 𝑦 1 < −𝑦 2 imply that √ |𝑦 1 − 𝑦 2 | = 𝑦 1 − 𝑦 2 ≥ 2𝑦 1 ≥ 2𝑦 1 , squaring both sides gives 12 (𝑦 1 − 𝑦 2 ) 2 < 𝑦 21 = 𝜆. This finishes the proof. Proof of Lemma 2.4.5 Consider the LLR 𝜆(𝜇∗ = −1, 𝒚) = min 𝑥 1 +𝑥 2 −𝑥3 =−1 𝒙≥0 ∥𝒙 − 𝒚∥ 22 − min ∥𝒙 − 𝒚∥ 22 . (2.84) 𝒙≥0 The goal of this proof is to show that 𝜒12 does not stochastically dominate (2.84) when 𝒚 ∼ N (𝑥 ∗ = (0, 0, 1), 𝑰3 ). By Corollary 4.26 in (Roch, 2024), 𝑋 ⪰ 𝑌 implies E[𝑥] > E[𝑦], so it suffices to show that E[𝜆(𝜇∗ = −1, 𝒚)] > E[ 𝜒12 ] = 1 to complete the proof. Observe that  ∗ E[𝜆(𝜇 = −1, 𝒚)] = E min 𝒉⊤ 𝒙=−1 𝒙≥0 ∥𝒙 − 𝒚∥ 22   −E min ∥𝒙 − 𝒚∥ 22 𝒙≥0  . (2.85) We begin by computing the second term. Since min ∥𝒙 − 𝒚∥ 22 = 3 ∑︁ 𝒙≥0 (𝑦𝑖 − max{𝑦𝑖 , 0}) 2 , 𝑖=1 we have   ∑︁   3 2 2 E min ∥𝒙 − 𝒚∥ 2 = E (𝑦𝑖 − max{𝑦𝑖 , 0}) 𝒙≥0 𝑖=1  = 2E𝑧∼N (0,1) (𝑧 − max{𝑧, 0}) 2   2  + E𝑧∼N (1,1) (𝑧 − max{𝑧, 0}) . Let 𝑔(𝑧) := (𝑧 − max{𝑧, 0}) 2 . Using in both cases, we obtain E[𝑔(𝑧)] = E[𝑔(𝑧) | 𝑧 ≥ 0] · P(𝑧 ≥ 0) + E[𝑔(𝑧) | 𝑧 < 0] · P(𝑧 < 0) = E[𝑧2 | 𝑧 < 0] · P(𝑧 < 0). | {z } 0 61 Note that E[𝑧2 | 𝑧 < 0] = (E[𝑧 | 𝑧 < 0]) 2 + Var[𝑧 | 𝑧 < 0]  2   2    𝜙(0) 𝜙(0)    − Φ(0) + 1 − Φ(0) , 𝑧 ∼ N (0, 1)  =  2  2   𝜙(−1) 𝜙(−1) 𝜙(−1)  1 − − + 1 +  Φ(−1) Φ(−1) Φ(−1) , 𝑧 ∼ N (1, 1)      1, 𝑧 ∼ N (0, 1)  = 𝜙(−1)   2 − Φ(−1) , 𝑧 ∼ N (1, 1),  where we used the formulas for mean and variance of a truncated Gaussian. Finally,   2 E min ∥𝒙 − 𝒚∥ 2 = 2 · 1/2 · 1 + (2 − 𝜙(−1)/Φ(−1)) · (Φ(−1)) 𝒙≥0 = 1 + 2Φ(−1) − 𝜙(−1) ≈ 1.0753. It suffices to prove that   2 E ⊤min ∥𝒙 − 𝒚∥ 2 > 2 + 2Φ(−1) − 𝜙(−1) ≈ 2.0753. 𝒉 𝒙=−1 𝒙≥0 We will prove that  E min 𝒉⊤ 𝒙=−1 𝒙≥0 ∥𝒙 − 𝒚∥ 22  = 13/6 ≈ 2.166. Note that the intersection of the plane 𝒉⊤ 𝒙 = 𝑥 1 + 𝑥 2 − 𝑥3 = −1 and 𝒙 ≥ 0 is the parametric surface S = {(𝑢, 𝑣, 𝑢 + 𝑣 + 1), 𝑢 ≥ 0, 𝑣 ≥ 0}, so we can write min ∥𝒙 − 𝒚∥ 22 = min (𝑦 1 − 𝑢) 2 + (𝑦 2 − 𝑣) 2 + (𝑦 3 − 𝑢 − 𝑣 − 1) 2 . 𝒙∈S (2.86) 𝑢≥0,𝑣≥0 It is convenient to define a new variable 𝑧3 = 1 − 𝑦 3 ∼ N (0, 1), so that (𝑦 1 , 𝑦 2 , 𝑧3 ) is sampled from a standard three dimensional Gaussian. Abusing notation we will still write 𝑦 3 for 𝑧3 and then 𝒚 ∼ N ((0, 0, 0), 𝑰). The optimization problem becomes: min (𝑦 1 − 𝑢) 2 + (𝑦 2 − 𝑣) 2 + (−𝑦 3 − 𝑢 − 𝑣) 2 . 𝑢≥0,𝑣≥0 (2.87) 62 This can be explicitly solved to yield   𝑦 21 + 𝑦 22 + 𝑦 23        1 2 + 2𝑦 𝑦 + 2𝑦 2 + 𝑦 2  𝑦  1 3  2 1 2 3      2 1 2 + 𝑦 2 + 2𝑦 𝑦 + 𝑦 2 min ∥𝒙 − 𝒚∥ = 2𝑦 2 3 2 2 1 2 3  𝒉⊤ 𝒙=−1   𝒙≥0     1  (𝑦 1 + 𝑦 2 + 𝑦 3 ) 2   3  𝑦 1 − 𝑦 3 ≤ 0 and 𝑦 2 − 𝑦 3 ≤ 0 𝑦 1 − 𝑦 3 ≥ 0 and 𝑦 1 − 2𝑦 2 + 𝑦 3 ≥ 0 𝑦 2 − 𝑦 3 ≥ 0 and 2𝑦 1 − 𝑦 2 − 𝑦 3 ≤ 0    2𝑦 1 − 𝑦 3 ≥ 𝑦 2 ≥ max{𝑦 1 , 𝑦 3 }  .   2𝑦 2 − 𝑦 3 ≥ 𝑦 1 ≥ max{𝑦 2 , 𝑦 3 }  (2.88) We split ∫ min ∥𝒙 − 𝒚∥ 22 𝜙(𝑦 1 )𝜙(𝑦 2 )𝜙(𝑦 3 ) d𝑦 ⊤ R3 𝒉 𝒙=−1 𝒙≥0 into the different domains given by (2.88), with the value of the expectation being equal to the sum of the different integrals, which we proceed to compute. Region 1: 1 2 2 2 𝑒 − 2 (𝑦1 +𝑦2 +𝑦3 ) 2 𝐼1 = (𝑦 1 + 𝑦 22 + 𝑦 23 ) d𝑦. √ 3/2 𝑦 3 ≥𝑦 1 ,𝑦 3 ≥𝑦 2 2 2𝜋 ∫ Note that by symmetry of the variables in the integrand, we have 1 2 2 2 𝑒 − 2 (𝑦1 +𝑦2 +𝑦3 ) 2 𝐼1 = (𝑦 1 + 𝑦 22 + 𝑦 23 ) d𝑦 √ 3/2 𝑦 2 ≥𝑦 1 ,𝑦 2 ≥𝑦 3 2 2𝜋 ∫ − 12 (𝑦 21 +𝑦 22 +𝑦 23 ) 𝑒 = (𝑦 21 + 𝑦 22 + 𝑦 23 ) d𝑦. √ 3/2 𝑦 1 ≥𝑦 3 ,𝑦 1 ≥𝑦 2 2 2𝜋 ∫ And since one of the 𝑦𝑖 will always be the largest one, the sum of the domains is R3 (modulo measure zero intersections that do not affect integration) and we can write 1 𝐼1 = 3 1 2 2 2 𝑒 − 2 (𝑦1 +𝑦2 +𝑦3 ) 2 1 (𝑦 1 + 𝑦 22 + 𝑦 23 ) d𝑦 = · 3 = 1. √ 3 R3 2 2𝜋 3/2 ∫ Here we used that the integral is the expected value of 𝑦 21 + 𝑦 22 + 𝑦 23 , which is 3 since the 𝑦𝑖 are centered with unit variance. Region 2: 1 2 2 2  𝑒 − 2 (𝑦1 +𝑦2 +𝑦3 )  2 2 2 𝑦 + 2𝑦 𝑦 + 2𝑦 + 𝑦 𝐼2 = √ 1 3 1 2 3 d𝑦. 𝑦 1 −𝑦 3 ≥0,𝑦 1 −2𝑦 2 +𝑦 3 ≥0 4 2𝜋 3/2 ∫ (2.89) 63 Partition R3 in four spaces with measure zero intersection, and we aim to argue that the integral of the integrand in (2.89) has the same value when integrating over any of them: n 𝑦1 + 𝑦3 o 𝐴 := 𝒚 : 𝑦 1 ≥ 𝑦 3 , 𝑦 2 ≥ 2 n 𝑦1 + 𝑦3 o 𝐵 := 𝒚 : 𝑦 1 ≥ 𝑦 3 , 𝑦 2 ≤ 2 n 𝑦1 + 𝑦3 o 𝐶 := 𝒚 : 𝑦 1 ≤ 𝑦 3 , 𝑦 2 ≥ 2 n 𝑦1 + 𝑦3 o 𝐷 := 𝒚 : 𝑦 1 ≤ 𝑦 3 , 𝑦 2 ≤ . 2 ∫ Clearly 𝐼2 = 𝐼 𝐵 = 𝐴 𝒉(𝑦 1 , 𝑦 2 , 𝑦 3 ) d𝑦. Since 𝒉 satisfies 𝒉(𝑥 1 , 𝑥2 , 𝑥3 ) = 𝒉(𝑥3 , 𝑥2 , 𝑥1 ), we can exchange 𝑦 1 and 𝑦 3 in the definitions of the sets, so 𝐼 𝐴 = 𝐼𝐶 and 𝐼 𝐵 = 𝐼 𝐷 . And since 𝒉 is even with respect to 𝑥 2 and odd with respect to 𝑥 1 , 𝑥3 we can exchange 𝑦𝑖 to −𝑦𝑖 for 𝑖 = 1, 2, 3 without the result changing. This flips both inequalities, proving 𝐼 𝐴 = 𝐼 𝐷 and 𝐼 𝐵 = 𝐼𝐶 . We therefore have 1 𝐼2 = 4 1 2 2 2  1 1 𝑒 − 2 (𝑦1 +𝑦2 +𝑦3 )  2 𝑦 1 + 2𝑦 1 𝑦 3 + 2𝑦 22 + 𝑦 23 d𝑦 = · 2 = . √ 4 2 R3 4 2𝜋 3/2 ∫ Here, in the integral, we factor out the sum and using that, the expected value of 𝑦𝑖 𝑦 𝑗 is 𝛿𝑖 𝑗 . Region 3: 1 2 2 2  𝑒 − 2 (𝑦1 +𝑦2 +𝑦3 )  2 2 2 𝐼3 = 2𝑦 1 + 2𝑦 2 𝑦 3 + 𝑦 2 + 𝑦 3 d𝑦. √ 𝑦 2 −𝑦 3 ≥0,𝑦 2 −2𝑦 1 +𝑦 3 ≥0 4 2𝜋 3/2 ∫ 1 This is exactly the same integral as 𝐼2 by switching 𝑦 2 with 𝑦 1 , so 𝐼3 = . 2 Region 4: 1 ∫ 𝐼4 = 2𝑦 1 −𝑦 3 ≥𝑦 2 ≥max(𝑦 1 ,𝑦 3 ) 2 2 2 𝑒 − 2 (𝑦1 +𝑦2 +𝑦3 ) (𝑦 1 + 𝑦 2 + 𝑦 3 ) 2 d𝑦 √ 3/2 6 2𝜋 ∫ + 2𝑦 2 −𝑦 3 ≥𝑦 1 ≥max(𝑦 2 ,𝑦 3 ) 1 2 2 2 𝑒 − 2 (𝑦1 +𝑦2 +𝑦3 ) (𝑦 1 + 𝑦 2 + 𝑦 3 ) 2 d𝑦. √ 3/2 6 2𝜋 (2.90) We partition R3 in 12 subspaces with measure 0 intersection and we aim to argue that the integral of the integrand in (2.90) (considering one of the integrals only) has the 64 same value when integrating over any of them. For 𝜎 a permutation of (𝑦 1 , 𝑦 2 , 𝑦 3 ), we define the first six subsets as: {𝒚 : 2𝑦 𝜎(1) − 𝑦 𝜎(2) ≥ 𝑦 𝜎(3) ≥ max{𝑦 𝜎(1) , 𝑦 𝜎(2) }}, and the last six subsets as: {𝒚 : 2𝑦 𝜎(1) − 𝑦 𝜎(2) ≤ 𝑦 𝜎(3) ≤ min{𝑦 𝜎(1) , 𝑦 𝜎(2) }}. We need to prove that the integral has the same value in any of the 12 subsets. Since that the integrand 1 2 2 2 𝑒 − 2 (𝑦1 +𝑦2 +𝑦3 ) 𝒉(𝑦 1 , 𝑦 2 , 𝑦 3 ) := (𝑦 1 + 𝑦 2 + 𝑦 3 ) 2 √ 3/2 6 2𝜋 satisfies 𝒉(𝑦 1 , 𝑦 2 , 𝑦 3 ) = 𝒉(𝑦 𝜎(1) , 𝑦 𝜎(2) , 𝑦 𝜎(3) ) for all permutations 𝜎, the value of the integral in between the first and second groups of 6 subsets is the same. For a fixed 𝜎 (say, the identity), since 𝒉(𝑦 1 , 𝑦 2 , 𝑦 3 ) = 𝒉(−𝑦 1 , −𝑦 2 , −𝑦 3 ), the value over {𝒚 : 2𝑦 1 − 𝑦 2 ≥ 𝑦 3 ≥ max{𝑦 1 , 𝑦 2 }} is the same as the value over {𝒚 : −2𝑦 1 + 𝑦 2 ≥ −𝑦 3 ≥ max{−𝑦 1 , −𝑦 2 }} = {𝒚 : 2𝑦 1 − 𝑦 2 ≤ 𝑦 3 ≤ min{𝑦 1 , 𝑦 2 }}, so the value over the 12 sets is complete. It remains to be seen that for a generic 𝒚 = (𝑦 1 , 𝑦 2 , 𝑦 3 ), 𝑦 1 ≠ 𝑦 2 ≠ 𝑦 3 ≠ 𝑦 1 (which can be assumed with probability 1 without affecting the integral), the point belongs to one and just one of the sets. Assume without loss of generality that 𝑦 1 is the greater of the three and 𝑦 3 is the smallest. Then since 𝑦 1 > max{𝑦 2 , 𝑦 3 } and 𝑦 3 < min{𝑦 1 , 𝑦 2 } the only subsets that 𝒚 can belong to are: 𝐴 := {𝒚 : 2𝑦 2 − 𝑦 3 ≥ 𝑦 1 ≥ max{𝑦 2 , 𝑦 3 }} 𝐵 := {𝒚 : 2𝑦 3 − 𝑦 2 ≥ 𝑦 1 ≥ max{𝑦 2 , 𝑦 3 }} 𝐶 := {𝒚 : 2𝑦 1 − 𝑦 2 ≤ 𝑦 3 ≤ min{𝑦 1 , 𝑦 2 }} 𝐷 := {𝒚 : 2𝑦 2 − 𝑦 1 ≤ 𝑦 3 ≤ min{𝑦 1 , 𝑦 2 }}. 2 But 𝒚 is not in 𝐵 because that would require 𝑦 3 ≥ 𝑦1 +𝑦 2 but 𝑦 3 < 𝑦 1 and 𝑦 3 < 𝑦 2 , and 3 it is also not in 𝐶 because that would require 𝑦 1 ≤ 𝑦2 +𝑦 2 and 𝑦 1 > 𝑦 2 and 𝑦 1 > 𝑦 3 . 𝑦 1 +𝑦 3 3 𝒚 will be in 𝐴 if 𝑦 2 > 𝑦1 +𝑦 2 and in 𝐷 if, on the contrary, 𝑦 2 < 2 , both of which 65 are possible, but not at the same time. We conclude by identifying 𝐼4 as the sum of two integrals over subsets that we have defined, and therefore 2 𝐼4 = 12 1 2 2 2 𝑒 − 2 (𝑦1 +𝑦2 +𝑦3 ) 1 1 (𝑦 1 + 𝑦 2 + 𝑦 3 ) 2 d𝑦 = · 1 = . √ 3/2 6 6 R3 6 2𝜋 ∫ Here we expand the sum and use again that the expected value of 𝑦𝑖 𝑦 𝑗 is 𝛿𝑖 𝑗 . The proof concludes by adding up   13 2 E ⊤min ∥𝒙 − 𝒚∥ 2 = 𝐼1 + 𝐼2 + 𝐼3 + 𝐼4 = . 6 𝒉 𝒙=−1 𝒙≥0 Proof of Proposition 2.4.6 We construct a series of counterexamples, indexed by the dimension 𝑝, and prove that as 𝑝 → ∞, the expected value of the LLR diverges. Since stochastic dominance implies inequality of expectations (when expectations are finite), we conclude that the distribution can not be stochastically dominated. For all 𝑝 ∈ N, consider the example in R 𝑝 (= R𝑚 ), 𝑲 = 𝑰 𝑝 , 𝒙 ∗ = (0, . . . , 0, 1), 𝒉 = (1, . . . , 1, −1) (such that 𝜇∗ = −1). Let 𝜆 𝑛 (𝜇∗ = −1, 𝒚) = Í min 𝑝−1 𝑥 −𝑥 𝑝 =−1 𝑖=1 𝑖 ∥𝒙 − 𝒚∥ 22 − min ∥𝒙 − 𝒚∥ 22 . (2.91) 𝒙≥0 𝒙≥0 And compute  E 𝒚∼N (𝒙 ∗ ,𝑰𝑛 ) [𝜆 𝑛 (−1, 𝒚)] = E 𝒚∼N (𝒙 ∗ ,𝑰𝑛 )  min Í 𝑝−1 𝑖=1 𝑥𝑖 −𝑥 𝑝 =−1 𝒙≥0 ∥𝒙−𝒚∥ 22 −E 𝒚∼N (𝒙 ∗ ,𝑰𝑛 )  min ∥𝒙−𝒚∥ 22 𝒙≥0  (2.92) For the second term, we have   ∑︁   𝑝 2 2 E min ∥𝒙 − 𝒚∥ 2 = E (𝑦𝑖 − max{𝑦𝑖 , 0}) 𝒙≥0 𝑖=1  = ( 𝑝 − 1)E𝑧∼N (0,1) (𝑧 − max{𝑧, 0}) 2   + E𝑧∼N (1,1) (𝑧 − max{𝑧, 0}) 1 = ( 𝑝 − 1) + (2 − 𝜙(−1)/Φ(−1)) · (Φ(−1)), 2 using similar arguments as the proof in Section 2.9. We will lower bound the first term using duality. For simplicity, define 𝒛 = (𝑦 1 , . . . , 𝑦 𝑝−1 , 𝑦 𝑛 − 1) ∼ N (0, 𝑰𝑛 ), and equivalently optimize min ∥𝒙 − 𝒛∥ 22 , (2.93) Í 𝑝−1 𝑥 =𝑥 𝑝 𝑖=1 𝑖 𝒙≥0 2  . 66 Í 𝑝−1 𝒙 by 𝒙, where we defined the feasible e 𝒙 = (𝑥1 , . . . , 𝑥 𝑛 − 1 = 𝑖=1 𝑥𝑖 ) and replaced e abusing notation. Using Fenchel duality, we have that min Í 𝑝−1 𝑥 =𝑥 𝑝 𝑖=1 𝑖 𝒙≥0 ∥𝒙 − 𝒛∥ 22 ≥ sup (− 𝑓 ∗ (𝝃) − 𝑔 ∗ (−𝝃)), (2.94) 𝝃∈R 𝑝 where we have noted by 𝑓 ∗ the convex conjugate of 𝑓 (𝒙) := ∥𝒙 − 𝒛∥ 22 and, letting 𝑆 be the feasible set, we denoted by 𝑔 ∗ the convex conjugate of   0  𝑔(𝒙) =  ∞  if 𝒙 ∈ 𝑆 (2.95) if 𝒙 ∉ 𝑆. Note that with these definitions, minÍ 𝑝−1 𝑥𝑖 =𝑥 𝑝 ∥𝒙 − 𝒛∥ 22 = inf 𝒙 ( 𝑓 (𝑥) + 𝑔(𝑥)) so the 𝑖=1 𝒙≥0 weak Fenchel duality applies. We compute 𝑓 ∗ (𝝃) = 14 ∥𝝃 ∥ 22 + 𝒛⊤ 𝝃 − 𝒛 ⊤ 𝒛, and   0  ∗ 𝑔 (𝝃) =  ∞  if 𝜉𝑖 + 𝜉 𝑝 ≤ 0 for 𝑖 ∈ [ 𝑝 − 1] (2.96) otherwise, so that  1 sup (− 𝑓 (𝝃) − 𝑔 (−𝝃)) = sup − ∥𝝃 ∥ 22 − 𝒛⊤ 𝝃 + 𝒛⊤ 𝒛 4 𝝃∈R 𝑝 𝜉𝑖 +𝜉 𝑛 ≥0, for all 𝑖∈[ 𝑝−1]   1 2 ⊤ ≥ sup − ∥𝝃 ∥ 2 − 𝒛 𝝃 . 4 𝜉𝑖 +𝜉 𝑛 ≥0, for all 𝑖∈[ 𝑝−1] ∗ ∗  (2.97) (2.98) Since the supremum is lower bounded by any feasible point, we can further bound by picking a feasible 𝝃 ∗ for each possible 𝒛. We define the following:    −𝒛  𝝃 (𝒛) =   (−𝑧 1 , . . . , −𝑧 𝑝−1 , max𝑖∈[ 𝑝−1] 𝑧𝑖 )  ∗ if − 𝒛 is feasible (−𝑧𝑖 ≥ 𝑧 𝑛 for all 𝑖) otherwise. (2.99) Observe that min Í 𝑝−1 𝑥 𝑖 =𝑥 𝑝 𝒙≥0 ∥𝒙 − 𝒛∥ 22 𝑖=1 1 ≥ − ∥𝝃 ∗ (𝒛)∥ 22 − 𝒛⊤ 𝝃 ∗ (𝒛) 4    34 ∥𝒛∥ 2   2 = Í 𝑝−1 2   34 𝑖=1 𝑧𝑖 + 𝑧 𝑛 max𝑖∈[ 𝑝−1] 𝑧𝑖 − 41 max𝑖∈[ 𝑝−1] 𝑧𝑖  if − 𝒛 is feasible (−𝑧𝑖 ≥ 𝑧 𝑛 for all 𝑖) otherwise. 67 We note that −𝒛 is feasible with probability 1/𝑝, by symmetry. Taking expected value over the inequality and using the law of total expectation yields   2 E Í min ∥𝒙 − 𝒛∥ 2 𝑝−1 𝑥 =𝑥 𝑝 𝑖=1 𝑖 𝒙≥0     ∑︁  𝑝−1  2  1  3 1 3  2 𝑝 − 1 2 𝑧 + E 𝑧 𝑛 max 𝑧𝑖 − E max 𝑧𝑖 E ≥ × E ∥𝑧∥ 2 + 𝑝 4 𝑝 4 𝑖=1 𝑖 4 𝑖∈[ 𝑝−1]) 𝑖∈[ 𝑝−1]     2 3 𝑝 − 1 3( 𝑝 − 1) 1 = + + 0 − E max 𝑧𝑖 . 4 𝑝 4 4 𝑖∈[ 𝑝−1] To bound the last term, we use      √︁ 2 E max 𝑧𝑖 = E max 𝑧𝑖 + Var max 𝑧𝑖 ≤ 2 log( 𝑝 − 1) + 1, 𝑖∈[ 𝑝−1] 𝑖∈[ 𝑝−1] 𝑖∈[ 𝑝−1] where the moment bounds are standard results: the expectation bound can be found √︁ using Jensen’s inequality on exp( 2 log 𝑝 max𝑖 𝑧𝑖 ) and then bounding max𝑖 𝑧𝑖 ≤ Í 𝑖 𝑧𝑖 , and the variance bound with Poincaré’s inequality applied to a smooth maximum, even though it can be refined (Boucheron and M. Thomas, 2012). Putting everything together, we obtain   𝑝 𝑝 − 1 3( 𝑝 − 1) 1 √︁ − 2 log( 𝑝 − 1) − + O (1), E 𝒚∼N (𝒙 ∗ ,𝑰𝑛 ) [𝜆 𝑛 (−1, 𝒚)] ≥ 𝑝 4 4 2 which is O ( 𝑝) and therefore tends to ∞ as 𝑝 → ∞. This completes the proof. 2.10 Additional numerical illustrations in Section 2.5 Constrained Gaussian in three dimensions We include the analog of Figure 2.9 with 1 − 𝛼 = 0.95 in Figure 2.12. 68 Figure 2.12: Estimated interval coverage (left) and expected lengths (right) for 95% intervals resulting from the SSB, OSB, MQ, and MQ𝜇 methods for the Gaussian linear model in (2.4) with 𝑲 = 𝑰3 , 𝜑(𝒙) = 𝒉⊤ 𝒙 = 𝑥 1 + 𝑥2 − 𝑥 3 , and X = {𝒙 ∈ R3 : 𝒙 ≥ 0}. 69 Chapter 3 CONFIDENCE INTERVALS FOR FUNCTIONALS IN CONSTRAINED INVERSE PROBLEMS VIA DATA-ADAPTIVE SAMPLING-BASED CALIBRATION We address functional uncertainty quantification for ill-posed inverse problems where it is possible to evaluate a possibly rank-deficient forward model, the observation noise distribution is known, and there are known parameter constraints. We present four constraint-aware confidence interval constructions extending the theoretical test inversion framework in (Batlle, Stanley, et al., 2023) by making the intervals both computationally feasible and less conservative. Our approach first shrinks the potentially unbounded constraint set compact in a data-adaptive way, obtains samples of the relevant test statistic inside this set to estimate a quantile function, and then uses these computed quantities to produce the constraint-aware confidence intervals. Our approach to bounding the constraint set in a data-adaptive way is based on the approach by (Roger L. Berger and Boos, 1994), and involves defining a subset of the constraint set where the true parameter is guaranteed to exist with high probability. The probabilistic guarantee of this compact subset is then incorporated into the final coverage guarantee in the form of an uncertainty budget. We then propose custom sampling algorithms to efficiently sample from this subset, even when the parameter space is high-dimensional. Optimization-based interval methods formulate confidence interval computation as two endpoint optimizations, where the optimization constraints can be set to achieve different types of interval calibration while seamlessly incorporating parameter constraints. However, choosing optimization constraints to obtain coverage for a single functional has been elusive. We show that all four proposed intervals achieve nominal coverage for a particular functional both theoretically and in practice, with several numerical examples demonstrating superior performance of our intervals over the OSB interval in terms of both coverage and expected interval length. In particular, we show the superior performance of our intervals in a realistic unfolding simulation from high-energy physics that is severely ill-posed and involves a rank-deficient forward model. 70 3.1 Introduction This paper proposes a novel uncertainty quantification (UQ) approach for ill-posed inverse problems characterized by a known parametric forward model1 𝑓 : R 𝑝 → R𝑛 mapping a parameter 𝒙 ∈ R 𝑝 to an observation 𝑓 (𝒙) ∈ R𝑛 , additive noise with a known distribution 𝜺 ∼ N (0, 𝚺), and constraints on the forward model parameters denoted by 𝒙 ∈ X, where X is primarily assumed to be of the form 𝑨𝒙 ≤ 𝒃. We consider scenarios where the parameter space dimension 𝑝 may exceed the observation space dimension 𝑛, which often results in an ill-posed problem. Additionally, we do not require the forward model 𝑓 to be injective, allowing for the possibility that 𝑓 is many-to-one. We assume observations are generated by 𝒚 = 𝑓 (𝒙 ∗ ) + 𝜺, where 𝒙 ∗ ∈ X is the true but unknown model parameter. Our UQ object of interest is a confidence interval2 , C𝛼 ( 𝒚), on a one-dimensional quantity of interest (QoI) derived from 𝑥 ∗ by 𝜑 : X → R. Our approach produces constraintaware intervals with a finite-sample frequentist coverage guarantee, namely, for each 𝛼 ∈ (0, 1) P(𝜑(𝒙 ∗ ) ∈ C𝛼 (𝒚)) ≥ 1 − 𝛼 for all 𝒙 ∗ ∈ X. (3.1) The probability in the coverage guarantee (3.1) is taken over the distribution of the observation 𝒚 = 𝑓 (𝒙 ∗ ) + 𝜺, where 𝜺 is the additive noise. The level 𝛼 ∈ (0, 1) corresponds to the desired coverage probability, meaning that the interval C𝛼 ( 𝒚) will contain the true value 𝜑(𝒙 ∗ ) with probability at least 1 − 𝛼 for any realization of the noise 𝜺. This paper builds on the confidence interval framework of (Batlle, Stanley, et al., 2023), which posits a more general data-generating process, 𝒚 ∼ 𝑃𝒙 ∗ , and proposes a novel test-inversion framework for constructing constraint-aware confidence intervals for inverse problem QoI’s. By proposing a testing framework to evaluate if 𝒙 ∗ ∈ Φ 𝜇 := {𝒙 ∈ X : 𝜑(𝒙) = 𝜇} via the test statistic 𝜆(𝜇, 𝒚), (Batlle, Stanley, et al., 2023) construct a confidence set as follows (see Equation (2.9)), C𝛼 (𝒚) := {𝜇 : 𝜆(𝜇, 𝒚) ≤ 𝑞 𝛼 (𝜇)}. (3.2) As shown in Lemma 2.2 of (Batlle, Stanley, et al., 2023), there are two ways to set 𝑞 𝛼 (𝜇) using a supremum over a particular quantile function, 𝑄 𝒙 (1 − 𝛼) (we show a particular realization of such a quantile function in the left-most panel of Figure 3.1), 1 Even though we write the forward model as a known function 𝑓 , we only require evaluation access to the map throughout the paper. 2 We interchangeably use the terms “confidence set” and “confidence interval” since the latter may be obtained by the former by simply retaining the set endpoints. 71 such that C𝛼 ( 𝒚) achieves the desired coverage, 𝑄 max 𝜇,1−𝛼 := sup 𝑄 𝒙 (1 − 𝛼), (3.3) 𝒙∈Φ 𝜇 ∩X max 𝑄 max 1−𝛼 := sup 𝑄 𝜇,1−𝛼 = sup𝑄 𝒙 (1 − 𝛼). 𝒙∈𝜑(X) (3.4) 𝒙∈X In practice, not only are Equations (3.3) and (3.4) difficult to compute because 𝑄 𝒙 (1− 𝛼) is typically only accessible via sampling, but the optimizations can be expressed equivalently as chance-constrained programs (see Lemma 3.1 in (Batlle, Stanley, et al., 2023)), which are typically challenging to solve. Furthermore, although 𝑄 max 𝜇,1−𝛼 Figure 3.1: (Left) A particular quantile surface, 𝑄 𝒙 (1 − 𝛼), where 𝒙 ≥ 0 and 𝛼 = 0.32. This surface was obtained via Monte Carlo sampling the LLR test statistic over a grid of 𝒙’s defined by the two-dimensional
⊤ constrained-Gaussian scenario similar to that in Section 3.6, but with 𝒉 := 1 1 . (Right-Top) An illustration of the Berger–Boos set and other, a 1−𝜂 confidence set for 𝒙 ∗ , which prevents having to contend with an unbounded parameter space. Additionally, it can eliminate “worstcase” parameter settings in X that are far from the data, thus potentially making the resulting intervals less conservative. (Right-Bottom) An illustration of an LLR test statistic curve, 𝜆(𝜇, 𝒚), over the QoI domain and the interval endpoints that result from using either Equation (3.5) for the “Sliced” interval or Equation (3.6) for the “Global” interval. and 𝑄 max 1−𝛼 are optimal with respect to the setup in which they are analyzed, they can still produce overly conservative (larger than necessary) confidence sets if the true 𝒙 ∗ is far away from the worst-case 𝒙 against which the above maximum quantiles protect. This paper addresses these practical and statistical challenges by including a bounded data-informed subset of the parameter space, B𝜂 with 𝜂 ∈ (0, 1), such 72 that P 𝒙 ∗ ∈ B𝜂 ≥ 1 − 𝜂 (see Section 3.3). We refer to B𝜂 as the “Berger–Boos” set due to its inspiration from (Roger L. Berger and Boos, 1994). We show that we can reformulate Equations (3.3) and (3.4) to be constrained over B𝜂 ⊂ X if we also use a slightly larger quantile level, 1−𝛾 for 𝛾 ∈ (𝛼, 1), and such that 1−𝛼 = (1−𝜂)(1−𝛾), producing the following maximum quantiles:
𝑞¯ 𝛾,𝜂 (𝜇) := sup 𝑄 𝒙 (1 − 𝛾), (3.5) 𝒙∈Φ 𝜇 ∩B 𝜂 𝑞¯ 𝛾,𝜂 := sup 𝑄 𝒙 (1 − 𝛾). (3.6) 𝒙∈B 𝜂 Since 𝑞¯ 𝛾,𝜂 (𝜇) and 𝑞¯ 𝛾,𝜂 are defined over bounded subsets of X, we can propose a novel sampling procedure to obtain design points covering B𝜂 and use quantile regression to estimate the desired quantile surface, 𝑄 𝒙 (1−𝛾). The estimated quantile surface is then used to produce estimates of either 𝑞¯ 𝛾,𝜂 (𝜇) or 𝑞¯ 𝛾,𝜂 and constructs a confidence set using a set definition similar to Equation (3.2), 𝐶𝛼 ( 𝒚; B𝜂 ) := {𝜇 : 𝜆(𝜇, 𝒚) ≤ 𝑞}, (3.7) where 𝑞 is set to one of 𝑞¯ 𝛾,𝜂 (𝜇) or 𝑞¯ 𝛾,𝜂 to obtain the desired interval. We present a variety of ways to perform the sampling and compare the different confidence set formulations and computations. Context and related work With advances in data collection and computational processing, high-dimensional ill-posed inverse problems have become more prevalent, especially in fields like remote sensing and data assimilation. This setting includes a wide array of physical science applications, spanning Earth science (Rodgers, 2000), atmospheric science and remote sensing (J. Liu, Bowman, Meemong, et al., 2016; Patil, Kuusela, and Hobbs, 2022), and high energy physics (Kuusela, 2016; Stanley, Patil, and Kuusela, 2022), among many others. Providing guarantees for UQ in parameter inference from indirect observations is essential for assessing the precision of scientific inferences made in these contexts. However, the inherent ill-posed nature of these problems often leads to inferences that are highly sensitive to noise, posing significant challenges for UQ. Namely, the ill-posedness leads to an identifiability issue in which statistical inference is impossible without providing some form of regularization, which usually takes either a deterministic (e.g., SVD truncation (Höcker and Kartvelishvili, 1996) and Tikhonov regularization (Schmitt, 2012)) or probabilistic (e.g., priors and Bayesian inference) form. Under some assumptions, these 73 two approaches are mathematically equivalent and are therefore subject to the same pitfalls, such as a disruption of the coverage guarantees of downstream intervals as a result of the incurred regularization bias as thoroughly discussed in (Kuusela, 2016). Our method’s focus on one-dimensional QoI’s and incorporation of parameter constraints allows for implicit regularization and therefore produces intervals with the promised coverage guarantee while avoiding the problem of regularization bias altogether. Although including parameter constraints provides implicit regularization and enables handling non-trivial null spaces, it shifts complexity to statistical inference under constraints, a non-trivial problem in even elementary settings (Gouriéroux, Holly, and Monfort, 1982; Wolak, 1987; Robertson, F. T. Wright, and Dykstra, 1988; Alexander Shapiro, 1988; Wolak, 1989; Molenberghs and Verbeke, 2007). One elegant solution to this problem originates in what we refer to as optimizationbased UQ, originating in the work of (W. R. Burrus, 1965) and (Burt W. Rust and Walter R. Burrus, 1972). (Philip B. Stark, 1992b) extended and generalized their approach, calling this method strict bounds, since it produced guaranteed simultaneous coverage interval estimators complying with known physical constraints on the model parameters. This collection of work defines endpoint optimization problems over the physically constrained parameter space to directly compute confidence intervals for one-dimensional QoI’s. Not only does the optimization form of the confidence interval computation shift statistical inference complexity to numerical optimization, but it also allows known physical constraints to be directly included in the endpoint optimizations. This practical advantage is dulled by the difficulty of proving the coverage properties of the intervals resulting from the defined endpoint optimizations in the one-at-a-time setting. Even under the relatively strong assumptions in (Burt W. Rust and Walter R. Burrus, 1972) of a linear forward model, non-negativity parameter constraints and linear QoI, the authors were only able to conjecture the coverage of their interval (known as the Burrus Conjecture). The interval coverage was unsuccessfully proven in (O’Leary and Bert W. Rust, 1986) (the error pointed out in (Tenorio, Fleck, and Moses, 2007)) and finally generally refuted in (Batlle, Stanley, et al., 2023). These optimized-based intervals that have gained recent attention take the form     2 𝑙 2 𝑢 2 I (𝜓𝛼 , 𝒚) := 𝜑 (𝜓𝛼 , 𝒚), 𝜑 (𝜓𝛼 , 𝒚) = min 𝜑(𝒙), max 𝜑(𝒙) , (3.8) 𝒙∈𝐷 (𝜓 2𝛼 ,𝒚) 𝒙∈𝐷 (𝜓 2𝛼 ,𝒚) where  𝐷 (𝜓𝛼2 , 𝒚) := 𝒙 ∈ X : ∥ 𝒚 − 𝑓 (𝒙)∥ 22 ≤ 𝜓𝛼2 . (3.9) 74 The statistical challenge for intervals of this form is the choosing of 𝜓𝛼2 such that I (𝜓𝛼2 , 𝒚) has the desired coverage guarantee. It is important to note that the definition of 𝐷 (𝜓𝛼2 , 𝒚) assumes that the noise has been standardized; in other words, the covariance matrix of the noise has been transformed to the identity matrix. The literature proposes two settings to guarantee coverage. One approach is to set 𝜓𝛼2 such that 𝐷 (𝜓𝛼2 , 𝒚) is itself a confidence set for 𝒙 ∗ in the parameter space. Since this choice would then automatically guarantee coverage for I (𝜓𝛼2 , 𝒚) regardless of the chosen QoI, this setting has been called “simultaneous” in (O’Leary and Bert W. Rust, 1986), and “Simultaneous Strict Bounds” (SSB) in (Stanley, Patil, and Kuusela, 2022; Batlle, Stanley, et al., 2023) since it also aligns with the setting of the strict bounds construction in (Philip B. Stark, 1992b). Under the Gaussian assumption, 2 2 , where 𝜒 2 is the upper 𝛼-quantile this can be achieved by setting 𝜓 𝑆𝑆𝐵,𝛼 := 𝜒𝑛,𝛼 𝑛,𝛼 for a chi-squared distribution with 𝑛 degrees of freedom. Although this approach is relatively simple and obtains the desired coverage guarantee, it is conservative since the guarantee holds for all possible QoI choices simultaneously. To tailor the interval to one particular QoI, there is the “one-at-a-time” setting as described in (Burt W. Rust and Walter R. Burrus, 1972; O’Leary and Bert W. Rust, 1986), or “one-at-a-time strict bounds” (OSB) as called in (Stanley, Patil, and Kuusela, 2022; Batlle, Stanley, et al., 2023). Under the Gaussian assumption this proposed 2 2 + 𝑠(𝒚) 2 , where 𝑠(𝒚) 2 := min 2 setting was 𝜓𝑂𝑆𝐵,𝛼 = 𝜒1,𝛼 𝒙∈X ∥ 𝒚 − 𝑓 (𝒙)∥ 2 . Unlike the 2 simultaneous setting, 𝐷 (𝜓𝑂𝑆𝐵,𝛼 , 𝒚) is not a 1 − 𝛼 confidence set for 𝒙 ∗ , which makes proving its coverage guarantee difficult. The validity of this claim was proposed by the Burrus conjecture (Burt W. Rust and Walter R. Burrus, 1972) and generally disproven by (Batlle, Stanley, et al., 2023). To address the challenge of calibrating strict bounds and optimization-based confidence intervals, (Batlle, Stanley, et al., 2023) approached these intervals as an inverted likelihood ratio test as shown in Equation (3.2). The distribution of the log-likelihood ratio (LLR) statistic is non-standard due to the presence of the constraints, adding to the complexity of controlling the type-I error of the test. This framework development allowed for the general disproving of the long-standing Burrus conjecture, invalidating the general calibration of the intervals proposed by (Burt W. Rust and Walter R. Burrus, 1972). However, the framework revealed that the characterization of 𝑄 𝒙 (1 − 𝛼) allowed for proper interval calibration via Equations (3.3) or (3.4) in much more general settings than those in which the Burrus conjecture applied (linear forward model, Gaussian noise, and linear functional). 75 Our approach and contributions To address the computational challenges and statistical conservatism following from the use of quantiles (3.3) or (3.4) in confidence set (3.2), this paper develops analogous maximum quantiles over the bounded Berger–Boos set (quantiles (3.5) and (3.6)) for the altered confidence set (3.7). We refer to the confidence intervals produced by quantile (3.5) as “Sliced”, since it considers the maximum quantile along slices of the Berger–Boos set as defined by level-sets of the QoI. We refer to the confidence intervals produced by quantile (3.6) as “Global”, since it considers the maximum quantile over the entire Berger–Boos set. For both the sliced and global forms of the confidence interval, we present and develop both optimization and sampling approaches. In total, these confidence interval construction options produce four different interval varieties. To validate the coverage guarantees of these interval varieties, we conduct a series of numerical experiments to demonstrate the effectiveness of the intervals in various scenarios. Since the OSB interval described in (Patil, Kuusela, and Hobbs, 2022) and (Stanley, Patil, and Kuusela, 2022) is the current standard approach in this setting, it is our primary point of comparison. First, we present low-dimensional examples. The first example is a two-dimensional constrained Gaussian noise model, a well-studied case in the literature (see (Tenorio, Fleck, and Moses, 2007), (Batlle, Stanley, et al., 2023)), illustrating how our intervals are competitive with the OSB interval in a scenario where the OSB interval is known to achieve nominal coverage. The second example is a three-dimensional constrained Gaussian noise model, illustrating the advantage of our method in achieving nominal coverage where the OSB interval fails. Second, we apply our method to a simulated version of particle unfolding from high-energy physics (see (Stanley, Patil, and Kuusela, 2022)), a binned deconvolution problem, in an 80-dimensional parameter space with a rankdeficient linear forward model. We use the previously studied realistic setting from (Stanley, Patil, and Kuusela, 2022) where the OSB interval is known to provide coverage to show that our Sliced intervals substantially outperform OSB in terms of expected length. Then, we use a parameter setting where the OSB interval does not achieve nominal coverage, but our intervals do, with the Sliced intervals still out-performing OSB in terms of expected length. This application highlights the practical significance of our approach, with our confidence intervals consistently achieving nominal coverage and often outperforming existing methods in terms of expected interval length. 76 The contributions of this work are both methodological and computational. Methodologically, we propose new confidence interval constructions for the setting described at the beginning of the section. Although the use of the Berger–Boos set is inspired by (Roger L. Berger and Boos, 1994), they originally applied it in a hypothesis testing setting for handling nuisance parameters. (Masserano, A. Shen, et al., 2024) applied a similar idea to nuisance parameters in a classification setting, but to the best of our knowledge, our paper is the first to apply the idea in the ill-posed inverse problem UQ setting. Inspired by the methods used in simulation-based inference (Niccolò Dalmasso, Izbicki, and A. B. Lee, 2020; Niccolo Dalmasso et al., 2024; Masserano, Dorigo, et al., 2023) we apply the approach of estimating the quantile function of a test statistic using simulated data. However, this work differs in the underlying model assumptions and composite nature of the null hypotheses. In addition, the prior work assumed bounded and relatively low-dimensional parameter spaces. The search for confidence interval endpoints is sometimes done via the Robbins-Monro (RM) procedure, which iteratively refines estimates to achieve desired coverage probabilities (Garthwaite and Buckland, 1992; Carpenter, 1999). Methods based on quantile regression (Roger Koenker, 2005) have been proposed to improve accuracy. For instance, (Fisher, Schweiger, and Rosset, 2020) introduced a technique that inverts estimated quantiles to determine the endpoints of the confidence intervals. This work is the first to combine test inversion, sampling, and quantile regression to estimate calibrated constraint-aware confidence intervals. Computationally, we propose two custom sampling methods designed specifically to draw design points within the defined Berger–Boos set. This set can be particularly challenging to sample since it is the intersection between a pre-image based on the observed data and linear parameter constraints, which can produce sharp edges and corners. This set is also elongated due to the ill-posedness of the underlying inverse problem. While there are some approaches to handle similar scenarios arising in more traditional MCMC sampling (e.g., nested sampling (Skilling, 2004; Buchner, 2023) and Hamiltonian Monte Carlo sampling with constraints (Pakman and Paninski, 2014)), these algorithms can be intricate, so we develop the following alternatives. In the low-dimensional setting (𝑝 < 10) where the forward model is linear and the noise distribution is Gaussian, we design an accept-reject sampler based on sampling from 𝑝–balls as described in (Voelker, Gosmann, and Stewart, 2017). Although the approach in (Voelker, Gosmann, and Stewart, 2017) can rapidly sample from the pre-image ellipsoids as defined by the data-generating model assumptions, with even a simple non-negativity constraint on the parameters, the proportion of 77 rejected points quickly get impractically large for even moderate dimensions. For moderate and higher dimensional settings (𝑝 ≥ 10), we leverage the Vaidya walk MCMC algorithm from (Yuansi Chen et al., 2018) to generate random walks about a polytope enclosing the Berger–Boos set. Using the Chebyshev ball center to define a notion of polytope centrality along with the extreme points of the Berger–Boos set with respect to the QoI, we create two lines along which starting positions are defined to generate a collection of parallel Markov chains. The design points resulting from these chains are then combined to create the complete set of sampled design points spanning the full range of the QoI values over the Berger–Boos set. The rest of the paper proceeds as follows. In Section 3.2, we recapitulate the framework and theoretical components of (Batlle, Stanley, et al., 2023) upon which this paper’s work is built. Section 3.3 then presents the theoretical Global and Sliced intervals under the Berger–Boos set formulation, along with the four interval constructions and theoretical justifications to guarantee that the constructed sampling-based intervals converge in probability to their theoretical counterparts. In Section 3.4, we give theoretical results regarding our proposed methods. Section 3.5 presents the custom sampling algorithms along with a brief description of how quantile regression works in our intervals. Section 3.6 presents four numerical experiments to demonstrate the coverage and length advantages of our intervals over the OSB interval. Finally, Section 3.7 provides some discussion and conclusions. 3.2 Background A key part of the approach in (Batlle, Stanley, et al., 2023) was to view the optimization-based intervals shown by Equation (3.8) as inverted hypothesis tests, more specifically, a particular inverted likelihood ratio test. This change in perspective allowed for interval analysis through the lens of the properties of a particular log-likelihood ratio (LLR) test. We summarize this connection here, as it is critical to our method development carried out in Section 3.3. Although the numerical examples in Section 3.6 focus on the linear-Gaussian version of the data-generating process, we follow the more general exposition of (Batlle, Stanley, et al., 2023) to indicate that this framework is not limited to that scenario. Suppose 𝒚 ∼ 𝑃𝒙 ∗ , where 𝑃𝒙 ∗ is a distribution parameterized by a fixed but unknown 𝒙 ∗ ∈ R 𝑝 . Let ℓ𝒙 ( 𝒚) denote the log-likelihood of 𝒙 evaluated at 𝒚. We furthermore suppose we know a set X ⊆ R 𝑝 such that 𝒙 ∗ ∈ X. The duality between hypothesis tests and confidence sets is well known in statistics 78 (see, e.g., Chapter 7 of (George Casella and Roger L. Berger, 2002) or Chapter 5 of (Victor M. Panaretos, 2016)). We invert the following hypothesis test: 𝐻0 : 𝒙 ∗ ∈ Φ 𝜇 ∩ X versus 𝐻1 : 𝒙 ∗ ∈ X − Φ 𝜇 , (3.10) where Φ 𝜇 := {𝒙 : 𝜑(𝒙) = 𝜇}. Since we are looking to obtain a confidence interval on the real line, it makes sense that each hypothesis test is defined by 𝜇 ∈ R. Notice that the composite structure of the null hypothesis includes all parameter settings within the 𝜇-level set of the functional of interest. We define the following test statistic to evaluate Test (3.10):

   inf −2ℓ𝒙 (𝒚) − inf −2ℓ𝒙 (𝒚) , if Φ 𝜇 ∩ X ≠ ∅,  𝒙∈Φ  𝒙∈X 𝜇 ∩X (3.11) 𝜆(𝜇, 𝒚) :=    ∞, otherwise,  where 𝜇 denotes the level set of the null hypothesis, 𝒚 is the observed data, and X is the constraint set that is known to contain the parameter. The test is rejected if the test statistic is large, and hence we automatically reject the null if Φ 𝜇 ∩ X = ∅ independently of the observed data. Alternatively, we can consider the test (3.10) only for 𝜇 ∈ 𝜑(X) to perform test inversion. We control the behavior of this test statistic, and therefore the test, by bounding from above the probability of erroneously rejecting the null hypothesis (i.e., type-1 error). As such, we consider the distribution of 𝜆(𝜇, 𝒚) under the null. For each 𝒙 ∈ X, let 𝜇 = 𝜑(𝒙). Define 𝑄 𝒙 : (0, 1) → R such that, for all 𝛼 ∈ (0, 1), P (𝜆(𝜇, 𝒚) ≤ 𝑄 𝒙 (1 − 𝛼)) = 1 − 𝛼, (3.12) where the probability is over 𝒚 ∼ 𝑃𝒙 . We refer to 𝑄 𝒙 as the quantile function of the LLR under the null hypothesis at (𝒙, 𝜑(𝒙) = 𝜇). Since the null hypothesis is composite, using this quantile function to define a cutoff is not enough to control type-1 error. Thus, we use the sliced maximum quantile function over the level-set under consideration: 𝑄 max (3.13) 𝜇,1−𝛼 := sup 𝑄 𝒙 (1 − 𝛼). 𝒙∈Φ 𝜇 ∩X Note, we refer to this version as “sliced” since we are considering the maximum of the quantile function defined by the level-set of the functional which defines a slice through the constrained parameter space. 𝑄 max 𝜇,1−𝛼 as defined above controls the type-1 error for a specific value of 𝜇. We can define a more conservative confidence set using the following global maximum quantile: max 𝑄 max 1−𝛼 := sup 𝑄 𝜇,1−𝛼 = sup 𝑄 𝒙 (1 − 𝛼). 𝜇∈𝜑(X) 𝒙∈X (3.14) 79 By Lemma 2.1 in (Batlle, Stanley, et al., 2023), the set C𝛼 (𝒚; X) := {𝜇 : 𝜆(𝜇, 𝒚) ≤ 𝑄 max 𝜇,1−𝛼 } ⊂ R 𝜇 (3.15) defines a 1 − 𝛼 confidence set for the true functional value, 𝜇∗ = 𝜑(𝒙 ∗ ).The global quantile (3.14) is more conservative than the sliced quantile (3.13) since it holds for all null hypotheses, and therefore the set C𝛼 (𝒚; X) := {𝜇 : 𝜆(𝜇, 𝒚) ≤ 𝑄 max 1−𝛼 } ⊂ R (3.16) is also a 1 − 𝛼 confidence set. To connect this framework back to Interval (3.8), suppose the data generating process is of the form, 𝑃𝒙 = N (𝑲𝒙, 𝑰). The LLR test statistic is then 𝜆(𝜇, 𝒚) = min ∥ 𝒚 − 𝑲𝒙∥ 22 − min ∥ 𝒚 − 𝑲𝒙∥ 22 . 𝒙∈Φ 𝜇 ∩X (3.17) 𝒙∈X (Batlle, Stanley, et al., 2023) (Theorem 2.4) proved that " # " inf 𝜇:𝜆(𝜇,𝒚)≤𝑄 max 1− 𝛼 𝜇, sup 𝜇:𝜆(𝜇,𝒚)≤𝑄 max 1− 𝛼 𝜇 = inf +𝑠(𝒚) 2 ,𝒚) 𝒙∈𝐷 (𝑄 max 1− 𝛼 𝜑(𝒙), # sup 𝜑(𝒙) , +𝑠(𝒚) 2 ,𝒚) 𝒙∈𝐷 (𝑄 max 1− 𝛼 (3.18) where 𝐷 (·, 𝒚) is defined by Equation (3.9). This equivalence asserts that by setting 2 2 2 𝜓𝛼2 := 𝑄 max 1−𝛼 + 𝑠( 𝒚) , where 𝑠(𝒚) := min𝒙∈X ∥ 𝒚 − 𝑲𝒙∥ 2 , we guarantee coverage for Interval (3.8). Furthermore, it asserts that the original OSB interval formulation is 2 ≥ 𝑄 max (see Lemma 2.2 and Corollary 2.3). (Batlle, only valid if and only if 𝜒1,𝛼 1−𝛼 Stanley, et al., 2023) showed that this inequality does not hold in general. As explored in (Batlle, Stanley, et al., 2023) and reiterated above, Interval (3.8) max can be calibrated by computing 𝑄 max 𝜇,1−𝛼 or 𝑄 1−𝛼 . However, pursuing calibration in this way is computationally challenging and statistically conservative. Both of these values require the ability to evaluate 𝑄 𝒙 . Without parameter constraints, this quantile function can be constant under Gaussian-linear assumptions (Batlle, Stanley, et al., 2023), i.e., the test statistic is pivotal. But even with a relatively simple twodimensional Gaussian noise model with non-negativity constraints, this quantile function becomes non-trivial (e.g., see Figure 5.3 in (Batlle, Stanley, et al., 2023)). Beyond the practical difficulty of dealing with the underlying quantile function, both (3.13) and (3.14) can be expressed as chance-constrained optimization problems, which are known to be NP-hard in general and would need to be solved over an unbounded constraint set (Geng and Xie, 2019b; Pena-Ordieres, Luedtke, and 80 Wachter, 2020; Batlle, Stanley, et al., 2023). Statistically, since both (3.13) and (3.14) are the largest quantile function values subject to their respective constraints, they are by definition conservative, especially in scenarios where the true 𝒙 ∗ is far from these most conservative points. Since we do not know 𝒙 ∗ , this conservatism is necessary under the framework of (Batlle, Stanley, et al., 2023). As we see in Section 3.3, our method addresses both of these challenges. In scenarios where evaluating 𝑄 𝒙 is difficult, we sample a collection of design points in a bounded subset of the constraint set, sample the LLR under the null at each design point and use quantile regression to estimate the quantile surface. Furthermore, we can potentially remove these most conservative points from consideration by only considering parameter values that are not unlikely given the observed data. 3.3 Interval constructions In this section, we present four related interval constructions to build on the theory developed in (Batlle, Stanley, et al., 2023) by addressing the key aforementioned challenges. The first implementation challenge is to handle the potentially unbounded constraint set, for which we define and apply the “Berger–Boos” set to create a data-dependent subset of the original constraint set. This set leads to our four interval definitions, which follow a two-stage taxonomy: Global versus Sliced and Inverted versus Optimized. The second implementation challenge is computing these interval constructions in practice, which we achieve using a combination of novel sampling algorithms and quantile regression. We will cover this in a later section (Section 3.5). We rewrite the data-generating process articulated in the introduction: 𝒚 = 𝑓 (𝒙 ∗ ) + 𝜺, 𝜺 ∼ N (0, 𝑰), 𝒙 ∗ ∈ X, (3.19) where 𝑓 : R 𝑝 → R𝑛 is a known forward model and Cov(𝜺) = 𝑰, without loss of generality. Let 𝑓 −1 ( 𝐴) := {𝒙 : 𝑓 (𝒙) ∈ 𝐴} be the pre-image of a set 𝐴 ⊂ R𝑛 under the forward map 𝑓 . Although the assumed data-generating process given by Equation (3.19) is less general than that of 𝒚 ∼ 𝑃𝒙 ∗ assumed in Section 3.2, it is still sufficiently general to contain the application areas mentioned in Section 3.1. Slightly generalizing the form of the additive noise distribution would make the construction of the Berger–Boos set more complicated, but is possible in principle. 81 Global and sliced confidence sets using the Berger–Boos set max Both 𝑄 max 𝜇,1−𝛼 and 𝑄 1−𝛼 in Section 3.2 suffer from the same theoretical and practical concerns. Theoretically, they are both conservative in the sense that they must control type-1 error probability under the worst case for the truth (i.e., the parameter setting with the largest quantile). Practically, not only can 𝑄 𝒙 be difficult to compute, but if the constraint set X is unbounded, computing the aforementioned quantiles becomes even more difficult. These challenges both stem from the composite nature of the null hypothesis. (Roger L. Berger and Boos, 1994) introduce a compelling solution in the context of hypothesis testing with nuisance parameters (a special case of hypothesis testing on 𝜑(𝒙)), which is to control type-1 error only over a data-informed region of the parameter space. Following their construction, we build a confidence interval that, instead of using the maximum 1 − 𝛼 quantile over X in (3.14) (or X ∩ Φ 𝜇 in (3.13)), uses a larger quantile (1 − 𝛾 with 𝛾 < 𝛼) but that is maximized over a smaller set. The construction follows a three-step process: (1) Choose 𝜂 ∈ [0, 𝛼] and build a 1 − 𝜂 confidence interval for 𝑥 ∗ , B𝜂 . Under the additive Gaussian noise assumption in Equation (3.19), letting Γ𝜂 (𝒚) := {𝒚′ ∈ R𝑛 :
2 } we have P 𝑓 (𝒙 ∗ ) ∈ Γ (𝒚) = 1 − 𝜂, and hence ∥ 𝒚 − 𝒚′ ∥ 22 ≤ 𝜒𝑛,𝜂 𝜂 2 B𝜂 := 𝑓 −1 (Γ𝜂 ( 𝒚)) ∩ X = {𝒙 ∈ X : ∥ 𝒚 − 𝑓 (𝒙)∥ 22 ≤ 𝜒𝑛,𝜂 } (3.20) is a 1 − 𝜂 confidence set for 𝑥 ∗3 . We refer to this pre-image confidence set, B𝜂 , as the “Berger–Boos” set. (2) Optimize the 1 − 𝛾 quantile of the test statistic only over B𝜂 (or B𝜂 ∩ Φ 𝜇 ), instead of X (or X ∩ Φ 𝜇 ) . Here, 𝛾 < 𝛼 is chosen to ensure calibration. As proved in lemma 3.3.1, 𝛾 = 𝛼 − 𝜂 is a valid choice. (3) Use the obtained quantiles, which we define as 𝑞¯ 𝛾,𝜂 (𝜇) := sup 𝑄 𝒙 (1 − 𝛾), (3.21) 𝒙∈B 𝜂 ∩Φ 𝜇 𝑞¯ 𝛾,𝜂 := sup 𝑄 𝒙 (1 − 𝛾), (3.22) 𝒙∈B 𝜂 that are used to construct the following sliced (sl) and global (gl) confidence sets: 𝐶𝛼sl (𝒚; B𝜂 ) := {𝜇 : 𝜆(𝜇, 𝒚) ≤ 𝑞¯ 𝛾,𝜂 (𝜇)}, gl 𝐶𝛼 (𝒚; B𝜂 ) := {𝜇 : 𝜆(𝜇, 𝒚) ≤ 𝑞¯ 𝛾,𝜂 }. 3 Intersecting the pre-image 𝑓 −1 (Γ probability since we know 𝒙 ∗ ∈ X. (3.23) (3.24) 𝜂 (𝒚)) with the constraint set X does not change the coverage 82 Analogous to the presentation in (Batlle, Stanley, et al., 2023), we define sliced and global max-quantiles that include the Berger–Boos set to control the type-1 error probability of Test (3.10). Both of these max-quantiles maximize a larger 𝛾-quantile over the set of interest instead of the 𝛼-quantile as shown in (3.13) and (3.14). The following Lemma (analogous to Lemma 2.1 in (Batlle, Stanley, et al., 2023)) gives a sufficient condition to select the values 𝛾 and 𝜂 to ensure a 1 − 𝛼 coverage. Lemma 3.3.1 (Setting 𝜂 and 𝛾 to guarantee 1 − 𝛼 coverage). Let 𝛼 ∈ (0, 1) and define 𝑞¯ 𝛾,𝜂 (𝜇) according (3.21) and 𝐶𝛼sl ( 𝒚; B𝜂 ) according to (3.23). For 𝜂 ∈ (0, 𝛼), 𝐶𝛼sl (𝒚; B𝜂 ) is a 1 − 𝛼 confidence set for 𝜇∗ = 𝜑(𝒙 ∗ ) if 𝛾 ≤ 𝛼 − 𝜂. The length of the obtained interval is a non-increasing function of 𝛾 so the tightest interval will be obtained when equality is satisfied. Proof. See Section 3.8. □ gl The coverage guarantee implied by Lemma 3.3.1 also implies coverage for 𝐶𝛼 ( 𝒚; B𝜂 ), as shown in the following corollary. gl Corollary 3.3.2. In the setting of Lemma 3.3.1. 𝐶𝛼 ( 𝒚; B𝜂 ) is a 1 − 𝛼 confidence set for 𝜇∗ = 𝜑(𝒙 ∗ ) if 𝛾 ≤ 𝛼 − 𝜂. Proof. See Section 3.8. □ Remark 6. Following the generality of the original construction, Lemma 3.3.1 generalizes to any 1 − 𝜂 confidence set of 𝑥 ∗ and test statistic calibrated with the 1 − 𝛾 quantile. In our particular case, since both the confidence set B𝜂 and the test statistic 𝜆(𝜇, 𝑦) depend on the Gaussian log-likelihood ∥ 𝒚 − 𝑓 (𝒙)∥ 22 , there is a stronger interpretation of the Berger-Boos construction. Whenever B𝜂 ∩ Φ 𝜇 ≠ ∅, the BergerBoos construction is equivalent to a data-dependant reduction of the constraint set, replacing X for B𝜂 both in the test statistic and the quantile optimization problems. Since this is done after seeing the data, the optimized quantile needs to increase to 1 − 𝛼 + 𝜂 to maintain 1 − 𝛼 coverage. Navigating the Berger–Boos parameter choices and trade-offs. Setting 𝜂 = 0 gl in 𝐶𝛼 ( 𝒚; B𝜂 ) we maximize 𝑄 𝒙 (1 − 𝛾) = 𝑄 𝒙 (1 − 𝛼) over X, which returns 𝑄 max 1−𝛼 from Equation (3.14). For 𝜂 > 0, the Berger–Boos construction restricts the quantile maximization to a smaller subset of the parameter space while maximizing a larger 1 − 𝛾 quantile to maintain the desired 1 − 𝛼 confidence level of the final confidence 83 set. Fundamentally, the choice of 𝜂 reflects a trade-off between these two opposing effects: as we increase 𝜂 from 0, the set over which the quantile function is optimized shrinks, but the quantile level is increased. In Section 3.6, we perform a numerical experiment comparing average interval length for different values of 𝜂, showing that a small 𝜂 > 0 can be beneficial not only numerically (since in certain cases it allows sampling over a bounded set) but also in terms of the average length of the resulting interval. Interval constructions gl There are possibly many ways to compute 𝐶𝛼 ( 𝒚; B𝜂 ) and 𝐶𝛼sl (𝒚; B𝜂 ) in practice. Obtaining either set comes down to the computation of 𝑞¯ 𝛾,𝜂 (𝜇) and 𝑞¯ 𝛾,𝜂 . If the quantile function 𝑄 𝒙 and its gradient ∇𝒙 𝑄 𝒙 could be evaluated, computing these quantities could potentially be achieved using a first-order numerical optimizer. However, we emphasize that since the sliced max-quantile, 𝑞¯ 𝛾,𝜂 (𝜇), is a function of the level-set parameter, 𝜇, such an optimization would have to be done for each possible functional value. Since such easy function and gradient evaluations rarely exist (see (Batlle, Stanley, et al., 2023) for some examples where such evaluations are possible), this paper develops a sampling-based approach to estimate these quantities to construct Confidence Sets (3.23) and (3.24). As we demonstrate below, once we estimate 𝑞¯ 𝛾,𝜂 (𝜇) and 𝑞¯ 𝛾,𝜂 , we can either use the output from the sampling algorithm to compute the Global or Sliced interval via classical test inversion, or we can use the estimated max-quantiles in optimizations similar to those in Interval (3.18). As such, we introduce four interval constructions: Global Inverted, Global Optimized, Sliced Inverted, and Sliced Optimized. These four options are summarized in Section 3.3. We leverage our ability to sample 𝜆(𝜇, 𝒚) by sampling 𝒚 ∼ N ( 𝑓 (𝒙), 𝑰) to estimate the desired max-quantiles. We present two algorithms, both of which first generate a random set of design points from the Berger–Boos set. If we can directly efficiently compute the desired quantile at each design point, this capability is leveraged in Algorithm 2. If we cannot afford such a computation, Algorithm 3 presents an alternative, which first samples one realization of the LLR test statistic at each sampled design point, and then performs quantile regression with the generated pairs of design points and LLR values to estimate the underlying quantile surface. The sampled design points and either the exact or estimated quantiles at the design points are then used to produce the final intervals. Both algorithms start by sampling B𝜂 uniformly at random to generate a collection 84
of 𝑀 design points across the Berger–Boos set, i.e., 𝒙¯ 1 , 𝒙¯ 2 , . . . , 𝒙¯ 𝑀 ∼ U B𝜂 . Even under the additive Gaussian noise assumption where the pre-image 𝑓 −1 (Γ𝜂 ( 𝒚)) is an ellipsoid, this step is non-trivial since the ill-posedness can produce an extremely narrow or unbounded ellipsoid, resulting in sharp boundaries when intersected with the known constraints in addition to a large portion of the pre-image lying outside of X, making rejection-sampling challenging. We address these sampling challenges in Section 3.5. If computing 𝜆(𝜇, 𝒚) for a given 𝜇 and 𝒚 is inexpensive, Algorithm 2 directly estimates the quantile function at each design point. This computation is most likely inexpensive when there are closed-form solutions to the LLR’s subordinate optimizations. As shown in Algorithm 2, an easy way to estimate each design point’s quantile is to sample its test statistic 𝑁 times and take the appropriate percentile. More often, computing 𝜆(𝜇, 𝒚) is expensive since it involves two constrained optimizations which are possibly non-convex due to either the constraints or the forward model, or just numerically challenging due to the ill-posedness of the problem. In this scenario, one can use Algorithm 3 to sample one realization of the test statistic at each design point and estimate the quantile function over the Berger–Boos set using quantile regression. While Algorithm 2 relies upon computational strength to compute the LLR 𝑁 × 𝑀 times, Algorithm 3 shifts complexity to the quantile regression and relieves the computational burden by assuming that there is information to be shared about the quantile surface between design points (namely that the quantile surface is smooth). The emphasis on the quantile regression further necessitates that the quantile regression be performed well. We discuss some considerations to this end in Section 3.5. We note that although Algorithm 2 and Algorithm 3 make use of the additive Gaussian noise assumption, they are not necessarily limited to this assumption given the proper adjustments to the Berger–Boos set, assuming one still has the ability to sample from the data-generating process. Since these approaches are sampling-based, it is necessary to show convergence as the number of samples gets large. The approach of Algorithm 3 is inspired by recent uncertainty quantification approaches in likelihood-free inference where one can sample from a likelihood but cannot easily compute it. Specifically, we draw inspiration from the use of quantile regression in (Niccolò Dalmasso, Izbicki, and A. B. Lee, 2020; Niccolo Dalmasso et al., 2024; Masserano, Dorigo, et al., 2023; Masserano, A. Shen, et al., 2024). Although these approaches differ in detail and implementation from our approach (e.g., they typically focus on settings with low-dimensional 𝒙), they overlap in the sampling and quantile regression perspectives, which effectively allow machine learning 85 to supplement for computationally intensive or intractable forward models. Namely, rather than assuming a purely stochastic forward model and therefore only being able to sample from the likelihood, we assume 𝑓 is a deterministic function involved in 𝜆(𝜇, 𝒚), which is a random quantity due to the additive noise. Since Algorithm 3 involves training a quantile regressor, it includes separate training and testing sets of design points over the Berger–Boos set and samples from their respective test statistics. Algorithm 2 Direct estimation of quantiles Input: 𝛼, 𝛾, 𝜂 ∈ (0, 1) such that 𝛾 = 𝛼 − 𝜂, 𝑀, 𝑁 ∈ N. Berger–Boos confidence set: Create Γ𝜂 ( 𝒚) ⊆ R𝑛 such that
∗ P 𝑓 (𝒙 ) ∈ Γ𝜂 ( 𝒚) ≥ 1 − 𝜂. 𝑓 −1 (Γ(𝒚)) is also a 1 − 𝜂 confidence set for 𝒙 ∗ , as is B𝜂 = 𝑓 −1 (Γ𝜂 ( 𝒚)) ∩ X. 1: Construct 2: Sample from the Berger–Boos confidence set, B𝜂 : Sample 𝒙¯ 1 , . . . , 𝒙¯ 𝑀 ∼
U B𝜂 . 3: for 𝑘 = 1, 2, . . . , 𝑀 do 4: Sample noise realizations: Sample 𝑁 noise realizations: 𝜺 1 , . . . , 𝜺 𝑁 ∼ N (0, 𝑰). 5: Sample from the LLR distribution: Create an ensemble of LLR draws 𝑁 , where 𝜆𝑖 := 𝜆 (𝜑( 𝒙¯ 𝑘 ), 𝑓 ( 𝒙¯ 𝑘 ) + 𝜺𝑖 ; X). under 𝒙¯ 𝑘 : {𝜆𝑖 }𝑖=1 6: Compute percentile estimate of the 𝛾-quantile: Compute the (1 − 𝛾) × 100 percentile of the LLR samples for the data generating process under 𝒙¯ 𝑘 , i.e., 𝑞b𝛾𝑘 := 𝜆 ({(1−𝛾)𝑁 }) , where {·} denotes the nearest whole number and 𝜆 (𝑖) denotes the 𝑖-th order statistic. 7: end for Output: Pairs of sampled design points and their respective 𝛾-quantiles, i.e., 𝑀 . {( 𝒙¯ 𝑘 , 𝑞b𝛾𝑘 )} 𝑘=1 With the generated pairs from Algorithms (2) and (3), we present two strategies   gl to estimate each of 𝐶𝛼 ( 𝒚; B𝜂 ) and 𝐶𝛼sl ( 𝒚; B𝜂 ). To streamline notation, let 𝒙 𝑘 , 𝑞 𝛾𝑘 denote the 𝑘-th pair from either algorithm. This is notationally helpful since Algorithm 2 only generates one set of parameter samples, whereas Algorithm 3 generates two. That is, using Algorithm 2, 𝒙 𝑘 := 𝒙¯ 𝑘 and 𝑞 𝛾𝑘 := 𝑞b𝛾𝑘 and using Algorithm 3, 𝒙 𝑘 := e 𝒙 𝑘 and 𝑞 𝛾𝑘 := 𝑞b𝛾 (e 𝒙 𝑘 ). To estimate the global confidence set, we estimate 𝑞¯ 𝛾,𝜂 using the empirical maximum, 𝑞b := max 𝑘 𝑞 𝛾𝑘 . This results in the following two interval constructions:  gl 𝐶opt ( 𝒚) := min/max 𝜑(𝒙) : 𝒙 ∈ 𝐷 (b 𝑞 + 𝑠( 𝒚) 2 , 𝒚)  gl 𝐶inv ( 𝒚) := min/max 𝜑(𝒙 𝑘 ) : 𝑘 = 1, . . . , 𝑀 and 𝜆(𝜑(𝒙 𝑘 ), 𝒚; X) ≤ 𝑞b . (3.25) (3.26) 86 Algorithm 3 Quantile regression estimate of quantile surface Input: 𝛼, 𝛾, 𝜂 ∈ (0, 1) such that 𝛾 = 𝛼 − 𝜂; 𝑀tr , 𝑀 ∈ N. Berger–Boos confidence set: Create Γ𝜂 ( 𝒚) ⊆ R𝑛 such that
P 𝑓 (𝒙 ∗ ) ∈ Γ𝜂 ( 𝒚) ≥ 1 − 𝜂. 𝑓 −1 (Γ(𝒚)) is also a 1 − 𝜂 confidence set for 𝒙 ∗ , as is B𝜂 = 𝑓 −1 (Γ𝜂 ( 𝒚)) ∩ X. 2: Sample from the Berger–Boos confidence set B𝜂 : Sample 𝒙¯ 1 , . . . , 𝒙¯ 𝑀tr ∼

U B𝜂 design points to train the quantile regressor and e 𝒙1 , . . . , e 𝒙 𝑀 ∼ U B𝜂 test points to invert the interval, generating 𝑀tr + 𝑀 total samples. Since the test points are used for the interval inversion, they are used as out-of-sample points for the quantile regressor. 3: for 𝑘 = 1, 2, . . . , 𝑀tr do 4: Sample a noise realization: Sample a noise realization: 𝜺 𝑘 ∼ N (0, 𝑰). 5: Sample from the LLR distribution: Compute the LLR under 𝒙¯ 𝑘 with sampled noise 𝜺 𝑘 : 𝜆 𝑘 := 𝜆 (𝜑( 𝒙¯ 𝑘 ), 𝑓 ( 𝒙¯ 𝑘 ) + 𝜺 𝑘 ; X). 6: end for 7: Estimate the quantile function using quantile regression: Using the gen𝑀tr erated pairs {( 𝒙¯ 𝑘 , 𝜆 𝑘 )} 𝑘=1 , estimate the upper 𝛾-conditional quantile function, 𝑞b𝛾 (𝒙), using quantile regression. Output: Generate 𝛾-quantile predictions at out-of-sample test points, 
𝑀 e 𝒙 𝑘 , 𝑞b𝛾 (e 𝒙 𝑘 ) 𝑘=1 . 1: Construct We refer to Interval (3.26) as the Global Inverted interval construction since it is endpoints defined by only those sampled parameter values that comport with the maximum estimated quantile LLR cutoff. We refer to Interval (3.25) as the Global Optimized interval since its endpoints are defined by the extreme functional values gl gl of a feasible region defined by 𝑞b. Although, 𝐶inv ( 𝒚) ≠ 𝐶opt (𝒚) due to finite sample, they are asymptotically equal and in practice show similar performance in terms of coverage and expected length, as shown in Section 3.6. We prove the consistency of the interval constructed via inversion and via optimization in Theorem 3.4.1. Since the endpoints of both the inverted and optimized intervals converge in probability gl to the endpoints of 𝐶𝛼 ( 𝒚; B𝜂 ), the two interval constructions are asymptotically equivalent. To estimate the sliced set, we estimate 𝑞¯ 𝛾,𝜂 (𝜇) using a rolling maximum of the sampled 𝑞 𝛾𝑘 values, as ordered by the sampled functional values, and directly accept or reject functional values based on their estimated quantile. These approaches result in two interval constructions:  sl 𝐶opt ( 𝒚) := min/max 𝜇 ∈ R : 𝜆(𝜑(𝜇), 𝒚; X) ≤ 𝑚 b𝛾 (𝜇) , (3.27)  sl 𝐶inv ( 𝒚) := min/max 𝜑(𝒙 𝑘 ) : 𝑘 = 1, . . . , 𝑀 and 𝜆(𝜑(𝒙 𝑘 , 𝒚; X) ≤ 𝑞 𝛾𝑘 ) , (3.28) 87 where 𝑚 b𝛾 (𝜇) denotes rolling estimate of 𝑞¯ 𝛾,𝜂 (𝜇) defined as follows. We refer to 𝐼 sl ( 𝒚) as the “sliced” index set, i.e., those indices for which the LLR at a particular functional value is less than the estimated quantile at a design point generating that functional value. The rolling maximum quantile is defined using estimated quantiles ordered by the sampled functional values. Choose “rolling” parameter, 𝑇 ∈ N, and let 𝜎(1), 𝜎(2), . . . , 𝜎(𝑀) define an ordering such that 𝜇𝜎(𝑘) ≤ 𝜇𝜎(𝑘+1) for all 𝑘 = 1, . . . , 𝑀 − 1. Define 𝑄 𝑘 :=   {𝑞 𝜎(𝑘) , 𝑞 𝜎(𝑘−1) , . . . , 𝑞 𝜎(𝑘−𝑇) } Then, for a given 𝜇 ∈ min𝒙∈B 𝜂 𝜑(𝒙), max𝒙∈B 𝜂 𝜑(𝒙) , 𝛾 𝛾 𝛾 define 𝑘 ∗ (𝜇) := argmin 𝑘 |𝜇 − 𝜇𝜎(𝑘) | and define 𝑚 b𝛾 (𝜇) := max{𝑞 ∈ 𝑄 𝑘 ∗ (𝜇) }. Using the rolling maximum quantile is one possible way to estimate 𝑞¯ 𝛾,𝜂 (𝜇). One could also bin the quantiles by functional value, compute the maximum predicted quantile in each bin, and then fit a nonparametric regression to fit the maximum binned quantiles to the functional values. sl ( 𝒚). To see why, note This estimator choice for 𝑚 b𝛾 (𝜇) affects how one computes 𝐶opt sl ( 𝒚) as follows: that we can re-express 𝐶opt  sl 𝐶opt ( 𝒚) := min/max 𝜑(𝒙) : 𝒙 ∈ 𝐷 ( 𝑚 b𝛾 (𝜑(𝒙)) + 𝑠(𝒚) 2 , 𝒚) . (3.29) As such, one could substitute the estimated 𝑚 b𝛾 (·) into each endpoint optimization. However, this estimated curve is likely not convex in 𝒙 and therefore complicates the optimizations. One could also pursue a root-finding approach to find the set of 𝜇 such that 𝜆(𝜇, 𝒚) = 𝑚 b𝛾 (𝜇) since these intersection points define a set of accepted functional values. This approach can also be complex in proportion to the complexity of the estimated curve. In our view, the most pragmatic approach is to simply determine all 𝜇 𝑘 such that 𝜆(𝜇 𝑘 , 𝒚; X) ≤ 𝑚 b𝛾 (𝜇 𝑘 ), and then define the endpoints of sl ( 𝒚) to be the minimum and maximum of those accepted sampled values. The 𝐶opt consistency of the computed endpoints for both the inverted and optimized sliced intervals is proven in Theorem 3.4.1. Similar to the global interval constructions, the consistency of both constructions to 𝐶𝛼sl (𝒚; B𝜂 ) also establishes the asymptotic equivalent of the inverted and optimized approaches. Although both Algorithms 2 and 3 can produce a quantile estimate, it is worth noting a few key differences between the two. First, Algorithm 2 involves a nested sampling loop. Thus, any statistical guarantee regarding the validity of its output has the two moving parts of the accuracy of 𝑞b𝛾𝑘 as 𝑁 gets large and the proximity of the approximated quantile to the true maximum quantile function value over B𝜂 as 𝑀 gets large. By contrast, Algorithm 3 only has one sampling loop and estimates the 88 Table 3.1: Summary of methods based on global and sliced max quantile, and whether they are optimization-based or inversion-based. Category Global max quantile Sliced max quantile Inversion-based methods Optimization-based methods Interval (3.26) Interval (3.25) Interval (3.28) Interval (3.27) full quantile function surface over B𝜂 , producing an estimate of the maximum as a consequence, but is reliant on smoothness to do an accurate regression with finitely many data points. 3.4 Theoretical justification In this section, we provide convergence results for the interval constructions of the previous section. We prove that, under certain assumptions, whenever Algorithm 2 or Algorithm 3 are used to estimate the maximum quantiles, the global interval constructions (3.25) and (3.26) converge in probability to the true global interval gl 𝐶𝛼 (𝒚; B𝜂 ) in (3.24), and the sliced interval constructions (3.27) and (3.28) converge in probability to the true sliced interval 𝐶𝛼sl (𝒚; B𝜂 ) in (3.23). Although our approach involving quantile regression is inspired by the approaches in (Niccolò Dalmasso, Izbicki, and A. B. Lee, 2020; Niccolo Dalmasso et al., 2024; Masserano, Dorigo, et al., 2023) and therefore requires similar theoretical results to connect the consistency of the quantile estimation with the interval validity, our approach is sufficiently different and requires novel theoretical insights. First, the inversion of a hypothesis test with a composite null adds a layer of complexity to the proofs. (Niccolo Dalmasso et al., 2024) does address composite null hypotheses, but in the context of nuisance parameters where they use a profile likelihood approach. Second and more fundamentally, (Niccolo Dalmasso et al., 2024; Masserano, Dorigo, et al., 2023) construct confidence sets which are shown to achieve the desired coverage level asymptotically in the number of samples used to train the quantile regressor. By contrast, we show that our computed intervals converge in probability to the theoretical Intervals (3.23) and (3.24) which achieve the correct coverage by definition. Third and finally, since we use the sampled points to invert the interval, it is insufficient to show that the cutoff is consistent as the results in (Niccolò Dalmasso, Izbicki, and A. B. Lee, 2020; Niccolo Dalmasso et al., 2024; Masserano, Dorigo, et al., 2023) show. Our inverted intervals require proof that the samplers can get arbitrarily close to the true endpoint boundaries. 89 Throughout the following results, we assume that for fixed 𝛾 ∈ (0, 1), 𝑄 𝒙 (1 − 𝛾) is a continuous function of 𝒙. We refer the reader to (Kibzun and Kan, 1997) for a full analysis of the properties of parametrized quantile functions. We furthermore assume that B𝜂 is a compact set without isolated points, so that the sampling methods eventually sample close to every point. Theorem 3.4.1. Assume that 𝑄 𝒙 (1 − 𝛾) is a continuous function of 𝒙, and let B𝜂 be compact and without isolated points. Let the quantile regression in Algorithm 2 be consistent for all 𝒙, i.e, such that P(|b 𝑞 𝛾 (𝒙) − 𝑄 𝒙 (1 − 𝛾)| > 𝜀) → 0 as 𝑀𝑡𝑟 → ∞ p is satisfied ∀𝜀 > 0. We will write → − for convergence in probability, understood as 𝑁, 𝑀 → ∞ if Algorithm 1 is used and as 𝑀𝑡𝑟 , 𝑀 → ∞ if Algorithm 2 is used. For either algorithm, we have, for a given observation 𝒚: gl p gl 1. 𝐶inv ( 𝒚) → − 𝐶𝛼 ( 𝒚), p sl ( 𝒚) → 2. 𝐶inv − 𝐶𝛼sl ( 𝒚). Further assume that there exists a point 𝜇¯ ∈ 𝜑(B𝜂 ) satisfying 𝜆( 𝜇, ¯ 𝒚; X) < 𝑞¯ 𝛾,𝜂 , and then gl p gl 3. 𝐶opt (𝒚) → − 𝐶𝛼 ( 𝒚). Finally, further assuming technical conditions discussed in Section 3.9, and then p sl (𝒚) → 4. 𝐶opt − 𝐶𝛼sl ( 𝒚). Proof. See Section 3.9. □ Note that if 𝜆(𝜇, 𝒚) is convex in 𝜇, as it is for linear forward models and quantities of interest (Batlle, Stanley, et al., 2023, Proposition 2.5), the condition of the existence of 𝜇¯ satisfying 𝜆( 𝜇, ¯ 𝒚; X) < 𝑞¯ 𝛾,𝜂 is equivalent to the interval not being empty. For nonlinear forward models, it is a slightly stronger condition. For the convergence of the sliced optimized version, one needs to show that inf 𝜇:𝜆(𝜇,𝒚)≤𝑚b𝛾 (𝜇) 𝜇 converges to inf 𝜇:𝜆(𝜇,𝒚)≤𝑚 𝛾 (𝜇) 𝜇 as 𝑚 b𝛾 (𝜇) converges to 𝑚 𝛾 (𝜇). In order to do so, we study the convergence of optimization problems of the form inf 𝜇: b𝑓 (𝜇)≥0 𝜇 to inf 𝜇: 𝑓 (𝜇)≥0 𝜇 as b 𝑓 converges to 𝑓 . Although the result is not true in general, we provide sufficient technical conditions about 𝑓 and the uniformness of convergence of b 𝑓 to 𝑓 for the result to hold. We discuss the details in Section 3.9. 90 3.5 Implementation methodology Sampling the pre-image Berger–Boos set The viability of this method directly relies upon our ability to sample from the Berger–Boos set, B𝜂 = 𝑓 −1 (Γ𝜂 ( 𝒚)) ∩ X. Under the assumed data generating process in Equation (3.19), the Berger–Boos set is defined as follows:  2 B𝜂 = 𝑓 −1 (Γ𝜂 ( 𝒚)) ∩ X = 𝒙 ∈ X : ( 𝒚 − 𝑓 (𝒙)) ⊤ 𝚺−1 (𝒚 − 𝑓 (𝒙)) ≤ 𝜒𝑛,𝜂 , (3.30) where we have generalized to a non-identity covariance matrix, 𝚺, to make the following exposition more general. This set is equivalent to the set over which the strict bounds intervals are optimized in (Philip B. Stark, 1992a), also called “SSB” intervals in (Stanley, Patil, and Kuusela, 2022; Batlle, Stanley, et al., 2023). Note, 2 instead of 𝜒 2 for 1 − 𝛼 interval computation in however, that one would use 𝜒𝑛,𝛼 𝑛,𝜂 that scenario. We discuss sampling B𝜂 in the linear-Gaussian case, where 𝑓 (𝒙) = 𝑲𝒙, as this scenario aligns with the numerical experiments presented in Section 3.6. We present two sampling approaches: the “Voelker-Gossman-Stewart” (VGS) algorithm based on (Voelker, Gosmann, and Stewart, 2017) for efficiently sampling ellipsoids uniformly at random in low-dimensional scenarios and an MCMC-based algorithm we call the Polytope sampler based on the Vaidya walk presented in (Yuansi Chen et al., 2018). When 𝑓 is linear and has full column rank, B𝜂 is an ellipsoid intersected with the constraint set, making the VGS algorithm an effective option in low-dimensional settings (see Section 3.5). In all other scenarios, especially higher dimensional ones, the Polytope sampler is a better option, as naive accept-reject algorithms become intractable (see Section 3.5). VGS Sampler for low-dimensional and full column rank settings Under the linear-Gaussian assumptions, when 𝑲 has full column rank, it can be shown that  2 B𝜂 = 𝒙 ∈ X : (𝒙 − b 𝒙 ) ⊤ 𝑲 ⊤ 𝚺−1 𝑲 (𝒙 − b 𝒙 ) ≤ 𝜒𝑛,𝜂 , (3.31) where b 𝒙 is the Generalized Least-Squares (GLS) estimator. Equation (3.31) describes an ellipsoid in R 𝑝 intersected with X with axis directions and lengths determined by 2 and the eigenvectors and eigenvalues of 𝑲 ⊤ 𝚺 −1 𝑲, respectively (see (Burt W. 𝜒𝑛,𝜂 Rust and Walter R. Burrus, 1972) for more details). An ellipsoid is nothing but a deformed ball. As such, we can sample uniformly at random from the ellipsoid in R 𝑝 using an algorithm sampling uniformly at random from a 𝑝–ball, followed 91 by an appropriate linear transformation and translation. We can then include an additional accept-reject step to account for the constraint set X. Note, all subsequent discussions of spheres and balls assume unit radii and centering at the origin. Also note that because the ellipsoid in (3.31) is centered around the GLS estimator, we can sample points first from within an ellipsoid of the same shape centered at the origin, and then translate those points by b 𝒙. (Voelker, Gosmann, and Stewart, 2017) propose a particularly efficient and clever algorithm to sample uniformly at random from the 𝑝–ball by proving a connection between uniform sampling on the ( 𝑝 + 1)-sphere (i.e., the surface of the ball in R 𝑝+2 ) and uniform sampling within the 𝑝–ball (i.e., within the unit ball in R 𝑝 ). Namely, one can sample a Gaussian e 𝒛 ∼ N (0, 𝑰 𝑝+2 ) and normalize 𝒛 := e 𝒛/∥e 𝒛 ∥ 2 to sample a point uniformly at random from the ( 𝑝 + 1)-sphere. Next, one simply drops the last two elements of 𝒛 to obtain a point that is uniformly sampled from within a 𝑝–ball. The validity of this approach is substantiated by Lemma 1 and Theorem 1 in (Voelker, Gosmann, and Stewart, 2017) and relies upon a distribution result about the ratio of chi-squared distributions and preservation of distribution under an orthogonal transformation. To denote sampling a point following this procedure, we use the notation 𝒙 ∼ VGS( 𝑝). To sample from our desired ellipsoid in (3.31), first consider   the eigendecomposition −1 ⊤ ⊤ 2 2 2 of 𝑲 𝚺 𝑲 = 𝑷𝛀𝑷 , where 𝛀 = diag 𝜔1 , 𝜔2 , . . . , 𝜔 𝑝 , 𝜔𝑖 is the 𝑖-th eigenvalue of 𝑲 ⊤ 𝚺−1 𝑲 and 𝑷 is an orthonormal matrix where the columns vectors are the
eigenvectors of 𝑲 ⊤ 𝚺−1 𝑲. Denote 𝛀1/2 = diag 𝜔1 , 𝜔2 , . . . , 𝜔 𝑝 and by extension, −1 −1 when 𝜔 > 0 for all 𝑖. These decompositions 𝛀−1/2 = diag 𝜔−1 𝑖 1 , 𝜔2 , . . . , 𝜔 𝑝 imply the correct transformation to √︃ apply to points sampled from the 𝑝–ball via 2 𝑷𝛀−1/2 𝒙. To know that 𝒘 is sampled from 𝒙 ∼ VGS( 𝑝). Namely, define 𝒘 := 𝜒𝑛,𝜂 2 , as then we can the correct ellipsoid, it should be the case that 𝒘 ⊤ 𝑲 ⊤ 𝚺−1 𝑲𝒘 ≤ 𝜒𝑛,𝜂 simply make the update 𝒘 ← 𝒘 + b 𝒙 to ensure that we have a point sampled from (3.31). This guarantee is verified as follows: 𝒘 ⊤ 𝑲 ⊤ 𝚺−1 𝑲𝒘 = 𝒘 ⊤ 𝑷𝛀1/2 𝛀1/2 𝑷⊤ 𝒘 2 = 𝜒𝑛,𝜂 · 𝒙 ⊤ 𝛀−1/2 𝑷⊤ 𝑷𝛀1/2 𝛀1/2 𝑷⊤ 𝑷𝛀−1/2 𝒙 (3.32) 2 2 = 𝜒𝑛,𝜂 · 𝒙 ⊤ 𝒙 ≤ 𝜒𝑛,𝜂 , where 𝑷⊤ 𝑷 = 𝑰 by definition and the last line follows since we know 𝒙 is sampled from the 𝑝–ball and therefore 𝒙 ⊤ 𝒙 ≤ 1. Equation (3.32) justifies that the 𝒘 samples lie within the desired ellipsoid, and their uniform distribution follows from the 92 uniform distribution of 𝒙 in the 𝑝–ball since the distribution is preserved under a linear transformation. Finally, we accept 𝒘 if 𝒘 ∈ X and reject if 𝒘 ∉ X. This procedure for sampling at random from B𝜂 is summarized in Algorithm 4. Algorithm 4 VGS Sampler 2 . Input: 𝑝 ∈ N, 𝑀 ∈ N, 𝑷, 𝛀−1/2 , b 𝒙 , 𝜒𝑛,𝜂 1: Define S := {} to be the initialized set in which the sampled points are to be placed. 2: for 𝑘 = 1, 2, . . . , 𝑀 do 3: Sample from the 𝑝–ball using VGS: 𝒙 𝑘 := 𝒛 1:𝑝 /∥𝒛∥ 2 , where 𝒛 ∼ N (0, 𝑰 𝑝+2 ) and 𝒛1:𝑝 denotes taking the first through 𝑝-th indices (inclusive). √︃ 2 𝑷𝛀−1/2 𝒙 . Transform VGS output: 𝒘 𝑘 := 𝜒𝑛,𝜂 𝑘 5: Translate 𝒘 𝑘 by the GLS estimator: 𝒘 𝑘 ← 𝒘 𝑘 + b 𝒙. 6: Accept-Reject to incorporate constraints: If 𝒘 𝑘 ∈ X, then add S ← S ∪ {𝒘 𝑘 }, else start loop iteration 𝑘 again. 7: end for Output: S containing uniformly sampled points over B𝜂 (as defined in (3.31)). 4: Generating samples using the VGS Sampler is efficient, but its feasibility diminishes in high dimensions because of the accept-reject step. To illustrate this point, consider 𝑝 the simple scenario where X = R+ , i.e., the non-negative orthant of R 𝑝 and suppose 𝑝 we sample from the 𝑝–ball intersected with R+ by sampling 𝒙 ∼ VGS( 𝑝) where 2 . Then, P(𝒙 ∈ R 𝑝 ) = 2−𝑝 and the 𝑷 = 𝛀 = 𝑰, b 𝒙 = 0 and we use 1 instead of 𝜒𝑛,𝜂 + acceptance probability of each sample goes to zero exponentially in 𝑝. To make this 𝑝 point slightly more general, we generate data 𝒚 ∼ N (𝒙 ∗𝑝 , 𝑰 𝑝 ) where 𝒙 ∗𝑝 ∈ R+ and is a vector of ones. For a collection of dimensions 𝑝 ∈ [2, 30], we sample 𝒙 ∼ VGS( 𝑝) 𝑝 with 𝑷 = 𝛀 = 𝑰 and b 𝒙 = 𝒚 to estimate the probability that 𝒙 is in R+ and plot the results in the left panel of Figure 3.2. Since the acceptance probability decays exponentially (under 10−4 for 30 dimensions), it becomes clear that this algorithm is inefficient in dimensions larger than 10, since the acceptance probability quickly becomes prohibitively small in regimes where a larger sample size is even more important. Dimension is only one of two primary complicating factors. Not only is there less of a (potentially) shifted 𝑝–ball’s volume in the non-negative orthant as 𝑝 𝑝 grows but if our data are generated via 𝒚 ∼ N (𝑲𝒙 ∗ , 𝑰 𝑝 ) with 𝒙 ∗ ∈ R+ where the condition number of 𝑲 is large, it is possible that even less of the ellipsoid from which the VGS Sampler draws points intersects with the parameter constraint. To illustrate this point, we generate a single observation from the aforementioned 93 model using a 𝑲 ∈ R40×40 as described in Section 3.6. The 𝒙 ∗ ∈ R40 + is created in the same way as that of Section 3.6. The right panel of Figure 3.2 shows a computed probability mass function for the number of coordinates in a VGS Sampler draw (with 𝑷 and 𝛀 defined such that 𝑲 ⊤ 𝑲 = 𝑷𝛀𝑷⊤ ) complying with the non–negativity parameter constraints. Equivalently, the right panel of Figure 3.2 shows the computed probability mass function for the number of coordinates lying within the Berger– Boos set. For this particular setup, we critically note that out of 5×104 draws from the VGS Sampler, none of the draws had all coordinates comply with the non-negativity constraint. By contrast, for the aforementioned noise model where we more simply sample from the 𝑝–ball intersected with the non-negative orthant, the computed acceptance probability is approximately 3.35 × 10−6 , in both cases emphasizing the VGS Sampler’s poor performance in high-dimensional and ill-conditioned forward model regimes. Figure 3.2: Numerical illustrations of the VGS Sampler’s infeasibility in high dimensional regimes. The left panel shows the computed acceptance probability of a point drawn by the VGS Sampler with data generated from a non-negatively constrained Gaussian noise model. Crucially, at only 30 dimensions, the acceptance probability is already less than 10−4 for this particular setup. The right panel shows the computed probability mass function for the number of non-negative constraint complying coordinates of a VGS sample with data generated from a non-negatively constrained linear Gaussian model in 40 dimensions with a non-identity forward model. Since this is an example using a forward model with a large condition number (≈ 1.6 × 104 ), we critically note that there is empirically zero probability of generating a sample within the non-negativity constraints. Polytope sampler for general settings In settings where the forward model is not linear and full column rank, Algorithm 4 fails. The ineffectiveness of this algorithm expands if the condition number of the linear forward model is large such that most of the pre-image ellipsoid defining the Berger–Boos set lies outside of the constraint set. Although these scenarios 94 induce particular geometric challenges, in the linear-Gaussian case with convex X we are still fundamentally sampling a convex set for which there is a vast literature. For example, there is a vast sampling literature for computing Bayesian posteriors in high dimensions via nested sampling (Skilling, 2004; Ashton et al., 2022; Buchner, 2023). In particular, nested sampling has been successfully applied in high-dimensional cosmology settings using sophisticated approaches to strategically sample the parameter space by restricting prior sampling in various ways (Buchner, 2023; Montel, Alvey, and Weniger, 2023). Although these approaches provide tools addressing a sampling setting similar to ours (i.e., the Berger–Boos set can be viewed as a portion of the parameter space defined by a cutoff on the likelihood) they are ultimately aimed at sampling from a particular distribution (i.e., the posterior), which is a stronger criterion than required here. For sampling general convex sets, simple algorithms like Hit-and-Run are available (Smith, 1984; Lovasz, 1999; Lovasz and Vempala, 2006). However, in the particular case considered here, more sophisticated and efficient algorithms can be devised. In particular, there exists a deep literature on random walks over polytopes such that the asymptotic stationary distribution of the walk is a uniform distribution over the polytope of interest (Kannan and Narayanan, 2012; Narayanan, 2016; Yuansi Chen et al., 2018). As such, we propose to first construct a bounding polytope, P 𝑑 composed of 𝑑 half-spaces around B𝜂 , sample 𝐶 random walks within the Berger–Boos set using the Vaidya walk as described in (Yuansi Chen et al., 2018) each starting at a point from a collection of strategically chosen locations, and then combine the parallel chains to create the final sample set. The detailed algorithm can be seen in Algorithm 5. Although MCMC algorithms are typically evaluated using trace plots on individual dimensions of the parameter space, given the high-dimensionality of the problem and the final step combining several MCMC chains in the parameter space started from different positions, it is more meaningful to evaluate this algorithm’s ability to sample fully from the functional space. Namely, we can solve for the largest and smallest values the functional can take within the Berger–Boos set as follows:     𝑙 𝑢 𝐼BB ( 𝒚) := 𝜇BB , 𝜇BB = min 𝜑(𝒙), max 𝜑(𝒙) . (3.33) 𝒙∈B 𝜂 𝒙∈B 𝜂 When 𝜑(𝒙) = 𝒉⊤ 𝒙 for some 𝒉 ∈ R 𝑝 , 𝐼BB ( 𝒚) corresponds to the SSB interval in gl (Stanley, Patil, and Kuusela, 2022). Computing the sets 𝐶𝛼sl (𝒚; B𝜂 ) and 𝐶𝛼 ( 𝒚; B𝜂 ) well is then contingent upon sampling functional values within 𝐼BB (𝒚) well since a functional value can only be included in the inverted set if the sampler has a non-zero probability of sampling arbitrarily close to it. By “well”, we informally 95 mean that the sampled functional values range at least between the endpoints of 𝐼BB (𝒚) and that if we partition 𝐼BB (𝒚) into 𝑛 sub–intervals, that most if not all of the sub–intervals contain at least one sample. In practice, it will often be the case that the sampled functional values can lie outside the endpoints of 𝐼BB ( 𝒚) since the MCMC chains are sampling a bounding polytope of the Berger–Boos set. Choosing the bounding polytope. Any bounded polytope of B𝜂 is the intersection of a finite number of half-spaces defined in R 𝑝 . The task of constructing a bounding polytope is then equivalent to choosing a collection of 𝑑 hyperplanes in R 𝑝 to construct a set P 𝑑 := {𝒙 ∈ R 𝑝 : 𝑨𝒙 ≤ 𝒃} such that B𝜂 ⊆ P 𝑑 , where 𝑨 ∈ R𝑑×𝑝 and 𝒃 ∈ R𝑑 . Let 𝒂𝑖⊤ ∈ R 𝑝 denote the 𝑖-th row vector of 𝑨. We compute 𝑏𝑖 (the 𝑖-th element of 𝒃) as 𝑏𝑖 = max 𝒂𝑖⊤ 𝒙. (3.34) 𝒙∈B 𝜂 This construction ensures the necessary inclusion. We consider three approaches to pick the vectors 𝒂𝑖 : (i) using the constraints X, (ii) using the known eigenvectors defining the bounded directions of the pre-image ellipsoid, and (iii) randomly. In practice, we combine these approaches to ensure that we consider only parameter settings in agreement with our physical constraints, and to tighten the bounding Berger–Boos set polytope as much as possible. There is a tradeoff with respect to the latter consideration since the mixing time and computational cost for the Vaidya walk increase with the number of hyperplanes (Yuansi Chen et al., 2018). To incorporate the known parameter constraints, consider the non-negativity con𝑝 straint used in Section 3.6, i.e., 𝒙 ∈ R+ . To enforce non-negativity, we set 𝒂𝑖 = −𝒆𝑖 for 𝑖 = 1, . . . , 𝑝, where 𝒆𝑖 is defined by its 𝑖-th element set to one and the rest of its elements set to zero and 𝑏𝑖 = 0. These choices produce 𝑝 rows in 𝑨 corresponding to the desired lower bounds (i.e., 𝑥𝑖 ≥ 0 for all 𝑖), but we can compute an additional 𝑝 constraints using (3.34) with 𝒂𝑖 := 𝒆𝑖 for 𝑖 = 𝑝 + 1, . . . 2𝑝. These 2𝑝 constraints define a hyperrectangle enclosing the Berger–Boos set in the parameter space. To incorporate polytope constraints based upon the forward model, we use the ellipsoidal definition of the pre-image as shown in (3.31) and eigendecomposition of 𝑲 ⊤ 𝚺−1 𝑲 = 𝑷𝛀𝑷⊤ shown in (3.32). Note that this ellipsoid form is valid under the linear-Gaussian noise assumption and for all 𝑲. In the event that 𝑲 is not full column rank, the ellipsoid defined by Equation (3.31) is still defined via the pseudo–inverse of of 𝑲 ⊤ 𝚺𝑲 but where the ellipsoid is unbounded in some directions. The column vectors of 𝑷 corresponding to the non-zero entries on the diagonal of 𝛀 are the 96 eigenvectors corresponding to the bounded principal axes. As such, both 𝒑𝑖 and − 𝒑𝑖 for 𝑖 = 1, . . . , 𝑝 (the column vectors of 𝑷) can be used as rows of 𝑨 with their corresponding bounds defined by (3.34). Finally, to further tighten the polytope around the Berger–Boos set, we sample a multivariate Gaussian, i.e., 𝒂𝑖 ∼ N (0, 𝑰 𝑝 ) to include random hyperplanes. Note, it is possible that the unbounded directions of the ellipsoid defined by Equation (3.31) are not bounded when intersecting with the parameter constraint set and hence there is no bounding polytope. In this case, our interval construction should be unbounded since there is not enough information to produce a bounded confidence set for the quantity of interest. Once the components ( 𝑨, 𝒃) have been defined using some or all of the above hyperplane generation strategies, the Vaidya walk can immediately be employed to perform a random walk around the polytope. The primary intuitive requirement we wish to satisfy with any sampling scheme in this context is that every region of the Berger–Boos set has a non-zero probability of being sampled. Asymptotically, the Vaidya walk samples the desired polytope uniformly at random, which satisfies a stronger requirement. In practice, the need to sample non-uniformly can arise if there are particularly meaningful parameter settings in regions that are difficult for the random walk to reach. We explore such a case in Section 3.6. Additionally, although the asymptotic distribution of the Vaidya sampler is theoretically sufficient, in practice it often has some difficulty reaching the corners of the generated polytope. Although MCMC chain mixing is typically evaluated by looking at trace plots, this diagnostic is insufficient here because the dimension is high and the defined polytope can make different dimensions difficult to compare. Instead, we consider how well the sampler samples the functional values 𝜑(𝒙). This motivates taking a collection of starting points between the SSB endpoints and the Chebyshev center of the polytope, as described in the following section. Constructing the parallel chain starting points. Although the random walks defined in (Yuansi Chen et al., 2018) asymptotically sample uniformly over P 𝑑 , given the long and thin shape of the pre-image, running the Vaidya walk from even a “good” starting position does not consistently sample the functional space well. Instead, we use the following heuristic to construct several parallel chains that constitute a complete sample when combined. We define an even number of starting points, 𝐶 ∈ 2N, to roughly span the Berger–Boos set, run the Vaidya walk for 𝑀 𝑝 steps from each starting point to collect parameter settings {e 𝒙1 , . . . , e 𝒙 𝑀 𝑝 }, and combine the samples from each walk, resulting in 𝐶 × 𝑀 𝑝 total samples. In practice, 97 𝐶 and 𝑀 𝑝 are chosen to yield a total of 𝑀 samples as desired for Algorithm 2 or Algorithm 3. Since we have designed this process to avoid the neccessity of any individual chain reaching its asymptotic distribution, we do not need a burn–in period for any chain as long as we are confident that the union of the sampled points sufficiently covers Interval (3.33) so that we may accept or reject any quantity of interest value in that interval. We define the collection of starting points using the line segments defined by the parameters generating the endpoints of 𝐼BB (𝒚) and the Chebyshev center of P 𝑑 as defined in (Boyd and Vandenberghe, 2004), and denoted by 𝒙 𝑐 . This point is the center of the largest ball contained within P 𝑑 and therefore acts as a way to characterize the center of P 𝑑 . We denote the parameter settings generating the endpoints of 𝐼BB (𝒚) by b 𝒙 𝑙 for the lower endpoint and b 𝒙𝑢 for the upper endpoint. We then define a uniform grid of values {𝜏𝑙 }𝐶/2 𝑙=1 such that ′ 𝜏𝑙 ∈ (0, 1) and 𝜏𝑙 < 𝜏𝑙+1 for all 𝑙. For each 𝑘 ∈ [𝐶], define 𝑘 := ⌈𝑘/2⌉ and set 𝒙 𝑘𝑠𝑡𝑎𝑟𝑡 := 𝜏𝑘 ′ b 𝒙 𝑙 + (1 − 𝜏𝑘 ′ )𝒙 𝑐 if 𝑘 is odd and 𝒙 𝑘𝑠𝑡𝑎𝑟𝑡 := 𝜏𝑘 ′ b 𝒙 𝑢 + (1 − 𝜏𝑘 ′ )𝒙 𝑐 if 𝑘 is even. Creating starting positions along the lines connecting these endpoints and the Chebyshev center accommodates the chosen polytope while helping ensure that samples are chosen spanning the range of possible functional values over the Berger–Boos set. The empirical performance of the Polytope sampler can be seen in Figure 3.3, showing (left) a histogram of the functional values sampled and (right) a trace plot for the sampled Vaidya walks starting from points constructed as described above. These plots are generated for one observation from the 80-dimensional ill-posed inverse problem in the coverage study performed in Section 3.6. Critically, the histogram shows that the Polytope sampler samples the functional space well and the trace plot shows that our starting point construction heuristic performs well in practice. Sampling from the Berger–Boos set. With P 𝑑 and the starting positions defined, we construct a sample S := {e 𝒙1 , . . . , e 𝒙𝐶×𝑀 𝑝 }. We leave the details of the Vaidya walk to the original paper Yuansi Chen et al. (2018), but note an essential radius tuning parameter of the algorithm that must be chosen. This radius impacts the spread of a Gaussian proposal distribution for the walk and thus affects a new proposed point’s acceptance probability. Choosing a too large radius results in a low acceptance rate of proposed steps, thus creating walk that does not mix well. In contrast, choosing a too small radius results in a high acceptance rate with relatively small step sizes. In practice, we find a radius setting of 0.5 works well as it produces 98 Figure 3.3: Polytope sampler output for a realization of the 80-dimensional illposed inverse problem studied in Section 3.6. The left panel contains a histogram of sampled functional values which both span and cover well the range of 𝐼BB ( 𝒚) (shown by the dashed gray lines in both plots). The right panel contains trace plots of the 14 Vaidya walks (each indicated by a different color) which together constitute the full sample. Our heuristic for choosing starting points along the lines connecting the parameter settings generating the endpoints of 𝐼BB ( 𝒚) and the Chebyshev center of the polytope provides a good initial spread of starting functional values. an acceptance probability of ≈ 33.3%, producing a reasonable trade-off between taking meaningful steps and not rejecting too many steps. Quantile regression Algorithm 3 explained in Section 3.3 involves using quantile regression to learn a quantile surface from a collection of pairs of design points and samples from the LLR test statistic. As previously mentioned, similar approaches have been taken in (Niccolò Dalmasso, Izbicki, and A. B. Lee, 2020; Niccolo Dalmasso et al., 2024; Masserano, Dorigo, et al., 2023; Masserano, A. Shen, et al., 2024), and since quantile regression is a technique facilitating our interval constructions, we will only give a brief overview of quantile regression and some different ways to implement it. Fundamentally, given a one-dimension random variable 𝑧 ∼ 𝑃𝒙 that depends on the parameter 𝒙 ∈ R 𝑝 , we are interested in the upper 𝛾-quantile at every parameter setting, i.e., P𝒙 (𝑧 > 𝑄 𝒙 (1 − 𝛾)) = 𝛾. (3.36) We note that the quantile surface itself is not random, so we can use draws from the distribution 𝑃𝒙 to estimate 𝑄 𝒙 at a given parameter setting 𝒙. However, as noted in Section 3.3, performing such an estimate is not always computationally feasible, and intuitively, we might expect quantiles to vary smoothly over the parameter space, which would imply that information about a quantile at 𝒙1 should be related to a quantile at 𝒙 2 if these points are close. In statistics literature, estimating 𝑄 𝒙 is 99 Algorithm 5 Polytope sampler Input: 𝑀 𝑝 ∈ N and 𝐶 ∈ 2N. 𝑨 ∈ R𝑑×𝑝 , 𝒃 ∈ R𝑑 . {𝜏𝑙 }𝐶/2 𝑙=1 , where 𝜏𝑙 ∈ (0, 1) and 𝜏𝑙 < 𝜏𝑙+1 for all 𝑙. 1: Let S := {} be the set in which we store all sampled parameter settings. 2: Construct Chebyshev center: Solve for 𝒙 𝑐 using the following optimization. maximize 𝑟 𝒙 𝑐 ,𝑟 subject to 𝒂𝑖⊤ 𝒙 𝑐 + 𝑟 ∥ 𝒂𝑖 ∥ 2 ≤ 𝑏𝑖 , 𝑖 = 1, . . . , 𝑑. 3: Compute 𝜑 extremes of Berger–Boos set: Extreme points are computed with respect to the functional of interest: b 𝒙 𝑙 := argmin 𝜑(𝒙) subject to 𝒙 ∈ B𝜂 , b 𝒙 𝑢 := argmax 𝜑(𝒙) subject to 𝒙 ∈ B𝜂 . (3.35) 4: for 𝑘 = 1, 2, . . . , 𝐶 do 5: Construct starting point: Define 𝑘 ′ := ⌈𝑘/2⌉. Define 𝒙 𝑘𝑠𝑡𝑎𝑟𝑡 = 𝜏𝑘 ′ b 𝒙 𝑙 + (1 − 𝜏𝑘 ′ )𝒙 𝑐 if 𝑘 is odd and 𝒙 𝑘𝑠𝑡𝑎𝑟𝑡 = 𝜏𝑘 ′ b 𝒙 𝑢 + (1 − 𝜏𝑘 ′ )𝒙 𝑐 if 𝑘 is even. 6: Run the Vaidya walk for 𝑀 𝑝 steps: Collect samples {e 𝒙1 , . . . , e 𝒙 𝑀 𝑝 } and add them to S. 7: end for Output: S containing sampled points over B𝜂 . framed as estimating a quantile function conditional on known covariates and is often thought of as a generalization of estimating the conditional median (Roger Koenker and Hallock, 2001). Just as conditional mean and conditional median estimation can be accomplished by using an appropriate loss function (sum of squares and absolute differences, respectively), estimating conditional quantiles can be accomplished by minimizing the pinball loss defined as follows:    (1 − 𝛾)(𝑞 − 𝑧),  𝐿 𝛾 (𝑧, 𝑞) :=   −𝛾(𝑧 − 𝑞),  𝑧 < 𝑞, (3.37) 𝑧 ≥ 𝑞, (Roger Koenker, 2005; Steinwart and Christmann, 2011). As such, estimating the quantile surface can be framed as a risk-minimization problem, leaving only standard modeling choices to fill in for 𝑞 in (3.37). Although initial efforts were focused on linear parametric quantile regressors (R. Koenker and Bassett Jr, 1978), in recent years, modeling efforts have focused on nonparametric varieties. (Meinshausen, 100 2006) adapted random forests to quantile regression. (Takeuchi et al., 2006) leveraged Reproducing Kernel Hilbert Spaces to construct smooth quantile regressors. Closer to our application, (Masserano, Dorigo, et al., 2023) used neural networks to optimize the pinball loss to learn the quantile surface for their application. Using the design points in the Berger–Boos set as sampled via the VGS or Polytope samplers, Algorithm 3 shows how we sample from the test statistic distribution defined at each design point to define a data set to fit a quantile regressor. We use the Gradient Boosting Regressor implemented in scikit-learn (Pedregosa et al., 2011) with the “quantile” loss function (i.e., pinball loss defined in eq. (3.37)) to fit the quantile surface for our numerical examples in Section 3.6. This algorithm involves a collection of hyperparameters (i.e., the minimum number of samples required to split an internal node, the minimum number of samples required to be a leaf node, the maximum depth of any individual estimator, the learning rate, and the number of estimators) which we determine using 10-fold cross validation in a pilot study ahead of our simulation experiments in Section 3.6. Although one may use any quantile regression approach to estimate the quantile surface, we emphasize the importance of choosing an approach that can accommodate a nonlinear surface in the parameters (such as the Gradient Boosting Regressor) as the parameter constraints are known to produce nonlinear quantile surfaces in even simple examples as seen in (Batlle, Stanley, et al., 2023). 3.6 Numerical experiments For scenarios within the linear-Gaussian case of data-generating process (3.19), the OSB interval can be regarded as the previous the state-of-the-art option for computing constraint-aware confidence intervals. Although these intervals have empirically achieved nominal coverage in applications (Patil, Kuusela, and Hobbs, 2022; Stanley, Patil, and Kuusela, 2022), they generally do not guarantee coverage (Batlle, Stanley, et al., 2023). As such, we use the OSB interval in the following numerical experiments as a main comparison point to the intervals defined in this paper. In scenarios where the OSB interval achieves at least nominal coverage, we show that our intervals are either competitive or better in terms of expected interval length. In scenarios where the OSB interval does not achieve nominal coverage, our intervals do achieve nominal coverage and can have shorter expected length. We provide four numerical experiments to make these points. The first set of two uses a constrained Gaussian noise model setup in two or three dimensions. These two experiments illustrate the aforementioned points in addition to constituent parts of 101 the interval computation process due to the relatively low dimensions. The second set of two considers a wide-bin deconvolution setup inspired by particle unfolding in high-energy physics (Stanley, Patil, and Kuusela, 2022). This setup features an 80-dimensional parameter space with a rank-deficient forward model and is thus a substantially more complicated computational scenario compared to the two or three-dimensional Gaussian noise models. These examples demonstrate the superior performance of our intervals over OSB in terms of both coverage and expected length. In the following experiments, we form 68% confidence intervals, set 𝜂 = 0.01 and compute 𝛾 according to Lemma 3.3.1. We draw 103 observations from each data-generating process to estimate both coverage and expected interval length for our four interval constructions and the OSB interval. We additionally provide 95% confidence intervals in the form of orange line segments to characterize statistical error for both coverage and expected length estimates. The coverage confidence intervals are Clopper-Pearson intervals for the success probability parameter of a binomial distribution, while the expected length confidence intervals are the average length plus/minus the appropriately scaled standard error of the mean. Constrained Gaussian in two dimensions The two-dimensional Gaussian noise model is defined as follows: 𝒚 = 𝒙 ∗ + 𝜺, 𝜺 ∼ N (0, 𝑰2 ) , 𝒙 ∗ ∈ R2+ , (3.38)  ⊤ where 𝜑(𝒙) = 𝑥 1 − 𝑥 2 and 𝒙 ∗ = 0.5 0.5 . The LLR is then given as follows: 𝜆(𝜇, 𝒚) = 𝑥 min ∥ 𝒚 − 𝒙∥ 22 − min ∥ 𝒚 − 𝒙∥ 22 . −𝑥 =𝜇 1 2 𝒙∈R2+ 𝒙∈R2+ (3.39) This example first appeared in (Tenorio, Fleck, and Moses, 2007) as a case where the OSB interval allegedly fails to achieve nominal coverage when the true parameter 𝒙 ∗ is such that 𝜑(𝒙 ∗ ) = 0. However, (Batlle, Stanley, et al., 2023) overturned this result by proving OSB validity in this case. As such, this example is important to include because of its historical context and OSB interval validity. The proof that the 2 OSB interval covers in this particular example relies upon showing that 𝑄 max 1−𝛼 = 𝜒1,𝛼 2 is the upper 𝛼-quantile of a chi-squared distribution for all 𝛼 ∈ (0, 1), where 𝜒1,𝛼 with one degree of freedom. Alternatively stated, it holds that 𝜆(𝜑(𝒙 ∗ ), 𝒚; R2+ ) is stochastically dominated by 𝜒12 . This result is shown in Lemma 4.4 of (Batlle, Stanley, et al., 2023). Estimated coverage and length results are shown in Figure 3.4. We note that all four of our interval constructions are competitive with OSB in terms of coverage, while 102 all of our interval constructions have higher estimated expected length, apart from the Sliced constructions which are within statistical error of OSB. Since the OSB 2 interval is defined using 𝑄 max 1−𝛼 = 𝜒1,𝛼 and the 𝛼-quantile surface of the LLR rapidly approaches this global max-quantile as one moves away from the origin (see Figure 5.3 in (Batlle, Stanley, et al., 2023)), the OSB interval lengths are difficult to beat in practice with intervals based on the Berger–Boos sets since these sets likely contain 2 . The left panel of Figure 3.5 shows four parameter settings with quantiles near 𝜒1,𝛼 realizations of the data-generating process with the observations shown as red points. For each observation, the blue points show uniformly distributed draws within its Berger–Boos set, sampled using the VGS sampler. Cross-referencing the spread of the Berger–Boos set samples in Figure 3.5 with Figure 5.3 in (Batlle, Stanley, et al., 2023), it is clear that there are always samples in the parameter space where the 2 quantile. Further, when including quantile surface is nearly the same as the 𝜒1,𝛼 the Berger–Boos set in the interval construction, we instead construct our intervals using the 𝛾-quantile, where 𝛾 < 𝛼, as the LLR cutoff, resulting in a more relaxed constraint. This fact can be clearly observed in the central panel of Figure 3.5, showing the sampled 𝛾-quantiles within the Berger–Boos set of one observation from the data generating process. Since a non-trivial portion of this distribution is 2 , the longer average length of the Global intervals is explained. In the right above 𝜒1,𝛼 panel of Figure 3.5, for the same observation, we show the estimated sliced max2 . Since this estimated function is quantile function, 𝑚 b𝛾 (𝜇), in orange alongside 𝜒1,𝛼 2 at their intersection points with the underlying LLR function shown in the above 𝜒1,𝛼 solid blue line, it further makes sense that the Sliced interval constructions provide no additional length improvement compared to the OSB interval in this particular setting. Constrained Gaussian in three dimensions As seen in the previous example, in a case where the OSB interval is known to achieve nominal coverage, its expected length can be difficult to beat. However, OSB coverage guarantee can be difficult to prove or disprove, since it amounts to proving stochastic dominance on the non-trivial LLR statistic. In such situations, our intervals immediately provide a clear theoretical advantage. One such case involving a three-dimensional constrained Gaussian case was explored in (Batlle, Stanley, et al., 2023), to which we now apply our four interval constructions. The three-dimensional Gaussian noise model is defined as follows: 𝒚 = 𝒙 ∗ + 𝜺, 𝜺 ∼ N (0, 𝑰3 ) , 𝒙 ∗ ∈ R3+ , (3.40) 103 Figure 3.4: Estimated coverages and expected lengths across all four interval constructions and OSB for comparison at the 68% level for the two-dimensional constrained Gaussian setting. All four of our interval constructions are comparable to OSB with respect to coverage, but OSB shows better-expected length performance, aside from our two sliced interval constructions. Although the OSB intervals are defined using the global max-quantile (𝑄 max 1−𝛼 ) and therefore can potentially be improved upon by limiting the considered parameter space via the Berger–Boos set, 2 quantile (see due to the rapidity with which the 𝛼-quantile surface meets the 𝜒1,𝛼 Figure 5.3 in (Batlle, Stanley, et al., 2023)), the OSB interval lengths are difficult to beat in practice.  ⊤ where 𝜑(𝒙) = 𝑥1 + 𝑥 2 − 𝑥 3 and 𝒙 ∗ = 0.03 0.03 1 . The LLR is then defined as follows: 𝜆(𝜇, 𝒚) = min ∥ 𝒚 − 𝒙∥ 22 − min ∥ 𝒚 − 𝒙∥ 22 . (3.41) 𝑥 1 +𝑥 2 −𝑥3 =𝜇 𝒙∈R3+ 𝒙∈R3+ Figure 3.6 shows estimated coverage and expected length across all four interval constructions and the OSB interval. While the OSB interval fails to attain nominal coverage in this example, all four of our interval constructions do, with the Sliced constructions providing the best calibration. While the Global constructions pay a fairly steep price for coverage in expected interval length, the Sliced constructions navigate the trade-off well, paying for coverage with only slightly longer intervals compared to the OSB interval. The setting of 𝒙 ∗ used here different than that of (Batlle, Stanley, et  is slightly ⊤ ∗ al., 2023), where 𝒙 = 0 0 1 was used. In (Batlle, Stanley, et al., 2023), this setting was used as a counter-example for OSB coverage, since 𝑄 𝒙 ∗ (1 − 𝛼) > 2 for at least some 𝛼 ∈ (0, 1). However, as shown in Figure 5.5 of (Batlle, 𝜒1,𝛼  ⊤ 2 ∗ Stanley, et al., 2023), when 𝛼 = 0.05, 𝑄 𝒙 ∗ (1 − 𝛼) ≤ 𝜒1,𝛼 for 𝒙 = 𝑡 𝑡 1 when 𝑡 is approximately greater than 𝑒 −2 ≈ 0.135, which indicates that parameter settings violating stochastic dominance by 𝜒12 exist close to the parameter constraint boundary. The location of these key parameter settings presents a challenge for the 104 Figure 3.5: (Left) Four realizations of the data-generating process where the observations are shown in red. For each realization, the blue points are uniformly distributed samples from its Berger–Boos set, sampled using the VGS sampler. (Center) For a realization of the data-generating process, we plot the distribution of 𝛾-quantiles for the points sampled by the VGS sampler. Notably, a non-trivial 2 defining the OSB interval. (Right) For the same percent of these are above 𝜒1,𝛼 realization, we plot the estimated sliced max-quantile function, 𝑚 b𝛾 (𝜇) in orange 2 alongside 𝜒1,𝛼 in red. The blue points correspond to sampled parameter values, each of which has a functional and quantile value, while the solid blue line shows the LLR over the functional varies. All intervals can be read immediately from this image by inspecting where the blue LLR curve intersects the sampled points. Figure 3.6: Estimated coverage and expected length across all four interval constructions and OSB for comparison at the 68% level for the three-dimensional constrained Gaussian example. All four of our interval constructions achieve nominal coverage while the OSB interval does not. While the Global interval constructions pay a steep price in expected length compared to OSB, the Sliced constructions are only slightly longer than OSB. Polytope sampler described by Algorithm 5. In Section 3.10, Algorithm 6 presents a modified version of Algorithm 5 that better handles sampling in this example. Berger–Boos set experiment. For this model, we investigate the effect of changing the parameter 𝜂 that controls the Berger–Boos construction in the global interval. We compute intervals for different 𝒚 and fixed 𝒙 ∗ = (2, 2, 0), 𝒙 ∗ = (3, 3, 0) and 105 𝒙 ∗ = (5, 5, 0) as 𝜂 ranges between 0 and 𝛼 = 0.32. The lengths of such intervals, averaged over the data 𝒚, are shown in Figure 3.7. In this example the maximum quantile is achieved at 𝒙 = (0, 0, 𝑡) for large 𝑡, and, as expected, the benefit of using a small 𝜂 > 0 becomes more pronounced as the true 𝒙 ∗ becomes farther from the point that achieves the maximum quantile. However, as 𝜂 grows too close to 𝛼 the downside of optimizing the 1 − 𝛼 + 𝜂 quantile instead of 1 − 𝛼 outweighs the benefit of optimizing it over a smaller set, resulting in larger intervals. This suggests that a small 𝜂 > 0 is a reasonable default, as suggested originally by (Roger L. Berger and Boos, 1994). Figure 3.7: Confidence interval lengths in the Berger–Boos setting, averaged over values of 𝒚, for varying 𝜂 and 𝒙 ∗ . The minimum average length occurs at a small 𝜂 > 0, showing that the construction is beneficial if 𝜂 is tuned correctly. This occurs because even for moderately small 𝜂, the Berger–Boos set, which is a threedimensional sphere intersected with the non-negative orthant, avoids the point with the highest 1 − 𝛼 quantile. Wide-bin deconvolution While the numerical experiments in Sections (3.6) and (3.6) show that our interval constructions are competitive with the OSB interval in scenarios where it is known to provide coverage and superior to the OSB interval by achieving nominal coverage when OSB does not, this section shows the superior performance of our interval constructions relative to OSB in a more complex high-dimensional setting. We consider the problem of computing a confidence interval for the sum of adjacent bins of a deconvolved histogram as described in (Stanley, Patil, and Kuusela, 2022). This problem is a core statistical problem of particle unfolding in high-energy physics. For more detailed information, we refer the reader to (Kuusela and Victor M. Panaretos, 2015b; Kuusela, 2016; Kuusela and Philip B. Stark, 2017b; CMS Collaboration, 2016; CMS Collaboration, 2019). 106 The data-generating process is linear with Gaussian noise, 𝒚 = 𝑲𝒙 ∗ + 𝜺, 𝜺 ∼ N (0, 𝑰), 𝒙 ∗ ≥ 0, (3.42) where 𝑲 ∈ R40×80 and 𝜑(𝒙) = 𝒉⊤ 𝒙. The vector 𝒉 defines the bin-adjacent aggregation. In particle unfolding, the vectors 𝒙 ∗ and 𝒚 represent particle counts within discretized bins. A collection of 𝒉 vectors can act to sum the contents of adjacent bins to effectively lower the resolution of the inference problem. The LLR is then defined as follows: 𝜆(𝜇, 𝒚) = min ∥ 𝒚 − 𝑲𝒙∥ 22 − min ∥ 𝒚 − 𝑲𝒙∥ 22 . ⊤ 𝒉 𝒙=𝜇 𝒙∈R80 + 𝒙∈R80 + (3.43) We emphasize two features of this setup that complicate the task of computing confidence intervals for 𝜑(𝒙). First, the forward model, 𝑲, has a non-trivial null space and a large condition number, making any inverse problem point estimation and UQ markedly challenging. Typically, this sort of ill-posedness is handled with regularization of some kind, but as is well-known in the inverse problem literature and specifically shown in (Kuusela, 2016), including such regularization induces a bias, which can undercut desired statistical guarantees (e.g., coverage) of the inference object of interest. Including constraints and focusing on a particular functional of the parameter vector implicitly regularizes the problem (Patil, Kuusela, and Hobbs, 2022; Stanley, Patil, and Kuusela, 2022), but shifts the problem difficulty to inference with constraints. Although (Batlle, Stanley, et al., 2023) and this paper proposes a theoretical framework to perform inference with constraints, the second challenge is in the practical implementation due to the high-dimensional parameter space in scenarios like this example. As we show in the following sections, the Polytope sampler described by Algorithm 5 and quantile regression do an adequate job producing samples and fitting quantile surfaces in this high-dimensional space to ensure the desired coverage of the interval constructions. As extensively discussed in (Stanley, Patil, and Kuusela, 2022), the OSB interval (i.e., 2 using 𝜒1,0.05 in the optimization-based interval construction) produces empirically valid confidence intervals in all tested scenarios, albeit typically with over-coverage. In the scenarios considered in (Stanley, Patil, and Kuusela, 2022), the underlying function generating the true histogram means (𝒙 ∗ ∈ R80 + ) was relatively smooth, likely contributing to the over-coverage. As such, we present two true parameter settings for 𝒙 ∗ in (3.42) to highlight two advantages of our interval constructions over the OSB interval. First, we use the original smooth parameter setting from (Stanley, 107 Patil, and Kuusela, 2022) to show how our intervals improve over-coverage relative to the OSB interval by reducing the expected interval length. Second, we present an “adversarial” setting where our interval constructions achieve nominal coverage while the OSB interval does not. Figure 3.8 shows the smooth and adversarial settings for 𝒙 ∗ . We constructed the adversarial setting by first computing our interval constructions on the smooth setting and then looking at the maximum out-of-sample predicted quantile for a generated observation with a large predicted quantile. For each observation drawn within both settings, we draw 2.1 × 104 samples using the Polytope sampler as described by Algorithm 5. Figure 3.8: Parameter values for the smooth and adversarial settings for 𝒙 ∗ used to illustrate our interval construction versus the OSB interval. The adversarial setting is made more difficult by the sharp jumps in parameter values. Smooth setting Using the smooth 𝒙 ∗ shown in Figure 3.8, (Stanley, Patil, and Kuusela, 2022) showed that the OSB interval over-covers at the 95% level. Furthermore, it was shown that the OSB interval was the shortest across a range of other interval options, including SSB, prior optimized, and minimax. As such, for this setting, we show that our interval constructions not only achieve nominal coverage, but the sliced constructions dramatically reduce over-coverage compared to OSB by producing substantially shorter intervals on average. Estimated coverage and expected interval lengths are shown in Figure 3.9. Both Global interval constructions and OSB dramatically over-cover which highlights the conservatism of the Global constructions. These estimated coverage values 108 indicate that within each observation’s Berger–Boos set, there is a parameter setting against which the method has to protect that is substantially more difficult to cover than the true realistic parameter setting. Interestingly, both Global constructions produce markedly longer intervals on average compared with the OSB interval. Aligning with the intuition from the Global and Sliced construction definitions, the Sliced intervals are less conservative as seen by their lower over-coverage and significantly smaller average lengths. Importantly, both Sliced constructions are shorter on average compared to the OSB interval, with the Sliced Inverted showing an 18.7% reduction in average length and the Sliced Optimized showing a 11.1% reduction in average length. Figure 3.9: Estimated coverage and expected length across all four interval constructions and OSB at the 68% level for the smooth wide-bin deconvolution experiment. While the Global interval constructions over-cover like the OSB interval, the Sliced interval constructions reduce both over-coverage and expected interval length. Adversarial setting The key result in Section 3.6 is that the Sliced interval constructions both reduce over-coverage and expected interval length compared with the OSB interval. In this section, we show that for the adversarial parameter setting the OSB interval does not achieve nominal coverage, whereas all four of our interval constructions do achieve nominal coverage while still reducing the expected interval length in the case of the Sliced interval constructions compared to the OSB interval. The corresponding estimated coverage and expected length results are shown in Figure 3.10. Both Global interval constructions and the Sliced Optimized interval over-cover, with the Sliced Optimized over-covering to a lesser extent than the Global intervals. The Sliced Inverted interval achieves nominal coverage within the statistical uncertainty. The estimated expected lengths tell a story similar to that of the smooth example, with the Global intervals showing the longest average interval lengths, the 109 Sliced intervals showing the shortest, and the OSB interval being between the two. Importantly, the Sliced intervals are again significantly shorter than the OSB interval, even though the OSB interval does not achieve nominal coverage. The Sliced Inverted interval shows a 18.9% average interval length reduction over OSB while the Sliced Optimized interval shows an 11.4% average interval length reduction. Figure 3.10: Estimated coverage and expected length across all four interval constructions and OSB at the 68% level for the adversarial wide-bin deconvolution experiment. While the OSB interval fails to achieve nominal coverage, all four of our interval constructions do. Interestingly, the Sliced interval constructions are meaningfully shorter than the OSB interval while also providing coverage. For both the smooth and adversarial experiments, we note the differences in coverage and expected length between the two Sliced interval constructions. Although both approaches theoretically compute the same interval, significant differences can arise from their construction. For instance, the Sliced Inverted intervals are constructed by accepting individual functional values, which makes each point’s acceptance dependent upon the quality of the quantile regressor at that point. By contrast, since the Sliced Optimized intervals essentially smooth over the max quantiles as a function of the functional space, the intervals are less sensitive to the quantile regressor’s performance at any individual point. In the above simulation studies, there were realizations of the data for which the Sliced Inverted construction only accepted a single functional value sample, thereby making the confidence interval a single point. We found the Sliced Optimized construction to be more robust in these settings as the max quantile values were shared in a sliding window over the functional space. 3.7 Conclusion This paper proposes several confidence interval constructions for functionals in constrained ill-posed inverse problems. Our approach is based on two key ideas: data-adaptive constraints using a Berger–Boos construction and sampling-based 110 inversion. Two independent decisions when constructing intervals provide four different valid intervals: Global versus Sliced, using the quantile function either over the entire or along level-set slices of the Berger–Boos set and Inverted versus Optimized, constructing the interval either by individually accepted functional values via the estimated quantile function or using the estimated quantile function in endpoint optimizations. All of the constructions are built upon the preliminary constraint by the data-informed Berger–Boos set, followed by a sampling procedure to estimate a quantile function that can be used to invert or optimize the interval endpoints. We have validated the method (including all four aforementioned interval constructions) through several numerical examples, demonstrating its ability to provide correct coverage, better calibration, and comparable or shorter interval length compared to the OSB interval baseline. Overall, our approach offers a flexible framework that can incorporate constraints directly and can be tailored to various types of inverse problems. The main takeaway is that data–adaptive constraining helps improve the length of the resulting confidence intervals, and enables sampling which makes it feasible to carry out the test inversion needed to construct confidence intervals with a desired nominal coverage. There are several promising directions for future work. One direction is to find ways to extend our method to even higher-dimensional problems, which are more challenging. This would involve developing improved techniques to handle the curse of dimensionality and exploring the trade-off between accuracy and computational complexity. For the approaches in this paper in particular, this extension would require a more tailored sampling approach. Another direction is to leverage more sophisticated machine learning algorithms (deep learning models or ensemble methods) to improve the estimate of the quantile function and thus improve the accuracy and efficiency of our confidence intervals. Additionally, applying our approach to other applications involving ill-posed inverse problems, such as medical imaging or geophysics, would provide further validation of the effectiveness of our approach. Finally, it is of interest to conduct a further theoretical analysis of our approach under different constraints and noise conditions to better understand its limitations and strengths. This would involve studying the statistical properties of our confidence intervals and investigating the impact of various assumptions on their performance. Of particular interest are relaxations of the linear forward model and Gaussian noise assumptions to extend our method’s application domain. Although the original theoretical foundation developed in (Batlle, Stanley, et al., 2023) does not make these assumptions, our implementation relies upon them for the tractability of the 111 sampling algorithms. Overall, these future research directions have the potential to demonstrate the applicability and robustness of our approach in a wide range of domains. 3.8 Proofs in Section 3.3 Proof of Lemma 3.3.1 We reproduce the original argument in (Roger L. Berger and Boos, 1994), originally stated in terms of p-values, translated into our quantile setting. Fix any 𝒙 ∗ ∈ X and consider the sets: • 𝐴1 = {𝒚 : 𝐵𝜂 ( 𝒚) ∋ 𝒙 ∗ } • 𝐴2 = {𝒚 : 𝜆(𝜇∗ , 𝒚) ≤ 𝑄 𝒙 (1 − 𝛾)} • 𝐴3 = {𝒚 : 𝜆(𝜇∗ , 𝒚) ≤ 𝑞¯ 𝛾,𝜂 (𝜇∗ )} = {𝒚 : 𝜆(𝜇∗ , 𝒚) ≤ sup𝒙∈B 𝜂 ∩Φ 𝜇∗ 𝑄 𝒙 (1 − 𝛾)} We now have that P(𝒚 ∈ 𝐴1 ) = 1 − 𝜂, P( 𝒚 ∈ 𝐴2 ) = 1 − 𝛾, and that ( 𝐴1 ∩ 𝐴2 ) ⊂ ( 𝐴1 ∩ 𝐴3 ). Therefore, P( 𝒚 ∉ 𝐴3 ) = P( 𝒚 ∉ 𝐴3 , 𝒚 ∈ 𝐴1 ) + P( 𝒚 ∉ 𝐴3 , 𝒚 ∉ 𝐴1 ) (3.44) ≤ P( 𝒚 ∉ 𝐴2 , 𝒚 ∈ 𝐴1 ) + P( 𝒚 ∉ 𝐴1 ) (3.45) ≤ P(𝒚 ∉ 𝐴2 ) + P( 𝒚 ∉ 𝐴1 ) (3.46) = 𝛾+𝜂 (3.47) so that P( 𝒚 ∈ 𝐴3 ) ≥ 1 − 𝛾 − 𝜂. Imposing 1 − 𝛾 − 𝜂 ≥ 1 − 𝛼 gives the desired result. Proof of Corollary 3.3.2 gl By definition, 𝑞¯ 𝛾,𝜂 ≤ 𝑞¯ 𝛾,𝜂 for all 𝜇 ∈ R. Therefore, 𝐶𝛼sl (𝒚; B𝜂 ) ⊆ 𝐶𝛼 ( 𝒚; B𝜂 ), and thus

gl P𝒙 ∗ 𝜇∗ ∈ 𝐶𝛼 ( 𝒚; B𝜂 ) ≥ P𝒙 ∗ 𝜇∗ ∈ 𝐶𝛼sl ( 𝒚; B𝜂 ) ≥ 1 − 𝛼. (3.48) 𝜇 3.9 Proof of Theorem 3.4.1 Throughout the proof, we make use of the following lemma: Lemma 3.9.1. Let 𝑓 : R𝑛 → R and X ⊂ R𝑛 such that min 𝑓 (𝒙) is achieved, and at 𝒙∈X least one of the minimizers 𝒙 ∗ satisfies: 1. 𝑓 is continuous at 𝒙 ∗ , 112 2. 𝒙 ∗ is not an isolated point of X (i.e., ∀𝛿 > 0, 𝐵𝛿 (𝒙 ∗ ) ∩ X ≠ ∅). Let 𝜇 be a measure on X such that 𝜇(𝐵) > 0 for all 𝐵 ⊆ X such that 𝜆 Leb (𝐵) > 0 (where 𝜆 Leb (𝐵) refers here to the Lebesgue measure of the set 𝐵). Let 𝑌𝑚 = 𝑝 min 𝑓 (𝒙𝑖 ), where 𝒙𝑖 are i.i.d. samples from 𝜇. Then 𝑌𝑚 → − 𝑓 (𝒙 ∗ ). 𝑖=1,...,𝑚 Proof. Fix 𝜀 > 0 and let us show that P(|𝑌𝑚 − 𝑓 (𝒙 ∗ )| > 𝜀) → 0. Since 𝑓 is continuous at 𝒙 ∗ there exists a 𝛿 > 0 such that 𝑓 (𝐵𝛿 (𝒙 ∗ )) ⊂ 𝐵𝜀 ( 𝑓 (𝒙 ∗ )) so that P(|𝑌𝑚 − 𝑓 (𝒙 ∗ )| ≥ 𝜀) ≤ P(𝒙𝑖 ∉ 𝐵𝛿 (𝒙 ∗ ), ∀𝑖 = 1, . . . , 𝑚) = (P𝒙∼𝜇 (𝒙 ∉ 𝐵𝛿 (𝒙 ∗ ))) 𝑚 . (3.49) ∗ ∗ Since 𝒙 is not an isolated point, we have 𝜆 Leb (𝐵𝛿 (𝒙 ) ∩ X) > 0 and therefore □ P𝒙∼𝜇 (𝒙 ∉ 𝐵𝛿 (𝒙 ∗ )) < 1 and P(|𝑌𝑚 − 𝑓 (𝒙 ∗ )| ≥ 𝜀) → 0. We begin by proving that the empirical maximums of the quantiles obtained both by Algorithm 1 and Algorithm 2 (assuming the quantile regressor is consistent) converge to the true max quantile. Algorithm 1 Let 𝑞¯ de b𝑖𝛾 (𝑁), where we explicitly write the dependence 𝛾,𝜂 := max 𝑞 𝑖=1,...,𝑀 with the number of samples and the index 𝑖 refers to the quantile estimated at the 𝑖-th p sampled point 𝒙𝑖 . We aim to show that 𝑞¯ de − 𝑞¯ 𝛾,𝜂 . We know that 𝑞b𝑖𝛾 (𝑁) converges 𝛾,𝜂 → in probability to 𝑄 𝑃 𝒙𝑖 (1 − 𝛾) as 𝑁 → ∞. We have de 𝑞¯ de 𝛾,𝜂 − 𝑞¯ 𝛾,𝜂 ≤ 𝑞¯ 𝛾,𝜂 − max 𝑄 𝑃 𝒙𝑖 (1 − 𝛾) + max 𝑄 𝑃 𝒙𝑖 (1 − 𝛾) − 𝑞¯ 𝛾,𝜂 . (3.50) 𝑖=1,...,𝑀 𝑖=1,...,𝑀 The first term can be made smaller than 𝜀/2 as 𝑁 → ∞ by convergence of the estimator, and the second term can be made smaller than 𝜀/2 as 𝑀 → ∞ by application of Lemma 3.9.1 to the quantile function (maximizing instead of minimizing). Algortihm 2 The proof is identical to that of Algorithm 1, with the only difference of replacing the quantiles estimated via Monte Carlo sampling to those estimated by the quantile regression, and 𝑁 to 𝑀tr , the number of samples needed to train the quantile regression. Since the quantile regression is assumed to be consistent, the first term can be made arbitrarily small as 𝑀𝑡𝑟 grows, and the result follows. Since identical convergence results apply for both algorithms, henceforth we will not explicitly distinguish: the proof is written in terms of 𝑁, which can be replaced by 𝑀𝑡𝑟 . 113 Proof of Statement 1 (Global Inverted) Recall, gl 𝐶inv (𝒚) =   𝜑(𝒙 𝑘 ), min 𝑘∈{1,...,𝑀 }:𝜆(𝜑(𝒙 𝑘 ),𝒚)≤b 𝑞 (𝑁) 𝜑(𝒙 𝑘 ) max 𝑘:𝜆(𝜑(𝒙 𝑘 ),𝒚)≤b 𝑞 (𝑁)  (3.51)  = min max 𝜇𝑘 , 𝑘∈{1,...,𝑀 }:𝜆(𝜇 𝑘 ,𝒚)≤b 𝑞 (𝑁) 𝜇𝑘 , 𝑘:𝜆(𝜇 𝑘 ,𝒚)≤b 𝑞 (𝑁) (3.52) where 𝑞b(𝑁) := max 𝑞b𝑖𝛾 (𝑁), the estimated quantiles of the sampled 𝒙𝑖 ∈ B𝜂 and 𝑖=1,...,𝑀 𝜇𝑖 := 𝜑(𝒙𝑖 ). Note that 𝜇𝑖 are samples in 𝜑(B𝜂 ) ⊂ R. Also note that we have more explicitly written out the interval definition (i.e., Equation (3.26)) to emphasize clarity rather than presentation. Consider the left extreme of the interval, a similar argument follows from the right extreme. Consider three quantities: 𝜇1 (𝑁, 𝑀) := min 𝜇 𝑘 s.t. 𝑘 = 1, . . . , 𝑀 and 𝜆(𝜇 𝑘 , 𝒚) ≤ 𝑞b(𝑁) (3.53) 𝜇2 (𝑀) := min 𝜇 𝑘 s.t. (3.54) 𝜇3 := min 𝜇 s.t. 𝑘 = 1, . . . , 𝑀 and 𝜆(𝜇 𝑘 , 𝒚) ≤ 𝑞¯ 𝛾,𝜂
𝜇 ∈ 𝜑 B𝜂 and 𝜆(𝜇, 𝒚) ≤ 𝑞¯ 𝛾,𝜂 , (3.55) p p − 𝜇3 . − 𝜇3 . Lemma 3.9.1 shows that 𝜇2 → Our goal is to show that as 𝑁, 𝑀 → ∞, 𝜇1 → Indeed, the minimization over the indices 𝑘 such that the condition is satisfied can be seen as a rejection sampling strategy in which all accepted samples are samples of the feasible region of the optimization in 𝜇3 . As 𝑀 grows, since the sampler eventually samples all areas of 𝜑(B𝜂 ), some samples are guaranteed to be close to the optimum with high probability. Finally, for fixed 𝑀 and 𝑁 going to infinity, p 𝜇1 → − 𝜇2 . This follows from the continuity of the optimization problem with respect p to the right-hand side of the constraint, and the fact that max 𝑞(𝒙𝑖 ) → − 𝑞¯ 𝛾,𝜂 . It 𝑖=1,...,𝑀 p follows that as 𝑁, 𝑀 → ∞, 𝜇1 → − 𝜇3 . Proof of Statement 2 (Sliced Inverted) The proof technique is similar to the one of Statement 1, replacing 𝑞b(𝑁) := max 𝑞b𝑖𝛾 (𝑁) for 𝑞b𝛾𝑘 (𝑁). Defined then, similarly as in the previous proof: 𝑖=1,...,𝑀 𝜇1 (𝑁, 𝑀) := min 𝜇 𝑘 s.t. 𝑘 = 1, . . . , 𝑀 and 𝜆(𝜇 𝑘 , 𝒚) ≤ 𝑞b𝛾𝑘 (𝑁) (3.56) 𝜇2 (𝑀) := min 𝜇 𝑘 s.t. (3.57) 𝜇3 := min 𝜇 s.t. 𝑘 = 1, . . . , 𝑀 and 𝜆(𝜇 𝑘 , 𝒚) ≤ 𝑞¯ 𝛾,𝜂 (𝜇 𝑘 )
𝜇 ∈ 𝜑 B𝜂 and 𝜆(𝜇, 𝒚) ≤ 𝑞¯ 𝛾,𝜂 (𝜇) (3.58) 114 where 𝑞¯ 𝛾,𝜂 (𝜇) = max𝒙∈Φ 𝜇 ∩B 𝜂 𝑄 𝒙 (1 − 𝛾). As 𝑁 → ∞, 𝑞b𝛾𝑘 (𝑁) → 𝑄 𝒙 𝑘 (1 − 𝛾). p Lemma 3.9.1 can be used to show 𝜇2 → − 𝜇3 , because the sampler strategy used that first samples 𝒙 𝑘 and then accepts 𝜇 𝑘 = 𝜑(𝒙 𝑘 ) as a sample if 𝜆(𝜑(𝒙 𝑘 ), 𝒚) < 𝑞 𝛾𝑘 , it can be shown that for every feasible point 𝜇, there is eventually a sample close to it. Finally, 𝜇1 will become arbitrarily close to 𝜇2 as 𝑀 grows large, since both are taking the minimum over samples that are sampled densely from the feasible set, meaning accepted 𝜇 𝑘 will eventually be close to 𝜇3 for both the case of 𝜇1 and the case of 𝜇2 Proof of Statement 3 (Global Optimized) We prove the continuity of the optimization problem, max 𝜇 s.t. 𝜆(𝜇, 𝒚) ≤ 𝑞, (3.59) 𝜇∈𝜑(B 𝜂 ) as a function of 𝑞 in the positive measure interval (𝜆( 𝜇, ¯ 𝒚), 𝑞¯ 𝛾,𝜂 ]. Therefore, convergence in probability follows as we have convergence in probability to 𝑞¯ 𝛾,𝜂 as 𝑁, 𝑀 → ∞, which in particular implies that the maximum quantile estimate is in the interval (𝜆( 𝜇, ¯ 𝒚), 𝑞¯ 𝛾,𝜂 ] almost surely. We do so by appealing to the maximum theorem (Ok, 2007, S E.3), which, in general, guarantees continuity of functions of the form 𝑓 ∗ (𝜃) = sup{ 𝑓 (𝑥, 𝜃) : 𝑥 ∈ 𝐶 (𝜃)} as long as 𝑓 is continuous, 𝐶 is a continuous compact-valued correspondence and 𝐶 (𝜃) is non-empty for all 𝜃 ∈ Θ. The continuity of 𝐶 comes from the continuity of the LLR, 𝑓 is equal to the identity and the strict feasibility condition ensures 𝐶 is non-empty in Θ := (𝜆( 𝜇, ¯ 𝒚), 𝑞¯ 𝛾,𝜂 ]. Proof of Statement 3 (Sliced Optimized) We will prove sufficient conditions for convergence of inf 𝜇 to 𝜇: b 𝑓 𝑘 (𝜇)≥0 inf 𝜇: 𝑓 (𝜇)≥0 𝜇 as b 𝑓𝑘 converges to 𝑓 , and the result will follow by taking b 𝑓 𝑘 (𝜇) = 𝑚 b𝛾 (𝜇) − 𝜆(𝜇, 𝒚) and 𝑓 (𝜇) = 𝑚 𝛾 (𝜇) − 𝜆(𝜇, 𝒚). A similar argument can be repeated for the supremum. Use use the notation b 𝑓 𝑘 to indicate that 𝑘 sampled points are used to estimate this function via the definition of 𝑚 b𝛾 (𝜇) (see Section 3.3). Lemma 3.9.2. Let 𝑓 : R → R be a function, and let 𝑓 𝑘 be a sequence of functions 𝑓 𝑘 : R → R. Let 𝜇∗ = inf 𝜇 and 𝜇 𝑘 = inf 𝜇. Let the sequence of functions 𝑓 (𝜇)≥0 𝑓 𝑘 (𝜇)≥0 { 𝑓 𝑘 } be such that for all 𝛿 > 0,   P sup | 𝑓 𝑘 (𝜇) − 𝑓 (𝜇)| > 𝛿 → 0 as 𝑘 → ∞, 𝜇 (3.60) 115 namely, 𝑓 𝑘 converges in probability uniformly to 𝑓 . Furthermore, let 𝑓 be such that for all 𝜀 > 0, there exists 𝛿′ > 0 such that | 𝑓 (𝜇)| > 𝛿′ if and only if |𝜇 − 𝜇∗ | < 𝜀. p Then, we have 𝜇 𝑘 → − 𝜇∗ . 𝑝 We then have 𝜇 𝑘 → − 𝜇∗ . Proof. Assume for the sake of contradiction that there exists 𝜀 > 0 such that P (|𝜇 𝑘 − 𝜇∗ | ≥ 𝜀) does not go to 0. Then, by the condition on 𝑓 , it must follow that P (| 𝑓 (𝜇 𝑘 )| > 𝛿) does not go to 0. But, P (| 𝑓 (𝜇 𝑘 )| > 𝛿) ≤ P (| 𝑓 (𝜇 𝑘 ) − 𝑓 𝑘 (𝜇 𝑘 )| > 𝛿) + P (| 𝑓 𝑘 (𝜇 𝑘 )| > 𝛿) , (3.61) and the right-hand side goes to 0 by uniform convergence in probability (first term) and by feasibility of 𝜇 𝑘 (second term). □ 3.10 Additional details and illustrations in Section 3.6 Importance-like sampler for the three-dimensional example in Section 3.6 2 are located Since the parameter settings with quantiles meaningfully larger than 𝜒1,𝛼 close to the constraint boundary, a sampling challenge is presented. Using the samplers as described in Section 3.5 results in under-sampling of this large-quantile region since both samplers provide uniform random samples over the Berger–Boos set. Algorithm 6 presents a modified version of Algorithm 2, an importance-like sampler to increase the probability mass of samples close to the constraint boundary. Note, we say “importance-like” because we do not provide any theoretical guarantee regarding this sampler’s ability to produce draws from a particular target distribution. We tailored Algorithm 6 to settings with a non–negativity constraint and hand-tuned the length scale parameter to the particular three-dimensional example in Section 3.6. The key to Algorithm 6 is the additional accept/reject step where the 𝑘-th sample is accepted with probability 𝑝 𝑘 . Additionally, by setting 𝑞 ∈ (0, 1), we prioritize retaining samples closer to the non-negativity boundary. This effect can be seen in Figure 3.11, where the vast majority of samples are found closer to the constraint boundary. The ability of Algorithm 6 to better sample the high-quantile regions of the Berger–Boos set can be seen in Figure 3.12. In particular, for the functional values 𝜇 ∈ (−4, −2), the importance-like sampler is substantially more effective than the Polytope sampler at finding parameter setting with 𝛾-quantiles greater than 1.1. 116 Figure 3.11: When sampling points using Algorithm 6, the vast majority of samples are found closer to the non-negativity constraint boundary. The true parameter setting is shown by the red point, while the parameter settings sampled by Algorithm 6 are shown by the blue points. This sampling prioritization helps adequately sample 2 . the regions of the Berger–Boos set where the quantile surface is larger than 𝜒1,𝛼 Furthermore, the vast majority of sampled points lie within the Berger–Boos set with some lying outside within the bounding polytope. Figure 3.12: The importance-like sampler described by Algorithm 6 is more effective than the Polytope sampler described by Algorithm 5 at sampling parameter settings with 𝛾-quantile greater than 1.1. Each parameter setting sampled by Algorithm 6 is shown by a blue point. This improved ability helps ensure the coverage guarantee shown in the left panel of Figure 3.6. 117 Algorithm 6 Importance-like sampler for three-dimensional constrained Gaussian Input: Number of samples: 𝑀 ∈ N, inverse length scale: 𝛾 𝑝 , and order of norm: 𝑞 ∈ (0, 1). 1: Instantiate a list S of length 𝑀 to store sampled points. 2: while |S| < 𝑀 do 3: Draw 𝑀 − |S| realizations from Algorithm 5: e 𝒙1 , . . . , e 𝒙 𝑀−|S| 4: for 𝑘 = 1, . . . , 𝑀 − |S|: do 5: Compute the probability of accepting the 𝑘-th draw:
exp −𝛾 𝑝 ∥e 𝒙𝑘 ∥𝑞 . 6: Draw 𝑧 𝑘 ∼ Bernoulli( 𝑝 𝑘 ). 7: if 𝑧 𝑘 = 1 then 8: S [𝑘] ← e 𝒙𝑘 9: end if 10: end for 11: end while Output: Sampled parameters in Berger–Boos set S. 𝑝𝑘 := 118 Chapter 4 UNCERTAINTY QUANTIFICATION OF THE 4TH KIND; OPTIMAL POSTERIOR ACCURACY-UNCERTAINTY TRADEOFF WITH THE MINIMUM ENCLOSING BALL Uncertainty quantification (UQ) is, broadly, the task of determining appropriate uncertainties to model predictions. There are essentially three kinds of approaches to Uncertainty Quantification: (A) robust optimization (min and max), (B) Bayesian (conditional average), and (C) decision theory (minmax). Although (A) is robust, it is unfavorable with respect to accuracy and data assimilation. (B) requires a prior, it is generally non-robust (brittle) with respect to the choice of that prior, and posterior estimations can be slow. Although (C) leads to the identification of an optimal prior, its approximation suffers from the curse of dimensionality and the notion of loss/risk used to identify the prior is one that is averaged with respect to the distribution of the data. We introduce a fourth kind which is a hybrid between (A), (B), (C), and hypothesis testing. It can be summarized as, after observing a sample 𝑥, (1) defining a likelihood region through the relative likelihood and (2) playing a minmax game in that region to define optimal estimators and their risk. The resulting method has several desirable properties: (a) an optimal prior is identified after measuring the data and the notion of loss/risk is a posterior one, (b) the determination of the optimal estimate and its risk can be reduced to computing the minimum enclosing ball of the image of the likelihood region under the quantity of interest map (such computations are fast and do not suffer from the curse of dimensionality). The method is characterized by a parameter in [0, 1] acting as an assumed lower bound on the rarity of the observed data (the relative likelihood). When that parameter is near 1, the method produces a posterior distribution concentrated around a maximum likelihood estimate (MLE) with tight but low confidence UQ estimates. When that parameter is near 0, the method produces a maximal risk posterior distribution with high confidence UQ estimates. In addition to navigating the accuracy-uncertainty tradeoff, the proposed method addresses the brittleness of Bayesian inference by navigating the robustness-accuracy tradeoff associated with data assimilation. 119 4.1 Introduction The past century has seen a steady increase in the need of estimating and predicting complex systems and making (possibly critical) decisions with limited information (H. Owhadi and C. Scovel, 2017c). These decisions are currently being formed based on increasingly complex models with imperfectly known parameters estimated based on available (limited) data whose distribution depends on the unknown/imperfectly known parameters of the model (if the model is well specified, i.e., if the distribution of the data belongs to the parametric family of distributions represented by the model). Making decisions and assessing the risk of these decisions requires identifying methods for data assimilation (estimating the parameters of the model based on data) and quantifying the risk/uncertainties of these decisions/parametric models. Such UQ methods are not unique, and they essentially differ through assumptions made on the generation of the true parameter of the model. In all inference/UQ methods, there is a tradeoff between robustness and accuracy (H. Owhadi and C. Scovel, 2017b), and these assumptions lead to the accuracy of the underlying method when they hold true but also to their lack of robustness when they do not hold true. In this paper, we introduce a new and rigorous UQ method that navigates (in a Pareto optimal manner) this tradeoff between accuracy and robustness in data assimilation and UQ for parametric models. Figure 4.1: The Uncertainty Quantification (UQ) problem. Here, Θ is the space of parameters, 𝜃 † is the true unknown parameter, 𝜑 a quantity of interest, and 𝑃 is the physical model determining the distribution 𝑝 from which the data 𝑥 is observed 120 The problem To describe this method we formalize the underlying UQ problem as follows (see Fig. 4.1). Given a parameter space Θ and a quantity of interest 𝜑 : Θ → 𝑉 we seek to estimate 𝜑(𝜃 † ), where 𝜃 † ∈ Θ is an unknown parameter, based on the observation of some data 𝑥 ∈ 𝑋 sampled from a probability distribution 𝑝(·|𝜃 † ) (given by our model 𝑃) depending on the unknown parameter 𝜃 † . Note that if our goal is to recover 𝜃 † itself, then we can let 𝜑 be the identity function. A simple example (detailed in Sec. 4.1) is to recover the probability 𝜃 † that a coin lands on heads, given the observation 𝑥 = (𝑥 1 , . . . , 𝑥 𝑛 ) ∈ {𝐻, 𝑇 }𝑛 of 𝑛 tosses of that coin. Note that this general setup combines parametric uncertainty (𝜃 † is unknown) with aleatoric uncertainty (the 𝑥 data is a sample from a random variable whose distribution depends on 𝜃 † ), and they need be merged to estimate 𝜑(𝜃 † ) and quantify the uncertainty/risk of the estimation. The three main approaches to UQ There are currently three main approaches (detailed in Sec. 4.2) to addressing this UQ problem. The worst case (robust optimization) approach is (if 𝜑 is realvalued) to compute, the minimum and maximum possible value of 𝜑(𝜃) over all possible values the parameter 𝜃 ∈ Θ. Although the data may be incorporated through empirical distribution inequalities (H. Owhadi and C. Scovel, 2017a), the worst-case approach is conservative and, due to its lack of assumptions on the generation of 𝜃 † , it is at the robust end of the tradeoff between accuracy and robustness. Indeed this approach is simply based on the observation that   𝜑(𝜃 † ) ∈ min 𝜑(𝜃), max 𝜑(𝜃) . (4.1) 𝜃∈Θ 𝜃∈Θ The Bayesian approach is to assume that 𝜃 † is a sample from a prior distribution 𝜋 on Θ, then estimate 𝜑(𝜃 † ) and quantify the uncertainty of that estimation by computing the posterior distribution of 𝜃 † given the data 𝑥. Writing 𝑑 (𝑥) for the estimation of 𝜑(𝜃 † ) (𝑑 : 𝑋 → 𝑉), the Bayesian decision theoretic variant of the Bayesian approach is to introduce a loss/cost   L (𝜃, 𝑑) = E𝑥∼𝑃(·|𝜃) E ∥𝜑(𝜃) − 𝑑 (𝑥)∥ 2 (4.2) for the choice of the estimator 𝑑 if the true value of unknown parameter is 𝜃, assume that 𝜃 † is sampled from a known prior distribution 𝜋 and identify an optimal estimator 𝑑 𝜋 as a minimizer   𝑑 𝜋 = argmin𝑑 E𝜃∼𝜋 L (𝜃, 𝑑) , (4.3) 121   of the 𝜋-averaged loss E𝜃∼𝜋 L (𝜃, 𝑑) , whose value at the minimum defines the risk of that estimator. Due to the strength of the assumption that 𝜃 † is sampled from a known prior distribution, the Bayesian approach is at the accurate end of the tradeoff between accuracy and robustness: in particular, it is brittle to the choice of prior (H. Owhadi and C. Scovel, 2017b; H. Owhadi, C. Scovel, and T. Sullivan, 2015b; H. Owhadi, C. Scovel, and T. Sullivan, 2015a; H. Owhadi and C. Scovel, 2016). The game/decision theoretic approach formulates the underlying UQ problem as a zero-sum game in which 𝜃 is chosen by an adversarial player (Player I) seeking to maximize the loss L (𝜃, 𝑑) and 𝑑 is chosen by Player II seeking to minimize that loss. As in classical game theory (Neumann, 1928), identifying a Nash equilibrium requires lifting this game by letting Player I randomize the selection of 𝜃 according to some mixed strategy/prior distribution 𝜋 on Θ and considering the average loss,   L (𝜋, 𝑑) = E𝜃∼𝜋,𝑥∼𝑃(·|𝜃) E ∥𝜑(𝜃) − 𝑑 (𝑥)∥ 2 , 𝜋 ∈ P (Θ) , 𝑑 : 𝑋 → 𝑉 . (4.4) A saddle point (𝜋 ∗ , 𝑑 𝜋∗ ) for (4.4) is then identified by letting 𝑑 𝜋 be the best Bayesian   response (4.3) to 𝜋 and 𝜋 ∗ be a maximizer of the average-loss E𝜃∼𝜋 L (𝜃, 𝑑 𝜋 ) , i.e.,   𝜋 ∗ ∈ argmax𝜃∈Θ E𝜃∼𝜋 L (𝜃, 𝑑 𝜋 ) . (4.5) Although this approach achieves a balance in the accuracy/robustness tradeoff by relaxing the assumption that 𝜃 is sampled from a known distribution, it does not explicitly enable a navigation of that tradeoff. Furthermore, (1) the numerical approximation of an optimal mixed strategy for Player II suffers from the curse of dimensionality, and (2) 𝑑 𝜋 is the best response to a data-averaged notion of risk rather than a data-given notion of risk. Our new approach to UQ In this paper, we present a new approach that does not suffer from weaknesses present in previous UQ methods such as brittleness and curse of dimensionality and that explicitly navigates the tradeoff between accuracy and robustness in the estimation of the quantity of interest. Motivated by the fact that the main cause of brittleness in inference is the possible rarity of the observed data (H. Owhadi and C. Scovel, 2017b; H. Owhadi, C. Scovel, and T. Sullivan, 2015b; H. Owhadi, C. Scovel, and T. Sullivan, 2015a; H. Owhadi and C. Scovel, 2016), the first step of this approach is to make the hypothesis that the parameter 𝜃 that has generated the data is such that the data is not rare and bound the probability that this hypothesis is false. To describe this, given the observation 𝑥, for 𝛼 ∈ [0, 1] let Θ𝑥 (𝛼) be the set 122 Figure 4.2: Example of the minimum enclosing ball 𝐵 about the image 𝜑(Θ𝑥 (𝛼)) (in Í green) with radius 𝑅 and center 𝑑 = 𝑧∗ . An optimal discrete measure 𝜇 := 𝑤 𝑖 δ𝑧𝑖 (𝑧𝑖 = 𝜑(𝜃 𝑖 )) on the range of 𝜑 for the maximum variance problem is characterized by the fact that it is supported on the intersection of 𝜑(Θ𝑥 (𝛼)) and 𝜕𝐵 and 𝑑 = Í 𝑧∗ = 𝑤 𝑖 𝑧𝑖 is the center of mass of the measure 𝜇. The size of the solid red balls indicates the size of the corresponding weights 𝑤 𝑖 . of parameters 𝜃 ∈ Θ whose relative likelihood 𝑝(𝑥|𝜃) ¯ := 𝑝(𝑥|𝜃) sup𝜃 ′ 𝑝(𝑥|𝜃 ′) (4.6) exceeds the threshold 𝛼, i.e.,  Θ𝑥 (𝛼) := 𝜃 ∈ Θ : 𝑝(𝑥|𝜃) ¯ ≥𝛼 (4.7) and let 𝛽𝛼 be the maximum (over 𝜃 ∈ Θ) probability that 𝜃 does not belong to Θ𝑥 (𝛼) when 𝑥 is randomized according to the model 𝑝(·|𝜃), i.e.,   𝛽𝛼 := sup 𝑃 𝑥 ′ ∈ 𝑋 : 𝜃 ∉ Θ𝑥 ′ (𝛼) 𝜃 . (4.8) 𝜃∈Θ 𝛽𝛼 is interpreted as the significan/p-value of the hypothesis that 𝜃 † ∈ Θ𝑥 (𝛼). In particular, for 𝛼 close to one Θ𝑥 (𝛼) concentrates around the Maximum Likelihood Estimators of 𝜃 † and the probability 𝛽𝛼 that the hypothesis is true goes to zero (which corresponds to accurate side of the tradeoff between accuracy and robustness). For 𝛼 close to zero, Θ𝑥 (𝛼) stretches over the whole set Θ, and the probability 𝛽𝛼 that the hypothesis is true goes to one (which corresponds to the robust side of the tradeoff). The next step of this approach is to employ the game/decision theoretic approach with Θ replaced by the smaller set Θ𝑥 (𝛼), i.e., replace (4.4) with   L (𝜋, 𝑑) = E𝜃∼𝜋,𝑥∼𝑃(·|𝜃) E ∥𝜑(𝜃) − 𝑑 (𝑥)∥ 2 , 𝜋 ∈ P (Θ𝑥 (𝛼)) , 𝑑 : 𝑋 → 𝑉 , (4.9) 123 compute a saddle point (𝜋 𝛼 , 𝑑 𝛼 ) for (4.9) (𝑑 𝛼 = 𝑑 𝜋 𝛼 ), and identify the optimal estimator as 𝑑 𝛼 and its risk/uncertainty R (𝑑 𝛼 ) as the value of the game: R (𝑑 𝛼 ) := L (𝜋 𝛼 , 𝑑 𝛼 ) . (4.10) Main result One of our main results (Theorems 4.3.3 and 4.3.5) is that this optimal decision and its associated risk/uncertainty (defined as the value of the game at the Nash equilibrium) can be identified as the center and the radius of the smallest ball enclosing the image of Θ𝑥 (𝛼) under 𝜑 (see Fig. 4.2). Furthermore, we present rigorous and practical algorithms (Algorithms 7 and 8)1 with approximation accuracy guarantees for computing that minimum enclosing ball based on the observation that optimal mixed strategies (priors) 𝜋 for Player I can be restricted to be supported at a maximum of dim(𝑉) + 1 points located on the boundary of that ball. Coin toss At the cost of some forward referencing, we will now describe an application of our proposed problem to the estimation of the probability that a coin lands on heads based on the observation of 𝑛 independent tosses of that coin. 𝑛 tosses of a single coin. In this example, we estimate the probability that a biased coin lands on heads from the observation of 𝑛 independent tosses of that coin. Specifically, we consider flipping a coin 𝑌 which has an unknown probability 𝜃 † of coming heads (𝑌 = 1) and probability 1 − 𝜃 † coming up tails (𝑌 = 0). Here Θ := [0, 1], 𝑋 = {0, 1}, and the model 𝑃 : Θ → P ({0, 1}) is 𝑃(𝑌 = 1|𝜃) = 𝜃 and 𝑃(𝑌 = 0|𝜃) = 1 − 𝜃. We toss the coin 𝑛 times, generating a sequence of i.i.d. Bernoulli variables (𝑌1 , . . . , 𝑌𝑛 ) all with the same unknown parameter 𝜃 † ∈ [0, 1], and let 𝑥 := (𝑥 1 , . . . , 𝑥 𝑛 ) ∈ {0, 1}𝑛 Í𝑛 𝑥𝑖 denote the number of heads denote the outcome of the experiment. Let ℎ = 𝑖=1 observed and 𝑡 = 𝑛 − ℎ the number of tails. Then the model for the 𝑛-fold toss is 𝑃(𝑥|𝜃) = 𝑛 Ö 𝜃 𝑥𝑖 (1 − 𝜃) 1−𝑥𝑖 = 𝜃 ℎ (1 − 𝜃) 𝑡 (4.11) 𝑖=1 1 Python implementation for these algorithms can be found in https://github.com/JPLMLIA/ UQ4K 124 and, given an observation 𝑥, the MLE is 𝜃 = 𝑛ℎ so that the relative likelihood (4.6) is2 𝜃 ℎ (1 − 𝜃) 𝑡 𝑝(𝑥|𝜃) ¯ = (4.12)
ℎ
𝑡 . ℎ 𝑛 𝑡 𝑛 We seek to estimate 𝜃, so let 𝑉 = R and let the quantity of interest 𝜑 : Θ → 𝑉 be the identity function 𝜑(𝜃) = 𝜃. In this case, given 𝛼 ∈ [0, 1], the likelihood region o 𝜃 ℎ (1 − 𝜃) 𝑡 Θ𝑥 (𝛼) = 𝜃 ∈ [0, 1] :

≥𝛼 ℎ ℎ 𝑡 𝑡 n 𝑛 (4.13) 𝑛 constrains the support of priors to points with relative likelihood larger than 𝛼. Using Theorem 4.3.5 with 𝑚 = 𝑑𝑖𝑚(𝑉) + 1 = 2, one can compute a saddle point (𝜋 𝛼 , 𝑑 𝛼 ) of the game (4.9) as 𝜋 𝛼 = 𝑤δ𝜃 1 + (1 − 𝑤)δ𝜃 2 and 𝑑 𝛼 = 𝑤𝜃 1 + (1 − 𝑤)𝜃 2 , (4.14) where 𝑤, 𝜃 1 , 𝜃 2 maximize the variance    Maximize      over       subject to  𝑤𝜃 12 + (1 − 𝑤)𝜃 22 − (𝑤𝜃 1 + (1 − 𝑤)𝜃 2 ) 2 0 ≤ 𝑤 ≤ 1, 𝜃 𝑖ℎ (1−𝜃 𝑖 ) 𝑡 ℎ 𝑛
ℎ 𝑡
𝑡 ≥ 𝛼, 𝜃 1 , 𝜃 2 ∈ [0, 1] (4.15) 𝑖 = 1, 2 . 𝑛 Equation (4.8) allows us to compute 𝛽 ∈ [0, 1] as a function of 𝛼 ∈ [0, 1]. The solution of the optimization problem can be found by finding the minimum enclosing ball of the set Θ𝑥 (𝛼), which in this 1-D case is also subinterval of the interval [0, 1]. For 𝑛 = 5 tosses resulting in ℎ = 4 heads and 𝑡 = 1 tails, Figure 4.3 plots (1) 𝛽, the relative likelihood, its level sets and minimum enclosing balls as a function of 𝛼, and (2) The risk R (𝑑 𝛼 ) =(4.10) and optimal decision 𝑑 𝛼 as a function of 𝛽. Three different points in the 𝛼 − 𝛽 curve are highlighted. Note that as 𝛼 goes from 0 to 1, the relative likelihood region Θ𝑥 (𝛼) gets smaller (it shrinks towards the MLE), the optimal estimator 𝑑 𝛼 goes from the center of the worst case interval to the MLE estimate, the risk (variance) of the estimator shrinks (which corresponds to an increase in accuracy), but the confidence 1 − 𝛽(𝛼) in that risk (the probability 𝛽(𝛼) that 𝜃 † ∈ Θ𝑥 (𝛼)) also shrinks towards zero (which corresponds to a loss of robustness). 2 Although the fact that 𝑝(𝑥|0) ¯ = 𝑝(𝑥|1) ¯ = 0 violates our positivity assumptions (described in Sec. 4.2) on the model in our framework, in this case this technical restriction can be removed, so we can still use this example as an illustration. 125 Figure 4.3: 𝛼 − 𝛽 relation, likelihood level sets, risk value and decision for different choices of 𝛼 (and consequently 𝛽) for the 1 coin problem after observing four heads and one tail. Three different values in the 𝛼 − 𝛽 curve are highlighted across the plots 𝑛1 and 𝑛2 tosses of two coins. We now consider the same problem with two independent coins with unknown probabilities 𝜃 1† , 𝜃 2† . After tossing each coin 𝑖 𝑛𝑖 times, the observation 𝑥 consists of ℎ𝑖 heads and 𝑡𝑖 tails for each 𝑖, produce a 2D relative likelihood function on Θ = [0, 1] 2 given by 𝜃 1ℎ1 (1 − 𝜃 1 ) 𝑡1 𝜃 2ℎ2 (1 − 𝜃 2 ) 𝑡2 𝑝(𝑥|𝜃 ¯ . 1, 𝜃2) = ℎ1
ℎ1 𝑡 1
𝑡 1 ℎ2
ℎ2 𝑡 2
𝑡 2 𝑛1 𝑛1 𝑛2 (4.16) 𝑛2 Figure 4.4 illustrates the level sets 𝑝(𝑥|𝜃 ¯ 1 , 𝜃 2 ) ≥ 𝛼 and their corresponding bounding balls for ℎ1 = 1, 𝑡1 = 3, ℎ2 = 5, 𝑡2 = 1 and different values of 𝛼 ∈ [0, 1]. Structure of the paper This article is organized as follows: In Sec. 4.2, we formalize the UQ problem and review the three previous approaches to the problem, emphasizing the limitations 126 Figure 4.4: 2D likelihood level sets and minimum enclosing balls for different values of 𝛼, visualized as level sets of the likelihood function (left) and projected onto a 2D plane (right) addressed with our method. In Sec. 4.3, we introduce a new kind of uncertainty quantification based on the minimum enclosing ball, and in Sections 4.4 and 4.5 we introduce the computational framework and our minimum enclosing ball algorithms. Sec. 4.6 presents numerical illustrations of the efficacy and scope of our approach. Sec. 4.7 generalizes the loss and rarity assumptions. Sec. 4.8 presents supporting theorems and proofs. 4.2 Previous approaches to UQ We begin by formalizing the UQ problem introduced in the previous section. Let 𝜑 : Θ → 𝑉 be a quantity of interest, where 𝑉 (the space of predictions) is a finitedimensional vector space and Θ (the space of parameters) is a compact set. Let 𝑋 (the space of data) be a measurable space and write P (𝑋) for the set of probability distributions on 𝑋. Consider a model 𝑃 : Θ → P (𝑋) representing the dependence of the distribution of a data point 𝑥 ∼ 𝑃(·|𝜃) on the value of the parameter 𝜃 ∈ Θ. Throughout, we use ∥ · ∥ to denote the Euclidean norm. We are then interested in solving the following problem. Problem 1. Let 𝜃 † be an unknown element of Θ. Given an observation 𝑥 ∼ 𝑃(·|𝜃 † ) of data, estimate 𝜑(𝜃 † ) and quantify the uncertainty (accuracy/risk) of the estimate. We assume that we can write a probability density function for any 𝑃(·|𝜃). More 127 formally, we assume that 𝑃 is a dominated model with positive densities, that is, for each 𝜃 ∈ Θ, 𝑃(·|𝜃) is defined by a (strictly) positive density 𝑝(·|𝜃) : 𝑋 → R>0 with respect to a measure 𝜈 ∈ P (𝑋), such that, for each measurable subset 𝐴 of 𝑋, ∫ 𝑃( 𝐴|𝜃) = 𝑝(𝑥 ′ |𝜃)𝑑𝜈(𝑥 ′), 𝜃 ∈ Θ. (4.17) 𝐴 The three main approaches to UQ Problem 1 is a fundamental Uncertainty Quantification (UQ) problem, and there are essentially three main approaches for solving it. We now describe them when 𝑉 is a Euclidean space with the ℓ 2 loss function. Worst-case In a different setting, essentially where the set Θ consists of probability measures, the OUQ framework (H. Owhadi, C. Scovel, T. J. Sullivan, et al., 2013b) provides a worst-case analysis for providing rigorous uncertainty bounds. In the setting of this paper, in the absence of data (or ignoring the data 𝑥), the (vanilla) worst-case (or robust optimization) answer is to estimate 𝜑(𝜃 † ) with the minimizer 𝑑 ∗ ∈ 𝑉 of the worst-case error   R (𝑑) := max ∥𝜑(𝜃) − 𝑑 ∥ 2 . (4.18) 𝜃∈Θ ∗ ∗ In that approach, (𝑑 , R (𝑑 )) are therefore identified as the center and squared radius of the minimum enclosing ball of 𝜑(Θ). Bayesian The (vanilla) Bayesian (decision theory) approach (see e.g. Berger (J. O. Berger, 2013, Sec. 4.4)) is to assume that 𝜃 is sampled from a prior distribution 𝜋 ∈ P (Θ), and approximate 𝜑(𝜃 † ) with the minimizer 𝑑 𝜋 (𝑥) ∈ 𝑉 of the Bayesian posterior risk   R 𝜋 (𝑑) := E𝜃∼𝜋 𝑥 ∥𝜑(𝜃) − 𝑑 ∥ 2 , 𝑑 ∈ 𝑉, (4.19) associated with the decision 𝑑 ∈ 𝑉, where 𝑝(𝑥|·)𝜋 𝑝(𝑥|𝜃)𝑑𝜋(𝜃) Θ 𝜋𝑥 := ∫ (4.20) is the posterior measure determined by the likelihood 𝑝(𝑥|·), the prior 𝜋 and the observation 𝑥. The minimizer 𝑑 𝜋 (𝑥) of (4.19) is the posterior distribution mean 𝑑 𝜋 (𝑥) := E𝜃∼𝜋 𝑥 [𝜑(𝜃)] (4.21) 128 and the uncertainty is quantified by the posterior variance   R 𝜋 (𝑑 𝜋 (𝑥)) := E𝜃∼𝜋 𝑥 ∥𝜑(𝜃) − 𝑑 𝜋 (𝑥)∥ 2 . (4.22) Game/decision theoretic The Wald’s game/decision theoretic approach is to consider a two-player zerosum game where player I selects 𝜃 ∈ Θ, and player II selects a decision function 𝑑 : 𝑋 → 𝑉 which estimates the quantity of interest 𝜑(𝜃) (given the data 𝑥 ∈ 𝑋), resulting in the loss   L (𝜃, 𝑑) := E𝑥∼𝑃(·|𝜃) ∥𝜑(𝜃) − 𝑑 (𝑥)∥ 2 , 𝜃 ∈ Θ, 𝑑 : 𝑋 → 𝑉, (4.23) for player II. Such a game will normally not have a saddle point, so following von Neumann’s approach (Neumann, 1928), one randomizes both players’ plays to identify a Nash equilibrium. To that end, first observe that, for the quadratic loss considered here (for ease of presentation), because of the convexity of the loss in 𝑑, only the choice of player I needs to be randomized. Letting 𝜋 ∈ P (Θ) be a probability measure randomizing the play of player I, we consider the lift   L (𝜋, 𝑑) := E𝜃∼𝜋 E𝑥∼𝑃(·|𝜃) ∥𝜑(𝜃) − 𝑑 (𝑥)∥ 2 , 𝜋 ∈ P (Θ), 𝑑 : 𝑋 → 𝑉, (4.24) of the game (4.23). A minmax optimal estimate of 𝜑(𝜃 † ) is then obtained by identifying a Nash equilibrium (a saddle point) for (4.24), i.e. 𝜋 ∗ ∈ P (Θ) and 𝑑 ∗ : 𝑋 → 𝑉 satisfying L (𝜋, 𝑑 ∗ ) ≤ L (𝜋 ∗ , 𝑑 ∗ ) ≤ L (𝜋 ∗ , 𝑑), 𝜋 ∈ P (Θ), 𝑑 : 𝑋 → 𝑉 . (4.25) Consequently, an optimal strategy of player II is then the posterior mean 𝑑 𝜋∗ (𝑥) of the form (4.21) determined by a worst-case measure and optimal randomized/mixed strategy for player I   𝜋 ∗ :∈ arg max E𝜃∼𝜋,𝑥∼𝑃(·,𝜃) ∥𝜑(𝜃) − 𝑑 𝜋 (𝑥)∥ 2 . (4.26) 𝜋∈P (Θ) To connect with the Bayesian framework we observe (by changing the order of integration) that the Wald’s risk (4.24) can be written as the average   L (𝜋, 𝑑) := E𝑥∼𝑋 𝜋 R 𝜋 (𝑑 (𝑥)) (4.27) of the Bayesian decision risk R 𝜋 (𝑑 (𝑥)) (=(4.19) for 𝑑 = 𝑑 (𝑥)) determined by the prior 𝜋 and decision 𝑑 (𝑥) with respect to the 𝑋-marginal distribution ∫ 𝑋𝜋 := 𝑃(·|𝜃)𝑑𝜋(𝜃) (4.28) Θ 129 associated with the prior 𝜋 and the model 𝑃. Therefore, the Wald framework identifies a worst-case prior (4.26), while the prior used in Bayesian decision theory is specified by the practitioner. Figure 4.5: Curse of dimensionality in discretizing the prior. The data is of the form 𝑥 = 𝑚(𝜃) + 𝜖N (0, 1) where 𝑚 is deterministic and 𝜖N (0, 1) is small noise. (a) For the continuous prior, the posterior concentrates around M := {𝜃 ∈ Θ|𝑚(𝜃) = 𝑥}. (b) For the discretized prior, the posterior concentrates on the delta Dirac that is the closest to M. Limitations of the three main approaches to UQ All three approaches described in Section 4.2 have limitations in terms of accuracy, robustness, and computational complexity. Although the worst-case approach is robust, it appears unfavorable in terms of accuracy and data assimilation. The Bayesian approach, on the other hand, suffers from the computational complexity of estimating the posterior distribution and from brittleness (H. Owhadi, C. Scovel, and T. Sullivan, 2015b) with respect to the choice of prior along with Stark’s admonition (P. Stark, 2020) “your prior can bite you on the posterior.” Although Kempthorne (Kempthorne, 1987) develops a rigorous numerical procedure with convergence guarantees for solving the equations of Wald’s statistical decision theory which appears amenable to computational complexity analysis, it suffers from the curse of dimensionality (see Fig. 4.5). This can be understood from the fact that the risk associated with the worst-case measure in the Wald framework is an average over the observational variable 𝑥 ∈ 𝑋 of the conditional risk, conditioned on the observation 𝑥. Consequently, for a discrete approximation of a worst-case measure, after 130 an observation is made, there may be insufficient mass near the places where the conditioning will provide a good estimate of the appropriate conditional measure. Indeed, in the proposal (H. Owhadi and C. Scovel, 2017c) to develop Wald’s statistical decision theory along the lines of Machine Learning, with its dual focus on performance and computation, it was observed that “Although Wald’s theory of Optimal Statistical Decisions has resulted in many important statistical discoveries, looking through the three Lehmann symposia of Rojo and Pérez-Abreu (Rojo and Pérez-Abreu, 2004) in 2004, and Rojo (Rojo, 2006; Rojo, 2009) in 2006 and 2009, it is clear that the incorporation of the analysis of the computational algorithm, both in terms of its computational efficiency and its statistical optimality, has not begun.” Moreover, one might ask why, after seeing the data, one is choosing a worst-case measure which optimizes the average (4.27) of the Bayesian risk (4.22), instead of choosing it to optimize the value of the risk R 𝜋 (𝑑 𝜋 (𝑥)) at the value of the observation 𝑥. It is therefore desirable for an approach to UQ to successfully assimilate the observed data, to avoid requiring having to manually select a prior and to have a data-dependent notion of risk. In Section 4.3, we will propose a framework with all these properties. A comparison of the properties of all the mentioned methods can be found in Table 4.1. Worst case Bayesian Decision Theory UQ4K (Section 4.3) Makes use of the observed data × ✓ ✓ ✓ No need to manually specify prior ✓ × ✓ ✓ Risk depends on the observed data × ✓ × ✓ Table 4.1: Comparison of the three previous approaches to uncertainty quantification with our proposed method in Section 4.3 4.3 Uncertainty Quantification of the 4th Kind Basic definitions In this paper, we introduce a framework which is a hybrid between Wald’s statistical decision theory (A. Wald and Wolfowitz, 1951), Bayesian decision theory (J. O. Berger, 2013, Sec. 4.4), robust optimization, and hypothesis testing. Here we de- 131 scribe its components for simplicity when the loss function is the ℓ 2 loss. Later in Section 4.7 we develop the framework for general loss functions. Rarity assumption on the data In (H. Owhadi, C. Scovel, and T. Sullivan, 2015b, Pg. 576) it was demonstrated that one could alleviate the brittleness of Bayesian inference (see (H. Owhadi, C. Scovel, and T. Sullivan, 2015a; H. Owhadi and C. Scovel, 2016)) by restricting to priors 𝜋 for which the observed data 𝑥 is not rare, that is, ∫ 𝑝(𝑥) := 𝑝(𝑥|𝜃)𝑑𝜋(𝜃) ≥ 𝛼 (4.29) Θ according to the density of the 𝑋-marginal determined by 𝜋 and the model 𝑃, for some 𝛼 > 0. In the proposed framework, we consider playing a game after observing the data 𝑥 whose loss function is defined by the Bayesian decision risk R 𝜋 (𝑑) (4.19), where player I selects a prior 𝜋 subject to a rarity assumption (𝜋 ∈ P𝑥 (𝛼)) and player II selects a decision 𝑑 ∈ 𝑉. The rarity assumption considered here is n o  P𝑥 (𝛼) := 𝜋 ∈ P (Θ) : support(𝜋) ⊂ 𝜃 ∈ Θ : 𝑝(𝑥|𝜃) ≥ 𝛼 . (4.30) Since 𝑝(𝑥|𝜃) ≥ 𝛼 for all 𝜃 in the support of any 𝜋 ∈ P𝑥 (𝛼) it follows that such a 𝜋 satisfies (4.29) and therefore is sufficient to prevent Bayesian brittleness. The relative likelihood for the rarity assumption Observe in (4.20) that the map from the prior 𝜋 to posterior 𝜋𝑥 is scale-invariant in the likelihood 𝑝(𝑥|·) and that the effects of scaling the likelihood in the rarity assumption can be undone by modifying 𝛼. Consequently, we scale the likelihood function 𝑝(𝑥|𝜃) 𝑝(𝑥|𝜃) ¯ := , 𝜃 ∈ Θ, (4.31) sup𝜃∈Θ 𝑝(𝑥|𝜃) to its relative likelihood function 𝑝(𝑥|·) ¯ : Θ → (0, 1] . (4.32) According to Sprott (Sprott, 2008, Sec. 2.4), the relative likelihood measures the plausibility of any parameter value 𝜃 relative to a maximum likely 𝜃 and summarizes the information about 𝜃 contained in the sample 𝑥. See Rossi (Rossi, 2018, p. 267) for its large sample connection with the X12 distribution and several examples of the relationship between likelihood regions and confidence intervals. 132 For 𝑥 ∈ 𝑋 and 𝛼 ∈ [0, 1], let (4.7) denote the corresponding likelihood region and, updating (4.30), redefine the rarity assumption by
P𝑥 (𝛼) := P Θ𝑥 (𝛼) . (4.33) That is, the rarity constraint P𝑥 (𝛼) constrains priors to have support on the likelihood region Θ𝑥 (𝛼). We will now define the confidence level of the family Θ𝑥 (𝛼), 𝑥 ∈ 𝑋. Significance/confidence level For a given 𝛼, let the significance 𝛽𝛼 at the value 𝛼 be the maximum (over 𝜃 ∈ Θ) of the probability that a data 𝑥 ′ ∼ 𝑃(·|𝜃) does not satisfy the rarity assumption 𝑝(𝑥 ¯ ′ |𝜃) ≥ 𝛼, i.e., ∫ 𝛽𝛼 := sup 1{ 𝑝(·|𝜃)<𝛼} (𝑥 ′) 𝑝(𝑥 ′ |𝜃)𝑑𝜈(𝑥 ′) , (4.34) ¯ 𝜃∈Θ where, for fixed 𝜃, 1{ 𝑝(·|𝜃)<𝛼} is the indicator function of the set {𝑥 ′ ∈ 𝑋 : 𝑝(𝑥 ¯ ′ |𝜃) < ¯ 𝛼}. Observe that, in the setting of hypothesis testing, (1) 𝛽𝛼 can be interpreted as the p-value associated with the hypothesis that the rarity assumption is not satisfied (i.e. the hypothesis that 𝜃 does not belongs to the set (4.7)), and (2) 1 − 𝛽𝛼 can be interpreted as the confidence level associated with the rarity assumption (i.e. the smallest probability that 𝜃 belongs to the set (4.7)). Therefore, to select 𝛼 ∈ [0, 1], we set a significance level 𝛽∗ (e.g. 𝛽∗ = 0.05) and choose 𝛼 to be the largest value such that the significance at 𝛼 satisfies 𝛽𝛼 ≤ 𝛽∗ . Connection to the Likelihood Ratio Test The connection of the likelihood region we are defining and confidence sets can be more explicitly seen via the likelihood ratio test and the inversion of that test to produce a confidence set. Namely, we define a hypothesis test, 𝐻0 : 𝜃 = 𝜃 0 versus 𝐻1 : 𝜃 ≠ 𝜃 0 , 𝜃 ∈ Θ. (4.35) The likelihood ratio test statistic is defined by Λ(𝜃 0 , 𝑥) = 𝑝(𝑥|𝜃 0 ) . sup𝜃∈Θ 𝑝(𝑥|𝜃) (4.36) Note, Λ(𝜃 0 , 𝑥) = 𝑝(𝑥|𝜃), ¯ i.e., Equation (4.31) and Equation (4.36) are equivalent. In the hypothesis testing setting, one would next define a critical value, 𝑐 > 0, such that when Λ(𝜃 0 , 𝑥) ≤ 𝑐, the null in test (4.35) is rejected. Ideally, 𝑐 is chosen such 133 that the probability of false rejection under the null hypothesis is capped at some probability, 𝛼 ∈ (0, 1), defining an 𝛼-level test. The type 1 error probability control is mathematically characterized as 𝑃 (Λ(𝜃 0 , 𝑥) ≤ 𝑐|𝜃 0 ) ≤ 𝛼. (4.37) The error control criterion of Equation (4.37) can be used to define an acceptance region in the sample space defined as follows: 𝐴(𝜃 0 , 𝑐) = {𝑥 ∈ 𝑋 : Λ(𝜃 0 , 𝑥) ≥ 𝑐}. (4.38) The type 1 error control then implies 𝑃(𝑥 ∈ 𝐴(𝜃 0 , 𝑐)|𝜃 0 ) ≥ 1 − 𝛼 and acts in a similar way to the control exerted by 𝛽𝛼 in Equation (4.34). In that equation, 𝛼 is chosen as the supremum over 𝜃 ∈ Θ of 𝑃 (Λ(𝜃, 𝑥) ≤ 𝑐|𝜃), thus making 𝛼 a constant independent of 𝜃. A viable alternative is to consider a curve 𝛼(𝜃), where 𝑃 (Λ(𝜃 0 , 𝑥) ≤ 𝑐|𝜃 0 ) ≤ 𝛼(𝜃 0 ) ∀ 𝜃0, (4.39) and then redefining (4.7) accordingly. In this work we consider the fixed 𝛼 model for simplicity. In some cases, the false rejection control can be done cleanly. For instance, in the context of a noise model where 𝑥 = 𝜃 + 𝜀, 𝜀 ∼ 𝑁 (0, 𝐼), 𝜃 ∈ R𝑛 , it can be shown that the log-likelihood ratio follows the distribution, −2 log Λ(𝜃 0 , 𝑥) ∼ 𝜒𝑛2 , (4.40) i.e., the log-likelihood ratio is distributed as a chi-squared distribution with 𝑛 degrees of freedom, allowing 𝑐 to be exactly chosen. In the event the test statistic distribution cannot be exactly known, asymptotic results such as that shown in Theorem 4.4.1 can provide similar results. As discussed in Chapter 9 of Casella/Berger (G. Casella and R. L. Berger, 2002), one can think about inverting an 𝛼-level hypothesis test such as Test (4.35) to obtain a 1 − 𝛼 confidence set 𝐶 (𝑥) ⊂ Θ, such that 𝑃 (𝜃 ∗ ∈ 𝐶 (𝑥)|𝜃 ∗ ) ≥ 1 − 𝛼, where 𝜃 ∗ is the true parameter value. The inverting is performed with the acceptance region of Equation (4.38) and the set is defined as follows: 𝐶 (𝑥) = {𝜃 0 ∈ Θ : 𝑥 ∈ 𝐴(𝜃 0 , 𝑐)}, (4.41) Note the equivalence between Equation (4.41) directly above and Equation (4.7) from the previous section. By the type 1 error control, we have 𝑃(𝜃 ∗ ∈ 𝐶 (𝑥)|𝜃 ∗ ) = 𝑃(𝑥 ∈ 𝐴(𝜃 ∗ , 𝑐)|𝜃 ∗ ) ≥ 1 − 𝛼, (4.42) 134 implying, 𝑃(𝜃 ∗ ∉ 𝐶 (𝑥)|𝜃 ∗ ) ≤ 𝛼, (4.43) providing an additional view of the equivalence with Equation (4.34). As such, the relative likelihood and rarity condition can be seen through the more traditional statistical lens of the likelihood ratio and type 1 error control in the classical hypothesis testing setting. Remark 4.3.1. For models where the maximum of the likelihood function 𝑀 (𝑥 ′) := sup 𝑝(𝑥 ′ |𝜃), 𝑥 ′ ∈ 𝑋, 𝜃∈Θ is expensive to compute but for which there exists an efficiently computable upper approximation 𝑀 ′ (𝑥 ′) ≥ 𝑀 (𝑥 ′), 𝑥 ′ ∈ 𝑋 available, the surrogate 𝑝(𝑥 ′ |𝜃) 𝑝¯ (𝑥 |𝜃) := ′ ′ , 𝑀 (𝑥 ) ′ 𝑥 ′ ∈ 𝑋, ′ (4.44) to the relative likelihood may be used in place of (4.31). If we let 𝛽′𝛼 denote the value determined in (4.34) using the surrogate (4.44) and Θ′𝑥 (𝛼) denote the corresponding likelihood region, then we have 𝛽𝛼 ≤ 𝛽′𝛼 and Θ′𝑥 (𝛼) ⊂ Θ𝑥 (𝛼), 𝛼 ∈ [0, 1]. Consequently, obtaining 𝛽′𝛼 ≤ 𝛽∗ for significance level 𝛽∗ implies that 𝛽𝛼 ≤ 𝛽∗ . As an example, for an 𝑁-dimensional Gaussian model with 𝑝(𝑥 ′ |𝜃) = 1 with Θ := [−𝜏, 𝜏] 𝑁 , the elementary upper bound 𝑀 (𝑥 ′) := sup 𝑝(𝑥|𝜃) ≤ 𝜃∈Θ 1 √ (𝜎 2𝜋) 𝑁 the surrogate relative likelihood defined in (4.44) becomes − 1 ∥𝑥 ′ −𝜃 ∥ 2 𝑝¯′ (𝑥 ′ |𝜃) := 𝑒 2𝜎2 . Posterior game and risk After observing 𝑥 ∈ 𝑋, we now consider playing a game using the loss   L (𝜋, 𝑑) := E𝜃∼𝜋 𝑥 ∥𝜑(𝜃) − 𝑑 ∥ 2 , ′ − ∥𝑥 −𝜃 ∥ √1 𝑒 2𝜎2 (𝜎 2𝜋) 𝑁 𝜋 ∈ P𝑥 (𝛼), 𝑑 ∈ 𝑉, (4.45) where 𝜋𝑥 is the posterior (4.20). In (4.45), we think about the maximizing player as choosing the prior 𝜋 and then the loss function depends on the posterior 𝜋𝑥 . Since the likelihood 𝑝(𝑥|·) is positive and the data 𝑥 is fixed, we have support(𝜋𝑥 ) = 2 135 support(𝜋) and the map (4.20) mapping the prior 𝜋 to the posterior 𝜋𝑥 is bijective. Therefore one can equivalently consider the choice of the maximizing player to be directly maximizing the posterior 𝜋𝑥 instead of the prior 𝜋 that is later mapped to the posterior using the data. The optimal choices of these two games can then be mapped by (4.20) and its inverse. Using the invariance of posterior (4.20) under the scaling of the likelihood function 𝑝(𝑥|·) we write the posterior in terms of the relative likelihood (4.31) as 𝑝(𝑥|·)𝜋 ¯ . 𝑝(𝑥|𝜃)𝑑𝜋(𝜃) ¯ Θ 𝜋𝑥 := ∫ (4.46) Therefore for simplicity one directly considers a game using the loss   L (𝜋, 𝑑) := E𝜃∼𝜋 ∥𝜑(𝜃) − 𝑑 ∥ 2 , 𝜋 ∈ P𝑥 (𝛼), 𝑑 ∈ 𝑉 . (4.47) Recall that a pair (𝜋 𝛼 , 𝑑 𝛼 ) ∈ P𝑥 (𝛼) × 𝑉 is a saddle point of the game (4.47) if L (𝜋, 𝑑 𝛼 ) ≤ L (𝜋 𝛼 , 𝑑 𝛼 ) ≤ L (𝜋 𝛼 , 𝑑), 𝜋 ∈ P𝑥 (𝛼), 𝑑 ∈ 𝑉 . We then have the following theorem. Theorem 4.3.2. Consider 𝑥 ∈ 𝑋, 𝛼 ∈ [0, 1], and suppose that the relative likelihood 𝑝(𝑥|·) and the quantity of interest 𝜑 : Θ → 𝑉 are continuous. The loss function L for the game (4.9) (=(4.86)) has saddle points and a pair (𝜋 𝛼 , 𝑑 𝛼 ) ∈ P𝑥 (𝛼) × 𝑉 is a saddle point for L if and only if 𝑑 𝛼 := E𝜋 𝛼 [𝜑] (4.48)   𝜋 𝛼 ∈ arg max E𝜋 ∥𝜑 − E𝜋 [𝜑] ∥ 2 . (4.49) and 𝜋∈P𝑥 (𝛼) Furthermore the associated risk (the value of the two person game (4.9) (=(4.86)) )   R (𝑑 𝛼 ) := L (𝜋 𝛼 , 𝑑 𝛼 ) = E𝜋 𝛼 ∥𝜑 − E𝜋 𝛼 [𝜑] ∥ 2 (4.50) is the same for all saddle points of L. Moreover, the second component 𝑑 𝛼 of the set of saddle points is unique and the set O𝑥 (𝛼) ⊂ P𝑥 (𝛼) of first components of saddle points is convex, providing a convex ridge O𝑥 (𝛼) × {𝑑 𝛼 } of saddle points. Duality with the minimum enclosing ball Although the Lagrangian duality between the maximum variance problem and the minimum enclosing ball problem on finite sets is known, see Yildirim (Yildirim, 136 2008), we now analyze the infinite case. Utilizing the recent generalization of the one-dimensional result of Popoviciu (Popoviciu, 1935) regarding the relationship between variance maximization and the minimum enclosing ball by Lim and McCann (Lim and McCann, 2021, Thm. 1), the following theorem demonstrates that essentially the maximum variance problem (4.49) determining a worst-case measure is the Lagrangian dual of the minimum enclosing ball problem on the image
𝜑(Θ𝑥 (𝛼)). Let 𝜑∗ : P Θ𝑥 (𝛼) → P (𝜑(Θ𝑥 (𝛼)) denote the pushforward map (change of variables) defined by (𝜑∗ 𝜋)( 𝐴) := 𝜋(𝜑−1 ( 𝐴)) for every Borel set 𝐴, mapping
probability measures on Θ𝑥 (𝛼) to probability measures on 𝜑 Θ𝑥 (𝛼) . Theorem 4.3.3. For 𝑥 ∈ 𝑋, 𝛼 ∈ [0, 1], suppose the relative likelihood 𝑝(𝑥|·) ¯ and the quantity of interest 𝜑 : Θ → 𝑉 are continuous. Consider a saddle point (𝜋 𝛼 , 𝑑 𝛼 ) of the game (4.9) (=(4.86)). The optimal decision 𝑑 𝛼 and its associated risk R (𝑑 𝛼 ) =(4.10) are equal to the center and squared radius, respectively, of the minimum enclosing ball of 𝜑(Θ𝑥 (𝛼)), i.e. the minimizer 𝑧∗ and the value 𝑅 2 of the minimum enclosing ball optimization problem    Minimize 𝑟 2     Subject to 𝑟 ∈ R, 𝑧 ∈ 𝜑(Θ𝑥 (𝛼)),      ∥𝑥 − 𝑧∥ 2 ≤ 𝑟 2 , 𝑥 ∈ 𝜑(Θ𝑥 (𝛼)).  (4.51) Moreover, the variance maximization problem on P𝑥 (𝛼) (4.49) pushes forward to the variance maximization problem on the image of the likelihood region P (𝜑(Θ𝑥 (𝛼))) under 𝜑, giving the identity     E𝜋 ∥𝜑 − E𝜋 [𝜑] ∥ 2 = E𝜑∗ 𝜋 ∥𝑣 − E𝜋′ [𝑣] ∥ 2 , 𝜋 ∈ P𝑥 (𝛼), and the latter is the Lagrangian dual to the minimum enclosing ball problem (4.51) on the image 𝜑(Θ𝑥 (𝛼)). Finally, let 𝐵, with center 𝑧∗ , denote the minimum enclosing ball of 𝜑(Θ𝑥 (𝛼)). Then a measure 𝜋 𝛼 ∈ P𝑥 (𝛼) is optimal for the variance maximization problem (4.49) if and only if
𝜑∗ 𝜋 𝛼 𝜑(Θ𝑥 (𝛼)) ∩ 𝜕𝐵 = 1 and ∗ 𝑧 = ∫ 𝑣𝑑 (𝜑∗ 𝜋 𝛼 )(𝑣), 𝑉 𝛼 that is, all the mass of 𝜑∗ 𝜋 lives on the intersection 𝜑(Θ𝑥 (𝛼)) ∩ 𝜕𝐵 of the image 𝜑(Θ𝑥 (𝛼)) of the likelihood region and the boundary 𝜕𝐵 of its minimum enclosing ball and the center of mass of the measure 𝜑∗ 𝜋 𝛼 is the center 𝑧∗ of 𝐵. 137 Remark 4.3.4. Note that once 𝛼, and therefore Θ𝑥 (𝛼), is determined that the computation of the risk and the minmax estimator is determined by the minimum enclosing ball about 𝜑(Θ𝑥 (𝛼)), which is also determined by the worst-case optimization problem (4.18) for Θ := Θ𝑥 (𝛼). Theorem 4.3.3 introduces the possibility of primal-dual algorithms, in particular, the availability of rigorous stopping criteria for the maximum variance problem (4.49).   To that end, for a feasible measure 𝜋 ∈ P𝑥 (𝛼), let Var(𝜋) := E𝜋 ∥𝜑 − E𝜋 [𝜑] ∥ 2 denote its variance and denote by Var∗ := sup𝜋∈P𝑥 (𝛼) Var(𝜋) =(4.50) the optimal variance. Let (𝑟, 𝑧) be feasible for the minimum enclosing ball problem (4.51). Then the inequality Var∗ = 𝑅 2 ≤ 𝑟 2 implies the rigorous bound Var∗ − Var(𝜋) ≤ 𝑟 2 − Var(𝜋) (4.52) quantifying the suboptimality of the measure 𝜋 in terms of known quantities 𝑟 and Var(𝜋). Finite-dimensional reduction Let Δ𝑚 (Θ) denote the set of convex sums of 𝑚 Dirac measures located in Θ and, let P𝑥𝑚 (𝛼) ⊂ P𝑥 (𝛼) defined by P𝑥𝑚 (𝛼) := Δ𝑚 (Θ) ∩ P𝑥 (𝛼) (4.53) denote the finite-dimensional subset of the rarity assumption set P𝑥 (𝛼) consisting of the convex combinations of 𝑚 Dirac measures supported in Θ𝑥 (𝛼). Theorem 4.3.5. Let 𝛼 ∈ [0, 1] and 𝑥 ∈ 𝑋, and suppose that the likelihood function 𝑝(𝑥|·) and quantity of interest 𝜑 : Θ → 𝑉 are continuous. Then for any 𝑚 ≥ 𝑑𝑖𝑚(𝑉) + 1, the variance maximization problem (4.49) has the finite-dimensional reduction     2 max E𝜋 ∥𝜑 − E𝜋 [𝜑] ∥ 2 = max E ∥𝜑 − E [𝜑] ∥ . (4.54) 𝜋 𝜋 𝑚 𝜋∈P𝑥 (𝛼) 𝜋∈P𝑥 (𝛼) Therefore one can compute a saddle point (𝑑 𝛼 , 𝜋 𝛼 ) of the game (4.9) (=(4.86)) as 𝑚 𝑚 ∑︁ ∑︁ 𝛼 𝛼 𝜋 = 𝑤 𝑖 δ𝜃 𝑖 and 𝑑 = 𝑤 𝑖 𝜑(𝜃 𝑖 ), (4.55) 𝑖=1 𝑖=1 where 𝑤 𝑖 ≥ 0, 𝜃𝑖 ∈ Θ, 𝑖 = 1, . . . , 𝑚 maximize Í𝑚 Í𝑚  2 2   Maximize 𝑖=1 𝑤 𝑖 𝜑(𝜃 𝑖 ) − 𝑖=1 𝑤 𝑖 𝜑(𝜃 𝑖 )     Í𝑚 Subject to 𝑤 𝑖 ≥ 0, 𝜃𝑖 ∈ Θ, 𝑖 = 1, . . . , 𝑚, 𝑖=1 𝑤𝑖 = 1      𝑝(𝑥|𝜃 ¯ 𝑖 = 1, . . . , 𝑚 . 𝑖 ) ≥ 𝛼,  (4.56) 138 As a consequence of Theorems 4.3.3 and 4.3.5, a measure with finite support Í 𝜇 := 𝑤 𝑖 δ𝑧𝑖 on 𝑉 is the pushforward under 𝜑 : Θ → 𝑉 of an optimal measure 𝜋 𝛼 for the maximum variance problem (4.49) if and only if, as illustrated in Figure 4.2, it is supported on the intersection of 𝜑(Θ𝑥 (𝛼)) and the boundary 𝜕𝐵 of the minimum enclosing ball of 𝜑(Θ𝑥 (𝛼)) and the center 𝑧∗ of 𝐵 is the center of mass Í 𝑧∗ = 𝑤 𝑖 𝑧𝑖 of the measure 𝜇. Relaxing MLE with an accuracy/robustness tradeoff For fixed 𝑥 ∈ 𝑋, assume that the model 𝑃 is such that the maximum likelihood estimate (MLE) 𝜃 ∗ := arg max 𝑝(𝑥|𝜃) (4.57) 𝜃∈Θ of 𝜃 † exists and is unique. Observe that for 𝛼 near one (1) the support of 𝜋 𝛼 and 𝑑 𝛼 concentrate around the MLE 𝜃 ∗ and 𝜑(𝜃 ∗ ), (2) the risk R (𝑑 𝛼 ) =(4.10) concentrates around zero, and (3) the confidence 1 − 𝛽𝛼 associated with the rarity assumption 𝜃 † ∈ Θ𝑥 (𝛼) is the smallest. In that limit, our estimator inherits the accuracy and lack of robustness of the MLE approach to estimating the quantity of interest. Conversely for 𝛼 near zero, since by (4.7) Θ𝑥 (𝛼) ≈ Θ, (1) the support of the pushforward of 𝜋 𝛼 by 𝜑 concentrates on the boundary of 𝜑(Θ) and 𝑑 𝛼 concentrate around the center of the minimum enclosing ball of 𝜑(Θ), (2) the risk R (𝑑 𝛼 ) =(4.50) is the highest and concentrates around the worst-case risk (4.18), and (3) the confidence 1 − 𝛽𝛼 associated with the rarity assumption 𝜃 † ∈ Θ𝑥 (𝛼) is the highest. In that limit, our estimator inherits the robustness and lack of accuracy of the worst-case approach to estimating the quantity of interest. For 𝛼 between 0 and 1, the proposed game-theoretic approach induces a minmax optimal tradeoff between the accuracy of MLE and the robustness of the worst case. 4.4 Computational framework The introduction developed this framework in the context of a model 𝑃 with density 𝑝 in terms of a single sample 𝑥. In Section 4.1, the single sample case was extended to 𝑁 i.i.d. samples by defining the multisample D := (𝑥1 , . . . , 𝑥 𝑁 ) and 𝑁 𝑝(𝑥 |𝜃). Extensions incorporatdefining the product model density 𝑝(D|𝜃) := Π𝑖=1 𝑖 ing correlations in the samples, such as Markov or other stochastic processes can easily be developed. Here we continue this development for the general model of the introduction for the ℓ 2 loss and also develop more fully a Gaussian noise model. 139 Later, in Sections 4.6 and 4.6 these models will be tested on estimating a quadratic function and a Lotka-Volterra predator-prey model based on noisy observations. In Section 4.7 the framework will be generalized to more general loss functions and rarity assumptions, which much of the current section generalizes to. Let the possible states of nature be a compact subset Θ ⊂ R 𝑘 , the decision space be 𝑉 := R𝑛 and the elements of the 𝑁-fold multisample D := (𝒙1 , . . . , 𝒙 𝑁 ) lie in 𝑋, that is, D lies in the multisample space 𝑋 𝑁 . Let the 𝑛 components of the quantity of interest 𝜑 : Θ → R𝑛 be indicated by 𝜑𝑡 : Θ → R, 𝑡 = 1, . . . 𝑛. 𝑁 𝑝(𝑥 |𝜃), Here, using the i.i.d. product model 𝑃(·|𝜃) with density 𝑝(D|𝜃) := Π𝑖=1 𝑖 the definition of 𝛽𝛼 in (4.34) becomes   ′ 𝑁 ′ 𝛽𝛼 := sup 𝑃 D ∈ 𝑋 : 𝜃 ∉ ΘD (𝛼) 𝜃 𝜃∈Θ ∫ = sup 1{ 𝑝(·|𝜃)<𝛼} (D ′) 𝑝(D ′ |𝜃)𝑑𝜈 𝑁 (D ′) , (4.58) ¯ 𝜃∈Θ where, for fixed 𝜃, 1{ 𝑝(·|𝜃)<𝛼} is the indicator function of the set {D ′ ∈ 𝑋 𝑁 : ¯ 𝑝(D ¯ ′ |𝜃) < 𝛼}. We use boldface, such as 𝒙𝑖 or 𝜽, to emphasize the vector nature of variables and functions in the computational framework. In this notation, the finite dimensional reduction guaranteed by Theorem 4.3.5 in (4.55) and (4.56) of the optimization problem (4.49) defining a worst-case measure Í𝑚 of the form 𝜋 𝛼 := 𝑖=1 𝑤 𝑖 δ𝜽 𝒊 takes the form
 Í𝑚 Í𝑛  Í 𝑚 2 (𝜽 ) (𝜽 ) maximize 𝜑 𝑤 − 𝑤 𝜑 𝜑 𝜽 𝑖 𝑖 𝑖 𝑡 𝑗 𝑤𝑗 𝑖, 𝑗=1 𝑖 𝑡 𝑡=1 𝑖=1 𝑡 𝑚 {(𝑤 𝑖 ,𝜽 𝑖 )}𝑖=1 𝑠.𝑡. 𝜽 𝑖 ∈ Θ, 𝑤 𝑖 ⩾ 0, 𝑖 = 1, . . . , 𝑚 Í𝑚 𝑖=1 𝑤 𝑖 = 1, 𝑝¯ (D|𝜽 𝑖 ) ⩾ 𝛼, 𝑖 = 1, . . . , 𝑚, (4.59) where the component of the objective function 𝑣𝑎𝑟 (𝜑𝑡 ) := 𝑚 ∑︁ 𝑖=1 𝜑𝑡2 (𝜽 𝑖 ) 𝑤 𝑖 − 𝑚 ∑︁
𝑤 𝑖 𝜑𝑡 (𝜽 𝑖 ) 𝜑𝑡 𝜽 𝑗 𝑤 𝑗 𝑖, 𝑗=1 is the variance of the random variable 𝜑𝑡 : Θ → R under the measure 𝜋 := Í𝑚 𝑖=1 𝑤 𝑖 δ𝜽 𝑖 . 140 Algorithm for solving the game We are now prepared to develop an algorithm for player II (the decision maker) to play the game (4.47), using the saddle point Theorem 4.3.2 and the finite dimensional reduction Theorem 4.3.5 after selecting the rarity parameter 𝛼 quantifying the rarity assumption (4.7) in terms of the relative likelihood (4.31) or a surrogate as described in Remark 4.3.1, to be the largest 𝛼 such that the significance 𝛽𝛼 (4.58) at 𝛼 satisfies 𝛽𝛼 ≤ 𝛽∗ , the significance level. At a high level the algorithm for computing a worst-case measure, its resulting risk (variance) and optimal estimator is as follows: 1. Observe a multisample D. 2. Find the largest 𝛼 such that 𝛽𝛼 defined in (4.58) satisfies 𝛽𝛼 ≤ 𝛽∗ . 3. Solve (4.59) determining a worst-case measure 𝜋 𝛼 := Í𝑚 𝑖=1 𝑤 𝑖 δ𝜽 𝑖 . 4. Output the Risk as the value of (4.59). 5. Output optimal decision 𝑑 𝛼 := Í𝑚 𝑖=1 𝑤 𝑖 𝜑(𝜽 𝑖 ). To solve (4.59) in Step 3 we apply the duality of the variance maximization problem with the minimum enclosing ball problem, Theorem 4.3.3, to obtain the following complete algorithm. It uses Algorithm 8 for computing the minimum enclosing ball about the (generally) infinite set 𝜑(Θ𝑥 (𝛼)), which in turn uses a minimum enclosing ball algorithm Miniball applied to sets of size at most 𝑑𝑖𝑚(𝑉) + 2, see e.g. Welzl (Welzl, 1991), Yildirim (Yildirim, 2008) and Gartner (Gärtner, 1999). Here we use that of Welzl (Welzl, 1991). See Section 4.5 for a discussion and a proof in Theorem 4.5.1 of the convergence of Algorithm 8. Theorem 4.5.1 also establishes a convergence proof when the distance maximization Step 8a in Algorithm 7 is performed approximately. Note that the likelihood region ΘD (𝛼) is defined by  ΘD (𝛼) := 𝜽 ∈ Θ : 𝑝(D|𝜽) ≥ 𝛼 𝑝(D|𝜽 ∗ ) , where 𝜽 ∗ ∈ arg max 𝑝 D|𝜽 𝜽 is a MLE.
141 Algorithm 7 UQ4K algorithm 1. Inputs: a) Multisample D := (𝒙1 , . . . , 𝒙 𝑁 ) b) 𝜀0 c) Significance level 𝛽∗ 2. Find MLE 𝜽 ∗ by 𝜽 ∗ ∈ arg max 𝑝 D|𝜽
𝜽 3. Find the largest 𝛼 such that 𝛽𝛼 defined in (4.58) satisfies 𝛽𝛼 ≤ 𝛽∗ 4. 𝒄 ← 𝝋 (𝜽 ∗ ) 5. 𝑆 ← {𝝋 (𝜽 ∗ )} 6. 𝜌0 ← 0 7. 𝑒 ← 2𝜀0 8. While 𝑒 ⩾ 𝜀0 𝜽¯ ∈ a) arg max ∥𝝋 (𝜽) − 𝒄∥ 2 𝜽 𝑠.𝑡. 𝑝(D|𝜽) ≥ 𝛼 𝑝(D|𝜽 ∗ )
b) if 𝝋 𝜽¯ − 𝒄 ⩾ 𝜌0 
i. 𝑆 ← 𝑆 ∪ 𝝋 𝜽¯ c) 𝒄, 𝜌 ← 𝑀𝑖𝑛𝑖𝑏𝑎𝑙𝑙 (𝑆) d) 𝑒 = |𝜌 − 𝜌0 | e) 𝜌0 ← 𝜌 f) if |𝑆| > 𝑛 + 1 i. find subset 𝑆′ ⊂ 𝑆 of size 𝑛 + 1 such that 𝑀𝑖𝑛𝑖𝑏𝑎𝑙𝑙 (𝑆′) = 𝑀𝑖𝑛𝑖𝑏𝑎𝑙𝑙 (𝑆) ii. 𝑆 ← 𝑆′
 Í𝑛 Í𝑛+1 2 Í𝑛+1 𝑛+1 9. Find {𝑤 𝑖 }𝑖=1 from maximize 𝑡=1 𝑖=1 𝜑𝑡 (𝜽 𝑖 ) 𝑤 𝑖 − 𝑖, 𝑗=1 𝑤 𝑖 𝜑𝑡 (𝜽 𝑖 ) 𝜑𝑡 𝜽 𝑗 𝑤 𝑗 𝑛+1 {𝑤 𝑖 }𝑖=1 142 Large sample simplifications Here we demonstrate that when the number of samples 𝑁 is large, under classic regularity assumptions, the significance 𝛽𝛼 is approximated by the value of a chisquared distribution, substantially simplifying the determination of 𝛼 in Step 3 of Algorithm 7. Let Θ ⊆ R 𝑘 and let data be generated by the model at the value 𝜃 ∈ Θ. Under standard regularity conditions, check (G. Casella and R. L. Berger, 2002, Sec. 10.6.2 & Thm. 10.1.12), the maximum likelihood estimator (MLE), b 𝜃 𝑁 , is asymptotically efficient for 𝜃. That is as the sample size 𝑁 → ∞ √ d 𝑁 (b 𝜃 𝑁 − 𝜃) → − 𝑁 (0, 𝐼 (𝜃) −1 ) , (4.60) where 𝐼 (𝜃) is the Fisher information matrix. Therefore, standard arguments, see (G. Casella and R. L. Berger, 2002, Thm. 10.3.1), for the asymptotic distribution of the likelihood ratio test result in the following approximation of 𝛽𝛼 . Theorem 4.4.1. Let Θ ⊆ R 𝑘 and assume that the model density 𝑝 satisfies the regularity conditions of (G. Casella and R. L. Berger, 2002, Section. 10.6.2). Then 1
(4.61) 𝛽𝛼 → 1 − 𝜒𝑘2 2 ln 𝛼 as 𝑁 → ∞, where 𝜒𝑘2 is the chi-square distribution with 𝑘 degrees of freedom. Consequently, under these conditions Step 3 of Algorithm 7 can take the simple form
(Step 3): Solve for 𝛼 satisfying 𝛽𝛼 := 1 − 𝜒𝑘2 2 ln 𝛼1 = 𝛽∗ . Algorithm 7 for a Gaussian noise model Consider a Gaussian noise model where, 𝑋 = R𝑟 and for 𝜽 ∈ Θ, the components of the multisample D := (𝒙1 , . . . , 𝒙 𝑁 ) ∈ R𝑟 𝑁 are i.i.d. samples from the Gaussian distribution N (𝒎 (𝜽) , 𝜎 2 𝐼𝑟 ), with mean 𝒎 (𝜽) and covariance 𝜎 2 𝐼𝑟 , where 𝒎 : Θ → R𝑟 is a measurement function, 𝜎 > 0 and 𝐼𝑟 is the 𝑟-dimensional identity matrix. The measurement function 𝒎 is a function such that the its value 𝒎(𝜃) can be computed when the model parameter 𝜽 is known. Therefore the i.i.d. multisample
𝑁 D := (𝒙1 , . . . , 𝒙 𝑁 ) is drawn from N (𝒎 (𝜽) , 𝜎 2 𝐼𝑟 ) and so has the probability density 𝑁 1 2ª © 1 ∑︁ 𝑝 (D|𝜽) =  √  𝑟 𝑁 exp ­− 2 𝒙 𝑗 − 𝒎 (𝜽) ® (4.62) 2𝜎 𝑗=1 𝜎 2𝜋 « ¬ 143  √ 𝑟𝑁 with respect to the Lebesgue measure 𝜈 on 𝑋 := R𝑟 𝑁 , and defining 𝜎 2𝜋 times the maximum likelihood 𝑁  ∑︁ 2 ª © 1 𝒙 𝑗 − 𝒎 (𝜽) ® , 𝑀 (D) := exp ­− 2 inf 2𝜎 𝜽∈Θ 𝑗=1 ¬ « (4.63) the relative likelihood (4.31) is 𝑝¯ (D|𝜽) =   Í 2 exp − 2𝜎1 2 𝑁𝑗=1 𝒙 𝑗 − 𝒎 (𝜽) (4.64) . 𝑀 (D) Taking the logarithm of the constraint 𝑝¯ (D|𝜽 𝑖 ) ⩾ 𝛼 defining the likelihood region ΘD (𝛼), using (4.64) we obtain 𝑁 n o ∑︁ 2 ΘD (𝛼) = 𝜽 ∈ Θ : 𝒙 𝑗 − 𝒎 (𝜽) ⩽ 𝑀𝛼 (4.65) 𝑗=1 in terms of 𝑀𝛼 := inf 𝑁 ∑︁ 𝜽∈Θ 𝒙 𝑗 − 𝒎 (𝜽) 2 + 2𝜎 2 ln 1 . 𝛼 (4.66) 𝑗=1 Consequently, for the Gaussian case, the worst-case measure optimization problem (4.59) becomes
 Í𝑚 Í𝑛  Í 𝑚 2 (𝜽 ) (𝜽 ) 𝑤 − 𝑤 𝜑 𝜑 𝜽 maximize 𝜑 𝑖 𝑖 𝑖 𝑡 𝑗 𝑤𝑗 𝑖, 𝑗=1 𝑖 𝑡 𝑡=1 𝑖=1 𝑡 𝑚 {(𝑤 𝑖 ,𝜽 𝑖 )}𝑖=1 𝑠.𝑡. 𝜽 𝑖 ∈ Θ, 𝑤 𝑖 ⩾ 0, 𝑖 = 1, . . . , 𝑚 Í𝑚 𝑖=1 𝑤 𝑖 = 1, Í𝑁 2 ⩽ 𝑀𝛼 , 𝑖 = 1, . . . , 𝑚. 𝑗=1 𝒙 𝑗 − 𝒎 (𝜽 𝑖 ) (4.67) Consequently, in the Gaussian noise case, Algorithm 7 appears with these modifications: 1. (Step 2): Find MLE 𝜽 ∗ by 𝜽 ∗ ∈ arg min Í𝑁 𝑗=1 𝒙 𝑗 − 𝒎 (𝜽) 2 𝜽 2. (Step 8a): Solve arg max ∥𝝋 (𝜽) − 𝒄∥ 2 Í𝑁 𝜽 2 𝑠.𝑡. ⩽ 𝑀𝛼 . 𝑗=1 𝒙 𝑗 − 𝒎 (𝜽) 𝜽¯ ∈ (4.68) 144 Farthest point optimization in the Gaussian model In Step 8a of Algorithm 7 we seek the farthest point 𝜽¯ from a center 𝒄: arg max ∥𝝋 (𝜽) − 𝒄∥ 2 Í𝑁 𝜽 2 𝑠.𝑡. ⩽ 𝑀𝛼 . 𝑗=1 𝒙 𝑗 − 𝒎 (𝜽) 𝜽¯ ∈ (4.69) To solve this optimization, we use the merit function technique (Nocedal and S. Wright, 2006) as follows:   𝑁      ∑︁  2 𝒙 𝑗 − 𝒎 (𝜽) − 𝑀𝛼 . minimize − ∥𝝋 (𝜽) − 𝒄∥ + 𝜇 max 0,   𝜽   𝑗=1   2 (4.70) In implementation, one should start with a small value of 𝜇 and increase it to find the optimum (Nocedal and S. Wright, 2006). The first term in (4.70) intends to increase the distance from the center 𝒄 and the second term keeps the solution feasible. Any algorithm picked to solve (4.70) must be able to slide near the feasibility region Í𝑁 2 ⩽ 𝑀𝛼 to guarantee a better performance. Suggestions of such 𝑗=1 𝒙 𝑗 − 𝒎 (𝜽) algorithms are the gradient descent (Kingma and Ba, 2014), if the gradients are available and differential evolution (Storn and Price, 1997) if gradients are not available. Surrogate relative likelihoods Although the computation of the maximum likelihood 𝜽 ∗ in Step 2 of Algorithm 7 is only done once; for the observed data D, the computation of 𝛽𝛼 in Step 3 requires it to be computed for all data D ′ generated by the statistical model. Simplification of this computation can be obtained by large sample 𝑁 approximations (see Section 4.4) or the utilization of a surrogate relative likelihood as discussed in Remark 4.3.1, which we now address. Let the generic multisample be D ′ := (𝑥 1′ , . . . , 𝑥 ′𝑁 ) in the computation of 𝛽𝛼 in (4.58), and consider the upper bound on the maximum likelihood of the Gaussian 145 noise model (4.62) ′ sup 𝑝 (D |𝜽) = 𝜃∈Θ ≤ = 𝑁 2ª © 1 ∑︁ ′ 𝒙 𝑗 − 𝒎 (𝜽) ®  √  𝑟 𝑁 sup exp ­− 2 2𝜎 𝑗=1 𝜃∈Θ 𝜎 2𝜋 « ¬ 1 𝑁 2ª © 1 ∑︁ ′ 𝒙𝑗 − 𝒎 ®  √  𝑟 𝑁 sup𝑟 exp ­− 2 2𝜎 𝑗=1 𝒎∈R 𝜎 2𝜋 ¬ « 1 𝑁 𝑁 © 1 ∑︁ ′ 1 ∑︁ ′ 2 ª exp − 𝒙 − 𝒙 ®, ­  √ 𝑟𝑁 2𝜎 2 𝑗=1 𝑗 𝑁 𝑘=1 𝑘 𝜎 2𝜋 « ¬ 1 so that the resulting surrogate relative likelihood (using the same symbol as the relative likelihood) discussed in Remark 4.3.1 becomes   2 1 Í𝑁 ′ exp − 2𝜎2 𝑗=1 𝒙 𝑗 − 𝒎 (𝜽)  , 𝑝¯ (D ′ |𝜽) = (4.71) Í Í 2 exp − 2𝜎1 2 𝑁𝑗=1 𝒙 ′𝑗 − 𝑁1 𝑁𝑘=1 𝒙 ′𝑘 and therefore the condition 𝑝¯ (·|𝜽) < 𝛼 in the computation of the surrogate significance 𝛽′𝛼 ≥ 𝛽𝛼 defined in (4.58) in terms of the surrogate relative likelihood (4.71) in Step 3 appears as 𝑁 ∑︁ 2 𝒙 ′𝑗 − 𝒎 (𝜽) − 𝑗=1 𝑁 ∑︁ 1 𝒙 ′𝑗 − 𝑁 ∑︁ 𝑁 𝑘=1 𝑗=1 2 𝒙 ′𝑘 > 2𝜎 2 ln 1 . 𝛼 (4.72) Rewriting in terms of the 𝑁 (0, 𝜎 2 𝐼𝑟 ) Gaussian random variables 𝜖𝑖 := 𝒙 ′𝑗 − 𝒎(𝜽), 𝑖 = 1, . . . , 𝑁 we obtain 𝑁 ∑︁ 2 𝒙 ′𝑗 − 𝒎 (𝜽) − 𝑗=1 𝑁 ∑︁ 𝑗=1 1 𝒙 ′𝑗 − 𝑁 ∑︁ 𝑁 𝑘=1 2 𝒙 ′𝑘 = 𝑁 ∑︁ 𝜖𝑗 2 𝑗=1 𝑁 ∑︁ 𝑁 ∑︁ 𝑁 1 ∑︁ 2 − 𝜖𝑗 − 𝜖𝑘 𝑁 𝑗=1 𝑘=1 𝑁 𝑁 1 ∑︁ 1 ∑︁ 2 = 2 𝜖𝑗, 𝜖𝑘 ⟩ − 𝜖𝑘 𝑁 𝑘=1 𝑁 𝑘=1 𝑗=1 𝑁 = (2𝑁 − 1) 1 ∑︁ 2 𝜖𝑘 , 𝑁 𝑘=1 that is 𝑁 ∑︁ 𝑗=1 2 𝒙 ′𝑗 − 𝒎 (𝜽) − 𝑁 ∑︁ 𝑗=1 1 𝒙 ′𝑗 − 𝑁 ∑︁ 𝑁 𝑘=1 𝒙 ′𝑘 2 = (2𝑁 − 1)∥𝑣∥ 2 , (4.73) 146 where 𝑁 𝑣 := 1 ∑︁ 𝜖𝑘 𝑁 𝑘=1 is Gaussian with mean zero and, since the 𝜖 𝑘 are i.i.d, have covariance 𝜎 2 𝐼𝑟 , that 2 is 𝑣 ∈ 𝑁 (0, 𝜎𝑁 𝐼𝑟 ). Since Schott (Schott, 2016, Thm. 9.9) implies that 𝜎𝑁2 ∥𝑣∥ 2 is distributed as 𝜒𝑟2 , it follows from (4.72), (4.73) and the definition of the surrogate significance 𝛽′𝛼 (4.58) that 𝛽′𝛼 = 1 − 𝜒𝑟2 2𝑁 1
ln ≥ 𝛽𝛼 . 2𝑁 − 1 𝛼 (4.74) Consequently, removing the prime indicating the surrogate significance 𝛽′𝛼 , denoting it as 𝛽𝛼 , the modifications (4.68) to Algorithm 7 are augmented to 1. (Step 2): Find MLE 𝜽 ∗ by 𝜽 ∗ ∈ arg min Í𝑁 𝑗=1 𝒙 𝑗 − 𝒎 (𝜽) 2 𝜽
2𝑁 ln 𝛼1 = 𝛽∗ 2. (Step 3): Solve for 𝛼 satisfying 𝛽𝛼 := 1 − 𝜒𝑟2 2𝑁−1 3. (Step 8a): Solve arg max ∥𝝋 (𝜽) − 𝒄∥ 2 Í𝑁 𝜽 2 𝑠.𝑡. ⩽ 𝑀𝛼 . 𝑗=1 𝒙 𝑗 − 𝒎 (𝜽) 𝜽¯ ∈ (4.75) Stochastic processes We now consider the case where Θ ⊆ R 𝑘 and the data is the multisample D := (𝑥 1 , . . . , 𝑥 𝑁 ) with 𝑥𝑖 = (𝑦𝑖 , 𝑡𝑖 ) ∈ Y × T where 𝑦𝑖 corresponds to the observation of a stochastic process at time 𝑡𝑖 . Letting 𝜃 parameterize the distribution of the stochastic process and assuming the 𝑦𝑖 to be independent given 𝜃 and the 𝑡𝑖 , the model density 𝑁 𝑝(𝑥 |𝜃, 𝑡 )𝑞(𝑡 ) if the 𝑡 are assumed to be i.i.d. takes the form (A) 𝑝(D|𝜃) := Π𝑖=1 𝑖 𝑖 𝑖 𝑖 with distribution 𝑄 (and density 𝑞 with respect to some given base measure on T ),
𝑁 𝑝(𝑥 |𝜃, 𝑡 ) if the 𝑡 are assumed to be (B) and 𝑝 (𝑥 1 , . . . , 𝑥 𝑁 ) 𝜃, (𝑡1 , . . . , 𝑡 𝑁 ) := Π𝑖=1 𝑖 𝑖 𝑖 arbitrary. Observe that the model densities for cases (A) and (B) are proportional and, as a consequence, given the 𝑡𝑖 (arbitrary or sampled), they share the same  𝑁 𝑝(𝑦 |𝜃, 𝑡 ) ≥ 𝛼 sup Π 𝑁 𝑝(𝑦 |𝜃 ′, 𝑡 ) . likelihood region ΘD (𝛼) = 𝜃 ∈ Θ : Π𝑖=1 𝑖 𝑖 𝑖 𝑖 𝜃 ′ 𝑖=1 δ𝑡 +···+δ𝑡 Let 𝑄 𝑁 := 1 𝑁 𝑁 ∈ P (T ) be the empirical probability distribution defined by (𝑡1 , . . . , 𝑡 𝑛 ), and assume T to be a compact subset of a finite dimensional Euclidean space. The following theorem indicates the result of Theorem 4.4.1 remains valid in case (B) if 𝑄 𝑁 → 𝑄 (e.g. when T = [0, 1] and 𝑄 is the uniform distribution and the 𝑡𝑖 = 𝑖/𝑁). 147 Theorem 4.4.2. Assume that the model density (𝑦, 𝑡) → 𝑝(𝑦|𝜃, 𝑡)𝑞(𝑡) satisfies the regularity conditions of (G. Casella and R. L. Berger, 2002, Section. 10.6.2), and that 𝑄 𝑁 → 𝑄 (in the sense of weak convergence) as 𝑁 → ∞. Then in both cases
(A) and (B) the limit 𝛽𝛼 → 1 − 𝜒𝑘2 2 ln 𝛼1 holds true as 𝑁 → ∞. 4.5 Minimum enclosing ball algorithm Let 𝐾 ⊂ R𝑛 be a compact subset and let 𝐵 ⊃ 𝐾, with center 𝑧 and radius 𝑅, be the smallest closed ball containing 𝐾. Together Theorem 4.3.3 and Theorem 4.7.2 demonstrate that the minimum enclosing ball exists and is unique. The problem of computing the minimum enclosing ball has received a considerable amount of attention, beginning with Sylvester (Sylvester, 1857) in 1857. Probably the most cited method is that of Welzl (Welzl, 1991), which, by (Welzl, 1991, Thm. 2), achieves the solution in expected O ((𝑛 + 1)(𝑛 + 1)!|𝐾 |) time, where |𝐾 | is the cardinality of the set 𝐾. Yildirim (Yildirim, 2008) provides two algorithms which converge to an 𝜖-approximate minimum enclosing ball in O ( |𝐾𝜖|𝑛 ) computations and provides a historical review of the literature along with extensive references. Although Yildirim does address the infinite 𝐾 situation, we provide a new algorithm, Algorithm 8, based on that of Bădoiu, Har-Peled and Indyk (Bădoiu, Har-Peled, and Indyk, 2002, p. 251), to approximately compute the minimum enclosing ball 𝐵 containing a (possibly infinite) compact set 𝐾 in R𝑛 , using the approximate computation of maximal distances from the set 𝐾 to fixed points in R𝑛 . To that end, let MINIBALL denote an existing algorithm for computing the minimum enclosing ball for sets of size ≤ 𝑛 + 2. As we will demonstrate, the FOR loop in Algorithm 8 always gets broken at Step 10 for some 𝑥 since, by Caratheodory’s theorem ( see e.g. (Rockafellar, 1970)) a minimum enclosing ball in 𝑛 dimensions is always determined by 𝑛 + 1 points. For 𝛿 ≥ 0, and a function 𝑓 : 𝑋 → R, let arg max𝛿 𝑓 denote a 𝛿-approximate maximizer in the following sense; 𝑥 ∗ ∈ arg max𝛿 𝑓 if 𝑓 (𝑥 ∗ ) ≥ 1 sup 𝑓 (𝑥). 1 + 𝛿 𝑥∈𝑋 For, 𝜖 > 0, the 𝜖-enlargement 𝐵 (1+𝜖) (𝑥, 𝑟) of a closed ball 𝐵(𝑥, 𝑟) with center 𝑥 and radius 𝑟 is the closed ball 𝐵(𝑥, (1 + 𝜖)𝑟). In the following algorithm, 𝛿 is a parameter quantifying the degree of optimality of distance maximizations and 𝜖 is a parameter specifying the accuracy required of the produced estimate to the minimum enclosing ball. The following theorem demonstrates that Algorithm 8 148 Algorithm 8 Miniball algorithm 1: Inputs: 𝜖 ∈ [0, 1), 𝛿 ≥ 0 , 𝐾 and MINIBALL for sets of size ≤ 𝑛 + 2 𝛿 ′ 2: (𝑥 𝛼 , 𝑥 𝛽 ) ← arg max𝑥,𝑥 ′ ∈𝐾 ∥𝑥 − 𝑥 ∥ 3: 𝐴0 ← {𝑥 𝛼 , 𝑥 𝛽 } 4: 0 ← 𝑘 (iteration counter) 5: repeat 6: 𝐵 𝑘 ← MINIBALL( 𝐴 𝑘 ) 7: while | 𝐴 𝑘 | > 𝑛 + 1 do 8: for 𝑥 ∈ 𝐴 𝑘 do 9: 𝐵𝑥𝑘 ← MINIBALL( 𝐴 𝑘 \ {𝑥}) 10: if 𝐵𝑥𝑘 = 𝐵 𝑘 then 𝐴 𝑘 ← 𝐴 𝑘 \ {𝑥} and break loop 11: end for 12: end while 13: 𝑧 𝑘 ← Center(𝐵 𝑘 ) 𝛿 14: 𝑥 𝑘+1 ← arg max𝑥∈𝐾 ∥𝑥 − 𝑧 𝑘 ∥ 15: 𝐴 𝑘+1 ← 𝐴 𝑘 ∪ {𝑥 𝑘+1 } 16: 𝑘 ← 𝑘 +1 (1+𝜖) 17: until 𝑥 𝑘 ∈ 𝐵 𝑘−1 (1+𝜖) (1+𝛿) 18: return 𝐵 𝑘−1 , 𝐴 𝑘−1 produces an approximation with guaranteed accuracy to the minimum enclosing ball in a quantified finite number of steps. Theorem 4.5.1. For a compact subset 𝐾 ⊂ R𝑛 , let 𝑅 denote the radius of the minimum enclosing ball of 𝐾. Then, for 𝜖 ∈ [0, 1), 𝛿 ≥ 0, Algorithm 8 converges to a ball 𝐵∗ , satisfying 𝐵∗ ⊃ 𝐾 and
𝑅 𝐵∗ ≤ (1 + 𝜖)(1 + 𝛿)𝑅 in at most 16 (1 + 2𝛿) steps of the REPEAT loop. Moreover, the size of the working 𝜖2 set 𝐴 𝑘 is bounded by   16 | 𝐴 𝑘 | ≤ min 2 + 2 (1 + 2𝛿), 𝑛 + 2 𝜖 for all 𝑘. 4.6 Examples Gaussian Mean Estimation Consider the problem of estimating the mean 𝜃 † of a Gaussian distribution N (𝜃 † , 𝜎 2 ) with known variance 𝜎 2 > 0 from the observation of one sample 𝑥 from that 149 distribution and from the information that 𝜃 † ∈ [−𝜏, 𝜏] for some given 𝜏 > 0. Note that this problem can be formulated in the setting of Problem 1 by letting (1) 𝑃(·|𝜃) be the Gaussian distribution on 𝑋 := R with mean 𝜃 and variance 𝜎 2 , (2) Θ := [−𝜏, 𝜏] and 𝑉 := R and (3) 𝜑 : Θ → 𝑉 be the identity map 𝜑(𝜃) = 𝜃. The relative likelihood (4.31) is − 1 |𝑥−𝜃| 2 𝑒 2𝜎2 𝑝(𝑥|𝜃) ¯ = − 1 |𝑥−𝜃| 2 (4.76) sup𝜃∈Θ 𝑒 2𝜎2 with the supremum in the denominator achieved at the closest 𝜃 ∈ Θ to 𝑥 (𝑥 itself if 𝑥 ∈ [−𝜏, 𝜏]). This defines the likelihood region Θ𝑥 (𝛼) := {𝜃 ∈ Θ : 𝑝(𝑥|𝜃) ¯ ≥ 𝛼}. A simple calculation yields, for the case 𝑥 ∈ [−𝜏, 𝜏] h √︁ √︁

i (4.77) Θ𝑥 (𝛼) = max −𝜏, 𝑥 − 2𝜎 2 ln(1/𝛼) , min 𝜏, 𝑥 + 2𝜎 2 ln(1/𝛼) . Using Theorem 4.3.5 with 𝑚 = 𝑑𝑖𝑚(𝑉) + 1 = 2, for 𝛼 ∈ [0, 1], one can compute a saddle point (𝜋 𝛼 , 𝑑 𝛼 ) of the game (4.47) as 𝜋 𝛼 = 𝑤δ𝜃 1 + (1 − 𝑤)δ𝜃 2 and 𝑑 𝛼 = 𝑤𝜃 1 + (1 − 𝑤)𝜃 2 , (4.78) where 𝑤, 𝜃 1 , 𝜃 2 maximize the variance    Maximize     over      subject to  𝑤𝜃 12 + (1 − 𝑤)𝜃 22 − (𝑤𝜃 1 + (1 − 𝑤)𝜃 2 ) 2 0 ≤ 𝑤 ≤ 1, (𝑥−𝜃 𝑖 ) 2 2𝜎 2 ≤ ln 𝛼1 , 𝜃 1 , 𝜃 2 ∈ [−𝜏, 𝜏] (4.79) 𝑖 = 1, 2, where the last two constraints are equivalent to the rarity assumption 𝜃 𝑖 ∈ Θ𝑥 (𝛼). Hence for 𝛼 near 0, Θ𝑥 (𝛼) = Θ = [−𝜏, 𝜏], and by Theorem 4.3.3, the variance is maximized by placing each Dirac on each boundary point of the region Θ, each receiving half of the total probability mass, that is by 𝜃 1 = −𝜏, 𝜃 2 = 𝜏 and 𝑤 = 1/2, in which case Var 𝜋 𝛼 = 𝜏 2 and 𝑑 𝛼 = 0. For 𝛼 = 1, the rarity constraint implies 𝜃 1 = 𝜃 2 = 𝑥 when 𝑥 ∈ [−𝜏, 𝜏], leading to the MLE 𝑑 𝛼 = 𝑥 with Var 𝜋 𝛼 = 0. Note that from (4.34) we have   𝛽𝛼 = sup P𝑥 ′ ∼N (𝜃,𝜎2 ) 𝑝(𝑥 ¯ ′ |𝜃) < 𝛼 𝜃∈[−𝜏,𝜏] which can be computed analytically for this example using 4.76 and separating into the three cases 𝑥 < −𝜏, 𝑥 ∈ [−𝜏, 𝜏] and 𝑥 > 𝜏. We illustrate in Figure 4.6 the different results of solving the optimization problem (4.79) in the case 𝜎 2 = 1, 150 Figure 4.6: 𝛼 − 𝛽 relation, likelihood level sets, risk value and decision for different choices of 𝛼 (and consequently 𝛽) for the normal mean estimation problem with 𝜏 = 3 and observed value 𝑥 = 1.5. Three different values in the 𝛼 − 𝛽 curve are highlighted across the plots 𝑥 = 1.5 and 𝜏 = 3. We plot the 𝛼 − 𝛽 curve (top left), the likelihood of the model in [−𝜏, 𝜏] and the 𝛼−level sets (top right), and the evolutions of the risk with 𝛽 (bottom left), and the optimal decision with 𝛽 (bottom right). Since, by Theorem 4.3.3, the optimal decision is the midpoint of the interval with extremes in either the 𝛼−level sets or ±𝜏, we observe that for low 𝛽, our optimal decision does not coincide with the MLE. Estimation of a quadratic function The measurement function 𝒎 of Section 4.6, being defined as the solution of the Lotka-Volterra predator-prey model as a function of its parameters 𝜽 ∈ Θ, does not appear simple to differentiate and therefore SciPy’s version of Storn and Price’s (Storn and Price, 1997) Differential Evolution optimizer (Virtanen et al., 2020) was used to perform the farthest point optimization problem in Step 8a in Algorithm 7. In this section, we test this framework on a problem which does not possess this 151 complication: estimating the parameters 𝜽 := (𝜃 0 , 𝜃 1 , 𝜃 2 ) of a quadratic function 𝒎(𝑡; 𝜽) := 𝜃 0 + 𝜃 1 𝑡 + 𝜃 2 𝑡 2 on a uniform grid 𝑇 of the interval (0, 5) consisting of 100 points, using noisy observational data. In this case, we can use automatic differentiation in the merit function technique of Section 4.4 to perform the farthest point optimization problem in Step 8a using gradient descent methods via automatic differentiating modules available in packages like autograd, or computing the gradient and applying a gradient descent method. We proceed as in Section 4.6 with Θ := [−30, 30] 3 and assume that, given 𝜽 ∈ Θ, a
100 single sample path D := {𝒙} := 𝒙 (𝑡) 𝑡=1 is generated on the grid 𝑇 to the stochastic process 𝒙 (𝑡) = 𝒎(𝑡; 𝜽) + 𝜖𝑡 , 𝜖𝑡 ∼ N (0, 𝜎 2 ), 𝑡 ∈ 𝑇, (4.80) with 𝜎 2 := 10. Consequently 𝑋 := R100 . We let the decision space be 𝑉 = R3 and the quantity of interest 𝜑 : Θ → R3 be the identity function. For the experiment, we generate a single (𝑁 := 1) full sample path D := {𝒙} :=
100 𝒙 (𝑡) 𝑡=1 according to (4.80) at the unknown values 𝜽 ∗ := (𝜃 1∗ , 𝜃 2∗ , 𝜃 3∗ ) = (1, .5, 1). As discussed in Section 4.4, we tune 𝜇 in the merit function (4.70), and set gradient descent with adaptive moment estimation optimizer (Paszke et al., 2019) with parameters (e.g. learning rate = 0.001, max epochs=50,000) to achieve full convergence. We observe the convergence plots for the increment of 𝜃s from the ball center as diagnostic. Figure 4.7 shows the 𝛽 vs 𝛼 relationship defined by the surrogate significance (4.74) derived from the surrogate likelihood method with 𝑁 := 1 and 𝑟 := 100, the risk as function of 𝛼, and the likelihood regions, their minimum enclosing balls and the optimal decisions (centers of the balls), for 𝛼 ∈ [0, 1]. As can be seen the optimal decisions, being the centers of the minimum enclosing balls, do not move and only the size of the minimum enclosing balls change, resulting in various risk values associated with the same optimal estimates. Finally, Figure 4.7 shows the results for the maximum likelihood solution and the two supporting points of the minimum enclosing balls for the case that 𝛽𝛼 = 𝛽∗ = 0.05. From this experiment, we obtained the optimal decision 𝑑 ∗ = (0.24, 1.22, 0.89)  along with the two support points 𝑆 = (−2.27, 3.52, 0.48), (2.75, −1.10, 1.31) of the minimum enclosing ball. For the sake of comparison, we also performed the 152 same experimentation with SciPy’s version of Storn and Price’s (Storn and Price, 1997) Differential Evolution optimizer (Virtanen et al., 2020) at the default settings, to perform the farthest point optimization problem in Step 8a in Algorithm 7, using the merit function (4.70) of Section 4.4, and obtained similar results. Figure 4.7: Quadratic model results: (left-top) 𝛽𝛼 vs 𝛼, (right-top) risk vs 𝛽𝛼 , (left-bottom) the supporting surfaces with the decision points at the center of each surface and (right-bottom) maximum-likelihood solution and supporting points of the minimum enclosing ball for 𝛽𝛼 = 𝛽∗ = .05 Estimation of a Lotka-Volterra predator-prey model Here we implement Algorithm 7 for the Gaussian noise model of Section 4.4, where the measurement function (𝜃 1 , 𝜃 2 ) ↦→ 𝒎(𝜃 1 , 𝜃 2 ) is defined as the solution map of the Lotka-Volterra (Lotka, 1920) predator-prey model 𝑑𝑥 = 𝜃 1 𝑥 − 𝜂𝑥𝑦 𝑑𝑡 𝑑𝑦 = 𝜉𝑥𝑦 − 𝜃 2 𝑦, 𝑑𝑡 (4.81) (4.82) 200 such that 𝑡 = 0 and 𝑡 evaluated on the uniform time grid 𝑇 := {𝑡𝑖 }𝑖=1 0 200 = 20, with fixed and known parameters 𝜂, 𝜉 and initial data 𝑥 0 , 𝑦 0 , describing the evolution of a prey population with variable 𝑥 and a predator population with variable 𝑦. As such, denoting 𝜽 := (𝜃 1 , 𝜃 2 ) ∈ Θ, we denote the solution map 𝜽 ↦→ 𝒎(𝑡; 𝜽), 𝑡 ∈ 𝑇, 153 by 𝒎 : Θ → (R2 )𝑇 . For the probabilistic model, we let Θ := [−5, 5] 2 and assume the Gaussian model (4.64) with 𝑁 = 1, where the data D consists of a single sample
200 path D := {𝒙} := 𝒙 (𝑡) 𝑡=1 of the 𝑇-indexed stochastic process " # 𝑥𝑡 𝒙 (𝑡) = = 𝒎(𝑡; 𝜽) + 𝜖𝑡 , 𝜖𝑡 ∼ N (0, 𝜎 2 I), 𝑡 ∈ 𝑇, (4.83) 𝑦𝑡 where I ∈ R2×2 is the identity matrix and 𝜎 = 5. Note that in the notation of (4.64) we have ∑︁ ∥𝒙 − 𝒎 (𝜽) ∥ 2 = ∥𝒙 (𝑡) − 𝒎 (𝑡; 𝜽) ∥ 2 . 𝑡∈𝑇 Let the decision space be 𝑉 := R2 and let the quantity of interest 𝜑 : Θ → R2 be the identity. For the experiment, we generate one sample path D := {𝒙} :=
200 𝒙 (𝑡) 𝑡=1 according to (4.83) at the unknown values 𝜽 ∗ := (𝜃 1∗ , 𝜃 2∗ ) = (0.55, 0.8), with 𝑥0 = 30, 𝑦 0 = 10, 𝜂 = 0.025 and 𝜉 = 0.02 known. We consider the evolution
200 𝑡 ↦→ 𝒎(𝑡; 𝜽 ∗ ), 𝑡 ∈ 𝑇, the true predator-prey values and the sample path 𝒙 (𝑡) 𝑡=1 as noisy observations of it. The resulting time series D is shown in the top image of Figure 4.8. For 𝛼 ∈ [0, 1], by taking the logarithm of the defining relation (4.7) of the likelihood region ΘD (𝛼), we obtain the representation (4.65), n o ∑︁ 2 (𝑡) ΘD (𝛼) = 𝜽 ∈ Θ : 𝒙 − 𝒎 (𝑡; 𝜽) ⩽ 𝑀𝛼 𝑡∈𝑇 in terms of 𝑀𝛼 := inf ∑︁ 𝜽∈Θ 𝒙 (𝑡) − 𝒎 (𝑡; 𝜽) 2 + 2𝜎 2 ln 1 , 𝛼 𝑡∈𝑇 for the likelihood region ΘD (𝛼) in terms of the data D. To determine 𝛼 at significance level 𝛽∗ := .05, we approximate the significance 𝛽𝛼 defined in (4.34) using the chi-squared approximation (4.74) and then select 𝛼 to be the value such that this approximation yields 𝛽𝛼 = 𝛽∗ = .05. The validity of this approximation for this example is additionally demonstrated in the right image of Figure 4.10, which shows via Monte Carlo simulation that the 1 − 𝛽𝛼 versus 𝛼 curve is well characterized by the 𝜒22 distribution. Having selected 𝛼, to implement Algorithm 7, we need to select an optimizer for Step 8a. Instead of computing the Jacobian of the solution map 𝒎, here we utilize the gradient-free method of SciPy’s version of Storn and Price’s (Storn and 154 Price, 1997) Differential Evolution optimizer (Virtanen et al., 2020) at the default settings. Given the data generating value 𝜽 ∗ = (0.55, 0.8), the primary feasible region Θ := [−5, 5] 2 is sufficiently non-suggestive of the the data generating value 𝜽 ∗ . Finally, since 𝑑𝑖𝑚(𝑉) = 2, Algorithm 7 produces: a set 𝑆 of at most three boundary points of ΘD (𝛼), the minimum enclosing ball 𝐵 of ΘD (𝛼), its center as the optimal estimate of 𝜽 ∗ , and the weights of the set 𝑆 corresponding to a worstcase measure, optimal for the variance maximization problem (4.49). The results are displayed in Figure 4.9. Figure 4.8: (Top image) Data D := 𝒙 (𝑡) , 𝑡 ∈ 𝑇, generated, according to the Gaussian noise model (4.83): solid red is the prey component and solid blue the predator component of the generated data 𝒙 (𝑡) , the dotted red and dotted blue are the preypredator components of the Lotka-Volterra solution 𝒎(𝑡, 𝜽 ∗ ) for 𝑡 ∈ 𝑇. (Bottom image) Uncertainty in the population dynamics corresponding to the worst-case measure: (1) red is prey and blue is predator, (2) solid line is the Lotka-Volterra evolution 𝒎(𝑡, 𝜽 ∗ ), 𝑡 ∈ 𝑇, fine dots 𝒎(𝑡, 𝜽 1 ), 𝑡 ∈ 𝑇, and coarse dots 𝒎(𝑡, 𝜽 2 ), where 𝑆 := {𝜽 1 , 𝜽 2 } is the set of support points of the worst case (posterior) measure (located on the boundary of the minimum enclosing ball). To get a sense of the output uncertainty of 𝒎(·) with these optimized results, we plot the predator and prey population dynamics associated with each optimized 155 Figure 4.9: Minimum enclosing ball for the Lotka-Volterra Model: (left) the dashed line indicates the boundary of the likelihood region ΘD (𝛼), the solid circle its minimum enclosing ball, the red point the data generating value 𝜽 ∗ and the blue point the center of the minimum enclosing ball and the optimal estimate of 𝜽 ∗ . The two yellow points comprise the set 𝑆 := {𝜽 1 , 𝜽 2 }. (right) a projected view with the yellow columns indicating the weights (.5, .5) of the set 𝑆 in their determination of a worst-case (posterior) measure. Figure 4.10: Monte Carlo numerically confirms the result from Theorem 4.4.1 in both the quadratic function estimation (left image) and Lotka-Volterra (right image) examples. 156 boundary point of ΘD (𝛼) in Figure 4.8. This figure shows that with significance value 𝛽𝛼 = 0.05, the optimized boundary points create population dynamics in a tight band around the true population dynamics. 4.7 General loss functions and rarity assumptions Here we generalize the framework introduced in Section 4.2 to allow more general loss functions than the ℓ 2 loss, in for example Equations (4.18), (4.19), and (4.23), and more general rarity assumptions than (4.33). The discussions of worst-case, robust Bayes, and Wald’s statistical decision theory generalize in a straightforward manner, so we focus on generalizing the current UQ of the 4th kind. Let ℓ : 𝑉 × 𝑉 → R+ be a loss function. In addition to the pointwise rarity assumption (4.33), consider an integral rarity assumption P𝑥Ψ (𝛼) ⊂ P (Θ) determined by a real-valued function Ψ, defined by ∫ n o
Ψ P𝑥 (𝛼) := 𝜋 ∈ P (Θ) : Ψ 𝑝(𝑥|𝜃) ¯ 𝑑𝜋(𝜃) ≥ 𝛼 (4.84) Θ generalizing (4.29). Note that for Ψ the identity function P𝑥 (𝛼) ⊂ P𝑥Ψ (𝛼) and by comparison with (4.29) it follows from the following remark that such integral rarity assumptions can also alleviate the brittleness of Bayesian inference. Remark 4.7.1. Jensen’s inequality implies that ∫  ∫
Ψ 𝑝(𝑥|𝜃)𝑑𝜋(𝜃) ¯ ≤ Ψ 𝑝(𝑥|𝜃)) ¯ 𝑑𝜋(𝜃) Θ Θ when the function Ψ is convex, and ∫  ∫
Ψ 𝑝(𝑥|𝜃)𝑑𝜋(𝜃) ¯ ≥ Ψ 𝑝(𝑥|𝜃)) ¯ 𝑑𝜋(𝜃) Θ Θ when the function Ψ is concave, such as when Ψ is a logarithm. Consequently, when Ψ is concave and strictly increasing, the assumption ∫
Ψ 𝑝(𝑥|𝜃) ¯ 𝑑𝜋(𝜃) ≥ 𝛼 Θ implies that the denominator in the conditional measure (4.46) satisfies ∫ 𝑝(𝑥|𝜃)𝑑𝜋(𝜃) ¯ ≥ Ψ−1 (𝛼) . Θ Consequently, such constraints, by keeping the denominator in the conditional measure bound away from zero, stabilize the numerical computation of the conditional measure in the numerical computation of a worst-case measure. 157 The following applies equally as well for the pointwise rarity assumption (4.33) and the integral rarity assumption (4.84). For simplicity of exposition, we restrict to the pointwise rarity assumption. Generalizing (4.45), consider playing a game using the loss   L (𝜋, 𝑑) := E𝜃∼𝜋 𝑥 ℓ(𝜑(𝜃), 𝑑) , 𝜋 ∈ P𝑥 (𝛼), 𝑑 ∈ 𝑉 . (4.85) For the pointwise rarity assumption, the same logic following (4.45) implies that this game is equivalent to the generalization (4.47) to a game using the loss   L (𝜋, 𝑑) := E𝜃∼𝜋 ℓ(𝜑(𝜃), 𝑑) , 𝜋 ∈ P𝑥 (𝛼), 𝑑 ∈ 𝑉 . (4.86) For the integral rarity assumption, we maintain the form (4.85). 𝛽𝛼 is defined before as in (4.34) and the selection of 𝛼 ∈ [0, 1] is as before. Under the mild conditions of Theorem 4.8.1, we can show that, for each 𝛼 ∈ [0, 1], a maxmin optimal solution 𝜋 𝛼 of max𝜋∈P𝑥 (𝛼) min𝑑∈𝑉 L (𝜋, 𝑑) can be computed. Moreover, by Theorem 4.8.1, it also follows that the saddle function L in (4.86) satisfies the conditions of Sion’s minmax theorem (Sion, 1958), resulting in a minmax result for the game (4.86): min max L (𝜋, 𝑑) = max min L (𝜋, 𝑑) . 𝑑∈𝑉 𝜋∈P𝑥 (𝛼) 𝜋∈P𝑥 (𝛼) 𝑑∈𝑉 (4.87) Consequently if, for each 𝜋 ∈ P𝑥 (𝛼), we select 𝑑 𝜋 ∈ arg min L (𝜋, 𝑑), (4.88) 𝑑∈𝑉 then a worst-case measure for the game (4.86), solving the maxmin problem on the right-hand side of (4.87), satisfies 𝜋 𝛼 ∈ arg max L (𝜋, 𝑑 𝜋 ) . 𝜋∈P𝑥 (𝛼) (4.89) Moreover let 𝑑 𝛼 ∈ arg min𝑑∈𝑉 max𝜋∈P𝑥 (𝛼) L (𝜋, 𝑑) denote any solution to the minmax problem on the left-hand side of (4.87). Then it is well known that the minmax equality (4.87) implies that the pair (𝜋 𝛼 , 𝑑 𝛼 ) is a saddle point of L in that L (𝜋, 𝑑 𝛼 ) ≤ L (𝜋 𝛼 , 𝑑 𝛼 ) ≤ L (𝜋 𝛼 , 𝑑), 𝜋 ∈ P𝑥 (𝛼), 𝑑 ∈ 𝑉 . (4.90) Since the solution to 𝑑 𝜋 𝛼 := arg min L (𝜋 𝛼 , 𝑑) 𝑑∈𝑉 (4.91) 158 is uniquely defined under the conditions of Theorem 4.7.2 (identical to those of Theorem 4.8.1), it follows from the right-hand side of the saddle equation (4.3) that 𝑑 𝜋 𝛼 = 𝑑 𝛼 and (𝜋 𝛼 , 𝑑 𝜋 𝛼 ) is a saddle point of L, that is we have L (𝜋, 𝑑 𝜋 𝛼 ) ≤ L (𝜋 𝛼 , 𝑑 𝜋 𝛼 ) ≤ L(𝜋 𝛼 , 𝑑), 𝜋 ∈ P𝑥 (𝛼), 𝑑 ∈ 𝑉 . (4.92) Moreover, its associated risk is the value R (𝑑 𝜋 𝛼 ) := L (𝜋 𝛼 , 𝑑 𝜋 𝛼 ) (4.93) in (4.92) of the two person game defined in (4.86), which is the same for all saddle points of L. Finite-dimensional reduction Let Δ𝑚 (Θ) denote the set of convex sums of 𝑚 Dirac measures located in Θ and, let P𝑥𝑚 (𝛼) ⊂ P𝑥 (𝛼) defined by n o 𝑚 𝑚 P𝑥 (𝛼) := 𝜋 ∈ Δ (Θ) ∩ P𝑥 (𝛼) , (4.94) denote the finite-dimensional subset of the rarity assumption set consisting of the convex combinations of 𝑚 Dirac measures in P𝑥 (𝛼). Then the following reduction Theorem 4.7.2 asserts that max min L (𝜋, 𝑑) = max min L (𝜋, 𝑑), 𝑚 𝜋∈P𝑥 (𝛼) 𝑑∈𝑉 𝜋∈P𝑥 (𝛼) 𝑑∈𝑉 (4.95) for any 𝑚 ≥ 𝑑𝑖𝑚(𝑉) + 2. We note that the improvement to 𝑚 ≥ 𝑑𝑖𝑚(𝑉) + 1 when the loss is the ℓ 2 loss follows from the Lagrangian duality, Theorem 4.3.3, of the maximization problem (4.95) with the minimum enclosing ball and Caratheodory’s theorem. In the following theorem, applicable to both pointwise and integral rarity assumptions, we provide sufficient conditions that the computation of a worst-case measure in the optimization problem (4.89) can be reduced to a finite-dimensional one. Theorem 4.7.2. Let Θ be compact, 𝑋 be a measurable space, P be a positive dominated model such that, for each 𝑥, its likelihood function is continuous, and let 𝜑 : Θ → R be continuous. Let 𝑉 be a finite-dimensional Euclidean space and let ℓ : 𝑉 × 𝑉 → R+ be continuously differentiable, strictly convex and coercive in its second variable and vanishing along the diagonal. Let Δ𝑚 (Θ) denote the set of convex sums of 𝑚 Dirac measures located in Θ and, for Ψ : (0, 1] → R upper 159 semicontinuous and 𝑥 ∈ 𝑋, consider both the pointwise rarity assumption subset P𝑥𝑚 (𝛼) ⊂ P𝑥 (𝛼) defined by n o 𝑚 𝑚 P𝑥 (𝛼) := 𝜋 ∈ Δ (Θ) ∩ P𝑥 (𝛼) , (4.96) and the integral rarity assumption subset P𝑥Ψ,𝑚 (𝛼) ⊂ P𝑥Ψ (𝛼), defined by ∫ n o
Ψ,𝑚 𝑚 P𝑥 (𝛼) := 𝜋 ∈ Δ (Θ) : Ψ 𝑝(𝑥|𝜃) 𝑑𝜋(𝜃) ≥ 𝛼 , (4.97) Θ where 𝑝(𝑥|·) : Θ → (0, 1] is the relative likelihood. Then we have max min L (𝜋, 𝑑) = max min L (𝜋, 𝑑), 𝑚 𝜋∈P𝑥 (𝛼) 𝑑∈𝑉 𝜋∈P𝑥 (𝛼) 𝑑∈𝑉 (4.98) for any 𝑚 ≥ 𝑑𝑖𝑚(𝑉) + 2, and max min L (𝜋, 𝑑) = 𝜋∈P𝑥Ψ (𝛼) 𝑑∈𝑉 max min L (𝜋, 𝑑), 𝜋∈P𝑥Ψ,𝑚 (𝛼) 𝑑∈𝑉 (4.99) for any 𝑚 ≥ 𝑑𝑖𝑚(𝑉) + 3, unless Ψ is the identity function, when 𝑚 ≥ 𝑑𝑖𝑚(𝑉) + 2. Remark 4.7.3. Theorem 4.7.2 easily generalizes to vector integral functions Ψ of more general form than (4.97), where the number of Diracs required is then 𝑚 ≥ 𝑑𝑖𝑚(𝑉) + dim(Ψ) + 1 and 𝑚 ≥ 𝑑𝑖𝑚(𝑉) + dim(Ψ) + 2 for the pointwise and integral cases, respectively. (A. Shapiro and Kleywegt, 2002, Thm.2.1) implies one can generalize Theorem 4.7.2 to the more general class of loss functions ℓ which are coercive and convex
in the second argument, but require more, (3 dim(𝑉) + 1) ) Dirac measures in the integral case. See (A. Shapiro and Kleywegt, 2002, Prop. 3.1). The following theorem generalizes the duality Theorem 4.3.3 to more general convex loss functions. Theorem 4.7.4. Let 𝜑 : Θ𝑥 (𝛼) → 𝑉 be continuous and suppose that the loss function satisfies function ℓ(𝑣 1 , 𝑣 2 ) = 𝑊 (𝑣 1 − 𝑣 2 ), where 𝑊 : 𝑉 → R+ is nonnegative, convex, and coercive. For 𝑥 ∈ 𝑋 and 𝛼 ∈ [0, 1], suppose that Θ𝑥 (𝛼) is
compact. Then 𝜑 Θ𝑥 (𝛼) ⊂ 𝑉 is compact. Let 𝜆 ≥ 0 be the smallest value such that there exists a 𝑧 ∈ 𝑉 with 𝜑(Θ𝑥 (𝛼)) + 𝑧 ⊂ 𝑊 −1 ([0, 𝜆]). Then 𝜆 is the value of the maxmin problem defined by the game (4.86)   𝜆 = max min E𝜃∼𝜋 ℓ(𝜑(𝜃), 𝑑) , 𝜋∈P𝑥 (𝛼) 𝑑∈𝑉 (4.100) 160 and 𝜋 ∗ is maxmin optimal for it if and only if there exists   𝑑∗ ∈ argmin𝑑∈𝑉 E𝜃∼𝜋∗ ℓ(𝜑(𝜃), 𝑑) with support(𝜑∗ 𝜋 ∗ ) ⊂ 𝑊 −1 (𝜆) − 𝑑∗ . Remark 4.7.5. Pass (Pass, 2020) generalizes these results to the case where the decision space 𝑉 is a not-necessarily affine metric space and the ℓ 2 distance is replaced by the metric. 4.8 Supporting theorems and proofs Minmax theorem Theorem 4.8.1. Consider the saddle function (4.86) for the pointwise and (4.85) for the integral rarity assumptions, respectively. Given the assumptions of Theorem 4.7.2 we have min max L (𝜋, 𝑑) = max min L (𝜋, 𝑑) 𝑑∈𝑉 𝜋∈P𝑥 (𝛼) 𝜋∈P𝑥 (𝛼) 𝑑∈𝑉 and min max L (𝜋, 𝑑) = max min L (𝜋, 𝑑) 𝑑∈𝑉 𝜋∈P𝑥Ψ (𝛼) 𝜋∈P𝑥Ψ (𝛼) 𝑑∈𝑉 for the pointwise and integral rarity assumptions, respectively. Proof. We prove the result for the integral rarity assumption case only, the pointwise case being much simpler. The assumptions imply Ψ( 𝑝(𝑥|·)) is upper semicontinuous, implying that P𝑥Ψ (𝛼) ⊂ P (Θ) is closed, see e.g. (Aliprantis and Border, 2006, Thm. 15.5). Since P (Θ) is a compact subset of the space of signed measures in the weak topology, see e.g. (Aliprantis and Border, 2006, Thm. 15.22), it follows that P𝑥Ψ (𝛼) ⊂ P (Θ) is compact. Consequently, to apply Sion’s minmax theorem (Sion, 1958) it is sufficient to establish that the map L (·, 𝑑) : P (Θ) → R is upper semicontinuous and quasiconcave for each 𝑑 ∈ 𝑉 and the map L (𝜋, ·) : 𝑉 → R is lower semicontinuous and quasiconvex for each 𝜋 ∈ P (Θ). To that end, observe that since 𝜙 is continuous and Θ compact, the function ℓ(𝜙(·), 𝑑) is bounded and continuous for each 𝑑 ∈ 𝑉. Fixing 𝑑, observe the positivity of the likelihood function implies that the set   𝜋 ∈ P (Θ) : E 𝜃∼ 𝜋𝑥 [ℓ(𝜙(𝜃), 𝑑)] ≥ 𝑟 = 𝜋 ∈ P (Θ) : E 𝜃∼ 𝜋 [ 𝑝(𝑥|·)ℓ(𝜙(𝜃), 𝑑)] ≥ 𝑟E 𝜃∼ 𝜋 [ 𝑝(𝑥|·)] is closed by the continuity of ℓ(𝜙(·), 𝑑) and 𝑝(𝑥|·), see e.g. (Aliprantis and Border, 2006, Thm. 15.5). Moreover, one can show that the reverse inequality also produces a closed set. Additionally, since it is a linear condition it is convex and therefore the 161 function L (·, 𝑑) : P (Θ) → R is upper and lower semicontinuous and quasiconcave for each 𝑑 ∈ 𝑉. Moreover, fixing 𝜋 ∈ P (Θ), since the function ℓ is continuous and Θ is compact and 𝜙 is continuous, it follows that the function L (𝜋, ·) : 𝑉 → R is continuous and convex and therefore lower semicontinuous and quasiconvex. Consequently, Sion (Sion, 1958, Cor. 3.3) implies that inf sup L (𝜋, 𝑑) = 𝑑∈𝑉 𝜋∈P Ψ (𝛼) sup inf L (𝜋, 𝑑). 𝜋∈P𝑥Ψ (𝛼) 𝑑∈𝑉 𝑥 Since, for fixed 𝑑, inner optimization sup𝜋∈P𝑥Ψ (𝛼) L (𝜋, 𝑑) is over of an upper semicontinuous function over a compact set, it achieves its supremum. Since have established in the proof of Theorem 4.7.2 that the inner optimization inf 𝑑∈𝑣 L (𝜋, 𝑑) achieves its infimum, we can write inf max L (𝜋, 𝑑) = 𝑑∈𝑉 𝜋∈P𝑥Ψ (𝛼) sup min L (𝜋, 𝑑) 𝜋∈P𝑥Ψ (𝛼) 𝑑∈𝑉 Since the inner minimum min𝑑∈𝑣 L (𝜋, 𝑑) is the minimum of a family upper semicontinuous functions, it produces an upper semicontinuous function, see e.g. (Aliprantis and Border, 2006, Lem. 2.41), since the maximization of the outer loop is over the compact set P𝑥Ψ (𝛼), we conclude that the supremum on the righthand side is attained. Moreover, since the inner maximum max𝜋∈P𝑥Ψ (𝛼) L (𝜋, 𝑑) is the maximum over a family of continuous functions, and therefore lower semicontinuous functions, it follows that it produces a lower semicontinuous function. Restricting to the compact subset 𝑉 ∗ from the proof of Theorem 4.7.2 we conclude the outer infimum is attained, thus establishing the assertion. □ Proof of Theorem 4.3.2 Since the ℓ 2 loss (𝑣 1 , 𝑣 2 ) ↦→ ∥𝑣 1 − 𝑣 2 ∥ 2 is strictly convex and coercive in its second argument and vanishes on the diagonal, the assumptions imply that the saddle   function L (𝜋, 𝑑) := E𝜃∼𝜋 ∥𝜑(𝜃) − 𝑑 ∥ 2 , 𝜋 ∈ P𝑥 (𝛼), 𝑑 ∈ 𝑉, (4.47) satisfies the conditions of Theorem 4.7.2, so that it follows from Theorem 4.8.1 that L satisfies the minmax equality, in particular establishing the existence of a worst-case measure 𝜋 ∗ ∈ arg max inf L (𝜋, 𝑑) (4.101) 𝑑 ∗ ∈ arg min sup L (𝜋, 𝑑). (4.102) 𝜋∈P𝑥 (𝛼) 𝑑∈𝑉 and a worst-case decision 𝑑∈𝑉 𝜋∈P𝑥 (𝛼) 162
In addition to establishing the existence of saddle points, where a pair 𝜋 ∗ , 𝑑 ∗ ∈ P𝑥 (𝛼) × 𝑉 is a saddle point of L if we have L (𝜋, 𝑑 ∗ ) ≤ L (𝜋 ∗ , 𝑑 ∗ ) ≤ L (𝜋 ∗ , 𝑑), 𝜋 ∈ P𝑥 (𝛼), 𝑑 ∈ 𝑉, (4.103) observe that Bertsekas et al. (Bertsekas, Nedić, and Ozdaglar, 2003, Prop. 2.6.1) assert that a pair (𝜋 ∗ , 𝑑 ∗ ) ∈ P𝑥 (𝛼) × 𝑉 is a saddle point if and only if they are a worst-case measure and worst-case decision, respectively, as defined in (4.101) and (4.102). Now let (𝜋 ∗ , 𝑑 ∗ ) be a saddle point. As demonstrated in the proof of Theorem 4.7.2, since the function 𝑑 ↦→ ∥𝜑(𝜃) − 𝑑 ∥ 2 is strictly convex for all 𝜃, it follows that its   expectation 𝑑 ↦→ L (𝜋, 𝑑) := E𝜃∼𝜋 ∥𝜑(𝜃) − 𝑑 ∥ 2 is strictly convex. Moreover a minimizer 𝑑 ∗∗ ∈ arg min L (𝜋 ∗ , 𝑑) 𝑑∈𝑉 exists and by strict convexity it is necessarily unique; see e.g. (Rockafellar and Wets, 1998). Consequently, by the right-hand side of the definition (4.103) of a saddle point it follows that 𝑑 ∗∗ = 𝑑 ∗ . Since 𝜋 ∗ satisfying (4.101) is equivalent to it satisfying (4.48) and 𝑑 ∗ satisfying the right-hand side of the definition (4.103) of a saddle point is equivalent to it satisfying (4.49), the assertion regarding the form (4.48) and (4.49) for saddle points is proved. Let (𝜋1 , 𝑑1 ) and (𝜋2 , 𝑑2 ) be two saddle points. Then by the saddle relation (4.103) we have 𝐿 (𝜋2 , 𝑑2 ) ≤ L (𝜋2 , 𝑑1 ) ≤ L (𝜋1 , 𝑑1 ) ≤ L(𝜋1 , 𝑑2 ) ≤ L (𝜋2 , 𝑑2 ), establishing equality of the value of the risk (4.50) for all saddle points. Finally, since 𝑑 ↦→ L (𝜋, 𝑑) is strictly convex it follows that its maximum 𝑑 ↦→ sup𝜋∈P𝑥 (𝛼) L (𝜋, 𝑑) is strictly convex, demonstrating the uniqueness of solutions to (4.102). Moreover, since P𝑥 (𝛼) is convex, the mapping 𝜋 ↦→ L (𝜋, 𝑑) :=   E𝜃∼𝜋 ∥𝜑(𝜃) − 𝑑 ∥ 2 is affine and therefore concave for all 𝑑 ∈ 𝑉, and therefore its minimum 𝜋 ↦→ inf 𝑑∈𝑉 L (𝜋, 𝑑) is also concave. Consequently, the set of all worst-case measures, that is, maximizers of (4.101), is convex, establishing the final assertion. Proof of Theorem 4.3.3 Since the relative likelihood is continuous, the likelihood region Θ𝑥 (𝛼) is closed and
therefore compact, and since 𝜑 is continuous, it follows that 𝜑 Θ𝑥 (𝛼) is compact 163 and therefore measurable. According to Bonnans and Shapiro (Bonnans and A. Shapiro, 2000, Sec. 5.4.1), because the constraint function (𝑟, 𝑧, 𝑥) ↦→ ∥𝑥 − 𝑧∥ 2 − 𝑟 2 is continuous, the Lagrangian of the minimum enclosing ball problem (4.51) is ∫

2 ∥𝑥 − 𝑧∥ 2 − 𝑟 2 𝑑𝜇(𝑥), 𝑟 ∈ R, 𝑧 ∈ 𝑉, 𝜇 ∈ M 𝜑(Θ𝑥 (𝛼)) . 𝐿(𝑟, 𝑧; 𝜇) := 𝑟 + 𝜑(Θ 𝑥 (𝛼)) (4.104) Define Ψ(𝜇) := inf 𝐿 (𝑟, 𝑧; 𝜇) 𝑟∈R,𝑧∈𝑉 and observe that ∫    𝜑(Θ (𝛼)) ∥𝑥 − 𝑧∥ 2 𝑑𝜇(𝑥),  𝑥 inf 𝐿 (𝑟, 𝑧; 𝜇) =  𝑟∈R  −∞,  ∫ 𝑑𝜇 = 1 ∫ 𝑑𝜇 ≠ 1 so that ∫  2   𝜑(Θ (𝛼)) 𝑥 − E 𝜇 [𝑥] 𝑑𝜇(𝑥),  𝑥 Ψ(𝜇) := inf 𝐿 (𝑟, 𝑧; 𝜇) =  𝑟∈R,𝑧∈𝑉  −∞,  ∫ 𝑑𝜇 = 1 ∫ 𝑑𝜇 ≠ 1 and therefore the dual problem to the minimum enclosing ball problem (4.51) is max Ψ(𝜇) = 𝜇∈M (𝜑(Θ 𝑥 (𝛼))) max 𝜇∈P (𝜑(Θ 𝑥 (𝛼)))   E 𝜇 ∥𝑥 − E 𝜇 [𝑥] ∥ 2 , establishing the Lagrangian duality assertion. Moreover, since Θ𝑥 (𝛼) is compact it is Polish, that is Hausdorff and completely
metrizable, and since 𝜑 : Θ𝑥 (𝛼) → 𝜑 Θ𝑥 (𝛼) is continuous (Aliprantis and Border,
2006, Thm. 15.14) asserts that 𝜑∗ : P Θ𝑥 (𝛼) → P (𝜑(Θ𝑥 (𝛼))) is surjective. The change of variables formula (Aliprantis and Border, 2006, Thm. 13.46) establishes that the objective function of (4.49) satisfies     E𝜋 ∥𝜑 − E𝜋 [𝜑] ∥ 2 = E𝜑∗ 𝜋 ∥𝑣 − E𝜑∗ 𝜋 [𝑣] ∥ 2 , so that the surjectivity of 𝜑∗ implies that the value of (4.49) is equal to max 𝜈∈P (𝜑(Θ 𝑥 (𝛼)))   E𝜈 ∥𝑣 − E𝜈 [𝑣] ∥ 2 . (4.105) The primary assertions then follow from Lim and McCann’s (Lim and McCann, 2021, Thm. 1) generalization of the one-dimensional result of Popoviciu (Popoviciu, 1935) regarding the relationship between variance maximization and the minimum enclosing ball of the domain 𝜑(Θ𝑥 (𝛼)) ⊂ 𝑉. 164 Proof of Theorem 4.3.5 The proof of Theorem 4.7.2 using rarity assumptions of the current form (4.33) does not require that the likelihood function 𝑝(𝑥 ′, ·) be continuous for all 𝑥 ′ ∈ 𝑋 but only at 𝑥. It asserts the finite-dimensional reduction (4.54) for 𝑚 ≥ 𝑑𝑖𝑚(𝑉) + 2 Dirac measures. On the other hand, the duality Theorem 4.3.3 implies that the Í𝑚 optimality of such a measure 𝜋 := 𝑖=1 𝑤 𝑖 δ𝜃 𝑖 is equivalent to the images of these Diracs δ𝜑(𝜃 𝑖 ) , 𝑖 = 1, . . . , 𝑚 lying on the intersection 𝜑(Θ𝑥 (𝛼)) ∩ 𝜕𝐵 of the image of the likelihood region and the boundary of its minimum enclosing ball 𝐵, and that the weights of these Diracs determine that the center of mass of this image measure 𝜑∗ 𝜋 is the center 𝑑 𝛼 of the ball 𝐵, expressed as the right-hand side of (4.55). Consequently, the center 𝑑 𝛼 is in the convex hull of the 𝑚 points 𝜑(𝜃 𝑖 ), 𝑖 = 1, . . . , 𝑚 and by Caratheodory’s theorem, see e.g. (Rockafellar, 1970), 𝑑 𝛼 is in the convex hull of 𝑑𝑖𝑚(𝑉) + 1 of these points. Let 𝑆 ⊂ {1, . . . , 𝑚} correspond to such a subset. Then by the if and only if characterization of duality Theorem 4.3.3 it follows that the subset 𝜑(𝜃 𝑖 ), 𝑖 ∈ 𝑆 of 𝑑𝑖𝑚(𝑉) + 1 image points, using the weights 𝑤 ′𝑖 , 𝑖 ∈ 𝑆 defining this convex combination to be the center 𝑑 𝛼 , corresponds to an optimal Í measure 𝜋′ := 𝑖∈𝑆 𝑤 ′𝑖 δ𝜃 𝑖 , thus establishing the assertion. Proof of Theorem 4.4.1 The following are standard results in the statistics literature, see (G. Casella and R. L. Berger, 2002). The idea is to write the Taylor expansion of the log-likelihood around the MLE, then consider properties of the MLE, and finally apply the law of large numbers and Slutsky’s theorem. For simplicity, first let 𝜃 ∈ Θ ⊆ R. Since the second term on the right-hand side in the Taylor expansion 𝑁 ∑︁ 𝑖=1 ln 𝑝(𝑥𝑖 |𝜃) = 𝑁 ∑︁ 𝑖=1 ln 𝑝(𝑥𝑖 | b 𝜃𝑁 ) + 𝑁 ∑︁ 𝜕 𝜕𝜃 𝑖=1 ln 𝑝(𝑥𝑖 |𝜃)𝜃=b𝜃 𝑁 (𝜃 − b 𝜃𝑁 ) 𝑁 1 ∑︁ 𝜕 2 + ln 𝑝(𝑥𝑖 |𝜃)𝜃=b𝜃 𝑁 (𝜃 − b 𝜃 𝑁 ) 2 + 𝑜 𝑝 (1) 2 2 𝑖=1 𝜕𝜃 of the log-likelihood around the MLE, b 𝜃 𝑁 , vanishes by the first order condition of the MLE, we obtain 𝑁 ∑︁ 𝑁 ∑︁ 𝑁 1 ∑︁ 𝜕 2 ln 𝑝(𝑥𝑖 |𝜃) − ln 𝑝(𝑥𝑖 | b 𝜃𝑁 ) = ln 𝑝(𝑥𝑖 |𝜃)𝜃=b𝜃 𝑁 (𝜃 − b 𝜃 𝑁 ) 2 + 𝑜 𝑝 (1), 2 2 𝜕𝜃 𝑖=1 𝑖=1 𝑖=1 165 which we write as 𝑁 √ 𝑝(𝑥𝑖 |𝜃)
1 ∑︁ 𝜕 2 −2 ln =− ln 𝑝(𝑥𝑖 |𝜃)𝜃=b𝜃 𝑁 ( 𝑁 (𝜃 − b 𝜃 𝑁 )) 2 + 𝑜 𝑝 (1). 2 𝑁 b 𝜕𝜃 𝑝(𝑥𝑖 | 𝜃 𝑁 ) 𝑖=1 𝑖=1 𝑁 ∑︁ By the law of large numbers and the consistency of the MLE we have 𝑁 1 ∑︁ 𝜕 2 𝜕2 P − ln 𝑝(𝑥𝑖 |𝜃)𝜃=b𝜃 𝑁 → − −𝐸 𝑋∼𝑃(·|𝜃) 2 ln 𝑝(𝑋 |𝜃) = 𝐼 (𝜃), 𝑁 𝑖=1 𝜕𝜃 2 𝜕𝜃 (4.106) P 2 𝜕 − represents where 𝐼 (𝜃) := −𝐸 𝑋∼𝑃(·|𝜃) 𝜕𝜃 2 ln 𝑝(𝑋 |𝜃) is the Fisher information and → convergence in probability. By the asymptotic efficiency of the MLE (under our regularity assumptions), the variance 𝑉0 (𝜃) of the MLE b 𝜃 𝑁 is the inverse of the Fisher information. That is 𝐼 (𝜃) = (𝑉0 (𝜃)) −1 . Moreover, under the regularity conditions of (G. Casella and R. L. Berger, 2002, Section. 10.6.2), we have √ 𝑁 (b 𝜃 𝑁 − 𝜃) d → − 𝑁 (0, 1), √︁ 𝑉0 (𝜃) d where → − represents convergence in distribution, and therefore Slutsky’s theorem implies   𝑁 ∑︁ 𝑝(𝑥𝑖 |𝜃) d 2 → − 𝜒1 . −2 ln 𝑝(D|𝜃) ¯ = −2 ln 𝑝(𝑥𝑖 | b 𝜃𝑁 ) 𝑖=1 In the more general case Θ ⊂ R 𝑘 , under the regularity conditions of (G. Casella and R. L. Berger, 2002, Section. 10.6.2), we have √ d 𝑁 (b 𝜃 𝑁 − 𝜃) → − 𝑁 (0, 𝐼 (𝜃) −1 ) , (4.107) where 𝐼 (𝜃) is the Fisher information matrix, and the same argument goes through obtaining d 𝑎 −2 ln 𝑝(D|𝜃) ¯ = 𝑁 (𝜃 − b 𝜃 𝑁 )𝐼 (𝜃) −1 (𝜃 − b 𝜃𝑁 ) → − 𝜒𝑘2 , 𝑎 where = represents asymptotic equality. Therefore, for large sample sizes, one may use the following approximation: −2 ln 𝑝(D|𝜃) ¯ ≈ 𝜒𝑘2 . 166 Finally, the condition 𝑝(D|𝜃) ¯ < 𝛼 in the computation of 𝛽𝛼 is the same as 1 −2 ln 𝑝(D|𝜃) ¯ > 2 ln( ) , 𝛼 so that consequently, for large sample sizes, 𝛽𝛼 may be approximated by 1
𝛽𝛼 ≈ 1 − 𝜒𝑘2 2 ln( ) . 𝛼 Proof of Theorem 4.4.2 For case (A) the proof is an application of Theorem 4.4.1. Case (B) follows from a direct adaptation of the proof of Theorem 4.4.1. The first main step (in this adaptation) is to use the convergence of 𝑄 𝑁 and the Independence of the 𝑦𝑖 given the 𝑡𝑖 to replace (4.106) by 𝑁 − 𝜕2 1 ∑︁ 𝜕 2 P ln 𝑝(𝑦 |𝜃, 𝑡 ) → − −𝐸 ln 𝑝(𝑦|𝜃, 𝑡) . 𝑖 𝑖 𝜃=b 𝑡∼𝑄,𝑦∼𝑃(·|𝜃,𝑡) 𝜃𝑁 𝑁 𝑖=1 𝜕𝜃 2 𝜕𝜃 2 (4.108) The second main step is to derive the asymptotic consistency and normality (4.107) of the MLE in case (B). This can be done by adapting the proofs of (Dudley, 2009, Lec. 5). Proof of Theorem 4.7.2 We prove the theorem for the integral rarity case only, the pointwise rarity case being much simpler. First note that Theorem 4.8.1 asserts that the saddle function L satisfies a minmax equality, in particular, that the inner loop min𝑑∈𝑉 L (𝜋, 𝑑) of the primary assertion (4.99) indeed has a solution. To analyze such a solution, recall, by (4.85), that L (𝜋, 𝑑) is the expectation L (𝜋, 𝑑) := E𝜃∼𝜋 𝑥 [ℓ(𝜙(𝜃), 𝑑)] . (4.109) It is easy to show that the expectation of a family of strictly convex functions is strictly convex, so that it follows, for fixed 𝜋, that L (𝜋, 𝑑) is strictly convex. Since 𝜙 is continuous and Θ is compact, it follows that the image 𝜙(Θ) ⊂ 𝑉 is compact and since ℓ is coercive in its second variable, continuous in its first and 𝜙(Θ) is compact, it follows that ℓ is uniformly coercive in 𝜃; that is, for every 𝑦 ∈ R+ , there exists an 𝑅 ∈ R+ such that |𝑑| ≥ 𝑅 =⇒ ℓ(𝜙(𝜃), 𝑑) ≥ 𝑦, 𝜃 ∈ Θ. It follows that L (𝜋, 𝑑) ≥ 𝑦, |𝑑| ≥ 𝑅, 𝜋 ∈ P (Θ). Since 𝑦 is arbitrary and L is convex, it follows that L achieves its minimum and since it is strictly convex this minimum is achieved at a unique point 𝑑 𝜋 ; see e.g. (Rockafellar and Wets, 1998). 167 Since ℓ is continuous in its first variable, 𝜙 is continuous, and Θ compact, it follows that ℓ(𝜙(𝜃), 0) is uniformly bounded in 𝜃 and therefore implies a uniform bound on L (𝜋, 0), 𝜋 ∈ P (Θ). Consequently, the coerciveness of L implies that we can uniformly bound the unique optima 𝑑 𝜋 , 𝜋 ∈ P (Θ). Let 𝑉 ∗ ⊃ 𝜙(Θ) be a closed cube in 𝑉 containing an open neighborhood of the image 𝜙(Θ) and this feasible set of optima just discussed. Since 𝜙 is continuous, Θ is compact, and ℓ is continuously differentiable, it follows that ∇𝑑 ℓ(𝜙(𝜃), 𝑑) is uniformly bounded in both 𝜃 and 𝑑 ∈ 𝑉 ∗ . Consequently, the Leibniz theorem for differentiation under the integral sign, (see e.g. (Aliprantis and Burkinshaw, 1998, Thm. 24.5)), implies, for fixed 𝜋, that   ∇𝑑 L (𝜋, 𝑑) := E𝜃∼𝜋 𝑥 ∇𝑑 ℓ(𝜙(𝜃), 𝑑) , 𝑑 ∈ 𝑉∗ . (4.110) Consequently, the relation 0 = ∇𝑑 L (𝜋, 𝑑 𝜋 ) at the unique minimum 𝑑 𝜋 of min𝑑∈𝑉 L (𝜋, 𝑑) implies that   E𝜃∼𝜋 𝑥 ∇𝑑 ℓ(𝜙(𝜃), 𝑑 𝜋 ) = 0, 𝜋 ∈ P (Θ) . (4.111) The formula (4.46) for the conditional measure and the positivity of its denominator imply that we can write (4.111) as   1 E𝜃∼𝜋 𝑝(𝑥|𝜃)∇𝑑 ℓ(𝜙(𝜃), 𝑑 𝜋 ) = 0 𝑝(𝑥|·)𝑑𝜋 Θ ∫ (4.112) which is equivalent to   E𝜃∼𝜋 𝑝(𝑥|𝜃)∇𝑑 ℓ(𝜙(𝜃), 𝑑 𝜋 ) = 0 . (4.113) Consequently, adding the constraint (4.113), equivalent to the minimization problem, the maxmin problem on the left-hand side of (4.99) can be written    Maximize L (𝜋, 𝑑)     Subject to 𝜋 ∈ P𝑥Ψ (𝛼), 𝑑 ∈ 𝑉 ∗        E𝜃∼𝜋 𝑝(𝑥|𝜃)∇𝑑 ℓ(𝜙(𝜃), 𝑑) = 0  (4.114) which again using the conditional formula (4.46) and the definition of L (𝜋, 𝑑) can be written    1  ∫ Maximize E 𝑝(𝑥|𝜃)ℓ(𝜙(𝜃), 𝑑)  𝜃∼𝜋  𝑝(𝑥|𝜃)𝑑𝜋(𝜃)  Θ   (4.115) Subject to 𝜋 ∈ P𝑥Ψ (𝛼), 𝑑 ∈ 𝑉 ∗        E𝜃∼𝜋 𝑝(𝑥|𝜃)∇𝑑 ℓ(𝜙(𝜃), 𝑑) = 0  168 which, introducing a new variable, can be written      Maximize 1𝜖 E𝜃∼𝜋 𝑝(𝑥|𝜃)ℓ(𝜙(𝜃), 𝑑)     Subject to 𝜋 ∈ P𝑥Ψ (𝛼), 𝑑 ∈ 𝑉 ∗ , 𝜖 > 0        E𝜃∼𝜋 𝑝(𝑥|𝜃)∇𝑑 ℓ(𝜙(𝜃), 𝑑) = 0, 𝜖 = E𝜃∼𝜋 [ 𝑝(𝑥|𝜃)] .  (4.116) Now fix 𝜖 > 0 and 𝑑 ∈ 𝑉 ∗ and consider the inner maximization loop      Maximize 1𝜖 E𝜃∼𝜋 𝑝(𝑥|𝜃)ℓ(𝜙(𝜃), 𝑑)     Subject to 𝜋 ∈ P𝑥Ψ (𝛼),        E𝜃∼𝜋 𝑝(𝑥|𝜃)∇𝑑 ℓ(𝜙(𝜃), 𝑑) = 0, 𝜖 = E𝜃∼𝜋 [ 𝑝(𝑥|𝜃)] .  (4.117) Since this is linear optimization of the integration of a non-negative, and thus integrable function, with possible integral value +∞ for all 𝜋, over the full simplex of probability measures subject to dim(𝑉) + 1 linear equality constraints defined by integration against measurable functions, plus one linear inequality constraint defined by integration against a measurable function, (H. Owhadi, C. Scovel, T. J. Sullivan, et al., 2013b, Thm. 4.1), which uses von Weizsacker and Winkler (Weizsäcker and Winkler, 1979, Cor. 3) (see also (Karr, 1983)) which is applicable under more assumptions on the model P, implies this optimization problem can be reduced to optimization over the convex combination of 𝑑𝑖𝑚(𝑉) + 3 Dirac measures supported on Θ. Since the full problem is the supremum of such problems, using the compactness of the space P𝑥Ψ,𝑚 (𝛼) in the weak topology, the primary assertion follows. When Ψ is the identity function one of the constraints disappears, and the assertion in that case follows. Proof of Theorem 4.7.4 The proof follows from the invariance of the variance under 𝜑∗ and the equality of their maximum variance problems established in the proof of Theorem 4.3.3 and Lim and McCann’s (Lim and McCann, 2021, Thm. 2) generalization of their ℓ 2 result (Lim and McCann, 2021, Thm. 1). Proof of Theorem 4.5.1 First consider the 𝜖 > 0 case. Our proof will use results from Bădoiu, Har-Peled and Indyk (Bădoiu, Har-Peled, and Indyk, 2002). Consider the REPEAT loop. As previously mentioned, the FOR loop always gets broken at Step 10 for some 𝑥 since, by Theorem 4.3.3, the center of the ball must lie in the convex hull of the 𝑛 + 2 points, but by Caratheodory’s theorem this center also lies in the convex hull of 𝑛 + 1 169 of those points, and Theorem 4.3.3 then asserts that this ball is also the minimum enclosing ball of those 𝑛 + 1 points. Clearly, the breaking of the step implies that the elimination of the point does not change the current ball. Consequently, the only change in the current ball is through the discovery in Step 14 of a distant point and its addition to the working set 𝐴 𝑘 followed by the calculation in Step 6 of a new minimum ball containing this enlarged working set. Let 𝐵( 𝐴 𝑘 ) denote the minimum enclosing ball of 𝐴 𝑘 , 𝑅( 𝐴 𝑘 ) denote its radius, and, overloading notation, let us denote 𝑅( 𝐴 𝑘 ) := 𝑅(𝐵 𝑘 ). Then when a new point is added to 𝐴 𝑘 to obtain 𝐴 𝑘+1 , it follows from 𝐴 𝑘 ⊂ 𝐴 𝑘+1 that 𝐵( 𝐴 𝑘+1 ) ⊃ 𝐴 𝑘+1 ⊃ 𝐴 𝑘 and therefore 𝑅( 𝐴 𝑘+1 ) ≥ 𝑅( 𝐴 𝑘 ). Likewise 𝐴 𝑘 ⊂ 𝐾 implies that 𝑅( 𝐴 𝑘 ) ≤ 𝑅, the radius of the minimum enclosing ball 𝐵 of 𝐾. Consequently the sequence 𝑅( 𝐴 𝑘 ) ≤ 𝑅 of the radii of the balls is monotonically increasing and bounded by 𝑅. Moreover, (Bădoiu, Har-Peled, and Indyk, 2002, pp. Clm. 2.4), using (Bădoiu, Har-Peled, and Indyk, 2002, Lem. 2.2) from Goel et al. (Goel, Indyk, and Varadarajan, 2001), implies that, until the stopping criterion in Step 17 is satisfied, we have 𝑅( 𝐴 𝑘+1 ) ≥ 1 + 𝜖2
𝑅( 𝐴 𝑘 ), 16 (4.118) and when the stopping criterion is satisfied it follows from Step 14 that the output in Step 18 satisfies (1+𝜖) (1+𝛿) 𝐵 𝑘−1 ⊃ 𝐾. (4.119) Observe that we have 𝑅 ≤ Δ where Δ := diam(𝐾) and the initialization implies that 1 Δ. Consequently (4.118) implies that the radius 𝑅( 𝐴 𝑘 ) increases by 𝑅( 𝐴0 ) ≥ 2(1+𝛿) 2 𝜖 Δ at each step. Since the sequence is bounded by 𝑅 ≤ Δ it follows at least 32((1+𝛿) 16 that at most 𝜖 2 (1 + 2𝛿) of the REPEAT loop can be taken before terminating at (1+𝜖) (1+𝛿) Step 17. Upon termination the returned ball 𝐵∗ := 𝐵 𝑘−1 in Step 18, by (4.119), satisfies 𝐵∗ ⊃ 𝐾, and since (1+𝜖) (1+𝛿) 𝑅(𝐵 𝑘−1 ) = (1 + 𝜖)(1 + 𝛿)𝑅(𝐵 𝑘−1 ) ≤ (1 + 𝜖)(1 + 𝛿)𝑅 we obtain
𝑅 𝐵∗ ≤ (1 + 𝜖)(1 + 𝛿)𝑅, establishing the primary assertion. Since each step in the REPEAT loop adds at most one new point to the working set 𝐴 𝑘 , it follows that the working set size is bounded 170 by 2 plus the number of steps in the REPEAT loop, that is 2 + 16 (1 + 2𝛿). Since the 𝜖2 WHILE loop keeps the bound ≤ 𝑛 + 2, the proof is finished. Now consider the 𝜖 = 0 case. First let 𝛿 = 0. By compactness of the set 𝐾, there exists a sub-sequence ( 𝐴𝑡1 ), indexed by 𝑇1 ⊆ N, of ( 𝐴𝑡 ) such that 𝐶 ( 𝐴𝑡1 ) → 𝐶1 , in our notation whenever the algorithm stops or we reach to a fixed point the sequence repeats the last set of points. For every 𝑡1 ∈ 𝑇1 , let 𝑦 𝑡1 be the point selected as a furthest point from 𝐵( 𝐴𝑡1 ) in the algorithm to form 𝐴𝑡1 +1 . Again by compactness of the set 𝐾, there exists a sub-sequence (𝑦 𝑡2 ), indexed by 𝑇2 ⊆ 𝑇1 ⊆ N, of (𝑦 𝑡1 ) such that 𝑦 𝑡2 → 𝑦 ∗ . By another application of compactness of 𝐾, there exists a sub-sequence ( 𝐴𝑡3 ), indexed by 𝑇3 ⊆ 𝑇2 ⊆ 𝑇1 ⊆ N, of ( 𝐴𝑡2 ) such that 𝐶 ( 𝐴𝑡3 ∪ {𝑦 𝑡3 }) → 𝐶2 . Since 𝐶 ( 𝐴𝑡3 ) → 𝐶1 and 𝑅( 𝐴𝑡3 ) ↑ 𝑅0 ≤ 𝑅(𝐾) as 𝑡3 ∈ 𝑇3 → ∞, it follows that 𝐵( 𝐴𝑡3 ) → 𝐵(𝐶1 , 𝑅0 ). To complete the proof it is sufficient to show that 𝐾 ⊆ 𝐵(𝐶1 , 𝑅0 ), since then 𝑅0 ≤ 𝑅(𝐾) implies that 𝐵(𝐶1 , 𝑅0 ) = 𝐵(𝐾). To that end, we demonstrate that 𝑑 := max 𝑑𝑖𝑠𝑡 (𝑥, 𝐵(𝐶1 , 𝑅0 )) = 0. 𝑥∈𝐾 Since 𝑑𝑖𝑠𝑡 () is a continuous function in both of its arguments, 𝐵( 𝐴𝑡3 ) → 𝐵(𝐶1 , 𝑅0 ) 𝑘 𝑘 as 𝑡3 ∈ 𝑇3 → ∞, 𝑦 𝑡3 ∈ argmax𝑥∈𝐾 𝑑𝑖𝑠𝑡 (𝑥, 𝐵( 𝐴𝑡 𝑘 )) for every 𝑡3𝑘 ∈ 𝑇3 and 𝑦 𝑡3 → 𝑦 ∗ , 3 it follows that 𝑑𝑖𝑠𝑡 (𝑦 ∗ , 𝐵(𝐶1 , 𝑅0 )) = 𝑑 . (4.120) By the choice of 𝑇3 , 𝑦 𝑡3 → 𝑦 ∗ , 𝐵( 𝐴𝑡3 ) → 𝐵(𝐶1 , 𝑅0 ), and 𝐵( 𝐴𝑡3 ∪ {𝑦 𝑡3 }) → 𝐵(𝐶2 , 𝑅0 ). Therefore, for 𝜖1 > 0, there exists a large number 𝑁𝜖1 ∈ N, such that for all 𝑡3 ∈ 𝑇3 with 𝑡3 ≥ 𝑁𝜖1 , we have 𝐴𝑡3 ⊆ 𝐵(𝐶1 , 𝑅0 + 𝜖 1 ) and 𝐴𝑡3 ∪ {𝑦 𝑡3 } ∪ 𝑦 ∗ ⊆ 𝐵(𝐶2 , 𝑅0 + 𝜖1 ). (4.121) Consequently (4.120), (4.121), and the triangle inequality imply 𝑑𝑖𝑠𝑡 (𝐶1 , 𝐶2 ) ≥ 𝑑𝑖𝑠𝑡 (𝑦 ∗ , 𝐶1 ) − 𝑑𝑖𝑠𝑡 (𝑦 ∗ , 𝐶2 ) ≥ (𝑑 + 𝑅0 ) − (𝑅0 + 𝜖1 ) = 𝑑 − 𝜖 1 and 𝐴𝑡3 ⊆ 𝐵(𝐶1 , 𝑅0 + 𝜖 1 ) ∩ 𝐵(𝐶2 , 𝑅0 + 𝜖1 ), 𝑡3 ∈ 𝑇3 , 𝑡3 ≥ 𝑁𝜖1 . 2 and consider the hyperplane orthogonal to the vector 𝐶1 − 𝐶2 Let 𝐶¯ = 𝐶1 +𝐶 2 ¯ passing 𝜖1 ) ∩ 𝐵(𝐶2 , 𝑅0 + 𝜖1 ) ⊆ √︃ through 𝐶. By Pythagoras’ Theorem, 𝐵(𝐶1 , 𝑅0 + √︃ 𝑑−𝜖 1 ¯ (𝑅0 + 𝜖1 ) 2 − ( ¯ (𝑅0 + 𝜖1 ) 2 − ( 𝑑−𝜖1 ) 2 ), 𝐵(𝐶, ) 2 ) and therefore, 𝐴𝑡3 ⊆ 𝐵(𝐶, 2 2 171 √︃ 1 2 (𝑅0 + 𝜖1 ) 2 − ( 𝑑−𝜖 2 ) for all 𝑡 3 ∈ 𝑇3 with 𝑡 3 ≥ 𝑁 𝜖1 . √︃ By sending 𝜖1 to zero, we obtain that 𝑅( 𝐴𝑡3 ) ≤ (𝑅0 ) 2 − ( 𝑑2 ) 2 as 𝑡3 ∈ 𝑇3 → ∞, but 𝑅( 𝐴𝑡3 ) ↑ 𝑅0 implies 𝑑 = 0, which completes the proof. which implies that 𝑅( 𝐴𝑡3 ) ≤ For the general case 𝛿 ≥ 0, we can use the same technique. Let 𝐾 ′ = (𝑦 𝑡 )𝑡∈N be a sequence of points selected as the furthest point in Step 14 (with the relative error size of 𝛿) in one complete execution of the algorithm. Using the result and the language of the case 𝛿 = 0 applied to the set 𝐾 ′, we obtain that 𝐵(𝐾 ′) = 𝐵(𝐶1 , 𝑅0 ), where 𝐶1 and 𝑅0 are the center and radius returned by the algorithm as 𝑡 → ∞, with the convention that whenever the algorithm stops we repeat the last set of points up to infinity. Note that max𝑥∈𝐾 𝑑𝑖𝑠𝑡 (𝐶1 , 𝑥) ≤ (1 + 𝛿)𝑅0 , since otherwise the algorithm would have not converged to 𝐶1 , and therefore, 𝐾 ⊆ 𝐵(𝐶1 , (1 + 𝛿)𝑅0 ) which implies that 𝑅(𝐾) ≤ (1+𝛿)𝑅0 . Moreover, by 𝐾 ′ ⊆ 𝐾 we have 𝑅0 = 𝑅(𝐾 ′) ≤ 𝑅(𝐾) and therefore 𝑅0 ≤ 𝑅(𝐾) ≤ (1 + 𝛿)𝑅0 , completing the proof. 172 Chapter 5 KERNEL METHODS ARE COMPETITIVE FOR OPERATOR LEARNING We present a general kernel-based framework for learning operators between Banach spaces along with a priori error analysis and comprehensive numerical comparisons with popular neural net (NN) approaches such as Deep Operator Networks (DeepONet) (L. Lu, Jin, et al., 2021) and Fourier Neural Operator (FNO) (Li, Kovachki, Azizzadenesheli, B. Liu, et al., 2020). We consider the setting where the input/output spaces of target operator G † : U → V are reproducing kernel Hilbert spaces (RKHS), the data comes in the form of partial observations 𝜙(𝑢𝑖 ), 𝜑(𝑣 𝑖 ) of input/output functions 𝑣 𝑖 = G † (𝑢𝑖 ) (𝑖 = 1, . . . , 𝑁), and the measurement operators 𝜙 : U → R𝑛 and 𝜑 : V → R𝑚 are linear. Writing 𝜓 : R𝑛 → U and 𝜒 : R𝑚 → V for the optimal recovery maps associated with 𝜙 and 𝜑, we approximate G † with Ḡ = 𝜒 ◦ 𝑓¯ ◦ 𝜙 where 𝑓¯ is an optimal recovery approximation of 𝑓 † := 𝜑 ◦ G † ◦ 𝜓 : R𝑛 → R𝑚 . We show that, even when using vanilla kernels (e.g., linear or Matérn), our approach is competitive in terms of cost-accuracy tradeoff and either matches or beats the performance of NN methods on a majority of benchmarks. Additionally, our framework offers several advantages inherited from kernel methods: simplicity, interpretability, convergence guarantees, a priori error estimates, and Bayesian uncertainty quantification. As such, it can serve as a natural benchmark for operator learning. 5.1 Introduction Operator learning is a well-established field going back at least to the 1970s with the articles (Almroth, Stern, and Brogan, 1978; Noor and J. M. Peters, 1980) who introduced the reduced basis method as a way speeding up expensive model evaluations. In the most broad sense operator learning arises in the solution of stochastic PDEs (Ghanem and Spanos, 2003), emulation of computer codes (Kennedy and O’Hagan, 2001), reduced order modeling (ROM) (Lucia, Beran, and Silva, 2004), and numerical homogenization (Houman Owhadi and Clint Scovel, 2019a). In recent years, and with the rise of machine learning, operator learning has become the focus of extensive research with the development of neural net (NN) methods such as Deep Operator Nets (L. Lu, Jin, et al., 2021) and Fourier Neural Nets (Li, Kovachki, 173 Azizzadenesheli, B. Liu, et al., 2020) among many others. While these NN methods are often benchmarked against each other (L. Lu, X. Meng, et al., 2022), they are rarely compared with the aforementioned classical approaches. Furthermore, the theoretical analysis of NN methods is often limited to density/universal approximation results, showing the existence of a network of a requisite size achieving a certain error rate, without guarantees whether this network is computable in practice (see for example (Deng et al., 2022; Kovachki, Lanthaler, and Mishra, 2021)). In order to alleviate the aforementioned shortcomings we present a mathematical framework for approximation of mappings between Banach spaces using the theory of operator valued reproducing Kernel Hilbert spaces (RKHS) and Gaussian Processes (GPs). Our abstract framework is (1) mathematically simple and interpretable, (2) convenient to implement, (3) encompasses some of the classical approaches such as linear methods, and (4) comes with a priori error analysis and convergence theory. We further present extensive benchmarking of our kernel method with the DeepONet and FNO approaches and show that the kernel approach either matches or outperforms NN methods in most benchmark examples. In the remainder of this section we give a summary of our methodology and results: we pose the operator learning problem in Section 5.1 before presenting a running example in Section 5.1 which is used to outline our proposed framework and main theoretical results in Section 5.1 as well as brief numerical results in Section 5.1. Our main contributions are summarized in Section 5.1 followed by a literature review in Section 5.1. The operator learning problem Let U and V be two (possibly infinite-dimensional) separable Banach spaces and suppose that G† : U → V (5.1) is an arbitrary (possibly nonlinear) operator. Then, broadly speaking, the goal of operator learning is to approximate G † from a finite number 𝑁 of input/output data on G † . For our framework, we consider the setting where the input/output data are only partially observed through a finite collection of linear measurements which we formalize as follows: 𝑁 be 𝑁 elements of U × V such that Problem 2. Let {𝑢𝑖 , 𝑣𝑖 }𝑖=1 G † (𝑢𝑖 ) = 𝑣 𝑖 , for 𝑖 = 1, . . . , 𝑁 . (5.2) 174 Let 𝜙 : U → R𝑛 and 𝜑 : V → R𝑚 be bounded linear operators. Given the data 𝑁 approximate G † . {𝜙(𝑢𝑖 ), 𝜑(𝑣 𝑖 )}𝑖=1 Running example To give context to the above problem and our solution method we briefly outline a running example to which the reader can refer to throughout the rest of this section. Consider the following elliptic PDE, which is of broad interest in geosciences and material science: ( −div 𝑒 𝑢 ∇𝑣 = 𝑤, in Ω, (5.3) 𝑣 = 0, on 𝜕Ω , where Ω = (0, 1) 2 , 𝑢 ∈ 𝐻 3 (Ω), 𝑤 ∈ 𝐻 1 (Ω) and 𝑣 ∈ 𝐻 3 (Ω) ∩ 𝐻01 (Ω). For a fixed forcing term 𝑤, we wish to approximate the nonlinear operator mapping the diffusion coefficient 𝑢 to the solution 𝑣, i.e., G † : 𝑢 ↦→ 𝑣. In this case we may take U ≡ 𝐻 3 (Ω) and V ≡ 𝐻 3 (Ω) ∩ 𝐻01 (Ω). We further assume that a training data set is available in the form of limited observations of input-out pairs. As a canonical example, consider the evaluation bounded and linear operators 𝜙 : 𝑢 ↦→ (𝑢(𝑋1 ), 𝑢(𝑋2 ), . . . , 𝑢(𝑋𝑛 ))𝑇 𝜑 : 𝑣 ↦→ (𝑣(𝑌1 ), 𝑣(𝑌2 ), . . . , 𝑣(𝑌𝑚 ))𝑇 , (5.4) 𝑛 𝑚 where the {𝑋 𝑗 } 𝑗=1 and {𝑌 𝑗 } 𝑗=1 are distinct collocation points in the domain Ω as 𝑁 that satisfy the PDE (5.3). Then our goal is to approximate well as pairs {𝑢𝑖 , 𝑣𝑖 }𝑖=1 𝑁 1. G † from the training data set {𝜙(𝑢𝑖 ), 𝜑(𝑣 𝑖 )}𝑖=1 and The proposed solution Our setup naturally gives rise to a commutative diagram depicted in Figure 5.1. Here the map 𝑓 † : R𝑛 → R𝑚 explicitely defined as 𝑓 † := 𝜑 ◦ G † ◦ 𝜓 (5.5) is a mapping between finite-dimensional Euclidean spaces, and is therefore amenable to numerical approximation. However, in order to approximate G † we also need the reconstruction maps 𝜓 : R𝑛 → U and 𝜒 : R𝑚 → V. Our proposed solution is to endow U and V with an RKHS structure and use kernel/GP regression to identify the maps 𝜓 and 𝜒. As a prototypical example we consider the situation where U is an RKHS of functions 𝑢 : Ω → R defined by a 1 Choosing 𝜙, 𝜑 as pointwise evaluation functionals is common to many applications, although our abstract framework readily accommodates other choices such as integral operators and basis projections 175 U 𝜓 𝜙 R𝑛 G† 𝑓† V 𝜒 𝜑 R𝑚 Figure 5.1: Commutative diagram of our operator learning setup. kernel 𝑄 : Ω × Ω → R and V is an RKHS of functions 𝑢 : 𝐷 → R defined by a kernel 𝐾 : 𝐷 × 𝐷 → R. For our running example, we have 𝐷 = Ω, and we can take 𝑄 and 𝐾 to be Matérn like kernels, e.g., the Green’s function of elliptic PDEs (possibly on Ω or restricted to Ω) with appropriate regularity. One can also choose 𝑄, 𝐾 to be smoother kernels such that their RKHSs are embedded in U and V. We then define 𝜓 and 𝜒 as the following optimal recovery maps 2 : 𝜓(𝑈) := argmin𝑤∈U ∥𝑤∥ 𝑄 s.t. 𝜙(𝑤) = 𝑈, 𝜒(𝑉) := argmin𝑤∈V ∥𝑤∥ 𝐾 s.t. 𝜑(𝑤) = 𝑉, (5.6) where ∥ · ∥ 𝑄 and ∥ · ∥ 𝐾 are the RKHS norms arising from their pertinent kernels. In the case where 𝜙 and 𝜑 are pointwise evaluation maps (𝜙(𝑢) = (𝑢(𝑋1 ), . . . , 𝑢(𝑋𝑛 )) and 𝜑(𝑣) = (𝑣(𝑌1 ), . . . , 𝑣(𝑌𝑚 )) where the 𝑋𝑖 and 𝑌 𝑗 are pairwise distinct collocation points in Ω and 𝐷), our optimal recovery maps can be expressed in closed form using standard representer theorems for kernel interpolation (Schölkopf, Herbrich, and Alex J. Smola, 2001): 𝜓(𝑈)(𝑥) = 𝑄(𝑥, 𝑋)𝑄(𝑋, 𝑋) −1𝑈, 𝜒(𝑉)(𝑦) = 𝐾 (𝑦, 𝑌 )𝐾 (𝑌 , 𝑌 ) −1𝑉, (5.7) where 𝑄(𝑋, 𝑋) and 𝐾 (𝑌 , 𝑌 ) are kernel matrices with entries 𝑄(𝑋, 𝑋)𝑖 𝑗 = 𝑄(𝑋𝑖 , 𝑋 𝑗 ) and 𝐾 (𝑌 , 𝑌 )𝑖 𝑗 = 𝐾 (𝑌𝑖 , 𝑌 𝑗 ) respectively, while 𝑄(𝑥, 𝑋) and 𝐾 (𝑦, 𝑌 ) denote rowvector fields with entries 𝑄(𝑥, 𝑋)𝑖 = 𝑄(𝑥, 𝑋𝑖 ) and 𝐾 (𝑦, 𝑌 )𝑖 = 𝐾 (𝑦, 𝑌𝑖 ). We further propose to approximate 𝑓 † by optimal recovery in a vector-valued RKHS. Let Γ : R𝑛 × R𝑛 → L (R𝑚 ) be a matrix valued kernel (Alvarez, Rosasco, Lawrence, et al., 2012); here L (R𝑚 ) is the space of 𝑚 × 𝑚 matrices) with RKHS HΓ equipped with the norm ∥ · ∥ Γ 3 and proceed to approximate 𝑓 † by the map 𝑓¯ defined as 𝑓¯ := argmin 𝑓 ∈HΓ ∥ 𝑓 ∥ Γ s.t. 𝑓 (𝜙(𝑢𝑖 )) = 𝜑(𝑣 𝑖 ) for 𝑖 = 1, . . . , 𝑁. 2 It is possible to define the optimal recovery maps 𝜓, 𝜒 in the setting where 𝜙 and 𝜓 are nonlinear, following the general framework of (Yifan Chen, Hosseini, et al., 2021a; Houman Owhadi, 2022; Houman Owhadi, 2023a). However, in this setting the closed form formulae (5.7) no longer hold. 3 See Section 5.6 for a review of operator-valued kernels or the reference (Kadri et al., 2016). 176 A simple and practical choice for Γ is the diagonal kernel Γ(𝑈, 𝑈 ′) = 𝑆(𝑈, 𝑈 ′)𝐼, (5.8) where 𝑆 : R𝑛 × R𝑛 → R is an arbitrary scalar-valued kernel, such as RBF, Laplace, or Matérn, and 𝐼 is the 𝑚 × 𝑚 identity matrix. More complicated choices, such as sums of kernels or replacing the identity matrix for a fixed positive definite matrix, implying correlations between various input or output correlations, are also possible. However, these may lead to greater computational cost and we observe empirically that the simple choice of the identity matrix already provides good performance. Then we can approximate the components of 𝑓¯ via the independent optimal recovery problems 𝑓¯𝑗 := argmin𝑔∈H𝑆 ∥𝑔∥ 𝑆 s.t. 𝑔(𝜙(𝑢𝑖 )) = 𝜑 𝑗 (𝑣 𝑖 ), for 𝑖 = 1, . . . , 𝑁 (5.9) for 𝑗 = 1, . . . , 𝑚. Here we wrote 𝜑 𝑗 (𝑣 𝑖 ) for the entry 𝑗 of the vector 𝜑(𝑣 𝑖 ) and, as our notation suggests, H𝑆 is the RKHS of 𝑆 equipped with the norm ∥ · ∥ 𝑆 . Since (5.9) is a standard optimal recovery problem, each 𝑓¯𝑗 can be identified by the usual representer formula: 𝑓¯𝑗 (𝑈) = 𝑆(𝑈, U)𝑆(U, U) −1 V·, 𝑗 , (5.10) where U := (𝜙(𝑢 1 ), . . . , 𝜙(𝑢 𝑁 )) and V·, 𝑗 := (𝜑 𝑗 (𝑣 1 ), . . . , 𝜑 𝑗 (𝑣 𝑁 ))𝑇 and 𝑆(𝑈, U) is a block-vector and 𝑆(U, U) is a block-matrix defined in an analogous manner to those in (5.7). By combining equations (5.7) and (5.10) we obtain the operator Ḡ := 𝜒 ◦ 𝑓¯ ◦ 𝜙 (5.11) as an approximation to G † . We provide further details and generalize the proposed framework in Section 5.2 to the setting where 𝜙 and 𝜑 are obtained from arbitrary linear measurements (e.g., integral operators as in tomography) and U and V may not be spaces of continuous functions. Convergence guarantee Under suitable regularity assumptions on G † , our method comes with worst-case convergence guarantees as the number of data points 𝑁 , i.e., input-output pairs and the number of collocations points 𝑛 and 𝑚 go to infinity. We present here a condensed version of this result and defer the proof to Section 5.3. Below we write 𝐵 𝑅 (H ) for the ball of radius 𝑅 > 0 in a normed space H . Theorem 5.1.1 (Condensed version of Thm. 5.3.4). Suppose it holds that: 177 (5.1.1.1) (Regularity of the domains Ω and 𝐷) Ω and 𝐷 are compact sets of finite dimensions 𝑑Ω and 𝑑 𝐷 and with Lipschitz boundary. (5.1.1.2) (Regularity of the kernels 𝑄 and 𝐾). Assume that H𝑄 ⊂ 𝐻 𝑠 (Ω) and H𝐾 ⊂ 𝐻 𝑡 (𝐷) for some 𝑠 > 𝑑Ω /2 and some 𝑡 > 𝑑 𝐷 /2 with inclusions indicating continuous embeddings. (5.1.1.3) (Space filling property of collocation points) The fill distance between the 𝑛 ⊂ Ω and the {𝑌 } 𝑚 ⊂ 𝐷 goes to zero as 𝑛 → ∞ collocation points {𝑋𝑖 }𝑖=1 𝑗 𝑗=1 and 𝑚 → ∞. ′ (5.1.1.4) (Regularity of the operator G † ) The operator G † is continuous from 𝐻 𝑠 (Ω) to H𝐾 for some 𝑠′ ∈ (0, 𝑠) as well as from U to V and all its Fréchet derivatives are bounded on 𝐵 𝑅 (H𝑄 ) for any 𝑅 > 0. (5.1.1.5) (Regularity of the kernels 𝑆 𝑛 ) Assume that for any 𝑛 ≥ 1 and any compact subset Υ of R𝑛 , the RKHS of 𝑆 𝑛 restricted to Υ is contained in 𝐻 𝑟 (Υ) for ′ some 𝑟 > 𝑛/2 and contains 𝐻 𝑟 (Υ) for some 𝑟 ′ > 0 that may depend on 𝑛. (5.1.1.6) (Resolution and space-filling property of the data) Assume that for 𝑛 suffi𝑁 ⊂ 𝐵 (H ) belong to the range of 𝜓 𝑛 ciently large, the data points (𝑢𝑖 )𝑖=1 𝑅 𝑄 and are space filling in the sense that they become dense in 𝜙𝑛 (𝐵 𝑅 (H𝑄 )) as 𝑁 → ∞. Then, for all 𝑡 ′ ∈ (0, 𝑡), lim lim sup 𝑛,𝑚→∞ 𝑁→∞ 𝑢∈𝐵 (H ) 𝑅 ∥G † (𝑢) − 𝜒 𝑚 ◦ 𝑓¯𝑁𝑚,𝑛 ◦ 𝜙𝑛 (𝑢)∥ 𝐻 𝑡 ′ (𝐷) → 0 , (5.12) 𝑄 where our notation makes the dependence of 𝜓, 𝜙, 𝜒, 𝑆, and 𝑓¯ on 𝑛, 𝑚, and 𝑁 explicit. We note that Assumptions (5.1.1.1)–(5.1.1.3) are standard, and concern the accuracy of the optimal recovery maps 𝜙𝑛 and 𝜒 𝑚 as 𝑛, 𝑚 → ∞. Assumptions (5.1.1.4)– (5.1.1.5) are less standard and amount to regularity assumptions on the map G † while Assumption (5.1.1.6) concerns the acquisition and regularity of the training data set. In Section 5.3 we also present Theorem 5.3.3 as the quantitative analogue of the above result which characterizes how the speed of convergence depends on the regularity of the operator G † and the choice of 𝜙 and 𝜑 in the setting of pointwise 178 measurement operators. We also comment on how this analysis could be extended to other linear measurements. Numerical Framework Returning to our running example, we implement the proposed framework for learning the non-linear operator mapping 𝑢 to 𝑣 in (5.3). We consider 1,000 inputs and outputs of 𝑢 and 𝑣. The data is taken from (L. Lu, X. Meng, et al., 2022) and the experimental setup is discussed further in Remark 8. We take 𝜑 to be of the form (5.4) with 𝑚 = 841 while we define 𝜙 through a PCA pre-processing step. More precisely, let 𝜙pointwise be of the form (5.4) with 𝑛 = 841. Choose 𝑛PCA = 202 (this value captures 95% of the empirical variance of our training data) and define 𝜙(𝑢) = ΠPCA ◦ 𝜙pointwise (𝑢) ∈ R202 . (5.13) In other words, we take our 𝜙 map to be the linear map that computes the first 202 PCA coefficients of the input functions 𝑢 given on a uniform grid; observe that we do not use PCA pre-processing on the output data here, although we do this for some of our other examples in Section 5.4 for better performance. With 𝜙 and 𝜑 identified (recall Figure 5.1) we proceed to implement our kernel method using the simple choice of a diagonal kernel 𝑆(𝑈, 𝑈 ′)𝐼 where 𝑆 is a rational quadratic (RQ) kernel (see Section 5.8). This choice transforms the problem into 841 independent kernel regression problems, each corresponding to one component of 𝑓 † (i.e., the 𝑓 𝑗† ’s). We used the PCA and kernel regression modules of the scikit-learn Python library (Pedregosa et al., 2011) to implement our algorithm. This implementation automatically selects the best kernel parameters by maximizing the marginal likelihood function (Rasmussen and Williams, 2006) jointly for all problems. Our proposed method can therefore be implemented conveniently using off-the-shelf software. Figure 5.2 illustrates examples of the inputs and outputs of our operator learning problem. Despite the simple implementation of our method, we are able to obtain competitive accuracy as shown in Table 5.1 where the relative testing 𝐿 2 loss of our method is compared to other popular algorithms. Moreover, our approach is amenable to well-known numerical analysis techniques, such as sparse or low-rank approximation of kernel matrices, to reduce its complexity. For the present example (and those in Section 5.4) we only consider “vanilla” kernel methods which compute (5.10) by computing the full Cholesky factors of the matrix 𝑆(U, U). 179 6 0 4 0.5 2 2 −0.25 0 2 x1 4 6 (a) Training input 6 1.5 4 1.0 4 1.0 6 0.06 4 0 0 2 x1 4 6 (b) Training output 0.0 2 0 0 2 x1 4 6 (c) True test 0.5 2 0.04 0.0 0 x2 0.00 1.5 x2 x2 4 6 1.0 x2 0.25 x2 6 0 2 x1 4 6 0.5 2 0.0 0 (d) Predicted test 0.02 0 2 x1 4 6 0.00 (e) Pointwise error Figure 5.2: Example of training data and test prediction and pointwise errors for the Darcy flow problem (5.3). Method DeepONet FNO POD-DeepONet Linear Rational quadratic Accuracy 2.91 % 2.41 % 2.32 % 6.74 % 2.87% Table 5.1: The 𝐿 2 relative test error of the Darcy flow problem in our running example. The kernel approach is compared with variations of DeepONet and FNO. Results of our kernel method are presented below the dashed line with the pertinent choice of the kernel 𝑆. Summary of contributions The main results of the article concern the properties, performance, and error analysis of the map Ḡ defined in (5.11). Our contributions can be summarized under four categories: 1. An abstract kernel framework for operator learning: In Section 5.2, we propose a framework for operator learning using kernel methods with several desirable properties. A family of methods of increasing complexity is proposed that includes linear models and diagonal kernels as well as nondiagonal kernels which capture output correlations. These properties make our approach ideal for benchmarking purposes. Furthermore, the methodology is (i) applicable to any choice of the linear functionals 𝜑 and 𝜙, (ii) minimax optimal with respect to an implicitly defined operator-valued kernel, and (iii) is mesh-invariant. We emphasize in remark 7 that our optimal recovery maps can be applied to any operator learning after training to obtain a mesh-invariant pipeline. 2. Error analysis and convergence rates for Ḡ: In Section 5.3, we develop rigorous worst-case a priori error bounds and convergence guarantees for 180 our method: Theorem 5.3.3 provides quantitative error bounds while Theorem 5.3.4 (the detailed version of Theorem 5.3.3) shows the convergence of Ḡ → G under appropriate conditions. 3. A simple to use vanilla kernel method: While our abstract kernel method is quite general, our numerical implementation in Section 5.4 focuses on a simple, easy-to-implement version using diagonal kernels of the form (5.8). Offthe-shelf software, such as the kernel regression modules of scikit-learn, can be employed for this task. We empirically observe low training times and robust choice of hyperparameters. These properties further suggest that kernel methods are a good baseline for benchmarking of more complex methods. 4. Competitive performance. In Section 5.4 we present a series of numerical experiments on benchmark PDE problems from the literature and observe that our simple implementation of the kernel approach is competitive in terms of complexity-accuracy tradeoffs in comparison to several NN-based methods. Since kernel methods can be interpreted as an infinite-width, one-layer NN, the results raise the question of how much of a role the depth of a deep NN plays in the performance of algorithms for the purposes of operator learning. Review of relevant literature In the most broad sense, operator learning is the problem of approximating a mapping between two infinite-dimensional function spaces (Bhattacharya et al., 2021; De Hoop et al., 2022). In recent years, this problem has become an area of intense research in the scientific machine learning community with a particular focus on parametric or stochastic PDEs. However, the approximation of such a parameter to solution maps has been an area of intense research in the computational mathematics and engineering communities, going back at least to the reduced basis method introduced in the 1970s (Almroth, Stern, and Brogan, 1978; Noor and J. M. Peters, 1980) as a way of speeding up the solution of families of parametric PDEs in applications that require many PDE solves such as design (Economon et al., 2016; Martins and Lambe, 2013; Bendsoe and Sigmund, 2003; Boncoraglio and Farhat, 2021), uncertainty quantification (UQ) (Sudret, Marelli, and Wiart, 2017; Martin et al., 2012; Huang, Schneider, and Andrew M Stuart, 2022), and multi-scale modeling (Weinan, 2011; Fish et al., 1997; Feyel and Chaboche, 2000; Kovachki, B. Liu, et al., 2022). In what follows we give a brief summary of the various areas and methodologies that overlap with operator learning; we cannot provide an exhaustive 181 list of references due to space, but refer the reader to key contributions and surveys where further references can be found. Deep learning techniques The use of NNs for operator learning goes back at least to the 90s and the seminal works of Chen and Chen (T. Chen and H. Chen, 1995b; T. Chen and H. Chen, 1995a) who proved a universal approximation theorem for NN approximations to operators. The use and design of NNs for operator learning has become popular in the last five years as a consequence of growing interest in NNs for scientific computing starting with the article (Zhu and Zabaras, 2018) which used autoencoders to build surrogates for UQ of subsurface flow models. Since then many different approaches have been proposed, some of which use specific architectures or target particular families of PDEs (J. Hesthaven and Ubbiali, 2018; Khoo and Ying, 2019; Li, Kovachki, Azizzadenesheli, B. Liu, et al., 2020; Khoo, J. Lu, and Ying, 2021; L. Lu, Jin, et al., 2021; Gin et al., 2021; Boullé and Townsend, 2022; Kröpfl, Maier, and Peterseim, 2022; Kissas et al., 2022). The most relevant of among these methods to our proposed framework are the DeepONet family (L. Lu, Jin, et al., 2021; S. Wang, H. Wang, and Perdikaris, 2021; L. Lu, X. Meng, et al., 2022; S. Wang, H. Wang, and Perdikaris, 2022), FNO (Li, Kovachki, Azizzadenesheli, B. Liu, et al., 2020), and PCA-Net (J. Hesthaven and Ubbiali, 2018; Bhattacharya et al., 2021) where the main novelty appears to be the use of novel, flexible, and expressive NN architectures that allow the algorithm to learn and adapt the bases that are selected for the input and outputs of the solution map as well as possible nonlinear dependencies between the basis coefficients. Although not part of our comparisons, we note that (Fan, Bohorquez, and Ying, 2019; Fan, Feliu-Faba, et al., 2019; Fan, Lin, et al., 2019) obtained competitive accuracy by using deep neural networks with architectures inspired by conventional fast solvers. Classical numerical approximation methods Operator learning has been the subject of intense research in the computational mathematics literature in the context of stochastic Galerkin methods (Ghanem and Spanos, 2003; Xiu and J. Shen, 2009), polynomial chaos (Xiu, 2010; Xiu and George Em Karniadakis, 2002), reduced basis methods (Noor and J. M. Peters, 1980; Maday, Patera, and Turinici, 2002) and numerical homogenization (Houman Owhadi and L. Zhang, 2007; Houman Owhadi, 2015a; Houman Owhadi and Clint Scovel, 2019a; Altmann, Henning, and Peterseim, 2021). In the setting of stochastic and parametric PDEs, the the goal is often to approximate the solution of a PDE as a function of a random or 182 uncertain parameter. The well-established approach to such problems is to pick or construct appropriate bases for the input parameter and the solution of the PDE and then construct a parametric, high-dimensional map that transforms the input basis coefficients to the output coefficients. Well-established methods such as polynomial chaos, stochastic finite element methods, and reduced basis methods (Ghanem and Spanos, 2003; Xiu, 2010; Cohen and DeVore, 2015; J. S. Hesthaven, Rozza, Stamm, et al., 2016; Lucia, Beran, and Silva, 2004) fall within this category. A vast amount of literature in applied mathematics exists on this subject, and the theoretical analysis of these methods is extensive; see for example (Beck et al., 2012; Chkifa, Cohen, DeVore, et al., 2012; Chkifa, Cohen, and Schwab, 2014; Nobile, Raúl Tempone, and Webster, 2008; Nobile, Raul Tempone, and Webster, 2008; Gunzburger, Webster, and G. Zhang, 2014) and references therein. Operator compression For solving PDEs, the objectives of operator learning are also similar to those of operator compression (Feischl and Peterseim, 2020; Kröpfl, Maier, and Peterseim, 2022) as formulated in numerical homogenization (Houman Owhadi and Clint Scovel, 2019a; Altmann, Henning, and Peterseim, 2021) and reduced order modeling (ROM) (Amsallem and Farhat, 2008; Lucia, Beran, and Silva, 2004), i.e., the approximation of the solution operator from pairs of solutions and source/forcing terms. While both ROM and numerical homogenization seek operator compression through the identification of reduced basis functions that are as accurate as possible (this translates into low-rank approximations with SVD and its variants (Boullé and Townsend, 2022)), numerical homogenization also requires those functions to be as localized as possible (Målqvist and Peterseim, 2014) and in turn leverages both low rank and sparse approximations. These localized reduced basis functions are known as Wannier functions in the physics literature (Marzari et al., 2012), and can be interpreted as linear combinations of eigenfunctions that are localized in both frequency space and the physical domain, akin to wavelets. The hierarchical generalization of numerical homogenization (Houman Owhadi, 2017) (gamblets) has led to the current state-of-the-art for operator compression of linear elliptic (Schaäfer, Katzfuss, and Houman Owhadi, 2021; Florian Schäfer, Timothy John Sullivan, and Houman Owhadi, 2021a) and parabolic/hyperbolic PDEs (Houman Owhadi and L. Zhang, 2017). In particular, for arbitrary (and possibly unknown) elliptic PDEs (Florian Schäfer and Houman Owhadi, 2021) shows that the solution operator (i.e., the Green’s function) can be approximated in near-linear complexity to accuracy 𝜖 from only O (log𝑑+1 ( 1𝜖 )) solutions of the PDE. 183 GP emulators In the case where the range of the operator of interest is finite dimensional, then operator learning coincides with surrogate modeling techniques that were developed in the UQ literature, such as GP surrogate modeling/emulation (Kennedy and O’Hagan, 2001; Bastos and O’hagan, 2009). When the kernels of the underlying GPs are also learned from data (Houman Owhadi and Yoo, 2019a; Yifan Chen, Houman Owhadi, and A. Stuart, 2021b), GP surrogate modeling has been shown to offer a simple, low-cost, and accurate solution to learning dynamical systems (Hamzi and Houman Owhadi, 2021), geophysical forecasting (Hamzi, Maulik, and Houman Owhadi, 2021a), and radiative transfer emulation (Susiluoto et al., 2021), and the inference of the structure of convective storms from passive microwave observations (Prasanth et al., 2021). Indeed, our proposed kernel framework for operator learning can be interpreted as an extension of these well-established GP surrogates to the setting where the range of the operator is a function space. Outline of the article The remainder of the article is organized as follows: we present our operator learning framework in Section 5.2 for the generalized setting where 𝜙, 𝜑 can be any collection of bounded and linear operators along with an interpretation of our method from the GP perspective. Our convergence analysis and quantitative error bounds are presented in Section 5.3 where we present the full version of Theorem 5.3.4. Our numerical experiments, implementation details, and benchmarks against FNO and DeepONet are collected in Section 5.4. We discuss future directions and open problems in Section 5.5. The appendix collects a review of operator valued kernels and GPs along with other auxiliary details. 5.2 The RKHS/GP framework for operator learning We now present our general kernel framework for operator learning, i.e., the proposed solution to Problem 2. We emphasize that here we do not require the spaces U and V to be spaces of continuous functions and in particular, we do not require the maps 𝜙 and 𝜑 to be obtained from pointwise measurements. To describe this, we will introduce the dual spaces of U and V to define optimal recovery with respect to kernel operators rather than just kernel functions. Write U ∗ and V ∗ for the duals of U and V, and write [·, ·] for the pertinent duality pairings. Assume that U is endowed with a quadratic norm ∥ · ∥ 𝑄 , i.e., there exists a linear bijection 𝑄 : U ∗ → U that is symmetric ([𝜙𝑎 , 𝑄𝜙 𝑏 ] = [𝜙 𝑏 , 𝑄𝜙𝑎 ]), positive 2 = [𝑄 −1 𝑢, 𝑢], ∀𝑢 ∈ U . ([𝜙𝑎 , 𝑄𝜙𝑎 ] > 0 for 𝜙𝑎 ≠ 0), and such that ∥𝑢∥ 𝑄 184 As in (Houman Owhadi and Clint Scovel, 2019a, Ch. 11), although U and U ∗ ∗ (with inner products are also Hilbert spaces under ∥ · ∥ 𝑄 and its dual norm ∥ · ∥ 𝑄 ∗ 𝑢, 𝑣 𝑄 = [𝑄 −1 𝑢, 𝑣] and 𝜙𝑎 , 𝜙 𝑏 𝑄 = [𝜙𝑎 , 𝑄𝜙 𝑏 ]), we will keep using the Banach space terminology to emphasize the fact that our dual pairings will not be based on the inner product through the Riesz representation theorem, but on a different realization of the dual space, as this setting is more practical. If U is a space of continuous functions on a subset Ω ⊂ R𝑑Ω then U ∗ contains delta Dirac functions and, to simplify notations, we also write 𝑄(𝑥, 𝑦) := [δ𝑥 , 𝑄δ𝑦 ] for 𝑥, 𝑦 ∈ R𝑑Ω to denote the kernel induced by the operator 𝑄. Note that in that case, U is a RKHS with norm ∥ · ∥ 𝑄 induced by the kernel 𝑄. Since 𝜙 is bounded and linear, its entries 𝜙𝑖 (write 𝜙 := (𝜙1 , . . . , 𝜙𝑛 )) must be elements of U ∗ . We assume those elements to be linearly independent. Write 𝜓 : R𝑛 → U for the linear operator defined by 𝜓(𝑌 ) := (𝑄𝜙) 𝑄(𝜙, 𝜙) −1𝑌 for 𝑌 ∈ R𝑛 , (5.14) where we write 𝑄(𝜙, 𝜙) for the 𝑛 × 𝑛 symmetric positive definite (SPD) matrix with entries 𝑄(𝜙𝑖 , 𝜙 𝑗 ) := [𝜙𝑖 , 𝑄𝜙 𝑗 ] 4 and 𝑄𝜙 for (𝑄𝜙1 , . . . , 𝑄𝜙𝑛 ) ∈ U 𝑛 . As described in (Houman Owhadi and Clint Scovel, 2019a, Chap. 11), for 𝑢 ∈ U, given 𝜙(𝑢) = 𝑌 , 𝜓(𝑌 ) is the minmax optimal recovery of 𝑢 when using the relative error in ∥·∥ 𝑄 -norm as a loss. Similarly, assume that V is endowed with a quadratic norm ∥ · ∥ 𝐾 , defined by the symmetric positive linear bijection 𝐾 : V ∗ → V. Write 𝜑 := (𝜑1 , . . . , 𝜑𝑚 ) and assume the entries of 𝜑 to be linearly independent elements of V ∗ . Using the same notations as in (5.14) write 𝜒 : R𝑚 → V for the linear operator defined by 𝜒(𝑍) := (𝐾 𝜑) 𝐾 (𝜑, 𝜑) −1 𝑍 for 𝑍 ∈ R𝑚 . (5.15) Then, as above, for 𝑣 ∈ V, given 𝜑(𝑣) = 𝑍, 𝜒(𝑍) is the minmax optimal recovery of 𝑣 when using the relative error in ∥ · ∥ 𝐾 -norm as a loss. Write L (R𝑚 ) for the space of bounded linear operators mapping R𝑚 to itself , i.e., 𝑚 × 𝑚 matrices. Let Γ : R𝑛 × R𝑛 → L (R𝑚 ) be a matrix-valued kernel (Alvarez, Rosasco, Lawrence, et al., 2012) defining an RKHS HΓ of continuous functions 𝑓 : R𝑛 → R𝑚 equipped with an RKHS norm ∥ · ∥ Γ . For 𝑖 ∈ {1, . . . , 𝑁 }, write 𝑈𝑖 := 𝜙(𝑢𝑖 ) and 𝑉𝑖 := 𝜑(𝑣 𝑖 ). Write U and V for the block-vectors with entries 𝑈𝑖 and 4 For linear measurements involving derivatives the computation of these kernel matrices requires the computation of derivatives of the kernels; see (Yifan Chen, Hosseini, et al., 2021a) for practical examples and considerations. 185 𝑉𝑖 . Write Γ(U, U) for the 𝑁 × 𝑁 block-matrix with entries Γ(𝑈𝑖 , 𝑈 𝑗 ) and assume Γ(𝑈, 𝑈) to be invertible (which is satisfied if Γ is non-degenerate and 𝑈𝑖 ≠ 𝑈 𝑗 for 𝑖 ≠ 𝑗). Let 𝑓 † be an element of HΓ and write 𝑓 † (U) for the block vector with entries 𝑓 † (𝑈𝑖 ). Then given 𝑓 † (U) = V it follows that 𝑓¯(𝑈) := Γ(𝑈, U)Γ(U, U) −1 V , (5.16) is the minimax optimal recovery of 𝑓 † , where Γ(·, U) is the block-vector with entries Γ(·, 𝑈𝑖 ). To this end, we propose to approximate the ground truth operator G † with Ḡ := 𝜒 ◦ 𝑓¯ ◦ 𝜙 , (5.17) also recall Figure 5.1. Combining (5.15) and (5.16) we further infer that Ḡ admits the following explicit representer formula Ḡ(𝑢) = (𝐾 𝜑) 𝐾 (𝜑, 𝜑) −1 Γ(𝜙(𝑢), U)Γ(U, U) −1 V. (5.18) In the remainder of this section we will provide more details and observations regarding our approximate operator Ḡ that is useful later in Section 5.3 and of independent interest. The kernel and RKHS associated with Ḡ The explicit formula (5.18) suggests that the operator Ḡ is an element of an RKHS defined by an operator-valued kernel, which we now characterize. For 𝑢 1 , 𝑢 2 ∈ U and 𝑣 ∈ V write 𝐺 (𝑢 1 , 𝑢 2 )𝑣 := (𝐾 𝜑) (𝐾 (𝜑, 𝜑)) −1 Γ(𝜙(𝑢 1 ), 𝜙(𝑢 2 ))(𝐾 (𝜑, 𝜑)) −1 𝜑(𝑣). (5.19) It turns out that 𝐺 : U × U → L (V) is a well-defined operator-valued kernel whose RKHS contains operators of the form Ḡ. Proposition 5.2.1. The kernel 𝐺 in (5.19) is an operator-valued kernel. Write H𝐺 for its RKHS and ∥ · ∥ 𝐺 for the associated norm. Then it holds that G ∈ H𝐺 if and only if G = 𝜒 ◦ 𝑓 ◦ 𝜙 for 𝑓 = 𝜑 ◦ G ◦ 𝜓 ∈ HΓ and ∥G∥ 𝐺 = ∥ 𝑓 ∥ Γ . Proof. Since 𝐺 is Hermitian and positive, we deduce that 𝐺 is an operator-valued kernel. Indeed for e 𝑢1, . . . , e 𝑢 𝑚 ∈ U and e 𝑣1, . . . , e 𝑣 𝑚 ∈ V, using e 𝑣 𝑖 , 𝐾 𝜑 𝑠 𝐾 = 𝜑 𝑠 (e 𝑣𝑖 ) and the fact that Γ is a matrix-valued kernel we have e 𝑣 𝑖 , 𝐺 (e 𝑢𝑖 , e 𝑢 𝑗 )e 𝑣 𝑗 𝐾 = 𝜑(e 𝑣 𝑖 )𝑇 (𝐾 𝜑) (𝐾 (𝜑, 𝜑)) −1 Γ(𝜙(e 𝑢𝑖 ), 𝜙(e 𝑢 𝑗 ))(𝐾 (𝜑, 𝜑)) −1 𝜑(e 𝑣𝑗) = 𝐺 (e 𝑢𝑗, e 𝑢𝑖 )e 𝑣𝑖 , e 𝑣𝑗 𝐾 , (5.20) 186 where we used e 𝑣 𝑖 , 𝐾 𝜑 𝑠 𝐾 = 𝜑 𝑠 (e 𝑣 𝑖 ) and the fact that Γ is a matrix-valued kernel. Í Furthermore, summing (5.20), we deduce that 𝑖,𝑚𝑗=1 e 𝑣 𝑖 , 𝐺 (e 𝑢𝑖 , e 𝑢 𝑗 )e 𝑣 𝑗 𝐾 ≥ 0. From (5.19) we infer 𝑚 ∑︁ 𝐺 (𝑢, e 𝑢 𝑗 )e 𝑣 𝑗 = 𝜒 ◦ 𝑓 ◦ 𝜙(𝑢) (5.21) 𝑗=1 with the function 𝑓 (𝑈) = 𝑚 ∑︁ Γ(𝑈, 𝜙(e 𝑢 𝑗 ))(𝐾 (𝜑, 𝜑)) −1 𝜑( 𝑣e𝑗 ) . (5.22) 𝑗=1 Furthermore using the reproducing property of 𝐺 and (5.20) we have 2 Í𝑚 𝑢 𝑗 )e 𝑣𝑗 𝑗=1 𝐺 (𝑢, e ∥ 𝑓 ∥ 2Γ . Therefore the closure of the space of operators of the form (5.21) with respect to the RKHS norm induced by 𝐺 is the space of functions of the form 𝜒 ◦ 𝑓 ◦ 𝜙 where 𝑓 lives in the closure of functions of the form (5.22) with respect to the RKHS norm induced by Γ. We deduce that H𝐺 = { 𝜒 ◦ 𝑓 ◦ 𝜙 | 𝑓 ∈ HΓ }. The uniqueness of 𝑓 in the representation G = 𝜒 ◦ 𝑓 ◦ 𝜙 for 𝑓 ∈ H𝐺 follows from 𝑓 = 𝜑 ◦ G ◦ 𝜓 following the identities 𝜑 ◦ 𝜒 = 𝐼 𝑑 and 𝜙 ◦ 𝜓 = 𝐼 𝑑 . □ Using the above result we can further characterize Ḡ and 𝑓¯ via optimal recovery problems in H𝐺 and HΓ respectively. In what follows we will write u for the 𝑁 vector whose entries are the 𝑢𝑖 , and G(u) for the 𝑁 vector whose entries are G † (𝑢𝑖 ). Proposition 5.2.2. The operator Ḡ is the minimizer of    Minimize    Over  2 ∥G∥ 𝐺 G ∈ H𝐺 such that 𝜑 ◦ G(u) = 𝜑 ◦ G † (u) (5.23) while the map 𝑓¯ is the minimizer of    Minimize    Over  ∥ 𝑓 ∥ 2Γ 𝑓 ∈ HΓ such that 𝑓 ◦ 𝜙(u) = 𝜑 ◦ G † (u) . (5.24) Proof. By Proposition 5.2.1 𝐺¯ is completely identified by 𝑓¯ and ∥ 𝐺¯ ∥ 𝐺 = ∥ 𝑓¯∥ Γ . Then solving (5.23) is equivalent to solving (5.24). The statement regarding 𝑓¯ follows directly from representer formulae for optimal recovery with matrix-valued kernels. □ = 𝐺 187 Regularizing 𝐺¯ by operator regression As is often the case with optimal recovery/kernel regression the estimator for 𝑓¯ in (5.16) is susceptible to numerical error due to ill-conditioning of the kernel matrix Γ(U, U). To overcome this issue we regularize our estimator by adding a small diagonal perturbation to this matrix. More precisely, let 𝛾 > 0 and write 𝐼 for the identity matrix. We then define the regularized map
−1 𝑓¯𝛾 (𝑈) := Γ(𝑈, U) Γ(U, U) + 𝛾𝐼 V . (5.25) This regularized map gives rise to the regularized approximate operator Ḡ𝛾 := 𝜒 ◦ 𝑓¯𝛾 ◦ 𝜙 , which admits the following representer formula: Ḡ𝛾 (𝑢) = (𝐾 𝜑) 𝐾 (𝜑, 𝜑) −1 Γ(𝜙(𝑢), U) Γ(U, U) + 𝛾𝐼
−1 V. (5.26) We can further characterize this operator as the solution to an operator regression problem. Proposition 5.2.3. Ḡ𝛾 is the solution to 2 Minimize G∈H𝐺 ∥G∥ 𝐺 + 𝛾 −1 |𝜑 ◦ G(u) − 𝜑 ◦ G † (u)| 2 . (5.27) Proof. By Proposition 5.2.1, G = 𝜒 ◦ 𝑓 ◦ 𝜙 solves (5.27) if and if 𝑓 solves Minimize 𝑓 ∈HΓ ∥ 𝑓 ∥ 2Γ + 𝛾 −1 | 𝑓 (U) − V| 2 . (5.28) It then follows by standard representer theorems for matrix-valued kernel regression (see Section 5.6) that 𝑓¯𝛾 is the minimizer of (5.28). □ Interpretation as conditioned operator valued GPs Our kernel approach to operator learning has a natural GP regression interpretation that is compatible with Bayesian inference and UQ pipelines. We present some facts and observations in this direction. Write 𝜉 ∼ N (0, 𝐺) for the centered operator-valued GP with covariance kernel 𝐺 5 and 𝜁 ∼ N (0, Γ) for a centered vector valued GP with covariance kernel Γ. Then it is straightforward to show that the law of 𝜉 is equivalent to that of 𝜒 ◦ 𝜁 ◦ 𝜙. Let 5 See Section 5.6 for a review of operator valued GPs. 188 𝑍 = (𝑍1 , . . . , 𝑍 𝑁 ) be a random block-vector, independent from 𝜉, with i.i.d. entries 𝑍 𝑗 ∼ N (0, 𝛾𝐼𝑚 ) for 𝑗 = 1, . . . , 𝑁; here 𝛾 ≥ 0 and 𝐼𝑚 is the 𝑚 × 𝑚 identity matrix. Then 𝜉 conditioned on 𝜑 ◦ 𝜉 (u) = 𝜑(v) + 𝑍 is an operator-valued GP with mean Ḡ𝛾 , as in (5.26), and conditional covariance kernel 𝐺 ⊥ (𝑢, 𝑢′)𝑣 = (𝐾 𝜑) (𝐾 (𝜑, 𝜑)) −1 Γ(𝜙(𝑢 1 ), 𝜙(𝑢 2 ))
Γ(𝜙(𝑢), 𝜙(𝑢′)) − Γ(𝜙(𝑢), U)(Γ(U, U) + 𝛾𝐼) −1 Γ(U, 𝜙(𝑢′)) (𝐾 (𝜑, 𝜑)) −1 𝜑(𝑣). Furthermore, the law of 𝜉 conditioned on 𝜑 ◦ 𝜉 (u) = 𝜑(v) + 𝑍 is equivalent to that of 𝜒 ◦ 𝜁 ⊥ ◦ 𝜙 where 𝜁 ⊥ ∼ N ( 𝑓¯𝛾 , Γ⊥ ) is the GP 𝜁 conditioned on 𝜁 (U) = V + 𝑍 ′, whose mean is 𝑓¯𝛾 as in (5.25) and conditional covariance kernel is Γ⊥ (𝑈, 𝑈 ′) = Γ(𝑈, 𝑈 ′) − Γ(𝑈, U)(Γ(U, U) + 𝛾𝐼) −1 Γ(U, 𝑈 ′). We also use the GP approach to derive an alternative regularization of (5.27) in Section 5.7. Measurement and mesh invariance As argued in (Li, Kovachki, Azizzadenesheli, B. Liu, et al., 2020), mesh invariance is a key property for operator learning methods, i.e, the learned operator should be generalizable at test time beyond the specific discretization that was used during training. In our framework, this translates to being able to predict the output of a test input function e 𝑢 given only a linear measurement 𝜙e(e 𝑢 ), where 𝜙e was unknown at training time. For example 𝜙e could be of the same form as 𝜙 (say (5.4)) but on a finer or coarser grid. Similarly, we may choose to output with an operator 𝜑 e which is a coarse/fine version of 𝜑. Our proposed framework can easily provide mesh invariance using additional optimal recovery and measurement operators at the input and outputs of the operator Ḡ as depicted in Figure 5.3. In fact, we can not only accommodate modification of the grid but completely different measurement operators at testing time. For example, while 𝜙, 𝜑 may be of the form (5.4) we may take 𝜙e and 𝜑 e to be integral operators such as Fourier or Radon transforms. Let us describe our approach to mesh invariance in detail. Given bounded and linear operators 𝜙e : U → Re𝑛 and 𝜑 e : V → R𝑚e we can approximate 𝜑 e(G † (e 𝑢 )) using the map 𝑓¯ obtained from (5.16) defined in terms of our training. To achieve mesh invariance we simply need a consistent approach to interpolate/extend the testing measurment operators to those used for training and we achieve this using e that is defined from 𝜙e analogously to 𝜓 in (5.14). the optimal recovery map 𝜓 189 G† Re𝑛 V U 𝜓 𝜙 e † 𝜒 𝜑 𝜑 e 𝜓 𝑓 𝑛 𝑚 R R e 𝜒 𝜙e† e ℎ := 𝜑 e◦ 𝜒 ◦ 𝑓 † ◦ 𝜙 ◦ 𝜓 R𝑚e Figure 5.3: Generalization of fig. 5.1 to the mesh invariant setting where the measurement functionals are different at test time. This setup gives rise to a natural approximation of G † in terms of the function ℎ† : Re𝑛 → R𝑚e depicted in Figure 5.3 which in turn can be approximated with e≡ 𝜑 e. This expression further gives rise to another ℎ¯ := 𝜑 e ◦ 𝜒 ◦ 𝑓¯ ◦ 𝜙 ◦ 𝜓 e ◦ Ḡ ◦ 𝜓 e= e approximation to G † given by the operator G 𝜒 ◦ ℎ¯ ◦ 𝜙e. Remark 7. Observe that the definition of ℎ¯ (and consequently Ḡ) is independent of the fact that 𝑓¯ is constructed using the kernel approach. Thus, the optimal recovery e can be used to retrofit any fixed-mesh operator learning algorithm, to maps 𝜒 and 𝜓 become mesh-invariant and able to use arbitrary linear measurements of the function e 𝑢 at test time. 5.3 Convergence and error analysis In this section, we present convergence guarantees and rigorous a priori error bounds for our proposed kernel method for operator learning and give a detailed statement and proof of Theorem 5.3.3. We assume that H𝑄 is a space of continuous functions from Ω ⊂ R𝑑Ω and that H𝐾 is a space of continuous functions from 𝐷 ⊂ R𝑑 𝐷 . Abusing notations we write 𝑄 : Ω × Ω → R𝑑Ω and 𝐾 : 𝐷 × 𝐷 → R𝑑 𝐷 for the kernels induced by the operators 𝑄 and 𝐾. Let 𝑋 = (𝑋1 , . . . , 𝑋𝑛 ) ⊂ Ω and 𝑌 = (𝑌1 , . . . , 𝑌𝑚 ) ⊂ 𝐷 be distinct collections of points and define their fill-distances ℎ 𝑋 := max min |𝑥 − 𝑥 ′ |, ′ 𝑥 ∈Ω 𝑥∈𝑋 ℎ𝑌 := max min |𝑦 − 𝑦′ |. ′ 𝑦 ∈𝐷 𝑦∈𝑌 This section focuses on operators 𝜙 and 𝜑 that are linear combinations of pointwise measurements in 𝑋 and 𝑌 . The presented results can be extended by using analogs of the sampling inequalities for other linear measurements; see (Houman Owhadi and Clint Scovel, 2019a, Theorem 4.11, Lemma 14.34) for a general framework that allows one to obtain such inequalities. 190 Let 𝐿 𝑄 and 𝐿 𝐾 be invertible 𝑛 × 𝑛 and 𝑚 × 𝑚 matrices. For 𝑢 ∈ H𝑄 write 𝑢(𝑋) for the 𝑛-vector with entries 𝑢(𝑋𝑖 ) and let 𝜙 : H𝑄 → R𝑛 be the bounded linear map defined by 𝜙(𝑢) = 𝐿 𝑄 𝑢(𝑋) . (5.29) For 𝑣 ∈ H𝐾 write 𝑣(𝑌 ) for the 𝑚-vector with entries 𝑣(𝑌 𝑗 ) and let 𝜑 : H𝐾 → R𝑚 be the bounded linear map defined by 𝜑(𝑣) = 𝐿 𝐾 𝑣(𝑌 ) . (5.30) Write ∥𝜙∥ := sup𝑢∈H𝑄 |𝜙(𝑢)|/∥𝑢∥ 𝑄 and ∥𝜓∥ := sup𝑈 ′ ∈R𝑛 ∥𝜓(𝑈 ′)∥ 𝑄 /|𝑈 ′ |, and similarly ∥𝜑∥ := sup𝑣∈H𝐾 |𝜑(𝑣)|/∥𝑣∥ 𝐾 and ∥ 𝜒∥ := sup𝑉 ′ ∈R𝑚 ∥ 𝜒(𝑉 ′)∥ 𝐾 /|𝑉 ′ |. We will also assume the following regularity conditions on the domains Ω, 𝐷, the kernels 𝑄, 𝐾, and the operator G † . Condition 5.3.1. Assume that the following conditions hold. (5.3.1.1) Ω and 𝐷 are compact sets with Lipschitz boundary. (5.3.1.2) There exist indices 𝑠 > 𝑑Ω /2 and 𝑡 > 𝑑 𝐷 /2 so that H𝑄 ⊂ 𝐻 𝑠 (Ω) and H𝐾 ⊂ 𝐻 𝑡 (𝐷), with inclusions indicating continuous embeddings. ′ (5.3.1.3) G † is a (possibly) nonlinear operator from 𝐻 𝑠 (Ω) to H𝐾 with 𝑠′ < 𝑠 that satisfies   † † ′ ∥G (𝑢) − G (𝑣)∥ 𝐾 ≤ 𝜔 ∥𝑢 − 𝑣∥ 𝐻 𝑠 (Ω) , (5.31) where 𝜔 : R → R+ is the modulus of continuity of G † . Note that conditions (5.3.1.2) and (5.3.1.3) imply ∥G † ∥ 𝐵 𝑅 (H𝑄 )→H𝐾 := sup ∥G † (𝑢)∥ 𝐾 < +∞ . 𝑢∈𝐵 𝑅 (H𝑄 ) Proposition 5.3.1. Suppose that Condition 5.3.1 holds. Let 0 < 𝑡 ′ < 𝑡. Then there exist constants ℎΩ , ℎ 𝐷 , 𝐶Ω , 𝐶𝐷 > 0 such that if ℎ 𝑋 < ℎΩ and ℎ𝑌 < ℎ 𝐷 , then  
′ ′ ∥G † (𝑢) − 𝜒◦ 𝑓 † ◦𝜙(𝑢)∥ 𝐻 𝑡 ′ (𝐷) ≤ 𝐶𝐷 𝜔 𝐶Ω ℎ 𝑠−𝑠 𝑅 +𝐶𝐷 ℎ𝑌𝑡−𝑡 ∥G † (0)∥ 𝐾 +𝜔(𝐶Ω 𝑅) , 𝑋 for any 𝑢 ∈ 𝐵 𝑅 (H𝑄 ), where 𝑓 † is defined as in (5.5). 191 Proof. By the definition of 𝑓 † and the triangle inequality we have ∥G † (𝑢) − 𝜒 ◦ 𝜑 ◦ G † ◦ 𝜓 † ◦ 𝜙(𝑢)∥ 𝐻 𝑡 ′ (Γ) ≤∥G † (𝑢) − G † ◦ 𝜓 ◦ 𝜙(𝑢)∥ 𝐻 𝑡 ′ (Γ) + ∥G † ◦ 𝜓 ◦ 𝜙(𝑢) − 𝜒 ◦ 𝜑 ◦ G † ◦ 𝜓 ◦ 𝜙(𝑢)∥ 𝐻 𝑡 ′ (Γ) =: 𝑇1 + 𝑇2 . Let us first bound 𝑇1 : By conditions (5.3.1.2) and (5.3.1.3), we have   𝑇1 ≤ 𝐶𝐷 ∥G † (𝑢) − G † ◦ 𝜓 ◦ 𝜙(𝑢)∥ 𝐾 ≤ 𝐶𝐷 𝜔 ∥𝑢 − 𝜓 ◦ 𝜙(𝑢)∥ 𝐻 𝑠′ (Ω) . At the same time, since (𝑢 − 𝜓 ◦ 𝜙(𝑢))(𝑋) = 0, condition (5.3.1.1) and the sampling inequality for interpolation in Sobolev spaces (Arcangéli, López de Silanes, and Torrens, 2007, Thm. 4.1), and condition (5.3.1.2) imply that there exists a constant ℎΩ > 0 so that if ℎ 𝑋 < ℎΩ then ′ ′ 𝑠−𝑠 ∥𝑢 − 𝜓 ◦ 𝜙(𝑢)∥ 𝐻 𝑠′ (Ω) ≤ 𝐶Ω′ ℎ 𝑠−𝑠 𝑋 ∥𝑢 − 𝜓 ◦ 𝜙(𝑢)∥ 𝐻 𝑠 (Ω) ≤ 𝐶Ω ℎ 𝑋 ∥𝑢 − 𝜓 ◦ 𝜙(𝑢)∥ 𝑄 , (5.32) where 𝐶Ω′ , 𝐶Ω > 0 are constants that are independent of 𝑢. Using ∥𝑢 − 𝜓 ◦ 𝜙(𝑢)∥ 𝑄 ≤ ∥𝑢∥ 𝑄 (Houman Owhadi and Clint Scovel, 2019a, Thm. 12.3) we deduce the desired bound   𝑠−𝑠′ 𝑇1 ≤ 𝐶𝐷 𝜔 𝐶Ω ℎΩ ∥𝑢∥ 𝑄 . (5.33) Let us now bound 𝑇2 : once again, by the continuous embedding of condition (5.3.1.2) and the sampling inequality for interpolation in Sobolev spaces, we have that there exists ℎ 𝐷 > 0 so that if ℎ𝑌 < ℎ 𝐷 , then for any 𝑣 ∈ 𝐻 𝑡 (𝐷) it holds that ′ ′ ′ ′ 𝑡−𝑡 ∥𝑣−𝜒◦𝜑(𝑣)∥ 𝐻 𝑡 ′ (𝐷) ≤ 𝐶𝐷 ℎ𝑌 ∥𝑣−𝜒◦𝜑(𝑣)∥ 𝐻 𝑡 (𝐷) ≤ 𝐶𝐷 ℎ𝑌𝑡−𝑡 ∥𝑣−𝜒◦𝜑(𝑣)∥ 𝐾 ≤ 𝐶𝐷 ℎ𝑌𝑡−𝑡 ∥𝑣∥ 𝐾 . Taking 𝑣 ≡ G † ◦ 𝜓 ◦ 𝜙(𝑢), we deduce that ′ 𝑇2 ≤ 𝐶𝐷 ℎ𝑌𝑡−𝑡 ∥G † ◦ 𝜓 ◦ 𝜙(𝑢)∥ 𝐾 ,
′ ≤ 𝐶𝐷 ℎ𝑌𝑡−𝑡 ∥G † (0)∥ 𝐾 + 𝜔(∥𝜓 ◦ 𝜙(𝑢)∥ 𝐻 𝑠′ (Ω) ) . Using ∥𝜓 ◦ 𝜙(𝑢)∥ 𝐻 𝑠′ (Ω) ≤ 𝐶Ω ∥𝜓 ◦ 𝜙(𝑢)∥ 𝑄 ≤ 𝐶Ω ∥𝑢∥ 𝑄 concludes the proof. □ While Proposition 5.3.1 gives an error bound for the distance between the maps G † and 𝜑 ◦ 𝑓 † ◦ 𝜙, we can never compute this map when 𝑁 < ∞ and so we have to approximate this map as well. Given the kernel Γ, our optimal recovery approximant for the map 𝑓 † is 𝑓¯ as in (5.16), which we recall is the minimizer of (5.24). 192 To proceed, we need to consider another intermediary problem that defines an approximation b 𝑓 to the map 𝑓 † : b 𝑓 :=    Minimize    Over  ∥ 𝑓 ∥ 2Γ 𝑓 ∈ HΓ such that 𝑓 ◦ 𝜙(u) = 𝑓 † ◦ 𝜙(u) . (5.34) We emphasize that the difference between the problems (5.24) and (5.34) is simply in the training data that is injected in the equality constraints, and this difference is quite subtle: In practical applications, observations may be taken from G † (𝑢𝑖 ), which is different from 𝑓 † ◦ 𝜙(𝑢𝑖 ) ≡ 𝜑 ◦ G † ◦ 𝜓 ◦ 𝜙(𝑢𝑖 ). To make our analysis simple, henceforth we assume the following condition on our input data. Condition 5.3.2. The input data points 𝑢𝑖 satisfy 𝑢𝑖 = 𝜓 ◦ 𝜙(𝑢𝑖 ) for 𝑖 = 1, . . . , 𝑁 . We observe that this condition implies G † (𝑢𝑖 ) = 𝑓 † ◦ 𝜙(𝑢𝑖 ) and 𝑓¯ = b 𝑓 . Removing † this assumption requires bounding some norm of the error 𝑓 − 𝑓¯, and we postpone that analysis to a sequel paper as this step can become very technical. The next step in our convergence analysis is then to control the error between the maps b 𝑓 and 𝑓 † which we will achieve using similar arguments as in the proof of Proposition 5.3.1. For our analysis, we take Γ to be a diagonal, matrix-valued kernel, of the form (5.8) which we recall for reference Γ(𝑈, 𝑈 ′) = 𝑆(𝑈, 𝑈 ′)𝐼, (5.35) where 𝐼 is the 𝑚 × 𝑚 identity matrix and 𝑆 : R𝑛 × R𝑛 → R is a real valued kernel. Proposition 5.3.2. Suppose that Condition 5.3.2 holds. Let Υ ⊂ R𝑛 be a compact set with Lipschitz boundary and consider 𝑈 = (𝑈1 , . . . , 𝑈𝑁 ) ⊂ Υ with fill distance ℎΥ := max min |𝑈𝑖 − 𝑈 ′ |. ′ 𝑈 ∈Υ 1≤𝑖≤𝑁 Let Γ be of the form (5.35), with 𝑆 restricted to the set Υ, and suppose H𝑆 ⊂ 𝐻 𝑟 (Υ) for 𝑟 > 𝑛/2 and that 𝑓 𝑗† ∈ H𝑆 for 𝑗 = 1, . . . , 𝑚. Then there exist constants ℎ′Υ , 𝐶Υ > 0 so that whenever ℎΥ < ℎ′Υ then for any 𝑟 ′ < 𝑟 it holds that ′ † ∥ 𝑓 𝑗† − b 𝑓 𝑗 ∥ 𝐻 𝑟 ′ (Υ) ≤ 𝐶Υ ℎ𝑟−𝑟 Υ ∥ 𝑓 𝑗 ∥𝑆. 193 Proof. The proof is a direct consequence of the fact that the components of b 𝑓 are given by the optimal recovery problems (5.34) and the sampling inequality for interpolation in Sobolev spaces (Arcangéli, López de Silanes, and Torrens, 2007, Thm. 4.1) following the same arguments used in the proof of Theorem 5.3.1. □ We can now combine the above results to obtain the following theorem. Theorem 5.3.3. Suppose that Conditions 5.3.1 and 5.3.2 hold in addition to those of
𝑁 ⊂ 𝐵 (H ), the set Υ = 𝜙 𝐵 (H ) , Proposition 5.3.2 with a set of inputs (𝑢𝑖 )𝑖=1 𝑅 𝑄 𝑅 𝑄 and index 𝑛/2 < 𝑟 ′ < 𝑟. Then for any 𝑢 ∈ 𝐵 𝑅 (H𝑄 ), it holds that  
′ † 𝑠−𝑠′ ¯ ′ ∥G (𝑢) − 𝜒 ◦ 𝑓 ◦ 𝜙(𝑢)∥ 𝐻 𝑡 (𝐷) ≤𝐶𝐷 𝜔 𝐶Ω ℎ 𝑋 𝑅 + 𝐶𝐷 ℎ𝑌𝑡−𝑡 ∥G † (0)∥ 𝐾 + 𝜔(𝐶Ω 𝑅) √ ′ + 𝑚𝐶𝐷 𝐶Υ ∥ 𝜒∥ℎΥ(𝑟−𝑟 ) max ∥ 𝑓 𝑗† ∥ 𝑆 . 1≤ 𝑗 ≤𝑚 (5.36) Proof. An application of the triangle inequality yields ∥G † (𝑢) − 𝜒 ◦ 𝑓¯ ◦ 𝜙(𝑢)∥ 𝐻 𝑡 ′ (𝐷) ≤∥G † (𝑢) − 𝜒 ◦ 𝑓 † ◦ 𝜙(𝑢)∥ 𝐻 𝑡 ′ (𝐷) + ∥ 𝜒 ◦ 𝑓 † ◦ 𝜙(𝑢) − 𝜒 ◦ b 𝑓 ◦ 𝜙(𝑢)∥ 𝐻 𝑡 ′ (𝐷) + ∥𝜒 ◦ b 𝑓 ◦ 𝜙(𝑢) − 𝜒 ◦ 𝑓¯ ◦ 𝜙(𝑢)∥ 𝐻 𝑡 ′ (𝐷) =: 𝐼1 + 𝐼2 + 𝐼3 . We can bound 𝐼1 immediately using Proposition 5.3.1. Furthermore, by Condition 5.3.2 we have that 𝐼3 = 0. So it remains for us to bound 𝐼2 : by the continuous ′ embedding of H𝐾 into 𝐻 𝑡 (𝐷) we can write 𝐼2 ≤ 𝐶𝐷 ∥ 𝜒 ◦ 𝑓 † ◦ 𝜙(𝑢) − 𝜒 ◦ b 𝑓 ◦ 𝜙(𝑢)∥ 𝐾 ≤ 𝐶𝐷 ∥ 𝜒∥| 𝑓 † ◦ 𝜙(𝑢) − b 𝑓 ◦ 𝜙(𝑢)| v u t∑︁ 𝑚 ∥ 𝑓 𝑗† − b 𝑓 𝑗 ∥ 2𝐻 𝑟 ′ (Υ) , ≤ 𝐶𝐷 ∥ 𝜒∥ 𝑗=1 where the last line follows from the Sobolev embedding theorem and the assumption that 𝑟 ′ > 𝑛/2. Then an application of Proposition 5.3.2 yields 𝐼2 ≤ √ ′ 𝑚𝐶𝐷 𝐶Υ ∥ 𝜒∥ℎΥ(𝑟−𝑟 ) max ∥ 𝑓 𝑗† ∥ 𝑆 . 1≤ 𝑗 ≤𝑚 □ Convergence theorem Our next step will be to consider the limits 𝑁, 𝑛, 𝑚 → ∞ and show the convergence of Ḡ to G † . To obtain this result we first need to make assumptions on the regularity of the true operator G † . 194 For 𝑘 ≥ 1 write 𝐷 𝑘 G † for the functional derivative of G † of order 𝑘. Recall 𝑘 H to H . that for 𝑢 ∈ H𝑄 , 𝐷 𝑘 G † (𝑢) is a multilinear operator mapping ⊗𝑖=1 𝑄 𝐾 𝑘 † 𝑘 For 𝑤 1 , . . . , 𝑤 𝑘 ∈ H𝑄 write [𝐷 G (𝑢), ⊗𝑖=1 𝑤 𝑖 ] for the (multilinear) action of 𝑘 𝑤 and write ∥𝐷 𝑘 G † (𝑢)∥ for the smallest constant such that for 𝐷 𝑘 G † (𝑢) on ⊗𝑖=1 𝑖 𝑤 1 , . . . , 𝑤 𝑘 ∈ H𝑄 , [𝐷 𝑘 † 𝑘 G (𝑢), ⊗𝑖=1 𝑤 𝑖 ] H𝐾 ≤ ∥𝐷 𝑘 G † (𝑢)∥ 𝑘 Ö ∥𝑤 𝑖 ∥ H𝑄 , (5.37) 𝑖=1 Similarly, for 𝑘 ≥ 1 write 𝐷 𝑘 𝑓 † for the derivation tensor of 𝑓 † of order 𝑘 (the gradient for 𝑘 = 1 and the Hessian for 𝑘 = 2, etc). Recall that for 𝑈 ∈ R𝑛 , 𝑘 R𝑛 to R𝑚 . For 𝑊 , . . . , 𝑊 ∈ R𝑛 𝐷 𝑘 𝑓 † (𝑈) is a multilinear operator mapping ⊗𝑖=1 1 𝑘 𝑘 † 𝑘 𝑘 † 𝑘 write [𝐷 𝑓 (𝑈), ⊗𝑖=1𝑊𝑖 ] for the (multilinear) action of 𝐷 𝑓 (𝑈) on ⊗𝑖=1𝑊𝑖 and write ∥𝐷 𝑘 𝑓 † (𝑈)∥ for the smallest constant such that for 𝑊1 , . . . , 𝑊 𝑘 ∈ R𝑛 , 𝑘 † [𝐷 𝑓 𝑘 (𝑈), ⊗𝑖=1 𝑊𝑖 ] 𝑘 † ≤ ∥𝐷 𝑓 (𝑈)∥ 𝑘 Ö |𝑊𝑖 |. (5.38) 𝑖=1 where | · | is the Euclidean norm. Lemma 5.3.3. It holds true that ∥𝐷 𝑘 𝑓 † (𝑈)∥ ≤ ∥𝜑∥∥𝜓∥ 𝑘 ∥𝐷 𝑘 G † ◦𝜓(𝑈)∥, ∀𝑈 ∈ R𝑛 . Proof. The chain rule and the linearity of 𝜑 and 𝜓 imply that 𝑘 𝑘 [𝐷 𝑘 𝑓 † (𝑈), ⊗𝑖=1 𝑊𝑖 ] = 𝜑[𝐷 𝑘 G † ◦ 𝜓(𝑈), ⊗𝑖=1 𝜓(𝑊𝑖 )] . We then conclude the proof by writing 𝑘 𝑊𝑖 ] ≤ ∥𝜑∥∥𝐷 𝑘 G † ◦ 𝜓(𝑈)∥ [𝐷 𝑘 𝑓 † (𝑈), ⊗𝑖=1 𝑘 Ö ∥𝜓(𝑊𝑖 )∥ H𝑄 𝑖=1 𝑘 𝑘 † ≤ ∥𝜑∥∥𝜓∥ ∥𝐷 G ◦ 𝜓(𝑈)∥ 𝑘 Ö |𝑊𝑖 | . 𝑖=1 □ Let us now consider an infinite and dense sequence of points 𝑋1 , 𝑋2 , 𝑋3 , . . . of Ω, ∞ {𝑋 } is the closure of Ω. Write 𝑋 𝑛 for the 𝑛-vector such that the closure of ∪𝑖=1 𝑖 formed by the first 𝑛 points, i.e., 𝑋 𝑛 := (𝑋1 , . . . , 𝑋𝑛 ) (5.39) 195 𝑛 be an arbitrary invertible 𝑛 × 𝑛 matrix. Further let 𝜙 𝑛 : H → R𝑛 be and let 𝐿 𝑄 𝑄 defined by 𝑛 𝜙𝑛 (𝑢) = 𝐿 𝑄 𝑢(𝑋 𝑛 ) . (5.40) Write 𝜓 𝑛 for the corresponding optimal recovery 𝜓-map. Similarly, we assume that we are given an infinite and dense sequence of points 𝑌1 , 𝑌2 , 𝑌3 , . . . of 𝐷, such that ∞ {𝑌 } is the closure of 𝐷. Write 𝑌 𝑚 for the 𝑚-vector formed by the the closure of ∪𝑖=1 𝑖 first 𝑚 points, i.e., 𝑌 𝑚 := (𝑌1 , . . . , 𝑌𝑚 ) . (5.41) 𝑚 𝑚 Let 𝐿 𝑚 𝐾 be an arbitrary invertible 𝑚 × 𝑚 matrix and let 𝜑 : H𝐾 → R be defined by 𝑚 𝜑𝑚 (𝑣) = 𝐿 𝑚 (5.42) 𝐾 𝑢(𝑌 ) . Write 𝜒 𝑚 for the corresponding optimal recovery 𝜒-map. We also assume that we are given a sequence of diagonal matrix-valued kernels Γ𝑚,𝑛 : R𝑛 × R𝑛 → L (R𝑚 ) with scalar-valued kernels 𝑆 𝑛 : R𝑛 × R𝑛 → R as diagonal entries. Write 𝑓¯𝑁𝑚,𝑛 for the corresponding minimizer of (5.24) (also identified by the formula (5.16)) for the above setup. Theorem 5.3.4. Let 𝑚, 𝑛 be the dimensionality of the input and output observations 𝑛 {𝑋 } is equal 𝜙 : U → R𝑛 and 𝜑 : V → R𝑚 . Suppose that the closure of lim𝑛↑∞ ∪𝑖=1 𝑖 𝑚 to the closure of Ω and that the closure of lim𝑚↑∞ ∪𝑖=1 {𝑌𝑖 } is equal to the closure of 𝐷. Suppose Condition 5.3.1 is satisfied and that [𝐷 𝑘 G † (𝑢)∥ < ∞ for all 𝑘 ≥ 1 , sup (5.43) 𝑢∈𝐵 𝑅 (H𝑄 ) for an arbitrary 𝑅 > 0. Assume that for any 𝑛 ≥ 1 and any compact set Υ of R𝑛 , the RKHS of 𝑆 𝑛 restricted to Υ (which we write H𝑆 𝑛 (Υ)) is contained in 𝐻 𝑟 (Υ) ′ for some 𝑟 > 𝑛/2 and contains 𝐻 𝑟 (Υ) for some 𝑟 ′ > 0 that may depend on 𝑛. Let 𝑁 be a sequence of inputs in 𝐵 (H ). Assume that there exists an integer 𝑛 (𝑢𝑖 )𝑖=1 𝑅 𝑄 0 𝑁 such that for 𝑛 ≥ 𝑛0 , the data points (𝑢𝑖 )𝑖=1 satisfy Condition 5.3.2, i.e., they satisfy 𝑢𝑖 = 𝜓 𝑛 ◦ 𝜙𝑛 (𝑢𝑖 ) for all 𝑖 ≥ 1. Further assume that the (𝜙𝑛 (𝑢𝑖 ))1≤𝑖≤𝑁 are space filling in the sense that for any 𝑛 ≥ 𝑛0 we have lim min 𝑢𝑖 (𝑋 𝑛 ) − 𝑢(𝑋 𝑛 )| = 0 . sup 𝑁→∞ 𝑢∈𝐵 𝑅 (H ) 1≤𝑖≤𝑁 (5.44) 𝑄 Then for any 𝑡 ′ ∈ (0, 𝑡), it holds that lim lim sup 𝑛,𝑚→∞ 𝑁→∞ 𝑢∈𝐵 (H ) 𝑅 𝑄 ∥G † (𝑢) − 𝜒 𝑚 ◦ 𝑓¯𝑁𝑚,𝑛 ◦ 𝜙𝑛 (𝑢)∥ 𝐻 𝑡 ′ (𝐷) = 0. (5.45) 196 Proof. Following (Houman Owhadi and Clint Scovel, 2019a, Chap. 12.1) define the projection 𝑃𝑛U = 𝜓 𝑛 ◦ 𝜙𝑛 onto the range of 𝜓 𝑛 . Since the points 𝑋𝑖 and 𝑌 𝑗 are dense in Ω and 𝐷 we have ℎ 𝑋 𝑛 ↓ 0 as 𝑛 → ∞ and ℎ𝑌 𝑚 ↓ 0 as 𝑚 → ∞. Given 𝑛, take
′ Υ = 𝜙𝑛 𝐵 𝑅 (H𝑄 ) . Then Lemma 5.3.3 and (5.43) imply that 𝑓¯𝑗𝑚,𝑛 ∈ 𝐻 𝑟 (Υ) for all 𝑟 ′ ≥ 0. Therefore 𝑓¯𝑗𝑚,𝑛 ∈ H𝑆 𝑛 (Υ). Now (5.44) implies that for any 𝑛, the fill distance,
in 𝜙𝑛 𝐵 𝑅 (H𝑄 ) , between the points (𝜙𝑛 (𝑢𝑖 ))1≤𝑖≤𝑁 goes to zero as 𝑁 → ∞. Since the conditions of Proposition 5.3.2 are satisfied, we conclude by taking the limit 𝑁 → ∞ in (5.36) before taking the limit 𝑚, 𝑛 → ∞. □ The effect of the 𝐿 𝑄 and 𝐿 𝐾 preconditioners We conclude this section and our discussion of convergence results by highlighting 𝑛 and 𝐿 𝑚 in (5.40) and (5.42). It is the importance of the choice of the matrices 𝐿 𝑄 𝐾 clear from the bounds (5.36) and (5.38) that our error estimates depend on the norms of the linear operators 𝜑𝑚 , 𝜓 𝑛 and 𝜒 𝑚 . To ensure that those norms do not blow up 𝑛 and 𝐿 𝑚 to be the Cholesky factors of as 𝑛, 𝑚 → ∞ we can select the matrices 𝐿 𝑄 𝐾 the precision matrices obtained from pointwise measurements of the kernels 𝑄 and 𝐾, i.e., 𝑛 𝑛 𝑇 𝐿𝑄 (𝐿 𝑄 ) = 𝑄(𝑋 𝑛 , 𝑋 𝑛 ) −1 and 𝑚 𝑇 𝑚 𝑚 −1 𝐿𝑚 𝐾 (𝐿 𝐾 ) = 𝐾 (𝑌 , 𝑌 ) . (5.46) We now obtain the following proposition. 𝑛 as in (5.46), then ∥𝜙 𝑛 ∥ = 1 and Proposition 5.3.4. If 𝜙𝑛 is as in (5.40) and 𝐿 𝑄 𝑚 𝑚 ∥𝜓 𝑛 ∥ = 1. If 𝜑𝑚 is as in (5.30) and 𝐿 𝑚 𝐾 as in (5.46), then ∥𝜑 ∥ = 1 and ∥ 𝜒 ∥ = 1. 2 . Since Proof. For 𝑢 ∈ H𝑄 , |𝜙𝑛 (𝑢)| 2 = 𝑢(𝑋 𝑛 )𝑇 𝑄(𝑋 𝑛 , 𝑋 𝑛 ) −1 𝑢(𝑋 𝑛 ) = ∥𝜓 𝑛 ◦ 𝜙𝑛 (𝑢)∥ 𝑄 𝜓 𝑛 ◦ 𝜙𝑛 is a projection (Houman Owhadi and Clint Scovel, 2019a, Chap. 12.1) we 𝑛 𝑈 ′ leads to ∥𝜓 𝑛 (𝑈 ′)∥ 2 = |𝑈 ′ | 2 deduce that ∥𝜙𝑛 ∥ = 1. Using 𝜓 𝑛 (𝑈 ′) = 𝑄(·, 𝑋 𝑛 )𝐿 𝑄 𝑄 and ∥𝜓 𝑛 ∥ = 1. The proof of ∥𝜑𝑛 ∥ = 1 and ∥ 𝜒 𝑛 ∥ = 1 is similar. □ We note that although useful for obtaining tighter approximation errors, this partic𝑛 and 𝐿 𝑚 is not required for convergence if one first ular choice for the matrices 𝐿 𝑄 𝐾 takes the limit 𝑁 → ∞ as in Theorem 5.3.4, which does not put any requirements 𝑛 and 𝐿 𝑚 beyond invertibility. on the matrices 𝐿 𝑄 𝐾 5.4 Numerics In this section, we present numerical experiments and benchmarks that compare a straightforward implementation of our kernel operator learning framework to stateof-the-art NN-based techniques. We discuss some implementation details of our 197 method in Section 5.4 followed by the setup of experiments and test problems in Section 5.4 and remark 8. A detailed discussion of our findings is presented in Remark 8. Implementation considerations Below we summarize some of the key details in the implementation of our kernel approach for operator learning for benchmark examples. Our code to reproduce the experiments can be found in a public repository6 . Choice of the kernel Γ Following our theoretical discussions in Sections 5.2 and 5.3, we primarily take Γ to be a diagonal kernel of the form (5.35). This implies that our estimation of 𝑓¯ can be split into independent problems for each of its components 𝑓¯𝑗 in the RKHS of the scalar kernel 𝑆. In our experiments, we investigate different choices of 𝑆 belonging to the families of the linear kernel, rational quadratic, and Matérn; see Section 5.8 for detailed expressions of these kernels. The rational quadratic kernel has two parameters: the lengthscale 𝑙 and the exponent 𝛼. We tuned these parameters using standard cross validation or log marginal likelihood maximization over the training data (see (Rasmussen and Williams, 2006, p.112) for a detailed description). The Matérn kernel is parameterized by two positive parameters: a smoothness parameter 𝜈 and the length scale 𝑙. The smoothness parameter 𝜈 controls the regularity of the  RKHS and we considered 𝜈 ∈ 12 , 23 , 52 , 27 , ∞ . In practice we found that 𝜈 = 52 almost always had the best performance. For a fixed choice of 𝜈 we tuned the length scale 𝑙 similarly to the rational quadratic kernel. We implemented the kernel regressions of the 𝑓¯𝑗 and parameter tuning algorithms in scikit-learn for low-dimensional examples and manually in JAX for high-dimensional examples. Preconditioning and dimensionality reduction Following (5.29) and (5.30) and the discussion in Section 5.3, we consider two preconditioning strategies for our pointwise measurements, i.e., choices of the matrices 𝐿 𝑄 and 𝐿 𝐾 : (1) we consider the Cholesky factors of the underlying covariance matrices as in (5.46), and (2) we use PCA projection matrices of the input and output functions computed from the training data. We truncated the PCA expansions to preserve (0.90, 0.95, 0.99) of the variance. The use of PCA in learning mappings 6 https://github.com/MatthieuDarcy/KernelsOperatorLearning/ 198 between infinite dimensional spaces was proposed in (Krischer et al., 1993) and recently revisited in (Bhattacharya et al., 2021; J. Hesthaven and Ubbiali, 2018). Experimental setup We compare the test performance of our method with different choices of the kernel 𝑆 of increasing complexity using the examples in (De Hoop et al., 2022) and (L. Lu, X. Meng, et al., 2022) and their reported test relative 𝐿 2 loss (see eq. (5.48) below). We use the data provided by these papers for the training set and the test set7 . Both articles provide performance comparisons between different variants of Neural Operators (most notably FNO and DeepONet) on a variety of PDE operator learning tasks, where the data is sampled independently from a distribution (𝐼𝑑, G † ) # 𝜇 supported on U × V, where 𝜇 is a specified (input) distribution on U. The example problems are outlined in detail in Remark 8; a summary of the specific PDEs, problem type, and distribution 𝜇 for each test is given in table 5.2. In some instances the train-test split of the data was not clear from the available online repositories in which case we re-sampled them from the assumed distribution 𝜇. The datasets from (L. Lu, X. Meng, et al., 2022) contain 1,000 training data-points per problem (which we will refer to as the “low-data regime”), whereas the datasets from (De Hoop et al., 2022) contain 20000 training data-points (which we will refer to as the “high-data” regime). We make this distinction because the complexity of kernel methods, unlike that of neural networks, may depend on the number of data-points. Following the suggestion of (De Hoop et al., 2022) we not only compare test errors and training complexity but also the complexity of operator learning at the inference/evaluation stage in Section 5.4. For the examples in (De Hoop et al., 2022), we investigate the accuracy-complexity trade-off of our method against the reported values of that article. 7 See https://github.com/Zhengyu-Huang/Operator-Learning and https://github. com/lu-group/deeponet-fno, respectively, for the data 199 Equation Burger’s Darcy problem Advection I Advection II Helmholtz Structural mechanics Navier Stokes Input Initial condition Coefficient Initial condition Initial condition Coefficient Initial force Forcing term Output Solution at time 𝑇 Solution Solution at time 𝑇 Solution at time 𝑇 Solution Stress field Solution at time 𝑇 Input Distribution 𝜇 Gaussian field (GF) Binary function of GF Random square waves Binary function of GF Function of Gaussian field Gaussian field Gaussian field Table 5.2: Summary of datasets used for benchmarking. The first three examples were considered in (L. Lu, X. Meng, et al., 2022), and the last four were taken from (De Hoop et al., 2022). Measures of accuracy As our first performance metric we measured the accuracy of models by a relative loss on the output space V: # " ||G † (𝑢) − G(𝑢)|| V , (5.47) R (G) = E𝑢∼𝜇 ||G † (𝑢)|| V where G † is true operator and G is a candidate operator. Following previous works, ∫ 1 we often took ||𝑢|| V = ||𝑢|| 𝐿 2 := ( 𝑢(𝑥) 2 𝑑𝑥) 2 , which in turn is discretized using the trapezoidal rule. In practice, we do not have the access to the underlying probability measure 𝜇 and we compute the empirical loss on a withheld test set: " # 𝑁 1 ∑︁ ||G † (𝑢 𝑛 ) − G(𝑢 𝑛 )|| V , 𝑢𝑖 ∼ 𝜇. (5.48) R 𝑁 (G) = 𝑁 𝑛=1 ||G † (𝑢 𝑛 )|| V Measures of complexity For our second performance metric we considered the complexity of operator learning algorithms at the inference stage (i.e., evaluating the learned operator). Complexity at inference time is the main metric used in (De Hoop et al., 2022) to compare numerical methods for operator learning. The motivation is that training of the methods can be performed in an offline fashion, and therefore the cost per test example dominates in the limit of many test queries. In particular, they compare the online evaluation costs of the neural networks by computing the requisite floating point operations (FLOPs) per test example. We adopt this metric as well for the methods not based on neural networks that we develop in this work, and we compare, when available, the cost-accuracy tradeoff with the numbers reported in (De Hoop et al., 200 2022). We computed the FLOPs with the same assumptions as in the original work: a matrix-vector product where the input vector is in R𝑛 and the output vector is in R𝑚 amounts to 𝑚(2 − 1) flops, and non-linear functions with 𝑛-dimensional inputs (activation functions for neural networks, kernel computations for kernel methods) are assumed to have cost O (𝑛). Remark 8 (Training complexity). While the inference complexity of a model eventually dominates the cost of training during applications, the training cost cannot be ignored since the allocated computational resources during this stage may still be limited and the resulting errors will have a profound impact on the quality and performance of the learned operators. Therefore numerical methods in which the offline data assimilation step is cheaper, faster, and more robust will always be preferred. Computing the exact number of FLOPs at training time is difficult to estimate for NN methods, as it depends on the optimization algorithms used, the hyperparameters and the optimization over such hyperparameters, among many other factors. Therefore in this work we limit the training complexity evaluation to the qualitative observation that kernel methods provided in this work are significantly simpler at training time, as they have no NN weights, they do not require the use of stochastic gradient descent, and have few or no hyperparameters which can be tuned using standard methods such as grid search or gradient descent in a low-dimensional space. Test problems and qualitative results Below we outline the setup of each of our benchmark problems. In all cases, U and V are spaces of real-valued functions with input domains Ω, 𝐷 ⊂ R 𝑘 for 𝑘 = 1 or 2. Whenever Ω = 𝐷, we simply write D for both. Burger’s equation Consider the one-dimensional Burger’s equation: 𝜕𝑤 𝜕2𝑤 𝜕𝑤 +𝑤 = 𝜈 2 , (𝑥, 𝑡) ∈ (0, 1) × (0, 1], 𝜕𝑡 𝜕𝑥 𝜕𝑥 𝑤(𝑥, 0) = 𝑢(𝑥), 𝑥 ∈ (0, 1) (5.49) with D = (0, 1), and periodic boundary conditions. The viscosity parameter 𝜈 is set to 0.1. We learn the operator mapping the initial condition 𝑢 to 𝑣 = 𝑤(·, 1), the solution at time 𝑡 = 1, i.e., G † : 𝑤(·, 0) ↦→ 𝑤(·, 1). 201 −0.5 0.0 −0.5 0 x 1 0 x 1 (a) Training input (b) Training output 0 −1 0 x 1 (c) True test Predicted - True 0.0 1 w(x, 1) 1 0.5 w(x, 1) 0.5 w(x, 1) u(x) = w(x, 0) The training data is generated by sampling the initial condition 𝑢 from a GP with a Riesz kernel, denoted by 𝜇 = GP (0, 625(−Δ + 25𝐼) −2 )). As in (L. Lu, X. Meng, et al., 2022), we used a spatial resolution with 128 grid points to represent the input and output functions, and used 1,000 instances for training and 200 instances for testing. Figure 5.4 shows an example of training input and output pairs as well as a test example along with its pointwise error. 0 −1 0 x 1 0.10 0.05 0.00 −0.05 −0.10 0 x 1 (d) Predicted test (e) Pointwise error Figure 5.4: Example of training data and test prediction and pointwise errors for the Burger’s equation (5.49). Darcy flow Consider the two-dimensional Darcy flow problem (5.3). Recall that in this example, we are interested in learning the mapping from the permeability field 𝑢 to the solution 𝑣 and the source term 𝑤 is assumed to be fixed, and hence D ≡ Ω = (0, 1) 2 and G † : 𝑢 ↦→ 𝑣. The coefficient 𝑢 is sampled by setting 𝑢 ∼ log ◦ℎ♯ 𝜇 where 𝜇 = GP (0, (−Δ + 9𝐼) −2 ) is a GP and ℎ is binary function mapping positive inputs to 12 and negative inputs to 3. The resulting permeability/diffusion coefficient 𝑒 𝑢 is therefore piecewise constant. As in (L. Lu, X. Meng, et al., 2022), we use a discretized grid of resolution 29 × 29, with the data generated by the MATLAB PDE Toolbox. We use 1,000 points for training and 200 points for testing. Figure 5.2 shows an example of training input and output of the map G † , and an example of predictions along with pointwise error at the test stage. Advection equations (I and II) Consider the one-dimensional advection equation: 𝜕𝑤 𝜕𝑤 + =0 𝑥 ∈ (0, 1), 𝑡 ∈ (0, 1] 𝜕𝑡 𝜕𝑥 𝑤(𝑥, 0) = 𝑢(𝑥) 𝑥 ∈ (0, 1) (5.50) with D = (0, 1) and periodic boundary conditions. Similar to the example for Burgers’ equation, we learn the mapping from the initial condition 𝑢 to 𝑣 = 𝑤(·, 0.5), the solution at 𝑡 = 0.5, i.e., G † : 𝑤(·, 0) ↦→ 𝑤(·, 0.5). 202 This problem was considered in (L. Lu, X. Meng, et al., 2022; De Hoop et al., 2022) with different distributions 𝜇 for the initial condition. We will show in the following section how these different distributions lead to different performances. In (L. Lu, X. Meng, et al., 2022), henceforth referred to as Advection I, the initial condition is a square wave centered at 𝑥 = 𝑐 of width 𝑏 and height ℎ: 𝑢(𝑥) = ℎ1{𝑐− 𝑏 ,𝑐+ 𝑏 } , 2 (5.51) 2 where the parameters (𝑐, 𝑏, ℎ) ∼ U ([0.3, 0.7] × [0.3, 06] × [1, 2]). In (De Hoop et al., 2022), henceforth referred to as Advection II, the initial condition is 𝑢 = −1 + 21{e 𝑢 0 ≥ 0}, (5.52) where e 𝑢 0 ∼ GP (0, (−Δ + 32 𝐼) −2 ). For Advection I, the spatial grid was of resolution 40, and we used 1,000 instances for training and 200 instances for testing. For Advection II, the resolution was of 200 and we used 20,000 training and test instances, following (De Hoop et al., 2022). 0 0.0 0.5 x 1.0 (a) Training input 1 0 0.0 0.5 x (b) Training output 1.0 2 1 0 0.0 0.5 x (c) True test 1.0 Predicted - True 1 2 w(x, 0.5) 2 w(x, 0.5) 2 w(x, 0.5) u(x) = w(x, 0) Figures 5.5 and 5.6 show an example of training input and output for Advection the I and II problems, respectively. Observe that the functional samples from the distribution in Advection I will have exactly two discontinuities almost surely, but the samples for Advection II can have many more jumps. We observe that prediction is challenging around discontinuities, and hence Advection II is a significantly harder problem (across all benchmarked methods) than Advection I. Figures 5.5 and 5.6 also show an instance of a test sample, along with a prediction and the pointwise errors. 1 0 0.0 0.5 x 1.0 (d) Predicted test ×10−15 0.0 −2.5 −5.0 0.0 0.5 x 1.0 (e) Pointwise error Figure 5.5: Example of training data and test prediction and pointwise errors for the Advection problem (5.50)-I. −1 0.0 0.5 −1 0.0 1.0 x 0 (a) Training input 0.5 0 −1 0.0 1.0 x 1 (b) Training output 0.5 1.0 x (c) True test 0 −1 0.0 0.5 1.0 x Predicted - True 0 1 w(x, 0.5) 1 w(x, 0.5) 1 w(x, 0.5) u(x) = w(x, 0) 203 (d) Predicted test 1 0 −1 0.0 0.5 1.0 x (e) Pointwise error Figure 5.6: Example of training data and test prediction and pointwise errors for the Advection problem (5.50)-II. Helmholtz’s equation For a given frequency 𝜔 and wavespeed field 𝑢 : D → R, with D = (0, 1) 2 , the excitation field 𝑣 : D → R solves   𝜔2 𝑣 = 0, 𝑥 ∈ (0, 1) 2 −Δ − 2 𝑢 (𝑥) 𝜕𝑣 𝜕𝑣 = 0, 𝑥 ∈ {0, 1} × [0, 1] ∪ [0, 1] × {0} and = 𝑣 𝑁 , 𝑥 ∈ [0, 1] × {1} 𝜕𝑛 𝜕𝑛 (5.53) 3 In the results that follow, we take 𝜔 = 10 , 𝑣 𝑁 = 1{0.35≤𝑥≤0.65} , and we aim to learn the map G : 𝑢 ↦→ 𝑣, i.e., the mapping from the wavespeed field to the excitation field. The distribution 𝜇 is specified as the law of 𝑢(𝑥) = 20 + tanh(e 𝑢 (𝑥)), where e 𝑢 is drawn from the GP, GP (0, (−Δ + 32 𝐼) −2 ). The training and test data were generated by solving (5.53) with a Finite Element Method on a discretization of size 100 × 100 of the unit square. Figure 5.7 shows an example of training input and output, a test prediction, and pointwise errors. 0.025 0.0 0.0 19.8 0.5 x1 1.0 (a) Training input 0.5 0.0 0.0 0.000 −0.025 0.5 x1 1.0 (b) Training output x2 x2 20.0 0.5 x2 0.025 0.000 0.5 0.0 0.0 −0.025 0.5 x1 1.0 (c) True test ×10−4 1.0 1.0 2.5 0.025 x2 1.0 x2 1.0 1.0 0.5 0.000 0.5 0.0 0.0 −0.025 0.5 x1 1.0 0.0 0.0 0.0 0.5 x1 1.0 (d) Predicted test (e) Pointwise error Figure 5.7: Example of training data and test prediction and pointwise errors for the Helmholtz problem (5.53). Structural mechanics We let Ω = [0, 1] 𝐷 = [0, 1] 2 , the equation that governs the displacement vector 𝑤 in an elastic solid undergoing infinitesimal deformations is ∇·𝜎 =0 in (0, 1) 2 , 𝑤 = 𝑤, ¯ on Γ𝑤 , ∇·𝑛=𝑢 on Γ𝑢 , (5.54) 204 where the boundary 𝜕𝐷 is split in [0, 1] × 1 = Γ𝑡 (the part of the boundary subject to stress) and its complement Γ𝑢 . The goal is to learn the operator that maps the one-dimensional load 𝑢 on Γ𝑢 to the two-dimensional von Mises stress field 𝑣 on Ω, i.e., G : 𝑢 ↦→ 𝑣. Here the distribution 𝜇 is GP (100, 4002 (−Δ + 32 𝐼) −1 ), with Δ being the Laplacian subject to homogeneous Neumann boundary conditions on the space of zero-mean functions. The function 𝑣 was obtained by a finite element code; see (De Hoop et al., 2022) for implementation details and the constitutive model used. Figure 5.8 shows an example of training input and outputs, a test prediction, and pointwise errors. 1.0 100 1.0 400 1.0 25 0.5 200 0.5 200 x2 0.5 250 x2 x2 t̄(x) 400 200 x2 1.0 500 0.5 0 0 −250 0.0 0.5 x 1.0 (a) Training input 0.0 0.0 0.5 x1 1.0 (b) Training output 0.0 0.0 0.5 x1 1.0 (c) True test 0.0 0.0 0.5 x1 1.0 0.0 0.0 0.5 x1 1.0 (d) Predicted test (e) Pointwise error Figure 5.8: Example of training data and test prediction and pointwise errors for the Structural Mechanics problem (5.54). Navier-Stokes equations Consider the vorticity-stream (𝜔, 𝜓) formulation of the incompressible NavierStokes equations:   ∫ 𝜕𝜓 𝜕𝜔 𝜕𝜓 + (𝑐 · ∇)𝜔 − 𝜈Δ𝜔 = 𝑢, 𝜔 = −Δ𝜓, 𝜓 = 0, 𝑐 = ,− , 𝜕𝑡 𝜕𝑥 2 𝜕𝑥1 𝐷 (5.55) where D = [0, 2𝜋] 2 , periodic boundary conditions are considered and the initial condition 𝑤(·, 0) is fixed. Here we are interested in the mapping from the forcing term 𝑢 to 𝑣 = 𝜔(·, 𝑇), the vorticity field at a given time 𝑡 = 𝑇, i.e., G † : 𝑢 ↦→ 𝜔(·, 𝑇). The distribution 𝜇 is GP (0, (−Δ+32 𝐼) −4 ). The viscosity 𝜈 is fixed and equal to 0.025, and the equation is solved on a 64×64 grid with a pseudo-spectral method and CrankNicholson time integration; see (De Hoop et al., 2022) for further implementation details. Figure 5.9 shows an example of input and output in the test set, along with an example of test prediction and pointwise errors. 205 2.5 x1 5.0 (a) Training input 0.0 0.0 2.5 x1 5.0 −0.1 (b) Training output 0.0 0.0 −0.05 2.5 x1 5.0 (c) True test 0.00 2.5 0.0 0.0 −0.05 2.5 x1 5.0 ×10 5.0 5 x2 2.5 −0.01 0.0 0.0 0.00 0.05 5.0 x2 0.0 2.5 2.5 0.05 5.0 x2 x2 0.00 5.0 x2 0.01 5.0 0 2.5 0.0 −5 −5 0 x1 5 (d) Predicted test (e) Pointwise error Figure 5.9: Example of training data and test prediction and pointwise errors for the Navier-Stokes problem (5.55). Results and discussion Below we discuss our main findings in benchmarking our kernel method against state-of-the-art NN based techniques. Performance against NNs Table 5.3 summarizes the 𝐿 2 relative test error of our vanilla implementation of the kernel method along with those of DeepONet, FNO, PCA-Net, and PARA-Net. We observed that our vanilla kernel method was reliable in terms of accuracy across all examples. In particular, observe that between the Matérn or rational quadratic kernel, we always managed to get close to the other methods, see for example the results for the Burgers’ equation or Darcy problem, and even outperform them in several examples such as Navier-Stokes and Helmholtz. Overall we observed that the performance of the kernel method is stable across all examples, suggesting that our method is reliable and provides a good baseline for a large class of problems. Moreover, we did not observe a significant difference in performance in terms of the choice of the particular kernel family once the hyper-parameters were tuned. This indicates that a large class of kernels are effective for these problems. Furthermore, we found the hyper-parameter tuning to be robust, i.e., results were consistent in a reasonable range of parameters such as length scales. In the high data regime, we found the vanilla kernel method to be the most accurate, although this comes with a greater cost, as seen in Figure 5.10. However, the kernel method appears to provide the highest accuracy for its level of complexity as the accuracy of NNs typically stagnates or even decreases after a certain level of complexity; see the Navier-Stokes and Helmholtz panels of Figure 5.10 where most of the NN methods seem to plateau after a certain complexity level. We also observed that the linear model did not provide the best accuracy as it quickly saturated in performance. Nonetheless, it provided surprisingly good accuracy at low 206 levels of complexity: for example, in the case of Navier-Stokes, the linear kernel provided the best accuracy below 106 FLOPS of complexity. This indicates that while simple, the linear model can be a valuable low-complexity model. Another notable example is the Advection equation (both I and II), where the operator G † is linear. In this case, the linear kernel had the best accuracy and the best complexityaccuracy tradeoff. We note, however, that while the linear model was close to machine precision on Advection I (error on the order 10−13 %), its performance was significantly worse on Advection II (error on the order of 10%). Moreover, the gap between the linear kernel and all other models was significantly smaller for Advection I; we conjecture this difference in performance is likely due to the setup of these problems. Finally, we note that the most challenging problem for our kernel method was the Structural Mechanics example. In this case, the vanilla kernel method has higher complexity but did not beat the NNs. In fact, the NNs seem to be able to reduce complexity without loss of accuracy compared to our method. DeepONet POD-DeepONet FNO PCA-Net PARA-Net Linear Best of Matérn/RQ Burger’s 2.15% 1.94% 1.93% n/a n/a 36.24% 2.15% Low-data regime Darcy problem Advection I 2.91% 0.66% 2.32% 0.04% 2.41% 0.22% n/a n/a n/a n/a 6.74% 2.15 × 10−13 % 2.75% 2.75 × 10−3 % Advection II 15.24% n/a 13.49% 12.53% 16.64% 11.28% 11.44% High-data regime Hemholtz Structural Mechanics 5.88% 5.20% n/a n/a 1.86% 4.76% 2.13% 4.67% 12.54% 4.55% 10.59% 27.11% 1.00% 5.18% Navier Stokes 3.63% n/a 0.26% 2.65% 4.09% 5.41% 0.12% Table 5.3: Summary of numerical results: we report the 𝐿 2 relative test error of our numerical experiments and compare the kernel approach with variations of DeepONet , FNO, PCA-Net, and PARA-Net. We considered two choices of the kernel 𝑆, the rational quadratic and the Matérn, but we observed little difference between the two. Effect of preconditioners Table 5.4 compares the performance of our method with the Matérn kernel family using various preconditioning steps. Overall we observed that both PCA and Cholesky preconditioning improved the performance of our vanilla kernel method. The Cholesky preconditioning generally offers the greatest improvement. However, we observed that getting the best results from the Cholesky approach required careful tuning of the parameters of the kernels 𝐾 and 𝑄 which we did using crossvalidation. While tuning the parameters does not increase the inference complexity, it does increase the training complexity. 207 Navier Stokes Test error 100 100 10−1 10 Helmholtz 100 Structural Mechanics 6 × 10−1 10−1 −2 10 10 10−3 10−4 4 10 106 108 1010 4 × 10−1 3 × 10−1 −1 −2 10−3 5 10 Advection 100 2 × 10−1 107 10−2 4 10 109 106 108 1010 10−1 104 106 108 Complexity (flops) PCANet DeepONet ParaNet FNO Linear Vanilla GP GP + PCA Figure 5.10: Accuracy complexity tradeoff achieved in the problems in (De Hoop et al., 2022). Data for NNs was obtained from the aforementioned article. Linear model refers to the linear kernel, vanilla GP is our implementation with the nonlinear kernels and minimal preprocessing, GP+PCA corresponds to preprocessing through PCA both the input and the output to reduce complexity. On the other hand, the PCA approach was more robust to changes in hyperparameters, i.e., the number of PCA components following Section 5.4. We observed that applying PCA on the input and output reduces complexity and has varying levels of effectiveness in providing a better cost-accuracy tradeoff. For example, for NavierStokes, it greatly reduced the complexity without affecting accuracy. But for the Helmholtz and Advection equations, PCA reduced the accuracy while remaining competitive with NN models. For structural mechanics, however, PCA significantly reduced accuracy and was worse than other models. We hypothesize that the loss in accuracy can be related to the decay of the eigenvalues of the PCA matrix in that example. No preprocessing PCA Cholesky Advection II 14.37% 14.50% 11.44% Burger’s 3.04% 2.41% 2.15% Darcy problem 4.47% 2.89% 2.75% Table 5.4: Comparison between Cholesky preconditioning and PCA dimensionality reduction on three examples for our vanilla kernel implementation with the Matérn kernel. 5.5 Conclusions In this work we presented a kernel/GP framework for the learning of operators between function spaces. We presented an abstract formulation of our kernel framework along with convergence proofs and error bounds in certain asymptotic limits. Numerical experiments and benchmarking against popular NN based algorithms revealed that our vanilla implementation of the kernel approach is competitive and either matches the performance of NN methods or beats them in several benchmarks. 208 Due to simplicity of implementation, flexibility, and the empirical results, we suggest that the proposed kernel methods are a good benchmark for future, perhaps more sophisticated, algorithms. Furthermore, these methods can be used to guide practitioners in the design of new and challenging benchmarks (e.g, identify problems where vanilla kernel methods do not perform well). Numerous directions of future research exist. In the theoretical direction it is interesting to remove the stringent Condition 5.3.2 and we anticipate this to require a particular selection of the kernel employed to obtain the map 𝑓¯. Moreover, obtaining error bounds for more general measurement functionals beyond pointwise evaluations would be interesting. One could also adapt our framework to non-vanilla kernel methods such as random features or inducing point methods to provide a low-complexity alternative to NNs in the large-data regime. Finally, since the proposed approach is essentially a generalization of GP Regression to the infinite-dimensional setting, we anticipate that some of the hierarchical techniques of (Houman Owhadi, 2017; Schaäfer, Katzfuss, and Houman Owhadi, 2021; Florian Schäfer, Timothy John Sullivan, and Houman Owhadi, 2021a) could be extended to this setting and provide a better cost-accuracy trade-off than current methods. 5.6 Review of operator valued kernels and GPs We review the theory of operator valued kernels and GPs (Houman Owhadi, 2023a) as these are utilized throughout the article. Operator-valued kernels were introduced in (Kadri et al., 2016) as a generalization of vector-valued kernels (Alvarez, Rosasco, Lawrence, et al., 2012). Operator valued kernels Let U and V be separable Hilbert spaces endowed with the inner products ·, · U and ·, · V . Write L (V) for the set of bounded linear operators mapping V to V. Definition 5.6.1. We call 𝐺 : U × U → L (V) an “operator-valued kernel” if 1. 𝐺 is Hermitian, i.e. 𝐺 (𝑢, 𝑢′) = 𝐺 (𝑢′, 𝑢)𝑇 for all 𝑢, 𝑢′ ∈ U , writing 𝐴𝑇 for the adjoint of the operator 𝐴 with respect to ·, · V . 𝑚 ⊂ U ×V 2. 𝐺 is non-negative, i.e., for all 𝑚 ∈ N and any set of points (𝑢𝑖 , 𝑣 𝑖 )𝑖=1 Í it holds that 𝑖,𝑚𝑗=1 𝑣 𝑖 , 𝐺 (𝑢𝑖 , 𝑢 𝑗 )𝑣 𝑗 V ≥ 0. We call 𝐺 non-degenerate if whenever 𝑢𝑖 ≠ 𝑢 𝑗 for 𝑖 ≠ 𝑗. Í𝑚 𝑖, 𝑗=1 𝑣 𝑖 , 𝐺 (𝑢𝑖 , 𝑢 𝑗 )𝑣 𝑗 V = 0 implies 𝑣 𝑖 = 0 for all 𝑖 209 RKHSs Each non-degenerate, locally bounded and separately continuous operator-valued kernel 𝐺 is in one to one correspondence with an RKHS H of continuous operators G : U → V obtained as the closure of the linear span of the maps 𝑧 ↦→ 𝐺 (𝑧, 𝑢)𝑣 with respect to the inner product identified by the reproducing property 𝑔, 𝐺 (·, 𝑢)𝑣 H = 𝑔(𝑢), 𝑣 V . (5.56) Feature maps Let F be a separable Hilbert space (with inner product ·, · F and norm ∥ · ∥ F ) and let 𝜓 : U → L (V, F ) be a continuous function mapping U to the space of bounded linear operators from V to F . Definition 5.6.2. We say that F and 𝜓 : U → L (V, F ) are a feature space and a feature map for the kernel 𝐺 if, for all (𝑢, 𝑢′, 𝑣, 𝑣 ′) ∈ U 2 × V 2 , 𝑣, 𝐺 (𝑢, 𝑢′)𝑣 ′ = 𝜓(𝑢)𝑣, 𝜓(𝑢′)𝑣 ′ F . Write 𝜓𝑇 (𝑢), for the adjoint of 𝜓(𝑢) defined as the linear function mapping F to V satisfying 𝜓(𝑢)𝑣, 𝛼 F = 𝑣, 𝜓𝑇 (𝑢)𝛼 V for 𝑢, 𝑣, 𝛼 ∈ U × V × F . Note that 𝜓𝑇 : U → L (F , V) is therefore a function mapping U to the space of bounded linear functions from F to V. Writing 𝛼𝑇 𝛼′ := 𝛼, 𝛼′ F for the inner product in F we can ease our notations by writing 𝐺 (𝑢, 𝑢′) = 𝜓𝑇 (𝑢)𝜓(𝑢′) (5.57) which is consistent with the finite-dimensional setting and 𝑣𝑇 𝐺 (𝑢, 𝑢′)𝑣 ′ = (𝜓(𝑢)𝑣)𝑇 (𝜓(𝑢′)𝑣 ′) (writing 𝑣𝑇 𝑣 ′ for the inner product in V). For 𝛼 ∈ F write 𝜓𝑇 𝛼 for the function U → V mapping 𝑢 ∈ U to the element 𝑣 ∈ V such that 𝑣 ′, 𝑣 V = 𝑣 ′, 𝜓𝑇 (𝑢)𝛼 V = 𝜓(𝑢)𝑣 ′, 𝛼 F for all 𝑣 ′ ∈ V . We can, without loss of generality, restrict F to be the range of (𝑢, 𝑣) → 𝜓(𝑢)𝑣 so that the RKHS H defined by 𝐺 is the closure of the pre-Hilbert space spanned by 𝜓𝑇 𝛼 for 𝛼 ∈ F . Note that the reproducing property (5.56) implies that for 𝛼 ∈ F 𝜓𝑇 (·)𝛼, 𝜓𝑇 (·)𝜓(𝑢)𝑣 H = 𝜓𝑇 (𝑢)𝛼, 𝑣 V = 𝛼, 𝜓(𝑢)𝑣 F for all 𝑢, 𝑣 ∈ U × V, which leads to the following theorem. 210 Theorem 5.6.3. The RKHS H defined by the kernel (5.57) is the linear span of 𝜓𝑇 𝛼 over 𝛼 ∈ F such that ∥𝛼∥ F < ∞. Furthermore, 𝜓𝑇 (·)𝛼, 𝜓𝑇 (·)𝛼′ H = 𝛼, 𝛼′ F and ∥𝜓𝑇 (·)𝛼∥ 2H = ∥𝛼∥ 2F for 𝛼, 𝛼′ ∈ F . Interpolation Let us consider the interpolation problem in operator valued RKHSs. Problem 3. Let G † be an unknown continuous operator mapping U to V. Given the information8 G † (u) = v with the data (u, v) ∈ U 𝑁 × V 𝑁 , approximate G † . Using the relative error in ∥ · ∥ H -norm as a loss, the minimax optimal recovery solution of Problem 3 is, by (Houman Owhadi and Clint Scovel, 2019a, Thm. 12.4, 12.5), given by    Minimize ∥G∥ 2H  (5.58)   subject to G(u) = v.  Í The minimizer is then of the form G(·) = 𝑁𝑗=1 𝐺 (·, 𝑢 𝑗 )𝑤 𝑗 , where the coefficients Í 𝑤 𝑗 ∈ V are identified by solving the system of linear equations 𝑁𝑗=1 𝐺 (𝑢𝑖 , 𝑢 𝑗 )𝑤 𝑗 = 𝑣 𝑖 for all 𝑖 ∈ {1, . . . , 𝑁 } . Using our compressed notation we can rewrite this equation as 𝐺 (u, u)w = v where w = (𝑤 1 , . . . , 𝑤 𝑁 ), v = (𝑣 1 , . . . , 𝑣 𝑁 ) ∈ V 𝑁 and 𝐺 (u, u) is the 𝑁 × 𝑁 block-operator matrix 9 with entries 𝐺 (𝑢𝑖 , 𝑢 𝑗 ). Therefore, writing 𝐺 (·, u) for the vector (𝐺 (·, 𝑢 1 ), . . . , 𝐺 (·, 𝑢 𝑁 )) ∈ H 𝑁 , the optimal recovery interpolant is given by Ḡ(·) = 𝐺 (·, u)𝐺 (u, u) −1 v , (5.59) which implies that the value of (5.58) at the minimum is ∥ Ḡ∥ 2H = v𝑇 𝐺 (u, u) −1 v , (5.60) where 𝐺 (u, u) −1 is the inverse of 𝐺 (u, u), whose existence is implied by the nondegeneracy of 𝐺 combined with 𝑢𝑖 ≠ 𝑢 𝑗 for 𝑖 ≠ 𝑗. 8 For a 𝑁-vector u = (𝑢 , . . . , 𝑢 ) ∈ U 𝑁 and a function G 1 𝑁
: U → V, write G(u) for the 𝑁 vector with entries G(𝑢 1 ), . . . , G(𝑢 𝑁 ) . 9 For 𝑁 ≥ 1 let V 𝑁 be the N-fold product space endowed with the inner-product v, w := V𝑁 Í𝑁 𝑁 𝑁 𝑖, 𝑗=1 𝑣 𝑖 , 𝑤 𝑗 V for v = (𝑣 1 , . . . , 𝑣 𝑁 ), w = (𝑤 1 , . . . , 𝑤 𝑁 ) ∈ V . A ∈ L (V ) given by A = 𝐴1,1 · · · 𝐴1, 𝑁 © . .. ª® where 𝐴 ∈ L (V), is called a block-operator matrix. Its adjoint AT with ­ . 𝑖, 𝑗 . ® ­ . « 𝐴 𝑁 ,1 · · · 𝐴 𝑁 , 𝑁 ¬ respect to ·, · V 𝑁 is the block-operator matrix with entries ( 𝐴𝑇 )𝑖, 𝑗 = ( 𝐴 𝑗,𝑖 ) 𝑇 . 211 Ridge regression Let 𝛾 > 0. A ridge regression (approximate) solution to Problem 3 can be found as the minimizer of 𝑁 ∑︁ inf 𝜆 ∥G∥ 2H + 𝛾 −1 ∥𝑣 𝑖 − G(𝑢𝑖 )∥ 2V . (5.61) G∈H 𝑖=1 This minimizer is given by the formula Ḡ(𝑢) = 𝐺 (𝑢, u) 𝐺 (u, u) + 𝛾𝐼
−1 v, (5.62) writing 𝐼 for the identity matrix. We can further compute directly ∥ Ḡ∥ 2H = v𝑇 𝐺 (u, u) + 𝛾𝐼
−1 v. Operator-valued GPs The following definition of operator-valued Gaussian processes is a natural extension of scalar-valued Gaussian fields (Houman Owhadi and Clint Scovel, 2019a). Definition 5.6.4. (Houman Owhadi, 2023a, Def. 5.1) Let 𝐺 : U × U → L (V) be an operator-valued kernel. Let 𝑚 be a function mapping U to V. We call 𝜉 : U → L (V, H) an operator-valued GP if 𝜉 is a function mapping 𝑢 ∈ U to 𝜉 (𝑢) ∈ L (V, H) where H is a Gaussian space and L (V, H) is the space of bounded linear operators from V to H. Abusing notations we write 𝜉 (𝑢), 𝑣 V for 𝜉 (𝑢)𝑣. We say that 𝜉 has mean 𝑚 and covariance kernel 𝐺 and write 𝜉 ∼ N (𝑚, 𝐺) if
𝜉 (𝑢), 𝑣 V ∼ N 𝑚(𝑢), 𝑣𝑇 𝐺 (𝑢, 𝑢)𝑣 and Cov
𝜉 (𝑢), 𝑣 V , 𝜉 (𝑢′), 𝑣′ V = 𝑣𝑇 𝐺 (𝑢, 𝑢′)𝑣 ′ . (5.63) We say that 𝜉 is centered if it is of zero mean. If 𝐺 (𝑢, 𝑢) is trace class (Tr[𝐺 (𝑢, 𝑢)] < ∞) then 𝜉 (𝑢) defines a measure on V, i.e. a V-valued random variable 10 . Theorem 5.6.5. (Houman Owhadi, 2023a, Thm. 5.2) The law of an operator-valued GP is uniquely determined by its mean 𝑚 and covariance kernel 𝐺. Conversely given 𝑚 and 𝐺 there exists an operator-valued GP having mean 𝑚 and covariance kernel 𝐺. In particular if 𝐺 has feature space F and map 𝜓, the 𝑒𝑖 form an orthonormal Í basis of F , and the 𝑍𝑖 are i.i.d. N (0, 1) random variables, then 𝜉 = 𝑚 + 𝑖 𝑍𝑖 𝜓𝑇 𝑒𝑖 is an operator-valued GP with mean 𝑚 and covariance kernel 𝐺. 10 Otherwise it only defines a (weak) cylinder-measure in the sense of Gaussian fields. 212 Theorem 5.6.6. (Houman Owhadi, 2023a, Thm. 5.3) Let 𝜉 be a centered operatorvalued GP with covariance kernel 𝐺 : U × U → L(V). Let u, v ∈ U 𝑁 × V 𝑁 . Let 𝑍 = (𝑍1 , . . . , 𝑍 𝑁 ) be a random Gaussian vector, independent from 𝜉, with i.i.d. N (0, 𝛾𝐼V ) entries (𝛾 ≥ 0 and 𝐼V is the identity map on V). Then 𝜉 conditioned on 𝜉 (u) + 𝑍 is an operator-valued GP with mean  
−1 E 𝜉 (𝑢) 𝜉 (u) + 𝑍 = v = 𝐺 (𝑢, u) 𝐺 (u, u) + 𝛾𝐼V v = (5.62) (5.64) and conditional covariance operator 𝐺 ⊥ (𝑢, 𝑢′) := 𝐺 (𝑢, 𝑢′) − 𝐺 (𝑢, u) 𝐺 (u, u) + 𝛾𝐼V
−1 𝐺 (u, 𝑢′) . (5.65) In particular, if 𝐺 is trace class, then h i   2 2 𝜎 (𝑢) := E 𝜉 (𝑢) − E[𝜉 (𝑢)|𝜉 (u) + 𝑍 = v] V 𝜉 (u) + 𝑍 = v = Tr 𝐺 ⊥ (𝑢, 𝑢) . (5.66) Deterministic error estimates for operator-valued regression The following theorem shows that the standard deviation (5.66) provides deterministic a prior error bounds on the accuracy of the ridge regressor (5.64) to G † in Problem 3. Local error estimates such as (5.67) below are classical in the Kriging literature (Wu and Schaback, 1993) where 𝜎 2 (𝑢) is known as the power function/kriging variance; see also (Houman Owhadi, 2015a)[Thm. 5.1] for applications to PDEs. Theorem 5.6.7. (Houman Owhadi, 2023a, Thm. 5.4) Let G † be the unknown function of Problem 3 and let G(𝑢) = (5.64) = (5.62) be its ridge regressor. Let H be the RKHS associated with 𝐺 and let H𝛾 be the RKHS associated with the kernel 𝐺 𝛾 := 𝐺 + 𝛾𝐼V . It holds true that G † (𝑢) − G(𝑢) V ≤ 𝜎(𝑢)∥G † ∥ H (5.67) and G † (𝑢) − G(𝑢) V ≤ √︁ 𝜎 2 (𝑢) + 𝛾 dim(V)∥G † ∥ H𝛾 , (5.68) where 𝜎(𝑢) is the standard deviation (5.66). 5.7 An alternative regularization of operator regression For 𝛾 > 0, the regularization implied by (5.27) is equivalent to adding noise on the 𝜑(v) measurements. If one could observe v (and not just 𝜑(v)), then an alternative approach to regularizing the problem is to add noise to 𝜉 (u). To describe this let 213 𝑍 ′ = (𝑍1′ , . . . , 𝑍 𝑁′ ) be a random block-vector, independent from 𝜉, with i.i.d. entries 𝑍 ′𝑗 ∼ N (0, 𝛾𝐼V ) for 𝑗 = 1, . . . , 𝑁 (where 𝐼V denotes the identity map on V). Then the GP 𝜉 conditioned on 𝜉 (u) = v + 𝑍 ′ is a GP with conditional covariance e𝛾 =(5.62) that is also the minimizer of (5.61). kernel (5.65) and conditional mean G e𝛾 = 𝜒 ◦ e 𝑓𝛾 ◦ 𝜙 where Observing11 that 𝜑(𝑍𝑖′) ∼ N (0, 𝛾𝐾 (𝜑, 𝜑)) , we deduce that G e 𝑓𝛾 minimizes     Minimize   Over  ∥ 𝑓 ∥ 2Γ + 𝛾 −1 Í𝑁 𝑖=1 ( 𝑓 (𝑈𝑖 ) − 𝑉𝑖 ) 𝑇 𝐾 (𝜑, 𝜑) −1 ( 𝑓 (𝑈 ) − 𝑉 ) . 𝑖 𝑖 (5.69) 𝑓 ∈ HΓ . Furthermore, the distribution of 𝜉 conditioned on 𝜉 (u) = v + 𝑍 ′ is that of 𝜒 ◦ 𝜁e⊥ ◦ 𝜙 where 𝜁e⊥ ∼ N ( e 𝑓𝛾 , e Γ⊥ ) is the GP 𝜁 conditioned on 𝜁 (U) = V+ 𝜑(𝑍 ′), whose mean is e 𝑓𝛾 and conditional covariance kernel is e Γ⊥ (𝑈, 𝑈 ′) = Γ(𝑈, 𝑈 ′) − Γ(𝑈, U)(Γ(U, U) + 𝛾 𝐴) −1 Γ(U, 𝑈 ′) where 𝐴 is a 𝑁 × 𝑁 block diagonal matrix with 𝐾 (𝜑, 𝜑) as diagonal entries. 5.8 Expressions for the kernels used in experiments Below we collect the expressions for the kernels that were referred to in the article or utilized for our numerical experiments. These can be found in many standard textbooks on GPs such as (Rasmussen and Williams, 2006). The linear kernel The linear kernel has the simple expression 𝐾linear (𝑥, 𝑥 ′) = 𝑥, 𝑥 ′ and may be defined on any inner product space. It has no hyper-parameters. The rational quadratic kernel The rational quadratic kernel has the expression 𝐾 (𝑥, 𝑥 ′) = 𝑘 RQ (∥𝑥 − 𝑥 ′ ∥) where   −𝛼 𝑟2 𝑘 RQ (𝑟) = 1 + 2 . (5.70) 2𝑙 It has hyper-parameters 𝛼 > 0 and 𝑙. The Matérn parametric family The Matérn kernel family is of the form 𝐾 (𝑥, 𝑥 ′) = 𝑘 (∥𝑥 − 𝑥 ′ ∥ where  √  √  𝑝−𝑖 𝑝 2𝜈𝑟 Γ( 𝑝 + 1) ∑︁ ( 𝑝 + 1)! 8𝜈𝑟 𝑘 𝜈 (𝑟) = exp − , 𝑙 Γ(2𝑝 + 1) 𝑖=0 𝑖!( 𝑝 − 𝑖)! 𝑙 (5.71) 11 This follows from 𝜑(𝑍 ′ ) ∼ N (0, 𝛾𝜑𝜑𝑇 ) where 𝜑𝑇 is the adjoint of 𝜑 identified as the linear 𝑖 map from R𝑚 to V satisfying 𝑊, 𝜑(𝑤) R𝑚 = 𝜑⊥𝑊, 𝑤 V for 𝑤 ∈ V and 𝑊 ∈ R𝑚 (i.e., 𝜑𝑇 (𝑊) = (𝐾 𝜑)𝑊). 214 for 𝜈 = 𝑝 + 12 . This kernel has hyper-parameters 𝑝 ∈ Z+ and 𝑙 > 0. In the limiting case where 𝜈 → ∞, the Matérn kernel, we obtain the Gaussian or squared exponential kernel:   𝑟2 𝑘 ∞ (𝑟) = exp − 2 , (5.72) 2𝑙 with hyper-parameter 𝑙 > 0. 215 Chapter 6 ERROR ANALYSIS OF KERNEL/GP METHODS FOR NONLINEAR AND PARAMETRIC PDES We introduce a priori Sobolev-space error estimates for the solution of arbitrary nonlinear, and possibly parametric, PDEs that are defined in the strong sense, using Gaussian process and kernel based methods. The primary assumptions are (1) a continuous embedding of the reproducing kernel Hilbert space of the kernel into a Sobolev space of sufficient regularity, and (2) the stability of the differential operator and the solution map of the PDE between corresponding Sobolev spaces. The proof is articulated around Sobolev norm error estimates for kernel interpolants and relies on the minimizing norm property of the solution. The error estimates demonstrate dimension-benign convergence rates if the solution space of the PDE is smooth enough. We illustrate these points with applications to high-dimensional nonlinear elliptic PDEs and parametric PDEs. Although some recent machine learning methods have been presented as breaking the curse of dimensionality in solving high-dimensional PDEs, our analysis suggests a more nuanced picture: there is a trade-off between the regularity of the solution and the presence of the curse of dimensionality. Therefore, our results are in line with the understanding that the curse is absent when the solution is regular enough. 6.1 Introduction In recent years the adoption of machine learning in the natural sciences and engineering has led to the development of new methods for solving PDEs (Raissi, Perdikaris, and George E Karniadakis, 2019; Weinan, Han, and Jentzen, 2017; Weinan and B. Yu, 2018; Li, Kovachki, Azizzadenesheli, Bhattacharya, et al., 2020; L. Lu, Jin, et al., 2021). The majority of these methods rely on the approximation power of artificial neural networks (ANNs) either as a function class to approximate the solution of the PDE or as a high-dimensional function class to approximate the solution map of the PDE. Despite the empirical success of the aforementioned ANN based methods, current theoretical understanding of these PDE solvers is scarce and, beyond particular PDEs (e.g., (Shin, Darbon, and George Em Karniadakis, 2020; Y. Lu et al., 2021; De Ryck and Mishra, 2022)), results are oftentimes limited to existence results rather than convergence guarantees or rates. 216 Similar to ANNs, kernel methods and Gaussian processes (GPs) have been very effective in scientific computing and machine learning (Scholkopf and Alexander J Smola, 2018; Muandet et al., 2017; Berlinet and Thomas-Agnan, 2011; Williams and Rasmussen, 2006) and at the same time they are supported by rigorous theoretical foundation (Berlinet and Thomas-Agnan, 2011; Wendland, 2004; Houman Owhadi and Clint Scovel, 2019b). Recently in (Yifan Chen, Hosseini, et al., 2021b), the authors introduced a kernel collocation method for solving arbitrary nonlinear PDEs with a rigorous convergence guarantee. The theory presented in that work was based on the assumptions that (1) the solution belongs to the reproducing kernel Hilbert space (RKHS) defined by the underlying kernel which in turn is embedded in the Sobolev space 𝐻 𝑠 for 𝑠 > 𝑑/2 +“order of the PDE” (where 𝑑 is the dimension of the domain of the PDE) and (2) the fill-distance between collocation points goes to zero. Convergence was proved via a compactness argument but no convergence rates were provided. The goal of this article is to provide quantitative convergence rates for the PDE solver introduced in (Yifan Chen, Hosseini, et al., 2021b). Our quantitaitve rates also reveal the interplay between the regularity of the solution of the PDE and the dimension 𝑑 of the problem. At the same time we make improvements to the methodology of (Yifan Chen, Hosseini, et al., 2021b) and extend it to the case of parametric PDEs. In the rest of this section we summarize our main contributions in Section 6.1 followed by a review of the relevant literature in Section 6.1, and an outline of the article in Section 6.1. Significance of Contributions The use of meshless, collocation methods with radial basis functions for solving PDEs dates back to the 1990s (Kansa, 1990a; Kansa, 1990b; Franke and Schaback, 1998b; Fasshauer, 1999). Typical approaches include symmetric collocation (Franke and Schaback, 1998b; Fasshauer, 1999) and unsymmetric collocation methods (also known as the Kansa method (Kansa, 1990a; Kansa, 1990b)); see (Schaback and Wendland, 2006; Fornberg and Flyer, 2015) for reviews. It is recognized that the unsymmetric collocation approach may encounter instability issues and require additional techniques (Ling, Opfer, and Schaback, 2006; Schaback, 2007; Ling and Schaback, 2008; Cheung, Ling, and Schaback, 2018), whereas the symmetric collocation approach always yields positive definite symmetric matrices and is stable. The reason is that the symmetric collocation approach includes higherorder derivatives of the kernel as basis functions and leads to solutions that can be 217 identified as optimal recovery (worst case minimax optimal) solutions. This optimal recovery property makes the algorithm generally applicable for any well-posed linear PDEs (Hon and Schaback, 2008; Schaback, 2015; Schaback, 2016). Furthermore the analysis becomes straightforward because of this optimality; see (Franke and Schaback, 1998a; Franke and Schaback, 1998b) for linear PDEs and (Böhmer and Schaback, 2013; Böhmer and Schaback, 2020) for some degree of generalization to quasi-linear/nonlinear problems. Note that the optimality has long been recognized, but not extensively acknowledged however, as highlighted in (Schaback, 2015): “This technique has been around since at least 1998, but its optimality properties went unnoticed.” The GP-PDE methodology proposed in (Yifan Chen, Hosseini, et al., 2021b) can be seen as a nonlinear generalization of the optimal recovery approach to solving (and learning) arbitrary classically/strongly defined nonlinear PDEs. This paper aims to offer a simple and transparent theoretical error analysis of this nonlinear optimal recovery method. This analysis extends the linear setting (Giesl and Wendland, 2007) and shares conceptual steps (in terms of the role of stability and sampling inequalities) with (Böhmer and Schaback, 2013) while explicitly focusing on optimal recovery solutions rather than solutions obtained from finitedimensional trial spaces and residual minimization. Such theoretical analyses are notably rare within the sphere of machine learning-based PDE solutions (where theoretical guarantees are typically limited to existence results). Furthermore, while there is no general theory for strongly defined arbitrary nonlinear PDEs, the optimal recovery approach provides a way of obtaining general theoretical guarantee for the numerical approximation of such PDEs. Beyond its wide scope, the proposed analysis also lays the groundwork for developing rigorous, efficient, and scalable (near-linear complexity) learning-based methods for arbitrary nonlinear PDEs. This can be achieved by integrating the proposed error estimates with the fast algorithms developed in (Yifan Chen, Houman Owhadi, and Florian Schäfer, 2024) for kernel matrices whose entries contain higher-order derivatives of the kernel, a setting well suited for the optimal recovery approach. We note that the analysis in the paper is focused on the minimizer of a loss function induced by kernel and GP methods. Understanding theoretically how iterative algorithms are able to achieve this minimizer represents another crucial stride towards our ultimate objective; this could be potentially done by combining analysis results for iterative linerization, for example the work in (Becker et al., 2023). 218 Summary of Contributions Throughout the article we consider parametric PDEs of the form ( P (𝑢★)(𝒙; 𝜽) = 𝑓 (𝒙; 𝜽), (𝒙, 𝜽) ∈ Ω × Θ, B (𝑢★)(𝒙; 𝜽) = 𝑔(𝒙; 𝜽), (𝒙, 𝜽) ∈ 𝜕Ω × Θ, (6.1) where Ω ⊂ R𝑑 is a bounded connected domain with an appropriately smooth boundary 𝜕Ω, P and B are the interior and boundary differential operators that define the PDE and 𝑓 , 𝑔 are the source and boundary data. 𝒙 denotes the spatial variable with 𝜽 denoting a parameter belonging to a compact set Θ ⊂ R 𝑝 . The function 𝑢★ denotes the exact, strong solution of this PDE. We view the solution 𝑢★ as a function on Ω × Θ and approximate it in an appropriate RKHS by imposing the PDE as a constraint on a set of collocation points in the product space Ω × Θ. Our main contributions are four-fold as summarized below: 1. We extend the kernel PDE solver of (Yifan Chen, Hosseini, et al., 2021b) to the case of the parametric PDE (6.1). This extension follows by viewing the solution 𝑢★ (𝒙; 𝜃) as a continuous function defined on Ω×Θ and approximating it with a function 𝑢 † in an appropriate RKHS U after imposing the PDE as a constraint on a set of collocation points. At the same time we improve the efficacy and performance of the Gauss-Newton (GN) algorithm of (Yifan Chen, Hosseini, et al., 2021b) through an approach of “linearize first then apply the kernel solver”. For many prototypical PDEs this new approach leads to smaller kernel matrices that can be factored or inverted more efficiently. These numerical strategies are outlined in Section 6.2, and our proposed methodology is summarized in Algorithms 9 and 10. 2. We provide explicit a priori convergence rates for the kernel estimator 𝑢 † ∈ U to the true solution 𝑢★. Our proof relies on three assumptions: (1) the RKHS U ⊂ 𝐻 𝑠 (Ω) for 𝑠 > (𝑑 + 𝑝)/2 + “order of the PDE”, (2) the true unique solution 𝑢★ ∈ U, and (3) the forward PDE operator and the associated solution map of the PDE are Lipschitz stable. Our error estimates are of the general form ∥𝑢 † − 𝑢★ ∥ 𝐿 2 (Ω) ≲ ℎ 𝑠 ∥𝑢★ ∥ U , (6.2) where ℎ is the fill-distance (mesh-norm) of our collocation points. Indeed, if expressed in terms of 𝑁, the number of collocation points, the rate will read as O (𝑁 −𝑠/(𝑑+𝑝) ). The above rate indicates a trade-off between the regularity 219 of the solution space and the dimension 𝑑 + 𝑝 of Ω × Θ stating that the convergence rate is dimension-benign so long as the solution 𝑢★ is sufficiently regular; these results are outlined in Section 6.3. 3. In fact, our method for proving the rate (6.2) is more general than the case of PDEs (see Figure 6.1 for the road map of the proof technique). The proof can be viewed as a recipe for convergence analysis of solutions to nonlinear functional equations of the form P (𝑢) = 𝑓 where 𝑢, 𝑓 belong to sufficiently regular function spaces and P is invertible (at least locally). Then the Lipschitz stability of P and P −1 plus RKHS interpolation bounds on 𝑓 yield convergence rates for 𝑢. Results at this level of generality are presented in (Schaback, 2016) for linear maps P which are then extended to nonlinear problems in (Böhmer and Schaback, 2013; Böhmer and Schaback, 2020). In these works, the stability of the discretization method is furthermore assumed. In the GP methodology, this property is guaranteed, due to the minimal RKHS norm and optimal recovery property of the solution. This has been pointed out in (Schaback, 2016, Sec. 10) for linear PDEs. Our theory can be seen as a generalization of the result in (Schaback, 2016) to the nonlinear case. 4. We present a suite of numerical experiments that elucidate and extend our theoretical analysis in item 2. We present an example of a nonlinear elliptic PDE with a prescribed solution of varying regularity in various dimensions. We then explore the interplay between regularity and dimensionality as well as the rate in (6.2). We further verify our result for a one dimensional parametric PDE by varying 𝑝, the dimension of the parameter space Θ. Because of this trade-off between regularity and dimensionality, showing that a numerical method remains accurate for a high dimensional PDE may not be an indication that it is breaking the curse of dimensionality but simply an indication that the problem being solved is very regular; see our experiments in Section 6.4 and in particular Section 6.4. Literature Review Below we present a brief review of the literature relevant to the current work. Kernel and Gaussian Process Solvers for PDEs As mentioned earlier our algorithmic and theoretical developments are focused on the kernel method introduced in (Yifan Chen, Hosseini, et al., 2021b) and extending 220 Figure 6.1: A summary of the main steps in our proof of convergence rates outlined in Theorems 6.3.1, 6.3.2, 6.3.4 and 6.3.6. The 1–4 norms denote arbitrary norms on appropriate Banach spaces while the ∥ · ∥ U -norm can be chosen as an RKHS norm or another desired norm with respect to which the numerical algorithm is stable. that approach to parametric PDEs. Further extensions and applications of the aforementioned framework can also be found in (Mou, Yang, and Zhou, 2022; R. Meng and Yang, 2022; Long, Mrvaljevic, et al., 2022; Yifan Chen, Houman Owhadi, and Florian Schäfer, 2024). When applied to linear PDEs our kernel method coincides with the so-called symmetric collocation method (Schaback and Wendland, 2006, Sec. 14) and is closely associated with radial basis function (RBF) PDE solvers (Fornberg and Flyer, 2015; Fasshauer, 1999; Franke and Schaback, 1998b). Various error analyses for RBF collocation methods can be found in (Franke and Schaback, 1998a; Franke and Schaback, 1998b). In particular, the article (Giesl and Wendland, 2007) is the closest to our work and their rates coincide with ours in the linear PDE setting. The articles (Ling, Opfer, and Schaback, 2006; Schaback, 2007; Ling and Schaback, 2008; Cheung, Ling, and Schaback, 2018) present similar bounds for the so-called Kansa method (Kansa, 1990a; Kansa, 1990b), a non-symmetric RBF collocation PDE solver. Finally, (Schaback, 2016) presents an abstract set of convergence rates for RBF interpolation of “well-posed” linear maps between regular function spaces that includes RBF PDE solvers as a special case. All of the aforementioned analyses consider linear PDEs and some generalizations to nonlinear problems are studied in (Böhmer and Schaback, 2013; Böhmer and Schaback, 2020). The deep connection of Kernels and RKHSs to the theory of GPs (Bogachev, 1998; Larkin, 1972; Williams and Rasmussen, 2006; Vaart and Zanten, 2008) suggests that kernel PDE solvers can be viewed from lens of probability theory as a conditioning problem for GPs. While not as extensively developed as the kernel solvers mentioned earlier, this direction has been explored for the solution of linear PDEs as well as 221 nonlinear ODEs (O. A. Chkrebtii et al., 2016; Cockayne, C. J. Oates, et al., 2019; Houman Owhadi, 2015b; Särkkä, 2011; Swiler et al., 2020) and recent works have extended this idea to some nonlinear and time-dependent PDEs (Cockayne, C. Oates, et al., 2017; Raissi, Perdikaris, and George Em Karniadakis, 2018; J. Wang et al., 2021). The GP interpretation is attractive due to the ability to provide rigorous uncertainty estimates along with the solution to the PDE. The idea here is that the uncertainties can serve as a posterior or a priori error indicators for the PDE solver. Some ideas related to this direction were discussed in (Yifan Chen, Hosseini, et al., 2021b; Cockayne, C. Oates, et al., 2017). A fully probabilistic GP interpretation of our kernel framework for linear PDEs can be found in (Houman Owhadi, 2015b; Cockayne, C. Oates, et al., 2017; Houman Owhadi and Clint Scovel, 2019b) but the case of nonlinear PDEs remains partially investigated (Yifan Chen, Hosseini, et al., 2021b; R. Meng and Yang, 2022; Mou, Yang, and Zhou, 2022; Long, Z. Wang, et al., 2022). Moreover we note that in the GP framework, hierarchical Bayes learning can be used to select kernels to get better convergence rates (Yifan Chen, Houman Owhadi, and A. Stuart, 2021a; Wilson et al., 2016; Houman Owhadi and Yoo, 2019b; Darcy et al., 2023). Parametric and High-dimensional PDEs Parametric PDEs are ubiquitous is physical sciences and engineering and in particular in the context of uncertainty quantification (UQ) and solution of stochastic PDEs (SPDEs) (Cialenco, Fasshauer, and Ye, 2012; Ghanem and Spanos, 2003; Le Maitre and Knio, 2010; Ye, 2013). A vast literature exists on the subject, connecting it to reduced basis models (Almroth, Stern, and Brogan, 1978; Noor and J. M. Peters, 1980), emulation of computer codes (Kennedy and O’Hagan, 2001), reduced order models (Lucia, Beran, and Silva, 2004), and numerical homogenization (Houman Owhadi and Clint Scovel, 2019b); for settings that most closely resemble our problems we refer the reader to (Ghanem and Spanos, 2003; Xiu, 2010; Cohen and DeVore, 2015) for a general overview. Broadly speaking, the dominant approaches for approximation of high-dimensional and parametric solution maps include polynomial/Taylor approximation methods (Beck et al., 2012; Chkifa, Cohen, DeVore, et al., 2012; Chkifa, Cohen, and Schwab, 2014; Nobile, Raúl Tempone, and Webster, 2008; Nobile, Raul Tempone, and Webster, 2008); Galerkin methods (Gunzburger, Webster, and G. Zhang, 2014; Cohen, DeVore, and Schwab, 2010); reduced basis methods (J. S. Hesthaven, Rozza, Stamm, et al., 2016); and more recently ANN operator learning techniques such as (Li, Kovachki, Azizzadenesheli, Bhattacharya, 222 et al., 2020; L. Lu, Jin, et al., 2021). In comparison to the aforementioned works we propose to directly approximate the solution of the parametric PDE as a function on the tensor product space of the physical and parameter domains in a similar spirit as (Kempf, Wendland, and Rieger, 2019). The recent article (Batlle, Darcy, et al., 2023) also presents a kernel based operator learning approach to various PDE problems including parametric PDEs. The Curse of Dimensionality Although the trade-off between regularity and accuracy is well understood in numerical approximation/integration, where it has led to the development of the Kolmogorov 𝑁-width and stress tests for finite-element methods (Pinkus, 2012; Melenk, 2000; Babuška and Osborn, 2000), its impact is oftentimes overlooked when communicating the convergence of Machine Learning and Deep Learning methods for high-dimensional PDEs. In particular, since artificial neural networks (ANNs) can be interpreted as kernel methods (Neal, 1996; J. Lee et al., 2017; Houman Owhadi, 2023b) with data-dependent parameterized kernels, our results raise the further question of understanding whether the (empirically observed) convergence of ANN-based methods for high-dimensional PDEs is an indication of the absence of the curse (i.e., the regularity of the solution in selected numerical experiments is high) or the breaking of that curse. In particular, empirically observing numerical accuracy for an algorithm and particular solutions is insufficient to prove that the curse of dimensionality is broken, and one must also show that the underlying problem and those solutions are not too regular. We emphasize that the curse of dimensionality referred to here is the one associated with the worsening of the accuracy of a numerical approximation algorithm as a function of the dimension of the domain of the PDE as opposed to the impact of the curse on the number of degrees of freedom in the implementation level (e.g., finite difference methods suffer from that second curse but ANN/kernel based methods do not). The Potential Value of Kernel/GP Methods The proposed work aims to further develop Gaussian Process (GP) and kernel methods for solving PDEs. We are motivated to do so because GP methods have the potential to offer the best of both worlds by combining the profound theory underlying traditional methods (and in particular finite element methods) with the ease of implementation of emerging Deep Learning (DL) methods. They also come equipped with automatic uncertainty quantification (UQ) capabilities, not readily 223 available in either traditional or deep-learning based methods. And finally they provide easily implementable meshless methods that can be used to benchmark other machine learning based algorithms such as PINNs (Raissi, Perdikaris, and George E Karniadakis, 2019). Compared to traditional methods (such as finite element methods (FEM), finite volume methods (FVM), finite difference methods (FDM), spectral methods, etc), GP methods generalize meshless, RBF, optimal recovery methods and are flexible and applicable in high dimensions. Compared to DL methods that use an expressive neural network representation, GPs offer transparent methods that are easy to reproduce and analyze. Furthermore the natural probabilistic interpretation of GPs enables convenient UQ and also facilitates the process of scientific discovery itself (Houman Owhadi and Clint Scovel, 2019b); FEM and DL methods do not interface so cleanly with UQ. Moreover, with hierarchical kernel learning (Yifan Chen, Houman Owhadi, and A. Stuart, 2021a; Wilson et al., 2016; Houman Owhadi and Yoo, 2019b; Darcy et al., 2023), GP methods can also be made highly expressive. In fact, as shown in the table below, GP methods can offer many advantages over traditional and DL methods. In the context of PDEs these advantages include greater flexibility, applicability in high dimensions, provable guarantees, near-linear complexity computation, Occam’s razor principle in the design of statistical models, mathematical transparency and interpretability, and ease of reproducibility; see Table 6.1. Although software support for GPs is currently not as advanced as that for DL and traditional methods, GPs are still easy to program and can be seamlessly integrated into an engineering pipeline. Table 6.1 should be interpreted in this light: as an argument for further deployment and development of software infrastructure for GP-PDE based methods. Method Trad. Kernel ANN Ease of implementation in high-dimensions ✗ ✓ ✓ Provable guarantees ✓ ✓ Limited Near linear complexity ✓ ✓ ✗ Occam’s razor Transparent Ease of reproducibility Built-in UQ Software support ✓ ✓ ✗ ✓ ✓ ✗ ✓ ✓ Limited ✗ ✓ ✗ ✓ Limited ✓ Table 6.1: A qualitative comparison of the properties of traditional PDE solvers (such as FEM, FVM, FDM, spectral methods, etc.) against kernel methods and ANNs. Given their long training times, ANN-based methods may not be competitive with FEM in low dimensions (Grossmann et al., 2023). In contrast, GP-based methods 224 can achieve near-linear complexity when combined with fast algorithms for kernel methods such as the sparse Cholesky factorization (Florian Schäfer, Katzfuss, and Houman Owhadi, 2021; Florian Schäfer, Timothy John Sullivan, and Houman Owhadi, 2021b; Yifan Chen, Houman Owhadi, and Florian Schäfer, 2024). In some applications, these algorithms can be competitive (both in terms of complexity and accuracy) even when compared to highly optimized algebraic multigrid solvers such as AMGCL and Trilinos (J. Chen et al., 2021). GP methods are naturally amenable to analysis and come with simple provable guarantees, while ANN-based methods involve complicated optimizations and many heuristics, which can make them hard to understand. GP methods fit Occam’s razor, offering a clarity of purpose in their structure. We can understand why and when they work, which is of scientific importance (Feynman, 1998). Therefore, it is of potential value to benchmark deep learning methods against kernel-based methods to ensure that the deep part of a DL method serves a significant purpose beyond adding complexity. Outline of the Article The rest of the article is organized as follows: Ww present a brief overview of our GP and kernel approach for solving nonlinear and parametric PDEs in Section 6.2; our error analysis is outlined in Section 6.3, followed by numerical experiments in Section 6.4 and conclusions in Section 6.5. Auxiliary results are collected in Sections 6.6 to 6.8. 6.2 Kernel Methods for Parametric PDEs In this section we extend the kernel methodology of (Yifan Chen, Hosseini, et al., 2021b) to the case of parametric PDEs as outlined in Section 6.2. Some numerical strategies and ideas for improving the efficiency of the solver are discussed in Section 6.2. Solving Parametric PDEs Let us consider bounded connected domains Ω ∈ R𝑑 with a Lipschitz boundary for 𝑑 ≥ 1 and Θ ⊂ R 𝑝 for 𝑝 ≥ 1. We consider nonlinear and parametric PDEs of the form ( P (𝑢★)(𝒙; 𝜽) = 𝑓 (𝒙; 𝜽), (𝒙, 𝜽) ∈ Ω × Θ, (6.3) B (𝑢★)(𝒙; 𝜽) = 𝑔(𝒙; 𝜽), (𝒙, 𝜽) ∈ 𝜕Ω × Θ, where P, B are nonlinear differential operators in the interior and boundary of Ω, 𝜽 is a parameter, and 𝑓 , 𝑔 are the PDE source and boundary data. For now we 225 assume that the above PDE is well-posed and has a unique solution 𝑢★ (𝒙; 𝜽) which is assumed to exist in the strong sense over Ω and for all values of 𝜽 ∈ Θ. In (Yifan Chen, Hosseini, et al., 2021b) the authors introduced a GP/kernel method for solving nonlinear PDEs of the form (6.3) without parametric dependence. Here we extend that approach to the parametric case. Let Υ := Ω × Θ and 𝜕Υ := 𝜕Ω × Θ and write s := (𝒙, 𝜽). Choose 𝑀 ≥ 1 collocation 𝑀Ω 𝑀 𝑀 points {s𝑚 } 𝑚=1 ∈ Υ such that {s𝑚 } 𝑚=1 ∈ Υ and {s𝑚 } 𝑚=𝑀 ∈ 𝜕Υ1 and consider Ω +1 a kernel 𝐾 : Υ × Υ → R with its corresponding RKHS denoted by U and norm ∥ · ∥ U . We then propose to approximate 𝑢★ (s) by solving the optimization problem   minimize ∥𝑢∥ U     𝑢∈U  (6.4) st P (𝑢)(s𝑚 ) = 𝑓 (s𝑚 ), 𝑚 = 1, . . . , 𝑀Ω ,      B (𝑢)(s𝑚 ) = 𝑔(s𝑚 ), 𝑚 = 𝑀Ω + 1, . . . , 𝑀.  Observe that our approach above approximates the solution 𝑢 as a function defined on the set product set Υ which is different from previous works (Beck et al., 2012; Chkifa, Cohen, DeVore, et al., 2012; Chkifa, Cohen, and Schwab, 2014; Nobile, Raúl Tempone, and Webster, 2008; Nobile, Raul Tempone, and Webster, 2008) where the solution map 𝜽 → 𝑢★ (·; 𝜽), as a mapping from Θ to an appropriate function space, is characterized and approximated. The latter approach requires different discretization methods for the 𝜽 parameter and the functions 𝑢★ (·; 𝜽) while our approach leads to a meshless collocation method on the product space which is desirable and convenient at the level of implementation, following (Yifan Chen, Hosseini, et al., 2021b, Sec. 3.1) (see also (Giesl and Wendland, 2007)). We make the following assumption on the differential operators P, B. Assumption 1. There exist bounded and linear operators 𝐿 1 , . . . , 𝐿 𝑄 Ω ∈ L U; 𝐶 (Υ)
and 𝐿 𝑄 Ω +1 , . . . , 𝐿 𝑄 ∈ L U; 𝐶 (𝜕Υ) for some 1 ≤ 𝑄 Ω < 𝑄 together with maps 𝑃 : R𝑄 Ω → R and 𝐵 : R𝑄−𝑄 Ω → R, which may be nonlinear, so that P, B can be written as   P (𝑢)(s) = 𝑃 𝐿 1 (𝑢)(s), . . . , 𝐿 𝑄 Ω (𝑢)(s) ∀s ∈ Υ,   (6.5) B (𝑢)(s) = 𝐵 𝐿 𝑄 Ω +1 (𝑢)(s), . . . , 𝐿 𝑄 (𝑢)(s) ∀s ∈ 𝜕Υ. We briefly introduce a running example of a parametric PDE for which the above assumptions can be verified easily. 1 Note that we do not specifically ask for collocation points on 𝜕Ω × 𝜕Θ since we may not have boundary data on the 𝜽 parameter.
226 Example 6.2.1 (Nonlinear Darcy flow). Consider the nonlinear Darcy flow PDE (
−div𝒙 exp(𝑎(𝒙, 𝜽))∇𝑢 (𝒙) + 𝜏(𝑢(𝒙)) = 1, 𝒙 ∈ Ω, (6.6) 𝑢(𝒙) = 0, 𝒙 ∈ 𝜕Ω, where Ω ⊂ R𝑑 is a bounded domain with a Lipschitz boundary and 𝜏 : R → R is a continuous and nonlinear map. We assume that the permeability field is parameterized as 𝑝 ∑︁ 𝜃 𝑗 𝜓 𝑗 (𝒙), (6.7) 𝑎(𝒙, 𝜽) = 𝑗=1 where 𝜃 𝑗 ∈ (0, 1) so that Θ = (0, 1) 𝑝 and 𝜓 𝑗 ∈ 𝐶 (Ω). Substituting into the PDE and expanding the differential operator we can rewrite our nonlinear PDE as  𝑝 𝑝  ∑︁ ∑︁  © ª   − exp ­ 𝜃 𝑗 𝜓 𝑗 (𝒙) ® 𝜃 𝑗 ∇𝒙 𝜓 𝑗 (𝒙) · ∇𝒙 𝑢(𝒙; 𝜽)      « 𝑗=1 ¬ 𝑗=1    𝑝 ©∑︁ ª   − exp ­ 𝜃 𝑗 𝜓 𝑗 (𝒙) ® Δ𝒙 𝑢(𝒙; 𝜽) + 𝜏(𝑢(𝒙; 𝜽)) = 1, (𝒙, 𝜽) ∈ Ω × Θ,      « 𝑗=1 ¬     𝑢(𝒙; 𝜽) = 0, (𝒙, 𝜽) ∈ 𝜕Ω × Θ,  where we used subscripts on the differential operators to highlight that derivatives are computed for the 𝒙 variable only and not 𝜽. We also did not use the compact notation s ≡ (𝒙, 𝜽) since it is more helpful to be able to distinguish between the 𝒙 and 𝜽 variables in this example. We can directly verify Assumption 1 with the bounded and linear operators 𝐿 1 : 𝑢(𝒙; 𝜽) ↦→ 𝑢(𝒙; 𝜽), 𝑝 𝑝 ©∑︁ ª ∑︁ 𝐿 2 : 𝑢(𝒙; 𝜽) → ↦ exp ­ 𝜃 𝑗 𝜓 𝑗 (𝒙) ® 𝜃 𝑗 ∇𝒙 𝜓 𝑗 (𝒙) · ∇𝒙 𝑢(𝒙; 𝜽), 𝑗=1 𝑗=1 « ¬ 𝑝 ©∑︁ ª 𝐿 3 : 𝑢(𝒙; 𝜽) ↦→ exp ­ 𝜃 𝑗 𝜓 𝑗 (𝒙) ® Δ𝒙 𝑢(𝒙; 𝜽) « 𝑗=1 ¬ 𝐿 4 : 𝑢(𝒙; 𝜽) ↦→ 𝑢(𝒙; 𝜽). Note that operators 𝐿 1 , 𝐿 4 are the same here since the point values of 𝑢 appear in both the interior and boundary conditions. Thus we have 𝑄 Ω = 3 and 𝑄 = 4 and the maps 𝑃(𝑡1 , 𝑡2 , 𝑡3 ) = −𝑡2 − 𝑡3 + 𝜏(𝑡1 ), 𝐵(𝑡 1 ) = 𝑡 1 . 227 If U is sufficiently regular and Assumption 1 holds, then we can define the func𝑞 tionals 𝜙𝑚 ∈ U ∗ for 1 ≤ 𝑞 ≤ 𝑄 as ( 1 ≤ 𝑚 ≤ 𝑀Ω if 1 ≤ 𝑞 ≤ 𝑄Ω 𝑞 𝜙𝑚 := 𝛿 (s𝑚 ) ◦ 𝐿 𝑞 , where (6.8) 𝑀Ω + 1 ≤ 𝑚 ≤ 𝑀 if 𝑄 Ω+1 ≤ 𝑞 ≤ 𝑄. In what follows we write [𝜙, 𝑢] to denote the duality pairing between U and U ∗ and 𝑞 further use the shorthand notation 𝝓 (𝑞) to denote the vector of dual elements 𝜙𝑚 for a fixed index 𝑞. Note that 𝝓 (𝑞) ∈ (U ∗ ) ⊗𝑀Ω if 𝑞 ≤ 𝑄 Ω but 𝝓 (𝑞) ∈ (U ∗ ) ⊗(𝑀−𝑀Ω ) if 𝑞 > 𝑄 Ω in order to accommodate different differential operators defining the PDE and the boundary conditions. We further write 𝑁 = 𝑀Ω 𝑄 Ω + (𝑀 − 𝑀Ω )(𝑄 − 𝑄 Ω ) and define
𝝓 = 𝝓 (1) , . . . , 𝝓 (𝑄) ∈ (U ∗ ) ⊗𝑁 . Henceforth we write 𝜙𝑛 for 𝑛 = 1, . . . , 𝑁 to denote the entries of the vector 𝝓 and write [𝝓, 𝑢] = ([𝜙1 , 𝑢], . . . , [𝜙 𝑁 , 𝑢]) ∈ R𝑁 . With this notation we rewrite problem (6.4) as ( minimize ∥𝑢∥ U st 𝐹 ([𝝓, 𝑢]) = 𝒚, where the data vector 𝒚 ∈ R 𝑀 has entries ( 𝑓 (s𝑚 ), if 1 ≤ 𝑚 ≤ 𝑀Ω , 𝑦 𝑚 := 𝑔(s𝑚 ), if 𝑀Ω + 1 ≤ 𝑚 ≤ 𝑀, and 𝐹 : R𝑁 → R 𝑀 is a nonlinear map whose output components are defined as   𝑄Ω 1   𝑃 [𝜙 , 𝑢], . . . , [𝜙 , 𝑢] if 1 ≤ 𝑚 ≤ 𝑀Ω , 𝑚   𝑚
  𝐹 ([𝝓, 𝑢]) 𝑚 := (6.9) 𝑄  Ω +1  𝐵 [𝜙𝑄 , 𝑢], . . . , [𝜙𝑚 , 𝑢] if 𝑀Ω + 1 ≤ 𝑚 ≤ 𝑀. 𝑚  Further define the kernel vector field 𝐾 (·, 𝝓) : Υ → U 𝑁 , 𝐾 (s, 𝝓) 𝑘 := [𝜙 𝑘 , 𝐾 (s, ·)] (6.10) 𝐾 (𝝓, 𝝓)𝑛𝑘 := [𝜙𝑛 , 𝐾 (·, 𝝓) 𝑘 ]. (6.11) and the kernel matrix 𝐾 (𝝓, 𝝓) ∈ R𝑁×𝑁 , We can then characterize the minimizers of (6.4) via the following representer theorem which is a direct consequence of (Yifan Chen, Hosseini, et al., 2021b, Prop. 2.3): 228 Proposition 6.2.2. Suppose Assumption 1 holds and 𝐾 (𝝓, 𝝓) is invertible. Then a function 𝑢 † : Υ → R is a minimizer of (6.4) if and only if 𝑢 † (s) = 𝐾 (s, 𝝓)𝐾 (𝝓, 𝝓) −1 𝒛† , (6.12) where 𝒛† ∈ R𝑁 solves   𝒛𝑇 𝐾 (𝝓, 𝝓) −1 𝒛,  minimize  𝑁 𝒛∈R   st 𝐹 (𝒛) = 𝒚,  with the nonlinear map 𝐹 defined in (6.9). (6.13) This result allows us to reduce the infinite-dimensional optimization problem (6.4) to a finite-dimensional optimization problem without incurring any approximation errors; it is an instance of the well-known family of representer theorems (Scholkopf and Alexander J Smola, 2018, Sec. 4.2). Thus, to find an approximation to 𝑢★ we simply need to solve (6.13) and apply the formula (6.12); algorithms for this task are discussed next. Numerical Strategies We now summarize various numerical strategies for solution of (6.13). These strategies are naturally applicable to non-parametric PDEs as they can be viewed as a special case of (6.3) with a fixed parameter. In Section 6.2 we summarize a GaussNewton algorithm that was introduced in (Yifan Chen, Hosseini, et al., 2021b) followed by a new and, often, more efficient strategy that linearizes the PDE first before formulating the optimization problem in Section 6.2. Gauss-Newton To solve the optimization problem (6.13), a Gauss-Newton algorithm was proposed in (Yifan Chen, Hosseini, et al., 2021b) which we recall briefly. The equality constraints can be dealt with either by elimination or relaxation. Suppose that there exists a map 𝐹 : R𝑁−𝑀 × R 𝑀 → R𝑁 so that 𝐹 (𝒛) = 𝒚 if and only if 𝒛 = 𝐹 (𝒘, 𝒚), for a unique 𝒘 ∈ R𝑁−𝑀 . Then, we rewrite (6.13) as the unconstrained optimization problem minimize 𝒘∈R 𝑁 − 𝑀 𝐹 (𝒘, 𝒚)𝑇 𝐾 (𝝓, 𝝓) −1 𝐹 (𝒘, 𝒚) . (6.14) 229 Then a minimizer 𝒘 † of (6.14) can be approximated with a sequence of elements 𝒘 ℓ defined iteratively via 𝒘 ℓ+1 = 𝒘 ℓ + 𝛼ℓ 𝛿𝒘 ℓ , where 𝛼ℓ > 0 is an appropriate step size while 𝛿𝒘 ℓ is the minimizer of the optimization problem minimize 𝛿𝒘∈R 𝑁 −𝑀  𝐹 (𝒘 ℓ , 𝒚) + ∇𝒘 𝐹 (𝒘 ℓ , 𝒚)𝛿𝒘 𝑇   𝐾 (𝝓, 𝝓) −1 𝐹 (𝒘 ℓ , 𝒚) + ∇𝒘 𝐹 (𝒘 ℓ , 𝒚)𝛿𝒘 . Alternatively, if the map 𝐹 does not exist or is hard to compute, i.e., eliminating the constraints is not feasible, then we consider the relaxed problem minimize 𝒛∈R 𝑁 1 𝑇 1 𝒛 𝐾 (𝝓, 𝝓) −1 𝒛 + 2 |𝐹 (𝒛) − 𝒚| 2 , 2 2𝛽 for a sufficiently small parameter 𝛽 > 0. Here ∥ · ∥ is the 𝐿 2 norm of the vector. A minimizer 𝒛 †𝛽 of the above problem can be approximated with a sequence 𝒛ℓ where 𝒛ℓ+1 = 𝒛ℓ + 𝛼ℓ 𝛿𝒛ℓ where 𝛿𝒛ℓ is the minimizer of minimize 𝛿𝒛∈R 𝑁 𝛿𝒛𝑇 𝐾 (𝝓, 𝝓) −1 𝒛ℓ + 1 |𝐹 (𝒛ℓ ) + ∇𝐹 (𝒛ℓ )𝛿𝒛 − 𝒚| 2 . 2𝛽2 We summarize the proposed Gauss-Newton algorithm for solution of parametric PDEs in Algorithm 9. Algorithm 9 Kernel Methods for Parametric PDEs using Gauss-Newton (Section 6.2) Input: PDE of the form Equation (6.3) defined on Υ = Ω × Θ with boundary condition on 𝜕Υ = 𝜕Ω × Θ, 𝑀 ≥ 1 collocation points in Υ, and kernel 𝐾 : Υ × Υ → R Output: Approximation 𝑢 † (𝑠) to exact solution 𝑢 ∗ (𝑠) 1: 𝑁 ← 𝑀Ω 𝑄 Ω + (𝑀 − 𝑀Ω ) (𝑄 − 𝑄 Ω ) 2: for 𝑖 = 1 to 𝑁 {Build kernel matrix 𝐾 (𝝓, 𝝓)} do 3: for 𝑗 = 1 to 𝑁 do 4: 𝐾 (𝝓, 𝝓)𝑖, 𝑗 ← [𝜙𝑖 , 𝐾 (·, 𝝓) 𝑗 ] 5: end for 6: end for 7: while not converged do  𝑇 8: 𝛿𝒘 ℓ ← arg min 𝛿𝒘 𝐹 (𝒘 ℓ , 𝒚) + ∇𝒘 𝐹 (𝒘 ℓ , 𝒚)𝛿𝒘   9: · 𝐾 (𝝓, 𝝓) −1 𝐹 (𝒘 ℓ , 𝒚) + ∇𝒘 𝐹 (𝒘 ℓ , 𝒚)𝛿𝒘 10: 𝒘 ℓ+1 ← 𝒘 ℓ + 𝛼ℓ 𝛿𝒘 ℓ 11: end while 12: 𝒛 † ← 𝐹 (𝒘, 𝒚) 13: 𝑢 † (s) ← 𝐾 (s, 𝝓)𝐾 (𝝓, 𝝓) −1 𝒛 † {Apply representer formula} 230 Linearize then Optimize The Gauss-Newton approach of Section 6.2 is applicable to wide families of nonlinear PDEs. The primary computational bottleneck of that approach is the construction and factorization of the kernel matrix 𝐾 (𝝓, 𝝓) which for some PDEs can be prohibitively large. To get around this difficulty we propose an alternative approach to approximating the solution of (6.4) by first linearizing the PDE operators before applying Proposition 6.2.2. The resulting approach is more intrusive in comparison to the Gauss-Newton method as it requires explicit calculations involving the PDE but often leads to smaller kernel matrices and better performance. This method can also be viewed as applying the methodology of (Yifan Chen, Hosseini, et al., 2021b; Giesl and Wendland, 2007) to discretize successive linearizations of the PDE. Let 𝑢 † denote the minimizer of (6.4) as before. Assuming that the operators P and B are Fréchet differentiable we then approximate 𝑢 † with a sequence of elements 𝑢 ℓ obtained by solving the problem ∥𝑢∥ U   P (𝑢 ℓ−1 ) + P ′ (𝑢 ℓ−1 )(𝑢 − 𝑢 ℓ−1 ) | s𝑚 = 𝑓 (s𝑚 ), 𝑚 = 1, . . . , 𝑀Ω ,   ℓ−1 ′ ℓ−1 ℓ−1 B (𝑢 ) + B (𝑢 )(𝑢 − 𝑢 ) | s𝑚 = 𝑔(s𝑚 ), 𝑚 = 𝑀Ω + 1, . . . , 𝑀, (6.15) ′ ′ where P and B are the Fréchet derivatives of P and B.   minimize   𝑢∈U     st        Let us further suppose that Assumption 1 holds. Observing that the constraints in (6.15) are linear in 𝑢 we obtain an explicit formula for 𝑢 ℓ by (Yifan Chen, Hosseini, et al., 2021b, Prop. 2.2): 𝑢 ℓ (s) = 𝐾 (s, 𝝓eℓ−1 )𝐾 ( 𝝓eℓ−1 , 𝝓eℓ−1 ) −1 𝒛ℓ−1 , ℓ−1 𝑇 where 𝒛ℓ−1 = (𝑧ℓ−1 1 , . . . , 𝑧 𝑀 ) has entries     𝑓 − P (𝑢 ℓ−1 ) + P ′ (𝑢 ℓ−1 )𝑢 ℓ−1 | s𝑚 , if 1 ≤ 𝑚 ≤ 𝑀Ω ,    𝑧ℓ−1 𝑚 =   𝑓 − P (𝑢 ℓ−1 ) + B ′ (𝑢 ℓ−1 )𝑢 ℓ−1 | s𝑚 , if 𝑀Ω + 1 ≤ 𝑚 ≤ 𝑀.  (6.16) (6.17) The vectors 𝝓eℓ−1 ∈ (U ∗ ) ⊗𝑀 are obtained by concatenating the dual elements ( 𝛿 (s𝑚 ) ◦ P ′ (𝑢 ℓ−1 ), if 1 ≤ 𝑚 ≤ 𝑀Ω , ℓ−1 e (6.18) 𝜙𝑚 := 𝛿 (s𝑚 ) ◦ B ′ (𝑢 ℓ−1 ), if 𝑀Ω + 1 ≤ 𝑚 ≤ 𝑀. We note that the above scheme implicitly assumes that U is sufficiently regular so that the derivatives P ′ (𝑢 ℓ−1 ) and B ′ (𝑢 ℓ−1 ) can be regarded as linear operators mapping 231 U to 𝐶 (Υ) where pointwise evaluation is well-defined. The most important feature of the linearize-then-optimize approach is that the kernel matrices 𝐾 ( 𝝓eℓ−1 , 𝝓eℓ−1 ) are of size 𝑀 × 𝑀 while the kernel matrix 𝐾 (𝝓, 𝝓), used in the Gauss-Newton approach of Section 6.2, is of size 𝑁 × 𝑁. Note that 𝑁/𝑀 ≈ 𝑄 Ω and for instance, in Example 6.2.1, 𝑄 Ω = 3 so there is an approximately 3× reduction in the kernel matrix size. Thus, the linearize-then-optimize approach requires the inversion of much smaller kernel matrices at each iteration but these matrices need to be updated successively since the 𝝓eℓ depend on the previous solution 𝑢 ℓ−1 . In the case where Assumption 1 holds and 𝑃, 𝐵 are differentiable, then the P ′ (𝑢) and B ′ (𝑢) operators can be written explicitly as    𝑇  𝐿 1 (𝑢)  . P ′ (𝑢 ℓ−1 ) : 𝑢 ↦→ ∇𝑃 𝐿 1 (𝑢 ℓ−1 ), . . . , 𝐿 𝑄 Ω (𝑢 ℓ−1 )  ..  ,    𝐿 𝑄 Ω (𝑢)    𝐿    𝑇  𝑄 Ω +1 (𝑢)  .. . B ′ (𝑢 ℓ−1 ) : 𝑢 → ↦ ∇𝐵 𝐿 𝑄 Ω +1 (𝑢 ℓ−1 ), . . . , 𝐿 𝑄 (𝑢 ℓ−1 )  .     𝐿 𝑄 (𝑢)     We note that while the “linearize then optimize” approach can reduce the size of kernel matrices by a constant factor, which is significant in practice, the size still scales with the number of collocation points. For very large-scale problems requiring many collocation points, we can further employ fast algorithms for these kernel matrices; see for example (Yifan Chen, Houman Owhadi, and Florian Schäfer, 2024). Remark 9. The “linearize then optimize” approach performs linearization first at the continuous level, while the Gauss-Newton iteration linearizes at the discrete level after applying the representor theorem and transforming the optimization problem into an unconstrained form, either through elimination or relaxation. The “linearize then optimize” approach and the Gauss-Newton iteration are mathematically equivalent if the latter is implemented using elimination with a specific choice of 𝐹. This equivalence is demonstrated in (Yifan Chen, Houman Owhadi, and Florian Schäfer, 2024, Sec. 5.1) for a nonlinear elliptic example, where the algorithm is also shown to be equivalent to a sequential quadratic programming approach for solving (6.13). In general, these approaches may differ in how the nonlinear operators P, B, or the nonlinear map 𝐹 are represented 232 Finally, we summarize our linearize-then-optimize approach as a pseudo-algorithm within Algorithm 10. Algorithm 10 The linearize-then-optimize approach to parametric PDEs Input: PDE of the form Equation (6.3) defined on Υ = Ω × Θ with boundary condition on 𝜕Υ = 𝜕Ω × Θ, 𝑀 ≥ 1 collocation points in Υ, and kernel 𝐾 : Υ × Υ → R Output: Approximation 𝑢 † (𝑠) to exact solution 𝑢 ∗ (𝑠) 1: while not converged {Iteratively approximate 𝑢 † } do 2: For 𝑚 = 1, . . . , 𝑀:   ℓ −1 ′ ℓ −1 ℓ −1   𝑓 − P (𝑢 ) + P (𝑢 )𝑢 (s𝑚 ), if 𝑚 ≤ 𝑀Ω    3: 𝑧ℓ𝑚−1 ←  {Build 𝒛ℓ −1 }  ℓ −1 ′ ℓ −1 ℓ −1  (s𝑚 ), if 𝑚 > 𝑀Ω  𝑓 − P (𝑢 ) + B (𝑢 )𝑢  4: For 𝑚 = 1, .(. . , 𝑀: 𝛿 (s𝑚 ) ◦ P ′ (𝑢 ℓ −1 ), if 𝑚 ≤ 𝑀Ω {Build 𝝓eℓ −1 } 5: 𝜙eℓ𝑚−1 ← 𝛿 (s𝑚 ) ◦ B ′ (𝑢 ℓ −1 ), if 𝑚 > 𝑀Ω 6: 𝑢 ℓ (s) ← 𝐾 (s, 𝝓eℓ −1 )𝐾 ( 𝝓eℓ −1 , 𝝓eℓ −1 ) −1 𝒛ℓ −1 7: end while 6.3 Error Analyses We now present our main theoretical results concerning convergence rates for the minimizers 𝑢 † of (6.4) to the respective true solutions 𝑢★. We start in Section 6.3 by articulating the abstract framework, main theorem, and proof. We then consider the simple setting of a nonlinear PDE in Section 6.3 where the RKHS U already satisfies the boundary conditions of the PDE to convey the main ideas of the proof in a simple setting. Non-trivial boundary conditions are then considered in Section 6.3 followed by the case of parametric PDEs in Section 6.3. Our proof technique is a generalization of the results of (Giesl and Wendland, 2007; Schaback, 2016) to the case of nonlinear and parametric PDEs that are Lipschitz stable and well-posed. An Abstract Framework for Obtaining Convergence Rates We present here an abstract theoretical result that allows us to obtain convergence rates for nonlinear operator equations. Our error analyses concerning the numerical solutions 𝑢 † and the true solution to the PDE 𝑢★ then follow as applications of this abstract result. Our main result here can also be viewed as a generalization of the results of (Schaback, 2016, Sec. 10), which focused on linear operators, to the nonlinear case. Let us consider operator equations of the form T (𝑣★) = 𝑤★, (6.19) 233 where 𝑣★, 𝑤★ are elements of appropriate Banach spaces and T is a nonlinear map. In the setting of PDEs the map T is defined by the differential operator of the PDE, 𝑣★ coincides with the solution and 𝑤★ is the source/boundary data. Broadly speaking our goal is to approximate the solution 𝑣★ under assumptions on its regularity and the stability properties of the map T . To this end, we present a general result that allows us to control the error of approximating 𝑣★ given an appropriate candidate 𝑣 † . Henceforth we write 𝐵𝑟 (𝑉) to denote the ball of radius 𝑟 centered at zero in a Banach space 𝑉. 4 as well as (U, ∥·∥ ). Theorem 6.3.1. Consider abstract Banach spaces (𝑉𝑖 , ∥·∥ 𝑖 )𝑖=1 U Suppose the following conditions are satisfied for any choice of 𝑟 > 0 (all the appeared constants 𝐶 (𝑟) are non-decreasing regarding 𝑟): (A1) For any pair 𝑣, 𝑣′ ∈ 𝐵𝑟 (𝑉1 ) there exists a constant 𝐶 = 𝐶 (𝑟) > 0 so that ∥𝑣 − 𝑣 ′ ∥ 1 ≤ 𝐶 ∥T (𝑣) − T (𝑣 ′)∥ 2 . (6.20) (A2) For any pair 𝑣, 𝑣′ ∈ 𝐵𝑟 (𝑉4 ) there exists a constant 𝐶 = 𝐶 (𝑟) > 0 so that ∥T (𝑣) − T (𝑣 ′)∥ 3 ≤ 𝐶 ∥𝑣 − 𝑣 ′ ∥ 4 . (6.21) (A3) For any 𝑣 ∈ 𝑉4 , there exists a constant 𝐶 > 0 so that ∥𝑣∥ 4 ≤ 𝐶 ∥𝑣∥ U . e ⊂ 𝑉2 ∩ 𝑉3 and a constant 𝜀 > 0, so that for all 𝑤, 𝑤′ ∈ 𝑉 e (A4) There exists a set 𝑉 it holds that ∥𝑤 − 𝑤 ′ ∥ 2 ≤ 𝜀∥𝑤 − 𝑤 ′ ∥ 3 . (6.22) Suppose problem (6.19) is uniquely solvable with 𝑣★ ∈ U and let 𝑣 † ∈ U be any other function such that e (A5) T (𝑣★), T (𝑣 † ) ∈ 𝑉. 234 (A6) There exists a constant 𝐶 > 0, independent of 𝑣★ and 𝑣 † , so that ∥𝑣 † ∥ U ≤ 𝐶 ∥𝑣★ ∥ U . (6.23) Then there exists a constant 𝐶 > 0, depending only on ∥𝑣★ ∥ U , such that ∥𝑣 † − 𝑣★ ∥ 1 ≤ 𝐶𝜀∥𝑣★ ∥ U . Proof. By (A1) we have that ∥𝑣 † − 𝑣★ ∥ 1 ≤ 𝐶 ∥T (𝑣 † ) − T (𝑣★)∥ 2 . (6.24) Then (A4) and (A5) imply that ∥T (𝑣 † ) − T (𝑣★)∥ 2 ≤ 𝐶𝜀∥T (𝑣 † ) − T (𝑣★)∥ 3 . By the triangle inequality we have ∥T (𝑣 † ) − T (𝑣★)∥ 3 ≤ ∥T (𝑣 † ) − T (0)∥ 3 + ∥T (𝑣★) − T (0)∥ 3 . Using (A2), (A3), and (A6) in that order, we get ∥T (𝑣 † ) − T (0)∥ 3 ≤ 𝐶 ∥𝑣 † ∥ 4 ≤ 𝐶 ∥𝑣 † ∥ U ≤ 𝐶 ∥𝑣★ ∥ U . Similarly, we have ∥T (𝑣★) − T (0)∥ 3 ≤ 𝐶 ∥𝑣★ ∥ U . Combining these bounds we obtain ∥T (𝑣 † ) − T (𝑣★)∥ 3 ≤ 𝐶𝜀∥𝑣★ ∥ U which yields the desired result due to (6.24). □ Let us provide some remarks regarding the assumptions of the theorem. In our PDE examples we often take the 𝑉𝑖 spaces to be Sobolev spaces of appropriate smoothness while U is taken as an RKHS that is sufficiently smooth and so 𝑣★ ∈ U amounts to an assumption on the regularity of the true solution to the problem. Conditions (A1) and (A2) amount to forward and inverse Lipschitz stability of the operator T while (A4) is often given by a sampling/Poincaré-type inequality for our numerical method. We treat the constant 𝜀 separately from the other constants in the theorem since in practice 𝜀 often coincides with some power of the resolution (filldistance/meshnorm) of our numerical scheme, constituting the rate of convergence of the method. Assumption (A3) also concerns the regularity of the RKHS and the choice of the space 𝑉4 (we simply ask for U to be continuously embedded in 𝑉4 ) and is a matter of the setup of the problem. Condition (A6) is less natural as it requires the norm of the approximate solution 𝑣 † to be controlled by the norm of 𝑣★. While this condition does not hold for many numerical approximation schemes, we will see that it follows easily from the setup of our collocation/optimal recovery scheme. In plain words, the most important message of Theorem 6.3.1 is that: given Condition (6.23) and the Lipschitz-continuity of T and its inverse, it follows that the 235 approximation error between 𝑣 † and 𝑣★ is bounded by the approximation error between T (𝑣 † ) and T (𝑣★). This result can be applied to both GP/kernel and ANN based collocation methods, since both seek to minimize the error between T (𝑣 † ) and T (𝑣★) at collocation points. This Condition (6.23) is automatically satisfied for our GP/Kernel based methods that solve problems of the form 𝑣 † = argmin∥𝑣∥ U s.t. [𝜙𝑖 , T (𝑣)] = [𝜙𝑖 , T (𝑣★)], 𝑖 = 1, . . . , 𝑀, with T denoting the differential operator of a PDE and 𝜙𝑖 denoting a set of dual elements (e.g. pointwise evaluations at collocation points). Then since the true solution 𝑣★ satisfies the PDE for an infinite collection of dual elements (e.g. pointwise within a set, or in a weak sense) then we immediately have that ∥𝑣 † ∥ U ≤ ∥𝑣★ ∥ U . One can also take U to be a Barron space (indeed the 𝑉𝑖 norms could be arbitrary) to obtain an analogous result for ANNs, but it is unclear if this setup coincides with (or leads to) any practical algorithms. The Case of Second Order Nonlinear PDEs We begin our error analysis in the case where (6.3) does not depend on the parameter 𝜽 and homogeneous Dirichlet boundary conditions are imposed, i.e., nonlinear second order PDEs of the form ( P (𝑢★)(𝒙) = 𝑓 (𝒙), 𝒙 ∈ Ω, (6.25) 𝑢★ (𝒙) = 0, 𝒙 ∈ 𝜕Ω. The choice of Dirichlet boundary conditions is only made for simplicity here and can be replaced with other conditions of interest. We will also consider approximate boundary conditions in Section 6.3. We further assume that the kernel 𝐾 is chosen so that the elements of U readily satisfy the boundary conditions of the PDE and consider optimization problems of the form   minimize ∥𝑢∥ U     𝑢∈U  st P (𝑢)(𝒙 𝑚 ) = 𝑓 (𝒙 𝑚 ), 𝑚 = 1, . . . , 𝑀Ω ,      𝑢(𝒙) = 0, 𝒙 ∈ 𝜕Ω,  (6.26) 𝑀Ω where 𝑋Ω := {𝒙 𝑚 } 𝑚=1 ⊂ Ω are a set of collocation points. We need to impose appropriate assumptions on the RKHS U, the domain Ω and the PDE operator P. Assumption 2. The following conditions hold: 236 (B1) (Regularity of the domain) Ω ⊂ R𝑑 is a compact set with a Lipschitz boundary. (B2) (Stability of P) There exist indices 𝛾 > 0 and 𝑘 ∈ N satisfying 𝑑/2 < 𝑘 + 𝛾 and 𝑠 ≥ 1, ℓ ∈ R, so that for any 𝑟 > 0 it holds that ∥𝑢 1 − 𝑢 2 ∥ 𝐻 ℓ (Ω) ≤ 𝐶 ∥P (𝑢 1 ) − P (𝑢 2 )∥ 𝐻 𝑘 (Ω) , ∀𝑢 1 , 𝑢 2 ∈ 𝐵𝑟 (𝐻 ℓ (Ω) ∩ 𝐻01 (Ω)) (6.27) ∥P (𝑢 1 ) − P (𝑢 2 )∥ 𝐻 𝑘+𝛾 (Ω) ≤ 𝐶 ∥𝑢 1 − 𝑢 2 ∥ 𝐻 𝑠 (Ω) , ∀𝑢 1 , 𝑢 2 ∈ 𝐵𝑟 (𝐻 𝑠 (Ω) ∩ 𝐻01 (Ω)) (6.28) where 𝐶 = 𝐶 (𝑟) > 0 is independent of the 𝑢𝑖 ’s. The space 𝐻 𝑠 (Ω) ∩ 𝐻01 (Ω) can be equipped with the norm ∥ · ∥ 𝐻 𝑠 (Ω) , which is used to define the balls above. (B3) U is continuously embedded in 𝐻 𝑠 (Ω) ∩ 𝐻01 (Ω). Item (B1) is standard while (B3) dictates the choice of the RKHS U, and in turn the kernel, which should be made based on a priori knowledge about regularity of the strong solution 𝑢★. We highlight that, asking elements of U to satisfy the boundary conditions is only practical for simple domains and boundary conditions such as periodic, Dirichlet, or Neumann conditions on hypercubes or spheres. Assumption (B2) on the other hand is a question in the analysis of nonlinear PDEs and is independent of our numerical scheme; simply put we require the PDE to be Lipschitz well-posed with respect to the right hand side/source term. We are now ready to present our first theoretical result characterizing the convergence of the minimizer 𝑢 † of (6.26) to 𝑢★ the strong solution of (6.25). Theorem 6.3.2. Suppose Assumption 2 is satisfied and let 𝑢★ ∈ U denote the unique strong solution of (6.25). Let 𝑢 † be a minimizer of (6.26) with a set of collocation points 𝑋Ω ⊂ Ω and define their fill-distance ℎΩ := sup inf |𝒙 ′ − 𝒙|. 𝒙 ′ ∈Ω 𝒙∈𝑋Ω Then there exists a constant ℎ0 > 0 so that if ℎΩ < ℎ0 then ∥𝑢 † − 𝑢★ ∥ 𝐻 ℓ (Ω) ≤ 𝐶ℎΩ ∥𝑢★ ∥ U , 𝛾 where the constant 𝐶 > 0 is independent of 𝑢 † , and ℎΩ . 237 Proof. We will obtain the result by applying Theorem 6.3.1 with the map T ≡ P and the spaces 𝑉1 ≡ 𝐻 ℓ (Ω), 𝑉2 ≡ 𝐻 𝑘 (Ω), 𝑉3 ≡ 𝐻 𝑘+𝛾 (Ω), and 𝑉4 ≡ 𝐻 𝑠 (Ω). With this setup we proceed to verify the conditions of Theorem 6.3.1: Condition (A1) follows from (6.27), (A2) follows from (6.28), and (A3) follows from (B3). Condition (A6) holds since 𝑢 † is a minimizer of (6.26) and so ∥𝑢 † ∥ U ≤ ∥𝑢★ ∥ U , since 𝑢★ is feasible but satisfies additional constraints compared with 𝑢 † , i.e., it solves the PDE over the entire set Ω. Thus, (6.23) is verified with constant 𝐶 = 1. It remains to verify (A4): Let 𝑓¯ = P (𝑢 † ) − P (𝑢★) and observe that 𝑓¯(𝒙) = 0 for all 𝒙 ∈ 𝑋Ω . Thus 𝑓¯ ∈ 𝐻 𝑘+𝛾 (Ω) is zero on 𝑋Ω and an application of Proposition 6.6.1 yields the existence of a constant ℎ0 > 0 so that whenever ℎΩ < ℎ0 then ∥ 𝑓¯∥ 𝐻 𝑘 (Ω) ≤ 𝛾 𝛾 𝐶ℎΩ ∥ 𝑓¯∥ 𝐻 𝑘+𝛾 (Ω) . This verifies (6.22) with 𝜀 ≡ 𝐶ℎΩ . □ Remark 10. We note that Item (B2) and in turn Theorem 6.3.2 can easily be modified to a local version where the stability estimates (6.28) and (6.28) are stated for 𝑢 1 , 𝑢 2 belonging to a ball of radius 𝑟 > 0 around the true solution 𝑢★. Then one can obtain an asymptotic rate for ∥𝑢 † − 𝑢★ ∥ 𝐻 ℓ (Ω) under the additional assumption that 𝑢 † is sufficiently close to 𝑢★. Remark 11. The assumptions and results of Assumption 2 are analogous to the one used to obtain error estimates in numerical homogenization for elliptic PDEs (Houman Owhadi and Clint Scovel, 2019b). In particular Theorem 6.3.2 can be extended to the setting where measurements on the PDE are not pointwise but involve integral operators and where the coefficients may be rough. We now present a brief example where Assumption 2 can be verified and so Theorem 6.3.2 is applicable to obtain convergence rates for our GP/kernel collocation solver. Example 6.3.3 (Nonlinear Darcy flow continued). Let us consider the nonlinear Darcy flow PDE (6.6) and assume that Ω has a smooth boundary and 𝑎(𝒙) ∈ 𝐶 ∞ (Ω) is fixed and satisfies 𝑎(𝒙) ≥ 1. Further suppose 𝜏(𝑧) = 1 + tanh(𝛽𝑧) for a fixed constant 𝛽 > 0 to be determined. Now pick 𝑘 = ⌈𝑑/2 + 𝛼⌉ from which it follows that 𝐻 𝑘 (Ω) is continuously embedded in 𝐶 𝛼 ( Ω̄) (Gilbarg and Trudinger, 1977, Thm. 7.26) and fix an integer 𝛾 > 02 . It is then straightforward to verify (6.28) with 𝑠 = 𝑘 + 𝛾 + 2 as well as Assumption 2(iii) by choosing the kernel 𝐾 to be the 2 We only assume the exponents are integers for simplicity but our arguments can be generalized to the case of non-integer indices 238 Green’s function of the operator (−Δ) 𝑠 on the domain Ω, subject to homogeneous Dirichlet boundary conditions. We note that approximating this Green’s function can be expensive in practice and in Section 6.3 we propose a way around this step by using collocation points on the boundary of Ω to impose the boundary conditions. Furthermore our assumptions on 𝑎 and 𝜏 imply that P is uniformly elliptic in Ω̄ (see (Gilbarg and Trudinger, 1977, Part II) for definition of ellipticity for nonlinear elliptic PDEs). Since 𝜏 is smooth it follows from (Gilbarg and Trudinger, 1977, Thm. 13.8) that, for any 𝛼 ∈ (0, 1) and 𝑓 ∈ 𝐶 𝛼 ( Ω̄), the PDE (
−div exp(𝑎)∇𝑢 + 𝜏(𝑢) = 𝑓 , 𝒙 ∈ Ω, (6.29) 𝑢 = 0, 𝒙 ∈ 𝜕Ω, has a solution 𝑢 ∈ 𝐶 2 ( Ω̄). Now pick 𝑓1 , 𝑓2 ∈ 𝐻 𝑘 (Ω) which, by the aforementioned Sobolev embedding result, belong to 𝐶 𝛼 ( Ω̄). Write 𝑢 1 , 𝑢 2 ∈ 𝐶 2 ( Ω̄) for the solution of the PDE with both right hand sides and observe that the difference 𝑤 := 𝑢 1 − 𝑢 2 solves the PDE
−div exp(𝑎)∇𝑤 = 𝑓1 − 𝑓2 + 𝜏(𝑢 2 ) − 𝜏(𝑢 1 ). Standard stability results for linear elliptic PDEs then imply the bound   ∥𝑤∥ 𝐻 2 (Ω) ≤ 𝐵 ∥ 𝑓1 − 𝑓2 ∥ 𝐿 2 (Ω) + ∥𝜏(𝑢 2 ) − 𝜏(𝑢 1 )∥ 𝐿 2 (Ω) , for a constant 𝐵 > 0 independent of 𝑤, 𝑓1 , 𝑓2 , 𝑢 1 , 𝑢 2 . Since 𝜏 is globally 𝛽-Lipschitz we infer that ∥𝜏(𝑢 1 ) − 𝜏(𝑢 2 )∥ 𝐿 2 (Ω) ≤ 𝛽∥𝑤∥ 𝐿 2 (Ω) which, together with the subsequent bound, yields 𝐵 ∥ 𝑓1 − 𝑓2 ∥ 𝐿 2 (Ω) . ∥𝑤∥ 𝐻 2 (Ω) ≤ 1 − 𝐵𝛽 Thus, assumption (6.27) is satisfied with ℓ = 2 as long as 𝛽𝐵 < 1. Handling Boundary Conditions We now turn our attention to the case where (6.3) is still independent of the 𝜽 parameter but involves non-trivial boundary conditions, i.e., ( P (𝑢★)(𝒙) = 𝑓 (𝒙), 𝒙 ∈ Ω, (6.30) B (𝑢★)(𝒙) = 𝑔(𝒙), 𝒙 ∈ 𝜕Ω. We will further assume that the elements of U do not satisfy the boundary conditions exactly and so boundary collocation points are utilized to approximate those 239 conditions leading to the problem   minimize     𝑢∈U  st       ∥𝑢∥ U P (𝑢)(𝒙 𝑚 ) = 𝑓 (𝒙 𝑚 ), 𝑚 = 1, . . . , 𝑀Ω , (6.31) B (𝑢)(𝒙 𝑚 ) = 𝑔(𝒙 𝑚 ), 𝑚 = 𝑀Ω + 1, . . . , 𝑀, 𝑀Ω where 𝑋Ω := {𝒙 𝑚 } 𝑚=1 ⊂ Ω are the interior collocation points as before and 𝑀 𝑋𝜕Ω := {𝒙 𝑚 } 𝑚=𝑀Ω +1 ⊂ 𝜕Ω are the boundary collocation points. We will state our assumptions and results for PDEs in 𝑑 > 1 dimensions since in the 1D case we can, in principle, impose the boundary conditions exactly by placing some collocation points on boundary. The main difference, in comparison to Theorem 6.3.2, is that here we need to impose new assumptions on the PDE operators P and B and the boundary of Ω to be able to use Proposition 6.6.1 (sampling inequality on manifolds) in the final step of the proof to obtain approximation rates for the boundary data. Assumption 3. The following conditions hold: (C1) (Regularity of the domain and its boundary) Ω ⊂ R𝑑 with 𝑑 > 1 is a compact set and 𝜕Ω is a smooth connected Riemannian manifold of dimension 𝑑 − 1 endowed with a geodesic distance 𝜌 𝜕Ω . (C2) (Stability of the PDE) There exist 𝛾 > 0 and 𝑘, 𝑡 ∈ N satisfying 𝑑/2 < 𝑘 + 𝛾 and (𝑑 − 1)/2 < 𝑡 + 𝛾, and 𝑠, ℓ ∈ R, so that for any 𝑟 > 0 it holds that ∥𝑢 1 − 𝑢 2 ∥ 𝐻 ℓ (Ω) ≤𝐶 ∥P (𝑢 1 ) − P (𝑢 2 )∥ 𝐻 𝑘 (Ω) + ∥B (𝑢 1 ) − B (𝑢 2 )∥ 𝐻 𝑡 (𝜕Ω)
∀𝑢 1 , 𝑢 2 ∈ 𝐵𝑟 (𝐻 ℓ (Ω)), (6.32) ∥P (𝑢 1 ) − P (𝑢 2 )∥ 𝐻 𝑘+𝛾 (Ω) + ∥B (𝑢 1 ) − B (𝑢 2 )∥ 𝐻 𝑡+𝛾 (𝜕Ω) ∀𝑢 1 , 𝑢 2 ∈ 𝐵𝑟 (𝐻 𝑠 (Ω)), (6.33) where 𝐶 = 𝐶 (𝑟) > 0 is a constant independent of the 𝑢𝑖 . ≤ 𝐶 ∥𝑢 1 − 𝑢 2 ∥ 𝐻 𝑠 (Ω) , (C3) U is continuously embedded in 𝐻 𝑠 (Ω). Observe that the above assumptions are analogous to Assumption 2 with the exception that we no longer work with the restricted Sobolev spaces 𝐻0𝑘 since we do not need to impose the boundary conditions. However, we need to state our stability results for both P and B. We emphasize that the verification of condition (C2) remains 240 a question in the analysis of PDEs. We are now ready to extend Theorem 6.3.2 to the case of non-trivial boundary conditions. Theorem 6.3.4. Suppose Assumption 3 is satisfied and let 𝑢★ ∈ U denote the unique strong solution of (6.25). Let 𝑢 † be a minimizer of (6.26) with a set of collocation points 𝑋 ⊂ Ω where 𝑋Ω ⊂ 𝑋 denotes the collocation points in the interior of Ω and 𝑋𝜕Ω denotes the collocation points on the boundary 𝜕Ω. Define the fill-distances ℎΩ := sup inf |𝒙 ′ − 𝒙|, 𝒙 ′ ∈Ω 𝒙∈𝑋Ω ℎ 𝜕Ω := sup inf 𝜌 𝜕Ω (𝒙 ′, 𝒙), 𝒙 ′ ∈𝜕Ω 𝒙∈𝑋𝜕Ω where 𝜌 𝜕Ω : 𝜕Ω×𝜕Ω → R+ is the geodesic distance defined on 𝜕Ω (see Section 6.6), and set ℎ¯ := max{ℎΩ , ℎ 𝜕Ω }. Then there exists a constant ℎ0 > 0 so that if ℎ¯ < ℎ0 then ∥𝑢 † − 𝑢★ ∥ 𝐻 𝑠 (Ω) ≤ 𝐶 ℎ¯ 𝛾 ∥𝑢★ ∥ U , ¯ where 𝐶 > 0 is independent of 𝑢 † and ℎ. Proof. The proof follows an identical approach to Theorem 6.3.2 and applies Theo
rem 6.3.1 with the appropriate setup. We take the operator T : 𝑢 ↦→ P (𝑢), B (𝑢) . We then choose the spaces 𝑉1 ≡ 𝐻 ℓ (Ω), 𝑉2 ≡ 𝐻 𝑘 (Ω) × 𝐻 𝑡 (𝜕Ω), 𝑉3 ≡ 𝐻 𝑘+𝛾 (Ω) × 𝐻 𝑡+𝛾 (Ω), and 𝑉4 ≡ 𝐻 𝑠 (Ω) where we equip 𝑉2 with the norm ∥( 𝑓 , 𝑔)∥ 2 := ∥ 𝑓 ∥ 𝐻 𝑘 (Ω) + ∥𝑔∥ 𝐻 𝑡 (𝜕Ω) and similarly for 𝑉3 with the 𝐻 𝑘 (Ω) and 𝐻 𝑡 (𝜕Ω) norms replaced by 𝐻 𝑘+𝛾 (Ω) and 𝐻 𝑡+𝛾 (𝜕Ω) norms. Analogously to the proof of Theorem 6.3.2, we can verify Conditions (A1), (A2), and (A3) by the hypothesis of the theorem. Condition (A6) is also satisfied since 𝑢 † is a minimizer of (6.31) and so ∥𝑢 † ∥ U ≤ ∥𝑢★ ∥ U as 𝑢 † satisfies more relaxed constraints. It remains for us to verify (A4). Repeating the same argument as in the proof of Theorem 6.3.2, in the interior of Ω, yields the bound ∥P (𝑢 † ) − P (𝑢★)∥ 𝐻 𝑘 (Ω) ≤ 𝐶ℎΩ ∥P (𝑢 † ) − P (𝑢★)∥ 𝐻 𝑘+𝛾 (Ω) , 𝛾 (6.34) whenever ℎΩ < ℎ1 and ℎ1 is a sufficiently small constant that is independent of 𝑢 † and 𝑢★. Let 𝑔¯ = B (𝑢 † )−B (𝑢★) which satisfies 𝑔(𝒙) ¯ = 0 for all 𝒙 ∈ 𝑋𝜕Ω and so 𝑔¯ ∈ 𝐻 𝑡+𝛾 (𝜕Ω) is zero on the set 𝑋𝜕Ω . Then Proposition 6.6.1 implies the existence of a constant ℎ2 > 0 so that whenever ℎ 𝜕Ω < ℎ2 we have 𝛾 ∥ 𝑔∥ ¯ 𝐻 𝑡 (𝜕Ω) ≤ 𝐶ℎ 𝜕Ω ∥ 𝑔∥ ¯ 𝐻 𝑡+𝛾 (𝜕Ω) . 241 Now take ℎ0 = min{ℎ1 , ℎ2 } and combine the above bound with (6.34), and substitute the definition of 𝑔¯ to get ∥P (𝑢 † ) − P (𝑢★)∥ 𝐻 𝑘 (Ω) + ∥B (𝑢 † ) − B (𝑢★)∥ 𝐻 𝑡 (𝜕Ω)
≤ 𝐶 ℎ¯ 𝛾 ∥P (𝑢 † ) − P (𝑢★)∥ 𝐻 𝑘+𝛾 (Ω) + ∥B (𝑢 † ) − B (𝑢★)∥ 𝐻 𝑡+𝛾 (𝜕Ω) , whenever ℎ¯ < ℎ0 . This verifies (A4) with 𝜀 ≡ 𝐶 ℎ¯ 𝛾 . □ Remark 12. We highlight that our statement of Theorem 6.3.4 can easily be extended to PDEs with mixed boundary conditions simply by modifying the norm that is chosen on the boundary, i.e., the spaces 𝑉2 and 𝑉4 , so long as we can prove the requisite stability estimates in condition (C2). In particular, this idea will allow us to obtain errors for time-dependent PDEs, cast as a static PDE in a space-time domain Ω with the initial and boundary conditions imposed as mixed conditions on 𝜕Ω. In fact, in the case of time-dependent PDEs we do not need to impose the boundary conditions on all of 𝜕Ω but only on a subset. We now return to our running example to verify Assumption 3 for the Darcy flow PDE. Example 6.3.5 (Nonlinear Darcy flow continued). Consider the PDE (6.29) but this time with the boundary condition 𝑢 = 𝑔 on 𝜕Ω for a function 𝑔 ∈ 𝐻 𝑡+𝛾 (𝜕Ω) with 𝑡 > min{3/2, (𝑑 − 1)/2} and 𝛾 > 0. Now fix a function 𝜑 ∈ 𝐻 𝑡+𝛾+1/2 (Ω) so that its trace coincides with 𝑔 and define 𝑣 = 𝑢 − 𝜑 and observe that 𝑢 solves the above PDE if 𝑣 solves (
−div exp(𝑎)∇𝑣 + 𝜏′ (𝑣) = 𝑓 ′, 𝒙 ∈ Ω, 𝑣 = 0, 𝒙 ∈ 𝜕Ω,
where we defined 𝜏′ (𝑣) := 𝜏(𝑣 + 𝜑) and 𝑓 ′ := 𝑓 + div exp(𝑎)∇𝜑 . Now observe that the functions 𝜏′ and 𝑓 ′ still satisfy the same conditions as 𝜏, 𝑓 in Example 6.3.3 and so we obtain existence and uniqueness of the solutions 𝑣 and in turn 𝑢. Now consider two solutions 𝑢 1 , 𝑢 2 arising from source terms 𝑓1 , 𝑓2 and boundary data 𝑔1 , 𝑔2 . Then the error 𝑤 = 𝑢 1 − 𝑢 2 solves the PDE (
−div exp(𝑎)∇𝑤 = 𝑓 + 𝜏(𝑢 2 ) − 𝜏(𝑢 1 ), 𝒙 ∈ Ω, 𝑤 = 𝑔1 − 𝑔2 , 𝒙 ∈ 𝜕Ω. By standard stability results for linear elliptic PDEs (McLean, 2000, Thm. 4.18) we have ∥𝑤∥ 𝐻 2 (Ω) ≤ 𝐵(∥ 𝑓1 − 𝑓2 ∥ 𝐿 2 (Ω) + ∥𝜏(𝑢 1 ) − 𝜏(𝑢 2 )∥ 𝐿 2 (Ω) + ∥𝑔1 − 𝑔2 ∥ 𝐻 3/2 (Ω) ). 242 We can now repeat the same argument as in the final steps of Example 6.3.3 to get the bound  𝐵  ∥𝑤∥ 𝐻 2 (Ω) ≤ ∥ 𝑓1 − 𝑓2 ∥ 𝐻 2 (Ω) + ∥𝑔1 − 𝑔2 ∥ 𝐻 3/2 (Ω) , 1 − 𝐵𝛽 which verifies Assumption 3(ii) with 𝑠 = 2 provided that 𝛽𝐵 < 1. The Case of Parametric PDEs We now consider the setting of the parametric PDE (6.3). Our error estimates can be viewed as further extending Theorem 6.3.4 with additional assumptions due to the fact that we will need to approximate the solutions on the set Υ = Ω × Θ as well as its relevant boundary which needs to be sufficiently regular for us to apply Proposition 6.6.1. Beyond this technical point, the statement and proof of the result for parametric PDEs is identically to PDEs with boundary conditions and so we state our results succinctly, starting with the requisite assumptions on the parametric PDE. Assumption 4. The following conditions hold: (D1) Ω ⊂ R𝑑 and Θ ⊂ R 𝑝 are compact sets such that 𝜕Ω and 𝜕Θ are smooth Riemannian manifolds of dimensions 𝑑 − 1 and 𝑝 − 1 respectively. (D2) (Stability of the parametric PDE) There exist 𝛾 > 0 and 𝑘, 𝑡 ∈ N satisfying (𝑑 + 𝑝)/2 < 𝑘 + 𝛾 and (𝑑 + 𝑝 − 1)/2 < 𝑡 + 𝛾, and Banach spaces 𝑉1 and 𝑉4 so that for any 𝑟 > 0 it holds that ∥𝑢 1 − 𝑢 2 ∥ 1 ≤ 𝐶 ∥P (𝑢 1 ) − P (𝑢 2 )∥ 𝐻 𝑘 (Υ) + ∥B (𝑢 1 ) − B (𝑢 2 )∥ 𝐻 𝑡 (𝜕Υ)
∀𝑢 1 , 𝑢 2 ∈ 𝐵𝑟 (𝑉1 ), (6.35) ∥P (𝑢 1 ) − P (𝑢 2 )∥ 𝐻 𝑘+𝛾 (Υ) + ∥B (𝑢 1 ) − B (𝑢 2 )∥ 𝐻 𝑡+𝛾 (𝜕Υ) ≤ 𝐶 ∥𝑢 1 − 𝑢 2 ∥ 4 , ∀𝑢 1 , 𝑢 2 ∈ 𝐵𝑟 (𝑉4 ), (6.36) where 𝐶 = 𝐶 (𝑟) > 0 is a constant independent of the 𝑢𝑖 . (D3) U is continuously embedded in 𝑉4 . Unlike Assumptions 2 and 3 here we left the function spaces 𝑉1 and 𝑉4 as generic Banach spaces of functions 𝑢 : Υ ↦→ R since, for parametric PDEs, we can often obtain the desired stability results in non-standard norms, such as the mixed norm in Example 6.3.7 below, as opposed to the Sobolev norms used for the non-parametric 243 PDE setting. More generally, one may also impose 𝑉2 , 𝑉3 to be generic Banach spaces rather than the standard Sobolev spaces. The Sobolev space setting suffices for applications in this paper. With the above assumptions we can now present our main result for the parametric PDE setting. The proof is omitted since it is identical to that of Theorem 6.3.4 except that (1) the argument on 𝜕Ω is now repeated for 𝜕Υ = 𝜕Ω × Θ which is in general a smooth manifold with boundary but this modification does not affect any of the steps in the proof, and (2) the results are stated in terms of the norm on the space 𝑉1 . Theorem 6.3.6. Suppose Assumption 4 is satisfied and let 𝑢★ ∈ U denote the unique strong solution of (6.3). Let 𝑢 † be a minimizer of (6.4) with a set of collocation points 𝑆 ⊂ Υ ∪ 𝜕Υ where 𝑆Υ ⊂ 𝑆 denotes the collocation points in the interior of Υ and 𝑆 𝜕Υ denotes the collocation points on the boundary 𝜕Υ. Define the fill-distances ℎ 𝜕Υ := sup inf 𝜌 𝜕Υ (s′, s), ℎΥ := sup inf |s′ − s|, s′ ∈𝜕Υ s∈𝑆 𝜕Υ s′ ∈Υ s∈𝑆Υ where 𝜌 𝜕Υ : 𝜕Υ×𝜕Υ → R+ is the geodesic distance defined on 𝜕Υ (see Section 6.6), and set ℎ¯ := max{ℎΥ , ℎ 𝜕Υ }. Then there exists a constant ℎ0 > 0 so that if ℎ¯ < ℎ0 then ∥𝑢 † − 𝑢★ ∥ 1 ≤ 𝐶 ℎ¯ 𝛾 ∥𝑢★ ∥ U , ¯ where 𝐶 > 0 is independent of 𝑢 † and ℎ. We end this section by returning to our example of the Darcy flow PDE but this time in the setting where the coefficient 𝑎 and the source 𝑓 are dependent on a finite dimensional parameter 𝜽. We will show that Assumption 4 can be verified in this case, with 𝑉1 and 𝑉4 taken as Banach spaces with mixed Sobolev and 𝐿 2 norms, and so Theorem 6.3.6 is applicable. Example 6.3.7 (1D Parametric Darcy flow PDE). Consider the parametric elliptic PDE (
−div 𝐴(𝒙, 𝜽)∇𝑢 = 𝑓 (𝒙, 𝜽) 𝒙 ∈ Ω, 𝑢 = 0, 𝒙 ∈ 𝜕Ω, over a compact domain Ω and 𝜽 ∈ Θ where both Ω and Θ are assumed to satisfy condition (D1); e.g., take Ω and Θ to be unit balls. For simplicity we are ignoring the boundary operator in this case and imposing homogeneous Dirichlet boundary conditions. In this example we assume 𝑎 is smooth in both 𝒙 and 𝜽, and there exists 𝑚, 𝑀 > 0 such that 𝑚 ≤ 𝐴(𝒙, 𝜽) ≤ 𝑀. As a concrete example we may take 244 Í𝑝 𝐴(𝒙, 𝜽) = 𝑗=1 𝜃 𝑗 𝜓 𝑗 (𝒙) where the 𝜓 𝑗 are a set of smooth functions on Ω that are uniformly bounded from below. First, the boundness of the operator P is straightforward to obtain, since 𝑎 is smooth and its derivatives will be bounded in the bounded domain Ω × Θ. More precisely,
for any 𝛾 > 0, since 𝑓 = −div 𝐴(𝒙, 𝜽)∇𝑢 = −∇𝒙 𝐴 · ∇𝒙 𝑢 − 𝐴Δ𝒙 𝑢, there exists some constant 𝐶 independent of 𝑢 and 𝑓 such that ∥ 𝑓 ∥ 𝐻 𝛾 (Ω×Θ) ≤ 𝐶 ∥𝑢∥ 𝐻 𝛾+2 (Ω×Θ) . Due to the linearity of the equation, by replacing 𝑢 by 𝑢 1 − 𝑢 2 and noting that 𝑓 = P𝑢 = P𝑢 1 − P𝑢 2 , we obtain the forward stability ∥P𝑢 1 − P𝑢 2 ∥ 𝐻 𝛾 (Ω×Θ) ≤ 𝐶 ∥𝑢 1 − 𝑢 2 ∥ 𝐻 𝛾+2 (Ω×Θ) . (6.37) For the backward stability estimate, via intergation by parts, we have ∫ ∫ 2 𝐴(𝒙, 𝜽)|∇𝒙 𝑢(𝒙, 𝜽)| d𝒙d𝜽 = 𝑢(𝒙, 𝜽) 𝑓 (𝒙, 𝜽) d𝒙d𝜽 Υ ∫Υ ≤ ∥𝑢(·, 𝜽)∥ 𝐿 2 (Ω) ∥ 𝑓 (·, 𝜽)∥ 𝐿 2 (Ω) d𝜽 Θ∫ ≤ 𝐶0 ∥∇𝒙 𝑢(·, 𝜽)∥ 𝐿 2 (Ω) ∥ 𝑓 (·, 𝜽)∥ 𝐿 2 (Ω) d𝜽 Θ ≤ 𝐶1 ∥𝑢∥ 𝐿 2 (Θ,𝐻 1 (Ω)) ∥ 𝑓 ∥ 𝐿 2 (Θ,𝐿 2 (Ω)) , 0 where in the first and third inequalities, we used the Cauchy-Schwarz inequality; in the second inequality, we used the Poincaré inequality as 𝑢(·, 𝜽) is zero on 𝜕Ω. Here we used the notation: ∫ ∫ 2 2 2 ∥𝑢∥ 𝐿 2 (Θ,𝐻 1 (Ω)) := ∥𝑢(·, 𝜽)∥ 𝐻 1 (Ω) d𝜽 and ∥ 𝑓 ∥ 𝐿 2 (Θ,𝐿 2 (Ω)) := ∥ 𝑓 (·, 𝜽)∥ 2𝐿 2 (Ω) d𝜽. 0 Θ 0 Θ Note that the 𝐿 2 (Θ, 𝐿 2 (Ω)) norm is also equivalent to the 𝐿 2 (Ω × Θ) norm. Now, using the bound on 𝐴, we obtain that there exists a constant 𝐶 such that ∥𝑢∥ 𝐿 2 (Θ,𝐻 1 (Ω)) ≤ 𝐶 ∥ 𝑓 ∥ 𝐿 2 (Ω×Θ) . Similar to the proof for the forward stability, the backward stability follows by the linearity of the equation. We have ∥𝑢 1 − 𝑢 2 ∥ 𝐿 2 (Θ,𝐻 1 (Ω)) ≤ 𝐶 ∥P𝑢 1 − P𝑢 2 ∥ 𝐿 2 (Ω×Θ) . (6.38) Thus it follows that we can verify condition (6.35) with the norm ∥ · ∥ 1 ≡ ∥ · ∥ 𝐿 2 (Θ,𝐻 1 (Ω)) , ∥ · ∥ 4 = ∥ · ∥ 𝐻 𝛾+2 (Ω×Θ) , and 𝑘 = 0. 0 245 Bounding Fill-distances Our bounds in Theorems 6.3.2 and 6.3.6 are given in terms of the fill distances ℎ of our collocation points. In this section, we provide an upper bound of these fill-in distances in terms of the number of collocation points, under the assumptions that the points are randomly drawn according to uniform distributions both in the interior of the domains and their pertinent boundaries. Throughout this section we only consider the case of non-parametric PDEs, and hence we work with Ω, assumed to be a compact subset of R𝑑 with boundary 𝜕Ω which is a compact smooth manifold of dimension 𝑑 − 1. We focus on the non-parametric setting for simplicity and our results can easily be extended to the parametric PDE setting by simply replacing Ω with Υ as a compact subset of R𝑑+𝑝 . Proposition 6.3.8. Suppose we sample 𝑀Ω points in Ω and 𝑀𝜕Ω points on 𝜕Ω, uniformly with respect to the canonical volume and surface measures. Let 𝛿 > 0. Then, with probability at least 1 − 𝛿, the fill-in distances ℎΩ and ℎ 𝜕Ω satisfy   1/𝑑  1/(𝑑−1)  log(𝑀𝜕Ω /𝛿) log(𝑀Ω /𝛿) , ℎ 𝜕Ω ≤ 𝐶 , ℎΩ ≤ 𝐶 𝑀Ω 𝑀𝜕Ω where 𝐶 is a constant independent of 𝑀Ω , 𝑀𝜕Ω , and 𝛿. The proof of Proposition 6.3.8 can be found in Section 6.7. Let’s combine Proposition 6.3.8 and previous error estimates to get error bounds regarding the number of collocation points. In the case of Theorem 6.3.2 where there is no boundary, we get   𝛾/𝑑 log(𝑀Ω /𝛿) † ★ ∥𝑢★ ∥ U , ∥𝑢 − 𝑢 ∥ 𝐻 𝑠 (Ω) ≤ 𝐶 𝑀Ω while in the case of Theorem 6.3.4 where boundary is considered, we have   𝛾/𝑑   𝛾/(𝑑−1) ! log(𝑀Ω /𝛿) log(𝑀𝜕Ω /𝛿) † ★ ∥𝑢 − 𝑢 ∥ 𝐻 𝑠 (Ω) ≤ 𝐶 ∥𝑢★ ∥ U . + 𝑀Ω 𝑀𝜕Ω More generally, we note that the bounds in Proposition 6.3.8 can be applied to the abstract setting in Theorem 6.3.1 when 𝜖 depends on the fill-in distance. If 𝑠 and 𝛾 are appropriately chosen such that the required assumptions hold, and 𝛾 ≥ 𝑑/2, then the convergence rate is at least as fast as the Monte Carlo rate, for uniformly sampled collocation points. There is no curse of dimensionality in this case. 246 6.4 Numerical Experiments In this section, we study several numerical examples to demonstrate the interplay between the dimensionality of the problem and the regularity of the solution. Our theory demonstrates that this interplay is central to determining the convergence rate, and hence accuracy, of the methodology studied in this paper. In Section 6.4, we consider a high dimensional elliptic PDE with smooth solutions. By varying the dimension of the problem and the frequency of the solution, we demonstrate dimension-benign convergence rates, and in particular the accuracy is better when the frequency of the solution is lower. In Section 6.4, we consider a high dimensional parametric PDE problem to illustrate the importance of choosing kernels that adapted to the regularity of the solution. In Section 6.4, we present a high dimensional Hamilton-Jacobi-Bellman (HJB) equation, which goes beyond our theory and demonstrates the interplay between dimensionality and regularity. High Dimensional PDEs Consider the variable coefficient nonlinear elliptic PDEs ( −∇ · ( 𝐴∇𝑢) + 𝑢 3 = 𝑓 , in Ω, (6.39) 𝑢 = 𝑔, on 𝜕Ω.  Í  We set 𝐴(𝒙) = exp sin 𝑑𝑗=1 cos(𝑥 𝑗 ) , and the ground truth solution ★ 𝑢 (𝒙; 𝛽) = exp(sin(𝛽 𝑑 ∑︁ cos(𝑥 𝑗 ))) , 𝑗=1 where we have a parameter 𝛽 to control the frequency of 𝑢. The right hand side and boundary data are obtained using 𝐴 and 𝑢★. In the experiment, we choose the domain Ω to be the unit ball in R𝑑 for 𝑑 = 2, 3, . . . , 6. We sample 𝑀Ω = 1000, 2000, 4000, 8000 points uniformly in the interior, and respectively 𝑀𝜕Ω = 200, 400, 800, 1600 points uniformly on the boundary. After selecting the kernel function, the number of iteration steps in our algorithm is set to be 3 with initial solution 0. We sample another set of 𝑀Ω test points and evaluate the 𝐿 2 error of the solution on these points. The results are averaged over 10 independent draws of the uniform collocation points. In the first experiment, we choose the Matérn kernel with 𝜈 = 7/2 and with length√ scale 𝜎 = 0.25 𝑑. We choose 𝛽 = 1, 4, to compare the convergence given ground 247 truth with different frequencies. The results are shown in fig. 6.2. It is clear that when 𝛽 is small, the accuracy is better. The slopes of convergence curves also have a tendency to improve for 𝑑 ≥ 3 if we increase 𝛽. solution accuracy 10−1 L2 error 10−2 10−3 d = 2, slope = -2.47 d = 3, slope = -1.75 d = 4, slope = -1.46 d = 5, slope = -1.23 d = 6, slope = -1.24 10−4 103 2 × 103 3 × 103 4 × 103 MΩ 6 × 103 solution accuracy 100 L2 error 10−1 10−2 d = 2, slope = -2.49 d = 3, slope = -1.66 d = 4, slope = -1.32 d = 5, slope = -1.36 d = 6, slope = -0.81 10−3 103 2 × 103 3 × 103 4 × 103 MΩ 6 × 103 Figure 6.2: 𝐿 2 test errors of solutions to Problem (6.39) as a function of the number of collocation points. Left: 𝛽 = 1; right: 𝛽 = 4. In both cases, we choose Matérn kernel with 𝜈 = 7/2. Reported slopes in the legend denote empirical convergence rates. In the second experiment, we fix 𝛽 = 4, and choose the Matérn kernel with √ 𝜈 = 5/2, 9/2 and with lengthscale 𝜎 = 0.25 𝑑. Results are shown in Figure 6.3. Comparing 𝜈 = 5/2, 9/2 and 𝜈 = 7/2 in the last example, we observe that increasing 𝜈 leads to faster convergence. This is due to the fact that the true solution is smooth. In dimension 𝑑 = 2, we can identify the exact convergence rate as 𝜈 − 1. In all dimensions, the rate is faster than the Monte Carlo rate. We observe that the regularity of the solution softens the effect of the curse of dimensionality, i.e., convergence rates are better in higher dimensions when 𝛽 is smaller. 248 solution accuracy L2 error 10 0 10 d = 2, slope = -1.49 d = 3, slope = -0.98 d = 4, slope = -1.02 d = 5, slope = -0.95 d = 6, slope = -0.68 −1 103 2 × 103 3 × 103 4 × 103 MΩ 6 × 103 solution accuracy 100 10−1 L2 error 10−2 10−3 d = 2, slope = -3.51 d = 3, slope = -2.07 d = 4, slope = -1.70 d = 5, slope = -1.60 d = 6, slope = -0.94 10−4 10−5 103 2 × 103 3 × 103 4 × 103 MΩ 6 × 103 Figure 6.3: 𝐿 2 test errors of solutions to Problem (6.39) as a function of the number of collocation points with 𝛽 = 4. Left: Matérn kernel with 𝜈 = 5/2; right: Matérn kernel with 𝜈 = 9/2. Reported slopes in the legend denote empirical convergence rates. Parametric PDEs We consider a parametric version of the linear (𝜏 = 0) darcy flow problem in example 6.2.1: (
−div exp(𝑎(𝒙, 𝜽))∇𝑢 (𝒙) = 𝑓 (𝒙), 𝒙 ∈ Ω, (6.40) 𝑢(𝒙) = 𝑔(𝒙), 𝒙 ∈ 𝜕Ω. Following the general form eq. (6.3), we aim to obtain the solution as a function taking values in the product space Υ. eq. (6.40) can be rewritten in terms of s = (𝒙, 𝜽) with new forcing terms b 𝑓 and b 𝑔 depending only on the first coordinate of s (
−div𝒙 𝐴(s)∇𝒙 𝑢(s) (s) = b 𝑓 (s) = 𝑓 (𝒙), s ∈ Υ, 𝑢(s) = b 𝑔 (s) = 𝑔(𝒙), s ∈ 𝜕Υ. (6.41) 249 Recall that we defined 𝜕Υ = 𝜕Ω × Θ. For our numerical example, we let 𝑑 = 1 and Í𝑝 𝜃 vary 𝑝. We set 𝐴(𝑥, 𝜃) = 2 + 𝜃 0 + 𝑗=1 𝑗 𝑘𝑗 sin(𝜋𝑥 + 𝑗), 𝑓 (𝑥) = 𝑥 and 𝑔(𝑥) = 0, a similar setting as in (Chkifa, Cohen, DeVore, et al., 2012). We choose Ω = [0, 1], and Θ = [0, 1] 𝑝 , for 𝑝 = 2, 3, . . . , 6. Note 1 ≤ 𝐴(s) ≤ 4 since the sum is in [−1, 1] for all 𝑝 and 𝜃 ∈ Θ, matching the setting of example 6.3.7. We sample different 𝑀Ω points uniformly in the interior, and 𝑀𝜕Ω = 𝑀Ω /10 points uniformly on the boundary of 𝑥. We do two experiments with different choices of kernel, in the first (fig. 6.4, left), a vanilla Gaussian kernel with different length scales for the 𝑥 and 𝜃 dimension, and with a scaling of the length scale in 𝜃 proportional to √ 𝑝. In the second one (fig. 6.4, right), we adapt the Gaussian kernel to the decay in 𝐴(𝑥, 𝜃), by including the decay of 1/ 𝑗 𝑘 in the norm in 𝜃 space used by the kernel. We see significant improvement in test error using this adaptation in high dimensions, which suggests future research directions of kernel adaptation to the specific form of the PDE. In all cases, we use a cross-validation procedure for hyperparameter tuning and we observe the average 𝐿 2 test error on an independent set of test points for different values of 𝑝 and 𝑀Ω . Since 𝑑 = 1 we computed our ground truth solution by numerically integrating Equation eq. (6.41) using quadrature. As mentioned, this problem was also explored by (Chkifa, Cohen, DeVore, et al., 2012), in which sparse multivariate polynomials are used to estimate the solution with a rate independent of the number of parameters, provided the decay of the coefficient functions is large enough (in ℓ 𝑝 for some 0 < 𝑝 < 1). While this assumption is satisfied in this example, our method’s convergence rate greatly depends on the dimension of 𝜃 when the kernel is not adapted to the particular equations and coefficients 𝐴(𝑥, 𝜃). Our results indicate improvement in the dependence of convergence rates on dimension when the kernel is adapted to the regularity of 𝐴. It remains open whether our kernel based approach (which is not specific to parametric equations) can achieve the same dimension independent convergence rates as the ones in (Chkifa, Cohen, DeVore, et al., 2012) (which apply even in the countably infinite dimensional case and which they refer to as breaking the curse of dimensionality) for parametric elliptic PDEs with rapidly decreasing parametric dependence as specified above (this assumption implies a finite number of effective parameters). 250 solution accuracy: non-adapted kernel solution accuracy: adapted kernel 10−4 L2 error L2 error 10−4 10−5 10−6 d = 1+2, slope = -2.24 d = 1+3, slope = -1.63 d = 1+4, slope = -0.61 d = 1+5, slope = -0.22 d = 1+6, slope = -0.10 103 2 × 103 MΩ 10−5 10−6 4 × 103 d = 1+2, slope = -2.17 d = 1+3, slope = -2.23 d = 1+4, slope = -1.98 d = 1+5, slope = -1.15 d = 1+6, slope = -0.71 103 8 × 103 2 × 103 MΩ 4 × 103 Figure 6.4: 𝐿 2 test error of solutions to Problem (6.41) as a function of the number of collocation points. Left: vanilla gaussian kernel; Right: Gaussian kernel adapted to the regularity of 𝐴. Reported slopes in the legend denote empirical convergence rates. High Dimensional HJB Equation Consider a prototypical HJB equation: (𝜕𝑡 + Δ)𝑉 (𝒙, 𝑡) − |∇𝑉 (𝒙, 𝑡)| 2 = 0 𝑉 (𝒙, 𝑇) = 𝑔(𝒙) , (6.42) where 𝑔(𝒙) = log( 12 + 12 |𝒙| 2 ), 𝒙 ∈ R𝑑 , 𝑡 ∈ [0, 𝑇]. We are interested in solving 𝑉 (𝒙 0 , 0) for some 𝒙0 ∈ R𝑑 . We adopt the stochastic differential equation (SDE) formula for representing the solution of the PDEs, following (Weinan, Han, and Jentzen, 2017; Richter, Sallandt, and Nüsken, 2021). More specifically, consider the SDE √ 𝑑𝑋𝑠 = 2𝑑𝑊𝑠 , 𝑋0 = 𝒙0 . (6.43) √ We define 𝑌𝑠 = 𝑉 (𝑋𝑠 , 𝑠), 𝑍 𝑠 = 2∇𝑉 (𝑋𝑠 , 𝑠). By Ito’s formula, one obtains 𝑑𝑌𝑠 = 1 |𝑍 𝑠 | 2 𝑑𝑠 + 𝑍 𝑠 · 𝑑𝑊𝑠 . 2 (6.44) The strategy is to integrate the above SDE backward to 𝑌0 . An implicit3 Euler discretization from time 𝑡 𝑛+1 to 𝑡 𝑛 (Δ𝑡 = 𝑡 𝑛+1 − 𝑡 𝑛 ) leads to the following equation: √ √ 𝑉 (𝑋𝑡 𝑛+1 , 𝑡 𝑛+1 ) = 𝑉 (𝑋𝑡 𝑛 , 𝑡 𝑛 ) + |∇𝑉 (𝑋𝑡 𝑛 , 𝑡 𝑛 )| 2 Δ𝑡 + 2∇𝑉 (𝑋𝑡 𝑛 , 𝑡 𝑛 ) · 𝜉𝑛+1 Δ𝑡 . (6.45) 3 Implicit because are integrating backwards in time. 8 × 103 251 Algorithmically, we sample 𝐽 different paths of the forward SDE in (6.43), namely ( 𝑗) 𝑋𝑡 𝑛 , 1 ≤ 𝑗 ≤ 𝐽, using the Euler–Maruyama scheme. Then, backward in time, we apply our kernel method, namely to solve the following optimization problem    minimize ∥𝑢∥ U  𝑢∈U √ √  ( 𝑗) ( 𝑗) ( 𝑗) ( 𝑗) 2  st 𝑢(𝑋 , 𝑡 ) + |∇𝑢(𝑋 , 𝑡 )| Δ𝑡 + 2∇𝑢(𝑋 , 𝑡 ) · 𝜉 Δ𝑡 = 𝑉 (𝑋𝑡 𝑛+1 , 𝑡 𝑛+1 ) 𝑛 𝑛 𝑛 𝑛+1 𝑡𝑛 𝑡𝑛 𝑡𝑛  (6.46) to get the solution 𝑉 (·, 𝑡 𝑛 ), assuming 𝑉 (·, 𝑡 𝑛+1 ) has been solved. Iterating this process, we end up with the solution 𝑉 (𝒙0 , 0). We can understand the algorithm as applying our kernel method iteratively with the sample path as the collocation points. Experimentally, we consider 𝑑 = 100 as in (Weinan, Han, and Jentzen, 2017; Richter, Sallandt, and Nüsken, 2021). We aim to solve 𝑉 (𝒙 0 , 0) for 𝒙 0 = 0. The ground truth is 𝑉 (𝒙0 , 0) = 4.589992 provided in (Weinan, Han, and Jentzen, 2017). We sample 𝐽 = 2000 paths from 𝒙 0 and choose the inverse quadratic ker −1  2 . We use the “linearize-then-optimize" approach to + 1 nel 𝑘 (𝒙, 𝒚; 𝜎) = ∥𝒙−𝒚∥ 2𝑑𝜎 2 compute an approximate solution to (6.46). The nugget term is set to be 𝜂 = 10−3 . The result is shown in Table 6.2. 𝜎 Computed solution 𝑉 (𝒙0 , 0) Relative accuracy 10 25 50 100 200 5.6042 4.6366 4.6039 4.6021 4.6021 22.10% 1.0154% 0.303% 0.2638% 0.2638% Table 6.2: Numerical results for the HJB equation (6.42), computing the quantity 𝑉 (𝒙 0 , 0). We observe that a suitable choice of the lengthscale of the kernel is crucial to obtain an accurate solution. Compared to the relative accuracy of 0.171% (reported in (Richter, Sallandt, and Nüsken, 2021)) using neural networks (DenseNet like architecture with four hidden layers) to solve (6.45), the accuracy of using kernel methods with a simple quadratic kernel is comparable. Moreover, the lengthscale of the kernel is very large, indicating that the solution behavior of this HJB equation is very smooth; similar “blessings of dimensionality” have been reported and discussed in (Richter, Sallandt, and Nüsken, 2021), where they used a constant function (and the terminal function 𝑔) as ansatz to solve (6.45) and obtained very high accuracy4 . Thus, this HJB example in dimension 100 demonstrates again the trade-off between the smoothness of the solution and the curse of dimensionality. 4 We anticipate that using the feature map perspective of kernel methods with constants and 𝑔 as features will achieve a similar accuracy as in (Richter, Sallandt, and Nüsken, 2021). We did not pursue this here to avoid using strong prior information on the solution beyond regularity. 252 6.5 Conclusions In this paper, we conducted an error analysis of GP and kernel based methods for solving PDEs. We provided convergence rates under the assumptions that (1) the solution belongs to the RKHS which is embedding to some Sobolev space of sufficient regularity, and (2) the underlying forward and inverse PDE operator is stable in corresponding Sobolev spaces. Our analysis relies on the crucial minimizing norm property of the numerical solution in the kernel/GP methodology. The analysis could be seamlessly generalized to the function class of NNs and other norms such as non-quadratic norms if we can formulate the training process as a minimization problem over the related norm. We emphasize that our convergence rates hold for the exact minimizer of the minimization problem. In practice, finding such a minimizer algorithmically can be a separate and challenging problem. Our numerical experience suggests that GaussNewton iterations usually perform well, and typically, 2-5 iterations are sufficient for convergence. Therefore, we can combine the error analysis in this paper and the fast implementation of the algorithm in (Yifan Chen, Houman Owhadi, and Florian Schäfer, 2024) to obtain a near-linear complexity solver for nonlinear PDEs with rigorous accuracy guarantee. It is worth mentioning that this paper focuses only on analyzing the MAP estimator within the GP interpretation. Exploring the posterior distribution of the GP can provide a means for quantifying uncertainty in the solution. In particular, analyzing the posterior contraction is an interesting direction for future research. Acknowledgments The authors gratefully acknowledge support by the Air Force Office of Scientific Research under MURI award number FA9550-20-1-0358 (Machine Learning and Physics-Based Modeling and Simulation). BH acknowledges support by the National Science Foundation grant number NSF-DMS-2208535 (Machine Learning for Bayesian Inverse Problems). HO also acknowedges support by the Department of Energy under award number DE-SC0023163 (SEA-CROGS: Scalable, Efficient and Accelerated Causal Reasoning Operators, Graphs and Spikes for Earth and Embedded Systems) and a Department of Defense Vannevar Bush Faculty Fellowship. 253 6.6 Sobolev Sampling Inequalities on Manifolds Below we collect useful sampling inequalities for Sobolev functions defined on smooth manifolds with corners. Following (J. M. Lee, 2012, Chs. 1, 16) we consider a smooth, compact Riemannian manifold M ⊂ R𝑑 of dimension 𝑘 ≤ 𝑑 with corners, i.e., a Riemannian manifold with a smooth structure with corners; see (J. M. Lee, 2012, Ch. 16). On such a manifold we define the natural geodesic distance ∫ 1 ¤ 𝜌M : M × M → R, 𝜌M (𝑥, 𝑦) := inf ∥ ℓ(𝑡)∥𝑑𝑡, 0 where the infimum is taken over all piecewise smooth paths ℓ : [0, 1] ↦→ M ¤ satisfying the boundary conditions ℓ(0) = 𝑥 and ℓ(1) = 𝑦, and ∥ ℓ(𝑡)∥ is the length ¤ of the tangent vector ℓ(𝑇) under the Riemannian metric. Following (Fuselier and G. B. Wright, 2012) (see also (Taylor, 2013, Sec. 4.3)) we further consider the Sobolev spaces 𝐻 𝑘 (M) of functions defined on M as follows: Let A = {𝑀 𝑗 , Ψ 𝑗 } 𝑁𝑗=1 be an atlas for M and let {𝜅 𝑗 } be a partition of unity of M, subordinate to 𝑀 𝑗 . Then given functions 𝑢 : M → R we define the Sobolev norms and the associated Sobolev spaces 𝐻 𝑠 (M) as 1/2 𝑁 𝐻 𝑠 (M) := {𝑢 : M → R | ∥𝑢∥ 𝐻 𝑠 (M) < +∞}, ©∑︁ ª ∥𝑢∥ 𝐻 𝑠 (M) := ­ ∥𝜋 𝑗 (𝑢)∥ 2𝐻 𝑠 (Ξ 𝑗 ) ® « 𝑗=1 ¬ where the maps 𝜋 𝑗 are defined as ( 𝜋 𝑗 ( 𝑓 ) := 𝜅 𝑗 ( 𝑓 (Ψ−1 𝑗 (𝑦))), if 𝑦 ∈ Ψ 𝑗 (𝑀 𝑗 ), 0 otherwise. and the sets Ξ 𝑗 are given by   R𝑘 if Ψ 𝑗 is an interior chart,     Ξ 𝑗 := {(𝑥 1 , . . . , 𝑥 𝑘 ) ∈ R 𝑘 |𝑥 1 ≥ 0} if Ψ 𝑗 is a boundary chart,     {(𝑥 1 , . . . , 𝑥 𝑘 ) ∈ R 𝑘 |𝑥 1 ≥ 0, . . . , 𝑥 𝑘 ≥ 0} if Ψ 𝑗 is a corner chart.  Put simply, the Sobolev spaces 𝐻 𝑠 (M) are functions on M that, locally after the flattening of the manifold belong to the standard Sobolev spaces 𝐻 𝑠 . With these notions at hand we then recall the following result of (Fuselier and G. B. Wright, 2012), which was proven by those authors for smooth embedded manifolds without boundary or corners. However, a brief investiation of the proof of that result reveals that it can immediately be generalized to our setting with manifolds with corners. In , 254 fact, the idea of the proof is to use the atlas to locally flatten the manifold and apply classic sampling theorems such as (Arcangéli, López de Silanes, and Torrens, 2007, Thm. 4.1) on each patch. The only difference in the case of manifolds with corners is that the patches do not only map to R 𝑘 but rather to the subspaces Ξ 𝑗 depending on whether the corresponding chart is an interior, boundary, or corner chart. Proposition 6.6.1 ((Fuselier and G. B. Wright, 2012, Lem. 10)). Suppose M ⊂ R𝑑 is a smooth, compact, Riemannian manifold with corners, of dimension 𝑘 and let 𝑠 > 𝑘/2 and 𝑟 ∈ N satisfy 0 ≤ 𝑟 ≤ ⌈𝑠⌉ − 1. Let 𝑋 ⊂ M be a discrete set with mesh norm ℎM defined as ℎM := sup inf 𝜌M (𝑥, 𝑥 ′). 𝑥 ′ ∈M 𝑥∈𝑋 Then there is a constant ℎ0 > 0 depending only on M such that if ℎM < ℎ0 and if 𝑢 ∈ 𝐻 𝑠 (M) satisfies 𝑢| 𝑋 = 0 then 𝑠−𝑟 ∥𝑢∥ 𝐻 𝑟 (M) ≤ 𝐶ℎM ∥𝑢∥ 𝐻 𝑠 (M) . Here 𝐶 > 0 is a constant independent of ℎM and 𝑢. 6.7 Bounds on Fill Distances This section collects a result from (Reznikov and Saff, 2016) for bounding the fill-in distance for randomly distributed points on a manifold. Assume (M, 𝜌) is a metric space, and 𝜇 is a finite positive Borel measure supported on M. Let 𝑋 = {𝑥 1 , ..., 𝑥 𝑁 } be a set of 𝑁 points, independently and randomly drawn from 𝜇. Define the fill-in distance ℎM = sup inf 𝜌(𝑥, 𝑥 ′) . (6.47) 𝑥 ′ ∈M 𝑥∈𝑋 Then, (Reznikov and Saff, 2016, Thm. 2.1) implies the following, Proposition 6.7.1. Suppose Φ is a continuous non-negative strictly increasing function on (0, ∞) satisfying Φ(𝑟) → 0 as 𝑟 → 0+ . If there exists a positive number 𝑟 0 such that 𝜇(𝐵(𝑥, 𝑟)) ≥ Φ(𝑟) holds for all 𝑥 ∈ M and every 𝑟 < 𝑟 0 , then there exist positive constants 𝑐 1 , 𝑐 2 , 𝑐 3 , and 𝛼0 such that for any 𝛼 > 𝛼0 , we have    −1 𝛼 log 𝑁 ≤ 𝑐 2 𝑁 1−𝑐3 𝛼 . (6.48) P ℎM ≥ 𝑐 1 Φ 𝑁 We use this proposition to prove Proposition 6.3.8. 255 Proof of Proposition 6.3.8. We apply Proposition 6.7.1. For the bounded domain Ω ⊂ R𝑑 , we know that there exists a constant 𝐶 such that Φ(𝑟) = 𝐶𝑟 𝑑 will satisfy the assumption in Proposition 6.7.1. Moreover, we choose 𝛼 such that 𝑐 2 𝑀Ω1−𝑐3 𝛼 ≤ 𝛿. 𝐶′ This implies that 𝛼 ≥ 𝑐3 log1 𝑀Ω log(𝑐 2 𝑀Ω /𝛿). Pick 𝛼 = 𝑐3 log 𝑀Ω log(𝑐 2 𝑀Ω /𝛿) for ′ some 𝐶 ≥ 1 such that 𝛼 ≥ 𝛼0 . Then Proposition 6.7.1 shows that with probability at least 1 − 𝛿, ℎΩ ≤ 𝑐 1 Φ −1     1/𝑑 𝛼 log 𝑀Ω ′′ log(𝑀Ω /𝛿) ≤𝐶 , 𝑀Ω 𝑀Ω where 𝐶 ′′ is a constant independent of 𝑀Ω and 𝛿. The bound on ℎ 𝜕Ω can be proved similarly by choosing Φ(𝑟) = 𝐶𝑟 𝑑−1 . □ 6.8 The Choice of Nugget Terms For numerical stability, we add a diagonal adaptive nugget term to the kernel matrix in our computation, such that !
ℓ + P ′ (𝑢 ℓ )𝑢 ℓ | 𝑓 − 𝑢 s Ω 𝑢 ℓ+1 (𝑥) = 𝐾 (𝑥, 𝝓𝑙 ) [𝐾 (𝝓𝑙 , 𝝓𝑙 ) + 𝜂diag(𝐾 (𝝓𝑙 , 𝝓𝑙 ))] −1
𝑔 − 𝑢 ℓ + B ′ (𝑢 ℓ )𝑢 ℓ | s𝜕Ω . Typically 𝜂 = 10−10 . This nugget term is similar to the adaptive nugget term proposed in (Yifan Chen, Hosseini, et al., 2021b). It is much more effective than the naive choice of 𝐾 (𝝓𝑙 , 𝝓𝑙 )+𝜂𝐼, since the conditioning of the interior block and the boundary block in the kernel matrix differs dramatically. 256 Chapter 7 DISCOVERING GRAPHICAL STRUCTURE AND FUNCTIONAL RELATIONSHIPS WITHIN DATA Most problems within and beyond the scientific domain can be framed into one of the following three levels of complexity of function approximation. Type 1: Approximate an unknown function given input/output data. Type 2: Consider a collection of variables and functions, some of which are unknown, indexed by the nodes and hyperedges of a hypergraph (a generalized graph where edges can connect more than two vertices). Given partial observations of the variables of the hypergraph (satisfying the functional dependencies imposed by its structure), approximate all the unobserved variables and unknown functions. Type 3: Expanding on Type 2, if the hypergraph structure itself is unknown, use partial observations of the variables of the hypergraph to discover its structure and approximate its unknown functions. These hypergraphs offer a natural platform for organizing, communicating, and processing computational knowledge. While most scientific problems can be framed as the data-driven discovery of unknown functions in a computational hypergraph whose structure is known (Type 2), many require the data-driven discovery of the structure (connectivity) of the hypergraph itself (Type 3). We introduce an interpretable Gaussian Process (GP) framework for such (Type 3) problems that does not require randomization of the data, nor access to or control over its sampling, nor sparsity of the unknown functions in a known or learned basis. Its polynomial complexity, which contrasts sharply with the super-exponential complexity of causal inference methods, is enabled by the nonlinear analysis of variance capabilities of GPs used as a sensing mechanism. Introduction The three levels of complexity of function approximation. As illustrated in Fig. 7.1.(a-c), Type 1, Type 2 and Type 3 problems can be formulated as completing or discovering hypergraphs where nodes represent variables and edges represent functional dependencies. The graph in Type 1 has only two variables and one unknown function. The graph in Type 2 has multiple variables and (some possibly unknown) functions, and the connectivity of the graph is known. The graph in Type 3 has an unknown connectivity (functional dependencies between variables 257 may be unknown) and this is the focus of this work. Current methods for solving Type 1 and 2 problems include Deep Learning (DL) methods, which benefit from extensive hardware and software support but have limited guarantees. Despite their prevalence, Type 3 challenges have been largely overlooked due to their inherent complexity. Causal inference methods (Morgan and Winship, 2015; M. Glymour, Pearl, and Jewell, 2016) and probabilistic graphs (Stegle et al., 2010; Lopez-Paz et al., 2015) and sparse regression methods (Doostan and Houman Owhadi, 2011; Brunton, Proctor, and Kutz, 2016), offer potential avenues for addressing Type 3 problems. However, it is important to note that their application to these problems necessitates additional assumptions. Causal inference models, for instance, typically assume randomized data and some level of access to the data generation process or its underlying distributions. Sparse regression methods, on the other hand, rely on the assumption that functional dependencies have a sparse representation within a known basis. In this paper, we do not impose these assumptions, and thus, these particular techniques may not be applicable. Furthermore while the complexity of Bayesian causal inference methods may grow super-exponentially with the number 𝑑 of variables, the complexity of our method is that of 𝑑 parallel computations of polynomial complexities bounded between O (𝑑) (best case) and O (𝑑 4 ) (worst case). Figure 7.1: The three levels of complexity of function approximation. Generalizing Gaussian Process methods. Although Gaussian Process (GP) methods are sometimes perceived as a wellfounded but old technology limited to curve fitting (Type 1 problems), they have recently been generalized, beyond Type 1 problems, to an interpretable framework (Computational Graph Completion or CGC (Houman Owhadi, 2022)) for solving Type 2 problems (Houman Owhadi, 2023a; Yifan Chen, Hosseini, et al., 2021c; Batlle, Yifan Chen, et al., 2023; Yifan Chen, Houman Owhadi, and Florian Schäfer, 2024; Darcy et al., 2023; Hamzi, Houman Owhadi, and Kevrekidis, 2023), all while maintaining the simple and transparent theoretical and computational guarantees 258 of kernel/optimal recovery methods (Micchelli and Rivlin, 1977; H. Owhadi and C. Scovel, 2019). This paper introduces a comprehensive GP framework for solving Type 3 problems, which is interpretable and amenable to analysis. This framework leverages the Uncertainty Quantification (UQ) properties of GP methods, which do not have an immediate natural counterpart in DL methods. It is based on a kernel generalization (Wahba, 2003; Houman Owhadi, Clint Scovel, and Yoo, 2021) of variance-based sensitivity analysis guiding the discovery of the structure of the hypergraph. Here, variables are linked via GPs, and those contributing to the highest data variance unveil the hypergraph’s structure. This GP variance decomposition of the data leads to signal-to-noise and a Z-score that can be employed to determine whether a given variable can be approximated as a nonlinear function of a subset of other variables. The scope of Type 1, 2 and 3 problems. The scope of Type 1, 2 and 3 problems is immense. Numerical approximation (H. Owhadi and C. Scovel, 2019; H. Owhadi, C. Scovel, and F. Schäfer, 2019; Florian Schäfer, Timothy John Sullivan, and Houman Owhadi, 2021c; Florian Schäfer and Houman Owhadi, 2021), Supervised Learning, and Operator Learning (Hamzi, Maulik, and Houman Owhadi, 2021b; Hamzi and Houman Owhadi, 2021; Florian Schäfer and Houman Owhadi, 2023; Batlle, Darcy, et al., 2023) can all be formulated as Type 1 problems, i.e., as approximating unknown functions given (possibly with noisy and infinite/high-dimensional) inputs/output data. The common GP based solution to these problems is to replace the underlying unknown function by a GP and compute its MAP estimator given available data. Type 2 problems include (Fig. 7.1.(d-h)) solving and learning (possibly stochastic) ordinary or partial differential equations (Yifan Chen, Hosseini, et al., 2021c; Darcy et al., 2023), Deep Learning (Houman Owhadi, 2023a), dimension reduction, reduced-ordered modeling, system identification (Houman Owhadi, 2022), closure modeling, etc. Indeed, all these problems can be formulated as completing a computational graph (Houman Owhadi, 2022). In this formulation, variables and functions are represented by the nodes and the edges of the graph whose structure corresponds to the functional dependencies between variables. Some of the functions and variables may be unknown, and by completing, we mean approximating the unknown functions (colored in red in Fig. 7.1) given samples from the observed variables. The common GP-based solution to Type 2 problems is to simply replace unknown functions by GPs and compute their MAP/MLE estimators given available data and constraints 259 imposed by the structure of the graph (Houman Owhadi, 2022). While most problems in Computational Sciences and Engineering (CSE) and Scientific Machine Learning (SciML) can be framed as Type 1 and Type 2 challenges, many problems in science can only be categorized as Type 3 problems, i.e., discovering the structure/connectivity of the graph itself from data prior to its completion. Indeed the scope of Type 3 problems extends well beyond Type 2 problems and includes equation discovery (Fig. 7.1.(i)); the modeling of land surface interactions in weather prediction (Fig. 7.1.(j) from (Dirmeyer et al., 2019), discovering possibly hidden functional dependencies between state variables for a finite number of snapshots of those variables); social network analysis (Fig. 7.1.(k) from (Gittell and Ali, 2021), discovering functional dependencies between quantitative markers associated with each individual in situations where the connectivity of the network may be hidden); economic network analysis (Fig. 7.1.(m) from (Schweitzer et al., 2009), discovering functional dependencies between the economic markers of different agents or companies, which is significant to systemic risk analysis); and computational biology (Fig. 7.1.(l) from (Sachs et al., 2005), identifying pathways and interactions between genes from their expression levels). Figure 7.2: Ancestors identification in Type 3 problem. Overview of the proposed approach for Type 3 problems. We first present an algorithmic overview of the proposed GP-based approach for Type 3 problems. For ease of presentation, we consider the simple setting of Fig. 7.2.(a) where we are given 𝑁 samples on the variables 𝑥 1 , . . . , 𝑥 6 . After measurements/collection, these variables are normalized to have zero mean and unit variance. Our objective is to uncover the underlying dependencies between them. A signal-to-noise ratio to decide whether or not a node has ancestors. Our algorithm’s core concept is the identification of ancestors for each node in the graph. Let’s explore this idea in the context of a specific node, say 𝑥 1 , as depicted in Fig. 7.2(b). Determining whether 𝑥 1 has ancestors is akin to asking if 𝑥 1 can be expressed as a function of 𝑥 2 , 𝑥3 , . . . , 𝑥 6 . In other words, can we find a function 𝑓 260 (living in a pre-specified space of functions that could be of controlled regularity) such that 𝑥 1 ≈ 𝑓 (𝑥2 , . . . , 𝑥 6 ) ? (7.1) To answer this question we regress 𝑓 with a centered GP 𝜉 ∼ N (0, Γ) whose covariance function Γ is an additive kernel of the form Γ = 𝐾 𝑠 + 𝛾δ(𝑥 − 𝑦) , where 𝐾 𝑠 is a smoothing kernel, 𝛾 > 0 is a regularization parameter and δ(𝑥 − 𝑦) is the white noise covariance operator. This is equivalent to assuming the GP 𝜉 to be the sum of two independent GPs, i.e., 𝜉 = 𝜉 𝑠 + 𝜉𝑛 where 𝜉 𝑠 ∼ N (0, 𝐾 𝑠 ) is a
smoothing/signal GP and 𝜉𝑛 ∼ N 0, 𝛾δ(𝑥 − 𝑦) is a noise GP. Writing H𝐾𝑠 for the Reproducing Kernel Hilbert Space (RKHS) induced by the kernel 𝐾 𝑠 , this is also equivalent to approximating 𝑓 with a minimizer of inf ∥ 𝑓 ∥ 2𝐾𝑠 + 𝑓 ∈H𝐾𝑠 1 2 𝑓 (𝑋) − 𝑌 R 𝑁 , 𝛾 (7.2) where ∥ · ∥ 2R 𝑁 is the Euclidean norm on R𝑁 , 𝑋 is the input data on 𝑓 obtained as an 𝑁 × 5-matrix whose rows 𝑋𝑖 are the samples on 𝑥2 , . . . , 𝑥 6 , 𝑌 is the output data on 𝑓 obtained as an 𝑁-vector whose entries are obtained from the samples on 𝑥 1 , and 𝑓 (𝑋) is a 𝑁-vector whose entries are the evaluations 𝑓 (𝑋𝑖 ). At the minimum V (𝑠) := ∥ 𝑓 ∥ 2𝐾𝑠 (7.3) quantifies the data variance explained by the signal GP 𝜉 𝑠 and V (𝑛) := 1 2 𝑓 (𝑋) − 𝑌 R 𝑁 𝛾 (7.4) quantifies the data variance explained by the noise GP 𝜉𝑛 (Houman Owhadi, Clint Scovel, and Yoo, 2021). This allows us to define the signal-to-noise ratio V (𝑠) ∈ [0, 1] . V (𝑠) + V (𝑛) (7.5) V (𝑠) 1 If V (𝑠)+V (𝑛) < 0.5 , then, as illustrated in Fig. 7.2.(c), we deduce that 𝑥 1 has no ancestors, i.e., 𝑥 1 cannot be approximated as function of 𝑥 2 , . . . , 𝑥 6 . Conversely if V (𝑠) V (𝑠)+V (𝑛) > 0.5, then, we deduce that 𝑥 1 has ancestors, i.e., 𝑥 1 can be approximated as function of 𝑥 2 , . . . , 𝑥 6 . 1 We will later present a version with a more sophisticated method for pruning, but we keep the 0.5 threshold in this example for simplicity. 261 Selecting the signal kernel 𝐾 𝑠 This process is repeated by selecting the kernel 𝐾 𝑠 to be linear (𝐾 𝑠 (𝑥, 𝑥 ′) = 1 + Í Í Í 𝛽1 𝑖 𝑥𝑖 𝑥𝑖′), quadratic (𝐾 𝑠 (𝑥, 𝑥 ′) = 1+ 𝛽1 𝑖 𝑥𝑖 𝑥𝑖′ + 𝛽2 𝑖≤ 𝑗 𝑥𝑖 𝑥 𝑗 𝑥𝑖′𝑥 ′𝑗 ) or fully nonlinear to identify 𝑓 as linear, quadratic, or nonlinear. In the case of a nonlinear kernel, we employ ∑︁ ∑︁ Ö 𝐾 𝑠 (𝑥, 𝑥 ′) = 1 + 𝛽1 𝑥𝑖 𝑥𝑖′ + 𝛽2 𝑥𝑖 𝑥 𝑗 𝑥𝑖′𝑥 ′𝑗 + 𝛽3 (1 + 𝑘 (𝑥𝑖 , 𝑥𝑖′)). (7.6) 𝑖 𝑖≤ 𝑗 𝑖 where 𝑘 is a universal kernel, such as a Gaussian or a Matérn kernel, with all parameters set to 1, and 𝛽𝑖 assigned the default value 0.1. We select 𝐾 𝑠 as the first kernel that surpasses a signal-to-noise ratio of 0.5. If no kernel reaches this threshold, we conclude that 𝑥 1 lacks ancestors. Pruning ancestors based on signal-to-noise ratio. Once we establish that 𝑥1 has ancestors, the next step is to prune its set of ancestors iteratively. We remove nodes with the least contribution to the signal-to-noise ratio and stop before that ratio drops below 0.5 as illustrated in Fig. 7.2.(d). To describe this, assume that 𝐾 𝑠 is as in (7.6). Then 𝐾 𝑠 is an additive kernel that can be decomposed into two parts: 𝐾 𝑠 = 𝐾1 + 𝐾2 , (7.7) Í Î Í where 𝐾1 = 1 + 𝛽1 𝑖≠1,2 𝑥𝑖 𝑥𝑖′ + 𝛽2 𝑖≤ 𝑗,𝑖, 𝑗≠1,2 𝑥𝑖 𝑥 𝑗 𝑥𝑖′𝑥 ′𝑗 + 𝛽3 𝑖≠1,2 (1 + 𝑘 (𝑥𝑖 , 𝑥𝑖′)) does not depend on 𝑥2 and 𝐾2 = 𝐾 𝑠 − 𝐾1 depends on 𝑥 2 . This decomposition allows us to express 𝑓 as the sum of two components: 𝑓 = 𝑓1 + 𝑓2 , (7.8) where 𝑓1 does not depend on 𝑥 2 , 𝑓2 depends on 𝑥 2 and ( 𝑓1 , 𝑓2 ) = argmin (𝑔1 ,𝑔2 )∈H𝐾 ×H𝐾 s.t. 𝑔1 +𝑔2 = 𝑓 ∥𝑔1 ∥ 2𝐾1 + 1 ∥ 𝑓2 ∥ 2𝐾 ∥𝑔2 ∥ 2𝐾2 . Furthermore, ∥ 𝑓 ∥ 2𝐾𝑠 = ∥ 𝑓1 ∥ 2𝐾1 + ∥ 𝑓2 ∥ 2𝐾2 , and ∥ 𝑓 ∥ 2 1 ∈ [0, 1] quantifies the 𝐾𝑠 contribution of 𝑥 2 to the signal data variance. Following the procedure illustrated in Fig. 7.2.(d), if, for example, 𝑥4 is found to have the least contribution to the signal data variance, we recompute the signal-to-noise ratio without 𝑥 4 in the set of ancestors for 𝑥 1 . If that ratio is below 0.5, we do not remove 𝑥 4 from the list of ancestors, and 𝑥2 , 𝑥3 , 𝑥4 , 𝑥5 , 𝑥6 is the final set of ancestors of 𝑥 1 . If this ratio remains above 0.5, we proceed with the removal. This iterative process continues, and we stop before the signal-to-noise ratio drops below 0.5 to identify the final list of ancestors of 𝑥 1 . The most efficient version of our proposed algorithm does not use a threshold of 0.5 on the signal-to-noise ratio to prune ancestors, but it rather employs 2 262 V (𝑛) an inflection point in the noise-to-signal ratio V (𝑠)+V (𝑛) (𝑞) as a function of the number 𝑞 of ancestors (Fig. 7.3.(d)). To put it simply, after ordering the ancestors in decreasing contribution to the signal, the final number 𝑞 of ancestors is determined V (𝑛) V (𝑛) as the maximizer of V (𝑠)+V (𝑛) (𝑞 + 1) − V (𝑠)+V (𝑛) (𝑞). Computational complexity We will now present a detailed analysis of the computational demands of the proposed method as a function of the number of variables, denoted as 𝑑, and the number of samples, 𝑁, pertaining to these variables. In the worst case, the proposed approach necessitates, for each of the 𝑑 variables: for 𝑖 = 1, . . . , 𝑑 − 1, regressing a function mapping 𝑑 − 𝑖 variables to the variable of interest and performing a mode decomposition, as exemplified in (7.8), to identify the variable with the minimal contribution to the signal. Since these two steps have the same cost, it follows that, in the worst case, the total computational complexity of the proposed method is O (d2 N3 ) which corresponds to product of the number of double-looping operations, 𝑑 2 , and the cost of kernel regression from 𝑁 samples which, without acceleration, is 𝑁 3 (i.e., the cost of inverting a 𝑁 × 𝑁 dense kernel matrix). However, if kernel scalability techniques are utilized, such as when the kernel has a rank 𝑘 (for example, 𝑘 = 𝑑 if the kernel is linear) or is approximated by a kernel of rank 𝑘 (e.g., via a random feature map), then this worst-case bound can be reduced to O (d2 Nk2 ) by reducing the complexity of each regression step from O (𝑁 3 ) to O (𝑁 𝑘 2 ). Note that the statistical accuracy of the proposed approach requires that 𝑁 > 𝑑 if the dependence of the unknown functions on their inputs is not sparse. Moreover, in the absence of kernel scalability techniques, the worst-case memory footprint of the method is O (𝑁 2 ) due to the necessity of handling dense kernel matrices. However, once the functional ancestors of each variable are determined, these matrices can be discarded. Consequently, only one such matrix needs to be retained in memory at any given time. Results The following examples and experiments illustrate the proposed approach. The Fermi-Pasta-Ulam-Tsingou system The Fermi-Pasta-Ulam-Tsingou (FPUT) system (Palais, 1997) is a prototypical chaotic dynamical system. It is composed of 𝑀 masses indexed by 𝑗 ∈ {0, . . . , 𝑀−1} with equilibrium position 𝑗 ℎ with ℎ = 1/𝑀. Each mass is tethered to its two adjacent masses by a nonlinear spring, and the displacement of the mass 𝑥 𝑗 adheres to the 263 Figure 7.3: (a-d) The Fermi-Pasta-Ulam-Tsingou system. (e-k) The Google Covid 19 open data. equation:
𝑐2 (𝑥 𝑗+1 + 𝑥 𝑗−1 − 2𝑥 𝑗 ) 1 + 𝛼(𝑥 𝑗+1 − 𝑥 𝑗−1 ) , (7.9) 2 ℎ where 𝛼(𝑥) = 𝑥 2 , 𝑐 = 1 and 𝑀 = 10. We use fixed boundary conditions by adding two more masses, with 𝑥−1 = 𝑥 𝑀 = 0. We take a total of 1000 snapshots from multiple trajectories and the observed variables are the positions, velocities, and accelerations of all the underlying masses. In the graph discovery phase, every other node is initially deemed a potential ancestor for a specified node of interest. We then proceed to iteratively remove the node with the least signal contribution. The step resulting in the largest surge in the noise-to-signal ratio is inferred as one eliminating a crucial ancestor, thereby pinpointing the final ancestor set. Fig. 7.3.(c) shows a plot V (𝑛) of the noise-to-signal ratio V (𝑠)+V (𝑛) (𝑞) as a function of the number 𝑞 of proposed ancestors for the variable 𝑥¥7 and with 𝑍-test quantiles (in the absence of signal, the noise-to-signal ratio should fall within the shaded area with probability 0.9). Removing a node essential to the equation of interest causes the noise-to-signal ratio to markedly jump from approximately 25% to 99%. Fig. 7.3.(d) shows a plot of V (𝑛) V (𝑛) the noise-to-signal ratio increments V (𝑠)+V (𝑛) (𝑞) − V (𝑠)+V (𝑛) (𝑞 − 1) as a function of the number 𝑞 of ancestors for the variable 𝑥¥7 . Note that the increase in the noise-tosignal ratio is significantly higher compared to previous removals when an essential node was removed. Therefore, while solely relying on a fixed threshold to decide 𝑥¥ 𝑗 = 264 when to cease the removals might prove challenging, evaluating the increments in noise-to-signal ratios offers a clear guideline for efficiently and reliably pruning ancestors. The recovered full graph, depicted in Fig. 7.3.(a), is remarkably accurate despite the nonlinear nature of the model and the fact that our prior only encodes that the nonlinearity is smooth. Therefore, our algorithm does not require a dictionary or extensive knowledge of the structure of the unknown functions. Notably, velocity variables are accurately identified as non-essential and omitted from the ancestors of position and acceleration variables. Fig. 7.3.(b), which omits velocity variables for clarity, further elucidates the accurate recovery of dependencies. The dependencies are the simplest and clearest possible. They match exactly those of the original equations except for the boundary particles for which we recover valid equivalent equations. The Google Covid 19 open data. Consider the COVID-19 data from Google2 . We focus on a single country, France, to ensure consistency in the data and avoid considering cross-border variations that are not directly reflected in the data. We select 31 variables that describe the state of the country during the pandemic, spanning over 500 data points, with each data point corresponding to a single day. These variables are categorized as the following datasets: (1) Epidemiology dataset: Includes quantities such as new infections, cumulative deaths, etc. (2) Hospital dataset: Provides information on the number of admitted patients, patients in intensive care, etc. (3) Vaccine dataset: Indicates the number of vaccinated individuals, etc. (4) Policy dataset: Consists of indicators related to government responses, such as school closures or lockdown measures, etc. Some of these variables are illustrated in Fig. 7.3.(e). The problem is then to analyze this data and identify possible hidden functional relations between these variables. Fig. 7.3.(f) shows the noise-to-signal ratio (and its increments) as function of the number of ancestors of the “cumulative number of hospitalized patients” variable. Even for this real dataset, the proposed approach gives a clear signal for stopping the pruning process. Fig. 7.3.(g) shows the full recovered graph, which is highly clustered. Fig. 7.3.(h) shows the cluster corresponding to the variable “schools closing” revealing that the government either implemented multiple restrictive measures simultaneously or lifted them in unison (except for mask mandates that were on the verge of being identified as noise). The vaccination cluster (Fig. 7.3.(j)) reveals a linear relationship between variables (signaling redundant information) and the 2 The dataset can be accessed here 265 hospitalization cluster (Fig. 7.3.(i)) reveals a nonlinear one. Eliminating redundant nodes leads to the sparse graph shown in Fig. 7.3.(k), which is interpretable and amenable to (both quantitative and qualitative) analysis, Figure 7.4: (a-c) Chemical reaction network. (d-g) Algebraic equations. (h-j) Cell signaling network. Chemical reaction network. In this example, we consider the recovery of a chemical reaction network from concentration snapshots. The reaction network, illustrated in Fig. 7.4.(a) is that of the hydrogenation of ethylene (C2 H4 ) into ethane (C2 H6 ). The problem is that of recovering the underlying chemical reaction network from snapshots (illustrated in Fig. 7.4.(b)) of concentrations [𝐻2 ], [𝐻], [𝐶2 𝐻4 ], and [𝐶2 𝐻5 ] and their time 𝑑 [𝐻] 𝑑 [𝐶2 𝐻4 ] 2] , and 𝑑 [𝐶𝑑𝑡2 𝐻5 ] . The proposed approach leads to a derivatives. 𝑑 [𝐻 𝑑𝑡 , 𝑑𝑡 , 𝑑𝑡 perfect recovery of the computational graph (shown in Fig. 7.4.(c)) and a correct identification of quadratic functional dependencies between variables. Algebraic equations. Fig. 7.4.(a-d) illustrate the application of the proposed approach to the recovery of functional dependencies from data satisfying hidden algebraic equations. In all these examples, we have 𝑑 = 6 or 𝑑 = 7 variables and 𝑁 = 1000 samples from those variables. For 𝑑 = 6 the variables are 𝑤 1 , 𝑤 2 , 𝑤 3 , 𝑤 4 , 𝑥1 , 𝑥2 . For 𝑑 = 7 the variables are 𝑤 1 , 𝑤 2 , 𝑤 3 , 𝑤 4 , 𝑥1 , 𝑥2 , 𝑥3 . The samples from the variables 𝑤 1 to 𝑤 4 are i.i.d. N(0, 1) random variables, and the samples from 𝑥 1 , 𝑥 2 (and 𝑥 3 for 𝑑 = 7) are functionally dependent on the other variables. In the first example, 𝑑 = 6 and the samples from 𝑥1 and 𝑥2 satisfy the equations 𝑥 1 = 𝑤 1 and 𝑥2 = 𝑤 2 . The algorithm selects the linear kernel and Fig. 7.4.(a) shows the recovered graph (which is exact). In the second example, 𝑑 = 7 and the samples from 𝑥 1 , 𝑥2 and 𝑥3 satisfy the equations 𝑥 1 = 𝑤 1 , 𝑥 2 = 𝑥 12 +1+0.1𝑤 2 , and 𝑥 3 = 𝑤 3 . The algorithm selects the quadratic kernel 266 and Fig. 7.4.(b) shows the recovered graph (which is exact). Even though 𝑥 2 can trace back its origin to either 𝑥 1 and 𝑤 2 or 𝑤 1 and 𝑤 2 , the algorithm recognizes 𝑥 1 , 𝑤 1 , and 𝑤 2 as its ancestors underscoring the importance of eliminating redundant variables when aiming at deriving the sparsest graph. In the third example, 𝑑 = 6 and the samples from 𝑥1 and 𝑥2 satisfy the equations 𝑥 1 = 𝑤 1 𝑤 2 and 𝑥 2 = 𝑤 2 sin(𝑤 4 ). The algorithm selects the nonlinear kernel and Fig. 7.4.(c) shows the recovered graph (which is exact). In the fourth example, 𝑑 = 7 and the samples from 𝑥1 , 𝑥2 and 𝑥3 satisfy the equations 𝑥 1 = 𝑤 1 , 𝑥 2 = 𝑥 13 + 1 + 0.1𝑤 2 and 𝑥 3 = (𝑥1 + 2) 3 + 0.1𝑤 3 . Although these equations appear to be cubic, the algorithm correctly selects the quadratic kernel and makes an exact recovery of the graph shown in Fig. 7.4. (d) revealing hidden quadratic dependencies between variables. Cell signaling network Next, we apply the proposed framework to the example illustrated in Fig. 7.1.(l) from (Sachs et al., 2005) and discover a hierarchy of functional dependencies in biological cellular signaling networks. We use single-cell data consisting of the 𝑑 = 11 phosphoproteins and phospholipids levels in the human immune system T-cells that were measured using flow cytometry. This dataset was studied from a probabilistic modeling perspective in previous works. While (Sachs et al., 2005) learned a directed acyclic graph to encode causal dependencies, (Friedman, Hastie, and Tibshirani, 2008) learned an undirected graph of conditional independencies between the 𝑑 molecule levels by assuming the underlying data follows a multivariate Gaussian distribution. The latter analysis encodes acyclic dependencies but does not identify directions. In this work, we aim to identify the functional dependencies without imposing strong distributional assumptions on the data. We simply use 𝑁 = 2, 000 samples chosen uniformly at random from the dataset consisting of 11 proteins and 7446 samples of their expressions. We apply the algorithm in two stages. The first stage of the algorithm uses only linear and quadratic kernels and recovers the graph shown in Fig. 7.4.(h). It consists of four disconnected clusters where the molecule levels in each cluster are closely related by linear or quadratic dependencies (all connections are linear except for the connection between Akt and PKA, which is quadratic). These edges match a subset of the edges found in the gold standard model identified in (Sachs et al., 2005). With perfect noiseless dependencies, one can define constraints that reduce the total number of variables in the system. Second, we learn the connections between groups of variables within each cluster with nonlinear kernels and obtain the graph shown in Fig. 7.4.(i) in which 267 solid arrows indicate strong intra-cluster connections identified in the first level, and dashed lines indicate weaker connections between nodes and clusters identified in the second level. The width and grayscale intensities of each edge correspond to its signal-to-noise ratio. We emphasize that while causal graph recovery methods rely on the control of the sampling of the underlying variables (i.e., the simultaneous measurement of multiple phosphorylated protein and phospholipid components in thousands of individual primary human immune system cells, and perturbing these cells with molecular interventions), the reconstruction obtained by our method did not use this information and recovered functional dependencies rather than causal dependencies. Interestingly, the information recovered through our method appears to complement and enhance the findings presented in (Sachs et al., 2005) (e.g., the linear and noiseless dependencies between variables in the JNK cluster is not something that could easily be inferred from the graph produced in (Sachs et al., 2005) shown in Fig. 7.1.(j) where we have colored the clusters for comparison). Comparisons. Using the expected graph reported in (Sachs et al., 2005) as the ground truth (acknowledging that it may not be entirely accurate), we compare the edges our approach incrementally added to the true graph. Figure 7.5.(a) reports the number of additional edges that have been added and are not present in the ground truth (false positives) and edges removed that are present in the ground truth graph (false negatives). The added edges are based on the two-stage procedure described above, where we first add the ten intra-cluster connections, followed by inter-cluster connections. Edges are added in decreasing order of signal-to-noise ratio, starting with the strongest. In the reported results, we do not account for the recovery of the direction of ground-truth edges. We note that, up to direction, all intra-cluster connections, along with the inter-cluster connections with the strongest signals are found in the ground truth graph, leading to the initial decrease in false negatives with only one false positive edge (the linear connection P38 → Jnk that is not reported in the true graph). With the addition of the remaining (possibly non-spurious) edges, the number of false negatives drops to one, having recovered all edges, except for the one between PKC and Raf, which is identified to be statistically non-informative in our approach. A large-scale chemical reaction network: the BCR reaction benchmark Lastly, we stress-test the scalability of our approach by applying it to a large-scale chemical reaction network: the BCR reaction benchmark from (Loman et al., 2023), which encompasses 1122 species. The dataset comprises 2,400 snapshots of species 268 Figure 7.5: (a) Cell signaling network comparisons. (b-h) The BCR reaction benchmark. concentrations and their corresponding time derivatives. We leveraged JAX’s inherent parallelization capabilities (Bradbury et al., 2018) to accelerate our computations, allowing for the simultaneous pruning of multiple nodes while abstracting the complexity of parallel execution. While the scaling with respect to the number of data points is straightforward, scaling with the number of variables introduces a trade-off between computational speed and memory footprint. Specifically, the process of identifying the ancestors of various nodes can be expedited by storing a large array for all nodes. Using a DGX workstation equipped with four Nvidia V100 GPUs, each with 32GB of memory, pruning 190 nodes took approximately three days, projecting a total experiment duration of around one month. Nonetheless, we can mitigate this computational burden by optimizing the computation of terms of the form 𝑦𝑇 𝐾 𝑦 for the specific quadratic kernel identified for this example. We include the details of such optimization in the supplementary material. By implementing this optimization, the duration of the entire experiment was reduced to just one hour. In the first experiment, we simulated five trajectories of the associated system of ODEs, recording 1000 snapshots per trajectory. Out of these 5000 snapshots, 2,400 were randomly selected as training data, and 2600 as testing data. Writing TP, TN, FP and FN for True/False Positives/Negatives and using the metrics True Positive Rate (TPR=TP/(TP+FN)), False Positive Rate (FPR=FP/(FP+TN)), and False Discovery Rate (FDR=FP/(TP+FP)), we observed a TPR of 39.9%, an FPR of 16.4%, and an FDR of 97.2% (indicating that 97.2% of predicted positives are false). This high FDR can be attributed to the limited exploration of the full variable range—1,122 in total—by the five trajectories. The trajectories explored a subset of the possible space 269 (near a limit cycle attractor), which led to the recovery of functional dependencies that represent both the chemical reactions and the specific subspace visited. Furthermore, with 1,122 variables, the 630,003 coefficients of the underlying quadratic equations are vastly under-determined with only 2,400 data points. Despite the high FDR in the recovered graph, as illustrated in Fig. 7.5.(b-c), the CHD pruning process vastly improves the accuracy (by orders of magnitude) of the estimated functions on the 2600 unseen snapshots by reducing the dimension of the regression problem whenever possible. We denote 𝑦𝑖 as an observed data point, 𝜎 2 as the variance of the observed data, b 𝑦𝑖 for a predicted data point without pruning, and 𝑦¯ 𝑖 for a predicted data point post-pruning. Fig. 7.5.(b) illustrates the histogram of the log-normalized
𝑦𝑖 | 2 /𝜎 2 and squared errors before and after pruning, expressed as log10 |𝑦𝑖 − b
log10 |𝑦𝑖 − 𝑦¯ 𝑖 | 2 /𝜎 2 . The 99th percentile of the normalized squared error is less than 10−2 for all species. Fig. 7.5.(c) displays the histogram of the log-normalized
squared error improvements due to pruning, calculated as log10 |𝑦𝑖 −b 𝑦𝑖 | 2 /|𝑦𝑖 − 𝑦¯ 𝑖 | 2 . Fig. 7.5.(d-e) display the quantiles of the histograms post-pruning, conditioned on the noise-to-signal ratio observed at the final pruning step. These plots reveal a clear trend: a higher noise-to-signal ratio at the time of pruning correlates with increased error and diminished improvements in accuracy. In a second experiment, we formed the data by randomly sampling concentrations uniformly in [0, 1] (independently across species and snapshots) and recorded the resulting time derivatives. While this sampling increased the variability of the 2,400 snapshots, the model remained vastly underdetermined. The noise-to-signal and bootstrapped (Z-test) ratios remained close to 0.5, suggesting insufficient data for statistically significant variable importance assessments. Nonetheless, as depicted in Fig. 7.5.(f-h), significant insights can still be gleaned from the activations, showing notable improvements when comparing the histograms of the values of TPR, FPR, and FDR obtained with pruning based on these ratios and pruning at random. This analysis reveals that even with high dimensionality and scarce data, between 10% and 80% of the true ancestors can still be accurately identified. Discussions Limitations In its present form, the proposed approach is limited by several factors. (1) Without access to the sampling of the data, the direction of some edges may not be identifiable. For instance the functional relationship 𝑥 − 2𝑦 = 0 can be represented as both 𝑦 = 2𝑥 (𝑥 → 𝑦) and 𝑥 = 𝑦/2 (𝑦 → 𝑥). (2) It assumes an additive noise 𝑊 270 on the functional relationship 𝑦 = 𝑓 (𝑥) + 𝑊 between the variables 𝑥 and 𝑦. In a fully probabilistic setting, this structure may be non-additive, i.e., of the form 𝑦 = 𝑓 (𝑥, 𝑊), which implies discovering a general transition kernel, i.e., a nonGaussian generative model. Although our method achieves polynomial complexity, in settings where one has access to the distribution of the data, the price to pay, when compared with information-theoretic methods, is a reduction in generality imposed by the stronger assumption made on the data-generating process. Furthermore, the price to pay for the weaker data requirements (i.e., the absence of interventional data) is that our method recovers functional relationships rather than causal ones or conditional dependencies. (3) If the (noisy) functional relationship 𝑦 = 𝑓 (𝑥) + 𝑊 is associated with a non-regular (e.g., discontinuous) function 𝑓 then the kernels discussed above (linear, quadratic, and fully nonlinear) will be misspecified and may lead to false negatives. The kernel selection and hyperparameter tuning problems in misspecified settings require further work. (4) As demonstrated in the BCR reaction application, while the method scales well computationally with an increase in the number of variables, it may still be impacted by the curse of dimensionality. This occurs particularly if the dataset only covers a limited subset of the full range of variable values. Given the results displayed in Fig. 7.5.(b-h) we suspect that this impact could be mitigated by adopting more advanced strategies in place of our current top-down pruning method. Such strategies could involve grouping variables and integrating both top-down and bottom-up iterative approaches. Conclusions We have developed a comprehensive Gaussian Process framework for solving Type 3 (hypergraph discovery) problems, which is interpretable and amenable to analysis. The breadth and complexity of Type 3 problems significantly surpass those encountered in Type 2 (hypergraph completion), and the initial numerical examples we present serve as a motivation for the scope of Type 3 problems and the broader applications made possible by this approach. Our proposed algorithm is designed to be fully autonomous, yet it offers the flexibility for manual adjustments to refine the graph’s structure recovery. We emphasize that our proposed approach is not intended to supplant causal inference methods (Pearl, 2009); see Methods for a complete overview. Instead, it aims to incorporate a distinct kind of information into the graph’s structure, namely, the functional dependencies among variables rather than their causal relationships. Additionally, our method eliminates the need for a predetermined ordering of variables, a common requirement in acyclic probabilistic 271 models where determining an optimal order is an NP-hard problem usually tackled using heuristic approaches. Furthermore, our approach can actually be utilized to generate such an ordering by quantifying the strength of the connections it recovers. The Uncertainty Quantification properties of the underlying Gaussian Processes are integral to the method and could also be employed to quantify uncertainties in the structure of the recovered graph. We also observe that forming clusters from highly interdependent variables helps to obtain a sparser graph. Additionally, the precision of the pruning process is enhanced by avoiding the division of node activation within the cluster among its separate constituents. We employed this strategy in the recovery of the gene expression graph in Fig. 7.4.(i). Given the polynomial complexity of our method, promising avenues for future work include applications to large datasets in genomics and in systems biology, particularly in the reconstruction and intervention of metabolic pathways. These applications benefit from the ability to handle large-scale datasets efficiently, enabling the analysis of complex biological networks. Data availability The data in the paper and the Supplementary Information are available in the Github repository of the paper. Code availability The code for the algorithm and its application to various examples are available for download (and as as an installable python library/package) in the Github repository of the paper. Online content Supplementary Information is available for this paper. Acknowledgements HO, TB, PB, XY, and RB acknowledge support from the Air Force Office of Scientific Research under MURI award number FA9550-20-1-0358 (Machine Learning and Physics-Based Modeling and Simulation). Additionally, HO, TB, and PB acknowledge support by the Department of Energy under award number DESC0023163 (SEA-CROGS: Scalable, Efficient and Accelerated Causal Reasoning Operators, Graphs and Spikes for Earth and Embedded Systems) and by the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration. HO and PB further acknowledge 272 support by Beyond Limits (Learning Optimal Models) through CAST (The Caltech Center for Autonomous Systems and Technologies). TB acknowledges support from the Kortschak Scholar Fellowship Program. NR acknowledges support from the JPL Researchers on Campus (JROC) program. HO is grateful for the Department of Defense Vannevar Bush Faculty Fellowship. We are grateful to two referees for their insightful comments and valuable suggestions. 273 Supplementary information This supplementary document provides an overview of refinements and generalizations on our proposed approach (Sec. 7.1) detailed in subsequent sections. It includes a summary of the principal components of our algorithm (Sec. 7.2). It includes a reminder on Type 2 problems (Sec. 7.3) and their common GP-based solutions. It discusses the hardness of Type 3 problems, presents an overview of causal inference methods, and a well-posed formulation of Type 3 problems (Sec. 7.4). Additionally, this document offers an in-depth description of our developed GP-based solution specifically designed for Type 3 problems (Section 7.5), along with the corresponding algorithmic pseudo-codes (Section 7.6). It also includes an analysis of the signal-to-noise ratio (SNR) test that is integral to our method (Section 7.7), and furnishes supplementary details concerning the examples discussed in the main manuscript (Section 7.8). 7.1 Additional details on our proposed approach. The efficacy of our proposed approach is enhanced through a series of refinements (implemented in all our examples), which are summarized below and detailed in sections 7.5, 7.6, and 7.7. Ancestor pruning. As discussed earlier, rather than using a threshold on the signal-to-noise ratio to prune ancestors, we order the ancestors in decreasing contribution to the signal, the final number 𝑞 of ancestors is determined as the maximizer of noise to signal ratio V (𝑛) V (𝑛) increment V (𝑠)+V (𝑛) (𝑞 + 1) − V (𝑠)+V (𝑛) (𝑞). Figure 7.6: Histogram of the eigenvalues of 𝐷 𝛾 =(7.11) for 𝛾 = 10−2 (good choice) and 𝛾 = 10−6 (bad choice). Parameter Selection. The choice of the parameter 𝛾 in (7.2) is a critical aspect of our proposed approach. We provide a structured approach for selecting 𝛾 based on the characteristics of 274 the kernel matrix 𝐾 𝑠 . Specifically, when 𝐾 𝑠 is derived from a finite-dimensional feature map 𝜓 (i.e., when 𝐾 𝑠 (𝑥, 𝑥 ′) := 𝜓(𝑥)𝑇 𝜓(𝑥 ′) where the range of 𝜓 is finitedimensional) and the data cannot be interpolated exactly with 𝐾 𝑠 (the dimension of the range of 𝜓 is smaller than the number of data points), we employ the regression residual to determine 𝛾 as follows: 2 𝛾 = min 𝑣𝑇 𝜓(𝑋) − 𝑌 R 𝑁 . (7.10) 𝑣 Write 𝐾 𝑠 (𝑋, 𝑋) for the 𝑁 × 𝑁 matrix with entries 𝐾 𝑠 (𝑋𝑖 , 𝑋 𝑗 ). Alternatively, when the data can be interpolated exactly with 𝐾 𝑠 (e.g., when 𝐾 𝑠 is a universal kernel), we select 𝛾 (see Fig. 7.6) by maximizing the variance of the eigenvalue histogram of the 𝑁 × 𝑁 matrix
−1 𝐷 𝛾 := 𝛾 𝐾 𝑠 (𝑋, 𝑋) + 𝛾𝐼 , (7.11) whose eigenvalues are bounded between 0 and 1 and converge towards 0 as 𝛾 ↓ 0 and towards 1 as 𝛾 ↑ ∞. We can also select 𝛾 as the median of the eigenvalues of 𝐷𝛾. Z-test quantiles. V (𝑛) The noise-to-signal ratio V (𝑠)+V (𝑛) associated with (7.2) admits the representer 𝑌 𝑇 𝐷2 𝑌 formula 𝑌 𝑇 𝐷 𝛾𝑌 . Therefore if the data is only composed of noise (if 𝑌 ∼ 𝜎 2 𝑍 where 𝛾 𝑍 is a random vector with i.i.d. N (0, 1) entries), then the distribution of the noiseto-signal ratio follows that of the random variable 𝐵 := 𝑍 𝑇 𝐷 2𝛾 𝑍 . 𝑍𝑇 𝐷𝛾 𝑍 (7.12) Therefore, the quantiles of 𝐵 can be used as an interval of confidence on the noiseto-signal ratio if 𝑌 ∼ 𝜎 2 𝑍. Fig. 7.3.(c) shows these 𝑍-test quantiles (in the absence of signal, the noise-to-signal ratio should fall within the shaded area with probability 0.9). Generalizations on our proposed approach. Complexity Reduction with Kernel PCA Variant. Write 𝐾 for the kernel associated with the RKHS H in Problem 4. We use a variant of Kernel PCA (Mika et al., 1998) to significantly reduces the computational complexity of our proposed method, making it primarily dependent on the number of principal nonlinear components in the kernel matrix 𝐾 (𝑋, 𝑋) (the 𝑁 × 𝑁 matrix with entries 𝐾 (𝑋𝑖 , 𝑋 𝑗 )) rather than the number of data points. To describe this 275 write 𝜆 1 ≥ · · · ≥ 𝜆𝑟 > 0 for the nonzero eigenvalues of 𝐾 (𝑋, 𝑋) indexed in decreasing order and write 𝛼·,𝑖 for the corresponding unit-normalized eigenvectors, i.e. 𝐾 (𝑋, 𝑋)𝛼·,𝑖 = 𝜆𝑖 𝛼·,𝑖 . Then | 𝑓 (𝑋)| 2 = | 𝑓 (𝜙)| 2 , where 𝑓 (𝜙) is the 𝑟 vector with Í𝑁 entries 𝑓 (𝜙𝑖 ) := 𝑠=1 𝑓 (𝑋𝑠 )𝛼𝑠,𝑖 . Furthermore, writing 𝑟 ′ ≤ 𝑟 for the smallest index 𝑖 such that 𝜆𝑖 /𝜆 1 < 𝜖 where 𝜖 > 0 is some small threshold, the complexity of the problem can be further reduced (as in PCA) by truncating 𝑓 (𝜙) to 𝑓 (𝜙′) = ( 𝑓 (𝜙1 ), . . . , 𝑓 (𝜙𝑟 ′ )) and approximating F with the space of functions 𝑓 ∈ H such that | 𝑓 (𝜙′)| 2 ≈ 0. Generalizing Descendants and Ancestors with Kernel Mode Decomposition. We can extend the concept of descendants and ancestors to cover more complex functional dependencies between variables, including implicit ones. This generalization is achieved through a Kernel-based adaptation of Row Echelon Form Reduction (REFR), initially designed for affine systems, and leveraging the principles of Kernel Mode Decomposition (Houman Owhadi, Clint Scovel, and Yoo, 2021). To describe the connection with REFR consider the example in which M is the manifold of R3 defined by the affine equations 𝑥1 + 𝑥2 + 3𝑥 3 − 2 = 0 and 𝑥1 − 𝑥 2 + 𝑥 3 = 0, which is equivalent to selecting F = span{ 𝑓1 , 𝑓2 } with 𝑓1 (𝑥) = 𝑥 1 + 𝑥2 + 3𝑥 3 − 2 and 𝑓2 (𝑥) = 𝑥 1 − 𝑥 2 + 𝑥3 in the problem formulation 4. Then, irrespective of how we recover the manifold from data, the hypergraph representation of that manifold is equivalent to the row echelon form reduction of the affine system, and this representation and this reduction require a possibly arbitrary choice of free and dependent variables. So, for instance, if we declare 𝑥 3 to be the free variables and 𝑥 1 and 𝑥2 to be the dependent variables, then we can represent the manifold via the equations 𝑥 1 = 1 − 2𝑥 3 and 𝑥 2 = 1 − 𝑥 3 which have the hypergraph representation depicted in Fig. 7.11.(b). To describe the kernel generalization of REFR assume that the kernel 𝐾 can be decomposed as the additive kernel 𝐾 = 𝐾𝑎 + 𝐾 𝑠 + 𝐾 𝑧 , (7.13) and write H𝑎 , H𝑠 , and H𝑧 for the RKHS induced by the kernels 𝐾𝑎 , 𝐾 𝑠 , 𝐾 𝑧 . Then a function 𝑓 ∈ H can be decomposed as 𝑓 = 𝑓𝑎 + 𝑓𝑠 + 𝑓𝑧 with ( 𝑓𝑎 , 𝑓𝑠 , 𝑓𝑧 ) ∈ H𝑎 × H𝑠 × H𝑧 . Then, generalizing REFR we can approximate the manifold M via a manifold parametrized by equations of the form 𝑓𝑎 + 𝑓 𝑠 + 𝑓 𝑧 = 0 ⇔ 𝑔𝑎 = 𝑓 𝑠 , (7.14) where 𝑓𝑎 = −𝑔𝑎 and 𝑔𝑎 is a given function in H𝑎 representing a dependent mode, 𝑓𝑧 = 0 represents a zero mode, and 𝑓𝑠 ∈ H𝑠 is identified (regularized) as the 276 minimizer of the following variational problem min ∥ 𝑓𝑠 ∥ 2𝐾𝑠 + 𝑓𝑠 ∈H𝑠 1 2 (−𝑔𝑎 + 𝑓𝑠 )(𝜙) . 𝛾 (7.15) Taking 𝑔𝑎 (𝑥) = 𝑥1 and H𝑠 + H𝑧 to be a space of functions that does not depend on 𝑥 1 recovers our initial example (7.1) (with the pruning process encoded into the selection of H𝑧 ). This generalization is motivated by its potential to recover implicit equations. For example, consider the implicit equation 𝑥 12 + 𝑥 22 = 1, which can be retrieved by setting the mode of interest to be 𝑔𝑎 (𝑥) = 𝑥 12 and allowing 𝑓𝑠 to depend only on the variable 𝑥 2 . 7.2 Algorithm Overview for Type 3 problems: An Informal Summary In this section, we provide an accessible overview of our algorithm’s key components, which are further detailed in Algorithms 11 and 12 in Section 7.6. Our method focuses on determining the edges within a hypergraph. To achieve this, we consider each node individually, finding its ancestors and establishing edges from these ancestors to the node in question. While we present the algorithm for a single node, it can be applied iteratively to all nodes within the graph. Algorithm for finding the ancestors of a node: 1. Initialization: We start by assuming that all other nodes are potential ancestors of the current node. 2. Selecting a Kernel: We choose a kernel function, such as linear, quadratic, or fully nonlinear kernels (refer to Example 7.5.4). The kernel selection process is analogous to the subsequent pruning steps, involving the determination of a parameter 𝛾, regression analysis, and evaluation based on signal-to-noise ratios. • Kernel Selection Method: The choice of kernel follows a process similar to the subsequent pruning steps, including 𝛾 selection, regression analysis, and signal-to-noise ratio evaluation. • Low Signal-to-Noise Ratio for All Kernels: If the signal-to-noise ratio is insufficient for all possible kernels, the algorithm terminates, indicating that the node has no ancestors. 3. Pruning Process: While there are potential ancestors left to consider (details in Section 7.5): 277 a) Identify the Least Important Ancestor: Ancestors are ranked based on their contribution to the signal (see Sec. 7.5). b) Noise prior: Determine the value of 𝛾 (see Section 7.7). c) Regression Analysis: Predict the node’s value using the current set of ancestors, excluding the least active one (i.e., the one contributing the least to the signal). We employ Kernel Ridge Regression with the selected kernel function and parameter 𝛾 (see Sec. 7.5 and 7.5). d) Evaluate Removal: Compute the regression signal-to-noise ratio (see Sec. 7.5 and 7.7): • Low Signal-to-Noise Ratio: If the signal-to-noise ratio falls below a certain threshold, terminate the algorithm and return the current set of ancestors (see Section 7.5). • Adequate Signal-to-Noise Ratio: If the signal-to-noise ratio is sufficient, remove the least active ancestor and continue the pruning process. Figure 7.7: Formal description of Type 2 problems. 7.3 Type 2 problems: Formal description and GP-based Computational Graph Completion Formal description of Type 2 problems Consider a computational graph (as illustrated in Fig. 7.7.(a)) where nodes represent variables and edges are directed and they represent functions. These functions may be known or unknown. In Fig. 7.7.(a), edges associated with unknown functions ( 𝑓5,1 , 𝑓1,2 , 𝑓3,6 ) are colored in red, and those associated with known functions ( 𝑓2,5 ) are colored in black. Round nodes are utilized to symbolize variables, which are derived from the concatenation of other variables (e.g, in Fig. 7.7.(a), 𝑥 3 = (𝑥2 , 𝑥4 )). Therefore, the underlying graph is, in fact, a hypergraph where functions may map groups of variables to other groups of variables, and we use round nodes to 278 illustrate the grouping step. Given partial observations derived from 𝑁 samples of the graph’s variables, we introduce a problem, termed a Type 2 problem, focused on approximating all unobserved variables and unknown functions. Using Fig. 7.7.(a)(b) as an illustration we call a vector (𝑋𝑠,1 , . . . , 𝑋𝑠,6 ) a sample from the graph if its entries are variables satisfying the functional dependencies imposed by the structure of the graph (i.e., 𝑋𝑠,1 = 𝑓5,1 (𝑋𝑠,5 ), 𝑋𝑠,2 = 𝑓1,2 (𝑋2,𝑠 ), 𝑋𝑠,3 = (𝑋𝑠,2 , 𝑋𝑠,4 ), 𝑋𝑠,5 = 𝑓𝑠,5 (𝑋𝑠,𝑠 ), and 𝑋𝑠,6 = 𝑓3,6 (𝑋𝑠,3 ). These samples can be seen as the rows of given matrix 𝑋 illustrated in Fig. 7.7.(b) for 𝑁 = 3. By partial observations, we mean that only a subset of the entries of each row may be observed, as illustrated in Fig. 7.7.(b)-(c). Note that a Type 2 problem combines a regression problem (approximating the unknown functions of the graph) with a matrix completion/data imputation problem (approximating the unobserved entries of the matrix 𝑋). Reminder on Computational Graph Completion for Type 2 problems Within the context of Sec. 7.3, the proposed GP solution to Type 2 problems is to simply replace unknown functions by GPs and compute their Maximum A Posteriori (MAP)/Maximum Likelihood Estimation (MLE) estimators given available data and constraints imposed by the structure of the graph. Taking into account the example depicted in Fig. 7.7, and substituting 𝑓5,1 , 𝑓1,2 , and 𝑓3,6 with independent GPs, each with kernels 𝐾, 𝐺, and Γ respectively, the objective of this MAP solution becomes 2 + ∥ 𝑓 ∥ 2 (writing ∥ 𝑓 ∥ for the RKHS norm of 𝑓 minimizing ∥ 𝑓5,1 ∥ 2𝐾 + ∥ 𝑓1,2 ∥ 𝐺 𝐾 3,6 Γ induced by the kernel 𝐾) subject to the constraints imposed by the data and the functional dependencies encoded into the structure of the graph. A system identification example. In order to exemplify Computational Graphical Completion (CGC), consider the system identification problem depicted in Fig. 7.8, sourced from (Houman Owhadi, 2022). Our objective is to identify a nonlinear electric circuit, as illustrated in Fig. 7.8.(a), from scarce measurement data. The nonlinearity of the circuit emanates from the resistance, capacitance, and inductances, which are nonlinear functions of currents and voltages, as shown in Fig. 7.8.(b). Assuming these functions to be unknown, along with all currents and voltages as unknown time-dependent functions, we operate the circuit between times 0 and 10. Measurements of a subset of variables, representing the system’s state, are taken at times 𝑡 𝑠 = 𝑠/10 for 𝑠 ∈ 0, . . . , 99. Given these measurements, the challenge arises in approximating all unknown functions that define currents and voltages as time functions, capacitance as a voltage function, 279 Figure 7.8: (a) Electric circuit. (b) Resistance, capacitance, and inductances are nonlinear functions of currents and voltages (c) Measurements. (d) Kirchhoff’s circuit laws. (e) The computational graph with unknown functions represented as red edges. (f) Recovered functions. and inductances and resistance as current functions. Fig. 7.8.(c) displays the available measurements, which are notably sparse, preventing us from reconstructing the underlying unknown functions independently. Thus, their interdependencies must be utilized for approximation. It is crucial to note that the system’s state variables are interconnected through functional relations, as per Kirchhoff’s laws for this nonlinear electric circuit, illustrated in Fig. 7.8.(d). These functional dependencies can be conceptualized as a computational graph, depicted in Fig. 7.8.(e), where nodes represent variables and directed edges represent functions. Known functions are colored in black, unknown functions in red, and round nodes aggregate variables, meaning edges map groups of variables, forming a hypergraph. The CGC solution involves substituting the graph’s unknown functions with Gaussian Processes (GPs), which may be independent or correlated, and then approximating the unknown functions with their Maximum A Posteriori (MAP) estimators, given the available data and the functional dependencies embedded in the graph’s structure. Fig. 7.8.(f) showcases the true and recovered functions, demonstrating a notably accurate approximation 280 despite the data’s scarcity. This simple example generalizes to an abstract framework detailed in (Houman Owhadi, 2022). This framework has a wide range of applications because most problems in CSE can also be formulated as completing computational graphs representing dependencies between functions and variables, and they can be solved in a similar manner by replacing unknown functions with GPs and by computing their MAP/EB estimator given the data. These problems include those illustrated in Fig. 7.1.(d-h). 7.4 Hardness and well-posed formulation of Type 3 problems. In this subsection, we describe why Type 3 problems are challenging and why they can even be intractable if not formalized and approached properly. Curse of combinatorial complexity. First, the problem suffers from the curse of combinatorial complexity in the sense that the number of hypergraphs associated with 𝑁 nodes blows up rapidly with 𝑁. As an illustration, Fig. 7.9 shows some of the hypergraphs associated with only three nodes. A lower bound on that number is the A003180 sequence, which answers the following question (Ishihara, 2001): given 𝑁 unlabeled vertices, how many different hypergraphs in total can be realized on them by counting the equivalent hypergraphs only once? For 𝑁 = 8, this lower bound is ≈ 2.78 × 1073 . Figure 7.9: Computational Hypergraph Discovery with three variables Nonidentifiability and implicit dependencies. Secondly, it is important to note that, even with an infinite amount of data, the exact structure of the hypergraph might not be identifiable. To illustrate this point, let’s consider a problem where we have 𝑁 samples from a computational graph with variables 𝑥 and 𝑦. The task is to determine the direction of functional dependency 𝑓 𝑦 ), or from 𝑦 to 𝑥 between 𝑥 and 𝑦. Does it go from 𝑥 to 𝑦 (represented as 𝑥 𝑓 𝑥 )? (represented as 𝑦 If we refer to Fig. 7.10.(a), we can make a decision because 𝑦 can only be expressed 281 as a function of 𝑥. In contrast, if we examine Fig. 7.10.(b), the decision is also straightforward because 𝑥 can solely be written as a function of 𝑦. However, if the data mirrors the scenario in Fig. 7.10.(c), it becomes challenging to decide as we can write both 𝑦 as a function of 𝑥 and 𝑥 as a function of 𝑦. Further complicating matters is the possibility of implicit dependencies between variables. As illustrated in Fig. 7.10.(d), there might be instances where neither 𝑦 can be derived as a function of 𝑥, nor 𝑥 can be represented as a function of 𝑦. Figure 7.10: The structure of the hypergraph is identifiable in (a), (b), and non-identifiable in (c). The relationship between variables is implicit in (d). Causal inference and probabilistic graphs. Causal inference methods broadly consist of two approaches: constraint and scorebased methods. While constraint-based approaches are asymptotically consistent, they only learn the graph up to an equivalence class (Spirtes and C. Glymour, 1991). Instead, score-based methods resolve ambiguities in the graph’s edges by evaluating the likelihood of the observed data for each graphical model. For instance, they may assign a higher evidence to 𝑦 → 𝑥 over 𝑥 → 𝑦 if the conditional distribution 𝑥|𝑦 exhibits less complexity than 𝑦|𝑥. The complexity of searching over all possible graphs, however, grows super-exponentially with the number of variables. Thus, it is often necessary to use approximate, but more tractable, search-based methods (Chickering, 2002; J. Peters, Janzing, and Schölkopf, 2017) or alternative criteria based on sensitivity analysis (Data et al., 2016). For example, the preference could lean towards 𝑦 → 𝑥 rather than 𝑥 → 𝑦 if 𝑦 demonstrates less sensitivity to errors or perturbations in 𝑥. In contrast, our proposed GP method avoids the growth in complexity by performing a guided pruning process that assesses the contribution of each node to the signal. We also emphasize that our method is not limited to learning acyclic graph structures as it can identify feedback loops between variables. Alternatively, methods for learning probabilistic undirected graphical models, also known as Markov networks, identify the graph structure by assuming the data is 282 randomly drawn from some probability distribution (Drton and Maathuis, 2017). In this case, edges in the graph (or lack thereof) encode conditional dependencies between the nodes. A common approach learns the graph structure by modeling the data as being drawn from a multivariate Gaussian distribution with a sparse inverse covariance matrix, whose zero entries indicate pairwise conditional independencies (Friedman, Hastie, and Tibshirani, 2008). Recently, this approach has been extended using models for non-Gaussian distributions, e.g., in (Baptista et al., 2021; Ren et al., 2021), as well as kernel-based conditional independence tests (K. Zhang et al., 2011). In this work, we learn functional dependencies rather than causality or probabilistic dependence. We emphasize that we also do not assume the data is randomized or impose strong assumptions, such as additive noise models, in the data-generating process. We complete this paragraph by comparing the hypergraph discovery framework to structure learning for Bayesian networks and structural equation models (SEM). Let 𝑥 ∈ R𝑑 be a random variable with probability density function 𝑝 that follows Î𝑑 the autoregressive factorization 𝑝(𝑥) = 𝑖=1 𝑝𝑖 (𝑥𝑖 |𝑥 1 , . . . , 𝑥𝑖−1 ) given a prescribed variable ordering. Structure learning for Bayesian networks aims to find the ancestors of variable 𝑥𝑖 , often referred to as the set of parents 𝑃𝑎(𝑖) ⊆ {1, . . . , 𝑖 − 1}, in the sense that 𝑝𝑖 (𝑥𝑖 |𝑥1 , . . . , 𝑥𝑖−1 ) = 𝑝𝑖 (𝑥𝑖 |𝑥 𝑃𝑎(𝑖) ). Thus, the variable dependence of the conditional density 𝑝𝑖 is identified by finding the parent set so that 𝑥𝑖 is conditionally independent of all remaining preceding variables given its parents, i.e., 𝑥𝑖 ⊥ 𝑥1:𝑖−1\𝑃𝑎(𝑖) |𝑥 𝑃𝑎(𝑖) . Finding ancestors that satisfy this condition requires performing conditional independence tests, which are computationally expensive for general distributions (SHAH and PETERS, 2020). Alternatively, SEMs assume that each variable 𝑥𝑖 is drawn as a function of its ancestors with additive noise, i.e. 𝑥𝑖 = 𝑓 (𝑥 𝑃𝑎(𝑖) ) +𝜖𝑖 for some function 𝑓 and noise 𝜖 (J. Peters, Janzing, and Schölkopf, 2017). For Gaussian noise 𝜖𝑖 ∼ N (0, 𝜎 2 ), each marginal conditional distribution in a Bayesian network is given by 𝑝𝑖 (𝑥𝑖 |𝑥1:𝑖−1 ) ∝ exp(− 2𝜎1 2 ∥𝑥𝑖 − 𝑓 (𝑥 1:𝑖−1 )∥ 2 ). Thus, finding the parents for such a model by maximum likelihood estimation corresponds to finding the parents that minimize the expected mean-squared error ∥𝑥𝑖 − 𝑓 (𝑥 𝑃𝑎(𝑖) )∥ 2 . Our approach minimizes a related objective, without imposing the strong probabilistic assumptions that are required in SEMs and Bayesian Networks. We also observe that while the graph structure identified in Bayesian networks is influenced by the specific sequence in which variables are arranged (a concept exploited in numerical linear algebra (Florian Schäfer, Timothy John Sullivan, and Houman Owhadi, 2021c; Florian Schäfer and Houman Owhadi, 2021) where Schur complementation 283 is equivalent to conditioning GPs and a carefully ordering leads to the accuracy of the Vecchia approximation 𝑝𝑖 (𝑥𝑖 |𝑥 1 , . . . , 𝑥𝑖−1 ) ≈ 𝑝𝑖 (𝑥𝑖 |𝑥𝑖−𝑘 , . . . , 𝑥𝑖−1 ) (Vecchia, 1988)), the graph recovered by our approach remains unaffected by any predetermined ordering of those variables. Figure 7.11: (a) CHD formulation as a manifold discovery problem and hypergraph representation, (b) The hypergraph representation of an affine manifold is equivalent to its Row Echelon Form Reduction. Well-posed formulation of the problem. In this paper, we focus on a formulation of the problem that remains well-posed even when the data is not randomized, i.e., we formulate the problem as the following manifold learning/discovery problem. Problem 4. Let H be a Reproducing Kernel Hilbert Space (RKHS) of functions mapping R𝑑 to R. Let F be a closed linear subspace of H and let M be a subset of R𝑑 such that 𝑥 ∈ M if and only if 𝑓 (𝑥) = 0 for all 𝑓 ∈ F . Given the (possibly noisy and nonrandom) observation of 𝑁 elements, 𝑋1 , . . . , 𝑋𝑁 , of M approximate M. To understand why problem 4 serves as the appropriate formulation for hypergraph discovery, consider a manifold M ⊂ R𝑑 . Suppose this manifold can be represented by a set of equations, expressed as a collection of functions ( 𝑓 𝑘 ) 𝑘 satisfying ∀𝑥 ∈ M, 𝑓 𝑘 (𝑥) = 0. To keep the problem tractable, we assume a certain level of regularity for these functions, necessitating they belong to a RKHS H , ensuring the applicability of kernel methods for our framework. Given that any linear combination of the 𝑓 𝑘 will also be evaluated to zero on M, the relevant functions are those within the span of the 𝑓 𝑘 , forming a closed linear subspace of H denoted as F . The manifold M can be subsequently represented by a graph or hypergraph (see Fig. 7.11.(a)), whose ambiguity can be resolved through a deliberate decision to classify some variables as free and others as dependent. This selection could be arbitrary, informed by expert knowledge, or derived from probabilistic models or sensitivity analysis. 284 7.5 A Gaussian Process method for Type 3 problems Affine case and Row Echelon Form Reduction. To describe the proposed solution to Problem 4, we start with a simple example. In this example H is a space of affine functions 𝑓 of the form ! 1 𝑓 (𝑥) = 𝑣𝑇 𝜓(𝑥) with 𝜓(𝑥) := and 𝑣 ∈ R𝑑+1 . (7.16) 𝑥 As a particular instantiation (see Fig. 7.11.(b)), we assume M to be the manifold of R3 (𝑑 = 3) defined by the affine equations (    𝑥 1 + 𝑥 2 + 3𝑥3 − 2  3 M= 𝑥∈R   𝑥1 − 𝑥2 + 𝑥3  =0 ) , (7.17) =0 which is equivalent to selecting F = span{ 𝑓1 , 𝑓2 } with 𝑓1 (𝑥) = 𝑥 1 + 𝑥2 + 3𝑥3 − 2 and 𝑓2 (𝑥) = 𝑥 1 − 𝑥 2 + 𝑥3 in the problem formulation 4. Then, irrespective of how we recover the manifold from data, the hypergraph representation of that manifold is equivalent to the row echelon form reduction of the affine system, and this representation and this reduction require a possibly arbitrary choice of free and dependent variables. So, for instance, for the system (7.17), if we declare 𝑥 3 to be the free variables and 𝑥1 and 𝑥2 to be the dependent variables, then we can represent the manifold via the equations (    𝑥1  3 M= 𝑥∈R   𝑥2  = 1 − 2𝑥 3 ) , (7.18) = 1 − 𝑥3 which have the hypergraph representation depicted in Fig. 7.11.(b). Now, in the 𝑁 > 𝑑 regime where the number of data points is larger than the number of variables, the manifold can simply be approximated via a variant of PCA. Take 𝑓 ∗ ∈ F , we have 𝑓 ∗ (𝑥) = 𝑣 ∗𝑇 𝜓(𝑥) for a certain 𝑣 ∗ ∈ R𝑑+1 . Then for 𝑋𝑠 ∈ M, 𝑓 ∗ (𝑋𝑠 ) = 𝜓(𝑋𝑠 )𝑇 𝑣 ∗ = 0. Defining 𝐶𝑁 := 𝑁 ∑︁ 𝜓(𝑋𝑠 )𝜓(𝑋𝑠 )𝑇 (7.19) 𝑠=1 we see that 𝑓 ∗ (𝑋𝑠 ) = 0 for all 𝑋𝑠 is equivalent to 𝐶𝑁 𝑣 ∗ = 0. Since 𝑁 > 𝑑, we can thus identify F exactly as {𝑣𝑇 𝜓 for 𝑣 ∈ 𝐾𝑒𝑟 (𝐶𝑁 )}. We then obtain the manifold  M 𝑁 = 𝑥 ∈ R𝑑 | 𝑣𝑇 𝜓(𝑥) = 0 for 𝑣 ∈ Span(𝑣 𝑟+1 , . . . , 𝑣 𝑑+1 ) , (7.20) 285 where Span(𝑣 𝑟+1 , . . . , 𝑣 𝑑+1 ) is the zero-eigenspace of 𝐶𝑁 . Here we write 𝜆 1 ≥ · · · ≥ 𝜆𝑟 > 0 = 𝜆𝑟+1 = · · · = 𝜆 𝑑+1 for the eigenvalues of 𝐶𝑁 (in decreasing order), and 𝑣 1 , . . . , 𝑣 𝑑+1 for the corresponding eigenvectors (𝐶𝑁 𝑣 𝑖 = 𝜆𝑖 𝑣 𝑖 ). The proposed approach extends to the noisy case (when the data points are perturbations of elements of the manifold) by simply replacing the zero-eigenspace of the covariance matrix by the linear span of the eigenvectors associated with eigenvalues that are smaller than some threshold 𝜖 > 0, i.e., by approximating M with (7.20) where 𝑟 is such that 𝜆 1 ≥ · · · ≥ 𝜆𝑟 ≥ 𝜖 > 𝜆𝑟+1 ≥ · · · ≥ 𝜆 𝑑+1 . In this affine setting (7.20) allows us to estimate M directly without RKHS norm minimization/regularization because linear regression does not require regularization in the sufficiently large data regime. Furthermore the process of pruning ancestors can be replaced by that of identifying sparse elements 𝑣 ∈ Span(𝑣 𝑟+1 , . . . , 𝑣 𝑑+1 ) such that 𝑣 𝑖 = 1. Figure 7.12: Feature map generalization Feature map generalization. This simple approach can be generalized by generalizing the underlying feature map 𝜓 used to define the space of functions (writing 𝑑S for the dimension of the range of 𝜓)  H = 𝑓 (𝑥) = 𝑣𝑇 𝜓(𝑥) | 𝑣 ∈ R𝑑 S . (7.21) For instance, if we use the feature map 𝜓(𝑥) := 1, . . . , 𝑥𝑖 , . . . , 𝑥𝑖 𝑥 𝑗 , . . .
𝑇 then H becomes a space of quadratic polynomials on R𝑑 , i.e., n o ∑︁ ∑︁ H = 𝑓 (𝑥) = 𝑣 0 + 𝑣 𝑖 𝑥𝑖 + 𝑣 𝑖, 𝑗 𝑥𝑖 𝑥 𝑗 | 𝑣 ∈ R𝑑 S , 𝑖 (7.22) (7.23) 𝑖≤ 𝑗 and, in the large data regime (𝑁 > 𝑑S ), identifying quadratic dependencies between variables becomes equivalent to (1) adding nodes to the hypergraph corresponding to secondary variables obtained from primary variables 𝑥𝑖 through known functions (for (7.22), these secondary variables are the quadratic monomials 𝑥𝑖 𝑥 𝑗 , see Fig. 7.12.(a)), 286 and (2) identifying affine dependencies between the variables of the augmented hypergraph. The problem can, therefore, be reduced to the previous affine case. Indeed, as in the affine case, the manifold can then be approximated in the regime where the number of data points is larger than the dimension 𝑑S of the feature map by (7.20), where 𝑣 𝑟 , . . . , 𝑣 𝑁 are the eigenvectors of 𝐶𝑁 =(7.19) whose eigenvalues are zero (noiseless case) or smaller than some threshold 𝜖 > 0 (noisy case). Furthermore, the hypergraph representation of the manifold is equivalent to a feature map generalization of Row Echelon Form Reduction to nonlinear systems of equations. For instance, choosing 𝑥 3 as the dependent variable and 𝑥 1 , 𝑥2 as the free variables, M = {𝑥 ∈ R3 | 𝑥3 − 5𝑥 12 + 𝑥 22 − 𝑥 1 𝑥2 = 0} can be represented as in Fig. 7.12.(b) where the round node represents the concatenated variable (𝑥1 , 𝑥2 ) and the red arrow represents a quadratic function. The generalization also enables the representation of implicit equations by selecting secondary variables as free variables. For instance, selecting 𝑥 32 as the free variable and 𝑥1 , 𝑥2 as the free variables, M = {𝑥 ∈ R3 | 𝑥 12 + 𝑥 22 + 𝑥 32 − 1 = 0} can be represented as in Fig. 7.12.(c). Kernel generalization and regularization. This feature-map extension of the previously discussed affine case can evidently be generalized to arbitrary degree polynomials and to other basis functions. However, as the dimension 𝑑S of the range of the feature map 𝜓 increases beyond the number 𝑁 of data points, the problem becomes underdetermined: the data only provides partial information about the manifold, i.e., it is not sufficient to uniquely determine the manifold. Furthermore, if the dimension of the feature map is infinite, then we are always in that low data regime, and we have the additional difficulty that we cannot directly compute with that feature map. On the other hand, if 𝑑S is finite (i.e., if the dictionary of basis functions is finite), then some elements of F (some constraints defining the manifold M) may not be representable or well approximated as equations of the form 𝑣𝑇 𝜓(𝑥) = 0. To address these conflicting requirements, we need to kernelize and regularize the proposed approach (as done in interpolation). The kernel associated with the feature map. To describe this kernelization, we assume that the feature map 𝜓 maps R𝑑 to some Hilbert space S that could be infinite-dimensional, and we write 𝐾 for the kernel defined by that feature map. To be precise, we now consider the setting where the feature map 𝜓 is a function from R𝑑 to a (possibly infinite-dimensional separable) 287 Hilbert (feature) space S endowed with the inner product ·, · S . To simplify notations, we will still write 𝑣𝑇 𝑤 for 𝑣, 𝑤 S and 𝑣𝑤𝑇 for the linear operator mapping 𝑣 ′ to 𝑣 𝑤, 𝑣′ S . Let H := {𝑣𝑇 𝜓(𝑥) | 𝑣 ∈ S} (7.24) be the space of functions mapping R𝑑 to R defined by the feature map 𝜓. To avoid ambiguity, assume (without loss of generality) that the identity 𝑣𝑇 𝜓(𝑥) = 𝑤𝑇 𝜓(𝑥) holds for all 𝑥 ∈ R𝑑 if and only if 𝑣 = 𝑤. It follows that for 𝑓 ∈ H there exists a unique 𝑣 ∈ S such that 𝑓 = 𝑣𝑇 𝜓. For 𝑓 , 𝑔 ∈ H with 𝑓 = 𝑣𝑇 𝜓 and 𝑔 = 𝑤𝑇 𝜓, we can then define 𝑓 , 𝑔 H := 𝑣𝑇 𝑤 . (7.25) Observe that H is a Hilbert space endowed with the inner product ·, · H . For 𝑥, 𝑥 ′ ∈ X, write 𝐾 (𝑥, 𝑥 ′) := 𝜓(𝑥)𝑇 𝜓(𝑥 ′) , (7.26) for the kernel defined by 𝜓 and observe that (H , ·, · H ) is the RKHS defined by the kernel 𝐾 (which is assumed to contain F in Problem 4). Observe in particular that for 𝑓 = 𝑣𝑇 𝜓 ∈ H , 𝐾 satisfies the reproducing property 𝑓 , 𝐾 (𝑥, ·) H = 𝑣𝑇 𝜓(𝑥) = 𝑓 (𝑥) . (7.27) Complexity Reduction with Kernel PCA Variant. We will now show that the previous feature-map PCA variant (characterizing the subspace of 𝑓 ∈ H such that 𝑓 (𝑋) = 0) can be kernelized as a variant of kernel PCA (Mika et al., 1998). To describe this write 𝐾 (𝑋, 𝑋) for the 𝑁 × 𝑁 matrix with entries 𝐾 (𝑋𝑖 , 𝑋 𝑗 ). Write 𝜆 1 ≥ 𝜆 2 ≥ · · · ≥ 𝜆𝑟 > 0 for the nonzero eigenvalues of 𝐾 (𝑋, 𝑋) indexed in decreasing order and write 𝛼·,𝑖 for the corresponding unit-normalized eigenvectors, i.e. 𝐾 (𝑋, 𝑋)𝛼·,𝑖 = 𝜆𝑖 𝛼·,𝑖 and |𝛼·,𝑖 | = 1 . (7.28) Write 𝑓 (𝑋) for the 𝑁 vector with entries 𝑓 (𝑋𝑠 ). For 𝑖 ≤ 𝑟, write 𝜙𝑖 := 𝑁 ∑︁ δ 𝑋𝑠 𝛼𝑠,𝑖 (7.29) 𝑓 (𝑋𝑠 )𝛼𝑠,𝑖 . (7.30) 𝑠=1 and 𝑓 (𝜙𝑖 ) := 𝑁 ∑︁ 𝑠=1 Write 𝑓 (𝜙) for the 𝑟 vector with entries 𝑓 (𝜙𝑖 ). Then, we have the following proposition. 288 Proposition 7.5.1. The subspace of functions 𝑓 ∈ H such that 𝑓 (𝜙) = 0 is equal to the subspace of 𝑓 ∈ H such that 𝑓 (𝑋) = 0. Furthermore for 𝑓 ∈ H with feature map representation 𝑓 = 𝑣𝑇 𝜓 with 𝑣 ∈ S we have the identity (where 𝐶𝑁 =(7.19)) 𝑣𝑇 𝐶𝑁 𝑣 = | 𝑓 (𝜙)| 2 = | 𝑓 (𝑋)| 2 . (7.31) b1 ≥ 𝜆 b2 ≥ · · · ≥ 𝜆 bb𝑟 > 0 for the nonzero eigenvalues of 𝐶𝑁 =(7.19) Proof. Write 𝜆 indexed in decreasing order. Write 𝑣 1 , . . . , 𝑣 𝑟 for the corresponding eigenvectors, i.e., b𝑖 𝑣 𝑖 . 𝐶𝑁 𝑣𝑖 = 𝜆 (7.32) Observing that 𝐶𝑁 = 𝑟 ∑︁ b𝑖 𝑣 𝑖 𝑣𝑇𝑖 𝜆 (7.33) 𝑖=1 we deduce that the zero-eigenspace of 𝐶𝑁 is the set of vectors 𝑣 ∈ S such that 𝑣𝑇 𝑣 𝑖 = 0 for 𝑖 = 1, . . . , 𝑟. Write 𝑓𝑖 := 𝑣𝑇𝑖 𝜓. Observe that for 𝑓 = 𝑣𝑇 𝜓, we have 𝑣𝑇𝑖 𝑣 = 𝑓𝑖 , 𝑓 𝐾 . Multiplying (7.32) by 𝜓𝑇 (𝑥) implies 𝑁 ∑︁ b𝑖 𝑓𝑖 (𝑥) 𝐾 (𝑥, 𝑋𝑠 ) 𝑓𝑖 (𝑋𝑠 ) = 𝜆 (7.34) 𝑠=1 (7.34) implies that for 𝑓 = 𝑣𝑇 𝜓 𝑣𝑇𝑖 𝑣 = 𝑁 ∑︁ b𝑖−1 𝑓𝑖 (𝑋𝑠 ) 𝐾 (·, 𝑋𝑠 ), 𝑓 𝜆 = 𝐾 𝑠=1 𝑁 ∑︁ b𝑖−1 𝑓𝑖 (𝑋𝑠 ) 𝑓 (𝑋𝑠 ), 𝜆 (7.35) 𝑠=1 where we have used the reproducing property (7.27) of 𝐾 in the last identity. Write b 𝛼𝑠,𝑖 := 𝜆𝑖−1/2 𝑓𝑖 (𝑋𝑠 ) . (7.36) Using (7.34) with 𝑥 = 𝑋𝑠′ implies that b 𝛼·,𝑖 is an eigenvector of the 𝑁 × 𝑁 matrix b𝑖 . Taking 𝑓 = 𝑓𝑖 in (7.35) implies that 1 = 𝑣𝑇 𝑣 𝑖 = |b 𝐾 (𝑋, 𝑋) with eigenvalue 𝜆 𝛼·,𝑖 | 2 . 𝑖 Therefore, the b 𝛼·,𝑖 are unit-normalized. Summarizing, this analysis (closely related to the one found in kernel PCA (Mika et al., 1998)) shows that the nonzero eigenvalues of 𝐾 (𝑋, 𝑋) coincide with those of 𝐶𝑁 and we have b 𝑟 = 𝑟, 𝜆b𝑖 = 𝜆𝑖 and b 𝛼·,𝑖 = 𝛼·,𝑖 . Furthermore, (7.35) and (7.36) imply that for 𝑖 ≤ 𝑟, 𝑣 ∈ S and 𝑓 = 𝑣𝑇 𝜓, we have 𝑣𝑇𝑖 𝑣 = 𝜆𝑖−1/2 𝑓 (𝑋)𝛼·,𝑖 . The identity (7.37) then implies (7.31). (7.37) □ Remark 7.5.2. As in PCA the dimension/complexity of the problem can be further reduced by truncating 𝜙 to 𝜙′ = (𝜙1 , . . . , 𝜙𝑟 ′ ) where 𝑟 ′ ≤ 𝑟 is identified as the smallest index 𝑖 such that 𝜆𝑖 /𝜆 1 < 𝜖 where 𝜖 > 0 is some small threshold. 289 Kernel Mode Decomposition. When the feature map 𝜓 is infinite-dimensional, the data only provides partial information about the constraints defining the manifold in the sense that 𝑓 (𝑋) = 0 or equivalently 𝑓 (𝜙) = 0 is a necessary but not sufficient condition for the zero level set of 𝑓 to be a valid constraint for the manifold (for 𝑓 to be such that 𝑓 (𝑥) = 0 for all 𝑥 ∈ M). So we are faced with the following problems: (1) How to regularize? (2) How do we identify free and dependent variables? (3) How do we identify valid constraints for the manifold? The proposed solution will be based on the Kernel Mode Decomposition (KMD) framework introduced in (Houman Owhadi, Clint Scovel, and Yoo, 2021) (which shares conceptual foundations with Smoothing Spline ANOVA (Wahba, 2003)). Reminder on KMD We will now present a quick reminder on KMD in the setting of the following mode decomposition problem. So, in this problem, we have an unknown function 𝑓 † mapping some input space X to the real line R. We assume that this function can be written as a sum of 𝑚 other unknown functions 𝑓𝑖† which we will call modes, i.e., 𝑚 ∑︁ 𝑓† = 𝑓𝑖† . (7.38) 𝑖=1 We assume each mode 𝑓𝑖† to be an unknown element of some RKHS H𝐾𝑖 defined by some kernel 𝐾𝑖 . Then we consider the problem in which given the data 𝑓 † (𝑋) = 𝑌 (with (𝑋, 𝑌 ) ∈ X 𝑁 × R𝑁 ) we seek to approximate the 𝑚 modes composing the target function 𝑓 † . Then, we have the following theorem. Theorem 7.5.3. (Houman Owhadi, Clint Scovel, and Yoo, 2021) Using the relative Í𝑚 ∥ 𝑓𝑖 ∥ 2𝐾𝑖 as a loss, the minimax error in the product norm ∥( 𝑓1 , . . . , 𝑓𝑚 )∥ 2 := 𝑖=1 optimal recovery of ( 𝑓1† , . . . , 𝑓𝑚† ) is ( 𝑓1 , . . . , 𝑓𝑚 ) with 𝑓𝑖 (𝑥) = 𝐾𝑖 (𝑥, 𝑋)𝐾 (𝑋, 𝑋) −1𝑌 , where 𝐾 is the additive kernel 𝐾= 𝑚 ∑︁ 𝐾𝑖 . (7.39) (7.40) 𝑖=1 The GP interpretation of this optimal recovery result is as follows. Let 𝜉𝑖 ∼ N (0, 𝐾𝑖 ) be 𝑚 independent centered GPs with kernels 𝐾𝑖 . Write 𝜉 for the additive GP 𝜉 := Í𝑚 † 𝑖=1 𝜉𝑖 . (7.39) can be recovered by replacing the modes 𝑓𝑖 by independent centered 290 GPs 𝜉𝑖 ∼ N (0, 𝐾𝑖 ) with kernels 𝐾𝑖 and approximating the mode 𝑖 by conditioning Í𝑚 𝜉𝑖 on the available data 𝜉 (𝑋) = 𝑌 where 𝜉 := 𝑖=1 𝜉𝑖 is the additive GP obtained by summing the independent GPs 𝜉𝑖 , i.e.,   𝑓𝑖 (𝑥) = E 𝜉𝑖 (𝑥) | 𝜉 (𝑋) = 𝑌 . (7.41) Furthermore ( 𝑓1 , . . . , 𝑓𝑚 ) can also be identified as the minimizer of    Minimize     over      s. t.  2 𝑖=1 ∥ 𝑓𝑖 ∥ 𝐾𝑖 Í𝑚 ( 𝑓1 , . . . , 𝑓𝑚 ) ∈ H𝐾1 × · · · × H𝐾𝑚 Í𝑚 ( 𝑖=1 𝑓𝑖 )(𝑋) = 𝑌 . (7.42) The variational formulation (7.42) can be interpreted as a generalization of Tikhonov regularization which can be recovered by selecting 𝑚 = 2, 𝐾1 to be a smoothing kernel (such as a Matérn kernel) and 𝐾2 (𝑥, 𝑦) = 𝜎 2 δ(𝑥 − 𝑦) to be a white noise kernel. Now, this abstract KMD approach (Houman Owhadi, Clint Scovel, and Yoo, 2021) is associated with a quantification of how much each mode contributes to the overall data or how much each individual GP 𝜉𝑖 explains the data. More precisely, the activation of the mode 𝑖 or GP 𝜉𝑖 can be quantified as 𝑝(𝑖) = ∥ 𝑓𝑖 ∥ 2𝐾𝑖 ∥ 𝑓 ∥ 2𝐾 (7.43) , Í𝑚 Í𝑚 where 𝑓 = 𝑖=1 𝑓𝑖 . These activations 𝑝(𝑖) satisfy 𝑝(𝑖) ∈ [0, 1] and 𝑖=1 𝑝(𝑖) = 1 they can be thought of as a generalization of Sobol sensitivity indices (Sobol, 2001; Soboĺ, 1993; Owen, 2013) to the nonlinear setting in the sense that they are associated with the following variance representation/decomposition (Houman Owhadi, Clint Scovel, and Yoo, 2021) (writing ·, · 𝐾 for the RKHS inner product induced by 𝐾): Var  𝜉, 𝑓 𝐾  = ∥ 𝑓 ∥ 2𝐾 = 𝑚 ∑︁ 𝑖=1 ∥ 𝑓𝑖 ∥ 2𝐾𝑖 = 𝑚 ∑︁ Var   𝜉𝑖 , 𝑓 𝐾 . (7.44) 𝑖=1 Application to CHD, general case. Now, let us return to our original manifold approximation problem 4 in the kernelized setting of (7.26). Given the data 𝑋 we cannot regress an element 𝑓 ∈ F directly since the minimizer of ∥ 𝑓 ∥ 2𝐾 + 𝛾 −1 ∥ 𝑓 (𝑋)∥ 2R 𝑁 is the null function. To identify the functions 𝑓 ∈ F , we need to decompose them into modes that can be interpreted as a generalization of the notion 291 of free and dependent variables. To describe this, assume that the kernel 𝐾 can be decomposed as the additive kernel 𝐾 = 𝐾𝑎 + 𝐾 𝑠 + 𝐾 𝑧 . (7.45) Then H𝐾 = H𝐾 𝑎 + H𝐾𝑠 + H𝐾 𝑧 implies that for all function 𝑓 ∈ H𝐾 , 𝑓 can be decomposed as 𝑓 = 𝑓𝑎 + 𝑓𝑠 + 𝑓𝑧 with ( 𝑓𝑎 , 𝑓𝑠 , 𝑓𝑧 ) ∈ H𝑎 × H𝑠 × H𝑧 . Example 7.5.4. As a running example, take 𝐾 to be the following additive kernel ∑︁ ∑︁ Ö 𝐾 (𝑥, 𝑥 ′) = 1 + 𝛽1 𝑥𝑖 𝑥𝑖′ + 𝛽2 𝑥𝑖 𝑥 𝑗 𝑥𝑖′𝑥 ′𝑗 + 𝛽3 (1 + 𝑘 (𝑥𝑖 , 𝑥𝑖′)) , (7.46) 𝑖 𝑖≤ 𝑗 𝑖 that is the sum of a linear kernel, a quadratic kernel, and a fully nonlinear kernel. Take 𝐾𝑎 to be the part of the linear kernel that depends only on 𝑥 1 , i.e., 𝐾𝑎 (𝑥, 𝑥 ′) = 𝛽1 𝑥 1 𝑥 1′ . (7.47) Take 𝐾 𝑠 to be the part of the kernel that does not depend on 𝑥 1 , i.e., ∑︁ ∑︁ Ö 𝐾 𝑠 = 1 + 𝛽1 𝑥𝑖 𝑥𝑖′ + 𝛽2 𝑥𝑖 𝑥 𝑗 𝑥𝑖′𝑥 ′𝑗 + 𝛽3 (1 + 𝑘 (𝑥𝑖 , 𝑥𝑖′)) . 𝑖≠1 𝑖≤ 𝑗,𝑖, 𝑗≠1 (7.48) 𝑖≠1 And take 𝐾 𝑧 to be the remaining portion, 𝐾 𝑧 = 𝐾 − 𝐾𝑎 − 𝐾 𝑠 . (7.49) Therefore the following questions are equivalent: • Given a function 𝑔𝑎 in the RKHS H𝐾 𝑎 defined by the kernel 𝐾𝑎 is there a function 𝑓𝑠 in the RKHS H𝐾𝑠 defined by the kernel 𝐾 𝑠 such that 𝑔𝑎 (𝑥) ≈ 𝑓𝑠 (𝑥) for 𝑥 ∈ M? • Given a function 𝑔𝑎 ∈ H𝐾 𝑎 is there a function 𝑓 in the RKHS H𝐾 defined by the kernel 𝐾 such that 𝑓 (𝑥) ≈ 0 for 𝑥 ∈ M and such that its 𝑓𝑎 mode is −𝑔𝑎 and its 𝑓𝑧 mode is zero? Then, the natural answer to the questions is to identify the modes of the constraint 𝑓 = 𝑓𝑎 + 𝑓𝑠 + 𝑓𝑧 ∈ H (such that 𝑓 (𝑥) ≈ 0 for 𝑥 ∈ M ) such that 𝑓𝑎 = −𝑔𝑎 and 𝑓𝑧 = 0 by selecting 𝑓𝑠 to be the minimizer of the following variational problem min ∥ 𝑓𝑠 ∥ 2𝐾𝑠 + 𝑓𝑠 ∈H𝑠 1 2 (−𝑔𝑎 + 𝑓𝑠 )(𝜙) . 𝛾 (7.50) 292 This is equivalent to introducing the additive GP 𝜉 = 𝜉𝑎 + 𝜉 𝑠 + 𝜉 𝑧 + 𝜉𝑛 whose modes are the independent GPs 𝜉𝑎 ∼ N (0, 𝐾𝑎 ), 𝜉 𝑠 ∼ N (0, 𝐾 𝑠 ), 𝜉 𝑧 ∼ N (0, 𝐾 𝑧 ), 𝜉𝑛 ∼ N (0, 𝛾δ(𝑥 − 𝑦)) (we use the label “n” in reference to “noise”), and then recovering 𝑓𝑠 as   𝑓𝑠 = E 𝜉 𝑠 | 𝜉 (𝑋) = 0, 𝜉𝑎 = −𝑔𝑎 , 𝜉 𝑧 = 0 . (7.51) Application to CHD, particular case. Taking 𝑔𝑎 (𝑥) = 𝑥 1 for our running example 7.5.4, the previous questions are, as illustrated in Fig. 7.2(b), equivalent to asking whether there exists a function 𝑓𝑠 ∈ H𝐾𝑠 that does not depend on 𝑥1 (since 𝐾 𝑠 does not depend on 𝑥 1 ) such that 𝑥1 ≈ 𝑓𝑠 (𝑥 2 , . . . , 𝑥 𝑑 ) for 𝑥 ∈ M . (7.52) Therefore, the mode 𝑓𝑎 can be thought of as a dependent mode (we use the label “a” in reference to “ancestors”), the mode 𝑓𝑠 as a free mode (we use the label “s” in reference to “signal”), the mode 𝑓𝑧 as a zero mode. While our numerical illustrations have primarily focused on the scenario where 𝑔𝑎 takes the form of 𝑔𝑎 (𝑥) = 𝑥𝑖 , and we aim to express 𝑥𝑖 as a function of other variables, the generality of our framework is motivated by its potential to recover implicit equations. For example, consider the implicit equation 𝑥12 + 𝑥 22 = 1, which can be retrieved by setting the mode of interest to be 𝑔𝑎 (𝑥) = 𝑥 12 and allowing 𝑓𝑠 to depend only on the variable 𝑥 2 . Signal-to-noise ratio. Now, we are led to the following question: since the mode 𝑓𝑠 (the minimizer of (7.50)) always exists and is always unique, how do we know that it leads to a valid constraint? To answer that question, we compute the activation of the GPs used to regress the data. We write V (𝑠) := ∥ 𝑓𝑠 ∥ 2𝐾𝑠 , (7.53) for the activation of the signal GP 𝜉 𝑠 and V (𝑛) := 1 2 (−𝑔𝑎 + 𝑓𝑠 )(𝑋) 𝛾 (7.54) for the activation of the noise GP 𝜉𝑛 , and then these allow us to define a signal-tonoise ratio defined as V (𝑠) . (7.55) V (𝑠) + V (𝑛) 293 Note that this corresponds to activation ratio of the noise GP defined in (7.43). This ratio can then be used to test the validity of the constraint in the sense that if 𝑉 (𝑠)/(𝑉 (𝑠) + 𝑉 (𝑛)) > 𝜏 (with 𝜏 = 0.5 as a prototypical example), then the data is mostly explained by the signal GP and the constraint is valid. If 𝑉 (𝑠)/(𝑉 (𝑠)+𝑉 (𝑛)) < 𝜏, then the data is mostly explained by the noise GP and the constraint is not valid. Iterating by removing the least active modes from the signal. If the constraint is valid, then we can next compute the activation of the modes composing the signal. To describe this, we assume that the kernel 𝐾 𝑠 can be decomposed as the additive kernel 𝐾 𝑠 = 𝐾 𝑠,1 + · · · + 𝐾 𝑠,𝑚 , (7.56) which results in H𝐾𝑠 = H𝐾𝑠,1 + · · · + H𝐾𝑠,𝑚 , which results in the fact that ∀ 𝑓𝑠 ∈ H𝑠 , 𝑓𝑠 can be decomposed as 𝑓𝑠 = 𝑓𝑠,1 + · · · + 𝑓𝑠,𝑚 , (7.57) with 𝑓𝑠,𝑖 ∈ H𝐾𝑠,𝑖 . The activation of the mode 𝑖 can then be quantified as 𝑝(𝑖) = Í𝑚 Í𝑚 ∥ 𝑓𝑠,𝑖 ∥ 2𝐾𝑠,𝑖 /∥ 𝑓𝑠 ∥ 2𝐾𝑠 , which combined with ∥ 𝑓𝑠 ∥ 2𝐾𝑠 = 𝑖=1 ∥ 𝑓𝑠,𝑖 ∥ 2𝐾𝑠,𝑖 leads to 𝑖=1 𝑝(𝑖) = 1. As our running example 7.5.4, we can decompose 𝐾 𝑠 =(7.48) as the sum of an affine kernel, a quadratic kernel, and a fully nonlinear kernel, i.e., 𝑚 = 3, 𝐾 𝑠,1 = Í Í Î 1 + 𝛽1 𝑖≠1 𝑥𝑖 𝑥𝑖′, 𝐾 𝑠,2 = 𝛽2 𝑖≤ 𝑗,𝑖, 𝑗≠1 𝑥𝑖 𝑥 𝑗 𝑥𝑖′𝑥 ′𝑗 and 𝐾 𝑠,3 = 𝛽3 𝑖≠1 (1 + 𝑘 (𝑥𝑖 , 𝑥𝑖′)). As another example for our running example, we can take 𝐾 𝑠 to be the sum of the portion of the kernel that does not depend on 𝑥 1 and 𝑥2 and the remaining portion, Í Í Î i.e., 𝑚 = 2, 𝐾 𝑠,1 = 1+ 𝛽1 𝑖≠1,2 𝑥𝑖 𝑥𝑖′ + 𝛽2 𝑖≤ 𝑗,𝑖, 𝑗≠1,2 𝑥𝑖 𝑥 𝑗 𝑥𝑖′𝑥 ′𝑗 + 𝛽3 𝑖≠1,2 (1+ 𝑘 (𝑥𝑖 , 𝑥𝑖′)) and 𝐾 𝑠,2 = 𝐾 𝑠 − 𝐾 𝑠,1 . Then, we can order these sub-modes from most active to least active and create a new kernel 𝐾 𝑠 by removing the least active modes from the signal and adding them to the mode that is set to be zero (see Fig. 7.13). To describe this, let 𝜋(1), · · · , 𝜋(𝑚) be an ordering of the modes by their activation, i.e., ∥ 𝑓𝑠,𝜋(1) ∥ 2𝐾𝑠, 𝜋 (1) ≥ ∥ 𝑓𝑠,𝜋(2) ∥ 2𝐾𝑠, 𝜋 (2) ≥ · · · . Í𝑚 Writing 𝐾𝑡 = 𝑖=𝑟+1 𝐾 𝑠,𝜋(𝑖) for the additive kernel obtained from the least active modes (with 𝑟 + 1 = 𝑚 as the value used for our numerical implementations), we update the kernels 𝐾 𝑠 and 𝐾 𝑧 by assigning the least active modes from 𝐾 𝑠 to 𝐾 𝑧 , i.e., 𝐾 𝑠 − 𝐾𝑡 → 𝐾 𝑠 and 𝐾 𝑧 + 𝐾𝑡 → 𝐾 𝑧 (we zero the least active modes). 294 Figure 7.13: Iterating by removing the least active modes from the signal Finally, we can iterate the process. This iteration can be thought of as identifying the structure of the hypergraph by placing too many hyperedges and removing them according to the activation of the underlying GPs. For our running example 7.5.4, where we try to identify the ancestors of the variable 𝑥 1 , if the sub-mode associated with the variable 𝑥 2 is found to be least active, then we can try to remove 𝑥 2 from the list of ancestors and try to identify 𝑥 1 as a function of 𝑥 3 to 𝑥 𝑑 . This is equivalent to selecting 𝐾𝑎 (𝑥, 𝑥 ′) = 𝛽1 𝑥1 𝑥 1′ , ∑︁ ∑︁ Ö 𝐾 𝑠/𝑡 = 1 + 𝛽1 𝑥𝑖 𝑥𝑖′ + 𝛽2 𝑥𝑖 𝑥 𝑗 𝑥𝑖′𝑥 ′𝑗 + 𝛽3 (1 + 𝑘 (𝑥𝑖 , 𝑥𝑖′)) , (7.58) 𝑖≠1,2 𝑖≤ 𝑗,𝑖, 𝑗≠1,2 𝑖≠1,2 and 𝐾 𝑧∪𝑡 = 𝐾 − 𝐾𝑎 − 𝐾 𝑠/𝑡 to assess whether there exists a function 𝑓𝑠 ∈ H𝐾 that does not depend on 𝑥 1 and 𝑥2 s.t. 𝑥 1 ≈ 𝑓𝑠 (𝑥 3 , . . . , 𝑥 𝑑 ) for 𝑥 ∈ M. Alternative determination of the list of ancestors. Our initial approach to determining the list of ancestors of a given node is to use a fixed threshold (e.g., 𝜏 = 0.5) to prune nodes. We propose a refined approach that mimics the strategy employed in Principal Component Analysis (PCA) for deciding which modes should be kept and which ones should be removed. The PCA approach is to order the modes in decreasing order of eigenvalues/variance and (1) either keep the smallest number modes holding/explaining a given fraction (e.g., 90%) of the variance in the data, (2) or use an inflection point/sharp drop in the decay of the eigenvalues to select which modes should be kept. Here, we propose a similar strategy. First we employ an alternative determination of the least active mode: we iteratively remove the mode that leads to the smallest increase in noise-to-signal ratio, i.e., we remove the mode 𝑡 such that 𝑡 = argmint V (𝑛) . V (𝑠/𝑡) + V (𝑛) (7.59) For our running example 7.5.4 in which we try to find the ancestors of the variable 𝑥 1 this is equivalent to removing the variables or node 𝑡 whose removal leads to 295 Figure 7.14: Computing the ancestors of the variable 𝑥¤0 in the Fermi-Pasta-UlamV (𝑛) Tsingou problem. (a) Noise-to-Signal Ratio, denoted as V (𝑠)+V (𝑛) (𝑞), with respect to the number of proposed ancestors, represented by 𝑞. Additionally, we include a visualization of the quantiles derived from the 𝑍-test, as described in Section 7.7. Notably, when there is no signal present, the noise-to-signal ratio is expected to fall within the shaded area with a probability of 0.9. (b) Increments in the NoiseV (𝑛) V (𝑛) to-Signal Ratio, defined as V (𝑠)+V (𝑛) (𝑞) − V (𝑠)+V (𝑛) (𝑞 − 1), as a function of the number of ancestors, denoted as 𝑞. The horizontal axis represents the number of proposed ancestors for 𝑥¤0 . Determining an appropriate stopping point based solely on absolute noise-to-signal ratio levels can be challenging. In contrast, the increments in the noise-to-signal ratio clearly exhibit a discernible maximum, offering a practical point for decision-making. the smallest loss in signal-to-noise ratio (or increase in noise-to-signal ratio) by selecting ∑︁ ∑︁ Ö 𝐾 𝑠/𝑡 = 1 + 𝛽1 𝑥𝑖 𝑥𝑖′ + 𝛽2 𝑥𝑖 𝑥 𝑗 𝑥𝑖′𝑥 ′𝑗 + 𝛽3 (1 + 𝑘 (𝑥𝑖 , 𝑥𝑖′)) . 𝑖≠1,𝑡 𝑖≤ 𝑗,𝑖, 𝑗≠1,𝑡 𝑖≠1,𝑡 Next, we iterate this process, and we plot (a) the noise-to-signal ratio, and (b) the increase in noise-to-signal ratio as a function of the number of ancestors ordered according to this iteration. Fig. 7.14 illustrates this process and shows that the removal of an essential node leads to a sharp spike in increase in the noise-to-signal ratio (the noise-to-signal ratio jumps from approximately 50-60% to 99%). The identification of this inflection point can be used as a method for effectively and reliably pruning ancestors. 7.6 Algorithm pseudocode. Our overall method is summarized in the pseudocode Alg. 11 and Alg. 12 that we will now describe. Alg. 11 takes the data 𝐷 (encoded into the samples 𝑋1 , . . . , 𝑋𝑁 of Problem 4) and the set of nodes 𝑉 as an input and produces, as described in 296 Algorithm 11 CHD by thresholding the signal-to-noise ratio Input: Data 𝐷, set of nodes 𝑉, threshold 𝜏 (default 𝜏 = 0.5) Output: Learned hypergraph Set of ancestors for each node 1: 𝐷 ← NormalizeData(𝐷) Normalize the data 2: for each 𝑣 ∈ 𝑉 do 3: for each kernel in {"linear", "quadratic", "nonlinear"} Find the kernel do 4: SetOfAncestors(𝑣) ← all other nodes 5: SignalToNoiseRatio ← ComputeSignalToNoiseRatio(kernel, node, 𝐷) 6: if SignalToNoiseRatio > 𝜏 then 7: Choose that kernel and exit the for loop 8: else 9: Remove all ancestors from node 10: end if 11: end for 12: while SignalToNoiseRatio > 𝜏 Prune ancestors do 13: Find least important ancestor 14: Recompute SignalToNoiseRatio without that ancestor 15: if SignalToNoiseRatio > 𝜏 then 16: Remove that ancestor 17: end if 18: end while 19: end for Sec. 7.5, for each node 𝑖 ∈ 𝑉 its set of minimal ancestors 𝐴𝑖 and the simplest
possible function 𝑓𝑖 such that 𝑥𝑖 ≈ 𝑓𝑖 (𝑥 𝑗 ) 𝑗 ∈𝐴𝑖 . It employs the default threshold of 0.5 on the signal-to-noise ratios for its operations. Line 1 normalizes the data (via an affine transformation) so that the samples 𝑋𝑖 are of mean zero and variance 1. Given a node with index 𝑖 = 1 in Line 2 (𝑖 runs through the set of nodes, and we select 𝑖 = 1 for ease of presentation), the command in Line 3 refers to selecting a signal kernel of the form 𝐾 𝑠 =(7.48) (where 𝑘 is selected to be a vanilla RBF kernel such as Gaussian or Matérn), with 1 ≥ 𝛽1 > 0 = 𝛽2 = 𝛽3 for the linear kernel, 1 ≥ 𝛽1 ≥ 𝛽2 > 0 = 𝛽3 for the quadratic kernel and 1 ≥ 𝛽1 ≥ 𝛽2 ≥ 𝛽3 > 0 for the fully nonlinear (interpolative) kernel. The ComputeSignalToNoiseRatio function in Line 5 computes the signal-to-noise ratio with 𝑔𝑎 (𝑥) = 𝑥 1 and with the kernel selected in Line 3. The value of 𝛾 is selected automatically by maximizing the variance of the histogram of eigenvalues of 𝐷 𝛾 as described in Sec. 7.7 (with the kernel 𝐾 = 𝐾 𝑠 =(7.48) selected in Line 3 and 𝑌 = 𝑔𝑎 (𝑋) with 𝑔𝑎 (𝑥) = 𝑥 1 ). The value of 𝛾 is re-computed whenever a node is removed from the list of ancestors, and 𝐾 𝑠 is nonlinear. Lines 13, 14 and 16 are described in Sec. 7.5. They correspond 297 to iteratively identifying the ancestor node 𝑡 contributing the least to the signal and removing that node from the set of ancestors of the node 1 if the removal of that node 𝑡 does not send the signal-to-noise ratio below the default threshold 0.5. Algorithm 12 CHD by inflection point in the noise-to-signal ratio Input: Data 𝐷, set of nodes 𝑉, threshold 𝜏 (default 𝜏 = 0.5) Set of ancestors for each node Output: Learned hypergraph 1: 𝐷 ← NormalizeData(𝐷) Normalize the data 2: for each node 𝑣 ∈ 𝑉 do 3: for each kernel in {"linear", "quadratic", "nonlinear"} Find the kernel do 4: SetOfAncestors ← all other nodes 5: SignalToNoiseRatio ← ComputeSignalToNoiseRatio(kernel, node, 𝐷) 6: if SignalToNoiseRatio > 𝜏 then 7: Choose that kernel and exit the for loop 8: else 9: Remove all ancestors from node 10: end if 11: end for 12: 𝑞 ← Cardinal(all other nodes) 13: SetOfAncestors(𝑞) ← all other nodes 14: while 𝑞 ≥ 1 do 15: NoiseToSignalRatio(𝑞) ← ComputeNoiseToSignalRatio(kernel, node, 𝐷) LeastImportantAncestor ← Find least important ancestor in SetOfAncestors(𝑞) 17: SetOfAncestors(𝑞 − 1) ← SetOfAncestors(𝑞)\ LeastImportantAncestor 18: 𝑞 ← 𝑞−1 19: end while 20: 𝑞 † ← inflection point in 𝑞 → NoiseToSignalRatio(𝑞) or spike in 𝑞 → NoiseToSignalRatio(𝑞)−NoiseToSignalRatio(𝑞 − 1) 21: FinalSetOfAncestors(𝑣) ← SetOfAncestors(𝑞 † ) 22: end for 16: Algorithm 12 distinguishes itself from Algorithm 11 in its approach to pruning ancestors based on signal-to-noise ratios. Instead of using a default threshold of 0.5 like Algorithm 11, Algorithm 12 computes the noise-to-signal ratio, represented as V (𝑛) V (𝑠)+V (𝑛) (𝑞). This ratio is calculated as a function of the number 𝑞 of ancestors, which are ordered based on their decreasing contribution to the signal. The detailed methodology behind this computation can be found in Section 7.5 and is visually depicted in Figure 7.14. The final number 𝑞 of ancestors is then determined by finding the value that maximizes the difference between successive noise-to-signal 298 V (𝑛) V (𝑛) ratios, V (𝑠)+V (𝑛) (𝑞 + 1) − V (𝑠)+V (𝑛) (𝑞). 7.7 Analysis of the signal-to-noise ratio test. The signal-to-noise ratio depends on the prior on the level of noise. The signal-to-noise ratio (7.55) depends on the value of 𝛾, which is the variance prior on the level of noise. The goal of this subsection is to answer the following two questions: (1) How do we select 𝛾? (2) How do we obtain a confidence level for the presence of a signal? Or equivalently for a hyperedge of the hypergraph? To answer these questions, we will now analyze the signal-to-noise ratio in the following regression problem in which we seek to approximate the unknown function 𝑓 † : X → R based on noisy observations 𝑓 † (𝑋) + 𝜎𝑍 = 𝑌 (7.60) of its values at collocation points 𝑋𝑖 ((𝑋, 𝑌 ) ∈ X 𝑁 ×R𝑁 , 𝑍 ∈ R𝑁 , and the entries 𝑍𝑖 of 𝑍 are i.i.d N(0, 1)). Assuming 𝜎 2 to be unknown and writing 𝛾 for a candidate for its value, recall that the GP solution to this problem is approximate 𝑓 † by interpolating the data with the sum of two independent GPs, i.e., √ (7.61) 𝑓 (𝑥) = E[𝜉 (𝑥)|𝜉 (𝑋) + 𝛾𝑍 = 𝑌 ] , √ where 𝜉 ∼ N (0, 𝐾) is the GP prior for the signal 𝑓 † and 𝛾𝑍 ∼ N (0, 𝛾𝐼 𝑁 ) is the GP prior for the noise 𝜎𝑍 in the measurements. Following Sec. 7.5 𝑓 can also be identified as a minimizer of 1 minimize 𝑓 ′ ∥ 𝑓 ′ ∥ 2𝐾 + ∥ 𝑓 ′ (𝑋) − 𝑌 ∥ 2R 𝑁 , (7.62) 𝛾 the activation of the signal GP can be quantified as 𝑠 = ∥ 𝑓 ∥ 2𝐾 , the activation of the noise GP can be quantified as V (𝑛) = 𝛾1 ∥ 𝑓 (𝑋) − 𝑌 ∥ 2R 𝑁 . We can then define the V (𝑛) noise to signal ratio V (𝑠)+V (𝑛) , which admits the following representer formula:
−2 𝑌 𝑇 𝐾 (𝑋, 𝑋) + 𝛾𝐼 𝑌 V (𝑛) =𝛾 (7.63)
−1 . V (𝑠) + V (𝑛) 𝑌 𝑇 𝐾 (𝑋, 𝑋) + 𝛾𝐼 𝑌 Observe that when applied to the setting of Sec. 7.5, this signal-to-noise ratio is calculated with 𝐾 = 𝐾 𝑠 and 𝑌 = 𝑔𝑎 (𝑋). Now we have the following proposition, which follows from (7.63). V (𝑛) Proposition 7.7.1. It holds true that V (𝑠)+V (𝑛) ∈ [0, 1], and if 𝐾 (𝑋, 𝑋) has full rank, V (𝑛) V (𝑛) lim = 0 and lim = 1. (7.64) 𝛾↓0 V (𝑠) + V (𝑛) 𝛾↑∞ V (𝑠) + V (𝑛) 299 Therefore, we are led to the following question: if the signal 𝑓 † and the level of noise 𝜎 2 are both unknown, how do we select 𝛾 to decide whether the data is mostly signal or noise? How do we select the prior on the level of noise? Our answer to this question depends on whether the feature-map associated with the base kernel 𝐾 is finite-dimensional or not. When the kernel is linear, quadratic, or associated with a finite-dimensional feature map. If the feature-map associated with the base kernel 𝐾 is finite-dimensional, then 𝛾 can be estimated from the data itself when the number of data-points is sufficiently large (at least larger than the dimension of the feature-space S). A prototypical example (when trying to identify the ancestors of the variable 𝑥1 ) is 𝐾 = 𝐾 𝑠 =(7.48) with 𝛽3 = 0. In the general setting assume that 𝐾 (𝑥, 𝑥 ′) := 𝜓(𝑥)𝑇 𝜓(𝑥 ′) where the range S of 𝜓 is finite-dimensional. Assume that 𝑓 † belongs to the RKHS defined by 𝜓, i.e., assume that it is of the form 𝑓 † = 𝑣𝑇 𝜓 for some 𝑣 in the feature-space. Then (7.60) reduces to 𝑣𝑇 𝜓(𝑋) + 𝜎𝑍 = 𝑌 , (7.65) and, in the large data regime, 𝜎 2 can be estimated by 𝜎 ¯ 2 := 1 2 inf 𝑤𝑇 𝜓(𝑋) − 𝑌 R 𝑁 . 𝑁 𝑤∈S (7.66) Our strategy, when the feature map is finite-dimensional, is then to select 2 𝛾 = 𝑁𝜎 ¯ 2 = inf 𝑤𝑇 𝜓(𝑋) − 𝑌 R 𝑁 . (7.67) 𝑤∈S When the kernel is interpolatory (associated with an infinite-dimensional feature map). If the feature-map associated with the base kernel 𝐾 is infinite-dimensional (or has more dimensions than we have data points) then it can interpolate the data exactly and the previous strategy cannot be employed since the minimum of (7.66) is zero. A prototypical example (when trying to identify the ancestors of the variable 𝑥 1 ) is 𝐾 = 𝐾 𝑠 =(7.48) with 𝛽3 > 0. In this situation, we do not attempt to estimate the level of noise 𝜎 but select a prior 𝛾 such that the resulting noise-to-signal ratio can effectively differentiate noise from signal. To describe this, observe that the 300 noise-to-signal ratio (7.63) admits the representer formula 𝑌 𝑇 𝐷 2𝛾𝑌 V (𝑛) , = 𝑇 V (𝑠) + V (𝑛) 𝑌 𝐷 𝛾𝑌 (7.68)
−1 . (7.69) lim 𝐷 𝛾 = 0 and lim 𝐷 𝛾 = 𝐼 . (7.70) involving the 𝑁 × 𝑁 matrix 𝐷 𝛾 := 𝛾 𝐾 (𝑋, 𝑋) + 𝛾𝐼 Observe that 0 ≤ 𝐷 𝛾 ≤ 𝐼, and 𝛾↓0 𝛾↑∞ Write (𝜆𝑖 , 𝑒𝑖 ) for the eigenpairs of 𝐾 (𝑋, 𝑋) (𝐾 (𝑋, 𝑋)𝑒𝑖 = 𝜆𝑖 𝑒𝑖 ) where the 𝜆𝑖 are ordered in decreasing order. Then the eigenpairs of 𝐷 𝛾 are (𝜔𝑖 , 𝑒𝑖 ) where 𝜔𝑖 := 𝛾 . 𝛾 + 𝜆𝑖 (7.71) Note that the 𝜔𝑖 are contained in [0, 1] and also ordered in decreasing order. Writing 𝑌¯𝑖 for the orthogonal projection of 𝑌 onto 𝑒𝑖 , we have Í𝑛 𝜔𝑖2𝑌¯𝑖2 V (𝑛) . = Í𝑖=1 𝑛 ¯2 V (𝑠) + V (𝑛) 𝑖=1 𝜔𝑖𝑌𝑖 (7.72) It follows that if the histogram of the eigenvalues of 𝐷 𝛾 is concentrated near 0 or near 1, then the noise-to-signal ratio is non-informative since the prior 𝛾 dominates it. To avoid this phenomenon, we select 𝛾 so that the eigenvalues of 𝐷 𝛾 are well spread out in the sense that the histogram of its eigenvalues has maximum or near-maximum variance (see Fig. 7.6 for a good choice and a bad choice for 𝛾). If the eigenvalues have an algebraic decay, then this is equivalent to taking 𝛾 to be the geometric mean of those eigenvalues. In practice, we use an off-the-shelf optimizer to obtain 𝛾 by maximizing the sample 𝑛 . If this optimization fails, we default to the median of the variance of (𝜔𝑖 )𝑖=1 eigenvalues. This ensures a balanced, well-spread spectrum for 𝐷𝛾, with half of the eigenvalues 𝜆𝑖 being lower and half being higher than the median. Rationale for the choices of 𝛾 The purpose of this section is to present a rationale for the proposed choices for 𝛾 in Sec. 7.7 and 7.7. For the choice Sec. 7.7, we present an asymptotic analysis of the 301 signal-to-noise ratio in the setting of a simple linear regression problem. According to (7.67), 𝛾 must scale linearly in 𝑁; this scaling is necessary to achieve a ratio that represents the signal-to-noise per sample. Without it (if 𝛾 remains bounded as a function of 𝑁), this scaling of the signal-to-noise would converge towards 0 as 𝑁 → ∞. To see how we will now consider a simple example in which we seek to linearly regress the variable 𝑦 as a function of the variable 𝑥, both taken to be scalar (in which case 𝜓(𝑥) = 𝑥). Assume that the samples are of the form 𝑌𝑖 = 𝑎𝑋𝑖 + 𝜎𝑍𝑖 for 𝑖 = 1, . . . , 𝑁, where 𝑎, 𝜎 ≠ 0, the 𝑍𝑖 are i.i.d. N (0, 1) random variables, and Í𝑁 Í𝑁 2 𝑋𝑖 = 0 and 𝑁1 𝑖=1 𝑋𝑖 = 1. Then, the signal-to-noise ratio is the 𝑋𝑖 satisfy 𝑁1 𝑖=1 V (𝑠) 1 Í𝑁 2 2 V (𝑠)+V (𝑛) with V (𝑠) = |𝑣| and V (𝑛) = 𝛾 𝑖=1 |𝑣𝑋𝑖 − 𝑌𝑖 | and 𝑣 is a minimizer of 𝑁 1 ∑︁ min |𝑣| + |𝑣𝑋𝑖 − 𝑌𝑖 | 2 . 𝑣∈R 𝛾 𝑖=1 2 (7.73) 𝑎𝑁 and In asymptotic 𝑁 → ∞ regime, we have 𝑣 ≈ 𝛾+𝑁 𝛾 2 V (𝑠) 𝑁𝑎 ≈ . V (𝑠) + V (𝑛) −𝑎 2 (𝛾/𝑁 + 1) + (𝑎 2 + 𝜎 2 )(𝛾/𝑁 + 1) 2 (7.74) V (𝑠) If 𝛾 is bounded independently from 𝑁, then V (𝑠)+V (𝑛) converges towards zero as 𝑁 → ∞, which is undesirable as it does not represent a signal-to-noise ratio per V (𝑠) 𝑎2 sample. If 𝛾 = 𝑁, then V (𝑠)+V (𝑛) ≈ 4𝜎 2 +2𝑎 2 , which does not converge to 1 as 𝑎 → ∞ and 𝜎 → 0, which is also undesirable. If 𝛾 is taken as in (7.67), then 𝛾 ≈ 𝑁𝜎 2 and V (𝑠) 𝑎2 , ≈ V (𝑠) + V (𝑛) (𝜎 2 + 1)(𝑎 2 + 𝜎 2 + 1) (7.75) which converges towards 0 as 𝜎 → ∞ and towards 1/(1 + 𝜎 2 ) as 𝑎 → ∞, which has, therefore, the desired properties. Moving to Sec. 7.7, because the kernel can interpolate the data exactly we can no longer use (7.66) to estimate the level of noise 𝜎. For a finite-dimensional feature map 𝜓, with data (𝑋, 𝑌 ), we can decompose 𝑌 = 𝑣𝑇 𝜓(𝑋) + 𝜎𝑍 into a signal part 𝑌𝑠 and noise part 𝑌𝑠 , s.t. 𝑌 = 𝑌𝑠 +𝑌𝑛 . While 𝑌𝑠 belongs to the linear span of eigenvectors of 𝐾 (𝑋, 𝑋) associated with non-zero eigenvalues, 𝑌𝑛 also activates the eigenvectors associated with with the null space of 𝐾 (𝑋, 𝑋) and the projection of 𝑌 onto that null-space is what allows us to derive 𝛾 in Sec. 7.7. Since in the interpolatory case, all eigenvalues are strictly positive, we need to choose which eigenvalues are associated with noise differently, as is described in the previous section. With a fixed 𝛾, we see that if 𝜆𝑖 ≫ 𝛾, then 𝜔𝑖 ≈ 0, which contributes in (7.72) to yield a low noiseto-signal ratio. Similarly, if 𝜆𝑖 ≪ 𝛾, this eigenvalue yields a high noise-to-signal 302 ratio. Thus, we see that the choice of 𝛾 assigns a noise level to each eigenvalue. While in the finite-dimensional feature map setting, this assignment is binary, here we perform soft thresholding using 𝜆 ↦→ 𝛾/(𝛾 + 𝜆) to indicate the level of noise of each eigenvalue. This interpretation sheds light on the selection of 𝛾 in equation (7.67). Let 𝜓 represent the feature map associated with 𝐾. Assuming the empirical mean of 𝜓(𝑋𝑖 ) is zero, the matrix 𝐾 (𝑋, 𝑋) corresponds to an unnormalized kernel covariance matrix 𝜓𝑇 (𝑋)𝜓(𝑋). Consequently, its eigenvalues correspond to 𝑁 times the variances of the 𝜓(𝑋𝑖 ) across various eigenspaces. After conducting Ordinary Least Squares regression in the feature space, if the noise variance is estimated as 𝜎 ¯ 2 , then any eigenspace of the normalized covariance matrix whose eigenvalue is lower than 𝜎 ¯ 2 cannot be recovered due to the noise. Given this, we set the soft thresholding cutoff to be 𝛾 = 𝑁 𝜎 ¯ 2 for the unnormalized covariance matrix 𝐾 (𝑋, 𝑋). Z-score/quantile bounds on the noise-to-signal ratio. If the data is only composed of noise, then an interval of confidence can be obtained on the noise-to-signal ratio. To describe this consider the problem of testing the null hypothesis H0 : 𝑓 † ≡ 0 (there is no signal) against the alternative hypothesis H1 : 𝑓 † . 0 (there is a signal). Under the null hypothesis H0 , the distribution of the noise-to-signal ratio (7.68) is known and it follows that of the random variable 𝐵 := 𝑍 𝑇 𝐷 2𝛾 𝑍 . 𝑍𝑇 𝐷𝛾 𝑍 (7.76) Therefore, the quantiles of 𝐵 can be used as an interval of confidence on the noiseto-signal ratio if H0 is true. More precisely, selecting 𝛽 such that P[𝐵 ≤ 𝛽𝛼 ] ≈ 𝛼 with 𝛼 = 0.05 as a prototypical example, we expect the noise to signal ratio (7.68) to be, under H0 , to be larger than 𝛽𝛼 with probability ≈ 1 − 𝛼. The estimation of 𝛽 requires Monte-Carlo sampling. An alternative approach (in the large data regime) to using the quantile 𝛽𝛼 is to use the Z-score 𝑌 𝑇 𝐷 2𝛾 𝑌 − E[𝐵] 𝑌 𝑇 𝐷 𝛾𝑌 Z := √︁ , (7.77) Var[𝐵] after estimating E[𝐵] and Var[𝐵] via Monte-Carlo sampling. In particular if H0 is true then |Z| ≥ 𝑧 𝛼 should occur with probability ≈ 𝛼 with 𝑧0.1 = 1.65, 𝑧0.05 = 1.96 and 𝑧0.01 = 2.58. Remark 7.7.2. Although the quantile 𝛽𝛼 or the Z-score Z can be employed to produce an interval of confidence on the noise-to-signal ratio under H0 we cannot 303 use them as thresholds for removing nodes from the list of ancestors as discussed in Sec. 7.5 Indeed, observing a noise-to-signal ratio (7.68) below the threshold 𝛽𝛼 does not imply that all the signal has been captured by the kernel; it only implies that some signal has been captured by the kernel 𝐾. To illustrate this point, consider the setting where one tries to approximate the variable 𝑥 1 as a function of the variable 𝑥 2 . If 𝑥1 is not a function of 𝑥 2 , but of 𝑥 2 and 𝑥3 , as in 𝑥 1 = cos(𝑥 2 ) + sin(𝑥3 ), then applying the proposed approach with 𝑌 encoding the values of 𝑥 1 , 𝑋 encoding the values of 𝑥 2 , and the kernel 𝐾 depending on 𝑥2 could lead to a noise-to-signal ratio below 𝛽𝛼 due to the presence of a signal in 𝑥 2 . Therefore, although we are missing the variable 𝑥 3 in the kernel 𝐾, we would still observe a possibly low noise-to-signal ratio due to the presence of some signal in the data. Summarizing if the data only V (𝑛) contains noise then V (𝑠)+V (𝑛) ≥ 𝛽𝛼 should occur with probability 1 − 𝛼. If the V (𝑛) event V (𝑠)+V (𝑛) < 𝛽𝛼 is observed in the setting of 𝐾 = 𝐾 𝑠/𝑡 =(7.58) where we try to identify the ancestors of 𝑥 1 , then we can only deduce that 𝑥 3 , . . . , 𝑥 𝑑 contain some signal but perhaps not all of it (we can use this a criterion for pruning 𝑥2 ). 7.8 Supplementary information on examples. Algebraic equations. Although we have used Alg. 12 for the algebraic equations examples presented in Fig. 7.4, Alg. 11 yields the same results with the default signal-to-noise threshold 𝜏 = 0.5. The chemical reaction network. Consider the chemical reaction network example illustrated in Fig. 7.4.(a). The proposed mechanism for the hydrogenation of ethylene (C2 H4 ) to ethane (C2 H6 ), is (writing [𝐻] for the concentration of 𝐻) modeled by the following system of differential equations: 𝑑 [𝐻2 ] = −𝑘 1 [𝐻2 ] + 𝑘 −1 [𝐻] 2 𝑑𝑡 𝑑 [𝐻] = 2𝑘 1 [𝐻2 ] − 2𝑘 −1 [𝐻] 2 − 𝑘 2 [𝐶2 𝐻4 ] [𝐻] − 𝑘 3 [𝐶2 𝐻5 ] [𝐻] 𝑑𝑡 (7.78) 𝑑 [𝐶2 𝐻4 ] = −𝑘 2 [𝐶2 𝐻4 ] [𝐻] 𝑑𝑡 𝑑 [𝐶2 𝐻5 ] = 𝑘 2 [𝐶2 𝐻4 ] [𝐻] − 𝑘 3 [𝐶2 𝐻5 ] [𝐻]. 𝑑𝑡 The primary variables are the concentrations [𝐻2 ], [𝐻], [𝐶2 𝐻4 ], and [𝐶2 𝐻5 ] and 𝑑 [𝐻] 𝑑 [𝐶2 𝐻4 ] 2] their time derivatives 𝑑 [𝐻 , and 𝑑 [𝐶𝑑𝑡2 𝐻5 ] . The computational hyper𝑑𝑡 , 𝑑𝑡 , 𝑑𝑡 graph encodes the functional dependencies (7.78) associated with the chemical 304 reactions. The hyperedges of the hypergraph are assumed to be unknown and the primary variables are assumed to be known. Given 𝑁 samples from the graph of the form
(7.79) [𝐻2 ] (𝑡𝑖 ), [𝐻] (𝑡𝑖 ), [𝐶2 𝐻4 ] (𝑡𝑖 ), [𝐶2 𝐻5 ] (𝑡𝑖 ) 𝑖=1,...,𝑁 , our objective is to recover the structure of the hypergraph given by (7.78), representing the functions by hyperedges. We create a dataset of the form (7.79) by integrating 50 trajectories of (7.78) for different initial conditions, and each equispaced 50 times from 𝑡 = 0 to 𝑡 = 5. The dataset is represented in Fig. 7.4.(b) (the time derivatives of concentrations are estimated by taking the derivatives of the interpolants of those concentrations). We impose the information that the derivative variables are function of the non-derivative variables to avoid ambiguity in the recovery, as (7.78) is not the unique representation of the functional relation between nodes in the graph. We implement Alg. 11 with weights 𝛽 = [0.1, 0.01, 0.001] for linear, quadratic, and nonlinear, respectively (Alg. 12 recovers the same hypergraph). The output graph can be seen in Fig. 7.4.(b). We obtain a perfect recovery of the computational graph and a correct identification of the relations being quadratic. The Google Covid 19 open data. Consider the example illustrated in Fig. 7.3.(e-k). Categorical data are treated as scalar values, with all variables scaled to achieve a mean of 0 and a variance of 1. We implement three distinct kernel types: linear, quadratic, and Gaussian, with a length scale of 1 for the latter. A weight ratio of 1/10 is assigned between kernels, signifying that the quadratic kernel is weighted ten times less than the linear kernel. Lastly, the noise parameter, 𝛾, is determined using the optimal value outlined in Sec. 7.7. Initially, a complete graph is constructed using all variables, depicted in Fig. 7.3.(g). This construction is done using only linear and quadratic kernels. The full graph is highly clustered and redundant information is eliminated by selecting representative nodes for each cluster. Eliminating redundant nodes is important for two reasons: firstly, it improves the graph’s readability, especially with 31 variables; secondly, it avoids hindering graph discovery. In an extreme case, treating two identical variables as distinct would result in one variable’s ancestor simply being its duplicate, yielding an uninformative graph. Subsequently, the graph discovery algorithm is rerun, with reduced variables due to eliminating redundancy, ushering us into a predominantly noisy regime. With fewer variables available, we use additionally the nonlinear kernel. Two indicators are employed to navigate our discovery process: the signal-to-noise ratio and the Z-test. The former quantifies the 305 degree to which our regression is influenced by noise, while the latter signals the existence of any signal. We follow the procedure in algorithm 12, resulting in the graph presented in Fig. 7.3.(k). Cell signaling network Consider the example Fig. 7.1.(l) from (Sachs et al., 2005) and Fig. 7.4.(h-j). To identify the ancestors of each node, we apply the algorithm in two stages. First, we learn the dependencies using only linear and quadratic kernels. Fig. 7.4.(h) identifies the resulting graph learned given a subset of 𝑁 = 2, 000 samples chosen uniformly at random from the dataset. We observe that the graph identified by the algorithm consists of four disconnected clusters where the molecule levels in each cluster are closely related by linear or quadratic dependencies (all connections are linear except for the connection between Akt and PKA, which is quadratic). These edges match a subset of the edges found in the gold standard model identified in (Sachs et al., 2005). With perfect dependencies that have no noise, one can define constraints that reduce the total number of variables in the system. For this noisy dataset, we treat these dependencies as forming groups of similar variables and introduce a hierarchical approach to learn the connections between groups. Second, we run the graph discovery algorithm after grouping the molecules into clusters. For each node in the graph, we identified the ancestors of each node by constraining the dependence to be a subset of the clusters. In other words, when identifying the ancestors of a given node 𝑖 in cluster 𝐶, the algorithm is only permitted to (1) use ancestors that do not belong to cluster 𝐶, and (2) include all or none of the variables in each cluster ( 𝑗 in cluster 𝐷 ≠ 𝐶 is listed as an ancestor if and only if all other nodes 𝑗 ′ in cluster 𝐷 are also listed as ancestors). The ancestors were identified using a Gaussian (fully nonlinear) kernel and the number of ancestors were selected manually based on the inflection point in the noise-to-signal ratio. The resulting graph is depicted in Fig. 7.4.(i). Each edge is weighted based on its signal-to-noise ratio. We observe that there is a stronger dependence of the Jnk, PKC, and P38 cluster on the PIP3, Plcg, and PIP2 cluster, which closely matches the gold standard model. As compared to approaches based on acyclic DAGs, however, the graph identified by our algorithm also contains feedback loops between the various molecule levels. Fig. 7.4.(i-j) displays a side-by-side comparison between the graph identified with our method and the graph generated in (Sachs et al., 2005). To aid in this comparison, we have highlighted different clusters in distinct colors. We emphasize that while the Bayesian network analysis in (Sachs et al., 2005) 306 relied on the control of the sampling of the underlying variables (the simultaneous measurement of multiple phosphorylated protein and phospholipid components in thousands of individual primary human immune system cells, and perturbing these cells with molecular interventions), the reconstruction obtained by our method did not use this information and recovered functional dependencies rather than causal dependencies. Interestingly, the information recovered through our method appears to complement and enhance the findings presented in (Sachs et al., 2005) (e.g., the linear and noiseless dependencies between variables in the JNK cluster is not something that could easily be inferred from the graph produced in (Sachs et al., 2005)). BCR reaction network In the high-dimensional example of the BCR reaction network, the computations of terms of the form 𝑦𝑇 𝑘 𝑜 (𝑋, 𝑋)𝑦 (i.e., the activations), where 𝑦 ∈ R𝑛 and 𝑘 𝑜 (𝑋, 𝑋) is the 𝑜-th coordinate of the quadratic kernel (𝑘 (𝑥𝑖 , 𝑥 𝑗 ) = (1 + ⟨𝑥𝑖 , 𝑥 𝑗 ⟩) 2 ) becomes the computational bottleneck of our method. If we let 𝑥 1 , . . . , 𝑥 𝑛 ∈ R 𝑝 be the points and 𝑥𝑖𝑜 be the 𝑜-th coordinate of 𝑥𝑖 , we can compute the activation of the 𝑜−th coordinate using 𝑘 𝑜 (𝑥𝑖 , 𝑥 𝑗 ) = (1 + 𝑥𝑖𝑜 𝑥 𝑜𝑗 ) 2 − 1 + 2𝑥𝑖𝑜 𝑥 𝑜𝑗 ⟨𝑥𝑖−𝑜 , 𝑥 −𝑜 (7.80) 𝑗 ⟩, where 𝑘 𝑜 is the 𝑜−th coordinate of the kernel and 𝑥𝑖−𝑜 represents the remaining coordinates of 𝑥𝑖 . To compute the 𝑛 × 𝑛 kernel matrix of 𝑘 𝑜 for each 𝑜 ∈ {1, .., 𝑝}, we must compute 𝑝 × 𝑛 × 𝑛 inner products in R 𝑝 , which is a very large computation. Instead, we may use the following reformulation to speed up computations. Notice 𝑜 𝑜 𝑜 𝑜 ⟨𝑥𝑖 , 𝑥 𝑗 ⟩ = 𝑥𝑖𝑜 𝑥 𝑜𝑗 + ⟨𝑥𝑖−𝑜 , 𝑥 −𝑜 𝑗 ⟩, and therefore 𝑘 𝑜 (𝑥𝑖 , 𝑥 𝑗 ) = 2𝑥𝑖 𝑥 𝑗 ⟨𝑥𝑖 , 𝑥 𝑗 ⟩ + 2𝑥𝑖 𝑥 𝑗 − 𝑝 𝑝 (𝑥𝑖𝑜 𝑥 𝑜𝑗 ) 2 . Now, define 𝑣 𝑜 = (𝑥𝑖𝑜 𝑦𝑖 )𝑖=1 and 𝑤 𝑜 = ((𝑥𝑖𝑜 ) 2 𝑦𝑖 )𝑖=1 , and note that ∑︁ ∑︁ (7.81) 𝑦𝑖 𝑦 𝑗 (𝑥𝑖𝑜 𝑥 𝑜𝑗 ) 2 𝑦𝑇 𝐾 𝑜 𝑦 = 2𝑦𝑖 𝑥𝑖𝑜 𝑦 𝑗 𝑥 𝑜𝑗 (1 + ⟨𝑥𝑖 , 𝑥 𝑗 ⟩) − 𝑖, 𝑗 𝑖, 𝑗 e = (2(1 + ⟨𝑥𝑖 , 𝑥 𝑗 ⟩)) 𝑛 we have that and so defining 𝐾 𝑖, 𝑗=1 e 𝑜− 𝑦𝑇 𝐾𝑜 𝑦 = 𝑣 𝑜𝑇 𝐾𝑣 𝑝 ∑︁ !2 𝑤 𝑖𝑜 . (7.82) 𝑖=1 e is computed just once for all 𝑝, and only 𝑣 𝑜 and 𝑤 𝑜 change for every Note that 𝐾 ancestor calculation, which is where the main computational gain comes from. One may find in the GitHub repository of the paper a comparison of the two methods of computations and observe a tenfold speedup. This speedup is even larger in 307 our implementation of the BCR example, as GPU acceleration enables the second method to run even faster. 308 BIBLIOGRAPHY Agrawal, Akshay, Robin Verschueren, Steven Diamond, and Stephen Boyd (2018). “A rewriting system for convex optimization problems”. In: Journal of Control and Decision 5.1, pp. 42–60. Aliprantis, C. D. and K. C. Border (2006). Infinite Dimensional Analysis: A Hitchhiker’s Guide. Third. Berlin: Springer, pp. xxii+703. Aliprantis, C. D. and O. Burkinshaw (1998). Principles of Real Analysis. 3rd. Academic Press. Almroth, Bo O, Perry Stern, and Frank A Brogan (1978). “Automatic choice of global shape functions in structural analysis”. In: Aiaa Journal 16.5, pp. 525– 528. Altmann, Robert, Patrick Henning, and Daniel Peterseim (2021). “Numerical homogenization beyond scale separation”. In: Acta Numerica 30, pp. 1–86. Alvarez, Mauricio A, Lorenzo Rosasco, Neil D Lawrence, et al. (2012). “Kernels for vector-valued functions: A review”. In: Foundations and Trends® in Machine Learning 4.3, pp. 195–266. Amsallem, David and Charbel Farhat (2008). “Interpolation method for adapting reduced-order models and application to aeroelasticity”. In: AIAA journal 46.7, pp. 1803–1813. Arcangéli, Rémi, María Cruz López de Silanes, and Juan José Torrens (2007). “An extension of a bound for functions in Sobolev spaces, with applications to (m, s)spline interpolation and smoothing”. In: Numerische Mathematik 107.2, pp. 181– 211. Ashton, Greg et al. (2022). “Nested sampling for physical scientists”. In: Nature Reviews Methods Primers 2. Babuška, Ivo and John Osborn (2000). “Can a finite element method perform arbitrarily badly?” In: Mathematics of computation 69.230, pp. 443–462. Bădoiu, M., S. Har-Peled, and P. Indyk (2002). “Approximate clustering via coresets”. In: Proceedings of the thirty-fourth annual ACM symposium on Theory of computing. ACM, pp. 250–257. Bajgiran, Hamed Hamze, Pau Batlle, Houman Owhadi, Mostafa Samir, Clint Scovel, Mahdy Shirdel, Michael Stanley, and Peyman Tavallali (2022a). “Uncertainty quantification of the 4th kind; optimal posterior accuracy-uncertainty tradeoff with the minimum enclosing ball”. In: Journal of Computational Physics 471. P.B contributed to the conceptualization, writing, and numerical experiments in Sections 1 and 6., p. 111608. doi: https://doi.org/10.1016/j.jcp.2022. 111608. 309 Bajgiran, Hamed Hamze, Pau Batlle, Houman Owhadi, Mostafa Samir, Clint Scovel, Mahdy Shirdel, Michael Stanley, and Peyman Tavallali (Dec. 2022b). “Uncertainty quantification of the 4th kind: Optimal posterior accuracy-uncertainty tradeoff with the minimum enclosing ball”. In: Journal of Computational Physics 471, p. 111608. doi: 10.1016/j.jcp.2022.111608. url: https://doi.org/10. 1016/j.jcp.2022.111608. Baptista, Ricardo, Youssef Marzouk, Rebecca E Morrison, and Olivier Zahm (2021). “Learning non-Gaussian graphical models via Hessian scores and triangular transport”. In: arXiv preprint arXiv:2101.03093. Bastos, Leonardo S and Anthony O’hagan (2009). “Diagnostics for Gaussian process emulators”. In: Technometrics 51.4, pp. 425–438. Batlle, Pau, Yifan Chen, Bamdad Hosseini, Houman Owhadi, and Andrew M Stuart (2023). “Error analysis of kernel/GP methods for nonlinear and parametric PDEs”. In: Journal of Computational Physics. P.B contributed to numerical results of Section 4, and writing. doi: https : //doi.org/10.1016/j.jcp.2024.113488. Batlle, Pau, Matthieu Darcy, Bamdad Hosseini, and Houman Owhadi (2023). “Kernel methods are competitive for operator learning”. In: Journal of Computational Physics. Batlle, Pau, Michael Stanley, Pratik Patil, Mikael Kuusela, and Houman Owhadi (2023). “Optimization-based frequentist confidence intervals for functionals in constrained inverse problems: Resolving the Burrus conjecture”. In: arXiv preprint arXiv:2310.02461. Beck, Joakim, Raul Tempone, Fabio Nobile, and Lorenzo Tamellini (2012). “On the optimal polynomial approximation of stochastic PDEs by Galerkin and collocation methods”. In: Mathematical Models and Methods in Applied Sciences 22.09, p. 1250023. Becker, Roland, Maximilian Brunner, Michael Innerberger, Jens Markus Melenk, and Dirk Praetorius (2023). “Cost-optimal adaptive iterative linearized FEM for semilinear elliptic PDEs”. In: ESAIM: Mathematical Modelling and Numerical Analysis 57.4, pp. 2193–2225. Bendsoe, Martin Philip and Ole Sigmund (2003). Topology optimization: theory, methods, and applications. Springer Science & Business Media. Berger, J. O. (2013). Statistical Decision Theory and Bayesian Analysis. Springer Science & Business Media. Berger, Roger L. and Dennis D. Boos (1994). “P Values Maximized Over a Confidence Set for the Nuisance Parameter”. In: Journal of the American Statistical Association 89, pp. 1012–1016. Berlinet, Alain and Christine Thomas-Agnan (2011). Reproducing kernel Hilbert spaces in probability and statistics. Springer Science & Business Media. 310 Bertsekas, D. P., A. Nedić, and A. Ozdaglar (2003). Convex Analysis and Optimization. Athena Scientific Optimization and Computation Series. Athena Scientific. Bhattacharya, Kaushik, Bamdad Hosseini, Nikola B. Kovachki, and Andrew M. Stuart (2021). “Model Reduction And Neural Networks For Parametric PDEs”. en. In: The SMAI Journal of computational mathematics 7, pp. 121–157. Bogachev, Vladimir Igorevich (1998). Gaussian measures. American Mathematical Society. Böhmer, Klaus and Robert Schaback (2013). “A nonlinear discretization theory”. In: Journal of computational and applied mathematics 254, pp. 204–219. – (2020). “A nonlinear discretization theory for meshfree collocation methods applied to quasilinear elliptic equations”. In: ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik 100.10, e201800170. Boncoraglio, Gabriele and Charbel Farhat (2021). “Active manifold and modelorder reduction to accelerate multidisciplinary analysis and optimization”. In: AIAA Journal 59.11, pp. 4739–4753. Bonnans, J. F. and A. Shapiro (2000). Perturbation Analysis of Optimization Problems. Springer-Verlag. Boucheron, Stéphane and Maud Thomas (2012). “Concentration inequalities for order statistics”. In: Electronic Communications in Probability 17.none, pp. 1–12. doi: 10.1214/ECP.v17-2210. url: https://doi.org/10.1214/ECP.v172210. Boullé, Nicolas and Alex Townsend (2022). “Learning elliptic partial differential equations with randomized linear algebra”. In: Foundations of Computational Mathematics, pp. 1–31. Bourdais, Théo, Pau Batlle, Xianjin Yang, Ricardo Baptista, Nicolas Rouquette, and Houman Owhadi (2023). “Codiscovering graphical structure and functional relationships within data: A Gaussian Process framework for connecting the dots”. In: Proceedings of the National Academy of Sciences (PNAS). P.B contributed to conceptualization, early prototyping, and writing. doi: https: //doi.org/10.1073/pnas.2403449121. Boyd, Stephen and Lievan Vandenberghe (2004). Convex Optimization. Cambridge University Press. Bradbury, James, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang (2018). JAX: composable transformations of Python+NumPy programs. Version 0.3.13. url: http://github.com/google/ jax. 311 Brunton, Steven L, Joshua L Proctor, and J Nathan Kutz (2016). “Discovering governing equations from data by sparse identification of nonlinear dynamical systems”. In: Proceedings of the national academy of sciences 113.15, pp. 3932– 3937. Buchner, Johannes (2023). “Nested Sampling Methods”. In: arXiv preprint arXiv:2101.09675v4. Burrus, W. R. (1965). Utilization of a Priori Information by Means of Mathematical Programming in the Statistical Interpretation of Measured Distributions. Oak Ridge National Laboratory. url: https://books.google.com/books?id= ubUPYiYzrRcC. Carpenter, James (1999). “Test inversion bootstrap confidence intervals”. In: Journal of the Royal Statistical Society: Series B (Statistical Methodology) 61.1, pp. 159– 172. Casella, G. and R. L. Berger (2002). Statistical Inference. Thomson Learning Inc. Casella, George and Roger L. Berger (2002). Statistical Inference. Second. Duxbery. Cash, Webster (1979). “Parameter estimation in astronomy through application of the likelihood ratio”. In: The Astrophysical Journal 228, pp. 939–947. Chen, Jiong, Florian Schäfer, Jin Huang, and Mathieu Desbrun (2021). “Multiscale cholesky preconditioning for ill-conditioned problems”. In: ACM Transactions on Graphics (TOG) 40.4, pp. 1–13. Chen, Tianping and Hong Chen (1995a). “Approximation capability to functions of several variables, nonlinear functionals, and operators by radial basis function neural networks”. In: IEEE Transactions on Neural Networks 6.4, pp. 904–910. – (1995b). “Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems”. In: IEEE Transactions on Neural Networks 6.4, pp. 911–917. Chen, Yifan, Bamdad Hosseini, Houman Owhadi, and Andrew M Stuart (2021a). Solving and Learning Nonlinear PDEs with Gaussian Processes. – (2021b). “Solving and learning nonlinear PDEs with Gaussian processes”. In: Journal of Computational Physics 447, p. 110668. issn: 0021-9991. – (2021c). “Solving and learning nonlinear PDEs with Gaussian processes”. In: Journal of Computational Physics 447, p. 110668. Chen, Yifan, Houman Owhadi, and Florian Schäfer (2024). “Sparse Cholesky factorization for solving nonlinear PDEs via Gaussian processes”. In: Mathematics of Computation. Chen, Yifan, Houman Owhadi, and Andrew Stuart (2021a). “Consistency of empirical Bayes and kernel flow for hierarchical parameter estimation”. In: Mathematics of Computation. 312 Chen, Yifan, Houman Owhadi, and Andrew Stuart (2021b). “Consistency of empirical Bayes and kernel flow for hierarchical parameter estimation”. In: Mathematics of Computation 90.332, pp. 2527–2578. Chen, Yuansi, Raaz Dwivedi, Martin J. Wainwright, and Bin Yu (2018). “Fast MCMC Sampling Algorithms on Polytopes”. In: Journal of Machine Learning Research 19, pp. 1–86. Cheung, Ka Chun, Leevan Ling, and Robert Schaback (2018). “𝐻 2 -Convergence of Least-Squares Kernel Collocation Methods”. In: SIAM Journal on Numerical Analysis 56.1, pp. 614–633. Chickering, David Maxwell (2002). “Optimal structure identification with greedy search”. In: Journal of machine learning research 3.Nov, pp. 507–554. Chkifa, Abdellah, Albert Cohen, Ronald DeVore, and Christoph Schwab (Nov. 2012). “Sparse adaptive Taylor approximation algorithms for parametric and stochastic elliptic PDEs”. In: ESAIM: Mathematical Modelling and Numerical Analysis 47.1, pp. 253–280. Chkifa, Abdellah, Albert Cohen, and Christoph Schwab (2014). “High-dimensional adaptive sparse polynomial interpolation and applications to parametric PDEs”. In: Foundations of Computational Mathematics 14.4, pp. 601–633. Chkrebtii, Oksana A, David A Campbell, Ben Calderhead, and Mark A Girolami (2016). “Bayesian solution uncertainty quantification for differential equations”. In: Bayesian Analysis 11.4, pp. 1239–1267. Cialenco, Igor, Gregory E Fasshauer, and Qi Ye (2012). “Approximation of stochastic partial differential equations by a kernel-based collocation method”. In: International Journal of Computer Mathematics 89.18, pp. 2543–2561. CMS Collaboration (2016). “Measurement of differential cross sections √ for Higgs boson production in the diphoton decay channel in pp collisions at 𝑠 = 8 TeV”. In: The European Physical Journal C 76.13. doi: https://doi.org/10.1140/ epjc/s10052-015-3853-3. – (2019). “Measurement of inclusive and differential Higgs boson production √ cross sections in the diphoton decay channel in proton-proton collisions at 𝑠 = 13 TeV”. In: Journal of High Energy Physics 2019.183. doi: https://doi.org/ 10.1007/JHEP01(2019)183. Cockayne, Jon, Chris Oates, Tim Sullivan, and Mark Girolami (2017). “Probabilistic numerical methods for PDE-constrained Bayesian inverse problems”. In: AIP Conference Proceedings, p. 060001. Cockayne, Jon, Chris J Oates, Timothy John Sullivan, and Mark Girolami (2019). “Bayesian probabilistic numerical methods”. In: SIAM Review 61.4, pp. 756–789. Cohen, Albert and Ronald DeVore (2015). “Approximation of high-dimensional parametric PDEs”. In: Acta Numerica 24, pp. 1–159. 313 Cohen, Albert, Ronald DeVore, and Christoph Schwab (2010). “Convergence rates of best 𝑁-term Galerkin approximations for a class of elliptic sPDEs”. In: Foundations of Computational Mathematics 10.6, pp. 615–646. Cranmer, Kyle, Johann Brehmer, and Gilles Louppe (2020). “The frontier of simulationbased inference”. In: Proceedings of the National Academy of Sciences 117.48, pp. 30055–30062. Cranmer, Kyle, Juan Pavez, and Gilles Louppe (2016). Approximating Likelihood Ratios with Calibrated Discriminative Classifiers. arXiv:1506.02169 [stat.AP]. Dalmasso, Niccolo, Luca Masserano, David Zhao, Rafael Izbicki, and Ann B. Lee (2024). “Likelihood-Free Frequentist Inference: Bridging Classical Statistics and Machine Learning for Reliable Simulator-Based Inference”. In: Electronic Journal of Statistics, pp. 5045–5090. Dalmasso, Niccolò, Rafael Izbicki, and Ann B. Lee (2020). “Confidence sets and hypothesis testing in a likelihood-free inference setting”. In: International Conference on Machine Learning. PMLR, pp. 2323–2334. Dalmasso, Niccolò, Luca Masserano, David Zhao, Rafael Izbicki, and Ann B. Lee (2023). “Likelihood-free frequentist inference: Bridging Classical Statistics and Machine Learning for Reliable Simulator-Based Inference”. In: arXiv preprint arXiv:2107.03920. Darcy, Matthieu, Boumediene Hamzi, Giulia Livieri, Houman Owhadi, and Peyman Tavallali (2023). “One-shot learning of stochastic differential equations with data adapted kernels”. In: Physica D: Nonlinear Phenomena 444, p. 133583. Data, MIT Critical, Justin D Salciccioli, Yves Crutain, Matthieu Komorowski, and Dominic C Marshall (2016). “Sensitivity analysis and model validation”. In: Secondary analysis of electronic health records, pp. 263–271. De Hoop, Maarten, Daniel Zhengyu Huang, Elizabeth Qian, and Andrew M Stuart (2022). “The cost-accuracy trade-off in operator learning with neural networks”. In: arXiv preprint arXiv:2203.13181. De Ryck, Tim and Siddhartha Mishra (2022). “Error analysis for physics-informed neural networks (PINNs) approximating Kolmogorov PDEs”. In: Advances in Computational Mathematics 48.6, pp. 1–40. Deng, Beichuan, Yeonjong Shin, Lu Lu, Zhongqiang Zhang, and George Em Karniadakis (2022). “Approximation rates of DeepONets for learning operators arising from advection–diffusion equations”. In: Neural Networks 153, pp. 411–426. Diamond, Steven and Stephen Boyd (2016). “CVXPY: A Python-embedded modeling language for convex optimization”. In: Journal of Machine Learning Research 17.83, pp. 1–5. Dirmeyer, Paul A, Pierre Gentine, Michael B Ek, and Gianpaolo Balsamo (2019). “Land surface processes relevant to sub-seasonal to seasonal (S2S) prediction”. In: Sub-Seasonal to Seasonal Prediction. Elsevier, pp. 165–181. 314 Donoho, David L. (1994). “Statistical estimation and optimal recovery”. In: The Annals of Statistics 22.1, pp. 238–270. Doostan, Alireza and Houman Owhadi (2011). “A non-adapted sparse approximation of PDEs with stochastic inputs”. In: Journal of Computational Physics 230.8, pp. 3015–3034. Drton, Mathias and Marloes H Maathuis (2017). “Structure learning in graphical modeling”. In: Annual Review of Statistics and Its Application 4, pp. 365–393. Dudley, R. (2009). Statistics for Applications. 18.443. MIT OpenCourseWare. Economon, Thomas D, Francisco Palacios, Sean R Copeland, Trent W Lukaczyk, and Juan J Alonso (2016). “SU2: An open-source suite for multiphysics simulation and design”. In: Aiaa Journal 54.3, pp. 828–846. Fan, Yuwei, Cindy Orozco Bohorquez, and Lexing Ying (2019). “BCR-Net: A neural network based on the nonstandard wavelet form”. In: Journal of Computational Physics 384, pp. 1–15. Fan, Yuwei, Jordi Feliu-Faba, Lin Lin, Lexing Ying, and Leonardo Zepeda-Núnez (2019). “A multiscale neural network based on hierarchical nested bases”. In: Research in the Mathematical Sciences 6.2, pp. 1–28. Fan, Yuwei, Lin Lin, Lexing Ying, and Leonardo Zepeda-Núnez (2019). “A multiscale neural network based on hierarchical matrices”. In: Multiscale Modeling & Simulation 17.4, pp. 1189–1213. Fasshauer, Gregory E (1999). “Solving differential equations with radial basis functions: multilevel methods and smoothing”. In: Advances in computational mathematics 11.2, pp. 139–159. Feischl, Michael and Daniel Peterseim (2020). “Sparse compression of expected solution operators”. In: SIAM Journal on Numerical Analysis 58.6, pp. 3144– 3164. Feldman, Gary J. and Robert D. Cousins (Apr. 1998). “Unified approach to the classical statistical analysis of small signals”. In: Physical Review D 57.7, pp. 3873– 3889. doi: 10 . 1103 / physrevd . 57 . 3873. url: https : / / doi . org / 10 . 11032Fphysrevd.57.3873. Feyel, Frédéric and Jean-Louis Chaboche (2000). “FE2 multiscale approach for modelling the elastoviscoplastic behaviour of long fibre SiC/Ti composite materials”. In: Computer methods in applied mechanics and engineering 183.3-4, pp. 309–330. Feynman, Richard P (1998). “Cargo cult science”. In: The art and science of analog circuit design. Elsevier, pp. 55–61. 315 Fish, Jacob, Kamlun Shek, Muralidharan Pandheeradi, and Mark S Shephard (1997). “Computational plasticity for composite structures based on mathematical homogenization: Theory and practice”. In: Computer methods in applied mechanics and engineering 148.1-2, pp. 53–73. Fisher, Eyal, Regev Schweiger, and Saharon Rosset (2020). “Efficient construction of test inversion confidence intervals using quantile regression”. In: Journal of Computational and Graphical Statistics 29.1, pp. 140–148. Forget, G., J.-M. Campin, P. Heimbach, C. N. Hill, R. M. Ponte, and C. Wunsch (2015). “ECCO version 4: an integrated framework for non-linear inverse modeling and global ocean state estimation”. In: Geoscientific Model Development. Fornberg, Bengt and Natasha Flyer (2015). “Solving PDEs with radial basis functions”. In: Acta Numerica 24, pp. 215–258. Franke, Carsten and Robert Schaback (1998a). “Convergence order estimates of meshless collocation methods using radial basis functions”. In: Advances in computational mathematics 8.4, pp. 381–399. – (1998b). “Solving partial differential equations by collocation using radial basis functions”. In: Applied Mathematics and Computation 93.1, pp. 73–82. Friedman, Jerome, Trevor Hastie, and Robert Tibshirani (2008). “Sparse inverse covariance estimation with the graphical lasso”. In: Biostatistics 9.3, pp. 432– 441. Fuselier, Edward and Grady B Wright (2012). “Scattered data interpolation on embedded submanifolds with restricted positive definite kernels: Sobolev error estimates”. In: SIAM Journal on Numerical Analysis 50.3, pp. 1753–1776. Garthwaite, Paul H. and Stephen T. Buckland (1992). “Generating Monte Carlo Confidence Intervals by the Robbins–Monro Process”. In: Journal of the Royal Statistical Society 41.1, pp. 159–171. Gärtner, B. (1999). “Fast and robust smallest enclosing balls”. In: European Symposium on Algorithms. Springer, pp. 325–338. Geng, Xinbo and Le Xie (2019a). “Data-driven decision making in power systems with probabilistic guarantees: Theory and applications of chance-constrained optimization”. In: Annual Reviews in Control 47, pp. 341–363. issn: 1367-5788. doi: 10.1016/j.arcontrol.2019.05.005. url: http://dx.doi.org/10. 1016/j.arcontrol.2019.05.005. – (2019b). “Data-driven decision making in power systems with probabilistic guarantees: Theory and applications of chance-constrained optimization”. In: Annual reviews in control 47, pp. 341–363. Ghanem, Roger G and Pol D Spanos (2003). Stochastic finite elements: a spectral approach. Dover Publications. 316 Giesl, Peter and Holger Wendland (2007). “Meshless collocation: Error estimates with application to dynamical systems”. In: SIAM Journal on Numerical Analysis 45.4, pp. 1723–1741. Gilbarg, David and Neil S Trudinger (1977). Elliptic Partial Differential Equations of second order. Springer. Gin, Craig R, Daniel E Shea, Steven L Brunton, and J Nathan Kutz (2021). “DeepGreen: deep learning of Green’s functions for nonlinear boundary value problems”. In: Scientific reports 11.1, p. 21614. Gittell, Jody Hoffer and Hebatallah Naim Ali (2021). Relational analytics: Guidelines for analysis and action. Routledge. Glymour, Madelyn, Judea Pearl, and Nicholas P Jewell (2016). Causal inference in statistics: A primer. John Wiley & Sons. Goel, A., P. Indyk, and K. R. Varadarajan (2001). “Reductions among high dimensional proximity problems”. In: Proceedings of the twelfth annual ACM-SIAM symposium on Discrete algorithms. SIAM, pp. 769–778. Gouriéroux, Christian, Alberto Holly, and Alain Monfort (1982). “Likelihood Ratio Test, Wald Test, and Kuhn–Tucker Test in Linear Models with Inequality Constraints on the Regression Parameters”. In: Econometrica 50, pp. 63–80. Grossmann, Tamara G, Urszula Julia Komorowska, Jonas Latz, and Carola-Bibiane Schönlieb (2023). “Can Physics-Informed Neural Networks beat the Finite Element Method?” In: arXiv preprint arXiv:2302.04107. Gunzburger, Max D, Clayton G Webster, and Guannan Zhang (2014). “Stochastic finite element methods for partial differential equations with random input data”. In: Acta Numerica 23, pp. 521–650. Gutmann, Michael U. and Jukka Corander (2015). Bayesian Optimization for LikelihoodFree Inference of Simulator-Based Statistical Models. eprint: arXiv : 1501 . 03291. Hahn, Gerald J. and William Q. Meeker (1991). Statistical Intervals. Wiley. Hamzi, Boumediene, Romit Maulik, and Houman Owhadi (2021a). “Simple, lowcost and accurate data-driven geophysical forecasting with learned kernels”. In: Proceedings of the Royal Society A 477.2252, p. 20210326. – (2021b). “Simple, low-cost and accurate data-driven geophysical forecasting with learned kernels”. In: Proceedings of the Royal Society A 477.2252, p. 20210326. Hamzi, Boumediene and Houman Owhadi (2021). “Learning dynamical systems from data: a simple cross-validation perspective, part I: parametric kernel flows”. In: Physica D: Nonlinear Phenomena 421, p. 132817. Hamzi, Boumediene, Houman Owhadi, and Yannis Kevrekidis (2023). “Learning dynamical systems from data: A simple cross-validation perspective, part iv: case with partial observations”. In: Physica D: Nonlinear Phenomena 454, p. 133853. 317 Hansen, Per Christian (1992). “Analysis of discrete ill-posed problems by means of the L-curve”. In: SIAM Review 34.4, pp. 561–580. Hansen, Per Christian and Dianne Prost O’Leary (1993). “The use of the L-curve in the regularization of discrete ill-posed problems”. In: SIAM Journal on Scientific Computing 14.6, pp. 1487–1503. Heinrich, Lukas (2022). “Learning Optimal Test Statistics in the Presence of Nuisance Parameters”. In: arXiv preprint arXiv:2203.13079. Hesthaven, J.S. and S. Ubbiali (2018). “Non-intrusive reduced order modeling of nonlinear problems using neural networks”. In: Journal of Computational Physics 363, pp. 55–78. Hesthaven, Jan S, Gianluigi Rozza, Benjamin Stamm, et al. (2016). Certified reduced basis methods for parametrized partial differential equations. Vol. 590. Springer. Höcker, Andreas and Vakhtang Kartvelishvili (1996). “SVD approach to data unfolding”. In: Nuclear Instruments and Methods in Physics Research A 372, pp. 469– 481. Hon, YC and Robert Schaback (2008). “Solvability of partial differential equations by meshless kernel methods”. In: Advances in Computational Mathematics 28.3, pp. 283–299. Huang, Daniel Zhengyu, Tapio Schneider, and Andrew M Stuart (2022). “Iterated Kalman methodology for inverse problems”. In: Journal of Computational Physics 463, p. 111262. Ishihara, Toru (2001). “Enumeration of hypergraphs”. In: European Journal of Combinatorics 22.4, pp. 503–509. Javanmard, Adel and Andrea Montanari (2014). “Confidence Intervals and Hypothesis Testing for High-Dimensional Regression”. In: Journal of Machine Learning Research 15.82, pp. 2869–2909. url: http : / / jmlr . org / papers / v15 / javanmard14a.html. Kadri, Hachem, Emmanuel Duflos, Philippe Preux, Stéphane Canu, Alain Rakotomamonjy, and Julien Audiffren (2016). “Operator-valued kernels for learning from functional response data”. In. Kalmikov, Alexander G. and Patrick Heimbach (2014). “A Hessian-based method for uncertainty quantification in global ocean state estimation”. In: SIAM Journal on Scientific Computing 36, S267–S295. Kannan, Ravindran and Hariharan Narayanan (2012). “Random Walks on Polytopes and an Affine Interior Point Method for Linear Programming”. In: Mathematics of Operations Research 37, pp. 1–20. 318 Kansa, Edward J (1990a). “Multiquadrics—A scattered data approximation scheme with applications to computational fluid-dynamics—I surface approximations and partial derivative estimates”. In: Computers & Mathematics with applications 19.8-9, pp. 127–145. – (1990b). “Multiquadrics—A scattered data approximation scheme with applications to computational fluid-dynamics—II solutions to parabolic, hyperbolic and elliptic partial differential equations”. In: Computers & mathematics with applications 19.8-9, pp. 147–161. Karr, A. F. (1983). “Extreme points of certain sets of probability measures, with applications”. In: Mathematics of Operations Research 8.1, pp. 74–85. Kaveh, Hojjat, Pau Batlle, Mateo Acosta, Pranav Kulkarni, Stephen J Bourne, and Jean Philippe Avouac (2024). “Induced seismicity forecasting with uncertainty quantification: Application to the Groningen gas field”. In: Seismological Research Letters 95.2A. P.B contributed to the conceptualization of the Uncertainty Quantification algorithm and writing of its corresponding section, pp. 773–790. doi: https : //doi.org/10.1785/0220230179. Kempf, Rüdiger, Holger Wendland, and Christian Rieger (2019). “Kernel-based reconstructions for parametric PDEs”. In: IWMMPDE 2017: Meshfree Methods for Partial Differential Equations IX 9. Springer, pp. 53–71. Kempthorne, P. J. (1987). “Numerical specification of discrete least favorable prior distributions”. In: SIAM Journal on Scientific and Statistical Computing 8.2, pp. 171–184. Kennedy, Marc C and Anthony O’Hagan (2001). “Bayesian calibration of computer models”. In: Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63.3, pp. 425–464. Khoo, Yuehaw, Jianfeng Lu, and Lexing Ying (2021). “Solving parametric PDE problems with artificial neural networks”. In: European Journal of Applied Mathematics 32.3, pp. 421–435. Khoo, Yuehaw and Lexing Ying (2019). “SwitchNet: a neural network model for forward and inverse scattering problems”. In: SIAM Journal on Scientific Computing 41.5, A3182–A3201. Kibzun, A I and Y S Kan (Aug. 1997). “Stochastic Programming Problems with Probability and Quantile Functions”. In: Journal of the Operational Research Society 48.8, pp. 849–849. issn: 1476-9360. doi: 10.1057/palgrave.jors. 2600833. url: http://dx.doi.org/10.1057/palgrave.jors.2600833. Kingma, D. P. and J. Ba (2014). “Adam: A method for stochastic optimization”. In: arXiv preprint arXiv:1412.6980. 319 Kissas, Georgios, Jacob H Seidman, Leonardo Ferreira Guilhoto, Victor M Preciado, George J Pappas, and Paris Perdikaris (2022). “Learning operators with coupled attention”. In: Journal of Machine Learning Research 23.215, pp. 1–63. Koenker, R. and G. Bassett Jr (1978). “Regression quantiles”. In: Econometrica: journal of the Econometric Society, pp. 33–50. Koenker, Roger (2005). Quantile regression. Vol. 38. Cambridge university press. Koenker, Roger and Kevin F. Hallock (2001). “Quantile Regression”. In: Journal of Economic Perspectives 15.4, pp. 143–156. Kovachki, Nikola, Samuel Lanthaler, and Siddhartha Mishra (2021). “On universal approximation and error bounds for fourier neural operators”. In: The Journal of Machine Learning Research 22.1, pp. 13237–13312. Kovachki, Nikola, Burigede Liu, Xingsheng Sun, Hao Zhou, Kaushik Bhattacharya, Michael Ortiz, and Andrew Stuart (2022). “Multiscale modeling of materials: Computing, data science, uncertainty and goal-oriented optimization”. In: Mechanics of Materials 165, p. 104156. Krischer, K, R Rico-Martínez, IG Kevrekidis, HH Rotermund, G Ertl, and JL Hudson (1993). “Model identification of a spatiotemporally varying catalytic reaction”. In: AIChE Journal 39.1, pp. 89–98. Kröpfl, Fabian, Roland Maier, and Daniel Peterseim (2022). “Operator compression with deep neural networks”. In: Advances in Continuous and Discrete Models 2022.1, pp. 1–23. Kuusela, Mikael (July 2016). “Uncertainty quantification in unfolding elementary particle spectra at the Large Hadron Collider”. PhD thesis. École Polytechnique Fédérale de Lausanne. Kuusela, Mikael and Victor M Panaretos (2015a). “Statistical unfolding of elementary particle spectra: Empirical Bayes estimation and bias-corrected uncertainty quantification”. In: The Annals of Applied Statistics 9.3, pp. 1671–1705. – (2015b). “Statistical Unfolding of Elementary Particle Spectra: Empirical Bayes Estimation and Bias-Corrected Uncertainty Quantification”. In: The Annals of Applied Statistics 9.3, pp. 1671–1705. Kuusela, Mikael and Philip B. Stark (2017a). “Shape-constrained uncertainty quantification in unfolding steeply falling elementary particle spectra”. In: Annals of Applied Statistics. – (2017b). “Shape-constrained uncertainty quantification in unfolding steeply falling elementary particle spectra”. In: The Annals of Applied Statistics 11.3, pp. 1671– 1710. Larkin, F. M. (1972). “Gaussian measure in Hilbert space and applications in numerical analysis”. In: Journal of Mathematics 2.3. 320 Le Maitre, Olivier and Omar M Knio (2010). Spectral methods for uncertainty quantification: with applications to computational fluid dynamics. Springer Science & Business Media. Lee, Jaehoon, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein (2017). “Deep neural networks as Gaussian processes”. In: arXiv preprint arXiv:1711.00165. Lee, John M (2012). Introduction to Smooth Manifolds. Springer. Lehmann, E. L. and Joseph P. Romano (2008). Testing Statistical Hypotheses. Springer. Li, Zongyi, Nikola Kovachki, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar (2020). “Fourier Neural Operator for Parametric Partial Differential Equations”. In: International Conference on Learning Representations. Li, Zongyi, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar (2020). Fourier Neural Operator for Parametric Partial Differential Equations. Lim, T. and R. J. McCann (2021). “Geometrical Bounds for Variance and Recentered Moments”. In: Mathematics of Operations Research. Ling, Leevan, Roland Opfer, and Robert Schaback (2006). “Results on meshless collocation techniques”. In: Engineering Analysis with Boundary Elements 30.4, pp. 247–253. Ling, Leevan and Robert Schaback (2008). “Stable and convergent unsymmetric meshless collocation methods”. In: SIAM Journal on Numerical Analysis 46.3, pp. 1097–1115. Liu, Junjie, Kevin Bowman, Lee Meemong, et al. (2016). “Carbon Monitoring system flux estimation and attribution: impact of ACOS-GOSAT X𝐶𝑂 2 sampling on the inference of terrestrial biospheric sources and sinks”. In: Tellus B: Chemical and Physical Meteorology 66. doi: https://doi.org/10.3402/tellusb. v66.22486. Loman, Torkel E., Yingbo Ma, Vasily Ilin, Shashi Gowda, Niklas Korsbo, Nikhil Yewale, Chris Rackauckas, and Samuel A. Isaacson (Oct. 2023). Catalyst: Fast and flexible modeling of reaction networks. en. Ed. by Christos A. Ouzounis. doi: 10.1371/journal.pcbi.1011530. url: http://dx.doi.org/10.1371/ journal.pcbi.1011530. Long, Da, Nicole Mrvaljevic, Shandian Zhe, and Bamdad Hosseini (2022). “A Kernel Approach for PDE Discovery and Operator Learning”. In: arXiv preprint arXiv:2210.08140. 321 Long, Da, Zheng Wang, Aditi Krishnapriyan, Robert Kirby, Shandian Zhe, and Michael Mahoney (2022). “AutoIP: A United Framework to Integrate Physics into Gaussian Processes”. In: International Conference on Machine Learning. PMLR, pp. 14210–14222. Lopez-Paz, David, Krikamol Muandet, Bernhard Schölkopf, and Iliya Tolstikhin (2015). “Towards a learning theory of cause-effect inference”. In: International Conference on Machine Learning. PMLR, pp. 1452–1461. Lotka, A. J. (1920). “Analytical Note on Certain Rhythmic Relations in Organic Systems”. In: Proceedings of the National Academy of Sciences of the United States of America 6.7, pp. 410–415. Lovasz, Laszlo (1999). “Hit-and-run mixes fast”. In: Mathematical Programming 86, pp. 443–461. Lovasz, Laszlo and Santosh Vempala (2006). “Hit-and-Run from a Corner”. In: SIAM Journal on Computing 35, pp. 985–1005. Lu, Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis (Mar. 2021). “Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators”. In: Nature Machine Intelligence 3.3, pp. 218–229. Lu, Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis (2022). “A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data”. In: Computer Methods in Applied Mechanics and Engineering 393, p. 114778. issn: 0045-7825. Lu, Yiping, Haoxuan Chen, Jianfeng Lu, Lexing Ying, and Jose Blanchet (2021). “Machine learning for elliptic PDEs: fast rate generalization bound, neural scaling law and minimax optimality”. In: arXiv preprint arXiv:2110.06897. Lucia, David J, Philip S Beran, and Walter A Silva (2004). “Reduced-order modeling: new approaches for computational physics”. In: Progress in aerospace sciences 40.1-2, pp. 51–117. Maday, Yvon, Anthony T Patera, and Gabriel Turinici (2002). “A priori convergence theory for reduced-basis approximations of single-parameter elliptic partial differential equations”. In: Journal of Scientific Computing 17, pp. 437–446. Målqvist, Axel and Daniel Peterseim (2014). “Localization of elliptic multiscale problems”. In: Mathematics of Computation 83.290, pp. 2583–2603. Martin, James, Lucas C Wilcox, Carsten Burstedde, and Omar Ghattas (2012). “A stochastic Newton MCMC method for large-scale statistical inverse problems with application to seismic inversion”. In: SIAM Journal on Scientific Computing 34.3, A1460–A1487. 322 Martinez-Cantin, Ruben (2014). BayesOpt: A Bayesian Optimization Library for Nonlinear Optimization, Experimental Design and Bandits. eprint: arXiv:1405. 7430. Martins, Joaquim RRA and Andrew B Lambe (2013). “Multidisciplinary design optimization: a survey of architectures”. In: AIAA journal 51.9, pp. 2049–2075. Marzari, Nicola, Arash A Mostofi, Jonathan R Yates, Ivo Souza, and David Vanderbilt (2012). “Maximally localized Wannier functions: Theory and applications”. In: Reviews of Modern Physics 84.4, p. 1419. Masserano, Luca, Tommaso Dorigo, Rafael Izbicki, Mikael Kuusela, and Ann Lee (2023). “Simulation-Based Inference with WALDO: Confidence Regions by Leveraging Prediction Algorithms or Posterior Estimators for Inverse Problems”. In: The International Conference on Artificial Intelligence and Statistics. Masserano, Luca, Alexander Shen, Michele Doro, Tommaso Dorigo, Rafael Izbiki, and Ann Lee (2024). “Classification Under Nuisance Parameters and Generalized Label Shift in Likelihood-Free Inference”. In: arXiv preprint arXiv:2402.05330. McLean, William (2000). Strongly elliptic systems and boundary integral equations. Cambridge university press. Meinshausen, Nicolai (2006). “Quantile Regression Forests”. In: Journal of Machine Learning Research 7, pp. 983–999. Melenk, Jens M (2000). “On n-widths for elliptic problems”. In: Journal of mathematical analysis and applications 247.1, pp. 272–289. Meng, Rui and Xianjin Yang (2022). “Sparse Gaussian processes for solving nonlinear PDEs”. In: arXiv preprint arXiv:2205.03760. Micchelli, Charles A and Theodore J Rivlin (1977). “A survey of optimal recovery”. In: Optimal estimation in approximation theory, pp. 1–54. Mika, Sebastian, Bernhard Schölkopf, Alexander J Smola, Klaus-Robert Müller, Matthias Scholz, and Gunnar Rätsch (1998). “Kernel PCA and De-noising in feature spaces.” In: NIPS. Vol. 11, pp. 536–542. Molenberghs, Geert and Geert Verbeke (2007). “Likelihood ratio, score, and Wald tests in a constrained parameter space”. In: The American Statistician 61.1, pp. 22– 27. Montel, Noemi Anau, James Alvey, and Christoph Weniger (2023). “Scalable Inference with Autoregressive Neural Ratio Estimation”. In: arXiv preprint arXiv:2308:08597v1. Morgan, Stephen L and Christopher Winship (2015). Counterfactuals and causal inference. Cambridge University Press. Mou, Chenchen, Xianjin Yang, and Chao Zhou (2022). “Numerical methods for mean field games based on Gaussian processes and Fourier features”. In: Journal of Computational Physics 460, p. 111188. 323 Muandet, Krikamol, Kenji Fukumizu, Bharath Sriperumbudur, Bernhard Schölkopf, et al. (2017). “Kernel mean embedding of distributions: A review and beyond”. In: Foundations and Trends® in Machine Learning 10.1-2, pp. 1–141. Murphy, S. A. (1995). “Likelihood Ratio-Based Confidence Intervals in Survival Analysis”. In: Journal of the American Statistical Association 90.432, pp. 1399– 1405. doi: 10 . 1080 / 01621459 . 1995 . 10476645. eprint: https : / / www . tandfonline.com/doi/pdf/10.1080/01621459.1995.10476645. url: https://www.tandfonline.com/doi/abs/10.1080/01621459.1995. 10476645. Narayanan, Hariharan (2016). “Ranomized Interior Point Methods for Sampling and Optimization”. In: The Annals of Applied Probability 26, pp. 597–641. Neal, Radford M (1996). “Priors for infinite networks”. In: Bayesian learning for neural networks, pp. 29–53. Neale, Michael C. and Michael B. Miller (1997). “The use of likelihood-based confidence intervals in genetic models”. In: Behavior Genetics 27, pp. 113–120. Neumann, J. von (1928). “Zur Theorie der Gesellschaftsspiele”. In: Math. Ann. 100.1, pp. 295–320. issn: 0025-5831. doi: 10.1007/BF01448847. url: http: //dx.doi.org/10.1007/BF01448847. Nobile, Fabio, Raul Tempone, and Clayton G Webster (2008). “An anisotropic sparse grid stochastic collocation method for partial differential equations with random input data”. In: SIAM Journal on Numerical Analysis 46.5, pp. 2411–2442. Nobile, Fabio, Raúl Tempone, and Clayton G Webster (2008). “A sparse grid stochastic collocation method for partial differential equations with random input data”. In: SIAM Journal on Numerical Analysis 46.5, pp. 2309–2345. Nocedal, J. and S. Wright (2006). Numerical Optimization. Springer Science & Business Media. Noor, Ahmed K and Jeanne M Peters (1980). “Reduced basis technique for nonlinear analysis of structures”. In: Aiaa journal 18.4, pp. 455–462. O’Leary, Dianne P. and Bert W. Rust (1986). “Confidence Intervals for InequalityConstrained Least Squares Problems, with Applications to Ill-Posed Problems”. In: SIAM Journal on Scientific Computing 7.2, pp. 473–489. Ok, Efe A. (2007). Real Analysis with Economic Applications. English. 1st. Econometric Society Monographs. Includes bibliographical references and index. Princeton, NJ: Princeton University Press, p. 832. isbn: 978-0-691-12949-5. url: https : / / press . princeton . edu / books / hardcover / 9780691129495 / real-analysis-with-economic-applications. Owen, Art B (2013). “Variance components and generalized Sobol’indices”. In: SIAM/ASA Journal on Uncertainty Quantification 1.1, pp. 19–41. 324 Owhadi, H. and C. Scovel (2016). “Brittleness of Bayesian inference and new Selberg formulas”. In: Communications in Mathematical Sciences 14.1, pp. 83–145. – (2017a). “Extreme points of a ball about a measure with finite support”. In: Communications in Mathematical Sciences 15.1. arXiv:1504.06745, pp. 77–96. – (2017b). “Qualitative robustness in Bayesian inference”. In: ESAIM: Probability and Statistics 21, pp. 251–274. – (2017c). “Toward Machine Wald”. In: Handbook of Uncertainty Quantification. Ed. by Owhadi H. Ghanem R. Higdon D. arXiv:1508.02449. Springer, pp. 157– 191. – (2019). Operator Adapted Wavelets, Fast Solvers, and Numerical Homogenization, from a game theoretic approach to numerical approximation and algorithm design. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press. Owhadi, H., C. Scovel, and F. Schäfer (2019). “Statistical Numerical Approximation”. In: Notices of the AMS 66.10. Owhadi, H., C. Scovel, and T. Sullivan (2015a). “Brittleness of Bayesian inference under finite information in a continuous world”. In: Electronic Journal of Statistics 9.1, pp. 1–79. – (2015b). “On the brittleness of Bayesian inference”. In: SIAM Review 57.4, pp. 566–582. Owhadi, H., C. Scovel, T. J. Sullivan, M. McKerns, and M. Ortiz (2013a). “Optimal Uncertainty Quantification”. In: SIAM Review 55.2, pp. 271–345. doi: 10.1137/ 10080782X. – (2013b). “Optimal uncertainty quantification”. In: Siam Review 55.2, pp. 271– 345. Owhadi, Houman (2015a). “Bayesian numerical homogenization”. In: Multiscale Modeling & Simulation 13.3, pp. 812–828. – (2015b). “Bayesian numerical homogenization”. In: Multiscale Modeling & Simulation 13.3, pp. 812–828. – (2017). “Multigrid with rough coefficients and multiresolution operator decomposition from hierarchical information games”. In: Siam Review 59.1, pp. 99– 149. – (2022). “Computational graph completion”. In: Research in the Mathematical Sciences 9.2, p. 27. – (2023a). “Do ideas have shape? Idea registration as the continuous limit of artificial neural networks”. In: Physica D: Nonlinear Phenomena 444, p. 133592. – (2023b). “Do ideas have shape? Idea registration as the continuous limit of artificial neural networks”. In: Physica D: Nonlinear Phenomena 444, p. 133592. 325 Owhadi, Houman and Clint Scovel (2019a). Operator-Adapted Wavelets, Fast Solvers, and Numerical Homogenization: From a Game Theoretic Approach to Numerical Approximation and Algorithm Design. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press. – (2019b). Operator-Adapted Wavelets, Fast Solvers, and Numerical Homogenization: From a Game Theoretic Approach to Numerical Approximation and Algorithm Design. Cambridge University Press. Owhadi, Houman, Clint Scovel, and Tim Sullivan (2015a). “Brittleness of Bayesian inference under finite information in a continuous world”. In: Electronic Journal of Statistics 9, pp. 1–79. – (2015b). “On the brittleness of Bayesian inference”. In: SIAM Review 57.4, pp. 566–582. Owhadi, Houman, Clint Scovel, and Gene Ryan Yoo (2021). Kernel Mode Decomposition and the programming of kernels. Springer. Owhadi, Houman and Gene Ryan Yoo (2019a). “Kernel flows: from learning kernels from data into the abyss”. In: Journal of Computational Physics 389, pp. 22–47. – (2019b). “Kernel flows: from learning kernels from data into the abyss”. In: Journal of Computational Physics 389, pp. 22–47. Owhadi, Houman and Lei Zhang (2007). “Metric-based upscaling”. In: Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences 60.5, pp. 675–723. – (2017). “Gamblets for opening the complexity-bottleneck of implicit schemes for hyperbolic and parabolic ODEs/PDEs with rough coefficients”. In: Journal of Computational Physics 347, pp. 99–128. Pakman, Ari and Liam Paninski (2014). “Exact Hamiltonian Monte Carlo for Truncated Multivariate Gaussians”. In: Journal of Computational and Graphical Statistics 23, pp. 518–542. Palais, Richard S. (1997). The Symmetries of Solitons. arXiv: dg - ga / 9708004 [dg-ga]. Panaretos, Victor M. (2016). Statistics for Mathematicians. Springer. doi: 10 . 1007/978- 3- 319- 28341- 8. url: https://doi.org/10.1007/978- 3319-28341-8. Pass, B. (2020). “Generalized barycenters and variance maximization on metric spaces”. In: arXiv:2006.02984. Paszke, A., S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala (2019). “PyTorch: An Imperative Style, High-Performance Deep Learning Library”. In: Advances in Neural Information Processing Systems 32. Ed. by H. Wallach, H. 326 Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett. Curran Associates, Inc., pp. 8024–8035. url: http://papers.neurips.cc/paper/ 9015- pytorch- an- imperative- %20style- high- performance- deeplearning-library.pdf. Patil, Pratik, Mikael Kuusela, and Jonathan Hobbs (2022). “Objective frequentist uncertainty quantification for atmospheric CO2 retrievals”. In: SIAM/ASA Journal on Uncertainty Quantification 10. doi: 10.1137/20M1356403. Pearl, Judea (2009). Causality. Cambridge university press. Pedregosa, F., G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay (2011). “Scikit-learn: Machine Learning in Python”. In: Journal of Machine Learning Research 12, pp. 2825–2830. Pena-Ordieres, Alejandra, James R Luedtke, and Andreas Wachter (2020). “Solving chance-constrained problems via a smooth sample-based nonlinear approximation”. In: SIAM Journal on Optimization 30.3, pp. 2221–2250. Peters, Jonas, Dominik Janzing, and Bernhard Schölkopf (2017). Elements of causal inference: foundations and learning algorithms. The MIT Press. Pinkus, Allan (2012). N-widths in Approximation Theory. Vol. 7. Springer Science & Business Media. Popoviciu, T. (1935). “Sur les équations algébriques ayant toutes leurs racines réelles”. In: Mathematica (Cluj), pp. 129–145. Prasanth, Sai, Ziad Haddad, Jouni Susiluoto, Amy Braverman, Houman Owhadi, Boumediene Hamzi, Svetla Hristova-Veleva, and Joseph Turk (2021). “Kernel flows to infer the structure of convective storms from satellite passive microwave observations”. In: AGU Fall Meeting Abstracts. Vol. 2021, A55F–1445. Raissi, Maziar, Paris Perdikaris, and George E Karniadakis (2019). “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations”. In: Journal of Computational Physics 378, pp. 686–707. Raissi, Maziar, Paris Perdikaris, and George Em Karniadakis (2018). “Numerical Gaussian processes for time-dependent and nonlinear partial differential equations”. In: SIAM Journal on Scientific Computing 40.1, A172–A198. Rasmussen, Carl Edward and Christopher K. I. Williams (2006). Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, pp. I–XVIII, 1–248. isbn: 026218253X. Ren, Christopher X, Sidhant Misra, Marc Vuffray, and Andrey Y Lokhov (2021). “Learning Continuous Exponential Families Beyond Gaussian”. In: arXiv preprint arXiv:2102.09198. 327 Reznikov, A and Edward B Saff (2016). “The covering radius of randomly distributed points on a manifold”. In: International Mathematics Research Notices 2016.19, pp. 6065–6094. Richter, Lorenz, Leon Sallandt, and Nikolas Nüsken (2021). “Solving high-dimensional parabolic PDEs using the tensor train format”. In: arXiv preprint arXiv:2102.11830. Robertson, Tim, F. T. Wright, and Richard Dykstra (1988). Order Restricted Statistical Inference. Wiley. Roch, Sebastien (2024). Modern Discrete Probability: An Essential Toolkit. To be published by Cambridge University Press. url: https://people.math.wisc. edu/~roch/mdp/index.html. Rockafellar, R. T. (1970). Convex Analysis. Princeton University Press. Rockafellar, R. T. and R. J.-B. Wets (1998). Variational Analysis. Vol. 317. Grundlehren der Mathematischen Wissenschaften. Berlin: Springer-Verlag. Rodgers, Clive D. (2000). Inverse Methods for Atmospheric Sounding. World Scientific Publishing. Rojo, J. (2006). “Optimality: The Second Erich L. Lehmann Symposium”. In: IMS. – (2009). “Optimality: The Third Erich L. Lehmann Symposium”. In: IMS. Rojo, J. and V. Pérez-Abreu (2004). “The First Erich L. Lehmann Symposium: Optimality”. In: IMS. Rossi, R. J. (2018). Mathematical Statistics: an Introduction to Likelihood Based Inference. John Wiley & Sons. Rust, Bert W. and Dianne P. O’Leary (1994). “Confidence intervals for discrete approximations to ill-posed problems”. In: Journal of Computational and Graphical Statistics 3.1, pp. 67–96. Rust, Burt W. and Walter R. Burrus (1972). Mathematical Programming and the Numerical Solution of Linear Equations. American Elsevier. Sachs, Karen, Omar Perez, Dana Pe’er, Douglas A Lauffenburger, and Garry P Nolan (2005). “Causal protein-signaling networks derived from multiparameter single-cell data”. In: Science 308.5721, pp. 523–529. Särkkä, Simo (2011). “Linear operators and stochastic partial differential equations in Gaussian process regression”. In: International Conference on Artificial Neural Networks. Springer, pp. 151–158. Schaäfer, Florian, Matthias Katzfuss, and Houman Owhadi (2021). “Sparse Cholesky Factorization by Kullback–Leibler Minimization”. In: SIAM Journal on scientific computing 43.3, A2019–A2046. Schaback, Robert (2007). “Convergence of unsymmetric kernel-based meshless collocation methods”. In: SIAM Journal on Numerical Analysis 45.1, pp. 333– 351. 328 Schaback, Robert (2015). “A computational tool for comparing all linear PDE solvers: Error-optimal methods are meshless”. In: Advances in Computational Mathematics 41, pp. 333–355. – (2016). “All well-posed problems have uniformly stable and convergent discretizations”. In: Numerische Mathematik 132.3, pp. 597–630. Schaback, Robert and Holger Wendland (2006). “Kernel techniques: from machine learning to meshless methods”. In: Acta numerica 15, p. 543. Schafer, Chad M. and Philip B. Stark (2009). “Constructing confidence regions of optimal expected size”. In: Journal of the American Statistical Association 104.487, pp. 1080–1089. Schäfer, Florian, Matthias Katzfuss, and Houman Owhadi (2021). “Sparse Cholesky Factorization by Kullback–Leibler Minimization”. In: SIAM Journal on Scientific Computing 43.3, A2019–A2046. Schäfer, Florian and Houman Owhadi (2021). “Sparse recovery of elliptic solvers from matrix-vector products”. In: arXiv preprint arXiv:2110.05351. – (2023). “Sparse recovery of elliptic solvers from matrix-vector products”. In: SIAM Journal on Scientific Computing. Schäfer, Florian, Timothy John Sullivan, and Houman Owhadi (2021a). “Compression, inversion, and approximate PCA of dense kernel matrices at near-linear computational complexity”. In: Multiscale Modeling & Simulation 19.2, pp. 688– 730. – (2021b). “Compression, inversion, and approximate PCA of dense kernel matrices at near-linear computational complexity”. In: Multiscale Modeling & Simulation 19.2, pp. 688–730. – (2021c). “Compression, inversion, and approximate PCA of dense kernel matrices at near-linear computational complexity”. In: Multiscale Modeling & Simulation 19.2, pp. 688–730. Schervish, Mark J. (1995). Theory of Statistics. Springer. Schmitt, S. (2012). “TUnfold, an algorithm for correcting migration effects in high energy physics”. In: Journal of Instrumentation 7, T10003. Scholkopf, Bernhard and Alexander J Smola (2018). Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT Press. Schölkopf, Bernhard, Ralf Herbrich, and Alex J. Smola (2001). “A Generalized Representer Theorem”. In: Computational Learning Theory. Ed. by David Helmbold and Bob Williamson. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 416– 426. isbn: 978-3-540-44581-4. Schott, J. R. (2016). Matrix Analysis for Statistics. John Wiley & Sons. 329 Schweiger, Regev, Eyal Fisher, Elior Rahmani, Liat Shenhav, Saharon Rosset, and Eran Halperin (2018). “Using Stochastic Approximation Techniques to Efficiently Construct Confidence Intervals for Heritability”. In: Journal of Computational Biology 25.7, pp. 794–808. Schweitzer, Frank, Giorgio Fagiolo, Didier Sornette, Fernando Vega-Redondo, Alessandro Vespignani, and Douglas R White (2009). “Economic networks: The new challenges”. In: science 325.5939, pp. 422–425. SHAH, RAJEN D and JONAS PETERS (2020). “THE HARDNESS OF CONDITIONAL INDEPENDENCE TESTING AND THE GENERALISED COVARIANCE MEASURE”. In: The Annals of Statistics 48.3, pp. 1514–1538. Shaked, M. and J.G. Shanthikumar (2007). Stochastic Orders. Springer Series in Statistics. Springer New York. isbn: 9780387346755. Shapiro, A. and A. Kleywegt (2002). “Minimax analysis of stochastic problems”. In: Optimization Methods and Software 17.3, pp. 523–542. Shapiro, Alexander (1988). “Towards a unified theory of inequality constrained testing in multivariate analysis”. In: International Statistical Review, pp. 49–62. Shin, Yeonjong, Jerome Darbon, and George Em Karniadakis (2020). “On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs”. In: arXiv preprint arXiv:2004.01806. Silvapulle, Mervyn J. and Pranab Kumar Sen (2011). Constrained Statistical Inference: Order, Inequality, and Shape Constraints. Wiley. Sion, M. (1958). “On general minimax theorems”. In: Pacific J. Math 8.1, pp. 171– 176. Skilling, J. (2004). “Nested Sampling”. In: AIP Conference Proceedings 735, p. 295. Smith, Robert L. (1984). “Efficient Monte Carlo Procedures for Generating Points Uniformly Distributed over Bounded Regions”. In: Operations Research 32, pp. 1296–1308. Sobol, Ilya M (2001). “Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates”. In: Mathematics and computers in simulation 55.1-3, pp. 271–280. Soboĺ, IM (1993). “Sensitivity estimates for nonlinear mathematical models”. In: Math. Model. Comput. Exp. 1, p. 407. Spano, Francesco (2014). “Unfolding in particle physics: a window on solving inverse problems.” en. In: Proc. of Top 2013, 256–261, DESY. doi: 10.3204/ DESY-PROC-2014-02/52. url: http://www-library.desy.de/preparch/ desy/proc/proc14-02/P52.pdf. Spirtes, Peter and Clark Glymour (1991). “An algorithm for fast recovery of sparse causal graphs”. In: Social science computer review 9.1, pp. 62–72. 330 Sprott, D. A. (2008). Statistical Inference in Science. Springer Verlag. Stanley, Michael, Pau Batlle, Pratik Patil, Houman Owhadi, and Mikael Kuusela (2025). “Confidence intervals for functionals in constrained inverse problems via data-adaptive sampling-based calibration”. In: arXiv preprint arXiv:2502.02674. P.B contributed to conceptualization, proof of the theoretical results, and numerical experiments. Stanley, Michael, Pratik Patil, and Mikael Kuusela (2022). “Uncertainty quantification for wide-bin unfolding: one-at-a-time strict bounds and prior-optimized confidence intervals”. In: Journal of Instrumentation 17. doi: 10.1088/17480221/17/10/P10013. Stark, P. (2020). “Your Prior Can Bite You on the Posterior: Contrasting Bayesian and Frequentist Measures of Uncertainty”. In: JPL Science Visitor and Colloquium Program - Earth Science Seminar, Sept. 1, 2020. https://www.stat. berkeley.edu/~stark/Seminars/uqJPL20.slides.html/. Stark, Philip B. (1992a). “Inference in Infinite-Dimensional Inverse Problems: Discretization and Duality”. In: Journal of Geophysical Research 97.B10, pp. 14055– 14082. – (1992b). “Inference in infinite-dimensional inverse problems: discretization and duality”. In: Journal of Geophysical Research: Solid Earth 97.B10, pp. 14055– 14082. – (1994). Simultaneous confidence intervals for linear estimates of linear functionals. Tech. rep. Citeseer. – (2015). “Constraints versus Priors”. In: SIAM/ASA Journal on Uncertainty Quantification 3 (1). Stegle, Oliver, Dominik Janzing, Kun Zhang, Joris M Mooij, and Bernhard Schölkopf (2010). “Probabilistic latent variable models for distinguishing between cause and effect”. In: Advances in neural information processing systems 23. Steinwart, Ingo and Andreas Christmann (2011). “Estimating conditional quantiles with the help of the pinball loss”. In: Bernoulli 117.1, pp. 211–225. Storn, R. and K. Price (1997). “Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces”. In: Journal of Global Optimization 11.4, pp. 341–359. Stuart, Andrew M. (2010). “Inverse Problems: A Bayesian Perspective”. In: Acta Numerica 19, pp. 451–559. Sudret, Bruno, Stefano Marelli, and Joe Wiart (2017). “Surrogate models for uncertainty quantification: An overview”. In: 2017 11th European conference on antennas and propagation (EUCAP). IEEE, pp. 793–797. 331 Susiluoto, Jouni, Amy Braverman, Philip Brodrick, Boumediene Hamzi, Maggie Johnson, Otto Lamminpaa, Houman Owhadi, Clint Scovel, Joaquim Teixeira, and Michael Turmon (2021). “Radiative transfer emulation for hyperspectral imaging retrievals with advanced kernel flows-based gaussian process emulation”. In: AGU Fall Meeting Abstracts. Vol. 2021, NG25A–0506. Swiler, Laura P, Mamikon Gulian, Ari L Frankel, Cosmin Safta, and John D Jakeman (2020). “A Survey of Constrained Gaussian Process Regression: Approaches and Implementation Challenges”. In: Journal of Machine Learning for Modeling and Computing 1.2. Sylvester, J. J. (1857). “A question in the geometry of situation”. In: Quarterly Journal of Pure and Applied Mathematics 1.1, pp. 79–80. Takeuchi, Ichiro, Quoc V. Le, Timothy D. Sears, and Alexander J. Smola (2006). “Nonparametric Quantile Estimation”. In: Journal of Machine Learning 7, pp. 1231– 1264. Tarantola, Albert (2005). Inverse problem theory and methods for model parameter estimation. SIAM. Taylor, Michael (2013). Partial differential equations I: Basic theory. Springer Science & Business Media. Tenorio, L., A. Fleck, and K. Moses (2007). “Confidence intervals for linear discrete inverse problems with a non-negativity constraint”. In: Inverse Problems 23.2, p. 669. Thomas, Owen, Ritabrata Dutta, Jukka Corander, Samuel Kaski, and Michael U. Gutmann (2022). “Likelihood-Free Inference by Ratio Estimation”. In: Bayesian Analysis 17.1, pp. 1–31. Vaart, Aad W van der and J Harry van Zanten (2008). “Reproducing Kernel Hilbert Spaces of Gaussian priors”. In: Pushing the limits of contemporary statistics: contributions in honor of Jayanta K. Ghosh. Institute of Mathematical Statistics, pp. 200–222. Vecchia, Aldo V (1988). “Estimation and model identification for continuous spatial processes”. In: Journal of the Royal Statistical Society Series B: Statistical Methodology 50.2, pp. 297–312. Venzon, D. J. and S. H. Moolgavkar (1988). “A method for computing profilelikelihood-based confidence intervals”. In: Journal of the Royal Statistical Society: Series C 37.1, pp. 87–94. Virtanen, P., R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, I. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors (2020). 332 “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python”. In: Nature Methods 17, pp. 261–272. doi: 10.1038/s41592-019-0686-2. Voelker, Aaron R., Jan Gosmann, and Terrence C. Stewart (2017). Simultaneous confidence intervals for linear estimates of linear functionals. Tech. rep. Centre for Theoretical Neuroscience. Wahba, Grace (2003). “An introduction to smoothing spline anova models in rkhs, with examples in geographical data, medicine, atmospheric sciences and machine learning”. In: IFAC Proceedings Volumes 36.16, pp. 531–536. Wald, A. (1945). “Statistical decision functions which minimize the maximum risk”. In: Annals of Mathematics, pp. 265–280. Wald, A. and J. Wolfowitz (1951). “Characterization of the Minimal Complete Class of Decision Functions When the Number of Distributions and Decisions Is Finite”. In: Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability. Berkeley, Calif.: University of California Press, pp. 149–157. Wald, Abraham (1943). “Tests of Statistical Hypotheses Concerning Several Parameters When the Number of Observations is Large”. In: Transactions of the American Mathematical Society 54, pp. 426–482. Wang, Junyang, Jon Cockayne, Oksana Chkrebtii, Timothy John Sullivan, Chris Oates, et al. (2021). “Bayesian numerical methods for nonlinear partial differential equations”. In: Statistics and Computing 31.5, pp. 1–20. Wang, Sifan, Hanwen Wang, and Paris Perdikaris (2021). “Learning the solution operator of parametric partial differential equations with physics-informed deeponets”. In: Science advances 7.40, eabi8605. – (2022). “Improved architectures and training algorithms for deep operator networks”. In: Journal of Scientific Computing 92.2, p. 35. Wasserman, Larry (2004). All of Statistics: A Concise Course in Statistical Inference. Vol. 26. Springer. Wasserman, Larry, Aaditya Ramdas, and Sivaraman Balakrishnan (2020). “Universal Inference”. In: PNAS 117, pp. 16880–16890. Weinan, E (2011). Principles of multiscale modeling. Cambridge University Press. Weinan, E, Jiequn Han, and Arnulf Jentzen (2017). “Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations”. In: Communications in Mathematics and Statistics 5.4, pp. 349–380. Weinan, E and Bing Yu (2018). “The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems”. In: Communications in Mathematics and Statistics 6.1, pp. 1–12. 333 Weizsäcker, H. von and G. Winkler (1979). “Integral representation in the set of solutions of a generalized moment problem”. In: Mathematische Annalen 246.1, pp. 23–32. Welzl, E. (1991). “Smallest enclosing disks (balls and ellipsoids)”. In: New Results and New Trends in Computer Science. Springer, pp. 359–370. Wendland, Holger (2004). Scattered Data Approximation. Cambridge University Press. Williams, Christopher K. I. and Carl Edward Rasmussen (2006). Gaussian Processes for Machine Learning. The MIT Press. Wilson, Andrew Gordon, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing (2016). “Deep kernel learning”. In: Artificial Intelligence and Statistics. PMLR, pp. 370– 378. Wolak, Frank A. (1987). “An Exact Test for Multiple Inequality and Equality Constraints in the Linear Regression Model”. In: Journal of the American Statistical Association 82, pp. 782–793. – (1989). “Testing Inequality Constraints in Linear Econometric Models”. In: Journal of Econometrics 41, pp. 205–235. Wu, Zong-min and Robert Schaback (1993). “Local error estimates for radial basis function interpolation of scattered data”. In: IMA journal of Numerical Analysis 13.1, pp. 13–27. Xiu, Dongbin (2010). Numerical Methods for Stochastic Computations: A Spectral Method Approach. Princeton University Press. Xiu, Dongbin and George Em Karniadakis (2002). “The Wiener–Askey polynomial chaos for stochastic differential equations”. In: SIAM journal on scientific computing 24.2, pp. 619–644. Xiu, Dongbin and Jie Shen (2009). “Efficient stochastic Galerkin methods for random diffusion equations”. In: Journal of Computational Physics 228.2, pp. 266– 281. Ye, Qi (2013). “Kernel-based methods for stochastic partial differential equations”. In: arXiv preprint arXiv:1303.5381. Yildirim, E. A. (2008). “Two algorithms for the minimum enclosing ball problem”. In: SIAM Journal on Optimization 19.3, pp. 1368–1391. Yu, Ming, Varun Gupta, and Mladen Kolar (2019). “Constrained high dimensional statistical inference”. In: arXiv preprint arXiv:1911.07319. Zhang, K, J Peters, D Janzing, and B Schölkopf (2011). “Kernel-based Conditional Independence Test and Application in Causal Discovery”. In: 27th Conference on Uncertainty in Artificial Intelligence (UAI 2011). AUAI Press, pp. 804–813. 334 Zhu, Yinhao and Nicholas Zabaras (2018). “Bayesian deep convolutional encoder– decoder networks for surrogate modeling and uncertainty quantification”. In: Journal of Computational Physics 366, pp. 415–447.