Learning to Sample in Computational Imaging: Measurement Acquisition and Posterior Estimation Citation Wu, Zihui (2026) Learning to Sample in Computational Imaging: Measurement Acquisition and Posterior Estimation. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/j42z-s192. https://resolver.caltech.edu/CaltechTHESIS:09062025-050711919 Abstract Many problems in science and engineering require visualizing objects that are not directly observable—such as black holes that are millions of light-years away from the Earth or internal anatomical structures hidden within the human body. Computational imaging is a powerful paradigm that combines sensor design with advanced computational algorithms to make the invisible visible. The typical computational imaging pipeline involves first collecting indirect measurements of the target object and then solving a reconstruction problem. This thesis focuses on two core challenges about sampling along this pipeline: (1) optimizing the sampling process for measurement acquisition, and (2) sampling the posterior distribution of possible reconstructions given noisy measurements. The first part of the thesis investigates how to design adaptive and task-specific acquisition strategies for computational imaging systems, with a focus on compressed sensing magnetic resonance imaging (CS-MRI). We propose a sequential sampling method that learns to select measurements in multiple stages, and an approach that tailors sampling patterns for specific downstream tasks such as region-of-interest reconstruction, segmentation, and classification. These methods enable a better selection of measurements taken during acquisition, leading to improved performance compared to conventional baselines. We have also implemented our learned sequences on a real MRI scanner and verified their improvement in practice. The second part of the thesis develops a principled framework for posterior sampling using diffusion models (DMs)—a state-of-the-art class of generative models. By revealing a key connection between DMs and the Split Gibbs Sampling, we introduce a posterior sampling method that rigorously incorporates pre-trained DMs as image priors for solving inverse problems, which exhibits strong performance on a variety of applications. We then show that this framework can be naturally extended into a series of instantiations for solving more general inverse problems, addressing topics like text conditioning, video inverse problems, non-differentiable forward models, and discrete-space sampling. We also present a comprehensive benchmark for systematically evaluating state-of-the-art DM-based posterior estimation methods. By leveraging machine learning to address challenges in both data acquisition and posterior estimation, this thesis provides new possibilities for building more intelligent and reliable imaging systems across science and engineering. Item Type: Thesis (Dissertation (Ph.D.)) Subject Keywords: Computational imaging, measurement acquisition, magnetic resonance imaging, posterior sampling, diffusion models. Degree Grantor: California Institute of Technology Division: Engineering and Applied Science Major Option: Computing and Mathematical Sciences Thesis Availability: Public (worldwide access) Research Advisor(s): Bouman, Katherine L. Thesis Committee: Yue, Yisong (chair) Bouman, Katherine L. Perona, Pietro Song, Yang Defense Date: 8 July 2025 Funders: Funding Agency Grant Number Kortschak Scholars Program UNSPECIFIED Amazon AI4Science Fellowship UNSPECIFIED Amazon AI4Science Discovery Award UNSPECIFIED U.S. National Science Foundation 2048237 U.S. National Science Foundation 2034306 U.S. National Science Foundation Cyber-Physical System Foundations and Connected Communities (CPS) 1918655 National Institutes of Health (NIH) 5R01AG064027 National Institutes of Health (NIH) 5R01AG070988 National Institutes of Health (NIH) R21EB029641 National Institutes of Health (NIH) R01HD099846 National Institutes of Health (NIH) R01HD085813 Heritage Medical Research Institute (HMRI) UNSPECIFIED Beyond Limits UNSPECIFIED Rockley Photonics Sensing to Intelligence (S2I) and Clinard Innovation Funds Schmidt Sciences UNSPECIFIED Record Number: CaltechTHESIS:09062025-050711919 Persistent URL: https://resolver.caltech.edu/CaltechTHESIS:09062025-050711919 DOI: 10.7907/j42z-s192 Related URLs: URL URL Type Description https://proceedings.mlr.press/v158/yin21a.html Publisher Paper adapted for Chapter 3 https://doi.org/10.1109/TCI.2024.3410521 DOI Paper adapted for Chapter 4 https://openreview.net/forum?id=Xq9HQf7VNV Publisher Paper adapted for Chapter 6 https://arxiv.org/abs/2504.07549 arXiv Paper adapted for Chapter 7.3 https://openreview.net/forum?id=3DfCxd0yxo Publisher Paper adapted for Chapter 7.4 https://arxiv.org/abs/2503.01161 arXiv Paper adapted for Chapter 7.5 https://openreview.net/forum?id=U3PBITXNG6 Publisher Paper adapted for Chapter 8 ORCID: Author ORCID Wu, Zihui 0000-0002-7622-3548 Default Usage Policy: No commercial reproduction, distribution, display or performance rights in this work are provided. ID Code: 17671 Collection: CaltechTHESIS Deposited By: Zihui Wu Deposited On: 29 Sep 2025 19:26 Last Modified: 07 Oct 2025 21:04 Thesis Files PDF (Thesis) - Final Version See Usage Policy. 113MB Repository Staff Only: item control page

Learning to Sample in Computational Imaging: Measurement Acquisition and Posterior Estimation Thesis by Zihui Wu In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy CALIFORNIA INSTITUTE OF TECHNOLOGY Pasadena, California 2026 Defended July 8th, 2025 ii © 2026 Zihui Wu ORCID: 0000-0002-7622-3548 All rights reserved except where otherwise noted iii ACKNOWLEDGEMENTS As I look back on my five years of Ph.D. study at Caltech, I am deeply grateful to a long list of people who made this journey an unforgettable experience. First and foremost, I would like to sincerely thank my advisor, Prof. Katie Bouman, for taking me as her student and giving me the opportunities to work on so many exciting projects. Without her guidance and mentorship, this thesis would not have been possible. Katie believed in me even when I doubted myself, and her perseverance pushed me to achieve what I once thought impossible. Working with her has shaped me both academically and personally, and I am truly a better version of myself because of her mentorship. I am also indebted to my thesis committee members: Prof. Yisong Yue, Prof. Pietro Perona, and Dr. Yang Song. Even though Yisong is not my official advisor, many of the works in this thesis were done with him. His advice and support—both on research and on navigating Ph.D. life—have been invaluable. I am grateful to Pietro for cultivating a vibrant vision and imaging community and for his insightful feedback during group meetings. I also appreciate the generosity of Dr. Song for serving on my committee. This thesis owes much to the knowledge and inspiration I drew from his insightful papers and blog posts on diffusion models. I would also like to thank my undergraduate research advisor, Prof. Ulugbek Kamilov, back at WashU and my mentor, Dr. Yu Sun, now a professor at Johns Hopkins University. I first met Prof. Kamilov in his class on optimization during my sophomore year, where his Ph.D. student Yu was a teaching assistant. The class sparked my curiosity and led me to join Prof. Kamilov’s research group, where they taught me everything from coding and running experiments to writing papers and making figures. Their mentorship laid the foundation for my research career, and without them, I would not have pursued a Ph.D. in computational imaging. It was also a pleasure to work with Yu again during his postdoc in Katie’s group, which helped me transition smoothly between the early and later stages of my Ph.D. I am grateful for the opportunities to collaborate with many brilliant researchers during my Ph.D. Thank you to Prof. Adrian Dalca, Prof. Robert Frost and Prof. Andre van de Kouwe from MGH for hosting me over two summers and teaching me MRI from the basics, which was instrumental to translating my research on MRI in Part I into real-world impact. Thank you to Dr. David Ouyang from Cedars Sinai iv for offering me an opportunity to learn about medical imaging in clinical practice, which was eye-opening to me. Thank you to Prof. Liam Connor for mentoring me on radio interferometry and introducing me to the beauty of astronomy. Thank you to Prof. Al Barr for attending both my candidacy exam and thesis defense and for offering encouragement and constructive feedback. Additionally, I would like to thank Dr. Tianwei Yin for collaborating with me on MRI projects—his motivation and technical skill, even as an undergraduate back then, were inspiring and set a high bar for my own work. I am also grateful to Dr. Yu Sun, Hongkai Zheng, Bingliang Zhang, and Wenda Chu—not only as collaborators on the diffusion model projects for Part II but also as friends. I learned so much from our stimulating discussions and was lucky to share this journey with them. This thesis was made possible by generous support from the Kortschak Scholars Program, the Amazon AI4Science Fellowship, the National Science Foundation, the National Institutes of Health, the Heritage Medical Research Institute, Beyond Limits, Rockley Photonics, and Schmidt Sciences. Beyond research, I have been fortunate to form deep and lasting friendships at Caltech. I am grateful to my lab mates in the Computational Cameras (Bouman) Group and the Caltech Vision (Perona) Lab, as well as friends across the CMS department and the broader Caltech community. My social life at Caltech was enriched particularly by friends from the Wednesday-Friday basketball group, where I very much enjoyed both the competitive basketball games and the camaraderie. I also appreciate the help from our department and group admins—Maria Lopez, Roberta Carvalho, Monica Nolasco, and Nuvia Alvarez—for their efforts in making my Ph.D. life smooth outside of research. I am immensely grateful to my parents, grandparents, and other family members for their unwavering love and care throughout my Ph.D. years. Their support has been a constant source of strength during difficult times and will continue to guide me in the years ahead. Finally, I would like to express my deepest gratitude to my girlfriend, Xinyi Wu. Her love, encouragement, and companionship have made my Ph.D. years truly cherishable. She has been by my side through every high and low, offering support in so many ways that I never expected but greatly appreciated—both personally and academically. I cannot imagine completing this journey without her, and I look forward to all the chapters of life ahead together. v ABSTRACT Many problems in science and engineering require visualizing objects that are not directly observable—such as black holes that are millions of light-years away from the Earth or internal anatomical structures hidden within the human body. Computational imaging is a powerful paradigm that combines sensor design with advanced computational algorithms to make the invisible visible. The typical computational imaging pipeline involves first collecting indirect measurements of the target object and then solving a reconstruction problem. This thesis focuses on two core challenges about sampling along this pipeline: (1) optimizing the sampling process for measurement acquisition, and (2) sampling the posterior distribution of possible reconstructions given noisy measurements. The first part of the thesis investigates how to design adaptive and task-specific acquisition strategies for computational imaging systems, with a focus on compressed sensing magnetic resonance imaging (CS-MRI). We propose a sequential sampling method that learns to select measurements in multiple stages, and an approach that tailors sampling patterns for specific downstream tasks such as region-of-interest reconstruction, segmentation, and classification. These methods enable a better selection of measurements taken during acquisition, leading to improved performance compared to conventional baselines. We have also implemented our learned sequences on a real MRI scanner and verified their improvement in practice. The second part of the thesis develops a principled framework for posterior sampling using diffusion models (DMs)—a state-of-the-art class of generative models. By revealing a key connection between DMs and the Split Gibbs Sampling, we introduce a posterior sampling method that rigorously incorporates pre-trained DMs as image priors for solving inverse problems, which exhibits strong performance on a variety of applications. We then show that this framework can be naturally extended into a series of instantiations for solving more general inverse problems, addressing topics like text conditioning, video inverse problems, non-differentiable forward models, and discrete-space sampling. We also present a comprehensive benchmark for systematically evaluating state-of-the-art DM-based posterior estimation methods. By leveraging machine learning to address challenges in both data acquisition and posterior estimation, this thesis provides new possibilities for building more intelligent and reliable imaging systems across science and engineering. vi PUBLISHED CONTENT AND CONTRIBUTIONS [1] Brady Bhalla*, Zachary Huang*, Elyes Serghine*, Bingliang Zhang, Zihui Wu, and Katherine L. Bouman. “Text-Guided Image Restoration via a Unified Plug-and-Play Diffusion Framework.” In: Computational Cameras and Displays (CCD) Workshop, CVPR 2025 (2025). Z.W. mentored the first authors, led the conception of the project, designed and supervised all the experiments, and led the writing (especially the figures) of the manuscript. [2] Wenda Chu, Zihui Wu, Yifan Chen, Yang Song, and Yisong Yue. Split Gibbs Discrete Diffusion Posterior Sampling. 2025. arXiv: 2503.01161 [cs.LG]. url: https://arxiv.org/abs/2503.01161. Z.W. led the theoretrical analysis and contributed to writing the manuscript. [3] Austin Wang, Hongkai Zheng, Zihui Wu, Ricardo Baptista, Daniel Zhengyu Huang, and Yisong Yue. “Ensemble Kalman Sampling and Diffusion Prior in Tandem: A Split Gibbs Framework.” In: Frontiers in Probabilistic Inference: Learning Meets Sampling, ICLR 2025. 2025. url: https://openreview. net/forum?id=3DfCxd0yxo. Z.W. participated in the conception and theoretical part of the project, and contributed to writing the manuscript. [4] Bingliang Zhang*, Zihui Wu*, Berthy T. Feng, Yang Song, Yisong Yue, and Katherine L. Bouman. STeP: A Framework for Solving Scientific Video Inverse Problems with Spatiotemporal Diffusion Priors. 2025. arXiv: 2504. 07549 [cs.CV]. url: https://arxiv.org/abs/2504.07549. Z.W. participated in the conception of the project, prepared the dataset for dynamic MRI, and co-led the writing (especially the figures) of the manuscript. [5] Hongkai Zheng*, Wenda Chu*, Bingliang Zhang*, Zihui Wu*, Austin Wang, Berthy Feng, Caifeng Zou, Yu Sun, Nikola Borislavov Kovachki, Zachary E Ross, Katherine Bouman, and Yisong Yue. “InverseBench: Benchmarking Plug-and-Play Diffusion Models for Scientific Inverse Problems.” In: The Thirteenth International Conference on Learning Representations. 2025. url: https://openreview.net/forum?id=U3PBITXNG6. Z.W. co-led the conception of the project, conducted all the experiments on compressed sensing MRI, and led the writing (especially the figures) of the manuscript. [6] Zihui Wu, Yu Sun, Yifan Chen, Bingliang Zhang, Yisong Yue, and Katherine Bouman. “Principled Probabilistic Imaging Using Diffusion Models as Plug-and-Play Priors.” In: The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024. url: https://openreview.net/ forum?id=Xq9HQf7VNV. vii Z.W. led the conception of the project, conducted all the experiments, proved the main theorem, and wrote the manuscript. [7] Zihui Wu, Tianwei Yin, Yu Sun, Robert Frost, Andre van der Kouwe, Adrian V. Dalca, and Katherine L. Bouman. “Learning Task-Specific Strategies for Accelerated MRI.” In: IEEE Transactions on Computational Imaging 10 (2024), pp. 1040–1054. doi: 10.1109/TCI.2024.3410521. Z.W. led the conception of the project, conducted all the experiments, and wrote the manuscript. [8] Tianwei Yin*, Zihui Wu*, He Sun, Adrian V. Dalca, Yisong Yue, and Katherine L. Bouman. “End-to-End Sequential Sampling and Reconstruction for MRI.” In: Proceedings of Machine Learning for Health. Vol. 158. Proceedings of Machine Learning Research. PMLR, Dec. 2021, pp. 261–281. url: https://proceedings.mlr.press/v158/yin21a.html. Z.W. participated in the conception of the project, conducted the experiments, and led the writing (especially the figures) of the manuscript. ∗ indicates co-first authors. viii TABLE OF CONTENTS Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v Published Content and Contributions . . . . . . . . . . . . . . . . . . . . . . vi Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii List of Illustrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxviii Chapter I: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Computational Imaging as Inverse Problems . . . . . . . . . . . . . 2 1.2 Computational Imaging Pipeline and Some Key Challenges . . . . . 2 1.3 Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 I Sampling: Measurement Acquisition 6 Chapter II: Overview and Preliminaries . . . . . . . . . . . . . . . . . . . . 7 2.1 Magnetic Resonance Imaging (MRI) . . . . . . . . . . . . . . . . . 7 2.2 Compressed Sensing MRI (CS-MRI) . . . . . . . . . . . . . . . . . 7 2.3 Part Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Chapter III: Learning Sequential Sampling and Reconstruction Strategies . . 10 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Chapter IV: Learning End-to-End Strategies for the Downstream Task of Interest 25 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.4 Experiments on Large-Scale Datasets . . . . . . . . . . . . . . . . . 33 4.5 Validation on an Experimentally Collected Out-of-Distribution Dataset 40 4.6 Ablation Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 4.7 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 II Sampling: Posterior Estimation 56 Chapter V: Overview and Preliminaries . . . . . . . . . . . . . . . . . . . . 5.1 Bayesian Inverse Problems . . . . . . . . . . . . . . . . . . . . . . . 5.2 Diffusion Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Solving Inverse Problems with Diffusion Models . . . . . . . . . . . 57 57 59 60 ix 5.4 Part Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Chapter VI: PnP-DM: A Principled Framework for Posterior Estimation Using Diffusion Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 6.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 6.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 6.4 Convergence Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 71 6.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Chapter VII: Beyond PnP-DM: Towards a Unified Framework for General Posterior Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 7.1 Limitations of PnP-DM . . . . . . . . . . . . . . . . . . . . . . . . 84 7.2 A Unified Diffusion-Based Posterior Estimation Framework . . . . . 85 7.3 Thrust 1: Incorporating More Information as Prior . . . . . . . . . . 86 7.4 Thrust 2: Harnessing Higher Dimension for Video Inverse Problems 97 7.5 Thrust 3: Accommodating Black-Box Forward Models . . . . . . . . 110 7.6 Thrust 4: Tackling Inverse Problems in Discrete Spaces . . . . . . . 116 7.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 Chapter VIII: InverseBench: Benchmarking Diffusion-Based Methods for Scientific Inverse Problems . . . . . . . . . . . . . . . . . . . . . . . . . 124 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 8.2 Plug-and-Play Diffusion Priors for Inverse Problems . . . . . . . . . 126 8.3 InverseBench . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 8.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 8.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 Chapter IX: Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 9.1 Key Takeaways . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 9.2 Futures Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 9.3 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . 142 IIIAppendices 143 Appendix A: Appendix for Chapter 3 . . . . . . . . . . . . . . . . . . . . . . 144 A.1 Model Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . 144 A.2 Baseline Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 A.3 Further Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 A.4 Additional Reconstruction Examples . . . . . . . . . . . . . . . . . 148 Appendix B: Appendix for Chapter 4 . . . . . . . . . . . . . . . . . . . . . . 152 B.1 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . 152 B.2 Subsampling Setup and Implementation . . . . . . . . . . . . . . . . 156 B.3 Additional Validation on Out-of-Distribution Data . . . . . . . . . . 157 B.4 Additional Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 Appendix C: Appendix for Chapter 6 . . . . . . . . . . . . . . . . . . . . . . 160 C.1 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 C.2 Inverse Problem Setup . . . . . . . . . . . . . . . . . . . . . . . . . 166 x C.3 Technical Details of PnP-DM . . . . . . . . . . . . . . . . . . . . . 171 C.4 Implementation Details of Baseline Methods . . . . . . . . . . . . . 178 C.5 Additional Related Works . . . . . . . . . . . . . . . . . . . . . . . 179 C.6 Additional Experimental Results . . . . . . . . . . . . . . . . . . . 181 Appendix D: Appendix for Chapter 7 . . . . . . . . . . . . . . . . . . . . . . 189 D.1 Appendix for Section 7.3 . . . . . . . . . . . . . . . . . . . . . . . . 189 D.2 Appendix for Section 7.4 . . . . . . . . . . . . . . . . . . . . . . . . 191 D.3 Appendix for Section 7.5 . . . . . . . . . . . . . . . . . . . . . . . . 201 D.4 Appendix for Section 7.6 . . . . . . . . . . . . . . . . . . . . . . . . 217 Appendix E: Appendix for Chapter 8 . . . . . . . . . . . . . . . . . . . . . . 228 E.1 Tables of Main Results . . . . . . . . . . . . . . . . . . . . . . . . . 228 E.2 Inverse Problem Details . . . . . . . . . . . . . . . . . . . . . . . . 231 E.3 Pre-Trained Diffusion Model Details . . . . . . . . . . . . . . . . . 236 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 xi LIST OF ILLUSTRATIONS Number Page 1.1 Pipeline of computational imaging. In Stage 1 (Measurement Acquisition), a physical sensor acquires indirect, noisy measurements of some unknown ground truth 𝒙0 . Mathematically, this can be modeled by 𝒚 = A (𝒙0 ) + 𝒏 where 𝒏 represents measurement noise. In Stage 2 (Image Reconstruction), an algorithm A † (·) is designed to obtain a reconstruction b 𝒙 from the measurements. . . . . . . . . . . . . . . 3 3.1 We propose a sequential sampling and reconstruction co-design framework for accelerated MRI that adapts to a target during acquisition. Here, we visualize the sampling policy and final reconstruction of rotated knees in a single-coil imaging setting with 8× acceleration (8× subsampling). The first four columns show the cumulative 𝑘-space measurements selected by the proposed learned sampler (pink) in acquisition steps 1 through 4 (during a 4-step acquisition). The fifth column shows the final image recovered by the proposed learned reconstructor, and the last column is the ground truth. This example illustrates how our model has learned to adapt to different 𝑘-space distributions: the final sampling patterns in the fourth column contain visible directional structure that aligns with the 𝑘-space power spectrum. Rotated anatomical images, such as these rotated knee images, were not included in the training set (or quantitatively evaluated test set). . . . . . . . . . . . . . . . . . . . 11 xii 3.2 Overview of the proposed sequential sampling framework. Lowfrequency samples are pre-selected and measured in 𝑘-space. The subsampled 𝑘-space is transformed into a zero-filled image, which is fed into a reconstructor R𝜽 (·) to produce an intermediate image reconstruction (Equation (3.1)). The intermediate reconstruction and measurements are passed into a sampler network S𝒒 (·), which outputs a discrete probability distribution representing suggested samples for the next iteration. An action is sampled from this distribution (Equation (3.2)), and the corresponding 𝑘-space measurements are acquired. The sampling and reconstruction process is repeated for 𝑇 steps. The sampler and reconstructor are neural networks learned via end-to-end training with a loss on the final reconstructed image. Weights are shared across all 𝑇 acquisition steps. . . . . . . . . . . . 3.3 Visualizations of two types of 𝑘-space sampling patterns: 1D line sampling and 2D point sampling. White regions are sampled from a uniform distribution over the space of possible actions. The center low-frequency samples are pre-selected in all experiments before any further sampling. DC corresponds to the (0, 0) frequency. . . . . . . 3.4 Visualizations of example reconstructions with an 4× acceleration for 1D line sampling. Two zoomed-in image patches are shown along with the cumulative 𝑘-space measurements selected by each policy. Our sequential approach often provides more accurate reconstructions with detailed local structures. More visualizations are included in the Appendix A.4. . . . . . . . . . . . . . . . . . . . . 3.5 Histograms of pair-wise SSIM differences between our sequential models and LOUPE [12] on all 1,851 test images. Positive numbers indicate improvement over LOUPE. The results are acquired by averaging three runs of 4× accelerated 2D point subsampling. More sequential steps lead to a bigger advantage over LOUPE, with the 4-step sequential model outperforming LOUPE on 96.96% of samples. This performance pattern holds for the 1D line scenario and other acceleration factors as well, as shown in Appendix A.3. More quantitative results are given in Table 3.3. . . . . . . . . . . . . . . 15 17 19 19 xiii 3.6 Comparison between our sequential model and the LOUPE model on the fastMRI knee test set. Our sequential model outperforms LOUPE for all acceleration ratios with an improvement comparable to 25% of the benefit of doubling the number of 𝑘-space measurements. The performance of our sequential model in the 1D line sampling case significantly outperforms LOUPE but plateaus after 2 sequential sampling steps, possibly due to the restricted action space of 1D line sampling. . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.7 Histograms of pair-wise SSIM comparison on all 1,851 test images with a different number of sequential steps (𝑇), using 2D point sampling with a 4× acceleration factor. The relative error between the 4-step and 1-step (left) or 2-step(right) demonstrates that additional sequential steps help to boost performance, but with diminishing returns as 𝑇 increases. . . . . . . . . . . . . . . . . . . 21 4.1 Comparison between (a) traditional CS-MRI, (b) a naïve approach to task-specific CS-MRI, and (c) the proposed Tackle framework. Compared with panel (a) which separately deals with reconstruction and task prediction, panel (b) is a simple extension of codesign methods for solving downstream tasks by adding a learnable mapping from measurements to task predictions. However, this naïve approach leads to a suboptimal performance and can even lead to a worse task prediction accuracy, as shown in the example above. On the other hand, we introduce Tackle for effectively learning taskspecific CS-MRI strategies. Tackle is first pre-trained for generic reconstruction, and then all three modules are fine-tuned for a more specific downstream task. We find that this training schedule allows Tackle to robustly learn generalizable task-specific strategies. In the above knee segmentation example, all three approaches are trained with the same architectures for the reconstructor (second module) and predictor (third module). Nevertheless, Tackle significantly outperforms the two baseline approaches. . . . . . . . . . . . . . . 48 xiv 4.2 Block diagram of the proposed framework Tackle and a summary of the investigated datasets and settings. Tackle uses a task-specific loss to jointly optimize a sampler, a retriever, and an optional predictor, ranging from scanner-level sampling to human-level diagnosis. A summary of the investigated settings is presented in the bottom left panel. FSE, GRE, DESS, and FLAIR stand for fast spin echo, gradient echo, double-echo steady-state, and fluid-attenuated inversion recovery, respectively. We comprehensively investigate multiple CS-MRI tasks on a variety of common MRI settings with six datasets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Visual examples of two Meniscus Tear samples reconstructed by different methods in the 16× acceleration single-coil setting. For each reconstruction, the full-FOV PSNR is labeled in white, and the local PSNR for the ROI is in orange. Note how TackleROI recovers the structure and details of the ROI more accurately than the two baselines, as indicated by the red arrows. The better recovery of TackleROI over the ROI leads to a more accurate diagnosis of the Meniscus Tear. We emphasize that the location of the ROI is not an input to any of these models and is only used for evaluating the accuracy of each method on the region that contains the pathology. . 4.4 Comparison of a subsampling PSF optimized for full-FOV reconstruction and another optimized for the reconstruction of menicus tear (MT) ROIs. Optimizing for MT ROI reconstruction leads to around 40% improvement on the vertical resolution in terms of the full width at half maximum (FWHM), as shown by the PSF profiles in the bottom panel. This improved vertical resolution leads to a better reconstruction of the meniscus that has horizontal anatomy. . . . . . 4.5 Box plots of the knee tissue segmentation results under 16× (a) and 64× (b). Within the rectangle between each pair of methods, the top number is the percentage of samples that get improved, and the bottom number is the 𝑝-value given by the paired samples 𝑡-test. A higher percentage and a lower 𝑝-value indicate a more significant improvement. We also provide the 95% confidence intervals for all methods below their names. For both acceleration ratios, Tackleseg. outperforms other baselines in terms of all the statistical measures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 49 50 50 xv 4.6 Comparison of segmentation results under 16× acceleration on one sample of the SKM-TEA dataset. We show the input of the predictor in the first row, a zoom-in on the region that contains the tissues to be segmented in the second row, and the output of the predictor in the third row. Note that Tackleseg. circumvents the typical “reconstruction” in terms of pixel-wise similarity with the ground truth image. Instead, it learns a feature map that accurately localizes the anatomy, leading to better segmentation prediction than other baselines both for this sample and on average over the test set (Table 4.2). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.7 Box plots of the brain tissue segmentation results under 16× (a) and 64× (b) accelerations. Within the rectangle between each pair of methods, the top number is the percentage of samples that get improved, and the bottom number is the 𝑝-value given by the paired samples 𝑡-test. A higher percentage and a lower 𝑝-value indicate a more significant improvement. We also provide the 95% confidence intervals for all methods below their names. Similar to the knee segmentation results, the proposed method Tackle outperforms other baselines in terms of all the statistical measures for both acceleration ratios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.8 Comparison of segmentation results under 16× acceleration on one sample from the OASIS dataset. Similar to the knee segmentation results, Tackleseg. circumvents the typical “reconstruction” in terms of pixel-wise similarity with the ground truth image. Instead, it learns an anatomically accurate feature map, which enables better segmentation prediction than other baselines both for this sample and on average over the test set (Table 4.3). The zoom-in panels highlight a region where Tackleseg. more accurately predicts the outline of white matter (in yellow) than other methods. This improvement leads to a more precise estimation of the thickness of the cortex (in orange), an important task for studying human cognition and neurodegeneration [10]. . . . . . . . . . . . . . . . . . . . . . . . . . . 53 xvi 4.9 Confusion matrices of the classification results by LOUPErecon. and Tackleclass. . Overall, Tackleclass. achieves greater accuracy in terms of both classification accuracy and 𝐹1 score than LOUPErecon. . Tackleclass. also has a significantly lower number of false negatives (bottom left) compared to LOUPErecon. , which could lead to more patients receiving early treatment. . . . . . . . . . . . . . . . . . . . 53 4.10 Per-slice difference histograms. (a): TackleFOV over LOUPEFOV on the full-FOV reconstruction task and (b): TackleROI over LOUPEFOV on the ROI-oriented reconstruction task. The 95% confidence intervals are given in the top left corner of each plot. In both cases, the vast majority of slices improve, and the 𝑝-values given by the paired samples 𝑡-test are highly significant. . . . . . . . . . . . . . . . . . . . . . . 54 4.11 Reconstruction comparison of two samples in the experimentally collected dataset (top: from subject 1; bottom: from subject 2) by different methods under 4× acceleration. The sampling mask, a zoom-in on the ROI, and the error map are presented for each method. By sampling more frequencies along the vertical direction in 𝑘-space, TackleROI has a higher vertical resolution in the image space and thus outperforms other baselines optimized for full-FOV reconstruction on the ROIs with directional anatomical structure. . . 55 4.12 Reconstruction comparison between the implemented prospective subsampling sequence and the retrospective subsampling sequence. Our learned sequence can be implemented on an MRI scanner and generates images of quality indistinguishable from those recovered from retrospectively sampled data. Compared to the ground truth image, our prospectively subsampled reconstruction recovers important features around the meniscus region, which is the ROI it is trained to enhance. . . . . . . . . . . . . . . . . . . . . . . . . . . 55 xvii 6.1 A schematic diagram of our method. Our method alternates between a likelihood step that enforces data consistency and a prior step that solves a denoising posterior sampling problem by leveraging the Split Gibbs Sampler [296]. An annealing schedule controls the strength of the two steps at each iteration to facilitate efficient and accurate sampling. A crucial part of our design is the prior step, where we identify a key connection to a general diffusion model framework called the EDM [151]. This connection allows us to easily incorporate a family of state-of-the-art diffusion models as priors to conduct posterior sampling in a principled way without additional training. Our method demonstrates strong performance on a variety of linear and nonlinear inverse problems. . . . . . . . . . . . . . . . . . . . . 6.2 A conceptual illustration of the non-stationary and stationary time-continuous processes as interpolations of 𝐾 discretize iterations of PnP-DM. . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Results on a synthetic problem with the ground truth posterior available. PnP-DM can sample it more accurately that DPS [64]. . . 6.4 Visual examples for the motion deblur problem (𝜎𝒚 = 0.05). We visualize one sample generated by each sampling algorithm. . . . . . 6.5 Comparison of uncertainty quantification (UQ) for the motion deblur. Left 3 columns: absolute error (|b 𝒙 − 𝒙0 |), standard deviation (std), and absolute z-score (|b 𝒙 − 𝒙0 |/std) with the outlier pixels in red. Right column: scatter plot of |b 𝒙 − 𝒙0 | versus std. Note that PnPDM leads to a better UQ performance than the baselines by having the lowest percentage of outliers while avoiding having overestimated per-pixel standard deviations. . . . . . . . . . . . . . . . . . . . . . 6.6 Results of the Fourier phase retrieval problem. (a) PnP-DM provides both upright and rotated reconstructions (two modes given by the invariance of the forward model to 180◦ rotation) with high fidelity, while the baseline methods cannot. (b) We visualize the percentages of upright and rotated reconstructions out of 90 runs for a test image with two samples for each orientation. . . . . . . . . . . 68 71 73 75 76 77 xviii 6.7 Results on the black hole imaging problem with simulated data. Due to severe noise corruption and measurement sparsity, this problem is non-convex and highly ill-posed, leading to a bi-modal posterior distribution as previously found in [269]. Here we compare our method, PnP-DM, with the DPS baseline [64]. A metric quantifying the mismatch with the observed measurements is labeled for each sample, which should be around 2 for an ideal measurement fit. Samples generated by PnP-DM exhibit two distinct modes with sharp details and a consistent ring structure, while samples given by DPS display inconsistent ring sizes and sometimes fail to capture the black hole structure entirely, with samples having poor measurement fit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.8 Results on the black hole imaging problem with the real data for the M87 black hole from April 6th , 2017 [71]. The posterior samples from PnP-DM contain fine-grained features that align with the prior distribution; see left for a few samples generated by the pretrained diffusion model from the prior. Besides having high visual quality, our posterior samples accurately capture key features of the official reconstruction by EHT as well, such as the bright spot location and ring diameter. . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.9 Visual examples of the DSA deconvolution problem in radio astronomy. All images undergo a nonlinear transformation to visualize the weaker galaxies. (a): The mean image of PnP-DM (based on 20 samples) significantly outperforms the reconstruction of the classic CLEAN algorithm [134]. (b): We visualize three posterior samples and the per-pixel standard deviation map computed from all 20 samples. Zoom-in regions highlight areas with notable sample variability. 6.10 Comparison of galaxy property estimation accuracy between CLEAN (blue) and PnP-DM (orange). Each scatter plot shows predicted versus true values for semi-major axis 𝜃 𝐴 (left), semi-minor axis 𝜃 𝐵 (middle), and flux (right), across all detected sources. The dashed line indicates perfect prediction. PnP-DM produces estimates that lie closer to the diagonal, indicating more accurate recovery of galaxy shapes and fluxes. Notably, PnP-DM avoids the strong overestimation biases seen in CLEAN. . . . . . . . . . . . . . . . . . . 78 79 81 82 xix 7.1 Illustration of the prior step with different text inputs. Starting from a noisy image 𝒗 (𝑘) , the prior step performs text-guided denoising through reverse diffusion to generate a cleaner sample 𝒖 (𝑘+1) on the clean image manifold. The images are visualized after decoding. Different text prompts steer the denoising process toward distinct yet plausible modes of the image. . . . . . . . . . . . . . . . . . . . . . 7.2 Visual examples for the super-resolution (16×), gaussian deblur (𝜎 = 6), and box inpainting tasks with matching prompts. The proposed instantiations, DAPS and DCDP in particular, produce more detailed and perceptually coherent reconstructions, while the baseline methods suffer from artifacts and a decrease in quality. . . . . . . . 7.3 Visual examples of DAPS on the 16× super-resolution task using different prompts. The examples show that varying descriptions lead to semantically different yet feasible reconstructions. Despite the severe degradation of the input measurements, the restored images are of high quality and closely align with the information from the provided text prompts, while still consistent with the original measurement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Visualization of the image generation process of DAPS. The top row shows outputs from the prior steps (𝒖 (𝑘+1) for 𝑘 ∈ {9, 19, 29, 39}), which denoise and integrate text conditioning to guide the sample toward the target distribution, while the bottom row shows outputs from the likelihood steps (𝒗 (𝑘) for 𝑘 ∈ {9, 19, 29, 39}), which enforce data consistency with added noise. Initialization starts from pure noise, and the two processes alternate to progressively generate a realistic image. The images are visualized after decoding. . . . . . . 89 91 94 95 xx 7.5 Effect of prompt specificity on posterior samples generated by DAPS for 16× super-resolution. We repeatedly generate samples using DAPS with prompts of varying specificity. As prompts become more generic, the samples exhibit greater semantic diversity—e.g., several yellow scoops in the “a plate of food” panel look like mashed potatoes, which are not seen in the more specific “a plate of desserts” panel. However, the overall diversity in appearance remains similar—i.e., the bottom two panels appear to be equally diverse at first glance. Modes with highly specific details, such as “two scoops of ice cream with a cannoli” or “a plate of macarons” (top right), are only recovered when given the corresponding prompts. These results underscore the importance of text conditioning in uncovering rare modes and improving mode coverage in posterior estimation. . . . . 96 7.6 An overview of our proposed framework with spatiotemporal diffusion priors, STeP, for scientific video inverse problems. Left: STeP combines the physics model of the target problem with a spatiotemporal diffusion prior that directly characterizes the video distribution. We show that such a prior can be efficiently obtained by fine-tuning a pre-trained image diffusion model with limited video data. Right: STeP can generate diverse solutions to a black hole video reconstruction problem that exhibit equally good fidelity with the measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 7.7 A schematic comparison between prior works (top) and our STeP framework (bottom) for video inverse problems. The bold texts highlight the key differences between them. While prior works use an image diffusion model and enforce temporal consistency using simple heuristics or warping noise with optical flow, we directly learn a spatiotemporal diffusion prior. . . . . . . . . . . . . . . . . . . . . 100 7.8 Architecture of the spatiotemporal module. Given a pre-trained image diffusion U-Net, we add a zero-initialized temporal module with an ON/OFF switch to each 2D spatial module and initialize the additive weight 𝛼 to zero. Thus, it will have no effect at the start of fine-tuning and gradually learn from the video training data. The number of frames, height, and width are denoted by 𝑛 𝑓 , 𝑛 ℎ , and 𝑛𝑤 , respectively. The numbers of channels for input features ( 𝑓in ) and output features ( 𝑓out ) are denoted by 𝑛in and 𝑛out , respectively. . . . . 101 xxi 7.9 Visual examples of STeP (bottom left) and baselines for black hole video reconstruction. To facilitate analysis of the reconstructed spatiotemporal structures, we present results in three ways: (1) a single frame to illustrate spatial fidelity, (2) an 𝑥-𝑡 slice depicting temporal evolution of a vertical line to evaluate temporal consistency, and (3) the averaged optical flow visualized using the standard color scheme from [278] to assess spatiotemporal coherence jointly. Compared to baselines, STeP exhibits clearer alignment with ground truth videos across all aspects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 7.10 Visual examples of STeP (bottom left) and baselines for dynamic MRI. We visualize a representative frame along with two zoomed-in regions for each method to better illustrate spatial fidelity. Benefiting from its robust spatiotemporal prior, STeP provide reconstructions with fewer structural artifacts and temporal fluctuations, as indicated by its averaged optical flow aligning more closely with the ground truth. This demonstrates the effectiveness of our learned spatiotemporal prior in enhancing both spatial and temporal consistency. . . . . 107 7.11 Detailed comparison on black hole video reconstruction and dynamic MRI. Left: We compare STeP (joint) and the BCS baseline [168] by visualizing the averaged delta frames (difference images) over an expanding window. The delta frames given by STeP (joint) better align with the ground truth, indicating better temporal consistency. Right: We also compare the spatial fidelity between STeP (joint) and its variant STeP (video-only). Trained on both images and videos, STeP (joint) provide reconstructions with less spatial hallucinations compared to STeP (video-only). . . . . . . . . . . . . . . . . . . . . 108 7.12 Consistent improvement in image-video joint fine-tuning. We evaluate intermediate checkpoints of (a) black hole video reconstruction and (b) dynamic MRI (8×). Spatial similarity (measured by PSNR), temporal consistency (measured by d-PSNR), and measurement data fit (measured by data misfit) all show steady improvement. 108 xxii 7.13 Comparison between running STeP with DAPS versus PnP-DM as the inference backbone. The DAPS version of STeP exhibits significantly better spatial consistency (shown by the first frames in the first row) and temporal consistency (shown by the 𝑥-𝑡 slice in the second row). This comparison illustrates the better compatibility of DAPS with latent video diffusion models than PnP-DM. . . . . . . . 110 7.14 Sampling results of the XOR task on the discretized MNIST dataset. SGDD faithfully recovers the structural information of the ground truth signal. . . . . . . . . . . . . . . . . . . . . . . . . . . 122 7.15 Diversified samples when the measurement 𝒚 is sparse. Samples are generated by SGDD when solving an MNIST inpainting task. . . 123 8.1 Illustration of five benchmark problems in the InverseBench. A represents the forward model that produces measurements from the underlying target. A † represents the inverse map. In the linear inverse scattering problem (left two), the measurements are the recorded data from the receivers, and the unknown source we aim to infer is the permittivity map of the object. The bottom panel displays the efficiency and accuracy plots for our benchmarked algorithms. Certain characteristics of the problem cause the efficiency and accuracy trade-offs of each algorithm to vary across tasks. In these plots, the larger radius of the points indicates greater interaction with the forward function A, as measured by the number of forward model evaluations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 8.2 Qualitative comparison showing representative examples of PnPDP methods and domain-specific baselines across five inverse problems. Note that for full waveform inversion, Adam∗ and LBFGS∗ are initialized with Gaussian-blurred ground truth, serving as references. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 8.3 Illustration of the failures of PnPDP methods (DAPS) as an example) on full waveform inversion. With a small learning rate, DAPS is numerically stable but does not solve the inverse problem effectively. With a slightly larger learning rate, DAPS produces a noisy velocity map that breaks the stability condition of the PDE solver, resulting in a complete failure. . . . . . . . . . . . . . . . . . 136 xxiii 8.4 Relative performance of plug-and-play diffusion prior methods compared with traditional baselines under different levels of measurement sparsity on different tasks. Metrics are averaged over multiple PnPDP methods. The performance difference increases in general as the measurement becomes sparser. . . . . . . . . . . . . 137 8.5 PnPDP methods on out-of-distribution test samples. (a) Blackhole imaging problem on digit inputs; and (b) inverse scattering on sources that contain 9 cells, while the prior model is trained on images with 1 to 6 cells. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 A.1 Flow diagram of the reconstructor, R𝜽 (·), in the proposed framework. We use a residual U-Net reconstructor for all of our models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 A.2 Flow diagram of the samplers, S𝒒 (·), in the proposed framework. We use a Multilayer Perceptron for 1D line sampling and a U-Net for 2D point sampling. Both networks take previous observations as inputs and output a probability map, which is rescaled and binarized into the final sub-sampling mask at the next iteration. . . . . . . . . 145 A.3 Histograms of pair-wise SSIM differences on all 1,851 test images using 1D line sampling with 4× acceleration factor. We calculate the improvement of our model with different sequential steps over LOUPE. Our sequential model and non-sequential baseline significantly outperform LOUPE for most subjects. . . . . . . . . . . . . . 147 A.4 Histograms of pair-wise SSIM differences on all 1,851 test images using 1D line sampling with 4× acceleration factor. We calculate the improvement of the Evaluator (left), PG-MRI (middle), and our best sequential model (4-step sequential) (right) over LOUPE. Our 4-step sequential model significantly outperforms LOUPE, while the other two baselines are substantially worse than LOUPE for most subjects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 xxiv A.5 Histograms of pair-wise SSIM differences on all 1,851 test images using 2D line sampling with 4× (first row), 8× (second row), and 16× (third row) acceleration factors. We calculate the improvement of our model with different sequential steps over LOUPE in each column. For all three acceleration factors, our sequential model outperforms the non-sequential baseline and LOUPE on an increasing percentage of test samples as the number of sequential steps increases. Our sequential models also have increasingly larger advantages over LOUPE as the number of sampled measurements increases (i.e., the acceleration factor decreases). . . . . . . . . . . . . . . . . . . . . . 149 A.6 Visualizations of the reconstructions of the 394th (top), 1083th (middle), 1506th (bottom) test images with an acceleration factor of 4× for 1D line sampling. Zoomed-in image patches highlight our significant improvement over previous methods. We find that our learned masks for the 1D line sampling case usually consist of adjacent low-frequency samples. However, only a few of the learned samples have their conjugate symmetric points sampled as well. Our learned policy appears to leverage the conjugate symmetry of the 𝑘-space and trade off taking more measurements with taking fewer measurements with higher SNR (by effectively sampling the same measurement twice). . . . . . . . . . . . . . . . . . . . . . . . . . 150 A.7 Visualizations of the reconstructions of the 1355th test sample with an acceleration factor of 4× (top) and 8× (bottom) for 2D point sampling. A zoomed-in image patch is shown along with the cumulative 𝑘-space measurements selected by each policy. Orange arrows point out the regions where our sequential approach provides more accurate and detailed local structures. . . . . . . . . . . . . . 151 B.1 Conceptual illustration of the subsampling setup with a knee example. The back dots on the 𝑘 𝑦 -𝑘 𝑧 plane represent 𝑘-space trajectories along 𝑘 𝑥 , which are illustrated by the black arrows. We consider subsampling in the two phase-encoding dimensions (𝑘 𝑦 and 𝑘 𝑧 ) of a 3D Cartesian sequence, where the subsampling pattern 𝒎 is learned from data for some specific downstream task. . . . . . . . . 156 xxv B.2 B.3 C.1 C.2 C.3 C.4 C.5 C.6 C.7 C.8 C.9 Box plots of the Meniscus Tear ROI reconstruction results under 8× (a) and 16× (b) accelerations. Within the rectangle between each pair of methods, the top number is the percentage of samples that get improved, and the bottom number is the 𝑝-value given by the paired samples t-test. A higher percentage and a lower 𝑝-value indicate a more significant improvement. The 95% confidence intervals for all methods are given below their names. . . . . . . . . . . . . . . . . 158 Visualization of the input of the predictor network for the brain tumor classification task under 16× acceleration on two samples of the BRATS dataset. Similar to the segmentation results, as a codesign method, Tackleclass. circumvents the typical “reconstruction” in terms of point-wise similarity with the ground truth image. Instead, the retriever learns a feature map that highlights the region around the tumor for the downstream prediction. . . . . . . . . . . . . . . . 159 Example images from the dataset for training the black hole diffusion model prior. . . . . . . . . . . . . . . . . . . . . . . . . . 175 Comparison of our method and DPS [64] on estimating the posterior distribution of a Gaussian deblurring problem under a Gaussian prior. While the mean estimations of the two methods are of roughly the same quality, our approach provides a much more accurate estimation of the posterior per-pixel standard deviation than DPS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 Visual comparison between our method and baselines on solving the Gaussian deblurring and super-resolution problems with i.i.d. Gaussian noise (𝜎𝒚 = 0.05). We visualize one sample generated by each algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 Additional visual examples for the Gaussian deblurring problem. 182 Additional visual examples for the motion deblurring problem. . 183 Additional visual examples for the 4× super-resolution problem. . 184 Visual comparison between our method and baselines on solving the coded diffraction pattern (CDP) reconstruction problems with i.i.d. Gaussian noise (𝜎𝒚 = 0.05). We visualize one sample generated by each algorithm. . . . . . . . . . . . . . . . . . . . . . 184 Additional visual examples for the Fourier phase retrieval problem.185 Additional visual examples for the Fourier phase retrieval problem.186 xxvi C.10 Additional visual examples given by PnP-DM and DPS using the simulated black hole data. . . . . . . . . . . . . . . . . . . . . . . 187 C.11 Additional visual examples given by PnP-DM using the real M87 black hole data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 C.12 Sensitivity analysis on the annealing schedule 𝜂 𝑘 with different decay rates 𝛼 (left) and minimum coupling strength 𝜂min (right) for a linear (super-resolution) and a nonlinear (coded diffraction patterns) inverse problem. Recall from Appendix C.3.3 that 𝜂 𝑘 := max(𝛼 𝑘 𝜂0 , 𝜂min ), where we set 𝜂0 = 10 for this experiment. . . . . . 188 C.13 Visual examples of intermediate 𝒙 (𝑘) and 𝒛 (𝑘) iterates (left) and convergence plots of PSNR, SSIM, and LPIPS for 𝒙 (𝑘) iterates (right) on the super-resolution problem. The vertical dashed lines show the iterations at which the 𝒙 (𝑘) and 𝒛 (𝑘) iterates are visualized. 188 D.1 The dirty images from the ideal visibilities. We use the standard implementation in EHT library to get dirty images for each selected frame. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 D.2 Subsampling masks of 6× (left) and 8× (right) accelerations used in dynamic MRI experiments. The white areas in the center indicate the auto-calibration (ACS) signals. The horizontal and vertical directions are the frequency (𝑘 𝑥 ) and phase (𝑘 𝑥 ) encoding directions, respectively. The same mask is applied to the sampling of each individual frame of all videos. . . . . . . . . . . . . . . . . . . . . 197 D.3 Visualization of VAE reconstructions. The VAE reconstructions are computed by first encoding the ground truth videos and then decoding them. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 D.4 Visualization of video diffusion model unconditional samples. The videos are sampled by solving the PF-ODE with 100 Euler steps. 202 D.5 Visualization of STeP posterior samples. The videos are sampled using the Algorithm 7. . . . . . . . . . . . . . . . . . . . . . . . . 203 E.1 Computational characteristics of each forward model. Fwd: runtime of a single forward model evaluation tested on a single A100 GPU. DM: runtime of a single diffusion model evaluation. Fwd Grad: runtime of a single forward model gradient evaluation. DM Grad: runtime of a single diffusion model gradient evaluation. Note that the inverse problem of the Navier-Stokes equation only permits black-box access to the forward model, so its Fwd Grad has no value. 231 xxvii E.2 Multi-modal example on black hole imaging. The image shows two image modes discovered by DAPS and PnP-DM. . . . . . . . . 234 xxviii LIST OF TABLES Number Page 3.1 SSIM comparison of 2D point sampling for 4×, 8×, and 16× accelerations. Our 4-step sequential model outperforms the previous approaches when tested on the fastMRI knee test set. For each model, we compute the test average and standard deviation obtained across three trained models with independent initialization. . . . . . . . . . 18 3.2 SSIM comparison of 1D line sampling under 4× acceleration. Our 4-step sequential model outperforms the previous approaches when tested on the fastMRI knee test set. A paired samples 𝑡-test shows a statistically significant difference between our 4-step sequential model and LOUPE [12], with a 𝑝-value smaller than 10−300 . For each model, we compute the test average and standard deviation obtained across three trained models with independent initialization. 20 3.3 The percentage of test samples on which our method with different numbers of sequential sampling steps (𝑇) outperforms the LOUPE [12] baseline for 2D point sampling. The percentage average and standard deviation are obtained using results from three trained models with independent initialization. . . . . . . . . . . . . 23 3.4 Ablation results showing the advantage of co-design with a 4× acceleration ratio and 2D point sampling. When co-design is specified as “Yes” the reconstruction network has been jointly optimized with the sampler. Otherwise, the sampler was optimized with a fixed reconstructor that was pre-trained with a random sampling policy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 4.1 Comparison of average test local peak signal-to-noise ratio (Local PSNR) in decibel (dB) within Meniscus Tear ROIs under different acceleration ratios (𝑅). . . . . . . . . . . . . . . . . . . . . . . . . 34 4.2 Comparison of average test Dice score on the SKM-TEA dataset [83] for segmenting four knee tissues under different acceleration ratios (𝑅). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.3 Comparison of average test Dice score on the brain segmentation task under different acceleration ratios (𝑅). . . . . . . . . . . . . 38 4.4 Comparison of average test accuracy on the pathology classification task under different acceleration ratios (𝑅). . . . . . . . . . 40 xxix 4.5 Comparison of average reconstruction accuracy on the experimentally collected dataset under 4× acceleration (top: full-FOV reconstruction; bottom: ROI-oriented reconstruction). . . . . . . 4.6 Ablation studies on two aspects of co-design for all the considered tasks under 16× acceleration. . . . . . . . . . . . . . . . . . . . . 4.7 Ablation studies on model architecture and pre-training for nonreconstruction tasks under 16× acceleration. . . . . . . . . . . . . 4.8 Comparison of average test PSNR (dB) between reconstruction models trained with task-specific masks and Tacklerecon. on the fastMRI knee dataset. . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Quantitative comparison on three noisy linear inverse problems for 100 FFHQ color test images. Bold: best; Underline: second best. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Quantitative evaluation on two noisy nonlinear inverse problems for 100 FFHQ grayscale test images. Bold: best; Underline: second best. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Quantitative results for the DSA deconvolution problem in radio astronomy. We compare PnP-DM with the well-known baseline CLEAN on detecting the galaxies and estimating their shapes and fluxes. PnP-DM outperform CLEANs in terms of all the metrics. . . 7.1 Quantitative evaluation for the super-resolution (16×), gaussian deblurring (𝜎 = 6), and box inpainting tasks with matching prompts. Bold: best; Underline: second best. For each task, we report the mean performance and standard deviation in PSNR, SSIM, and LPIPS across 14 test images. The results show that methods based on our unified framework outperform the baseline approaches, TReg and PSLD, especially on the super-resolution and deblurring problems. The unified methods maintain strong performance without relying on additional components, highlighting their efficiency and generality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 44 45 45 73 77 80 91 xxx 7.2 Quantitative results on black hole video reconstruction and dynamic MRI. We compare our method against baselines by reporting the mean and standard deviation (shown in parentheses) of selected evaluation metrics computed over 10 test videos (FVD is reported without standard deviation since it evaluates the set of 10 videos collectively). The results clearly demonstrate that by leveraging the spatiotemporal prior, STeP consistently achieves improvements in both spatial quality and temporal consistency relative to baseline methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 7.3 Comparison on the Navier-Stokes inverse problem. Metrics are abbreviated as follows: Rel ℓ2 (relative ℓ2 error), CRPS (continuous ranked probability score), and SSR (spread-skill ratio). − indicates either that probabilistic metrics are inapplicable (deterministic models) or that it is too expensive to generate enough samples from the algorithm for reliable calculation. Bold and Underline indicate best and second best among methods that accommodate unpaired data, respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 7.4 Quantitative results for XOR and AND problems on discretized MNIST. We report the mean and standard deviation (shown in parentheses) of PSNR and class accuracy across 1,000 generated samples. SGDD demonstrates superior performance on both tasks. . . . . . . 122 8.1 Requirements on the forward model of the algorithms evaluated in our experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . 126 8.2 Characteristics of different inverse problems in InverseBench. From left to right: whether the forward model is linear, whether one can efficiently compute the SVD from the forward model, the domain in which the inverse problem is defined, whether the forward model can be solved in closed form, whether one can access gradients from the forward model, and the noise type. . . . . . . . . . . . . . . . . 128 B.1 Comparison of TackleROI on in- and out-of-distribution samples under different acceleration ratios (𝑅). . . . . . . . . . . . . . . . 157 C.1 List of hyperparameters for the likelihood step of PnP-DM. . . . 172 C.2 List of hyperparameters for the annealing schedule of 𝜂 in PnP-DM.176 C.3 Comparison of computational efficiency between PnP-DM and other baseline methods. . . . . . . . . . . . . . . . . . . . . . . . . 178 D.1 Hyperparameters used by each method on each task. . . . . . . . 191 xxxi D.2 Summary of the hyperparameters of Algorithm 7 (STeP) for black hole video reconstruction and dynamic MRI. The HMCrelated parameters are tuned on 3 leave-out validation videos. The run time and memory are tested using one NVIDIA A100 GPU. . . . 193 D.3 Summary of the training of spatiotemporal diffusion prior. We provide and group the hyperparameters according to each component in the model. The model is trained with 1 NVIDIA A100-SCM480GB GPU. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 D.4 Additional results on dynamic MRI with 6× acceleration. Following the same setup as in Table 7.2, we report the quantitative results on 10 test videos. . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 D.5 The data misfit values for samples shown in Figure 7.6. We report the data misfit metrics for the three obtained modes, which demonstrate that all modes fit the measurement data equally well. . . 200 D.6 Hyperparameters of Blade for the Navier-Stokes experiments in Table 7.3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 D.7 Hyperparameters of SGDD used in each experiment. . . . . . . . 227 E.1 Results on linear inverse scattering. PSNR and SSIM of different algorithms on linear inverse scattering. Noise level 𝜎𝒚 = 10−4 . . . . 228 E.2 Results on compressed sensing MRI. Mean and standard deviation are reported over 94 test cases. . . . . . . . . . . . . . . . . . . . . 228 E.3 Generalization results on compressed sensing MRI with ×4 acceleration and raw measurements. Mean and standard deviation are reported over 94 test cases. . . . . . . . . . . . . . . . . . . . . 229 E.4 Results on black hole imaging. PSNR and Chi-squared of different algorithms on black hole imaging. Gain and phase noise and thermal noise are added based on the EHT library. . . . . . . . . . . . . . . 229 E.5 Results on FWI. Mean and standard deviation are reported over 10 test cases. †: initialized from data blurred by Gaussian filters with 𝜎 = 20. ∗: one test case is excluded from the results due to numerical instability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 E.6 Results on Navier-Stokes equation. Relative ℓ2 error of different algorithms on 2D Navier-Stokes inverse problem, reported over 10 test cases. ∗: one or two test cases are excluded from the results due to numerical instability. . . . . . . . . . . . . . . . . . . . . . . . . 230 xxxii E.7 Table of metrics we use to capture the computation complexity of each algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 E.8 Diagnostic performance of compressed sensing MRI reconstructions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 E.9 Model card for pre-trained diffusion models. . . . . . . . . . . . . 236 E.10 Hyperparameter search space and final choices of the diffusionmodel-based algorithms on all five inverse problems. Columns marked with task names present the chosen values for the reported main results in Appendix E.1. These values are selected by a hybrid hyperparameter search strategy described in Appendix E.3.1.2. . . . 238 1 Chapter 1 INTRODUCTION Imaging technologies enable us to visualize objects that cannot be observed directly—such as distant stars or internal anatomical structures hidden within the human body. For example, the German scientist Wilhelm Conrad Röntgen discovered X-rays in 1895 and found that bones and soft tissues absorb X-rays differently. X-ray images allow doctors to see inside the body and diagnose bone fractures, infections, and other medical conditions. In astronomy, radio telescopes have allowed scientists to survey the sky, leading to the discoveries of new stars, distant galaxies, and signals from deep space that help us understand the origins and structure of the universe. Traditional imaging techniques have greatly benefited humanity, but they are approaching their limits as the demands of science and engineering continue to grow. For instance, while X-rays can distinguish between bones and soft tissues, they struggle to differentiate among various soft tissues, which often share similar physical properties. To see more subtle and nuanced structures, more sophisticated imaging techniques are needed. Similarly, imaging the suspected black hole at the center of the Milky Way would be impossible with a traditional radio telescope—unless its dish were as large as the Earth [32]. Computational Imaging refers to a class of techniques that integrate sensor design (hardware) with advanced computational algorithms (software) to form images [30]. Unlike traditional imaging techniques that rely on minimal post-processing, computational imaging techniques leverage complex reconstruction algorithms that are tightly coupled with the sensor system. These techniques have enabled unprecedented capabilities that are far beyond the reach of traditional techniques [71, 204, 304, 355]. To image a black hole, scientists connected radio telescopes distributed across the globe to create an Earth-sized computational telescope. By combining measurements from different telescopes in a physically meaningful way, the computational telescope achieves a high enough resolution to see black holes that are thousands or even millions of light-years away from us. Even so, sophisticated algorithms are required to recover the black holes from the combined measurements. In medicine, magnetic resonance imaging (MRI) was developed to visualize different 2 soft tissues in the human body [171, 186]. MRI uses electromagnetic waves and their interaction with tissue to acquire frequency-domain measurements, carefully designed so that the formed measurements correspond to the Fourier components of the target region. An image reconstruction algorithm is necessary to invert the Fourier measurements back to the image space. In both cases, computation plays a central role and is deeply integrated with the sensor system. 1.1 Computational Imaging as Inverse Problems Many problems in computational imaging can be formulated as inverse problems [268], such as astronomical imaging [49], optical microscopy [62], and medical imaging [204]. Inverse problems are also common in many other domains of science and engineering, including geophysics [294] and fluid dynamics [143]. Mathematically, inverse problems often take the following form 𝒚 = A (𝒙0 ) + 𝒏. (1.1) In this equation, 𝒙0 ∈ C𝑛 denotes the target ground truth that we want to image. Since 𝒙0 cannot be directly observed, we collect indirect measurements 𝒚 ∈ C𝑚 using a sensor system described by the forward model A : C𝑛 → C𝑚 . The term 𝒏 ∈ C𝑚 accounts for measurement noise, which includes model mismatch, hardware imperfection, and other sources of error. For simplicity, we model the noise in an additive form, although more complex noise models may apply in practice [200, 280]. Solving inverse problems amounts to finding a mapping A † : C𝑚 → C𝑛 such that b 𝒙 := A † ( 𝒚) ≈ 𝒙 0 (1.2) where we often refer to b 𝒙 as a reconstruction of 𝒙0 . Directly inverting A is usually infeasible because the problem is under-constrained and A can be highly nonlinear. Therefore, the design of the inverse mapping A † becomes challenging and requires the knowledge of both the forward model A and the underlying image 𝒙0 . 1.2 Computational Imaging Pipeline and Some Key Challenges Conceptually, computational imaging systems consist of two main stages. Figure 1.1 provides a schematic illustration using MRI as an example. Stage 1: Measurement Acquisition Acquiring measurements is the first stage of a computational imaging system. The total number of measurements is usually 3 Ground truth x0 <latexit sha1_base64="vhLHt7/0ENojMbrDJPGKa7JlXxU=">AAAF63icbdTJThtBEAbghuCEOBskx1ysoEg5ITuK4Mq+L2bxAhihnnHbDJ6eGaZ7wMaad4iUU1CueZ6c8zYZwP+vsJTUreqvqqQ+lRP5nrHF4t+h4WcjuecvRl/mX71+8/bd2Pj7qgmT2FUVN/TDuO5Io3wvUBXrWV/Vo1hJ7fiq5nTmb+q1CxUbLwz2bS9Sx1q2A6/ludJmVG10HX1SPBmbKE4Wb6PwOCkNkgkxiPLJ+MifRjN0E60C6/rSmKNSMbLHfRlbz/VVmm8kRkXS7ci2OsrSQGpljvu3300LnzNpFlphnJ3AFm71/4m+1Mb0tJN1amlPzcPaDT5ZU+eBjGPZe7Lo6PT++yHoxLdeHF7eVyudxJdxN83nG03VakhH9/uN7JJpmt6RA3JILsglNUFNkgIpUgvUIrVBbdIp6JTkgTzSGeiM1AF1SD7IJ2mQJgWggBSCQlIEikjnoHNSDIpJBmRIFmRJCSghXYAuSJegS1IX1CX1QD3SFejqhu5sFjbLtjnQHGkeNE9aAC2QFkGLpCXQEmkZtExaAa2QVkGrpDXQGmkdtE7aAG2QNkGbpC3QFmkbtE0qg8qkHdAOaRe0S9oD7ZH2QfukCqhCqoKqpBqoRqqD6qQD0AHpEHSYUbYCSw8X3uOk+nWyNDU5tfNtYmZusAxHxUfxSXwRJTEtZsSKKIuKcMWZ+C5+iuuczv3IXed+3bUODw1mPoh7kfv9Dwb2JZc=</latexit> ? Physical sensor (hardware) A(·) + n <latexit sha1_base64="XvT+/52nY6v0jtpR/U5YBWKqITg=">AAAF/nicbdTLThsxFAZgQ0lL01toF110E4EqUSGhhAXdBsL9mkBuwCDkmTjJkPHMZOwhCdFIfYi+QaWuirrtW3TN23SS8B+Vy5FsHX/nWLI3x/QdW+lM5nZi8tlU4vmL6ZfJV6/fvH2XmnlfUV4YWKJseY4X1EyuhGO7oqxt7YiaHwguTUdUzXZ+WK9eiUDZnlvSfV+cS9507YZtcR3TReqjIbluWdwZrETzhlX39JcFwzXlRWous5gZRfpxkr1L5nKzxsL321y/cDEz9deoe1Yohasthyt1ls34+nzAA21bjoiSRqiEz602b4qzOHW5FOp8MPpBlP4cSz3d8IJ4uTo90v9vDLhUqi/NuHP4XvWwNsQna6Lj8iDg/SeLpozunx+CDB1tB173vmpuhg4PelEyadRFw+CmHAyMeONRFI3JBJlEFsgiqoPqRAIkiBqgBlET1CRqgVpENsgmugRdErVBbSIH5BBJkCRyQS6RB/KIfJBP1AF1iAJQQKRAikiDNFEIComuQFdEXVCXqAfqEfVBfaJr0PWQxrYCW6G2VdAqUR6UJ1oDrRGtg9aJNkAbRJugTaIt0BbRNmibaAe0Q7QL2iXaA+0R7YP2iQ5AB0SHoEOiAqhAVAQViY5AR0THoGOiEqhEVAaViSqgClEVVCWqgWpEJ6ATolPQaUzxCMw+HHiPk8rSYnZ5cbkYz8JVNo5p9onNsnmWZV9Zjm2xAiszi0XsB/vFbhLfEj8TN4nf49bJibs7H9i9SPz5B1L4L9U=</latexit> Stage 1: Measurement Acquisition Measurements y Reconstruction Algorithm (software) A† (·) <latexit sha1_base64="Ph4tFluBVIfBkniDM5tYnLpBStU=">AAAGAXicbdTJThtBEAbghuCEOBskt+RigSIRRUJ2DuTqhX21AS/AOKhnpm0Pno3uHsBYI0XKK+QZIuUUlGseImfeJmObvxSWkrpV/VW11H0pM3QdpbPZ67HxRxOpx08mn6afPX/x8tXU9OuaCiJpiaoVuIFsmFwJ1/FFVTvaFY1QCu6Zrqib3dKgXj8TUjmBv697oWh6vO07LcfiOqHjqbeGx3XH4m6/EH8xbN5uCzlnWHagPxxPzWbns8PI3E9yN8lsfsb4+P063ysfT0/8NezAijzha8vlSh3lsqFu9rnUjuWKOG1ESoTc6vK2OEpSn3tCNfvDT8SZ94nYmVYgk+XrzFD/v9HnnlI9z0w6B09Wd2sDfLAmTn0uJe89WDS9+Pb5LniRqx0ZnN9Wzc3I5fIiTqcNW7QMbnr9vpFsPI7jEZkgk8gCWUQ2yCYSIEHUArWI2qA2UQfUIXJADtEJ6ISoC+oSuSCXyAN5RD7IJwpAAVEIColOQadEEiSJFEgRaZAmikAR0RnojOgcdE50Abog6oF6RJegywGNrAArUFsRVCQqgUpEi6BFoiXQEtEyaJloBbRCtApaJVoDrRGtg9aJNkAbRJugTaIt0BbRNmibaAe0Q1QGlYkqoArRLmiXaA+0R7QP2ieqgqpENVCNqA6qEzVADaID0AHRIegwoWQE5u4OvPtJ7dN8bmF+oZLMwiIbxSR7x2bYHMuxzyzPVlmZVZnFvrIf7Be7Sn1L/UxdpX6PWsfHbu68Ybci9ecfgO8xUw==</latexit> ! x Stage 2: Image Reconstruction Figure 1.1: Pipeline of computational imaging. In Stage 1 (Measurement Acquisition), a physical sensor acquires indirect, noisy measurements of some unknown ground truth 𝒙0 . Mathematically, this can be modeled by 𝒚 = A (𝒙 0 ) + 𝒏 where 𝒏 represents measurement noise. In Stage 2 (Image Reconstruction), an algorithm A † (·) is designed to obtain a reconstruction b 𝒙 from the measurements. limited because each measurement comes with a cost in, e.g., time and hardware. Therefore, it is important to wisely choose informative measurements of 𝒙0 for accurate recovery. But how do we define or quantify the amount of information in a set of measurements? How to choose the most informative measurements? Mathematically, different forward models have different numerical properties, some of which will make the recovery of 𝒙 0 easier. In the case of linear forward models, i.e., A := 𝑨 ∈ R𝑚×𝑛 , a matrix with a smaller condition number (defined as the ratio between the largest and the smallest singular value) is known to be easier to invert. But what about more general, nonlinear forward models? What if we have prior knowledge about the distribution of 𝒙0 ? If we already have some measurements about 𝒙 0 , could we use initial measurements to guide the selection of future ones? What if we particularly care about some local regions and/or specific features of 𝒙0 ? Answering these questions requires both domain-specific insights and a general optimization framework. In this thesis, we aim to address them by leveraging techniques in machine learning. Stage 2: Image Reconstruction The second stage involves reconstructing an estimate b 𝒙 from the acquired measurements 𝒚. Due to the non-trivial relationship between 𝒚 and 𝒙0 , having the most informative measurements does not guarantee a successful recovery. Reconstruction algorithms still play an important role because they are directly responsible for image formation and significantly impact the imaging quality. One central challenge is the ill-posed nature of Equation (1.1), meaning that multiple solutions could explain the same 𝒚 and they depend sensitively on 𝒚 [268]. There has been a rich literature on various kinds of image reconstruction algorithms. Some focus on specific domains [107, 134, 205, 267] and others are designed for generic problems [2, 19, 248, 291]. However, many open problems 4 remain. How to solve challenging inverse problems where a significant amount of information is lost in the forward model? Given that most existing methods focus on recovering static images, how to solve inverse problems on videos? How to discover multiple solutions to an inverse problem? Is there a reliable and efficient way of quantifying the likelihood of different solutions and their degrees of uncertainty? These questions are not only interesting from a theoretical perspective but also grounded in the needs of real-world applications. Addressing these challenges is essential for unlocking the next generation of computational imaging techniques. 1.3 Thesis Overview The word “sample” in the title carries two different meanings. In the context of measurement acquisition, sampling means choosing a set of measurements from a space of possible measurements. In the context of image reconstruction, sampling means generating samples from the distribution of all possible solutions that agree with the observed measurements. This thesis presents two lines of work that leverage various machine learning techniques to address both notions of sampling in computational imaging. In Part I, we investigate sampling pattern optimization in compressed sensing MRI (CS-MRI). Chapter 2 presents an overview of this part and some preliminaries of CSMRI. We mainly consider two aspects that are particularly relevant for MRI. First, in Chapter 3, we present a method that takes advantage of the sequential nature of MRI acquisition and learns a sequential sampling strategy. Dividing the entire acquisition process into a few steps, the proposed method sequentially selects the next batch of measurements based on the previously observed ones. We show that this approach significantly outperforms the baseline that determines the choice of measurements ahead of time. Compared to other reinforcement learning-based methods, our approach is trained in a fully end-to-end manner, leading to better efficiency and reconstruction quality. Second, in Chapter 4, we propose a new framework for optimizing sampling patterns with respect to downstream diagnostic tasks beyond standard reconstruction. We recognize the fact that having a reconstruction is not the end of the workflow and that common image similarity metrics do not fully reflect the reconstruction quality from a downstream task perspective. We demonstrate the effectiveness of the proposed framework on three types of tasks—reconstruction that focuses on a region-of-interest (ROI), segmentation, and classification. Our framework shows consistent improvements over baselines that are limited to image reconstruction in terms of task-relevant metrics. 5 In Part II, we present a framework for sampling from posterior distributions using diffusion models (DMs), a state-of-the-art family of generative models. Chapter 5 covers relevant background on Bayesian inverse problems and DMs. In Chapter 6, we introduce PnP-DM, a principled posterior sampling framework inspired by the Split Gibbs sampler (SGS) [296], a rigorous type of Markov chain Monte Carlo (MCMC) sampler. Our main contribution is to identify a key connection between SGS and DMs via a unified formulation called the EDM framework [151]. This connection allows us to easily incorporate pre-trained DMs as image priors for solving inverse problems. We show both a convergence theory and a validation of our approach on a synthesis Gaussian prior example. Experimental results on a few linear and nonlinear inverse problems demonstrate that our method provides both better individual reconstructions and more accurate estimations of posterior distributions. We then work towards a unified framework for diffusion-based posterior estimation in Chapter 7. Specifically, we present four instantiations of the unified framework that handle more general inverse problems. Topics include (1) incorporating text as prior, (2) harnessing high dimension for video inverse problems, (3) accommodating black box forward models, and (4) generalizing to discrete spaces. Adopting similar alternating-update structures as PnP-DM, these instantiations exhibit similar convergence guarantees and strong empirical performance. Finally, in Chapter 8, we propose a comprehensive benchmark for systematically evaluating diffusion-based methods for solving scientific inverse problems. The insights gained from these experiments point to promising directions for further improving posterior estimation in computational imaging. Part I Sampling: Measurement Acquisition 6 7 Chapter 2 OVERVIEW AND PRELIMINARIES In this part, we investigate the first topic around sampling in computational imaging—measurement acquisition—in the context of magnetic resonance imaging. 2.1 Magnetic Resonance Imaging (MRI) Magnetic resonance imaging (MRI) is a widely used imaging technology for clinical diagnosis and biomedical research [171]. MRI provides powerful tools to noninvasively visualize anatomy and physiology without ionizing radiation. However, a central challenge of MRI is its slow acquisition process. This is due to the fact that the raw measurements of MRI must be sampled one at a time rather than simultaneously, leading to a total scan time on the order of minutes. This long acquisition time leads to high cost, patient discomfort, and significant reconstruction artifacts when patients move. Therefore, there is a strong demand for accelerating the MRI acquisition process. 2.2 Compressed Sensing MRI (CS-MRI) Compressed sensing magnetic resonance imaging (CS-MRI) is a popular accelerated MRI technology based on compressed sensing (CS) [41], which aims to reconstruct the underlying image from a set of subsampled 𝑘-space measurements [204]. 2.2.1 Basics The common setup of CS-MRI involves reconstructing a target image 𝒙0 ∈ C𝑛 from its subsampled, noisy 𝑘-space measurements 𝒚 := 𝑴 𝑭𝒙 0 + 𝒏 ∈ C𝑚 (𝑚 ≪ 𝑛), (2.1) where 𝑭 is the Fourier transform, 𝑴 ∈ {0, 1} 𝑚×𝑛 is the subsampling matrix with 𝒎 ∈ {0, 1}𝑛 denoting its subsampling pattern, and 𝒏 ∈ C𝑚 is the complex measurement noise. In the parallel imaging setup, the measurements are collected from multiple receiver coils. For the 𝑗-th coil, the measurements 𝒚 𝑗 can be expressed as 𝒚 𝑗 := 𝑴 𝑭𝑺 𝑗 𝒙0 + 𝒏 𝑗 ∈ C𝑚 , (2.2) where 𝑺 𝑗 is a diagonal matrix that represents the pixel-wise sensitivity map and 𝒏 𝑗 is the measurement noise of the 𝑗-th coil. For both settings, we refer to 𝑏 := ∥𝒎∥ 1 8 as the sampling budget and 𝑅 := 𝑏𝑛 as the acceleration ratio of the acquisition. Once properly implemented1 , the accelerated sequence will shorten the scan time by a factor of 𝑅, leading to significantly higher throughput. 2.2.2 Image Reconstruction Traditional image reconstruction techniques in CS-MRI include solving a regularized optimization problem [205, 243]. Recently, deep learning (DL) methods have achieved state-of-the-art performance on CS-MRI reconstruction. One line of work combines data-driven priors with model-based iterative reconstruction (MBIR) [4, 149, 248, 291]. Another line of work learns a model-free reconstruction network via end-to-end training [173, 174, 235, 305, 329]. A third line of work, known as deep unrolling (DU), combines the characteristics of MBIR and end-to-end training [1, 2, 117, 123, 136, 191, 255, 267, 328, 331, 343]. The idea is to “unroll” an iterative optimization procedure into a cascade of mappings and train these mappings end-to-end so that they can gradually map a low-resolution input image to a high-quality output reconstruction. Inheriting the advantage of both MBIR and end-to-end learning, these methods exhibit state-of-the-art performance on CS-MRI reconstruction. 2.2.3 Sampling Pattern Design Subsampling patterns in traditional CS-MRI are often generated randomly or handcrafted to have a point spread function (PSF) with a high degree of incoherence defined under the compressed sensing theory [41]. Popular subsampling patterns include the 2D variable density [204], bidirectional Cartesian [301], Poisson-disc [288], and continuous-trajectory variable density [54], among others [74, 239]. These subsampling patterns are designed for generic image reconstruction and not optimized for any specific body part or diagnostic purpose. Therefore, these patterns may lead to suboptimal performance for downstream tasks where specific anatomical or pathological information is relevant. Recently, a new group of DL-based methods, known as co-design, has been proposed to jointly optimize the subsampling pattern 𝒎 and a downstream reconstruction module, leading to better reconstruction performance than the traditional CS-MRI methods [3, 8, 12, 216, 229, 237, 299, 300, 302, 307, 327, 330, 344, 351, 362]. 1 The actual implementation of subsampling in MRI involves physical and hardware constraints, which can be complicated. Here we omit the details. We will provide details on the implementation in Chapter 4. 9 Due to the sequential nature of MRI acquisition, reinforcement learning (RL) based methods have also been considered to learn an adaptive policy for determining 𝒎 [13, 195, 231]. 2.3 Part Outline For the rest of this part, we will discuss two topics around the optimization of sampling patterns for CS-MRI: • In Chapter 3, we propose an end-to-end approach to learning a sequential sampling strategy. Our method adaptively selects which measurements to take on the fly based on the previous selections. We show that the proposed method not only outperforms previous non-adaptive end-to-end methods but also improves upon RL-based methods. The improvements are reflected in terms of both statistical significance and visual quality. • In Chapter 4, we present a framework for optimizing the sampling patterns directly for downstream diagnostic tasks. We show that if there is a clear downstream objective for an MRI scan, such as inspecting a particular region of interest or type of tissue, optimizing directly for that objective often leads to significantly better performance than optimizing for reconstruction as a surrogate. Besides experiments on publicly available datasets, we also program our learned sequence on an MRI machine. Notably, our actual implementation achieves a 4× scan time reduction as expected while providing high-fidelity reconstructions. 10 Chapter 3 LEARNING SEQUENTIAL SAMPLING AND RECONSTRUCTION STRATEGIES In this chapter, we investigate one way to improve measurement acquisition: leveraging the sequential nature of the acquisition process. Intuitively, if some measurements are observed before we have a chance to choose others, leveraging the information in the observed measurements allows us to make a wiser choice about which measurements to observe next. Based on this intuition, we propose a fully differentiable framework that jointly learns a sequential sampling policy simultaneously with a reconstruction strategy. This co-designed framework can adapt during acquisition to capture the most informative measurements for a particular target (Figure 3.1). Experimental results on the fastMRI knee dataset demonstrate that the proposed approach successfully utilizes intermediate information during the sampling process to boost reconstruction performance. In particular, our proposed method outperforms the learning-based LOUPE baseline [12] on over 96% of test samples. We also investigate the individual and collective benefits of the sequential sampling and co-design strategies. This chapter is based on our work [335], published in the Proceedings of the 1st Machine Learning for Health symposium 2021 (ML4H 2021), PMLR volume 158, ISSN: 2640-3498. This work also received the Best Paper Award for ML4H 2021. The appendix for this chapter is Appendix A. The code for the work presented in this chapter is available at https://github.com/tianweiy/SeqMRI. 3.1 Introduction The success of CS-MRI depends on two critical factors: (a) a carefully designed 𝑘-space subsampling pattern to collect informative measurements, and (b) a reconstruction method that accurately recovers high-quality images from subsampled data. Current MRI protocols collect measurements over time using static subsampling patterns that were designed a priori. To further accelerate a scan, we are interested in sequential sampling patterns that adapt to a target based on intermediate information collected during acquisition. A high-fidelity MRI reconstruction stems from the cooperation between the 𝑘-space 11 8× subsampled k-space Ground truth Right-rotated Left-rotated Final reconstruction Step 1 Step 2 Step 3 Step 4 Measurements: Pre-selected Model-selected Knee rotation direction Figure 3.1: We propose a sequential sampling and reconstruction co-design framework for accelerated MRI that adapts to a target during acquisition. Here, we visualize the sampling policy and final reconstruction of rotated knees in a single-coil imaging setting with 8× acceleration (8× subsampling). The first four columns show the cumulative 𝑘-space measurements selected by the proposed learned sampler (pink) in acquisition steps 1 through 4 (during a 4-step acquisition). The fifth column shows the final image recovered by the proposed learned reconstructor, and the last column is the ground truth. This example illustrates how our model has learned to adapt to different 𝑘-space distributions: the final sampling patterns in the fourth column contain visible directional structure that aligns with the 𝑘-space power spectrum. Rotated anatomical images, such as these rotated knee images, were not included in the training set (or quantitatively evaluated test set). sampling strategy and the reconstruction method. Traditionally, MRI subsampling patterns and reconstruction methods have been largely independently designed. We are instead interested in co-design, where jointly designing the two components can synergistically boost reconstruction quality. Our approach builds on neural networkbased co-design frameworks that have shown strong empirical performance and take advantage of efficient differentiable training [12, 157, 158, 270]. In this chapter, we propose an end-to-end differentiable framework that successfully combines co-design and sequential sampling. Specifically, we design an explicit sequential structure of 𝑇 steps, with each step consisting of a jointly learned 𝑘-space sampler and reconstructor. Comparing our model with prior work in accelerated MRI, we investigate the individual and collective benefits of sequential sampling and co-design. We evaluate the proposed model on the NYU fastMRI datasets and find that: (1) even a single sequential step consistently improves performance compared to using a pre-designed sampling pattern; (2) more sequential steps can improve reconstruction quality, but with diminishing returns; and (3) a fully differentiable approach enables more efficient and effective co-design than non-differentiable 12 methods. Notably, despite various published works on sequential sampling using reinforcement learning [13, 231], we are among the first to demonstrate consistent and statistically significant improvement over a state-of-the-art learned non-sequential baseline [12] through the use of a fully-differentiable sequential computation graph. The rest of the chapter is organized as follows. In Section 3.2, we review past literature in accelerated MRI from the perspectives of co-design and sequential sampling. We introduce our proposed framework and its training procedure in Section 3.3. We then present our experimental settings, comparisons between our model and other baselines, and ablation studies in Section 3.4. Finally, we conclude with a discussion on future directions of our framework in Section 3.5. 3.2 Related Work Prior work in accelerated MRI can be organized into four quadrants, split across two dimensions: methods that (1) independently (and/or manually) design the sampler and reconstructor versus data-driven co-design, and (2) specify the sampling pattern before a scan (pre-designed) versus adapt samples to the target during acquisition. In Section 3.2.1, we cover traditional methods that independently (and/or manually) design the sampler and reconstructor. In Section 3.2.2, we discuss previous methods that perform pre-designed acquisition in a co-design framework. In Section 3.2.3, we introduce recent work on sequential sampling for accelerated MRI. We conclude in Section 3.2.4 with an overview of methods that attempt to combine co-design and sequential sampling, but without end-to-end learning. Our end-to-end framework efficiently combines co-design and sequential sampling, successfully inheriting the advantages of both approaches. 3.2.1 Traditional Methods Accelerated MRI sampling patterns implemented on commercial scanners are motivated by ideas in compressed sensing (CS) [41]. Since anatomical images are sparse in a linearly transformed space, it is possible to reconstruct a high-fidelity image with incoherent 𝑘-space data sampled below the Nyquist-Shannon rate [205]. In the context of 2D CS-MRI, prior work has investigated uniform density random sampling, variable density sampling [204], Poisson-disc sampling [288], continuous-trajectory variable density sampling [54], and equi-spaced sampling [121]. These sampling patterns are easy to implement, but not adaptive to specific datasets or target images. Once sparse 𝑘-space measurements have been acquired, an image is typically reconstructed via an optimization problem that involves two objectives: the first encour- 13 ages a reconstruction that matches the observed data, while the second addresses the ill-posed nature of the under-determined system through image regularization. Common regularization terms include total variation (TV) [29] and the ℓ1 -norm after a sparsifying transformation (obtained using wavelets [204, 207] or dictionary decompositions [140, 243, 340]). Recently, convolutional neural networks (CNNs) have demonstrated impressive performance in MRI reconstruction. Strategies include unrolled networks [123, 191, 255, 331], UNet-based networks [142, 174], GAN-based networks [235, 329], among others [190, 305, 360]. These learning methods have achieved state-of-theart performance on public MRI challenge datasets [338]. In our proposed co-design model, we employ a convolutional UNet for image reconstruction. 3.2.2 Co-Design The goal of co-design is to jointly identify the optimal sampling and reconstruction strategies. This is an NP-hard combinatorial optimization problem due to the discrete nature of the sampling pattern. Theoretically, one could identify an optimized reconstructor for every possible sampling strategy, and then pick the overall strategy that performs best. However, this brute-force optimization approach is not practical, as it requires enumerating an exponential number of possible sampling combinations. Early work formulated the co-design as a nested (or bi-level) optimization problem and alternated between optimizing a sampler and a reconstructor [242]. More recently, deep learning has enabled a data-driven solution to the co-design problem, where the sampler and reconstructor can be jointly learned through endto-end training. For example, [12, 312, 344] proposed co-design frameworks for 2D Cartesian 𝑘-space sampling and [299, 311] applied co-design to 2D radial 𝑘space sampling.1 These methods have shown superior performance over previous baselines that combine an individually-optimized sampler and reconstructor pair [12, 270, 299, 312, 344]. However, these methods do not take advantage of the sequential nature of data collection during an MRI scan, and only solve for a generic sampling pattern for an entire dataset. 3.2.3 Sequential Sampling Since MRI scanners acquire measurements in a sequential manner, recent work has modeled the sampling process in the context of sequential decision making. 1 Differentiable co-design of discrete sensing and reconstruction methods has also been success- fully applied to other imaging domains as well [270]. 14 Sequential decisions enable the sampling pattern to adapt to different input images by choosing the next 𝑘-space sample based on prior measurements. Reinforcement learning (RL) methods have primarily been employed for this purpose. For example, [13, 231] formulate the sampling problem as a Partially Observable Markov Decision Process (POMDP) and use Policy Gradient [18] and DDQN [124] methods, respectively. These RL methods heavily rely on a pre-trained reconstructor, which leads to a training mismatch (and thus potentially suboptimal performance), since the reconstructor was trained with a sampling strategy that does not match the strategy eventually employed by the RL-learned sampler. Furthermore, these RL methods are difficult and costly to train, as they are non-differentiable. As a consequence, in the context of accelerated MRI, these methods either fail to be adaptive to different input images or have only limited improvement over simple baselines [13, 231]. 3.2.4 Co-Design & Sequential Sampling Several approaches have attempted to combine co-design with sequential sampling strategies, but they have achieved only limited success to date. The work of [146] draws inspiration from AlphaGo [258] and trains a sampler to emulate the policy distribution obtained through a Monte Carlo Tree Search (MCTS); the reconstructor is trained during alternating optimization steps. However, according to the results in [13], the MCTS method in [146] has limited improvement over simple baselines, and is outperformed by the sequential sampling method in [13] without co-design. This poor performance may be due to the overall MCTS framework not being end-to-end differentiable. Alternatively, [351] proposes a framework that trains a ResNet to reconstruct the anatomical image simultaneously with an evaluator network that is trained to select the most uncertain measurement in 𝑘-space. Although the authors demonstrate how this framework can be used to sequentially choose the next sample, it is not explicitly trained end-to-end and is outperformed by [231], which does not use co-design. This training-testing mismatch limits the potential improvement of sequential sampling. In contrast, we design a fully differentiable end-to-end framework that leverages the sequential nature of 𝑘-space MRI acquisition during both training and testing. 3.3 Method Figure 3.2 summarizes the co-design framework for our sequential sampling and reconstruction model. We partition the 𝑘-space sampling budget into 𝑇 steps and 15 Masked k-space Mask m(t) Step 0 ⊙ Zero-filled reconstruction y(t) F−1 x̃(t) Reconstruction ̂ x (t) Reconstructor θ F Sampler Step 1 F−1 ⊙ q Reconstructor θ F Sampler Step 2 F−1 ⊙ q Reconstructor θ F Sampler Step T F−1 ⊙ q Reconstructor θ Loss Input without back-propagation Input / Output <latexit sha1_base64="p1iaaD1m7ojNsHnn9453ho6TnzU=">AAADS3icfZLNbtNAEMe3LoUSvlI4crGIkDgUy3b8lVsFEuJYJNJWSkK0Xo/TVddra3ddESy/CVd4Gx6A5+CGOLB2oihxECtZMzvzm52/RxMXjEpl2z8PjMM7R3fvHd/vPXj46PGT/snTC5mXgsCY5CwXVzGWwCiHsaKKwVUhAGcxg8v45m2Tv7wFIWnOP6plAbMMLzhNKcFKh+b9/jTD6ppgVr2rP1WvnXreH9iW3R5z33HWzgCtz/n8xBhMk5yUGXBFGJZy4tiFmlVYKEoY1L1pKaHA5AYvYKJdjjOQs6qVXpsvdSQx01zojyuzjW5XVDiTcpnFmmyEym6uCf4rNylVGs0qyotSASerRmnJTJWbzRzMhAogii21g4mgWqtJrrHAROlp7XT5vJK6E4szfU8g1VNvs9VCAPC6Eou4ruxT2wqi0NcmdIOgQ+IvpYBt0vMaMoqCsEPGrNwGh6MWDIZuVPd2yVxgvtiwlu7dFPhB4GnjuNGo87KAZAOHo6GGT+29H8LLDeP7Xqhf8kK3NcPQ/49Uy3W8qNEbhG6jIhppvXqtnO4S7TsXruUElv3BG5y9WS/YMXqOXqBXyEEhOkPv0TkaI4Ju0Vf0DX03fhi/jN/GnxVqHKxrnqGdc3j0F+d4BNo=</latexit> F 1 <latexit sha1_base64="p1iaaD1m7ojNsHnn9453ho6TnzU=">AAADS3icfZLNbtNAEMe3LoUSvlI4crGIkDgUy3b8lVsFEuJYJNJWSkK0Xo/TVddra3ddESy/CVd4Gx6A5+CGOLB2oihxECtZMzvzm52/RxMXjEpl2z8PjMM7R3fvHd/vPXj46PGT/snTC5mXgsCY5CwXVzGWwCiHsaKKwVUhAGcxg8v45m2Tv7wFIWnOP6plAbMMLzhNKcFKh+b9/jTD6ppgVr2rP1WvnXreH9iW3R5z33HWzgCtz/n8xBhMk5yUGXBFGJZy4tiFmlVYKEoY1L1pKaHA5AYvYKJdjjOQs6qVXpsvdSQx01zojyuzjW5XVDiTcpnFmmyEym6uCf4rNylVGs0qyotSASerRmnJTJWbzRzMhAogii21g4mgWqtJrrHAROlp7XT5vJK6E4szfU8g1VNvs9VCAPC6Eou4ruxT2wqi0NcmdIOgQ+IvpYBt0vMaMoqCsEPGrNwGh6MWDIZuVPd2yVxgvtiwlu7dFPhB4GnjuNGo87KAZAOHo6GGT+29H8LLDeP7Xqhf8kK3NcPQ/49Uy3W8qNEbhG6jIhppvXqtnO4S7TsXruUElv3BG5y9WS/YMXqOXqBXyEEhOkPv0TkaI4Ju0Vf0DX03fhi/jN/GnxVqHKxrnqGdc3j0F+d4BNo=</latexit> FFT F ⊙ Element-wise multiplication with ground truth k-space x0 1 iFFT Compute loss Pre-selected measurements Ground Truth Figure 3.2: Overview of the proposed sequential sampling framework. Lowfrequency samples are pre-selected and measured in 𝑘-space. The subsampled 𝑘-space is transformed into a zero-filled image, which is fed into a reconstructor R𝜽 (·) to produce an intermediate image reconstruction (Equation (3.1)). The intermediate reconstruction and measurements are passed into a sampler network S𝒒 (·), which outputs a discrete probability distribution representing suggested samples for the next iteration. An action is sampled from this distribution (Equation (3.2)), and the corresponding 𝑘-space measurements are acquired. The sampling and reconstruction process is repeated for 𝑇 steps. The sampler and reconstructor are neural networks learned via end-to-end training with a loss on the final reconstructed image. Weights are shared across all 𝑇 acquisition steps. denote a model with 𝑇 sequential steps as “𝑇-Step Seq” hereafter. At each step 𝑡, the pipeline applies a sampler S𝒒 (·) and a reconstructor R𝜽 (·). The reconstructor aims to remove aliasing artifacts in the zero-filled reconstruction e 𝒙 (𝑡) of the acquired measurements 𝒚 (𝑡) :     b 𝒙 (𝑡) = R𝜽 e 𝒙 (𝑡) = R𝜽 𝑭 −1 𝒚 (𝑡) . (3.1) The sampler, in turn, seeks to intelligently select which 𝑘-space samples to observe next, based on previously observed measurements and the 𝑘-space of the 16 reconstruction so far:   𝒎 (𝑡) ∼ S𝒒 𝒎 (𝑡−1) , 𝒚 (𝑡−1) , 𝑭b 𝒙 (𝑡−1) s.t. ∥𝒎 (𝑡) − 𝒎 (𝑡−1) ∥ 1 = 𝑏 (𝑡) (3.2) where 𝒎 (𝑡) is a binary mask representing the sampling pattern collected up until step 𝑡 and 𝑏 (𝑡) is the sampling budget at step 𝑡. We model the sampler S𝒒 (·) and reconstructor R𝜽 (·) as neural networks, and jointly optimize their weights, 𝒒 and 𝜽, by minimizing the image reconstruction error between the final-step reconstruction b 𝒙 (𝑇) and the ground truth target image 𝒙0 :   𝒒 ∗ , 𝜽 ∗ = arg min Lrecon. b 𝒙 (𝑇) , 𝒙0 , (3.3) 𝒒,𝜽 where Lrecon. is an image reconstruction loss function based on, e.g., structural similarity index measure (SSIM) [310] or peak signal-to-noise ratio (PSNR). We choose to share sampler and reconstructor weights across all steps. The sampler and reconstructor are described in more detail in Section 3.3.1 and Section 3.3.2, respectively. 3.3.1 Sampler Subsampling Patterns We follow prior work to consider two types of 𝑘-space sampling: 1D line sampling and unconstrained 2D point sampling [12, 13, 338, 351]. Figure 3.3 illustrates these two sampling scenarios, which enable different levels of sampling flexibility. In 1D line sampling, only vertical trajectories along the vertical (frequency encoding) direction can be sampled. This corresponds to the Cartesian subsampling setting in 2D MRI. On the other hand, 2D point sampling allows any measurement on the 2D grid in 𝑘-space to be acquired. Unconstrained 2D point sampling represents an upper bound on sampling flexibility and can be implemented on 3D Cartesian sequences. We note that our sequential sampling framework is applicable to other patterns, such as radial sampling [26]. As low-frequency 𝑘-space measurements contain the most information about largescale anatomical structure, it is common practice in accelerated MRI to fix a small number of low-frequency 𝑘-space samples to always be collected [13, 231, 351]. We follow this strategy by allocating 1/8 of the total sampling budget to the central low-frequency region in all experiments. Probabilistic Modeling We follow [12] to model the subsampling strategy at step 𝑡 as an element-wise Bernoulli policy. To learn the optimal probabilities at 17 1D line sampling 2D point sampling Pre-selected measurements DC component Figure 3.3: Visualizations of two types of 𝑘-space sampling patterns: 1D line sampling and 2D point sampling. White regions are sampled from a uniform distribution over the space of possible actions. The center low-frequency samples are pre-selected in all experiments before any further sampling. DC corresponds to the (0, 0) frequency. each step, the sampler takes in the sampling mask 𝒎 (𝑡−1) , 𝑘-space measurements 𝒚 (𝑡−1) , and the 𝑘-space of the reconstruction 𝑭b 𝒙 (𝑡−1) up to date (see Equation (3.2)). The sampler then outputs a set of logits 𝒒𝑖(𝑡) ∈ R𝑙 , where 𝑙 denotes the number of possible sampling indices (i.e., number of columns for the 1D line pattern and number of pixels for the 2D point pattern). These logits are subsequently turned into probabilities e 𝒑𝑖(𝑡) := Sigmoid(𝒒𝑖(𝑡) ). We then rescale e 𝒑 (𝑡) to obtain a probability map 𝒑 (𝑡) with 𝑏 (𝑡) measurements in expectation: 𝒑 (𝑡 ) (𝑡)    𝛼𝛽 e  𝒑 (𝑡) =   1 − 1−𝛼 𝒑 (𝑡) ) 1−𝛽 (1 − e  if 𝛽 ≥ 𝛼 otherwise (𝑡 ) where 𝛼 := 𝑏𝑛 , 𝛽 := ∥ e𝒑 𝑛 ∥ 1 , and 1 is the all-one vector. Additionally, to avoid acquiring the same measurements again, the probabilities of sampling previously acquired lines are set to zero: 𝒑 (𝑡) = 𝒑 (𝑡) ⊙ (1 − 𝒎 (𝑡−1) ) (3.4) where ⊙ is the element-wise multiplication. We then draw Bernoulli random samples for all sampling locations according to the probability map 𝒑 (𝑡) , where 1 indicates sampling a location and 0 indicates otherwise. Mathematically, we can write this sampling process as the 𝑘-space sampling mask 𝒎 𝑡 for acquisition step 𝑡 via: 𝒎 (𝑡) = 1 [𝒖 ≤ 𝒑 (𝑡) ] + 𝒎 (𝑡−1) , (3.5) where 𝒖 ∈ [0, 1] 𝑙 is a vector of 𝑙 independent realizations of the uniform distribution on the interval [0, 1]. We repeatedly sample 𝒖 until ∥𝒎 (𝑡) − 𝒎 (𝑡−1) ∥ 1 ≈ 𝑏 (𝑡) under a small tolerance. This sampling process encourages exploration of different patterns 18 Table 3.1: SSIM comparison of 2D point sampling for 4×, 8×, and 16× accelerations. Our 4-step sequential model outperforms the previous approaches when tested on the fastMRI knee test set. For each model, we compute the test average and standard deviation obtained across three trained models with independent initialization. Method 4× 8× 16× Random 90.40 (0.02) 87.43 (0.05) 84.25 (0.00) Spectrum 92.39 (0.01) 90.38 (0.01) 88.37 (0.01) LOUPE [12] 92.44 (0.01) 90.60 (0.03) 88.73 (0.04) 4-Step Seq. (ours) 92.91 (0.01) 91.07 (0.02) 89.10 (0.03) and ensures that the sampling patterns approximately satisfy the budget constraint. Note that the indicator function 1 [·] is not differentiable, which hinders the training of the model through back-propagation. To overcome the non-differentiability, we use a straight-through estimator that applies the indicator function in the forward pass, while approximating its gradients by treating the binary indicator function as a sigmoid during back-propagation [20]. In this way, we can capture binary sampling in real MR scanning, while retaining gradients for end-to-end training. Sampler Architecture For the 1D line sampler, we use a Multilayer Perceptron (MLP) network with five layers separated by ReLU activation functions. For the 2D point sampler, we instead use an eight-block UNet with ReLU activation functions whose architecture is more scalable on higher-dimensional action spaces. Further details of the network architectures for both samplers are included in Appendix A.1. 3.3.2 Reconstructor Our proposed co-design sequential framework learns the parameters of a reconstructor jointly with the sampler. Although many networks have been proposed for MR image reconstruction [123, 255, 267, 331], the choice of reconstructor architecture is not the main focus of this chapter. We following [12, 13, 338] to adopt a standard 8-block U-Net architecture [250] . The input to the reconstructor at each step 𝑡 is the complex-valued zero-filled image, b 𝒙 (𝑡) , and the output is a single-channel real-valued image, e 𝒙 (𝑡) . The UNet reconstructor contains four downsampling blocks and four upsampling blocks, each consisting of two 3×3 convolutions separated by ReLU and instance normalization [286]. We note that our framework is agnostic to the specific reconstructor architecture. 8x Random Spectrum 8× Reconstruction 89.85 LOUPE 93.65 19 ID: 1834 1-Step Seq. (ours) 4-Step Seq. (ours) 94.63 95.19 Ground truth 95.66 SSIM Sampling pattern k-space 500500 Non-seq. over LOUPE 1-Step Seq. over LOUPE 2-Step Seq. over LOUPE 400400 4-Step Seq. over LOUPE 500 500 500 77.42% 74.05% 88.74% 96.96% 400 400 400 300300 300 300 300 200200 200 200 200 100100 100 100 min=-1.22 0 0-2 -1.5 -1 ≤-2 -1 max=1.87 -0.5 0 0 0.5 1 1 1.5 2 2 2.5 min=-1.06 3 0 -2 ≥3 ≤-2 -1.5 -1 -1 max=2.83 -0.5 0 0 0.5 1 1 1.5 2 2 2.5 100 min=-1.11 3 0 -2 4× acceleration factor # of test samples Figure 3.4: Visualizations of example reconstructions with an 4× acceleration for 1D line sampling. Two zoomed-in image patches are shown along with the cumulative 𝑘-space measurements selected by each policy. Our sequential approach often provides more accurate reconstructions with detailed local structures. More visualizations are included in the Appendix A.4. ≥3 ≤-2 -1.5 -1 -1 max=2.15 -0.5 0 0 0.5 1 1 1.5 2 2 2.5 min=-0.94 3 0 -2 ≥3 ≤-2 -1.5 -1 -1 max=2.74 -0.5 0 0 0.5 1 1 1.5 2 2 2.5 3 ≥3 SSIM improvement distribution on the test set (averaged over 3 runs) Figure 3.5: Histograms of pair-wise SSIM differences between our sequential models and LOUPE [12] on all 1,851 test images. Positive numbers indicate improvement over LOUPE. The results are acquired by averaging three runs of 4× accelerated 2D point subsampling. More sequential steps lead to a bigger advantage over LOUPE, with the 4-step sequential model outperforming LOUPE on 96.96% of samples. This performance pattern holds for the 1D line scenario and other acceleration factors as well, as shown in Appendix A.3. More quantitative results are given in Table 3.3. 3.4 Experiments 3.4.1 Setup and Implementation Details We evaluate our sequential sampling and reconstruction method on the NYU fastMRI open dataset [338]2 . The dataset provides raw single-coil 𝑘-space measurements for knee images, with 973 training set volumes and 97 validation set volumes [338]. We follow the setup of [231] and split the original validation set into a new validation set with 48 volumes and a test set with 49 volumes, which results in 34,742 2D slices for training, 1,785 slices for validation, and 1,851 slices 2 https://fastmri.org/ 20 Table 3.2: SSIM comparison of 1D line sampling under 4× acceleration. Our 4-step sequential model outperforms the previous approaches when tested on the fastMRI knee test set. A paired samples 𝑡-test shows a statistically significant difference between our 4-step sequential model and LOUPE [12], with a 𝑝-value smaller than 10−300 . For each model, we compute the test average and standard deviation obtained across three trained models with independent initialization. Random Equispaced Evaluator [351] PG-MRI [13] LOUPE [12] 4-Step Seq. (ours) 85.95 (0.05) 86.86 (0.06) 85.99 (0.04) 93 93 92.44 92.56 87.97 (0.09) 92.66 92.74 89.52 (0.02) 92.91 991 1 SSIM SSIM 92 92 90.84 90.66 90.73 88.67 88.82 90.60 90 90 89.52 89 89 88.73 88 90.89 91.20 91.08 90.95 91.07 88.88 91.08 (0.09) 2D 1D 4× 8× 16× 89.10 . . . Non-Seq. qSeq. q. 1-Step qSeq.2-Step qSeq.4-Step PE Se Se Se Se U p n p p LO te te te No 4-S 1-S 2-S LOUPE Figure 3.6: Comparison between our sequential model and the LOUPE model on the fastMRI knee test set. Our sequential model outperforms LOUPE for all acceleration ratios with an improvement comparable to 25% of the benefit of doubling the number of 𝑘-space measurements. The performance of our sequential model in the 1D line sampling case significantly outperforms LOUPE but plateaus after 2 sequential sampling steps, possibly due to the restricted action space of 1D line sampling. for testing. For computational efficiency, we follow [13, 351] to crop the 𝑘-space to the center 128×128 region. We use the structural similarity index measure (SSIM) for the primary evaluation metric, which has been found to correlate well with expert evaluations [164]. We define the loss function Lrecon. (𝒙, 𝒙0 ) := − SSIM(𝒙, 𝒙0 ) following [13, 231, 267], which is computed using a window size of 7×7 and hyperparmeters 𝑘 1 = 0.01, 𝑘 2 = 0.03 following the fastMRI challenge’s official implementation. We use the Adam optimizer [162] and train our model for 50 epochs with a learning rate of 1𝑒−3 for 2D point sampling experiments and 5𝑒 − 5 for 1D line sampling experiments. The learning rate is decreased by half every ten epochs. Training each model takes at most one day on a single NVIDIA RTX 2080Ti GPU. 21 4-Step Seq. over 2-Step Seq. 500 92.13% 400400 90.17% 400 300300 300 200200 200 100100 100 min=-0.90 0 0-0.5 ≤-0.5 max=1.84 0 0 0.5 0.5 1 1 1.5 min=-0.96 0 -0.5 ≥1.5 ≤-0.5 max=1.36 0 0 0.5 0.5 1 1 4× acceleration factor # of test samples 4-Step Seq. over 1-Step Seq. 500500 1.5 ≥1.5 SSIM improvement distribution on the test set (averaged over 3 runs) Figure 3.7: Histograms of pair-wise SSIM comparison on all 1,851 test images with a different number of sequential steps (𝑇), using 2D point sampling with a 4× acceleration factor. The relative error between the 4-step and 1-step (left) or 2step(right) demonstrates that additional sequential steps help to boost performance, but with diminishing returns as 𝑇 increases. 3.4.2 Results In Figure 3.1, we visualize our framework’s sequential sampling masks and final reconstruction for rotated knees in the 8× acceleration setting. Starting from preselected measurements, our model sequentially samples 2D 𝑘-space measurements based on previous observations. Here, we demonstrate that our model can accurately estimate and leverage the 𝑘-space structure during the sequential sampling steps. In particular, the final sampling patterns contain visible directional structures that align with the true 𝑘-space power spectrum induced by knee rotation. This highlights the adaptivity of our sequential model, as no rotated anatomical images are included in the training set. 2D Point Sampling In Table 3.1, we compare our method to several baselines for 4×, 8×, and 16× accelerations, including: (1) Random [110]: randomly select points from a uniform distribution, (2) Spectrum [289]: select points with the largest 𝑘-space magnitude over the training set, (3) LOUPE [12]: select points before acquisition using a distribution learned via co-design. For each baseline, the reconstruction network has been trained with the specified sampling policy. Please refer to Appendix A.2 for the implementation details of these baseline methods. Our 4-step sequential model achieves the best reconstruction performance across different acceleration ratios. A paired samples 𝑡-test between our method and the previous state-of-the-art pre-designed sampling approach, LOUPE [12], indicates a statistically significant difference in performance, with a 𝑝-value less than 10−160 for all acceleration ratios. By inspecting Figure 3.6, one can see that our 4-step model outperforms LOUPE for all acceleration ratios, with an improvement comparable 22 to 25% of the benefit of doubling the number of 𝑘-space measurements from 8× to 4×. 1D Line Sampling We compare our model to previous methods for the 1D line sampling with a 4× acceleration factor in Table 3.2. The baselines we consider include: (1) Random: randomly select 𝑘-space lines from a uniform distribution, (2) Equispaced [121]: select equidistant lines, (3) Evaluator [351]: sequentially select lines following a learned evaluation function, (4) PG-MRI [13]: sequentially select lines using a conditional distribution trained by a policy gradient algorithm, (5) LOUPE [12]: select lines before acquisition using a distribution learned via codesign. The implementation details of these baselines are included in Appendix A.2. Our 4-step sequential framework significantly outperforms prior methods, with an SSIM improvement of roughly 1.6 over the previous learning-based method, LOUPE [12]. A paired samples 𝑡-test also indicates a highly statistically significant boost in performance compared to LOUPE with a 𝑡-score of 64.01 and a 𝑝-value smaller than 10−300 . Note that our differentiable end-to-end framework also significantly outperforms a sequential reinforcement learning optimization approach, PG-MRI [13]. Figure 3.4 shows sample images reconstructed using the approaches mentioned above. Using the same number of 𝑘-space samples, our 4-step sequential model most accurately recovers important anatomical structures and details. The orange and blue patches under each reconstruction highlight certain regions where our method significantly outperforms other baselines. Adaptive vs. Pre-Designed Sampling Figure 3.5 shows histograms of pair-wise SSIM differences on each test sample, computed between our sequential method and LOUPE. Here we introduce a non-sequential baseline referred to as “non-seq." which uses the same network architecture as our sequential model but replaces the prior 𝑘-space measurements used as input with a random tensor. The “non-seq." baseline demonstrates a performance comparable to that of LOUPE in Figure 3.5, Figure 3.6, and Table 3.3. However, more sequential steps consistently lead to a higher percentage of improved samples. Thus, we can conclude that the improvement is not merely due to a better framework architecture but the adaptive sampling strategy of our approach. 23 Table 3.3: The percentage of test samples on which our method with different numbers of sequential sampling steps (𝑇) outperforms the LOUPE [12] baseline for 2D point sampling. The percentage average and standard deviation are obtained using results from three trained models with independent initialization. Method 4× 8× 16× Non-Seq. 74.05 (2.56) 60.18 (3.03) 46.98 (8.58) 1-Step Seq. 77.42 (7.89) 57.05 (4.36) 51.09 (4.16) 2-Step Seq. 88.74 (0.45) 83.04 (3.78) 56.42 (4.62) 4-Step Seq. 96.96 (0.73) 92.62 (0.46) 76.91 (2.29) Number of Sequential Steps We further ablate the impact of the number of sequential steps. For the case of 2D point sampling in Figure 3.6, the accuracy consistently increases as the number of sequential sampling steps increases. To further understand the improvements seen with additional sequential steps, we perform a pair-wise SSIM comparison between our sequential models. Figure 3.7 shows the result of 2D point sampling with a 4× acceleration ratio. Additional sequential steps boost the reconstruction performance for almost all subjects, with diminishing returns as 𝑇 increases. Table 3.3 shows quantitative results that compare the percentage of test samples that outperform the LOUPE baseline. On 2D point sampling, our 4-step sequential model outperforms LOUPE roughly 97%, 89%, and 77% of the time for the 4×, 8× and 16× acceleration factors, respectively. Ablation Study on Co-Design We demonstrate the advantage of co-designing the sampler and reconstructor in Table 3.4. Specifically, we pre-train a reconstructor using a uniform sampling policy and demonstrate the performance improvement that occurs when jointly learning the reconstructor weights with the sampler. Codesigning the reconstructor with the sampler significantly improves performance, with an increase of 2.33–2.51 SSIM for 2D point sampling with a 4× acceleration factor. 3.5 Conclusion Accelerating the MRI acquisition process has the potential to reduce patient discomfort, increase throughput, and expand the use of MRI worldwide. In this chapter, we proposed an end-to-end sequential sampling and reconstruction framework for accelerated MR imaging. We leveraged the sequential nature of MRI acquisition and design a model with an explicit sequential structure that jointly optimizes a neural network-based sampler simultaneously with a network-based reconstructor. 24 Table 3.4: Ablation results showing the advantage of co-design with a 4× acceleration ratio and 2D point sampling. When co-design is specified as “Yes” the reconstruction network has been jointly optimized with the sampler. Otherwise, the sampler was optimized with a fixed reconstructor that was pre-trained with a random sampling policy. Co-design 1-Step Seq. 4-Step Seq. ✓ ✗ 92.66 (0.06) 90.33 (0.01) 92.91 (0.01) 90.40 (0.02) In our experiments, this simple framework outperforms previous state-of-the-art MR sampling approaches for up to nearly 97% of the test samples on the fastMRI single-coil knee dataset. Overall, our results suggest that future methods for MRI sampling can benefit from the collaboration of sequential sampling and co-design via end-to-end learning. 25 Chapter 4 LEARNING END-TO-END STRATEGIES FOR THE DOWNSTREAM TASK OF INTEREST In the previous chapter, we discussed one way of improving measurement acquisition via learning sequential sampling strategies. In this chapter, we explore another approach to improving measurement acquisition: optimizing the entire pipeline end-to-end for the downstream task of interest. Traditional CS-MRI methods design the measurement subsampling strategy independently of the downstream task prediction [12, 204, 305]. For example, these methods are optimized for reconstruction accuracy over the entire field-of-view, even when the goal of the MRI scan is, e.g., to inspect a certain kind of tissue or region of interest. This often results in suboptimal performance on the task that ultimately matters to the user. To address this limitation, we propose Tackle, a unified co-design framework that jointly optimizes subsampling, reconstruction, and prediction strategies to maximize downstream task performance. The naïve approach of simply appending a task prediction module and training with a task-specific loss leads to suboptimal downstream performance. Instead, we develop a training procedure where a backbone architecture is first trained for a generic pre-training task (image reconstruction in our case), and then finetuned for different downstream tasks with a prediction head. Experimental results on multiple public MRI datasets show that Tackle achieves an improved performance on various tasks over traditional CS-MRI methods. We also demonstrate that Tackle is robust to some distribution shift by showing that it generalizes to a new dataset we experimentally collected using different acquisition setups from the training data. Without additional fine-tuning, Tackle leads to both numerical and visual improvements compared to existing baselines. We have further implemented a learned 4×-accelerated sequence on a Siemens 3T MRI Skyra scanner. Compared to the fully-sampling scan that takes 335 seconds, our optimized sequence only takes 84 seconds, achieving a four-fold scan time reduction as desired, while maintaining high performance. This chapter is based on our work [323], published in the IEEE Transactions on Computational Imaging, vol. 10, pp. 1040-1054, 2024, doi: 10.1109/TCI.2024.3410521. The appendix for this chapter is Appendix B. The code for the work presented in this chapter is available at https://github.com/zihuiwu/TACKLE. 26 4.1 Introduction In the existing co-design literature for CS-MRI, task prediction is often viewed as a post-processing step decoupled from image reconstruction. Existing methods focus predominantly on image reconstruction and rely on standard image similarity metrics such as mean square error (MSE) or peak signal-to-noise ratio (PSNR) as a proxy for performance on a downstream task. Such a reconstruction-oriented formulation lacks a direct connection with the downstream tasks that reflect actual clinical needs [219]. We are thus motivated to ask: Can one improve the accuracy of downstream task prediction by optimizing the entire CS-MRI pipeline in an end-to-end fashion? With end-to-end co-design methods, it seems like we are only one step away from incorporating downstream tasks as part of the optimization. Namely, one can simply append a task prediction module and add a task-specific loss. However, as shown in Figure 4.1 and Table 4.2, this approach leads to a suboptimal performance on the task prediction and is sometimes even worse than the traditional approach of separate reconstruction and task prediction. These results indicate that it remains a challenge to learn task-specific strategies robustly for CS-MRI. In this chapter, we address this challenge by proposing a unified framework, taskspecific codesign of 𝒌-space subsampling and prediction (Tackle), for designing task-specific CS-MRI systems. Different from existing works that focus on specific tasks, Tackle is a general framework that accommodates different downstream tasks. To do so, we design a two-step training strategy that mimics the training of modern language and vision models. Tackle is first trained for a generic task of image reconstruction, and then fine-tuned for specific downstream tasks. We find that this approach can effectively learn generalizable task-specific strategies that lead to significant and consistent improvements, with an example shown in Figure 4.1 (c). Besides the standard task of reconstructing the full field-of-view (which we call full-FOV reconstruction hereafter), we demonstrate Tackle on three other tasks covering both pixel-level and image-level imaging problems: region-of-interest (ROI) oriented reconstruction, tissue segmentation, and pathology classification. Our experimental results show that end-to-end optimization for task prediction sometimes circumvents the typical reconstruction in terms of pixel-wise accuracy, but leads to improved accuracy on the task of interest by effectively extracting key visual information for task prediction. 27 The main contributions of this chapter are as follows: • We provide a general framework (Tackle) that learns specific strategies for a variety of CS-MRI tasks. Tackle optimizes the entire CS-MRI pipeline, from measurement acquisition to label prediction, in an end-to-end fashion directly for a user-defined task. • We validate Tackle on multiple MRI datasets, covering different body parts, scanning sequences, and hardware setups. Experimental results show that Tackle outperforms the reconstruction-oriented baseline methods on all considered settings. We evaluate the proposed end-to-end architecture and training procedure through ablation studies. Our results offer guidance for designing effective task-specific CS-MRI systems in the future. • We show the generalization of Tackle to out-of-distribution data by deploying it to a dataset we experimentally acquired using a different acquisition sequence from that of the training data. We further implement a learned 4×-accelerated sequence on a Siemens 3T MRI Skyra scanner. The sequence shortens the scan time from 335 seconds to 84 seconds, a four-fold time reduction as desired, while maintaining high performance. These experiments highlight the realworld practicality of our method. 4.2 4.2.1 Related Work Reconstruction-Oriented Co-Design The success of DL methods in CS-MRI reconstruction motivates the idea of jointly optimizing acquisition together with reconstruction via end-to-end training. Recently, there has been a rapidly growing literature on optimizing a parameterized sampling strategy jointly with a CNN reconstructor [3, 8, 12, 216, 229, 237, 299, 300, 302, 307, 327, 330, 344, 351, 353, 362]. These methods have different architectural designs and applicable scenarios, but all rely on the differentiable nature of neural networks to optimize the reconstruction accuracy over the choice of 𝑘-space measurements. The learned subsampling pattern and reconstruction network are thus specific to the dataset. The end-to-end training enables synergistic cooperation between the learned subsampling pattern and reconstructor, achieving state-of-the-art reconstruction performance. From a task perspective, however, having a reconstruction is not the end of the workflow. These methods rely on either human evaluation, a traditional task prediction algorithm, or a CNN for task predictions, which are out of the scope of these papers. 28 4.2.2 Task-Oriented Co-Design Recent work has investigated the co-design idea in the context of limited tasks beyond full-FOV reconstruction, such as physical parameter estimation [325, 353, 363] and segmentation [98, 271, 308, 309, 311]. Using task-specific loss functions in their training procedures, these proposed methods demonstrate stronger task performance than methods trained by a reconstruction-only loss. Most of these proposed approaches leave either subsampling or prediction as a pre-determined fixed module, and focus on co-designing the other modules [98, 271, 325, 353, 363]. On the other hand, the authors of [309, 311] proposed to jointly optimize all three steps and investigated a brain segmentation task using a U-Net reconstructor and predictor. Although these methods show the potential of extending co-design beyond reconstruction, they are each fine-tuned for one particular task, do not easily accommodate different types of data (e.g., multi-coil), and have not been demonstrated on real out-of-distribution datasets. The most relevant work to ours in the literature is a concurrent work by Wang et al. [308], in which the authors presented a thorough investigation of optimizing the entire CS-MRI pipeline for various segmentation problems. In this work, we cast a wider net for the taskspecific CS-MRI co-design problem. In particular, we demonstrate our unified framework for designing generalized CS-MRI pipelines, Tackle, on three different tasks beyond full FOV reconstruction. Tackle performs robustly on this broad range of tasks and experiments, and is implemented and tested on a Siemens scanner. 4.3 Method Figure 4.2 illustrates the architecture of Tackle. As a co-design CS-MRI method, Tackle jointly optimizes the sampler, retriever, and predictor for a task-dependent loss. In the following subsections, we describe each module in order, and more implementation details can be found in Appendix B.1.2. 4.3.1 Sampler We consider 2D Cartesian subsampling patterns, i.e., 𝒎 ∈ {0, 1}𝑛 . Similar to the previous chapter, we model the subsampling strategy as the element-wise Bernoulli distribution with a probability vector 𝒑 ∈ [0, 1] 𝑛 , i.e., 𝒎𝑖 ∼ Bernoulli( 𝒑𝑖 ), following [12, 327, 335]. To learn the optimal sampling probabilities, we directly optimize a set of logit parameters 𝒒𝑖 that first give us a set of probabilities e 𝒑𝑖 := Sigmoid(𝒒𝑖 ). We then rescale e 𝒑 to obtain a probabilistic sampling mask 𝒑 that would result in 𝑏 29 measurements in expectation via Bernoulli sampling:   𝛼 e  𝒑 𝒑= 𝛽   1 − 1−𝛼 𝒑) 1−𝛽 (1 − e  if 𝛽 ≥ 𝛼 otherwise where 𝛼 := 𝑏𝑛 , 𝛽 := ∥ e𝒑𝑛∥ 1 , and 1 is the all-one vector. During training, the sampler draws a 𝑘-space sampling mask 𝒎 by sampling 𝒎𝑖 ∼ Bernoulli( 𝒑𝑖 ). We repeatedly sample 𝒎 until ∥𝒎∥ 1 ≈ 𝑏 under a small tolerance. This sampling process encourages exploration of different patterns and ensures that the sampling patterns approximately satisfy the budget constraint. Since the sampling process is not differentiable, we use the same straight-through estimator technique as in the previous chapter to overcome the non-differentiability [20]. During testing, we set the top 𝑏 indices of 𝒑 with the highest probabilities to 1 (to sample) and others to 0 (not to sample). This binarization guarantees that the sampling mask strictly satisfies the sampling budget constraint and all slices of a volume share the same sampling mask. We also allocate 1/8 of the sampling budget for the low-frequency region around the DC component, which we refer to as the pre-select region. The pre-selected measurements provide auto-calibration signals (ACS) for multi-coil reconstruction and stabilize the training of some baselines. Therefore, we include the pre-select region for all experiments for consistency. More discussion on this can be found in Appendix B.1.3. We denote the sampler as S𝒒 where 𝒒 is the vector of learnable parameters. 4.3.2 Retriever After acquiring measurements, we employ a retriever to extract visual information from noisy and subsampled 𝑘-space measurements. We note that we name the module “retriever” instead of “reconstructor” because it is jointly optimized with the downstream predictor for non-reconstruction tasks. Hence, the retriever should not be interpreted as a reconstructor as its output may not be a typical “reconstruction” in terms of pixel-wise accuracy. We denote the retriever as R𝜽 where 𝜽 is its weights. We select the E2E-VarNet [267] since it is a model-based DU architecture that combines the MRI forward model and deep learning architectures, and achieves excellent performance on CS-MRI reconstruction [338]. E2E-VarNet also accommodates multi-coil 𝑘-space data with its ability to estimate coil sensitivity maps. Specifically, our E2E-VarNet retriever operates in 𝑘-space and consists of 12 refinement steps, each of which includes a U-Net [250] with independent weights from each other. For each U-Net, we use the standard architecture with the follow- 30 ing parameters: 2 input and output channels, 18 channels after the first convolution filter, 4 average down-pooling layers, and 4 up-pooling layers. The final output layer of the retriever is an inverse Fourier transform followed by a root-sum-squares reduction for each pixel over all coils. The output of the retriever is a batch of magnitude images. For reconstruction tasks, a loss function will be directly applied to the output. For non-reconstruction tasks, the output will be fed into an additional predictor module described in the next section. 4.3.3 Task-Specific Design: Predictor and Loss Function We demonstrate Tackle on three tasks that together represent a gradual progression from generic full-FOV reconstruction to clinically relevant tasks. 4.3.3.1 ROI-Oriented Reconstruction For many MRI scans, only a small region of the FOV is relevant to the reader, so we define a task where we aim to maximize reconstruction quality around that region. In contrast to the full-FOV reconstruction task, the reconstruction accuracy in this task is only measured over the region of interest (ROI) of each image instead of the entire FOV. We hereafter refer to this task as ROI-oriented reconstruction. This task is a first step from generic full-FOV reconstruction to more specific downstream tasks in CS-MRI. There is no predictor for this reconstruction task, and the output of the retriever will directly be used for evaluation. The evaluation metric we use is the local peak signal-to-noise ratio (PSNR), which is the PSNR within the ROI of an underlying image 𝒙. Let R 𝒙 be the set of indices 𝑖 that are within the ROI of an image 𝒙. Note that R 𝒙 varies from one image 𝒙 to another. We define the local PSNR within the ROI as LocalPSNR(b 𝒙 , 𝒙; R 𝒙 ) := 10 log10 max(𝒙) 2 LocalMSE(b 𝒙 , 𝒙; R 𝒙 ) (4.1) Í where LocalMSE(b 𝒙 , 𝒙; R 𝒙 ) := |R1𝒙 | 𝑖∈R 𝒙 (b 𝒙𝑖 − 𝒙𝑖 ) 2 and max(𝒙) is the largest pixel value of 𝒙. We optimize our model for the local reconstruction quality using LROI (b 𝒙 , 𝒙0 ) := −LocalPSNR(b 𝒙 , 𝒙 0 ; R 𝒙0 ) as the training loss where 𝒙0 is the ground truth image. 31 4.3.3.2 Tissue Segmentation For this task, we aim to predict segmentation maps of different body tissues. Accurately segmenting a tissue from the rest of the organ provides important anatomical and pathological information [128, 238, 254]. Conventional segmentation workflow involves human evaluation and traditional algorithms, which often require standard reconstructions of certain contrasts as input [105]. On the contrary, Tackle does not require reconstruction as a necessary intermediate step, and is optimized for segmentation performance in an end-to-end fashion. We include an additional predictor P𝝓 with weights 𝝓 after the retriever. We choose the U-Net architecture due to its ability to solve medical image analysis tasks [14, 116, 250]. The specific parameters are: 1 input channel, 𝑐 output channels (where 𝑐 is the number of segmentation classes), 64 channels after the first convolution filter, 4 average down-pooling layers, and 4 up-pooling layers. We use the Dice score [88, 222, 364] as the evaluation metric. The Dice score measures the degree of overlap between two segmentation maps and takes a value between 0 (no overlap) and 1 (perfect overlap). During training, we employ the Dice loss Lseg. (b 𝒔, 𝒔0 ) := 1 − DiceScore(b 𝒔, 𝒔0 ) where 𝒔0 is the ground truth segmentation map. For both training and evaluation, we apply a Softmax function across all the classes for each pixel and then calculate the Dice loss/score. During the evaluation, we apply an additional binarization step where we set the class with the highest value after Softmax as 1 and others as 0. In this way, we assign each pixel of the predicted segmentation map b 𝒔 to exactly one class. 4.3.3.3 Pathology Classification The third task we consider is to determine whether a potential pathology exists in an MRI image, such as a suspected tumor. Using algorithms to automatically analyze MRI scans could lead to improved diagnosis accuracy in clinical practice [219]. We formulate this task as a binary image classification problem, where the negative class means the underlying image 𝒙 does not contain any pathology lesion, and the positive class means it does contain a lesion. Through this proof-ofconcept classification task, we go beyond pixel-level problems and show the benefit of task-specific co-design for solving an image-level problem. Similar to the segmentation task, we include an additional predictor in the pipeline, which we also denote as P𝝓 to simplify notations. Specifically, we choose the ResNet 32 [126], which is an established architecture for computer vision tasks, especially image classification. We use the standard ResNet18 architecture except for using 1 input channel and 2 output dimensions. We use the binary cross entropy (BCE) as the loss function for this classification task, Lclass. (b 𝒄, 𝒄 0 ) := BCE(b 𝒄, 𝒄 0 ) where 𝒄 0 is the ground truth classification label. For evaluation metrics, we consider both the classification accuracy TP+TN 2TP (ClsAcc := TP+TN+FP+FN ) and the 𝐹1 score (F1 score := 2TP+FP+FN ) where TP, TN, FP, and FN are the number of True Positive, True Negative, False Positive, and False Negative, respectively. The classification accuracy is more interpretable, while the 𝐹1 score is more robust to class imbalance. So we include both metrics for a more comprehensive evaluation. 4.3.4 Training Procedure We summarize the training objective for each task as follows: • ROI-oriented reconstruction:

min LROI R𝜽 S𝒒 ⊙ 𝒌 , 𝒙0 𝒒,𝜽 • Segmentation: min Lseg. P𝝓 R𝜽 S𝒒 ⊙ 𝒌

, 𝒔0

, 𝒄0 𝒒,𝜽,𝝓
• Classification: min Lclass. P𝝓 R𝜽 S𝒒 ⊙ 𝒌 𝒒,𝜽,𝝓
where 𝒌 ∈ C𝑛 contains all 𝑘-space measurements of 𝒙 and ⊙ denotes element-wise multiplication. When performing end-to-end training over multiple stages, we empirically observed that a model trained from scratch tends to run into either optimization (hard to train) or generalization (unable to generalize) issues. Some prior works address these problems using a hybrid of reconstruction and task-dependent loss [271, 309, 311, 325, 363]. This approach requires tuning a weight parameter that balances the two losses. We adopt an alternative approach that avoids tuning this additional parameter. Specifically, we first train the sampler and retriever jointly with a full-FOV PSNR loss until convergence:

min LFOV R𝜽 S𝒒 ⊙ 𝒌 , 𝒙 0 𝒒,𝜽 33 Task Training procedure and loss Notation example (ROI) recon. S&R: PSNR loss S&R: local PSNR loss (w/ pre-training) LP+UNFOV TackleROI Tissue seg. S&R: PSNR loss → P: Dice loss S&R&P: Dice loss (w/ pre-training) PD+UNrecon. Tackleseg. S&R: PSNR loss → P: BCE loss LOUPErecon. S&R&P: BCE loss (w/ pre-training) Tackleclass. S: sampler, R: retriever, P: predictor &: joint training in an end-to-end fashion →: separate training stages for reconstruction and prediction Patho. class. where LFOV (b 𝒙 , 𝒙 0 ) := −PSNR(b 𝒙 , 𝒙 0 ). We refer to this as the pre-training step in later sections. With the weights learned for the sampler and retriever, we then add the predictor (initialized with random weights) into the framework and fine-tune all three components. We find that the pre-training step allows the model to better learn task-specific strategies, as demonstrated by an ablation study in Section 4.6.2. This training procedure mimics the training of foundation models in state-of-theart language and vision models, which are first pre-trained on a general task and then fine-tuned for more specific tasks. Similar procedures can be found in other task-specific co-design papers, such as [98, 271]. 4.4 Experiments on Large-Scale Datasets We first demonstrate the effectiveness of our framework on the three considered tasks using large-scale datasets. We categorize all the investigated datasets and settings in the bottom left panel of Figure 4.2. For each task, we demonstrate that the proposed task-specific co-design framework achieves better performance than baselines that separately design reconstruction and prediction. We abbreviate different variants of the proposed method and baselines in the following way, based on their task and training procedure: To clarify, the subscript “recon.” for the segmentation and classification methods means that the sampler and retriever are trained for full-FOV reconstruction, and a predictor is subsequently trained for the downstream task with the sampler and retriever fixed. This is equivalent to training a predictor with the reconstructed images by these methods as input for the downstream task. 34 Table 4.1: Comparison of average test local peak signal-to-noise ratio (Local PSNR) in decibel (dB) within Meniscus Tear ROIs under different acceleration ratios (𝑅). 4.4.1 Data 𝑅 LP+UNFOV PD+UNFOV LOUPEFOV TackleROI Single-coil 8 16 26.95 25.16 28.23 26.05 30.32 27.32 34.04 31.54 Multi-coil 8 16 27.55 26.02 32.68 30.00 34.88 31.79 40.65 37.89 ROI-oriented Reconstruction Dataset and Setup For the ROI-oriented reconstruction task, we use the images and raw single- and multi-coil 𝑘-space data from the fastMRI+ knee dataset [338, 354], which contains bounding box annotations for knee pathologies. Specifically, we investigate the most common knee pathology in the dataset called “Meniscus Tear” (MT). Each image 𝒙0 in the dataset contains at least one rectangular bounding box annotation R 𝒙0 , which is drawn to include all the pathology but exclude the normal surrounding anatomy [354]. Therefore, the local image quality within each bounding box (i.e., ROI) is more indicative of the quality for pathology assessment than a metric over the entire FOV. We emphasize that the location of the bounding box R 𝒙0 varies sample by sample and is never an input to any method during inference. R 𝒙0 is only used for calculating the training loss and evaluating the local PSNR during test time according to Equation (4.1). Hence, the local PSNR performance reflects the quality of reconstructions by different methods for assessing the considered pathological lesions in the ROIs. Baselines We compare TackleROI with three full-FOV reconstruction-oriented baselines. • LOUPEFOV : Proposed in [12], LOUPEFOV jointly optimizes a sampler and a residual U-Net reconstructor. • Low-pass + U-NetFOV (LP+UNFOV ): substitute the sampler in LOUPEFOV with a fixed low-pass filter sampling pattern. • Poisson-disc + U-NetFOV (PD+UNFOV ): substitute the sampler in LOUPEFOV with a Poisson-disc sampling pattern drawn from a variable density distribution 35 and generated with the sigpy.mri.poisson function in the SigPy package1 . Results We compare the average local PSNR of our method and other baselines over the test set in Table 4.1. For all settings, Tackle outperforms other baselines designed for full-FOV reconstruction by at least 3 dB, indicating a significant improvement of image quality within the ROI. In Figure 4.3, we provide example reconstructions by our method and three baseline methods. For each reconstruction, a zoom-in on its ROI is provided at the bottom with the corresponding local PSNR value labeled above in orange, and its full-FOV PSNR is labeled on the top right corner. As shown in the ground truth of the MT example, a meniscus tear is indicated by a streak (dark in the top row and bright in the bottom row) that is present on the meniscus (bright in the top row and dark in the bottom row), as indicated by the red pointers. To accurately detect the existence and assess the severity of a meniscus tear, a reconstruction should clearly show the boundaries of the meniscus and the details of the tear. However, the ROIs of both LP+UNFOV and LOUPEFOV reconstructions contain significant reconstruction artifacts that disguise the tear (see the red arrows). On the other hand, TackleROI preserves the details of the tear and contains fewer artifacts than the baselines, providing a more accurate ROI reconstruction with a higher diagnostic value. In Appendix B.3 we also include a validation of TackleROI on images that either are healthy or contain pathologies other than the meniscus tear. Although TackleROI is not designed to generalize across different pathologies, we empirically find that TackleROI still yields high-fidelity reconstructions for out-of-distribution images so that the pathologies on these images remain detectable. This generalization of TackleROI is consistent across the three acceleration ratios (4×, 8×, and 16×) for this fastMRI+ dataset. Discussion Enhancing local ROIs for MRI may seem counterintuitive because the acquisition happens in 𝑘-space; each frequency measurement in theory corresponds to the entire FOV. Here, we understand the feasibility via a PSF analysis. Consider the zero-filled reconstruction e 𝒙 from some (noiseless) single-coil 𝑘-space data:   e 𝒙 := 𝑭 −1 (𝒎 ⊙ (𝑭𝒙)) = 𝑭 −1 𝒎 ∗ 𝒙 1 https://github.com/mikgroup/sigpy (BSD-3-Clause license) 36 Table 4.2: Comparison of average test Dice score on the SKM-TEA dataset [83] for segmenting four knee tissues under different acceleration ratios (𝑅). 𝑅 PD+UNrecon. LOUPErecon. 16 64 0.7843 0.7486 0.7888 0.6715 SemuNet Tacklerecon. 0.8108 0.7741 0.8232 0.8145 Tackleseg. 0.8532 0.8357 where ∗ denotes convolution and the second equality holds due to the Fourier convolution theorem. Here, 𝑭 −1 𝒎 is the PSF of the subsampling mask 𝒎 and determines the resolution of the CS-MRI system. We visualize the PSF of a sampling mask trained for full-FOV reconstruction and another trained for MT ROIs reconstruction with the same sampling budget in Figure 4.4. We plot the PSF profiles in the vertical direction around the main lobes. The PSF learned for MT ROIs reconstruction has around 40% improvement in vertical resolution in terms of full width at half maximum (FWHM) of the PSF profiles. Since MT ROIs contain the thin horizontal anatomy of the meniscus, it makes sense that the learned subsampling pattern has a narrower PSF profile (and thus higher resolution) in the vertical direction. This comparison demonstrates that the improvement on ROIs is partly due to the capability of our model to optimize the subsampling PSF for local ROI anatomy via co-design. This is particularly beneficial when there is a mismatch between the optimal subsampling PSF for full-FOV reconstruction and that for ROI reconstruction due to directional anatomical structure, which is the case for MT ROI reconstruction. 4.4.2 Knee Tissue Segmentation Dataset and Setup We then show the performance of Tackle on solving a task that involves segmenting four types of knee tissues: the patellar cartilage, the femoral cartilage, the tibial cartilage, and the meniscus. We use the Stanford Knee MRI with Multi-Task Evaluation (SKM-TEA) dataset [83], which contains pixel-level segmentation maps of the four tissues. Specifically, we use the raw 3D multi-coil 𝑘-space measurements of knee images and take a 1D inverse Fourier transform along the left-to-right direction to obtain the 2D 𝑘-space of sagittal slices. We train each method to minimize the Dice loss until convergence and select the model with the highest Dice score on the validation set. Baselines We compare Tackleseg. with four baselines. • LOUPErecon. : LOUPErecon. is a baseline based on LOUPEFOV . We first train 37 a LOUPEFOV model for the full-FOV reconstruction task and then use the reconstructed images to separately train a segmentation network. • Poisson-disc + U-Netrecon. (PD+UNrecon. ): same as LOUPErecon. except that the sampler is fixed to be a Poisson-disc sampling mask. • Tacklerecon. : same as LOUPErecon. except for using the proposed architecture of Tackle. • SemuNet: Proposed in [309], SemuNet uses a hybrid of ℓ1 reconstruction loss and cross-entropy segmentation loss. Results We provide a quantitative comparison in Table 4.2 and a box-plot comparison in Figure 4.5. Within the rectangle between each pair of methods in Figure 4.5, the top number is the percentage of samples that get improved, and the bottom number is the 𝑝-value given by the paired samples 𝑡-test. With an improved architecture, Tacklerecon. already outperforms the other baselines. Nevertheless, the segmentation-oriented method Tackleseg. achieves even better performance on both 16× and 64× accelerations. Tackleseg. also significantly outperforms SemuNet on both acceleration ratios and has a much smaller performance drop from 16× to 64× than SemuNet, indicating that the proposed approach is more robust to high acceleration ratios. We further provide some visual examples in Figure 4.6. The first row visualizes the input of the predictor by different methods, where each image is labelled by its PSNR value on the top right corner. The last row shows the predicted segmentation maps by different methods, where each prediction is labelled by its Dice score on the top right corner. The blue arrows point out the locations where Tackleseg. provides more accurate reconstructions than other reconstructionoriented baselines. We also provide a zoom-in on the region that contains the segmented tissues in the second row. Discussion We note that Tackleseg. learns an intermediate feature map as the input to the predictor, which circumvents a typically “good” reconstruction. It is interesting how the retriever produces an image where different knee tissues to be segmented have distinctive textures, which are easy to distinguish both from the background and from each other. Even though this feature map is not a typical “reconstruction” in terms of pixel-wise accuracy, it still accurately localizes the anatomy of the tissues to be segmented. We highlight that Tacklerecon. provides a high-fidelity reconstruction of the entire FOV with a PSNR of 33.00 dB, which 38 Table 4.3: Comparison of average test Dice score on the brain segmentation task under different acceleration ratios (𝑅). 𝑅 PD+UNrecon. LOUPErecon. 16 64 0.8952 0.8377 0.9244 0.8733 SemuNet Tacklerecon. 0.9196 0.3824 0.9350 0.9181 Tackleseg. 0.9395 0.9218 demonstrates that our model is well capable of doing the full-FOV reconstruction task accurately. However, Tackleseg. still outperforms Tacklerecon. in terms of segmentation performance in Figure 4.6 and on average over the dataset in Table 4.6 (see Section 4.6.1 for more details). This observation demonstrates that finding the most accurate full-FOV reconstruction does not necessarily lead to the optimal result on the considered segmentation task. 4.4.3 Brain Tissue Segmentation Dataset and Setup We demonstrate Tackle on another task that involves segmenting four brain tissues: the cortex, the white matter, the subcortical gray matter, and the cerebrospinal fluid (CSF). Following [135], we use the 109th coronal slice of each full 𝑘-space sampled volume in the OASIS dataset [213] and the segmentation maps generated with SAMSEG in FreeSurfer [105]. SAMSEG, which stands for Sequence Adaptive Multimodal SEGmentation, is an established method for brain tissue segmentation and is considered a standard method for this task [234]. We use the segmentation maps generated by SAMSEG as the supervised labels for training. We use the same measurement simulation procedure as in the tumor classification experiments. We simulate the single-coil 𝑘-space data for each image by taking the Fourier transform of the image and adding complex additive white Gaussian noise (AWGN), according to the forward model in Equation (1). The standard deviation of the noise for each image is 0.05% of the magnitude of the DC component. We train each method to minimize the Dice loss until convergence and select the model with the highest Dice score on the validation set. Baselines We compare Tackleseg. with the same baselines as the ones in Section 4.4.2: LOUPErecon. , PD+UNrecon. , Tacklerecon. , and SemuNet. Results We first provide a numerical comparison in Table 4.3 and a box-plot comparison in Figure 4.7. Within the rectangle between each pair of methods in Figure 4.7, the top number is the percentage of improved samples, and the bottom 39 number is the 𝑝-value given by the paired samples 𝑡-test. With an improved architecture, Tacklerecon. significantly outperforms the other reconstruction-oriented baselines. Nevertheless, Tackleseg. still outperforms Tacklerecon. under both accelerations with significant 𝑝-values, highlighting the benefit of task-specific training. Compared to SemuNet [309], Tackleseg. learns better segmentation strategies for both acceleration ratios and is more robust to high acceleration. We further provide some visual examples in Figure 4.8, visualizing the input and output of the predictor across different methods. The zoom-in regions highlight a location where the segmentation prediction of Tackleseg. outperforms other baselines. Specifically, Tackleseg. more accurately predicts the outline of the white matter (in yellow) than other methods. Such an improvement leads to more precise estimation of the thickness of the cortex (in orange), an important task for studying human cognition and neurodegeneration [10]. 4.4.4 Pathology (Tumor) Classification Dataset and Setup In this section, we demonstrate the effectiveness of the proposed method at detecting the existence of gliomas, a common type of brain tumor in adults. We use the images acquired by the FLAIR sequence in the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) dataset [219]. To obtain an image-level label of the existence of a tumor, we aggregate the pixel-level peritumoral edema (ED) segmentation annotations in the BRATS dataset by checking whether there exists any positive pixel in the segmentation map: negative (healthy) means there is no ED pixel, while positive (unhealthy) means there is at least one ED pixel. We simulate the single-coil 𝑘-space data for each image by taking the Fourier transform of the image and adding complex additive white Gaussian noise (AWGN), according to the forward model in Equation (2.1). The standard deviation of the noise for each image is 0.05% of the magnitude of the DC component. We train all models using the BCE loss and evaluate them using the classification accuracy and 𝐹1 score as described in Section 4.3.3.3. Baselines We compare the proposed method Tackleclass. with the first three baselines as in Section 4.4.2 and Section 4.4.3 except that the predictor of each baseline is subsequently trained for pathology classification rather than tissue segmentation (with input images optimized for full-FOV reconstruction). We do not include SemuNet here because it was originally proposed for the segmentation task only. 40 Table 4.4: Comparison of average test accuracy on the pathology classification task under different acceleration ratios (𝑅). Metric 𝑅 PD+UNrecon. LOUPErecon. Tacklerecon. Tackleclass. Cls. acc. 16 64 0.9016 0.8809 0.9024 0.8930 0.9062 0.9054 0.9159 0.9136 𝐹1 score 16 64 0.8853 0.8628 0.8846 0.8768 0.8929 0.8910 0.9039 0.8992 Results In Table 4.4, we compare Tackleclass. with reconstruction-oriented baselines, and find that Tackleclass. achieves higher classification accuracy under both performance metrics. Specifically, Tackleclass. outperforms the existing reconstructionoriented baseline LOUPErecon. by around 2% in the extreme 64× accelerated acquisition scenario. Both variants of Tackle maintain competitive performance under the highly accelerated setting (𝑅=64), while PD+UNrecon. and LOUPErecon. suffer from significant performance degradation. Note that Tackleclass. outperforms Tacklerecon. by more than 0.8% in both cases, despite having the same architecture. We also visualize and compare the classification performance of Tackleclass. and LOUPErecon. under 16× acceleration in Figure 4.9, using confusion matrices. The results show that Tackleclass. has substantially fewer false negatives (bottom left) and a higher overall accuracy compared to LOUPErecon. . 4.5 Validation on an Experimentally Collected Out-of-Distribution Dataset In practice, creating a large, well-annotated training set for a specific task can be very time-consuming or even infeasible. To demonstrate the immediate benefit of our method in a real-world setting, we conduct a validation of Tackle on the ROI-oriented reconstruction task using experimentally collected data that is out of the distribution of the training data. Specifically, we train a Tackle model on a large-scale dataset (fastMRI in this case) and directly test it on raw 𝑘-space data collected by different hardware using a different type of sequence from that of the training. Even without extra fine-tuning or test-time optimization, the learned ROI-specific model provides improved reconstructions on meniscus ROIs. In the following subsections, we present the details of this experiment. Data Acquisition and Processing Two subjects were scanned at the Massachusetts General Hospital in accordance with institutional review board guidelines. Their right knees were scanned by a 3D-encoded Cartesian gradient-echo sequence with a 41 3 Tesla MRI scanner (Model: Skyra; Siemens Healthcare, Erlangen, Germany) and a single-channel extremity coil. To implement the 2D subsampling pattern in the coronal plane, we used a transversal orientation with the frequency encoding direction (𝑘 𝑥 ) pointing into the knee cap (anterior-posterior), so that the two phase encoding directions were left-right (𝑘 𝑦 ) and superior-inferior (𝑘 𝑧 ), respectively. The acquisition parameters were as follows: TE/TR=4.8/9.1ms, FOV=192×192×192mm3 , resolution=1×1×1mm3 , flip angle=10◦ . The total acquisition time of obtaining the fully sampled data for each subject was 5 minutes and 35 seconds. The raw 𝑘-space data has the shape of 192×192×192 (𝑘 𝑥 × 𝑘 𝑦 × 𝑘 𝑧 ). We apply the 1D inverse Fourier transform along 𝑘 𝑥 for downstream processing. Specifically, we take the middle 40 slices of each volume and annotated bounding boxes around the meniscus region using an image labelling tool2 . Efforts were made such that the locations and sizes of the bounding boxes roughly match those in the fastMRI MT dataset. We emphasize that these bounding boxes are only for the purpose of measuring the accuracy of different models on reconstructing the meniscus region. The locations of the annotated ROIs are not the input to any of the tested models. Generalization Gaps There are multiple generalization gaps between the training (fastMRI single-coil data) and the test data: • Different hardware: The acquired data are collected directly with a singlechannel extremity coil, while the training data are simulated from 𝑘-space data collected by multi-channel receive coils [165]. • Different sequence and resolution: The acquired data are given by a gradientecho sequence with 1 mm isotropic resolution, while the training data are given by a spin-echo sequence with 0.5mm in-plane resolution [165]. • Different distribution of the ROI anatomy: The acquired data are collected from two subjects whose menisci are healthy and have no tears, while the ROIs in the training data contain meniscus tears. Despite these generalization gaps, TackleROI works robustly and leads to both numerical and visual improvement. Baselines In this section, we compare TackleROI with the following baselines under 4× acceleration. 2 https://github.com/heartexlabs/labelImg (MIT license) 42 Table 4.5: Comparison of average reconstruction accuracy on the experimentally collected dataset under 4× acceleration (top: full-FOV reconstruction; bottom: ROI-oriented reconstruction). Full-FOV recon. PD+TVFOV LOUPEFOV TackleFOV TackleROI PSNR (dB) 27.94 28.00 28.70 28.18 ROI recon. PD+TVFOV LOUPEFOV TackleFOV TackleROI Local PSNR (dB) 24.45 24.67 25.16 25.72 indicates the variant of Tackle with matching training and evaluation metrics • Poisson-disc + Total VariationFOV (PD+TVFOV ): The subsampling pattern is the same as the Poisson-disc sampling pattern generated by sigpy.mri.poisson for PD+UNFOV in Section 4.4.1. The reconstruction is obtained by solving a total variation (TV) regularized optimization problem with the Sparse MRI toolbox3 . • LOUPEFOV : the same LOUPEFOV baseline as in Section 4.4.1. • TackleFOV : a Tackle model trained for full-FOV reconstruction. • LOUPEROI : the same architecture as LOUPEFOV but trained for ROI reconstruction following the same training procedure as TackleROI . Results We present a quantitative comparison in Table 4.5. For both the full-FOV and ROI-oriented reconstruction tasks, Tackle outperforms the baselines under the corresponding metric. For each task, we highlight the variant of Tackle trained for the evaluation metric in green. Our results show that the highlighted variant outperforms the other variant of Tackle, indicating a tradeoff between full-FOV and ROI reconstruction accuracy. We further conduct a slice-wise PSNR analysis in Figure 4.10. For both histograms, the horizontal axis is the improvement on the respective metric, and the vertical axis is the count. We also quantify the significance of the improvements using the paired samples 𝑡-test. For the full-FOV reconstruction, TackleFOV outperforms LOUPEFOV on all 80 slices, giving a highly significant 𝑝-value of 3.10e-57. We then compare TackleROI with the better full-FOV reconstruction method, TackleFOV , 3 https://people.eecs.berkeley.edu/~mlustig/Software.html (unknown license) 43 on the ROI-oriented reconstruction task. Despite having the same architecture, TackleROI still outperforms TackleFOV on 72.5% of slices, leading to a 𝑝-value of 5.12e-8, which is also statistically significant. This result indicates that the ROI-oriented model TackleROI indeed provides more accurate ROI reconstructions on this out-of-distribution dataset. We further provide some visual examples in Figure 4.11. Below each reconstruction is a zoom-in on the region around the ROI and the error map of the region with respect to the ground truth. Tackle not only achieves higher PSNR values in both cases but also visually recovers the ROIs with fewer artifacts. Implementation Besides the above results based on retrospective subsampling for quantitative comparison, we have also tested the learned sequence on a Siemens 3T MRI Skyra scanner. Specifically, we implement a re-ordering loop that iterates through all the trajectories based on our learned subsampling mask 𝒎. The implemented sequence prospectively subsamples in 𝑘-space and shortens the scan time from 335 seconds to 84 seconds. In Figure 4.12, we compare the reconstruction given by the prospectively subsampling sequence we implement with the reconstruction given by the retrospectively subsampled measurements from the fully sampling sequence. We note that the images labelled as “TackleROI (retrospective)” and “TackleROI (prospective)” are taken by two consecutive but separate scans, so there might be some subtle motion between them. Nevertheless, the two images have no significant visual difference, indicating that the improvement we show on retrospective simulations translates into actual improvement in practice. The prospective reconstruction successfully recovers important anatomical features around the meniscus region while only taking a quarter of the scan time compared to the full-sampled image. 4.6 4.6.1 Ablation Studies Effectiveness of Co-Design We evaluate the effectiveness of two aspects of co-design used in the proposed framework: learnable subsampling and task-specific training. In Table 4.6, we compare four variants of the proposed method that have neither, one, or both aspects of co-design. The meanings of having or not having each aspect are summarized as follows: • Learnable subsampling (column 2) 44 Table 4.6: Ablation studies on two aspects of co-design for all the considered tasks under 16× acceleration. Method Ablated component Learned subsampling ROI-oriented reconstruction (Local PSNR in dB) Task-specific Single-coil training Multi-coil Tissue segmentation (Dice score) Knee Brain PD+VN♭ ✗ (Poisson-disc) ✗ 29.91 36.48 0.8018 PD+VN♯ ✗ (Poisson-disc) ✓ 30.15 36.51 0.8474 Tackle♭ ✓ ✗ 30.14 37.53 0.8232 Tackle♯ ✓ ✓ 31.54 37.89 0.8532 ♭ indicates full-FOV reconstruction oriented versions of PD+VN and Tackle ♯ indicates task-specific versions of PD+VN and Tackle 0.9257 0.9256 0.9350 0.9395 Pathology classification (Cls. acc.) (𝐹1 score) Gliomas tumor 0.9024 0.9072 0.9062 0.9159 0.8871 0.8966 0.8929 0.9039 ✗ (Poisson-disc): use a Poisson-disc subsampling pattern that is randomly generated and then fixed ✓: learn the subsampling pattern from data • Task-specific training (column 3) ✗: separately optimize retriever and predictor ✓: jointly optimize retriever and predictor To eliminate the effect of different network architectures, all four variants have exactly the same architectures. Overall, we find both aspects of co-design are beneficial. For the task of reconstructing meniscus tear ROIs, learning the subsampling pattern is particularly helpful. Task-specific training, on the other hand, is more important for the knee segmentation task. Highlighted in cyan, the last row is the full-fledged version of Tackle, which achieves the best performance for all considered scenarios with both aspects of co-design. 4.6.2 Effectiveness of the Proposed Architecture and Training Procedure The proposed architecture of the mapping from measurements 𝒚 to prediction b 𝒛, which we denote as T𝝍 , consists of an E2E-VarNet retriever and a U-Net predictor. A natural question is how this architecture compares with a single model-free neural network with a comparable number of parameters that directly maps subsampled measurements to the final prediction. We consider the following comparisons in Table 4.7: • Single larger predictor (row 1) – Tissue seg.: U-Net with 128 channels after the first convolution layer and the same number of pooling layers (42.2M parameters) 45 Table 4.7: Ablation studies on model architecture and pre-training for nonreconstruction tasks under 16× acceleration. Ablated component Arch. of T𝝍 Pre-train Tissue segmentation (Dice score) Knee Pathology classification (𝐹1 score) (Cls. acc.) Brain Gliomas tumor Predictor only‡ ✗ 0.7539 0.9005 0.8966 0.8788 ‡ VN+predictor ✗ 0.8163 0.9371 0.9102 0.8969 VN+predictor§ ✓ 0.8532 0.9395 0.9159 0.9039 ‡ U-Net(128) / ResNet(101) for tissue seg. / patho. class. § E2E-VarNet + U-Net(64) / ResNet(18) for tissue seg. / patho. class. Table 4.8: Comparison of average test PSNR (dB) between reconstruction models trained with task-specific masks and Tacklerecon. on the fastMRI knee dataset. Method Task-specific mask+VN Tacklerecon. Brain seg. Knee seg. Tumor class. 16× 64× 16× 64× 16× 64× 38.47 33.04 38.44 33.13 32.53 32.63 30.10 30.24 44.20 44.48 37.07 37.26 – Patho. class.: ResNet101 (42.5M parameters) • VN+predictor (rows 2&3) – Tissue seg.: E2E-VarNet + standard U-Net (29.9M + 10.6M = 40.5M parameters) – Patho. class.: E2E-VarNet + ResNet18 (29.9M + 11.2M = 41.1M parameters) Comparing the first two rows, we find that the proposed “VN+predictor” architecture significantly outperforms the “single larger predictor” baseline on all settings. This is likely due to the model-based nature of the “VN+predictor” architecture, which more effectively extracts useful information from subsampled measurements for downstream tasks. Finally, we include the pre-training step discussed in Section 4.3.4. Highlighted in cyan, the full-fledged version of Tackle in the last row significantly outperforms the ablated baselines on both non-reconstruction tasks, indicating the importance of both the proposed architecture and training procedure. 46 4.6.3 Using Task-specific Sequences for Reconstruction Our optimized task-specific pipeline learns to adjust the image representation from a conventional form to one that is more readily interpretable by the predictor network. This often adds additional textures to the images, making them look different from traditional reconstructions. However, this does not imply there is a significant loss in information that could be used for image reconstruction. Despite being optimized for task-specific objectives, our learned task-specific subsampling patterns can be used retrospectively for generating high-fidelity reconstructions. To show this, we conduct an experiment where we take the learned subsampling patterns of Tackleseg. and Tackleclass. and train an additional reconstruction network for each subsampling pattern. The subsampling pattern is fixed during the training. This experiment mimics the scenario if one wants a traditional reconstruction out of the collected 𝑘-space samples from our task-specific sequences. In Table 4.8, we provide a comparison with Tacklerecon. , which jointly optimizes the subsampling pattern and reconstructor, on the fastMRI knee dataset. One can see that the reconstruction models trained with task-specific masks (row 1) come close to Tacklerecon. (row 2) in terms of reconstruction performance. These results indicate that our taskspecific models do not incur a significant loss of image information but achieve a better trade-off for the downstream task accuracy. It is thus possible to recover better images retrospectively using the 𝑘-space measurements collected by the task-specific sequences. 4.7 Limitations Building on the promising results we have achieved, we acknowledge opportunities for further improvement of our current study. Data Usage Similar to other works on task-specific CS-MRI co-design, our approach requires matched 𝑘-space, image, and annotation labels, which are of limited quantity in the research community. Due to this limitation, two of our experiments (brain segmentation and tumor classification tasks) are conducted with 𝑘-space data simulated from magnitude images. Sequence Implementation Although we have implemented a prospectively subsampling sequence with a learned sampling pattern by TackleROI on a Siemens MRI scanner, it was done using only one type of 3D gradient echo sequence. Other physical constraints affect the deployment of our method for general MRI sequences. 47 For example, in spin-echo sequences, the order of sampling should be considered to mitigate spin-relaxation effects. Controlled Study The evaluation in the current study is based on conventional quantitative metrics and qualitative visual comparisons. The number of volunteers for testing our learned sequences on a Siemens MRI scanner is relatively small. To further assess prospective subsampling, future evaluations should involve controlled studies of image quality with radiologists. 4.8 Conclusion In this chapter, we generalized the objective of CS-MRI co-design to a variety of tasks beyond full-FOV reconstruction. We introduced Tackle as a unified approach for robustly learning task-specific strategies. Through comprehensive experiments, we showed that Tackle outperforms existing DL techniques that separately learn subsampling pattern, reconstruction, and prediction. Additionally, Tackle outperforms naive approaches to co-design that directly learn mappings from measurements to predictions. We found that the optimized strategies sometimes circumvent the typical reconstruction in terms of pixel-wise accuracy, but effectively extract key visual information useful for task prediction. Through ablation studies, we justified multiple design choices about architecture and training procedure, and showed their importance in effectively learning CS-MRI strategies for tasks that go beyond full-FOV reconstruction. We further implemented a learned subsampling sequence and tested it on a Siemens 3T MRI Skyra scanner, which led to a four-fold scan time reduction without sacrificing visual quality. Our study demonstrates the exciting promise of employing end-to-end co-design techniques, suggesting a future where clinical CS-MRI requirements are addressed with enhanced efficiency while maintaining accuracy. 48 (a) Separate reconstruction & task prediction (traditional CS-MRI) Poisson-disc random sampling Recon. loss Dice score (↑): 0.8022 Error map Task loss Separately designed (b) Background: g.t. image Naïve approach to end-to-end task-specific co-design Joint seg.-recon. sampling Dice score (↑): 0.7970 Error map Task loss Task-specific training (c) Proposed end-to-end task-specific co-design (Tackle) Task-specific sampling Pre-trained w/ recon. loss Dice score (↑): 0.8563 Error map Task loss Recon. pre-training + task specific fine-tuning Forward pass types: No gradient back-propagation With gradient back-propagation Figure 4.1: Comparison between (a) traditional CS-MRI, (b) a naïve approach to task-specific CS-MRI, and (c) the proposed Tackle framework. Compared with panel (a) which separately deals with reconstruction and task prediction, panel (b) is a simple extension of co-design methods for solving downstream tasks by adding a learnable mapping from measurements to task predictions. However, this naïve approach leads to a suboptimal performance and can even lead to a worse task prediction accuracy, as shown in the example above. On the other hand, we introduce Tackle for effectively learning task-specific CS-MRI strategies. Tackle is first pre-trained for generic reconstruction, and then all three modules are fine-tuned for a more specific downstream task. We find that this training schedule allows Tackle to robustly learn generalizable task-specific strategies. In the above knee segmentation example, all three approaches are trained with the same architectures for the reconstructor (second module) and predictor (third module). Nevertheless, Tackle significantly outperforms the two baseline approaches. 49 Task-specific co-design of k-space subsampling and prediction (Tackle) End-to-end optimization over the full pipeline Sampler MRI scanner Retriever ( (ROI-oriented) recon. denotes the learnable modules) Annotations / labels on g.t. Reconstruction task ROI (Local) PSNR loss Measurements (E2E-VarNet) ① ② ③ ④ ⑤ ⑥ ①: fastMRI knee (single-coil) ②: fastMRI knee (multi-coil) ③: OASIS ④: SKM-TEA ⑤: BraTS ⑥: Own Different tasks ROI recon. ① ② ⑥ Knee seg. ③ Different body parts Brain seg. ①②④⑥ Knee ③⑤ Brain ⑤ Tumor class. ④ Different sequences Different hardwares FSE ①② ③⑥ GRE ①③⑤⑥ Single-coil DESS ④ ⑤ FLAIR ②④ Multi-coil Classification task Investigated datasets and settings Segmentation task Sampling mask (Meniscus) Segmenter Segmentation map Dice loss Visual information for segmentation (U-Net) Classifier Classification prediction Unhealthy (82%) Visual information for classification BCE loss (ResNet) Unhealthy (100%) (Glioma tumor) (annotations / labels are given by either human experts or established methods) Task-specific architecture and training Figure 4.2: Block diagram of the proposed framework Tackle and a summary of the investigated datasets and settings. Tackle uses a task-specific loss to jointly optimize a sampler, a retriever, and an optional predictor, ranging from scanner-level sampling to human-level diagnosis. A summary of the investigated settings is presented in the bottom left panel. FSE, GRE, DESS, and FLAIR stand for fast spin echo, gradient echo, double-echo steady-state, and fluid-attenuated inversion recovery, respectively. We comprehensively investigate multiple CS-MRI tasks on a variety of common MRI settings with six datasets. LP+UNFOV PD+UNFOV 34.17 dB LOUPEFOV 33.37 dB TackleROI 36.13 dB Ground Truth 36.01 dB Mask Full-FOV PSNR ROI 32.12 dB 31.48 dB 29.69 dB 33.61 dB 29.20 dB 39.95 dB 31.02 dB Local PSNR 30.44 dB Full-FOV PSNR ROIs 23.10 dB 24.92 dB 25.13 dB 32.28 dB Local PSNR Figure 4.3: Visual examples of two Meniscus Tear samples reconstructed by different methods in the 16× acceleration single-coil setting. For each reconstruction, the full-FOV PSNR is labeled in white, and the local PSNR for the ROI is in orange. Note how TackleROI recovers the structure and details of the ROI more accurately than the two baselines, as indicated by the red arrows. The better recovery of TackleROI over the ROI leads to a more accurate diagnosis of the Meniscus Tear. We emphasize that the location of the ROI is not an input to any of these models and is only used for evaluating the accuracy of each method on the region that contains the pathology. 50 PSF of TackleFOV PSF of TackleROI (Meniscus Tear) Vertical PSF profiles TackleFOV TackleROI (MT) ≈40% improvement on vertical resolution in terms of FWHM Figure 4.4: Comparison of a subsampling PSF optimized for full-FOV reconstruction and another optimized for the reconstruction of menicus tear (MT) ROIs. Optimizing for MT ROI reconstruction leads to around 40% improvement on the vertical resolution in terms of the full width at half maximum (FWHM), as shown by the PSF profiles in the bottom panel. This improved vertical resolution leads to a better reconstruction of the meniscus that has horizontal anatomy. SKM-TEA — Knee Tissue Segmentation Dice score (a) Top: percentage of improved samples (↑) Bottom: p-value of the paired t-test (↓) 58.3% 9.17e-8 95% CI PD+UNrecon. (0.7843±0.0046) 91.3% 1.24e-171 LOUPErecon. (0.7888±0.0046) 87.5% 4.95e-131 Tacklerecon. (0.8232±0.0046) 16× Tackleseg. (0.8532±0.0043) Dice score (b) 5.47% 1 95% CI PD+UNrecon. (0.7486±0.0051) 98.8% 0 LOUPErecon. (0.6715±0.0044) 74.8% 2.20e-55 Tacklerecon. (0.8145±0.0048) 64× Tackleseg. (0.8357±0.0043) Figure 4.5: Box plots of the knee tissue segmentation results under 16× (a) and 64× (b). Within the rectangle between each pair of methods, the top number is the percentage of samples that get improved, and the bottom number is the 𝑝-value given by the paired samples 𝑡-test. A higher percentage and a lower 𝑝-value indicate a more significant improvement. We also provide the 95% confidence intervals for all methods below their names. For both acceleration ratios, Tackleseg. outperforms other baselines in terms of all the statistical measures. 51 PD+UNrecon. LOUPErecon. Tacklerecon. Tackleseg. Ground Truth 30.99 33.00 —— PSNR (dB) 0.8332 0.7943 0.8334 0.8933 Dice score Predicted seg. map Zoom-in Predictor input 30.01 Background: ground truth image Figure 4.6: Comparison of segmentation results under 16× acceleration on one sample of the SKM-TEA dataset. We show the input of the predictor in the first row, a zoom-in on the region that contains the tissues to be segmented in the second row, and the output of the predictor in the third row. Note that Tackleseg. circumvents the typical “reconstruction” in terms of pixel-wise similarity with the ground truth image. Instead, it learns a feature map that accurately localizes the anatomy, leading to better segmentation prediction than other baselines both for this sample and on average over the test set (Table 4.2). 52 OASIS — Brain Tissue Segmentation Dice score (a) Top: percentage of improved samples (↑) Bottom: p-value of the paired t-test (↓) 100% 7.95e-57 95% CI PD+UNrecon. (0.8952±0.0033) 100% 8.63e-39 LOUPErecon. (0.9244±0.0027) 16× 86.9% 5.68e-14 Tacklerecon. (0.9350±0.0025) Tackleseg. (0.9395±0.0024) Dice score (b) 64× 100% 3.28e-43 95% CI PD+UNrecon. (0.8377±0.0045) 100% 1.33e-63 LOUPErecon. (0.8733±0.0033) 81.0% 4.14e-11 Tacklerecon. (0.9181±0.0027) Tackleseg. (0.9218±0.0025) Figure 4.7: Box plots of the brain tissue segmentation results under 16× (a) and 64× (b) accelerations. Within the rectangle between each pair of methods, the top number is the percentage of samples that get improved, and the bottom number is the 𝑝-value given by the paired samples 𝑡-test. A higher percentage and a lower 𝑝-value indicate a more significant improvement. We also provide the 95% confidence intervals for all methods below their names. Similar to the knee segmentation results, the proposed method Tackle outperforms other baselines in terms of all the statistical measures for both acceleration ratios. 53 PD+UNrecon. LOUPErecon. Tacklerecon. Tackleseg. Ground Truth 31.82 36.77 —— PSNR (dB) 0.8985 0.9291 0.9365 0.9461 Dice score Predictor input 27.71 Predicted seg. map Test_50 Background: g.t. image Figure 4.8: Comparison of segmentation results under 16× acceleration on one sample from the OASIS dataset. Similar to the knee segmentation results, Tackleseg. circumvents the typical “reconstruction” in terms of pixel-wise similarity with the ground truth image. Instead, it learns an anatomically accurate feature map, which enables better segmentation prediction than other baselines both for this sample and on average over the test set (Table 4.3). The zoom-in panels highlight a region where Tackleseg. more accurately predicts the outline of white matter (in yellow) than other methods. This improvement leads to a more precise estimation of the thickness of the cortex (in orange), an important task for studying human cognition and neurodegeneration [10]. Healthy Unhealthy Healthy True label Ours Precision: 0.9453 Recall: 0.8660 F1-score: 0.9039 Accuracy: 0.9159 Tackleclass. F1 score: 0.9039, Accuracy: 91.59% Unhealthy LOUPE Precision: 0.9629 Recall: 0.8180 F1-score: 0.8846 Accuracy: 0.9024 LOUPErecon. F1 score: 0.8846, Accuracy: 90.24% Healthy Unhealthy Predicted label Healthy Unhealthy Predicted label Figure 4.9: Confusion matrices of the classification results by LOUPErecon. and Tackleclass. . Overall, Tackleclass. achieves greater accuracy in terms of both classification accuracy and 𝐹1 score than LOUPErecon. . Tackleclass. also has a significantly lower number of false negatives (bottom left) compared to LOUPErecon. , which could lead to more patients receiving early treatment. 54 (a) TackleFOV over LOUPEFOV (full-FOV recon.) 95% CI TackleFOV: 28.70±0.14 Count LOUPEFOV: 28.00±0.15 100% slices improved p-value: 3.10e-57 Subject 1 Subject 2 Per-slice PSNR improvement (dB) (b) TackleROI over TackleFOV (ROI-oriented recon.) 95% CI TackleROI: 25.72±0.35 Count TackleFOV: 25.16±0.29 72.5% slices improved p-value: 5.12e-8 Per-slice Local PSNR improvement (dB) Figure 4.10: Per-slice difference histograms. (a): TackleFOV over LOUPEFOV on the full-FOV reconstruction task and (b): TackleROI over LOUPEFOV on the ROIoriented reconstruction task. The 95% confidence intervals are given in the top left corner of each plot. In both cases, the vast majority of slices improve, and the 𝑝-values given by the paired samples 𝑡-test are highly significant. 55 PD+TVFOV LOUPEFOV 28.45 dB TackleFOV 28.61 dB TackleROI 29.17 dB Ground Truth 28.78 dB Full-FOV PSNR Subject 1 Mask ROI 25.25 dB 25.43 dB 27.50 dB 27.11 dB 27.43 dB Local PSNR w/i ROI 29.16 dB 28.35 dB 27.87 dB Full-FOV PSNR Subject 2 ROI 23.66 dB 23.69 dB 26.58 dB 28.09 dB Local PSNR w/i ROI Figure 4.11: Reconstruction comparison of two samples in the experimentally collected dataset (top: from subject 1; bottom: from subject 2) by different methods under 4× acceleration. The sampling mask, a zoom-in on the ROI, and the error map are presented for each method. By sampling more frequencies along the vertical direction in 𝑘-space, TackleROI has a higher vertical resolution in the image space and thus outperforms other baselines optimized for full-FOV reconstruction on the ROIs with directional anatomical structure. TackleROI (retrospective) TackleROI (prospective) Scan time: 84 seconds Scan time: 84 seconds Fully sampled Scan time: 335 seconds Figure 4.12: Reconstruction comparison between the implemented prospective subsampling sequence and the retrospective subsampling sequence. Our learned sequence can be implemented on an MRI scanner and generates images of quality indistinguishable from those recovered from retrospectively sampled data. Compared to the ground truth image, our prospectively subsampled reconstruction recovers important features around the meniscus region, which is the ROI it is trained to enhance. Part II Sampling: Posterior Estimation 56 57 Chapter 5 OVERVIEW AND PRELIMINARIES In this part, we investigate the second topic around sampling in computational imaging—posterior estimation—in the context of Bayesian inverse problems. 5.1 Bayesian Inverse Problems The Bayesian formulation provides a principled framework for solving inverse problems via a probabilistic viewpoint [268]. We refer to this formulation as Bayesian Inverse Problems (BIPs) and formally introduce it in this section. 5.1.1 Basics Recall from Chapter 1 that we consider inverse problems of reconstructing b 𝒙 ≈ 𝒙0 from measurements 𝒚 = A (𝒙0 ) + 𝒏, (5.1) where A is the forward model and 𝒏 is the noise term. Assume that 𝒏 is a random variable with density 𝜋. Then the probability density of observing measurements 𝒚 given 𝒙0 is 𝑝(𝒚 | 𝒙0 ) := 𝜋(𝒚 − A (𝒙 0 )) = 𝜋(𝒏), (5.2) commonly referred to as the data likelihood (or likelihood in short). Let 𝑝(𝒙0 ) denote the prior distribution (or prior in short) of 𝒙 0 , which encodes our knowledge about 𝒙0 before any measurements are acquired. According to Bayes’ theorem, the posterior distribution (or posterior in short) of 𝒙0 given 𝒚 is 𝑝(𝒙0 | 𝒚) = 𝑝(𝒚 | 𝒙0 ) 𝑝(𝒙0 ) 𝑝(𝒙 0 , 𝒚) 𝑝(𝒚 | 𝒙0 ) 𝑝(𝒙 0 ) = =∫ , 𝑝(𝒚) 𝑝(𝒚) 𝑝( 𝒚 | 𝒙 0 ) 𝑝(𝒙0 )d𝒙0 (5.3) where the denominator 𝑝( 𝒚) is the model evidence (or evidence in short)—a normalization constant ensuring that the posterior is a valid probability distribution. The posterior distribution represents our updated belief about 𝒙0 after incorporating the information provided by the measurements 𝒚. Solving BIPs amounts to generating samples from the posterior distribution, i.e., sampling b 𝒙 ∼ 𝑝(𝒙 0 | 𝒚). Despite being a standard Bayesian inference problem theoretically, solving BIPs in computational imaging still faces several challenges: 58 • In imaging applications, the unknown target 𝒙0 often represents a high-resolution image or video. It is common for 𝒙 0 to have thousands, millions, or even higher dimensions. Characterizing a high-dimensional distribution 𝑝(𝒙0 ) of images or videos is hard in the first place. • The posterior distribution involves the normalizing constant (evidence) 𝑝( 𝒚), which is an integration over all candidates 𝒙0 ’s that could potentially lead to the measurements 𝒚. Evaluating this term is almost impossible in real-world problems due to the high dimensionality and lack of a closed form for 𝑝(𝒙0 ). It is a challenge to estimate 𝑝(𝒙0 | 𝒚) without access to 𝑝( 𝒚). • Equation (5.3) only elucidates the connection among prior, posterior, likelihood, and evidence in terms of numerical values. It is non-trivial how to design a sampler that actually provides samples from the posterior distribution, especially for high-dimensional problems. • In many real-world applications, the measurements 𝒚 only provides limited information for 𝒙 0 . Successful recovery of 𝒙0 relies on a sophisticated prior 𝑝(𝒙 0 ) that captures the set of possible solutions. One important challenge in computational imaging is to design and leverage priors with sufficient expressiveness for posterior estimation. 5.1.2 Maximum a Posteriori Estimation Traditional approaches to Bayesian inverse problems circumvent these challenges by finding the maximum a posteriori (MAP) estimator. Instead of sampling the full posterior distribution, the MAP approach aims to maximize the (logarithmic) posterior, i.e., finding b 𝒙 MAP := arg max log 𝑝(𝒙 | 𝒚) = arg max [log 𝑝(𝒚 | 𝒙) + log 𝑝(𝒙)] . 𝒙 𝒙 (5.4) Note that the intractable evidence term is eliminated because it does not depend on 𝒙. Assuming Gaussian noise with zero mean and covariance 𝚺, we have that 1 log 𝑝( 𝒚 | 𝒙) = − ∥A (𝒙) − 𝒚∥ 2𝚺 −1 + 𝐶 2 (5.5) where ∥ · ∥ 𝚺 −1 := ⟨·, 𝚺−1 ·⟩ and 𝐶 is a constant that does not depend on 𝒙. The resulting optimization problem is   1 2 b 𝒙MAP := arg max − ∥A (𝒙) − 𝒚∥ 𝚺 −1 + log 𝑝(𝒙) . (5.6) 𝒙 2 59 This optimization problem can be solved by existing algorithms for relatively simple priors [19, 143, 162]. For more sophisticated priors, the plug-and-play prior (PnP) [291] and regularization by denoising (RED) [248] frameworks provide algorithmic tools for solving Equation (5.6). Despite its clear path toward a solution, the MAP approach has some fundamental limitations: • Instability: Unlike the full posterior, the MAP estimate does not possess Lipschitz continuity with respect to 𝒚 [268]. As a result, small perturbations in the measurements due to the noise could lead to large variations in the solution b 𝒙MAP . This is undesirable from a theoretical perspective. • Lack of uncertainty quantification: Most MAP estimation algorithms only provide a deterministic solution, which cannot represent the full solution space. This problem is particularly prominent for ill-posed problems where the reconstruction comes with significant uncertainty. Even if one can heuristically obtain multiple solutions by introducing randomness or choosing different initializations and hyperparameters, these solutions may be biased and lack a principled interpretation. These limitations motivate the need for methods that can sample from the full posterior distribution, enabling both accurate reconstructions and rigorous uncertainty quantification. 5.2 5.2.1 Diffusion Models Basics Diffusion models (DMs) are a class of generative models that can capture complicated high-dimensional distributions [131, 151, 266]. They have achieved remarkable success across a variety of domains, including natural image synthesis [249], protein structure generation [108], molecular design [203], and robotic trajectory modeling [59]. We adopt the continuous-time formulation of DMs based on stochastic differential equations (SDEs), as introduced by Song et al. [266]. First consider a forward diffusion process that gradually transforms a data distribution 𝒙0 ∼ 𝑝(𝒙0 ) into an approximately Gaussian distribution 𝒙𝑇 ∼ N (0, 𝜎𝑇2 𝑰) where 𝜎𝑇 is the noise level at 60 time 𝑇. Mathematically, the forward process can be characterized by d𝒙𝑡 = 𝑓 (𝒙𝑡 , 𝑡)d𝑡 + 𝑔(𝑡)d𝒘 𝑡 , (5.7) where 𝑓 is the drift coefficient, 𝑔 is the diffusion coefficient, and 𝒘 is the standard Wiener process with time 𝑡 flowing from 0 to 𝑇. A key observation is that this forward SDE has a reverse time SDE with the same marginal distributions [9]. The backward process removes noise and gradually transforms a Gaussian sample back to a clean sample, defined by the reverse-time SDE   1 2 d𝒙𝑡 = 𝑓 (𝒙𝑡 , 𝑡) − 𝑔 (𝑡)∇𝒙𝑡 log 𝑝 𝑡 (𝒙𝑡 ) d𝑡 + 𝑔(𝑡)d𝒘¯ 𝑡 , (5.8) 2 where 𝑝 𝑡 (𝒙𝑡 ) is the probability density of 𝒙𝑡 at time 𝑡 and 𝒘¯ 𝑡 is the reverse-time Wiener process. A neural network is trained to learn the score function ∇𝒙𝑡 log 𝑝 𝑡 (𝒙𝑡 ) for 𝑡 ∈ [0, 𝑇] via denoising score matching [292]. Once trained, we can generate new samples from the learned data distribution by first sampling 𝒙𝑇 ∼ N (0, 𝜎𝑇2 𝑰) and solving Equation (5.8) with numeric solvers. 5.2.2 Latent Diffusion Models Latent diffusion models (LDMs) [249] offer a way to reduce the computational cost otherwise necessary to model the original high-dimensional data distribution. LDMs generate an efficient, low-dimensional latent representation 𝒛0 ∈ R𝑑 of data 𝒙0 ∈ R𝑛 with a pre-trained perceptual compression encoder E and decoder D, which satisfy 𝒛0 = E (𝒙0 ) and D (𝒛0 ) ≈ 𝒙 0 . The compression models E and D can be trained as VAE variants [38, 163, 246] with KL divergence regularization or VQGAN variants [95, 120, 228] with quantization regularization. The generation process solves the reverse-time SDE (5.8) in the latent space, followed by decoding with the decoder D. 5.3 Solving Inverse Problems with Diffusion Models The expressive power of diffusion models for modeling complex, high-dimensional distributions makes them a promising choice for solving Bayesian inverse problems [64, 265]. Prior work has largely focused on two main strategies1 : conditional diffusion models (CDMs) and plug-and-play diffusion priors (PnPDP). 1 Additional formulations include Variational Bayes [101, 103, 214] and Sequential Monte Carlo (SMC) [44, 89, 283, 318]. 61 5.3.1 Conditional Diffusion Models Inspired by classifier-free guidance [132], conditional diffusion models (CDMs) consider the conditional reverse diffusion process   1 2 d𝒙𝑡 = 𝑓 (𝒙𝑡 , 𝑡) − 𝑔 (𝑡)∇𝒙𝑡 log 𝑝 𝑡 (𝒙𝑡 | 𝒚) d𝑡 + 𝑔(𝑡)d𝒘¯ 𝑡 . (5.9) 2 where the conditional score function ∇𝒙𝑡 log 𝑝 𝑡 (𝒙𝑡 |𝒚) can be learned by applying denoising score matching to pairs (𝒙0 , 𝒚). CDMs naturally extend DMs to conditional sampling, but suffer two major limitations: (1) Training a CDM requires the joint distribution of 𝒙0 and 𝒚, which depends on the specific forward model A and noise profile of the inverse problem. As a result, a CDM is learned for a specific inverse problem and does not generalize if the forward model A or the noise distribution changes. One needs to retrain the model for each problem, even when the same prior distribution is considered, making this approach inflexible. (2) If ∇𝒙𝑡 log 𝑝 𝑡 (𝒙𝑡 | 𝒚) is learned directly with a neural network, the measurements 𝒚 will be an input to the network. In general, especially for nonlinear inverse problems, 𝒚 may belong to a totally different space from 𝒙𝑡 . This mismatch poses practical challenges when designing neural networks to learn the conditional score, as the network must infer a complex relationship between measurements and image space. Due to these limitations, CDMs are mainly considered for relatively simpler problems where the spatial structure of 𝒚 aligns with the target image 𝒙0 , such as image inpainting or super-resolution [17, 79, 253]. 5.3.2 Plug-and-Play Diffusion Priors Plug-and-play diffusion priors (PnPDP) extend the philosophy of plug-and-play priors [291] to diffusion models. Rather than training a conditional model, PnPDP relies on pre-trained unconditional diffusion models as stand-alone priors for inverse problems [34, 64, 262, 265, 306, 361]. Instead of solving Equation (5.9) directly, one popular approach is to apply Bayes’ theorem and rewrite the conditional score function as ∇𝒙𝑡 log 𝑝 𝑡 (𝒙𝑡 | 𝒚) = ∇𝒙𝑡 log 𝑝 𝑡 ( 𝒚 | 𝒙𝑡 ) + ∇𝒙𝑡 log 𝑝 𝑡 (𝒙𝑡 ). (5.10) Note that the second term is exactly the score function of the prior term, which can be modeled directly by the pre-trained DM. This decomposition decouples the prior from the likelihood, allowing the same diffusion model to be reused across 62 different inverse problems without re-training. However, the first term is difficult to evaluate as it depends on 𝒙𝑡 , an intermediate noisy variable, instead of 𝒙0 . In fact, the likelihood 𝑝 𝑡 ( 𝒚 | 𝒙𝑡 ) can be expressed as ∫ ∫ 𝑝 𝑡 (𝒚 | 𝒙𝑡 ) = 𝑝 𝑡 (𝒚 | 𝒙𝑡 , 𝒙0 ) 𝑝(𝒙0 | 𝒙𝑡 )d𝒙0 = 𝑝(𝒚 | 𝒙0 ) 𝑝 𝑡 (𝒙0 | 𝒙𝑡 )d𝒙0 , (5.11) which involves integration over all possible clean images 𝒙0 . This integral is analogous to the model evidence term in Equation (5.10), and is similarly intractable in practice. Various approximations have been proposed to overcome the intractability, but they often lead to significant sampling errors even in simple, low-dimensional settings [44, 341]. 5.4 Part Outline In the remainder of this part, we present a series of works that push the boundary of posterior estimation in computational imaging: • In Chapter 6, we introduce a novel PnPDP method for posterior sampling with DMs called PnP-DM. We leverage a principled Markov chain Monte Carlo (MCMC) approach, called Split Gibbs Sampler (SGS), and identify its key connection with the EDM formulation of DMs. This perspective allows us to use pre-trained DMs as image priors without re-training or heuristic approximations. We provide both theoretical guarantees and empirical results, showing that our method achieves more accurate posterior estimates and higher-quality reconstructions across a range of linear and nonlinear inverse problems. • In Chapter 7, we generalize PnP-DM to a unified PnPDP framework for diffusionbased posterior estimation that can accommodate a broader class of inverse problems. This framework gives rise to four instantiations, each targeting a specific challenge beyond the original PnP-DM formulation: leveraging semantic information through text conditioning, scaling to high-dimensional video settings, operating with non-differentiable (black-box) forward models, and performing posterior inference in discrete domains. Despite tackling different problem settings, all four variants share a common alternating-update structure inspired by PnP-DM, and they demonstrate both theoretical convergence properties and strong empirical results across a variety of tasks. • In Chapter 8, we introduce a comprehensive benchmark that systematically evaluates diffusion-based methods for solving inverse problems. Specifically, 63 we consider 14 popular methods and 5 representative inverse problems in various scientific domains. Through extensive comparisons and ablation studies, we highlight key strengths and limitations of existing techniques and identify promising directions for advancing posterior sampling in computational imaging. 64 Chapter 6 PNP-DM: A PRINCIPLED FRAMEWORK FOR POSTERIOR ESTIMATION USING DIFFUSION MODELS In this chapter, we propose a new framework that leverages diffusion models (DMs) for sampling the posterior distribution of an inverse problem. DMs have recently shown outstanding capability in modeling complex image distributions, making them expressive image priors for solving Bayesian inverse problems. However, most existing DM-based methods rely on approximations in the generative process to be generic to different inverse problems, leading to inaccurate sample distributions that deviate from the target posterior defined within the Bayesian framework. To harness the generative power of DMs while avoiding such approximations, we propose a Markov chain Monte Carlo algorithm that performs posterior sampling for general inverse problems by reducing it to sampling the posterior of a Gaussian denoising problem. Crucially, we leverage a general DM formulation as a unified interface that allows for rigorously solving the denoising problem with a range of state-ofthe-art DMs. We demonstrate the effectiveness of the proposed method on seven inverse problems (four linear and three nonlinear), including a real-world black hole imaging problem. Experimental results indicate that our proposed method offers more accurate reconstructions and posterior estimation compared to existing DM-based imaging inverse methods. This chapter is based on our work [320], published in the Proceedings of the 38th Annual Conference on Neural Information Processing Systems 2024 (NeurIPS 2024). The appendix for this chapter is Appendix C. The code for the work presented in this chapter is available at https://github.com/zihuiwu/PnP-DM-public. 6.1 Introduction Diffusion models generate samples from a distribution by reversing a diffusion process from the target distribution to a simple (usually Gaussian) distribution [131, 266]. In particular, it estimates a clean image 𝒙0 from a Gaussian noise image 𝒙𝑇 by successively denoising noisy images, where 𝒙𝑡 ∼ 𝑝 𝑡 is the intermediate noisy image at time 𝑡 ∈ [0, 𝑇]. Reversing diffusion requires one to estimate the time-varying gradient log density (score function) ∇ log 𝑝 𝑡 (𝒙𝑡 ) along the diffusion process, or ∇ log 𝑝 𝑡 (𝒙𝑡 | 𝒚) in the case of sampling the posterior 𝑝(𝒙 | 𝒚). 65 To design generic DM-based inverse problem solvers, most existing methods attempt to approximate the time-varying gradient log density ∇ log 𝑝 𝑡 (𝒙𝑡 | 𝒚) [34, 60, 64, 65, 155, 189, 251, 260, 262, 265, 306, 324, 361]. In particular, they first apply Bayes’ rule to separate the forward operator from an unconditional prior over the intermediate noisy image 𝒙𝑡 : ∇ log 𝑝 𝑡 (𝒙𝑡 | 𝒚) = ∇ log 𝑝 𝑡 (𝒚 | 𝒙𝑡 ) + ∇ log 𝑝 𝑡 (𝒙𝑡 ). (6.1) By instead aiming to evaluate the right-hand side, one can leverage the existing pre-trained DMs for the unconditional term ∇ log 𝑝 𝑡 (𝒙𝑡 ). However, the main challenge in this case is that ∇ log 𝑝 𝑡 ( 𝒚 | 𝒙𝑡 ) is intractable to compute in general, as 𝑝 𝑡 ( 𝒚 | 𝒙𝑡 ) involves an integral over all possible 𝒙 0 ’s that could give rise to 𝒙𝑡 [64]. Various methods have been proposed to circumvent the intractability and can mostly be categorized into two groups. One group of methods explicitly approximate ∇ log 𝑝 𝑡 ( 𝒚 | 𝒙𝑡 ) by making simplifying assumptions [34, 64, 262, 265]. However, even for arguably the finest approximation to date proposed in the recent work [34], it is exact only when the prior distribution 𝑝(𝒙) is Gaussian. For general prior distributions beyond Gaussian, these methods do not sample the true posterior 𝑝(𝒙 | 𝒚). The other group of methods do not make explicit approximations but instead substitute ∇ log 𝑝 𝑡 ( 𝒚 | 𝒙𝑡 ) with empirically designed updates where 𝒚 is treated as a guidance signal [60, 65, 155, 189, 251, 260, 306, 324, 361]. Although these methods may have strong empirical performance, they have deviated from the Bayesian formulation and no longer aim to sample the target posterior. In summary, these existing DM-based inverse methods should be best viewed as guidance methods, where the generative process is guided towards the regions where the measurement 𝒚 is more likely to be observed, not as posterior sampling methods [34]. We also note that some recent work considered combining DMs with Sequential Monte Carlo to ensure asymptotic consistency in posterior sampling [44, 89], but the investigation has been limited to linear imaging inverse problems. Chapter Summary In this chapter, we pursue a different path towards posterior sampling with DM priors by proposing a new Markov chain Monte Carlo (MCMC) algorithm, which we call Plug-and-Play Diffusion Models (PnP-DM). It incorporates DMs in a principled way and circumvents the approximation required when taking the approach in Equation (6.1). The proposed algorithm is based on the Split Gibbs Sampler [296] that alternates between two sampling steps that separately involve the likelihood and prior. While the likelihood step can be tackled with traditional sampling techniques, the prior step involves a Bayesian denoising problem that 66 requires careful design. Importantly, we identify a connection between the Bayesian denoising problem and the unconditional image generation problem under a general formulation of DMs presented in [151] (which is referred to as the EDM formulation hereafter). This connection allows us to perform rigorous posterior sampling for denoising using DMs without approximating the generative process and enables the use of a wide range of pre-trained DMs through the unified EDM formulation. We present an analysis of the non-asymptotic behavior of PnP-DM by establishing a stationarity guarantee in terms of the average Fisher divergence. We further demonstrate the strong empirical performance of PnP-DM by investigating four linear and three nonlinear noisy inverse problems, including a black hole interferometric imaging problem involving real data that is both nonlinear and severely ill-posed. Overall, PnP-DM outperforms existing baseline methods, achieving higher accuracy in posterior estimation. 6.2 Preliminaries Split Gibbs Sampler (SGS) is an MCMC approach developed for Bayesian inference [296]. It is also related to the Proximal Sampler [58, 97, 176, 336] and serves as the backbone for the Generative Plug-and-Play (GPnP) [31] and Diffusion Plug-andPlay (DPnP) [326] frameworks in computational imaging. The goal of SGS is to sample the posterior distribution 𝑝(𝒙 | 𝒚) ∝ 𝑝( 𝒚 | 𝒙) 𝑝(𝒙) = exp(− 𝑓 (𝒙; 𝒚) − 𝑔(𝒙)) (6.2) where 𝑓 (𝒙; 𝒚) := − log 𝑝(𝒚 | 𝒙) and 𝑔(𝒙) := − log 𝑝(𝒙) are the potential functions of the likelihood and prior distribution, respectively. The dual dependence of Equation (6.2) on both the likelihood and prior makes it nontrivial to directly sample from it in general. Instead, SGS leverages the composite structure of the posterior distribution by adopting a variable-splitting strategy and considers sampling an alternative distribution   1 2 (6.3) 𝜋(𝒙, 𝒛) ∝ exp − 𝑓 (𝒛; 𝒚) − 𝑔(𝒙) − 2 ∥𝒙 − 𝒛∥ 2 2𝜂 where 𝒛 ∈ R𝑛 is an augmented variable and 𝜂 > 0 is a hyperparameter that controls the strength of the coupling between 𝒙 and 𝒛. We denote the 𝒙- and 𝒛-marginal ∫ ∫ distributions of Equation (6.3) as 𝜋 𝑋 (𝒙) := 𝜋(𝒙, 𝒛)d𝒛 and 𝜋 𝑍 (𝒙) := 𝜋(𝒙, 𝒛)d𝒙, respectively. As 𝜂 → 0, 𝜋 𝑋 converges to the target posterior 𝑝(𝒙 | 𝒚) in terms of total variation distance [296], so one can obtain approximate samples from the target posterior by sampling Equation (6.3) instead. 67 SGS samples Equation (6.3) via Gibbs sampling. Specifically, SGS starts from an initialization 𝒙 (0) and, for iteration 𝑘 = 0, · · · , 𝐾 − 1, alternates between   (𝑘 ) 1. Likelihood step: sample 𝒛 (𝑘) ∼ 𝜋 𝑍 |𝑋=𝒙 (𝒛) ∝ exp − 𝑓 (𝒛; 𝒚) − 2𝜂1 2 ∥𝒙 (𝑘) − 𝒛∥ 22 2. Prior step: (𝑘 ) sample 𝒙 (𝑘+1) ∼ 𝜋 𝑋 |𝑍=𝒛 (𝒙) ∝ exp   −𝑔(𝒙) − 2𝜂1 2 ∥𝒙 − 𝒛 (𝑘) ∥ 22 . Note that the two conditional distributions separately involve 𝑓 (·; 𝒚) and 𝑔(·). The likelihood and prior are decoupled so that these two steps can be designed in a modular way. A similar variable-splitting strategy is also adopted in optimization methods such as the Half-Quadratic Splitting (HQS) method [114] and the Alternating Direction Method of Multipliers (ADMM) [33, 109]. In fact, SGS can be viewed as a sampling analogue of HQS. SGS is a principled approach to posterior sampling if the two sampling steps are rigorously implemented. Existing Works Related to SGS Several works have designed algorithms for solving imaging inverse problems based on SGS [31, 69, 100, 230, 326]. The key distinction among these methods lies in their approaches to the prior step. For instance, the works [31, 100, 230] applied Langevin-based updates for sampling 𝜋 𝑋 |𝑍=𝒛 such that the prior information is encoded by either traditional regularizers or off-the-shelf image denoisers. The work [69] tackled the prior step by heuristically customizing a diffusion model (i.e., DDPM [131]) for sampling 𝜋 𝑋 |𝑍=𝒛 . A concurrent work [326] improved the implementation by devising two diffusion processes that rigorously solve the prior step. Our method differs from [326] by connecting the prior step to the EDM formulation [151]. This connection allows us to seamlessly integrate state-of-the-art DMs as expressive image priors for Bayesian inference through a unified interface, eliminating the need for additional customization for each model and leading to better empirical performance. We also note the recent work [181] that adopted the optimization-based variable-splitting formulation of HQS and utilized general DMs as image priors. We instead consider the SGS formulation from a Bayesian posterior sampling standpoint. Additionally, while SGS-based methods theoretically accommodate general inverse problems, empirical evidence on real-world nonlinear inverse problems remains scarce in the literature. In this work, we demonstrate our method on three nonlinear inverse problems, including a black hole imaging problem. For a more comprehensive review of related works, see Appendix C.5. 68 Arbitrary initialization iteration 0 ⌘0 iteration 1 ⌘1 ⌘1 ⌘0 <latexit sha1_base64="2aeY3f9RAUtm2+JtZRfkP3HOqZE=">AAAGA3icbdTJbhNBEAbgTshAMFsCRy4WERIny0YouWaPszuJtyRjWT3jtj3xbO7uieNYc+QFuMIbcENceRAegPdgvPwFTlLSjGq+qpJaak1Zoesonc3+npl9NGc8fjL/NPXs+YuXrxYWX5dVEElblOzADWTV4kq4ji9K2tGuqIZScM9yRcXqbAzrlWshlRP4Rd0PRc3jLd9pOjbXCVVNoXl9kI3rC0vZTHYU6ftJbpIssUkU6otzf8xGYEee8LXtcqUuc9lQ1wZcasd2RZwyIyVCbnd4S1wmqc89oWqD0YHj9PtEGulmIJPH1+mR/j8x4J5Sfc9KOj2u2+pubYgP1kTX51Ly/oNFy4unv++CF7nakUFvWjW3IpfLmziVMhuiaXLLGwzM5MXjOB6TBbKIbJBN1AA1iARIEDVBTaIWqEXUBrWJHJBDdAW6IuqAOkQuyCXyQB6RD/KJAlBAFIJCoi6oSyRBkkiBFJEGaaIIFBFdg66JeqAe0Q3ohqgP6hPdgm6HNLY12Bq1rYPWiTZAG0SboE2iLdAW0TZom2gHtEOUB+WJdkG7RHugPaJ90D7RAeiA6BB0SHQEOiI6Bh0TFUAFohPQCdEp6JToDHRGVAQViUqgElEZVCaqgCpEVVCV6Bx0TnQBuvh33Y1EhqtDjn/TZDHm7q7B+0n5Yya3nFk++bS0uj5ZkfPsLXvHPrAcW2GrLM8KrMRs5rIv7Cv7Znw2vhs/jJ/j1tmZycwbNhXGr79S4S+q</latexit> <latexit sha1_base64="LWSFwx3yzVCwvYM5XY4QZw6Aom0=">AAAGA3icbdTJbhNBEAbgTshAMFsCRy4WFhKnyEYouSZ2Fmd3Fi9JxrJ6xm174tnc3ZPYsebIC3CFN+CGuPIgPADvwTj2X+AkJc2o5qsqqaXWlBW6jtLZ7O+Z2UdzxuMn809Tz56/ePlqYfF1RQWRtEXZDtxA1iyuhOv4oqwd7YpaKAX3LFdUrW5hVK9eCamcwD/Vg1DUPd72nZZjc51QzRSaN4a5uLGQyS5lbyN9P8lNkgybRKmxOPfHbAZ25Alf2y5X6iKXDXV9yKV2bFfEKTNSIuR2l7fFRZL63BOqPrw9cJx+n0gz3Qpk8vg6fav/Twy5p9TAs5JOj+uOulsb4YM10fO5lHzwYNHy4unvu+BFrnZkcD2tmluRy2U/TqXMpmiZ3PKGQzN58TiOx2SBLCIbZBM1QU0iARJELVCLqA1qE3VAHSIH5BBdgi6JuqAukQtyiTyQR+SDfKIAFBCFoJCoB+oRSZAkUiBFpEGaKAJFRFegK6Jr0DVRH9QnGoAGRDegmxGNbQ22Rm15UJ6oACoQrYPWiTZAG0SboE2iLdAWURFUJNoGbRPtgHaIdkG7RHugPaJ90D7RAeiA6BB0SFQClYiOQEdEx6BjohPQCdEp6JSoDCoTVUAVoiqoSlQD1YjOQGdE56Dzf9fdTGS0OuT4N00WY+7uGryfVD4u5ZaXlo8+ZVbzkxU5z96yd+wDy7EVtsqKrMTKzGYu+8K+sm/GZ+O78cP4OW6dnZnMvGFTYfz6C1h6L6s=</latexit> <latexit sha1_base64="2aeY3f9RAUtm2+JtZRfkP3HOqZE=">AAAGA3icbdTJbhNBEAbgTshAMFsCRy4WERIny0YouWaPszuJtyRjWT3jtj3xbO7uieNYc+QFuMIbcENceRAegPdgvPwFTlLSjGq+qpJaak1Zoesonc3+npl9NGc8fjL/NPXs+YuXrxYWX5dVEElblOzADWTV4kq4ji9K2tGuqIZScM9yRcXqbAzrlWshlRP4Rd0PRc3jLd9pOjbXCVVNoXl9kI3rC0vZTHYU6ftJbpIssUkU6otzf8xGYEee8LXtcqUuc9lQ1wZcasd2RZwyIyVCbnd4S1wmqc89oWqD0YHj9PtEGulmIJPH1+mR/j8x4J5Sfc9KOj2u2+pubYgP1kTX51Ly/oNFy4unv++CF7nakUFvWjW3IpfLmziVMhuiaXLLGwzM5MXjOB6TBbKIbJBN1AA1iARIEDVBTaIWqEXUBrWJHJBDdAW6IuqAOkQuyCXyQB6RD/KJAlBAFIJCoi6oSyRBkkiBFJEGaaIIFBFdg66JeqAe0Q3ohqgP6hPdgm6HNLY12Bq1rYPWiTZAG0SboE2iLdAW0TZom2gHtEOUB+WJdkG7RHugPaJ90D7RAeiA6BB0SHQEOiI6Bh0TFUAFohPQCdEp6JToDHRGVAQViUqgElEZVCaqgCpEVVCV6Bx0TnQBuvh33Y1EhqtDjn/TZDHm7q7B+0n5Yya3nFk++bS0uj5ZkfPsLXvHPrAcW2GrLM8KrMRs5rIv7Cv7Znw2vhs/jJ/j1tmZycwbNhXGr79S4S+q</latexit> General reverse diffusion under EDM iteration K-1 ⌘K 1 <latexit sha1_base64="LWSFwx3yzVCwvYM5XY4QZw6Aom0=">AAAGA3icbdTJbhNBEAbgTshAMFsCRy4WFhKnyEYouSZ2Fmd3Fi9JxrJ6xm174tnc3ZPYsebIC3CFN+CGuPIgPADvwTj2X+AkJc2o5qsqqaXWlBW6jtLZ7O+Z2UdzxuMn809Tz56/ePlqYfF1RQWRtEXZDtxA1iyuhOv4oqwd7YpaKAX3LFdUrW5hVK9eCamcwD/Vg1DUPd72nZZjc51QzRSaN4a5uLGQyS5lbyN9P8lNkgybRKmxOPfHbAZ25Alf2y5X6iKXDXV9yKV2bFfEKTNSIuR2l7fFRZL63BOqPrw9cJx+n0gz3Qpk8vg6fav/Twy5p9TAs5JOj+uOulsb4YM10fO5lHzwYNHy4unvu+BFrnZkcD2tmluRy2U/TqXMpmiZ3PKGQzN58TiOx2SBLCIbZBM1QU0iARJELVCLqA1qE3VAHSIH5BBdgi6JuqAukQtyiTyQR+SDfKIAFBCFoJCoB+oRSZAkUiBFpEGaKAJFRFegK6Jr0DVRH9QnGoAGRDegmxGNbQ22Rm15UJ6oACoQrYPWiTZAG0SboE2iLdAWURFUJNoGbRPtgHaIdkG7RHugPaJ90D7RAeiA6BB0SFQClYiOQEdEx6BjohPQCdEp6JSoDCoTVUAVoiqoSlQD1YjOQGdE56Dzf9fdTGS0OuT4N00WY+7uGryfVD4u5ZaXlo8+ZVbzkxU5z96yd+wDy7EVtsqKrMTKzGYu+8K+sm/GZ+O78cP4OW6dnZnMvGFTYfz6C1h6L6s=</latexit> start from t⇤1 at noise level ⌘1 Prior step: denoising with noise level ⌘k <latexit sha1_base64="j6fqKXB3waq6DblVi6pU/QcAmDQ=">AAACXHicbZHLTgIxFIbLiIogCpq4cTORkLgiM8agK0N04xITuSSA5Ewp0NB2xvaMSggv4VZfzI3PYrks5HKSJn+//5yk528QCW7Q834Szk5yd28/dZDOHGaPjnP5k7oJY01ZjYYi1M0ADBNcsRpyFKwZaQYyEKwRjB5mfuONacND9YzjiHUkDBTvcwpoURNf2mCw63dzBa/kzcvdFP5SFMiyqt184q7dC2ksmUIqwJiW70XYmYBGTgWbptuxYRHQEQxYy0oFkpnOZP7gqVu0pOf2Q22PQndO/09MQBozloHtlIBDs+7N4FaPvSrQGsZbzUBOV+/rQMYCuQ7fVylCEAvQH9N0cWUp7N92JlxFMSJXg8VS/Vi4GLqzpN0e14yiGFsBVHObi0uHoIGi/Y+0Tdxfz3dT1K9KfrlUfrouVO6X2afIObkgl8QnN6RCHkmV1AglgnySL/Kd+HWSTsbJLlqdxHLmlKyUc/YHr2u4DQ==</latexit> noise level ⌘0 <latexit sha1_base64="LWSFwx3yzVCwvYM5XY4QZw6Aom0=">AAAGA3icbdTJbhNBEAbgTshAMFsCRy4WFhKnyEYouSZ2Fmd3Fi9JxrJ6xm174tnc3ZPYsebIC3CFN+CGuPIgPADvwTj2X+AkJc2o5qsqqaXWlBW6jtLZ7O+Z2UdzxuMn809Tz56/ePlqYfF1RQWRtEXZDtxA1iyuhOv4oqwd7YpaKAX3LFdUrW5hVK9eCamcwD/Vg1DUPd72nZZjc51QzRSaN4a5uLGQyS5lbyN9P8lNkgybRKmxOPfHbAZ25Alf2y5X6iKXDXV9yKV2bFfEKTNSIuR2l7fFRZL63BOqPrw9cJx+n0gz3Qpk8vg6fav/Twy5p9TAs5JOj+uOulsb4YM10fO5lHzwYNHy4unvu+BFrnZkcD2tmluRy2U/TqXMpmiZ3PKGQzN58TiOx2SBLCIbZBM1QU0iARJELVCLqA1qE3VAHSIH5BBdgi6JuqAukQtyiTyQR+SDfKIAFBCFoJCoB+oRSZAkUiBFpEGaKAJFRFegK6Jr0DVRH9QnGoAGRDegmxGNbQ22Rm15UJ6oACoQrYPWiTZAG0SboE2iLdAWURFUJNoGbRPtgHaIdkG7RHugPaJ90D7RAeiA6BB0SFQClYiOQEdEx6BjohPQCdEp6JSoDCoTVUAVoiqoSlQD1YjOQGdE56Dzf9fdTGS0OuT4N00WY+7uGryfVD4u5ZaXlo8+ZVbzkxU5z96yd+wDy7EVtsqKrMTKzGYu+8K+sm/GZ+O78cP4OW6dnZnMvGFTYfz6C1h6L6s=</latexit> <latexit sha1_base64="2aeY3f9RAUtm2+JtZRfkP3HOqZE=">AAAGA3icbdTJbhNBEAbgTshAMFsCRy4WERIny0YouWaPszuJtyRjWT3jtj3xbO7uieNYc+QFuMIbcENceRAegPdgvPwFTlLSjGq+qpJaak1Zoesonc3+npl9NGc8fjL/NPXs+YuXrxYWX5dVEElblOzADWTV4kq4ji9K2tGuqIZScM9yRcXqbAzrlWshlRP4Rd0PRc3jLd9pOjbXCVVNoXl9kI3rC0vZTHYU6ftJbpIssUkU6otzf8xGYEee8LXtcqUuc9lQ1wZcasd2RZwyIyVCbnd4S1wmqc89oWqD0YHj9PtEGulmIJPH1+mR/j8x4J5Sfc9KOj2u2+pubYgP1kTX51Ly/oNFy4unv++CF7nakUFvWjW3IpfLmziVMhuiaXLLGwzM5MXjOB6TBbKIbJBN1AA1iARIEDVBTaIWqEXUBrWJHJBDdAW6IuqAOkQuyCXyQB6RD/KJAlBAFIJCoi6oSyRBkkiBFJEGaaIIFBFdg66JeqAe0Q3ohqgP6hPdgm6HNLY12Bq1rYPWiTZAG0SboE2iLdAW0TZom2gHtEOUB+WJdkG7RHugPaJ90D7RAeiA6BB0SHQEOiI6Bh0TFUAFohPQCdEp6JToDHRGVAQViUqgElEZVCaqgCpEVVCV6Bx0TnQBuvh33Y1EhqtDjn/TZDHm7q7B+0n5Yya3nFk++bS0uj5ZkfPsLXvHPrAcW2GrLM8KrMRs5rIv7Cv7Znw2vhs/jJ/j1tmZycwbNhXGr79S4S+q</latexit> <latexit sha1_base64="C+mn9EqqL/CfCS8fIBYjKPioVaQ=">AAAGBXicbdTJbhNBEAbgToghmC2BIxcLC4kLlo1QuCbO5sRZ7MRbEkdRz7htTzxbunu8xJozL8AV3oAb4spz8AC8B+PlL7CTkmZU81WV1FJryvBtS+l0+vfC4oOl2MNHy4/jT54+e/5iZfVlRXmBNEXZ9GxP1gyuhG25oqwtbYuaLwV3DFtUjc7mqF7tCqkszy3pgS8uHd5yraZlch3ReV1ofjXMv8+EVyvJdCo9jsTdJDNNkmwahavVpT/1hmcGjnC1aXOlLjJpX18OudSWaYswXg+U8LnZ4S1xEaUud4S6HI6PHCbeRtJIND0ZPa5OjPX/iSF3lBo4RtTpcN1W87UR3lsTNy6Xkg/uLRpOOPs9D05ga0t6vVnV3AhsLvthPF5viGadG85wWI9ePAzDCRkgg8gEmUQNUINIgARRE9QkaoFaRG1Qm8gCWUTXoGuiDqhDZINsIgfkELkgl8gDeUQ+yCe6Ad0QSZAkUiBFpEGaKAAFRF1Ql6gH6hH1QX2iAWhAdAu6HdHENmAb1JYFZYk2QZtEW6Atom3QNtEOaIdoF7RLlAPliPZAe0T7oH2iPChPdAA6IDoEHRIdgY6IjkHHRAVQgagIKhKdgE6ITkGnRCVQiagMKhNVQBWiKqhKVAPViM5AZ0TnoPN/192IZLQ65OQ3jRZjZn4N3k0qH1KZtdRa8WNyPTtdkcvsNXvD3rEM+8TWWY4VWJmZzGVf2Ff2LfY59j32I/Zz0rq4MJ15xWYi9usvUBAwNw==</latexit> Likelihood step: enforcing data-consistency General reverse diffusion under EDM start from t⇤0 at <latexit sha1_base64="fBIXyKhIbKLps0RTBlo586BOrMw=">AAACXHicbZHLTgIxFIbLiIogCpq4cTORkLgiM8agK0N04xITuSSA5Ewp0NB2xvaMSggv4VZfzI3PYrks5HKSJn+//5yk528QCW7Q834Szk5yd28/dZDOHGaPjnP5k7oJY01ZjYYi1M0ADBNcsRpyFKwZaQYyEKwRjB5mfuONacND9YzjiHUkDBTvcwpoURNf2mCw63VzBa/kzcvdFP5SFMiyqt184q7dC2ksmUIqwJiW70XYmYBGTgWbptuxYRHQEQxYy0oFkpnOZP7gqVu0pOf2Q22PQndO/09MQBozloHtlIBDs+7N4FaPvSrQGsZbzUBOV+/rQMYCuQ7fVylCEAvQH9N0cWUp7N92JlxFMSJXg8VS/Vi4GLqzpN0e14yiGFsBVHObi0uHoIGi/Y+0Tdxfz3dT1K9KfrlUfrouVO6X2afIObkgl8QnN6RCHkmV1AglgnySL/Kd+HWSTsbJLlqdxHLmlKyUc/YHrXq4DA==</latexit> ⌘K 1 <latexit sha1_base64="C+mn9EqqL/CfCS8fIBYjKPioVaQ=">AAAGBXicbdTJbhNBEAbgToghmC2BIxcLC4kLlo1QuCbO5sRZ7MRbEkdRz7htTzxbunu8xJozL8AV3oAb4spz8AC8B+PlL7CTkmZU81WV1FJryvBtS+l0+vfC4oOl2MNHy4/jT54+e/5iZfVlRXmBNEXZ9GxP1gyuhG25oqwtbYuaLwV3DFtUjc7mqF7tCqkszy3pgS8uHd5yraZlch3ReV1ofjXMv8+EVyvJdCo9jsTdJDNNkmwahavVpT/1hmcGjnC1aXOlLjJpX18OudSWaYswXg+U8LnZ4S1xEaUud4S6HI6PHCbeRtJIND0ZPa5OjPX/iSF3lBo4RtTpcN1W87UR3lsTNy6Xkg/uLRpOOPs9D05ga0t6vVnV3AhsLvthPF5viGadG85wWI9ePAzDCRkgg8gEmUQNUINIgARRE9QkaoFaRG1Qm8gCWUTXoGuiDqhDZINsIgfkELkgl8gDeUQ+yCe6Ad0QSZAkUiBFpEGaKAAFRF1Ql6gH6hH1QX2iAWhAdAu6HdHENmAb1JYFZYk2QZtEW6Atom3QNtEOaIdoF7RLlAPliPZAe0T7oH2iPChPdAA6IDoEHRIdgY6IjkHHRAVQgagIKhKdgE6ITkGnRCVQiagMKhNVQBWiKqhKVAPViM5AZ0TnoPN/192IZLQ65OQ3jRZjZn4N3k0qH1KZtdRa8WNyPTtdkcvsNXvD3rEM+8TWWY4VWJmZzGVf2Ff2LfY59j32I/Zz0rq4MJ15xWYi9usvUBAwNw==</latexit> <latexit sha1_base64="Di71KvXtkWTRAo7IdlMiYS0ILIk=">AAAGA3icbdTLTttAFAbggeKWpjdol91ERZW6ipKqgi13wj2B3IAgNLYnwcS3zIwJwfKyL9Bt+wbdVd32QfoAfY86l/+0BI5k6/g750gjjXzM0HWUzud/T00/mjEeP5l9mnn2/MXLV3Pzr2sqiKQlqlbgBrJhciVcxxdV7WhXNEIpuGe6om521gb1+rWQygn8iu6H4tzjbd9pORbXKTWaQvOLuJNczC3kc/lhZO8nhXGywMZRupif+dO0AyvyhK8tlyt1VsiH+jzmUjuWK5JMM1Ii5FaHt8VZmvrcE+o8Hh44yb5Pxc62Apk+vs4O9f+JmHtK9T0z7fS4vlSTtQE+WBNdn0vJ+w8WTS+5+z0JXuRqRwa9u6q5Gblc3iSZTNMWrSY3vThupi+eJMmITJBJZIEsIhtkEwmQIGqBWkRtUJvoEnRJ5IAcoivQFVEH1CFyQS6RB/KIfJBPFIACohAUEnVBXSIJkkQKpIg0SBNFoIjoGnRN1AP1iG5AN0R9UJ/oFnQ7oJGtwFaobRW0SrQGWiNaB60TbYA2iDZBm0RboC2iIqhItA3aJtoB7RDtgnaJ9kB7RPugfaID0AHRIeiQqAQqEZVBZaIj0BHRMeiYqAKqEFVBVaIaqEZUB9WJGqAG0QnohOgUdPrvuu1UBqtDjn7TdDEWJtfg/aT2MVdYzC2WPy0sr45X5Cx7y96xD6zAltgyK7ISqzKLuewL+8q+GZ+N78YP4+eodXpqPPOG3Qnj11+dMy/l</latexit> {⌘k } <latexit sha1_base64="GyJVUypZCYAH5CFzcR4YCEa2y/w=">AAAGB3icbdTJbhNBEAbgToghmC2BIxcLC4lTZCOUXLPH2Z3FS5KJrJ6Ztj3xbO7uie2M5gF4Aa7wBtwQVx6DB+A9GC9/QZaSZlTzVZXUUmvKDF1H6ULh99T0o5nM4yezT7PPnr94+Wpu/nVVBZG0RMUK3EDWTa6E6/iioh3tinooBfdMV9TMztqwXrsWUjmBf6oHobj0eMt3mo7FdUqGERtC80bcSYykMZcvLBRGkbufFCdJnk2i3Jif+WPYgRV5wteWy5W6KBZCfRlzqR3LFUnWiJQIudXhLXGRpj73hLqMR4dOcu9TsXPNQKaPr3Mj/X8i5p5SA89MOz2u2+pubYgP1kTX51LywYNF00tuf98FL3K1I4PebdXcjFwu+0k2a9iiaXDTi2MjffEkScZkgkwiC2QR2SCbSIAEURPUJGqBWkRtUJvIATlEV6Arog6oQ+SCXCIP5BH5IJ8oAAVEISgk6oK6RBIkiRRIEWmQJopAEdE16JqoB+oR9UF9ogFoQHQDuhnS2FZgK9S2ClolWgOtEa2D1ok2QBtEm6BNoi3QFlEJVCLaBm0T7YB2iHZBu0R7oD2ifdA+0QHogOgQdEhUBpWJjkBHRMegY6IT0AnRKeiUqAKqEFVBVaIaqEZUB9WJzkBnROeg83/XbacyXB1y/Jumi7F4dw3eT6ofF4qLC4tHn/LLq5MVOcvesnfsAyuyJbbMSqzMKsxiIfvCvrJvmc+Z75kfmZ/j1umpycwbdisyv/4CwjsxvQ==</latexit> <latexit sha1_base64="59oDi08mYKKJNgwBIFlEwpUifVY=">AAACZ3icbVHLSgMxFE3Hd320VRDBzWgRdFNmRKorEd24VLBVsKXcSdMamseY3LEtw3yKW/0mP8G/MK1d2NYDISfn3Av3nkSx4BaD4CvnLSwuLa+sruXXNza3CsXSdt3qxFBWo1po8xSBZYIrVkOOgj3FhoGMBHuMejcj//GNGcu1esBhzJoSuop3OAV0UqtYio8bkRZtO5TuSgfZSatYDirBGP48CSekTCa4a5Vyl422polkCqkAa5/DIMZmCgY5FSzLNxLLYqA96LJnRxVIZpvpePbMP3JK2+9o445Cf6z+7UhB2tFwrlICvthZbyT+67FXBcbA8F8zktn0e1aQiUBudH9aRYgSAWaQ5Y+mlsLORTPlKk4Quer+LtVJhI/aH4Xut7lhFMXQEaCGu1x8+gIGKLqvybvEw9l850n9tBJWK9X7s/LV9ST7VbJPDskxCck5uSK35I7UCCV98k4+yGfu2yt4u97eb6mXm/TskCl4Bz/zMrxW</latexit> <latexit sha1_base64="59oDi08mYKKJNgwBIFlEwpUifVY=">AAACZ3icbVHLSgMxFE3Hd320VRDBzWgRdFNmRKorEd24VLBVsKXcSdMamseY3LEtw3yKW/0mP8G/MK1d2NYDISfn3Av3nkSx4BaD4CvnLSwuLa+sruXXNza3CsXSdt3qxFBWo1po8xSBZYIrVkOOgj3FhoGMBHuMejcj//GNGcu1esBhzJoSuop3OAV0UqtYio8bkRZtO5TuSgfZSatYDirBGP48CSekTCa4a5Vyl422polkCqkAa5/DIMZmCgY5FSzLNxLLYqA96LJnRxVIZpvpePbMP3JK2+9o445Cf6z+7UhB2tFwrlICvthZbyT+67FXBcbA8F8zktn0e1aQiUBudH9aRYgSAWaQ5Y+mlsLORTPlKk4Quer+LtVJhI/aH4Xut7lhFMXQEaCGu1x8+gIGKLqvybvEw9l850n9tBJWK9X7s/LV9ST7VbJPDskxCck5uSK35I7UCCV98k4+yGfu2yt4u97eb6mXm/TskCl4Bz/zMrxW</latexit> <latexit sha1_base64="TkRJyhtNyacOHV6/hMfCp/4nKik=">AAAGDHicbdTLbtNAFAbgaWmghEtbWLKJiJDKpkpQVRZser9fkra5tHUVjZ1J6sZjuzPjNGnkV+AF2MIbsENseQcegPfAufwHejmSrePvnCONNPKxQ8/VJpf7PTb+aCL1+Mnk0/Sz5y9eTk3PvCrrIFKOKDmBF6iqzbXwXF+UjGs8UQ2V4NL2RMVurfTrlbZQ2g38Y9MNxbnkTd9tuA43CdWmp8JZq2PLTxlLGF7Lva9NZ3NzuUFk7if5UZJloyjUZib+WPXAiaTwjeNxrc/yudCc97gyruOJOG1FWoTcafGmOEtSn0uhz3uDk8eZd4nUM41AJY9vMgP9f6LHpdZdaSedkpsLfbfWxwdr4srnSvHug0Vbxre/74KMPOOq4Pq2Gm5HHledOJ226qJhcVv2elby4nEcD8kG2UQOyCGqg+pEAiSIGqAGURPUJLoAXRC5IJfoEnRJ1AK1iDyQRyRBksgH+UQBKCAKQSHRFeiKSIEUkQZpIgMyRBEoImqD2kTXoGuiDqhD1AV1iW5AN30a2hJsidqWQctEK6AVolXQKtEaaI1oHbROtAHaINoEbRJtgbaItkHbRDugHaJd0C7RHmiPaB+0T3QAOiAqgApERVCR6BB0SHQEOiI6Bh0TlUAlojKoTFQBVYiqoCrRCeiE6BR0+u+664n0V4ca/qbJYszfXYP3k/KHufzC3EJxPru4PFqRk+wNe8tmWZ59ZItskxVYiTksYl/YV/Yt9Tn1PfUj9XPYOj42mnnNbkXq119zSDHo</latexit> p(x; ⌘0 ) p(x) <latexit sha1_base64="o1Ir8/uSrI8MOSedkmHHxjnC/sk=">AAAGDHicbdTJbhNBEAbgToghmCUJHLlYREjhEtkoCgcu2ePszuIlyURWz7jtTDw9M+nu8RJrXoEX4ApvwA1x5R14AN6D8fIXZClpRjVfVUkttabs0HO1yWZ/j40/mkg9fjL5NP3s+YuXU9Mzr0o6iJQjik7gBapicy081xdF4xpPVEIluLQ9Ubabq/16uSWUdgP/xHRDcSF5w3frrsNNQtXpqXDO6tjyU8YShldz76vTs9n57CAy95PcKJlloyhUZyb+WLXAiaTwjeNxrc9z2dBc9LgyruOJOG1FWoTcafKGOE9Sn0uhL3qDk8eZd4nUMvVAJY9vMgP9f6LHpdZdaSedkptLfbfWxwdr4trnSvHug0Vbxre/74KMPOOqoH1bDbcjj6tOnE5bNVG3uC17PSt58TiOh2SDbCIH5BDVQDUiARJEdVCdqAFqEF2CLolckEt0BboiaoKaRB7II5IgSeSDfKIAFBCFoJDoGnRNpECKSIM0kQEZoggUEbVALaI2qE3UAXWIuqAu0Q3opk9DW4YtU9sKaIVoFbRKtAZaI1oHrRNtgDaINkGbRHlQnmgLtEW0Ddom2gHtEO2Cdon2QHtE+6B9ogPQAVEBVCA6BB0SHYGOiI5Bx0QnoBOiIqhIVAKViMqgMlEFVCE6BZ0SnYHO/l13LZH+6lDD3zRZjLm7a/B+Uvown1ucXzxcmF1aGa3ISfaGvWVzLMc+siWWZwVWZA6L2Bf2lX1LfU59T/1I/Ry2jo+NZl6zW5H69Rd44THp</latexit> p(x; ⌘1 ) p(x) Annealing schedule Figure 6.1: A schematic diagram of our method. Our method alternates between a likelihood step that enforces data consistency and a prior step that solves a denoising posterior sampling problem by leveraging the Split Gibbs Sampler [296]. An annealing schedule controls the strength of the two steps at each iteration to facilitate efficient and accurate sampling. A crucial part of our design is the prior step, where we identify a key connection to a general diffusion model framework called the EDM [151]. This connection allows us to easily incorporate a family of state-of-the-art diffusion models as priors to conduct posterior sampling in a principled way without additional training. Our method demonstrates strong performance on a variety of linear and nonlinear inverse problems. 6.3 Method A schematic diagram for the proposed method is shown in Figure 6.1. Our method, dubbed PnP-DM, builds upon the SGS framework with rigorous implementations of the two sampling steps and an annealing schedule for the coupling parameter 𝜂. We start with our implementations of the first step for solving both linear and nonlinear inverse problems. 6.3.1 Likelihood Step: Enforcing Data Consistency For the likelihood step at iteration 𝑘, we sample   1 (𝑘) 2 (𝑘) 𝑍 |𝑋=𝒙 (𝑘 ) 𝒛 ∼𝜋 (𝒛) ∝ exp − 𝑓 (𝒛; 𝒚) − 2 ∥𝒙 − 𝒛∥ 2 . 2𝜂 (6.4) Linear Forward Model and Gaussian Noise We first consider a simple yet common case where the forward model A is linear and the noise distribution is zero-mean Gaussian, i.e., A := 𝑨 ∈ R𝑚×𝑛 and 𝒏 ∼ N (0, 𝚺). In this case, the potential function of the likelihood term is 𝑓 (𝒙; 𝒚) = 21 ∥ 𝒚 − 𝑨𝒙∥ 2𝚺 (up to an additive constant that does not depend on 𝒙 and 𝒚) where ∥ · ∥ 2𝚺 := ⟨·, 𝚺−1 ·⟩. It is then straightforward to show that 𝜋 𝑍 |𝑋=𝒙 = N (𝒎(𝒙), 𝚲−1 ) 69 where 𝚲 := 𝑨𝑇 𝚺−1 𝑨 + 𝜂12 𝑰 and 𝒎(𝒙) := 𝚲−1 ( 𝑨𝑇 𝚺−1 𝒚 + 𝜂12 𝒙). The problem of sampling from Gaussian distributions has been systematically studied [295]. We refer readers to Appendix C.3.1 for a more detailed discussion. General Case For general nonlinear inverse problems, the likelihood step is not sampling from a Gaussian distribution anymore. Nevertheless, since we have access to 𝜋 𝑍 |𝑋=𝒙 in closed form up to a multiplicative factor, we can use Monte Carlo methods based on Langevin dynamics to draw samples from it as long as the likelihood potential is differentiable. Specifically, we first set up the following Langevin SDE that admits 𝜋 𝑍 |𝑋=𝒙 as the stationary distribution   √ √ 1 𝑍 |𝑋=𝒙 d𝒛𝑡 = ∇ log 𝜋 (𝒛𝑡 )d𝑡 + 2d𝒘 𝑡 = −∇ 𝑓 (𝒛; 𝒚) − 2 (𝒛 − 𝒙) d𝑡 + 2d𝒘 𝑡 . 𝜂 We then initialize the SDE at 𝒛0 = 𝒙 and run it with Euler discretization. The pseudocode is provided in Appendix C.3.1. 6.3.2 Prior Step: Denoising via the EDM Framework For the prior step at iteration 𝑘, we sample   1 (𝑘+1) 𝑋 |𝑍=𝒛 (𝑘 ) (𝑘) 2 𝒙 ∼𝜋 (𝒙) ∝ exp −𝑔(𝒙) − 2 ∥𝒙 − 𝒛 ∥ 2 . 2𝜂 (6.5) A closer examination of Equation (6.5) reveals that this prior step is essentially to draw posterior samples for a Gaussian denoising problem, where the “measurement” is 𝒛 (𝑘) , the noise level is 𝜂, and the prior distribution is 𝑝(𝒙) ∝ exp(−𝑔(𝒙)). We tackle this denoising posterior sampling problem within SGS using DMs as image priors. In particular, we leverage the EDM framework [151], which was originally proposed to unify various formulations of DMs for unconditional image generation. To see the connection of the EDM framework to Equation (6.5), consider a family of mollified distributions 𝑝(𝒙; 𝜎) given by adding i.i.d. Gaussian noise of standard deviation 𝜎 to the prior distribution 𝑝(𝒙), i.e., 𝒙 + 𝜎𝝐 ∼ 𝑝(𝒙; 𝜎). The core idea of the EDM framework is that a variety of state-of-the-art DMs can be unified into the following reverse SDE:    √︁ 𝒙𝑡 𝑠¤ (𝑡) 2 d𝒙𝑡 = 𝒙𝑡 − 2𝑠(𝑡) 𝜎(𝑡)𝜎(𝑡)∇ ¤ log 𝑝 ; 𝜎(𝑡) d𝑡 + 𝑠(𝑡) 2𝜎(𝑡)𝜎(𝑡)d ¤ 𝒘¯ 𝑡 𝑠(𝑡) 𝑠(𝑡) (6.6) where 𝒘¯ 𝑡 is an 𝑛-dimensional Wiener process running backward in time, 𝜎(𝑡) > 0 is a pre-defined noise level schedule with 𝜎(0) = 0, 𝑠(𝑡) is a pre-defined scaling 70 schedule, and 𝜎(𝑡), ¤ 𝑠¤ (𝑡) are their time derivatives. As shown in [151], the defining property of Equation (6.6) is that 𝒙𝑡 /𝑠(𝑡) ∼ 𝑝(𝒙; 𝜎(𝑡)) for any time 𝑡. Therefore, solving this SDE backward in time allows us to travel from any noise level 𝜎(𝑡) to the clean image distribution at 𝑡 = 0. This means that we can use Equation (6.6) to solve Equation (6.5) with arbitrary noise level 𝜂 as long as 𝜂 is within the range of 𝜎(𝑡). Indeed, the distribution of 𝒙0 conditioned on 𝒙𝑡 is 𝑝(𝒙0 | 𝒙𝑡 ) ∝ 𝑝(𝒙𝑡 | 𝒙0 ) 𝑝(𝒙0 ) ∝ N (𝑠(𝑡)𝒙0 , 𝑠(𝑡) 2 𝜎(𝑡) 2 𝑰) exp(−𝑔(𝒙0 ))   1 2 ∝ exp −𝑔(𝒙0 ) − ∥𝒙 0 − 𝒙𝑡 /𝑠(𝑡)∥ 2 . 2𝜎(𝑡) 2 We highlight that the last line exactly matches Equation (6.5) when 𝒙𝑡 = 𝑠(𝑡)𝒛 (𝑘) and 𝜎(𝑡) = 𝜂. Therefore, we can naturally design a practical algorithm that samples Equation (6.5) by following these three steps: (1) find 𝑡 ∗ such that 𝜎(𝑡 ∗ ) = 𝜂, (2) initialize at 𝒙𝑡 ∗ = 𝑠(𝑡 ∗ )𝒛 (𝑘) , and (3) solve Equation (6.6) backward from 𝑡 ∗ to 0 by choosing the discretization time steps and integration scheme. Through this unified interface, any DMs, once converted to the EDM formulation, can be directly turned into a rigorous solver for Equation (6.5). Leveraging the connection with EDM, our prior step implementation comes with a large design space that encompasses a variety of existing DMs, such as DDPM (or VP-SDE) [131], VE-SDE [266], and iDDPM [224]. In our experiments, we conduct posterior sampling with all these different models within our framework and all of them provide high-quality samples. The pseudocode of our implementation and more details on the EDM formulation for the prior step is given in Appendix C.3.2. 6.3.3 Overall Algorithm The pseudocode of PnP-DM in complete form is presented in Algorithm 1. PnPDM alternates between the two sampling steps with an annealing schedule {𝜂 𝑘 } for the coupling parameter. We find that the annealing schedule on 𝜂 accelerates the mixing time of the Markov chain and prevents the algorithm from getting stuck in bad local minima for solving highly ill-posed inverse problems. This is a common practice in both Langevin-based [145, 156, 272] and SGS-based [31, 326] MCMC algorithms to improve the empirical performance in solving inverse problems. Our work shares some similarities with PnP-SGS [69] but contains three main key differences. First, as demonstrated in our experiments, we investigate three nonlinear inverse problems, while nonlinear inverse problems are beyond the scope of [69]. 71 Non-stationary process µX 1 <latexit sha1_base64="kt18+z8l/fky9dHmZBgH/pt9NuA=">AAAGAnicbdTLbtNAFAbgaamhhFsLSzYRFRKrKkGobNukl/SetM2lbUo0diaJG4/tzIzTppF3vABbeAN2iC0vwgPwHjhN/gO9HMnW8XfOkUYa+dih52qTyfyemn4wYz18NPs49eTps+cv5uZfVnQQKUeUncALVM3mWniuL8rGNZ6ohUpwaXuianfzo3q1L5R2A//IDEJxJnnbd1uuw01C1bqMGtlPtcbcQmYxcx3pu0l2kiywSRQb8zN/6s3AiaTwjeNxrU+zmdCcDbkyruOJOFWPtAi50+VtcZqkPpdCnw2vzxun3ybSTLcClTy+SV/r/xNDLrUeSDvplNx09O3aCO+tiZ7PleKDe4u2jG9+3wYZecZVwcVNNdyOPK4u41Sq3hStOrflcFhPXjyO4zHZIJvIATlETVCTSIAEUQvUImqD2kQdUIfIBblE56Bzoi6oS+SBPCIJkkQ+yCcKQAFRCAqJeqAekQIpIg3SRAZkiCJQRNQH9YkuQBdEl6BLogFoQHQFuhrR2FZgK9SWA+WI8qA80SpolWgNtEa0Dlon2gBtEBVABaJN0CbRFmiLaBu0TbQD2iHaBe0S7YH2iPZB+0RFUJGoBCoRHYAOiA5Bh0RHoCOiMqhMVAFViKqgKlENVCM6Bh0TnYBO/l13M5HR6lDj3zRZjNnba/BuUnm/mF1aXCp9WFjOTVbkLHvN3rB3LMs+smVWYEVWZg7rsi/sK/tmfba+Wz+sn+PW6anJzCt2I6xffwHO1i8H</latexit> µX 0 µZ 0 ⇡X ⇡Z <latexit sha1_base64="uRjdSp8eg/FIiNE+/lABuopjyyM=">AAAGBHicbdTNbtNAEAfwbWmghK8WjlwiKiROVYJQubZJvz+TNt9NidbOJnHjtZ3ddds08pUX4ApvwA1x5T14AN4Dp8l/IG1HsjX+zYy00spjBa6jTTr9e2b2wVzi4aP5x8knT589f7Gw+LKs/VDZomT7rq+qFtfCdTxRMo5xRTVQgkvLFRWrlxvVKxdCacf3imYQiDPJO57TdmxuYqo1ZNgcpqNP9ebCUno5fROpu0lmkiyxSeSbi3N/Gi3fDqXwjO1yrU8z6cCcDbkyju2KKNkItQi43eMdcRqnHpdCnw1vThyl3sbSSrV9FT+eSd3o/xNDLrUeSCvulNx09e3aCO+tib7HleKDe4uWjKa/b4MMXeMo/3JaDbdCl6urKJlstES7wS05HDbiF4+iaEwWyCKyQTZRC9QiEiBB1Aa1iTqgDlEX1CVyQA7ROeicqAfqEbkgl0iCJJEH8oh8kE8UgAKiPqhPpECKSIM0kQEZohAUEl2ALoguQZdEV6ArogFoQHQNuh7R2NZga9SWBWWJcqAc0TponWgDtEG0Cdok2gJtEW2Dtol2QDtEu6Bdoj3QHtE+aJ/oAHRAdAg6JDoCHRHlQXmiAqhAdAw6JjoBnRAVQUWiEqhEVAaViSqgClEVVCWqgWpEdVD933W3YhmtDjX+TePFmLm9Bu8m5ffLmZXllcKHpdXsZEXOs9fsDXvHMuwjW2XbLM9KzGaSfWFf2bfE58T3xI/Ez3Hr7Mxk5hWbisSvv5hgMBQ=</latexit> <latexit sha1_base64="utnBD4ecW5WEXOZW/tXzmvW4IKo=">AAAGAnicbdTLbtNAFAbgaamhhFsLSzYRFRKrKkGobNukl/SetM2lbUo0diaJG4/tzIzTppF3vABbeAN2iC0vwgPwHjhN/gO9HMnW8XfOkUYa+dih52qTyfyemn4wYz18NPs49eTps+cv5uZfVnQQKUeUncALVM3mWniuL8rGNZ6ohUpwaXuianfzo3q1L5R2A//IDEJxJnnbd1uuw01C1bqMGplPtcbcQmYxcx3pu0l2kiywSRQb8zN/6s3AiaTwjeNxrU+zmdCcDbkyruOJOFWPtAi50+VtcZqkPpdCnw2vzxun3ybSTLcClTy+SV/r/xNDLrUeSDvplNx09O3aCO+tiZ7PleKDe4u2jG9+3wYZecZVwcVNNdyOPK4u41Sq3hStOrflcFhPXjyO4zHZIJvIATlETVCTSIAEUQvUImqD2kQdUIfIBblE56Bzoi6oS+SBPCIJkkQ+yCcKQAFRCAqJeqAekQIpIg3SRAZkiCJQRNQH9YkuQBdEl6BLogFoQHQFuhrR2FZgK9SWA+WI8qA80SpolWgNtEa0Dlon2gBtEBVABaJN0CbRFmiLaBu0TbQD2iHaBe0S7YH2iPZB+0RFUJGoBCoRHYAOiA5Bh0RHoCOiMqhMVAFViKqgKlENVCM6Bh0TnYBO/l13M5HR6lDj3zRZjNnba/BuUnm/mF1aXCp9WFjOTVbkLHvN3rB3LMs+smVWYEVWZg7rsi/sK/tmfba+Wz+sn+PW6anJzCt2I6xffwHJPC8G</latexit> µX K 1 <latexit sha1_base64="Ud5BbrAVR89U88EDFddtbjlz9UE=">AAAGBnicbdTNTttAEAfwhZKWpl/QHnuJGlXqpSipKnqFhI9A+EiAfAAJaO1sgonXdnbXQIh87wv02r5Bb1WvfY0+QN+jDsl/2gAj2Rr/ZkZaaeWxAtfRJpP5PTX9YCbx8NHs4+STp8+ev5ibf1nVfqhsUbF911d1i2vhOp6oGMe4oh4owaXliprVzQ/rtQuhtON7B6YfiKbkHc9pOzY3MR03ZHg6KL7PRif107l0ZiFzE6m7SXacpNk4SqfzM38aLd8OpfCM7XKtj7OZwDQHXBnHdkWUbIRaBNzu8o44jlOPS6Gbg5szR6m3sbRSbV/Fj2dSN/r/xIBLrfvSijslN2f6dm2I99ZEz+NK8f69RUtGk9+3QYaucZR/OamGW6HL1VWUTDZaot3glhwMGvGLR1E0IgtkEdkgm6gFahEJkCBqg9pEHVCH6Ax0RuSAHKJz0DlRF9QlckEukQRJIg/kEfkgnygABUQ9UI9IgRSRBmkiAzJEISgkugBdEF2CLomuQFdEfVCf6Bp0PaSRLcOWqS0HyhHlQXmiFdAK0SpolWgNtEa0DlonKoAKRBugDaJN0CZREVQk2gJtEW2Dtol2QDtEu6BdohKoRFQGlYn2QHtE+6B9ogPQAVEFVCGqgqpENVCNqA6qEx2CDomOQEf/rrsVy3B1qNFvGi/G7O01eDepfljILi4slj+ml3LjFTnLXrM37B3Lsk9siRVYiVWYzXz2hX1l3xKfE98TPxI/R63TU+OZV2wiEr/+AoqlMJ8=</latexit> µZ K 1 µX K ⇡Z ⇡X <latexit sha1_base64="zAwtEtkEiCVtccP0qUunSy2PUaI=">AAAGBnicbdTLbtNAFAbgaamhhFsLSzYRERIbogShsu29adNLesmlbdJq7ExSNx7bmRmnTSPveQG28AbsEFtegwfgPXCa/Ad6OZKt4++cI4008rFDz9Uml/s9Mflgynr4aPpx6snTZ89fzMy+rOggUo4oO4EXqJrNtfBcX5SNazxRC5Xg0vZE1e4sDevVnlDaDfwD0w9FQ/K277Zch5uEjusyOh0U3+fjk6PTmUwum7uO9N0kP04ybByl09mpP/Vm4ERS+MbxuNbH+VxoGgOujOt4Ik7VIy1C7nR4Wxwnqc+l0I3B9Znj9NtEmulWoJLHN+lr/X9iwKXWfWknnZKbM327NsR7a6Lrc6V4/96iLeOb37dBRp5xVXBxUw23I4+ryziVqjdFq85tORjUkxeP43hENsgmckAOURPUJBIgQdQCtYjaoDbRGeiMyAW5ROegc6IOqEPkgTwiCZJEPsgnCkABUQgKibqgLpECKSIN0kQGZIgiUETUA/WILkAXRJegS6I+qE90Bboa0sgWYAvUtghaJFoCLREtg5aJVkArRKugVaI10BpRAVQgWgetE22ANoiKoCLRJmiTaAu0RbQN2ibaAe0QlUAlol3QLtEeaI9oH7RPdAA6ICqDykQVUIWoCqoS1UA1okPQIdER6OjfdTcTGa4ONfpNk8WYv70G7yaVD9n8XHZu92NmfnG8IqfZa/aGvWN59onNswIrsTJzWMC+sK/sm/XZ+m79sH6OWicnxjOv2I2wfv0FldUwoQ==</latexit> <latexit sha1_base64="SxjCaA0j2/hdbkJ7y3Qfo8o/mxw=">AAAGBHicbdTLbtNAFAbgaWmghFsLSzYRFRKrKkGobNv0lt6Ttrm1KdHYmSRuPLY7M06aRt7yAmzhDdghtrwHD8B74Fz+A017JFvH3zlHGmnkYwWuo006/Xtm9sFc4uGj+cfJJ0+fPX+xsPiypP1Q2aJo+66vKhbXwnU8UTSOcUUlUIJLyxVlq7M+rJe7QmnH905NPxAXkrc8p+nY3MRUrcmwPtiLPlXqC0vp5fQoUneTzCRZYpPI1xfn/tQavh1K4Rnb5VqfZ9KBuRhwZRzbFVGyFmoRcLvDW+I8Tj0uhb4YjE4cpd7G0kg1fRU/nkmN9P+JAZda96UVd0pu2nq6NsR7a+LK40rx/r1FS0a3v6dBhq5xlN+7rYZbocvVdZRM1hqiWeOWHAxq8YtHUTQmC2QR2SCbqAFqEAmQIGqCmkQtUIuoDWoTOSCH6BJ0SdQBdYhckEskQZLIA3lEPsgnCkAB0RXoikiBFJEGaSIDMkQhKCTqgrpEPVCP6Bp0TdQH9YluQDdDGtsabI3asqAs0TponWgDtEG0Cdok2gJtEW2DtolyoBzRDmiHaBe0S7QH2iPaB+0THYAOiA5Bh0RHoCOiPChPVAAViI5Bx0QnoBOiU9ApURFUJCqBSkRlUJmoAqoQVUFVojPQ2b/rbsQyXB1q/JvGizEzvQbvJqX3y5mV5ZXCh6XV7GRFzrPX7A17xzLsI1tlOZZnRWYzyb6wr+xb4nPie+JH4ue4dXZmMvOK3YrEr78kmDAt</latexit> Stationary process <latexit sha1_base64="z8IAAZ3yHrMscQ8+Frupods7G9o=">AAACWXicbZHNSgMxFIUz418df6tLN4Ol4KrMiFRXIrpxWcG2YjuWO2naBpPMmNxRS5lncKuPJr6MaZ2FrV4InHznXsg9iVPBDQbBp+MuLa+srpXWvY3Nre2d3fJeyySZpqxJE5HouxgME1yxJnIU7C7VDGQsWDt+vJr67WemDU/ULY5TFkkYKj7gFNCiZjflD/e93UpQC2bl/xVhISqkqEav7Jx3+wnNJFNIBRjTCYMUowlo5FSw3OtmhqVAH2HIOlYqkMxEk9lrc79qSd8fJNoehf6M/p6YgDRmLGPbKQFHZtGbwn899qRAaxj/a8Yyn78vApkJ5Dp5macIcSZAv+ZedW4pHJxFE67SDJGr4c9Sg0z4mPjTmP0+14yiGFsBVHObi09HoIGi/QzPJh4u5vtXtI5rYb1WvzmpXFwW2ZfIATkkRyQkp+SCXJMGaRJKOHkj7+TD+XIdt+R6P62uU8zsk7ly978BFlO21g==</latexit> <latexit sha1_base64="z8IAAZ3yHrMscQ8+Frupods7G9o=">AAACWXicbZHNSgMxFIUz418df6tLN4Ol4KrMiFRXIrpxWcG2YjuWO2naBpPMmNxRS5lncKuPJr6MaZ2FrV4InHznXsg9iVPBDQbBp+MuLa+srpXWvY3Nre2d3fJeyySZpqxJE5HouxgME1yxJnIU7C7VDGQsWDt+vJr67WemDU/ULY5TFkkYKj7gFNCiZjflD/e93UpQC2bl/xVhISqkqEav7Jx3+wnNJFNIBRjTCYMUowlo5FSw3OtmhqVAH2HIOlYqkMxEk9lrc79qSd8fJNoehf6M/p6YgDRmLGPbKQFHZtGbwn899qRAaxj/a8Yyn78vApkJ5Dp5macIcSZAv+ZedW4pHJxFE67SDJGr4c9Sg0z4mPjTmP0+14yiGFsBVHObi09HoIGi/QzPJh4u5vtXtI5rYb1WvzmpXFwW2ZfIATkkRyQkp+SCXJMGaRJKOHkj7+TD+XIdt+R6P62uU8zsk7ly978BFlO21g==</latexit> <latexit sha1_base64="m0DJhVhpgMOUVm49XmpkUycU1+I=">AAACWXicbZHLTgIxFIY74w3HG8jSzURC4orMGIOuDNGNS0zkkgCSM6VAQ9sZ2zMqITyDW30048tYLgtBTtLk7/efk/T8jRLBDQbBt+Nube/s7mX2vYPDo+OTbO60buJUU1ajsYh1MwLDBFeshhwFayaagYwEa0Sj+5nfeGXa8Fg94ThhHQkDxfucAlpUayf8udnNFoJSMC//vwiXokCWVe3mnNt2L6apZAqpAGNaYZBgZwIaORVs6rVTwxKgIxiwlpUKJDOdyfy1U79oSc/vx9oehf6c/p2YgDRmLCPbKQGHZt2bwY0ee1GgNYw3mpGcrt7XgUwFch2/rVKEKBWg36decWUp7N90JlwlKSJXg8VS/VT4GPuzmP0e14yiGFsBVHObi0+HoIGi/QzPJh6u5/tf1C9LYblUfrwqVO6W2WfIGTknFyQk16RCHkiV1AglnHyQT/Ll/LiOm3G9RavrLGfyZKXc/C8ScbbU</latexit> ⇡X ⇡X <latexit sha1_base64="m0DJhVhpgMOUVm49XmpkUycU1+I=">AAACWXicbZHLTgIxFIY74w3HG8jSzURC4orMGIOuDNGNS0zkkgCSM6VAQ9sZ2zMqITyDW30048tYLgtBTtLk7/efk/T8jRLBDQbBt+Nube/s7mX2vYPDo+OTbO60buJUU1ajsYh1MwLDBFeshhwFayaagYwEa0Sj+5nfeGXa8Fg94ThhHQkDxfucAlpUayf8udnNFoJSMC//vwiXokCWVe3mnNt2L6apZAqpAGNaYZBgZwIaORVs6rVTwxKgIxiwlpUKJDOdyfy1U79oSc/vx9oehf6c/p2YgDRmLCPbKQGHZt2bwY0ee1GgNYw3mpGcrt7XgUwFch2/rVKEKBWg36decWUp7N90JlwlKSJXg8VS/VT4GPuzmP0e14yiGFsBVHObi0+HoIGi/QzPJh6u5/tf1C9LYblUfrwqVO6W2WfIGTknFyQk16RCHkiV1AglnHyQT/Ll/LiOm3G9RavrLGfyZKXc/C8ScbbU</latexit> <latexit sha1_base64="m0DJhVhpgMOUVm49XmpkUycU1+I=">AAACWXicbZHLTgIxFIY74w3HG8jSzURC4orMGIOuDNGNS0zkkgCSM6VAQ9sZ2zMqITyDW30048tYLgtBTtLk7/efk/T8jRLBDQbBt+Nube/s7mX2vYPDo+OTbO60buJUU1ajsYh1MwLDBFeshhwFayaagYwEa0Sj+5nfeGXa8Fg94ThhHQkDxfucAlpUayf8udnNFoJSMC//vwiXokCWVe3mnNt2L6apZAqpAGNaYZBgZwIaORVs6rVTwxKgIxiwlpUKJDOdyfy1U79oSc/vx9oehf6c/p2YgDRmLCPbKQGHZt2bwY0ee1GgNYw3mpGcrt7XgUwFch2/rVKEKBWg36decWUp7N90JlwlKSJXg8VS/VT4GPuzmP0e14yiGFsBVHObi0+HoIGi/QzPJh6u5/tf1C9LYblUfrwqVO6W2WfIGTknFyQk16RCHkiV1AglnHyQT/Ll/LiOm3G9RavrLGfyZKXc/C8ScbbU</latexit> Time from initialization ( ⌧ ) ⌧ =0 <latexit sha1_base64="Qj5M6Kg7+FTRc8bRu2+9j1+P8pc=">AAACWnicbZHLSgMxFIbT8daLl1bduRksBVdlRqS6UYpuXFawF2hLOZOmbWySGZMzail9B7f6ZoIPY3pZ2MuBwJ/vPwdy/gSR4AY97yfhbG3v7O4lU+nM/sHhUTZ3XDNhrCmr0lCEuhGAYYIrVkWOgjUizUAGgtWD4cPUr78xbXionnEUsbaEvuI9TgEtqrUQ4luvk817RW9W7rrwFyJPFlXp5BJ3rW5IY8kUUgHGNH0vwvYYNHIq2CTdig2LgA6hz5pWKpDMtMez507cgiVdtxdqexS6M/p/YgzSmJEMbKcEHJhVbwo3euxVgdYw2mgGcrJ8XwUyFsh1+L5MEYJYgP6YpAtLS2Hvpj3mKooRuerPl+rFwsXQnebsdrlmFMXICqCa21xcOgANFO1vpG3i/mq+66J2WfRLxdLTVb58v8g+Sc7IObkgPrkmZfJIKqRKKHkhn+SLfCd+HcdJOZl5q5NYzJyQpXJO/wCHaLcG</latexit> <latexit sha1_base64="RmYIRwjACx5v1BsIQ0g8soVMBlY=">AAAEMXicrVPLbtNAFJ3YPIp5tIUlmxFRpEQRldNFYRNoYcOySKStFIfoejxxRh2P3fF12sj11/AJfASfgMoKseUnGDtZ5NFIIHEla+6cc9++4ydSpOi6NzXLvnP33v2tB87DR4+fbO/sPj1J40wz3mOxjPWZDymXQvEeCpT8LNEcIl/yU//8fcmfTrhORaw+4TThgwhCJUaCARpouFv70fD8KJ8WXS8CHDOQ+VHRzL0qcq55UJT0VVG02qWiCqdRGeooDwrq+bEM0mlkDmMzxK4n+Qj7WRNba9zLiUE/71NPgS+BejIOaTLEyqO5auxpEY6xNTsGdCEltidV9MUiQOeLAS4LE8HU+TeJrmfdz/PR7qbySqPrDVXS9j81tbGy/5nDQ8iGO3V3z62EriuduVJ/+83pJl++O8dmG954QcyyiCtkEtK033ETHOSgUTDJC8fLUp4AO4eQ942qIOLpIK92paANgwR0FGvzKaQVuuiRQ5SWlRrL8uelq1wJ3srxCwVaw/RW0o+K5fsqEGUShY4vl1EEP5Ogr8p9XmwKR68HuVBJhihUOGtqlEmKMS2fDw2E5gzl1CjAtDBzoWwMGhiaR+aYiXdW57uunOzvdQ72Dj669cN3ZCZb5Dl5QZqkQ16RQ/KBHJMeYdaRFVqJdWF/tW/sn/avmalVm/s8I0ti//4DxtxxZQ==</latexit> Likelihood step Prior step <latexit sha1_base64="m0DJhVhpgMOUVm49XmpkUycU1+I=">AAACWXicbZHLTgIxFIY74w3HG8jSzURC4orMGIOuDNGNS0zkkgCSM6VAQ9sZ2zMqITyDW30048tYLgtBTtLk7/efk/T8jRLBDQbBt+Nube/s7mX2vYPDo+OTbO60buJUU1ajsYh1MwLDBFeshhwFayaagYwEa0Sj+5nfeGXa8Fg94ThhHQkDxfucAlpUayf8udnNFoJSMC//vwiXokCWVe3mnNt2L6apZAqpAGNaYZBgZwIaORVs6rVTwxKgIxiwlpUKJDOdyfy1U79oSc/vx9oehf6c/p2YgDRmLCPbKQGHZt2bwY0ee1GgNYw3mpGcrt7XgUwFch2/rVKEKBWg36decWUp7N90JlwlKSJXg8VS/VT4GPuzmP0e14yiGFsBVHObi0+HoIGi/QzPJh6u5/tf1C9LYblUfrwqVO6W2WfIGTknFyQk16RCHkiV1AglnHyQT/Ll/LiOm3G9RavrLGfyZKXc/C8ScbbU</latexit> ⌧ =T <latexit sha1_base64="DG+baFKQQ3BNH5kbamkj+EuctQo=">AAACWnicbZHLTgIxFIbLeOPiBdSdm4mExBWZMQbdaIhuXGLCLQFCzpQClbYztmdUQngHt/pmJj6M5bIQ5CRN/n7/OUnP3yAS3KDnfSecre2d3b1kKp3ZPzg8yuaO6yaMNWU1GopQNwMwTHDFashRsGakGchAsEYwepj5jVemDQ9VFccR60gYKN7nFNCiehshvq12s3mv6M3L/S/8pciTZVW6ucRduxfSWDKFVIAxLd+LsDMBjZwKNk23Y8MioCMYsJaVCiQzncn8uVO3YEnP7YfaHoXunP6dmIA0ZiwD2ykBh2bdm8GNHntRoDWMN5qBnK7e14GMBXIdvq1ShCAWoN+n6cLKUti/6Uy4imJErgaLpfqxcDF0Zzm7Pa4ZRTG2AqjmNheXDkEDRfsbaZu4v57vf1G/LPqlYunpKl++X2afJGfknFwQn1yTMnkkFVIjlDyTD/JJvhI/juOknMyi1UksZ07ISjmnv81Mtyo=</latexit> Prior step with error Figure 6.2: A conceptual illustration of the non-stationary and stationary timecontinuous processes as interpolations of 𝐾 discretize iterations of PnP-DM. Our experiments show that PnP-SGS struggles with challenging nonlinear inverse problems such as Fourier phase retrieval. Second, we adopt the EDM formulation to ensure that the prior step of PnP-DM is a rigorous mapping from the image manifold with the desired noise level to the clean image manifold, aligning with the theory of SGS. In contrast, the prior step of PnP-SGS [69] is heuristic (which is also pointed out by [326]) and not rigorously designed to sample Equation (6.5). Third, unlike PnP-SGS [69] that uses a constant 𝜂, we consider an annealing schedule {𝜂 𝑘 } for the coupling parameter, which is important for highly ill-posed inverse problems. Algorithm 1 Plug-and-Play Diffusion Models (PnP-DM) Require: initialization 𝒙0 ∈ R𝑛 , total number of iterations 𝐾 > 0, coupling strength 𝑚 schedule {𝜂 𝑘 > 0} 𝐾−1 𝑘=0 , likelihood potential 𝑓 ( · ; 𝒚) with measurements 𝒚 ∈ R , pre-trained model 𝐷 𝜽 ( · ; · ) that approximates ∇ log 𝑝 (𝒙; 𝜎) with (𝐷 𝜽 (𝒙; 𝜎) − 𝒙)/𝜎 2 . 1: for 𝑘 = 0, ..., 𝐾 − 1 do 2: 𝒛 (𝑘) ← LikelihoodStep(𝒙 (𝑘) , 𝑓 ( · ; 𝒚), 𝜂 𝑘 ) ⊲ Section 6.3.1 (𝑘+1) (𝑘) 3: 𝒙 ← PriorStep(𝒛 , 𝐷 𝜽 ( · ; · ), 𝜂 𝑘 ) ⊲ Section 6.3.2 4: end for 5: return 𝒙 (𝑘+1) 6.4 Convergence Analysis We provide some theoretical insights on the non-asymptotic behavior of PnP-DM via a convergence analysis. We start with the following definitions. For two probability measures 𝜇 and 𝜋 such that 𝜇 ≪ 𝜋, the Kullback–Leibler (KL) divergence and Fisher divergence (or relative Fisher information) of 𝜇 with respect to 𝜋 are defined, respectively, as ∫ ∫ 𝜇 𝜇 2 KL(𝜇||𝜋) := 𝜇 log and FI(𝜇||𝜋) := 𝜇 ∇ log . 𝜋 𝜋 2 72 Both divergences are equal to zero if and only if 𝜇 = 𝜋. KL divergence is a common metric for quantifying the difference of one distribution with respect to another. Fisher divergence has been used for analyzing the stationarity of sampling algorithms [15, 273]. We analyze PnP-DM via a continuous-time perspective, leveraging the interpolation techniques introduced for Langevin Monte Carlo [15, 273, 290]. We assume that the likelihood step Equation (6.4) can be implemented exactly and the prior step Equation (6.5) involves running the reverse diffusion process Equation (6.6) with an approximated score function 𝒔𝑡 ≈ ∇ log 𝑝 𝑡 := ∇ log 𝑝( · ; 𝜎(𝑡)). Let 𝜇0𝑋 be the 𝑋 be the distributions of 𝒛 (𝑘) and distribution of the initialization 𝒙 (0) . Let 𝜇 𝑘𝑍 and 𝜇 𝑘+1 𝒙 (𝑘+1) at the 𝑘 th iteration. Recall that the stationary distributions are 𝜋 𝑋 and 𝜋 𝑍 . Our analysis is concerned with two continuous-time processes: (1) the non-stationary process from 𝜇0𝑋 , a non-stationary initialization, to 𝜇 𝐾𝑋 where Equation (6.6) is run with the approximated score function 𝒔𝑡 and (2) the stationary process that alternates between stationary distributions 𝜋 𝑋 and 𝜋 𝑍 . These two processes are the interpolation PnP-DM in non-stationary and stationary states and define continuous transitions over discrete iterations. A conceptual illustration of the two processes is provided in Figure 6.2 with the exact formulations in Appendix C.1. Now we present our main result: Theorem 6.4.1. Consider running 𝐾 iterations of PnP-DM with 𝜂 𝑘 ≡ 𝜂 > 0 and a score estimate 𝒔𝑡 ≈ ∇ log 𝑝 𝑡 := ∇ log 𝑝( · ; 𝜎(𝑡)). Let 𝑡 ∗ > 0 be such that 𝜎(𝑡 ∗ ) = 𝜂 √︁ ¤ Define 𝜇𝜏 and 𝜋𝜏 as the and 𝛿 := inf 𝑡∈[0,𝑡 ∗ ] 𝑣(𝑡) where 𝑣(𝑡) := 𝑠(𝑡) 2𝜎(𝑡)𝜎(𝑡). distributions at time 𝜏 of the non-stationary and stationary process, respectively. Then, for over 𝐾 iterations of PnP-DM, or equivalently over 𝜏 ∈ [0, 𝑇𝐾 ] with 𝑇𝐾 := 𝐾 (𝑡 ∗ + 1), we have ∫ 𝑇𝐾 4KL(𝜋 𝑋 ||𝜇0𝑋 ) 4𝜖score 1 + ∗ , (6.7) FI (𝜋𝜏 ||𝜇𝜏 ) d𝜏 ≤ ∗ 2 𝑇𝐾 0 𝐾 (𝑡 + 1) min(𝜂, 𝛿) (𝑡 + 1)𝛿2 | {z } | {z } | {z } average Fisher divergence over 𝐾 iterations of PnP-DM convergence from initialization where we assume that the score estimation error 𝜖 score := ∇ log 𝑝 𝜏 ∥ 22 d𝜏 < ∞. score error ∫ 𝑡 ∗ +1 1 𝑣(𝜏) 2 E𝜋 𝜏 ∥𝒔𝜏 − The proof is provided in Appendix C.1. This theorem states that the average distance (measured by Fisher divergence) of the non-stationary process with respect to the stationary process over 𝐾 iterations of PnP-DM goes to zero at a rate of 𝑂 (1/𝐾) 73 Compressed sensing GT posterior DPS Ours (EDM) 1.2 Mean GT image -1.2 std. dev. <latexit sha1_base64="fKDpb1fx6i7jC7XaRYFk3YSvlh4=">AAAMYHicjZZbb9s2FMfV7pYl2dpsb9sLkSJAB2SZXezyNKyJ3SYBgsDLpS1adQElUxIRSVRIKrYj6GPsdftce90n2aEkHkmxN0yA7aPfOYeH/PNQlpfFXOnB4K8HDz/48KOPP1n7dH1j87PPHz3e+uKVErn02aUvYiHfeFSxmKfsUnMdszeZZDTxYvbaux4Z/+tbJhUX6YVeZOx9QsOUB9ynGtA7N6E6UkFxPi6vHj8Z7A2qiywbw8Z44jTX5Gpr4xd3Kvw8Yan2Y6rUu+Eg0+8LKjX3Y1auu7liGfWvacgKT4hrTT1VkvraIZkUAVNmYjT+9ianMdcLAiExU71MmqhApLqT6cXg8gSVU2KmT9Qi8US8lGWwTYKsWypToSOehr3AlPsskNRvh/dFAk5thyWBkGT43bNdwrS/18tNuC+FUbW0uZaIUNIsWvQlSDqz4RoW7BMoMO0F+Vy3o4VS5BkBVG1Wf4FVAe7Pyzo04GEu7ylnxNEClmBjYCHbpmVSdnOz3ZcrDoXkOoL5lyYU79XqsPK+2IbZaeuICcnupbI0T2BtSTMXLUhEbxmJmdZMEuNlcsUykzzWXIpZuz2QGbLUBLNV7QIoj6mcl/8zPorM0bm3HrmYqihOm7lOBcxxSkycIjwltB6nlzKvjmK5pL9UgSrX13d2dshEChFAcSuPXsSsyGLKoQ4cB2OQn5vG2CVTFvCUG0EASpHQdN1N2axJLuD0ZEJV/rKYdG56Ue0gZTFu7V5M81sWF43R88YsSWhZnFQ/PY8vpIhB6UVZjNDsRcAjIU+yuvp+a/dHgWVD7VH1U+skMrNZQoJUMH/zSQt317WN6+6WtYPKMOHgMjKLrLCPMsDuLjjKshNH5/8SR+cYF/NkKQgY+GF7wOC6jltVtK5HmoAV1epCTYDKs6UAYDgTngZLfmDoh82fo8Pc4BJE2E5dhBbfMYkYbIsD3o4CtsVTFSB2oREtH6ej01OqdFG4ms11Ud2Xv7mAymbrfDi+msLDus44MUoUJ1ftcJqnC1Jr0aT0HoTVdnkJVIAviuv1LPIQ+Rb5iJhFDFFkUYRIWaQQzS1qeyG0KES0sGiB6M6iO0TSIonoxqIbRKlFbZfmFuWITiw6QXRr0S2imUUzRCJhIcpY36EPDp5ufdVd2ekTYSh8Bpbt29h9DBtbNEZ0bNHxqo0lM/jPINAnTPI5al63UiW7bSIrNLoWfdddx3XXb72lchHVWApsSJrxKTOW3eomU9G06dcLaFHs1Qtc3HkXn7cydHErxcsufon4qIuPEJ928Sniwy4+bGXu4mPEky6eID7r4jPEl118+R8TrOXpvG1V4j5VTCsiAgJ/2R68WX5Tp4/qXM8rRlgf0ZlFLxC9QAUQnVr0FtFbfLLA7javPE09IM18jTVqFe07Opr2Ha2qB33HAXZG+1Aae1Ycr7vXR13cbuphF7d19rvYHCd45R7ef8FeNl492xv+uPfDr98/eX7QvHyvOV87285TZ+j85Dx3jpyJc+n4jnB+d/5w/tz4e3Nt89HmVh368EGT86XTuza/+gdKMYdo</latexit> Std. deviation ( SD) GT prior mean 0.15 0.05 Figure 6.3: Results on a synthetic problem with the ground truth posterior available. PnP-DM can sample it more accurately that DPS [64]. Table 6.1: Quantitative comparison on three noisy linear inverse problems for 100 FFHQ color test images. Bold: best; Underline: second best. Gaussian deblur Method PSNR (↑) SSIM (↑) PnP-ADMM [52] DPIR [345] DDRM [155] DPS [64] PnP-SGS [69] DPnP [326] PnP-DM (VP) PnP-DM (VE) PnP-DM (iDDPM) PnP-DM (EDM) 26.88 28.74 27.05 28.83 27.46 29.24 29.46 29.65 29.60 29.66 0.7855 0.8348 0.7819 0.8212 0.8356 0.8360 0.8215 0.8399 0.8383 0.8411 Motion deblur Super-resolution (4×) LPIPS (↓) PSNR (↑) SSIM (↑) LPIPS (↓) PSNR (↑) SSIM (↑) LPIPS (↓) 0.3472 0.2677 0.2570 0.2330 0.2445 0.2098 0.2202 0.2090 0.2203 0.2170 26.55 29.97 – 27.87 28.98 30.21 30.06 30.38 30.26 30.35 0.7655 0.8529 – 0.8035 0.8447 0.8527 0.8336 0.8547 0.8507 0.8547 0.3600 0.2404 – 0.2542 0.2190 0.2010 0.2099 0.1971 0.2103 0.2062 26.61 28.75 29.47 29.45 28.30 29.32 29.40 29.57 29.53 29.60 0.7634 0.8378 0.8437 0.8379 0.8349 0.8407 0.8238 0.8431 0.8404 0.8435 0.3766 0.2577 0.2322 0.2274 0.2160 0.2127 0.2219 0.2108 0.2213 0.2191 under certain conditions up to the score approximation error. Note that our theory only requires 𝐿 2 -accurate score estimate under the measure 𝜋𝜏 , which is a relatively weaker condition than the common 𝐿 ∞ -accurate score estimate assumption in prior analysis of sampling methods involving score estimates [28, 273]. This result resembles the first-order stationarity for Langevin Monte Carlo [15]. Unlike the non-asymptotic analysis in [326], we utilize the average Fisher divergence instead of the total variation distance, enabling us to obtain an explicit convergence rate. Here 𝛿 is the infimum of the diffusion coefficient along the reverse diffusion in Equation (6.6); see further discussions on the role of 𝛿 in Appendix C.1.4. Our theory shows that the accurate implementations of the two sampling steps lead to a sampler that provably converges to the stationary process that alternates between the two target stationary distributions. 74 6.5 6.5.1 Experiments Validation with Ground Truth Posterior We first demonstrate the accuracy of PnP-DM for posterior sampling on a simulated compressed sensing problem with a Gaussian prior where the posterior distribution can be expressed in a closed form. The mean and per-pixel standard deviation of the prior are visualized on the bottom left of Figure 6.3. The linear forward model 𝑨 ∈ R𝑚×𝑛 is a Gaussian matrix (𝑚 = 𝑛/2), i.e., 𝑨𝑖 𝑗 ∼ N (0, 1). A test image is randomly generated from the prior (see top left of Figure 6.3), and the measurement is calculated according to Equation (1.1) with 𝒏 ∼ N (0, 0.012 𝑰). We compare our method with the popular DM-based method DPS [64]. We draw 1,000 samples and visualize the empirical mean and per-pixel standard deviation for both algorithms. Compared with the true posterior (second column), we find that both methods accurately estimate the mean. However, the standard deviation image estimated by DPS significantly deviates from the ground truth. In contrast, our standard deviation image matches the ground truth in terms of both absolute magnitude and spatial distribution. These results highlight the accuracy of our method over DPS by taking a more principled Bayesian approach. 6.5.2 Benchmark Experiments Dataset and Inverse Problems We test our proposed algorithm and several baseline methods on 100 images from the validation set of the FFHQ dataset [152] for five inverse problems: (1) Gaussian deblur with kernel size 61×61 and standard deviation 3.0, (2) Motion deblur with kernel size 61×61 and intensity of 0.5, (3) Super-resolution with 4× downsampling ratio, (4) the coded diffraction patterns (CDP) reconstruction problem (nonlinear) in [42, 221] (phase retrieval with a phase mask), and (5) the Fourier phase retrieval (nonlinear) with 4× oversampling. We add i.i.d. Gaussian noise to all the simulated measurements 𝒚. In particular, i.e., 𝒏 ∼ N (0, 𝜎𝒚2 𝑰). For all problems except for Fourier phase retrieval, the noise standard deviation is set as 𝜎𝒚 = 0.05. Due to the severe ill-posedness of Fourier phase retrieval, we consider a smaller noise standard deviation 𝜎𝒚 = 0.01. Baselines and Comparison Protocols We consider four variants of DMs as plugin priors for our method, namely VP-SDE (VP) [131], VE-SDE (VE) [266], iDDPM [224], and EDM [151]. We compare our method with various baselines, including (1) optimization-based methods: PnP-ADMM [52], DPIR [345]; (2) conditional DMs: DDRM [155], DPS [65]; and (3) SGS-based method: PnP-SGS [69], DPnP 75 DPIR DPS PnP-SGS Ours: PnP-DM (VE) Ours: PnP-DM (EDM) Ground truth Motion deblur Measurement Figure 6.4: Visual examples for the motion deblur problem (𝜎𝒚 = 0.05). We visualize one sample generated by each sampling algorithm. [326]. For fair comparison, we use the same pre-trained score function checkpoint for all DM-based methods. Since the pre-trained score function was trained with the DDPM formulation (VP-SDE) [131], we convert it to the EDM formulation by applying the VP preconditioning [151]. We use the Peak Signal-to-Noise Ratio (PSNR), the Structural Similarity Index Measure (SSIM), and the Learned Perceptual Image Patch Similarity (LPIPS) distance for quantitative comparison. For each sampling method, we draw 20 random samples, calculate their mean, and report the metrics on the mean image. More experimental details are provided in Appendices C.2, C.3, and C.4. Results: Linear Problems A quantitative comparison is provided in Table 6.1. PnP-DM generally outperforms the baseline methods, and the VE and EDM variants consistently outperform the other two variants on these linear problems. Figure 6.4 contains visual examples for the motion deblur problem (see Appendix C.6.2 for the other two linear problems). PnP-DM provides high-quality reconstructions that are both sharp and consistent with the ground truth image. We also provide an uncertainty quantification analysis based on pixel-wise statistics in Figure 6.5. In the left three columns, we visualize the absolute error (|b 𝒙 −𝒙0 |), standard deviation (std), and absolute z-score (|b 𝒙 −𝒙0 |/std). In the third column, red pixels highlight locations where the ground truth pixel values are outliers of the 3-sigma credible interval (CI) under the estimated posterior uncertainty. The fourth column contains scatter plots of |b 𝒙 − 𝒙0 | versus std for each pixel of the reconstructions, where red boxes show the percentages of outliers (outside of 3-sigma CI) and gray boxes indicate the percentages within the 3-sigma CI. Similar to the synthetic prior experiment, DPS tends to have larger standard deviation estimations, as shown by the less concentrated distribution of gray points around the origin. Compared with baselines, especially PnP-SGS, our approach captures a higher percentage (97.46%) of ground truth pixels than the baselines (96.20% and 88.77%). If the true posterior were truly Gaussian, 76 x0 |/std |b x 0.45 x0 | vs. std <latexit sha1_base64="4A7GQ21R7DHMNdvP7sXD5ot/2xQ=">AAACYHicbVFNS8NAEN3Erxo/2upNL8FS8FQSkepJRC8eFawtmCKT7aYu7m7i7kQtIX/Dq/4tr/4St7UH2/pg4c17MzDzNs4ENxgEX467tLyyulZZ9zY2t7artfrOnUlzTVmHpiLVvRgME1yxDnIUrJdpBjIWrBs/XY797gvThqfqFkcZ60sYKp5wCmilKJKAjyYpDA7Kh1ojaAUT+IsknJIGmeL6oe6cRYOU5pIppAKMuQ+DDPsFaORUsNKLcsMyoE8wZPeWKpDM9IvJ0qXftMrAT1Jtn0J/ov6dKEAaM5Kx7ZwsOe+NxX899qxAaxj9a8aynK3nBZkL5Dp9nVUR4lyAfiu95sxRmJz2C66yHJGr4e9RSS58TP1x2v6Aa0ZRjCwBqrnNxaePoIGi/RPPJh7O57tI7o5aYbvVvjlunF9Ms6+QfXJADklITsg5uSLXpEMoycg7+SCfzrdbcatu/bfVdaYzu2QG7t4PbaK6UQ==</latexit> = |e x0 | <latexit sha1_base64="OxE6hKc0gq/831/BcoKk/ktmi7U=">AAAGAHicbdTJThtBEAbghuCEOBskpygXKyhSLkFjFJErYPbVBrwAY1k9M2178Gye7vGCGUXKu0TKKSjXvETOeZuMsf9SMJTU3dVfVR/6Ukbg2FJp2t+p6UczqcdPZp+mnz1/8fLV3PzrkvSj0BRF03f8sGJwKRzbE0VlK0dUglBw13BE2WjlhvVyR4TS9r1T1Q9E1eUNz67bJlcJ1ebeXutd2xJNrgZ6z3DjzKfM8Kxp17W5BW1Ru43M/SQ7ThbYOPK1+Zk/uuWbkSs8ZTpcyousFqjqgIfKNh0Rp/VIioCbLd4QF0nqcVfI6uD2D3HmQyJWpu6HyfJU5lb/fzHgrpR910g6Xa6acrI2xAdrou3xMOT9B4vJj+/eJ8GNHGWHfveuKm5EDg97cTqtW6Kuc8MdDPRk43Ecj8gAGUQmyCSyQBaRAAmiOqhO1AA1iJqgJpENsokuQZdELVCLyAE5RC7IJfJAHpEP8okCUEDUBrWJQlBIJEGSSIEUUQSKiDqgDlEX1CXqgXpEfVCf6Ap0NaSRrcJWqW0NtEaUA+WI1kHrRBugDaJN0CbRFmiLaBu0TbQD2iHaBe0S7YH2iPZB+0QHoAOiQ9Ah0RHoiCgPyhMVQAWiY9Ax0QnohOgUdEpUBBWJSqASURlUJqqAKkRnoDOic9B5QskIzE4OvPtJaWkxu7y4XPi8sLI2Hoaz7B17zz6yLPvCVtg2y7MiM9lX9p39ZDepb6kfqZvUr1Hr9NT4zRt2J1K//wFzuC3I</latexit> |b x DPS 3 3.80 % td |b x <latexit sha1_base64="OxE6hKc0gq/831/BcoKk/ktmi7U=">AAAGAHicbdTJThtBEAbghuCEOBskpygXKyhSLkFjFJErYPbVBrwAY1k9M2178Gye7vGCGUXKu0TKKSjXvETOeZuMsf9SMJTU3dVfVR/6Ukbg2FJp2t+p6UczqcdPZp+mnz1/8fLV3PzrkvSj0BRF03f8sGJwKRzbE0VlK0dUglBw13BE2WjlhvVyR4TS9r1T1Q9E1eUNz67bJlcJ1ebeXutd2xJNrgZ6z3DjzKfM8Kxp17W5BW1Ru43M/SQ7ThbYOPK1+Zk/uuWbkSs8ZTpcyousFqjqgIfKNh0Rp/VIioCbLd4QF0nqcVfI6uD2D3HmQyJWpu6HyfJU5lb/fzHgrpR910g6Xa6acrI2xAdrou3xMOT9B4vJj+/eJ8GNHGWHfveuKm5EDg97cTqtW6Kuc8MdDPRk43Ecj8gAGUQmyCSyQBaRAAmiOqhO1AA1iJqgJpENsokuQZdELVCLyAE5RC7IJfJAHpEP8okCUEDUBrWJQlBIJEGSSIEUUQSKiDqgDlEX1CXqgXpEfVCf6Ap0NaSRrcJWqW0NtEaUA+WI1kHrRBugDaJN0CbRFmiLaBu0TbQD2iHaBe0S7YH2iPZB+0QHoAOiQ9Ah0RHoiCgPyhMVQAWiY9Ax0QnohOgUdEpUBBWJSqASURlUJqqAKkRnoDOic9B5QskIzE4OvPtJaWkxu7y4XPi8sLI2Hoaz7B17zz6yLPvCVtg2y7MiM9lX9p39ZDepb6kfqZvUr1Hr9NT4zRt2J1K//wFzuC3I</latexit> |/s x0 | ) Standard deviation <latexit sha1_base64="r6dkQxfuQ7dP5LU49GRz6YAlq80=">AAAGD3icbdRJT9tAFAfwgZKWpltoj71YRZW4lCZVRa8Q9j1ANsAoGtuTxMQbnjFJMP4KlfpdKvVU1GtPPffb1Fn+T2V5kj3Pv/dG8szhGYFjS5XP/52YfDSVefxk+mn22fMXL1/lZl5XpR+FpqiYvuOHdYNL4dieqChbOaIehIK7hiNqRmd5UK9dilDavldW/UCcubzl2U3b5CqlRm7uWu/almhzFes9w020D9pgbeSvtY+a7nLVls1YKitp5Gbz8/lhaPeTwjiZZeMoNWam/uiWb0au8JTpcClPC/lAncU8VLbpiCSrR1IE3OzwljhNU4+7Qp7FwyMl2vtULK3ph+njKW2o/++IuStl3zXSzuFP3q0N8MGauPB4GPL+g8X0Am5/3wU3cpQd+t3bqrgROTzsJdmsbommzg03jvX0xZMkGZEBMohMkElkgSwiARJETVCTqAVqEbVBbSIbZBOdg86JOqAOkQNyiFyQS+SBPCIf5BMFoIDoAnRBFIJCIgmSRAqkiCJQRHQJuiTqgrpEPVCPqA/qE12BrgY0siXYErUVQUWiZdAy0QpohWgVtEq0BlojWgetE22ANog2QZtEW6Atom3QNtEOaIdoF7RLtAfaI9oH7ROVQCWiA9AB0SHokOgIdERUBpWJKqAKURVUJaqBakR1UJ3oGHRMdAI6SSkdgYW7A+9+Uv00X1iYXzj4PLtYHA/DafaWvWNzrMC+sEW2wUqswkz2lX1nP9lN5lvmR+Ym82vUOjkx3vOG3YrM73830DPv</latexit> rr or x Abs. error ( |b <latexit sha1_base64="OxE6hKc0gq/831/BcoKk/ktmi7U=">AAAGAHicbdTJThtBEAbghuCEOBskpygXKyhSLkFjFJErYPbVBrwAY1k9M2178Gye7vGCGUXKu0TKKSjXvETOeZuMsf9SMJTU3dVfVR/6Ukbg2FJp2t+p6UczqcdPZp+mnz1/8fLV3PzrkvSj0BRF03f8sGJwKRzbE0VlK0dUglBw13BE2WjlhvVyR4TS9r1T1Q9E1eUNz67bJlcJ1ebeXutd2xJNrgZ6z3DjzKfM8Kxp17W5BW1Ru43M/SQ7ThbYOPK1+Zk/uuWbkSs8ZTpcyousFqjqgIfKNh0Rp/VIioCbLd4QF0nqcVfI6uD2D3HmQyJWpu6HyfJU5lb/fzHgrpR910g6Xa6acrI2xAdrou3xMOT9B4vJj+/eJ8GNHGWHfveuKm5EDg97cTqtW6Kuc8MdDPRk43Ecj8gAGUQmyCSyQBaRAAmiOqhO1AA1iJqgJpENsokuQZdELVCLyAE5RC7IJfJAHpEP8okCUEDUBrWJQlBIJEGSSIEUUQSKiDqgDlEX1CXqgXpEfVCf6Ap0NaSRrcJWqW0NtEaUA+WI1kHrRBugDaJN0CbRFmiLaBu0TbQD2iHaBe0S7YH2iPZB+0QHoAOiQ9Ah0RHoiCgPyhMVQAWiY9Ax0QnohOgUdEpUBBWJSqASURlUJqqAKkRnoDOic9B5QskIzE4OvPtJaWkxu7y4XPi8sLI2Hoaz7B17zz6yLPvCVtg2y7MiM9lX9p39ZDepb6kfqZvUr1Hr9NT4zRt2J1K//wFzuC3I</latexit> 96.20 % 0 std 0.15 |/s td = 3 11.23 % <latexit sha1_base64="OxE6hKc0gq/831/BcoKk/ktmi7U=">AAAGAHicbdTJThtBEAbghuCEOBskpygXKyhSLkFjFJErYPbVBrwAY1k9M2178Gye7vGCGUXKu0TKKSjXvETOeZuMsf9SMJTU3dVfVR/6Ukbg2FJp2t+p6UczqcdPZp+mnz1/8fLV3PzrkvSj0BRF03f8sGJwKRzbE0VlK0dUglBw13BE2WjlhvVyR4TS9r1T1Q9E1eUNz67bJlcJ1ebeXutd2xJNrgZ6z3DjzKfM8Kxp17W5BW1Ru43M/SQ7ThbYOPK1+Zk/uuWbkSs8ZTpcyousFqjqgIfKNh0Rp/VIioCbLd4QF0nqcVfI6uD2D3HmQyJWpu6HyfJU5lb/fzHgrpR910g6Xa6acrI2xAdrou3xMOT9B4vJj+/eJ8GNHGWHfveuKm5EDg97cTqtW6Kuc8MdDPRk43Ecj8gAGUQmyCSyQBaRAAmiOqhO1AA1iJqgJpENsokuQZdELVCLyAE5RC7IJfJAHpEP8okCUEDUBrWJQlBIJEGSSIEUUQSKiDqgDlEX1CXqgXpEfVCf6Ap0NaSRrcJWqW0NtEaUA+WI1kHrRBugDaJN0CbRFmiLaBu0TbQD2iHaBe0S7YH2iPZB+0QHoAOiQ9Ah0RHoiCgPyhMVQAWiY9Ax0QnohOgUdEpUBBWJSqASURlUJqqAKkRnoDOic9B5QskIzE4OvPtJaWkxu7y4XPi8sLI2Hoaz7B17zz6yLPvCVtg2y7MiM9lX9p39ZDepb6kfqZvUr1Hr9NT4zRt2J1K//wFzuC3I</latexit> |b x |e x0 | PnP-SGS 0.45 rr or 0 88.77 % 0 0.45 std 0.15 |/s td = 3 2.54 % <latexit sha1_base64="OxE6hKc0gq/831/BcoKk/ktmi7U=">AAAGAHicbdTJThtBEAbghuCEOBskpygXKyhSLkFjFJErYPbVBrwAY1k9M2178Gye7vGCGUXKu0TKKSjXvETOeZuMsf9SMJTU3dVfVR/6Ukbg2FJp2t+p6UczqcdPZp+mnz1/8fLV3PzrkvSj0BRF03f8sGJwKRzbE0VlK0dUglBw13BE2WjlhvVyR4TS9r1T1Q9E1eUNz67bJlcJ1ebeXutd2xJNrgZ6z3DjzKfM8Kxp17W5BW1Ru43M/SQ7ThbYOPK1+Zk/uuWbkSs8ZTpcyousFqjqgIfKNh0Rp/VIioCbLd4QF0nqcVfI6uD2D3HmQyJWpu6HyfJU5lb/fzHgrpR910g6Xa6acrI2xAdrou3xMOT9B4vJj+/eJ8GNHGWHfveuKm5EDg97cTqtW6Kuc8MdDPRk43Ecj8gAGUQmyCSyQBaRAAmiOqhO1AA1iJqgJpENsokuQZdELVCLyAE5RC7IJfJAHpEP8okCUEDUBrWJQlBIJEGSSIEUUQSKiDqgDlEX1CXqgXpEfVCf6Ap0NaSRrcJWqW0NtEaUA+WI1kHrRBugDaJN0CbRFmiLaBu0TbQD2iHaBe0S7YH2iPZB+0QHoAOiQ9Ah0RHoiCgPyhMVQAWiY9Ax0QnohOgUdEpUBBWJSqASURlUJqqAKkRnoDOic9B5QskIzE4OvPtJaWkxu7y4XPi8sLI2Hoaz7B17zz6yLPvCVtg2y7MiM9lX9p39ZDepb6kfqZvUr1Hr9NT4zRt2J1K//wFzuC3I</latexit> |b x |e x0 | rr or Ours: PnP-DM (EDM) 0 97.46 % 0 0.0 0.3 0.0 0.15 0.0 ≥3.0 0 std 0.15 Figure 6.5: Comparison of uncertainty quantification (UQ) for the motion deblur. Left 3 columns: absolute error (|b 𝒙 − 𝒙0 |), standard deviation (std), and absolute z-score (|b 𝒙 − 𝒙0 |/std) with the outlier pixels in red. Right column: scatter plot of |b 𝒙 − 𝒙0 | versus std. Note that PnP-DM leads to a better UQ performance than the baselines by having the lowest percentage of outliers while avoiding having overestimated per-pixel standard deviations. 99% of the ground-truth pixels should lie within the 3-sigma CI; however, as the posterior is not Gaussian with a DM-based prior, we do not necessarily expect to reach 99% coverage. Results: Nonlinear Problems We provide a quantitative comparison in Table 6.2. For the CDP reconstruction problem, PnP-DM performs on par with DPS but outperforms other SGS-based methods. We then consider the Fourier phase retrieval (FPR) problem, which is known to be a challenging nonlinear inverse problem. One challenge lies in its invariance to 180◦ rotation, so the posterior distribution has two modes, one with upright images and another with 180◦ -rotated images, that equally fit the measurement. To increase the chance of getting properly-oriented reconstructions, we run each algorithm with four different random initializations and report the metrics for the best run, following the practice in [65]. We find that PnP- 77 Table 6.2: Quantitative evaluation on two noisy nonlinear inverse problems for 100 FFHQ grayscale test images. Bold: best; Underline: second best. Coded diffraction patterns Method PSNR (↑) HIO [104] DPS [64] PnP-SGS [69] DPnP [326] PnP-DM (VP) PnP-DM (VE) PnP-DM (iDDPM) PnP-DM (EDM) (a) DPS – 33.43 32.19 32.19 32.91 33.13 33.35 33.25 PnP-SGS Fourier phase retrieval SSIM (↑) LPIPS (↓) PSNR (↑) SSIM (↑) LPIPS (↓) – 0.9049 0.8889 0.8853 0.8846 0.8971 0.9083 0.9050 Ours: PnP-DM (EDM) – 0.1374 0.2010 0.2000 0.1906 0.1663 0.1471 0.1386 Ground truth 20.66 23.60 15.36 29.28 30.36 29.88 30.61 31.14 0.4308 0.6804 0.3659 0.8397 0.8553 0.8464 0.8718 0.8731 (b) Rotated Upright 0.6469 0.3126 0.5730 0.2180 0.2115 0.2186 0.1975 0.2024 Fourier PR 18.9% Rotated 81.1% Upright Figure 6.6: Results of the Fourier phase retrieval problem. (a) PnP-DM provides both upright and rotated reconstructions (two modes given by the invariance of Explain in caption and that flipping is the forward model to 180◦ rotation) with high fidelity, while paragraph the baseline methods normal cannot. (b) We visualize the percentages of upright and rotated reconstructions out of 90 runs for a test image with two samples for each orientation. Put PSNR and show average psnr DM significantly outperforms the baselines on this highly ill-posed inverse problem. Add flipped to the corner As shown in Figure 6.6 (a), our method can provide high-quality reconstructions for both orientations, while the baseline methods fail to capture at least one of the two modes. We further run our method for a test image with 100 different random initializations and collect reconstructions in both orientations that are above 28 dB in PSNR (90 out of 100 runs). The percentage of upright and rotated reconstructions is visualized by the pie chart in Figure 6.6 (b). With a prior on upright face images, our method generates mostly samples with the upright orientation. Nevertheless, it can also find the other mode that has an equal likelihood, demonstrating its ability to capture multi-modal posterior distributions. 6.5.3 Experiments on Black Hole Imaging Problem Setup We validate PnP-DM on a real-world nonlinear imaging inverse problem: black hole imaging (BHI) (see Appendix C.2 for more details). A visual illustration of BHI is provided in Figure 6.7 (a). This BHI inverse problem is severely ill-posed. Even with an Earth-sized telescope, only a small fraction of the 78 (a) Ground truth (b) Data mismatch Data-fidelity(↓ (↓)) Mode 1 Mode 2 Mode 1 Mode 2 4.19 3.24 60μas 60μas 60μas 60μas Fourier spectrum 2.60 2.60 3.22 2.38 Mode 3 41.17 788.44 3.24 19.79 5.58 2.83 3.23 45.79 1087.77 2.92 6.19 15.67 550.14 Mean image 3.12 9e-3 0.0 u <latexit sha1_base64="e45MGNrssxesSN5lX0lJ/w2Yr/A=">AAAMVnicjZbfb9s2EMfVbl07J9vS7XEvRIIAHZBldrEfT8OaOG0SIAjcNGmLzm1AyZRERBIVkoqtCPoL9rr9b9s/M+woiScp9oYJSHT63B2P/PIoy00jrvRw+Ne9+x99/OCTh48+Haytf/b5FxuPv3ytRCY9duGJSMi3LlUs4gm70FxH7G0qGY3diL1xr8bG/+aGScVFcq7zlL2PaZBwn3tUA3qZXW5sDXeH1UWWjVFjbDnNNbl8vPbLdCa8LGaJ9iKq1K+jYarfF1Rq7kWsHEwzxVLqXdGAFa4QV5q6qiT1tU1SKXymzHRo9O11RiOucwIhEVO9TBorXyS6k+lG4HIFlTMSUx0SlceuiJayDLZJkHVDZSJ0yJOgF5hwj/mSeu3wnojBqe2wxBeSjL57ukOY9nZ7uTH3pDBaljbXEhFImoZ5X4K4MxuuYcEegQKzXpDHdTtaIEWWEkDVFvUXWBXg3qKsQ30eZPKOckYcLWAJNgYWsmkaJWHX15t9uaJASK5DmH9pQvFZrQ4r74ptmJ22DpmQ7E4qS7IY1hY3c9GChPSGkYhpzSQxXiZXLDPOIs2lmLfbA5kBS0wwW9UugLKIykX5P+PD0ByYO+uR+UyFUdLMdSZgjjNi4hThCaH1OL2URXUAyyX9pfJVORhsb2+TiRTCh+JWHp1HrEgjyqEOHAdjkJ+bxtghM+bzhBtBAEoR02QwTdi8SS7g9KRCVf6ymHQeelHtIGVx0Nq9mOZeFueN0fNGLI5pWZxUt57HE1JEoHReFmM0exHwSsjitK6+19r9UWDZUHtc3WqdRGo2S0iQCuZv/pJiujO1jTvdKWsHlUHMwWVkFml9Vz5sXjDdAUdZduLo4l/i6ALjIh4vBQEDP2wPGFzXcauK1vVIE7CiWl2oCVBZuhQADGfCE3/JDwz9sPkLdJgHXIII2qmLwOJbJhGDbbHP21HAtnimfMRTaETLD5Lx6SlVuiimmi10UT2XH6aAymbrPDi+msLLus44MUoUJ5ftcJonOam1aFJ6L8Jqu9wYKsA/iut1LXIReRZ5iJhFDFFoUYhIWaQQLSxqeyGwKECUW5QjurXoFpG0SCK6tugaUWJR26WZRRmiE4tOEN1YdINobtEckYhZgDLWT+iDg6dbX/VUdvpEGAp/Q8v2bOwehh1YdIDo2KLjVRtL5vCbQaBPmOQL1LxupUp220RWaHTlfddtx3Xbb72lciHVWApsSJrzGTOW3eomU9Gk6ddzaFHs1XNc3KsuftXK0MWtFC+6+AXioy4+QnzaxaeID7v4sJW5i48RT7p4gvisi88QX3TxxX9MsJan87VViftEMa2I8An8ZLvwPflNnT6uc123GGN9RGcWPUf0HBVAdGrRO0Tv8M0Cu9t88jT1gDTzNda4VbTv6Gjad7Sq7vcd+9gZ7UvpwLXiuN29PuridlMPu7its9fF5jjBJ/fo7gf2svH66e7ox90fXn6/9Wy/+fh+5HztbDpPnJHzk/PMOXImzoXjOcz5zfnd+WPtz7W/1x+sP6xD799rcr5yetf6xj8H+YQK</latexit> v <latexit sha1_base64="ypQ60mojgKSM35Id65I/+e61i8Q=">AAAMVnicjZbfb9s2EMfVbl27JNvS7XEvRIIAHZBldrEfT8OaOG0SIAjcNGmLzm1AyZRERBIVkrKtCPoL9rr9b9s/M+woiScp9oYJSHT63B2P/PIoy00jrvRg8Ne9+x99/OCTh48+XVvf+OzzLzYff/laiUx67NITkZBvXapYxBN2qbmO2NtUMhq7EXvjXo+M/82MScVFcqHzlL2PaZBwn3tUA3o5u9rcHuwNqossG8PG2Haaa3z1eP2XyVR4WcwS7UVUqV+Hg1S/L6jU3ItYuTbJFEupd00DVrhCXGvqqpLU1w5JpfCZMtOh0bc3GY24zgmEREz1MmmsfJHoTqYbgcsVVE5JTHVIVB67IlrKMtgmQdaMykTokCdBLzDhHvMl9drhPRGDU9thiS8kGX73dJcw7e31cmPuSWG0LG2uJSKQNA3zvgRxZzZcw4I9AgWmvSCP63a0QIosJYCqLeovsCrAvUVZh/o8yOQd5Yw4WsASbAwsZMs0SsJubrb6ckWBkFyHMP/ShOKzWh1W3hXbMDttHTIh2Z1UlmQxrC1u5qIFCemMkYhpzSQxXiZXLDPOIs2lmLfbA5kBS0wwW9UugLKIykX5P+PD0ByYO+uR+VSFUdLMdSpgjlNi4hThCaH1OL2URXUAyyX9pfJVuba2s7NDxlIIH4pbeXQesSKNKIc6cByMQX5uGmOXTJnPE24EAShFTJO1ScLmTXIBpycVqvKXxbjz0ItqBymLw9buxTT3srhojJ43YnFMy+K0uvU8npAiAqXzshih2YuAV0IWp3X1/dbujwLLhtqj6lbrJFKzWUKCVDB/85cUk92JbdzJblk7qAxiDi4js0jru/Jh84LJLjjKshNHF/8SRxcYF/F4KQgY+GF7wOC6jltVtK5HmoAV1epCTYDK0qUAYDgTnvhLfmDoh81foMM84BJE0E5dBBbfMokYbIt93o4CtsVT5SOeQCNafpiMzs6o0kUx0Wyhi+q5/DABVDZb58Hx1RRe1nXGqVGiOL1qh9M8yUmtRZPSexFW2+XGUAH+UVyva5GLyLPIQ8QsYohCi0JEyiKFaGFR2wuBRQGi3KIc0a1Ft4ikRRLRjUU3iBKL2i7NLMoQnVp0imhm0QzR3KI5IhGzAGWsn9AHB0+3vuqp7PSJMBT+Bpbt29h9DDu06BDRiUUnqzaWzOE3g0CfMMkXqHndSpXstoms0OjK+67bjuu233pL5UKqsRTYkDTnU2Ysu9VNpqJJ068X0KLYqxe4uFdd/KqVoYtbKV508QvEx118jPisi88QH3XxUStzF58gHnfxGPF5F58jvuziy/+YYC1P52urEveJYloR4RP4yXbhe/KbOn1U57puMcL6iM4teo7oOSqA6Myid4je4ZsFdrf55GnqAWnma6xRq2jf0dG072hVPeg7DrAz2pfSoWvFcbt7fdzF7aYedXFbZ7+LzXGCT+7h3Q/sZeP1073hj3s/vPx++9lB8/H9yPna2XKeOEPnJ+eZc+yMnUvHc5jzm/O788f6n+t/bzzYeFiH3r/X5Hzl9K6NzX8AE+uECw==</latexit> <latexit (Reconstruction from fully sampled low-freq.) sha1_base64="5Xk/EpwtMrd+Mzt6SkJoAh8NewQ=">AAAMgnicjZbhbts2EMfVrtu61NvS7eO+EAkCdFvm2WmTJsCGNXHaJEAQeGnSFp26gJIoiYgkKiQV2xH0EHuafd0eY2+zoyRSUuwMM2CL+t0dj/zfUZaTRlTIweCfe/c/evDxJ58+/GzpUe/zL75cfvzVG8Ey7pJzl0WMv3OwIBFNyLmkMiLvUk5w7ETkrXM5Uva314QLypIzOUvJhxgHCfWpiyWgi+Xv7SzxCHc4dkluX2XYu/OnuFheHfR3tp9vD7bQoL+xsznY2VaDZ083Nwdo2B+Un1Wr/owvHj/6xfaYm8UkkW6EhfhtOEjlhxxzSd2IFEt2JkiK3UsckNxh7FJiRxSo+qyhlDOfCLV0HP0AK4ionCFwiYjoROJY+CyRrUgnApPDMPdQjGWIxCx2WDQXpbAOgqhrzBMmQ5oEHceEusQHfZrpXRaDUeppkc84Gv64sY6IdPud2Ji6nCndCx2rCQs4TsNZV4K4tRoqYcMuggRex8mlspkt4CxLEaCynN0NlgmoOy0qV58GGb+lnBJHMtiC9oGNrKimSsjV1UpXrihgnMoQ1l8oV3MvFrsVt8VWTC9bhoRxciuUJFkMe4vrtUiGQnxNUESkJBwpK+ELthlnkaScTZryQGRAEuVMFrULoCzCfFr8T/8wVIfr1n74zBNhlNRr9Ris0UPKTyCaIFzN0wmZloe1mNOfC18US0tra2tozBnzIbmWR84ikqcRppAHjoMaoJ/rxlhHHvFpQpUgADmLcbJkJ2RSB+dwelImSnuRj1s3Ha9mkiLfb8Ydn/pa5Gf1oGONSBzjIj8uLx2LyziLQOlZkY/MsOMBj4QsTqvsu824OwtsG3KPykulE0tVsRgHqWD96pvk9rqtG9deLyoD5kFMwaRkZml1FT4UL7DXwVAULT88vcMPT41fROM5J2Bgh/LAgMrKb1HSKh+qHRZkqxLVDiJL5xyAmZXQxJ+zAzN2KP7UGNSN2QILmqWzQOMbwg2GscY+bWaBscae8A22oRE1309GJydYyDy3JZnKvLwvfrcBFXXpXDi+EsPDuoo4VkrkxxfNdJImM1RpUYd0HoRluZwYMsAPNvt1NHIMcjVyDSIaEYNCjUKDhEbCoKlGTS8EGgUGzTSaGXSj0Y1BXCNu0JVGVwYlGjVdmmmUGXSs0bFB1xpdGzTRaGIQi0lgZKzujA0Onmxs5V3R6hOmKHwHmu1q313jtq/RvkFHGh0tKiyawH8Ggj4hnE6N5lUrlbLrJtJCG9Osa7ppmW66rTeXLsTSpIIxBE2oR9RIl7qOFDip+/UMWtT06pnZ3Os2ft3I0MaNFK/a+JXBh218aPBJG58YfNDGB43MbXxk8LiNxwaftvGpwedtfP4fC6zkab1tleI+EUQKxHwEf9kOvHt+W4WPqljHyUcmv0GnGr006KVRwKATjd4b9N48WaC69StPnQ9IvV41GjWKdg0tTbuGRtW9rmHPdEbzUNp3tDhOu9aHbdwU9aCNmzy7bayOE7xy6/dqdPfgzUZ/uNXf+vXZ6ou9+uX7ofWNtWI9sYbWc+uFdWiNrXPLtf6w/rT+sv7uPeh91xv2nlau9+/VMV9bnU/vp38BsHyVYA==</latexit> | {z Ours: PnP-DM (EDM) } <latexit sha1_base64="1MZpKbTtjWY7+NgJzh09sxd8OlE=">AAAMgnicjZb/btxEEMfdUqCkB6TwJ/+sEkUqEI67lIREAtHk0iaRouhok7YqLtHat7ZXsb3O7jp3F8sPwdPwLzwGb8P4x47tXII46c7rz8zs7H5n1mcnCbnSg8E/9+5/8ODDjz5++MnSo96nn32+/PiL10qk0mVnrgiFfOtQxUIeszPNdcjeJpLRyAnZG+diVNjfXDGpuIhP9Txh7yPqx9zjLtWAzpe/tdN4wqQjqcsy+zKlkzt/8vPl1UF/sLW9/XSbDPqbm8OdjQEMdna2Nwc7ZNgflJ9Vq/6Mzx8/+sWeCDeNWKzdkCr123CQ6PcZlZq7IcuX7FSxhLoX1GeZI8SFpo7KSfVZI4kUHlPF0mn4Hawg5HpOwCVkqhNJI+WJWLcinRBMjqByQiKqA6LmkSPChagCmyCIuqIyFjrgsd9xjLnLPNCnmd4VERi1mZZ4QpLh9xvrhGm334mNuCtFoXtuYg0RvqRJMO9KELVWwzVs2CWQYNJxcrluZvOlSBMCqCxnd4NlAu7O8srV434qbyhXiKMFbMH4wEZWiqaK2eXlSleu0BeS6wDWnxeueK9ud8tvil0ws2wdMCHZjVAWpxHsLarXogUJ6BUjIdOaSVJYmbxlm1Eaai7FtCkPRPosLpzZbe0CKA2pnOX/0z8IisN1Yz9yPlFBGNdrnQhY44QUforwmNBqnk7IrDys+YL+UnkqX1paW1sjYymEB8mNPHoesiwJKYc8cByKAfm5box1MmEej3khCEApIhov2TGb1sEZnJ5EqNKeZ+PWTcermSTP9ptxx6e+5tlpPehYQxZFNM+Oy0vH4gopQlB6nmcjHHY84JGQRkmVfbcZd2eBbUPuUXmpdBJJUSwhQSpYf/GNM3vdNo1rr+eVgUo/4mAqZBZJdVUeFM+318GQ5y0/OrvDj87QL+TRghMwsEN5YMB15Xdb0iofqR1uyVYlqh1Umiw4AMOV8NhbsANDOxR/hobiBrcg/Gbpwjf4mknEMDbY480sMDZ4ojzENjSi4fvx6OSEKp1ltmYznZX3+e82oLwunQvHV1N4WFcRx4US2fF5M53m8ZxUWtQhnQdhWS4nggzwQ3G/jkEOItcgFxEziCEKDAoQKYMUoplBTS/4BvmI5gbNEV0bdI1IGiQRXRp0iSg2qOnS1KAU0bFBx4iuDLpCNDVoikhEzEcZqzu0wcHTja28y1t9IgoK34Fhu8Z3F932DdpHdGTQ0W2FJVP4zyDQJ0zyGWpetVIpu2kiIzSa5l3Tdct03W29hXQB1ZgKxhA05RNWjEyp60hF47pfT6FFsVdPcXOv2vhVI0MbN1K8aOMXiA/b+BDxSRufID5o44NG5jY+Qjxu4zHil238EvFZG5/9xwIreVpvW6W4TxTTigiPwF+2A++eX1fhoyrWcbIR5kf00qDniJ6jAohODHqH6B0+WaC69StPnQ9Ivd5iNGoU7RpamnYNjap7XcMedkbzUNp3jDhOu9aHbdwU9aCNmzy7bVwcJ3jlNu/V5O7B643+cKu/9esPq8/26pfvh9ZX1or1xBpaP1rPrENrbJ1ZrvWH9af1l/V370Hvm96w97RyvX+vjvnS6nx6P/0L6viVZQ==</latexit> | {z DPS Posterior samples Zoom-in on low-freq. } Figure 6.7: Results on the black hole imaging problem with simulated data. Due to severe noise corruption and measurement sparsity, this problem is non-convex and highly ill-posed, leading to a bi-modal posterior distribution as previously found in [269]. Here we compare our method, PnP-DM, with the DPS baseline [64]. A metric quantifying the mismatch with the observed measurements is labeled for each sample, which should be around 2 for an ideal measurement fit. Samples generated by PnP-DM exhibit two distinct modes with sharp details and a consistent ring structure, while samples given by DPS display inconsistent ring sizes and sometimes fail to capture the black hole structure entirely, with samples having poor measurement fit. Fourier frequencies of the target black hole can be measured (region within the red box); in reality, this region is further subsampled with a highly sparse pattern (black lines). Additionally, the atmospheric noise causes nonlinearity of this BHI problem that sometimes results in a multi-modal posterior distribution of the reconstructed image [269]. Here we demonstrate the effectiveness of PnP-DM in capturing a multi-modal posterior distribution. For brevity, we restrict our choice of diffusion models in PnP-DM to EDM and use DPS as the baseline. The pre-trained diffusion model is trained on images from the GRMHD simulation [72]. Results on Simulated Data We first use the simulated data from [269] where the measurements are generated assuming that the ground-truth black hole image was at the location of the Sagittarius A∗ black hole. Figure 6.7 (b) visually compares the 79 Prior samples Posterior samples Mean image 40μas Official image by EHT 40μas * Experiment is performed with real data for the M87 black hole Figure 6.8: Results on the black hole imaging problem with the real data for the M87 black hole from April 6th , 2017 [71]. The posterior samples from PnPDM contain fine-grained features that align with the prior distribution; see left for a few samples generated by the pre-trained diffusion model from the prior. Besides having high visual quality, our posterior samples accurately capture key features of the official reconstruction by EHT as well, such as the bright spot location and ring diameter. results obtained by PnP-DM and DPS. We use the t-SNE method [210] to cluster the generated samples (100 for each method) and identify two modes in the samples generated by PnP-DM and three modes in those generated by DPS. We visualize the mean and three samples for each image mode. A metric for quantifying the degree of data mismatch is labeled on the top right corner of each image. As illustrated by both the mean and sample images, PnP-DM successfully captures the two modes previously identified for this dataset [269]. Note that PnP-DM generates high-fidelity samples from both modes with sharp details of the flux ring, and its samples from “Mode 1” align well with the ground truth image. In contrast, two out of the three modes sampled by DPS fail to exhibit a meaningful black hole structure and do not correspond with the observed measurements, as indicated by the significantly larger data mismatch values. Results on Real Data We finally apply PnP-DM to the real M87 black hole data from April 6th , 2017 [71], where the results are shown in Figure 6.8. By leveraging an expressive DM-based image prior, PnP-DM generates high-quality samples that are both visually plausible and consistent with the ring diameters observed in the official EHT reconstruction. These results highlight the robustness and effectiveness of our method in tackling a highly ill-posed real-world inverse problem. 80 Table 6.3: Quantitative results for the DSA deconvolution problem in radio astronomy. We compare PnP-DM with the well-known baseline CLEAN on detecting the galaxies and estimating their shapes and fluxes. PnP-DM outperform CLEANs in terms of all the metrics. Detection Precision ↑ Recall ↑ CLEAN PnP-DM 6.5.4 0.8675 0.8866 0.9791 0.9991 Shape & flux estimation 𝐹1 score ↑ Semi-major axis MSE ↓ Semi-minor axis MSE ↓ Flux MSE ↓ 0.9199 0.9395 26.84 4.81 21.94 3.20 1.48 × 105 897 Experiments on Deconvolution with the Deep Synoptic Array (DSA) Problem Setup We finally show the effectiveness of PnP-DM on solving a realworld deconvolution problem that arises in radio astronomy (see Appendix C.2 for more details). We adopt the same experimental setup as [73] based on the Deep Synoptic Array (DSA) [122]. We first generate a dataset of 800 true sky images of size 512×512 with a spatial resolution of 0.25 arcsecond, used for training the diffusion model in PnP-DM, and a separate dataset from the same distribution for testing. The true sky images contain 2D Gaussian ellipsoids with random orientations on the sky. The brightness, ellipticity, and angular size of each galaxy are chosen to match the empirically measured distributions for radio galaxies [284]. We simulate dirty sky measurements by convolving the true sky images with the full-band point spread function (PSF) of DSA with a bandwidth of 1,300 MHz. The main difference from the deblurring problems in Section 6.5.2 is that the additive Gaussian noise is applied before the blurring operation for this problem, which models the random fluctuations in the background of the true sky before reaching the telescopes. The standard deviation of the noise is chosen to have a signal-to-noise ratio of around 6, which is the expected level for DSA. Since some galaxies in the true sky could have intensities on par with the level of the background noise, there are uncertainties about their existence. Posterior sampling methods are desirable for quantifying the likelihoods of their existence and discovering new galaxies. Results We compare PnP-DM with CLEAN [134], a widely used baseline in radio astronomy, which suppresses PSF-shaped artifacts in the dirty image by iteratively subtracting the PSF near the bright galaxies. We first provide a visual comparison in Figure 6.9. The top panel displays the mean of 20 posterior samples generated by PnP-DM. Since the PSF in DSA lacks strong side lobes, CLEAN is less effective in this setting where the dirty image does not contain any obvious PSF-induced artifacts. In contrast, PnP-DM faithfully reconstructs both the positions and shapes 81 (a) Measurement CLEAN Ours: PnP-DM (EDM) 22.00 (b) Sample 1 42.48 Sample 2 42.37 Ground truth 52.41 Sample 3 PSNR Standard deviation 42.35 0.05 0 Figure 6.9: Visual examples of the DSA deconvolution problem in radio astronomy. All images undergo a nonlinear transformation to visualize the weaker galaxies. (a): The mean image of PnP-DM (based on 20 samples) significantly outperforms the reconstruction of the classic CLEAN algorithm [134]. (b): We visualize three posterior samples and the per-pixel standard deviation map computed from all 20 samples. Zoom-in regions highlight areas with notable sample variability. of the galaxies, achieving a peak signal-to-noise ratio (PSNR) of 52.41dB. The bottom panel of Figure 6.9 illustrates three representative posterior samples and the per-pixel standard deviation map computed from all 20 samples. Zoom-in regions highlight areas with notable structural differences. As indicated by the yellow arrows, certain galaxies only appear in some samples but not others, illustrating the ability of PnP-DM to recover diverse structures in its reconstructions. This diversity enables us to understand and quantify the uncertainty of the reconstruction. We further assess the quality of the reconstructed astronomical images in a taskrelevant manner using the extract function from the SEP package1 . Specifically, we quantify their accuracies for detecting galaxies and estimating their shapes and fluxes and provide the quantitative results in Table 6.3. A detected galaxy is considered 1 https://github.com/sep-developers/sep (Lesser GNU Public License) 82 Figure 6.10: Comparison of galaxy property estimation accuracy between CLEAN (blue) and PnP-DM (orange). Each scatter plot shows predicted versus true values for semi-major axis 𝜃 𝐴 (left), semi-minor axis 𝜃 𝐵 (middle), and flux (right), across all detected sources. The dashed line indicates perfect prediction. PnP-DM produces estimates that lie closer to the diagonal, indicating more accurate recovery of galaxy shapes and fluxes. Notably, PnP-DM avoids the strong overestimation biases seen in CLEAN. a true positive if its center lies within 10 pixels of the nearest galaxy in the ground truth image; otherwise, it is counted as a false positive. Conversely, a false negative is recorded when no detection falls within the 10-pixel radius of a ground truth galaxy. Based on these criteria, we compute standard detection metrics: Precision, Recall, and 𝐹1 score. Additionally, we report the mean squared errors (MSE) of the semi-major axis (𝜃 𝐴 ), semi-minor axis (𝜃 𝐵 ), and flux based on the metrics provided by the source detection algorithm. PnP-DM significantly outperforms CLEAN in terms of all metrics. The improvements are further shown by the scatter plots in Figure 6.10, where each point represents a galaxy detection. The estimates given by PnP-DM align with the ground truth much better than those given by CLEAN. 6.6 Conclusion In this chapter, we introduced PnP-DM, a posterior sampling method for solving imaging inverse problems. The backbone of our method is a split Gibbs sampler that iteratively alternates between two steps that separately involve the likelihood and prior. Crucially, we established a link between the prior step and a general DM framework known as the EDM formulation. By leveraging this connection, we seamlessly integrated a diverse range of state-of-the-art DMs as priors through a unified interface. Experimental results demonstrate that our method outperforms existing DM-based methods across both linear and nonlinear inverse problems, including a nonlinear and severely ill-posed black hole interferometric imaging problem. 83 Limitations PnP-DM can be further improved in the following two aspects. First, PnP-DM currently requires evaluating the likelihood and prior steps for the entire image at a time. This potentially poses computational challenges in solving large-scale inverse problems (e.g., 3D imaging) or those with expensive likelihood evaluation (e.g., PDE inverse problems). Second, the current theoretical analysis does not consider the approximation error introduced in the likelihood step for general nonlinear inverse problems when running Langevin MCMC for finite iterations. Explicit incorporation of this error would offer further insights into the empirical performance of PnP-DM. 84 Chapter 7 BEYOND PNP-DM: TOWARDS A UNIFIED FRAMEWORK FOR GENERAL POSTERIOR ESTIMATION In Chapter 6, we introduced PnP-DM as a general method for solving inverse problems using diffusion models. While it has proven effective across a range of linear and nonlinear imaging problems, several important challenges remain unaddressed. In this chapter, we take the core idea of PnP-DM further and work toward a unified PnPDP framework capable of handling more general classes of inverse problems. We begin in Section 7.1 by outlining four challenges that are beyond the reach of the original PnP-DM in Chapter 6. We then formulate a unified framework in Section 7.2 for addressing these challenges. The remainder of the chapter presents four instantiations (Section 7.3, Section 7.4, Section 7.5, Section 7.6), each designed to address one of these challenges. This chapter is based on our works [22, 63, 297, 352], which collectively expand the scope and applicability of diffusion-based posterior estimation. 7.1 Limitations of PnP-DM Recall that our goal is to sample from the posterior distribution 𝑝(𝒙0 | 𝒚) given measurements 𝒚 = A (𝒙 0 ) + 𝒏. PnP-DM turns a pre-trained diffusion model (DM) that characterizes the prior 𝑝(𝒙0 ) into a principled sampler for the posterior. However, a few important issues remain unaddressed by the original formulation. First, there may be additional sources of information about 𝒙0 that we would like to incorporate, such as a text description of what 𝒙 0 looks like. Text could provide rich semantic information that significantly constrains the set of possible solutions, but it often requires working with more advanced latent DMs, which were not considered in Chapter 6. Can we develop a latent analogue of PnP-DM that can support state-of-the-art text-conditioned DMs? Second, many imaging inverse problems are defined for videos, which introduce higher dimensionality and, more importantly, an extra time dimension compared to static images. How can we efficiently obtain expressive diffusion priors for videos? And how can we effectively incorporate them into the PnP-DM framework 85 to reconstruct videos that are consistent in both spatial and temporal dimensions? Third, in many applications (especially those governed by partial differential equations (PDEs)), the forward model A may only permit forward evaluation but not differentiation. The likelihood step in Chapter 6 assumes access to the gradient of A, making it incompatible with such black-box settings. Can we overcome this limitation and extend PnP-DM to handle non-differentiable forward models? Fourth, many inverse problems have solution spaces that are inherently discrete, whereas PnP-DM is designed for continuous domains R𝑛 . Modeling data distributions in discrete spaces requires discrete DMs, which differ substantially from their continuous counterparts. Can we design an analogous version of PnP-DM for discrete spaces that also offers strong theoretical guarantees and empirical performance? We answer all of these questions affirmatively by introducing four methods in the remainder of this chapter. To highlight their shared structure and enable future generalizations, we first present a unifying alternating-update framework that unifies all four approaches. 7.2 A Unified Diffusion-Based Posterior Estimation Framework Algorithm 2 A Unified Diffusion-Based Posterior Estimation Framework Require: initialization 𝒖 (0) , total number of iterations 𝐾 > 0, noise schedule 𝑚 {𝜂 𝑘 > 0} 𝐾−1 𝑘=0 , measurements 𝒚 ∈ R , pre-trained diffusion model 𝒔 𝜽 . 1: for 𝑘 = 0, ..., 𝐾 − 1 do 2: 𝒗 (𝑘) ← LikelihoodStep(𝒖 (𝑘) , 𝒚, 𝜂 𝑘 ) 3: 𝒖 (𝑘+1) ← PriorStep(𝒗 (𝑘) , 𝒔𝜽 , 𝜂 𝑘 ) 4: end for 5: return 𝒖 (𝐾) We present the unified framework in Algorithm 2. In a nutshell, it alternates between two updates—the likelihood step and the prior step—until a final sample is obtained. The likelihood step enforces measurement consistency in a stochastic way, producing noisy output with noise level 𝜂 𝑘 . The prior step denoises the sample back from noise level 𝜂 𝑘 to the clean image manifold. The noise schedule {𝜂 𝑘 > 0} 𝐾−1 𝑘=0 controls the progression of the reconstruction from a starting manifold (e.g., the manifold with noise level 𝜂0 ) to an ending manifold with noise level 𝜂 𝐾−1 ≈ 0. In Section 7.3 and Section 7.4, we retain the likelihood step from PnP-DM and extend the prior step. Rather than unconditional image-space DMs for images 86 considered in Chapter 6, we investigate either text-conditioned latent-space DMs (Section 7.3) or latent-space video DMs (Section 7.4). Such extensions enable us to incorporate text description as a form of prior and solve video inverse problems, respectively. In Section 7.5, we keep the prior step the same as in PnP-DM and generalize the likelihood step. Instead of using a gradient-based MCMC sampler for the likelihood step, we leverage the Ensemble Kalman technique [143] to implement the likelihood step via statistical linearization when the gradient of A is unavailable. This generalization enables us to solve inverse problems based on, for example, the Navier-Stokes equation. Finally, in Section 7.6, we propose a discrete analogue of PnP-DM that requires re-designing both the likelihood and prior steps. We first construct a discrete version of Split Gibbs Sampler and then follow the same recipe in PnP-DM to obtain an implementable algorithm, where the likelihood step involves a Metropolis-Hasting sampler and the prior step involves a discrete DM. 7.3 Thrust 1: Incorporating More Information as Prior Text descriptions provide rich semantic information for characterizing image distributions, which may be helpful as priors for solving inverse problems. For example, for a blurry photograph, we can more accurately resolve it into a sharper image if we are given a description of its content. However, the instantiation of PnP-DM in Chapter 6 assumes unconditional DMs and does not take text as an input. It remains an open question whether and how state-of-the-art text-conditioned DMs, such as StableDiffusion [249], can be effectively leveraged for solving inverse problems. In this section, we show how to employ a text-conditioned DM in the prior updates of Algorithm 2 for incorporating text as a form of prior knowledge. Specifically, we present three instantiations of Algorithm 2 on the update rules of DCDP [181], PnPDM (Section 6.3), and DAPS [341]. We show that they have highly similar likelihood steps and the same prior steps. Compared to existing text-guided diffusion-based techniques such as TReg [160], our approach does not require an additional CLIP model [236]. We demonstrate the effectiveness of our approach on three image restoration problems with severe ill-posedness. We find that our proposed methods can generate high-quality solutions while also being able to resolve ambiguity according to text prompts. The appendix for this section is Appendix D.1. 87 7.3.1 Instantiation of Algorithm 2 with Text-Conditioned Diffusion Models To incorporate text as a form of prior, we sample from the following posterior distribution conditioned on both measurements and text 𝑝(𝒙0 | 𝒚, 𝒕) ∝ 𝑝( 𝒚 | 𝒙 0 , 𝒕) 𝑝(𝒙0 | 𝒕) = 𝑝( 𝒚 | 𝒙 0 ) 𝑝(𝒙0 | 𝒕) (7.1) where the equality holds because 𝒚 and 𝒕 are independent conditioned on 𝒙0 . The text-conditioned prior 𝑝(𝒙0 | 𝒕) can be modeled by text-conditioned DMs. Solve Inverse Problems in Latent Space Since most text-conditioned DMs are latent-space models, we formulate inverse problems also in latent space. Assuming that the set of likely targets 𝒙0 ’s is in the range of a decoder D, we have that ∃ 𝒛0 s.t. 𝒙0 = D (𝒛 0 ) and can thus rewrite Equation (1.1) as: 𝒚 = A (D (𝒛0 )) + 𝒏. (7.2) It follows that the posterior 𝑝(𝒙0 | 𝒚) is the pushforward of the latent posterior 𝑝(𝒛0 | 𝒚) through D. Then it suffices to first generate latent samples from 𝑝(𝒛 0 | 𝒚) and then decode them by D. This latent-space approach shares a similar spirit as the Deep Image Prior (DIP) approach [177], which (in our notations) chooses D to be a convolutional neural network and 𝒛0 to be its weights. However, there are two key distinctions. First, DIP is deterministic and only provides a single solution b 𝒙 = D (𝒛0 ), while we aim to sample from the posterior 𝑝(𝒙 0 | 𝒚). Second, DIP employs an untrained network and optimizes its randomly initialized weights, while our decoder D is pre-trained in a VAE and fixed when we optimize its input 𝒛0 . Similar to Equation (7.1), the latent posterior 𝑝(𝒛0 | 𝒚) can be written as 𝑝(𝒛0 | 𝒚, 𝒕) ∝ 𝑝( 𝒚 | 𝒛 0 ) 𝑝(𝒛0 | 𝒕) (7.3) where 𝑝(𝒚 | 𝒛0 ) is given by Equation (7.2) and 𝑝(𝒛 0 | 𝒕) can be modelled by state-of-the-art text-conditioned DMs [240, 249]. Text-Conditioned Latent Diffusion Models (LDMs) Given a text prompt 𝒕, textconditioned LDMs sample from the text-guided prior distribution with the classifierfree guidance technique [132], which models the prior distribution as 𝑝(𝒛0 | 𝒕) ∝ 𝑝(𝒛0 ) · 𝑝(𝒕 | 𝒛0 ) 𝑤 where the parameter 𝑤 ≥ 0 controls the strength of the text guidance. During the reverse diffusion process, where the time-dependent score function ∇𝒙0 log 𝑝 𝑡 (𝒙0 | 𝒕) 88 is required, these models approximate it as ∇ 𝒛0 log 𝑝 𝑡 (𝒛 0 | 𝒕) = ∇ 𝒛0 log 𝑝 𝑡 (𝒛0 ) + 𝑤 · ∇ 𝒛0 log 𝑝 𝑡 (𝒕 | 𝒛0 ) ≈ 𝒔𝜽 (𝒛0 , ∅, 𝑡) + 𝑤 · 𝒔𝜽 (𝒛0 , 𝒕, 𝑡), where 𝒔𝜽 is a neural network that takes text as input and ∅ denotes an empty string. Note that setting 𝑤 = 0 disables text conditioning, reducing the model to an unconditional diffusion model. Overall Approach With the latent-space formulations of inverse problems and DMs in place, we can then follow the same procedure in Section 6.2 to get a latent version of the Split Gibbs Sampler in the format of Algorithm 2. Below we present three instantiations of the prior and likelihood steps based on the update rules of DCDP [181], PnP-DM (Section 6.3), and DAPS [341]. We show that they share the same prior step and only differ slightly for the likelihood step. We refer the readers to Appendix D.1.1 for a detailed derivation of each algorithm. 7.3.1.1 Prior Step via Text-Conditioned Denoising Diffusion The prior step (Line 3 of Algorithm 2) generates a sample 𝒖 (𝑘+1) from the following distribution ! 2 1 /𝑍 (7.4) exp log 𝑝(𝒖 | 𝒕) − 2 𝒖 − 𝒗 (𝑘) 2 2𝜂 𝑘 where 𝑍 is a normalizing constant and 𝒕 is the text prompt. As explained in Section 6.3, this sampling task is equivalent to solving a Gaussian denoising problem and can be implemented rigorously using the EDM framework given a text-conditioned DM for 𝑝(𝒖 | 𝒕) [151]. To sample from this distribution, we first find the timestep 𝑡 = 𝑡 ∗ where 𝜎𝑡 ∗ = 𝜂 𝑘 and set 𝒖 𝑡 ∗ = 𝒗 (𝑘) . Then we run the following updated reverse diffusion probability flow ODE1 from 𝑡 = 𝑡 ∗ to 𝑡 = 0    𝑠¤𝑡 𝒖𝑡 2 d𝒖 𝑡 = 𝒖 𝑡 − 𝑠𝑡 𝜎 ¤ 𝑡 𝜎𝑡 𝒔𝜽 , 𝒕, 𝜎𝑡 d𝑡 (7.5) 𝑠𝑡 𝑠𝑡 where the score function 𝒔𝜽 takes the text embedding 𝒕 as input. The score term 𝒔𝜽 (𝒙𝑡 /𝑠𝑡 , 𝒕, 𝜎𝑡 ) can be readily obtained using the built-in support for text prompts through classifier-free guidance of many state-of-the-art LDMs, such as StableDiffusion [249]. As illustrated in Figure 7.1, starting from a noisy iterate 𝒗 (𝑘) , the prior 1 This is the ODE version of Equation (6.6). distribution at any time 𝑡. These two processes share the same marginal 89 Figure 7.1: Illustration of the prior step with different text inputs. Starting from a noisy image 𝒗 (𝑘) , the prior step performs text-guided denoising through reverse diffusion to generate a cleaner sample 𝒖 (𝑘+1) on the clean image manifold. The images are visualized after decoding. Different text prompts steer the denoising process toward distinct yet plausible modes of the image. step can sample distinct yet plausible modes on the clean image manifold matching different text prompts. In scenarios where the measurement 𝒚 is highly ambiguous due to the ill-posedness of the forward operator A, the text prompt 𝒕 serves as a powerful signal to guide the restoration process and enables controllability within the solution space. Our approach resembles the technique used in the SDEdit algorithm [218], where text prompts are used to guide the image generation process. Unlike prior methods such as TReg [160], our framework does not rely on a CLIP image encoder and introduces no additional hyperparameters besides those already in the LDMs (e.g., classifier-free guidance scale 𝑤). 7.3.1.2 Likelihood Step with Two Sub-Steps For the likelihood step (Line 2 of Algorithm 2), we summarize the three instantiations in the following table. Method DCDP [181] PnP-DM (Section 6.3) DAPS [341] Sub-Step 1 Sub-Step 2 Find argmax of (7.6) Sample from (7.6) Sample from (7.6) Add noise 𝝐 ∼ N (0, 𝜂2𝑘 𝑰) — Add noise 𝝐 ∼ N (0, 𝜂2𝑘 𝑰) 90 Sub-Step 1: Enforcing Data Fidelity The first part of the likelihood step targets the following distribution ! 2 1 1 exp − 2 ∥A (D (𝒗)) − 𝒚∥ 22 − 2 𝒖 (𝑘) − 𝒗 /𝑍 (7.6) 2 2𝜎𝒚 2𝜂 𝑘 where 𝑍 is a normalizing constant. This distribution balances fidelity to the observed measurements 𝒚 with proximity to the current prior sample 𝒖 (𝑘) . The way this target distribution is handled varies slightly across different methods. DCDP performs MAP estimation by maximizing the log-posterior using gradient-based optimization. In contrast, PnP-DM and DAPS treat this as a sampling problem and use algorithms such as Langevin Monte Carlo or Hamiltonian Monte Carlo to draw samples from Equation (7.6). Although their formulations differ, they can be unified under our framework by identifying a correspondence between 𝜂 𝑘 and the method-specific hyperparameters used to control the relative weighting between data fidelity and prior proximity. Specifically, we have:  √   𝜎𝒚 / 𝜇     𝜂𝑘 ≡ 𝜂𝑘     𝑟 𝑡  for DCDP [181], for PnP-DM (Chapter 6), for DAPS [341] where 𝑡 ≡ 𝑘. This correspondence reveals that all three methods are effectively targeting the same class of distributions, differing only in their choices of hyperparameters. Sub-Step 2: Adding Noise The second part of the likelihood step involves adding Gaussian noise to the intermediate sample obtained in sub-step 1. The intuition is that Equation (7.6) is equivalent to solving the original inverse problem with a gaussian prior N (𝒖 (𝑘) , 𝜂2𝑘 𝑰), so ideally the output 𝒗 (𝑘) should contain additional noise with a standard deviation of 𝜂 𝑘 upon 𝒖 (𝑘) . However, this may not be the case in practice, so DCDP and DAPS manually add the expected amount of noise to maintain a consistent noise level 𝜂 𝑘 across iterations, aligning the sample with the expected input distribution of the subsequent prior denoising step. On the other hand, PnP-DM strictly follows the Split Gibbs Sampling derivation and thus does not have this noise adding step. We find in the experiments that this noise-adding step is helpful empirically when using LDMs as priors. 91 Figure 7.2: Visual examples for the super-resolution (16×), gaussian deblur (𝜎 = 6), and box inpainting tasks with matching prompts. The proposed instantiations, DAPS and DCDP in particular, produce more detailed and perceptually coherent reconstructions, while the baseline methods suffer from artifacts and a decrease in quality. Table 7.1: Quantitative evaluation for the super-resolution (16×), gaussian deblurring (𝜎 = 6), and box inpainting tasks with matching prompts. Bold: best; Underline: second best. For each task, we report the mean performance and standard deviation in PSNR, SSIM, and LPIPS across 14 test images. The results show that methods based on our unified framework outperform the baseline approaches, TReg and PSLD, especially on the super-resolution and deblurring problems. The unified methods maintain strong performance without relying on additional components, highlighting their efficiency and generality. PSNR (↑) Super-resolution (16×) SSIM (↑) LPIPS (↓) Data fit (↓) PSNR (↑) Gaussian deblur (𝜎 = 6) SSIM (↑) LPIPS (↓) TReg PSLD 20.96 (1.88) 14.74 (3.33) 0.48 (0.14) 0.24 (0.20) 0.42 (0.05) 0.53 (0.08) 1.99 (0.22) 3.52 (1.76) 23.81 (2.18) 20.42 (4.54) 0.60 (0.14) 0.41 (0.18) DCDP PnP-DM DAPS 22.44 (2.44) 22.04 (2.80) 22.87 (2.43) 0.55 (0.16) 0.49 (0.19) 0.47 (0.15) 0.45 (0.07) 0.45 (0.08) 0.39 (0.08) 1.32 (0.10) 1.76 (0.50) 1.46 (0.15) 24.98 (2.51) 23.85 (2.43) 24.33 (2.61) 0.62 (0.15) 0.58 (0.16) 0.63 (0.15) Method Data fit (↓) PSNR (↑) Box inpainting SSIM (↑) LPIPS (↓) 0.26 (0.06) 0.44 (0.10) 39.44 (5.57) 79.83 (204.19) 2 15.89 (1.37) 19.72 (2.41) 0.46 (0.11) 0.83 (0.03) 0.32 (0.04) 0.16 (0.02) 116.22 (32.73) 88.51 (38.55) 0.34 (0.07) 0.39 (0.06) 0.35 (0.09) 16.74 (1.30) 29.71 (4.73) 31.74 (2.78) 19.18 (2.34) 18.41 (2.83) 18.90 (2.57) 0.70 (0.11) 0.66 (0.13) 0.71 (0.11) 0.27 (0.07) 0.32 (0.06) 0.27 (0.07) 56.57 (21.59) 69.51 (19.44) 60.70 (22.39) Data fit (↓) 92 7.3.2 7.3.2.1 Experiments Setup Inverse Problems We test our methods on super-resolution, Gaussian deblurring, and box inpainting. For super-resolution, we consider 16× downsampling. For deblurring, we use a Gaussian blur kernel of 61×61 pixels and 𝜎 = 6. The box inpainting is done with a 256×256 pixel mask in the center of each test image. Each task is repeated with both a matching and non-matching text prompt, and measurement noise is assumed to be a zero-mean Gaussian with standard deviation 𝜎𝒚 = 0.01. Datasets and Metrics For validation and testing, we use 512×512 images from the ImageNet dataset [81]. The output images are evaluated using the peak signal-tonoise ratio (PSNR), structural similarity index measure (SSIM) [310], and Learned Perceptual Image Patch Similarity (LPIPS) [349] metrics. Note that the images are resized to 256 × 256 using bicubic interpolation before being input to LPIPS. We also report the “data fit” for each task, defined as ∥A (b 𝒙 ) − 𝒚∥ 2 where b 𝒙 is the reconstructed image, which quantifies how well the reconstruction is consistent with the measurement 𝒚. Likelihood Step We perform the likelihood step according to the table in Section Section 7.3.1.2. In sub-step 1, sampling is done using Hamiltonian Monte Carlo (DAPS, PnP-DM) and optimization is performed using the stochastic gradient descent optimizer with momentum (DCDP). See Table D.1 for the inverse problemspecific hyperparameters used in each of these. Sub-step 2 varies by method, but does not need to change for different inverse problems. Prior Step The prior step is performed with reverse diffusion using StableDiffusion 1.5 [249] with the DPMSolverMultistepScheduler. The number of steps in the scheduler is set to 𝐾 = 50 in order to match the 𝜂-schedule. Baselines We compare our methods with PSLD [251] and TReg [160]. The hyperparameters for these methods were selected per inverse problem through Bayesian optimization. TReg is supplied with the same text prompts as the other methods. PSLD is not given text prompts, as we found this generates significantly worse results. 93 7.3.2.2 Main Results The quantitative results are shown in Table 7.1. A few qualitative samples for each method can be found in Figure 7.2. Overall, we see that the methods based on our unified framework outperform the two baseline approaches, PSLD and TReg, especially for super-resolution and Gaussian deblurring. Additionally, these methods avoid the added complexity introduced by external components such as CLIP, which is required by TReg to function. We also observe that DAPS, PnP-DM, and DCDP work more robustly than TReg and PSLD. We see that the images recovered by PSLD tend to contain artifacts throughout the entire image. Additionally, as mentioned before, we were not able to use text prompts to guide PSLD, resulting in an inpainting result that does not contain “two teenagers" as specified in the prompt (in Figure 7.2). TReg performed well on deblurring, but struggled on box inpainting. Note that the original TReg paper used a diffusion model fine-tuned specifically for inpainting, while our models can perform inpainting with the same base StableDiffusion model used for all of the other tasks. Within the unified methods, the reconstructions produced by PnP-DM appear blurrier and contain fewer details than those from DAPS and DCDP. We hypothesize that this is because PnP-DM is the only method that does not add noise in sub-step 2 of the likelihood step. Without this noise-adding step, we find that the output of the likelihood step, 𝒗 (𝑘) , in PnP-DM tends to exhibit less noise than expected. As a result, the prior step over-denoises the image—removing too many features under the assumed noise standard deviation—leading to overly smooth and blurry reconstructions. This issue was less prominent for solving inverse problems with pixel-space DMs in Chapter 6. However, it becomes more significant in this latent-space setting, where the decoder in Equation (7.2) introduces a high degree of nonlinearity. 7.3.2.3 Discussions Through the unified framework, we can naturally control image generation via text guidance in a consistent manner. For convenience, we use DAPS as a representative to explore the following three aspects of the proposed approach. 2 Note that PSLD failed to converge on two of the test images for the Gaussian deblur test, resulting in a high data fit term. If these outliers were removed, the data fit term would be 15.92 (1.82). 94 Figure 7.3: Visual examples of DAPS on the 16× super-resolution task using different prompts. The examples show that varying descriptions lead to semantically different yet feasible reconstructions. Despite the severe degradation of the input measurements, the restored images are of high quality and closely align with the information from the provided text prompts, while still consistent with the original measurement. 95 Figure 7.4: Visualization of the image generation process of DAPS. The top row shows outputs from the prior steps (𝒖 (𝑘+1) for 𝑘 ∈ {9, 19, 29, 39}), which denoise and integrate text conditioning to guide the sample toward the target distribution, while the bottom row shows outputs from the likelihood steps (𝒗 (𝑘) for 𝑘 ∈ {9, 19, 29, 39}), which enforce data consistency with added noise. Initialization starts from pure noise, and the two processes alternate to progressively generate a realistic image. The images are visualized after decoding. Controllability with Text Guidance We first demonstrate its ability to reconstruct different image modes from significantly degraded measurements using both matching and non-matching text prompts, as shown in Figure 7.3. The matching text prompt was created as a simple description of the image, and the non-matching text prompt was created by making a slight modification to the matching text prompt. When there is enough ambiguity in the measurement, DAPS can generate detailed information that matches each of the given prompts. Sometimes this involves reinterpreting what the colors of the image represent: in the second row, the original image has a single cannoli, but when DAPS is run with a non-matching prompt, it is changed into two separate macarons. Different Prompt Specificity We then investigate how the specificity of text prompts influences the posterior samples generated by DAPS, where the results are shown in Figure 7.5. With highly specific prompts, such as “two scoops of ice cream with a cannoli” or “a plate of macarons” (top right), DAPS effectively reconstructs the corresponding image modes in a controllable manner. As we provide increasingly generic prompts, such as “a plate of desserts” (middle) or “a plate of food” (bottom), the generated samples exhibit roughly the same level of diversity in appearance—i.e., the bottom two panels appear to be equally diverse at first 96 Figure 7.5: Effect of prompt specificity on posterior samples generated by DAPS for 16× super-resolution. We repeatedly generate samples using DAPS with prompts of varying specificity. As prompts become more generic, the samples exhibit greater semantic diversity—e.g., several yellow scoops in the “a plate of food” panel look like mashed potatoes, which are not seen in the more specific “a plate of desserts” panel. However, the overall diversity in appearance remains similar—i.e., the bottom two panels appear to be equally diverse at first glance. Modes with highly specific details, such as “two scoops of ice cream with a cannoli” or “a plate of macarons” (top right), are only recovered when given the corresponding prompts. These results underscore the importance of text conditioning in uncovering rare modes and improving mode coverage in posterior estimation. 97 glance—but an increased level of semantic diversity. For example, several yellow scoops in the “a plate of food” panel look like mashed potatoes, which are not seen in the more specific “a plate of desserts” panel. However, less likely modes such as “a plate of macarons” are not recovered by the samples of the more generic prompts. This highlights the importance of text conditioning in discovering less likely solutions and improving posterior mode coverage. In addition, samples from the most generic prompt (“a plate of food”) tend to exhibit structurally incoherent elements, such as speckled or shredded textures in the brown and yellow objects. This is consistent with prior observations that conditional DMs often outperform unconditional ones, as conditioning reduces the complexity of the target distribution [16]. Hence, beyond improving mode coverage, text conditioning also enhances the quality of individual samples. Visualization of Image Generation Process We also visualize the image generation process in Figure 7.4 where the bottom row contains the inputs to the prior steps 𝒗 (𝑘) and the top row shows the outputs 𝒖 (𝑘+1) . The text prompt for this example is “A dog sitting on the grass.” In the early iterations where 𝜂 𝑘 is large, the 𝒗 iterates are noisier and the prior step is closer to text-conditioned image generation. As 𝜂 𝑘 anneals down to 0, the 𝒖-iterates become increasingly similar to the target image. 7.4 Thrust 2: Harnessing Higher Dimension for Video Inverse Problems Reconstructing high-quality videos from time-varying measurements is a core challenge in many scientific domains, such as black hole video reconstruction [92] and dynamic magnetic resonance imaging (MRI) [110]. In these applications, the accuracy of the recovered spatiotemporal features significantly affects downstream scientific analysis or medical interpretation. These problems are inherently difficult due to the high dimensionality of the underlying video and the severe loss of both spatial and temporal information during acquisition, necessitating a prior on both the spatial and temporal dimensions for meaningful recovery. However, the investigation in Chapter 6 was limited to image DMs. It remains an open question how to efficiently obtain DMs for videos and use them for solving video inverse problems (VIPs). In this section, we introduce an instantiation of Algorithm 2 for solving highdimensional scientific VIPs with SpatioTemporal video diffusion Priors (STeP). Existing methods for VIPs rely on extracting temporal consistency directly from measurements, which limits their effectiveness on scientific tasks with high spa- 98 tiotemporal uncertainty. We instead learn a spatiotemporal diffusion prior for videos. To do so efficiently, we are inspired by [303] to first train an image latent diffusion model with a 2D U-Net and then turn it into a spatiotemporal video diffusion model by adding a zero-initialized temporal module to each 2D convolution module in the U-Net. This enables us to fine-tune a spatiotemporal diffusion prior from a pre-trained IDM in a data- and time-efficient manner, using only hundreds to thousands of videos and a few hours of training on a single NVIDIA A100 GPU. STeP enforces learned priors over both spatial and temporal dimensions and does not rely on heuristics to ensure temporal consistency, making it well-suited for tasks with high spatiotemporal uncertainty. Due to its plug-and-play nature, STeP can handle general inverse problems with nonlinear forward models without the need for task-specific design or temporal heuristics to enforce temporal consistency. We demonstrate the effectiveness of STeP on two challenging scientific video inverse problems: black hole video reconstruction (previewed in Figure 7.6) and dynamic MRI, where the underlying targets exhibit significantly different spatiotemporal characteristics. As Figure 7.6 illustrates, STeP not only achieves state-of-the-art results with improved spatial and temporal consistency but also effectively captures the multi-modal nature of highly ill-posed problems, recovering diverse plausible solutions from the posterior distribution. Notably, our approach achieves substantial improvements in both spatial and temporal consistency, significantly outperforming baselines across most evaluation metrics. For example, our method demonstrates 1.57 dB and 2.38 dB improvements in PSNR on black hole video reconstruction and dynamic MRI, respectively. More importantly, for the challenging black hole video reconstruction, while baseline methods have difficulty accurately recovering temporal dynamics, our method demonstrates spatiotemporal structure that closely aligns with ground truth video (Figure 7.9), under extremely sparse measurement conditions. The appendix for this section is Appendix D.2. The code for the work presented in this section is available at https://github.com/zhangbingliang2019/STeP. 7.4.1 Background Existing Approaches for Video Inverse Problems Existing approaches for video inverse problems have primarily focused on restoration and editing problems, such as super-resolution and JPEG artifact removal, for natural videos [53, 77, 82, 168, 169, 365]. Since it is computationally expensive to obtain a well-trained video 99 Large amount of image data Physics of the inverse problem Diverse (multi-modal) and high-fidelity video reconstructions Selected frames over time y = A(x0 ) + n <latexit sha1_base64="1d9lc30kVe3D32Ml22Gbf8dEKnw=">AAAGBnicbdRLb9NAEAfwbWmghFcKRw5EREhFSFWCUDkhNUnfz6RtHm1dVWtnk7jx2o533SaNfETiuyBxouLKN+DMt8Fp8x+RtCN5PfubGcm+jOk7ttLZ7N+p6QcziYePZh8nnzx99vxFau5lVXlhYImK5TleUDe5Eo7tioq2tSPqfiC4NB1RMzvFYb12IQJle+6h7vviVPKWazdti+uYzlJvjL4p01/ShuS6bXFnkI/mjZ4pz7LvPxhu/E5lsgvZm0jfTXKjJMNGUTqbm/ljNDwrlMLVlsOVOsllfX064IG2LUdESSNUwudWh7fESZy6XAp1Orj5kyj9LpZGuukF8ePq9I3+PzHgUqm+NOPO4QerydoQ762JrsuDgPfvLZoyGr9PggwdbQfe5bhqboYOD3pRMmk0RNPgphwMjPjgURTdkgkyiSyQRdQANYgESBA1QU2iFqhF1Aa1iWyQTXQOOifqgDpEDsghkiBJ5IJcIg/kEfkgn6gL6hIFoIBIgRSRBmmiEBQSXYAuiC5Bl0Q9UI+oD+oTXYGuhnRreVie2gqgAlERVCRaBi0TrYBWiFZBq0RroDWiddA60QZog2gTtEm0Bdoi2gZtE+2Adoh2QbtEe6A9ohKoRFQGlYn2QftEB6ADokPQIVEFVCGqgqpENVCNqA6qEx2BjoiOQccxxSswN7nw7ibVjwu5xYXF8qfMUmG0DGfZa/aWzbMc+8yW2DorsQqz2Ff2nf1k14lviR+J68Sv29bpqdHMKzYWid//ADsIL2c=</latexit> Noise (e.g., Gaussian OR non-Gaussian) Limited amount of video data Spatiotemporal diffusion prior Mode ① (≈30%) (d-PSNR = 40.43) Mode ② (≈60%) (d-PSNR = 37.48) Higher d-PSNR Video diffusion inverse solver Equal data delity Forward model (e.g., black hole imaging OR MR imaging) Avg. optical ow Ground truth Mode ③ (≈10%) (d-PSNR = 34.65) fifl Figure 7.6: An overview of our proposed framework with spatiotemporal diffusion priors, STeP, for scientific video inverse problems. Left: STeP combines the physics model of the target problem with a spatiotemporal diffusion prior that directly characterizes the video distribution. We show that such a prior can be efficiently obtained by fine-tuning a pre-trained image diffusion model with limited video data. Right: STeP can generate diverse solutions to a black hole video reconstruction problem that exhibit equally good fidelity with the measurements. diffusion model (VDM) for natural videos, existing methods instead rely on an image diffusion model (IDM) to process each video frame and propose various techniques to enforce temporal consistency. For example, the batch-consistent sampling (BCS) technique fixes the injected noise across all video frames [168, 169]. Another common approach, which we refer to as noise warping, first extracts optical flow from measurements and uses it to warp the injected noise across all video frames [53, 77, 82, 334]. It is observed empirically that the dynamic of the injected noise translates into that of the generated video. These methods work well in restoration problems where the dynamic is mostly preserved in the measurements or video editing tasks where the dynamic is given. However, these methods face challenges when dealing with more challenging problems in scientific domains because the measurements may belong to a different domain that is nontrivial to invert, or substantial spatiotemporal information may be lost in the measurement process. Plug-and-Play Diffusion Priors for Video Inverse Problems As reviewed in Section 5.3.2, plug-and-play diffusion priors (PnPDP) constitute a family of methods that leverage diffusion models as priors for solving inverse problems. A major advantage of PnPDP is its ability to handle inverse problems in a general way without task-specific design. Video inverse problems fit into the PnPDP framework in principle, but existing PnPDP methods have mainly focused on the static image 100 Prior works Spatial (image) UNet Per-frame processing Input latent noise Output video / latent High-res. video Decode if in latent space D <latexit sha1_base64="jcvHCVPFVk4MNpW/cCw6JKT6SgA=">AAAF8HicbdTJThtBEAbghuCEOBskx1ysoEg5ITuK4Mq+L2bxAhihnnHbDJ6N7h7AWPMYkXIKyjUvk3PeJmPw/yssJXWr+qsqqU/lxL5nbLH4d2j42Uju+YvRl/lXr9+8fTc2/r5qokS7quJGfqTrjjTK90JVsZ71VT3WSgaOr2pOZ75fr10obbwo3LfdWB0Hsh16Lc+VNqOjRiDtqSv93kJ6MjZRnCzeRuFxUhokE2IQ5ZPxkT+NZuQmgQqt60tjjkrF2B73pLae66s030iMiqXbkW11lKWhDJQ57t3+OS18zqRZaEU6O6Et3Or/Ez0ZGNMNnKyz/0fzsNbHJ2vqPJRay+6TRSdI778fQpD41tPR5X210kl8qa/SfL7RVK2GdIJer5FdMk3TO3JADskFuaQmqElSIEVqgVqkNqhNOgWdkjyQRzoDnZE6oA7JB/mkABSQQlBIikARKQbFpHPQOUmDNMmADMmCLCkBJaQL0AXpEnRJugJdkbqgLukadN2nO5uFzbJtDjRHmgfNkxZAC6RF0CJpCbREWgYtk1ZAK6RV0CppDbRGWgetkzZAG6RN0CZpC7RF2gZtk8qgMmkHtEPaBe2S9kB7pH3QPqkCqpCqoCqpBqqR6qA66QB0QDoEHWaUrcDSw4X3OKl+nSxNTU7tfJuYmRssw1HxUXwSX0RJTIsZsSLKoiJcEYnv4qe4yencj9xN7tdd6/DQYOaDuBe53/8ASOAoCQ==</latexit> Implicitly enforced by correlated noise OR optical flow information Identical OR correlated i.i.d Gaussian Spatiotemporal modules ✕ N steps Output latent Measurement consistency Spatiotemporal UNet Whole-video processing Ours Temporal consistency Measurement consistency Input noise OPTIONAL Decoder High-res. video D <latexit sha1_base64="jcvHCVPFVk4MNpW/cCw6JKT6SgA=">AAAF8HicbdTJThtBEAbghuCEOBskx1ysoEg5ITuK4Mq+L2bxAhihnnHbDJ6N7h7AWPMYkXIKyjUvk3PeJmPw/yssJXWr+qsqqU/lxL5nbLH4d2j42Uju+YvRl/lXr9+8fTc2/r5qokS7quJGfqTrjjTK90JVsZ71VT3WSgaOr2pOZ75fr10obbwo3LfdWB0Hsh16Lc+VNqOjRiDtqSv93kJ6MjZRnCzeRuFxUhokE2IQ5ZPxkT+NZuQmgQqt60tjjkrF2B73pLae66s030iMiqXbkW11lKWhDJQ57t3+OS18zqRZaEU6O6Et3Or/Ez0ZGNMNnKyz/0fzsNbHJ2vqPJRay+6TRSdI778fQpD41tPR5X210kl8qa/SfL7RVK2GdIJer5FdMk3TO3JADskFuaQmqElSIEVqgVqkNqhNOgWdkjyQRzoDnZE6oA7JB/mkABSQQlBIikARKQbFpHPQOUmDNMmADMmCLCkBJaQL0AXpEnRJugJdkbqgLukadN2nO5uFzbJtDjRHmgfNkxZAC6RF0CJpCbREWgYtk1ZAK6RV0CppDbRGWgetkzZAG6RN0CZpC7RF2gZtk8qgMmkHtEPaBe2S9kB7pH3QPqkCqpCqoCqpBqqR6qA66QB0QDoEHWaUrcDSw4X3OKl+nSxNTU7tfJuYmRssw1HxUXwSX0RJTIsZsSLKoiJcEYnv4qe4yencj9xN7tdd6/DQYOaDuBe53/8ASOAoCQ==</latexit> ✕ N steps Figure 7.7: A schematic comparison between prior works (top) and our STeP framework (bottom) for video inverse problems. The bold texts highlight the key differences between them. While prior works use an image diffusion model and enforce temporal consistency using simple heuristics or warping noise with optical flow, we directly learn a spatiotemporal diffusion prior. setting. The main reason is that characterizing the prior distribution 𝑝(𝒙0 ) for videos is challenging due to their high dimensionality and a potentially limited number of samples for training. Prior work [167] has explored the applicability of PnPDP to an optical scattering problem with a pixel-space video diffusion prior. Video diffusion models are commonly believed to be hard to train due to the computational overhead of 3D modules and the requirement of a large video dataset [24, 25, 133, 350, 358]. Many recent methods tend to solve video modeling by fine-tuning from a pre-trained IDM with a video dataset [24, 25, 317]. In the following sections, we propose a recipe for efficiently obtaining spatiotemporal diffusion prior and solving VIPs with an instantiation of Algorithm 2. 7.4.2 Instantiation of Algorithm 2 with Spatiotemporal Diffusion Priors Instead of taking a per-frame processing approach using IDMs, we propose to directly learn the video distribution 𝑝(𝒙 0 ) from data (Section 7.4.2.1) and draw samples from the posterior distribution 𝑝(𝒙0 | 𝒚) ∝ 𝑝(𝒚 | 𝒙0 ) 𝑝(𝒙0 ) via an instantiation of Algorithm 2 (Section 7.4.2.2). Figure 7.7 illustrates the conceptual difference between STeP and the prior approaches on video inverse problems. Designed mainly for restoration problems on natural videos, prior works rely on IDMs to incorporate spatial prior and enforce temporal consistency by injecting correlated noise in the diffusion process. STeP instead adopts a whole-video formulation and simultaneously handles spatial and temporal dimensions within a PnPDP framework. This enables us to deal with more general problems in which the extraction 101 Input feature † Spatiotemporal module Output feature fin fout <latexit sha1_base64="wm0nY7CDa872X8dE/XgqRlMK8hk=">AAAIR3icjdRJT9tAFAdw49KGphu0x16sIqQgIZRUFb2y70tYsgAO1tgZJybe8IxJguWvV6mnnvsdKvVU1GPH4PeyQEQtxZr5vf+MZyJ7dN+2GM/nf47Jz8afv8hMvMy+ev3m7bvJqfdl5oWBQUuGZ3tBVSeM2pZLS9ziNq36ASWObtOK3lpJ6pVrGjDLc09416c1hzRcy7QMwgVpU7I2o5iaymmHR5Ybq5arqA7hTV2PjuKLyNVMReWWQ5niasYF5nrY7DXbcbY3mRfy/5stCY6eLu2pcyqx/Sa5iNQ5RSWMJ7WB6Z5Ymoifi/xcfzLpNJNbu/agnKxqqJ5LZhSd/ilmk7KZDu8tIU0+WlSNuseTBDRGxHAXvU1AMDd6J7PD1cGNiLKSe3oxaQw3MfCgEVV4kCj3vVBZbXI6P5+/u5SHjULamJbSq6hNjf9Q654ROtTlhk0YOy/kfV6LSMAtw6ZxVg0Z9YnRIg16LpouEf9OLbr7FGJlRkhdMb1A/Fyu3Gn/iIg4jHUdXSSTF5MN1xJ8tEavXBIEpPtoUXfiwf4wOKHNrcBrDyonemiToBNns2qdmirRnShSxY3EcXxPOpCOZAAZSHWgOhIFokgmkInUAGogNYGaSBaQhXQJdInUAmoh2UA2kgPkILlALpIH5CH5QD7SFdAVUgAUIDEghsSBOFIIFCJdA10jtYHaSB2gDlIXqIt0A3ST0L0tgS1hbBloGWkFaAVpFWgVaQ1oDWkdaB1pA2gDaRNoE2kLaAtpG2gbaQdoB2kXaBdpD2gPaR9oH+kA6ACpCFREOgQ6RDoCOkI6BjpGOgE6QSoBlZDKQGWkClAFqQpURToFOkU6AzoTJI7AwvCB97BR/jxfWJhfOPwyvbicHoYT0kfpk5STCtJXaVHalIpSSTLkb/Iv+Y98m/me+Z25zfy9j8pj6ZgP0sA1MfYPqJf81g==</latexit> <latexit sha1_base64="3kZ8lVDgFZA+LRSeq/7EHbH1TqQ=">AAAISHicjdRJT9tAFAdw49KGphu0x16sIqQgIZSgil7Z9yUsWQAn6dgZJybe8IxJguXPV6mXnvshKvVU1FvH4PeyQEQtxZr5vf+MZyJ7NM8yGc9mf47Jz8afv0hNvEy/ev3m7bvJqfdF5ga+Tgu6a7l+WSOMWqZDC9zkFi17PiW2ZtGS1lqN66Vr6jPTdU5516MVmzQc0zB1wgXVpuSvM4pRUznt8NB0ItV0FNUmvKlp4XFUDZ2aoajctClTnJpexVwPm71mO0r3JnMD/n+zxcHR0yU9dU4lltck1VCdU1TCeFwbmO6JpYn4hcjP9SfjTjO+tSsPyvGqhuqZeEbR6Z9iNi4byfDeEpLko0VVr7s8TkBjRAx30dsEBDOjdzI7XB3ciCgrmacXk8RwEwMPGlGFB4ly/zuQrk1OZ+ezd5fysJFLGtNScuVrU+M/1LqrBzZ1uG4Rxi5yWY9XQuJzU7dolFYDRj2it0iDXoimQ8TfUwnvvoVImRFSVwzXFz+HK3faPyIkNmNdWxPJ+M1kw7UYH63RK4f4Puk+WtTsaLA/DHZgcdN324PKiRZYxO9E6bRap4ZKNDsMVXEjURTdkwakIelAOlIdqI5EgSiSAWQgNYAaSE2gJpIJZCJdAl0itYBaSBaQhWQD2UgOkIPkArlIHpCHdAV0heQD+UgMiCFxII4UAAVI10DXSG2gNlIHqIPUBeoi3QDdxHRvy2DLGFsBWkFaBVpFWgNaQ1oHWkfaANpA2gTaRNoC2kLaBtpG2gHaQdoF2kXaA9pD2gfaRzoAOkA6BDpEygPlkY6AjpCOgY6RToBOkE6BTpEKQAWkIlARqQRUQioDlZHOgM6QzoHOBYkjMDd84D1sFBfmc4vzi0efp5dWksNwQvoofZIyUk76Ii1JW1JeKki6/E3+Jf+Rb1PfU79Tt6m/91F5LBnzQRq4JuR/ckr9YQ==</latexit> Input shape <latexit sha1_base64="yUcO3/UtXCkH0xFp4N5Vd/QiQBk=">AAAH8XicjdTLTttAFAZg49IG0hu0y26sIqQgIZSgim4h4X4Nl1wAh2jsjBMTe2w8Y5Jg+TUqdVXUbR+m675NbeJzQgKUjmTrzHf+Gc1IljXXMrnIZv+MyS/GX75KTUymX795++791PSHMnd8T6cl3bEcr6oRTi2T0ZIwhUWrrkeJrVm0orULcb9yTT1uOuxE9Fxas0mTmYapExFRfVqenFWMuipoVwQmC1WTKapNREvTgqPwImB1Q1GFaVOusLp+gbkBtgZlJ0wPNnN88X+7xcGnt0tm6rxKLLdFIhna5JkDRfHzKD9/PxlPWvGrU3vQjs8y0s/EO0aT+1vMxW0jWT44QpJ8tKnqDUfECSieiOEtBpdIgpmnLzKXns384x799nNHmatPzWQXsndDeVjkkmJGSkaxPj3+W204um9TJnSLcH6ey7qiFhBPmLpFw7Tqc+oSvU2a9DwqGYluVAvuPtpQmY2koRiOFz1MKHd6f0VAbM57thYl40+Ij/ZifLRHrxjxPNJ7tKnZ4fB8FGzfEqbndIZVEM23iNcN02m1QQ2VaHYQqNGLhGHYJw1IQ9KBdKQGUAOJAlEkA8hAagI1kVpALSQTyES6BLpEagO1kSwgC8kGspEYEENygBwkF8hFugK6QvKAPCQOxJEEkEDygXyka6BrpA5QB6kL1EXqAfWQboBuYurbCtgKxvJAeaQCUAFpFWgVaQ1oDWkdaB1pA2gDaRNoE2kLaAtpG2gbaQdoB2kXaBdpD2gPaR9oH+kA6ACpCFREOgQ6RDoCOkI6BjpGOgE6QSoBlZDKQGWkClAFqQpURToFOkU6AzqLKPoF5kZ/eA+L8uJCbmlh6fDLzHI++RlOSJ+kz1JGyklfpWVpUypKJUmXXfmb/EO+TfHU99Rt6mc/Ko8laz5KQyP16y/MEt3G</latexit> (nf , nin , nh , nw ) Output shape Spatial module conv2d ON/OFF <latexit sha1_base64="gxP166q8e2hPfO1joxd56qeQwHc=">AAAH8XicjdRJT9tAFAdw49IG0g3aYy9WEVIiIZSgil7Z9yUsWQCHaOyMkyHe8IxJguWvUamnol77YXrut6kd/F5IAqUj2Xrze/8ZzUiWNddkXORyf8bkF+MvX6UmJtOv37x9935q+kOJO76n06LumI5X0QinJrNpUTBh0orrUWJpJi1rrdW4X76hHmeOfSq6Lq1apGEzg+lERFSblidnFaOmCtoRAbNDldmKahHR1LTgOLwM7JqhqIJZlCt2Tb/EXB+b/bIdpvubOb74v93i4NPbJTN1TiWm2ySRDGzyzIGi+EWUn3uYjCfN+NWujrTjswz1M/GO0eThFtm4bSTL+0dIko82Vb3uiDgBxRMxvEX/EhDMPH2TbDrzj3tke2ufO0q2NjWTm8/1hjJa5JNiRkpGoTY9/lutO7pvUVvoJuH8Ip9zRTUgnmC6ScO06nPqEr1FGvQiKm0S3aga9D7aUJmNpK4Yjhc9tlB6+nBFQCzOu5YWJeNPiA/3Yny0R69t4nmk+2hTs8LB+TBYvimY57QHVRDNN4nXCdNptU4NlWhWEKjRi4RheE8akIakA+lIdaA6EgWiSAaQgdQAaiA1gZpIDIghXQFdIbWAWkgmkIlkAVlINpCN5AA5SC6Qi3QNdI3kAXlIHIgjCSCB5AP5SDdAN0htoDZSB6iD1AXqIt0C3cZ0b8tgyxhbAVpBWgVaRVoDWkNaB1pH2gDaQNoE2kTaAtpC2gbaRtoB2kHaBdpF2gPaQ9oH2kc6ADpAOgQ6RCoAFZCOgI6QjoGOkU6ATpBOgU6RikBFpBJQCakMVEaqAFWQzoDOkM6BziOKfoH54R/eaFFamM8vzi8efZlZWkl+hhPSJ+mzlJHy0ldpSdqSClJR0mVX/ib/kO9SPPU9dZf6eR+Vx5I1H6WBkfr1F8ab3cY=</latexit> (nf , nout , nh , nw ) fspat ⇥ (1 <latexit sha1_base64="z7wwB8APZjET+D6w5JnDDS5zs28=">AAAGBnicbdTJbhNBEAbgToghmM2BIwcsIqRwwLIjFK6JncXZ7cRbkrGsnnHbnng2T/d4iTVHJN4FiRMRV96AM2/D2PFfIktJ3ar+qlqtvpTuWaZU6fTfuflHC7HHTxafxp89f/HyVWLpdUW6gW+IsuFarl/TuRSW6YiyMpUlap4vuK1boqp3c5N6tS98abpOSY08Ubd52zFbpsFVRI3Eu1ZDU2KoxtLjKkxqyrSFTK5kPmnc8jr8YyOxnE6lp5G8n2RmyTKbRaGxtPBHa7pGYAtHGRaX8iKT9lR9zH1lGpYI41oghceNLm+Liyh1ePRgfTz9SZj8EEkz2XL9aDkqOdX/b4y5LeXI1qNOm6uOvFub4IM10XO47/PRg0XdDm+f74IdWMr03cFtVVwPLO4Pw3hca4qWxnV7PNaijYdheEM6SCcyQAZRE9QkEiBB1AK1iNqgNlEH1CEyQSbRJeiSqAvqElkgi8gG2UQOyCFyQS6RB/KIeqAekQ/yiSRIEimQIgpAAVEf1CcagAZEQ9CQaAQaEV2BriZ0YxuwDWrLgrJEOVCOaBO0SbQF2iLaBm0T7YB2iPKgPNEuaJdoD7RHtA/aJzoAHRAdgg6JjkBHRMegY6ICqEBUBBWJTkAnRKegU6ISqERUBpWJKqAKURVUJaqBakRnoDOic9B5RNEIzNwdePeTymoqs5ZaK35eXs/OhuEie8vesxWWYV/YOsuzAiszg31l39lPdh37FvsRu479ummdn5vdecNuRez3P1wjL/U=</latexit> ↵) ftemp ⇥ ↵ ⇤ <latexit sha1_base64="WJPwdTRHSfIRnZbDMzhOpuxaNec=">AAAISXicjdRJT9tAFAdwE2hD0w3aYy9WEVKQEEpQRa/s+xKWLMCEaOyMExNveMYkwfL3q9RDz/0SlXoq6qlj8HshgYhaijXze/8Zz0T2aJ5lcpHL/RxJjY69eJkef5V5/ebtu/cTkx9K3A18nRV113L9ikY5s0yHFYUpLFbxfEZtzWJlrbUS18vXzOem65yIrseqNm04pmHqVEiqTabotGrUiGAdEZpORExHJTYVTU0Lj6KL0KkZKhGmzbjq1PQLzPWw2Wu2o0xvMjcQ/zdbHBw+XdIjs4RaXpNehGRWJZSLuNY33TNLk/FzmZ99mIw7zfjWrj4qx6saqGfjGWXn4RQzcdlIhveWkCSfLBK97oo4AY0hMdxFbxMQzA7fycxgtX8jsqxmn19MEsNN9D1oSBUeJMvwDghme1GmNjGVm8vdXerjRj5pTCnJVahNjv0gdVcPbOYI3aKcn+dznqiG1BembrEoQwLOPKq3aIOdy6ZD5f9TDe8+hkidllJXDdeXP0eod/pwREhtzru2JpPxq8kHazE+WWNXDvV92n2yqNlRf38Q7MASpu+2+1VQLbCo34kyGVJnBqGaHYZE3mgURfekAWlIOpCOVAeqIzEghmQAGUgNoAZSE6iJZAKZSJdAl0gtoBaSBWQh2UA2kgPkILlALpIH5CFdAV0h+UA+EgfiSAJIIAVAAdI10DVSG6iN1AHqIHWBukg3QDcx3dsS2BLGloGWkVaAVpBWgVaR1oDWkNaB1pE2gDaQNoE2kbaAtpC2gbaRdoB2kHaBdpH2gPaQ9oH2kQ6ADpAKQAWkQ6BDpCOgI6RjoGOkE6ATpCJQEakEVEIqA5WRKkAVpFOgU6QzoDNJ8gjMDx54jxul+bn8wtzC4ZepxeXkMBxXPimflaySV74qi8qmUlCKip76lvqV+pO6TX9P/07fpv/eR1MjyZiPSt81PvoPezr9yQ==</latexit> <latexit sha1_base64="YZ39ywcMtrr9aIVGMGYOYDRqTBQ=">AAAH+3icjdRJT9tAFABg49KGphuUYy9WEVKQEEqqil7Z9yUsWQCHaOyMExN7bDxjkmD5v1TqqajX/o6e+286Br8XEkB0pFhvvvfmeUaxxvAdm4t8/u+I+mL05avM2Ovsm7fv3n8Yn/hY5l4YmLRkeo4XVA3CqWMzWhK2cGjVDyhxDYdWjPZykq9c0YDbHjsWPZ/WXNJktmWbREiqT6iT05pV1wXtishmsW4zTXeJaBlGdBifR6xuabqwXco1VjfPsa6PrX7YibP9Zl4o/q9bUvhUuzTWZ3Xi+C1yHumzmk64SF400OyZjcnyM1k/e78ymbSSR6f2IJ3saSifSzrKyf0WM0naSpf3t5BWPprUzYYnkgoInijDU/QPAYW5p08yM5wdPIhMa7nnNzNTH5/Kz+Vvh/YwKKTBlJKOYn1i9I/e8MzQpUyYDuH8rJD3RS0igbBNh8o/MuTUJ2abNOmZDBmRZ6pFt59vrE1LaWiWF8gfE9qt3l8REZfznmvIyuRj4sO5BB/N0UtGgoD0Hk0abjw4HwY3dIQdeJ1BFcQIHRJ042xWb1BLJ4YbRbp8kDiO78gAMpBMIBOpAdRAokAUyQKykJpATaQWUAvJBrKRLoAukNpAbSQHyEFygVwkBsSQPCAPyQfykS6BLpECoACJA3EkASSQQqAQ6QroCqkD1EHqAnWRekA9pGug64TubBFsEcuWgJaQloGWkVaAVpBWgVaR1oDWkNaB1pE2gDaQNoE2kbaAtpC2gbaRdoB2kHaBdpH2gPaQ9oH2kYpARaQDoAOkQ6BDpCOgI6RjoGOkElAJqQxURqoAVZCqQFWkE6ATpFOgU0nyCiwMX3gPg/KXucL83PzB16mFpfQyHFM+KZ+VnFJQvikLyoZSVEqKqV6r39Wf6k0mzvzI3GR+3ZWqI+maSWVgZH7/A4zy4cw=</latexit> Temporal module Reshape to <latexit sha1_base64="bRYljDlBtzZ7c6X05t9aRO09WnU=">AAAIGnicjdRJT9tAFABgk9KGphu0x16sIqQgIZSgil7Z9yUsWYAJ0dgZJybe8IxJguUfUqn/pVJPRT300kv/Tcfg97IAoiPFevO9Ny/zJMuaZ5lc5HJ/R1LPRp+/SI+9zLx6/ebtu/GJ9yXuBr7OirpruX5Fo5xZpsOKwhQWq3g+o7ZmsbLWWo7z5Svmc9N1jkXXY1WbNhzTMHUqJNUmUnNTqlEjgnVEaDoRMR2V2FQ0NS08jM5Dp2aoRJg246pT08+xrofNXtiOMr1mbiD+r1tc+Hi7ZEdmCLW8Jj0PyYxKKBdxbqDdE1eT5Weyfqa/Mt4040e7ei8d32oon407yk1/i+k4bSTHe1dIKh9MEr3uirgCgkfKcIreEFCYfXyS6eHs4CAyrWafvsx0Jts3wsDfyFxtfDI3m7td6v0gnwSTSrIKtYnR36Tu6oHNHKFblPOzfM4T1ZD6wtQtFmVIwJlH9RZtsDMZOlQOXA1v3+5InZJSVw3Xlz9HqLfafyKkNuddW5OV8bvGh3MxPphjlw71fdp9MKnZ0eB+GOzAEqbvtgdVUC2wqN+JMhlSZwahmh2GRD5oFEV3pAFpSDqQjlQHqiMxIIZkABlIDaAGUhOoiWQCmUgXQBdILaAWkgVkIdlANpID5CC5QC6SB+QhXQJdIvlAPhIH4kgCSCAFQAHSFdAVUhuojdQB6iB1gbpI10DXMd3ZItgili0BLSEtAy0jrQCtIK0CrSKtAa0hrQOtI20AbSBtAm0ibQFtIW0DbSPtAO0g7QLtIu0B7SHtA+0jFYAKSAdAB0iHQIdIR0BHSMdAx0hFoCJSCaiEVAYqI1WAKkgnQCdIp0CnkuQnMD/8wbsflOZm8/Oz8wefJxeWko/hmPJR+aRklbzyRVlQNpSCUlT01NfU99TP1E36W/pH+ib96640NZKc+aAMrPSff6Dk7M0=</latexit> (nh nw , nin , nf ) † Broadcasted to conv1d nf frames for image input <latexit sha1_base64="uRYu2ikS+Q4Z8I6Kr/G97819e84=">AAAIAXicjdRJT9tAFABgk9KGphu0p6oXqwgpSAglVUWv7PsSliyQCdHYGScm3vCMSYJlqVL/S6Weinrtj+i5/6Zj8HshAURH8ujN9948zUjWaJ5lcpHL/R1JPRl9+iw99jzz4uWr12/GJ96WuBv4OivqruX6FY1yZpkOKwpTWKzi+YzamsXKWnspzpcvmM9N1zkSPY/VbNp0TMPUqZBUn0i9n1KNOhGsK0LTiYjpqMSmoqVp4UF0Gjp1QyXCtBlXnbp+inV9bPXDTpTpN3MD8X/d4sKH2yUrMkOo5bXoaUhmVEK5iHMD7R45miyvyvqZ25XxohVPndqddHyqoXw27igXt1tMx2kj2d4/QlJ5b5LoDVfEFRA8UIa36F8CCrMP32R6ODt4EZlWs48fZjojp/r4ZG42dz3Uu0E+CSaVZBTqE6N/SMPVA5s5Qrco59V8zhO1kPrC1C0WZUjAmUf1Nm2yqgwdKu9VC69/4kidktJQDdeXnyPUa729I6Q25z1bk5XxL8WHczHem2PnDvV92rs3qdnR4HoY7MASpu92BlVQLbCo340yGdJgBqGaHYZETjSKohvSgDQkHUhHagA1kBgQQzKADKQmUBOpBdRCMoFMpDOgM6Q2UBvJArKQbCAbyQFykFwgF8kD8pDOgc6RfCAfiQNxJAEkkAKgAOkC6AKpA9RB6gJ1kXpAPaRLoMuYbmwBbAHLFoEWkZaAlpCWgZaRVoBWkFaBVpHWgNaQ1oHWkTaANpA2gTaRtoC2kLaBtpF2gHaQdoF2kfaA9pAKQAWkfaB9pAOgA6RDoEOkI6AjpCJQEakEVEIqA5WRKkAVpGOgY6QToBNJ8gnMDz94d4PSp9n83Ozc/ufJ+cXkMRxTPigflaySV74o88q6UlCKip76mvqe+pm6Sn9L/0hfpX/dlKZGkj3vlIGR/v0Ps1zjig==</latexit> Reshape to output shape <latexit sha1_base64="imrhv0p14cgkD0FzJzZoeP+gsxM=">AAAIOXicjdRbT9swFAfw0LGVdTfYHvcSDSEVCaF2mtgr9/ulBXphuFRO6rShuRE7tCXK95q0D7HnvUza09Be9wXmQM4pbUHMUiP79z927Cqy5lkmF7ncj7HUk/Gnz9ITzzMvXr56/WZy6m2Zu4Gvs5LuWq5f1ShnlumwkjCFxaqez6itWayitVfivHLJfG66zrHoeaxm06ZjGqZOhaT6VKo4oxp1IlhXhKYTEdNRiU1FS9PCw+gsdOqGSoRpM646df0M6/rY6nc7Uaa/mBuI/1stLnx4uWRE5gi1vBY9C8mcSigXcTaw3CNbk+Wnsn7ubmU8aMWPTm0kjnc1lGfjFeXg7hKzcWwk0/tbSCrvDYnecEVcAZ0HyvAU/UNAYfbhk8wOp4MHkbGafXwzSRkeYuBFMh3N4DUyrE9O5+ZzN00d7eSTzrSStEJ9avw7abh6YDNH6Bbl/DSf80QtpL4wdYtFGRJw5lG9TZvsVHYdKv+PWnjz8UfqjJSGari+/DlCvdG7M0Jqc96zNVkZf4p8OIvx3oxdONT3ae/eULOjwfEw2IElTN/tDKqgWmBRvxtlMqTBDEI1OwyJfNAoim5JA9KQdCAdqQHUQGJADMkAMpCaQE2kFlALyQQykc6BzpHaQG0kC8hCsoFsJAfIQXKBXCQPyEO6ALpA8oF8JA7EkQSQQAqAAqRLoEukDlAHqQvUReoB9ZCugK5iurUlsCUsWwZaRloBWkFaBVpFWgNaQ1oHWkfaANpA2gTaRNoC2kLaBtpG2gHaQdoF2kXaA9pD2gfaRzoAOkAqABWQikBFpEOgQ6QjoCOkY6BjpBJQCakMVEaqAFWQqkBVpBOgE6QvQF8kySswP3zhjXbKH+fzC/MLxU/Ti8vJZTihvFc+KFklr3xWFpVNpaCUFD31NfUz9Tt1nf6W/pW+Tv+5LU2NJXPeKQMt/fcfGgv4RQ==</latexit> from (nh nw , nout , nf ) * Trainable and initialized at 0 Figure 7.8: Architecture of the spatiotemporal module. Given a pre-trained image diffusion U-Net, we add a zero-initialized temporal module with an ON/OFF switch to each 2D spatial module and initialize the additive weight 𝛼 to zero. Thus, it will have no effect at the start of fine-tuning and gradually learn from the video training data. The number of frames, height, and width are denoted by 𝑛 𝑓 , 𝑛 ℎ , and 𝑛𝑤 , respectively. The numbers of channels for input features ( 𝑓in ) and output features ( 𝑓out ) are denoted by 𝑛in and 𝑛out , respectively. of temporal information directly from measurements is hard. 7.4.2.1 Efficient Training of Spatiotemporal Diffusion Priors We propose to obtain spatiotemporal diffusion priors for scientific video inverse problems efficiently via the following three steps. Step 1: Training a latent diffusion model (LDM) as an image prior We start by training a VAE [163] using the standard ℓ1 reconstruction loss with a scaled KL divergence loss on an image dataset. The KL divergence scaling factor is set to much less than 1 to prevent excessive regularization of the latent space. This allows us to obtain an image encoder E and decoder D. Once they are trained, we fix their parameters and train a 2D U-Net model 𝒔𝜽 (𝒛𝑡 ; 𝜎𝑡 ) using the standard denoising score matching loss [292]. Despite recent progress in 3D spatiotemporal encoders and decoders [55, 319, 332], we opt for a 2D spatial encoder and decoder to process each frame independently. This choice is due to efficiency considerations, as the decoder D will be called repeatedly during inference. Step 2: Turning an image prior into a spatiotemporal prior Upon obtaining an LDM as an image prior, we use a spatiotemporal U-Net architecture to parameterize 102 the time-dependent video score function, i.e., 𝒔𝜽 (𝒛𝑡 ; 𝜎𝑡 ) ≈ ∇ 𝒛𝑡 log 𝑝 𝑡 (𝒛𝑡 ; 𝜎𝑡 ), leveraging recent advancements in video generation [133, 303]. The key component in the architecture is the spatiotemporal module for 3D modeling, as illustrated in Figure 7.8. Given a pre-trained IDM with a 2D U-Net, we introduce a zero-initialized temporal module for each 2D spatial module within the U-Net. Specifically, for an input feature 𝑓in , let 𝑓out be the output of the spatiotemporal module, with 𝑓spat and 𝑓temp representing the outputs of the spatial and temporal branches, respectively. These features are combined using an 𝛼-blending mechanism: 𝑓out = (1 − 𝛼) · 𝑓spat + 𝛼 · 𝑓temp , (7.7) where 𝛼 ∈ R is a learnable parameter initialized as 0 in each spatiotemporal module. This design allows us to inherit the weights of the 2D spatial modules from the pretrained IDM, significantly reducing training time. Additionally, by factorizing the 3D module into a 2D spatial module and a 1D temporal module, the spatiotemporal U-Net has marginal computational overhead compared to the original 2D U-Net, striking a good balance between model capacity and efficiency. Step 3: Image-video joint fine-tuning To further improve performance and be compatible with both image and video inputs, we introduce an ON/OFF switch signal in the spatiotemporal module. When the switch is set to OFF (indicating image inputs), the temporal module is disabled (or equivalently 𝛼 = 0). This ensures that 𝑓out = 𝑓spat and reduces the spatiotemporal module to the original 2D spatial module, which processes each frame independently. During training, we initialize the weights of the spatial modules based on a pre-trained image diffusion model and fine-tune all parameters of the spatiotemporal U-Net using both image and video data. During fine-tuning, the model receives video data with probability 𝑝 joint ∈ [0, 1] and receives images (with switch set to OFF accordingly) with probability 1 − 𝑝 joint . The probability 𝑝 joint is a tunable hyperparameter controlling the proportion of real video data in training. Pseudo-video regularization helps the spatiotemporal U-Net retain the spatial capabilities of the initialized spatial UNet. This strategy, proven effective in previous work [303], stabilizes training and prevents overfitting to the video dataset. 7.4.2.2 Likelihood and Prior Steps As shown in Section 7.3.1, DCDP [181], PnP-DM (Chapter 6), and DAPS [341] share the same prior step and have slight differences in the likelihood step. We 103 find that STeP works the best with the DAPS instantiation due to its better compatibility with latent diffusion models. See the last paragraph of Section 7.3.2.2 for a discussion on this. For the likelihood step (Line 2 of Algorithm 2), STeP first samples ! 2 1 1 e 𝒗 (𝑘) ∼ exp − 2 ∥A (D (𝒗)) − 𝒚∥ 22 − 2 𝒖 (𝑘) − 𝒗 /𝑍 (7.8) 2 2𝜎𝒚 2𝜂 𝑘 where 𝑍 is a normalizing constant, and then samples 𝒗 (𝑘) ∼ N (e 𝒗 (𝑘) , 𝜂2𝑘 𝑰). (7.9) For the prior step (Line 3 of Algorithm 2), STeP samples 𝒖 (𝑘+1) from the following distribution ! 2 1 /𝑍 (7.10) 𝒖 (𝑘+1) ∼ exp log 𝑝(𝒖) − 2 𝒖 − 𝒗 (𝑘) 2 2𝜂 𝑘 where 𝑍 is a normalizing constant. As shown in Section 6.3, this sampling task is equivalent to solving a Gaussian denoising problem and can be implemented rigorously using the EDM framework given a pre-trained DM for 𝑝(𝒖) [151]. The pseudocode and more technical details of the STeP with DAPS are provided in Appendix D.2.1. 7.4.3 7.4.3.1 Experiments Tasks and Setup We consider two scientific video inverse problems: black hole video reconstruction [91] and dynamic MRI [110]. Although both are scientific imaging tasks, they have significantly different characteristics. Black hole video reconstruction involves simple spatial structures (usually a ring structure) but complex temporal dynamics that obey certain physical constraints. In contrast, dynamic MRI requires higher spatial fidelity with relatively simpler temporal dynamics, such as periodic heartbeat motion. Black Hole Video Reconstruction We consider the problem of observing the Sagittarius A* black hole using the Event Horizon Telescope (EHT) array in 2017 [91]. The measurements consist of sparse vectors with a dimensionality of 1,856 derived from an underlying black hole video comprising 64 frames, each with a spatial resolution of 256×256 pixels, capturing the black hole’s dynamics during the 100-minute observation period. These measurements are given by calculating 104 the closure quantities based on the complex visibilities (a detailed description of the problem is available in Appendix D.2.2.1). For the training dataset, we employ the General Relativistic MagnetoHydroDynamic (GRMHD) simulations [316] of Sagittarius A*, covering various black hole models and observational conditions. The dataset comprises 648 simulated black hole videos and 50,000 black hole images. Dynamic MRI We consider a standard compressed sensing MRI setup with two acceleration scenarios: 8× acceleration with 12 auto-calibration signal (ACS) lines and 6× acceleration with 24 ACS lines. Further details are provided in Appendix D.2.2.2. We utilize the publicly available cardiac cine MRI dataset from the CMRxRecon Challenge 2023 [298] for training. This dataset includes 3,324 cardiac MRI sequences, each containing fully sampled, ECG-triggered 𝑘-space data from 300 patients, featuring various canonical cardiac imaging views. Each sequence is processed into a video consisting of 12 frames with a spatial resolution of 192×192 pixels. We construct the image dataset by extracting the individual frames from the videos, resulting in 39,888 images in total. 7.4.3.2 Baselines and Our Method Recall from Section 7.4.2 that our method leverages a learned spatiotemporal prior to avoid explicitly estimating temporal dynamics from measurements at inference time. To evaluate the effectiveness of this strategy, we compare our method to existing IDM-based approaches. Based on how temporal information is incorporated, we categorize these baseline methods into two groups as follows. Group 1: Simple Heuristics We consider two baselines: Batch Independent Sampling (BIS) and Batch Consistent Sampling (BCS) [168]. BIS reconstructs each frame independently, while BCS implicitly incorporates a static temporal prior to enforce consistency across frames. ∫ Group 2: Noise Warping We include another line of baselines: -noise [53, 82], and GP-Warp [77]. These baselines enforce temporal consistency by warping the noise using optical flow estimated directly from the measurements. Besides, we include more conventional warping strategies such as Bilinear, Bicubic, Nearest from work [53]. For the black hole video reconstruction task, a direct inversion from the sparse measurement vector is infeasible. Thus, to demonstrate the upper-bound performance of these methods, we leverage the ground truth video as an oracle to 105 derive the optical flow. For the dynamic MRI task, we utilize a naïve inversion obtained via inverse Fourier transformation to estimate the optical flow. We follow the methodology of [53], leveraging a pre-trained model [278] to extract optical flow. Ours: Two Variants of STeP We evaluate two variants differentiated by their spatiotemporal priors: STeP (video-only), which uses a spatiotemporal prior trained exclusively on video data, and STeP (image-video joint), which initializes with a pretrained IDM and undergoes joint image-video fine-tuning. We follow the procedure in Appendix D.2.3 to train our priors until convergence. The detailed training hyperparameters are summarized in Table D.3. To make a fair comparison, we run all the baselines with DAPS [341]. The detailed implementation of each baseline can be found at Appendix D.2.1 and Appendix D.2.2.3. 7.4.3.3 Metrics We evaluate our results on three aspects: (1) spatial similarity, (2) temporal consistency, and (3) measurement data fit. Spatial Spatial similarity is assessed by calculating Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) [310], and Learned Perceptual Image Patch Similarity (LPIPS) [349]. Metrics are computed per frame and averaged across all frames. We utilize implementations from piq [153], normalizing images to the range [0, 1]. For grayscale images, frames are replicated across three channels before computing LPIPS scores. Temporal Temporal consistency is quantified using delta-based PSNR (d-PSNR) and delta-based SSIM (d-SSIM), measuring the similarity of normalized differences between consecutive frames, with results averaged over all frames. Additionally, we compute the Fréchet Video Distance (FVD) [287] between the reconstructed videos and our test dataset to evaluate distributional similarity. Data Fit Lastly, we report measurement data fit using task-specific metrics. For the black hole video reconstruction task, we employ the unified average 𝝌˜ 2 statistic (defined in Equation (D.14)), where values closer to 1 indicate better data fidelity. For dynamic MRI, we measure data misfit by computing the mean squared error in measurement space. 106 Table 7.2: Quantitative results on black hole video reconstruction and dynamic MRI. We compare our method against baselines by reporting the mean and standard deviation (shown in parentheses) of selected evaluation metrics computed over 10 test videos (FVD is reported without standard deviation since it evaluates the set of 10 videos collectively). The results clearly demonstrate that by leveraging the spatiotemporal prior, STeP consistently achieves improvements in both spatial quality and temporal consistency relative to baseline methods. Tasks Black hole MRI (8×) 7.4.3.4 Methods PSNR (↑) SSIM (↑) LPIPS (↓) d-PSNR (↑) d-SSIM (↑) FVD (↓) Data Misfit (↓) BIS [168] BCS [168] Bilinear [53] Bicubic [53] Nearest [53] ∫ -noise [53, 82] GP-Warp [77] 23.79 (1.41) 27.66 (2.04) 26.11 (2.05) 25.68 (1.88) 25.29 (1.80) 24.90 (1.52) 23.98 (1.28) 0.718 (0.047) 0.816 (0.053) 0.718 (0.067) 0.730 (0.058) 0.754 (0.059) 0.745 (0.058) 0.721 (0.043) 0.179 (0.031) 0.124 (0.040) 0.151 (0.044) 0.163 (0.041) 0.164 (0.042) 0.163 (0.034) 0.176 (0.029) 29.26 (1.51) 41.71 (1.99) 33.14 (2.33) 31.70 (1.88) 30.76 (1.75) 31.87 (1.93) 29.21 (1.24) 0.938 (0.015) 0.979 (0.008) 0.958 (0.013) 0.952 (0.015) 0.943 (0.017) 0.945 (0.017) 0.938 (0.014) 1429.42 564.43 1335.95 1521.14 1171.07 1253.81 1395.15 1.719 (1.277) 1.426 (0.784) 1.384 (0.742) 1.736 (1.259) 1.691 (1.020) 1.655 (1.020) 1.721 (1.225) STeP (video only) STeP (image-video joint) 28.71 (1.81) 30.28 (2.71) 0.802 (0.079) 0.865 (0.063) 0.120 (0.041) 0.095 (0.039) 41.41 (2.39) 42.09 (2.63) 0.975 (0.011) 0.976 (0.011) 238.36 170.67 1.124 (0.136) 1.114 (0.154) BIS [168] BCS [168] Bilinear [53] Bicubic [53] Nearest [53] ∫ -noise [53, 82] GP-Warp [77] 35.04 (0.76) 35.43 (0.97) 35.30 (0.96) 35.31 (0.91) 34.87 (0.90) 35.55 (1.03) 34.49 (0.65) 0.889 (0.016) 0.893 (0.016) 0.896 (0.016) 0.896 (0.016) 0.895 (0.018) 0.892 (0.020) 0.886 (0.016) 0.100 (0.012) 0.099 (0.012) 0.098 (0.012) 0.098 (0.011) 0.099 (0.013) 0.099 (0.014) 0.102 (0.012) 38.91 (1.13) 39.91 (1.09) 39.87 (1.14) 40.11 (1.10) 40.09 (1.14) 39.77 (1.36) 38.71 (1.14) 0.918 (0.012) 0.931 (0.011) 0.931 (0.011) 0.934 (0.010) 0.933 (0.011) 0.929 (0.015) 0.916 (0.013) 82.30 95.68 87.90 108.90 108.57 89.59 92.19 9.203 (0.698) 9.225 (0.693) 9.157 (0.670) 9.189 (0.653) 9.188 (0.656) 9.208 (0.693) 9.209 (0.659) STeP (video only) STeP (image-video joint) 37.00 (1.46) 39.38 (1.16) 0.927 (0.019) 0.951 (0.009) 0.086 (0.013) 0.078 (0.011) 43.50 (2.70) 44.86 (1.92) 0.963 (0.015) 0.974 (0.006) 75.27 78.60 8.817(0.650) 8.753 (0.603) Main Results We summarize the main results as the following four observations. STeP reconstructs videos with better spatial and temporal coherence. We provide quantitative results in Table 7.2 and visual comparisons in Figure 7.9 and Figure 7.10. STeP outperforms baselines in overall video quality across most metrics and generates video reconstructions with significantly better spatiotemporal coherence. Figure 7.9 visualizes an 𝑥-𝑡 slice, representing the temporal evolution (horizontal axis) of a spatial slice (marked by the cyan vertical line). We find that the noise warping baselines fail to constrain temporal consistency, as indicated by the fluctuating ring diameters in the 𝑥-𝑡 slices, while BCS provides almost static reconstructions. In contrast, STeP exhibits closer alignment with the ground truth and improved temporal consistency. This is further illustrated by the averaged optical flow visualization, highlighting that STeP faithfully captures the underlying temporal dynamic. Similar trends can be observed in Figure 7.10. Noise warping is less effective in scientific problems. We observe that the performance of noise-warping strategies in challenging scientific VIPs does not cor∫ relate with noise-warping accuracy. For example, although the -noise approach has demonstrated superior noise-warping capability given optical flow (as shown 107 First frame x-t slice Avg. optical ow BCS Ground truth Avg. optical ow x-t slice Avg. optical ow t BIS ∫-noise STeP (joint) x First frame GP-Warp x-t slice First frame Figure 7.9: Visual examples of STeP (bottom left) and baselines for black hole video reconstruction. To facilitate analysis of the reconstructed spatiotemporal structures, we present results in three ways: (1) a single frame to illustrate spatial fidelity, (2) an 𝑥-𝑡 slice depicting temporal evolution of a vertical line to evaluate temporal consistency, and (3) the averaged optical flow visualized using the standard color scheme from [278] to assess spatiotemporal coherence jointly. Compared to baselines, STeP exhibits clearer alignment with ground truth videos across all aspects. fl GP-Warp BIS First frame BCS Avg. optical ow ∫-noise First frame Ground truth Avg. optical ow STeP (joint) First frame Avg. optical ow Figure 7.10: Visual examples of STeP (bottom left) and baselines for dynamic MRI. We visualize a representative frame along with two zoomed-in regions for each method to better illustrate spatial fidelity. Benefiting from its robust spatiotemporal prior, STeP provide reconstructions with fewer structural artifacts and temporal fluctuations, as indicated by its averaged optical flow aligning more closely with the ground truth. This demonstrates the effectiveness of our learned spatiotemporal prior in enhancing both spatial and temporal consistency. fl in [53]), it underperforms simpler strategies such as Bilinear and Bicubic interpolation in challenging VIPs. This discrepancy arises primarily due to: (1) inaccuracies in the optical flow, and (2) the inherent difficulty of effectively manipulating noise in latent space through pixel space optical-flow-guided warping. Consequently, methods relying on precise optical flow and carefully designed warping strategies struggle with tasks requiring high temporal consistency, such as black hole video reconstruction. In contrast, our approach employs a data-driven spatiotemporal prior learned from training data, effectively overcoming these limitations and enabling broader applicability to scientific tasks with significant temporal uncertainty. 108 Averaged delta frames Avg. optical ow Selected frames over time STeP (joint) (Window: 4) (Window: 16) (Window: 64) (a) better temporal consistency STeP (video only) BCS STeP (joint) Ground truth larger Δ Ground truth First frame (b) less spatial hallucinations Figure 7.11: Detailed comparison on black hole video reconstruction and dynamic MRI. Left: We compare STeP (joint) and the BCS baseline [168] by visualizing the averaged delta frames (difference images) over an expanding window. The delta frames given by STeP (joint) better align with the ground truth, indicating better temporal consistency. Right: We also compare the spatial fidelity between STeP (joint) and its variant STeP (video-only). Trained on both images and videos, STeP (joint) provide reconstructions with less spatial hallucinations compared to STeP (video-only). fl (a) Black hole imaging (b) Dynamic MRI max: 1.795 max: 9.261 nal: 8.993 min: 8.968 min: 1.114 nal: 1.120 Figure 7.12: Consistent improvement in image-video joint fine-tuning. We evaluate intermediate checkpoints of (a) black hole video reconstruction and (b) dynamic MRI (8×). Spatial similarity (measured by PSNR), temporal consistency (measured by d-PSNR), and measurement data fit (measured by data misfit) all show steady improvement. fi Joint fine-tuning benefits both spatial and temporal consistency. As shown in Table 7.2, STeP with an image-video jointly fine-tuned spatiotemporal prior outperforms both IDM-based baselines and STeP trained video data only. To further illustrate these improvements, we provide a detailed visual comparison in Figure 7.11. In Figure 7.11 (a), we analyze the temporal dynamics using averaged delta frames, which visualize the averaged differences between consecutive frames over a window. We observe that delta frames from BCS remain largely unchanged as the temporal window expands, indicating limited temporal variation. In contrast, 109 STeP (joint) effectively captures more complex and coherent temporal dynamics. In Figure 7.11 (b), we compare STeP (joint) and STeP (video only) with a focus on spatial fidelity. Although their reconstructions have similar average optical flow, STeP (joint) have fewer artifacts and hallucinations as pointed out by the arrows. Such an improvement highlights the value of using image data in both the pretraining and fine-tuning stages. STeP provides diverse and equally plausible solutions. Due to the highly illposed and extremely sparse measurements in black hole video reconstruction, STeP can produce multiple semantically diverse reconstructions, each visually plausible and consistent with measurement data. Since the true posterior distribution is unknown, directly quantifying mode coverage is infeasible. Instead, we draw 100 i.i.d. samples and cluster them based on spatial appearance and temporal dynamics. As illustrated in Figure 7.6, we identify three distinct modes with nearly identical data fidelity (see Table D.5 exact data misfit values). One recovered mode aligns closely with the ground truth, while the others differ in rotation direction or spatial structure. This demonstrates that our method not only accurately reconstructs the underlying ground truth but also discovers additional plausible solutions. 7.4.3.5 Ablation on Image-Video Joint Finetuning To show the effectiveness of the proposed image-video joint finetuning technique, we show the quantitative results of using checkpoints of the spatiotemporal U-Net that were fine-tuned for different numbers of epochs. We assess performance using PSNR (blue curve), d-PSNR (red curve), and a data-fitting metric (green curve), as shown in Figure 7.12. Since the spatiotemporal U-Net is initialized from a pre-trained image diffusion model, these curves indicate steady improvement in spatiotemporal consistency and data fitting as the prior is fine-tuned. 7.4.3.6 Discussion on the Choice of Inference Algorithm In this section, we use DAPS rather than PnP-DM as the base algorithm for inference due to the former’s better compatibility with latent DMs (as discussed in Section 7.3.2.2 for text-conditioned DMs). Here we deepen the investigation by running STeP with PnP-DM as the inference algorithm and compare it to the DAPSbased variant. As shown in Figure 7.13, DAPS leads to significantly better visual quality in terms of both improved spatial consistency in individual frames (top row) 110 STeP with PnP-DM 28.41 STeP with DAPS Ground truth 30.47 PSNR x STeP with PnP-DM STeP with DAPS 29.72 t Ground truth 38.57 PSNR x t Figure 7.13: Comparison between running STeP with DAPS versus PnP-DM as the inference backbone. The DAPS version of STeP exhibits significantly better spatial consistency (shown by the first frames in the first row) and temporal consistency (shown by the 𝑥-𝑡 slice in the second row). This comparison illustrates the better compatibility of DAPS with latent video diffusion models than PnP-DM. and enhanced temporal coherence in the 𝑥–𝑡 slices (bottom row). Quantitatively, DAPS achieves higher PSNR in both the black hole video (30.47 dB vs. 28.41 dB) and dynamic MRI reconstruction (38.57 dB vs. 29.72 dB), highlighting its superior compatibility with latent video diffusion models over PnP-DM. 7.5 Thrust 3: Accommodating Black-Box Forward Models Many inverse problems in physical sciences involve partial differential equations (PDEs), such as weather forecasting [313], geophysics [192], and fluid reconstruction [94]. The forward models in these problems may involve complicated numerical algorithms where reliable computation of derivatives is challenging or even infeasible. These problems are beyond the scope of PnP-DM in Chapter 6, which assumes the differentiability of A. In this section, we introduce Blade, an instantiation of Algorithm 2 for derivative-free, ensemble-based posterior estimation. We provide a convergence analysis of Blade and provide explicit bounds on the sampling error. We also show the performance of Blade on a challenging nonlinear fluid dynamics problem. Blade achieves both higher predictive accuracy and better probabilistic calibration compared to competing methods. The appendix for this section is Appendix D.3. 7.5.1 Instantiation of Algorithm 2 with Derivative-Free Likelihood Step The name Blade is derived from the key components of the algorithm: Bayesian inversion, Linearization, Alternating updates, Derivative-free, and Ensemble. We refer readers to Appendix D.3.2 for a brief review of the relevant background. In 111 Algorithm 3 Blade for Derivative-Free Diffusion-Based Posterior Estimation Require: initial ensemble 𝑿 (0) = {𝒙 ( 𝑗) ∈ R𝑛 } 𝐽𝑗=1 , number of iterations 𝐾, {𝜂 𝑘 } 𝐾−1 𝑘=0 , 𝑚 likelihood potential 𝑓 ( · ; 𝒚) with measurements 𝒚 ∈ R , pre-trained diffusion model 𝒔𝜽 . 1: for 𝑘 = 0, ..., 𝐾 − 1 do 2: Z (𝑘) ← LikelihoodStep( 𝑿 (𝑘) , 𝒚, 𝜂 𝑘 ) ⊲ Algorithm 8 (Section 7.5.1.1) 3: 𝑿 (𝑘+1) ← PriorStep(Z (𝑘) , 𝒔𝜽 , 𝜂 𝑘 ) ⊲ Algorithm 9 (Section 7.5.1.2) 4: end for 5: return 𝑿 (𝐾) a nutshell, Blade iteratively updates an ensemble of interacting particles by alternating between a derivative-free likelihood sampling step and a denoising diffusion prior step. We provide pseudocode for the complete sampling algorithm in Algorithm 3. Section 7.5.1.1 details the derivative-free likelihood step achieved through statistical linearization using an ensemble of particles. Section 7.5.1.2 describes the prior step with a denoising DM. At the same time, the noise schedule 𝜂 𝑘 is gradually annealed towards zero. Further details are deferred to Appendix D.3.3. 7.5.1.1 Derivative-Free Likelihood Step via Statistical Linearization Let 𝑿 (𝑘) = {𝒙 ( 𝑗) } 𝐽𝑗=1 denote the ensemble of 𝐽 particles at 𝑘-th alternating iteration of the SGS framework. Recall from Chapter 6 that, in the likelihood step, we ( 𝑗) aim to sample 𝒛 ( 𝑗) from 𝜋 𝑍 |𝑋=𝒙 (𝒛) ∝ exp(− 𝑓 (𝒛; 𝒚) − 2𝜂1 2 ∥𝒛 − 𝒙 ( 𝑗) ∥ 22 ) for each 𝑗 ∈ {1, . . . , 𝐽} where 𝑓 (𝒛; 𝒚) = 2𝜎1 2 ∥A (𝒛) − 𝒚∥ 22 for inverse problems of form 𝒚 Equation (5.1) assuming that 𝒏 ∼ N (0, 𝜎𝒚2 𝑰). Statistical Linearization Following [111, 143], we consider the covariancepreconditioned Langevin dynamics with the large particle limit   √︁ 1 ( 𝑗) ( 𝑗) ( 𝑗) ( 𝑗) 2 d𝒛𝑡 = −𝐶𝑡 ∇ 𝑓 (𝒛𝑡 ; 𝒚) + 2 ∥𝒛𝑡 − 𝒙 ∥ 2 d𝑡 + 2𝐶𝑡 d𝒘 𝑡 , (7.11) 2𝜂 where 𝐶𝑡 := E𝑞 𝑡 [(𝒛𝑡 − 𝒛¯𝑡 )(𝒛𝑡 − 𝒛¯𝑡 )𝑇 ] with 𝒛¯𝑡 := E𝑞 𝑡 [𝒛𝑡 ] and 𝑞 𝑡 is the particle distribution. Since the gradient of 𝑓 with respect to 𝑧 involves the gradient of A with respect to 𝑧, we approximate the gradient of 𝑓 using the statistical linearization 112 technique, i.e., 1 ( 𝑗) (𝐷 𝑇 A) (A (𝒛𝑡 ) − 𝒚) 2 𝜎𝒚 1 ( 𝑗) ≈ 2 𝐶𝑡−1 E𝑞 𝑡 [e 𝒛𝑡 (A (𝒛𝑡 ) − E𝑞 𝑡 A (𝒛𝑡 ))𝑇 ] (A (𝒛𝑡 ) − 𝒚). 𝜎𝒚 ( 𝑗) ∇ 𝑓 (𝒛𝑡 ; 𝒚) = (7.12) where 𝐷 𝑇 A denotes the adjoint of the Jacobian of A and e 𝒛𝑡 = 𝒛𝑡 − 𝒛¯𝑡 . The approximation is given by substituting 𝐷 𝑇 A with the adjoint (i.e., transpose) of 𝑨𝑡 := E𝑞 𝑡 [(A (𝒛𝑡 ) − E𝑞 𝑡 A (𝒛𝑡 ))(𝒛𝑡 − 𝒛¯𝑡 )𝑇 ]𝐶𝑡−1 , (7.13) which is the least-square linear approximation of A. Substituting Equation (7.12) into Equation (7.11) gives # " 1 1 ( 𝑗) ( 𝑗) ( 𝑗) 𝒛𝑡 (A (𝒛𝑡 ) − E𝑞 𝑡 A (𝒛𝑡 ))𝑇 ] (A (𝒛𝑡 ) − 𝒚) + 2 𝐶𝑡 (𝒛𝑡 − 𝒙 ( 𝑗) ) d𝑡 d𝒛𝑡 = − 2 E𝑞 𝑡 [e 𝜂 𝜎𝒚 √︁ + 2𝐶𝑡 d𝒘 𝑡 . (7.14) Thanks to the covariance preconditioner, the dynamics in Equation (7.14) avoid computing 𝐶𝑡−1 in 𝑨𝑡 . Further, Equation (7.14) eliminates the reliance on the forward model’s derivatives, allowing us to run the algorithm with only black-box access to A. Practical Implementation We then implement the dynamics in Equation (7.14) with a finite-particle system in practice and ensure that the invariant measure of the finite-particle system remains the same as that of Equation (7.14). As shown in [226], the covariance-preconditioned stochastic process requires an additional correction term as the diffusion term depends on the evolving particle. For the 𝑗-th particle, we add a correction term to the drift of Equation (7.14), yielding # " √︁ 1 1 ( 𝑗) ( 𝑗) ( 𝑗) 𝑇 ( 𝑗) (A (𝒛 ) − 𝒚) + 𝐶 𝑨 𝐶 (𝒛 − 𝒙 ) d𝑡 + 2𝐶𝑡 d𝒘 𝑡 d𝒛𝑡 = − 𝑡 𝑡 𝑡 𝑡 𝑡 𝜂2 𝜎𝒚2 𝑛 + 1 ( 𝑗) (𝒛𝑡 − 𝒛¯𝑡 )d𝑡 (7.15) 𝐽 where 𝑛 is the dimensionality of 𝒛 and 𝐽 is the ensemble size. Lemma D.3.7 verifies that Equation (7.15) has an invariant measure that is identical to that of ( 𝑗) Equation (7.14). Intuitively, the correction term 𝑛+1 𝐽 (𝒛 𝑡 − 𝒛¯𝑡 ) pushes the particles √ away from each other and vanishes when 𝐽 ≫ 𝑛. For theh computation of 𝐶𝑡 , iwe √ use the construction proposed in [112] where 𝐶𝑡 = √1 𝒛𝑡(1) − 𝒛¯𝑡 , . . . , 𝒛𝑡(𝐽) − 𝒛¯𝑡 ∈ 𝐽 R𝑛×𝐽 , which avoids explicit matrix square roots. Further implementation details can be found in Appendix D.3.3.1, and the pseudocode is provided in Algorithm 8. + 113 7.5.1.2 Ensemble-Based Prior Step via Denoising Diffusion 𝐽 Let 𝒁 (𝑘) = 𝒛 ( 𝑗) 𝑗=1 denote the ensemble of 𝐽 particles at the 𝑘-th alternating iteration of the SGS framework. Recall from Chapter  6 that, in the prior step, our goal ( 𝑗 ) is to sample 𝒙 ( 𝑗) from 𝜋 𝑋 |𝑍=𝒛 (𝒙) ∝ exp −𝑔(𝒙) − 2𝜂1 2 ∥𝒙 − 𝒛 ( 𝑗) ∥ 22 for each 𝑗 ∈ 1, . . . , 𝐽, where 𝑔(𝒙) := − log 𝑝(𝒙) denotes the potential function of the prior distribution. This corresponds to applying the prior step of PnP-DM independently to each particle in 𝒁 (𝑘) 3 , and we therefore refer the readers to Section 6.3.2 for details. Further implementation details are provided in Appendix D.3.3.2 and the pseudocode is available in Algorithm 9. 7.5.2 Convergence Analysis In this sub-section, we analyze the non-asymptotic behavior of the proposed algorithm through the lens of its continuous-time and large particle limit for the ease of understanding. In practice, the proposed algorithm incurs two bias terms: 𝜖model from the statistical linearization and 𝜖score from the learned prior. By extending the existing interpolation proof techniques from Chapter 6, our analysis also quantifies how these errors affect the deviation from the reference process over 𝐾 iterations. The technical definitions and notations are collected in Appendix D.3.1.1. Theorem 7.5.1. Given 𝜂 > 0, consider the following two processes that alternate between the likelihood step with horizon 𝑡 † and the prior step with horizon 𝑡 ∗ , where 𝜎(𝑡 ∗ ) = 𝜂: • The approximate process that implements the likelihood step as in Equation (7.14) (with forward model approximation) and the prior step as in Equation (6.6) (with diffusion model score approximation). Let 𝜇 e𝜏 denote its distribution at time 𝜏, ∗ 𝐶𝜏 the associated covariance matrix, 𝜆 𝜏 the smallest non-zero eigenvalue of 𝐶𝜏 . • The reference process that starts from the stationary distribution 𝜋 𝑋 𝑍 and implements the likelihood step as Equation (7.11) with the preconditioner 𝐶𝜏 , and the prior step as Equation (6.6), assuming exact knowledge of both the prior score function and forward model derivative. Let 𝜇𝜏 denote its distribution at time 𝜏. Let 𝑇𝑘 = 𝑘 (𝑡 † + 𝑡 ∗ ), 𝑘 = 0, . . . , 𝐾, 𝜆∗ = inf 𝑡∈∪𝑘 [𝑇𝑘 ,𝑇𝑘 +𝑡 † ] 𝜆∗𝑡 , and 𝛿 = inf 𝑡∈[0,𝑡 ∗ ] 𝛿(𝑡) where 𝛿(𝑡) is the diffusion term defined in Equation (D.21). We denote by 𝜖score 3 For Blade , we set 𝑠(𝑡) = 1 for simplicity. 114 the score approximation error of the diffusion model defined in Assumption D.3.1, and 𝜖model the forward model derivative approximation error defined in Assumption D.3.2. Assuming that KL(𝜋 𝑋 ||𝜇0 ) < +∞ and Assumption D.3.3 holds, for 𝐾 iterations of Blade, we have ∫ 𝑇𝐾 1 𝑐KL(𝜋 𝑋 ||e 𝜇0 ) FI(𝜇𝜏 ||e 𝜇𝜏 )d𝜏 ≤ + 𝑐𝑡 † 𝜖model + 𝑐𝑡 ∗ 𝜖 score (7.16) | {z } | {z } 𝑇𝐾 0 𝐾 | {z } | {z } model error score error average Fisher divergence over 𝐾 iterations of Blade convergence from initialization 4 is a constant and FI, KL are Fisher divergence and KL where 𝑐 := min(𝜆∗ ,𝛿) (𝑡 † +𝑡 ∗ ) divergence, respectively. The proof is provided in Appendix D.3.1.3. Theorem 7.5.1 accounts for the effect of the two generally unavoidable approximations: the statistical linearization of the forward model and the learned diffusion prior. Equation (7.16) indicates that the time-average Fisher divergence between the approximate process and reference process decays at an 𝑂 (1/𝐾) rate up to a weighted sum of two approximation errors. While this bound is structurally similar to that of PnP-DM (Theorem 6.4.1), there are two key distinctions. First, our method considers fundamentally different likelihood step dynamics (with linearization and covariance preconditioning); consequently, the resulting error term depends on additional factors such as covariance matrix eigenvalues and the linearization error. Second, unlike the analysis in Section 6.4 that assumes exact likelihood steps, our analysis incorporates the effects of model error 𝜖model and a finite time horizon 𝑡 † . Furthermore, this analysis offers theoretical guarantees with explicit bounds on the approximation errors, which has not been done in the existing works on the derivative-free algorithms with diffusion priors. 7.5.3 Experiments We evaluate Blade on the inverse problem of recovering the initial vorticity field in the 2D Navier–Stokes equations from partial, noisy observations taken at a later time. This setting mirrors many practical inverse problems in science and engineering, including weather data assimilation [313], geophysics [192], and fluid reconstruction [94]. Problem Setup We follow the general experimental setup established in InverseBench [357] (Chapter 8), using its publicly released dataset and pre-trained diffusion prior for all experiments. The true initial vorticity 𝒙0 in resolution 128×128 115 Table 7.3: Comparison on the Navier-Stokes inverse problem. Metrics are abbreviated as follows: Rel ℓ2 (relative ℓ2 error), CRPS (continuous ranked probability score), and SSR (spread-skill ratio). − indicates either that probabilistic metrics are inapplicable (deterministic models) or that it is too expensive to generate enough samples from the algorithm for reliable calculation. Bold and Underline indicate best and second best among methods that accommodate unpaired data, respectively. 𝜎𝒚 = 0 𝜎𝒚 = 1.0 𝜎𝒚 = 2.0 Rel ℓ2 ↓ CRPS ↓ SSR → 1 Rel ℓ2 ↓ CRPS ↓ SSR → 1 Rel ℓ2 ↓ CRPS ↓ SSR → 1 Paired data CDM U-Net 1.362 0.585 2.900 – 0.983 – 1.409 0.702 2.872 – 1.059 – 1.542 0.709 2.993 – 1.087 – Unpaired data EKI EKS + DM DPG SCG EnKG Blade (ours) 0.577 0.539 0.325 0.961 0.120 0.110 2.303 1.900 – – 0.395 0.216 0.012 0.181 – – 0.164 0.955 0.586 0.606 0.408 0.928 0.191 0.229 2.350 2.088 – – 0.651 0.453 0.118 0.218 – – 0.154 0.950 0.673 0.685 0.466 0.966 0.294 0.306 2.700 2.255 – – 1.032 0.608 0.011 0.280 – – 0.144 0.949 is evolved forward with a numerical solver, then subsampled and corrupted with Gaussian noise of standard deviation 𝜎𝒚 = 0, 1, 2. The measurements 𝒚 thus constitute a partial, noisy snapshot of the flow field. More details can be found in Appendix D.3.5. Baselines We compare our algorithm against two classes of methods. The first class is the methods that only require diffusion prior trained on unpaired data, including DPG [277], SCG [141], EnKG [356], EKI [143], EKS [111] (initialized from diffusion prior). The second class is provided as reference points, which requires additional training on paired data, including a conditional diffusion model (CDM) and an end-to-end UNet. The conditional diffusion learns the posterior distribution through conditional score matching. The UNet directly learns to predict the ground truth from the observation. For each noise regime, we re-train both the conditional diffusion model and the U-Net from scratch, using the same training configuration. More details can be found in Appendix D.3.4. Evaluation Metrics For comprehensive evaluation, we consider three different metrics in the literature [241, 357] to assess the performance from both deterministic and probabilistic perspectives: relative ℓ2 error, continuous ranked probability score (CRPS), and spread-skill ratio (SSR). CRPS is a proper scoring rule that rewards both sharp predictive distribution and well-calibrated predictions, whereas SSR diagnoses calibration only. An SSR near one is desirable, but must be interpreted 116 in conjunction with the other error metrics. Formal definitions and implementation details of these metrics are in Appendix D.3.5.2. Results Table 7.3 summarizes performance under three observation noise levels. Blade offers the best calibrated ensemble predictions: its CRPS is the best among all, and its SSR remains close to one. The other competing methods are too confident (SSR < 0.2) and their predictions do not represent the true uncertainty. The CRPS of CDM has a very high CRPS despite SSR near one, which means that it produces an overly diffuse distribution with large errors. Overall, Blade achieves both sharp predictive performance and reliable uncertainty calibration. 7.6 Thrust 4: Tackling Inverse Problems in Discrete Spaces Finally, we present SGDD, an instantiation of Algorithm 2 for posterior estimation in discrete spaces using discrete diffusion models (DMs). While existing works on diffusion-based posterior estimation have been focusing on continuous domains [64, 320, 341], designing analogous techniques to discrete-state spaces remains challenging due to the lack of well-defined gradients in both the likelihood and prior. To address this, we formulate a discrete version of the Split Gibbs Sampler (SGS) with a generalized regularization potential 𝐷 (𝒙, 𝒛; 𝜂), extending the standard ℓ2 term to functions compatible with discrete DMs under certain limiting conditions. This relaxed formulation preserves the alternating update structure of Algorithm 2 and enables seamless integration of pre-trained discrete DMs. We prove theoretical convergence guarantees for SGDD, with error bounds accounting for discretization and score approximation. Empirically, we show the effectiveness of SGDD on inverse problems on discrete images, where SGDD outperforms baselines by more than 8 dB in PSNR. The appendix for this section is Appendix D.4. 7.6.1 Discrete Diffusion Models Recent works on discrete diffusion models extend score-based generative methods from modeling continuous distributions in Euclidean spaces to categorical distributions in discrete-state spaces [11, 40, 198, 259]. Specifically, when the data distribution lies in a finite support X = {1, . . . , 𝑁 }, one can evolve a family of categorical distributions 𝑝 𝑡 over X following a continuous-time Markov chain 𝜕𝑡 𝑝 𝑡 = 𝑸 𝑡fw 𝑝 𝑡 , (7.17) where 𝑝 0 = 𝑝 data and 𝑸 𝑡fw ∈ R𝑁×𝑁 are diffusion matrices with a simple stationary distribution. To reverse this continuous-time Markov chain, it suffices to learn the 117 concrete score 𝑠(𝒙; 𝑡) := h i ˜ 𝑝 𝑡 ( 𝒙) , as the reverse process is given by 𝑝 𝑡 (𝒙) 𝒙≠𝒙 ˜ 𝜕𝑡 𝑝𝑇−𝑡 = 𝑸𝑇−𝑡 𝑝𝑇−𝑡 [𝑖, 𝑗] with 𝑸 𝑡 (𝒙 𝑖 ) 𝑸𝑡 = 𝑝𝑝𝑡𝑡 (𝒙 𝑗) fw[ 𝑗,𝑖] and 𝑸 𝑡[𝑖,𝑖] = − Í (7.18) [ 𝑗,𝑖] . 𝒙 𝑗 ≠𝒙𝑖 𝑸 𝑡 For sequential data 𝒙 ∈ X 𝑛 , there are 𝑁 𝑛 states in total. Instead of constructing an exponentially large diffusion matrix, we use a sparse matrix 𝑸 𝑡fw that perturbs tokens independently in each dimension [198]. Example: Uniform Kernel An example of such a diffusion matrix is 𝑸 𝑡fw = 𝜎 ¤ 𝑡 𝑸 uniform = 𝜎 ¤ 𝑡 (11𝑇 /𝑁 − 𝑰), (7.19) where 𝜎𝑡 ≡ 𝜎(𝑡) is a predefined noise schedule with 𝜎(0) = 0 and 𝜎(𝑇) = 𝜎max . This uniform kernel transfers any distribution to a uniform distribution as 𝜎 → ∞. Moreover, ∫ 𝑡    𝑇 fw uniform −𝜎𝑡 −𝜎𝑡 11 𝑝 𝑡 = exp 𝑸 𝜏 d𝜏 𝑝 0 = exp(𝜎𝑡 𝑸 ) 𝑝 0 = 𝑒 𝑰 + (1 − 𝑒 ) 𝑝0 . 𝑁 0 When 𝒙 ∈ X 𝑛 is a discrete object of 𝑛 dimensions, we have 𝑝 𝑡 (𝒙𝑡 | 𝒙0 ) ∝ 𝛽𝑡𝑑 (𝒙𝑡 ,𝒙0 ) (1− 𝛽𝑡 ) 𝑛−𝑑 (𝒙𝑡 ,𝒙0 ) , where 𝑑 (·, ·) is the Hamming distance between two sequences and −𝜎𝑡 ). 𝛽𝑡 = 𝑁−1 𝑁 (1 − 𝑒 7.6.2 7.6.2.1 Instantiation of Algorithm 2 for Discrete-Space Inverse Problems A Generalized Split Gibbs Sampler Recall that our goal is to sample from the posterior distribution 𝑝(𝒙 | 𝒚) ∝ 𝑝( 𝒚 | 𝒙) 𝑝(𝒙) = exp(− 𝑓 (𝒙; 𝒚) − 𝑔(𝒙)), (7.20) where 𝑓 (𝒙; 𝒚) = − log 𝑝( 𝒚 | 𝒙) and 𝑔(𝒙) = − log 𝑝(𝒙). The Split Gibbs Sampler (SGS) relaxes this sampling problem by introducing an auxiliary variable 𝒛, allowing sampling from an augmented distribution 𝜋(𝒙, 𝒛; 𝜂) ∝ exp(− 𝑓 (𝒛; 𝒚) − 𝑔(𝒙) − 𝐷 (𝒙, 𝒛; 𝜂)), (7.21) where 𝐷 (𝒙, 𝒛; 𝜂) measures the distance between 𝒙 and 𝒛, and 𝜂 > 0 is a parameter that controls the strength of regularization. While PnP-DM (Chapter 6) and other prior works [69, 326] consider 𝐷 (𝒙, 𝒛; 𝜂) = ∥𝒙−𝒛∥ 22 , we generalize the potential function to any continuous function 𝐷, such that 2𝜂2 118 𝐷 (𝒙, 𝒛; 𝜂) → ∞ as 𝜂 → 0 for any 𝒙 ≠ 𝒛. As shown in Appendix D.4.1.3, this ensures that both marginal distributions, ∫ ∫ 𝑋 𝑍 𝜋 (𝒙; 𝜂) := 𝜋(𝒙, 𝒛; 𝜂)d𝒛, and 𝜋 (𝒛; 𝜂) := 𝜋(𝒙, 𝒛; 𝜂)d𝒙 converge to the posterior 𝑝(𝒙 | 𝒚). The decoupling of the prior and likelihood in Equation (7.21) enables Gibbs sampling, which alternates between two steps: 1. Likelihood step: sample 𝒛 (𝑘) ∼ 𝜋(𝒙 = 𝒙 (𝑘) , 𝒛; 𝜂) ∝ exp(− 𝑓 (𝒛; 𝒚)−𝐷 (𝒙 (𝑘) , 𝒛; 𝜂)) 2. Prior step: sample 𝒙 (𝑘+1) ∼ 𝜋(𝒙, 𝒛 = 𝒛 (𝑘) ; 𝜂) ∝ exp(−𝑔(𝒙) − 𝐷 (𝒙, 𝒛 (𝑘) ; 𝜂)) A key feature of SGS is that it does not rely on the gradient of the guidance term 𝑓 (𝒛; 𝒚), which is highly desirable in our setting with discrete data. The key challenge that we tackle is to develop an effective approach for discrete-space DMs that is easy to implement and enjoys rigorous guarantees (i.e., sampling from the correct posterior distribution). 7.6.2.2 Prior Step with Discrete Diffusion Models Suppose 𝑝(𝒙) is a discrete-state distribution over X 𝑛 modeled by a diffusion process. We consider a discrete DM with uniform transition kernel 𝑸 𝑡fw = 𝑁1 11𝑇 − 𝑰. To connect SGS to discrete DMs, we specify the potential function 𝐷 (𝒙, 𝒛; 𝜂) as 𝐷 (𝒙, 𝒛; 𝜂) := 𝑑 (𝒙, 𝒛) log 1 + (𝑁 − 1)𝑒 −𝜂 (𝑁 − 1)(1 − 𝑒 −𝜂 ) (7.22) where 𝑑 (𝒙, 𝒛) denotes the Hamming distance between 𝒙 and 𝒛. When 𝜂 → 0+ , the regularization potential 𝐷 (𝒙, 𝒛; 𝜂) goes to infinity unless 𝑑 (𝒙, 𝒛) = 0, ensuring the convergence of marginal distributions to 𝑝(𝒙 | 𝒚). Given Equation (7.22), the prior step can be written as 𝒙 (𝑘+1) ∼ 𝜋(𝒙, 𝒛 = 𝒛 (𝑘) ; 𝜂) ∝ 𝑝 0 (𝒙) 𝛽e ! 𝑑 (𝒛 (𝑘 ) ,𝒙) 1 − 𝛽e (7.23) −𝜂 where 𝛽e = 𝑁−1 𝑁 (1 − 𝑒 ). On the other hand, the distribution of clean data 𝒙 0 conditioned on 𝒙𝑡 for discrete DMs is given by 𝑝(𝒙0 | 𝒙𝑡 ) ∝ 𝑝 0 (𝒙 0 ) 𝑝(𝒙𝑡 | 𝒙0 ) ∝ 𝑝 0 (𝒙0 ) 𝛽𝑡𝑑 (𝒙𝑡 ,𝒙0 ) (1 − 𝛽𝑡 ) 𝑛−𝑑 (𝒙𝑡 ,𝒙0 ) (7.24) −𝜎𝑡 ). Note that the term (1 − 𝛽 ) 𝑛 does not depend on 𝒙 where 𝛽𝑡 = 𝑁−1 𝑡 0 𝑁 (1 − 𝑒 and can thus be dropped. Therefore, sampling from Equation (7.23) is equivalent 119 to unconditional generation from 𝑝(𝒙0 | 𝒙𝑡 ) when 𝛽e = 𝛽𝑡 , i.e., 𝜂 = 𝜎𝑡 . We can then solve the prior sampling problem by simulating a partial discrete diffusion sampler that starts from 𝜎𝑡 = 𝜂, 𝒙𝑡 = 𝒛 (𝑘) and solves backward to 𝑡 = 0. This is ∥𝒙−𝒛∥ 2 analogous to the continuous-space setting in Chapter 6 (i.e., 𝐷 (𝒙, 𝒛; 𝜂) = 2𝜂2 2 ), where the prior step can be formulated as a Gaussian denoising problem solvable by a continuous-space DM. 7.6.2.3 Likelihood Step with Metropolis-Hastings With the potential function 𝐷 (𝒙, 𝒛; 𝜂) specified in Section 7.6.2.2, at iteration 𝑘, the likelihood sampling step can be written as:   1 + (𝑁 − 1)𝑒 −𝜂 (𝑘) (𝑘) . (7.25) 𝒛 ∼ 𝜋(𝒙 = 𝒙 , 𝒛; 𝜂) ∝ exp − 𝑓 (𝒛; 𝒚) − 𝑑 (𝒙, 𝒛) log (𝑁 − 1)(1 − 𝑒 −𝜂 ) Since the unnormalized probability density function of 𝜋(𝒙 = 𝒙 (𝑘) , 𝒛; 𝜂) is available, we can efficiently sample from Equation (7.25) using the Metropolis-Hastings algorithm [125, 220]. 7.6.2.4 Overall Algorithm Algorithm 4 Split Gibbs Discrete Diffusion Posterior Sampling (SGDD) Require: initialization 𝒙 (0) ∈ X 𝑁 , total number of iterations 𝐾 > 0, noise schedule 𝑚 {𝜂 𝑘 } 𝐾−1 𝑘=0 , likelihood potential 𝑓 ( · ; 𝒚) with measurements 𝒚 ∈ R , pre-trained discrete diffusion model 𝒔𝜽 . 1: for 𝑘 = 0, ..., 𝐾 − 1 do 2: 𝒛 (𝑘) ← LikelihoodStep(𝒙 (𝑘) , 𝒚, 𝜂 𝑘 ) ⊲ Sample (7.25) (Section 7.6.2.3) 3: 𝒙 (𝑘+1) ← PriorStep(𝒛 (𝑘) , 𝒔𝜽 , 𝜂 𝑘 ) ⊲ Sample (7.23) (Section 7.6.2.2) 4: end for 5: return 𝒙 (𝐾) We now summarize the complete SGDD algorithm. Like PnP-DM, SGDD alternates between a likelihood step and a prior step while employing an annealing schedule {𝜂 𝑘 }, which starts at a large 𝜂0 and gradually decays to 𝜂 𝑘 → 0. This annealing scheme accelerates the mixing time of the Markov chain and ensures the convergence of 𝜋 𝑋 (𝒙; 𝜂) and 𝜋 𝑍 (𝒛; 𝜂) to 𝑝(𝒙 | 𝒚) as 𝜂 → 0. We present the complete pseudocode of our method in Algorithm 4. 7.6.3 Convergence Analysis We provide theoretical guarantees on the convergence of SGDD. For two probability measures in a finite domain X, we consider the Kullback-Leibler (KL) divergence 120 and the Fisher divergence (or relative Fisher information), respectively, as (where we define 𝑓 := 𝜇/𝜋) h 𝜇i KL(𝜇∥𝜋) := E 𝜇 log , 𝜋   ∑︁ 𝑓 (𝒙 𝑗 ) [ 𝑗,𝑖] FI𝑸 (𝜇∥𝜋) := . 𝜋(𝒙𝑖 )𝑸 𝑓 (𝒙 𝑗 ) − 𝑓 (𝒙𝑖 ) − 𝑓 (𝒙𝑖 ) log 𝑓 (𝒙𝑖 ) 𝒙 ,𝒙 ∈X 𝑖 𝑗 Both divergences are nonnegative and equal to zero if and only if 𝜇 = 𝜋, when 𝑸 is irreducible. For the continuous case in Section 6.4 and Section 7.5.2, Fisher divergence can be written as a quadratic form FI(𝜇∥𝜋) = E 𝜇 ∥∇ log(𝜇/𝜋)∥ 2 . However, it does not have an analogous quadratic form in finite spaces, which poses additional challenges to the analysis of SGDD. To address this challenge, we adopt a generalized definition of Fisher divergence [27, 130] and analyze the convergence of SGDD to the stationary distribution using a more general technique that encompasses both continuous and discrete settings. We define a distribution 𝜇𝜏 over X that evolves according to likelihood steps and prior steps alternatively where 𝜏 is the index for time over 𝐾 iterations of SGDD. We assume each likelihood step is implemented with the Metropolis-Hastings algorithm, and that each prior step is solved by the Euler method with an approximated score function. We compare 𝜇𝜏 to the continuous-time stationary distribution 𝜋𝜏 , which alternates between 𝜋 𝑋 and 𝜋 𝑍 . The definitions of 𝜇𝜏 and 𝜋𝜏 are provided in Appendix D.4.1. Theorem 7.6.1. Consider running 𝐾 iterations of SGDD with a fixed 𝜂 > 0 and an estimated concrete score 𝒔𝜽 (𝒙𝑡 ; 𝑡), and suppose that each prior step is solved by an 𝐻 step Euler method. Let 𝑡 ∗ > 0 with 𝜎(𝑡 ∗ ) = 𝜂. Let 𝑇𝑘 = 𝑘 (𝑡 ∗ + 1) + 1 be the starting time of the 𝑘-th prior step. Define 𝜋𝜏 and 𝜇𝜏 as stationary and nonstationary distributions. Over 𝐾 iterations of SGDD, the average Fisher divergence between 𝜇𝜏 and 𝜋𝜏 satisfies 𝐾−1 1 ∑︁ 1 𝐾 𝑘=0 𝑡 ∗ | ∫ 𝑇𝑘 +𝑡 ∗ FI𝑸 𝜏 (𝜇𝜏 ∥𝜋𝜏 )d𝜏 ≤ 𝑇𝑘 {z average Fisher divergence over the k-th prior step 𝑡)−𝒔(·; 𝑡) where ∥ 𝒔𝜽 (·;𝒔(·; ∥∞ 𝑡) Appendix D.4.1. } 2KL(𝜇0 ∥𝜋0 ) 4𝑀𝜖 + + ∗ 𝐾𝑡 𝑐 | {z } |{z} convergence from initialization score error 2𝑀 𝐿𝑡 ∗ 𝑐𝐻 | {z } . discretization error (7.26) ≤ 𝜖 < 1, and 𝐿, 𝑀, 𝑐 are positive constants defined in 121 The proof is provided in Appendix D.4.1. Theorem 7.6.1 states that the average Fisher divergence of the non-stationary process with respect to the stationary process in all prior steps converges at the rate of 𝑂 (1/𝐾), up to a constant error term. This result extends our theoretical understanding of diffusion posterior sampling by generalizing the analysis in [273] and Section 6.4 on SDEs to general Markov processes using the free-energy-rate-functional-relative-Fisher-information (FIR) inequality [129]. Moreover, compared to existing analyses for continuous diffusion models, our analysis accounts not only for the imperfect score function but also for the discretization error in solving continuous-time Markov chains as well. 7.6.4 Experiments We apply SGDD to inverse problems on discrete images to show its effectiveness on posterior estimation in discrete domains. Implementation details for the experiments are provided in Appendix D.4.2.3. Problem Setup We convert the MNIST dataset [172] to binary strings by discretizing the images, and train a discrete diffusion prior on 60k training data using the SEDD [198] model with the uniform transition kernel. We consider AND and XOR operators as examples of linear and nonlinear forward models on binary strings. We randomly pick 𝛾𝑛 pairs of positions (𝑖 𝑝 , 𝑗 𝑝 ) over {1, . . . , 𝑛}, and compute AAND (𝒙) = [𝒙𝑖 𝑝 ∧ 𝒙 𝑗 𝑝 ] 𝑝=1,...,𝛾𝑛 , AXOR (𝒙) = [𝒙𝑖 𝑝 ⊕ 𝒙 𝑗 𝑝 ] 𝑝=1,...,𝛾𝑛 . (7.27) The likelihood function is defined as 𝑝(𝒚 | 𝒙) ∝ exp(−∥A (𝒙) − 𝒚∥ 0 /𝜎𝒚 ). Baselines We compare SGDD to existing approaches that apply to discrete diffusion posterior sampling: DPS [64], SVDD-PM [182], and SMC [318]. Details on how we adapt these methods to our setting are provided in Appendix D.4.2.2. Evaluation Metrics We use 1,000 binary images from the test set of MNIST and calculate the Peak Signal-to-Noise Ratio (PSNR) of the reconstructed image. Furthermore, we train a simple convolutional neural network on MNIST as a surrogate and report the classifier accuracy of the generated samples. Results As shown in Table 7.4, SGDD outperforms baseline methods by a large margin in both XOR and AND tasks. We present the samples generated by SGDD for the XOR task in Figure 7.14. The reconstructed samples are visually consistent 122 Table 7.4: Quantitative results for XOR and AND problems on discretized MNIST. We report the mean and standard deviation (shown in parentheses) of PSNR and class accuracy across 1,000 generated samples. SGDD demonstrates superior performance on both tasks. XOR SVDD-PM [182] SMC [318] DPS [64] SGDD AND PSNR ↑ Accuracy (%) ↑ PSNR ↑ Accuracy (%) ↑ 11.81 (2.54) 10.05 (1.54) 9.04 (1.21) 20.17 (3.47) 51.4 27.8 30.0 91.2 10.04 (1.49) 10.25 (1.63) 8.67 (0.91) 17.25 (3.82) 33.7 24.4 24.5 79.4 Figure 7.14: Sampling results of the XOR task on the discretized MNIST dataset. SGDD faithfully recovers the structural information of the ground truth signal. with the underlying ground truth signal, which explains the high class accuracy of SGDD shown in Table 7.4. Furthermore, we demonstrate that SGDD generates diversified samples when the measurement 𝒚 is sparse. For example, when a digit is masked with a large box, as shown in Figure 7.15, the measurement lacks sufficient information to fully recover the original digit. In this scenario, SGDD generate samples from multiple plausible modes, including digits 1, 4, 7, and 9. This highlights the ability of SGDD to produce diverse samples from the posterior distribution while preserving consistency with the measurement. 7.7 Conclusion In this chapter, we presented four instantiations of Algorithm 2, each designed to extend the applicability of diffusion-based posterior estimation to broader classes 123 Figure 7.15: Diversified samples when the measurement 𝒚 is sparse. Samples are generated by SGDD when solving an MNIST inpainting task. of inverse problems. These methods address key limitations of the original PnPDM method in Chapter 6 by incorporating richer forms of prior information, scaling to higher-dimensional video data, accommodating non-differentiable forward models, and enabling inference in discrete domains. Despite targeting distinct challenges, all four instantiations share a common alternating-update structure that provides a modular template for future extensions. Collectively, these developments demonstrate the versatility of our unified PnPDP framework, which we believe paves the way for future advances in diffusion-based posterior estimation. 124 Chapter 8 INVERSEBENCH: BENCHMARKING DIFFUSION-BASED METHODS FOR SCIENTIFIC INVERSE PROBLEMS Plug-and-play diffusion priors (PnPDP), including the methods we presented in Chapter 6 and Chapter 7, have emerged as a promising paradigm for solving inverse problems. While many existing methods claim to handle general inverse problems, they were investigated in different domains (primarily natural image restoration). They have never been compared in a controlled manner, and it is unclear how they perform on scientific inverse problems in computational imaging. To address this gap, we introduce InverseBench, a framework that evaluates diffusion models across five distinct scientific inverse problems. These problems present unique structural challenges that differ from existing benchmarks, arising from critical scientific applications such as optical tomography, medical imaging, black hole imaging, seismology, and fluid dynamics. With InverseBench, we benchmark 14 inverse problem algorithms that use plug-and-play diffusion priors against strong, domain-specific baselines, offering valuable new insights into the strengths and weaknesses of existing algorithms. To facilitate further research and development, we open-source the codebase, along with datasets and pre-trained models, at https: //devzhk.github.io/InverseBench/. This chapter is based on our work [357], published as a Spotlight paper in the Proceedings of the 13th International Conference on Learning Representations 2025 (ICLR 2025). The appendix for this chapter is Appendix E. The code for the work presented in this chapter is available at https://github.com/devzhk/ InverseBench. 8.1 Introduction The existing PnPDP methods are primarily evaluated and compared on a fairly narrow set of image restoration tasks—such as inpainting, super-resolution, and deblurring [44, 64, 147, 214, 262, 306]. These problems differ greatly from those in science and engineering applications such as geophysics [294], astronomy [233], oceanography [47], and many other fields, which have very different structural challenges arising from the underlying physics. It is unclear how much insight can be carried over from image restoration to scientific inverse problems. 125 In this chapter, we introduce InverseBench, a comprehensive benchmarking framework designed to evaluate PnPDP methods in a systematic and easily extensible manner. We curate a diverse set of five inverse problems from distinct scientific domains: optical tomography, black hole imaging, medical imaging, seismology, and fluid dynamics. These problems present structural challenges that differ significantly from natural image restoration tasks (cf. Figure 8.1 and Table 8.2), and encompass a broad spectrum of complexities across multiple scientific fields. Most notably, the forward model (which maps the target image to measurements) is defined using various types of physics-based models, which can be highly nonlinear and difficult to evaluate. We select 14 representative PnPDP algorithms proposed for solving inverse problems, providing a thorough comparison of their performance across different scientific inverse problems and further insights into their efficacy and limitations. Additionally, we establish strong, domain-specific baselines for each inverse problem, providing a meaningful reference point for assessing the effectiveness of diffusion model-based approaches against traditional methods. Through extensive experiments, we find that PnPDP methods generally exhibit strong performance given a suitable dataset for training a diffusion prior. This performance is consistent even as we vary the forward model (which is a strength of a PnP approach), given appropriate tuning. However, for forward models that require certain constraints on the input (e.g., use a PDE solver), performance can be very sensitive to hyperparameter tuning. Moreover, the strength of using a diffusion prior can also be a limitation, as PnPDP methods have difficulty when the source image is out of the prior distribution (i.e., the use of diffusion models makes it difficult to recover “surprising” results). Additionally, we find that PnP methods that use multiple queries of the forward model tend to outperform simpler methods like DPS, at the cost of requiring additional tuning and computation, which points to an interesting direction for future method development. InverseBench is implemented as a highly modular framework that can interface with new inverse problems and algorithms to run experiments at scale. We opensource the codebase, along with datasets and pre-trained models, at https:// devzhk.github.io/InverseBench/. 126 Linear inverse scattering Scattered light field Compressed sensing MRI k-space measurements MRI image Black hole imaging Black hole image Visibilities A G Compressional wave velocity G† A † Navier-Stokes equation Initial vortocity field Observed vortocity field G A <latexit sha1_base64="76fLj5oFKMQr4i3cMV9HlgCW1pQ=">AAAF8HicbdTJThtBEAbghuCEOBskx1ysoEg5ITuK4Mq+L2bxAhihnnHbDJ6N7h7AWPMYkXIKyjUvk3PeJmPw/yssJXWr+qsqqU/lxL5nbLH4d2j42Uju+YvRl/lXr9+8fTc2/r5qokS7quJGfqTrjjTK90JVsZ71VT3WSgaOr2pOZ75fr10obbwo3LfdWB0Hsh16Lc+VNqOjRiDtqSv93mx6MjZRnCzeRuFxUhokE2IQ5ZPxkT+NZuQmgQqt60tjjkrF2B73pLae66s030iMiqXbkW11lKWhDJQ57t3+OS18zqRZaEU6O6Et3Or/Ez0ZGNMNnKyz/0fzsNbHJ2vqPJRay+6TRSdI778fQpD41tPR5X210kl8qa/SfL7RVK2GdIJer5FdMk3TO3JADskFuaQmqElSIEVqgVqkNqhNOgWdkjyQRzoDnZE6oA7JB/mkABSQQlBIikARKQbFpHPQOUmDNMmADMmCLCkBJaQL0AXpEnRJugJdkbqgLukadN2nO5uFzbJtDjRHmgfNkxZAC6RF0CJpCbREWgYtk1ZAK6RV0CppDbRGWgetkzZAG6RN0CZpC7RF2gZtk8qgMmkHtEPaBe2S9kB7pH3QPqkCqpCqoCqpBqqR6qA66QB0QDoEHWaUrcDSw4X3OKl+nSxNTU7tfJuYmRssw1HxUXwSX0RJTIsZsSLKoiJcEYnv4qe4yencj9xN7tdd6/DQYOaDuBe53/8AOFcoBg==</latexit> <latexit sha1_base64="y0ihnytmEtYb/XROIpDkBvRbpNI=">AAAF+nicbdTJThtBEAbghuCEOBuEYy5WrEg5ITuKyBVsFrPb4A0YYvWM2+PBs7m7B9s48yyRcgrKNe+Rc94mY/BfCktJ3ar+qkrqU5mh6yidy/2dmX0yl3r6bP55+sXLV6/fLCy+rasgkpaoWYEbyKbJlXAdX9S0o13RDKXgnumKhtkrTuqNSyGVE/hVPQrFucdt3+k4FtcJtRaWDI/rrsXd8Vr81Whz2xaytZDNLeduIvMwyU+TLJtGubU498doB1bkCV9bLlfqLJ8L9fmYS+1YrojTRqREyK0et8VZkvrcE+p8fPP7OPMhkXamE8jk+Dpzo/9PjLmn1Mgzk87JX9X92gQfrYm+z6Xko0eLphfffd8HL3K1I4PBXdXcjFwuh3E6bbRFx+CmNx4bycXjOL4lE2QSWSCLqA1qEwmQIOqAOkQ2yCbqgrpEDsghugBdEPVAPSIX5BJ5II/IB/lEASggCkEhUR/UJ5IgSaRAikiDNFEEioguQZdEA9CAaAgaEo1AI6Ir0NWEbm0NtkZtBVCBqAgqEq2D1ok2QBtEm6BNoi3QFlEJVCLaBm0T7YB2iHZBu0R7oD2ifdA+0QHogOgQdEhUBpWJKqAK0RHoiOgYdExUBVWJaqAaUR1UJ2qAGkRNUJPoBHRCdAo6TShZgfn7C+9hUv+0nF9ZXql8zq4Wpstwnr1j79lHlmdf2CorsTKrMYuN2Hf2k12nvqV+pK5Tv25bZ2emM0vsTqR+/wPvqSur</latexit> <latexit sha1_base64="y0ihnytmEtYb/XROIpDkBvRbpNI=">AAAF+nicbdTJThtBEAbghuCEOBuEYy5WrEg5ITuKyBVsFrPb4A0YYvWM2+PBs7m7B9s48yyRcgrKNe+Rc94mY/BfCktJ3ar+qkrqU5mh6yidy/2dmX0yl3r6bP55+sXLV6/fLCy+rasgkpaoWYEbyKbJlXAdX9S0o13RDKXgnumKhtkrTuqNSyGVE/hVPQrFucdt3+k4FtcJtRaWDI/rrsXd8Vr81Whz2xaytZDNLeduIvMwyU+TLJtGubU498doB1bkCV9bLlfqLJ8L9fmYS+1YrojTRqREyK0et8VZkvrcE+p8fPP7OPMhkXamE8jk+Dpzo/9PjLmn1Mgzk87JX9X92gQfrYm+z6Xko0eLphfffd8HL3K1I4PBXdXcjFwuh3E6bbRFx+CmNx4bycXjOL4lE2QSWSCLqA1qEwmQIOqAOkQ2yCbqgrpEDsghugBdEPVAPSIX5BJ5II/IB/lEASggCkEhUR/UJ5IgSaRAikiDNFEEioguQZdEA9CAaAgaEo1AI6Ir0NWEbm0NtkZtBVCBqAgqEq2D1ok2QBtEm6BNoi3QFlEJVCLaBm0T7YB2iHZBu0R7oD2ifdA+0QHogOgQdEhUBpWJKqAK0RHoiOgYdExUBVWJaqAaUR1UJ2qAGkRNUJPoBHRCdAo6TShZgfn7C+9hUv+0nF9ZXql8zq4Wpstwnr1j79lHlmdf2CorsTKrMYuN2Hf2k12nvqV+pK5Tv25bZ2emM0vsTqR+/wPvqSur</latexit> Observed waveforms G A <latexit sha1_base64="76fLj5oFKMQr4i3cMV9HlgCW1pQ=">AAAF8HicbdTJThtBEAbghuCEOBskx1ysoEg5ITuK4Mq+L2bxAhihnnHbDJ6N7h7AWPMYkXIKyjUvk3PeJmPw/yssJXWr+qsqqU/lxL5nbLH4d2j42Uju+YvRl/lXr9+8fTc2/r5qokS7quJGfqTrjjTK90JVsZ71VT3WSgaOr2pOZ75fr10obbwo3LfdWB0Hsh16Lc+VNqOjRiDtqSv93mx6MjZRnCzeRuFxUhokE2IQ5ZPxkT+NZuQmgQqt60tjjkrF2B73pLae66s030iMiqXbkW11lKWhDJQ57t3+OS18zqRZaEU6O6Et3Or/Ez0ZGNMNnKyz/0fzsNbHJ2vqPJRay+6TRSdI778fQpD41tPR5X210kl8qa/SfL7RVK2GdIJer5FdMk3TO3JADskFuaQmqElSIEVqgVqkNqhNOgWdkjyQRzoDnZE6oA7JB/mkABSQQlBIikARKQbFpHPQOUmDNMmADMmCLCkBJaQL0AXpEnRJugJdkbqgLukadN2nO5uFzbJtDjRHmgfNkxZAC6RF0CJpCbREWgYtk1ZAK6RV0CppDbRGWgetkzZAG6RN0CZpC7RF2gZtk8qgMmkHtEPaBe2S9kB7pH3QPqkCqpCqoCqpBqqR6qA66QB0QDoEHWaUrcDSw4X3OKl+nSxNTU7tfJuYmRssw1HxUXwSX0RJTIsZsSLKoiJcEYnv4qe4yencj9xN7tdd6/DQYOaDuBe53/8AOFcoBg==</latexit> Input light field Full waveform inversion G A <latexit sha1_base64="76fLj5oFKMQr4i3cMV9HlgCW1pQ=">AAAF8HicbdTJThtBEAbghuCEOBskx1ysoEg5ITuK4Mq+L2bxAhihnnHbDJ6N7h7AWPMYkXIKyjUvk3PeJmPw/yssJXWr+qsqqU/lxL5nbLH4d2j42Uju+YvRl/lXr9+8fTc2/r5qokS7quJGfqTrjjTK90JVsZ71VT3WSgaOr2pOZ75fr10obbwo3LfdWB0Hsh16Lc+VNqOjRiDtqSv93mx6MjZRnCzeRuFxUhokE2IQ5ZPxkT+NZuQmgQqt60tjjkrF2B73pLae66s030iMiqXbkW11lKWhDJQ57t3+OS18zqRZaEU6O6Et3Or/Ez0ZGNMNnKyz/0fzsNbHJ2vqPJRay+6TRSdI778fQpD41tPR5X210kl8qa/SfL7RVK2GdIJer5FdMk3TO3JADskFuaQmqElSIEVqgVqkNqhNOgWdkjyQRzoDnZE6oA7JB/mkABSQQlBIikARKQbFpHPQOUmDNMmADMmCLCkBJaQL0AXpEnRJugJdkbqgLukadN2nO5uFzbJtDjRHmgfNkxZAC6RF0CJpCbREWgYtk1ZAK6RV0CppDbRGWgetkzZAG6RN0CZpC7RF2gZtk8qgMmkHtEPaBe2S9kB7pH3QPqkCqpCqoCqpBqqR6qA66QB0QDoEHWaUrcDSw4X3OKl+nSxNTU7tfJuYmRssw1HxUXwSX0RJTIsZsSLKoiJcEYnv4qe4yencj9xN7tdd6/DQYOaDuBe53/8AOFcoBg==</latexit> G† A † <latexit sha1_base64="y0ihnytmEtYb/XROIpDkBvRbpNI=">AAAF+nicbdTJThtBEAbghuCEOBuEYy5WrEg5ITuKyBVsFrPb4A0YYvWM2+PBs7m7B9s48yyRcgrKNe+Rc94mY/BfCktJ3ar+qkrqU5mh6yidy/2dmX0yl3r6bP55+sXLV6/fLCy+rasgkpaoWYEbyKbJlXAdX9S0o13RDKXgnumKhtkrTuqNSyGVE/hVPQrFucdt3+k4FtcJtRaWDI/rrsXd8Vr81Whz2xaytZDNLeduIvMwyU+TLJtGubU498doB1bkCV9bLlfqLJ8L9fmYS+1YrojTRqREyK0et8VZkvrcE+p8fPP7OPMhkXamE8jk+Dpzo/9PjLmn1Mgzk87JX9X92gQfrYm+z6Xko0eLphfffd8HL3K1I4PBXdXcjFwuh3E6bbRFx+CmNx4bycXjOL4lE2QSWSCLqA1qEwmQIOqAOkQ2yCbqgrpEDsghugBdEPVAPSIX5BJ5II/IB/lEASggCkEhUR/UJ5IgSaRAikiDNFEEioguQZdEA9CAaAgaEo1AI6Ir0NWEbm0NtkZtBVCBqAgqEq2D1ok2QBtEm6BNoi3QFlEJVCLaBm0T7YB2iHZBu0R7oD2ifdA+0QHogOgQdEhUBpWJKqAK0RHoiOgYdExUBVWJaqAaUR1UJ2qAGkRNUJPoBHRCdAo6TShZgfn7C+9hUv+0nF9ZXql8zq4Wpstwnr1j79lHlmdf2CorsTKrMYuN2Hf2k12nvqV+pK5Tv25bZ2emM0vsTqR+/wPvqSur</latexit> <latexit sha1_base64="76fLj5oFKMQr4i3cMV9HlgCW1pQ=">AAAF8HicbdTJThtBEAbghuCEOBskx1ysoEg5ITuK4Mq+L2bxAhihnnHbDJ6N7h7AWPMYkXIKyjUvk3PeJmPw/yssJXWr+qsqqU/lxL5nbLH4d2j42Uju+YvRl/lXr9+8fTc2/r5qokS7quJGfqTrjjTK90JVsZ71VT3WSgaOr2pOZ75fr10obbwo3LfdWB0Hsh16Lc+VNqOjRiDtqSv93mx6MjZRnCzeRuFxUhokE2IQ5ZPxkT+NZuQmgQqt60tjjkrF2B73pLae66s030iMiqXbkW11lKWhDJQ57t3+OS18zqRZaEU6O6Et3Or/Ez0ZGNMNnKyz/0fzsNbHJ2vqPJRay+6TRSdI778fQpD41tPR5X210kl8qa/SfL7RVK2GdIJer5FdMk3TO3JADskFuaQmqElSIEVqgVqkNqhNOgWdkjyQRzoDnZE6oA7JB/mkABSQQlBIikARKQbFpHPQOUmDNMmADMmCLCkBJaQL0AXpEnRJugJdkbqgLukadN2nO5uFzbJtDjRHmgfNkxZAC6RF0CJpCbREWgYtk1ZAK6RV0CppDbRGWgetkzZAG6RN0CZpC7RF2gZtk8qgMmkHtEPaBe2S9kB7pH3QPqkCqpCqoCqpBqqR6qA66QB0QDoEHWaUrcDSw4X3OKl+nSxNTU7tfJuYmRssw1HxUXwSX0RJTIsZsSLKoiJcEYnv4qe4yencj9xN7tdd6/DQYOaDuBe53/8AOFcoBg==</latexit> G† A † <latexit sha1_base64="y0ihnytmEtYb/XROIpDkBvRbpNI=">AAAF+nicbdTJThtBEAbghuCEOBuEYy5WrEg5ITuKyBVsFrPb4A0YYvWM2+PBs7m7B9s48yyRcgrKNe+Rc94mY/BfCktJ3ar+qkrqU5mh6yidy/2dmX0yl3r6bP55+sXLV6/fLCy+rasgkpaoWYEbyKbJlXAdX9S0o13RDKXgnumKhtkrTuqNSyGVE/hVPQrFucdt3+k4FtcJtRaWDI/rrsXd8Vr81Whz2xaytZDNLeduIvMwyU+TLJtGubU498doB1bkCV9bLlfqLJ8L9fmYS+1YrojTRqREyK0et8VZkvrcE+p8fPP7OPMhkXamE8jk+Dpzo/9PjLmn1Mgzk87JX9X92gQfrYm+z6Xko0eLphfffd8HL3K1I4PBXdXcjFwuh3E6bbRFx+CmNx4bycXjOL4lE2QSWSCLqA1qEwmQIOqAOkQ2yCbqgrpEDsghugBdEPVAPSIX5BJ5II/IB/lEASggCkEhUR/UJ5IgSaRAikiDNFEEioguQZdEA9CAaAgaEo1AI6Ir0NWEbm0NtkZtBVCBqAgqEq2D1ok2QBtEm6BNoi3QFlEJVCLaBm0T7YB2iHZBu0R7oD2ifdA+0QHogOgQdEhUBpWJKqAK0RHoiOgYdExUBVWJaqAaUR1UJ2qAGkRNUJPoBHRCdAo6TShZgfn7C+9hUv+0nF9ZXql8zq4Wpstwnr1j79lHlmdf2CorsTKrMYuN2Hf2k12nvqV+pK5Tv25bZ2emM0vsTqR+/wPvqSur</latexit> AG† † Figure 8.1: Illustration of five benchmark problems in the InverseBench. A represents the forward model that produces measurements from the underlying target. A † represents the inverse map. In the linear inverse scattering problem (left two), the measurements are the recorded data from the receivers, and the unknown source we aim to infer is the permittivity map of the object. The bottom panel displays the efficiency and accuracy plots for our benchmarked algorithms. Certain characteristics of the problem cause the efficiency and accuracy trade-offs of each algorithm to vary across tasks. In these plots, the larger radius of the points indicates greater interaction with the forward function A, as measured by the number of forward model evaluations. Table 8.1: Requirements on the forward model of the algorithms evaluated in our experiments. SVD of A Pseudo inverse of A Linear A Gradient DDRM [155] DDNM [306] ΠGDM [262] ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ – – – General guidance DPS [64] LGD [263] DPG [277] SCG [141] EnKG [356] ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ Variable-splitting DiffPIR [361] PnP-DM [320] DAPS [341] ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ Variational Bayes RED-diff [214] ✗ ✗ ✗ ✓ Sequential Monte Carlo FPS [89] MCGdiff [44] ✗ ✓ ✗ ✓ ✓ ✓ – – Category Method Linear guidance 8.2 Plug-and-Play Diffusion Priors for Inverse Problems We use the term Plug-and-Play Diffusion Prior (PnPDP) to refer to the class of recent methods that use diffusion models (or the denoising network within) as plugand-play priors [291] for solving inverse problems. See Section 5.3.2 for a brief overview. Table 8.1 lists the 14 representative PnPDP methods we selected and notes 127 their different requirements on the forward model A. Broadly speaking, existing PnPDP approaches can be grouped into four categories described below. Guidance-Based Methods Arguably the most popular approach to solving inverse problems with a pre-trained diffusion model is guidance-based methods [64, 155, 251, 262, 306], which modify Equation (5.8) by adding a likelihood score term, ∇𝒙𝑡 log 𝑝 𝑡 (𝒚 | 𝒙𝑡 ), along the diffusion trajectory using Equation (5.10). This term is related to the forward model A if the final clean 𝒙0 is a candidate source, in which case 𝑝(𝒚 | 𝒙 0 ) can be estimated by querying A. However, as we discussed, log 𝑝 𝑡 ( 𝒚 | 𝒙𝑡 ) is generally intractable so various approximations have been proposed [34, 64, 262, 265]. Variable Splitting Variable splitting is a widely used strategy for solving regularized optimization problems and conducting Bayesian inference [176, 296]. The core idea is to split the inference into two alternating steps. The first step uses the forward model to update or sample in the neighborhood of the most recent 𝒙𝑡 . The second step runs unconditional inference on 𝑝(𝒙𝑡 ), which amounts to running Equation (5.8) for a small amount of time. Example methods include those presented in Chapter 6, Chapter 7, and others in the literature [181, 260, 326, 341, 361]. Variational Bayes Variational Bayes methods approximate intractable distributions such as 𝑝(𝒙 | 𝒚) using some simpler parameterized distribution 𝑞 𝜃 [342]. The key idea is to find a 𝑞 𝜃 ∗ that both fits the measurements 𝒚 and agrees with the prior 𝑝(𝒙) in a KL divergence sense. Instead of directly sampling according to Equation (5.8), it uses the diffusion model as a prior within a variational inference framework [101, 103, 214]. Sequential Monte Carlo Sequential Monte Carlo (SMC) methods draw samples iteratively from a sequence of probability distributions. These methods represent probability distributions by a set of particles with associated weights, which asymptotically converge to a target distribution following a sequence of proposal and reweighting steps. Recent works have extended SMC methods to the sequential diffusion sampling process [44, 89, 283, 318], enabling zero-shot posterior sampling with diffusion priors. However, these methods are typically applicable only to inverse problems with linear forward models. 128 Table 8.2: Characteristics of different inverse problems in InverseBench. From left to right: whether the forward model is linear, whether one can efficiently compute the SVD from the forward model, the domain in which the inverse problem is defined, whether the forward model can be solved in closed form, whether one can access gradients from the forward model, and the noise type. Problem Linear A Efficient SVD of A Domain Closed-form A Gradient of A Noise type Linear inverse scattering Compressed sensing MRI Black hole imaging Full waveform inversion Navier-Stokes equation ✓ ✓ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ C𝑛 C𝑛 R𝑛 R𝑛 R𝑛 ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✗ Gaussian Real-world Non-additive Noise-free Gaussian 8.3 InverseBench In this section, we introduce the formulation and specific challenges of the five scientific inverse problems considered in InverseBench: linear inverse scattering, compressed sensing MRI, black hole imaging, full waveform inversion, and the Navier-Stokes equation. The characteristics of these inverse problems are summarized in Table 8.2. Their computational characteristics are summarized in Figure E.1. Detailed descriptions and formal definitions can be found in Appendix E.2. Linear Inverse Scattering Inverse scattering is an inverse problem that arises from optical microscopy, where the goal is to recover the unknown permittivity contrast 𝒙0 ∈ R𝑛 from the measured scattered lightfield 𝒚 sc ∈ C𝑚 . We consider the following formulation of inverse scattering 𝒚 sc = 𝑯( 𝒇tot ⊙ 𝒙0 ) + 𝒏 ∈ C𝑚 where 𝒇tot = 𝑮 ( 𝒇in ⊙ 𝒙0 ). (8.1) Here 𝑮 ∈ C𝑛×𝑛 and 𝑯 ∈ C𝑚×𝑛 are the discretized Green’s functions that model the responses of the optical system, 𝒇in and 𝒇tot are the input and total lightfields, ⊙ is the elementwise product, and 𝒏 is the measurement noise. Since this problem is a linearized version of the general nonlinear inverse scattering problem based on the first Born approximation, we refer to it as linear inverse scattering. This problem allows us to test algorithms designed specifically for linear problems. Compressed Sensing MRI Compressed sensing MRI is a technique that accelerates the scan time of MRI via subsampling. We consider the parallel imaging (PI) setup of CS-MRI (same as Section 2.2), which is widely adopted in research and practice. Mathematically, PI CS-MRI can be formulated as an inverse problem that aims to recover an image 𝒛 ∈ C𝑛 from 𝒚 𝑗 = 𝑴 𝑭𝑺 𝑗 𝒙0 + 𝒏 𝑗 ∈ C𝑚 for 𝑗 = 1, ..., 𝐽 129 where 𝑴 ∈ {0, 1} 𝑚×𝑛 is a subsampling operator and 𝑭 is the Fourier transform; 𝒚 𝑗 , 𝑺 𝑗 , and 𝒏 𝑗 are the measurements, sensitivity map, and the noise of the 𝑗-th coil, respectively. Compressed sensing MRI is a linear problem, but it poses significant challenges due to its high-dimensional nature, involvement of priors in the complex domain, and attention to fine-grained details. Black Hole Imaging Black hole imaging (BHI) is a technique of imaging black holes using Very Long Baseline Interferometry (VLBI). Each measurement for BHI, 𝑡 ∈ C (often referred to as visibility) is given by cross-correlating the recorded 𝑽{𝑎,𝑏} scalar electric fields of a pair of telescopes {𝑎, 𝑏} at time 𝑡 and measures a Fourier component of the target image 𝒙0 . However, the measurements are corrupted by the gain errors and phase errors due to the atmosphere. To correct for these errors, multiple noisy visibilities can be combined into quantities that are invariant to these errors, which are called closure phase and log closure amplitude measurements [23, 50] 𝑡 𝑡 𝑡 𝑡 𝒚 𝑡cp,{𝑎,𝑏,𝑐} := ∠(𝑽{𝑎,𝑏} 𝑽{𝑏,𝑐} 𝑽{𝑎,𝑐} ) := Acp,{𝑎,𝑏,𝑐} (𝒙0 ) ∈ R, © 𝑽{𝑎,𝑏} 𝑽{𝑐,𝑑} ª 𝑡 ® := A 𝑡 𝒚 logca,{𝑎,𝑏,𝑐,𝑑} := log ­­ logca,{𝑎,𝑏,𝑐,𝑑} (𝒙 0 ) ∈ R, ® 𝑡 𝑡 𝑽{𝑎,𝑐} 𝑽{𝑏,𝑑} « ¬ 𝑡 𝑡 where ∠ computes the angle of a complex number. Additionally, because the closure quantities do not constrain the total flux (i.e., summation of the pixel values) of the underlying black hole image, we add a constraint on the total flux defined as ∫ ∫ 𝒚 flux := 𝒙0 (𝜌, 𝛿)d𝜌d𝛿 ∈ R. (8.2) 𝜌 𝛿 Aggregating data over time intervals and telescope combinations, the overall forward model of BHI can be expressed as     𝒚 := Acp (𝒙0 ), Alogca (𝒙0 ), Aflux (𝒙 0 ) := 𝒚 cp , 𝒚 logca , 𝒚 flux , (8.3) h i where 𝒚 cp = 𝒚 𝑡cp,{𝑎,𝑏,𝑐} , ∀𝑡 ∈ T , {𝑎, 𝑏, 𝑐} is the set of all closure phase measureh i ments and 𝒚 cp = 𝒚 𝑡logca,{𝑎,𝑏,𝑐,𝑑} , ∀𝑡 ∈ T , {𝑎, 𝑏, 𝑐, 𝑑} is the set of all log closure amplitude measurements over the observation period T and combinations of telescopes. Since the closure quantities are nonlinear transformations of the visibilities, the forward model of BHI is non-convex. The inverse problem is further complicated by the need for super-resolution imaging beyond the intrinsic resolution of 130 the observations (i.e., maximum probed spatial frequency), as well as phase ambiguities, which can lead to multiple modes in the posterior distribution [269, 273]. Another challenge of BHI is that measurement noise is non-additive due to the usage of the closure quantities. We refer readers to Appendix C.2 for more details on the BHI setup. Full Waveform Inversion Full waveform inversion (FWI) aims to infer subsurface physical properties—such as compressional and shear wave velocities—using the full information of recorded waveforms. We consider the problem of recovering the discrete wave velocity 𝒙0 ∈ R𝑛 , or equivalently, the square slowness of the wave 𝒎 := 𝒙12 , from measurements of the pressure wavefield. The measurements 𝒚 are 0 given by ! −1 1 𝒚 = 𝑷𝑟 𝒈 = 𝑷𝑟 𝑨(𝒎) −1 𝑷𝑇𝑠 𝒒 𝑠 = 𝑷𝑟 𝑨 2 𝑷𝑇𝑠 𝒒 𝑠 ∈ R𝑚 , (8.4) 𝒙0 where 𝑷𝑟 is the sampling operator at the receiver locations, 𝑷𝑇𝑠 is the injection operator at the source locations, sampling operator at the receiver locations, 𝒈 is the discretized synthetic pressure wavefield (which is a function of location  and time), and 𝒒 𝑠 is the corresponding pressure source. Matrix 𝑨 𝒙12 is the 0 discretized operator for the acoustic (scalar) wave equation that models seismic wave propagation in heterogeneous acoustic media with constant density 1 2 𝜕𝑡 𝑔 − ∇2𝒙 𝑔 = 𝑞 𝑠 , (8.5) 2 𝒙0 where   𝑣, 𝑔 := 𝑔(𝒙, 𝑡), and 𝑞 𝑠 are the continuous counterpart of 𝒙 0 , 𝒈, and 𝒒 𝑠 . So, 𝑨 𝒙12 is the discretization of operator 𝒙12 𝜕𝑡2 − ∇2𝒙 , where ∇2𝒙 is the Laplacian oper0 0 ator. Since we only have measurements at the free surface, the inverse problem has non-unique solutions. One of the major challenges for FWI is the prohibitive computational expense, especially for large problems, as it usually requires numerous calls to the forward modeling process. Moreover, the conventional method for FWI, called the adjoint-state method, casts it as a local optimization problem [293, 294]. This means that a sufficiently accurate initial model is required, as the solution is only sought in its vicinity. FWI conventionally needs to start with a smoothed model derived from simpler ray-based methods [194, 211], which imposes a significantly strong prior. A general method with less reliance on initialization is highly desired. Navier-Stokes Equation Navier-Stokes equation is a classic benchmarking problem from fluid dynamics [143]. Its applications range from ocean dynamics to 131 climate modeling, where measurements of the atmosphere are used to calibrate the initial condition for the downstream numerical forecasting. We consider the forward model that is given by the following 2D Navier-Stokes equation for a viscous, incompressible fluid in vorticity form on a torus. 𝜕𝑡 𝝎(𝒙, 𝑡) + 𝒗(𝒙, 𝑡) · ∇𝒙 𝝎(𝒙, 𝑡) = 𝜈∇2𝒙 𝝎(𝒙, 𝑡) + 𝑓 (𝒙), 𝒙 ∈ (0, 2𝜋) 2 , 𝑡 ∈ (0, 𝑇] ∇𝒙 · 𝒗(𝒙, 𝑡) = 0, 𝒙 ∈ (0, 2𝜋) 2 , 𝑡 ∈ [0, 𝑇] 𝝎(𝒙, 0) = 𝝎0 (𝒙), 𝒙 ∈ (0, 2𝜋) 2 (8.6)   𝑟 ((0, 2𝜋) 2 ; R2 ) for any 𝑟 > 0 is the velocity field, 𝝎 = where 𝒗 ∈ 𝐶 [0, 𝑇]; 𝐻per
2 ∇𝒙 × 𝒗 is the vorticity, 𝝎0 ∈ 𝐿 per (0, 2𝜋) 2 ; R is the initial vorticity, 𝜈 ∈ R+ is
2 (0, 2𝜋) 2 ; R is the forcing function. The the viscosity coefficient, and 𝑓 ∈ 𝐿 per solution operator G is defined as the operator mapping the vorticity from the initial vorticity to the vorticity at time 𝑇, i.e., G : 𝝎0 → 𝝎𝑇 . We consider the problem of recovering the initial vorticity field 𝒙 0 := 𝝎0 from the noisy partial measurements 𝒚 of the vorticity field 𝒘𝑇 at time 𝑇 given by 𝒚 = 𝑷𝑮 (𝒙 0 ) + 𝒏 where 𝑷 is the sampling operator, 𝒏 is the measurement noise, and 𝑮 (·) is the discretization of G. The Navier-Stokes equation does not admit a closed-form solution, so there is no closed-form gradient available for the solution operator. We implement the forward model using a pseudo-spectral solver with adaptive time stepping [127]. Obtaining an accurate numerical gradient via automatic differentiation through the numerical solver is challenging due to the extensive computation graph expanded after thousands of discrete time steps. 8.4 8.4.1 Experiments Experimental Setup Here we provide a summary of our experimental setup. The details about each inverse problem and its corresponding datasets can be found in Appendix E.2. For each problem, we train a diffusion model on the training set using the pipeline from [151], and use the same checkpoint for all PnPDP methods on each problem for a fair comparison. Technical details of DM pre-training can be found in Appendix E.3. Linear Inverse Scattering We create a dataset of fluorescence microscopy images using the online simulator [314]. The training set consists of 10,000 HL60 nucleus 132 permittivity images. The test and validation sets contain 100 and 10 permittivity images, respectively. We curate the test and validation samples so that all test samples have less than 0.6 cosine similarities to those in the training set. Compressed Sensing MRI We use the multi-coil raw 𝑘-space data from the fastMRI knee dataset [338]. We exclude the first and last 5 slices of each volume for training and validation, since they do not contain much anatomical information, and resize all images down to 320 × 320 following the preprocessing procedure of [145]. In total, we use 25,012 images for training, 6 images for hyperparameter search, and 94 images for testing. Black Hole Imaging We leverage a dataset of General Relativistic MagnetoHydroDynamic (GRMHD) [223] simulated black hole images as our training data. The training set consists of 50,000 resized 64×64 images. Since this dataset is not publicly available, we generate synthetic images from a pre-trained diffusion model for both the validation and test datasets. Specifically, we use 5 sampled images for the validation set and 100 sampled images for the test set. Full Waveform Inversion We adapt the CurveFaultB dataset [80], which presents the velocity maps that contain faults caused by shifted rock layers. We resize the original data to a resolution of 128×128 with bilinear interpolation and anti-aliasing. The training set consists of 50,000 velocity maps. The test and validation sets contain 10 and 1 velocity maps, respectively. Navier-Stokes Equation We create a dataset of non-trivial initial vorticity fields by first sampling from a Gaussian random field and then evolving Equation (8.6) for five time units. The equation setup follows Iglesias, Law, and Stuart [143] and Li et al. [185]. We set the Reynolds number to 200 and the spatial resolution to 128×128. The training set consists of 10,000 samples. The test and validation sets contain 10 and 1 samples, respectively. 8.4.2 Evaluation Metrics Accuracy Metrics We use the Peak Signal-to-Noise Ratio (PSNR) and Structure Similarity Index Measure (SSIM) [310] as the generic ways to quantify recovery of the true source. For all the problems except for black hole imaging, we use the ℓ2 error ∥A (b 𝒙 ) − 𝒚∥ 2 to measure the consistency of a reconstruction b 𝒙 with the measurements 𝒚. For black hole imaging, the closure quantities are invariant 133 under translation, and so we measure the best fit under any shift alignment. We also assess the Blur PSNR, where images are blurred to match the target resolution of the telescope. We evaluate data misfit via the 𝝌2 statistic on two closure quantities: 2 ) and the log closure amplitude ( 𝝌 2 2 the closure phase ( 𝝌cp logca ). A 𝝌 value close to 1 indicates better data fitting. To facilitate a comparison between underfitting ( 𝝌2 > 1) and overfitting ( 𝝌2 < 1), we report a unified metric defined as 𝝌˜ 2 = 𝝌2 · 1{ 𝝌2 ≥ 1} + 1 · 1{ 𝝌2 < 1}. 𝝌2 (8.7) For FWI and Navier-Stokes experiments, we also use the relative ℓ2 error ∥b 𝒙− 𝒙0 ∥ 2 /∥𝒙 0 ∥ 2 as it is a commonly used primary accuracy metric in PDE problems [143]. Efficiency Metrics We define a set of efficiency metrics in Table E.7 to evaluate the computational complexity of inverse algorithms more thoroughly. These metrics fall into two categories: (1) total metrics that measure the overall computational cost; (2) sequential metrics that help identify bottlenecks where forward model or diffusion model queries cannot be parallelized. Ranking Score To assess the relative ranking of different PnP diffusion models across various problems, we define the following ranking score for each problem. Given a set of accuracy or efficiency metrics {ℎ 𝑘 } 𝐾𝑘=1 , we rank the algorithms according to each individual metric. Suppose algorithm 𝑙 has the rank 𝑅 𝑘 (𝑙) out of 𝐿 algorithms under the metric 𝑘. Its ranking score on this metric is given by score 𝑘 (𝑙) = 100 × (𝐿 − 𝑅 𝑘 (𝑙) + 1) /𝐿. For each problem, we calculate the average ranking score to assess overall performance: 𝐾 score 8.4.3 problem 1 ∑︁ score 𝑘 (𝑙). (𝑙) = 𝐾 𝑘=1 Main Findings The full experimental results for each problem are provided in Appendix E.1 as tables. Below, we highlight some key insights distilled from these results. How do PnPDP methods work compared to conventional baselines? Our primary finding is that, given a suitable dataset for training a DM prior, PnPDP methods generally outperform conventional baselines. This is evident in Figure 8.1, where Ground truth FISTA DDRM DDNM ∏GDM DPS LGD Pseudo-inverse DiffPIR PnP-DM DAPS RED-diff FPS MCG-diff Ground truth L1-Wavelet TV UNet VarNet ScoreMRI CSGM DPS DiffPIR PnP-DM DAPS RED-diff Ground truth EHT-Imaging SMILI DPS LGD Ground truth Adam Adam* LBFGS* DPS LGD DiffPIR PnP-DM DAPS RED-diff DiffPIR Navier-Stokes equation Full waveform inversion Black hole imaging Compressed sensing MRI Linear inverse scattering 134 PnP-DM DAPS RED-diff Ground truth EKI DPS-GSG SCG DPG EnKG Figure 8.2: Qualitative comparison showing representative examples of PnPDP methods and domain-specific baselines across five inverse problems. Note that for full waveform inversion, Adam∗ and LBFGS∗ are initialized with Gaussianblurred ground truth, serving as references. the PnPDP methods generally lie higher along the vertical axis. This finding is as expected, given that the baselines do not incorporate such strong prior information. However, if the classic optimization baselines are initialized well, then they sometimes outperform PnPDP methods, most of which cannot naturally incorporate an 135 initialization beyond white noise. For example, in FWI, PnPDP methods clearly outperform the classic baseline methods if the baselines are initialized randomly or from a constant. But if initialized with a good guess (e.g., a heavily blurred ground truth image), they consistently outperform the current PnPDP methods. That being said, the fact that PnPDP methods rely much less on initialization than the traditional optimization methods is already an intriguing property. See qualitative comparison in Figure 8.1 and quantitative comparison in Table E.5. How do PnPDP methods compare with each other? In the problems where the forward model has a closed-form expression, methods that require more gradient queries, such as DAPS and PnP-DM, tend to be more accurate. However, since they have more queries to the forward model, they are also more expensive, as shown in Figure 8.1. Additionally, these methods require more careful tuning as they usually have larger hyperparameter spaces, as shown in Table E.101 . In the problems where the forward model has no closed-form expression, particularly a forward model defined by a PDE system and implemented as a numerical PDE solver, this trend does not hold. In fact, DAPS and PnP-DM perform poorly, as shown in Figure 8.1 and Table E.5. These methods also demonstrate an increased level of numerical instability and sensitivity to hyperparameters, as shown in Figure 8.3: minor adjustments in step size can lead to either unconditional generation results that ignore measurements (with slightly smaller steps) or complete failure (with slightly larger steps). This performance degradation stems from a critical limitation in many current PnPDP algorithms: they do not account for stability conditions required to query a forward model. For example, in the FWI and Navier-Stokes equation, the input of the forward model must satisfy the Courant–Friedrichs–Lewy (CFL) condition [76] to produce stable solutions. This issue is particularly pronounced for methods like DAPS and PnP-DM, which incorporate Langevin Monte Carlo (LMC) as a subroutine, because LMC introduces additional Gaussian noise at each step and thus exacerbates instability compared to other PnPDP methods. How does the performance vary with different levels of measurement sparsity? As measurement sparsity increases, making the inverse problem more ill-posed, we observe an increasingly wide performance gap between PnPDP methods and baselines. Figure 8.4 illustrates this trend across three problems, showing that 1 Note that tuning the hyperparameters of PnPDP approaches is still much more efficient than retraining a neural network that is typically required for end-to-end approaches. 136 Figure 8.3: Illustration of the failures of PnPDP methods (DAPS) as an example) on full waveform inversion. With a small learning rate, DAPS is numerically stable but does not solve the inverse problem effectively. With a slightly larger learning rate, DAPS produces a noisy velocity map that breaks the stability condition of the PDE solver, resulting in a complete failure. the average performance gain of top PnPDP methods over baselines grows with increasing measurement sparsity. How well do PnPDP methods deal with different forward models? For linear inverse problems, our results demonstrate that PnPDP methods can effectively deal with varying forward models without the need for parameter tuning. To validate this, we conduct a controlled experiment in CS-MRI, where we maintain a consistent measurement sparsity while altering the subsampling pattern (from vertical to horizontal lines). We assess the average performance variation across three method categories: traditional baselines, end-to-end approaches, and PnPDP methods. The average absolute performance change for PnPDP methods is 0.48dB (PSNR) and 0.016 (SSIM), comparable to the traditional baseline methods at 1.62dB (PSNR) and 0.027 (SSIM), but significantly smaller than the end-to-end methods, which exhibit changes of 9.58dB (PSNR) and 0.21 (SSIM). These findings indicate that PnPDP methods are more robust than both baseline and end-to-end methods when handling different forward models.2 How well do PnPDP methods handle out-of-distribution sources? In general, if the unknown source falls outside the diffusion prior distribution, PnPDP methods tend to generate solutions that are biased toward the prior. As illustrated in Figure 8.5a, most solutions produced by PnPDP methods exhibit a black hole ring 2 For end-to-end approaches, this is considered as an out-of-distribution test. 137 Figure 8.4: Relative performance of plug-and-play diffusion prior methods compared with traditional baselines under different levels of measurement sparsity on different tasks. Metrics are averaged over multiple PnPDP methods. The performance difference increases in general as the measurement becomes sparser. Ground truth DPS PnP-DM (a) DAPS RED-diff Ground truth DDRM DDNM ∏GDM DPS LGD PSNR=27.55 PSNR=32.24 PSNR=25.20 PSNR=26.34 PSNR=22.13 DiffPIR PnP-DM DAPS RED-diff FPS PSNR=29.04 PSNR=28.24 PSNR=30.04 PSNR=30.48 PSNR=28.28 (b) Figure 8.5: PnPDP methods on out-of-distribution test samples. (a) Black-hole imaging problem on digit inputs; and (b) inverse scattering on sources that contain 9 cells, while the prior model is trained on images with 1 to 6 cells. feature characteristic of the diffusion prior. This suggests that while PnPDP approaches are flexible in capturing high-dimensional priors, they are limited in their ability to reliably recover “surprising” sources that lie outside the support of the diffusion prior distribution. However, when the unknown source is close to the diffusion prior distribution, PnPDP methods can recover it effectively, as demonstrated in Figure 8.5b. 8.5 Conclusion We conclude this chapter by highlighting key research opportunities for advancing PnPDP methods in solving inverse problems. In this chapter, we only consider the setting where the forward model is exact and explicit, while the forward model in real-world problems often involve approximations. Studying the robustness of PnPDP methods when there is a mismatch between the assumed and actual forward model would be an essential step toward their practical deployment. This includes evaluating how well they handle imperfect and/or partially known forward model representations. Another research challenge we identify is that the current PnPDP 138 methods do not account for stability conditions required to query a forward model, which leads to degraded performance and numerical instability. However, many scientific inverse problems are based on PDE systems that require certain conditions on the inputs to simulate numerically and violating these constraints can result in meaningless solutions. Another direction for improvement is inference speed. As shown in Figure 8.1, almost all the PnPDP methods are less computationally efficient than the conventional baselines. There remains substantial room for optimization. We hope that InverseBench can serve as a systematic benchmark and catalyst for developing more advanced diffusion-based posterior samplers capable of addressing these challenges. 139 Chapter 9 CONCLUSION Computational imaging systems push the boundaries of imaging techniques by enhancing physical sensors with computational algorithms. In this thesis, we presented two lines of research that improve sampling in the computational imaging pipeline using machine learning (ML)—one around measurement acquisition with physical sensors and another around posterior estimation with computational algorithms. Together, these contributions address key challenges and unlock new capabilities for existing computational imaging systems. This chapter concludes the thesis by highlighting the key takeaways and proposing some promising directions for future research. 9.1 Key Takeaways In Part I, we investigated how to improve sampling strategies for compressed sensing MRI (CS-MRI) through end-to-end ML approaches. Traditional CS-MRI methods use fixed, predetermined subsampling patterns that are agnostic to both the underlying target and downstream tasks. In contrast, we showed that ML-based approaches can effectively learn sampling strategies that are adaptive to the underlying target or specific to the downstream objectives: • Sequential sampling: In Chapter 3, we proposed a method for learning sequential sampling strategies for CS-MRI, leveraging the fact that measurements in CS-MRI must be taken sequentially rather than simultaneously. Our learned multi-step strategies decide which measurements to acquire based on the previously observed samples, thereby making better selections compared to nonadaptive baselines. In addition, our method outperforms reinforcement learning–based approaches in both accuracy and training efficiency. • Task-specific sampling: In Chapter 4, we developed a framework that directly optimizes sampling patterns of CS-MRI for a downstream task, rather than reconstruction metrics that may not align with clinical or diagnostic objectives. While this may sound like an easy tweak, we showed that the naïve end-to-end optimization approach does not lead to a robust strategy. Instead, we introduced a two-stage training procedure: first pre-training a backbone model for reconstruc- 140 tion, followed by task-specific fine-tuning with a prediction head—analogous to modern foundation model workflows. We showed that our approach consistently outperforms existing methods that do not consider downstream tasks in terms of task-specific metrics. We also implemented the learned sampling strategies on a real MRI scanner and validated their effectiveness with experimentally collected data. In Part II, we introduced a new approach of using diffusion models (DMs) as priors for estimating the posterior distributions of inverse problems in Bayesian inference. The proposed approach addresses several core challenges: • Principled method: In Chapter 6, we proposed PnP-DM, a posterior sampling method based on the Split Gibbs Sampler (SGS) [296]. Unlike previous methods, our method avoids making uncontrolled approximations that could lead to significant sampling errors, thereby enabling more accurate posterior estimation. We drew a key connection between SGS and the EDM framework [151], which provides a unified and principled way of incorporating DMs as priors for solving inverse problems. • Unified framework: In Chapter 7, we built on the core ideas of PnP-DM and extended them into a unified sampling framework for diffusion-based posterior estimation. We presented four instantiations of this framework, each designed to address one limitation in PnP-DM. These include incorporating text as a semantic prior, scaling to high-dimensional video data, handling nondifferentiable (black-box) forward models, and enabling posterior inference in discrete spaces. While addressing diverse challenges, all four methods share a common alternating-update structure inspired by PnP-DM, and they exhibit both strong theoretical guarantees and robust empirical performance across a range of tasks. • Comprehensive benchmark: In Chapter 8, we presented a large-scale benchmark designed to rigorously evaluate diffusion-based approaches for solving scientific inverse problems. The benchmark encompasses 14 widely used methods and 5 representative inverse problems spanning multiple scientific fields. Through comprehensive evaluations and ablation studies, we provided critical insights into the capabilities and shortcomings of current techniques and highlighted future opportunities for improving posterior sampling in computational imaging. 141 9.2 Futures Directions While the contributions above advance both measurement acquisition and posterior estimation in computational imaging, many open problems remain. We outline a few promising directions below. 9.2.1 Measurement Acquisition (Part I) Optimizing Measurement Acquisition for Posterior Sampling Methods The proposed methods in Part I rely on end-to-end network architectures, which require paired data to train and could be prone to overfitting due to a lack of uncertainty quantification in the decision-making process. Incorporating posterior sampling methods like PnP-DM from Part II could address problems where paired data is unavailable and lead to better performance and robustness. It remains a challenge to optimize the sampling patterns with posterior sampling methods that do not permit backpropagation. Implementation of Sequential Strategies While our learned sequential sampling strategies showed significant improvements in retrospective experiments, it remains an open question whether these improvements translate to real-world performance when deployed on an actual MRI scanner. A key implementation challenge lies in integrating deep learning-based sequential policies into MRI sampling sequences without causing disruptions for the underlying acquisition protocols. Extension to Other Imaging Modalities While our methods are developed for CS-MRI, they may generalize to other modalities. For example, in computed tomography (CT), one could optimize the angular views of X-ray acquisition for downstream diagnostic tasks. Exploring these extensions could broaden the impact of adaptive and task-driven sampling strategies. 9.2.2 Posterior Estimation (Part II) Partially-Known or Mismatched Forward Models The proposed methods in Part II assume the forward model A to be accurate, which is sometimes unrealistic in practice. Many real-world problems, such as photographic deblurring and radio astronomy, involve uncertainty or a mismatch in the forward process. Developing posterior estimation methods that can handle these imperfections remains an open challenge. 142 More General Noise Distributions The current framework is designed under the assumption of additive Gaussian measurement noise. However, many realworld imaging problems involve structured or non-Gaussian noise. Extending the framework to accommodate more general and complex noise distributions (e.g., Poisson, impulse, structured noise) is an important direction. 4D Imaging While the proposed framework can handle inverse problems on 3D videos, many real-world imaging problems involve inherently 4D data. Scaling up the current framework to handle the spatiotemporal evolution of 3D volumes would be an interesting future direction. For example, one potentially viable idea is to partition 3D volumes into lower-dimensional slabs or cubes that are more computationally tractable. Compatibility with Autoregressive Generative Models Autoregressive models have recently achieved state-of-the-art performance on generative modeling [281]. Even though they are not strictly DMs, they share a similar iterative generation procedure over a sequence of image manifolds. It would be worthwhile to explore whether the insights and techniques we developed for DM-based posterior estimation can be leveraged for designing more powerful techniques with autoregressive models. 9.3 Concluding Remarks As computational imaging continues to push the boundaries of science and engineering, the opportunities to improve how we see and interpret the world are expanding rapidly. This thesis contributes to that progress by proposing new approaches for more intelligent measurement acquisition and principled posterior estimation—two foundational stages of the imaging pipeline. We believe that our results not only address concrete challenges and demonstrate practical impact, but also open the door to promising future directions that further enhance the performance, robustness, and efficiency of computational imaging systems. Realizing these advances will require even deeper integration between physical models and learning-based algorithms. Looking ahead, we hope the insights and frameworks developed in this work will serve as building blocks for the next generation of intelligent imaging systems. Part III Appendices 143 144 Appendix A APPENDIX FOR CHAPTER 3 A.1 Model Architectures Architecture of the reconstructor (𝖱θ) Reconstruction Reconstructor Zero-filled reconstruction 𝖱θ x̃(t) ̂ x (t) 2 64 64 64 128 64 64 1 1 128×128 Output Image 128 128 128 256 128 128 64×64 64 256 256 256 512 32×32 128 256 256 Conv2D + Instance Norm + ReLU 16×16 Concatenation 256 512 512 512 1024 512 512 AveragePooling2D UpSampling2D Element-wise `2-norm <latexit sha1_base64="f35eFNbD4watLzyoLMgoLnv07ao=">AAADP3icfZLNjtMwEMe94WspX7tw5BJRIXFYRUmar95WcOG4SLS7UltVjjPJmnWcyHYQJco7cIW34TF4Am6IKzectKraFDFSNJOZ33j+tiYuGZXKtn8cGbdu37l77/j+4MHDR4+fnJw+ncqiEgQmpGCFuIqxBEY5TBRVDK5KATiPGVzGN2/a+uVHEJIW/L1albDIccZpSglWOjWdA2NLd3kytC27M/MwcDbBEG3sYnlqDOdJQaocuCIMSzlz7FItaiwUJQyawbySUGJygzOY6ZDjHOSi7uQ25kudScy0EPrjyuyyux01zqVc5bEmc6yuZb/WJv9Vm1UqjRY15WWlgJP1oLRipirM9u5mQgUQxVY6wERQrdUk11hgovQL7U35tJa6l4tz/Z9Aql+6q9aZAOBNLbK4qe0z2wqi0NcudIOgR+LPlYBd0vNaMoqCsEfGrNoFR+MODEZu1Az2yUJgnm1ZS89uG/wg8LRz3GjcO1lAsoXD8UjDZ/bBhfBqy/i+F+qTvNDt3Cj0/yPVch0vavUGoduqiMZar14rp79Eh8HUtZzAct55w/PXmwU7Rs/RC/QKOShE5+gtukATRNAH9AV9Rd+M78ZP45fxe40aR5ueZ2jPjD9/AUw0AUc=</latexit> 8×8 512 1024 1024 Addition Figure A.1: Flow diagram of the reconstructor, R𝜽 (·), in the proposed framework. We use a residual U-Net reconstructor for all of our models. A.1.1 Reconstructor In Figure A.1, we provide the network diagram for the reconstructor. As stated in the main paper, we use the standard U-Net architecture following [12, 13, 338]. The input to the reconstructor is the complex-valued zero-filled image, and the output is a single-channel real-valued image. The initial convolutional layer has 64 channels, which are doubled after every downsampling layer. The reconstructor uses skipconnections, depicted as white horizontal arrows, that concatenate feature maps at different levels for easier optimization. A.1.2 Samplers Figure A.2 shows the detailed architecture of our neural samplers. On top, we show the sampler architecture for the 1D line sampling setting, which is a fivelayer Multilayer Perceptron (MLP) network with 512 channels in the hidden layers. The input to the sampler includes the up-to-date sampling mask 𝑴 (𝑡) , 𝑘-space 145 Architecture of the 2D sampler ( q) 2D sampler 𝖱q Previous mask (t) m Rescale Previous masked k-space (t) y Previous reconstruction (t) x˜ p(t+1) Binarize <latexit sha1_base64="uqsqAJbBUNtcrSZDN6M2jfRCAHw=">AAAF8HicbdRJTxsxFAdwQ0lL0w3aYy9RUaWeUFJVcGXfl7BkAYKQZ+KEIbNhe4AQzceo1FNRr/0yPffbdAL5/1WWJ9l6/r1nyfLhObHvGVss/h0afjaSe/5i9GX+1es3b9+Njb+vmijRrqq4kR/puiON8r1QVaxnfVWPtZKB46ua05nv12sXShsvCvdtN1bHgWyHXstzpc3oqBFIe+o4Qa+UnoxNFCeLt1F4nJQGyYQYRPlkfORPoxm5SaBC6/rSmKNSMbbHPamt5/oqzTcSo2LpdmRbHWVpKANljnu3b04LnzNpFlqRzlZoC7f6/42eDIzpBk7W2X+jeVjr45M1dR5KrWX3yaITpPfPDyFIfOvp6PK+WukkvtRXaT7faKpWQ2Y/1mtkm0zT9I4ckENyQS6pCWqSFEiRWqAWqQ1qk05BpyQP5JHOQGekDqhD8kE+KQAFpBAUkiJQRIpBMekcdE7SIE0yIEOyIEtKQAnpAnRBugRdkq5AV6QuqEu6Bl336c5mYbNsmwPNkeZB86QF0AJpEbRIWgItkZZBy6QV0AppFbRKWgOtkdZB66QN0AZpE7RJ2gJtkbZB26QyqEzaAe2QdkG7pD3QHmkftE+qgCqkKqhKqoFqpDqoTjoAHZAOQYcZZSOw9HDgPU6qXydLU5NTO98mZuYGw3BUfBSfxBdREtNiRqyIsqgIV0Tiu/gpbnI69yN3k/t11zo8NLjzQdyL3O9/5Zwn9w==</latexit> New mask (t+1) [u 𝖲 p(t+1)] m Uniformly random matrix u 5 64 64 64 128 64 64 1 128×128 Output Image 128 128 128 256 128 128 64×64 64 256 256 256 512 256 256 32×32 128 16×16 Conv2D + Instance Norm + ReLU 256 512 512 512 1024 512 Concatenation 512 AveragePooling2D UpSampling2D 1024 8×8 512 1024 Architecture of the 1D sampler ( q) 1D sampler 𝖱q Previous mask (t) y Previous reconstruction (t) x˜ Flatten Previous masked k-space (t) Uniformly random matrix u Flatten 𝖲 Rescale m Flattening layer 128×128×5 512 512 FC layer + ReLU 512 p(t+1) Binarize <latexit sha1_base64="uqsqAJbBUNtcrSZDN6M2jfRCAHw=">AAAF8HicbdRJTxsxFAdwQ0lL0w3aYy9RUaWeUFJVcGXfl7BkAYKQZ+KEIbNhe4AQzceo1FNRr/0yPffbdAL5/1WWJ9l6/r1nyfLhObHvGVss/h0afjaSe/5i9GX+1es3b9+Njb+vmijRrqq4kR/puiON8r1QVaxnfVWPtZKB46ua05nv12sXShsvCvdtN1bHgWyHXstzpc3oqBFIe+o4Qa+UnoxNFCeLt1F4nJQGyYQYRPlkfORPoxm5SaBC6/rSmKNSMbbHPamt5/oqzTcSo2LpdmRbHWVpKANljnu3b04LnzNpFlqRzlZoC7f6/42eDIzpBk7W2X+jeVjr45M1dR5KrWX3yaITpPfPDyFIfOvp6PK+WukkvtRXaT7faKpWQ2Y/1mtkm0zT9I4ckENyQS6pCWqSFEiRWqAWqQ1qk05BpyQP5JHOQGekDqhD8kE+KQAFpBAUkiJQRIpBMekcdE7SIE0yIEOyIEtKQAnpAnRBugRdkq5AV6QuqEu6Bl336c5mYbNsmwPNkeZB86QF0AJpEbRIWgItkZZBy6QV0AppFbRKWgOtkdZB66QN0AZpE7RJ2gJtkbZB26QyqEzaAe2QdkG7pD3QHmkftE+qgCqkKqhKqoFqpDqoTjoAHZAOQYcZZSOw9HDgPU6qXydLU5NTO98mZuYGw3BUfBSfxBdREtNiRqyIsqgIV0Tiu/gpbnI69yN3k/t11zo8NLjzQdyL3O9/5Zwn9w==</latexit> New mask (t+1) [u 𝖲 p(t+1)] m 128 Rescale Rescaling layer Binarize Binarizing layer Figure A.2: Flow diagram of the samplers, S𝒒 (·), in the proposed framework. We use a Multilayer Perceptron for 1D line sampling and a U-Net for 2D point sampling. Both networks take previous observations as inputs and output a probability map, which is rescaled and binarized into the final sub-sampling mask at the next iteration. measurements 𝒚 (𝑡) , and the 𝑘-space of reconstruction 𝑭b 𝒙 (𝑡) . A uniformly random vector 𝒖 is also given as an input for Bernoulli sampling. The output is a binary sampling mask 𝒎 (𝑡+1) generated through stochastic binarization with the random vector 𝒖. The bottom shows the 2D point sampling architecture. For the 2D setting, the sampler uses a U-Net architecture [250]. Inputs and outputs are the same as the 1D line sampler. A.2 A.2.1 Baseline Details Random Random subsampling is a widely used 𝑘-space sampling pattern that utilizes stochastic subsampling for creating incoherent artifacts that can be easily recognized and 146 removed through post-processing techniques [110]. In our implementation, we first pre-select the central low-frequency 𝑘-space region and then uniformly sample from the remaining lines or points until exhausting the sampling budget. We pair this sampler with a U-Net reconstructor trained with this random sampling pattern for 50 epochs. The reconstructor architecture and training schedule are the same as those of our sequential models. A.2.2 Equispaced Equispaced subsampling is another widely used 1D line sampling baseline [338]. Lines are sampled equidistantly from each other with an offset to achieve the desired sampling budget. We choose the equispaced baseline due to its ease of implementation on existing MRI scanners [338]. A.2.3 Spectrum Spectrum is a data-driven 𝑘-space sampling approach introduced in [289]. The spectrum method utilizes the fact that 𝑘-space samples with higher power often contain more information about the image’s large-scale structure. To identify the 𝑘space samples, we average the magnitude spectrum of all fully-sampled 𝑘-space data in the training set. We then select samples with the largest average power, which will form the final subsampling mask. We pair this sampler with a U-Net reconstructor trained using measurements acquired according to this learned sampling pattern. A.2.4 LOUPE LOUPE [12] is the state-of-the-art single-shot sampling method. It jointly optimizes an subsampling pattern along with an image reconstruction network. We follow the official implementation in [12] but replace the binarization function in the subsampling mask generation with a straight-through estimator following [20, 344]. The same modification is applied to our method as described in Section 3.3.1. The reconstructor architecture and other hyperparameters are the same as those of our sequential methods. A.2.5 PG-MRI PG-MRI [13] formulates the 𝑘-space sample selection as a partially observable Markov decision process and learns a sequential sampling policy using the policy gradient algorithm [18]. According to their evaluations, PG-MRI outperforms multiple baseline approaches, including uniform random [110], equispaced sampling [338] and another Monte-Carlo tree search-based reinforcement learning ap- 147 proach [146]. We use the author’s official code for our implementation. The reconstructor is pre-trained using the uniform random policy. We then plug the pretrained reconstructor into the pipeline to evaluate the image reconstruction reward for sampling policy training. All other hyperparameters are the same as the original paper [13]. A.2.6 Evaluator Non-seq. over LOUPE 250250 1-Step Seq. over LOUPE 2-Step Seq. over LOUPE 200200 4-Step Seq. over LOUPE 250 250 250 95.46% 96.54% 93.01% 200 200 200 150150 150 150 150 100100 100 100 100 5050 50 50 min=-5.48 0 0-2 ≤-2 -1 max=8.38 0 0 1 2 2 3 4 4 5 min=-1.28 6 0 -2 ≥6 ≤-2 -1 max=8.38 0 0 1 2 2 3 4 4 5 0 -2 97.19% 50 min=-3.92 6 4× acceleration factor # of test samples [351] proposed a greedy acquisition framework that trains a ResNet to reconstruct the anatomical image simultaneously with an Evaluator network trained to select the most uncertain measurements in 𝑘-space. As there is no official code available for [351], we use the reimplementation in [231]. The reconstructor uses a cascade ResNet architecture with four cascade blocks, each composed of three residual bottleneck layers [126] followed by a data consistency layer [255]. The evaluator contains four convolutional blocks, and each consists of a 4×4 convolution, instance normalization, and a LeakyReLU activation layer [209]. We use a batch size of 128 and train the model for 200 epochs with a learning rate of 1𝑒 − 4 using the Adam optimizer [162]. ≥6 ≤-2 -1 max=8.46 0 0 1 2 2 3 4 4 5 min=-1.21 6 0 -2 ≥6 ≤-2 -1 max=8.59 0 0 1 2 2 3 4 4 5 6 ≥6 SSIM improvement distribution on the test set (averaged over 3 runs) Figure A.3: Histograms of pair-wise SSIM differences on all 1,851 test images using 1D line sampling with 4× acceleration factor. We calculate the improvement of our model with different sequential steps over LOUPE. Our sequential model and non-sequential baseline significantly outperform LOUPE for most subjects. A.3 A.3.1 Further Analyses Pair-Wise Comparison for 1D Line Sampling We report extended pair-wise SSIM comparisons for 1D line sampling on the test set. Figure A.3 and Figure A.4 show the SSIM improvement distribution on the test set. Here, we compare our method with the previous sequential sampling approach Evaluator [351] and PG-MRI [13] by measuring their improvements over the stateof-the-art single-shot sampling baseline LOUPE [12]. Our model outperforms LOUPE for 97.19% of the targets while both previous sequential sampling baselines perform substantially worse than the LOUPE baseline, with only 1.58% and 5.64% 500500 Evaluator over LOUPE PG-MRI over LOUPE 4-Step Seq. (ours) over LOUPE 500 500 1.58% 400400 97.19% 5.64% 400 400 300300 300 300 200200 200 200 100100 100 min=-23.78 0 0-12 -10 -8 ≤-12 -8 max=0.94 -6 -4 -4 -2 0 0 2 100 min=-17.16 4 4 6 8 4× acceleration factor # of test samples 148 0 -12 ≥8 ≤-12 -10 -8 -8 min=-1.21 max=3.5 -6 -4 -4 -2 0 0 2 4 4 6 8 0 -12 ≥8 ≤-12 -10 -8 -8 -6 -4 -4 -2 max=8.59 0 0 2 4 4 6 8 ≥8 SSIM improvement distribution on the test set (averaged over 3 runs) Figure A.4: Histograms of pair-wise SSIM differences on all 1,851 test images using 1D line sampling with 4× acceleration factor. We calculate the improvement of the Evaluator (left), PG-MRI (middle), and our best sequential model (4-step sequential) (right) over LOUPE. Our 4-step sequential model significantly outperforms LOUPE, while the other two baselines are substantially worse than LOUPE for most subjects. of the subjects outperforming LOUPE for Evaluator and PG-MRI, respectively. This highlights the importance of combining co-design and sequential sampling in an end-to-end fashion for MR 𝑘-space sampling. A.3.2 Pair-Wise Comparison for 2D Point Sampling In Figure A.5, we show the SSIM improvement distribution for different methods compared to the LOUPE baseline. The histograms across each row show that, for all three acceleration factors, the non-sequential model has marginal improvement over the LOUPE baseline; in contrast, our sequential model significantly outperforms LOUPE as we increase the number of sequential sampling steps. By inspecting Figure A.5 down each column, our models demonstrate increasingly larger advantages over LOUPE as the number of sampled measurements increases from 16× to 4× (i.e., the acceleration factor decreases). A.4 Additional Reconstruction Examples We present some additional reconstruction examples in Figure A.6 and Figure A.7. 149 Non-seq. over LOUPE 400400 2-Step Seq. over LOUPE 4-Step Seq. over LOUPE 500 500 77.42% 88.74% 96.96% 400 400 400 300300 300 300 300 200200 200 200 200 100100 100 100 0 0-2 min=-1.22 -1.5 -1 max=1.87 -0.5 0 0.5 500500 1 1.5 2 2.5 min=-1.06 3 0 -2 -1.5 -1 max=2.83 -0.5 0 0.5 1 1.5 2 2.5 3 -1.5 -1 max=2.15 -0.5 0 0.5 1 1.5 2 2.5 min=-0.94 3 57.05% 60.18% 0 -2 83.04% 400 400 400 300300 300 300 300 200200 200 200 200 100100 100 100 0 0-2 min=-1.82 -1.5 -1 max=1.93 -0.5 0 0.5 1.5 2 2.5 min=-1.53 3 0 -2 -1.5 -1 max=2.91 -0.5 0 0.5 1 1.5 2 2.5 -1.5 -1 max=3.06 -0.5 0 0.5 1 1.5 2 2.5 300300 300 200200 200 200 200 100100 100 100 max=4.46 0 0.5 1 1 1.5 2 2 2.5 min=-3.17 3 0 -2 ≥3 ≤-2 -1.5 -1 -1 max=5.10 -0.5 0 0 0.5 1 1 1.5 2 2 2.5 0 -2 2 2.5 3 -1 max=3.42 -0.5 0 0.5 1 1.5 2 2.5 3 100 min=-2.04 3 1.5 76.91% 400 300 0 -1.5 56.42% 400 -0.5 1 89.26% 0 -2 300 min=-3.21 0.5 min=-0.94 3 400 0 0-2 -1.5 -1 ≤-2 -1 0 500 51.08% 46.98% 400400 0 -2 500 500 max=2.74 -0.5 100 min=-0.64 3 -1 16× acceleration factor 500500 1 -1.5 500 500 500 400400 100 min=-1.11 0 -2 8× acceleration factor # of test samples 1-Step Seq. over LOUPE 500 74.05% 4× acceleration factor 500500 ≥3 ≤-2 -1.5 -1 -1 max=5.20 -0.5 0 0 0.5 1 1 1.5 2 2 2.5 min=-1.22 3 0 -2 ≥3 ≤-2 -1.5 -1 -1 max=5.34 -0.5 0 0 0.5 1 1 1.5 2 2 2.5 3 ≥3 SSIM improvement distribution on the test set (averaged over 3 runs) Figure A.5: Histograms of pair-wise SSIM differences on all 1,851 test images using 2D line sampling with 4× (first row), 8× (second row), and 16× (third row) acceleration factors. We calculate the improvement of our model with different sequential steps over LOUPE in each column. For all three acceleration factors, our sequential model outperforms the non-sequential baseline and LOUPE on an increasing percentage of test samples as the number of sequential steps increases. Our sequential models also have increasingly larger advantages over LOUPE as the number of sampled measurements increases (i.e., the acceleration factor decreases). 150 ID: 394 (4x) Random Evaluator 90.28 PG-MRI 91.10 LOUPE 94.26 4-Step Seq. (ours) Ground truth 97.12 SSIM Sampling pattern 4× reconstruction 87.48 k-space ID: 1083 (4x) 89.57 90.95 93.35 96.68 SSIM Sampling pattern 4× reconstruction 87.80 k-space ID: 1506 (4x) 91.42 92.99 93.51 96.61 SSIM Sampling pattern 4× reconstruction 89.02 k-space Figure A.6: Visualizations of the reconstructions of the 394th (top), 1083th (middle), 1506th (bottom) test images with an acceleration factor of 4× for 1D line sampling. Zoomed-in image patches highlight our significant improvement over previous methods. We find that our learned masks for the 1D line sampling case usually consist of adjacent low-frequency samples. However, only a few of the learned samples have their conjugate symmetric points sampled as well. Our learned policy appears to leverage the conjugate symmetry of the 𝑘-space and trade off taking more measurements with taking fewer measurements with higher SNR (by effectively sampling the same measurement twice). 151 ID: 1355 (4x) Random 97.50 LOUPE 1-Step Seq. (ours) 4-Step Seq. (ours) 97.63 98.03 98.03 Ground truth SSIM 4× reconstruction 92.81 Spectrum Sampling pattern k-space ID: 1355 (8x) 95.60 95.95 96.38 96.55 SSIM Sampling pattern 8× reconstruction 87.83 k-space Figure A.7: Visualizations of the reconstructions of the 1355th test sample with an acceleration factor of 4× (top) and 8× (bottom) for 2D point sampling. A zoomed-in image patch is shown along with the cumulative 𝑘-space measurements selected by each policy. Orange arrows point out the regions where our sequential approach provides more accurate and detailed local structures. 152 Appendix B APPENDIX FOR CHAPTER 4 B.1 Implementation Details In this section, we describe the implementation details of Tackle and the baseline methods. B.1.1 Further Information on Datasets and Their Preparation For each dataset in Section 4.4, we randomly split the data into training, validation, and test sets on the patient level, which means that each validation or test slice comes from a patient whose images are not used for training. B.1.1.1 ROI-Oriented Reconstruction For this task, we use all images with Meniscus Tear (MT) annotations in the fastMRI+ dataset [338, 354]. We follow the specific data splitting in [231], which results in 4,158 slices for training, 210 slices for validation, and 201 slices for testing. We crop the center of the 𝑘-space of each image and adjust the size and position of each bounding box accordingly. B.1.1.2 Brain Tissue Segmentation For this task, we use the 109th coronal slice of each volume in the OASIS dataset [213]. The access to the dataset can be found here1 . Specifically, we use the 4-label tissue-type segmentation maps, which include segments of the cortex, the white matter, the subcortical gray matter, and the cerebrospinal fluid (CSF). We split the data into 248 slices for training (≈60%), 82 slices for validation (≈20%), and 84 slices for testing (≈20%). B.1.1.3 Knee Tissue Segmentation For this task, we use all the sagittal slices in the SKM-TEA dataset [83] that contains all four segmentation labels (the patellar cartilage, the femoral cartilage, the tibial cartilage, and the meniscus). We split the data into 2,935 slices for training (≈60%), 1,040 slices for validation (≈20%), and 987 slices for testing (≈20%). 1 https://github.com/adalca/medical-datasets/blob/master/neurite-oasis.md 153 B.1.1.4 Pathology Classification For this task, we use all the images acquired by the FLAIR sequence in the BRATS dataset [219] to detect the existence of the Glioma tumor. FLAIR stands for fluid attenuated inversion recovery, a kind of inversion recovery sequence that is commonly used for detecting various brain lesions due to its ability to suppress the CSF signal and enhance lesion-to-background contrast [154]. Empirically, we find that it is more accurate to detect the existence of the Glioma on FLAIR images than on images with the other contrasts in the BRATS dataset. We split the data into 30,495 slices for training (≈60%), 9,996 slices for validation (≈20%), and 10,353 slices for testing (≈20%). B.1.2 B.1.2.1 Training and Implementation Details of Tackle Training Details For all experiments, the models were trained by the Adam [162] optimizer with 𝛽1 = 0.9, 𝛽2 = 0.999 on a single NVIDIA A6000 GPU. We choose the best learning rate among {1e-2, 1e-3, 1e-4}, and trained all models until convergence (i.e., no improvement for 10 epochs on the validation set according to the task-specific evaluation metric). For instance, if a Tackleseg. model achieves a higher Dice score on the validation set than all previous epochs on epoch 42, the model will be saved as a checkpoint. If it has no further improvement until epoch 52, then the training will be terminated, and the saved checkpoint on epoch 42 will be used for reporting the final results. The training of our proposed framework is conducted by retrospective subsampling on fully sampled measurements. The first module is the sampler, which requires no input and directly learns a matrix that contains the probability of sampling each 𝑘-space frequency. The output of the sampler is the subsampling mask 𝒎, in which 1 represents the measurements to be sampled and 0 represents those not to be sampled. Sampling amounts to taking the element-wise product between 𝒎 and the fully sampled measurements 𝒌, which gives us the subsampled measurements 𝒚 := 𝒎 ⊙ 𝒌. The retriever will then take the two-channel complex measurements 𝒚 as the input and output a single-channel real image b 𝒙 . In the multi-coil case, 𝒚 contains signals from multiple coils with different sensitivity maps, and b 𝒙 is reconstructed by taking the root sum of squares across all coils. For reconstruction tasks (full-FOV reconstruction and ROI-oriented reconstruction), b 𝒛=b 𝒙 will be the final output for loss calculation and back-propagation. For downstream tasks beyond reconstruction, 154 we feed b 𝒙 into an additional predictor which gives a prediction b 𝒛 . In this case, b 𝒛 will be the final output for loss calculation and back-propagation. B.1.2.2 Retriever Architecture Following the E2E-VarNet architecture [267], our retriever operates in 𝑘-space and contains 12 refinement steps, each of which includes a U-Net [250] with independent weights from each other. The update rule of the 𝑡-th refinement step is     𝒌 (𝑡+1) = 𝒌 (𝑡) − 𝜂 (𝑡) 𝒎 ⊙ 𝒌 (𝑡) − 𝒚 + G (𝑡) 𝒌 (𝑡) where 𝒎 is the subsampling mask, 𝒚 is the measurement, 𝒌 (𝑡) is the reconstructed 𝑘-space, 𝜂 (𝑡) is a data consistency parameter, and G (𝑡) is the refinement module defined as      (𝑡) −1 (𝑡) (𝑡) (𝑡) . 𝑹𝑭 𝒌 := 𝑭𝑬 UN 𝒌 G Here, 𝑬 and 𝑹 are the expand and reduce operations across all coils (see [267] for more details), and UN (𝑡) is the U-Net model at the 𝑡-th step. Specifically, we use the standard U-Net [250] architecture with 2 input and output channels, 4 average downpooling layers, and 4 up-pooling layers. The model starts with an 18-channel output for the input layer and doubles the number of channels with each downsampling layer. Between every two pooling layers are two convolution modules, each of which consists of a 3 × 3 convolution, an instance normalization [286], and a LeakyReLU activation with negative slope of 0.2. The input to each U-Net is first normalized to zero mean and standard deviation of 1 before being fed into the network, and will be normalized back to the original mean and standard deviation after passing through the network. After 12 refinement steps, the final output layer of the retriever is an inverse Fourier transform followed by a root-sum-squares reduction for each pixel over all coils. The output of the retriever is a batch of single-channel images. For reconstruction tasks, a loss function will be directly applied to the output. For non-reconstruction tasks, there is an additional predictor module. B.1.2.3 Predictor Architecture For tissue segmentation tasks, the predictor is a U-Net model that has the same architecture as the refinement network described above, except for the following differences: There is 1 input channel and 𝑐 output channels (where 𝑐 is the number of segmentation classes). The model starts with a 64-channel output for the input layer. The convolution modules use the Parametric ReLU activation. There is no 155 normalization after the output. We use the U-Net implementation in the MONAI package [45]. For the pathology classification task, the predictor is a ResNet18 model with 1 input channel and 2 output dimensions. We normalize the input to zero mean and a standard deviation of 1 before feeding it into the network. We use the ResNet implementation in the torchvision package [212]. B.1.3 Pre-Select Region and Sensitivity Map Estimation Among all the datasets considered in this manuscript, fastMRI+ [338, 354] and SKM-TEA [83] contain multi-coil 𝑘-space measurements. Reconstruction from multi-coil 𝑘-space data requires estimation of the coil sensitivity maps, i.e., 𝑺𝑖 in Equation (2.2), using the central low-frequency region of the 𝑘-space, called the Auto-Calibration Signal (ACS). We set the ACS region as a square around the DC component that contains 1/8 of the subsampling budget. For example, if a dataset contains 𝑘-space measurements of size 256 × 256, for 8× acceleration, we select the center 32 × 32 low frequencies as the ACS. We also include the pre-determined ACS region for single-coil 𝑘-space experiments because we find that it stabilizes the training of some baselines. Given the ACS, we estimate coil sensitivity maps using the Sensitivity Map Estimation (SME) module introduced in [267]. In contrast to the ESPIRiT algorithm [285], SME estimates the sensitivity maps with a CNN applied to each coil image independently. The architecture of the CNN in SME is the same as the U-Net in each E2E-VarNet cascade, except with an 8-channel output instead of an 18-channel output for the input layer. B.1.4 Further Details on the Implementation of SemuNet [309] For the brain and knee segmentation tasks, we compare the proposed method with SemuNet [309]. Originally demonstrated for a brain segmentation task, it also aims to jointly optimize a subsampling mask, a reconstructor, and a task predictor for the downstream accuracy. SemuNet uses a hybrid of ℓ1 loss for reconstruction and cross-entropy loss for segmentation. Since the code of SemuNet has not been released, we have tried to reproduce the results in the original paper to our best efforts. Specifically, we follow their proposed loss function and architecture of the sampler, the residual U-Net reconstructor, and a U-Net predictor. We follow the original paper to use an Adam optimizer [162] and not pre-select low-frequency measurements. However, since our tasks and datasets are different from those in [309], we empirically find that the learning rate and the parameter 𝜆 that adjusts 156 kz <latexit sha1_base64="liwQ68ktjtzoZopy27b+sQe4b28=">AAACKXicbVDLTgJBEJzFF+IL9OhlIjHxRHaNryPRi0eMgiSwIb3DLEyYmV1nZk1wwyd41Y/wa7ypV3/EAfYgj0o6qa7qTroriDnTxnW/ndzK6tr6Rn6zsLW9s7tXLO03dJQoQusk4pFqBqApZ5LWDTOcNmNFQQScPgaDm7H/+EyVZpF8MMOY+gJ6koWMgLHS/aDz0imW3Yo7AV4kXkbKKEOtU3KK7W5EEkGlIRy0bnlubPwUlGGE01GhnWgaAxlAj7YslSCo9tPJrSN8bJUuDiNlSxo8Uf9vpCC0HorATgowfT3vjcWlHn2SoBQMl5qBGM32c0IrMeGVnzIZJ4ZKMj0zTDg2ER7HhrtMUWL40BIgitlPMemDAmJsuAWboTef2CJpnFa8i8r53Vm5ep2lmUeH6AidIA9doiq6RTVURwT10Ct6Q+/Oh/PpfDk/09Gck+0coBk4v3/Pq6aL</latexit> m <latexit sha1_base64="fmViNNZAH/JPjC2rh0PDQYdZ3TQ=">AAACZXicbVHJTsMwEHXDXrYWEBcORBQkTlVSie2AhODCESRakNqqctxJsWo7wZ4goihfw9dwhRNfwG/gLgdaOpKt5/dmPJ7nIBbcoOd9F5y5+YXFpeWV4ura+sZmqbzVMFGiGdRZJCL9FFADgiuoI0cBT7EGKgMBj0H/ZqA/voI2PFIPmMbQlrSneMgZRUt1SpetLoS2dnhTJtOepmmejXavenGRF1sIbzgpH7YCmcn8MO+UKl7VG4b7H/hjUCHjuOuUC6VWN2KJBIVMUGOavhdjO6MaORNguyUGYsr6tAdNCxWVYNrZsH3uHlmm64aRtkuhO2T/VmRUGpPKwGZKis9mWhuQMzV4UVQPRpslBjKfPE8RzQTD83bGVZwgKDZ6ZpgIFyN3YLnb5RoYitQCyjS3k7rsmWrK0H5M0XroTzv2HzRqVf+0enJfq1xdj91cJnvkgBwTn5yRK3JL7kidMPJOPsgn+Sr8OOvOjrM7SnUK45ptMhHO/i9dy7xT</latexit> kx <latexit sha1_base64="Oj0xAAhAA0FXJMsuYCk8mgzKWts=">AAACKXicbVDJTsMwEHXKVsrWwpGLRYXEqUoQ27GCC8ciaKnURpXjTlqrthNsBxFF/QSu8BF8DTfgyo/gLge6PGmkN+/NSDMviDnTxnW/ndzK6tr6Rn6zsLW9s7tXLO03dJQoCnUa8Ug1A6KBMwl1wwyHZqyAiIDDYzC4GfmPz6A0i+SDSWPwBelJFjJKjJXuB52XTrHsVtwx8CLxpqSMpqh1Sk6x3Y1oIkAayonWLc+NjZ8RZRjlMCy0Ew0xoQPSg5alkgjQfja+dYiPrdLFYaRsSYPH6v+NjAitUxHYSUFMX897I3GpB0+SKEXSpWYghrP9nNBKTHjlZ0zGiQFJJ2eGCccmwqPYcJcpoIanlhCqmP0U0z5RhBobbsFm6M0ntkgapxXvonJ+d1auXk/TzKNDdIROkIcuURXdohqqI4p66BW9oXfnw/l0vpyfyWjOme4coBk4v3/MJaaJ</latexit> ky <latexit sha1_base64="+AEj9Deg7gkzcQwiZe9109+Z2D4=">AAACKXicbVDJTsMwELXLVsrWwpFLRIXEqUoQ27GCC8ci6CK1UeW4k9aq7QTbQYqqfgJX+Ai+hhtw5Udw2xzo8qSR3rw3I828IOZMG9f9xrm19Y3Nrfx2YWd3b/+gWDps6ChRFOo04pFqBUQDZxLqhhkOrVgBEQGHZjC8m/jNF1CaRfLJpDH4gvQlCxklxkqPw27aLZbdijuFs0y8jJRRhlq3hIudXkQTAdJQTrRue25s/BFRhlEO40In0RATOiR9aFsqiQDtj6a3jp1Tq/ScMFK2pHGm6v+NERFapyKwk4KYgV70JuJKD54lUYqkK81AjOf7BaGdmPDGHzEZJwYknZ0ZJtwxkTOJzekxBdTw1BJCFbOfOnRAFKHGhluwGXqLiS2TxnnFu6pcPlyUq7dZmnl0jE7QGfLQNaqie1RDdURRH72iN/SOP/An/sI/s9EcznaO0Bzw7x/N6KaK</latexit> Figure B.1: Conceptual illustration of the subsampling setup with a knee example. The back dots on the 𝑘 𝑦 -𝑘 𝑧 plane represent 𝑘-space trajectories along 𝑘 𝑥 , which are illustrated by the black arrows. We consider subsampling in the two phase-encoding dimensions (𝑘 𝑦 and 𝑘 𝑧 ) of a 3D Cartesian sequence, where the subsampling pattern 𝒎 is learned from data for some specific downstream task. the trade-off between the two losses are suboptimal for our settings. Therefore, we conduct a grid search on the learning rate in {0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5} and 𝜆 ∈ {0.0001, 0.001, 0.01}. For both 16× and 64× accelerations, we choose the best combination of parameters based on the performance on the validation set, and report the Dice score on the test set. B.2 Subsampling Setup and Implementation In this work, we optimize the subsampling mask 𝒎 over all 2D subsampling patterns. We implement 2D subsampling patterns in practice by subsampling in the two phase encoding dimensions of a 3D Cartesian sequence based on the 2D pattern, as illustrated in Figure B.1. We denote the number of trajectories along 𝑘 𝑦 and 𝑘 𝑧 (the two phase encoding directions) as 𝑛 𝑘 𝑦 and 𝑛 𝑘 𝑧 , respectively. For the fully sampling scenario, one needs to sequentially sample a total of 𝑛 𝑘 𝑦 𝑛 𝑘 𝑧 trajectories, which could take a long time to acquire in practice. Given a 2D subsampling mask 𝒎, we subsample in the 𝑘 𝑦 -𝑘 𝑧 plane according to 𝒎. If 𝒎 has an acceleration ratio of 𝑅, the subsampling sequence only takes 𝑛 𝑘 𝑦 𝑛 𝑘 𝑧 /𝑅 trajectories, and the acquisition time will be reduced by a factor 𝑅 in practice. One can obtain the slice-wise 2D 𝑘-space measurements 𝒚 by taking the 1D inverse Fourier Transform of the raw 3D 𝑘-space data along 𝑘 𝑥 . We have implemented a 4×-accelerated version of the sequence 157 Table B.1: Comparison of TackleROI on in- and out-of-distribution samples under different acceleration ratios (𝑅). Metric 𝑅 Samples w/ MT (in-distribution) Samples w/o MT (out-of-distribution) PSNR (dB) 4 8 16 37.65 33.28 32.06 36.92 32.88 31.74 that we use in Section 4.5 on the Siemens IDEA sequence programming platform, using the subsampling scheme we describe above. Figure 4.12 demonstrates that the prospective subsampling version of our learned sequence achieves the same level of visual quality as the fully sampling version but only takes a quarter of the scan time. This result highlights the real-world practicality of our approach. B.3 Additional Validation on Out-of-Distribution Data In Chapter 4, we show that TackleROI improves the reconstruction of ROIs that contain the meniscus tear (MT). In practice, it is likely that a healthy subject or someone with a different pathological lesion from the meniscus tear will get scanned. So it is important that the learned sequence should also generalize to out-of-distribution subjects. Here we take our trained TackleROI models with 4×, 8×, and 16× accelerations from our ROI reconstruction experiments, and directly test them on images that do not contain the meniscus tear, without additional fine-tuning. The results are summarized in Table B.1. Although it is not surprising that TackleROI performs better on the in-distribution data (samples w/ MT), we want to point out that the two numbers above correspond to two different test sets and are thus not directly comparable. The main takeaway is that TackleROI can robustly recover samples without MT, even if it is trained on samples with MT. As discussed in Section 4.4.1, TackleROI improves the ROI reconstruction by trading off 𝑘-space frequencies for the local anatomy to attain improved resolution. We find that such a strategy can lead to satisfactory reconstruction quality even when the underlying subject does not contain the target pathology. B.4 Additional Results We provide a box-plot comparison for the Meniscus Tear ROI reconstruction in Figure B.2. We also provide some visual examples for the tumor classification task 158 FastMRI — Meniscus Tear ROI Reconstruction PSNR (dB) (a) Top: percentage of improved samples (↑) Bottom: p-value of the paired t-test (↓) 70.1% 9.43e-13 95% CI LP+UNFOV (26.95±0.63) 86.6% 1.86e-37 PD+UNFOV (28.23±0.44) 8× 95.0% 7.12e-61 LOUPEFOV (30.32±0.55) TackleROI (34.04±0.63) PSNR (dB) (b) 16× 68.2% 3.33e-9 95% CI LP+UNFOV (25.16±0.60) 98.5% 2.29e-83 77.6% 1.55e-19 PD+UNFOV (26.05±0.48) LOUPEFOV (27.32±0.57) TackleROI (31.53±0.61) Figure B.2: Box plots of the Meniscus Tear ROI reconstruction results under 8× (a) and 16× (b) accelerations. Within the rectangle between each pair of methods, the top number is the percentage of samples that get improved, and the bottom number is the 𝑝-value given by the paired samples t-test. A higher percentage and a lower 𝑝-value indicate a more significant improvement. The 95% confidence intervals for all methods are given below their names. in Figure B.3. For the three reconstruction-oriented baselines (left three columns), the inputs to the predictor network are typical reconstructions. Optimized end-toend for classification accuracy, the retriever of Tackleclass. learns a feature map that highlights the region where a tumor could exist. Similar to the segmentation results, we find that the end-to-end model Tackleclass. circumvents the typical reconstruction but preserves image-level features that are helpful for downstream classification prediction. 159 Predictor input PD+UNrecon. Predictor input Test_20 LOUPErecon. Tacklerecon. Tackleclass. 32.83 34.38 37.88 —— 38.48 40.62 46.50 —— Ground Truth PSNR (dB) Figure B.3: Visualization of the input of the predictor network for the brain tumor classification task under 16× acceleration on two samples of the BRATS dataset. Similar to the segmentation results, as a co-design method, Tackleclass. circumvents the typical “reconstruction” in terms of point-wise similarity with the ground truth image. Instead, the retriever learns a feature map that highlights the region around the tumor for the downstream prediction. 160 Appendix C APPENDIX FOR CHAPTER 6 C.1 C.1.1 Theory Interpolation of PnP-DM In this section, we formally introduce the interpolation of PnP-DM. We consider the case where the coupling strength 𝜂 is constant, i.e., 𝜂 𝑘 ≡ 𝜂 and make the following assumption. Assumption C.1.1. There exists a unique 𝑡 ∗ such that 𝜎(𝑡 ∗ ) = 𝜂. This assumption is satisfied for common diffusion models. Popular choices of √ the noise level schedule include 𝜎(𝑡) = 𝑡 or 𝜎(𝑡) = 𝑡, which are monotonically increasing functions of 𝑡. We first present two propositions showing that the two steps in SGS can be implemented by running two SDEs. Proposition C.1.2 (Brownian bridge for the likelihood step). For iteration 𝑘 with iterate 𝒙 (𝑘) , the likelihood step of SGS is equivalent to solving the following SDE from 𝑡 = 0 to 𝑡 = 1: d𝒙𝑡 = 𝜂2 ∇ log 𝜙𝑡 (𝒙𝑡 )d𝑡 + 𝜂d𝒘 𝑡 ∫ 1 ∥𝒙 − 𝒛∥ 22 ]d𝒛. where 𝒙0 = 𝒙 (𝑘) and 𝜙𝑡 (𝒙) := exp[− 𝑓 (𝒛; 𝒚) − 2𝜂2 (1−𝑡) (C.1) Proof. This proposition is due to the Brownian bridge construction presented in   Lemma 4 of [336]. This SDE satisfies that 𝑝(𝒙1 |𝒙0 ) ∝ exp − 𝑓 (𝒙1 ; 𝒚) − 2𝜂1 2 ∥𝒙0 − 𝒙1 ∥ 22 . Therefore, solving Equation (C.1) from 𝑡 = 0 to 𝑡 = 1 is equivalent to taking a likelihood step. Proposition C.1.3 (EDM reverse diffusion for the prior step). For iteration 𝑘 with iterate 𝒛 (𝑘) , the prior step of SGS is equivalent to solving the following SDE from 𝑡 = 𝑡 ∗ to 𝑡 = 0:   d𝒙𝑡 = 𝑢(𝑡)𝒙𝑡 − 𝑣(𝑡) 2 ∇ log 𝑝 𝑡 (𝒙𝑡 ) d𝑡 + 𝑣(𝑡)d𝒘¯ 𝑡 (C.2) √︁ ¤ (𝑡) where 𝒙𝑡 ∗ = 𝑠(𝑡 ∗ )𝒛 (𝑘) , 𝑢(𝑡) := 𝑠𝑠(𝑡) , 𝑣(𝑡) := 𝑠(𝑡) 2𝜎(𝑡)𝜎(𝑡), ¤ and 𝑝 𝑡 is the distribution of 𝑠(𝑡)𝒙 + 𝑠(𝑡)𝜎(𝑡)𝝐 with 𝒙 following the prior distribution 𝑝(𝒙) ∝ exp(−𝑔(𝒙)) and 𝝐 ∼ N (0, 𝑰). 161 Proof. First note that Equation (C.2) is exactly Equation (6.6) written in terms of 𝑢(𝑡) and 𝑣(𝑡). We know that the Equation (C.2) is the reverse SDE of the following SDE d𝒙𝑡 = 𝑢(𝑡)𝒙𝑡 d𝑡 + 𝑣(𝑡)d𝒘 𝑡 , (C.3) where 𝒙0 ∼ 𝑝(𝒙) and 𝑝 𝑡 is the marginal distribution of 𝒙𝑡 . As shown in Section 6.3.2, it holds for Equation (C.3) that   1 2 𝑝(𝒙0 |𝒙𝑡 ) ∝ exp −𝑔(𝒙0 ) − ∥𝒙0 − 𝒙𝑡 /𝑠(𝑡)∥ 2 . 2𝜎(𝑡) 2 As Equation (C.2) is the time-reversed process of Equation (C.3), they share the same path distribution and thus the same conditional distribution 𝑝(𝒙0 |𝒙𝑡 ). So, if we set 𝒙𝑡 ∗ = 𝑠(𝑡 ∗ )𝒛 (𝑘) , we have that     1 1 (𝑘) 2 (𝑘) 2 𝑝(𝒙0 |𝒙𝑡 ∗ ) ∝ exp −𝑔(𝒙0 ) − ∥𝒙0 − 𝒛 ∥ 2 ∝ exp −𝑔(𝒙0 ) − 2 ∥𝒙0 − 𝒛 ∥ 2 , 2𝜎(𝑡 ∗ ) 2 2𝜂 which is the desired conditional distribution of the prior step. Therefore, solving Equation (C.2) from 𝑡 = 𝑡 ∗ to 𝑡 = 0 is equivalent to taking a prior step. Due to Proposition C.1.2 and Proposition C.1.3, the SDEs Equation (C.1) and Equation (C.2) implement the two desired conditional distributions in SGS. In PnPDM, the prior step involves a network that approximates the score function of the prior distribution, i.e., 𝒔𝑡 ≈ ∇ log 𝑝 𝑡 , so the continuous-time process for the actual update is   d𝒙𝑡 = 𝑢(𝑡)𝒙𝑡 − 𝑣(𝑡) 2 𝒔𝑡 (𝒙𝑡 ) d𝑡 + 𝑣(𝑡)d𝒘¯ 𝑡 . (C.4) We can then interpolate PnP-DM by considering a dynamic that alternates between running Equation (C.1) and Equation (C.4). Since each likelihood step takes 1 unit of time and each prior step takes 𝑡 ∗ units of time, the total time of the interpolating process for 𝐾 iterations of PnP-DM is 𝑇𝐾 := 𝐾 (𝑡 ∗ + 1). We use 𝜏 to denote the time that has elapsed from initializing PnP-DM with 𝒙 (0) . We define {𝜇𝜏 } and {𝜋𝜏 } as the distributions at time 𝜏 of the non-stationary process initialized at 𝒙 (0) ∼ 𝜇0𝑋 (Figure 6.2 top) and the stationary process initialized at 𝒙 (0) ∼ 𝜋 𝑋 (Figure 6.2 bottom), respectively. Therefore, we have • 𝜇𝜏 = 𝜇 𝑘𝑋 , 𝜋𝜏 = 𝜋 𝑋 for 𝜏 = 𝑘 (𝑡 ∗ + 1) with 𝑘 = 0, · · · , 𝐾, and • 𝜇𝜏 = 𝜇 𝑘𝑍 , 𝜋𝜏 = 𝜋 𝑍 for 𝜏 = 𝑘 (𝑡 ∗ + 1) + 1 with 𝑘 = 0, · · · , 𝐾 − 1. 162 C.1.2 A Key Lemma for General Diffusion Processes Before proving our main result, we present a key lemma for our analysis, which quantifies the time derivative of the KL divergence in terms of the Fisher divergence along a pair of general diffusion processes. Lemma C.1.4. Given the following pair of diffusion processes d𝒙𝑡 = 𝑏(𝒙𝑡 , 𝑡)d𝑡 + 𝑐(𝑡)d𝒘 𝑡 (C.5) de 𝒙𝑡 = e 𝑏(e 𝒙𝑡 , 𝑡)d𝑡 + 𝑐(𝑡)d𝒘 𝑡 (C.6) where 𝑏(·, ·) : R𝑛 × R → R𝑛 , e 𝑏(·, ·) : R𝑛 × R → R𝑛 , and 𝑐(·) : R → R. Let 𝜇𝑡 be the distribution of 𝒙𝑡 initialized with 𝒙 0 ∼ 𝜇0 for Equation (C.5), and let 𝜇e𝑡 be the distribution of e 𝒙𝑡 initialized with e 𝒙0 ∼ 𝜇 e0 for Equation (C.6). Then we have ∫ 2 1 𝑐(𝑡) 2 e FI (𝜇𝑡 ||e 𝜇𝑡 ) + 𝜕𝑡 KL(𝜇𝑡 ||e 𝜇𝑡 ) ≤ − 𝑏 − 𝑏 𝜇𝑡 . (C.7) 𝑡 𝑡 2 4 𝑐(𝑡) 2 Proof of Lemma C.1.4. Writing 𝑏(·, 𝑡) as 𝑏 𝑡 and e 𝑏(·, 𝑡) as e 𝑏 𝑡 , by the Fokker-Planck equations of Equation (C.5) and Equation (C.6), we have that       𝑐(𝑡) 2 𝑐(𝑡) 2 ∇ log 𝜇𝑡 − 𝑏 𝑡 𝜇𝑡 and 𝜕𝑡 𝜇 e𝑡 = div ∇ log 𝜇 e𝑡 − e 𝑏𝑡 𝜇 e𝑡 . 𝜕𝑡 𝜇𝑡 = div 2 2 163 d Defining 𝜙(𝑥) := 𝑥 log 𝑥 and 𝜙′ (𝑥) = d𝑥 𝜙(𝑥) = log 𝑥 + 1, we can calculate   ∫ 𝜇𝑡 𝜇 e𝑡 𝜕𝑡 KL(𝜇𝑡 ||e 𝜇𝑡 ) = 𝜕𝑡 𝜙 𝜇 e𝑡    ∫   ∫ 𝜇𝑡 𝜇𝑡 ′ 𝜇𝑡 = 𝜙 𝜕𝑡 𝜇𝑡 − 𝜕𝑡 𝜇 e𝑡 + 𝜙 𝜕𝑡 𝜇 e𝑡 𝜇 e𝑡 𝜇 e𝑡 𝜇 e𝑡         ∫ 𝜇𝑡 𝑐(𝑡) 2 𝑐(𝑡) 2 ′ 𝜇𝑡 e = 𝜙 div ∇ log 𝜇𝑡 − 𝑏 𝑡 𝜇𝑡 − div ∇ log 𝜇 e𝑡 − 𝑏 𝑡 𝜇 e𝑡 𝜇 e𝑡 2 𝜇 e𝑡 2      ∫ 𝑐(𝑡) 2 𝜇𝑡 div ∇ log 𝜇 e𝑡 − e 𝑏𝑡 𝜇 e𝑡 + 𝜙 𝜇 e𝑡 2     ∫  ∫      𝑐(𝑡) 2 𝑐(𝑡) 2 ′ 𝜇𝑡 ′ 𝜇𝑡 𝜇𝑡 =− ∇𝜙 , ∇ log 𝜇𝑡 − 𝑏 𝑡 𝜇𝑡 + ∇ 𝜙 , ∇ log 𝜇 e𝑡 − e 𝑏𝑡 𝜇 e𝑡 𝜇 e𝑡 2 𝜇 e𝑡 𝜇 e𝑡 2    ∫  𝜇𝑡 𝑐(𝑡) 2 e − ∇𝜙 , ∇ log 𝜇 e𝑡 − 𝑏 𝑡 𝜇 e𝑡 𝜇 e𝑡 2         ∫  ∫  2 𝜇𝑡 𝜇𝑡 𝑐(𝑡) 2 𝜇 𝑐(𝑡) 𝑡 ′ 𝜇𝑡 ∇ log ∇ log 𝜇 e𝑡 − e 𝑏 𝑡 𝜙′ =− ∇𝜙 , − 𝑏𝑡 + e 𝑏 𝑡 𝜇𝑡 + ∇ , 𝜇 e𝑡 𝜇 e𝑡 2 𝜇 e𝑡 𝜇 e𝑡 2 𝜇 e𝑡    ∫  𝜇𝑡 𝜇𝑡 𝑐(𝑡) 2 e ∇ log 𝜇 e𝑡 − 𝑏 𝑡 𝜙′ 𝜇 e𝑡 − ∇ , 𝜇 e𝑡 2 𝜇 e𝑡    2   ∫ ∫  𝑐(𝑡) 2 𝜇𝑡 𝜇𝑡 e =− ∇ log , 𝑏 𝑡 − 𝑏 𝑡 𝜇𝑡 𝜇𝑡 − ∇ log 2 𝜇 e𝑡 2 𝜇 e𝑡   2 ∫ ∫ 2 𝑐(𝑡) 2 𝜇𝑡 1 e ≤− ∇ log 𝜇𝑡 + 𝑏 − 𝑏 𝜇𝑡 𝑡 𝑡 2 4 𝜇 e𝑡 2 𝑐(𝑡) 2   2 ∫ ∫ 2 𝑐(𝑡) 2 𝜇𝑡 1 e ∇ log 𝑏 − 𝑏 =− 𝜇𝑡 + 𝜇𝑡 𝑡 𝑡 2 4 𝜇 e𝑡 2 𝑐(𝑡) 2 ∫ 2 1 𝑐(𝑡) 2 e 𝑏 − 𝑏 =− FI (𝜇𝑡 ||e 𝜇𝑡 ) + 𝜇𝑡 𝑡 𝑡 2 4 𝑐(𝑡) 2 where we used the fact that − 21 𝑎 2 − 𝑎𝑏 ≤ − 14 𝑎 2 + 𝑏 2 , ∀𝑎, 𝑏 ∈ R for the inequality. C.1.3 Proof of Theorem 6.4.1 Now we are ready to prove Theorem 6.4.1. Proof. We first consider the likelihood steps over 𝐾 iterations of PnP-DM. Applying Lemma 2 of [336] to the likelihood steps Equation (C.1) of the non-stationary and stationary processes, we have that 𝜕𝜏 KL(𝜋𝜏 ||𝜇𝜏 ) = − 𝜂2 𝜂2 FI(𝜋𝜏 ||𝜇𝜏 ) ≤ − FI(𝜋𝜏 ||𝜇𝜏 ), 2 4 164 for 𝜏 ∈ [𝑘 (𝑡 ∗ + 1), 𝑘 (𝑡 ∗ + 1) + 1] with 𝑘 = 0, ..., 𝐾 − 1. Integrating both sides over 𝜏 ∈ [𝑘 (𝑡 ∗ + 1), 𝑘 (𝑡 ∗ + 1) + 1], we get ∫ 𝑘 (𝑡 ∗ +1)+1 4[KL(𝜋 𝑋 ||𝜇 𝑘𝑋 ) − KL(𝜋 𝑍 ||𝜇 𝑘𝑍 )] FI (𝜋𝜏 ||𝜇𝜏 ) d𝜏 = (C.8) 𝜂2 𝑘 (𝑡 ∗ +1) for 𝑘 = 0, ..., 𝐾 − 1. Then, applying Lemma C.1.4 to the prior steps Equation (C.4) with 𝑏(𝒙𝑡 , 𝑡) := 𝑢(𝑡)𝒙𝑡 − 𝑣(𝑡) 2 ∇ log 𝑝 𝑡 (𝒙𝑡 ) e 𝑏(𝒙𝑡 , 𝑡) := 𝑢(𝑡)𝒙𝑡 − 𝑣(𝑡) 2 𝒔𝑡 (𝒙𝑡 ) 𝑐(𝑡) := 𝑣(𝑡) 𝛿 := inf ∗ 𝑣(𝑡), 𝑡∈[0,𝑡 ] we have that ∫ 1 𝑣(𝜏) 2 2 FI (𝜋𝜏 ||𝜇𝜏 ) + 𝜕𝜏 KL(𝜋𝜏 ||𝜇𝜏 ) ≤ − 𝑣(𝜏) 2 (𝒔𝜏 − ∇ log 𝑝 𝜏 ) 2 𝜋𝜏 2 4 𝑣(𝜏) ∫ 2 𝑣(𝜏) 2 ∥𝒔𝜏 − ∇ log 𝑝 𝜏 ∥ 22 𝜋𝜏 FI (𝜋𝜏 ||𝜇𝜏 ) + 𝑣(𝜏) ≤− 4 𝛿2 ≤ − FI (𝜋𝜏 ||𝜇𝜏 ) + 𝑣(𝜏) 2 E𝜋 𝜏 ∥𝒔𝜏 − ∇ log 𝑝 𝜏 ∥ 22 , 4 for 𝜏 ∈ [𝑘 (𝑡 ∗ + 1) + 1, (𝑘 + 1)(𝑡 ∗ + 1)] with 𝑘 = 0, ..., 𝐾 − 1. Integrating both sides over 𝜏 ∈ [𝑘 (𝑡 ∗ + 1) + 1, (𝑘 + 1)(𝑡 ∗ + 1)], we get ∫ (𝑘+1) (𝑡 ∗ +1) 𝑋 )] 4[KL(𝜋 𝑍 ||𝜇 𝑘𝑍 ) − KL(𝜋 𝑋 ||𝜇 𝑘+1 4𝜖score FI (𝜋𝜏 ||𝜇𝜏 ) d𝜏 ≤ + (C.9) 𝛿2 𝛿2 𝑘 (𝑡 ∗ +1)+1 where ∫ (𝑘+1) (𝑡 ∗ +1) 𝜖 score := 𝑘 (𝑡 ∗ +1)+1 ∫ 𝑡 ∗ +1 = 1 𝑣(𝜏) 2 E𝜋 𝜏 ∥𝒔𝜏 − ∇ log 𝑝 𝜏 ∥ 22 d𝜏 𝑣(𝜏) 2 E𝜋 𝜏 ∥𝒔𝜏 − ∇ log 𝑝 𝜏 ∥ 22 d𝜏. Finally, combining Equation (C.8) and Equation (C.9) for 𝑘 = 0, ..., 𝐾 − 1, we obtain ∫ 𝑇𝐾 4[KL(𝜋 𝑋 ||𝜇0𝑋 ) − KL(𝜋 𝑋 ||𝜇 𝐾𝑋 )] 4𝐾𝜖 score FI (𝜋𝜏 ||𝜇𝜏 ) d𝜏 ≤ + min(𝜂, 𝛿) 2 𝛿2 0 4KL(𝜋 𝑋 ||𝜇0𝑋 ) 4𝐾𝜖 score ≤ + min(𝜂, 𝛿) 2 𝛿2 where 𝑇𝐾 := 𝐾 (𝑡 ∗ + 1). The proof is concluded by dividing 𝑇𝐾 on both sides. 165 C.1.4 Discussion To facilitate the discussion, we first present the following proposition. Proposition C.1.5. Define a weighting function 𝜆(𝜏) over 𝜏 ∈ [0, 𝑇𝐾 ] such that for 𝑘 = 0, ..., 𝐾 − 1,    𝜂2  𝜆(𝜏) =   𝑣(𝜏) 2  if 𝜏 ∈ [𝑘 (𝑡 ∗ + 1), 𝑘 (𝑡 ∗ + 1) + 1], if 𝜏 ∈ [𝑘 (𝑡 ∗ + 1) + 1, (𝑘 + 1)(𝑡 ∗ + 1)]. Then, under the same settings of Theorem 6.4.1, we have ∫ 𝑇𝐾 4KL(𝜋 𝑋 ||𝜇0𝑋 ) 4𝜖score 1 + ∗ 𝜆(𝜏)FI (𝜋𝜏 ||𝜇𝜏 ) d𝜏 = 𝑇𝐾 0 𝐾 (𝑡 ∗ + 1) 𝑡 +1 ∫ 𝑡 ∗ +1 where 𝜖 score := 1 𝑣(𝜏) 2 E𝜋 𝜏 ∥𝒔𝜏 − ∇ log 𝑝 𝜏 ∥ 22 d𝜏. (C.10) Proof. With the definition of 𝜆(𝜏), we can apply Lemma 2 of [336] to the likelihood steps and obtain ∫ 𝑘 (𝑡 ∗ +1)+1 𝜆(𝜏)FI (𝜋𝜏 ||𝜇𝜏 ) d𝜏 = 4[KL(𝜋 𝑋 ||𝜇 𝑘𝑋 ) − KL(𝜋 𝑍 ||𝜇 𝑘𝑍 )] (C.11) 𝑘 (𝑡 ∗ +1) for 𝑘 = 0, ..., 𝐾 − 1. Similarly, we can apply Lemma C.1.4 to the prior steps and obtain ∫ (𝑘+1) (𝑡 ∗ +1) 𝑋 𝜆(𝜏)FI (𝜋𝜏 ||𝜇𝜏 ) d𝜏 ≤ 4[KL(𝜋 𝑍 ||𝜇 𝑘𝑍 ) − KL(𝜋 𝑋 ||𝜇 𝑘+1 )] + 4𝜖score 𝑘 (𝑡 ∗ +1)+1 (C.12) where ∫ (𝑘+1) (𝑡 ∗ +1) 𝜖 score := 𝑘 (𝑡 ∗ +1)+1 ∫ 𝑡 ∗ +1 = 1 𝑣(𝜏) 2 E𝜋 𝜏 ∥𝒔𝜏 − ∇ log 𝑝 𝜏 ∥ 22 d𝜏 𝑣(𝜏) 2 E𝜋 𝜏 ∥𝒔𝜏 − ∇ log 𝑝 𝜏 ∥ 22 d𝜏. Together, for 𝜏 ∈ [0, 𝑇𝐾 ]. We can then get Equation (C.10) by combining Equation (C.11) and Equation (C.12) for 𝑘 = 0, ..., 𝐾 − 1 and dividing by 𝑇𝐾 := 𝐾 (𝑡 ∗ + 1). Unlike Theorem 6.4.1, this proposition calculates the weighted average of the Fisher divergence along the two processes with the weighting function 𝜆(𝜏). The bound in Theorem 6.4.1 on the unweighted average of Fisher divergence can be obtained 166 by further lower-bounding the left-hand side of Equation (C.10) using the infimum of 𝜆(𝜏) over 𝜏 ∈ [0, 𝑇𝐾 ]. Given this observation, we can see the role of 𝛿 in Theorem 6.4.1. With a strictly positive 𝛿, the weighting function 𝜆(𝜏) is always strictly positive, so the (unweighted) average Fisher divergence must converge to 0. This is precisely the case for the VP- and VE-SDE [266]. On the other hand, if 𝛿 = 0, the Fisher divergence FI (𝜋𝜏 ||𝜇𝜏 ) may be increasingly large as 𝜆(𝜏) gets closer to 0. For iDDPM and EDM, this could happen near 𝑡 = 0 in the reverse diffusion at 𝑣(0) = 0 for these diffusion processes. Nevertheless, we can instead consider a slightly adjusted diffusion coefficient 𝑣˜ (𝑡) := 𝑣(𝑡) + 𝜖 with 𝜖 > 0. Using the relation between scores and diffusions div( 𝑝∇ log 𝑝) = Δ𝑝, we get the following reverse SDE which has the same law as Equation (6.6) at each 𝑡:  2       √︁ 𝜖 𝒙𝑡 𝑠¤ (𝑡) 2 ∇ log 𝑝 𝒙𝑡 + − 2𝑠(𝑡) 𝜎(𝑡)𝜎(𝑡) ¤ ; 𝜎(𝑡) d𝑡+ 𝑠(𝑡) 2𝜎(𝑡)𝜎(𝑡) ¤ + 𝜖 d𝒘¯ 𝑡 . d𝒙𝑡 = 𝑠(𝑡) 2 𝑠(𝑡) √︁ In this case, 𝑣˜ (𝑡) = 𝑠(𝑡) 2𝜎(𝑡)𝜎(𝑡) ¤ + 𝜖 is strictly positive, so the convergence on the unweighted average Fisher divergence is also guaranteed. C.2 Inverse Problem Setup Test Data Here is a summary of the test data for all experiments: • For the synthetic prior experiment, we take images from the CelebA dataset [196], turn them into grayscale, rescale them to [−1, 1], and resize them to 32 × 32 pixels for efficient computation. We then find the empirical mean and covariance of the images to construct the Gaussian image prior. The ground truth image is randomly drawn from this Gaussian prior. • For the benchmark experiments, we use the first 100 images (index 00000 to 00099) in the FFHQ dataset [152]. For all linear inverse problems, the test images are in RGB and normalized to the range [−1, 1]. For all nonlinear problems, the test images are in grayscale and normalized to the range [0, 1]. • For the radio interferometry experiments, we generate synthetic sky images using the code1 . • For the black hole experiments, we use the simulated data used in [269] and the publicly available EHT 2017 data2 that was used to produce the first image of the M87 black hole. 1 https://github.com/liamconnor/polish-torch (unknown license) 2 https://eventhorizontelescope.org/blog/public-data-release-event- horizon-telescope-2017-observations 167 Gaussian and Motion Deblur The forward model is defined as 𝒚 ∼ N (𝑩𝒙 0 , 𝜎𝒚2 𝑰) where 𝑩 ∈ R𝑛×𝑛 is a circulant matrix that effectively implements a convolution with kernel 𝒌 under the circular boundary condition. For the Gaussian deblurring problem, we fix the kernel 𝒌 as a Gaussian kernel with standard deviation 3.0 and size 61 × 61. For the motion deblurring problem, we randomly generate the kernel 𝒌 for each test image using the code3 with intensity of 0.5 and size 61 × 61. For fair comparison, the blur kernel for each test image is set to the same for all compared methods. Super-Resolution The forward model is defined as 𝒚 ∼ N (𝑷 𝑓 𝒙0 , 𝜎𝒚2 𝑰), 𝑛 where 𝑷 𝑓 ∈ R 𝑓 ×𝑛 is a matrix that implements a block averaging filter to downscale the images by a factor of 𝑓 . Specifically, we set 𝑓 = 4 and used the SVD implementation from the code4 of [155]. Coded Diffraction Patterns (CDP) CDP is a measurement model originally proposed in [42]. The target 𝒙 is illuminated by a coherent source and modulated by a phase mask 𝑫. The light field then undergoes the far-field Fraunhofer diffraction and is measured by a standard camera. Mathematically, the forward model of CDP is defined as 𝒚 ∼ N (|𝑭𝑫𝒙0 |, 𝜎𝒚2 𝑰), where 𝑭 denotes the 2D Fourier transform. We follow [321] to set 𝑫 as a diagonal matrix with entries drawn randomly from the complex unit circle. Fourier Phase Retrieval We adopt a similar setting as [65]. In particular, the forward model is defined as 𝒚 ∼ N (|𝑭𝑷𝒙0 |, 𝜎𝒚2 𝑰), where 𝑷 denotes the oversampling matrix that effectively pads 𝒙 in 2D matrix form with zeros. We consider a 4× oversampling ratio for grayscale images of size 256 × 256, so 𝑷𝒙 has a size of 512 × 512. 3 https://github.com/LeviBorodenko/motionblur (unknown license) 4 https://github.com/bahjat-kawar/ddrm (MIT license) 168 Black Hole Imaging We adopt the same BHI setup as in [269, 273]. Measurements for black hole imaging are obtained using Very Long Baseline Interferometry (VLBI). The cross-correlation of the recorded scalar electric fields at two telescopes, referred to as the (ideal) visibility, is related to the source image 𝒙0 through a Fourier transform, as given by the van Cittert-Zernike theorem [66, 339] ∫ ∫ h  i 𝑡 𝒌 {𝑎,𝑏} (𝒙0 ) = exp −𝑖2𝜋 𝑢 𝑡{𝑎,𝑏} 𝜂 + 𝑣 𝑡{𝑎,𝑏} 𝛿 𝒙0 (𝜂, 𝛿)d𝜂d𝛿 ∈ C. (C.13) 𝜂 𝛿 Here, (𝜂, 𝛿) denotes the angular coordinates of the source image, and (𝑢 𝑡{𝑎,𝑏} , 𝑣 𝑡{𝑎,𝑏} ) is the baseline vector between two telescopes {𝑎, 𝑏}, orthogonal to the source direction. In practice, these measurements can be time-averaged over short intervals during which we assume that the target 𝒙0 is static. The relationship between the black hole image and each interferometric measurement, or visibility, is given by   𝑡 𝑽{𝑎,𝑏} = 𝑔𝑎𝑡 𝑔𝑡𝑏 · exp −𝑖(𝜙𝑡𝑎 − 𝜙𝑡𝑏 ) · 𝒌 𝑡{𝑎,𝑏} (𝒙0 ) + 𝒏𝑡{𝑎,𝑏} ∈ C, (C.14) where 𝑎 and 𝑏 denote a pair of telescopes, 𝑡 represents the time of measurement acquisition, 𝑖 is the imaginary unit, and 𝒌 𝑡{𝑎,𝑏} (𝒙 0 ) is the Fourier component of the target image 𝒙0 corresponding to the baseline between telescopes 𝑎 and 𝑏 at time 𝑡. In practice, there are three main sources of noise in Equation (C.14): gain errors 𝑔𝑎𝑡 and 𝑔𝑡𝑏 at the telescopes, phase errors 𝜙𝑡𝑎 and 𝜙𝑡𝑏 , and baseline-based additive white Gaussian noise 𝒏𝑡{𝑎,𝑏} . The gain and phase errors stem from atmospheric turbulence and instrument miscalibration and often cannot be ignored. To correct for these two errors, multiple noisy visibilities can be combined into data products that are invariant to these errors, which are called closure phase and log closure amplitude measurements [23, 50] 𝑡 𝑡 𝑡 𝑡 𝑽{𝑏,𝑐} 𝑽{𝑎,𝑐} ) := Acp,{𝑎,𝑏,𝑐} (𝒙0 ) ∈ R, 𝒚 𝑡cp,{𝑎,𝑏,𝑐} := ∠(𝑽{𝑎,𝑏} © 𝑽{𝑎,𝑏} 𝑽{𝑐,𝑑} ª ® := A 𝑡 logca,{𝑎,𝑏,𝑐,𝑑} (𝒙 0 ) ∈ R, ® 𝑡 𝑡 𝑽{𝑎,𝑐} 𝑽{𝑏,𝑑} ¬ « 𝑡 𝑡 𝒚 𝑡logca,{𝑎,𝑏,𝑐,𝑑} := log ­­ where ∠ computes the angle of a complex number. Given a total of 𝑀 telescopes, there are in total (𝑀−1)2(𝑀−2) closure phase and 𝑀 (𝑀−3) log closure amplitude mea2 surements at time 𝑡, after eliminating repetitive measurements. In our experiments, we use a 9-telescope array (𝑀 = 9) from the Event Horizon Telescope (EHT) and construct the data likelihood term based on these nonlinear closure quantities. Since closure quantities are nonlinear transformations of the visibilities, the forward 169 model in black hole imaging becomes non-convex. Additionally, because the closure quantities do not constrain the total flux (i.e., summation of the pixel values) of the underlying black hole image, we add a constraint on the total flux defined as ∫ ∫ 𝒚 flux := 𝒙 0 (𝜂, 𝛿)d𝜂d𝛿. (C.15) 𝜂 𝛿 To aggregate data over time intervals and telescope combinations, the forward model of black hole imaging can be expressed as     𝒚 := Acp (𝒙0 ), Alogca (𝒙0 ), Aflux (𝒙0 ) := 𝒚 cp , 𝒚 logca , 𝒚 flux , (C.16) h i where 𝒚 cp = 𝒚 𝑡cp,{𝑎,𝑏,𝑐} , ∀𝑡 ∈ T , {𝑎, 𝑏, 𝑐} is the set of all closure phase measureh i ments and 𝒚 cp = 𝒚 𝑡logca,{𝑎,𝑏,𝑐,𝑑} , ∀𝑡 ∈ T , {𝑎, 𝑏, 𝑐, 𝑑} is the set of all log closure amplitude measurements over the observation period T and combinations of telescopes. Overall, the potential function of the data likelihood is given by the sum of the 𝝌2 statistics 𝑓 (𝒙0 ; 𝒚) = − log 𝑝( 𝒚 | 𝒙0 ) 1 2 = Acp (𝒙0 ) − 𝒚 cp 2 𝑑cp 𝜎cp | {z } 2 𝝌cp + 1 2 Alogca (𝒙0 ) − 𝒚 logca 2 𝑑logca 𝜎logca | {z } 2 𝝌logca + 1 ∥Aflux (𝒙0 ) − 𝒚 flux ∥ 2 2 𝜎flux | {z } (C.17) 2 𝝌flux where 𝜎cp , 𝜎logca , and 𝜎flux are the estimated standard deviations of the measured closure phase, log closure amplitude, and flux, respectively. Additionally, 𝑑cp and 𝑑logca represent the total number of time intervals and telescope combinations for the closure phase and log closure amplitude measurements. 2 and The data mismatch metric reported in Figure 6.7 is defined as the sum of 𝝌cp 2 𝝌logca measurements, which are calculated using the ehtim.obsdata.Obsdata.chisq 170 function of the ehtim package5 . Both 𝝌2 values should ideally be around 1 for data with a high signal-to-noise ratio (SNR). Therefore, a data mismatch value around 2 to 3 is considered as fitting the measurements well. Radio Interferometry We consider a deconvolution problem in radio interferometry using the Deep Synoptic Array (DSA), following the setup in [73]. Similar to black hole imaging, each interferometric measurement is given by a pair of antennas and samples a Fourier coefficient of the underlying astronomical image. The spatial frequency sampled by each measurement depends on both the locations of the antennas with respect to each other and the observing wavelength. Given enough antennas, one could densely sample the entire spatial frequency domain. However, in practice, the array consists of a finite number of antenna pairs and operates over a limited range of radio frequencies, resulting in incomplete Fourier coverage. In the image domain, this translates to a convolution with a blur kernel defined by the point spread function (PSF) of the antenna array. Mathematically, the forward model can be formulated as 𝒚 = 𝑩(𝒙0 + 𝝐) (C.18) where 𝑩 ∈ R𝑛×𝑛 represents convolution with the PSF under circular boundary conditions, and 𝝐 ∼ N (0, 𝜎𝒚2 𝑰) models background noise in the true sky image 𝒙0 before the signals reach the telescopes. Due to the linearity of 𝑩, Equation (C.18) can be equivalently written as: 𝒚 = 𝑩𝒙0 + 𝒏 where 𝒏 ∼ N (0, 𝜎𝒚2 𝑩𝑩𝑇 ). To apply PnP-DM to this problem, we implement the likelihood step by setting the forward model as 𝑨 = 𝑩 and noise covariance as 𝚺 = 𝑩𝑩𝑇 . One caveat is that the likelihood step involves 𝚺−1 , which can be problematic if 𝑩 is low-rank. To ensure numerical stability, we regularize the inverse computation by adding a small constant of 1e-10 to the diagonal of 𝚺. In our experiments, we set 𝜎𝒚 = 0.05, corresponding to a signal-to-noise ratio of approximately 5, which aligns with the expected noise level in DSA-2000 observations. 5 https://github.com/achael/eht-imaging (GPL-3.0 license) 171 C.3 C.3.1 Technical Details of PnP-DM Likelihood Step Linear Forward Model and Gaussian Noise As discussed in Section 6.3.1, in the case of linear forward models and Gaussian noise, the likelihood step is 𝜋 𝑍 |𝑋=𝒙 = N (𝒎(𝒙), 𝚲−1 ) where 𝚲 := 𝑨𝑇 𝚺−1 𝑨 + 𝜂12 𝑰 and 𝒎(𝒙) := 𝚲−1 ( 𝑨𝑇 𝚺−1 𝒚 + 𝜂12 𝒙). The bottleneck here is that both the mean and the covariance involve the matrix inverse 𝚲−1 , which can be prohibitive to compute directly for high-dimensional problems. Nevertheless, the computational cost can be significantly alleviated when the noise is i.i.d. Gaussian, i.e., 𝚺 = 𝜎𝒚2 𝑰, and 𝑨 can be efficiently decomposed. For example, if one can efficiently calculate the SVD of the forward model 𝑨, i.e., 𝑨 = 𝑼𝑺𝑽 𝑇 , one can find the Cholesky decomposition of 𝚲−1 as  − 21  1 2 1 −1 𝑇 . 𝑺 + 2𝑰 𝚲 = 𝑳𝑳 where 𝑳 := 𝑽 𝜎2 𝜂 Since 𝑺 is a diagonal matrix, the second term can be calculated with only 𝑂 (𝑛) complexity. Then, leveraging the property of multivariate Gaussian distribution, we can sample 𝝐 ∼ N (0, 𝑰) and calculate 𝒛 = 𝒎(𝒙) + 𝑳𝝐 as a sample that exactly follows the target Gaussian distribution N (𝒎(𝒙), 𝚲−1 ). An analogous derivation with the Fourier transform can be done when 𝑨 is a circulant convolution matrix. Nonlinear Forward Model We provide the pseudocode of the LMC algorithm for sampling the likelihood step with general differentiable forward models in Algorithm 5. Algorithm 5 Langevin Monte Carlo for the likelihood step under general A Require: state 𝒙, coupling strength 𝜂 > 0, likelihood potential 𝑓 ( · ; 𝒚) with measurements 𝒚 Ensure: step size 𝛾 > 0, number of iterations 𝐽 > 0 1: 𝒖 (0) ← 𝒙 2: for 𝑗 = 0, · · · , 𝐽 − 1 do 3: 𝝐 𝑗 ∼ N (0, 𝑰) √︁ 4: 𝒖 ( 𝑗+1) ← 𝒖 ( 𝑗) − 𝛾∇ 𝑓 (𝒖 ( 𝑗) ; 𝒚) − 𝜂𝛾2 (𝒖 ( 𝑗) − 𝒙) + 2𝛾𝝐 𝑗 5: end for 6: return 𝒖 (𝐽) Table C.1 summarizes the hyperparameters we used for solving the nonlinear inverse problems considered in this chapter. 172 Table C.1: List of hyperparameters for the likelihood step of PnP-DM. C.3.2 Inverse problem Step size (𝛾) Number of iterations (𝐽) Coded diffraction patterns Fourier phase retrieval Black hole imaging 1.0e-3 1.0e-4 1.0e-5 100 100 200 Prior step The EDM Framework We formally introduce the EDM formulation [151] using our notations. The forward diffusion process is defined as the following linear Itô SDE d𝒙𝑡 = 𝑢(𝑡)𝒙𝑡 d𝑡 + 𝑣(𝑡)d𝒘 𝑡 , (C.19) where 𝑢(𝑡) : R → R, 𝑣(𝑡) : R → R are the drift and diffusion coefficients. The generative process is the time-reversed version of Equation (C.19). According to [9], it is another Itô SDE of the form   d𝒙𝑡 = 𝑢(𝑡)𝒙𝑡 − 𝑣(𝑡) 2 ∇𝒙𝑡 log 𝑝 𝑡 (𝒙𝑡 ) d𝑡 + 𝑣(𝑡)d𝒘¯ 𝑡 , (C.20) where 𝑝 𝑡 (𝒙𝑡 ) is the marginal distribution of 𝒙𝑡 . There also exists a reverse probability flow ODE   1 2 (C.21) d𝒙𝑡 = 𝑢(𝑡)𝒙𝑡 − 𝑣(𝑡) ∇𝒙𝑡 log 𝑝 𝑡 (𝒙𝑡 ) d𝑡, 2 which shares the same marginal distributions as Equation (C.20). Based on Equation (C.19), we have 𝑝(𝒙𝑡 |𝒙0 ) = N (𝑠(𝑡)𝒙0 , 𝑠(𝑡) 2 𝜎(𝑡) 2 𝑰), √︃∫  ∫ 𝑡 𝑡 𝑣(𝜉) 2 where 𝑠(𝑡) := exp 0 𝑢(𝜉)d𝜉 and 𝜎(𝑡) := d𝜉. We also have 𝒙𝑡 /𝑠(𝑡) ∼ 0 𝑠(𝜉) 2 𝑝(𝒙; 𝜎(𝑡)) where 𝑝(𝒙; 𝜎(𝑡)) is the distribution obtained by adding i.i.d. Gaussian noise of standard deviation 𝜎(𝑡) to the prior data. The idea of the EDM formulation is to write the reverse diffusion directly in terms of the scaling and noise level of 𝒙𝑡 with respect to 𝒙 0 , which are more important than the drift and diffusion coefficients. With the relations between 𝑢(𝑡), 𝑣(𝑡), 𝑝 𝑡 and 𝑠(𝑡), 𝜎(𝑡), 𝑝(·; 𝜎(𝑡)), we can rewrite Equation (C.20) and Equation (C.21) as    √︁ 𝑠¤ (𝑡) 𝒙𝑡 2 d𝒙𝑡 = 𝒙𝑡 − 2𝑠(𝑡) 𝜎(𝑡)𝜎(𝑡)∇ ¤ ; 𝜎(𝑡) d𝑡 + 𝑠(𝑡) 2𝜎(𝑡)𝜎(𝑡)d ¤ 𝒘¯ 𝑡 𝒙 𝑡 log 𝑝 𝑠(𝑡) 𝑠(𝑡) (C.22) 173 and    𝑠¤ (𝑡) 𝒙𝑡 2 d𝒙𝑡 = 𝒙𝑡 − 𝑠(𝑡) 𝜎(𝑡)𝜎(𝑡)∇ ¤ ; 𝜎(𝑡) d𝑡. 𝒙 𝑡 log 𝑝 𝑠(𝑡) 𝑠(𝑡) (C.23) Note that Equation (C.22) is precisely the SDE 6.6 we considered for the prior step. Finally, due to the Tweedie’s formula [90], we can approximate ∇𝒙𝑡 log 𝑝 ( · ; 𝜎(𝑡)) by a denoiser [𝐷 𝜃 ( · ; 𝜎(𝑡))− · ]/𝜎(𝑡) 2 trained to minimize the ℓ2 error of a denoising objective. Substituting the score function with the approximation by the denoiser and using the chain rule, we can further rewrite Equation (C.22) and Equation (C.23) as     √︁ 𝑠¤ (𝑡) 2𝜎(𝑡)𝑠(𝑡) ¤ 𝒙𝑡 2𝜎(𝑡) ¤ + 𝒙𝑡 − 𝐷𝜃 ; 𝜎(𝑡) d𝑡 + 𝑠(𝑡) 2𝜎(𝑡)𝜎(𝑡)d ¤ 𝒘¯ 𝑡 d𝒙𝑡 = 𝜎(𝑡) 𝑠(𝑡) 𝜎(𝑡) 𝑠(𝑡) (C.24) and  d𝒙𝑡 =    𝜎(𝑡) ¤ 𝑠¤ (𝑡) 𝒙𝑡 𝜎(𝑡)𝑠(𝑡) ¤ + 𝐷𝜃 ; 𝜎(𝑡) d𝑡. 𝒙𝑡 − 𝜎(𝑡) 𝑠(𝑡) 𝜎(𝑡) 𝑠(𝑡) (C.25) Pseudocode We provide the pseudocode for our prior step in Algorithm 6. Note that the update rule is precisely the Euler discretization of Equation (C.24) and 𝑁 , scaling schedule 𝑠(·), and Equation (C.25). The discretization time steps {𝑡𝑖 }𝑖=0 noise schedule 𝜎(·), are kept the same as in Table 1 of [151]. For all experiments, we set the total number of time steps to 100, i.e., 𝑁 = 100. We note that this does not imply that the number of function evaluations (NFE) of each prior step is 100. Since 𝜂 is to a small value as the algorithm runs, the number of steps in later iterations of the algorithm is much fewer than 100. The prior step is similar to the image synthesis process in SDEdit [218] that starts from the middle of the reverse diffusion process. We use the pf-ODE solver for the CDP problem and the SDE solver for all other problems. Part of our code implementation is based on the repository6 . Model Checkpoint For experiments with FFHQ color images, we use the pretrained checkpoint from [61] available at the repository7 . For experiments with synthetic data, FFHQ grayscale images, and black hole images, we train our own models using the same repository. The model network is based on the U-Net 6 https://github.com/NVlabs/edm/tree/main (Creative Commons NonCommercial-ShareAlike 4.0 International Public license) 7 https://github.com/jychoi118/P2-weighting (MIT license) Attribution- 174 Algorithm 6 EDM for the prior step (Bayesian denoising with noise level 𝜂) Require: noisy image 𝒛 ∈ R𝑛 , assumed noise level 𝜂 > 0, pre-trained model 𝐷 𝜃 ( · ; · ) that approximates ∇ log 𝑝 (𝒙; 𝜎) with (𝐷 𝜃 (𝒙; 𝜎) − 𝒙)/𝜎 2 𝑁 (monotonically decreasing to 𝑡 Ensure: discretization time steps {𝑡𝑖 }𝑖=0 𝑁 = 0), scaling schedule 𝑠(·), noise schedule 𝜎(·), solver (SDE or pf-ODE) 1: 𝑖 ∗ ← min {𝑖 ∈ [𝑁] | 𝜎(𝑡𝑖 ) ≤ 𝜂} ⊲ Find the starting point of the reverse diffusion ∗ 2: 𝒗 (𝑖 ) ← 𝑠(𝑡𝑖 ∗ )𝒛 ⊲ Initialize at time 𝑡𝑖∗ ∗ 3: for 𝑖 = 𝑖 , · · · , 𝑁 − 1 do 4: 𝜆 ← 2if solver is SDE  else 1   𝜎(𝑡 ¤ 𝑖) 𝑠¤ (𝑡𝑖 ) 𝜆 𝜎(𝑡 ¤ 𝑖 )𝑠(𝑡𝑖 ) 𝒗𝑖 (𝑡 ) + 𝒗 − 𝐷 ; 𝜎 5: 𝒅𝑖 ← 𝜆𝜎(𝑡 𝑖 𝜃 𝑠(𝑡 𝑖 ) 𝑖 𝑠(𝑡𝑖 ) 𝜎(𝑡𝑖 ) 𝑖) 6: 𝒗 (𝑖+1) ← 𝒗 (𝑖) + (𝑡𝑖+1 − 𝑡𝑖 ) 𝒅𝑖 7: if 𝑖 ≠ 𝑁 − 1 and solver is SDE then 8: 𝝐𝑖 ∼ N (0, 𝑰) √︁ ¤ 9: 𝒗 (𝑖+1) ← 𝒗 (𝑖+1) + 𝑠(𝑡) 2𝜎(𝑡)𝜎(𝑡)(𝑡 𝑖 − 𝑡𝑖+1 )𝝐𝑖 10: end if 11: end for 12: return 𝒗 (𝑁) ⊲ Drift ⊲ Diffusion architecture in [224] with BigGAN [36] residual blocks, multi-resolution attention, and multi-head attention with fixed channels per head. See the appendix of [61] for architecture details. Specifically, we change the input and output channels to 1 and 2, respectively, to accommodate grayscale inputs, and reduce the number of down-pooling and up-pooling levels in the U-Net for smaller images. We train all models until convergence using an exponential moving average (EMA) rate of 0.9999, 32-bit precision, and the AdamW optimizer [197]. Here is a list of training data we use for each model: • For the Gaussian prior model, we randomly generate images from the constructed Gaussian prior distribution. • For the FFHQ grayscale model, we use the images with index 01000 to 69999 in the FFHQ dataset. • For the black hole model, we use 3,068 simulated black hole images from the GRMHD simulation, which stands for general relativistic magnetohydrodynamic simulation [72]. See Figure C.1 for some example training images. We apply data augmentation with random flipping and resizing, so that the flux spin rotation and the ring diameter vary from sample to sample. 175 Figure C.1: Example images from the dataset for training the black hole diffusion model prior. Preconditioning Since the checkpoints we use are all trained based on the DDPM (or VP-SDE) formulation [131], we convert them to the denoiser 𝐷 𝜃 under the EDM formulation via the VP preconditioning [151]. Specifically, if we denote the pre-trained model as 𝐹𝜃 ( · ; · ), the model we use for Algorithm 6 is 𝐷 𝜃 (𝒙; 𝜎) := 𝑐 skip (𝜎)𝒙 + 𝑐 out (𝜎)𝐹𝜃 (𝑐 in (𝜎)𝒙; 𝑐 noise (𝜎)) , (C.26) √ −1 (𝜎). where 𝑐 skip (𝜎) = 1, 𝑐 out (𝜎) = −𝜎, 𝑐 in (𝜎) = 1/ 𝜎 2 + 1, and 𝑐 noise (𝜎) = 999𝜎VP −1 (·) is the inverse of the VP-SDE noise schedule defined as 𝜎 (𝑡) := Here 𝜎VP VP √︁ 1 2 𝑒 2 𝛽d 𝑡 +𝛽min 𝑡 − 1 with 𝛽d = 19.9 and 𝛽min = 0.1. This adaptation allows us to make a fair comparison with other DM-based methods using the same pre-trained models. One can also incorporate DMs trained with other formulations into PnP-DM by properly setting the preconditioning parameters. Connection to DDS-DDPM in [326] A concurrent work [326] introduced a rigorous implementation of the prior step, called DDS-DDPM, by converting the DDPM [131] (or VP-SDE [266]) sampler into a reverse diffusion based on the VE-SDE [266]. The diffusion process after the conversion can be used to solve Equation (6.5) rigorously by properly choosing the starting point. In fact, our formulation admits DDS-DDPM as a special case with the VP preconditioning and reverse diffusion based on the VE-SDE. Here we explicitly show this connection. For the VE-SDE, we have √ 𝑠VE (𝑡) = 1, 𝜎VE (𝑡) = 𝑡, 𝑢 VE (𝑡) = 0, and 𝑣 VE (𝑡) = 1. So Equation (C.24) becomes   √  1 1 d𝒙𝑡 = 𝒙𝑡 − 𝐷 𝜃 𝒙𝑡 ; 𝑡 d𝑡 + d𝒘¯ 𝑡 . (C.27) 𝑡 𝑡 Applying the VP preconditioning Equation (C.26) to Equation (C.27), we obtain    √ 1 𝒙𝑡 −1 d𝒙𝑡 = √ 𝐹𝜃 √ ; 999𝜎VP ( 𝑡) d𝑡 + d𝒘¯ 𝑡 . (C.28) 𝑡 𝑡+1 We can then rescale the time range from [0, 1] to [0, 1000], discretize Equation (C.28) backward in time over the time steps {𝜏𝑡 } from [326], and apply the 176 exponential integrator [348] to the drift term, resulting in the following update rule:   √ √ √ b 𝒙𝑡 −1 √ b 𝒙𝑡−1 = b 𝒙𝑡 − 2( 𝜏𝑡 − 𝜏𝑡−1 )𝐹𝜃 √ ; 𝜎VP ( 𝜏𝑡 ) + 𝜏𝑡 − 𝜏𝑡−1 𝝐 𝜏𝑡 + 1 where 𝝐 ∼ N (0, 𝑰). Based on the definition 𝜏𝑡 := 𝛼¯ 𝑡−1 − 1 = 𝜎VP (𝑡) 2 in DDS-DDPM, we get √  √ √ √ b 𝒙𝑡−1 = b 𝒙𝑡 − 2( 𝜏𝑡 − 𝜏𝑡−1 )𝐹𝜃 𝛼¯ 𝑡 b 𝒙𝑡 ; 𝑡 + 𝜏𝑡 − 𝜏𝑡−1 𝝐 . (C.29) This is exactly the update rule of DDS-DDPM with 𝐹𝜃 ( · ; 𝑡) denoting the noise estimate b 𝜖𝑡 (·) of DDPM. One can also verify that the initialization in DDS-DDPM is equivalent to ours by checking that 𝛼¯ 𝑡 ≥ 𝜂21+1 is equivalent to 𝜏𝑡 = 𝜎VP (𝑡) 2 ≤ 𝜂2 where 𝜂 ≡ 𝜂 is the assumed noise level in Equation (6.5). As one can see, DDS-DDPM is equivalent to our prior step by choosing the VP-preconditioning, VE reverse diffusion, and a particular integration scheme. In fact, our prior step allows for more general definitions of diffusion processes and includes both the ODE and SDE solvers. C.3.3 Others Annealing Schedule for 𝜂 In this chapter, we consider an exponential annealing schedule8 for the coupling strength 𝜂. By specifying a starting level 𝜂0 , decay rate 𝛼, and a minimum value 𝜂min , we set 𝜂 𝑘 = max(𝛼 𝑘 𝜂0 , 𝜂min ) for 𝑘 = 0, · · · , 𝐾 − 1. Table C.2 summarizes the annealing hyperparameters that we use for all the inverse problems considered in this chapter. Table C.2: List of hyperparameters for the annealing schedule of 𝜂 in PnP-DM. Inverse problem Synthetic prior experiments Gaussian deblur Motion deblur Super-resolution Coded diffraction patterns Fourier phase retrieval Radio interferometry Black hole imaging Starting level (𝜂0 ) Minimum level (𝜂min ) Decay rate (𝛼) 0.03 10 10 10 10 10 0.1 10 0.03 0.3 0.3 0.3 0.1 0.1 0.1 0.02 1 0.9 0.9 0.9 0.9 0.9 1 0.93 8 PnP-DM is compatible with more general schedules, such as linear decay and parabolic decay. 177 Initialization For the linear inverse problems, including radio interferometry, we use the zero initialization, i.e., 𝒙 (0) = 0 ∈ R𝑛 . For the CDP and Fourier phase retrieval problems, we use the Gaussian initialization, i.e., 𝒙 (0) ∼ N (0, 𝑰). For black hole imaging experiments, we use the uniform random initialization between 0 and 1 for each pixel. We find that PnP-DM, as an MCMC algorithm, is insensitive to the initialization. Except for the black hole experiments, where we found the negative values would cause problems, any reasonable initialization would lead to comparable results. This observation corroborates our convergence result, which holds for any initialization 𝜇0𝑋 . Number of Iterations We run 500 iterations for the synthetic prior experiments, 200 iterations for the black hole experiments, and 100 iterations for all other experiments. The numbers were chosen so that the algorithm was fully converged. Sample Collection To collect multiple samples using our method, there are two main approaches: (1) Run a single Markov chain and collect samples after a certain number of iterations, known as the burn-in period, to ensure the chain has converged. (2) Run several independent Markov chains and collect one sample from each chain after convergence. The first approach is more efficient, but the collected samples are not entirely independent and thus may have a small effective sample size. The second approach ensures all samples are fully independent but takes longer to run. In our experiments, we use the first approach for all tests involving 256 × 256 images to enhance efficiency. Specifically, we set the burn-in period to 40 iterations and collect 20 random samples from the remaining 60 iterations (one every 3 iterations). For other experiments, due to the smaller image sizes, we employ the second approach to obtain fully independent samples. Compute All experiments were performed on NVIDIA RTX A6000 and A100 GPUs. The runtime per image depends on several factors, such as the choice of GPU, the total number of iterations and the coupling strength schedule {𝜂 𝑘 } (as it takes more network evaluations for larger 𝜂 for our EDM-based denoiser). In our actual experiments, we ran each image for at least 100 iterations to ensure convergence, which took around 1 minute for a single Markov chain. Here we present a comparison of computational efficiency with the major baselines on a linear superresolution and a nonlinear coded diffraction patterns problem in Table C.3. The clock time in seconds and the number of function evaluations (NFE) are calculated 178 for each method to measure its computational efficiency. All hyperparameters are kept the same for each method as those used for Table 6.1 and Table 6.2 in the manuscript. As expected, DM-based approaches (DDRM & DPS) generally yield shorter runtimes due to their lower NFEs. Nevertheless, our PnP-DM method significantly outperforms these methods while achieving comparable runtimes with DPS (≈ 1.5×), despite its larger NFEs (≈ 3×). This is primarily due to two factors: (1) PnP-DM avoids running the full diffusion process by adapting the starting noise level to 𝜂 𝑘 at each iteration, and (2) the runtime is further reduced by using an annealing schedule of 𝜂 𝑘 . We also note that the runtime reported for DDRM and DPS below is the time it takes to generate one sample. For the linear inverse problem experiments, where we generate 20 samples for each sampling method, PnP-DM is faster than DPS because we take 20 samples that PnP-DM generated along one Markov chain of batch size 1 (hence same runtime as below, around 50 seconds) but DPS requires running a diffusion process with batch size 20, which is significantly slower (around 330 seconds). Table C.3: Comparison of computational efficiency between PnP-DM and other baseline methods. Inverse problem Metric Clock time (s) NFE Clock time (s) Coded diffraction patterns NFE Super-resolution C.4 DDRM DPS PnP-SGS DPnP PnP-DM (ours) 0.4 20 – – 39 1,000 37 1,000 20 1,030 54 2,572 322 18,372 261 14,596 55 3,032 50 2,482 Implementation Details of Baseline Methods PnP-ADMM We set the ADMM penalty parameter as 2 and run for 500 iterations to ensure convergence. We use the pre-trained DnCNN denoiser [346] available at the deepinv library9 . An additional batch dimension is introduced to collect multiple samples. DPIR We follow the annealing schedule in [345] and run for 40 iterations. We use the pre-trained DRUNet denoiser [347] available at the deepinv library. An additional batch dimension is introduced to collect multiple samples. DDRM We adopt the default hyperparameters: 𝜂 𝐵 = 1.0, 𝜂 = 0.85, and 20 steps for the DDIM sampler [261]. For the Gaussian deblur problem, we use the 9 https://github.com/deepinv/deepinv (BSD-3-Clause license) 179 SVD-based forward model implementation based on separable 1D convolution. An additional batch dimension is introduced to collect multiple samples. DPS We follow the original paper to use a 1000-step DDPM sampler backbone. For the linear inverse problems, we use the step size given in [64], i.e., 𝜁 ′ = 1. For the nonlinear inverse problems, we optimized the step size 𝜁 ′ by performing a grid search, which led to 𝜁 ′ = 3 for CDP and Fourier phase retrieval and 𝜁 ′ = 0.001 for black hole imaging. For the synthetic prior experiments, we also optimized the step size and obtained 𝜁 ′ = 0.1 for compressed sensing and 𝜁 ′ = 1 for Gaussian deblur. An additional batch dimension is introduced to collect multiple samples. PnP-SGS We performed a grid search for the coupling parameter 𝜂 and found that 𝜂 = 0.1 worked the best for all problems. We follow the practice in [69] to have a burn-in period of 20 iterations during which the reverse diffusion is early-stopped. We run the algorithm for 100 iterations in total and collect 20 samples in the 80 iterations after the burn-in period. DPnP We implement the DDS-DDPM sampler for the prior step. For fair comparison, we use the same annealing schedule for the coupling strength (denoted as 𝜂 𝑘 in [326]) as PnP-DM. We run the algorithm for the same number of iterations for each inverse problem in the same way of collecting samples as PnP-DM. HIO We set 𝛽 = 0.7 and applied both the non-negative constraint and the finite support constraint. To mitigate the instability of reconstruction depending on initialization, we first repeatedly run the algorithm with 100 different random initializations and choose the reconstruction with the best measurement fit. Then we run another 10,000 iterations with the chosen reconstruction to ensure convergence and report the metrics on the final reconstruction. C.5 Additional Related Works Image Reconstruction with Plug-and-Play Priors Plug-and-Play priors (PnP) [291] is an algorithmic framework that leverages off-the-shelf denoisers for solving imaging inverse problems. Recognizing the equivalence between the proximal operator and finding the maximum a posteriori (MAP) solution to a denoising problem, PnP substitutes the proximal update in many optimization algorithms, such as ADMM [52, 252] and HQS [345, 347], with generic denoising algorithms, particularly those based on deep learning [217, 345, 347]. The PnP framework enjoys both 180 convergence guarantees [252, 272] and strong empirical performance [4, 322] due to its compatibility with state-of-the-art learning-based denoising priors. Recent works have also proposed learning-based PnP frameworks that have direction interpretations from an optimization perspective [70, 99]. See [149] for a comprehensive review on the theory and practice of PnP. Posterior Sampling with MCMC and Learned Priors Learning-based priors have also been considered in the Bayesian context, where one seeks to sample the posterior distribution defined under a learned prior. An important technique is denoising score matching (DSM) [292], which connects image denoising with learning the score function of an image distribution. Based on DSM, prior works have incorporated deep denoising priors into MCMC formulations, particularly focusing on the Langevin Monte Carlo and its variants as they involve the score function of the target distribution [145, 156, 170, 244, 273]. Recently, methods based on SGS have also gained increasing popularity [31, 69, 100, 230, 326]. Unlike PnP methods based on optimization, these sampling methods possess the ability to generate diverse solutions and quantify the uncertainty of the solution space. Solving Inverse Problems with Diffusion Models The remarkable performance of diffusion models [131, 266] on modeling image distributions makes them desirable choices as image priors for solving inverse problems. One popular approach is to leverage a pre-trained unconditional model and modify the reverse diffusion process during inference to enforce data consistency [34, 64, 65, 155, 189, 260, 262, 263, 265, 306, 361]. Despite the promising performance of these methods, they usually involve approximations and empirically driven designs that are hard to justify theoretically and may lead to inconsistent sample distributions. Another line of work learns task-specific models, which achieves higher accuracy at the cost of re-training models for new problems [17, 189, 253]. Methods based on Particle Filtering and Sequential Monte Carlo are also considered to ensure asymptotic consistency [44, 89, 318]. Diffusion models have also been considered as a prior for variational inference [103, 214] and pluy-and-play image reconstruction [119, 215]. 181 Gaussian deblur GT posterior DPS Ours (EDM) 1.2 Mean GT image -1.2 mean std. dev. 0.15 Std. deviation GT prior 0.05 Figure C.2: Comparison of our method and DPS [64] on estimating the posterior distribution of a Gaussian deblurring problem under a Gaussian prior. While the mean estimations of the two methods are of roughly the same quality, our approach provides a much more accurate estimation of the posterior per-pixel standard deviation than DPS. C.6 Additional Experimental Results C.6.1 Synthetic Prior Experiment In addition to the compressed sensing experiment presented in the main paper, we show another comparison on a Gaussian deblurring problem in Figure C.2. Here, the linear forward model 𝑨 ∈ R𝑚×𝑛 is a 2D convolution matrix with a Gaussian blur kernel of size 7 × 7 and standard deviation 3.0. Similar to the compressed sensing experiment, both methods yield accurate reconstructions of the mean. However, in terms of the posterior standard deviation, DPS exhibits a notable difference from the ground truth, whereas our method achieves a significantly more accurate result. C.6.2 Linear Inverse Problems We provide visual comparisons for the Gaussian deblur and super-resolution problems in Figure C.3. Additional visual examples are provided in Figure C.4 (Gaussian deblurring), Figure C.5 (motion deblurring), and Figure C.6 (super-resolution). C.6.3 Nonlinear inverse problems We provide visual comparisons for the CDP reconstruction problem in Figure C.7, where we visualize one sample for each method. As shown by the red zoom-in boxes, PnP-DM can recover fine-grained features such as the hair threads that are missing in the reconstructions by the baselines. Additional reconstruction examples are given in Figure C.8 for the CDP reconstruction problem. 182 DPIR DPS PnP-SGS Ours: PnP-DM (VE) Ground truth Super-resolution Gaussian deblur Measurement Figure C.3: Visual comparison between our method and baselines on solving the Gaussian deblurring and super-resolution problems with i.i.d. Gaussian noise (𝜎𝒚 = 0.05). We visualize one sample generated by each algorithm. Measurement Ours (VE) Ours (EDM) Ground truth Measurement Ours (VE) Ours (EDM) Ground truth Figure C.4: Additional visual examples for the Gaussian deblurring problem. We then show some additional reconstruction examples with comparison to DPS in Figure C.9. For each method, we visualize the best reconstruction out of four runs for each test image according to the PSNR value. While DPS failed on around half of the test images, our proposed method provided high-fidelity reconstructions on almost all test images. This comparison highlights the better robustness of our method over DPS. 183 Measurement Ours (VE) Ours (EDM) Ground truth Measurement Ours (VE) Ours (EDM) Ground truth Figure C.5: Additional visual examples for the motion deblurring problem. C.6.4 Black Hole Imaging Finally, we present visual examples from the black hole imaging experiments. Samples generated by PnP-DM (EDM) and DPS using the simulated data are shown in Figure C.10, with the data mismatch metric labeled at the top right corner of each sample. Consistent with the results in Figure 6.7, DPS can only capture one of the two posterior modes. DPS samples from Mode 2 and Mode 3 significantly deviate from the measurements and lack the expected black hole structure. In contrast, PnPDM successfully samples both posterior modes and consistently produces samples that fit the measurements well. Additionally, Figure C.11 presents more samples obtained by applying PnP-DM to the real M87 black hole data. The generated samples are not only diverse but also fit the measurements well, with data mismatch values around 2. These samples exhibit a ring diameter consistent with the official EHT reconstruction in Figure 6.8 and share a common bright spot location at the lower half of the ring. C.6.5 Further Analysis Sensitivity Analysis on the Annealing Schedule {𝜂 𝑘 } In Figure C.12, we present a sensitivity analysis on the annealing schedule {𝜂 𝑘 }. In particular, we show the PSNR curves of 𝒙 𝑘 with different exponential decay rates 𝛼 (left) and minimum coupling levels 𝜂min (right) for one linear (super-resolution) and one nonlinear 184 Measurement Ours (VE) Ours (EDM) Ground truth Measurement Ours (VE) Ours (EDM) Ground truth Figure C.6: Additional visual examples for the 4× super-resolution problem. PnP-SGS DPnP Ours: PnP-DM (iDDPM) Ours: PnP-DM (EDM) Ground truth Coded diffraction patterns DPS Figure C.7: Visual comparison between our method and baselines on solving the coded diffraction pattern (CDP) reconstruction problems with i.i.d. Gaussian noise (𝜎𝒚 = 0.05). We visualize one sample generated by each algorithm. (coded diffraction patterns) problem. We have the following conclusions based on the results. First, different decay rates lead to different rates of convergence, which corroborates our theoretical insights that 𝜂 plays the same role as the step size. The final level of PSNR is not sensitive to different decay rates, as all curves converge to the same level. Second, as 𝜂min decreases, the final PSNR becomes higher. This is as expected because the stationary distribution of the 𝒙 𝑘 , 𝜋 𝑋 , should converge to the true target posterior, 𝑝(𝒙|𝒚), as 𝜂 decreases. Convergence Curves with Intermediate Visual Examples In Figure C.13, we show some visual examples of intermediate 𝒙 𝑘 and 𝒛 𝑘 iterates (left) and convergence plots of PSNR, SSIM, and LPIPS for 𝒙 𝑘 (right) on the super-resolution problem. 185 Ours (VE) Ours (iDDPM) Ours (EDM) Ground truth Ours (VE) Ours (iDDPM) Ours (EDM) Ground truth Figure C.8: Additional visual examples for the Fourier phase retrieval problem. As 𝜂 𝑘 decreases, 𝒙 𝑘 becomes closer to the ground truth and 𝒛 𝑘 gets less noisy. Both the visual quality and metric curves stabilize after the minimum coupling strength is achieved. Despite being run for 100 iterations in total, our method generates good images in around 40 iterations, which is around 30 seconds and 1600 NFEs. 186 DPS Ours (iDDPM) Ours (EDM) Ground truth DPS Ours (iDDPM) Ours (EDM) Ground truth Figure C.9: Additional visual examples for the Fourier phase retrieval problem. 187 Mode 1 <latexit sha1_base64="5Xk/EpwtMrd+Mzt6SkJoAh8NewQ=">AAAMgnicjZbhbts2EMfVrtu61NvS7eO+EAkCdFvm2WmTJsCGNXHaJEAQeGnSFp26gJIoiYgkKiQV2xH0EHuafd0eY2+zoyRSUuwMM2CL+t0dj/zfUZaTRlTIweCfe/c/evDxJ58+/GzpUe/zL75cfvzVG8Ey7pJzl0WMv3OwIBFNyLmkMiLvUk5w7ETkrXM5Uva314QLypIzOUvJhxgHCfWpiyWgi+Xv7SzxCHc4dkluX2XYu/OnuFheHfR3tp9vD7bQoL+xsznY2VaDZ083Nwdo2B+Un1Wr/owvHj/6xfaYm8UkkW6EhfhtOEjlhxxzSd2IFEt2JkiK3UsckNxh7FJiRxSo+qyhlDOfCLV0HP0AK4ionCFwiYjoROJY+CyRrUgnApPDMPdQjGWIxCx2WDQXpbAOgqhrzBMmQ5oEHceEusQHfZrpXRaDUeppkc84Gv64sY6IdPud2Ji6nCndCx2rCQs4TsNZV4K4tRoqYcMuggRex8mlspkt4CxLEaCynN0NlgmoOy0qV58GGb+lnBJHMtiC9oGNrKimSsjV1UpXrihgnMoQ1l8oV3MvFrsVt8VWTC9bhoRxciuUJFkMe4vrtUiGQnxNUESkJBwpK+ELthlnkaScTZryQGRAEuVMFrULoCzCfFr8T/8wVIfr1n74zBNhlNRr9Ris0UPKTyCaIFzN0wmZloe1mNOfC18US0tra2tozBnzIbmWR84ikqcRppAHjoMaoJ/rxlhHHvFpQpUgADmLcbJkJ2RSB+dwelImSnuRj1s3Ha9mkiLfb8Ydn/pa5Gf1oGONSBzjIj8uLx2LyziLQOlZkY/MsOMBj4QsTqvsu824OwtsG3KPykulE0tVsRgHqWD96pvk9rqtG9deLyoD5kFMwaRkZml1FT4UL7DXwVAULT88vcMPT41fROM5J2Bgh/LAgMrKb1HSKh+qHRZkqxLVDiJL5xyAmZXQxJ+zAzN2KP7UGNSN2QILmqWzQOMbwg2GscY+bWaBscae8A22oRE1309GJydYyDy3JZnKvLwvfrcBFXXpXDi+EsPDuoo4VkrkxxfNdJImM1RpUYd0HoRluZwYMsAPNvt1NHIMcjVyDSIaEYNCjUKDhEbCoKlGTS8EGgUGzTSaGXSj0Y1BXCNu0JVGVwYlGjVdmmmUGXSs0bFB1xpdGzTRaGIQi0lgZKzujA0Onmxs5V3R6hOmKHwHmu1q313jtq/RvkFHGh0tKiyawH8Ggj4hnE6N5lUrlbLrJtJCG9Osa7ppmW66rTeXLsTSpIIxBE2oR9RIl7qOFDip+/UMWtT06pnZ3Os2ft3I0MaNFK/a+JXBh218aPBJG58YfNDGB43MbXxk8LiNxwaftvGpwedtfP4fC6zkab1tleI+EUQKxHwEf9kOvHt+W4WPqljHyUcmv0GnGr006KVRwKATjd4b9N48WaC69StPnQ9IvV41GjWKdg0tTbuGRtW9rmHPdEbzUNp3tDhOu9aHbdwU9aCNmzy7bayOE7xy6/dqdPfgzUZ/uNXf+vXZ6ou9+uX7ofWNtWI9sYbWc+uFdWiNrXPLtf6w/rT+sv7uPeh91xv2nlau9+/VMV9bnU/vp38BsHyVYA==</latexit> | Mode 2 Mode 1 Mode 2 Mode 3 3.24 2.75 2.82 12.48 660.61 3.08 2.83 4.03 21.28 842.39 2.43 3.84 3.23 39.95 505.21 2.91 3.02 4.47 37.42 654.74 {z Ours: PnP-DM (EDM) } <latexit sha1_base64="1MZpKbTtjWY7+NgJzh09sxd8OlE=">AAAMgnicjZb/btxEEMfdUqCkB6TwJ/+sEkUqEI67lIREAtHk0iaRouhok7YqLtHat7ZXsb3O7jp3F8sPwdPwLzwGb8P4x47tXII46c7rz8zs7H5n1mcnCbnSg8E/9+5/8ODDjz5++MnSo96nn32+/PiL10qk0mVnrgiFfOtQxUIeszPNdcjeJpLRyAnZG+diVNjfXDGpuIhP9Txh7yPqx9zjLtWAzpe/tdN4wqQjqcsy+zKlkzt/8vPl1UF/sLW9/XSbDPqbm8OdjQEMdna2Nwc7ZNgflJ9Vq/6Mzx8/+sWeCDeNWKzdkCr123CQ6PcZlZq7IcuX7FSxhLoX1GeZI8SFpo7KSfVZI4kUHlPF0mn4Hawg5HpOwCVkqhNJI+WJWLcinRBMjqByQiKqA6LmkSPChagCmyCIuqIyFjrgsd9xjLnLPNCnmd4VERi1mZZ4QpLh9xvrhGm334mNuCtFoXtuYg0RvqRJMO9KELVWwzVs2CWQYNJxcrluZvOlSBMCqCxnd4NlAu7O8srV434qbyhXiKMFbMH4wEZWiqaK2eXlSleu0BeS6wDWnxeueK9ud8tvil0ws2wdMCHZjVAWpxHsLarXogUJ6BUjIdOaSVJYmbxlm1Eaai7FtCkPRPosLpzZbe0CKA2pnOX/0z8IisN1Yz9yPlFBGNdrnQhY44QUforwmNBqnk7IrDys+YL+UnkqX1paW1sjYymEB8mNPHoesiwJKYc8cByKAfm5box1MmEej3khCEApIhov2TGb1sEZnJ5EqNKeZ+PWTcermSTP9ptxx6e+5tlpPehYQxZFNM+Oy0vH4gopQlB6nmcjHHY84JGQRkmVfbcZd2eBbUPuUXmpdBJJUSwhQSpYf/GNM3vdNo1rr+eVgUo/4mAqZBZJdVUeFM+318GQ5y0/OrvDj87QL+TRghMwsEN5YMB15Xdb0iofqR1uyVYlqh1Umiw4AMOV8NhbsANDOxR/hobiBrcg/Gbpwjf4mknEMDbY480sMDZ4ojzENjSi4fvx6OSEKp1ltmYznZX3+e82oLwunQvHV1N4WFcRx4US2fF5M53m8ZxUWtQhnQdhWS4nggzwQ3G/jkEOItcgFxEziCEKDAoQKYMUoplBTS/4BvmI5gbNEV0bdI1IGiQRXRp0iSg2qOnS1KAU0bFBx4iuDLpCNDVoikhEzEcZqzu0wcHTja28y1t9IgoK34Fhu8Z3F932DdpHdGTQ0W2FJVP4zyDQJ0zyGWpetVIpu2kiIzSa5l3Tdct03W29hXQB1ZgKxhA05RNWjEyp60hF47pfT6FFsVdPcXOv2vhVI0MbN1K8aOMXiA/b+BDxSRufID5o44NG5jY+Qjxu4zHil238EvFZG5/9xwIreVpvW6W4TxTTigiPwF+2A++eX1fhoyrWcbIR5kf00qDniJ6jAohODHqH6B0+WaC69StPnQ9Ivd5iNGoU7RpamnYNjap7XcMedkbzUNp3jDhOu9aHbdwU9aCNmzy7bVwcJ3jlNu/V5O7B643+cKu/9esPq8/26pfvh9ZX1or1xBpaP1rPrENrbJ1ZrvWH9af1l/V370Hvm96w97RyvX+vjvnS6nx6P/0L6viVZQ==</latexit> | {z } DPS Indices used in teaser: given 5, 11, 32, 59, 88 and DPS using the Figure C.10: Additional visual examples by 50, PnP-DM simulated black hole data. Indices used here: 9, 14, 25, 29, 52, 54, 64, 74, 86, 99 2.15 1.98 2.05 1.82 1.79 1.90 2.08 1.92 2.02 1.96 Figure C.11: Additional visual examples given by PnP-DM using the real M87 black hole data. 188 Different decay rate ↵ Different minimum coupling strength ⌘min <latexit sha1_base64="syAVhY817BIoeANCcXJ3hmA9fsQ=">AAACWnicbZHLTgIxFIbLeOPiBdSdm4mExBWZMQZdGaIbl5jIJQFCzpQClbYztmdUQuYd3OqbmfgwlstCkJM0+fv95yQ9f4NIcIOe951ytrZ3dvfSmWxu/+DwKF84bpgw1pTVaShC3QrAMMEVqyNHwVqRZiADwZrB+H7mN1+ZNjxUTziJWFfCUPEBp4AWNTogohH08kWv7M3L/S/8pSiSZdV6hdRtpx/SWDKFVIAxbd+LsDsFjZwKlmQ7sWER0DEMWdtKBZKZ7nT+3MQtWdJ3B6G2R6E7p38npiCNmcjAdkrAkVn3ZnCjx14UaA2TjWYgk9X7OpCxQK7Dt1WKEMQC9HuSLa0shYOb7pSrKEbkarhYahALF0N3lrPb55pRFBMrgGpuc3HpCDRQtL+RtYn76/n+F43Lsl8pVx6vitW7ZfZpckbOyQXxyTWpkgdSI3VCyTP5IJ/kK/XjOE7GyS1andRy5oSslHP6CyDMt1U=</latexit> ↵ = 0.85 ↵ = 0.9 ↵ = 0.95 <latexit sha1_base64="xJPalkzCd6OKxpkcn4de8c9n9C8=">AAACX3icbZHJSgNBEIY74xbjluhJvAwGwVOYEZdclKAXjxHMAkmQmk4nadLL2F2jhpDH8KrP5dE3sbMczFLQ8Pf3V0HX31EsuMUg+El5a+sbm1vp7czO7t7+QTZ3WLU6MZRVqBba1COwTHDFKshRsHpsGMhIsFrUfxj7tTdmLNfqGQcxa0noKt7hFNChRhNE3IPboFC8esnmg0IwKX9ZhDORJ7Mqv+RSd822polkCqkAaxthEGNrCAY5FWyUaSaWxUD70GUNJxVIZlvDyZtH/pkjbb+jjTsK/Qn9PzEEae1ARq5TAvbsojeGKz32qsAYGKw0Izmavy8CmQjkRr/PU4QoEWA+RpmzuaWwU2wNuYoTRK6606U6ifBR++Ow/TY3jKIYOAHUcJeLT3tggKL7koxLPFzMd1lULwrhdeH66TJfup9lnyYn5JSck5DckBJ5JGVSIZRo8km+yHfq19vy9r3stNVLzWaOyFx5x3/qT7iP</latexit> ↵ = 0.85 ↵ = 0.9 ↵ = 0.95 <latexit sha1_base64="xJPalkzCd6OKxpkcn4de8c9n9C8=">AAACX3icbZHJSgNBEIY74xbjluhJvAwGwVOYEZdclKAXjxHMAkmQmk4nadLL2F2jhpDH8KrP5dE3sbMczFLQ8Pf3V0HX31EsuMUg+El5a+sbm1vp7czO7t7+QTZ3WLU6MZRVqBba1COwTHDFKshRsHpsGMhIsFrUfxj7tTdmLNfqGQcxa0noKt7hFNChRhNE3IPboFC8esnmg0IwKX9ZhDORJ7Mqv+RSd822polkCqkAaxthEGNrCAY5FWyUaSaWxUD70GUNJxVIZlvDyZtH/pkjbb+jjTsK/Qn9PzEEae1ARq5TAvbsojeGKz32qsAYGKw0Izmavy8CmQjkRr/PU4QoEWA+RpmzuaWwU2wNuYoTRK6606U6ifBR++Ow/TY3jKIYOAHUcJeLT3tggKL7koxLPFzMd1lULwrhdeH66TJfup9lnyYn5JSck5DckBJ5JGVSIZRo8km+yHfq19vy9r3stNVLzWaOyFx5x3/qT7iP</latexit> <latexit sha1_base64="u1HZIAN9sELMLcBSwdwyegqeuYY=">AAACXnicbZHBTgIxEIbLqoirCOjFxMtGYuKJ7BqDetAQvXjERNQoxMyWAg1td21nVbLhLbzqe3nzUSywB0EnafL3+2eSzt8wFtyg73/lnKXllfxqYc1d3yhulsqVrVsTJZqyFo1EpO9DMExwxVrIUbD7WDOQoWB34fBy4t+9MG14pG5wFLOOhL7iPU4BLXpog4gHcObXTp/KVb/mT8v7K4JMVElWzadK7rzdjWgimUIqwJjHwI+xk4JGTgUbu+3EsBjoEPrs0UoFkplOOn3y2Nu3pOv1Im2PQm9Kf0+kII0ZydB2SsCBWfQm8F+PPSvQGkb/mqEcz98XgUwEch29zlOEMBGg38bu/txS2DvppFzFCSJX/dlSvUR4GHmTrL0u14yiGFkBVHObi0cHoIGi/RHXJh4s5vtX3B7Wgnqtfn1UbVxk2RfILtkjByQgx6RBrkiTtAgliryTD/KZ+3byTtEpzVqdXDazTebK2fkBXMO4UQ==</latexit> <latexit sha1_base64="u1HZIAN9sELMLcBSwdwyegqeuYY=">AAACXnicbZHBTgIxEIbLqoirCOjFxMtGYuKJ7BqDetAQvXjERNQoxMyWAg1td21nVbLhLbzqe3nzUSywB0EnafL3+2eSzt8wFtyg73/lnKXllfxqYc1d3yhulsqVrVsTJZqyFo1EpO9DMExwxVrIUbD7WDOQoWB34fBy4t+9MG14pG5wFLOOhL7iPU4BLXpog4gHcObXTp/KVb/mT8v7K4JMVElWzadK7rzdjWgimUIqwJjHwI+xk4JGTgUbu+3EsBjoEPrs0UoFkplOOn3y2Nu3pOv1Im2PQm9Kf0+kII0ZydB2SsCBWfQm8F+PPSvQGkb/mqEcz98XgUwEch29zlOEMBGg38bu/txS2DvppFzFCSJX/dlSvUR4GHmTrL0u14yiGFkBVHObi0cHoIGi/RHXJh4s5vtX3B7Wgnqtfn1UbVxk2RfILtkjByQgx6RBrkiTtAgliryTD/KZ+3byTtEpzVqdXDazTebK2fkBXMO4UQ==</latexit> Iteration (k) 0 100 100 35 ⌘min = 0.1 ⌘min = 0.5 ⌘min = 0.9 <latexit sha1_base64="A7hnmAXTN65z/HTmx8jHwJSawhI=">AAAGEHicbdTbbtNAEAbgbWmghFNLL7mJiJC4ihJA5Qqp56bn9JBDW1fR2tmkbry2u7vOoZFfghfgFt6AO8Qtb8AD8B44Tf6BHkayNf5mRlpp5bFDz9Umn/89MflgKvXw0fTj9JOnz56/mJl9WdFBpBxRdgIvUDWba+G5vigb13iiFirBpe2Jqt1eHtarHaG0G/hHph+KM8lbvtt0HW4Sqs/MWcLwumVEzwyk68ef8rn39ZlsPpe/jszdpDBOsmwcpfrs1B+rETiRFL5xPK71aSEfmrMBV8Z1PBGnrUiLkDtt3hKnSepzKfTZ4Pr0ceZNIo1MM1DJ45vMtf4/MeBS6760k07Jzbm+XRvivTVx6XOleP/eoi3jm9+3QUaecVXQvamG25HHVS9Op62GaFrcloOBlbx4HMcjskE2kQNyiBqgBpEACaImqEnUArWIzkHnRC7IJboAXRC1QW0iD+QRSZAk8kE+UQAKiEJQSHQJuiRSIEWkQZrIgAxRBIqIOqAOURfUJeqBekR9UJ/oCnQ1pJEtwhapbQm0RLQMWiZaAa0QrYJWidZAa0TroHWiIqhItAHaINoEbRJtgbaItkHbRDugHaJd0C7RHmiPqAQqEe2D9okOQAdEh6BDoiPQEVEZVCaqgCpEVVCVqAaqER2DjolOQCf/rruRyHB1qNFvmizGwu01eDepvMsV5nPz+x+yC0vjFTnNXrHX7C0rsI9sgRVZiZWZw/rsC/vKvqU+p76nfqR+jlonJ8Yzc+xGpH79BbmtNEw=</latexit> <latexit sha1_base64="wfPuNA6hpYWYeDCMBH0po1FqLIw=">AAAGEHicbdTbbtNAEAbgbWmghFNLL7mJiJC4ihKE2iuknpuek7Y5tHUVrZ1N6sZru7vrJmnkl+AFuIU34A5xyxvwALwHTpN/oIeRbI2/mZFWWnns0HO1yed/T0w+mko9fjL9NP3s+YuXr2ZmX1d1EClHVJzAC1Td5lp4ri8qxjWeqIdKcGl7omZ3Vob12pVQ2g38I9MPxZnkbd9tuQ43CTVm5ixheMMyomcG0vXjT/lcoTGTzefyN5G5nxTGSZaNo9SYnfpjNQMnksI3jse1Pi3kQ3M24Mq4jifitBVpEXKnw9viNEl9LoU+G9ycPs68S6SZaQUqeXyTudH/JwZcat2XdtIpuTnXd2tDfLAmLn2uFO8/WLRlfPv7LsjIM64KurfVcDvyuOrF6bTVFC2L23IwsJIXj+N4RDbIJnJADlET1CQSIEHUArWI2qA20TnonMgFuUQXoAuiDqhD5IE8IgmSRD7IJwpAAVEICokuQZdECqSINEgTGZAhikAR0RXoiqgL6hL1QD2iPqhPdA26HtLIlmBL1LYMWiZaAa0QrYJWidZAa0TroHWiDdAGURFUJNoEbRJtgbaItkHbRDugHaJd0C7RHmiPaB+0T1QClYjKoDLRAeiA6BB0SHQEOiKqgCpEVVCVqAaqEdVBdaJj0DHRCejk33U3ExmuDjX6TZPFWLi7Bu8n1Q+5wnxuvvwxu7g8XpHT7A17y96zAltgi6zISqzCHNZnX9hX9i31OfU99SP1c9Q6OTGemWO3IvXrL659NEo=</latexit> <latexit sha1_base64="/yuM/RBGMutlPjoNxvrm7+anpc8=">AAAGEHicbdTbbtNAEAbgbWmghFNLL7mJiJC4ihIE5Qqp56bn9JBDW1fR2tmkbry2u7vOoZFfghfgFt6AO8Qtb8AD8B44Tf6BHkayNf5mRlpp5bFDz9Umn/89MflgKvXw0fTj9JOnz56/mJl9WdFBpBxRdgIvUDWba+G5vigb13iiFirBpe2Jqt1eHtarHaG0G/hHph+KM8lbvtt0HW4Sqs/MWcLwumVEzwyk68ef8rkP9ZlsPpe/jszdpDBOsmwcpfrs1B+rETiRFL5xPK71aSEfmrMBV8Z1PBGnrUiLkDtt3hKnSepzKfTZ4Pr0ceZNIo1MM1DJ45vMtf4/MeBS6760k07Jzbm+XRvivTVx6XOleP/eoi3jm9+3QUaecVXQvamG25HHVS9Op62GaFrcloOBlbx4HMcjskE2kQNyiBqgBpEACaImqEnUArWIzkHnRC7IJboAXRC1QW0iD+QRSZAk8kE+UQAKiEJQSHQJuiRSIEWkQZrIgAxRBIqIOqAOURfUJeqBekR9UJ/oCnQ1pJEtwhapbQm0RLQMWiZaAa0QrYJWidZAa0TroHWiIqhItAHaINoEbRJtgbaItkHbRDugHaJd0C7RHmiPqAQqEe2D9okOQAdEh6BDoiPQEVEZVCaqgCpEVVCVqAaqER2DjolOQCf/rruRyHB1qNFvmizGwu01eDepvMsV5nPz+++zC0vjFTnNXrHX7C0rsI9sgRVZiZWZw/rsC/vKvqU+p76nfqR+jlonJ8Yzc+xGpH79BcTdNE4=</latexit> <latexit sha1_base64="sMSn5QvOac/pTY228K+V73LA+5E=">AAAGEHicbdTbbtNAEAbgbWmghFNLL7mJiJC4ihKE2iuknpue00MObV1Fa2eTuvHa7u46h0Z+CV6AW3gD7hC3vAEPwHvgNPkHehjJ1vibGWmllccOPVebfP73xOSjqdTjJ9NP08+ev3j5amb2dUUHkXJE2Qm8QNVsroXn+qJsXOOJWqgEl7YnqnZ7ZVivdoTSbuAfm34oziVv+W7TdbhJqD4zZwnD65YRPTOQrh9/yufm6zPZfC5/E5n7SWGcZNk4SvXZqT9WI3AiKXzjeFzrs0I+NOcDrozreCJOW5EWIXfavCXOktTnUujzwc3p48y7RBqZZqCSxzeZG/1/YsCl1n1pJ52Smwt9tzbEB2viyudK8f6DRVvGt7/vgow846qge1sNtyOPq16cTlsN0bS4LQcDK3nxOI5HZINsIgfkEDVADSIBEkRNUJOoBWoRXYAuiFyQS3QJuiRqg9pEHsgjkiBJ5IN8ogAUEIWgkOgKdEWkQIpIgzSRARmiCBQRdUAdoi6oS9QD9Yj6oD7RNeh6SCNbgi1R2zJomWgFtEK0ClolWgOtEa2D1ok2QBtERVCRaBO0SbQF2iLaBm0T7YB2iHZBu0R7oD2ifdA+UQlUIjoAHRAdgg6JjkBHRMegY6IyqExUAVWIqqAqUQ1UIzoBnRCdgk7/XXcjkeHqUKPfNFmMhbtr8H5S+ZArzOfmDz5mF5fHK3KavWFv2XtWYAtskRVZiZWZw/rsC/vKvqU+p76nfqR+jlonJ8Yzc+xWpH79Bcp1NE8=</latexit> <latexit sha1_base64="mOIGelPeJsrbQljkc26DbyP+wog=">AAAGEHicbdTbbtNAEAbgbWmghFNLL7mJiJC4ihKEirhA6rnpOT3k0NZVtHY2qRuv7e6uc2jkl+AFuIU34A5xyxvwALwHTpN/oIeRbI2/mZFWWnns0HO1yed/T0w+mEo9fDT9OP3k6bPnL2ZmX1Z0EClHlJ3AC1TN5lp4ri/KxjWeqIVKcGl7omq3l4f1akco7Qb+kemH4kzylu82XYebhOozc5YwvG4Z0TMD6frxp3zuY30mm8/lryNzNymMkywbR6k+O/XHagROJIVvHI9rfVrIh+ZswJVxHU/EaSvSIuROm7fEaZL6XAp9Nrg+fZx5k0gj0wxU8vgmc63/Twy41Lov7aRTcnOub9eGeG9NXPpcKd6/t2jL+Ob3bZCRZ1wVdG+q4XbkcdWL02mrIZoWt+VgYCUvHsfxiGyQTeSAHKIGqEEkQIKoCWoStUAtonPQOZELcokuQBdEbVCbyAN5RBIkiXyQTxSAAqIQFBJdgi6JFEgRaZAmMiBDFIEiog6oQ9QFdYl6oB5RH9QnugJdDWlki7BFalsCLREtg5aJVkArRKugVaI10BrROmidqAgqEm2ANog2QZtEW6Atom3QNtEOaIdoF7RLtAfaIyqBSkT7oH2iA9AB0SHokOgIdERUBpWJKqAKURVUJaqBakTHoGOiE9DJv+tuJDJcHWr0myaLsXB7Dd5NKu9yhfnc/P777MLSeEVOs1fsNXvLCuwDW2BFVmJl5rA++8K+sm+pz6nvqR+pn6PWyYnxzBy7EalffwHbPTRS</latexit> <latexit sha1_base64="mOIGelPeJsrbQljkc26DbyP+wog=">AAAGEHicbdTbbtNAEAbgbWmghFNLL7mJiJC4ihKEirhA6rnpOT3k0NZVtHY2qRuv7e6uc2jkl+AFuIU34A5xyxvwALwHTpN/oIeRbI2/mZFWWnns0HO1yed/T0w+mEo9fDT9OP3k6bPnL2ZmX1Z0EClHlJ3AC1TN5lp4ri/KxjWeqIVKcGl7omq3l4f1akco7Qb+kemH4kzylu82XYebhOozc5YwvG4Z0TMD6frxp3zuY30mm8/lryNzNymMkywbR6k+O/XHagROJIVvHI9rfVrIh+ZswJVxHU/EaSvSIuROm7fEaZL6XAp9Nrg+fZx5k0gj0wxU8vgmc63/Twy41Lov7aRTcnOub9eGeG9NXPpcKd6/t2jL+Ob3bZCRZ1wVdG+q4XbkcdWL02mrIZoWt+VgYCUvHsfxiGyQTeSAHKIGqEEkQIKoCWoStUAtonPQOZELcokuQBdEbVCbyAN5RBIkiXyQTxSAAqIQFBJdgi6JFEgRaZAmMiBDFIEiog6oQ9QFdYl6oB5RH9QnugJdDWlki7BFalsCLREtg5aJVkArRKugVaI10BrROmidqAgqEm2ANog2QZtEW6Atom3QNtEOaIdoF7RLtAfaIyqBSkT7oH2iA9AB0SHokOgIdERUBpWJKqAKURVUJaqBakTHoGOiE9DJv+tuJDJcHWr0myaLsXB7Dd5NKu9yhfnc/P777MLSeEVOs1fsNXvLCuwDW2BFVmJl5rA++8K+sm+pz6nvqR+pn6PWyYnxzBy7EalffwHbPTRS</latexit> 8 8 Iteration (k) 0 Coded diffraction patterns ⌘min = 0.3 ⌘min = 0.6 ⌘min = 0.9 <latexit sha1_base64="LgY8gX1Z6ffQfKtgHGWp+EaGwqw=">AAACX3icbZHLTgIxFIbLeAHxBroybiYSE1dkxijqQkN04xITARMg5kwp0NDL2J5RCeEx3OpzufRNLJeFgCdp8vf7z0l6/kax4BaD4DvlrayuraczG9nNre2d3Vx+r2Z1YiirUi20eYrAMsEVqyJHwZ5iw0BGgtWj/t3Yr78yY7lWjziIWUtCV/EOp4AONZog4h5cB8Wr8+dcISgGk/KXRTgTBTKrynM+ddNsa5pIppAKsLYRBjG2hmCQU8FG2WZiWQy0D13WcFKBZLY1nLx55B870vY72rij0J/QvxNDkNYOZOQ6JWDPLnpj+K/HXhQYA4N/zUiO5u+LQCYCudFv8xQhSgSY91H2eG4p7Fy2hlzFCSJX3elSnUT4qP1x2H6bG0ZRDJwAarjLxac9MEDRfUnWJR4u5rssaqfFsFQsPZwVyrez7DPkkByRExKSC1Im96RCqoQSTT7IJ/lK/Xhpb8fLTVu91Gxmn8yVd/AL7EG4kA==</latexit> 8 8 <latexit sha1_base64="LgY8gX1Z6ffQfKtgHGWp+EaGwqw=">AAACX3icbZHLTgIxFIbLeAHxBroybiYSE1dkxijqQkN04xITARMg5kwp0NDL2J5RCeEx3OpzufRNLJeFgCdp8vf7z0l6/kax4BaD4DvlrayuraczG9nNre2d3Vx+r2Z1YiirUi20eYrAMsEVqyJHwZ5iw0BGgtWj/t3Yr78yY7lWjziIWUtCV/EOp4AONZog4h5cB8Wr8+dcISgGk/KXRTgTBTKrynM+ddNsa5pIppAKsLYRBjG2hmCQU8FG2WZiWQy0D13WcFKBZLY1nLx55B870vY72rij0J/QvxNDkNYOZOQ6JWDPLnpj+K/HXhQYA4N/zUiO5u+LQCYCudFv8xQhSgSY91H2eG4p7Fy2hlzFCSJX3elSnUT4qP1x2H6bG0ZRDJwAarjLxac9MEDRfUnWJR4u5rssaqfFsFQsPZwVyrez7DPkkByRExKSC1Im96RCqoQSTT7IJ/lK/Xhpb8fLTVu91Gxmn8yVd/AL7EG4kA==</latexit> Super-resolution 32 35 Coded diffraction patterns PSNR (dB) PSNR (dB) 32 Super-resolution <latexit sha1_base64="9VRYP5Su5DLy3b/JFl+qeFAXGQw=">AAAGDHicbdTbbtNAEAbgbWmghENbuOQmokLiKkoQKrc9Nz2nhxzauorWziZ147Wd3XWTNPIr8ALcwhtwh7jlHXgA3gPn8A+k7Ui2xt/MSCutPHboudrkcr+nph/NpB4/mX2afvb8xcu5+YVXZR1EyhElJ/ACVbW5Fp7ri5JxjSeqoRJc2p6o2K21Qb1yI5R2A//U9EJxKXnTdxuuw01Ctfk5Sxhes4zomr50/bg2v5jL5oaRuZ/kx8kiG0extjDzx6oHTiSFbxyPa32Rz4Xmss+VcR1PxGkr0iLkTos3xUWS+lwKfdkfnjzOvEuknmkEKnl8kxnq/xN9LrXuSTvplNxc6bu1AT5YE22fK8V7DxZtGU9+3wUZecZVQWdSDbcjj6tunE5bddGwuC37fSt58TiOR2SDbCIH5BDVQXUiARJEDVCDqAlqEl2BrohckEt0DbomaoFaRB7II5IgSeSDfKIAFBCFoJCoDWoTKZAi0iBNZECGKAJFRDegG6IOqEPUBXWJeqAe0S3odkAjW4GtUNsqaJVoDbRGtA5aJ9oAbRBtgjaJtkBbRAVQgWgbtE20A9oh2gXtEu2B9oj2QftEB6ADokPQIVERVCQ6Ah0RHYOOiU5AJ0SnoFOiEqhEVAaViSqgClEVVCU6A50RnYPO/113PZHB6lCj3zRZjPm7a/B+Uv6QzS9ll44+Li6vjlfkLHvD3rL3LM8+sWVWYEVWYg6L2Bf2lX1LfU59T/1I/Ry1Tk+NZ16ziUj9+gt5rTNW</latexit> Iteration (k) 0 100 Iteration (k) 0 100 Figure C.12: Sensitivity analysis on the annealing schedule 𝜂 𝑘 with different decay rates 𝛼 (left) and minimum coupling strength 𝜂min (right) for a linear (super-resolution) and a nonlinear (coded diffraction patterns) inverse problem. Recall from Appendix C.3.3 that 𝜂 𝑘 := max(𝛼 𝑘 𝜂0 , 𝜂min ), where we set 𝜂0 = 10 for this experiment. <latexit sha1_base64="opYhLCgae4wSkmshIAsdO2T/6oY=">AAAF73icbdRLT9tAEAfwheKWpi9oj71ERZXoBSVVRa9AeD8TyAtwitbOJph4bce7JgmWv0Wlnop67afpud+mzuM/Ko+RdjX7mxlpT2MFrqN0Lvd3avrJjPH02ezzzIuXr16/mZt/W1V+FNqiYvuuH9YtroTreKKiHe2KehAKLi1X1KxOYVivXYtQOb5X1oNANCRve07LsblO6czsW/JbvNj5lFzMLeSWcqPIPkzyk2SBTaJ4MT/zx2z6diSFp22XK3WezwW6EfNQO7YrkowZKRFwu8Pb4jxNPS6FasSjLyfZj6k0sy0/TI+nsyP9fyLmUqmBtNJOyfWlul8b4qM10fV4GPLBo0VLJnff90FGrnZCv3dXNbcil4f9JJMxm6JlckvGsZlePEmSMVkgi8gG2URNUJNIgARRC9QiaoPaRJegSyIH5BBdga6IOqAOkQtyiSRIEnkgj8gH+UQBKCDqgrpEISgkUiBFpEGaKAJFRNega6IeqEfUB/WJBqAB0Q3oZkhjW4WtUtsaaI2oACoQrYPWiTZAG0SboE2iLdAW0TZom2gHtEO0C9ol2gPtEe2D9okOQAdEh6BDoiPQEVERVCQqgUpEx6BjohPQCVEZVCaqgCpEVVCVqAaqEdVBdaJT0CnRGegspXQF5u8vvIdJ9fNSfnlpufRlYWVtsgxn2Xv2gS2yPPvKVtg2K7IKs5nHvrOf7NboGj+MW+PXuHV6ajLzjt0J4/c/CiUnQg==</latexit> <latexit sha1_base64="ga/uKs/yWNcidwd8arf+0i3l5t4=">AAAF73icbdRLT9tAEAfwhZKWpi9oj71EjSrRS5RUFb0C4f1MIC9CUrR2NsHEazveNUmw/C0q9VTUaz9Nz/02dR7/UQmMtKvZ38xIexrDsy2ls9m/c/NPFhJPny0+T754+er1m6XltxXlBr4pyqZru37N4ErYliPK2tK2qHm+4NKwRdXo5kf16o3wleU6JT30RFPyjmO1LZPrmOqNgSG/hSu5T9HlUjqbyY4j9TDJTZM0m0bhcnnhT6PlmoEUjjZtrtRFLuvpZsh9bZm2iJKNQAmPm13eERdx6nApVDMcfzlKfYyllWq7fnwcnRrr/xMhl0oNpRF3Sq6v1GxthI/WRM/hvs+HjxYNGd1/z4IMbG35bv++am4ENvcHUTLZaIl2gxsyDBvxxaMompABMohMkEnUArWIBEgQtUFtog6oQ3QFuiKyQBbRNeiaqAvqEtkgm0iCJJEDcohckEvkgTyiHqhH5IN8IgVSRBqkiQJQQHQDuiHqg/pEA9CAaAgaEt2Cbkc0sXXYOrVtgDaI8qA80SZok2gLtEW0Ddom2gHtEO2Cdon2QHtE+6B9ogPQAdEh6JDoCHREdAw6JjoBnRAVQAWiIqhIdAo6JToDnRGVQCWiMqhMVAFViKqgKlENVCM6B50T1UH1mOIVmJtdeA+TyudMbjWzWvySXtuYLsNF9p59YCssx76yNbbLCqzMTOaw7+wnu0v0Ej8Sd4lfk9b5uenMO3YvEr//AcofJwg=</latexit> <latexit sha1_base64="UtxnJrjVEsFPr21HkiCJWZ3gkYw=">AAAF8HicbdTJbhNBEAbgToghmC2BIxcLCylcLA9C4Zp9X5zFSxKbqGfcdiaenhl398R2rHkMJE5EXHkZzrwNE9t/CTspqVvVX1VJfSo79Fxt8vm/U9NPZlJPn80+T794+er1m7n5tyUdRMoRRSfwAlWxuRae64uicY0nKqESXNqeKNut1ft6+UYo7Qb+qemFoiZ503cbrsNNQhfVri2/9Rcs61N8OZfN5/KDyDxMrFGSZaMoXM7P/KnWAyeSwjeOx7W+sPKhqfW5Mq7jiThdjbQIudPiTXGRpD6XQtf6gz/HmY+J1DONQCXHN5mB/j/R51LrnrSTTsnNlZ6s3eOjNdH2uVK892jRlvH4exJk5BlXBZ1xNdyOPK66cTpdrYtGlduy368mF4/jeEg2yCZyQA5RHVQnEiBB1AA1iJqgJtEV6IrIBblE16BrohaoReSBPCIJkkQ+yCcKQAFRCAqJ2qA2kQIpIg3SRAZkiCJQRHQDuiHqgDpEXVCXqAfqEd2Cbu9paMuwZWpbAa0QrYJWidZAa0TroHWiDdAG0SZok2gLtEW0Ddom2gHtEO2Cdon2QHtE+6B9ogPQAdEh6JCoACoQHYGOiI5Bx0QnoBOiU9ApURFUJCqBSkRlUJmoAqoQnYHOiM5B5wklK9CaXHgPk9LnnLWYWzz6kl1aGS3DWfaefWALzGJf2RLbYgVWZA4L2Hf2k92lVOpH6i71a9g6PTWaecfGIvX7HwRKJ0M=</latexit> <latexit sha1_base64="hMSmKtumgKFuXrJ8eGGEV/CTOUA=">AAAF8HicbdRLTxsxEAdwQ0lL0xe0x16iRpXoJUpQRa9AeIR3gLyABOTdOGHJendjeyEh2o9RqaeiXvtleu636SbkPyqBkWyNfzMj+TRW4DraZLN/p6afzSSev5h9mXz1+s3bd3Pz7yvaD5Utyrbv+qpmcS1cxxNl4xhX1AIluLRcUbU6+WG9ei2UdnyvZPqBaEje9pyWY3MT01m9Z8nzwcJi7kt0MZfOZrKjSD1OcuMkzcZRvJif+VNv+nYohWdsl2t9lssGpjHgyji2K6JkPdQi4HaHt8VZnHpcCt0YjP4cpT7H0ky1fBUfz6RG+v/EgEut+9KKOyU3l3qyNsQna6LrcaV4/8miJaOH70mQoWsc5d88VMOt0OWqFyWT9aZo1bklB4N6fPEoiu7JAllENsgmaoKaRAIkiFqgFlEb1Ca6BF0SOSCH6Ap0RdQBdYhckEskQZLIA3lEPsgnCkABURfUJVIgRaRBmsiADFEIComuQddEN6Aboh6oR9QH9YluQbdDurcV2Aq1rYJWifKgPNEaaI1oHbROtAHaINoEbRIVQAWiLdAW0TZom2gHtEO0C9ol2gPtEe2D9okOQAdERVCR6BB0SHQEOiI6Bh0TlUAlojKoTFQBVYiqoCpRDVQjOgGdEJ2CTmOKV2BucuE9TiqLmdxSZunwa3p5dbwMZ9lH9oktsBz7xpZZgRVZmdnMZ9/ZT3aXUIkfibvEr/vW6anxzAf2IBK//wEJzydE</latexit> x(11) <latexit sha1_base64="qdiixQH5UpHN2ZYpGRYLvSdvvBM=">AAAF8HicbdRLTxsxEAdwQ0lL0xe0x16iokr0gpJS0SskvJ8JkAeQFHk3TrJkvbuxvSEh2o9RqaeiXvtleu636YbkP2qAkWyNfzMj+TRW4DrapNN/p6afzCSePpt9nnzx8tXrN3Pzb0vaD5Utirbv+qpicS1cxxNF4xhXVAIluLRcUbbauWG93BVKO753avqBqEne9JyGY3MT00W1Z8lvg8XlzKfocm4hvZS+i9TDJDNOFtg48pfzM3+qdd8OpfCM7XKtLzLpwNQGXBnHdkWUrIZaBNxu86a4iFOPS6Frg7s/R6mPsdRTDV/FxzOpO/1/YsCl1n1pxZ2Sm5a+XxviozXR8bhSvP9o0ZLR5Ps+yNA1jvKvJ9VwK3S56kXJZLUuGlVuycGgGl88iqIRWSCLyAbZRHVQnUiABFED1CBqgppELVCLyAE5RFegK6I2qE3kglwiCZJEHsgj8kE+UQAKiDqgDpECKSIN0kQGZIhCUEjUBXWJrkHXRD1Qj6gP6hPdgG6GNLI12Bq1ZUFZohwoR7QOWifaAG0QbYI2ibZAW0TboG2iHdAO0S5ol2gPtEe0D9onOgAdEB2CDomOQEdEeVCeqAAqEB2DjolOQCdEp6BToiKoSFQClYjKoDJRBVQhOgOdEZ2DzmOKV2Dm/sJ7mJQ+L2VWllYKXxZWs+NlOMvesw9skWXYV7bKtlmeFZnNfPad/WS3CZX4kbhN/Bq1Tk+NZ96xiUj8/gcPVCdF</latexit> x(21) <latexit sha1_base64="I7r1/7ZE7WEE5TxF9DUIWKa2jcY=">AAAF8HicbdRLTxsxEAdwQ0lL0xe0x16iokr0gpIKUfUGCe9nAuQBJEXejZMsWe9ubG9IiPZjVOqpqNd+mZ77bboh+Y8aYCRb49/MSD6NFbiONun036npJzOJp89mnydfvHz1+s3c/NuS9kNli6Ltu76qWFwL1/FE0TjGFZVACS4tV5Stdm5YL3eF0o7vnZp+IGqSNz2n4djcxHRR7Vny22Dxa+ZTdDm3kF5K30XqYZIZJwtsHPnL+Zk/1bpvh1J4xna51heZdGBqA66MY7siSlZDLQJut3lTXMSpx6XQtcHdn6PUx1jqqYav4uOZ1J3+PzHgUuu+tOJOyU1L368N8dGa6HhcKd5/tGjJaPJ9H2ToGkf515NquBW6XPWiZLJaF40qt+RgUI0vHkXRiCyQRWSDbKI6qE4kQIKoAWoQNUFNohaoReSAHKIr0BVRG9QmckEukQRJIg/kEfkgnygABUQdUIdIgRSRBmkiAzJEISgk6oK6RNega6IeqEfUB/WJbkA3QxrZGmyN2rKgLFEOlCNaB60TbYA2iDZBm0RboC2ibdA20Q5oh2gXtEu0B9oj2gftEx2ADogOQYdER6AjojwoT1QAFYiOQcdEJ6ATolPQKVERVCQqgUpEZVCZqAKqEJ2BzojOQecxxSswc3/hPUxKn5cyK0srheWF1ex4Gc6y9+wDW2QZ9oWtsm2WZ0VmM599Zz/ZbUIlfiRuE79GrdNT45l3bCISv/8BMHInSw==</latexit> x(31) x(91) Convergence for x(k) SSIM 1 8 PSNR (dB) x(1) 32 <latexit sha1_base64="Yk5fb9c4/I3KIDExbrGlXfcr9nc=">AAAF73icbdRLT9tAEAfwhZKWpi9oj71EjSrRS5RUFb0C4f1MIC9CUrR2NsHEazveNUmw/C0q9VTUaz9Nz/02dR7/UQmMtKvZ38xIexrDsy2ls9m/c/NPFhJPny0+T754+er1m6XltxXlBr4pyqZru37N4ErYliPK2tK2qHm+4NKwRdXo5kf16o3wleU6JT30RFPyjmO1LZPrmOqNgSG/hSvZT9HlUjqbyY4j9TDJTZM0m0bhcnnhT6PlmoEUjjZtrtRFLuvpZsh9bZm2iJKNQAmPm13eERdx6nApVDMcfzlKfYyllWq7fnwcnRrr/xMhl0oNpRF3Sq6v1GxthI/WRM/hvs+HjxYNGd1/z4IMbG35bv++am4ENvcHUTLZaIl2gxsyDBvxxaMompABMohMkEnUArWIBEgQtUFtog6oQ3QFuiKyQBbRNeiaqAvqEtkgm0iCJJEDcohckEvkgTyiHqhH5IN8IgVSRBqkiQJQQHQDuiHqg/pEA9CAaAgaEt2Cbkc0sXXYOrVtgDaI8qA80SZok2gLtEW0Ddom2gHtEO2Cdon2QHtE+6B9ogPQAdEh6JDoCHREdAw6JjoBnRAVQAWiIqhIdAo6JToDnRGVQCWiMqhMVAFViKqgKlENVCM6B50T1UH1mOIVmJtdeA+TyudMbjWzWvySXtuYLsNF9p59YCssx76yNbbLCqzMTOaw7+wnu0v0Ej8Sd4lfk9b5uenMO3YvEr//AcSbJwc=</latexit> x(0) <latexit sha1_base64="LfF11tICqij7C+362g9YoTVJtnc=">AAAF73icbdRLT9tAEAfwheKWpi9oj71EjSrRC3Kqil4hvN8J5EVIitbOJjHx2o53HQiRv0Wlnop67afpud+mTsh/1AAj7Wr2NzPSnsYKXEdp0/w7M/tkznj6bP556sXLV6/fLCy+LSs/Cm1Rsn3XD6sWV8J1PFHSjnZFNQgFl5YrKlZ3fVSv9EWoHN8r6kEgGpK3Pafl2FwnVKvfWPLbcMn8FF8sZMxlcxzph0l2kmTYJPIXi3N/6k3fjqTwtO1ypc6zZqAbQx5qx3ZFnKpHSgTc7vK2OE9Sj0uhGsPxl+P0x0Sa6ZYfJsfT6bH+PzHkUqmBtJJOyXVH3a+N8NGa6Hk8DPng0aIl4+n3fZCRq53Qv5pWza3I5eF1nErVm6JV55YcDuvJxeM4viMLZBHZIJuoCWoSCZAgaoFaRG1Qm6gD6hA5IIfoEnRJ1AV1iVyQSyRBksgDeUQ+yCcKQAFRD9QjCkEhkQIpIg3SRBEoIuqD+kRXoCuia9A10QA0ILoB3YzoztZga9SWA+WI1kHrRBugDaJN0CbRFmiLaBu0TbQD2iHaBe0S7YH2iPZB+0QHoAOiQ9Ah0RHoiOgYdEyUB+WJCqAC0QnohOgUdEpUBBWJSqASURlUJqqAKkRVUJXoDHRGVAPVEkpWYPb+wnuYlD8vZ1eWVwpfMqu5yTKcZ+/ZB7bEsuwrW2U7LM9KzGYe+85+slujZ/wwbo1fd62zM5OZd2wqjN//AM+vJwk=</latexit> z (0) <latexit sha1_base64="RKjz1B0NlZw+98XaAUTC4vwkNuw=">AAAF8HicbdRLTxsxEAdwQ0lL0xe0x16iokr0EiVVRa+Q8AjvBMgDSIq8GycsWe8utjdP7ceo1FNRr/0yPffbdAn5j0pgJFvj38xIPo0VuI42mczfmdknc4mnz+afJ1+8fPX6zcLi24r2Q2WLsu27vqpZXAvX8UTZOMYVtUAJLi1XVK1O/rZe7QqlHd87MYNANCRve07LsbmJ6bw+tOS30XI28ym6WFjKpDPjSD1MspNkiU2ieLE496fe9O1QCs/YLtf6PJsJTGPElXFsV0TJeqhFwO0Ob4vzOPW4FLoxGv85Sn2MpZlq+So+nkmN9f+JEZdaD6QVd0puLvV07RYfrYlrjyvFB48WLRndf0+DDF3jKL93Xw23QperfpRM1puiVeeWHI3q8cWjKLojC2QR2SCbqAlqEgmQIGqBWkRtUJvoEnRJ5IAcoivQFVEH1CFyQS6RBEkiD+QR+SCfKAAFRNegayIFUkQapIkMyBCFoJCoC+oS9UA9oj6oTzQADYiGoOEt3dkabI3acqAcUR6UJ1oHrRNtgDaINkGbRFugLaICqEC0Ddom2gHtEO2Cdon2QHtE+6B9ogPQAdEh6JCoCCoSlUAloiPQEdEx6JjoBHRCVAaViSqgClEVVCWqgWpEp6BTojPQWUzxCsxOL7yHSeVzOruSXil9WVrNTZbhPHvPPrBllmVf2SorsCIrM5v57Dv7yW4SKvEjcZP4ddc6OzOZecfuReL3PwncJ0Q=</latexit> z (10) <latexit sha1_base64="nIeLrLXQuz3AAUOrMBjK88EXu5o=">AAAF8HicbdRLTxsxEAdwQ0lL0xe0x16iokr0ghJU0SskPMI7QF5AAvJunGTJendje0lCtB+jUk9FvfbL9Nxv0yXJf1QeI9ka/2ZG8mmswHW0Saf/Tk0/m0k8fzH7Mvnq9Zu37+bm35e1HypblGzf9VXV4lq4jidKxjGuqAZKcGm5omJ1cnf1yrVQ2vG9ohkEoi55y3Oajs1NTOe1G0teDBeX01+iy7mF9FJ6FKnHSWaSLLBJFC7nZ/7UGr4dSuEZ2+Van2fSgakPuTKO7YooWQu1CLjd4S1xHqcel0LXh6M/R6nPsTRSTV/FxzOpkf4/MeRS64G04k7JTVs/rN3hkzXR9bhSfPBk0ZLR/fdDkKFrHOX37qvhVuhy1Y+SyVpDNGvcksNhLb54FEVjskAWkQ2yiRqgBpEACaImqEnUArWI2qA2kQNyiK5AV0QdUIfIBblEEiSJPJBH5IN8ogAUEHVBXSIFUkQapIkMyBCFoJDoGnRN1AP1iPqgPtEANCC6Ad3c0djWYGvUlgVliXKgHNE6aJ1oA7RBtAnaJNoCbRHlQXmibdA20Q5oh2gXtEu0B9oj2gftEx2ADogOQYdEBVCB6Ah0RHQMOiY6AZ0QFUFFohKoRFQGlYkqoApRFVQlOgWdEp2BzmKKV2Dm4cJ7nJSXlzIrSytHXxdWs5NlOMs+sk9skWXYN7bK8qzASsxmPvvOfrLbhEr8SNwmfo1bp6cmMx/YvUj8/gcPYSdF</latexit> z (20) <latexit sha1_base64="ICRH+XOwh0+Y07RJFS00sW8iHig=">AAAF8HicbdRLTxsxEAdwQ0lL0xe0x16iokr0EiWlolcgPMI7QF5AUuTdOGHJendjewMh2o9RqaeiXvtleu636YbkP2oCI9ka/2ZG8mmswHW0yWT+Tk0/mUk8fTb7PPni5avXb+bm35a1HypblGzf9VXV4lq4jidKxjGuqAZKcGm5omK1c4N6pSuUdnyvaHqBqEve8pymY3MT03nt1pLf+otLmU/RxdxCJp25j9TDJDtKFtgoChfzM39qDd8OpfCM7XKtz7OZwNT7XBnHdkWUrIVaBNxu85Y4j1OPS6Hr/fs/R6mPsTRSTV/FxzOpe/1/os+l1j1pxZ2Sm0s9WRvgozXR8bhSvPdo0ZLR+HsSZOgaR/nX42q4Fbpc3UTJZK0hmjVuyX6/Fl88iqIhWSCLyAbZRA1Qg0iABFET1CRqgVpEl6BLIgfkEF2BrojaoDaRC3KJJEgSeSCPyAf5RAEoIOqAOkQKpIg0SBMZkCEKQSFRF9QlugZdE92Aboh6oB7RLeh2QENbha1S2xpojSgHyhGtg9aJNkAbRJugTaIt0BZRHpQn2gZtE+2Adoh2QbtEe6A9on3QPtEB6IDoEHRIVAAViI5AR0THoGOiE9AJURFUJCqBSkRlUJmoAqoQVUFVolPQKdEZ6CymeAVmJxfew6T8OZ1dTi8ffVlYWRstw1n2nn1giyzLvrIVlmcFVmI289l39pPdJVTiR+Iu8WvYOj01mnnHxiLx+x8U5idG</latexit> z (30) <latexit sha1_base64="Xv5hvcck/17CzRKd1xWTsaQHcLQ=">AAAF8HicbdRLTxsxEAdwQ0lL0xe0x16iRpXoBSUVouoNEt7PBMgDSIq8GydZst5dbC95aT9GpZ6Keu2X6bnfpkvIf9QAI9ka/2ZG8mmswHW0yWT+Tk0/mUk8fTb7PPni5avXb+bm35a1HypblGzf9VXV4lq4jidKxjGuqAZKcGm5omJ18rf1yrVQ2vG9E9MPRF3yluc0HZubmM5rA0t+Gy58zXyKLubSmcXMKFIPk+w4SbNxFC7mZ/7UGr4dSuEZ2+Van2czgakPuTKO7YooWQu1CLjd4S1xHqcel0LXh6M/R6mPsTRSTV/FxzOpkf4/MeRS67604k7JTVvfr93iozVx5XGleP/RoiWjyfd9kKFrHOV3J9VwK3S56kXJZK0hmjVuyeGwFl88iqI7skAWkQ2yiRqgBpEACaImqEnUArWI2qA2kQNyiC5Bl0QdUIfIBblEEiSJPJBH5IN8ogAUEF2BrogUSBFpkCYyIEMUgkKia9A1URfUJeqBekR9UJ9oABrc0p2twlapLQfKEeVBeaI10BrROmidaAO0QbQJ2iTaAm0RbYO2iXZAO0S7oF2iPdAe0T5on+gAdEB0CDokKoAKREVQkegIdER0DDomOgGdEJVAJaIyqExUAVWIqqAq0SnolOgMdBZTvAKz9xfew6T8eTG7vLhcXEqv5MbLcJa9Zx/YAsuyL2yFbbECKzGb+ew7+8luEirxI3GT+HXXOj01nnnHJiLx+x82BCdM</latexit> z (90) 0 LPIPS 0.8 0.2 Initialization 0 Iteration (k) 100 Figure C.13: Visual examples of intermediate 𝒙 (𝑘) and 𝒛 (𝑘) iterates (left) and convergence plots of PSNR, SSIM, and LPIPS for 𝒙 (𝑘) iterates (right) on the super-resolution problem. The vertical dashed lines show the iterations at which the 𝒙 (𝑘) and 𝒛 (𝑘) iterates are visualized. 189 Appendix D APPENDIX FOR CHAPTER 7 D.1 Appendix for Section 7.3 D.1.1 D.1.1.1 DCDP, PnP-DM, and DAPS as Instantiations of Algorithm 2 DCDP Likelihood Step The likelihood step with LDMs is given in the “Approach I: Enforcing DC in the Latent Space” section from the DCDP paper [181]. Switching this to the notation in this paper, we have 1 𝜇 𝒗 (𝑘) = arg min ∥A (D (𝒗)) − 𝒚∥ 22 + ∥𝒖 (𝑘) − 𝒗∥ 22 2 2 𝒗 (D.1) for some constant 𝜇. Setting 𝜇 = 𝜎𝒚2 /𝜂2𝑘 makes this equivalent to 𝒗 (𝑘) = arg min 𝒗 1 1 ∥A (D (𝒗)) − 𝒚∥ 22 + 2 ∥𝒖 (𝑘) − 𝒗∥ 22 2 2𝜎𝒚 2𝜂 𝑘 ! 1 1 = arg max exp − 2 ∥A (D (𝒗)) − 𝒚∥ 22 − 2 ∥𝒖 (𝑘) − 𝒗∥ 22 /𝑍. 2𝜎𝒚 2𝜂 𝑘 𝒗 (D.2) Note that Equation (D.2) is exactly finding the MAP estimation of Equation (7.6) √ with 𝜂 𝑘 = 𝜎𝒚 / 𝜇. This shows the equivalence of sub-step 1. In sub-step 2, noise is added as the first half of the Diffusion Purification step of DCDP. DCDP runs forward diffusion up to an annealed timestep 𝑇𝑘 , which is equivalent to adding 𝜂 𝑘 noise for the correct choice of 𝜂 𝑘 . Prior Step In the second half of the Diffusion Purification step of DCDP, reverse diffusion is run using the VP-SDE formulation. This is the same √︁ 1 √︁ 1 as reverse diffusion 2 2 in the EDM framework with 𝑠𝑡 = 1/ 𝑒 2 𝛽𝑑 𝑡 +𝛽min 𝑡 and 𝜎𝑡 = 𝑒 2 𝛽𝑑 𝑡 +𝛽min 𝑡 − 1 [151]. Since this formulation is representable in the EDM framework, it is equivalent to Equation (7.4). D.1.1.2 PnP-DM Likelihood Step The likelihood step in PnP-DM in Chapter 6, originally designed for pixel-space diffusion, introduces a hyperparameter 𝜂 with exponential decay 190 𝜂 𝑘 = max(𝛼 𝑘 𝜂0 , 𝜂min ), where 𝛼, 𝜂0 , and 𝜂min are preset. Extending this to latent diffusion amounts to replacing A with A (D (·)), leading to ! 1 1 𝒗 (𝑘) ∼ exp − 2 ∥A (D (𝒗)) − 𝒚∥ 22 − 2 |𝒖 (𝑘) − 𝒗∥ 22 /𝑍. (D.3) 2𝜎𝒚 2𝜂 𝑘 This likelihood step matches the distribution in Equation (7.6). No additional noise is introduced in sub-step 2. Prior Step The prior step of PnP-DM utilizes various diffusion schedulers, such as VP-SDE and EDM-SDE. By adopting the EDM scheduler with 𝑠𝑡 = 1 and 𝜎𝑡 = 𝑡, the prior step matches the form of Equation (7.4). D.1.1.3 DAPS Likelihood Step In DAPS [341], the likelihood and prior steps are coupled, as sampling 𝒙0 ∼ 𝑝(𝒙0 | 𝒙𝑡 , 𝒚) is required. However, by adopting the Gaussian approximation 𝑝(𝒙 0 | 𝒙𝑡 ) ≈ N (b 𝒙 0 (𝒙𝑡 ), 𝜎𝑡2 𝑰) (where b 𝒙0 (𝒙𝑡 ) is obtained via a fewstep ODE solver), separation becomes possible. This approximation aligns with the prior step formulation in Equation (7.4). With 𝒖 (𝑘) = b 𝒙0 (𝒙𝑡 ), the DAPS update rule is: ! 1 1 𝒗 (𝑘) ∼ exp − 2 ∥A (D (𝒗)) − 𝒚∥ 22 − 2 ∥𝒖 (𝑘) − 𝒗∥ 22 /𝑍, 2𝛽 𝒚 2𝑟 𝑡 (D.4) where 𝛽 𝒚 and 𝑟 𝑡 are hyperparameters. Setting 𝛽 𝒚 = 𝜎𝒚 and 𝑟 𝑡 = 𝜂 𝑘 recovers the likelihood step in Equation (7.6). Prior Step The prior step in DAPS follows from setting 𝒖 (𝑘) = b 𝒙0 (𝒙𝑡 ) via a fewstep ODE solver using the EDM scheduler. Choosing an appropriate 𝜂 𝑘 makes this sampling procedure equivalent to that described by Equation (7.4). D.1.2 Hyperparameters To optimize hyperparameters, we run two Bayesian optimization processes for each task to select the combinations that had the highest PSNR and LPIPS, respectively. These caused different forms of artifacts in the results, so the parameters that yielded the best qualitative outputs are chosen. TReg, PSLD, and PnP-DM are optimized for LPIPS while DCDP is optimized for PSNR. For DAPS, neither of these optimizations leads to results better than previously handpicked parameters, so those are used instead. 191 Table D.1: Hyperparameters used by each method on each task. Method Task Super-resolution (16×) Box inpainting Gaussian deblur (𝜎 = 6) TReg Classifier free guidance scale (𝑤) 𝜆 for conjugate gradient [160] Adaptive negation learning rate (𝜂) [160] 2.87 9.97e-04 1.56e-04 4.92 2.05e-05 4.17e-04 3.31 1.12e-04 3.33e-04 PSLD 𝛾 defined in [251] 𝜂 defined in [251] 0.298 0.896 0.0187 0.102 0.124 0.132 DCDP SGD learning rate SGD momentum Likelihood optimization steps 9.95e-02 0.964 49 1.35e-02 0.872 28 3.29e-02 0.642 46 PnP-DM HMC learning rate HMC momentum Likelihood sampling steps 4.83e-04 0.496 14 1.99e-05 0.440 15 1.07e-04 0.404 26 DAPS HMC learning rate HMC momentum Likelihood sampling steps 1.00e-04 0.45 30 2.00e-06 0.45 30 1.00e-05 0.45 30 D.2 D.2.1 Appendix for Section 7.4 Detailed Implementation of STeP Here, we summarize the proposed framework for solving video inverse problems in Algorithm 7. Algorithm 7 STeP: a Framework for Solving Video Inverse Problems with SpatioTemporal Prior 𝑁 where 𝑡 = 0 and 𝑡 = 𝑇, noise schedule Require: discretization time steps {𝑡𝑖 }𝑖=1 0 𝑁 𝜎𝑡 , likelihood 𝑝(𝒚 | ·) with measurements 𝒚, HMC step size 𝜂 and damping factor 𝛾, number of HMC updates 𝑀, pre-trained latent score function 𝒔𝜽 (𝒛; 𝜎) ≈ ∇ 𝒛 log 𝑝 (𝒛; 𝜎) with image decoder D. 1: 𝒛 𝑡 𝑁 ∼ N (0, 𝜎𝑡2𝑁 𝑰) ⊲ Initialization 2: for 𝑖 = 𝑁, ..., 1 do b 3: 𝒛0 ← Backward(𝒛𝑡𝑖 ; 𝒔𝜽 ) ⊲ Solve PF-ODE (D.5) backward from 𝑡 = 𝑡𝑖 to 𝑡 = 0 to enforce prior 4: 𝒑 ∼ N (0, 𝑰) 5: for 𝑗 = 1, ..., 𝑀 do 6: 𝝐 𝑗 ∼ N (0, 𝑰) 7: (b 𝒛 0 , 𝒑) = HamiltonianDynamics(b 𝒛0 , 𝒚, 𝒑, 𝝐 𝑗 ; 𝜂, 𝛾) ⊲ HMC updates to enforce data consistency 8: end for 9: 𝒛𝑡𝑖−1 ∼ N (b 𝒛0 , 𝜎𝑡2𝑖−1 𝑰) ⊲ Proceed to the next noise level at time 𝑡 = 𝑡𝑖−1 10: end for 11: return D (𝒛 𝑡0 ) ⊲ Return the decoded video The algorithm’s main loop alternates between three key steps: (1) solving the PFODE backward from 𝑡 = 𝑡𝑖 to 𝑡 = 0 (line 3), (2) performing multi-step MCMC updates (lines 4–8), and (3) advancing to the next noise level (line 9). We will discuss each step in detail. 192 Through the lens of Algorithm 2, line 3 can be viewed as the prior step, and lines 4-9 together can be viewed as the likelihood step. The equivalence can be seen by pattern matching with 𝐾 ≡ 𝑁, 𝑘 ≡ 𝑁 − 𝑖 and 𝜂 𝑘 ≡ 𝜎𝑡 𝑁 −𝑖 . Solving PF-ODE Backward from 𝒕 = 𝒕 𝒊 to 𝒕 = 0 The probability flow ordinary differential equation (PF-ODE) [151] of the diffusion model governs the continuous increase or reduction of noise in the image when moving forward or backward in time, given by d𝒛𝑡 = −𝜎 ¤ 𝑡 𝜎𝑡 ∇ 𝒛𝑡 log 𝑝(𝒛𝑡 ; 𝜎𝑡 ) d𝑡, (D.5) where 𝜎 ¤ 𝑡 denotes the time derivative of 𝜎𝑡 , and ∇ 𝒛𝑡 log 𝑝(𝒛𝑡 ; 𝜎𝑡 ) represents the time-dependent score function [264, 266]. Our goal is to solve the probability flow ODE (PF-ODE), as defined in Equation (D.5), backward from 𝑡 = 𝑡𝑖 to 𝑡 = 0, given the intermediate state 𝒛𝑡𝑖 and the pre-trained latent score function 𝒔𝜽 (𝒛; 𝜎) ≈ ∇ 𝒛 log 𝑝(𝒛; 𝜎). Any ODE solver, such as Euler’s method or the fourth-order RungeKutta method (RK4) [39], can be used to solve this problem. Following previous conventions [341], we adopt a few-step Euler method for solving it efficiently. Multi-Step MCMC Updates Any MCMC samplers can be used, such as Langevin Dynamic Monte Carlo (LMC) and Hamiltonian Monte Carlo (HMC). For example, the LMC update with step size 𝜂 is √︁ 𝒛+0 = 𝒛0 + 𝜂∇ 𝒛0 log 𝑝( 𝒚 | D (𝒛0 )) + 𝜂∇ 𝒛0 log 𝑝(𝒛0 | 𝒛𝑡 ) + 2𝜂𝝐 . (D.6) Note that the first gradient term can be computed with a differentiable forward model A and the neural network decoder D. The second gradient term, on the other hand, can be calculated by ∇ 𝒛0 log 𝑝(𝒛0 | 𝒛𝑡 ) = ∇ 𝒛0 log 𝑝(𝒛𝑡 | 𝒛 0 )+∇ 𝒛0 log 𝑝(𝒛0 ) ≈ ∇ 𝒛0 log 𝑝(𝒛𝑡 | 𝒛0 )+𝒔𝜽 (𝒛0 , 𝑡min ). This approximation holds for 𝑡min ≈ 0, assuming that 𝒛0 lies close to the clean latent manifold [264]. To improve both convergence speed and approximation accuracy, the MCMC samplers are initialized with the solutions obtained from the previous PF-ODE step, leveraging its outputs as a warm start. Note that during MCMC updates, the decoder D needs to be evaluated multiple times in the backward pass. To accelerate this process, we adopt Hamiltonian Monte Carlo (HMC), which typically requires fewer steps for convergence, thereby speeding up the algorithm. For each multi-step MCMC update, we introduce an additional 193 Table D.2: Summary of the hyperparameters of Algorithm 7 (STeP) for black hole video reconstruction and dynamic MRI. The HMC-related parameters are tuned on 3 leave-out validation videos. The run time and memory are tested using one NVIDIA A100 GPU. Black hole video reconstruction Dynamic MRI 20 𝑡 20 𝑡 HMC related number of steps 𝑀 scaling factor 1 − 𝛾𝜂 step size square 𝜂2 observation noise level 𝜎𝒚 60 0.00 1.2e-5 0.02 53 0.83 1.2e-3 0.01 Annealing Schedule related number of steps 𝑁 final time 𝑇 25 100 20 100 PF-ODE related number of steps 𝑁ode scheduler 𝜎𝑡 discretization time {𝑡𝑖 }, 𝑖 = 1, · · · , 𝑁 Inference related decoder NFE 𝑁dec diffusion model NFE 𝑁dm time (s) per sample memory (GB)  1 𝑁−𝑖 7 𝑁 ·𝑇 1500 500 645 48 7  1 𝑁−𝑖 7 𝑁 ·𝑇 7 1060 400 332 21 momentum variable 𝒑, initialized as N (0, 𝑰). The HamiltonianDynamics(b 𝒛0 , 𝒚, 𝒑, 𝝐; 𝜂, 𝛾) update with step size 𝜂 and damping factor 𝛾 is given by: √︁ 𝒑 + = (1 − 𝛾𝜂) · 𝒑 + 𝜂∇ 𝒛0 log 𝑝(𝒚 | D (𝒛 0 )) + 𝜂∇ 𝒛0 log 𝑝(𝒛0 | 𝒛𝑡 ) + 2𝛾𝜂𝝐, 𝒛+0 = 𝒛0 + 𝜂 𝒑 + . Proceeding to Next Noise Level According to Proposition 1 in [341], one can obtain a sample 𝒛𝑡𝑖−1 ∼ 𝑝(𝒛𝑡𝑖−1 | 𝒚) by simply adding Gaussian noise from a sample 𝒛ˆ0 ∼ 𝑝(𝒛0 | 𝒛𝑡𝑖 , 𝒚), given 𝒛𝑡𝑖 ∼ 𝑝(𝒛𝑡𝑖 | 𝒚) from last step. Thus, we solve the target posterior sampling by gradually sampling from the time-marginal posterior of the diffusion trajectory. The full hyperparameters and running cost STeP is summarized in Table D.2. The HMC-related parameters are searched on a leave-out validation dataset consisting of three videos that are different from the testing videos. D.2.2 D.2.2.1 Experimental Details Black Hole Video Reconstruction We introduce the black hole video reconstruction problem in more detail. The goal is to reconstruct a video 𝒙0 ∈ R𝑛 𝑓 ×𝑛ℎ ×𝑛𝑤 of a rapidly moving black hole. Each 194 measurement, or visibility, is given by correlating the measurements from a pair of telescopes to sample a particular spatial Fourier frequency of the source with very long baseline interferometry (VLBI) [66, 339]. In VLBI, the cross-correlation of the recorded scalar electric fields at two telescopes, known as the ideal visibility, is related to the ideal source video 𝒙0 through a 2D Fourier transform, as given by the van Cittert-Zernike theorem [66, 339]. Specifically, the ideal visibility of the 𝑗-th frame of the target video is ∫ ∫ h  i [ 𝑗] [ 𝑗] [ 𝑗] [ 𝑗] 𝒌 {𝑎,𝑏} (𝒙0 ) := exp −𝑖2𝜋 𝑢 {𝑎,𝑏} 𝜌 + 𝑣 {𝑎,𝑏} 𝛿 𝒙 0 (𝜌, 𝛿)d𝜌d𝛿 ∈ C, (D.7) 𝜌 𝛿 where (𝜌, 𝛿)  denotes the angular coordinates of the source video frame, and  [ 𝑗] [ 𝑗] 𝑢 {𝑎,𝑏} , 𝑣 {𝑎,𝑏} is the dimensionless baseline vector between two telescopes {𝑎, 𝑏}, orthogonal to the source direction. Due to atmospheric turbulence and instrumental calibration errors, the observed visibility is corrupted by gain error, phase error, and additive Gaussian thermal noise [91, 273]: h  i [ 𝑗] [ 𝑗] [ 𝑗] [ 𝑗] [ 𝑗] [ 𝑗] [ 𝑗] 𝑽{𝑎,𝑏} := 𝑔𝑎 𝑔𝑏 exp −𝑖 𝜙𝑎 − 𝜙 𝑏 𝒌 {𝑎,𝑏} (𝒙0 ) + 𝒏 {𝑎,𝑏} ∈ C, (D.8) [ 𝑗] [ 𝑗] [ 𝑗] [ 𝑗] where gain errors are denoted by 𝑔𝑎 , 𝑔𝑏 , phase errors are denoted by 𝜙𝑎 , 𝜙 𝑏 , [ 𝑗] and thermal noise is denoted by 𝒏 {𝑎,𝑏} . While the phase of the observed visibility cannot be directly used due to phase errors, the product of three visibilities among any combination of three telescopes, known as the bispectrum, can be computed to retain useful information. Specifically, the phase of the bispectrum, termed the closure phase, effectively cancels out the phase errors in the observed visibilities. Similarly, a strategy can be employed to cancel out amplitude gain errors and extract information from the visibility amplitude [23]. Formally, these quantities are defined as [ 𝑗] [ 𝑗] [ 𝑗] [ 𝑗] [ 𝑗] 𝒚 cp,{𝑎,𝑏,𝑐} := ∠(𝑽{𝑎,𝑏} 𝑽{𝑏,𝑐} 𝑽{𝑎,𝑐} ) := Acp,{𝑎,𝑏,𝑐} (𝒙0 ) ∈ R, [ 𝑗] [ 𝑗] 𝑽{𝑎,𝑏} 𝑽{𝑐,𝑑} ª ® := A [ 𝑗] (𝒙 ) ∈ R. logca,{𝑎,𝑏,𝑐,𝑑} 0 [ 𝑗] [ 𝑗] ® 𝑽{𝑎,𝑐} 𝑽{𝑏,𝑑} ¬ « © [ 𝑗] 𝒚 logca,{𝑎,𝑏,𝑐,𝑑} := log ­­ (D.9) (D.10) Here, ∠(·) denotes the complex angle, and | · | computes the complex amplitude. For [ 𝑗] a total of 𝑀 telescopes, the number of closure phase measurements 𝒚 cp,{𝑎,𝑏,𝑐} at is [ 𝑗] (𝑀−1) (𝑀−2) , and the number of log closure amplitude measurements 𝒚 logca,{𝑎,𝑏,𝑐,𝑑} is 2 195 [ 𝑗] [ 𝑗] 𝑀 (𝑀−3) , after accounting for redundancy. Let 𝑑cp and 𝑑logca to indicate the dimen2 [ 𝑗] [ 𝑗] sion of 𝒚 cp and 𝒚 logca . Since closure quantities are nonlinear transformations of the visibilities, the black hole video reconstruction problem is non-convex. Additionally, because the closure quantities do not constrain the total flux (i.e., summation of the pixel values) of the underlying black hole video, we add a constraint on the total flux for each frame, defined as ∫ ∫ [ 𝑗] [ 𝑗] (D.11) 𝒚 flux := 𝒙0 (𝜌, 𝛿)d𝜌d𝛿. 𝜌 𝛿 To aggregate data over different measurement times and telescope combinations, the forward model of black hole video reconstruction for the 𝑗-th frame can be expressed as h i h i [ 𝑗] [ 𝑗] [ 𝑗] [ 𝑗] [ 𝑗] [ 𝑗] [ 𝑗] 𝒚 := Acp (𝒙0 ), Alogca (𝒙0 ), Aflux (𝒙0 ) := 𝒚 cp , 𝒚 logca , 𝒚 flux , (D.12) h i [ 𝑗] [ 𝑗] where 𝒚 cp = 𝒚 cp,{𝑎,𝑏,𝑐} , ∀{𝑎, 𝑏, 𝑐} is the set of all closure phase measurements h i [ 𝑗] [ 𝑗] and 𝒚 cp = 𝒚 logca,{𝑎,𝑏,𝑐,𝑑} , ∀{𝑎, 𝑏, 𝑐, 𝑑} is the set of all log closure amplitude measurements over all combinations of telescopes for the 𝑗-th frame. The overall data consistency is an aggregation over all frames and typically expressed using the 𝝌2 statistics − log 𝑝(𝒚 | 𝒙0 ) ∝ 𝑛𝑓 ∑︁ 1 [ 𝑗] [ 𝑗] 2 𝑗=1 𝑛 𝑓 𝑑 cp 𝜎cp | + 𝑛𝑓 ∑︁ {z +𝛽 2 } 2 𝝌cp 1 [ 𝑗] [ 𝑗] 2 Alogca (𝒙0 ) − 𝒚 logca [ 𝑗] 2 𝑗=1 𝑛 𝑓 𝑑 logca 𝜎logca | [ 𝑗] Acp (𝒙 0 ) − 𝒚 cp {z 2 𝝌logca 𝑛𝑓 ∑︁ 2 1 [ 𝑗] [ 𝑗] A (𝒙 ) − 𝒚 0 flux flux 𝑛 𝜎2 𝑗=1 𝑓 flux | {z } } (D.13) 2 𝝌flux where 𝜎cp , 𝜎logca , and 𝜎flux are the estimated standard deviations of the measured closure phase, log closure amplitude, and flux, respectively, and 𝛽 is a hyperparameter that controls the strength of the flux regularization, which is empirically 196 Dirty image Ground truth Selected frames over time Figure D.1: The dirty images from the ideal visibilities. We use the standard implementation in EHT library to get dirty images for each selected frame. determined. To evaluate the data fitting, we introduce a unified 𝝌˜ 2 statistics 𝝌˜ 2 = 𝝌2 · 1{ 𝝌2 ≥ 1} + 1 · 1{ 𝝌2 < 1}. 𝝌2 (D.14) The 𝝌˜ 2 is no less than 1 and closer to 1, indicating better measurement data fit. In 2 + 𝝌 2 ˜ logca our experiments we use an average ( 𝝌˜ cp )/2 for the data misfit metrics for evaluation. Our experiments are based on the simulation of observing the Sagittarius A∗ black hole with the EHT 2017 array of eight radio telescopes over an observation period of around 100 minutes. We refer the readers to Figure 5 of [178] for a visualization of the measurement patterns in Fourier space over time. To show the difficulty of this black hole video reconstruction problem, we visualize the dirty video frames obtained by applying the inverse Fourier transform to the ideal visibilities, assuming no measurement errors, in Figure D.1. One can see that substantial spatiotemporal information is lost during the measurement process, so obtaining high-quality reconstructions relies on the effectiveness of incorporating prior information in the reconstruction process. D.2.2.2 Dynamic MRI MRI is an important imaging technique for clinical diagnosis and biomedical research, where the objective is to recover a video 𝒙0 ∈ C𝑛 𝑓 ×𝑛ℎ ×𝑛𝑤 of the heart from the subsampled Fourier space (a.k.a 𝑘-space) measurements 𝒚. Despite its many advantages, MRI is known to be slow because of the physical limitations of the data acquisition in 𝑘-space. This leads to low patient throughput and sensitivity to patients’ motion [298]. To accelerate the scan speed, instead of fully sampling 𝑘space, the compressed subsampling MRI (CS-MRI) technique subsamples 𝑘-space 197 kx <latexit sha1_base64="1WxsgSQFxKVWkPS9JiPgVsEVYOU=">AAAIo3icjdRZb9pAEABgJ6QtpVfSPvbFShSJqBECVKVPkXKH3EA4cixYa7MGB1+x1wFi+Zf1R/S5/6Y28QwBglJLoNlvZse7xqxs65rLs9m/c/OJhTdv3yXfpz58/PT5y+LS15preY7CqoqlW86VTF2mayarco3r7Mp2GDVkndXl7m6Urz8wx9Uss8IHNmsYtG1qqqZQHpK0NP97VVQlwlmf+5oZEM0UiUF5R5b9ctD0TUkVCdcM5oqmpDSxboSdUdgLUqNmlsf/r1tUOLtdPCLrhOp2hzZ9si4S6vIoN9bulaWF5bdh/frzymjQib56jal0tKqJfDrqGA6et1iL0mo8fbSEuPLFJFFaFo8qIJhRhrsYbQIL07O3sjaVHt9KnH9tOXEZbmPsTjOycKNhGt4Dzgx7+EMO3wPVHwTiJg62gzSEpC8bUjZYE39g1hzOkw2fKB0taOYlX6Fhs81cJp/Np7pSX1pcyWayw0ucDnJxsCLEV1FaWvhDWpbiGczkik5d9zaXtXnDpw7XFJ0FKeK5zKZKl7bZbRiaNHzyDX/4NwvE1VBaomo54cfk4lCfz/Cp4boDQw4rox24k7kIX8yxe5M6Dh28mJSNYHw8CYanc82xeuPKqezp1OkHqRRpMZXQ8Dn60cOkQRA8kQwkIylAClILqIXEgBiSCqQitYHaSB2gDpIGpCHdAd0hdYG6SDqQjmQAGUgmkIlkAVlINpCNdA90j+QAOUgukIvEgTiSB+QhPQA9IPWAekh9oD7SAGiA9Aj0GNGTbYNtY9kO0A7SLtAu0h7QHtI+0D7SAdAB0iHQIVIBqIB0BHSEdAx0jHQCdIJ0CnSKdAZ0hnQOdI50AXSBVAQqIpWASkhloDLSJdAlUgWoglQFqiLVgGpIdaA60hXQFdI10DXSDdBNSOERmJs88KaDWj6T28hslH6ubO3Eh2FS+C4sC2khJ/wStoSCUBSqgpJYThQSpUQ5uZo8SZaTlafS+bl4zjdh7Eo2/gFoEBff</latexit> 6× acceleration kx <latexit sha1_base64="1WxsgSQFxKVWkPS9JiPgVsEVYOU=">AAAIo3icjdRZb9pAEABgJ6QtpVfSPvbFShSJqBECVKVPkXKH3EA4cixYa7MGB1+x1wFi+Zf1R/S5/6Y28QwBglJLoNlvZse7xqxs65rLs9m/c/OJhTdv3yXfpz58/PT5y+LS15preY7CqoqlW86VTF2mayarco3r7Mp2GDVkndXl7m6Urz8wx9Uss8IHNmsYtG1qqqZQHpK0NP97VVQlwlmf+5oZEM0UiUF5R5b9ctD0TUkVCdcM5oqmpDSxboSdUdgLUqNmlsf/r1tUOLtdPCLrhOp2hzZ9si4S6vIoN9bulaWF5bdh/frzymjQib56jal0tKqJfDrqGA6et1iL0mo8fbSEuPLFJFFaFo8qIJhRhrsYbQIL07O3sjaVHt9KnH9tOXEZbmPsTjOycKNhGt4Dzgx7+EMO3wPVHwTiJg62gzSEpC8bUjZYE39g1hzOkw2fKB0taOYlX6Fhs81cJp/Np7pSX1pcyWayw0ucDnJxsCLEV1FaWvhDWpbiGczkik5d9zaXtXnDpw7XFJ0FKeK5zKZKl7bZbRiaNHzyDX/4NwvE1VBaomo54cfk4lCfz/Cp4boDQw4rox24k7kIX8yxe5M6Dh28mJSNYHw8CYanc82xeuPKqezp1OkHqRRpMZXQ8Dn60cOkQRA8kQwkIylAClILqIXEgBiSCqQitYHaSB2gDpIGpCHdAd0hdYG6SDqQjmQAGUgmkIlkAVlINpCNdA90j+QAOUgukIvEgTiSB+QhPQA9IPWAekh9oD7SAGiA9Aj0GNGTbYNtY9kO0A7SLtAu0h7QHtI+0D7SAdAB0iHQIVIBqIB0BHSEdAx0jHQCdIJ0CnSKdAZ0hnQOdI50AXSBVAQqIpWASkhloDLSJdAlUgWoglQFqiLVgGpIdaA60hXQFdI10DXSDdBNSOERmJs88KaDWj6T28hslH6ubO3Eh2FS+C4sC2khJ/wStoSCUBSqgpJYThQSpUQ5uZo8SZaTlafS+bl4zjdh7Eo2/gFoEBff</latexit> ky 8× acceleration ky <latexit sha1_base64="RulI99z0Wp9qHoeJGeM0d9WRwBY=">AAAIo3icjdRZb9pAEABgJ6QtpVfSPvbFShSJqBECVKVPkXKH3EA4cixYa7MGB1+x1wFi+Zf1R/S5/6Y28QwBglJLoNlvZse7xqxs65rLs9m/c/OJhTdv3yXfpz58/PT5y+LS15preY7CqoqlW86VTF2mayarco3r7Mp2GDVkndXl7m6Urz8wx9Uss8IHNmsYtG1qqqZQHpK0NP97VVQlwlmf+5oZEM0UiUF5R5b9ctD0TUkVCdcM5oqmpDSxboSdUdgLUqNmlsf/r1tUOLtdPCLrhOp2hzZ9si4S6vIoN9bulaWF5bdh/frzymjQib56jal0tKqJfDrqGA6et1iL0mo8fbSEuPLFJFFaFo8qIJhRhrsYbQIL07O3sjaVHt9KnH9tOXEZbmPsTjOycKNhGt4Dzgx7+EMO3wPVHwTiJg62gzSEpC8bUjZYE39g1hzOkw2fKB0taOYlX6Fhs81cJp/Np7rSQFpcyWayw0ucDnJxsCLEV1FaWvhDWpbiGczkik5d9zaXtXnDpw7XFJ0FKeK5zKZKl7bZbRiaNHzyDX/4NwvE1VBaomo54cfk4lCfz/Cp4boDQw4rox24k7kIX8yxe5M6Dh28mJSNYHw8CYanc82xeuPKqezp1OkHqRRpMZXQ8Dn60cOkQRA8kQwkIylAClILqIXEgBiSCqQitYHaSB2gDpIGpCHdAd0hdYG6SDqQjmQAGUgmkIlkAVlINpCNdA90j+QAOUgukIvEgTiSB+QhPQA9IPWAekh9oD7SAGiA9Aj0GNGTbYNtY9kO0A7SLtAu0h7QHtI+0D7SAdAB0iHQIVIBqIB0BHSEdAx0jHQCdIJ0CnSKdAZ0hnQOdI50AXSBVAQqIpWASkhloDLSJdAlUgWoglQFqiLVgGpIdaA60hXQFdI10DXSDdBNSOERmJs88KaDWj6T28hslH6ubO3Eh2FS+C4sC2khJ/wStoSCUBSqgpJYThQSpUQ5uZo8SZaTlafS+bl4zjdh7Eo2/gFtkhfg</latexit> <latexit sha1_base64="RulI99z0Wp9qHoeJGeM0d9WRwBY=">AAAIo3icjdRZb9pAEABgJ6QtpVfSPvbFShSJqBECVKVPkXKH3EA4cixYa7MGB1+x1wFi+Zf1R/S5/6Y28QwBglJLoNlvZse7xqxs65rLs9m/c/OJhTdv3yXfpz58/PT5y+LS15preY7CqoqlW86VTF2mayarco3r7Mp2GDVkndXl7m6Urz8wx9Uss8IHNmsYtG1qqqZQHpK0NP97VVQlwlmf+5oZEM0UiUF5R5b9ctD0TUkVCdcM5oqmpDSxboSdUdgLUqNmlsf/r1tUOLtdPCLrhOp2hzZ9si4S6vIoN9bulaWF5bdh/frzymjQib56jal0tKqJfDrqGA6et1iL0mo8fbSEuPLFJFFaFo8qIJhRhrsYbQIL07O3sjaVHt9KnH9tOXEZbmPsTjOycKNhGt4Dzgx7+EMO3wPVHwTiJg62gzSEpC8bUjZYE39g1hzOkw2fKB0taOYlX6Fhs81cJp/Np7rSQFpcyWayw0ucDnJxsCLEV1FaWvhDWpbiGczkik5d9zaXtXnDpw7XFJ0FKeK5zKZKl7bZbRiaNHzyDX/4NwvE1VBaomo54cfk4lCfz/Cp4boDQw4rox24k7kIX8yxe5M6Dh28mJSNYHw8CYanc82xeuPKqezp1OkHqRRpMZXQ8Dn60cOkQRA8kQwkIylAClILqIXEgBiSCqQitYHaSB2gDpIGpCHdAd0hdYG6SDqQjmQAGUgmkIlkAVlINpCNdA90j+QAOUgukIvEgTiSB+QhPQA9IPWAekh9oD7SAGiA9Aj0GNGTbYNtY9kO0A7SLtAu0h7QHtI+0D7SAdAB0iHQIVIBqIB0BHSEdAx0jHQCdIJ0CnSKdAZ0hnQOdI50AXSBVAQqIpWASkhloDLSJdAlUgWoglQFqiLVgGpIdaA60hXQFdI10DXSDdBNSOERmJs88KaDWj6T28hslH6ubO3Eh2FS+C4sC2khJ/wStoSCUBSqgpJYThQSpUQ5uZo8SZaTlafS+bl4zjdh7Eo2/gFtkhfg</latexit> Figure D.2: Subsampling masks of 6× (left) and 8× (right) accelerations used in dynamic MRI experiments. The white areas in the center indicate the autocalibration (ACS) signals. The horizontal and vertical directions are the frequency (𝑘 𝑥 ) and phase (𝑘 𝑥 ) encoding directions, respectively. The same mask is applied to the sampling of each individual frame of all videos. 𝑛 𝑓 with masks {𝒎 [ 𝑗] } 𝑗=1 . Mathematically, this can be formulated as   [ 𝑗] 𝒚 [ 𝑗] = 𝒎 [ 𝑗] ⊙ 𝑭𝒙0 + 𝒏 [ 𝑗] ∈ C𝑛 for 𝑗 = 1, ..., 𝑛 𝑓 , (D.15) where 𝒎 [ 𝑗] ∈ {0, 1}𝑛ℎ ×𝑛𝑤 is the subsampling mask for the 𝑗-th frame, ⊙ denotes element-wise multiplication, and 𝒏 [ 𝑗] is the measurement noise. In our experiments, we used subsampling masks with an equi-spaced pattern (similar to those visualized in [298]) of both 6× acceleration with 24 auto-calibration signal (ACS) lines (Table D.4) and 8× acceleration with 12 ACS lines (Table 7.2). For dynamic MRI, we use the Gaussian likelihood function 𝑛𝑓 2 1 ∑︁ A [ 𝑗] (𝒙0 ) − 𝒚 [ 𝑗] , − log 𝑝( 𝒚 | 𝒙0 ) ∝ 2 𝑛 𝑓 𝑗=1 (D.16)   [ 𝑗] where A [ 𝑗] (𝒙0 ) := 𝒎 [ 𝑗] ⊙ 𝑭𝒙0 . Figure D.2 visualizes the subsampling masks used in our experiments, where 𝑘 𝑥 , 𝑘 𝑦 indicate the frequency encoding and phase encoding directions, respectively. The same mask is applied to the sampling of each frame of all videos. D.2.2.3 Baseline Implementations We provide details on implementations of our baseline methods, following the same grouping in Section 7.4.3.1. 198 Group 1: Simple Heuristics We implement batch independent sampling (BIS) and batch consistent sampling (BCS) following [168, 169]. BIS is implemented to reconstruct each frame independently, whereas BCS promotes temporal consistency by using identical ("batch-consistent") noise across the temporal dimension. To ensure a fair comparison, we adapt Algorithm 7 by initializing the noise variables 𝒛𝑡 𝑁 (line 1) and 𝒛𝑡𝑖−1 (line 9) with batch-consistent noise. Group 2: Noise Warping We follow [53, 82] in performing noise warping based on optical flow estimated from measurements. As described in Section 7.4.3.1, we derive optical flow using ground-truth black hole videos and inverse Fouriertransformed dynamic MRI videos with a pre-trained model [278]. Since we utilize a latent diffusion model, we further downsample the optical flow via interpolation to match the latent noise dimension. We adapt Algorithm 7 accordingly by replacing the original i.i.d. noise with the warped noise while keeping other components unchanged. We implement Bilinear, Bicubic, and Nearest warping strategies using their corresponding interpolation methods, and follow the implementation from https://github.com/yitongdeng-projects/infinite_resolution_ ∫ integral_noise_warping_code (MIT License) for -noise. Lastly, since [77] does not provide publicly available code, we implement GP-Warp following Equation (2) from their paper. D.2.3 Training Details for Spatiotemporal Diffusion Prior In this section, we show the details of getting a video diffusion prior on black hole video reconstruction and dynamic MRI, and we summarize the training hyperparameters in Table D.3. We define 𝐷 image and 𝐷 video as the image and video datasets, containing 𝑁image and 𝑁video data points, respectively. The image dataset 𝐷 image includes all individual frames from the video dataset 𝐷 video , along with additional large-scale image data to enhance generalization. For data augmentation, we apply random horizontal/vertical flipping and random zoom-in-and-out to improve robustness and diversity in training. We first train the compression functions, the encoder E and decoder D, on an image dataset. The training objective consists of an ℓ1 reconstruction loss combined with a KL divergence term scaled by a factor 𝛽𝐾 𝐿 . The loss function for training is as defined in Equation (D.17). The Adam optimizer is used as the default optimizer throughout the paper. The loss function for training the variational autoencoder 199 (VAE) is given by LVAE = E𝑞 𝜙 (𝒛0 |𝒙0 ),𝒙0 ∼𝐷 image [∥D (𝒛0 ) − 𝒙0 ∥ 1 ] + 𝛽KL KL 𝑞 𝜙 (𝒛 0 |𝒙0 )∥ 𝑝(𝒛0 )
(D.17) where 𝑝(𝒛0 ) is the standard Gaussian N (0, 𝑰) and 𝑞 𝜙 (𝒛0 |𝒙0 ) is the isotropic Gaussian distribution over 𝒛0 where the mean and standard deviation is given by E (𝒙 0 ). Next, we train the image diffusion U-Net 𝒔𝜽 using the standard score-matching loss h i 2 LIDM = E 𝒛0 ∼𝑞 𝜙 (𝒛0 |𝒙0 ),𝑥0 ∼𝐷 image ,𝜖∼N (0,𝑰),𝑡∼U (0,1) 𝜎𝑡2 𝒔𝜽 (𝒙𝑡 , 𝑡) − ∇𝒙𝑡 log 𝑝 𝑡 (𝒙𝑡 |𝒙0 ) , (D.18) following [131, 266]. After pre-training, the image diffusion U-Net 𝒔𝜽 is then converted to a spatiotemporal U-Net by adding zero-initialized temporal modules to 2D spatial modules and fine-tuned jointly with video and image datasets. We use the same Equation (D.18) without changing 𝒙0 to video or pseudo-video input and use the encoder to process each frame independently. Table D.3: Summary of the training of spatiotemporal diffusion prior. We provide and group the hyperparameters according to each component in the model. The model is trained with 1 NVIDIA A100-SCM4-80GB GPU. Hyper-parameters Black hole video reconstruction Dynamic MRI Dataset Related frames 𝑛 𝑓 resolution 𝑛 ℎ × 𝑛𝑤 𝑁image 𝑁video VAE Training Related latent channels block channels down sampling factor batch size epochs 𝛽KL 64 256×256 50000 648 12 192×192 39888 3324 1 [64, 128, 256, 256] 8 16 25 0.06 2 [256, 512, 512] 4 16 10 0.03 IDM Training Related block channels batch size epochs [128, 256, 512, 512] 16 200 [128, 256, 512, 512] 16 50 Joint Fine-tuning Related 𝑝 joint epochs 0.8 500 0.8 300 14.8M 131.7M 4.5h 5.5h 13.7h 57.5M 131.7M 8.9h 3.8h 22.8h Other Info VAE parameters diffusion model parameters VAE training time image diffusion model training time joint fine-tuning time 200 Table D.4: Additional results on dynamic MRI with 6× acceleration. Following the same setup as in Table 7.2, we report the quantitative results on 10 test videos. Tasks MRI (6×) Methods PSNR (↑) SSIM (↑) LPIPS (↓) d-PSNR (↑) d-SSIM (↑) FVD (↓) Data Misfit (↓) BIS [168] BCS [168] Bilinear [53] Bicubic [53] Nearest [53] ∫ -noise [53, 82] GP-Warp [77] 39.47 (0.59) 40.69 (0.57) 40.85 (0.57) 40.71 (0.67) 40.37 (0.56) 40.09 (0.50) 39.50 (0.48) 0.958 (0.007) 0.959 (0.006) 0.960 (0.006) 0.959 (0.007) 0.960 (0.007) 0.960 (0.006) 0.959 (0.007) 0.086 (0.011) 0.081 (0.012) 0.080 (0.012) 0.079 (0.012) 0.080 (0.012) 0.082 (0.012) 0.080 (0.012) 43.26 (1.23) 44.73 (1.31) 44.84 (1.28) 44.74 (1.38) 44.81 (1.41) 44.77 (1.34) 44.53 (1.33) 0.962 (0.005) 0.974 (0.004) 0.975 (0.004) 0.974 (0.005) 0.974 (0.005) 0.974 (0.005) 0.973 (0.005) 113.17 110.01 114.37 106.82 110.91 111.92 105.70 11.071 (0.740) 11.085 (0.720) 11.038 (0.735) 11.068 (0.755) 11.050 (0.739) 11.059 (0.731) 11.070 (0.727) STeP (video only) STeP (image-video joint) 40.76 (0.43) 41.39 (0.52) 0.967 (0.005) 0.969 (0.005) 0.077 (0.012) 0.076 (0.012) 46.38 (1.82) 46.61 (1.72) 0.981 (0.005) 0.982 (0.004) 101.83 98.15 10.788 (0.713) 10.808 (0.723) After pre-training, the image diffusion U-Net 𝒔𝜽 is transformed into a spatiotemporal U-Net by integrating zero-initialized temporal modules into the existing 2D spatial modules. The model is then fine-tuned jointly using both video and image datasets. We use the same loss as in Equation (D.18), by changing 𝒙0 to a video or a pseudovideo input. Each frame is independently processed using the encoder E, ensuring that spatial representations remain aligned while temporal consistency is learned through the added temporal modules. D.2.4 Limitations Though STeP is a general framework for solving scientific VIPs with spatiotemporal diffusion prior, the sampling cost of STeP is relatively high due to the requirement of backpropagation through decoder D in MCMC updates in Algorithm 7. So we have to strike a balance between the capability of the decoder and its computational cost. D.2.5 More Results and Visualization Data Misfit Values for Samples in Figure 7.6 We report the data misfit values in Table D.5. Table D.5: The data misfit values for samples shown in Figure 7.6. We report the data misfit metrics for the three obtained modes, which demonstrate that all modes fit the measurement data equally well. Metrics Mode 1 Mode 2 Mode 3 2 𝝌cp 2 𝝌logca Data Misfit 1.045 1.007 1.026 0.987 1.001 1.007 1.084 1.202 1.143 Dynamic MRI with Higher Acceleration To access the capability of using a spatiotemporal prior for solving more challenging inverse problems, we increase VAE Recon Ground truth 201 VAE Recon Ground truth (a) black hole imaging (b) dynamic MRI Figure D.3: Visualization of VAE reconstructions. The VAE reconstructions are computed by first encoding the ground truth videos and then decoding them. the acceleration times in Dynamic MRI, which makes the observation more sparse. The results are summarized in Table D.4. More Visualizations Here, we show the VAE reconstruction results in Figure D.3, unconditional samples in Figure D.4 and additional posterior samples in Figure D.5. D.3 Appendix for Section 7.5 D.3.1 D.3.1.1 Theory Notation Recall that for the prior step, we consider the following SDE that corresponds to Equation (6.6) with 𝑠(𝑡) = 1    √︁  √︁ 𝒙𝑡 d𝒙𝑡 = −(2𝜎(𝑡)𝜎(𝑡) ¤ + 𝛽(𝑡))∇ log 𝑝 𝑡 ; 𝜎(𝑡) d𝑡 + 2𝜎(𝑡)𝜎(𝑡) ¤ + 2𝛽(𝑡) d𝒘¯ 𝑡 . 𝑠(𝑡) (D.19) We denote the drift coefficient and the diffusion coefficient as ℎ(𝑡) and 𝛿(𝑡), respectively: ℎ(𝑡) := − (2𝜎(𝑡)𝜎(𝑡) ¤ + 𝛽(𝑡)) √︁ √︁ 𝛿(𝑡) := 2𝜎(𝑡)𝜎(𝑡) ¤ + 2𝛽(𝑡). (D.20) (D.21) Sample 5 Sample 4 Sample 3 Sample 2 Sample 1 202 Sample 5 Sample 4 Sample 3 Sample 2 Sample 1 (a) black hole imaging (b) dynamic MRI Figure D.4: Visualization of video diffusion model unconditional samples. The videos are sampled by solving the PF-ODE with 100 Euler steps. STeP Ground truth STeP Ground truth STeP Ground truth 203 STeP Ground truth STeP Ground truth STeP Ground truth (a) black hole imaging (b) dynamic MRI Figure D.5: Visualization of STeP posterior samples. The videos are sampled using the Algorithm 7. 204 Recall from Section 6.4 that the Kullback–Leibler (KL) divergence between two distributions 𝜇 and 𝜋 is defined by ∫ h 𝜇 𝜇i KL(𝜇||𝜋) = 𝜇 log = E 𝜇 log . 𝜋 𝜋 The Fisher divergence between two distributions 𝜇 and 𝜋 is defined by ∫ 𝜇 2 𝜇 2 = E 𝜇 ∇ log . FI(𝜇||𝜋) = 𝜇 ∇ log 𝜋 2 𝜋 2 For 𝒙 ∈ R𝑛 , we define the weighted norm induced by a positive semi-definite matrix 𝑴 ∈ R𝑛×𝑛 as ∥𝒙∥ 2𝑴 = 𝒙𝑇 𝑴𝒙, (D.22) and the divergence of a matrix 𝑇 (𝒙) ∈ R𝑛×𝑛 as the vector field (∇𝒙 · 𝑇)𝑖 = 𝑛 ∑︁ 𝜕𝑇𝑖 𝑗 𝑗=1 D.3.1.2 𝒙𝑗 . (D.23) Assumptions and Lemmas Assumption D.3.1. The average score approximation error of the diffusion model 𝒔𝑡 := 𝒔𝜽 ( · ; 𝑡) is bounded,  ∫ 𝑇𝑘+1  ℎ(𝑡) 2 1 2 𝜖score = sup E ∥𝒔 − ∇𝒙𝑡 log 𝑝 𝜎(𝑡) ∥ 2 d𝑡 < +∞, (D.24) ∗ 2 𝜇𝑡 𝑡 𝑘=0,...,𝐾−1 𝑡 𝑇𝑘 +𝑡 † 𝛿(𝑡) where ℎ(𝑡) is defined in Equation (D.20) and 𝛿(𝑡) is defined in Equation (D.21). Assumption D.3.2. The average derivative approximation error of the linear surrogate model 𝑨𝑡 is bounded,     2   1 ∫ 𝑇𝑘 +𝑡 †    1 𝑇 ( 𝑗) 𝜖model = sup E ∇ 𝑓 (𝒛 ; 𝒚) − 𝑨 A (𝒛 ) − 𝒚 d𝑡 < +∞, 𝜇 𝑡 𝑡 𝑡 𝑡  𝜎𝒚2 𝑘=0,...,𝐾−1   𝑡 † 𝑇𝑘  𝐶𝑡   (D.25) where ∥ · ∥ 𝐶𝑡 is the weighted norm defined in Equation (D.22). 𝑡 Assumption D.3.3. The Radon–Nikodym derivative d𝜇 de 𝜇𝑡 is constant along the null space of 𝐶𝑡 almost surely, where 𝐶𝑡 is the covariance matrix of 𝜇 e𝑡 . d𝜇𝑡 d𝜇𝑡 (𝒙) = (𝒙 + 𝒗), ∀𝒗 ∈ Ker(𝐶𝑡 ). de 𝜇𝑡 de 𝜇𝑡 205 Remark D.3.4. Assumption D.3.1 coincides with the usual bounded score-matching error condition that underpins convergence results [57, 175, 320]. Assumption D.3.2 characterizes the ℓ2 accuracy of the linear proxy. Assumption D.3.3 is the weakest assumption to bound the weighted Fisher divergence in our analysis. Two common sufficient (but not necessary) scenarios are: (1) 𝐶𝑡 is full-rank and (2) 𝜇𝑡 is absolutely continuous with respect to 𝜇 e𝑡 , and both log densities are continuously differentiable. Lemma D.3.5 (Stationary distribution of the likelihood step). Assume the particle distribution is not a Dirac measure, the dynamics of Equation (7.11) admits ( 𝑗) 𝜋 𝑍 |𝑋=𝒙 (𝒛) ∝ exp(− 𝑓 (𝒛; 𝒚) − 2𝜂1 2 ∥𝒛 − 𝒙 ( 𝑗) ∥ 22 ) as a stationary distribution. Further, if the covariance matrix is positive definite, the stationary distribution is unique. Proof. This result has been proved in various forms in the literature [111, 206]. Here, we provide a simple proof of our use case for ease of understanding. Suppose 𝜇𝑡 (𝒛) is the probability density of 𝒛 at time 𝑡. For the ease of notation, we ignore the particle index 𝑗 in 𝒛𝑡 . Let Φ(𝒛) := 𝑓 (𝒛; 𝒚) + 2𝜂1 2 ∥𝒛 − 𝒙 ( 𝑗) ∥ 22 . The corresponding Fokker-Planck equation for Equation (7.11) reads 𝜕𝑡 𝜇𝑡 = ∇ · (𝜇𝑡 𝐶𝑡 ∇Φ(𝒛)) + ∇ · (𝐶𝑡 ∇𝜇𝑡 ) , which can be rewritten as 𝜕𝑡 𝜇𝑡 = ∇ · (𝜇𝑡 𝐶𝑡 (∇Φ(𝒛) + ∇ log 𝜇𝑡 )) . (D.26) Let 𝜇∞ denote the stationary distribution of Equation (D.26). We have 0 = ∇ · (𝜇𝑡 𝐶𝑡 (∇Φ(𝒛) + ∇ log 𝜇∞ )) . If the particle distribution is not Dirac, 𝐶𝑡 ≠ 0 due to Lemma 2.1 in [111]. Therefore, ∇Φ(𝒛) + ∇ log 𝜇∞ = 𝑐, where 𝑐 is a constant. Integrating both sides gives   1 ( 𝑗) 2 𝜇∞ (𝒛) ∝ exp(−Φ(𝒛)) = exp − 𝑓 (𝒛; 𝒚) − 2 ∥𝒛 − 𝒙 ∥ 2 , 2𝜂 ( 𝑗) showing that 𝜋 𝑍 |𝑋=𝒙 (𝒛) is a stationary distribution of the dynamics of Equation (7.11). Further, if 𝐶𝑡 is positive definite, it ensures the irreducibility and strong Feller property, and the stationary distribution is unique [206, 247]. 206 Lemma D.3.6. Given the following pair of stochastic processes d𝒙𝑡 = 𝑏(𝒙𝑡 , 𝑡)d𝑡 + 𝐻 (𝑡)d𝒘 𝑡 , (D.27) de 𝒙𝑡 = e 𝑏(e 𝒙𝑡 , 𝑡)d𝑡 + 𝐻 (𝑡)d𝒘 𝑡 , (D.28) where 𝑏, e 𝑏 : R𝑛 × R+ → R𝑛 are the drift terms, 𝐻 : R+ → R𝑛 × R𝑛 is the diffusion term, 𝒘 𝑡 is the standard Wiener process. Let 𝜇𝑡 (respectively 𝜇 e𝑡 ) be the law of 𝒙𝑡 (respectively e 𝒙𝑡 ), 𝐶 (𝑡) := 𝐻 (𝑡)𝐻 (𝑡)𝑇 , and 𝜆∗𝑡 be the smallest non-zero eigenvalue of 𝐶 (𝑡). Assuming that 𝑏 𝑡 − e 𝑏 𝑡 ∈ Range(𝐶 (𝑡)) and Assumption D.3.3 holds, we have 2 𝜆∗ 𝜕𝑡 KL(𝜇𝑡 ||e 𝜇𝑡 ) ≤ − 𝑡 FI(𝜇𝑡 ||e 𝜇𝑡 ) + E 𝜇𝑡 𝑏 𝑡 − e 𝑏𝑡 , 𝐶 (𝑡) † 4 (D.29) where 𝐶 (𝑡) † is the pseudo-inverse of 𝐶 (𝑡). Proof. Since the diffusion terms only depend on 𝑡 and 𝐶 (𝑡) = 𝐻 (𝑡)𝐻 (𝑡)𝑇 , the Fokker-Planck equations of Equation (D.27) and Equation (D.28) read    1 𝜕𝑡 𝜇𝑡 = ∇ · 𝐶 (𝑡)∇ log 𝜇𝑡 − 𝑏 𝑡 𝜇𝑡 , (D.30) 2    1 e 𝜕𝑡 𝜇 e𝑡 = ∇ · 𝐶 (𝑡)∇ log 𝜇 e𝑡 − 𝑏 𝑡 𝜇 e𝑡 . (D.31) 2 Let 𝑟 𝑡 := 𝜇𝜇e𝑡𝑡 and 𝜙(𝑟 𝑡 ) := 𝑟 𝑡 log 𝑟 𝑡 (so 𝜙′ (𝑟 𝑡 ) = d𝑟d𝑡 𝜙(𝑟 𝑡 ) = log 𝑟 𝑡 + 1). Differentiating the KL divergence gives ∫ 𝜕𝑡 KL(𝜇𝑡 ||e 𝜇𝑡 ) = 𝜕𝑡 𝜙(𝑟 𝑡 ) 𝜇 e𝑡 ∫ [𝜙(𝑟 𝑡 )𝜕𝑡 𝜇 = e𝑡 + 𝜙′ (𝑟 𝑡 )𝜕𝑡 𝜇 e𝑡 ] ∫ [𝜙(𝑟 𝑡 )𝜕𝑡 𝜇 = e𝑡 + 𝜙′ (𝑟 𝑡 )𝜕𝑡 𝜇𝑡 − 𝜙′ (𝑟 𝑡 )𝑟 𝑡 𝜕𝑡 𝜇 e𝑡 ] ∫ [(log 𝑟 𝑡 + 1)𝜕𝑡 𝜇𝑡 − 𝑟 𝑡 𝜕𝑡 𝜇 = e𝑡 ] , (D.32) where the last step uses the fact that 𝜙(𝑟 𝑡 ) −𝑟 𝑡 𝜙′ (𝑟 𝑡 ) = −𝑟 𝑡 . Plugging Equation (D.30) and Equation (D.31) into Equation (D.32) and applying integration by parts further, 207 we have 𝜕𝑡 KL(𝜇𝑡 ||e 𝜇𝑡 )    ∫    ∫ 1 1 = (log 𝑟 𝑡 + 1)∇ · 𝐶 (𝑡)∇ log 𝜇𝑡 − 𝑏 𝑡 𝜇𝑡 − 𝑟𝑡 ∇ · 𝐶 (𝑡)∇ log 𝜇 e𝑡 − e 𝑏𝑡 𝜇 e𝑡 2 2   ∫  ∫  1 1 e ∇𝑟 𝑡 , 𝐶 (𝑡)∇ log 𝜇 e𝑡 − 𝑏 𝑡 𝜇 e𝑡 =− ∇ log 𝑟 𝑡 , 𝐶 (𝑡)∇ log 𝜇𝑡 − 𝑏 𝑡 𝜇𝑡 + 2 2   ∫  ∫  1 1 e =− ∇ log 𝑟 𝑡 , 𝐶 (𝑡)∇ log 𝜇𝑡 − 𝑏 𝑡 𝜇𝑡 + ∇ log 𝑟 𝑡 , 𝐶 (𝑡)∇ log 𝜇 e𝑡 − 𝑏 𝑡 𝜇𝑡 2 2  ∫ D ∫  E 1 ∇ log 𝑟 𝑡 , 𝑏 𝑡 − e 𝑏 𝑡 𝜇𝑡 e𝑡 ) 𝜇𝑡 + =− ∇ log 𝑟 𝑡 , 𝐶 (𝑡) (∇ log 𝜇𝑡 − ∇ log 𝜇 2 ∫ ∫ D E 1 ⟨∇ log 𝑟 𝑡 , 𝐶 (𝑡)∇ log 𝑟 𝑡 ⟩ 𝜇𝑡 + =− ∇ log 𝑟 𝑡 , 𝑏 𝑡 − e 𝑏 𝑡 𝜇𝑡 . (D.33) 2 The weighted Young’s inequality states that, for any 𝑢, 𝑣 ∈ R𝑛 , when 𝑣 ∈ Range(𝐶), we have 1 ⟨𝑢, 𝑣⟩ ≤ ⟨𝑢, 𝐶𝑢⟩ + 𝑣, 𝐶 † 𝑣 , 4 where 𝐶 † is the pseudo-inverse. By Assumption D.3.3, Equation (D.33) can be bounded as follows ∫ ∫ D E 1 e ⟨∇ log 𝑟 𝑡 , 𝐶 (𝑡)∇ log 𝑟 𝑡 ⟩ 𝜇𝑡 + − ∇ log 𝑟 𝑡 , 𝑏 𝑡 − 𝑏 𝑡 𝜇𝑡 2 ∫ ∫ D E 1 ⟨∇ log 𝑟 𝑡 , 𝐶 (𝑡)∇ log 𝑟 𝑡 ⟩ 𝜇𝑡 + 𝑏𝑡 − e 𝑏 𝑡 , 𝐶 (𝑡) † (𝑏 𝑡 − e 𝑏 𝑡 ) 𝜇𝑡 ≤− 4 2 𝜆∗ ≤ − 𝑡 FI(𝜇𝑡 ||e 𝜇𝑡 ) + E 𝜇𝑡 𝑏 𝑡 − e 𝑏𝑡 (D.34) 𝐶 (𝑡) † 4 where 𝜆∗𝑡 is the smallest non-zero eigenvalue of 𝐶 (𝑡). This is a generalization of Lemma C.1.4 to the general matrix-valued diffusion term. Intuitively, the condition that 𝑏 𝑡 − e 𝑏 𝑡 belongs to the range of 𝐶 (𝑡) means that the two drift terms may only differ along the directions that are actually driven by noise. In the context of our proof below, this is always satisfied because the drift terms are either preconditioned with 𝐶 (𝑡) or 𝐶 (𝑡) is full-rank. 208 D.3.1.3 Proof of Theorem 7.5.1 Proof. For 𝜏 ∈ [𝑇𝑘 , 𝑇𝑘 +𝑡 † ], 𝑘 = 0, ..., 𝐾 −1, we apply Lemma D.3.6 to the likelihood step with 1 𝐶𝑡 (𝒛𝑡 − 𝒙 ( 𝑗) ) 𝜂2 1 1 e 𝑏(𝒛𝑡 , 𝑡) := −𝐶𝑡 2 𝑨𝑇𝑡 (A (𝒛𝑡 ) − 𝒚) − 2 𝐶𝑡 (𝒛𝑡 − 𝒙 ( 𝑗) ) 𝜂 𝜎𝒚 √︁ 𝐻 (𝑡) = 𝐶𝑡 , 𝑏(𝒛𝑡 , 𝑡) := −𝐶𝑡 ∇ 𝑓 (𝒛𝑡 ; 𝒚) − where 𝑨𝑡 = E 𝜇e𝑡 [(A (𝒛𝑡 ) −E𝑞 𝑡 A (𝒛𝑡 ))𝒛𝑇𝑡 ]𝐶𝑡−1 as defined in Equation (7.13). Note that the condition 𝑏 − e 𝑏 ∈ Range(𝐶𝑡 ) is satisfied as both drift terms are preconditioned with 𝐶𝑡 . Thus, by Assumption D.3.3, we have D E 𝜆∗ 𝜇𝜏 ) + E𝜇 𝜏 𝑏 𝜏 − e 𝑏 𝜏 , 𝐶𝜏† (𝑏 𝜏 − e 𝑏𝜏 ) 𝜕𝜏 KL(𝜇𝜏 ||e 𝜇𝜏 ) ≤ − 𝜏 FI(𝜇𝜏 ||e 4 1 𝜆∗ ≤ − FI(𝜇𝜏 ||e 𝜇𝜏 ) + E 𝜇 𝜏 ∇ 𝑓 (𝒛 𝜏 ; 𝒚) − 2 𝑨𝑇𝜏 (A (𝒛 𝜏 ) − 𝒚) 4 𝜎𝒚 2 𝐶𝜏 where 𝜆∗𝜏 is the smallest non-zero eigenvalue of 𝐶𝜏 and 𝜆∗ := inf 𝜏∈[𝑇𝑘 ,𝑇𝑘 +𝑡 † ],𝑘=0,...,𝐾−1 𝜆∗𝜏 . By Assumption D.3.2 , integrating both sides over [𝑇𝑘 , 𝑇𝑘 + 𝑡 † ] gives KL(𝜇𝑇𝑘 +𝑡 † ||e 𝜇𝑇𝑘 +𝑡 † ) − KL(𝜇𝑇𝑘 ||e 𝜇𝑇𝑘 ) 2 ∫ ∫ 𝑇𝑘 +𝑡 † † 𝜆∗ 𝑇𝑘 +𝑡 1 𝑇 ≤− FI(𝜇𝜏 ||e 𝜇𝜏 )d𝜏 + d𝜏 E 𝜇 𝜏 ∇ 𝑓 (𝒛 𝜏 ; 𝒚) − 2 𝑨𝜏 (A (𝒛 𝜏 ) − 𝒚) 4 𝑇𝑘 𝜎𝒚 𝑇𝑘 𝐶𝜏 ∫ 𝑇𝑘 +𝑡 † ∗ 𝜆 ≤− FI(𝜇𝜏 ||e 𝜇𝜏 )d𝜏 + 𝑡 † 𝜖model , (D.35) 4 𝑇𝑘 where 𝜖model is defined in Equation (D.25). For 𝜏 ∈ [𝑇𝑘 + 𝑡 † , 𝑇𝑘+1 ], 𝑘 = 0, ..., 𝐾 − 1, we apply Lemma D.3.6 to the prior step (D.19) with 𝑏(𝒙𝑡 , 𝑡) := ℎ(𝑡)∇𝒙𝑡 log 𝑝 𝑡 (𝒙𝑡 ; 𝜎(𝑡)) e 𝑏(𝒛𝑡 , 𝑡) := ℎ(𝑡)𝒔𝜽 (𝒙𝑡 ; 𝑡) 𝐻 (𝑡) := 𝛿(𝑡) 𝑰, where ℎ(𝑡) is the drift coefficient defined in Equation (D.20), 𝛿(𝑡) is the diffusion coefficient defined in Equation (D.21), 𝒔𝜽 is the pre-trained diffusion model with score approximation error 𝜖score . Note that 𝐻 (𝑡)𝐻 (𝑡)𝑇 is full-rank so that the 209 condition of Lemma D.3.6 is satisfied. Therefore, we have 𝛿(𝜏) 2 ℎ(𝜏) 2 E 𝜇 ∥∇𝒙 𝜏 log 𝑝 𝜎(𝜏) − 𝒔𝜏 ∥ 22 FI(𝜇𝜏 ||e 𝜇𝜏 ) + 4 𝛿(𝜏) 2 𝜏 𝛿 ℎ(𝜏) 2 ≤ − FI(𝜇𝜏 ||e E 𝜇 ∥∇𝒙 𝜏 log 𝑝 𝜎(𝜏) − 𝒔𝜏 ∥ 22 , 𝜇𝜏 ) + 4 𝛿(𝜏) 2 𝜏 𝜕𝜏 KL(𝜇𝜏 ||e 𝜇𝜏 ) ≤ − where 𝛿 := inf 𝜏∈[0,𝑡 ∗ ] 𝛿(𝜏) 2 . Integrating both sides over [𝑇𝑘 + 𝑡 † , 𝑇𝑘+1 ] and applying Assumption D.3.1 gives KL(𝜇𝑇𝑘+1 ||e 𝜇𝑇𝑘+1 ) − KL(𝜇𝑇𝑘 +𝑡 † ||e 𝜇𝑇𝑘 +𝑡 † ) ∫ 𝑇𝑘+1 ∫ 𝑇𝑘+1 𝛿 ℎ(𝜏) 2 ≤− E 𝜇 𝜏 ∥ log 𝑝 𝜎(𝜏) − 𝒔𝜏 ∥ 22 d𝜏 FI(𝜇𝜏 ||e 𝜇𝜏 )d𝜏 + 2 4 𝑇𝑘 +𝑡 † 𝑇𝑘 +𝑡 † 𝛿(𝜏) ∫ 𝑇𝑘+1 𝛿 ≤− FI(𝜇𝜏 ||e 𝜇𝜏 )d𝜏 + 𝑡 ∗ 𝜖 score , (D.36) 4 𝑇𝑘 +𝑡 † where 𝜖score is defined in Equation (D.24). Summing up both sides of Equation (D.35) and Equation (D.36) for 𝑘 = 0, ..., 𝐾 − 1 gives ∫ min(𝜆∗ , 𝛿) 𝑇𝐾 FI(𝜇𝜏 ||e 𝜇𝜏 )d𝜏+𝐾 (𝑡 † 𝜖model +𝑡 ∗ 𝜖 score ). KL(𝜇𝑇𝐾 ||e 𝜇𝑇𝐾 )−KL(𝜇0 ||e 𝜇0 ) ≤ − 4 0 Rearranging the terms gives ∫ 𝑇𝐾 1 FI(𝜇𝜏 ||e 𝜇𝜏 )d𝜏 𝑇𝐾 0
4 4 (𝜖model + 𝜖score ) ≤ KL(𝜇 ||e 𝜇 ) − KL(𝜇 ||e 𝜇 ) + 0 0 𝑇 𝑇 𝐾 𝐾 ∗ ∗ 𝑇𝐾 min(𝜆 , 𝛿) min(𝜆 , 𝛿)(𝑡 † + 𝑡 ∗ )   4 KL(𝜇0 ||e 𝜇0 ) 𝑡 † 𝜖model + 𝑡 ∗ 𝜖score ≤ + min(𝜆∗ , 𝛿) 𝐾 (𝑡 † + 𝑡 ∗ ) 𝑡† + 𝑡∗ 𝑐KL(𝜇0 ||e 𝜇0 ) + 𝑐𝑡 † 𝜖model + 𝑐𝑡 ∗ 𝜖score = 𝐾 4 where 𝑐 := min(𝜆∗ ,𝛿) . The proof is concluded by the fact that 𝜇0 = 𝜋 𝑋 . (𝑡 † +𝑡 ∗ ) Lemma D.3.7. Let 𝜋(𝒛; 𝒙 ( 𝑗) ) denote the h invariant measure associated with i the 1 𝑇 1 ( 𝑗) ( 𝑗) ( 𝑗) potential Φ(𝒛; 𝒙 ) where ∇ 𝒛 Φ(𝒛; 𝒙 ) = 𝜎2 𝑨𝑡 (A (𝒛) − 𝒚) + 𝜂2 (𝒛 − 𝒙 ) . Then 𝒚 𝜋(𝒛; 𝒙 ( 𝑗) ) is an invariant measure of the finite-particle system in Equation (7.15) as well as its large particle limit in Equation (7.14). Proof. In the large particle limit, the covariance 𝐶𝑡 does not depend on any specific particle but depends on the particle distribution only. Therefore, the Fokker-Planck 210 equation of Equation (7.14) reads:   ( 𝑗) ( 𝑗) 𝜕𝑡 𝑝 𝑡 = ∇ · 𝑝 𝑡 𝐶𝑡 ∇Φ(𝒛𝑡 ; 𝒙 ) + 𝐶𝑡 ∇2 𝑝 𝑡    ( 𝑗) = ∇ · 𝑝 𝑡 𝐶𝑡 ∇Φ(𝒛𝑡 ; 𝒙 ( 𝑗) ) + ∇ log 𝑝 𝑡 where 𝑝 𝑡 is the probability density at time 𝑡. We can see that 𝜋(𝒛; 𝒙 ( 𝑗) ) is an invariant measure by setting both sides to zero. In the finite-particle system, the covariance Í ( 𝑗) ( 𝑗) ( 𝑗) 𝐶𝑡 = 1𝐽 𝐽𝑗=1 (𝒛𝑡 − 𝒛¯𝑡 )(𝒛𝑡 − 𝒛¯𝑡 )𝑇 , which depends on the current state 𝒛𝑡 . Therefore, the Fokker-Plank equation of the finite-particle dynamics in Equation (7.15) is    𝑛 + 1 ( 𝑗) ( 𝑗) ( 𝑗) (𝒛𝑡 − 𝒛¯𝑡 ) + ∇ · (∇ · ( 𝑝 𝑡 𝐶𝑡 )) 𝜕𝑡 𝑝 𝑡 =∇ · 𝑝 𝑡 𝐶𝑡 ∇Φ(𝒛𝑡 ; 𝒙 ) − 𝐽    𝑛 + 1 ( 𝑗) ( 𝑗) ( 𝑗) (𝒛𝑡 − 𝒛¯𝑡 ) + ∇ · (𝐶𝑡 ∇𝑝 𝑡 + 𝑝 𝑡 ∇ · 𝐶𝑡 ) =∇ · 𝑝 𝑡 𝐶𝑡 ∇Φ(𝒛𝑡 ; 𝒙 ) − 𝐽   h  i 𝑛 + 1 ( 𝑗) ( 𝑗) ( 𝑗) =∇ · 𝑝 𝑡 𝐶𝑡 ∇Φ(𝒛𝑡 ; 𝒙 ) + ∇ log 𝑝 𝑡 − ∇ · (𝒛𝑡 − 𝒛¯𝑡 ) + ∇ · ( 𝑝 𝑡 ∇ · 𝐶𝑡 ) 𝐽   h  i 𝑛 + 1 ( 𝑗) ( 𝑗) ( 𝑗) (𝒛𝑡 − 𝒛¯𝑡 ) =∇ · 𝑝 𝑡 𝐶𝑡 ∇Φ(𝒛𝑡 ; 𝒙 ) + ∇ log 𝑝 𝑡 − ∇ · 𝐽 ! 𝐽 1 ∑︁ (𝑖) + ∇ · 𝑝𝑡 ∇𝒛 ( 𝑗 ) · (𝒛 − 𝒛¯𝑡 )(𝒛𝑡(𝑖) − 𝒛¯𝑡 )𝑇 𝑡 𝐽 𝑖=1 𝑡   h  i 𝑛 + 1 ( 𝑗) ( 𝑗) ( 𝑗) =∇ · 𝑝 𝑡 𝐶𝑡 ∇Φ(𝒛𝑡 ; 𝒙 ) + ∇ log 𝑝 𝑡 − ∇ · 𝑝 𝑡 (𝒛𝑡 − 𝒛¯𝑡 ) 𝐽   1 ( 𝑗) + ∇ · 𝑝 𝑡 (𝑛 + 1)(𝒛𝑡 − 𝒛¯𝑡 ) 𝐽 h  i ( 𝑗) =∇ · 𝑝 𝑡 𝐶𝑡 ∇Φ(𝒛𝑡 ; 𝒙 ( 𝑗) ) + ∇ log 𝑝 𝑡 where the divergence of a matrix is defined in Equation (D.23), and we use the following properties:   ( 𝑗) ( 𝑗) 𝑇 ( 𝑗) ∇𝒛 ( 𝑗 ) · 𝒛𝑡 𝒛𝑡 = (𝑛 + 1)𝒛𝑡 , 𝑡   𝑇 ( 𝑗) ∇ 𝒛 ( 𝑗 ) · 𝒛𝑡 𝒛𝑡(𝑖) = 𝒛𝑡(𝑖) , 𝑡   (𝑖) ( 𝑗) 𝑇 ∇𝒛 ( 𝑗 ) · 𝒛𝑡 𝒛𝑡 = 𝑛𝒛𝑡(𝑖) , 𝑡  𝑛+1  𝑇 ∇ 𝒛 ( 𝑗 ) · 𝒛¯𝑡 𝒛¯𝑡 = 𝒛¯𝑡 , 𝑡 𝐽 where 𝑖 ≠ 𝑗. By taking both sides to zero, we have that 𝜋(𝒛; 𝒙 ( 𝑗) ) is an invariant measure of Equation (7.15) as well. This proof is largely adapted from [112, 226], which apply to more general scenarios. We tailor the proof to our case for ease of understanding. 211 Algorithm 8 LikelihoodStep of Algorithm 3 Require: initial ensemble 𝑿 = {𝒙 ( 𝑗) } 𝐽𝑗=1 , forward model A, observation 𝒚, effective observation noise e 𝜎𝒚 , coupling strength 𝜂, number of discretization steps 𝑁, step size scale 𝛾. ( 𝑗) 1: 𝒁 0 = {𝒛 0 } 𝐽𝑗=1 ← 𝑿 2: for 𝑖 = 0, ..., 𝑁 − 1 do Í ( 𝑗) ( 𝑗 ′) ( 𝑗) ( 𝑗 ′) 3: 𝒅1 ← − 1𝐽 𝐽𝑗′ =1 e𝜎12 ⟨A (𝒛𝑖 ) − Ā, A (𝒛𝑖 ) − 𝒚⟩(𝒛𝑖 − 𝒛¯𝑖 ) for 𝑗 = 1, ..., 𝐽 𝒚   (1) (𝐽) 4: 𝐶𝑖 ← cov 𝒛𝑖 , ..., 𝒛𝑖 ( 𝑗) ( 𝑗) ( 𝑗) 𝒅2 ← − 𝜂12 𝐶𝑖 (𝒙 ( 𝑗) − 𝒛𝑖 ) + 𝑛+1 − 𝒛¯𝑖 ) for 𝑗 = 1, ..., 𝐽 𝐽 (𝒛 h i𝑖 √ 6: 𝐶𝑖 ← √1 𝒛𝑖(1) − 𝒛¯𝑖 , ..., 𝒛𝑖(𝐽) − 𝒛¯𝑖 ∈ R𝑛×𝐽 𝐽 7: 𝝐𝑖 𝑗 ∼ N (0, 𝑰 𝐽 ) for 𝑗 = 1, ..., 𝐽 8: 𝜂 ← 𝛾/∥ 𝒅1 + 𝒅2 ∥ 22 √︁ √ ( 𝑗) ( 𝑗) ( 𝑗) ( 𝑗) 9: 𝒛𝑖+1 ← 𝒛𝑖 + 𝜂( 𝒅1 + 𝒅2 ) + 2𝜂 𝐶𝑖 𝝐𝑖 𝑗 for 𝑗 = 1, ..., 𝐽 10: end for ( 𝑗) 11: return 𝒁 𝑁 = {𝒛 𝑁 } 𝐽𝑗=1 5: Algorithm 9 PriorStep of Algorithm 3 Require: initial ensemble 𝒁 = {𝒛 ( 𝑗) } 𝐽𝑗=1 , diffusion model 𝒔𝜽 , assumed noise level 𝜂 > 0, number of discretization steps 𝑁, noise schedule 𝜎(𝑡) = 𝑡, discretization 𝑁 (monotonically decreasing to 𝑡 = 0), solver (SDE or ODE) time steps {𝑡𝑖 }𝑖=0 𝑁 ∗ 1: 𝑖 ← min {𝑖 ∈ [𝑁] | 𝜎(𝑡𝑖 ) ≤ 𝜂} ( 𝑗) 2: 𝑿𝑖 ∗ = {𝒙𝑖 ∗ } 𝐽𝑗=1 ← 𝒁 3: for 𝑖 = 𝑖 ∗ , ..., 𝑁 − 1 do 4: 𝜆 ← 2 if solver is SDE else 1 ( 𝑗) ( 𝑗) 5: 𝒅𝑖 ← −𝜆𝑡𝑖 𝒔𝜽 (𝒙𝑖 ; 𝜎(𝑡𝑖 )) for 𝑗 = 1, ..., 𝐽 ( 𝑗) ( 𝑗) ( 𝑗) for 𝑗 = 1, ..., 𝐽 6: 𝒙𝑖+1 ← 𝒙𝑖 + (𝑡𝑖+1 − 𝑡𝑖 ) 𝒅𝑖 7: if 𝑖 ≠ 𝑁 − 1 and solver is SDE then 8: 𝝐𝑖 𝑗 ∼ N (0, 𝑰) for 𝑗 = 1, ..., 𝐽 √︁ ( 𝑗) ( 𝑗) 9: 𝒙𝑖+1 ← 𝒙𝑖+1 + 2𝑡𝑖 (𝑡𝑖 − 𝑡𝑖+1 )𝝐𝑖 𝑗 for 𝑗 = 1, ..., 𝐽 10: end if 11: end for ( 𝑗) 12: return 𝑿 𝑁 = {𝒙 𝑁 } 𝐽𝑗=1 212 D.3.2 D.3.2.1 Background Traditional Methods for Derivative-Free Posterior Estimation Traditional methods for derivative-free posterior estimation include Markov chain Monte Carlo (MCMC) samplers [75, 113, 115] and Sequential Monte Carlo (SMC) approaches [78]. These methods offer theoretical convergence guarantees but face significant scalability challenges in real-world applications, especially in high dimensions. On the other hand, approximate Bayesian methods [46, 111, 138] offer better efficiency, but often struggle to capture complex posteriors. Additionally, these methods require access to the prior density (up to a normalizing constant). However, in practice, prior knowledge is often implicit in data (e.g., simulation archives and historical measurements) and difficult to directly model the density in high dimensions. D.3.2.2 Diffusion-Based Derivative-Free Posterior Estimation Many recent derivative-free algorithms [141, 277, 356] leverage diffusion models (DMs) as plug-and-play priors for solving high-dimensional inverse problems with complex prior distributions. DMs can flexibly capture complex prior distributions, but require optimization or sampling for posterior inference, mainly due to modeling the score function rather than the density. Optimization-based approaches typically introduce approximations that can lead to mis-calibrated posterior samples even in simple linear-Gaussian settings (see Section 5.1 of [297]). Sampling-based approaches are often asymptotically correct sampling [44, 89, 283, 318], but are typically strictly restricted to linear problems and do not generalize to nonlinear settings. D.3.2.3 Ensemble Kalman Methodology The Ensemble Kalman methodology was first introduced by [96] in the context of filtering problems and has gained popularity in applications such as reservoir modeling [227] and weather forecasting [137] due to its derivative-free nature and effectiveness in practical settings. In the context of inverse problems, [143] revisits this idea to propose Ensemble Kalman Inversion (EKI), spawning a variety of extensions: momentum-augmented updates for neural network training [166], Tikhonov and other regularization schemes for improved stability and efficiency [48, 139, 144]. More recently, [161, 356] have explored this idea to create derivative-free 213 diffusion guidance from an optimization perspective to guide the generation of a diffusion model for different applications. D.3.3 Implementation Details of Blade In this sub-section, we provide details on the practical implementation of Blade. The choices on key hyperparameters are summarized in Table D.6. All the experiments are conducted on single NVIDIA GH200 GPU. D.3.3.1 Likelihood Step Initialization As Theorem 7.5.1 indicates, the initialization of Blade is quite flexible, provided the initial distribution maintains a finite KL divergence from the target distribution. We used samples from the prior distribution as initializations for the Navier-Stokes equation experiments, indicated by “DM” in Table D.6. Discretization We discretize the SDE in Equation (7.15) using the standard Euler method with an adaptive step size defined as 𝛾 , Step size = ∥drift∥ 22 where 𝛾 is the hyperparameter that controls the scale of the step size, drift is the drift term of the SDE in Equation (7.15). This adaptive step size is effective across all our experiments. Further design of adaptive step sizes could potentially reduce discretization error with fewer steps. Resample During the likelihood step, we employ a resampling strategy to ensure that the particles are at the correct noise level 𝜂. Resampling is a commonly used method that has been shown to help improve the performance of algorithms such as DAPS [341], DiffPIR [361], and ReSample [260]. Specifically, we define the following resampling strategy: ( 𝑗) 𝒛 resample = 𝒛 ( 𝑗) + 𝜂′𝝐, 𝑡) where 𝝐 ∼ N (0, 𝑰), 𝜂′ = max(0, 𝜂 − tr(𝐶 𝑛 ), and 𝑛 is the dimension of the variable 𝒛. Intuitively, we approximate the current noise level in 𝒛 and add a corresponding ( 𝑗) amount of noise to bring 𝒛 resample to noise level 𝜂. A key distinction from the prior work is that our 𝜂′ is estimated from the ensemble while the existing methods need to tune it as part of the hyperparameters. In our main experiments, we apply a resampling strategy since it introduces minimal additional computation cost and yields slightly better results. 214 Table D.6: Hyperparameters of Blade for the Navier-Stokes experiments in Table 7.3. 𝜎𝒚 𝛾 e 𝜎𝒚 𝜂max 𝜂min 0 1 2 20 0.03 30 0.17 30 0.30 4.8 4.8 4.8 0.08 0.08 0.08 Anneal schedule # anneal steps linear linear linear 25 25 25 # iterations Initialization 50 50 50 DM DM DM Effective Observation Noise In practice, we observe that weighting the likelihood with a smaller 𝜎𝒚 yields better performance. We denote this adjusted value as the effective observation noise, e 𝜎𝒚 . Using an effective noise smaller than 𝜎𝒚 potentially compensates for the smoothing effect introduced by statistical linearization. In practice, we treat e 𝜎𝒚 as a hyperparameter and tune it so that the spread-skill ratio is close to 1. D.3.3.2 Prior Step The prior step is implemented as a denoising diffusion process, with its pseudocode detailed in Algorithm 9. We set 𝜎(𝑡) = 𝑡 for simplicity and employ the Euler ODE sampler for faster sampling. We discretize the denoising diffusion process with the standard Euler method. Following [151], we use the following step size:  7 𝑖  1/7 1/7  1/7 𝑡 −𝑡 for 𝑖 = 0, ..., 𝑁 − 1. 𝑡𝑖 = 𝑡max + 𝑁 − 1 min max D.3.3.3 Annealing Schedule We use the linear annealing schedule for the coupling strength 𝜂. Given the number of iterations 𝐾, the maximum value 𝜂max , and minimum value 𝜂min , the linear decay schedule reduces 𝜂 𝑘 as 𝜂 𝑘 = 𝜂max + D.3.4 𝑘 (𝜂min − 𝜂max ) 𝐾 −1 for 𝑘 = 0, ..., 𝐾 − 1. Implementation Details of Baselines For baseline methods that do not require additional training on paired data, such as DPG [277], SCG [141], EnKG [356], EKI [143], EKS [111] (with diffusion prior initialization), we follow the implementation provided in InverseBench (Chapter 8). For methods that do require training on paired data, specifically the end-toend U-Net and conditional diffusion model (CDM), we first generate a collection of measurement-target pairs by simulating measurements from the prior training 215 dataset available in InverseBench. To evaluate their in-distribution performance, we retrain the U-Net and CDM for each noise level, which takes around 7-10 hours on a single NVIDIA GH200 GPU. The end-to-end U-Net architecture is adapted from the U-Net used in our diffusion model by removing the time conditioning branch. The measurements are upsampled to the same resolution before being fed into the U-Net. However, it is important to note that observations are not always spatially aligned with the unknown signal in a general setting. Consequently, end-to-end neural networks typically require additional design considerations for different types of observations. The CDM is also adapted from the U-Net architecture of the prior diffusion model. This involved replacing the self-attention module with cross-attention and incorporating a CNN-based observation encoder, following the conditioning mechanism used in Rombach et al. [249]. D.3.5 D.3.5.1 Navier-Stokes Equation Problem Setup We follow the experimental setup in InverseBench (Chapter 8), which we include here for completeness. We consider the 2D Navier-Stokes equation for a viscous, incompressible fluid in vorticity form on a torus, 𝜕𝑡 𝝎(𝒙, 𝑡) + 𝒗(𝒙, 𝑡) · ∇𝒙 𝝎(𝒙, 𝑡) = 𝜈∇2𝒙 𝝎(𝒙, 𝑡) + 𝑓 (𝒙), 𝒙 ∈ (0, 2𝜋) 2 , 𝑡 ∈ (0, 𝑇] ∇𝒙 · 𝒗(𝒙, 𝑡) = 0, 𝒙 ∈ (0, 2𝜋) 2 , 𝑡 ∈ [0, 𝑇] 𝝎(𝒙, 0) = 𝝎0 (𝒙), 𝒙 ∈ (0, 2𝜋) 2 (D.37)   𝑟 ((0, 2𝜋) 2 ; R2 ) for any 𝑟 > 0 is the velocity field, 𝝎 = where 𝒗 ∈ 𝐶 [0, 𝑇]; 𝐻per
2 ∇𝒙 × 𝒗 is the vorticity, 𝝎0 ∈ 𝐿 per (0, 2𝜋) 2 ; R is the initial vorticity, 𝜈 ∈ R+ is the
2 viscosity coefficient, and 𝑓 ∈ 𝐿 per (0, 2𝜋) 2 ; R is the forcing function. The solution operator G is defined as the operator mapping the vorticity from the initial vorticity to the vorticity at time 𝑇, i.e., G : 𝝎0 → 𝝎𝑇 . Numerically, it is realized as a pseudo-spectral solver [127]. This Navier-Stokes equation is a standard benchmark problem widely used in the literature [143, 184, 276]. The forward model is given by 𝒚 = 𝑷𝑮 (𝒘 0 ) + 𝒏, (D.38) where 𝑷 is the sampling operator and 𝑮 (·) is the discretization of G. 216 D.3.5.2 Evaluation Metrics We adopt the following three standard metrics to evaluate the results from different perspectives. Relative ℓ2 Error Suppose 𝒙0 is the ground truth function and b 𝒙 is the predicted function. The relative ℓ2 error measures the norm of the error relative to the norm of the ground truth, defined as ∥b𝒙∥𝒙−𝒙0 ∥02∥ 2 . Continuous Ranked Probability Score (CRPS) The CRPS [118] is a standard probabilistic metric to assess the quality of the entire predicted distribution for inverse problems, which is defined as 1 𝒙−b 𝒙 ′ |, CRPS = E|b 𝒙 − 𝒙0 | − E|b 2 where b 𝒙, b 𝒙 ′ are independent random predictions. Intuitively, it measures the distance between a predicted distribution and the single ground truth 𝒙0 that actually occurred. It is minimized when the ensemble prediction is drawn from the same distribution as the ground truth, i.e., 𝒙 ( 𝑗) ∼ 𝑝(𝒙0 ) for all 𝑗. We consider the multi-dimensional version of CRPS defined in [241]. For an ensemble prediction {𝒙 ( 𝑗) } 𝐽𝑗=1 where 𝒙 ( 𝑗) ∈ R𝑛 , the CRPS for the single ground truth 𝒙0 is given by 𝑛 𝐽 𝐽 ∑︁ 𝐽 ∑︁ 1 1 ∑︁ © 1 ∑︁ ( 𝑗) ª |𝒙 (𝑖) − 𝒙 (𝑖)| − |𝒙 ( 𝑗) (𝑖) − 𝒙 (𝑘) (𝑖)| ® , ­ 0 𝑛 𝑖=1 𝐽 𝑗=1 2𝐽 (𝐽 − 1) 𝑗=1 𝑘=1 « ¬ (D.39) which can be implemented in 𝑂 (𝑛𝐽 log 𝐽) complexity using the equivalent form introduced in [337]. In our experiments, we report the CRPS averaged over all test cases. CRPS = Spread-Skill Ratio (SSR) The spread-skill ratio (SSR) is a simple yet powerful metric for quantifying how well an ensemble prediction’s stated uncertainty (spread) matches its actual error (skill) [106]. Intuitively, if the ensemble distribution truly captures the variability of the ground truth, then ensemble members should be 𝑁 denote statistically indistinguishable from observed outcomes. Formally, let {𝒙𝑖∗ }𝑖=1 a set of observed ground truths. Suppose, for each observed ground truth 𝒙𝑖∗ , we Í have an ensemble prediction {𝒙𝑖, 𝑗 } 𝐽𝑗=1 . Let 𝒙¯ 𝑖 := 1𝐽 𝑗 𝒙𝑖, 𝑗 . The unbiased estimator of SSR can be written as √︄ SSR = spread2 , skill2 (D.40) 217 where 𝑁 𝐽 1 ∑︁ 1 ∑︁ spread = ∥𝒙𝑖, 𝑗 − 𝒙¯ 𝑖 ∥ 22 , 𝑁 𝑖=1 𝐽 − 1 𝑗=1 2 𝑁 1 1 ∑︁ 1 ∑︁ ∥ skill = 𝒙𝑖, 𝑗 − 𝒙𝑖∗ ∥ 22 + spread2 . 𝑁 𝑖=1 𝐽 𝑗 𝐽 (𝐽 − 1) 2 A value of SSR = 1 indicates the perfect calibration. Small SSR means that the ensemble prediction is over-confident, while large SSR indicates that the ensemble prediction is over-cautious. D.4 D.4.1 Appendix for Section 7.6 Theory Notations We consider a linear stochastic process whose forward Kolmogorov equation can be written as 𝜕𝑡 𝜋𝑡 = 𝑸 𝑡 𝜋𝑡 with boundary condition 𝜋𝑡=0 = 𝜋0 , where 𝜋𝑡 ∈ P (X) is a probability measure on X. For the simplicity of notation, we [𝑖, 𝑗] let 𝑸 𝑡 to generate reverse continuous-time Markov chain, which means 𝑸 𝑡 = [ 𝑗,𝑖] 𝒔(𝒙 𝑗 ; 𝑡) 𝒙𝑖 𝑸 uniform where 𝒔(𝒙 𝑗 ; 𝑡) 𝒙𝑖 is the concrete score. Time 𝑡 flows forward as the sampling algorithm progresses, which means that for unconditional generation 𝜋0 is the data distribution and 𝜋𝑇 is the uniform distribution. For two probability mass functions 𝜇 and 𝜋, we define the KL divergence of 𝜇 with respect to 𝜋 as i h 𝜇 (D.41) KL(𝜇∥𝜋) := E𝒙∼𝜇 log (𝒙) . 𝜋 We define the Fisher divergence (or relative Fisher information) of 𝜇 with respect to 𝜋 as   ∑︁ 𝑓 (𝒙 𝑗 ) [ 𝑗,𝑖] FI𝑸 (𝜇∥𝜋) := 𝜋(𝒙𝑖 )𝑸 𝑓 (𝒙 𝑗 ) − 𝑓 (𝒙𝑖 ) − 𝑓 (𝒙𝑖 ) log , (D.42) 𝑓 (𝒙𝑖 ) 𝒙 ,𝒙 ∈X 𝑖 𝑗 where 𝑓 := 𝜇/𝜋. Note that when 𝑸 is irreducible, the Fisher divergence FI𝑸 (𝜇∥𝜋) ≥ 0, and FI𝑸 (𝜇∥𝜋) = 0 if and only if 𝜇 = 𝜋. It is also important to see that this definition of KL divergence for discrete spaces is the same as its continuous version in Section 6.4 and Appendix D.3.1.1. However, the definition of Fisher divergence is significantly different because the gradient is not well-defined in discrete spaces. In continuous state spaces, the Fisher divergence has been used to derive general first-order guarantees for non-log-concave sampling [15]. This line of analysis has been adapted to provide theoretical insights for posterior sampling methods 218 using diffusion models [273, 320]. The formula in Equation (D.42) represents a discrete state space analogue of Fisher divergence. We refer interested readers to [27, 87, 130] for more discussions on this topic and the relation between KL and Fisher divergence in the discrete state space. Our analysis in this paper adapts these techniques and provides general first-order guarantees for posterior sampling with discrete diffusion models. Time Interpolation of SGDD SGDD alternates between likelihood steps and prior steps. Let 𝑡 ∗ > 0 such that 𝜎(𝑡 ∗ ) = 𝜂. We define {𝜋𝜏 } as the distributions at time 𝜏 of the stationary process, and {𝜇𝜏 } as the distributions of the non-stationary process. • In time intervals 𝜏 ∈ [𝑘 (𝑡 ∗ +1) +1, (𝑘 +1)(𝑡 ∗ +1)]. The stationary distribution is initialized with 𝜋 𝑘 (𝑡 ∗ +1)+1 (𝒙) = 𝜋𝜂𝑍 (𝒙). We run a prior step on 𝜇𝜏 with the learned concrete score function for 𝐻 steps, while 𝜋𝜏 evolves in continuous time with the true concrete score function. • In time intervals 𝜏 ∈ [𝑘 (𝑡 ∗ + 1), 𝑘 (𝑡 ∗ + 1) + 1], we run a Metropolis-Hastings sampling algorithm on both 𝜋𝜏 and 𝜇𝜏 . Assumptions Our analysis relies on the following assumptions: (i) Concrete score is well estimated: 𝒔𝜽 ( · ; 𝑡)−𝒔(·; 𝑡) ≤ 𝜖 < 1. 𝒔( · ; 𝑡) ∞ · ; 𝑡) (ii) Smoothness of score function in 𝑡: ∥ 𝒔( · ; 𝑡+Δ𝑡)−𝒔( ∥ ∞ ≤ 𝐿 · Δ𝑡. 𝒔( · ; 𝑡) [𝑖, 𝑗] (iii) Strong irreducibility: 𝑸 𝑡 > 0 for 𝑖 ≠ 𝑗. (𝒙) (iv) Bounded probability ratio: sup𝑡 log 𝜋𝜇𝑡𝑡 (𝒙) ≤ 𝐵. ∞ (v) The entry-wise absolute of the reverse-time transition matrix is bounded: sup𝑡 ∥|𝑸 𝑡 |∥ 1 ≤ 𝑀. D.4.1.1 Lemmas Lemma D.4.1 (Data processing inequality of Metropolis-Hastings). Running MetropolisHastings algorithm on two distributions 𝜋𝜏 and 𝜇𝜏 does not increase their KL divergence, i.e., KL(𝜇 𝑘 (𝑡 ∗ +1)+1 ∥𝜋 𝑘 (𝑡 ∗ +1)+1 ) ≤ KL(𝜇 𝑘 (𝑡 ∗ +1) ∥𝜋 𝑘 (𝑡 ∗ +1) ). (D.43) 219 Lemma D.4.2 (Free-energy-rate-functional-relative-Fisher-information (FIR) inequality (from Theorem 6.2.3. in [129])). Consider two continuous time Markov e𝑡 𝜇𝑡 . Suppose Assumption (iv) holds, then there chains: 𝜕𝑡 𝜋𝑡 = 𝑸 𝑡 𝜋𝑡 and 𝜕𝑡 𝜇𝑡 = 𝑸 exists a constant 𝑐 > 0, such that 1 2 𝜕𝑡 KL(𝜇𝑡 ∥𝜋𝑡 ) ≤ − FI𝑸 𝑡 (𝜇𝑡 ∥𝜋𝑡 ) + L𝑸 𝑡 (𝜇𝑡 , 𝜕𝑡 𝜇𝑡 ), 2 𝑐 (D.44) where L𝑸 (𝜇𝑡 , 𝜕𝑡 𝜇𝑡 ) ≥ 0 is the Lagrangian defined as   ∑︁   [ 𝑗,𝑖]  L𝑸 (𝜇𝑡 , 𝜕𝑡 𝜇𝑡 ) = sup  ⟨𝜑, 𝜕𝑡 𝜇𝑡 ⟩ − 𝜇𝑡 (𝒙𝑖 )𝑸 exp(𝜑(𝒙 𝑗 ) − 𝜑(𝒙𝑖 ))  . 𝜑∈C𝑏 (X)   𝒙𝑖 ,𝒙 𝑗 ∈X   (D.45) Proof Sketch. This lemma follows from Theorem 6.2.3. in [129]. For completeness, we provide a sketch of the proof here. By a direct calculation of derivatives, we have   ∑︁ 𝜇𝑡 (𝒙𝑖 ) ∑︁ 𝜇𝑡 (𝒙𝑖 ) 𝜇𝑡 (𝒙𝑖 ) 𝜕𝑡 KL(𝜇𝑡 ∥𝜋𝑡 ) = 𝜕𝑡 𝜇𝑡 (𝒙𝑖 ) log = − 𝜕𝑡 𝜋𝑡 (𝒙𝑖 ) 𝜕𝑡 𝜇𝑡 (𝒙𝑖 ) log 𝜋𝑡 (𝒙𝑖 ) 𝜋𝑡 (𝒙𝑖 ) 𝜋𝑡 (𝒙𝑖 ) 𝒙𝑖 𝒙𝑖  ∑︁  𝜇𝑡 (𝒙𝑖 ) 𝜇𝑡 (𝒙𝑖 ) [𝑖, 𝑗] [𝑖, 𝑗] e = 𝑸 𝑡 𝜇𝑡 (𝒙 𝑗 ) log − 𝑸 𝜋𝑡 (𝒙 𝑗 ) 𝜋𝑡 (𝒙𝑖 ) 𝜋𝑡 (𝒙𝑖 ) 𝑡 𝒙𝑖 ,𝒙 𝑗   ∑︁ 𝜇𝑡 (𝒙𝑖 ) 𝜇𝑡 (𝒙𝑖 ) 𝜇𝑡 (𝒙𝑖 ) [𝑖, 𝑗] = 𝑸 𝑡 𝜋𝑡 (𝒙 𝑗 ) log − 𝜋𝑡 (𝒙𝑖 ) 𝜋𝑡 (𝒙𝑖 ) 𝜋𝑡 (𝒙𝑖 ) 𝒙𝑖 ,𝒙 𝑗  ∑︁  [𝑖, 𝑗] e[𝑖, 𝑗] 𝜇𝑡 (𝒙 𝑗 ) log 𝜇𝑡 (𝒙𝑖 ) . 𝑸𝑡 − 𝑸 − 𝑡 𝜋𝑡 (𝒙𝑖 ) 𝒙 ,𝒙 Using the fact that equality ∑︁ 𝒙 𝑖 ,𝒙 𝑗 𝑖 𝑗 Í [𝑖, 𝑗] = 0 and the definition in Equation (D.42), we have the 𝒙𝑖 𝑸 𝑡 [𝑖, 𝑗] 𝑸 𝑡 𝜋𝑡 (𝒙 𝑗 )   𝜇𝑡 (𝒙𝑖 ) 𝜇𝑡 (𝒙𝑖 ) 𝜇𝑡 (𝒙𝑖 ) log − = −FI𝑸 𝑡 (𝜇𝑡 ∥𝜋𝑡 ). 𝜋𝑡 (𝒙𝑖 ) 𝜋𝑡 (𝒙𝑖 ) 𝜋𝑡 (𝒙𝑖 ) e𝑡 𝜇𝑡 then leads to Using the relation 𝜕𝑡 𝜇𝑡 = 𝑸 𝑇 𝜇𝑡 𝜕𝑡 KL(𝜇𝑡 ∥𝜋𝑡 ) = −FI𝑸 𝑡 (𝜇𝑡 ∥𝜋𝑡 ) − log (𝑸 𝑡 − 𝜕𝑡 )𝜇𝑡 . 𝜋𝑡 | {z }  error term This formula is similar to Lemma 1 in [333]. (D.46) 220 We define L𝑸 𝑡 (𝜇𝑡 , 𝜕𝑡 𝜇𝑡 ) = sup [⟨𝜑, 𝜕𝑡 𝜇𝑡 ⟩ − (𝑒 −𝜑 𝜇𝑡 )𝑇 𝑸𝑇𝑡 𝑒 𝜑 ] 𝜑∈C𝑏 (X) = sup [⟨𝜑, 𝜕𝑡 𝜇𝑡 − 𝑸 𝑡 𝜇𝑡 ⟩ − 𝜇𝑇𝑡 (𝑒 −𝜑 𝑸𝑇𝑡 𝑒 𝜑 − 𝑸𝑇𝑡 𝜑) ]. | {z } 𝜑∈C𝑏 (X) (D.47) e (𝜇𝑡 ,𝜑) denoted as H By the variational characterization of the Lagrangian, for any continuous, bounded 𝜑, e (𝜇𝑡 , 𝜑). ⟨𝜑, 𝜕𝑡 𝜇𝑡 − 𝑸 𝑡 𝜇𝑡 ⟩ ≤ L𝑸 𝑡 (𝜇𝑡 , 𝜕𝑡 𝜇𝑡 ) + H (D.48) If we choose 𝜑 = log 𝜋𝜇𝑡𝑡 , the inequality above gives a bound to the error term in Equation (D.46). Moreover, it is easy to verify that   𝜇𝑡 e = FI(𝜇𝑡 ∥𝜋𝑡 ). (D.49) H 𝜇𝑡 , log 𝜋𝑡 In fact, when the space X is continuous, choosing 𝜑(𝒙) = log 𝜋𝜇𝑡𝑡 (𝒙) exactly recovers Lemma A.4 in [320]. However, as pointed out in [129], it is necessary to consider a rescaled 𝜑 = 𝜆 log 𝜋𝜇𝑡𝑡 with 𝜆 ∈ (0, 1) to derive a bound in finite space X. Fortunately, according to Lemma 6.2.2. in [129], for any 𝜑 ∈ C𝑏 (X) with ∥𝜑∥ ∞ ≤ 𝐵, there exists a positive constant 𝑐 = 𝑐(𝐵) ∈ (0, 1), such that 2 e (𝜇𝑡 , 𝜆𝜑) ≤ 𝜆 H e (𝜇𝑡 , 𝜑). H 𝑐 (D.50) Plugging in 𝜑 = 𝜆 log 𝜋𝜇𝑡𝑡 in Equation (D.48), we have     𝜇𝑡 𝜇 𝑡 e 𝜇𝑡 , 𝜆 log 𝜆 log , 𝜕𝑡 𝜇𝑡 − 𝑸 𝑡 𝜇𝑡 ≤ L𝑸 𝑡 (𝜇𝑡 , 𝜕𝑡 𝜇𝑡 ) + H 𝜋𝑡 𝜋𝑡   2 𝜆 e 𝜇𝑡 ≤ L𝑸 𝑡 (𝜇𝑡 , 𝜕𝑡 𝜇𝑡 ) + H 𝜇𝑡 , log . 𝑐 𝜋𝑡 (D.51) (D.52) Combining this with Equation (D.46), we get   𝜆 e 𝜇𝑡 1 𝜕𝑡 KL(𝜇𝑡 ∥𝜋𝑡 ) ≤ −FI𝑸 𝑡 (𝜇𝑡 ∥𝜋𝑡 ) + L𝑸 𝑡 (𝜇𝑡 , 𝜕𝑡 𝜇𝑡 ) + H 𝜇𝑡 , log 𝜆 𝑐 𝜋𝑡 𝜆 1 = −(1 − )FI𝑸 𝑡 (𝜇𝑡 ∥𝜋𝑡 ) + L𝑸 𝑡 (𝜇𝑡 , 𝜕𝑡 𝜇𝑡 ). (D.53) 𝑐 𝜆 Finally, choosing 𝜆 = 2𝑐 ∈ (0, 1) recovers the statement of Lemma D.4.2. 221 Remark D.4.3. The Lagrangian L𝑸 (𝜇𝑡 , 𝜕𝑡 𝜇𝑡 ) = 0 if and only if 𝜕𝑡 𝜇𝑡 = 𝑸𝜇𝑡 . Lemma D.4.4. When the learned score function 𝒔𝜽 satisfies Assumption (i), e𝑡 𝜇𝑡 ) ≤ 𝑀𝜖 . L𝑸 𝑡 (𝜇𝑡 , 𝑸 (D.54) Proof. By definition, e𝑡 𝜇𝑡 ) L𝑸 𝑡 (𝜇𝑡 , 𝑸 i ∑︁ h [ 𝑗,𝑖] [ 𝑗,𝑖] 𝜑(𝒙 𝑗 )−𝜑(𝒙 𝑖 ) e = sup 𝜇𝑡 (𝒙𝑖 ) 𝑸 𝑡 𝜑(𝒙 𝑗 ) − 𝜇𝑡 (𝒙𝑖 )𝑸 𝑡 𝑒 𝜑∈C𝑏 (X) 𝒙 ,𝒙 ∈X 𝑖 = 𝑗 ∑︁ h sup e[ 𝑗,𝑖] (𝜑(𝒙 𝑗 ) − 𝜑(𝒙𝑖 )) − 𝜇𝑡 (𝒙𝑖 )𝑸 [ 𝑗,𝑖] 𝑒 𝜑(𝒙 𝑗 )−𝜑(𝒙𝑖 ) 𝜇𝑡 (𝒙𝑖 ) 𝑸 𝑡 𝑡 i 𝜑∈C𝑏 (X) 𝒙 ,𝒙 ∈X 𝑖 𝑗 e𝑡 𝒖 = 0 for any 𝒖) (1𝑇 𝑸 " = ∑︁ sup [ 𝑗,𝑖] 𝜇𝑡 (𝒙𝑖 )𝑸 𝑡 − !# e[ 𝑗,𝑖] 𝑸 𝑡 𝑧 − 𝑒 −𝑧𝑖 𝑗 [ 𝑗,𝑖] 𝑖 𝑗 + 𝑸𝑡 𝜑∈C𝑏 (X) 𝒙 𝑖 ≠𝒙 𝑗 ∑︁ e[𝑖,𝑖] − 𝑸 [𝑖,𝑖] ) 𝜇𝑡 (𝒙𝑖 )( 𝑸 𝑡 𝑡 𝒙𝑖 (𝑧𝑖 𝑗 := 𝜑(𝒙𝑖 ) − 𝜑(𝒙 𝑗 )) " ≤ ∑︁ [ 𝑗,𝑖] 𝜇𝑡 (𝒙𝑖 )𝑸 𝑡 sup − 𝑧 𝑖 𝑗 ∈R 𝒙 𝑖 ≠𝒙 𝑗 e[ 𝑗,𝑖] 𝑸 𝑡 𝑧 − 𝑒 −𝑧𝑖 𝑗 [ 𝑗,𝑖] 𝑖 𝑗 !# +𝜖 𝑸𝑡 ∑︁ 𝜇𝑡 (𝒙𝑖 ) 𝑸 𝑡[𝑖,𝑖] (D.55) 𝒙𝑖 where the inequality is due to swapping supremum and summation, and that [ 𝑗,𝑖] 𝜇𝑡 (𝒙𝑖 )𝑸 𝑡 ≥ 0 for 𝑖 ≠ 𝑗. Consider the function 𝑔(𝑧) = −𝑢𝑧 − 𝑒 −𝑧 . When 𝑢 ≥ 0, this function is maximized e[ 𝑗,𝑖] /𝑸 [ 𝑗,𝑖] = when 𝑧 = − log 𝑢. So, −𝑢𝑧 − 𝑒 −𝑧 ≤ (𝑢 − 1) log 𝑢. Setting 𝑢 = 𝑸 𝑡 𝑡 𝒔𝜽 (𝒙𝑖 ; 𝑡) 𝒙 𝑗 /𝒔(𝒙𝑖 ; 𝑡) 𝒙 𝑗 , we can invoke Assumption (i) to obtain sup − 𝑧𝑖 𝑗 ∈R ! e[ 𝑗,𝑖] 𝑸 𝑡 𝑧 − 𝑒 −𝑧𝑖 𝑗 ≤ (𝑢 − 1) log 𝑢 ≤ |𝑢 − 1| = [ 𝑗,𝑖] 𝑖 𝑗 𝑸𝑡 𝒔𝜽 (𝒙𝑖 ; 𝑡) 𝒙 𝑗 − 𝒔(𝒙𝑖 ; 𝑡) 𝒙 𝑗 𝒔(𝒙𝑖 ; 𝑡) 𝒙 𝑗 ≤𝜖 (D.56) for 𝒔𝜽 (𝒙𝑖 ; 𝑡) 𝒙 𝑗 −𝒔(𝒙 𝑖 ; 𝑡) 𝒙 𝑗 𝒔(𝒙 𝑖 ; 𝑡) 𝒙 𝑗 ≤ 1. Combining with Equation (D.55), we have e𝑡 𝜇𝑡 ) ≤ 𝜖1𝑇 |𝑸 𝑡 |𝜇𝑡 ≤ 𝜖 ∥|𝑸 𝑡 |∥ 1 ∥𝜇𝑡 ∥ 1 ≤ 𝜖 𝑀. L𝑸 𝑡 (𝜇𝑡 , 𝑸 (D.57) 222 Remark D.4.5. The Lagrangian L𝑸 (𝜇𝑡 , 𝜕𝑡 𝜇𝑡 ) can potentially be bounded by the score entropy defined in Definition 3.1 of [198]. To see the this, we calculate e𝑡 𝜇𝑡 ) L𝑸 𝑡 (𝜇𝑡 , 𝑸 " !# ∑︁ ∑︁ e[ 𝑗,𝑖] 𝑸 [ 𝑗,𝑖] 𝑡 −𝑧 𝑖 𝑗 e[𝑖,𝑖] − 𝑸 [𝑖,𝑖] ) = sup 𝜇𝑡 (𝒙𝑖 )𝑸 𝑡 − [ 𝑗,𝑖] 𝑧𝑖 𝑗 − 𝑒 + 𝜇𝑡 (𝒙𝑖 )( 𝑸 𝑡 𝑡 𝜑∈C𝑏 (X) 𝒙 𝑖 ≠𝒙 𝑗 𝑸𝑡 𝒙𝑖 !# " [ 𝑗,𝑖] ∑︁ e (1) ∑︁ 𝑸 [ 𝑗,𝑖] e[𝑖,𝑖] − 𝑸 [𝑖,𝑖] ) 𝜇𝑡 (𝒙𝑖 )( 𝑸 ≤ 𝜇𝑡 (𝒙𝑖 )𝑸 𝑡 sup − 𝑡[ 𝑗,𝑖] 𝑧𝑖 𝑗 − 𝑒 −𝑧𝑖 𝑗 + 𝑡 𝑡 𝑧 𝑖 𝑗 ∈R 𝑸𝑡 𝒙𝑖 𝒙 𝑖 ≠𝒙 𝑗  ∑︁  ∑︁ 𝒔𝜽 (𝒙𝑖 ; 𝑡) 𝒙 𝑗 (2) ∑︁ [ 𝑗,𝑖] 𝒔 𝜽 (𝒙𝑖 ; 𝑡) 𝒙 𝑗 e[ 𝑗,𝑖] − 𝑸 [ 𝑗,𝑖] ) − ≤ log 𝜇𝑡 (𝒙𝑖 ) (𝑸 𝜇𝑡 (𝒙𝑖 )𝑸 𝑡 𝑡 𝑡 𝒔(𝒙 ; 𝑡) 𝒔(𝒙 ; 𝑡) 𝑖 𝒙𝑗 𝑖 𝒙𝑗 𝒙𝑖 𝒙 𝑖 ≠𝒙 𝑗 𝑗≠𝑖   ∑︁  𝒔𝜽 (𝒙𝑖 ; 𝑡) 𝒙 𝑗 (3) fw[𝑖, 𝑗] − 𝒔𝜽 (𝒙𝑖 ; 𝑡) 𝒙 𝑗 + 𝒔(𝒙𝑖 ; 𝑡) 𝒙 𝑗 = 𝜇𝑡 (𝒙𝑖 )𝑸 𝑡 𝒔𝜽 (𝒙𝑖 ; 𝑡) 𝒙 𝑗 log 𝒔(𝒙𝑖 ; 𝑡) 𝒙 𝑗 𝒙𝑖 ≠𝒙 𝑗  ∑︁   𝒔(𝒙𝑖 ; 𝑡) 𝒙 𝑗  (4) fw[𝑖, 𝑗] ≤ E𝒙𝑖 ∼𝜇𝑡 𝐾 𝑸𝑡 𝒔𝜽 (𝒙𝑖 ; 𝑡) 𝒙 𝑗 𝒔𝜽 (𝒙𝑖 ; 𝑡) 𝒙 𝑗 𝒙 ≠𝒙 𝑖 𝑗 where (1) is due to swapping supremum and summation, (2) is due to Equation (D.56) and the property of transition-rate matrices, (3) is due to the definitions of 𝑸 𝑡 and e𝑡 , and (4) is due to the definition of 𝐾 (𝑥) := 𝑥 − log 𝑥 − 1. Note that the last line 𝑸 is the score entropy of 𝒔 with respect to 𝒔𝜽 defined as  ∑︁   𝒔𝜃 (𝒙𝑖 ; 𝑡) 𝒙 𝑗  fw[𝑖, 𝑗] Q𝑡 𝒔(𝒙𝑖 ; 𝑡) 𝒙 𝑗 . SE𝑸 fw (𝒔( · ; 𝑡)||𝒔𝜽 ( · ; 𝑡)) := 𝐾 𝑡 𝒔(𝒙𝑖 ; 𝑡) 𝒙 𝑗 𝒙 ≠𝒙 𝑖 𝑗 e𝑡 , we have Therefore, by swapping 𝜇𝑡 with 𝜋𝑡 and 𝑸 𝑡 with 𝑸 L𝑸e𝑡 (𝜋𝑡 , 𝜕𝑡 𝜋𝑡 ) = L𝑸e𝑡 (𝜋𝑡 , 𝑸 𝑡 𝜋𝑡 ) ≤ E𝒙𝑖 ∼𝜋𝑡 SE𝑸 fw (𝒔𝜽 ( · ; 𝑡)||𝒔( · ; 𝑡)) 𝑡 (D.58) where SE𝑸 fw (𝒔𝜽 ( · ; 𝑡)||𝒔( · ; 𝑡)) is the score entropy of 𝒔𝜽 with respect to 𝒔, which 𝑡 is the loss function for training the score entropy estimator 𝒔𝜽 . This implies that if e𝑡 , we can potentially substitute Assumption (i) we swap 𝜇𝑡 with 𝜋𝑡 and 𝑸 𝑡 with 𝑸 and Assumption (v) with a more interpretable assumption on the score entropy error and obtain a convergence result that explicitly incorporates this error, which would resemble that of discrete DMs for unconditional generation [245]. Nevertheless, we underscore that the right-hand side of Equation (D.58) is an integration over the stationary process 𝜋𝑡 , which may be different from the path on which the DM was trained. This is in fact expected and highlights the difference between analyzing the generative process of discrete DMs versus analyzing the process of using discrete DMs for solving inverse problems. The ideal process in the generative process of 223 discrete DMs is the same as the process for training (only time going in different directions). In contrast, when using discrete DMs as priors for solving inverse problems, they conduct inference on arbitrary distributions that may be different from the training distributions, which is a generalization gap that requires stronger condition on the DMs for convergence. For the purpose of this chapter, we proceed with Lemma D.4.4 to simply the subsequent discretization analysis. D.4.1.2 Proof of Theorem 7.6.1 Proof. We denote the time index in this proof by 𝜏 to align with the time interpolation of 𝐾 iterations of SGDD introduced earlier. Consider discretizing a prior step [0, 𝑡 ∗ ] uniformly into 𝐻 intervals, [ℎ𝛿, (ℎ + 1)𝛿], for ℎ = 0, . . . , 𝐻 − 1, where 𝛿 = 𝑡 ∗ /𝐻. In the ℎ-th interval, we have eℎ𝛿 𝜇𝜏 . 𝜕𝜏 𝜇𝜏 = 𝑸 (D.59) Applying Lemma D.4.2 to the ℎ-th interval for 𝜇𝜏 and 𝜋𝜏 , we get 1 2 eℎ𝛿 𝜇𝜏 ). 𝜕𝜏 KL(𝜇𝜏 ∥𝜋𝜏 ) ≤ − FI𝑸 𝜏 (𝜇𝜏 ∥𝜋𝜏 ) + L𝑸 𝜏 (𝜇𝜏 , 𝑸 2 𝑐 (D.60) Denoting the starting time of the 𝑘-th prior step as 𝑇𝑘 = 𝑘 (𝑡 ∗ + 1) + 1, we then integrate both sides by 𝜏 and obtain  ∫ 𝑇𝑘 +(ℎ+1)𝛿 ∫ 𝑇𝑘 +(ℎ+1)𝛿  4 e𝑇𝑘 +ℎ𝛿 𝜇𝜏 ) d𝜏 FI𝑸𝜏 (𝜇𝜏 ∥𝜋𝜏 )d𝜏 ≤ −2𝜕𝜏 KL(𝜇𝜏 ∥𝜋𝜏 ) + L𝑸 𝜏 (𝜇𝜏 , 𝑸 𝑐 𝑇𝑘 +ℎ𝛿 𝑇𝑘 +ℎ𝛿   = 2 KL(𝜇𝑇𝑘 +ℎ𝛿 ∥𝜋𝑇𝑘 +ℎ𝛿 ) − KL(𝜇𝑇𝑘 +(ℎ+1)𝛿 ∥𝜋𝑇𝑘 +(ℎ+1)𝛿 ) ∫ 4 𝑇𝑘 +(ℎ+1)𝛿 e𝑇𝑘 +ℎ𝛿 𝜇𝜏 )d𝜏. + L𝑸 𝜏 (𝜇𝜏 , 𝑸 (D.61) 𝑐 𝑇𝑘 +ℎ𝛿 Taking a summation over ℎ gives us ∫ 𝑇𝑘 +𝑡 ∗   FI𝑸 𝜏 (𝜇𝜏 ∥𝜋𝜏 )d𝜏 ≤ 2 KL(𝜇𝑇𝑘 ∥𝜋𝑇𝑘 ) − KL(𝜇𝑇𝑘 +𝑡 ∗ ∥𝜋𝑇𝑘 +𝑡 ∗ ) 𝑇𝑘 + 𝐻−1 ∑︁ ∫ 𝑇𝑘 +(ℎ+1)𝛿 ℎ=0 𝑇𝑘 +ℎ𝛿 4 e𝑇𝑘 +ℎ𝛿 𝜇𝜏 )d𝜏. L𝑸 𝜏 (𝜇𝜏 , 𝑸 𝑐 (D.62) Therefore, by Lemma D.4.4, we have ∫ 𝑇𝑘 +(ℎ+1)𝛿 𝑇𝑘 +ℎ𝛿 4 e𝑇𝑘 +ℎ𝛿 𝜇𝜏 )d𝜏 ≤ L𝑸 𝜏 (𝜇𝜏 , 𝑸 𝑐 ∫ 𝛿 0   4 4𝑀 𝜖𝑡 ∗ 𝐿𝑡 ∗ 2 𝑀 (𝜖 + 𝐿𝑠)d𝑠 = + . 𝑐 𝑐 𝐻 2𝐻 2 (D.63) 224 𝑡) where we used the fact that 𝒔𝜽 (·; 𝑡+Δ𝑡)−𝒔(·; ≤ 𝜖 + 𝐿 · Δ𝑡 due to both Assumption 𝒔(·; 𝑡) ∞ (i) and Assumption (ii). It follows that   ∫ 𝑇𝑘 +𝑡 ∗   4𝑀 𝐿𝑡 ∗ 2 ∗ FI𝑸 𝜏 (𝜇𝜏 ∥𝜋𝜏 )d𝜏 ≤ 2 KL(𝜇𝑇𝑘 ∥𝜋𝑇𝑘 ) − KL(𝜇𝑇𝑘 +𝑡 ∗ ∥𝜋𝑇𝑘 +𝑡 ∗ ) + 𝜖𝑡 + . 𝑐 2𝐻 𝑇𝑘 Finally, taking summation over 𝑘 = 0, . . . , 𝐾 − 1 and applying Lemma D.4.1 to each likelihood step, we have   𝐾−1 ∑︁ ∫ 𝑇𝑘 +𝑡 ∗ 4𝑀 𝐿𝐾𝑡 ∗ 2 ∗ FI𝑸 𝜏 (𝜇𝜏 ∥𝜋𝜏 )d𝜏 ≤ 2KL(𝜇0 ∥𝜋0 ) + 𝜖 𝐾𝑡 + . 𝑐 2𝐻 𝑇 𝑘 𝑘=0 Dividing by 𝐾𝑡 ∗ on both sides gives us ∫ 𝑇𝑘 +𝑡 ∗ 𝐾−1 1 ∑︁ 1 2KL(𝜇0 ∥𝜋0 ) 4𝑀𝜖 2𝑀 𝐿𝑡 ∗ FI (𝜇 ∥𝜋 )d𝜏 ≤ + + . 𝑸𝜏 𝜏 𝜏 𝐾 𝑘=0 𝑡 ∗ 𝑇𝑘 𝐾𝑡 ∗ 𝑐 𝑐𝐻 D.4.1.3 (D.64) Potential Function of Split Gibbs Samplers As defined in Equation (7.21), the Split Gibbs Sampler draws samples from the augmented distribution 𝜋(𝒙, 𝒛; 𝜂) ∝ exp(− 𝑓 (𝒛; 𝒚) − 𝑔(𝒙) − 𝐷 (𝒙, 𝒛; 𝜂)). The key requirement of split Gibbs samplers is that the potential function 𝐷 (𝒙, 𝒛; 𝜂) satisfies lim+ 𝐷 (𝒙, 𝒛; 𝜂) = ∞, ∀𝒙 ≠ 𝒛. (D.65) 𝜂→0 Or more precisely, lim+ ∫ 𝜂→0 exp(−𝐷 (𝒙, 𝒛; 𝜂)) = 𝛿 𝒙 (𝒛). exp(−𝐷 (𝒙, 𝒛; 𝜂))d𝒛 (D.66) Therefore, ∫ 𝑋 lim 𝜋 (𝒙; 𝜂) ∝ lim+ 𝑝(𝒙) 𝑝( 𝒚 | 𝒛) exp(−𝐷 (𝒙, 𝒛; 𝜂))d𝒛 𝜂→0 ∫ = 𝑝(𝒙) 𝑝( 𝒚 | 𝒛)𝛿(𝒙 − 𝒛)d𝒛 = 𝑝(𝒙) 𝑝(𝒚 | 𝒙). 𝜂→0+ (D.67) Also, the similar derivation holds for 𝜋 𝑍 (𝒛; 𝜂). This result has also been shown in [296]. Combining this with Theorem 7.6.1, SGDD is guaranteed to sample from the true posterior distribution when 𝜂 goes to zero. 225 D.4.2 Experimental Details D.4.2.1 Pre-Trained Models We learn prior distributions for each dataset using SEDD [198] discrete diffusion models. We use the SEDD small architecture with around 90M parameters for all experiments, and the models are trained with AdamW [197] with batch size 32 and a learning rate of 3 × 10−4 . D.4.2.2 Baseline Methods DPS DPS [64] is designed to solve general inverse problems with a pre-trained (continuous) diffusion model. It performs posterior sampling from 𝑝(𝒙 | 𝒚) by modifying the reverse SDE √︁ d𝒙𝑡 = −2𝜎 ¤ 𝑡 𝜎𝑡 ∇𝒙𝑡 log 𝑝(𝒙𝑡 | 𝒚)d𝑡 + 2𝜎 ¤ 𝑡 𝜎𝑡 d𝑡 (D.68) √︁ = −2𝜎 ¤ 𝑡 𝜎𝑡 (∇𝒙𝑡 log 𝑝(𝒙𝑡 ; 𝜎𝑡 )) + ∇𝒙𝑡 log 𝑝( 𝒚 | 𝒙𝑡 ))d𝑡 + 2𝜎 ¤ 𝑡 𝜎𝑡 d𝑡. (D.69) To estimate the intractable guidance term ∇𝒙𝑡 log 𝑝( 𝒚 | 𝒙𝑡 )d𝑡, DPS proposes to approximate it with 𝑝(𝒚 | 𝒙𝑡 ) ≈ 𝑝( 𝒚 | E[𝒙0 |𝒙𝑡 ]). The guidance term is thus approximated by ∇𝒙𝑡 log 𝑝( 𝒚|𝒙𝑡 ) ≈ −∇𝒙𝑡 ∥A (b 𝒙0 (𝒙𝑡 ; 𝜎𝑡 )) − 𝒚∥ 22 2𝜎𝒚2 , (D.70) where b 𝒙0 (𝒙𝑡 ; 𝜎𝑡 ) is a one-step approximation of E[𝒙0 |𝒙𝑡 ] using the pre-trained diffusion model, and the measurement is assumed to be 𝒚 = A (𝒙) + 𝒏 with 𝒏 ∼ N (0, 𝜎𝒚2 𝑰). However, DPS is not directly applicable to inverse problems in discretestate spaces since propagating gradients through A and 𝐷 𝜃 in Equation (D.70) is impossible. Therefore, we consider the counterpart of DPS in discrete spaces. We modify the continuous-time Markov chain of the discrete diffusion model by 𝒚 𝜕𝑡 𝑝𝑇−𝑡 = 𝑸𝑇−𝑡 𝑝𝑇−𝑡 , (D.71) in which 𝒚 [𝑖, 𝑗] 𝑸𝑇−𝑡 = 𝑝𝑇−𝑡 (𝒙𝑖 |𝒚) 𝑝𝑇−𝑡 (𝒙𝑖 ) 𝑝𝑇−𝑡 ( 𝒚|𝒙𝑖 ) [𝑖, 𝑗] 𝑝𝑇−𝑡 (𝒚|𝒙𝑖 ) = = 𝑸𝑇−𝑡 · . 𝑝𝑇−𝑡 (𝒙 𝑗 | 𝒚) 𝑝𝑇−𝑡 (𝒙 𝑗 ) 𝑝𝑇−𝑡 (𝒚|𝒙 𝑗 ) 𝑝𝑇−𝑡 ( 𝒚|𝒙 𝑗 ) (D.72) Similar ideas are appliedh to classifier i guidance for discrete diffusion models [225], 𝒚 𝑝 𝑇 −𝑡 (𝒚|𝒙𝑖 ) where the matrix 𝑹𝑡 = 𝑝𝑇 −𝑡 (𝒚|𝒙 𝑗 ) is called a guidance rate matrix. We compute 𝑖, 𝑗 226 𝒚 𝒚|𝒙 𝑖 ) 𝑹𝑡 at 𝒙 𝑗 -column by enumerating every neighboring 𝒙𝑖 and calculating 𝑝𝑝𝑡𝑡 ((𝒚|𝒙 for 𝑗) each 𝒙𝑖 . The discrete version of DPS can be summarized by 𝒚 𝜕𝑡 𝑝𝑇−𝑡 = 𝑸 𝑡 𝑹𝑡 𝑝𝑇−𝑡 . (D.73) However, the discrete version of DPS is very time-consuming, especially when the vocabulary size is large, since it enumerates (𝑁 − 1) × 𝑛 number of neighboring 𝒙 𝒚 when computing 𝑹𝑡 . We find it slow for the discrete image reconstruction problems on binary MNIST where 𝑁 = 2. SVDD SVDD [182] aims to sample from the distribution 𝑝 𝛽 (𝒙0 ) ∝ 𝑝(𝒙 0 ) exp(𝛽𝑟 (𝒙0 )), which is equivalent to the regularized MDP problem: 𝑝 𝛽 (𝒙0 ) = arg max E𝒙0 ∼𝜋 𝑟 (𝒙 0 ) − KL(𝜋∥ 𝑝)/𝛽. (D.74) 𝜋 They calculate the soft value function as 𝑣 𝑡 (𝒙𝑡 ) = log E𝒙0 ∼𝑝(𝒙0 |𝒙𝑡 ) [exp(𝛽𝑟 (𝒙0 ))]/𝛽 (D.75) and propose to sample from the optimal policy 𝑝★𝑡 (𝒙𝑡 | 𝒙𝑡+1 ) ∝ 𝑝 𝑡 (𝒙𝑡 | 𝒙𝑡+1 ) exp(𝛽𝑣 𝑡 (𝒙𝑡 )). (D.76) In time step 𝑡, SVDD samples a batch of 𝑀 particles from the unconditional distribution 𝑝 𝑡 (𝒙𝑡 | 𝒙𝑡+1 ), and conduct importance sampling according to exp(𝛽𝑣 𝑡 (𝒙𝑡 )). Although [182] is initially designed for guided diffusion generation, it also applies to solving inverse problems by carefully choosing reward functions. We consider 𝑟 (𝒙) = −∥A (𝒙)−𝒚∥ 0 /𝜎𝒚 and 𝛽 = 1, so that it samples from the posterior distribution 𝑝(𝒙 | 𝒚). As recommended in [182], we choose 𝛽 = ∞ (𝛼 = 0 in their notation) in practice, so the importance sampling reduces to finding the particle with the maximal value in each iteration. We use SVDD-PM, a training-free method provided by [182] in our experiments. It approximates the value function by 𝑣 𝑡 (𝒙𝑡 ) = 𝑟 ((𝒙𝑡 ; 𝜎𝑡 )), where (𝒙𝑡 ; 𝜎𝑡 ) is an approximation of E[𝒙0 |𝒙𝑡 ]. In practice, we find that approximating (𝒙𝑡 ; 𝜎𝑡 ) by Monte Carlo sampling with a few-step Euler sampler achieves slightly better results. We use 3 Monte Carlo samples to estimate 𝑣 𝑡 (𝒙𝑡 ) in our experiments. SMC Sequential Monte Carlo (SMC) methods evolve multiple particles to approximate a series of distributions, eventually converging to the target distribution. Specifically, in our experiments, we implement the SMC method to sample from 227 𝑝(𝒙𝑡 | 𝒚) for 𝑡 = 𝑇, 𝑇 − 1 . . . , 0, using the unconditional discrete diffusion sampler 𝑝(𝒙𝑡 | 𝒙𝑡+1 ) as the proposal function. ( 𝑗) We maintain a batch of 𝐽 = 20 particles {𝒙 ( 𝑗) }. At time 𝑡, we sample 𝒙𝑡 ∼ ( 𝑗) 𝑝(𝒙𝑡 | 𝒙𝑡+1 ) by the pre-trained discrete diffusion model and estimate the likelihood ( 𝑗) 𝑝( 𝒚 | 𝒙𝑡 ) = E𝒙 ∼𝑝 𝒙 |𝒙 ( 𝑗 )  𝑝( 𝒚 | 𝒙0 ) by Monte Carlo sampling. We then resample 0 0 𝑡 ( 𝑗) ( 𝑗) the particles {𝒙𝑡 } according to their weights 𝑤 𝑡 ( 𝑗) ( 𝑗) = 𝑝(𝒚|𝒙𝑡 )/𝑝( 𝒚|𝒙𝑡+1 ). In 𝛽 practice, we find that we have to carefully tune a hyperparameter 𝛽, where 𝑤 𝑡 . Otherwise, the resampling step can easily degenerate to finding arg max or uniform random sampling. D.4.2.3 Hyperparameters 𝑘 1− 𝑘 𝐾 −1 We use an annealing noise schedule of 𝜂 𝑘 = 𝜂min 𝜂max𝐾 −1 with 𝜂min = 10−4 and 𝜂max = 20. We run SGDD for 𝐾 iterations. In each likelihood sampling step, we run Metropolis-Hastings for 𝑇 steps, while in each prior sampling step, we run a few-step Euler discrete diffusion sampler with 𝐻 steps. The hyperparameters used for each experiment are listed in Table D.7. We also include the dimension of data spaces X 𝑛 for each experiment in Table D.7, where |X| = 𝑁. Table D.7: Hyperparameters of SGDD used in each experiment. MNIST XOR MNIST AND Metropolis-Hastings 𝑇 SGDD iterations 𝐾 Euler sampler 𝐻 2000 50 20 5000 100 20 Sequence length 𝑛 Vocab size 𝑁 1024 2 1024 2 228 Appendix E APPENDIX FOR CHAPTER 8 E.1 Tables of Main Results We present the main experimental results in Tables E.1, E.2, E.3, E.4, E.5, and E.6. Bold indicates the best across all PnPDP methods. Table E.1: Results on linear inverse scattering. PSNR and SSIM of different algorithms on linear inverse scattering. Noise level 𝜎𝒚 = 10−4 . Number of receivers 360 180 60 Methods PSNR ↑ SSIM ↑ Meas err (%) ↓ PSNR ↑ SSIM ↑ Meas err (%) ↓ PSNR ↑ SSIM ↑ Meas err (%) ↓ Traditional FISTA-TV 32.126 (2.139) 0.979 (0.009) 1.23 (0.25) 26.523 (2.678) 0.914 (0.040) 2.65 (0.30) 20.938 (2.513) 0.709 (0.103) 6.05 (0.65) PnPDP DDRM DDNM ΠGDM DPS LGD DiffPIR PnP-DM DAPS RED-diff FPS MCGdiff 32.598 (1.825) 36.381 (1.098) 27.925 (3.211) 32.061 (2.163) 27.901 (2.346) 34.241 (2.310) 33.914 (2.054) 34.641 (1.693) 36.556 (2.292) 33.242 (1.602) 30.937 (1.964) 0.929 (0.012) 0.935 (0.017) 0.889 (0.072) 0.846 (0.127) 0.812 (0.037) 0.988 (0.006) 0.988 (0.006) 0.957 (0.006) 0.981 (0.005) 0.870 (0.026) 0.751 (0.029) 1.04 (0.26) 0.78 (0.22) 2.74 (1.23) 4.35 (1.19) 1.17 (0.20) 1.11 (0.24) 1.21 (0.25) 1.03 (0.25) 0.89 (0.23) 0.70 (0.01) 0.70 (0.01) 28.080 (1.516) 35.024 (0.993) 26.412 (3.430) 31.798 (2.163) 27.837 (2.337) 34.010 (2.269) 31.817 (2.073) 33.160 (1.704) 35.411 (2.166) 29.624 (1.651) 28.057 (1.672) 0.890 (0.019) 0.895 (0.027) 0.816 (0.114) 0.862 (0.123) 0.803 (0.034) 0.987 (0.006) 0.981 (0.008) 0.944 (0.009) 0.984 (0.004) 0.710 (0.040) 0.631 (0.042) 1.57 (0.39) 0.58 (0.16) 3.66 (1.79) 4.28 (1.20) 1.06 (0.16) 1.04 (0.23) 1.42 (0.26) 1.11 (0.25) 0.87 (0.21) 0.37 (0.01) 0.38 (0.01) 20.436 (1.210) 29.235 (3.376) 20.074 (2.608) 27.372 (3.415) 20.491 (3.031) 26.321 (3.272) 24.715 (2.874) 25.875 (3.110) 27.072 (3.330) 21.323 (1.445) 21.004 (1.571) 0.545 (0.037) 0.917 (0.022) 0.540 (0.198) 0.813 (0.133) 0.552 (0.077) 0.918 (0.028) 0.909 (0.046) 0.885 (0.030) 0.935 (0.037) 0.460 (0.030) 0.445 (0.028) 3.04 (0.92) 0.28 (0.07) 6.90 (3.38) 4.53 (1.31) 1.45 (0.68) 1.27 (0.23) 2.20 (0.34) 1.51 (0.25) 1.18 (0.23) 0.15 (0.02) 0.21 (0.06) Table E.2: Results on compressed sensing MRI. Mean and standard deviation are reported over 94 test cases. ×4 Subsampling ratio Measurement type Methods PSNR ↑ SSIM ↑ Raw Data Misfit ↓ Traditional Wavelet+ℓ1 TV 29.45 (1.776) 0.690 (0.121) 27.03 (1.635) 0.518 (0.123) 0.306 (0.049) 5.748 (1.283) End-to-end Residual U-Net E2E-VarNet 32.27 (1.810) 0.808 (0.080) 33.40 (2.097) 0.836 (0.079) – – PnPDP CSGM ScoreMRI RED-diff DiffPIR DPS DAPS PnP-DM 28.78 (6.173) 0.710 (0.147) 25.97 (1.681) 0.468 (0.087) 29.36 (7.710) 0.733 (0.131) 28.31 (1.598) 0.632 (0.107) 26.13 (4.247) 0.620 (0.105) 31.48 (1.988) 0.762 (0.089) 31.80 (3.473) 0.780 (0.096) E.1.1 ×8 Simulated (noiseless) PSNR ↑ SSIM ↑ Simulated (noiseless) Data Misfit ↓ PSNR ↑ SSIM ↑ Raw Data Misfit ↓ PSNR ↑ SSIM ↑ Data Misfit ↓ 26.47 (1.508) 0.598 (0.122) 31.601 (15.286) 25.97 (1.761) 0.575 (0.105) 0.318 (0.042) 24.08 (1.602) 0.511 (0.106) 22.362 (10.733) 26.22 (1.578) 0.509 (0.123) 32.269 (15.414) 24.12 (1.900) 0.432 (1.112) 5.087 (1.049) 23.70 (1.857) 0.427 (0.112) 23.048 (10.854) 31.70 (1.970) 31.71 (2.540) 0.785 (0.095) 0.756 (0.102) – – 1.518 (0.433) 25.17 (6.246) 0.582 (0.167) 31.642 (15.382) 10.828 (1.731) 25.60 (1.618) 0.463 (0.086) 33.697 (15.209) 0.509 0.077) 28.71 (2.755) 0.626 (0.126) 31.591 (15.368) 10.545 (2.466) 27.60 (1.470) 0.624 (0.111) 34.015 (15.522) 9.900 (2.925) 25.83 (2.197) 0.548 (0.116) 35.095 (15.967) 1.566 (0.390) 28.61 (2.197) 0.689 (0.102) 31.115 (15.497) 4.701 (0.675) 27.62 (3.425) 0.679 (0.117) 32.261 (15.169) 29.75 (1.675) 30.67 (1.761) 0.750 (0.088) 0.769 (0.085) — — 29.36 (1.746) 30.45 (1.940) 0.733 (0.100) 0.736 (0.103) – – 26.15 (6.383) 25.01 (1.526) 26.76 (6.696) 26.78 (1.556) 20.82 (4.777) 29.01 (1.712) 29.33 (3.081) 0.625 (0.158) 0.405 (0.079) 0.647 (0.124) 0.588 (0.113) 0.536 (0.111) 0.681 (0.098) 0.704 (0.105) 1.142 (1.078) 8.360 (1.381) 0.485 (0.068) 7.787 (1.741) 6.737 (1.928) 1.280 (0.301) 3.421 (0.504) 21.17 (8.314) 24.74 (1.481) 27.33 (2.441) 26.26 (1.458) 23.00 (3.205) 27.10 (2.034) 25.28 (3.102) 0.425 (0.192) 0.403 (0.080) 0.563 (0.117) 0.590 (0.113) 0.507 (0.109) 0.629 (0.107) 0.607 (0.117) 22.088 (10.740) 24.028 (10.663) 22.336 (10.838) 24.208 (10.922) 24.842 (11.263) 22.729 (10.926) 22.879 (10.712) Extended Evaluation of CS-MRI For compressed sensing MRI, achieving good performance on distortion metrics such as PSNR and SSIM is not always a sufficient signal for high-quality reconstruction, as hallucinations might lead to wrong diagnoses. We quantify the degree of hallucination by employing a pathology detector on the reconstructed images of different methods. Specifically, we finetune a medium-size YOLOv11 model [159] on a training set of fully sampled images with the fastMRI+ pathology annotations [354] (22 classes in total). We calculate the mAP50 metric over the reconstructed results 229 Table E.3: Generalization results on compressed sensing MRI with ×4 acceleration and raw measurements. Mean and standard deviation are reported over 94 test cases. Vertical → Horizontal Generalization Knee → Brain ×4 → ×8 Methods PSNR ↑ SSIM ↑ Data Misfit ↓ PSNR ↑ SSIM ↑ Data Misfit ↓ PSNR ↑ SSIM ↑ Data Misfit ↓ Traditional Wavelet+ℓ1 TV 27.75 (1.683) 28.18 (1.777) 0.627 (0.133) 0.533 (0.138) 31.744 (15.362) 32.311 (15.487) 25.96 (1.253) 25.56 (1.302) 0.747 (0.026) 0.686 (0.049) 7.986 (0.965) 8.396 (0.990) 24.08 (1.602) 23.70 (1.857) 0.511 (0.106) 0.427 (0.112) 22.362 (10.733) 23.048 (10.854) End-to-end Residual U-Net E2E-VarNet 22.06 (1.682) 22.13 (2.925) 0.603 (0.049) 0.543 (0.103) – – 30.07 (1.364) 31.97 (1.452) 0.881 (0.019) 0.857 (0.038) – – 23.93 (2.176) 24.59 (2.012) 0.610 (0.064) 0.637 (0.069) – – PnPDP CSGM ScoreMRI RED-diff DiffPIR DPS DAPS PnP-DM 26.56 (3.647) 25.60 (1.647) 28.95 (2.480) 27.93 (1.502) 26.77 (1.546) 28.78 (2.209) 27.93 (3.444) 0.629 (0.129) 0.473 (0.091) 0.628 (0.126) 0.637 (0.113) 0.571 (0.117) 0.696 (0.105) 0.689 (0.121) 31.866 (15.479) 33.707 (15.274) 31.740 (15.421) 34.188 (15.479) 35.233 (16.006) 32.198 (15.538) 32.391 (15.235) 27.19 (7.521) 28.52 (0.885) 30.61 (0.982) 27.75 (0.854) 26.77 (1.137) 29.29 (0.911) 29.96 (0.984) 0.779 (0.189) 0.674 (0.045) 0.811 (0.048) 0.823 (0.026) 0.738 (0.031) 0.882 (0.025) 0.882 (0.028) 7.779 (1.043) 9.472 (0.948) 7.750 (0.996) 10.972 (1.016) 10.806 (1.159) 8.255 (0.986) 8.789 (0.978) 21.17 (8.314) 24.74 (1.481) 27.33 (2.441) 26.26 (1.458) 23.00 (3.205 ) 27.10 (2.034) 25.28 (3.102) 0.425 (0.192) 0.403 (0.080) 0.563 (0.117) 0.590 (0.113) 0.507 (0.109) 0.629 (0.107) 0.607 (0.117) 22.088 (10.740) 24.028 (10.663) 22.336 (10.838) 24.208 (10.922) 24.842 (11.263) 22.729 (10.926) 22.879 (10.712) Table E.4: Results on black hole imaging. PSNR and Chi-squared of different algorithms on black hole imaging. Gain and phase noise and thermal noise are added based on the EHT library. Observation time ratio 3% 10% 100% PSNR ↑ Blur PSNR ↑ 2 𝝌˜ cp 2 𝝌˜ logca PSNR ↑ Blur PSNR ↑ 2 𝝌˜ cp Traditional SMILI EHT-Imaging 18.51 (1.39) 21.72 (3.39) 23.08 (2.12) 25.66 (5.04) 1.478 (0.428) 1.507 (0.485) 4.348 (3.827) 1.695 (0.539) 20.85 (2.90) 22.67 (3.46) 25.24 (3.86) 26.66 (3.93) 1.209 (0.169) 1.166 (0.156) PnPDP DPS LGD RED-diff PnPDM DAPS DiffPIR 24.20 (3.72) 22.51 (3.76) 20.74 (2.62) 24.25 (3.45) 23.54 (3.28) 24.12 (3.25) 30.83 (5.58) 28.50 (5.49) 26.10 (3.35) 30.49 (4.93) 29.48 (4.88) 30.45 (4.88) 8.024 (24.336) 5.007 (5.750) 24.36 (3.72) 15.825 (16.838) 12.862 (12.663) 22.08 (3.75) 6.713 (6.925) 9.128 (19.052) 22.53 (3.02) 2.201 (1.352) 1.668 (0.551) 24.57 (3.47) 3.647 (3.287) 2.329 (1.354) 23.99 (3.56) 14.085 (14.105) 10.545 (8.860) 23.84 (3.59) 30.79 (5.75) 27.48 (5.09) 27.67 (4.53) 30.80 (5.22) 30.10 (5.13) 30.04 (5.03) 13.052 (43.087) 10.775 (21.684) 2.488 (2.925) 1.433 (0.417) 1.545 (0.705) 5.374 (3.733) Methods 2 𝝌˜ logca Blur PSNR ↑ 2 𝝌˜ cp 2 𝝌˜ logca 21.788 (12.491) 22.67 (3.13) 1.240 (0.205) 24.28 (3.63) 27.79 (4.02) 28.57 (4.52) 1.878 (0.952) 1.251 (0.250) 17.612 (10.299) 1.259 (0.316) 6.614 (26.789) 25.86 (3.90) 13.375 (56.397) 21.22 (3.64) 4.916 (13.221) 23.77 (4.13) 1.336 (0.478) 26.07 (3.70) 2.253 (9.903) 25.60 (3.64) 5.205 (5.556) 25.01 (4.64) 32.94 (6.19) 26.06 (4.98) 29.13 (6.22) 32.88 (6.02) 32.78 (5.68) 31.86 (6.56) 8.759 (37.784) 13.239 (17.231) 1.853 (0.938) 1.311 (0.195) 1.300 (0.324) 3.271 (1.623) 5.456 (24.185) 13.233 (39.107) 2.050 (2.361) 1.199 (0.221) 1.229 (0.532) 2.970 (1.202) PSNR ↑ Table E.5: Results on FWI. Mean and standard deviation are reported over 10 test cases. †: initialized from data blurred by Gaussian filters with 𝜎 = 20. ∗: one test case is excluded from the results due to numerical instability. Methods Relative ℓ2 ↓ PSNR ↑ SSIM ↑ Traditional Adam Adam† LBFGS† 0.333 (0.086) 0.089 (0.021) 0.070 (0.023) 9.968 (2.083) 0.305 (0.120) 115.14 (52.10) 21.273 (2.045) 0.679 (0.073) 15.89 (10.16) 23.398 (2.749) 0.704 (0.077) 9.18 (6.47) PnPDP DPS LGD DiffPIR DAPS† PnP-DM REDDiff 0.250 (0.154) 0.244 (0.024) 0.204 (0.129) 0.201 (0.103) 0.259 (0.075) 0.319 (0.102) 14.111 (6.820) 12.288 (0.889) 16.113 (6.962) 14.914 (4.184) 11.983 (2.269) 10.372 (2.650) 0.491 (0.161) 0.341 (0.047) 0.554 (0.191) 0.321 (0.067) 0.431 (0.073) 0.280 (0.108) Data Misfit ↓ 155.08 (92.17) 258.47 (26.40) 88.53 (56.91) 111.13 (71.33) 308.84 (26.34) 94.67 (41.33) on 14 selected volumes with severe knee pathologies, which includes 171 test images in total. For each method, we report the Precision, Recall, and mAP50 metrics for detection, and PSNR, SSIM, and Data Misfit for reconstruction, as shown in Table E.8. We also provide the rankings based on mAP50 and PSNR. Overall, the two rankings are correlated, which means that better pixel-wise accuracy indeed leads to a more accurate diagnosis. However, there are a few algorithms for which 230 Table E.6: Results on Navier-Stokes equation. Relative ℓ2 error of different algorithms on 2D Navier-Stokes inverse problem, reported over 10 test cases. ∗: one or two test cases are excluded from the results due to numerical instability. ×2 Subsampling ratio Measurement noise ×4 ×8 𝜎 = 0.0 𝜎 = 1.0 𝜎 = 2.0 𝜎 = 0.0 𝜎 = 1.0 𝜎 = 2.0 𝜎 = 0.0 𝜎 = 1.0 𝜎 = 2.0 Traditional EKI 0.577 (0.138) 0.609 (0.119) 0.673 (0.107) 0.579 (0.145) 0.669 (0.131) 0.805 (0.112) 0.852 (0.167) 0.940(0.115) 1.116(0.090) PnPDP DPS-fGSG DPS-cGSG DPG SCG EnKG 1.687 (0.156) 1.612 (0.173) 2.203* (0.314) 2.117 (0.295) 0.325 (0.188) 0.408* (0.173) 0.908 (0.600) 0.928 (0.557) 0.120 (0.085) 0.191 (0.057) 1.454 (0.154) 1.203* (0.122) 1.746 (0.191) 1.175* (0.079) 0.466 (0.171) 0.322 (0.200) 0.966 (0.546) 0.869 (0.513) 0.294 (0.061) 0.115 (0.064) 1.209* (0.116) 1.133* (0.095) 0.361 (0.187) 0.926 (0.546) 0.271 (0.053) 1.200* (0.100) 1.114* (0.144) 0.454 (0.207) 0.929 (0.505) 0.522 (0.136) 1.246* (0.108) 1.186* (0.117) 0.596 (0.301) 1.260 (0.135) 0.287 (0.273) 1.221* (0.082) 1.260 (0.117) 1.204* (0.115) 1.218 (0.113) 0.591 (0.262) 0.846 (0.251) 1.284 (0.117) 1.347 (0.141) 0.546 (0.212) 0.773 (0.170) Table E.7: Table of metrics we use to capture the computation complexity of each algorithm. Metric Description # Fwdtotal # DMtotal # Fwd Gradtotal # DM Gradtotal Costtotal total forward model evaluations total diffusion model evaluations total forward model gradient evaluations total diffusion model gradient evaluations total runtime # Fwdseq # DMseq # Fwd Gradseq # DM Gradseq Costseq sequential forward model evaluations sequential diffusion model evaluations sequential forward model gradient evaluations sequential diffusion model gradient evaluations sequential runtime Table E.8: Diagnostic performance of compressed sensing MRI reconstructions. Methods Precision Traditional Wavelet+ℓ1 TV Recall mAP50 mAP50 Ranking PSNR SSIM Data Misfit 0.532 0.447 0.332 0.251 0.385 0.263 9 11 28.16 (1.724) 0.685 (0.064) 23.501 (10.475) 28.31 (1.834) 0.662 (0.079) 24.182 (10.613) 8 7 End-to-End Residual U-Net E2E-VarNet 0.482 0.610 0.462 0.514 0.439 0.500 8 1 31.62 (1.635) 0.803 (0.050) 32.25 (1.901) 0.805 (0.056) 2 1 PnPDP CSGM ScoreMRI RED-diff DiffPIR DPS DAPS PnP-DM 0.501 0.412 0.478 0.536 0.346 0.514 0.527 0.528 0.554 0.468 0.484 0.380 0.556 0.579 0.454 0.470 0.448 0.496 0.362 0.480 0.500 6 5 7 3 10 4 1 27.34 (2.770) 26.86 (2.583) 31.56 (2.337) 28.41 (1.403) 26.49 (1.550) 30.15 (1.429) 29.85 (2.934) Fully sampled 0.573 0.581 0.535 – – – – 0.673 (0.082) 23.483 (10.651) 0.547 (0.092) 25.677 (10.491) 0.764 (0.080) 23.406 (10.571) 0.632 (0.061) 26.376 (10.555) 0.540 (0.067) 27.603 (11.127) 0.725 (0.053) 23.978 (10.630) 0.730 (0.056) 24.324 (10.413) – 23.721 (10.824) PSNR Ranking 9 10 3 6 11 4 5 – the two rankings disagree: Residual U-Net, Score MRI, and RED-diff. The best methods for pathology detection are E2E-VarNet and PnP-DM. 231 Linear inverse scattering Magnetic resonance imaging Black hole imaging 10 2 Full waveform inversion 100 Navier-Stokes equation Fwd DM Fwd Grad DM Grad Run Time (s) 10 2 6 × 10 3 10 2 100 10 1 4 × 10 3 10 3 3 × 10 3 10 1 10 3 2 × 10 3 ad Gr dG rad DM Fw d DM Fw ad Gr dG rad DM Fw d DM Fw ad Gr dG rad Fw DM d DM Fw ad Gr dG rad DM Fw d DM Fw ad Gr dG rad Fw DM d Fw DM 10 2 Figure E.1: Computational characteristics of each forward model. Fwd: runtime of a single forward model evaluation tested on a single A100 GPU. DM: runtime of a single diffusion model evaluation. Fwd Grad: runtime of a single forward model gradient evaluation. DM Grad: runtime of a single diffusion model gradient evaluation. Note that the inverse problem of the Navier-Stokes equation only permits black-box access to the forward model, so its Fwd Grad has no value. E.2 E.2.1 Inverse Problem Details Linear Inverse Scattering Problem Details Consider a 2D object with permittivity distribution 𝜖 (𝒓) in a bounded sample plane Ω ∈ R2 , which is immersed in the background medium with permittivity 𝜖 𝑏 . The permittivity contrast is given by Δ𝜖 (𝒓) = 𝜖 (𝒓)−𝜖 𝑏 . At each time, the object is illuminated by an incident light field 𝒇in (𝒓) emitted by one of 𝑁 > 0 transmitters, and the scattered light field 𝒇sc (𝒓) is measured by 𝑀 > 0 receivers. We adopt the experimental setup in [274] where the transmitters and receivers are arranged along a circle Γ ∈ R2 that surrounds the object. Here, 𝒓 := (𝑥, 𝑦) denotes the spatial coordinates. Under the first Born approximation [315], the interaction between the light and the object is governed by the following equation ∫ 𝒇tot (𝒓) = 𝒇in (𝒓) + 𝑔(𝒓 − 𝒓 ′) 𝑠(𝒓 ′) 𝒇in (𝒓 ′) 𝑑 𝒓 ′, 𝒓 ∈ Ω, (E.1) Ω 1 2 𝑘 Δ𝝐 (𝒓) is the scattering potential. where 𝒇tot (𝒓) is the total light field, and 𝑠(𝒓) = 4𝜋 Here, 𝑘 = 2𝜋/𝜆 is the wavenumber in free space, and 𝜆 is the wavelength of the illumination. In the 2D space, the Green’s function is given by 𝑔(𝒓 − 𝒓 ′) = 𝑖 (1) 𝐻 (𝑘 𝑏 ∥ 𝒓 − 𝒓 ′ ∥ 2 ) 4 0 (E.2) √ where 𝑘 𝑏 = 𝜖 𝑏 𝑘 is the wavenumber of the background medium, and 𝐻0(1) is the zero-order Hankel function of the first kind. Given the total field 𝒇tot inside the sample domain Ω, the scattered field at the sensor plane Γ is given by ∫ 𝒇sc (𝒓) = 𝑔(𝒓 − 𝒓 ′) 𝑠(𝒓 ′) 𝒇tot (𝒓 ′) 𝑑 𝒓 ′, 𝒓 ∈ Γ. (E.3) Ω 232 Note that 𝒓 denotes the sensor location in Γ, and the integral is computed over Ω. By discretizing Equation (E.1) and Equation (E.3), we obtain a vectorized system that describes the linear inverse scattering problem. We denote the 2D vectorized permittivity distribution of the object by 𝒙0 := 𝑠(𝒓), and the corresponding measurement by 𝒚 sc = 𝒇sc (𝒓) for notational consistency. The forward model can thus be written as 𝒚 sc = 𝑯( 𝒇tot ⊙ 𝒙 0 ) + 𝒏 = 𝑨𝒙 0 + 𝒏, (E.4) where 𝒇tot = 𝑮 ( 𝒇in ⊙ 𝒙0 ), matrices 𝑮 and 𝑯 are discretizations of the Green’s function at Γ and Ω, respectively, and 𝑨 := 𝑯diag( 𝒇tot ). We split and concatenate the real and imaginary parts of 𝑨, and pre-compute the singular value decomposition of 𝑨 to facilitate the plug-and-play diffusion methods that exploit SVD of linear inverse problems. We set the physical size of test images to 18cm×18cm, and the wavelength of the illumination to 𝜆 = 0.84cm as specified in [272]. The forward model consists of 𝑁 = 20 transmitters, placed uniformly on a circle of radius 𝑅 = 1.6m. We further assume the background medium to be air with permittivity 𝜖 𝑏 = 1. We specify the number of receivers to be 𝑀 = 360, 180, 60 in our experiments. Related Work Linear inverse scattering aims to reconstruct the spatial distribution of an object’s dielectric permittivity from the measurements of the light it scatters [148, 315]. This problem arises in various applications, such as ultrasound imaging [37], optical microscopy [62, 275], and digital holography [35]. Due to the physical constraints on the number and placement of sensors, the problem is often ill-posed, as the scattered light field is undersampled. Linear inverse scattering is commonly formulated as a linear inverse problem using scattering models based on the first Born [315] or Rytov [84] approximations. These models enable efficient computation and facilitate the use of convex optimization algorithms. On the other hand, nonlinear approaches have been developed to image strongly scattering objects [56, 150, 188, 208, 282], although these methods generally have a higher computational complexity. Deep learning-based methods have also been explored for linear inverse scattering. A common approach is to train convolutional neural networks (CNNs) to directly invert the scattering process by learning an inverse mapping from the measurements to permittivity distribution [179, 183, 274, 322]. Recent research has extended these efforts to more advanced deep learning techniques, such as neural fields [43, 193] and deep image priors [359]. 233 E.2.2 Compressed Sensing Multi-Coil MRI Problem Details We refer readers to Section 2.2 for details on the compressed sensing multi-coil MRI problem. We use the raw multi-coil 𝑘-space data from the fastMRI knee dataset [338]. We then estimate the coil sensitivity maps of each slice using the ESPIRiT [285] method implemented in SigPy1 . Since different volumes in the dataset have different shapes, we adopt the preprocessing procedure in [145], leading to images with a shape of 320×320. The ground truth image is given by calculating the magnitude image of the Minimum Variance Unbiased Estimator (MVUE), which is used for all numbers reported in Table E.2 and Table E.3. The MVUE images are also used as ground truths for training the end-to-end deep learning methods Residual U-Net and E2E-VarNet [267]. Related Work Compressed sensing magnetic resonance imaging (CS-MRI) is a medical imaging technology that enables high-resolution visualization of human tissues with faster acquisition time than traditional MRI [204]. Instead of fully sampling the measurement space (a.k.a. 𝑘-space), CS-MRI only takes sparse measurements and then solves an inverse problem that recovers the underlying image [205]. The traditional approach is to solve a regularized optimization problem that involves a data-fit term and a regularization term, such as the total variation (TV) [29], and the ℓ1 -norm after a sparsifying transformation, such as the Wavelet transform [207] and dictionary decomposition [140, 243, 340]). End-to-end deep learning methods have also demonstrated strong performance in MRI reconstruction. Prior works have proposed unrolled networks [2, 123, 191, 255, 331], U-Net-based networks [142, 174], GAN-based networks [235, 329], among others [190, 202, 279, 305, 360]. These learning methods have achieved state-of-the-art performance on the fastMRI dataset [338]. Another line of work is to employ image denoisers as plug-and-play prior [145, 190, 273] Recently, diffusion model-based methods have been designed for CS-MRI reconstruction [64, 65, 201]. E.2.3 Black Hole Imaging Related Work The Event Horizon Telescope (EHT) Collaboration aims to image black holes using a global network of radio telescopes operating at around a 1mm wavelength. Using traditional imaging techniques, the EHT Collaboration has successfully imaged the supermassive black holes M87* [71, 93] and SgrA* [92]. The classical imaging algorithm is CLEAN [67, 134], as implemented in the DIFMAP 1 https://github.com/mikgroup/sigpy (BSD-3-Clause license) 234 PnP-DM DAPS Ground truth Figure E.2: Multi-modal example on black hole imaging. The image shows two image modes discovered by DAPS and PnP-DM. [256, 257] software. DIFMAP is an inverse modeling approach that starts with the “dirty” image (given by the inverse Fourier transform of the visibilities) and iteratively deconvolves the image with an estimate point-spread function to “clean” the image. Since DIFMAP often requires a human-in-the-loop, we choose not to present results from DIFMAP. The EHT has also developed and used regularized maximum-likelihood approaches, namely eht-imaging [49–51] and SMILI [5–7]. Although they regularize and optimize the image differently [71], eht-imaging and SMILI both iteratively update an estimated image to agree with the measured data and regularization assumptions. Because of the simple regularization they choose to use, these baseline methods are limited in the amount of visual detail they can recover and do not recover detail far beyond the intrinsic resolution of the measurements. Some deep-learning-based regularization approaches have been proposed for VLBI [86, 101, 102], but most plug-and-play inverse diffusion solvers have not been validated on black hole imaging. Multi-Modal Solutions As previously discussed, the non-convex and sparse measurement characteristics of black hole imaging may lead to multi-modal posterior distributions. While the solutions potentially look quite different from the ground truth, they may fit the measurements equally well and exhibit high prior likelihood. Figure E.2 illustrates two modes of solutions discovered by DAPS and PnP-DM. This multi-modal behavior has not been extensively formulated or discussed in previous literature, and we believe it represents a phenomenon worthy of further investigation. 235 E.2.4 Full Waveform Inversion Problem Details We use an open-source software, Devito [199], for both the forward and adjoint modeling of FWI2 . We discretize a physical domain of 2.54km × 1.27km with a 128×128 mesh, leading to a horizontal spacing of ≈20m and a vertical spacing of ≈10m. The time step is set to 0.001s which satisfies the Courant–Friedrichs–Lewy (CFL) condition [76]. We use a Ricker wavelet with a central frequency of 5Hz to excite the wavefield and model it for 1s. The natural boundary condition is set for the top boundary (free surface), which will generate reflected waves, while the absorbing boundary condition [68] is set for the rest boundaries to avoid artificial reflections. The absorbing boundary width is set to 80 grid points. Inferring the subsurface velocity from observed data at receivers defines the inverse problem. We place 129 receivers evenly near the free surface (at a depth of 10m) to model a realistic scenario. We excite 16 sources evenly at a depth of 1270m. This source-receiver geometry is designed for the entire physical medium to be sampled by seismic waves, making it theoretically feasible to invert for 𝑣. Devito uses the adjoint-state method to estimate the gradient by cross-correlating the forward and adjoint wavefields at zero time lag [232]: 𝑁𝑡 ∑︁ 1 2 −1 𝑇 𝒈[𝑡] 𝒉𝑡𝑡 [𝑡], ∇ 𝒎 𝑷𝑟 𝑨(𝒎) 𝑷𝑠 𝒒 𝑠 − 𝒚 2 = 2 𝑡=1 (E.5) where 𝒎 = 𝒙12 , 𝑁𝑡 is the number of computational time steps, and 𝒉𝑡𝑡 is the second0 order time derivative of the adjoint wavefield that solves 𝑨𝑇 (𝒎) 𝒉 = 𝑷𝑟𝑇 (𝑷𝑟 𝒈 − 𝒚). Recall that 𝒈 = 𝑨(𝒎) −1 𝑷𝑇𝑠 𝒒 𝑠 is the synthetic pressure wavefield. E.2.5 Navier-Stokes Equation To generate the training and test samples, we first draw independent identically
distributed samples from the Gaussian random field N 0, (−Δ + 9𝑰) −4 , where −Δ denotes the negative Laplacian. Then, we evolve them according to Equation (8.6) for 5 time units to get the final vorticity field, which generates an empirical distribution of the vorticity field with rich flow features. We set the forcing function as 𝑓 (𝒙) = −4 cos(4𝒙2 ) where 𝒙2 indicates the second dimension of 𝒙. 2 See https://www.devitoproject.org/examples/seismic/tutorials/03_fwi.html for a tutorial. 236 E.3 Pre-Trained Diffusion Model Details We train diffusion models following the pipeline from [151], using U-Net architectures from [85] and [266]. Detailed network configurations can be found in Table E.9. Table E.9: Model card for pre-trained diffusion models. Inverse scattering Black hole Input resolution # Attention blocks in encoder/decoder # Residual blocks per resolution Attention resolutions # Parameters # Training steps E.3.1 E.3.1.1 128 × 128 5 1 {16} 26.8M 50,000 64 × 64 3 1 {16} 20.0M 50,000 MRI FWI 2D Navier-Stokes 2 × 320 × 320 5 1 {16} 26.0M 100,000 128 × 128 5 1 {16} 26.8M 50,000 128 × 128 5 1 {16} 26.8M 50,000 Algorithms and Parameter Choices Problem-Specific Baselines Linear Inverse Scattering We include FISTA-TV [272] as a traditional optimizationbased method. We set batch size 𝐵 = 20 and 𝜏 = 5 × 10−7 for all experiments. Compressed Sensing Multi-Coil MRI We utilize both traditional methods, such as Wavelet+ℓ1 [204, 205] and TV, as well as end-to-end models like Residual U-Net and E2E-VarNet [267]. For the traditional methods, we apply the same hyperparameter search strategy for fine-tuning, while the end-to-end models are trained using the Adam optimizer with a learning rate of 1 × 10−4 until convergence. Black Hole Imaging We use SMILI [5–7] and eht-imaging [49–51] as our baseline methods. To ensure compatibility with the default hyperparameters of these methods, we preprocess the test dataset accordingly. Full Waveform Inversion A classic baseline for full waveform inversion is LBFGS [187]. We set the maximum iteration to 5 and perform 100 global update steps with a Wolfe line search. The second baseline we consider is the Adam optimization algorithm [162]. We implement the Adam optimizer with a learning rate of 0.02 with the learning rate decay to minimize the data misfit term. For the traditional method, the initialization is a smoothed version of the ground truth, which is blurred using a Gaussian filter with 𝜎 = 20. We perform the inversion for 300 iterations. Navier-Stokes Equation The traditional baseline we implement is the Ensemble Kalman Inversion (EKI) first proposed in [143]. It is implemented with 2,048 237 particles, 500 update steps, and an adaptive step size used in [166] to ensure a similar computation budget. Additional baselines include DPS-fGSG and DPScGSG, which are natural extensions based on DPS that replace gradient by zerothorder gradient estimation first introduced in [356]. More specifically, we use the forward and central Gaussian Smoothed Gradient estimation technique [21]. E.3.1.2 Hyperparameter Selection To ensure sufficient tuning of the hyperparameters for each algorithm, we employ a hybrid strategy combining grid search with Bayesian optimization and an early termination technique, using a small validation dataset. Specifically, we first perform a coarse grid search to narrow down the search space and then apply Bayesian optimization. For problems where the forward model is fast, such as linear inverse scattering, MRI, and black hole imaging, we conduct 50 to 100 iterations of Bayesian optimization to select the best hyperparameters. For computationally intensive problems such as full waveform inversion and Navier-Stokes equation, we use 10-30 iterations of Bayesian optimization combined with an early termination technique [180], based on data misfit. The details of the search spaces for Bayesian optimization and the optimized hyperparameter choices are listed in Table E.10. 238 Table E.10: Hyperparameter search space and final choices of the diffusionmodel-based algorithms on all five inverse problems. Columns marked with task names present the chosen values for the reported main results in Appendix E.1. These values are selected by a hybrid hyperparameter search strategy described in Appendix E.3.1.2. Methods / Parameters Search space Linear inverse scattering (360 / 180 / 60) Black hole MRI (Sim. / Raw) FWI 2D Navier-Stokes DPS Guidance scale [10−3 , 103 ] 280/380/625 0.003 0.589/0.428 10−2 – LGD Guidance scale # MC samples [10−3 , 104 ] [1, 20] 3200/6400/13000 20 0.0082 8 – – 11.73 5 – – RED-diff Learning rate Regularization 𝜆 base Regularization schedule Gradient weight [10−4 , 1.0] [10−3 , 1.0] constant, linear, sqrt [10−2 , 102 ] 0.04 0.0005 constant 1500 0.05 0.25 constant 0.0004 4 × 10−2 / 2.96 × 10−2 2.33 × 10−1 / 2.72 × 10−3 sqrt 6.68 × 101 / 1.7 × 10−2 0.01 0.1 linear 1 – – – – DiffPIR # sampling steps Regularization 𝜆 Stochasticity 𝜁 Noise level 𝜎𝑦 {200, 400, . . . , 1000} [1, 105 ] [10−5 , 1] [10−2 , 101 ] 200 4 × 10−4 /2 × 10−4 /10−4 1 0.01 1000 113.6 0.34 1.4 1000 163 / 1.31 0.114 / 0.478 1.05 × 10−2 / 1.36 × 10−1 1000 80.6 0.11 0.28 – – – – PnP-DM Annealing step Annealing sigma max Annealing decay rate Langevin step size Langevin step number Noise level [50, 200] [10, 50] [0.60, 0.99] [10−6 , 10−3 ] [10, 500] [10−4 , 101 ] 100 10 0.9 2 × 10−5 /4 × 10−5 /10−4 200 10−4 100 10 0.93 10−5 200 1 100 10 0.93 10−6 200 1.02 × 10−3 / 1.15 × 10−2 150 25 0.99 3 × 10−4 10 1 – – – – – – DAPS Annealing step Diffusion step Langevin step size Langevin step number Noise level Step size decay [50, 200] [1, 10] [10−6 , 10−3 ] [10, 500] [10−4 , 101 ] [0.1, 1] 200 10 200 5 50 10−4 1/1/0.5 100 5 10−4 20 1 1 100 1.63 × 10−3 / 4.77 × 10−3 1 150 5 3 × 10−4 50 1 1 – – – - 4 × 10−5 /8 × 10−5 /2 × 10−4 1.03 × 10−5 / 1.52 × 10−5 DDRM Stochasticity 𝜂 [0, 1] 0.85 – – – – DDNM Stochasticity 𝜂 # time-travel steps 𝐿 [0, 1] [0, 5] 0.95 1 – – – – – – – – 𝚷GDM Stochasticity 𝜂 [0, 1] 0.2 – – – – FPS Stochasticity 𝜂 # particles [0, 1] [1, 20] 0.9 20 – – – – – – – – MCGdiff # particles [1, 64] 16 – – – – DPS-fGSG Guidance scale [10−2 , 102 ] – – – – 0.1 DPS-cGSG Guidance scale [10−2 , 102 ] – – – – 0.1 DPG # MC samples Guidance scale {1000, 2000, . . . , 6000} [10−1 , 103 ] – – – – – – – – 4000 64 SCG # MC samples {128, 256, 512} – – – – 512 EnKG Guidance scale # particles {1.0, 2.0, 4.0} {512, 1024, 2048} - – – – – – – 2.0 2048 239 BIBLIOGRAPHY [1] Jonas Adler and Oktem Ozan. “Learned Primal-Dual Reconstruction.” In: IEEE Transactions on Medical Imaging PP (July 2017). doi: 10.1109/ TMI.2018.2799231. [2] Hemant K. Aggarwal, Merry P. Mani, and Mathews Jacob. “MoDL: ModelBased Deep Learning Architecture for Inverse Problems.” In: IEEE Transactions on Medical Imaging 38.2 (2019), pp. 394–405. doi: 10.1109/TMI. 2018.2865356. [3] Hemant Kumar Aggarwal and Mathews Jacob. “J-MoDL: Joint ModelBased Deep Learning for Optimized Sampling and Reconstruction.” In: IEEE Journal of Selected Topics in Signal Processing 14.6 (2020), pp. 1151– 1162. doi: 10.1109/JSTSP.2020.3004094. [4] Rizwan Ahmad, Charles Bouman, Gregery Buzzard, Stanley Chan, Sizhuo Liu, Edward Reehorst, and Philip Schniter. “Plug-and-Play Methods for Magnetic Resonance Imaging: Using Denoisers for Image Recovery.” In: IEEE Signal Processing Magazine 37 (Jan. 2020), pp. 105–116. doi: 10. 1109/MSP.2019.2949470. [5] Kazunori Akiyama, Shiro Ikeda, Mollie Pleau, Vincent L. Fish, Fumie Tazaki, Kazuki Kuramochi, Avery E. Broderick, Jason Dexter, Monika Mościbrodzka, Michael Gowanlock, Mareki Honma, and Sheperd S. Doeleman. “Superresolution Full-Polarimetric Imaging for Radio Interferometry with Sparse Modeling.” In: The Astronomical Journal 153.4 (Mar. 2017), p. 159. doi: 10.3847/1538-3881/aa6302. url: https://dx.doi.org/10. 3847/1538-3881/aa6302. [6] Kazunori Akiyama, Kazuki Kuramochi, Shiro Ikeda, Vincent L. Fish, Fumie Tazaki, Mareki Honma, Sheperd S. Doeleman, Avery E. Broderick, Jason Dexter, Monika Mościbrodzka, Katherine L. Bouman, Andrew A. Chael, and Masamichi Zaizen. “Imaging the Schwarzschild-Radius-Scale Structure of M87 with the Event Horizon Telescope Using Sparse Modeling.” In: The Astrophysical Journal 838.1 (Mar. 2017), p. 1. doi: 10.3847/1538-4357/ aa6305. url: https://dx.doi.org/10.3847/1538-4357/aa6305. [7] Kazunori Akiyama, Fumie Tazaki, Kotaro Moriyama, Ilje Cho, Shiro Ikeda, Mahito Sasada, Hiroki Okino, and Mareki Honma. SMILI: Sparse Modeling Imaging Library for Interferometry. Version v0.0.0. Mar. 2019. doi: 10. 5281/zenodo.2616725. url: https://doi.org/10.5281/zenodo. 2616725. [8] Cagan Alkan, Morteza Mardani, Shreyas Vasanawala, and John M. Pauly. “Learning to Sample MRI via Variational Information Maximization.” In: 240 NeurIPS 2020 Workshop on Deep Learning and Inverse Problems. 2020. url: https://openreview.net/forum?id=1AOReNkDmh_. [9] Brian D. O. Anderson. “Reverse-Time Diffusion Equation Models.” In: Stochastic Processes and their Applications 12.3 (1982), pp. 313–326. issn: 0304-4149. doi: 10.1016/0304- 4149(82)90051- 5. url: https:// www.sciencedirect.com/science/article/pii/0304414982900515. [10] Liana Apostolova and Paul Thompson. “Brain Mapping as a Tool to Study Neurodegeneration.” In: Neurotherapeutics: The journal of the American Society for Experimental NeuroTherapeutics 4 (Aug. 2007), pp. 387–400. doi: 10.1016/j.nurt.2007.05.009. [11] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. “Structured Denoising Diffusion Models in Discrete StateSpaces.” In: Advances in Neural Information Processing Systems. Ed. by A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan. 2021. url: https://openreview.net/forum?id=h7-XixPCAL. [12] Cagla D. Bahadir, Alan Q. Wang, Adrian V. Dalca, and Mert R. Sabuncu. “Deep-Learning-Based Optimization of the Under-Sampling Pattern in MRI.” In: IEEE Transactions on Computational Imaging 6 (2020), pp. 1139–1152. doi: 10.1109/TCI.2020.3006727. [13] Tim Bakker, Herke van Hoof, and Max Welling. “Experimental Design for MRI by Greedy Policy Search.” In: Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS ’20. Vancouver, BC, Canada: Curran Associates Inc., 2020. isbn: 9781713829546. url: https://dl.acm.org/doi/abs/10.5555/3495724.3497315. [14] Guha Balakrishnan, Amy Zhao, Mert R. Sabuncu, John Guttag, and Adrian V. Dalca. “VoxelMorph: A Learning Framework for Deformable Medical Image Registration.” In: IEEE Transactions on Medical Imaging 38.8 (2019), pp. 1788–1800. doi: 10.1109/TMI.2019.2897538. [15] Krishna Balasubramanian, Sinho Chewi, Murat A Erdogdu, Adil Salim, and Shunshi Zhang. “Towards a Theory of Non-Log-Concave Sampling:FirstOrder Stationarity Guarantees for Langevin Monte Carlo.” In: Proceedings of Thirty Fifth Conference on Learning Theory. Ed. by Po-Ling Loh and Maxim Raginsky. Vol. 178. Proceedings of Machine Learning Research. PMLR, July 2022, pp. 2896–2923. url: https://proceedings.mlr. press/v178/balasubramanian22a.html. [16] Fan Bao, Chongxuan Li, Jiacheng Sun, and Jun Zhu. “Why Are Conditional Generative Models Better Than Unconditional Ones?” In: NeurIPS 2022 Workshop on Score-Based Methods. 2022. url: https://openreview. net/forum?id=sbDyvrvvKn7. 241 [17] Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann. Conditional Image Generation with Score-Based Diffusion Models. 2021. arXiv: 2111.13606 [cs.LG]. url: https://arxiv.org/abs/ 2111.13606. [18] Jonathan Baxter and Peter L. Bartlett. “Infinite-Horizon Policy-Gradient Estimation.” In: Journal of Artificial Intelligence Research 15.1 (Nov. 2001), pp. 319–350. issn: 1076-9757. doi: 10.1613/jair.806. [19] Amir Beck and Marc Teboulle. “A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems.” In: SIAM Journal on Imaging Sciences 2.1 (2009), pp. 183–202. doi: 10.1137/080716542. eprint: https: //doi.org/10.1137/080716542. url: https://doi.org/10.1137/ 080716542. [20] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. 2013. arXiv: 1308.3432 [cs.LG]. url: https://arxiv.org/ abs/1308.3432. [21] Albert S. Berahas, Liyuan Cao, Krzysztof Choromanski, and Katya Scheinberg. “A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-free Optimization.” In: Foundations of Computational Mathematics 22.2 (2022), pp. 507–560. doi: 10 . 1007 / s10208 - 021 09513-z. [22] Brady Bhalla*, Zachary Huang*, Elyes Serghine*, Bingliang Zhang, Zihui Wu, and Katherine L. Bouman. “Text-Guided Image Restoration via a Unified Plug-and-Play Diffusion Framework.” In: Computational Cameras and Displays (CCD) Workshop, CVPR 2025 (2025). [23] Lindy Blackburn, Dominic W. Pesce, Michael D. Johnson, Maciek Wielgus, Andrew A. Chael, Pierre Christian, and Sheperd S. Doeleman. “Closure Statistics in Interferometric Data.” In: The Astrophysical Journal 894.1 (May 2020), p. 31. doi: 10 . 3847 / 1538 - 4357 / ab8469. url: https : //dx.doi.org/10.3847/1538-4357/ab8469. [24] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets. 2023. arXiv: 2311.15127 [cs.CV]. url: https://arxiv.org/abs/2311.15127. [25] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. “Align Your Latents: HighResolution Video Synthesis with Latent Diffusion Models.” In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2023, pp. 22563–22575. doi: 10.1109/CVPR52729.2023.02161. 242 [26] Kai Tobias Block, Martin Uecker, and Jens Frahm. “Undersampled Radial MRI with Multiple Coils. Iterative Image Reconstruction Using a Total Variation Constraint.” In: Magnetic Resonance in Medicine 57.6 (2007), pp. 1086–1098. doi: 10 . 1002 / mrm . 21236. eprint: https : / / onlinelibrary . wiley . com / doi / pdf / 10 . 1002 / mrm . 21236. url: https : / / onlinelibrary . wiley . com / doi / abs / 10 . 1002 / mrm . 21236. [27] Sergey Bobkov and Prasad Tetali. “Modified Logarithmic Sobolev Inequalities in Discrete Settings.” In: Journal of Theoretical Probability 19 (June 2006), pp. 289–336. doi: 10.1007/s10959-006-0016-3. [28] Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. “Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling.” In: Advances in Neural Information Processing Systems. Ed. by A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan. 2021. url: https://openreview.net/forum?id=9BnCwiXB0ty. [29] Charles Bouman and Ken Sauer. “A Generalized Gaussian Image Model for Edge-Preserving MAP Estimation.” In: IEEE Transactions on Image Processing 2.3 (1993), pp. 296–310. doi: 10.1109/83.236536. [30] Charles A. Bouman. Foundations of Computational Imaging: A ModelBased Approach. Philadelphia, PA: Society for Industrial and Applied Mathematics, 2022. doi: 10 . 1137 / 1 . 9781611977134. eprint: https : / / epubs.siam.org/doi/pdf/10.1137/1.9781611977134. url: https: //epubs.siam.org/doi/abs/10.1137/1.9781611977134. [31] Charles A. Bouman and Gregery T. Buzzard. “Generative Plug and Play: Posterior Sampling for Inverse Problems.” In: 2023 59th Annual Allerton Conference on Communication, Control, and Computing (Allerton). 2023, pp. 1–7. doi: 10.1109/Allerton58177.2023.10313413. [32] Katherine L. Bouman, Michael D. Johnson, Daniel Zoran, Vincent L. Fish, Sheperd S. Doeleman, and William T. Freeman. “Computational Imaging for VLBI Image Reconstruction.” In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016, pp. 913–922. doi: 10.1109/ CVPR.2016.105. [33] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. “Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers.” In: Foundations and Trends in Machine Learning 3 (Jan. 2011), pp. 1–122. doi: 10.1561/2200000016. [34] Benjamin Boys, Mark Girolami, Jakiw Pidstrigach, Sebastian Reich, Alan Mosca, and Omer Deniz Akyildiz. “Tweedie Moment Projected Diffusions for Inverse Problems.” In: Transactions on Machine Learning Research (2024). Featured Certification. issn: 2835-8856. url: https : / / openreview.net/forum?id=4unJi0qrTE. 243 [35] David J. Brady, Kerkil Choi, Daniel L. Marks, Ryoichi Horisaki, and Sehoon Lim. “Compressive Holography.” In: Optics Express 17.15 (July 2009), pp. 13040–13049. doi: 10.1364/OE.17.013040. url: https://opg. optica.org/oe/abstract.cfm?URI=oe-17-15-13040. [36] Andrew Brock, Jeff Donahue, and Karen Simonyan. “Large Scale GAN Training for High Fidelity Natural Image Synthesis.” In: International Conference on Learning Representations. 2019. url: https://openreview. net/forum?id=B1xsqj09Fm. [37] Michael M. Bronstein, Alexander M. Bronstein, Michael Zibulevsky, and Haim Azhari. “Reconstruction in Diffraction Ultrasound Tomography Using Nonuniform FFT.” In: IEEE Transactions on Medical Imaging 21.11 (2002), pp. 1395–1401. doi: 10.1109/TMI.2002.806423. [38] Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 𝛽-VAE. 2018. arXiv: 1804.03599 [stat.ML]. url: https: //arxiv.org/abs/1804.03599. [39] John C. Butcher. Numerical Methods for Ordinary Differential Equations. Third. Hoboken, New Jersey: John Wiley & Sons, 2016. isbn: 9781119121503. doi: 10.1002/9781119121534. url: https://onlinelibrary.wiley. com/doi/book/10.1002/9781119121534. [40] Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. “A Continuous Time Framework for Discrete Denoising Models.” In: Advances in Neural Information Processing Systems. Ed. by Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho. 2022. url: https://openreview.net/forum?id= DmT862YAieY. [41] Emmanuel J. Candes, Justin Romberg, and Terrence Tao. “Robust Uncertainty Principles: Exact Signal Reconstruction from Highly Incomplete Frequency Information.” In: IEEE Transactions on Information Theory 52.2 (2006), pp. 489–509. doi: 10.1109/TIT.2005.862083. [42] Emmanuel Candès, Xiaodong Li, and Mahdi Soltanolkotabi. “Phase Retrieval from Coded Diffraction Patterns.” In: Applied and Computational Harmonic Analysis 39 (Oct. 2013). doi: 10.1016/j.acha.2014.09.004. [43] Ruiming Cao, Nikita Divekar, James Nuñez, Srigokul Upadhyayula, and Laura Waller. “Neural Space–Time Model for Dynamic Multi-Shot Imaging.” In: Nature Methods 21 (Sept. 2024), pp. 2336–2341. doi: 10.1038/ s41592-024-02417-0. [44] Gabriel Cardoso, Yazid Janati El Idrissi, Sylvain Le Corff, and Eric Moulines. “Monte Carlo Guided Denoising Diffusion Models for Bayesian Linear Inverse Problems.” In: The Twelfth International Conference on Learning 244 Representations. 2024. url: https : / / openreview . net / forum ? id = nHESwXvxWK. [45] Manuel Jorge Cardoso, Wenqi Li, Richard Brown, Nic Ma, Eric Kerfoot, Yiheng Wang, Benjamin Murrey, Andriy Myronenko, Can Zhao, Dong Yang, Vishwesh Nath, Yufan He, Ziyue Xu, Ali Hatamizadeh, Wentao Zhu, Yun Liu, Mingxin Zheng, Yucheng Tang, Isaac Yang, and Andrew Feng. “MONAI: An Open-Source Framework for Deep Learning in Healthcare.” In: (Nov. 2022). doi: 10.48550/arXiv.2211.02701. [46] Jose A. Carrillo, Franca Hoffmann, Andrew M. Stuart, and Urbain Vaes. “Consensus Based Sampling.” In: Studies in Applied Mathematics 148.3 (2022), pp. 1069–1140. doi: 10.1111/sapm.12470. eprint: https:// onlinelibrary.wiley.com/doi/pdf/10.1111/sapm.12470. url: https : / / onlinelibrary . wiley . com / doi / abs / 10 . 1111 / sapm . 12470. [47] James Carton and Benjamin Giese. “A Reanalysis of Ocean Climate Using Simple Ocean Data Assimilation (SODA).” In: Monthly Weather Review 136 (Aug. 2008). doi: 10.1175/2007MWR1978.1. [48] Neil K. Chada, Andrew M. Stuart, and Xin T. Tong. “Tikhonov Regularization within Ensemble Kalman Inversion.” In: SIAM Journal on Numerical Analysis 58.2 (2020), pp. 1263–1294. doi: 10.1137/19M1242331. eprint: https://doi.org/10.1137/19M1242331. url: https://doi.org/ 10.1137/19M1242331. [49] Andrew Chael, Katie Bouman, Michael Johnson, Maciek Wielgus, Lindy Blackburn, Chi-kwan Chan, Joseph Rachid Farah, Daniel Palumbo, and Dominic Pesce. eht-imaging: v1.1.0: Imaging Interferometric Data with Regularized Maximum Likelihood. Version v1.1.0. Mar. 2019. doi: 10 . 5281/zenodo.2614016. url: 10.5281/zenodo.2614016. [50] Andrew A. Chael, Michael D. Johnson, Katherine L. Bouman, Lindy L. Blackburn, Kazunori Akiyama, and Ramesh Narayan. “Interferometric Imaging Directly with Closure Phases and Closure Amplitudes.” In: The Astrophysical Journal 857.1 (Apr. 2018), p. 23. doi: 10.3847/1538- 4357/ aab6a8. url: https://dx.doi.org/10.3847/1538-4357/aab6a8. [51] Andrew A. Chael, Michael D. Johnson, Ramesh Narayan, Sheperd S. Doeleman, John F. C. Wardle, and Katherine L. Bouman. “High-Resolution Linear Polarimetric Imaging for the Event Horizon Telescope.” In: The Astrophysical Journal 829.1 (Sept. 2016), p. 11. doi: 10.3847/0004-637X/829/1/ 11. url: https://dx.doi.org/10.3847/0004-637X/829/1/11. [52] Stanley Chan, Xiran Wang, and Omar Elgendy. “Plug-and-Play ADMM for Image Restoration: Fixed Point Convergence and Applications.” In: IEEE Transactions on Computational Imaging PP (May 2016). doi: 10.1109/ TCI.2016.2629286. 245 [53] Pascal Chang, Jingwei Tang, Markus Gross, and Vinicius C. Azevedo. “How I Warped Your Noise: a Temporally-Correlated Noise Prior for Diffusion Models.” In: The Twelfth International Conference on Learning Representations. 2024. url: https://openreview.net/forum?id=pzElnMrgSD. [54] Nicolas Chauffert, Philippe Ciuciu, Jonas Kahn, and Pierre Weiss. “Variable Density Sampling with Continuous Trajectories.” In: SIAM Journal on Imaging Sciences 7.4 (2014), pp. 1962–1992. doi: 10.1137/130946642. eprint: https://doi.org/10.1137/130946642. url: https://doi. org/10.1137/130946642. [55] Liuhan Chen, Zongjian Li, Bin Lin, Bin Zhu, Qian Wang, Shenghai Yuan, Xing Zhou, Xinhua Cheng, and Li Yuan. OD-VAE: An Omni-Dimensional Video Compressor for Improving Latent Video Diffusion Model. 2024. arXiv: 2409.01199 [cs.CV]. url: https://arxiv.org/abs/2409.01199. [56] Michael Chen, David Ren, Hsiou-Yuan Liu, Shwetadwip Chowdhury, and Laura Waller. “Multi-Layer Born Multiple-Scattering Model for 3D phase Microscopy.” In: Optica 7 (Apr. 2020), pp. 394–403. doi: 10 . 1364 / OPTICA.383030. [57] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. “Sampling is as Easy as Learning the Score: Theory for Diffusion Models with Minimal Data Assumptions.” In: The Eleventh International Conference on Learning Representations. 2023. url: https : / / openreview . net/forum?id=zyLVMgsZ0U_. [58] Yongxin Chen, Sinho Chewi, Adil Salim, and Andre Wibisono. “Improved Analysis for a Proximal Algorithm for Sampling.” In: Proceedings of Thirty Fifth Conference on Learning Theory. Ed. by Po-Ling Loh and Maxim Raginsky. Vol. 178. Proceedings of Machine Learning Research. PMLR, July 2022, pp. 2984–3014. url: https://proceedings.mlr.press/ v178/chen22c.html. [59] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion.” In: Proceedings of Robotics: Science and Systems. Daegu, Republic of Korea, July 2023. doi: 10.15607/RSS.2023.XIX.026. [60] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. “ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models.” In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). 2021, pp. 14347–14356. doi: 10.1109/ICCV48922. 2021.01410. [61] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. “Perception Prioritized Training of Diffusion Models.” In: 2022 IEEE/CVF Conference on Computer Vision and Pattern 246 Recognition (CVPR). 2022, pp. 11462–11471. doi: 10.1109/CVPR52688. 2022.01118. [62] Wonshik Choi, Christopher Fang-Yen, Kamran Badizadegan, Seungeun Oh, Niyom Lue, Ramachandra R. Dasari, and Michael S. Feld. “Tomographic Phase Microscopy.” In: Nature Methods 4.9 (Sept. 2007), pp. 717–719. doi: 10.1038/nmeth1078. [63] Wenda Chu, Zihui Wu, Yifan Chen, Yang Song, and Yisong Yue. Split Gibbs Discrete Diffusion Posterior Sampling. 2025. arXiv: 2503.01161 [cs.LG]. url: https://arxiv.org/abs/2503.01161. [64] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. “Diffusion Posterior Sampling for General Noisy Inverse Problems.” In: The Eleventh International Conference on Learning Representations. 2023. url: https : / / openreview . net / forum ? id = OnD9zGAGT0k. [65] Hyungjin Chung and Jong Chul Ye. “Score-Based Diffusion Models for Accelerated MRI.” In: Medical Image Analysis 80 (May 2022), p. 102479. doi: 10.1016/j.media.2022.102479. [66] Pieter Hendrik van Cittert. “Die Wahrscheinliche Schwingungsverteilung in Einer von Einer Lichtquelle Direkt Oder Mittels Einer Linse Beleuchteten Ebene.” In: Physica 1.1 (1934), pp. 201–210. issn: 0031-8914. doi: 10. 1016/S0031-8914(34)90026-4. url: https://www.sciencedirect. com/science/article/pii/S0031891434900264. [67] Barry G. Clark. “An Efficient Implementation of the Algorithm “CLEAN”.” In: Astronomy and Astrophysics, vol. 89, no. 3, Sept. 1980, p. 377, 378. 89 (1980), p. 377. url: https://adsabs.harvard.edu/full/1980A% 26A....89..377C. [68] Robert Clayton and Björn Engquist. “Absorbing Boundary Conditions for Acoustic and Elastic Wave Equations.” In: Bulletin of the Seismological Society of America 67.6 (Dec. 1977), pp. 1529–1540. issn: 0037-1106. doi: 10.1785/BSSA0670061529. eprint: https://pubs.geoscienceworld. org/ssa/bssa/article-pdf/67/6/1529/5320934/bssa0670061529. pdf. url: https://doi.org/10.1785/BSSA0670061529. [69] Florentin Coeurdoux, Nicolas Dobigeon, and Pierre Chainais. “Plug-andPlay Split Gibbs Sampler: Embedding Deep Generative Priors in Bayesian Inference.” In: IEEE Transactions on Image Processing 33 (2024), pp. 3496– 3507. doi: 10.1109/TIP.2024.3404338. [70] Regev Cohen, Yochai Blau, Daniel Freedman, and Ehud Rivlin. “It Has Potential: Gradient-Driven Denoisers for Convergent Solutions to Inverse Problems.” In: Advances in Neural Information Processing Systems. Ed. by A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan. 2021. url: https://openreview.net/forum?id=MYvpQVjCK0_. 247 [71] The Event Horizon Telescope Collaboration. “First M87 Event Horizon Telescope Results. IV. Imaging the Central Supermassive Black Hole.” In: The Astrophysical Journal Letters 875.1 (Apr. 2019), p. L4. doi: 10.3847/ 2041- 8213/ab0e85. url: https://dx.doi.org/10.3847/20418213/ab0e85. [72] The Event Horizon Telescope Collaboration. “First M87 Event Horizon Telescope Results. V. Physical Origin of the Asymmetric Ring.” In: The Astrophysical Journal Letters 875.1 (Apr. 2019), p. L5. doi: 10 . 3847 / 2041- 8213/ab0f43. url: https://dx.doi.org/10.3847/20418213/ab0f43. [73] Liam Connor, Katherine L Bouman, Vikram Ravi, and Gregg Hallinan. “Deep Radio-Interferometric Imaging with POLISH: DSA-2000 and Weak Lensing.” In: Monthly Notices of the Royal Astronomical Society 514.2 (May 2022), pp. 2614–2626. issn: 0035-8711. doi: 10.1093/mnras/stac1329. eprint: https://academic.oup.com/mnras/article- pdf/514/2/ 2614/44147595/stac1329.pdf. url: https://doi.org/10.1093/ mnras/stac1329. [74] Robert L. Cook. “Stochastic Sampling in Computer Graphics.” In: 5.1 (Jan. 1986), pp. 51–72. issn: 0730-0301. doi: 10 . 1145 / 7529 . 8927. url: https://doi.org/10.1145/7529.8927. [75] Simon L. Cotter, Gareth O. Roberts, Andrew M. Stuart, and David White. “MCMC Methods for Functions: Modifying Old Algorithms to Make Them Faster.” In: Statistical Science 28.3 (2013), pp. 424–446. doi: 10.1214/13STS421. url: https://doi.org/10.1214/13-STS421. [76] Richard Courant, Kurt Friedrichs, and Hans Lewy. “On the Partial Difference Equations of Mathematical Physics.” In: IBM Journal of Research and Development 11.2 (1967), pp. 215–234. doi: 10.1147/rd.112.0215. [77] Giannis Daras, Weili Nie, Karsten Kreis, Alex Dimakis, Morteza Mardani, Nikola Borislavov Kovachki, and Arash Vahdat. “Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models.” In: The Thirtyeighth Annual Conference on Neural Information Processing Systems. 2024. url: https://openreview.net/forum?id=LH94zPv8cu. [78] Pierre Del Moral, Arnaud Doucet, and Ajay Jasra. “Sequential Monte Carlo Samplers.” In: Journal of the Royal Statistical Society Series B: Statistical Methodology 68.3 (May 2006), pp. 411–436. issn: 1369-7412. doi: 10 . 1111 / j . 1467 - 9868 . 2006 . 00553 . x. eprint: https : / / academic . oup . com / jrsssb / article - pdf / 68 / 3 / 411 / 49795343 / jrsssb \ _68 \ _3 \ _411 . pdf. url: https : / / doi . org / 10 . 1111 / j . 1467 9868.2006.00553.x. 248 [79] Mauricio Delbracio and Peyman Milanfar. “Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration.” In: Transactions on Machine Learning Research (2023). Featured Certification. issn: 28358856. url: https://openreview.net/forum?id=VmyFF5lL3F. [80] Chengyuan Deng, Shihang Feng, Hanchen Wang, Xitong Zhang, Peng Jin, Yinan Feng, Qili Zeng, Yinpeng Chen, and Youzuo Lin. “OpenFWI: Large-Scale Multi-structural Benchmark Datasets for Full Waveform Inversion.” In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2022. url: https://openreview.net/ forum?id=7w-a8PYPlP. [81] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. “ImageNet: A Large-Scale Hierarchical Image Database.” In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009, pp. 248– 255. doi: 10.1109/CVPR.2009.5206848. [82] Yitong Deng, Winnie Lin, Lingxiao Li, Dmitriy Smirnov, Ryan D Burgert, Ning Yu, Vincent Dedun, and Mohammad H. Taghavi. “Infinite-Resolution Integral Noise Warping for Diffusion Models.” In: The Thirteenth International Conference on Learning Representations. 2025. url: https:// openreview.net/forum?id=Y6LPWBo2HP. [83] Arjun D. Desai, Andrew M. Schmidt, Elka B. Rubin, Christopher Michael Sandino, Marianne Susan Black, Valentina Mazzoli, Kathryn J. Stevens, Robert Boutin, Christopher Re, Garry E. Gold, Brian Hargreaves, and Akshay Chaudhari. “SKM-TEA: A Dataset for Accelerated MRI Reconstruction with Dense Image Labels for Quantitative Clinical Evaluation.” In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021. url: https : / / openreview . net/forum?id=YDMFgD_qJuA. [84] Anthony J. Devaney. “Inverse-Scattering Theory within the Rytov Approximation.” In: Optics Letters 6.8 (Aug. 1981), pp. 374–376. doi: 10.1364/ OL.6.000374. [85] Prafulla Dhariwal and Alexander Quinn Nichol. “Diffusion Models Beat GANs on Image Synthesis.” In: Advances in Neural Information Processing Systems. Ed. by A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan. 2021. url: https : / / openreview . net / forum ? id = AAWuCvzaVt. [86] Noe Dia, M. J. Yantovski-Barth, Alexandre Adam, Micah Bowles, Pablo Lemos, Anna M. M. Scaife, Yashar Hezaveh, and Laurence PerreaultLevasseur. Bayesian Imaging for Radio Interferometry with Score-Based Priors. 2023. arXiv: 2311 . 18012 [astro-ph.IM]. url: https : / / arxiv.org/abs/2311.18012. 249 [87] Persi Diaconis and Laurent Saloff-Coste. “Logarithmic Sobolev Inequalities for Finite Markov Chains.” In: The Annals of Applied Probability 6.3 (1996), pp. 695–750. doi: 10.1214/aoap/1034968224. url: https://doi. org/10.1214/aoap/1034968224. [88] Lee R. Dice. “Measures of the Amount of Ecologic Association Between Species.” In: Ecology 26.3 (1945), pp. 297–302. issn: 00129658, 19399170. url: http://www.jstor.org/stable/1932409. [89] Zehao Dou and Yang Song. “Diffusion Posterior Sampling for Linear Inverse Problem Solving: A Filtering Perspective.” In: The Twelfth International Conference on Learning Representations. 2024. url: https : / / openreview.net/forum?id=tplXNcHZs1. [90] Bradley Efron. “Tweedie’s Formula and Selection Bias.” In: Journal of the American Statistical Association 106 (Dec. 2011), pp. 1602–1614. doi: 10.1198/jasa.2011.tm11181. [91] The Event Horizon Telescope Collaboration EHTC. “First M87 Event Horizon Telescope Results. III. Data Processing and Calibration.” In: The Astrophysical Journal Letters 875.1 (Apr. 2019), p. L3. doi: 10.3847/20418213/ ab0c57. url: https :/ /dx .doi. org/ 10. 3847/ 2041- 8213/ ab0c57. [92] The Event Horizon Telescope Collaboration EHTC. “First Sagittarius A* Event Horizon Telescope Results. III. Imaging of the Galactic Center Supermassive Black Hole.” In: The Astrophysical Journal Letters 930.2 (May 2022), p. L14. doi: 10.3847/2041-8213/ac6429. url: https://dx. doi.org/10.3847/2041-8213/ac6429. [93] The Event Horizon Telescope Collaboration EHTC. “The Persistent Shadow of the Supermassive Black Hole of M 87-I. Observations, Calibration, Imaging, and Analysis.” In: Astronomy & Astrophysics 681 (2024), A79. doi: 10.1051/0004-6361/202347932. [94] Gerrit Elsinga, Fulvio Scarano, Bernhard Wieneke, and Bas Oudheusden. “Tomographic Particle Image Velocimetry.” In: Experiments in Fluids 41 (Dec. 2006), pp. 933–947. doi: 10.1007/s00348-006-0212-z. [95] Patrick Esser, Robin Rombach, and Björn Ommer. “Taming Transformers for High-Resolution Image Synthesis.” In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021, pp. 12868–12878. doi: 10.1109/CVPR46437.2021.01268. [96] Geir Evensen. “Sequential Data Assimilation with a Nonlinear Quasi-Geostrophic Model Using Monte Carlo Methods to Forecast Error Statistics.” In: Journal of Geophysical Research: Oceans 99.C5 (1994), pp. 10143–10162. doi: 10 . 1029 / 94JC00572. eprint: https : / / agupubs . onlinelibrary . wiley.com/doi/pdf/10.1029/94JC00572. url: https://agupubs. onlinelibrary.wiley.com/doi/abs/10.1029/94JC00572. 250 [97] Jiaojiao Fan, Bo Yuan, and Yongxin Chen. “Improved Dimension Dependence of a Proximal Algorithm for Sampling.” In: Proceedings of Thirty Sixth Conference on Learning Theory. Ed. by Gergely Neu and Lorenzo Rosasco. Vol. 195. Proceedings of Machine Learning Research. PMLR, July 2023, pp. 1473–1521. url: https://proceedings.mlr.press/ v195/fan23a.html. [98] Zhiwen Fan, Liyan Sun, Xinghao Ding, Yue Huang, Congbo Cai, and John Paisley. “A Segmentation-Aware Deep Fusion Network for Compressed Sensing MRI.” In: Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8–14, 2018, Proceedings, Part VI. Munich, Germany: Springer-Verlag, 2018, pp. 55–70. isbn: 978-3-030-01230-4. doi: 10.1007/978-3-030-01231-1_4. url: https://doi.org/10. 1007/978-3-030-01231-1_4. [99] Zhenghan Fang, Sam Buchanan, and Jeremias Sulam. “What’s in a Prior? Learned Proximal Networks for Inverse Problems.” In: The Twelfth International Conference on Learning Representations. 2024. url: https: //openreview.net/forum?id=kNPcOaqC5r. [100] Elhadji C. Faye, Mame Diarra Fall, and Nicolas Dobigeon. “Regularization by Denoising: Bayesian Model and Langevin-Within-Split Gibbs Sampling.” In: IEEE Transactions on Image Processing 34 (2025), pp. 221–234. doi: 10.1109/TIP.2024.3520012. [101] Berthy Feng and Katherine Bouman. “Variational Bayesian Imaging with an Efficient Surrogate Score-Based Prior.” In: Transactions on Machine Learning Research (2024). issn: 2835-8856. url: https://openreview. net/forum?id=db2pFKVcm1. [102] Berthy Feng, Katherine Bouman, and William Freeman. “Event-HorizonScale Imaging of M87* under Different Assumptions via Deep Generative Image Priors.” In: The Astrophysical Journal 975 (Nov. 2024), p. 201. doi: 10.3847/1538-4357/ad737f. [103] Berthy T. Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine L. Bouman, and William T. Freeman. “Score-Based Diffusion Models as Principled Priors for Inverse Imaging.” In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). 2023, pp. 10486–10497. doi: 10. 1109/ICCV51070.2023.00965. [104] James R. Fienup. “Phase Retrieval Algorithms: A Comparison.” In: Applied Optics 21 15 (1982), pp. 2758–69. doi: 10.1364/AO.21.002758. [105] Bruce Fischl. “FreeSurfer.” In: NeuroImage 62 (Jan. 2012), pp. 774–81. doi: 10.1016/j.neuroimage.2012.01.021. [106] Vincent Fortin, Mabrouk Abaza, F. Anctil, and Richard Turcotte. “Why Should Ensemble Spread Match the RMSE of the Ensemble Mean? (vol 15, 251 pg 1708, 2014).” In: Journal of Hydrometeorology 16 (Feb. 2015), pp. 484– 484. doi: 10.1175/JHM-D-14-0161.1. [107] William T. Freeman, Thouis R. Jones, and Egon C. Pasztor. “ExampleBased Super-Resolution.” In: IEEE Computer Graphics and Applications 22.2 (2002), pp. 56–65. doi: 10.1109/38.988747. [108] Cong Fu, Keqiang Yan, Limei Wang, Wing Yee Au, Michael Curtis McThrow, Tao Komikado, Koji Maruhashi, Kanji Uchino, Xiaoning Qian, and Shuiwang Ji. “A Latent Diffusion Model for Protein Structure Generation.” In: Proceedings of the Second Learning on Graphs Conference. Ed. by Soledad Villar and Benjamin Chamberlain. Vol. 231. Proceedings of Machine Learning Research. PMLR, Nov. 2024, 29:1–29:17. url: https : //proceedings.mlr.press/v231/fu24a.html. [109] Daniel Gabay and Bertrand Mercier. “A Dual Algorithm for the Solution of Nonlinear Variational Problems via Finite Element Approximation.” In: Computers & Mathematics with Applications 2.1 (1976), pp. 17–40. issn: 0898-1221. doi: 10.1016/0898- 1221(76)90003- 1. url: https:// www.sciencedirect.com/science/article/pii/0898122176900031. [110] Urs Gamper, Peter Boesiger, and Sebastian Kozerke. “Compressed Sensing in Dynamic MRI.” In: Magnetic Resonance in Medicine 59.2 (2008), pp. 365–373. doi: 10.1002/mrm.21477. [111] Alfredo Garbuno-Inigo, Franca Hoffmann, Wuchen Li, and Andrew M. Stuart. “Interacting Langevin Diffusions: Gradient Structure and Ensemble Kalman Sampler.” In: SIAM Journal on Applied Dynamical Systems 19.1 (2020), pp. 412–441. doi: 10 . 1137 / 19M1251655. eprint: https : / / doi.org/10.1137/19M1251655. url: https://doi.org/10.1137/ 19M1251655. [112] Alfredo Garbuno-Inigo, Nikolas Nüsken, and Sebastian Reich. “Affine Invariant Interacting Langevin Dynamics for Bayesian Inference.” In: SIAM Journal on Applied Dynamical Systems 19.3 (2020), pp. 1633–1658. doi: 10.1137/19M1304891. eprint: https://doi.org/10.1137/19M1304891. url: https://doi.org/10.1137/19M1304891. [113] Andrew Gelman, Walter R. Gilks, and Gareth O. Roberts. “Weak Convergence and Optimal Scaling of Random Walk Metropolis Algorithms.” In: The Annals of Applied Probability 7.1 (1997), pp. 110–120. doi: 10. 1214/aoap/1034625254. url: https://doi.org/10.1214/aoap/ 1034625254. [114] D. Geman and Chengda Yang. “Nonlinear Image Recovery with Halfquadratic Regularization.” In: IEEE Transactions on Image Processing 4.7 (1995), pp. 932–946. doi: 10.1109/83.392335. 252 [115] Charles J. Geyer. “Practical Markov Chain Monte Carlo.” In: Statistical Science 7.4 (1992), pp. 473–483. doi: 10.1214/ss/1177011137. url: https://doi.org/10.1214/ss/1177011137. [116] Vahid Ghodrati, Jiaxin Shao, Mark Bydder, Ziwu Zhou, Wotao Yin, KimLien Nguyen, Yingli Yang, and Peng Hu. “MR Image Reconstruction Using Deep Learning: Evaluation of Network Structure and Loss Functions.” In: Quantitative Imaging in Medicine and Surgery 9 (Sept. 2019), pp. 1516– 1527. doi: 10.21037/qims.2019.08.10. [117] Davis Gilton, Gregory Ongie, and Rebecca Willett. “Deep Equilibrium Architectures for Inverse Problems in Imaging.” In: IEEE Transactions on Computational Imaging 7 (2021), pp. 1123–1133. doi: 10 . 1109 / TCI . 2021.3118944. [118] Tilmann Gneiting and Adrian Raftery. “Strictly Proper Scoring Rules, Prediction, and Estimation.” In: Journal of the American Statistical Association 102 (Mar. 2007), pp. 359–378. doi: 10.1198/016214506000001437. [119] Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. “Diffusion Models as Plug-and-Play Priors.” In: Advances in Neural Information Processing Systems. Ed. by Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho. 2022. url: https://openreview.net/ forum?id=yhlMZ3iR7Pu. [120] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. “Vector Quantized Diffusion Model for Text-to-Image Synthesis.” In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022, pp. 10686–10696. doi: 10. 1109/CVPR52688.2022.01043. [121] Justin P. Haldar, Diego Hernando, and Zhi-Pei Liang. “Compressed-Sensing MRI With Random Encoding.” In: IEEE Transactions on Medical Imaging 30.4 (2011), pp. 893–903. doi: 10.1109/TMI.2010.2085084. [122] Gregg Hallinan et al. “The DSA-2000 — A Radio Survey Camera.” In: Bulletin of the AAS 51.7 (Sept. 2019). https://baas.aas.org/pub/2020n7i255. url: https://arxiv.org/abs/1907.07648. [123] Kerstin Hammernik, Teresa Klatzer, Erich Kobler, Michael P. Recht, Daniel K. Sodickson, Thomas Pock, and Florian Knoll. “Learning a Variational Network for Reconstruction of Accelerated MRI Data.” In: Magnetic Resonance in Medicine 79.6 (2018), pp. 3055–3071. doi: 10.1002/mrm.26977. eprint: https : / / onlinelibrary . wiley . com / doi / pdf / 10 . 1002 / mrm.26977. url: https://onlinelibrary.wiley.com/doi/abs/10. 1002/mrm.26977. [124] Hado van Hasselt, Arthur Guez, and David Silver. “Deep Reinforcement Learning with Double Q-Learning.” In: Proceedings of the Thirtieth AAAI 253 Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA. AAAI Press, 2016, pp. 2094–2100. url: https://aaai.org/ papers/10295-deep-reinforcement-learning-with-double-qlearning/. [125] Wilfred K. Hastings. “Monte Carlo Sampling Methods Using Markov Chains and Their Applications.” In: Biometrika 57.1 (Apr. 1970), pp. 97–109. issn: 0006-3444. doi: 10.1093/biomet/57.1.97. eprint: https:// academic.oup.com/biomet/article-pdf/57/1/97/23940249/571-97.pdf. url: https://doi.org/10.1093/biomet/57.1.97. [126] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. “Deep Residual Learning for Image Recognition.” In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016, pp. 770–778. doi: 10.1109/ CVPR.2016.90. [127] Yinnian He and Weiwei Sun. “Stability and Convergence of the Crank–Nicolson/Adams–Bashforth Scheme for the Time-Dependent Navier–Stokes Equations.” In: SIAM Journal on Numerical Analysis 45.2 (2007), pp. 837–869. doi: 10 . 1137 / 050639910. eprint: https : / / doi . org / 10 . 1137 / 050639910. url: https://doi.org/10.1137/050639910. [128] Tobias Heimann, Bryan J. Morrison, Martin A. Styner, Marc Niethammer, and S. Warfield. “Segmentation of Knee Images: A Grand Challenge.” In: Proc. MICCAI Workshop on Medical Image Analysis for the Clinic. Beijing, China. 2010, pp. 207–214. [129] Bastian Hilder, Mark Peletier, Mikola Schlottke, Guido Schneider, and Upanshu Sharma. “An FIR Inequality for Markov Jump Processes on Discrete State Spaces.” In: (2017). url: https : / / research . tue . nl / en / studentTheses / an - fir - inequality - for - markov - jump processes-on-discrete-state-spa. [130] Bastian Hilder, Mark A. Peletier, Upanshu Sharma, and Oliver Tse. “An Inequality Connecting Entropy Distance, Fisher Information and Large Deviations.” In: Stochastic Processes and their Applications 130.5 (2020), pp. 2596–2638. issn: 0304-4149. doi: 10.1016/j.spa.2019.07.012. url: https : / / www . sciencedirect . com / science / article / pii / S0304414919300687. [131] Jonathan Ho, Ajay Jain, and Pieter Abbeel. “Denoising Diffusion Probabilistic Models.” In: Advances in Neural Information Processing Systems. Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin. Vol. 33. Curran Associates, Inc., 2020, pp. 6840–6851. url: https : / / proceedings . neurips . cc / paper _ files / paper / 2020 / file / 4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf. 254 [132] Jonathan Ho and Tim Salimans. “Classifier-Free Diffusion Guidance.” In: NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications. 2021. url: https : / / openreview . net / forum ? id = qw8AKxfYbI. [133] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. “Video Diffusion Models.” In: Advances in Neural Information Processing Systems. Ed. by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh. Vol. 35. Curran Associates, Inc., 2022, pp. 8633–8646. url: https : / / proceedings . neurips . cc / paper _ files / paper / 2022 / file / 39235c56aef13fb05a6adc95eb9d8d66 Paper-Conference.pdf. [134] Jan A. Högbom and Tim J. Cornwell. “Aperture Synthesis with a NonRegular Distribution of Interferometer Baselines.” In: Astronomy and Astrophysics 500 (1974), pp. 55–66. url: https://api.semanticscholar. org/CorpusID:116934453. [135] Andrew Hoopes, Malte Hoffmann, Bruce Fischl, John Guttag, and Adrian V. Dalca. “HyperMorph: Amortized Hyperparameter Learning for Image Registration.” In: Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28–June 30, 2021, Proceedings. Berlin, Heidelberg: Springer-Verlag, 2021, pp. 3–17. isbn: 978-3-030-78190-3. doi: 10 . 1007 / 978 - 3 - 030 - 78191 - 0 _ 1. url: https://doi.org/10.1007/978-3-030-78191-0_1. [136] Seyed Amir Hossein Hosseini, Burhaneddin Yaman, Steen Moeller, Mingyi Hong, and Mehmet Akçakaya. “Dense Recurrent Neural Networks for Accelerated MRI: History-Cognizant Unrolling of Optimization Algorithms.” In: IEEE Journal of Selected Topics in Signal Processing PP (June 2020), pp. 1–1. doi: 10.1109/JSTSP.2020.3003170. [137] Pieter Houtekamer and Fuqing Zhang. “Review of the Ensemble Kalman Filter for Atmospheric Data Assimilation.” In: Monthly Weather Review 144 (June 2016). doi: 10.1175/MWR-D-15-0440.1. [138] Daniel Huang, Jiaoyang Huang, Sebastian Reich, and Andrew Stuart. “Efficient derivative-free Bayesian inference for large-scale inverse problems.” In: Inverse Problems 38 (Oct. 2022). doi: 10.1088/1361-6420/ac99fa. [139] Daniel Zhengyu Huang, Tapio Schneider, and Andrew M. Stuart. “Iterated Kalman Methodology for Inverse Problems.” In: Journal of Computational Physics 463 (2022), p. 111262. issn: 0021-9991. doi: 10.1016/j.jcp. 2022 . 111262. url: https : / / www . sciencedirect . com / science / article/pii/S0021999122003242. [140] Yue Huang, John Paisley, Qin Lin, Xinghao Ding, Xueyang Fu, and XiaoPing Zhang. “Bayesian Nonparametric Dictionary Learning for Compressed 255 Sensing MRI.” In: IEEE Transactions on Image Processing 23.12 (2014), pp. 5007–5019. doi: 10.1109/TIP.2014.2360122. [141] Yujia Huang, Adishree Ghatare, Yuanzhe Liu, Ziniu Hu, Qinsheng Zhang, Chandramouli Shama Sastry, Siddharth Gururani, Sageev Oore, and Yisong Yue. “Symbolic Music Generation with Non-Differentiable Rule Guided Diffusion.” In: Proceedings of the 41st International Conference on Machine Learning. Ed. by Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp. Vol. 235. Proceedings of Machine Learning Research. PMLR, July 2024, pp. 19772–19797. url: https : / / proceedings . mlr . press / v235 / huang24g.html. [142] Chang Min Hyun, Hwa Pyung Kim, Sung Min Lee, Sungchul Lee, and Jin Keun Seo. “Deep Learning for Undersampled MRI Reconstruction.” In: Physics in Medicine & Biology 63.13 (June 2018), p. 135007. doi: 10.1088/1361-6560/aac71a. url: https://dx.doi.org/10.1088/ 1361-6560/aac71a. [143] Marco Iglesias, Kody Law, and Andrew Stuart. “Ensemble Kalman Methods for Inverse Problems.” In: Inverse Problems 29 (Mar. 2013), p. 045001. doi: 10.1088/0266-5611/29/4/045001. [144] Marco A Iglesias. “A Regularizing Iterative Ensemble Kalman Method for PDE-constrained Inverse Problems.” In: Inverse Problems 32.2 (2016), p. 025002. doi: 10.1088/0266-5611/32/2/025002. [145] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alex Dimakis, and Jonathan Tamir. “Robust Compressed Sensing MRI with Deep Generative Priors.” In: Advances in Neural Information Processing Systems. Ed. by A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan. 2021. url: https://openreview.net/forum?id=wHoIjrT6MMb. [146] Kyong Hwan Jin, Michael Unser, and Kwang Moo Yi. “Self-Supervised Deep Active Accelerated MRI.” In: arXiv preprint arXiv:1901.04547 (2019). [147] Zahra Kadkhodaie and Eero P. Simoncelli. “Stochastic Solutions for Linear Inverse Problems Using the Prior Implicit in a Denoiser.” In: Advances in Neural Information Processing Systems. Ed. by A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan. 2021. url: https://openreview. net/forum?id=x5hh6N9bUUb. [148] Avinash C. Kak and Malcolm Slaney. Principles of Computerized Tomographic Imaging. Society for Industrial and Applied Mathematics, 2001. doi: 10.1137/1.9780898719277. eprint: https://epubs.siam.org/ doi/pdf/10.1137/1.9780898719277. url: https://epubs.siam. org/doi/abs/10.1137/1.9780898719277. 256 [149] Ulugbek Kamilov, Charles Bouman, Gregery Buzzard, and Brendt Wohlberg. “Plug-and-Play Methods for Integrating Physical and Learned Models in Computational Imaging: Theory, Algorithms, and Applications.” In: IEEE Signal Processing Magazine 40 (Jan. 2023), pp. 85–97. doi: 10 . 1109 / MSP.2022.3199595. [150] Ulugbek Kamilov, Ioannis Papadopoulos, Morteza Shoreh, Alexandre Goy, Cedric Vonesch, Michael Unser, and Demetri Psaltis. “Learning Approach to Optical Tomography.” In: Optica 2 (May 2015), pp. 517–522. doi: 10. 1364/OPTICA.2.000517. [151] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. “Elucidating the Design Space of Diffusion-Based Generative Models.” In: Advances in Neural Information Processing Systems. Ed. by Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho. 2022. url: https : //openreview.net/forum?id=k7FuTOWMOc7. [152] Tero Karras, Samuli Laine, and Timo Aila. “A Style-Based Generator Architecture for Generative Adversarial Networks.” In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018), pp. 4396–4405. url: https://api.semanticscholar.org/CorpusID: 54482423. [153] Sergey Kastryulin, Jamil Zakirov, Denis Prokopenko, and Dmitry V. Dylov. PyTorch Image Quality: Metrics for Image Quality Assessment. 2022. doi: 10.48550/ARXIV.2208.14818. url: https://arxiv.org/abs/2208. 14818. [154] Rosemary Kates, David Atkinson, and Michael N. Brant-Zawadzki. “FluidAttenuated Inversion Recovery (FLAIR): Clinical Prospectus of Current and Future Applications.” In: Topics in Magnetic Resonance Imaging: TMRI 86 (1996), pp. 389–396. url: https : / / pubmed . ncbi . nlm . nih . gov / 9402679/. [155] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. “Denoising Diffusion Restoration Models.” In: Advances in Neural Information Processing Systems. Ed. by Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho. 2022. url: https://openreview.net/forum?id= kxXvopt9pWK. [156] Bahjat Kawar, Gregory Vaksman, and Michael Elad. “SNIPS: Solving Noisy Inverse Problems Stochastically.” In: Advances in Neural Information Processing Systems. Ed. by A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan. 2021. url: https : / / openreview . net / forum ? id = pBKOx_dxYAN. [157] Michael Kellman, Emrah Bostan, Michael Chen, and Laura Waller. “DataDriven Design for Fourier Ptychographic Microscopy.” In: 2019 IEEE Inter- 257 national Conference on Computational Photography (ICCP). 2019, pp. 1–8. doi: 10.1109/ICCPHOT.2019.8747339. [158] Michael R. Kellman, Emrah Bostan, Nicole A. Repina, and Laura Waller. “Physics-Based Learned Design: Optimized Coded-Illumination for Quantitative Phase Imaging.” In: IEEE Transactions on Computational Imaging 5.3 (2019), pp. 344–353. doi: 10.1109/TCI.2019.2905434. [159] Rahima Khanam and Muhammad Hussain. YOLOv11: An Overview of the Key Architectural Enhancements. 2024. arXiv: 2410.17725 [cs.CV]. url: https://arxiv.org/abs/2410.17725. [160] Jeongsol Kim, Geon Yeong Park, Hyungjin Chung, and Jong Chul Ye. “Regularization by Texts for Latent Diffusion Inverse Solvers.” In: The Thirteenth International Conference on Learning Representations. 2025. url: https://openreview.net/forum?id=TtUh0TOlGX. [161] Won Jun Kim, Hyungjin Chung, Jaemin Kim, Sangmin Lee, Byeongsu Sim, and Jong Chul Ye. “Derivative-Free Diffusion Manifold-Constrained Gradient for Unified XAI.” In: Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR). June 2025, pp. 23795–23805. url: https://openaccess.thecvf.com/content/CVPR2025/html/Kim_ Derivative-Free_Diffusion_Manifold-Constrained_Gradient_ for_Unified_XAI_CVPR_2025_paper.html. [162] Diederik P. Kingma and Jimmy Ba. “Adam: A Method for Stochastic Optimization.” In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. 2015. url: http:// arxiv.org/abs/1412.6980. [163] Diederik P. Kingma and Max Welling. “Auto-Encoding Variational Bayes.” In: 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. 2014. url: http://arxiv.org/abs/ 1312.6114. [164] Florian Knoll, Tullie Murrell, Anuroop Sriram, Nafissa Yakubova, Jure Zbontar, Michael Rabbat, Aaron Defazio, Matthew J. Muckley, Daniel K. Sodickson, C. Lawrence Zitnick, and Michael P. Recht. “Advancing Machine Learning for MR Image Reconstruction with an Open Competition: Overview of the 2019 fastMRI Challenge.” In: Magnetic Resonance in Medicine 84.6 (2020), pp. 3054–3070. doi: 10.1002/mrm.28338. eprint: https : / / onlinelibrary . wiley . com / doi / pdf / 10 . 1002 / mrm . 28338. url: https : / / onlinelibrary . wiley . com / doi / abs / 10 . 1002/mrm.28338. 258 [165] Florian Knoll et al. “fastMRI: A Publicly Available Raw K-space and DICOM Dataset of Knee Images for Accelerated MR Image Reconstruction Using Machine Learning.” In: Radiology: Artificial Intelligence 2.1 (2020). PMID: 32076662, e190007. doi: 10 . 1148 / ryai . 2020190007. eprint: https://doi.org/10.1148/ryai.2020190007. url: https://doi. org/10.1148/ryai.2020190007. [166] Nikola Kovachki and Andrew Stuart. “Ensemble Kalman Inversion: A Derivative-Free Technique for Machine Learning Tasks.” In: Inverse Problems 35 (Aug. 2019). doi: 10.1088/1361-6420/ab1c3a. [167] Taesung Kwon, Gookho Song, Yoosun Kim, Jong Chul Ye, and Mooseok Jang. Seeing Video Through Optical Scattering Media Using Spatio-Temporal Diffusion Models. 2024. url: https://openreview.net/forum?id= DHCp41nv1M. [168] Taesung Kwon and Jong Chul Ye. “Solving Video Inverse Problems Using Image Diffusion Models.” In: The Thirteenth International Conference on Learning Representations. 2025. url: https://openreview.net/ forum?id=TRWxFUzK9K. [169] Taesung Kwon and Jong Chul Ye. VISION-XL: High Definition Video Inverse Problem Solver Using Latent Image Diffusion Models. 2024. arXiv: 2412.00156 [cs.CV]. url: https://arxiv.org/abs/2412.00156. [170] Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. “Bayesian Imaging Using Plug & Play Priors: When Langevin Meets Tweedie.” In: SIAM Journal on Imaging Sciences 15.2 (2022), pp. 701–737. doi: 10 . 1137 / 21M1406349. eprint: https : //doi.org/10.1137/21M1406349. url: https://doi.org/10.1137/ 21M1406349. [171] Paul C. Lauterbur. “Image Formation by Induced Local Interactions: Examples Employing Nuclear Magnetic Resonance.” In: Nature 242 (1973), pp. 190–191. doi: 10.1038/242190a0. [172] Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. “GradientBased Learning Applied to Document Recognition.” In: Proceedings of the IEEE 86.11 (1998), pp. 2278–2324. doi: 10.1109/5.726791. [173] Dongwook Lee, Jaejun Yoo, Sungho Tak, and Jong Chul Ye. “Deep Residual Learning for Accelerated MRI Using Magnitude and Phase Networks.” In: IEEE Transactions on Biomedical Engineering 65.9 (2018), pp. 1985–1995. doi: 10.1109/TBME.2018.2821699. [174] Dongwook Lee, Jaejun Yoo, and Jong Chul Ye. “Deep Residual Learning for Compressed Sensing MRI.” In: 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017). 2017, pp. 15–18. doi: 10.1109/ISBI. 2017.7950457. 259 [175] Holden Lee, Jianfeng Lu, and Yixin Tan. “Convergence for Score-Based Generative Modeling with Polynomial Complexity.” In: Advances in Neural Information Processing Systems. Ed. by Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho. 2022. url: https://openreview. net/forum?id=dUSI4vFyMK. [176] Yin Tat Lee, Ruoqi Shen, and Kevin Tian. “Structured Logconcave Sampling with a Restricted Gaussian Oracle.” In: Proceedings of Thirty Fourth Conference on Learning Theory. Vol. 134. Proceedings of Machine Learning Research. PMLR, Aug. 2021, pp. 2993–3050. url: https://proceedings. mlr.press/v134/lee21a.html. [177] Victor Lempitsky, Andrea Vedaldi, and Dmitry Ulyanov. “Deep Image Prior.” In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, pp. 9446–9454. doi: 10.1109/CVPR.2018.00984. [178] Aviad Levis, Daeyoung Lee, Joel A. Tropp, Charles F. Gammie, and Katherine L. Bouman. “Inference of Black Hole Fluid-Dynamics from Sparse Interferometric Measurements.” In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). 2021, pp. 2320–2329. doi: 10.1109/ ICCV48922.2021.00234. [179] Lianlin Li, Long Gang Wang, Fernando L. Teixeira, Che Liu, Arye Nehorai, and Tie Jun Cui. “DeepNIS: Deep Neural Network for Nonlinear Electromagnetic Inverse Scattering.” In: IEEE Transactions on Antennas and Propagation 67.3 (2019), pp. 1819–1825. doi: 10.1109/TAP.2018.2885437. [180] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization.” In: Journal of Machine Learning Research 18.185 (2018), pp. 1–52. url: http://jmlr.org/papers/v18/16-558.html. [181] Xiang Li, Soo Min Kwon, Ismail R. Alkhouri, Saiprasad Ravishankar, and Qing Qu. Decoupled Data Consistency with Diffusion Purification for Image Restoration. 2024. arXiv: 2403.06054 [eess.IV]. url: https://arxiv. org/abs/2403.06054. [182] Xiner Li, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokcen Eraslan, Surag Nair, Tommaso Biancalani, Shuiwang Ji, Aviv Regev, Sergey Levine, and Masatoshi Uehara. Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding. 2024. arXiv: 2408 . 08252 [cs.LG]. url: https://arxiv.org/abs/2408.08252. [183] Yunzhe Li, Yujia Xue, and Lei Tian. “Deep Speckle Correlation: A Deep Learning Approach toward Scalable Imaging through Scattering Media.” In: Optica 5 (Sept. 2018), pp. 1181–1190. doi: 10.1364/OPTICA.5.001181. [184] Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. “Fourier 260 Neural Operator for Parametric Partial Differential Equations.” In: International Conference on Learning Representations. 2021. url: https : / / openreview.net/forum?id=c8P9NQVtmnO. [185] Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. “Physicsinformed Neural Operator for Learning Partial Differential Equations.” In: ACM/JMS Journal of Data Science 1.3 (2024), pp. 1–27. doi: 10.1145/ 3648506. [186] Zhi-Pei Liang and Paul C. Lauterbur. Principles of Magnetic Resonance Imaging: A Signal Processing Perspective. IEEE Press series in biomedical engineering. SPIE Optical Engineering Press, 2000. isbn: 9780819435163. url: https://books.google.com/books?id=sRyEQgAACAAJ. [187] Dong C. Liu and Jorge Nocedal. “On the Limited Memory BFGS Method for Large Scale Optimization.” In: Mathematical Programming 45.1–3 (Aug. 1989), pp. 503–528. issn: 0025-5610. doi: 10.1007/BF01589116. [188] Hsiou-Yuan Liu, Dehong Liu, Hassan Mansour, Petros T. Boufounos, Laura Waller, and Ulugbek S. Kamilov. “SEAGLE: Sparsity-Driven Image Reconstruction Under Multiple Scattering.” In: IEEE Transactions on Computational Imaging 4.1 (2018), pp. 73–86. doi: 10.1109/TCI.2017.2764461. [189] Jiaming Liu, Rushil Anirudh, Jayaraman J. Thiagarajan, Stewart He, K. Aditya Mohan, Ulugbek S. Kamilov, and Hyojin Kim. “DOLCE: A Model-Based Probabilistic Diffusion Framework for Limited-Angle CT Reconstruction.” In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). 2023, pp. 10464–10474. doi: 10.1109/ICCV51070.2023.00963. [190] Jiaming Liu, Yu Sun, Cihat Eldeniz, Weijie Gan, Hongyu An, and Ulugbek S. Kamilov. “RARE: Image Reconstruction Using Deep Priors Learned Without Groundtruth.” In: IEEE Journal of Selected Topics in Signal Processing 14.6 (2020), pp. 1088–1099. doi: 10.1109/JSTSP.2020.2998402. [191] Jiaming Liu, Yu Sun, Weijie Gan, Xiaojian Xu, Brendt Wohlberg, and Ulugbek S. Kamilov. “SGD-Net: Efficient Model-Based Deep Learning With Theoretical Guarantees.” In: IEEE Transactions on Computational Imaging 7 (2021), pp. 598–610. doi: 10.1109/TCI.2021.3085534. [192] Lijun Liu and Michael Gurnis. “Simultaneous Inversion of Mantle Properties and Initial Conditions Using an Adjoint of Mantle Convection.” In: Journal of Geophysical Research: Solid Earth 113.B8 (2008). doi: 10.1029/ 2008JB005594. eprint: https : / / agupubs . onlinelibrary . wiley . com / doi / pdf / 10 . 1029 / 2008JB005594. url: https : / / agupubs . onlinelibrary.wiley.com/doi/abs/10.1029/2008JB005594. [193] Renhao Liu, Yu Sun, Jiabei Zhu, Lei Tian, and Ulugbek Kamilov. “Recovery of Continuous 3D Refractive Index Maps from Discrete Intensity-Only Mea- 261 surements Using Neural Fields.” In: Nature Machine Intelligence 4 (Sept. 2022), pp. 1–11. doi: 10.1038/s42256-022-00530-3. [194] Yaning Liu, Fenglin Niu, Min Chen, and Wencai Yang. “3-D Crustal and Uppermost Mantle Structure beneath NE China Revealed by Ambient Noise Adjoint Tomography.” In: Earth and Planetary Science Letters 461 (2017), pp. 20–29. issn: 0012-821X. doi: 10 . 1016 / j . epsl . 2016 . 12 . 029. url: https : / / www . sciencedirect . com / science / article / pii / S0012821X16307427. [195] Yiming Liu, Yanwei Pang, Ruiqi Jin, Yonghong Hou, and Xuelong Li. “Reinforcement Learning and Transformer for Fast Magnetic Resonance Imaging Scan.” In: IEEE Transactions on Emerging Topics in Computational Intelligence 8.3 (2024), pp. 2310–2323. doi: 10.1109/TETCI.2024.3358180. [196] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. “Deep Learning Face Attributes in the Wild.” In: 2015 IEEE International Conference on Computer Vision (ICCV). 2015, pp. 3730–3738. doi: 10 . 1109 / ICCV . 2015.425. [197] Ilya Loshchilov and Frank Hutter. “Decoupled Weight Decay Regularization.” In: International Conference on Learning Representations. 2019. url: https://openreview.net/forum?id=Bkg6RiCqY7. [198] Aaron Lou, Chenlin Meng, and Stefano Ermon. “Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution.” In: Forty-first International Conference on Machine Learning. 2024. url: https : / / openreview.net/forum?id=CNicRIVIPA. [199] Mathias Louboutin, Michael Lange, Fabio Luporini, Navjot Kukreja, Philipp A. Witte, Felix J. Herrmann, Paulius Velesko, and Gerard J. Gorman. “Devito (v3.1.0): An Embedded Domain-specific Language for Finite Differences and Geophysical Exploration.” In: Geoscientific Model Development 12.3 (2019), pp. 1165–1187. doi: 10.5194/gmd- 12- 1165- 2019. url: https://www.geosci-model-dev.net/12/1165/2019/. [200] Jan Lukas, Jessica Fridrich, and Miroslav Goljan. “Digital Camera Identification from Sensor Pattern Noise.” In: IEEE Transactions on Information Forensics and Security 1.2 (2006), pp. 205–214. doi: 10 . 1109 / TIFS . 2006.873602. [201] Guanxiong Luo, Moritz Blumenthal, Martin Heide, and Martin Uecker. “Bayesian MRI Reconstruction with Joint Uncertainty Estimation Using Diffusion Models.” In: Magnetic Resonance in Medicine 90 (Mar. 2023), pp. 295–311. doi: 10.1002/mrm.29624. [202] Guanxiong Luo, Na Zhao, Wenhao Jiang, Edward Hui, and Peng Cao. “MRI Reconstruction Using Deep Bayesian Estimation.” In: Magnetic Resonance in Medicine 84 (Apr. 2020), pp. 2246–2261. doi: 10.1002/mrm.28274. 262 [203] Yanchen Luo, Junfeng Fang, Sihang Li, Zhiyuan Liu, Jiancan Wu, An Zhang, Wenjie Du, and Xiang Wang. “Text-Guided Small Molecule Generation via Diffusion Model.” In: iScience 27 (Sept. 2024), p. 110992. doi: 10.1016/ j.isci.2024.110992. [204] Michael Lustig, David Donoho, and John Pauly. “Sparse MRI: The Application of Compressed Sensing for Rapid MR Imaging.” In: Magnetic resonance in medicine : official journal of the Society of Magnetic Resonance in Medicine / Society of Magnetic Resonance in Medicine 58 (Dec. 2007), pp. 1182–95. doi: 10.1002/mrm.21391. [205] Michael Lustig, David L. Donoho, Juan M. Santos, and John M. Pauly. “Compressed Sensing MRI.” In: IEEE Signal Processing Magazine 25.2 (2008), pp. 72–82. doi: 10.1109/MSP.2007.914728. [206] Yi-An Ma, Tianqi Chen, and Emily Fox. “A Complete Recipe for Stochastic Gradient MCMC.” In: Advances in Neural Information Processing Systems. Ed. by C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett. Vol. 28. Curran Associates, Inc., 2015. url: https://proceedings.neurips. cc/paper_files/paper/2015/file/9a4400501febb2a95e79248486a5f6d3Paper.pdf. [207] Shiqian Ma, Wotao Yin, Yin Zhang, and Amit Chakraborty. “An Efficient Algorithm for Compressed MR Imaging Using Total Variation and Wavelets.” In: 2008 IEEE Conference on Computer Vision and Pattern Recognition. 2008, pp. 1–8. doi: 10.1109/CVPR.2008.4587391. [208] Yanting Ma, Hassan Mansour, Dehong Liu, Petros T. Boufounos, and Ulugbek S. Kamilov. “Accelerated Image Reconstruction for Nonlinear Diffractive Imaging.” In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2018, pp. 6473–6477. doi: 10.1109/ ICASSP.2018.8462400. [209] Andrew L. Maas, Awni Y. Hannun, Andrew Y. Ng, et al. “Rectifier Nonlinearities Improve Neural Network Acoustic Models.” In: Proceedings of the Thirtieth International Conference on Machine Learning (ICML). Vol. 30. 1. Atlanta, GA. 2013, p. 3. [210] Laurens van der Maaten and Geoffrey Hinton. “Visualizing Data Using tSNE.” In: Journal of Machine Learning Research 9.86 (2008), pp. 2579– 2605. url: http://jmlr.org/papers/v9/vandermaaten08a.html. [211] Ross Maguire, Brandon Schmandt, Jiaqi Li, Chengxin Jiang, Guoliang Li, Justin Wilgus, and Min Chen. “Magma Accumulation at Depths of Prior Rhyolite Storage beneath Yellowstone Caldera.” In: Science (New York, N.Y.) 378 (Dec. 2022), pp. 1001–1004. doi: 10.1126/science.ade0347. [212] Sébastien Marcel and Yann Rodriguez. “Torchvision the Machine-Vision Package of Torch.” In: Proceedings of the 18th ACM International Conference on Multimedia. MM ’10. Firenze, Italy: Association for Computing 263 Machinery, 2010, pp. 1485–1488. isbn: 9781605589336. doi: 10.1145/ 1873951 . 1874254. url: https : / / doi . org / 10 . 1145 / 1873951 . 1874254. [213] Daniel Marcus, Tracy Wang, Jamie Parker, John Csernansky, John Morris, and Randy Buckner. “Open Access Series of Imaging Studies (OASIS): Cross-Sectional MRI Data in Young, Middle Aged, Nondemented, and Demented Older Adults.” In: Journal of Cognitive Neuroscience 19 (Oct. 2007), pp. 1498–507. doi: 10.1162/jocn.2007.19.9.1498. [214] Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. “A Variational Perspective on Solving Inverse Problems with Diffusion Models.” In: The Twelfth International Conference on Learning Representations. 2024. url: https://openreview.net/forum?id=1YO4EE3SPB. [215] Ségolène Martin, Anne Gagneux, Paul Hagemann, and Gabriele Steidl. PnPFlow: Plug-and-Play Image Restoration with Flow Matching. 2024. arXiv: 2410.02423 [cs.CV]. url: https://arxiv.org/abs/2410.02423. [216] Filippo Martinini, Mauro Mangia, Alex Marchioni, Riccardo Rovatti, and Gianluca Setti. “A Deep Learning Method for Optimal Undersampling Patterns and Image Recovery for MRI Exploiting Losses and Projections.” In: IEEE Journal of Selected Topics in Signal Processing 16.4 (2022), pp. 713– 724. doi: 10.1109/JSTSP.2022.3171082. [217] Tim Meinhardt, Michael Moeller, Caner Hazirbas, and Daniel Cremers. “Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems.” In: 2017 IEEE International Conference on Computer Vision (ICCV). 2017, pp. 1799–1808. doi: 10 . 1109 / ICCV . 2017.198. [218] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. “SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations.” In: International Conference on Learning Representations. 2021. url: https://api.semanticscholar. org/CorpusID:245704504. [219] Bjoern Menze, András Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahaniy, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboomy, Roland Wiest, Levente Lancziy, Elizabeth Gerstnery, Marc-Andr´e Webery, Tal Arbel, Brian Avants, Nicholas Ayache, Patricia Buendia, Louis Collins, Nicolas Cordier, and Koen Van Leemput. “The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS).” In: IEEE Transactions on Medical Imaging 99 (Dec. 2014). doi: 10.1109/TMI.2014.2377694. [220] Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Edward Teller. “Equation of State Calculations by Fast Computing Machines.” In: The Journal of Chemical Physics 21.6 (June 1953), pp. 1087–1092. issn: 0021-9606. doi: 10.1063/1.1699114. eprint: 264 https : / / pubs . aip . org / aip / jcp / article - pdf / 21 / 6 / 1087 / 18802390/1087\_1\_online.pdf. url: https://doi.org/10.1063/ 1.1699114. [221] Christopher Metzler, Phillip Schniter, Ashok Veeraraghavan, and Richard Baraniuk. “prDeep: Robust Phase Retrieval with a Flexible Deep Network.” In: Proceedings of the 35th International Conference on Machine Learning. Ed. by Jennifer Dy and Andreas Krause. Vol. 80. Proceedings of Machine Learning Research. PMLR, July 2018, pp. 3501–3510. url: https : / / proceedings.mlr.press/v80/metzler18a.html. [222] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. “V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation.” In: 2016 Fourth International Conference on 3D Vision (3DV). 2016, pp. 565–571. doi: 10.1109/3DV.2016.79. [223] Yosuke Mizuno. “GRMHD Simulations and Modeling for Jet Formation and Acceleration Region in AGNs.” In: Universe 8.2 (Jan. 2022), p. 85. issn: 2218-1997. doi: 10.3390/universe8020085. url: http://dx. doi.org/10.3390/universe8020085. [224] Alexander Quinn Nichol and Prafulla Dhariwal. “Improved Denoising Diffusion Probabilistic Models.” In: Proceedings of the 38th International Conference on Machine Learning. Ed. by Marina Meila and Tong Zhang. Vol. 139. Proceedings of Machine Learning Research. PMLR, July 2021, pp. 8162– 8171. url: https : / / proceedings . mlr . press / v139 / nichol21a . html. [225] Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. “Unlocking Guidance for Discrete State-Space Diffusion and Flow Models.” In: The Thirteenth International Conference on Learning Representations. 2025. url: https://openreview.net/forum?id=XsgHl54yO7. [226] Nikolas Nüsken and Sebastian Reich. Note on Interacting Langevin Diffusions: Gradient Structure and Ensemble Kalman Sampler by GarbunoInigo, Hoffmann, Li and Stuart. 2019. arXiv: 1908 . 10890 [math.DS]. url: https://arxiv.org/abs/1908.10890. [227] Dean Oliver and Yan Chen. “Recent Progress on Reservoir History Matching: A Review.” In: Computational Geosciences 15 (July 2011), pp. 185– 221. doi: 10.1007/s10596-010-9194-2. [228] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. “Neural Discrete Representation Learning.” In: Advances in Neural Information Processing Systems. Ed. by I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett. Vol. 30. Curran Associates, Inc., 2017. url: https://proceedings.neurips.cc/paper_ files / paper / 2017 / file / 7a98af17e63a0ac09ce2e96d03992fbc Paper.pdf. 265 [229] Wei Peng, Li Feng, Guoying Zhao, and Fang Liu. “Learning Optimal K-space Acquisition and Reconstruction Using Physics-Informed Neural Networks.” In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022, pp. 20762–20771. doi: 10.1109/CVPR52688.2022. 02013. [230] Marcelo Pereyra, Luis A. Vargas-Mieles, and Konstantinos C. Zygalakis. “The Split Gibbs Sampler Revisited: Improvements to Its Algorithmic Structure and Augmented Target Distribution.” In: SIAM Journal on Imaging Sciences 16.4 (2023), pp. 2040–2071. doi: 10.1137/22M1506122. eprint: https://doi.org/10.1137/22M1506122. url: https://doi.org/ 10.1137/22M1506122. [231] Luis Pineda, Sumana Basu, Adriana Romero, Roberto Calandra, and Michal Drozdzal. “Active MR K-Space Sampling with Reinforcement Learning.” In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. Ed. by Anne L. Martel, Purang Abolmaesumi, Danail Stoyanov, Diana Mateus, Maria A. Zuluaga, S. Kevin Zhou, Daniel Racoceanu, and Leo Joskowicz. Cham: Springer International Publishing, 2020, pp. 23–33. isbn: 978-3-030-59713-9. doi: 10.1007/978-3-030-59713-9_3. [232] Rene-Edouard Plessix. “A Review of the Adjoint-State Method for Computing the Gradient of a Functional with Geophysical Applications.” In: Geophysical Journal International 167.2 (2006), pp. 495–503. doi: 10 . 1111/j.1365-246X.2006.02978.x. [233] Oliver Porth et al. “The Event Horizon General Relativistic Magnetohydrodynamic Code Comparison Project.” In: The Astrophysical Journal Supplement Series 243.2 (Aug. 2019), p. 26. doi: 10.3847/1538-4365/ab29fd. url: https://dx.doi.org/10.3847/1538-4365/ab29fd. [234] Oula Puonti, Juan Iglesias, and Koen Van Leemput. “Fast and SequenceAdaptive Whole-Brain Segmentation Using Parametric Bayesian Modeling.” In: NeuroImage 143 (Sept. 2016). doi: 10.1016/j.neuroimage. 2016.09.011. [235] Tran Minh Quan, Thanh Nguyen-Duc, and Won-Ki Jeong. “Compressed Sensing MRI Reconstruction Using a Generative Adversarial Network With a Cyclic Loss.” In: IEEE Transactions on Medical Imaging 37.6 (2018), pp. 1488–1497. doi: 10.1109/TMI.2018.2820120. [236] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. “Learning Transferable Visual Models From Natural Language Supervision.” In: Proceedings of the 38th International Conference on Machine Learning. Ed. by Marina Meila and Tong Zhang. Vol. 139. Proceedings of Machine Learning Research. PMLR, July 2021, pp. 8748–8763. url: https://proceedings.mlr. press/v139/radford21a.html. 266 [237] Chaithya G. Radhakrishna, Zaccharie Ramzi, and Philippe Ciuciu. “Hybrid Learning of Non-Cartesian K-Space Trajectory and MR Image Reconstruction Networks.” In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI). 2022, pp. 1–5. doi: 10.1109/ISBI52829.2022.9761408. [238] Archit Raj, Srikrishnan Vishwanathan, Bhavya Ajani, Karthik Krishnan, and Harsh Agarwal. “Automatic Knee Cartilage Segmentation Using Fully Volumetric Convolutional Neural Networks for Evaluation of Osteoarthritis.” In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). 2018, pp. 851–854. doi: 10.1109/ISBI.2018.8363705. [239] Rajikha Raja and Neelam Sinha. “Adaptive K-Space Sampling Design for Edge-Enhanced DCE-MRI Using Compressed Sensing.” In: Magnetic Resonance Imaging 32 (Sept. 2014). doi: 10.1016/j.mri.2013.12.022. [240] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents. 2022. arXiv: 2204.06125 [cs.CV]. url: https://arxiv.org/abs/ 2204.06125. [241] Stephan Rasp, Stephan Hoyer, Alexander Merose, Ian Langmore, Peter Battaglia, Tyler Russell, Alvaro Sanchez-Gonzalez, Vivian Yang, Rob Carver, Shreya Agrawal, Matthew Chantry, Zied Ben Bouallegue, Peter Dueben, Carla Bromberg, Jared Sisk, Luke Barrington, Aaron Bell, and Fei Sha. “WeatherBench 2: A Benchmark for the Next Generation of Data-Driven Global Weather Models.” In: Journal of Advances in Modeling Earth Systems 16.6 (2024). e2023MS004019 2023MS004019, e2023MS004019. doi: 10.1029/2023MS004019. eprint: https://agupubs.onlinelibrary. wiley . com / doi / pdf / 10 . 1029 / 2023MS004019. url: https : / / agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2023MS004019. [242] Saiprasad Ravishankar and Yoram Bresler. “Adaptive Sampling Design for Compressed Sensing MRI.” In: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2011, pp. 3751– 3755. doi: 10.1109/IEMBS.2011.6090639. [243] Saiprasad Ravishankar and Yoram Bresler. “MR Image Reconstruction From Highly Undersampled K-Space Data by Dictionary Learning.” In: IEEE Transactions on Medical Imaging 30.5 (2011), pp. 1028–1041. doi: 10.1109/TMI.2010.2090538. [244] Benjamin Remy, Francois Lanusse, Niall Jeffrey, Jia Liu, Jean-Luc Starck, Ken Osato, and Tim Schrabback. “Probabilistic Mass-Mapping with Neural Score Estimation.” In: Astronomy & Astrophysics 672 (Dec. 2022). doi: 10.1051/0004-6361/202243054. [245] Yinuo Ren, Haoxuan Chen, Grant M. Rotskoff, and Lexing Ying. “How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Framework.” In: The Thirteenth 267 International Conference on Learning Representations. 2025. url: https: //openreview.net/forum?id=6awxwQEI82. [246] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. “Stochastic Backpropagation and Approximate Inference in Deep Generative Models.” In: Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014. Vol. 32. JMLR Workshop and Conference Proceedings. JMLR.org, 2014, pp. 1278–1286. url: http: //proceedings.mlr.press/v32/rezende14.html. [247] Gareth O. Roberts and Richard L. Tweedie. “Exponential Convergence of Langevin Distributions and Their Discrete Approximations.” In: Bernoulli 2.4 (1996), pp. 341–363. doi: 10.2307/3318418. [248] Yaniv Romano, Michael Elad, and Peyman Milanfar. “The Little Engine that Could: Regularization by Denoising (RED).” In: SIAM Journal on Imaging Sciences 10 (Nov. 2016). doi: 10.1137/16M1102884. [249] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. “High-Resolution Image Synthesis with Latent Diffusion Models.” In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022, pp. 10674–10685. doi: 10.1109/CVPR52688. 2022.01042. [250] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. “U-Net: Convolutional Networks for Biomedical Image Segmentation.” In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Springer. 2015, pp. 234–241. doi: 10.1007/978-3-31924574-4_28. [251] Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. “Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models.” In: Thirty-seventh Conference on Neural Information Processing Systems. 2023. url: https: //openreview.net/forum?id=XKBFdYwfRo. [252] Ernest K. Ryu, Jialin Liu, Sicheng Wang, Xiaohan Chen, Zhangyang Wang, and Wotao Yin. “Plug-and-Play Methods Provably Converge with Properly Trained Denoisers.” In: International Conference on Machine Learning. 2019. url: https : / / api . semanticscholar . org / CorpusID : 153311703. [253] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. “Image Super-Resolution via Iterative Refinement.” In: IEEE Transactions on Pattern Analysis and Machine Intelligence 45.4 (2023), pp. 4713–4726. doi: 10.1109/TPAMI.2022.3204461. [254] Fritz Schick. “Tissue Segmentation: A Crucial Tool for Quantitative MRI and Visualization of Anatomical Structures.” In: Magma (New York, N.Y.) 29 (Apr. 2016). doi: 10.1007/s10334-016-0549-0. 268 [255] Jo Schlemper, Jose Caballero, Joseph V. Hajnal, Anthony N. Price, and Daniel Rueckert. “A Deep Cascade of Convolutional Neural Networks for Dynamic MR Image Reconstruction.” In: IEEE Transactions on Medical Imaging 37.2 (2018), pp. 491–503. doi: 10.1109/TMI.2017.2760978. [256] Martin Shepherd. “Difmap: An Interactive Program for Synthesis Imaging.” In: Astronomical Data Analysis Software and Systems VI. Vol. 125. 1997, p. 77. url: https://www.cv.nrao.edu/adass/adassVI/shepherdm. html. [257] Martin Shepherd. “Difmap: Synthesis Imaging of Visibility Data.” In: Astrophysics Source Code Library (2011), p. 1103. url: https://ascl. net/1103.001. [258] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. “Mastering the Game of Go with Deep Neural Networks and Tree Search.” In: Nature 529.7587 (Jan. 2016), pp. 484–489. doi: 10.1038/nature16961. [259] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. “Deep Unsupervised Learning Using Nonequilibrium Thermodynamics.” In: Proceedings of the 32nd International Conference on Machine Learning. Ed. by Francis Bach and David Blei. Vol. 37. Proceedings of Machine Learning Research. Lille, France: PMLR, July 2015, pp. 2256–2265. url: https://proceedings.mlr.press/v37/sohl- dickstein15. html. [260] Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. “Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency.” In: The Twelfth International Conference on Learning Representations. 2024. url: https://openreview.net/forum?id= j8hdRqOUhN. [261] Jiaming Song, Chenlin Meng, and Stefano Ermon. “Denoising Diffusion Implicit Models.” In: International Conference on Learning Representations. 2021. url: https://openreview.net/forum?id=St1giarCHLP. [262] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. “PseudoinverseGuided Diffusion Models for Inverse Problems.” In: International Conference on Learning Representations. 2023. url: https : / / openreview . net/forum?id=9_gsMA8MRKQ. [263] Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, MingYu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. “Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation.” In: Proceedings of the 40th International Conference on Machine Learning. Ed. by 269 Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett. Vol. 202. Proceedings of Machine Learning Research. PMLR, July 2023, pp. 32483–32498. url: https : //proceedings.mlr.press/v202/song23k.html. [264] Yang Song and Stefano Ermon. “Generative Modeling by Estimating Gradients of the Data Distribution.” In: Neural Information Processing Systems. 2019. url: https : / / api . semanticscholar . org / CorpusID : 196470871. [265] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. “Solving Inverse Problems in Medical Imaging with Score-Based Generative Models.” In: International Conference on Learning Representations. 2022. url: https: //openreview.net/forum?id=vaRCHVj0uGI. [266] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. “Score-Based Generative Modeling through Stochastic Differential Equations.” In: International Conference on Learning Representations. 2021. url: https://openreview.net/forum?id= PxTIG12RRHS. [267] Anuroop Sriram, Jure Zbontar, Tullie Murrell, Aaron Defazio, C. Lawrence Zitnick, Nafissa Yakubova, Florian Knoll, and Patricia Johnson. “End-toEnd Variational Networks for Accelerated MRI Reconstruction.” In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part II. Lima, Peru: Springer-Verlag, 2020, pp. 64–73. isbn: 978-3030-59712-2. doi: 10 . 1007 / 978 - 3 - 030 - 59713 - 9 _ 7. url: https : //doi.org/10.1007/978-3-030-59713-9_7. [268] Andrew Stuart. “Inverse Problems: A Bayesian Perspective.” In: Acta Numerica 19 (May 2010), pp. 451–559. doi: 10.1017/S0962492910000061. [269] He Sun and Katherine L. Bouman. “Deep Probabilistic Imaging: Uncertainty Quantification and Multi-Modal Solution Characterization for Computational Imaging.” In: Proceedings of the AAAI Conference on Artificial Intelligence 35.3 (May 2021), pp. 2628–2637. doi: 10.1609/aaai.v35i3. 16366. url: https://ojs.aaai.org/index.php/AAAI/article/ view/16366. [270] He Sun, Adrian V. Dalca, and Katherine L. Bouman. “Learning a Probabilistic Strategy for Computational Imaging Sensor Selection.” In: 2020 IEEE International Conference on Computational Photography (ICCP). 2020, pp. 1–12. doi: 10.1109/ICCP48838.2020.9105133. [271] Liyan Sun, Zhiwen Fan, Xinghao Ding, Yue Huang, and John Paisley. “Joint CS-MRI Reconstruction and Segmentation with a Unified Deep Network.” In: Information Processing in Medical Imaging. Ed. by Albert C. S. Chung, James C. Gee, Paul A. Yushkevich, and Siqi Bao. Cham: Springer In- 270 ternational Publishing, 2019, pp. 492–504. isbn: 978-3-030-20351-1. doi: 10.1007/978-3-030-20351-1_38. [272] Yu Sun, Brendt Wohlberg, and Ulugbek S. Kamilov. “An Online Plug-andPlay Algorithm for Regularized Image Reconstruction.” In: IEEE Transactions on Computational Imaging 5.3 (2019), pp. 395–408. doi: 10.1109/ TCI.2019.2893568. [273] Yu Sun, Zihui Wu, Yifan Chen, Berthy T. Feng, and Katherine L. Bouman. “Provable Probabilistic Imaging Using Score-Based Generative Priors.” In: IEEE Transactions on Computational Imaging 10 (2024), pp. 1290–1305. doi: 10.1109/TCI.2024.3449114. [274] Yu Sun, Zhihao Xia, and Ulugbek Kamilov. “Efficient and Accurate Inversion of Multiple Scattering with Deep Learning.” In: Optics Express 26 (May 2018), pp. 14678–14688. doi: 10.1364/OE.26.014678. [275] Yongjin Sung, Wonshik Choi, Christopher Fang-Yen, Kamran Badizadegan, Ramachandra Dasari, and Michael Feld. “Optical Diffraction Tomography for High Resolution Live Cell Imaging.” In: Optics Express 17 (Jan. 2009), pp. 266–277. doi: 10.1364/OE.17.000266. [276] Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. “PDEBench: An Extensive Benchmark for Scientific Machine Learning.” In: Advances in Neural Information Processing Systems. Ed. by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh. Vol. 35. Curran Associates, Inc., 2022, pp. 1596–1611. url: https : / / proceedings . neurips . cc / paper _ files / paper / 2022 / file / 0a9747136d411fb83f0cf81820d44afb Paper-Datasets_and_Benchmarks.pdf. [277] Haoyue Tang, Tian Xie, Aosong Feng, Hanyu Wang, Chenyang Zhang, and Yang Bai. “Solving General Noisy Inverse Problem via Posterior Sampling: A Policy Gradient Viewpoint.” In: Proceedings of The 27th International Conference on Artificial Intelligence and Statistics. Ed. by Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li. Vol. 238. Proceedings of Machine Learning Research. PMLR, May 2024, pp. 2116–2124. url: https: //proceedings.mlr.press/v238/tang24b.html. [278] Zachary Teed and Jia Deng. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. 2020. arXiv: 2003.12039 [cs.CV]. url: https://arxiv. org/abs/2003.12039. [279] Kerem Tezcan, Christian Baumgartner, Roger Luechinger, Klaas Pruessmann, and Ender Konukoglu. “MR Image Reconstruction Using Deep Density Priors.” In: IEEE Transactions on Medical Imaging PP (Dec. 2018), pp. 1–1. doi: 10.1109/TMI.2018.2887072. 271 [280] A. Richard. Thompson, James M. Moran, and SpringerLink (Online service). Interferometry and Synthesis in Radio Astronomy. 3rd edition 2017. Astronomy and Astrophysics Library. Springer International Publishing, 2017. doi: 10.1007/978-3-319-44431-4. [281] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. “Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction.” In: The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024. url: https://openreview.net/forum?id= gojL67CfS8. [282] Lei Tian and Laura Waller. “3D Intensity and Phase Imaging from Light Field Measurements in an LED Array Microscope.” In: Optica 2 (Jan. 2015), pp. 104–111. doi: 10.1364/OPTICA.2.000104. [283] Brian L. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi S. Jaakkola. “Diffusion Probabilistic Modeling of Protein Backbones in 3D for the Motif-scaffolding Problem.” In: The Eleventh International Conference on Learning Representations. 2023. url: https://openreview.net/forum?id=6TxBxqNME1Y. [284] Ben Tunbridge, Ian Harrison, and Michael L. Brown. “Radio–Optical Galaxy Shape Correlations in the COSMOS Field.” In: Monthly Notices of the Royal Astronomical Society 463.3 (Sept. 2016), pp. 3339–3353. issn: 0035-8711. doi: 10.1093/mnras/stw2224. eprint: https://academic.oup.com/ mnras / article - pdf / 463 / 3 / 3339 / 18243295 / stw2224 . pdf. url: https://doi.org/10.1093/mnras/stw2224. [285] Martin Uecker, Peng Lai, Mark Murphy, Patrick Virtue, Michael Elad, Shreyas Vasanawala, and Michael Lustig. “ESPIRiT—An Eigenvalue Approach to Autocalibrating Parallel MRI: Where SENSE Meets GRAPPA.” In: Magnetic Resonance in Medicine 71 (Mar. 2014). doi: 10.1002/mrm. 24751. [286] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. “Instance Normalization: The Missing Ingredient for Fast Stylization.” In: (July 2016). doi: 10.48550/arXiv.1607.08022. [287] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards Accurate Generative Models of Video: A New Metric & Challenges. 2019. arXiv: 1812.01717 [cs.CV]. url: https://arxiv.org/abs/1812.01717. [288] Shreyas S. Vasanawala, Mark J. Murphy, Marcus T. Alley, Peng Lai, Kurt Keutzer, John M. Pauly, and Michael Lustig. “Practical Parallel Imaging Compressed Sensing MRI: Summary of Two Years of Experience in Accelerating Body MRI of Pediatric Patients.” In: 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro. 2011, pp. 1039– 1043. doi: 10.1109/ISBI.2011.5872579. 272 [289] Jaganathan Vellagoundar and Ramasubba Reddy Machireddy. “A Robust Adaptive Sampling Method for Faster Acquisition of MR Images.” In: Magnetic Resonance Imaging 33.5 (June 2015), pp. 635–643. issn: 0730-725X. doi: 10.1016/j.mri.2015.01.008. url: https://doi.org/10. 1016/j.mri.2015.01.008. [290] Santosh Vempala and Andre Wibisono. “Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Suffices.” In: Advances in Neural Information Processing Systems. Ed. by H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett. Vol. 32. Curran Associates, Inc., 2019. url: https://proceedings.neurips.cc/paper_ files / paper / 2019 / file / 65a99bb7a3115fdede20da98b08a370f Paper.pdf. [291] Singanallur V. Venkatakrishnan, Charles A. Bouman, and Brendt Wohlberg. “Plug-and-Play Priors for Model Based Reconstruction.” In: 2013 IEEE Global Conference on Signal and Information Processing. 2013, pp. 945– 948. doi: 10.1109/GlobalSIP.2013.6737048. [292] Pascal Vincent. “A Connection Between Score Matching and Denoising Autoencoders.” In: Neural Computation 23.7 (2011), pp. 1661–1674. doi: 10.1162/NECO_a_00142. [293] Jean Virieux, Amir Asnaashari, Romain Brossier, Ludovic Métivier, Alessandra Ribodetti, and Wei Zhou. “An Introduction to Full Waveform Inversion.” In: Encyclopedia of Exploration Geophysics. Society of Exploration Geophysicists, Jan. 2014. isbn: 9781560803010. doi: 10.1190/1.9781560803027. entry6. url: https://doi.org/10.1190/1.9781560803027.entry6. [294] Jean Virieux and Stéphane Operto. “An Overview of Full-waveform Inversion in Exploration Geophysics.” In: Geophysics 74 (Nov. 2009), WCC1– WCC26. doi: 10.1190/1.3238367. [295] Maxime Vono, Nicolas Dobigeon, and Pierre Chainais. “High-Dimensional Gaussian Sampling: A Review and a Unifying Approach Based on a Stochastic Proximal Point Algorithm.” In: SIAM Review 64.1 (2022), pp. 3–56. doi: 10 . 1137 / 20M1371026. eprint: https : / / doi . org / 10 . 1137 / 20M1371026. url: https://doi.org/10.1137/20M1371026. [296] Maxime Vono, Nicolas Dobigeon, and Pierre Chainais. “Split-and-Augmented Gibbs Sampler—Application to Large-Scale Inference Problems.” In: IEEE Transactions on Signal Processing 67.6 (2019), pp. 1648–1661. doi: 10. 1109/TSP.2019.2894825. [297] Austin Wang, Hongkai Zheng, Zihui Wu, Ricardo Baptista, Daniel Zhengyu Huang, and Yisong Yue. “Ensemble Kalman Sampling and Diffusion Prior in Tandem: A Split Gibbs Framework.” In: Frontiers in Probabilistic Inference: Learning Meets Sampling, ICLR 2025. 2025. url: https://openreview. net/forum?id=3DfCxd0yxo. 273 [298] Chengyan Wang, Jun Lyu, Shuo Wang, Chen Qin, Kunyuan Guo, Xinyu Zhang, Xiaotong Yu, Yan Li, Fanwen Wang, Jianhua Jin, Zhang Shi, Ziqiang Xu, Yapeng Tian, Sha Hua, Zhensen Chen, Meng Liu, Mengting Sun, Xutong Kuang, Kang Wang, and Xiaobo Qu. “CMRxRecon: A Publicly Available Kspace Dataset and Benchmark to Advance Deep Learning for Cardiac MRI.” In: Scientific Data 11 (June 2024). doi: 10.1038/s41597-024-03525-4. [299] Guanhua Wang, Tianrui Luo, Jon-Fredrik Nielsen, Douglas C. Noll, and Jeffrey A. Fessler. “B-Spline Parameterized Joint Optimization of Reconstruction and K-Space Trajectories (BJORK) for Accelerated 2D MRI.” In: IEEE Transactions on Medical Imaging 41.9 (2022), pp. 2318–2330. doi: 10.1109/TMI.2022.3161875. [300] Guanhua Wang, Jon-Fredrik Nielsen, and Jeff Fessler. “Stochastic Optimization of Three-Dimensional Non-Cartesian Sampling Trajectory.” In: Magnetic Resonance in Medicine 90 (Apr. 2023). doi: 10.1002/mrm.29645. [301] Haifeng Wang, Dong Liang, and Leslie Ying. “Pseudo 2D Random Sampling for Compressed Sensing MRI.” In: 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2009, pp. 2672– 2675. doi: 10.1109/IEMBS.2009.5334086. [302] Jiechao Wang, Qinqin Yang, Qizhi Yang, Lina Xu, Congbo Cai, and Shuhui Cai. “Joint Optimization of Cartesian Sampling Patterns and Reconstruction for Single-Contrast and Multi-Contrast Fast Magnetic Resonance Imaging.” In: Computer Methods and Programs in Biomedicine (Sept. 2022). doi: 10.1016/j.cmpb.2022.107150. [303] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. ModelScope Text-to-Video Technical Report. 2023. arXiv: 2308.06571 [cs.CV]. url: https://arxiv.org/abs/2308.06571. [304] Lihong V. Wang and Song Hu. “Photoacoustic Tomography: In Vivo Imaging from Organelles to Organs.” In: Science 335.6075 (2012), pp. 1458– 1462. doi: 10.1126/science.1216210. eprint: https://www.science. org / doi / pdf / 10 . 1126 / science . 1216210. url: https : / / www . science.org/doi/abs/10.1126/science.1216210. [305] Shanshan Wang, Zhenghang Su, Leslie Ying, Xi Peng, Shun Zhu, Feng Liang, Dagan Feng, and Dong Liang. “Accelerating Magnetic Resonance Rmaging via Deep Learning.” In: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). 2016, pp. 514–517. doi: 10.1109/ISBI. 2016.7493320. [306] Yinhuai Wang, Jiwen Yu, and Jian Zhang. “Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model.” In: The Eleventh International Conference on Learning Representations. 2023. url: https : / / openreview.net/forum?id=mRieQgMtNTQ. 274 [307] Zhiwen Wang, Bowen Li, Wenjun Xia, Chenyu Shen, Mingzheng Hou, Hu Chen, Yan Liu, Jiliu Zhou, and Yi Zhang. “Leaders: Learnable Deep Radial Subsampling for MRI Reconstruction.” In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI). 2022, pp. 1–5. doi: 10.1109/ ISBI52829.2022.9761544. [308] Zhiwen Wang, Bowen Li, Hui Yu, Zhongzhou Zhang, Maosong Ran, Wenjun Xia, Ziyuan Yang, Jingfeng Lu, Hu Chen, Jiliu Zhou, Hongming Shan, and Yi Zhang. “Promoting Fast MR Imaging Pipeline by Full-Stack AI.” In: iScience 27.1 (2024), p. 108608. issn: 2589-0042. doi: 10.1016/j.isci. 2023 . 108608. url: https : / / www . sciencedirect . com / science / article/pii/S2589004223026858. [309] Zhiwen Wang, Wenjun Xia, Zexin Lu, Yongqiang Huang, Yan Liu, Hu Chen, Jiliu Zhou, and Yi Zhang. “One Network to Solve Them All: A Sequential Multi-Task Joint Learning Network Framework for MR Imaging Pipeline.” In: Machine Learning for Medical Image Reconstruction: 4th International Workshop, MLMIR 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, October 1, 2021, Proceedings. 2021, pp. 76–85. isbn: 978-3-030-88551-9. doi: 10.1007/978-3-030-88552-6_8. url: https://doi.org/10.1007/978-3-030-88552-6_8. [310] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. “Image Quality Assessment: From Error Visibility to Structural Similarity.” In: IEEE Transactions on Image Processing 13.4 (2004), pp. 600–612. doi: 10.1109/TIP.2003.819861. [311] Tomer Weiss, Ortal Senouf, Sanketh Vedula, Oleg Michailovich, Michael Zibulevsky, and Alex Bronstein. “PILOT: Physics-Informed Learned Optimized Trajectories for Accelerated MRI.” In: Machine Learning for Biomedical Imaging 1 (April 2021 issue 2021), pp. 1–23. issn: 2766-905X. doi: 10.59275/j.melba.2021-1a1f. url: https://melba-journal.org/ 2021:006. [312] Tomer Weiss, Sanketh Vedula, Ortal Senouf, Oleg Michailovich, Michael Zibulevsky, and Alex Bronstein. “Joint Learning of Cartesian Undersampling and Reconstruction for Accelerated MRI.” In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2020, pp. 8653–8657. doi: 10 . 1109 / ICASSP40776 . 2020 . 9054542. [313] Peter W. White and ECMWF. IFS Documentation: Part III: Dynamics and Numerical Procedures (CY21R4). Meteorological bulletin. ECMWF, 2000. url: https://books.google.com/books?id=yiXczAEACAAJ. [314] David Wiesner, David Svoboda, Martin Maška, and Michal Kozubek. “CytoPacq: A Web-Interface for Simulating Multi-Dimensional Cell Imaging.” In: Bioinformatics 35.21 (May 2019), pp. 4531–4533. issn: 1367-4803. doi: 10.1093/bioinformatics/btz417. eprint: https://academic. 275 oup.com/bioinformatics/article- pdf/35/21/4531/50721742/ bioinformatics\_35\_21\_4531.pdf. url: https://doi.org/10. 1093/bioinformatics/btz417. [315] Emil Wolf. “Three-Dimensional Structure Determination of Semi-Transparent Objects from Holographic Data.” In: Optics Communications 1.4 (1969), pp. 153–156. issn: 0030-4018. doi: https : / / doi . org / 10 . 1016 / 0030- 4018(69)90052- 2. url: https://www.sciencedirect.com/ science/article/pii/0030401869900522. [316] George N. Wong, Ben S. Prather, Vedant Dhruv, Benjamin R. Ryan, Monika Mościbrodzka, Chi-kwan Chan, Abhishek V. Joshi, Ricardo Yarza, Angelo Ricarte, Hotaka Shiokawa, Joshua C. Dolence, Scott C. Noble, Jonathan C. McKinney, and Charles F. Gammie. “PATOKA: Simulating Electromagnetic Observables of Black Hole Accretion.” In: The Astrophysical Journal. Supplement Series 259.2 (Apr. 2022). doi: 10.3847/1538-4365/ac582e. [317] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. “Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-toVideo Generation.” In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). 2023, pp. 7589–7599. doi: 10.1109/ICCV51070. 2023.00701. [318] Luhuan Wu, Brian Trippe, Christian Naesseth, David Blei, and John P Cunningham. “Practical and Asymptotically Exact Conditional Sampling in Diffusion Models.” In: Advances in Neural Information Processing Systems. Ed. by A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine. Vol. 36. Curran Associates, Inc., 2023, pp. 31372–31403. url: https://proceedings.neurips.cc/paper_files/paper/2023/ file / 63e8bc7bbf1cfea36d1d1b6538aecce5 - Paper - Conference . pdf. [319] Pingyu Wu, Kai Zhu, Yu Liu, Liming Zhao, Wei Zhai, Yang Cao, and ZhengJun Zha. Improved Video VAE for Latent Video Diffusion Model. 2024. arXiv: 2411.06449 [cs.CV]. url: https://arxiv.org/abs/2411.06449. [320] Zihui Wu, Yu Sun, Yifan Chen, Bingliang Zhang, Yisong Yue, and Katherine Bouman. “Principled Probabilistic Imaging Using Diffusion Models as Plug-and-Play Priors.” In: The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024. url: https://openreview.net/ forum?id=Xq9HQf7VNV. [321] Zihui Wu, Yu Sun, Jiaming Liu, and Ulugbek Kamilov. “Online Regularization by Denoising with Applications to Phase Retrieval.” In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). 2019, pp. 3887–3895. doi: 10.1109/ICCVW.2019.00482. 276 [322] Zihui Wu, Yu Sun, Alex Matlock, Jiaming Liu, Lei Tian, and Ulugbek S. Kamilov. “SIMBA: Scalable Inversion in Optical Tomography Using Deep Denoising Priors.” In: IEEE Journal of Selected Topics in Signal Processing 14.6 (2020), pp. 1163–1175. doi: 10.1109/JSTSP.2020.2999820. [323] Zihui Wu, Tianwei Yin, Yu Sun, Robert Frost, Andre van der Kouwe, Adrian V. Dalca, and Katherine L. Bouman. “Learning Task-Specific Strategies for Accelerated MRI.” In: IEEE Transactions on Computational Imaging 10 (2024), pp. 1040–1054. doi: 10.1109/TCI.2024.3410521. [324] Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, and Luc Van Gool. “DiffIR: Efficient Diffusion Model for Image Restoration.” In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). 2023, pp. 13049–13059. doi: 10.1109/ ICCV51070.2023.01204. [325] Xiaojian Xu, Weijie Gan, Satya V. V. N. Kothapalli, Dmitriy A. Yablonskiy, and Ulugbek S. Kamilov. “CoRRECT: A Deep Unfolding Framework for Motion-Corrected Quantitative R2* Mapping.” In: Journal of Mathematical Imaging and Vision 67.2 (Apr. 2025). issn: 0924-9907. doi: 10 . 1007 / s10851- 025- 01236- y. url: https://doi.org/10.1007/s10851025-01236-y. [326] Xingyu Xu and Yuejie Chi. “Provably Robust Score-Based Diffusion Posterior Sampling for Plug-and-Play Image Reconstruction.” In: The Thirtyeighth Annual Conference on Neural Information Processing Systems. 2024. url: https://openreview.net/forum?id=SLnsoaY4u1. [327] Shengke Xue, Zhaowei Cheng, Guangxu Han, Chaoliang Sun, Ke Fang, Yingchao Liu, Jian Cheng, Xinyu Jin, and Ruiliang Bai. “2D Probabilistic Undersampling Pattern Optimization for MR Image Reconstruction.” In: Medical Image Analysis 77 (Jan. 2022), p. 102346. doi: 10 . 1016 / j . media.2021.102346. [328] Burhaneddin Yaman, Seyed Amir Hossein Hosseini, Steen Moeller, Jutta Ellermann, Kâmil Uğurbil, and Mehmet Akçakaya. “Self-Supervised Learning of Physics-Guided reconstruction Neural Networks without Fully Sampled Reference Data.” In: Magnetic Resonance in Medicine 84 (July 2020). doi: 10.1002/mrm.28378. [329] Guang Yang, Simiao Yu, Hao Dong, Greg Slabaugh, Pier Dragotti, Xujiong Ye, Fangde Liu, Simon Arridge, Jennifer Keegan, Yike Guo, and David Firmin. “DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction.” In: IEEE Transactions on Medical Imaging PP (Dec. 2017), pp. 1–1. doi: 10 . 1109 / TMI . 2017 . 2785879. 277 [330] Junwei Yang, Xiao-Xin Li, Feihong Liu, Dong Nie, Pietro Lio, Haikun Qi, and Dinggang Shen. “Fast Multi-Contrast MRI Acquisition by Optimal Sampling of Information Complementary to Pre-acquired MRI Contrast.” In: IEEE Transactions on Medical Imaging (2022), pp. 1–1. doi: 10.1109/ TMI.2022.3227262. [331] Yan Yang, Jian Sun, Huibin Li, and Zongben Xu. “Deep ADMM-Net for Compressive Sensing MRI.” In: Advances in Neural Information Processing Systems. Ed. by D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett. Vol. 29. Curran Associates, Inc., 2016. url: https : / / proceedings . neurips . cc / paper _ files / paper / 2016 / file / 1679091c5a880faf6fb5e6087eb1b2dc-Paper.pdf. [332] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan.Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. “CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer.” In: The Thirteenth International Conference on Learning Representations. 2025. url: https://openreview.net/ forum?id=LQzN6TRFg9. [333] Horng-Tzer Yau. “Relative Rntropy and Hydrodynamics of Ginzburg-Landau Models.” In: Letters in Mathematical Physics 22.1 (1991), pp. 63–80. doi: 10.1007/BF00400379. url: https://doi.org/10.1007/BF00400379. [334] Chang-Han Yeh, Chin-Yang Lin, Zhixiang Wang, Chi-Wei Hsiao, TingHsuan Chen, Hau-Shiang Shiu, and Yu-Lun Liu. DiffIR2VR-Zero: ZeroShot Video Restoration with Diffusion-based Image Restoration Models. 2024. arXiv: 2407.01519 [cs.CV]. url: https://arxiv.org/abs/ 2407.01519. [335] Tianwei Yin*, Zihui Wu*, He Sun, Adrian V. Dalca, Yisong Yue, and Katherine L. Bouman. “End-to-End Sequential Sampling and Reconstruction for MRI.” In: Proceedings of Machine Learning for Health. Vol. 158. Proceedings of Machine Learning Research. PMLR, Dec. 2021, pp. 261–281. url: https://proceedings.mlr.press/v158/yin21a.html. [336] Bo Yuan, Jiaojiao Fan, Jiaming Liang, Andre Wibisono, and Yongxin Chen. “On a Class of Gibbs Sampling over Networks.” In: Proceedings of Thirty Sixth Conference on Learning Theory. Ed. by Gergely Neu and Lorenzo Rosasco. Vol. 195. Proceedings of Machine Learning Research. PMLR, July 2023, pp. 5754–5780. url: https://proceedings.mlr.press/ v195/yuan23a.html. [337] Michaël Zamo and Philippe Naveau. “Estimation of the Continuous Ranked Probability Score with Limited Information and Applications to Ensemble Weather Forecasts.” In: Mathematical Geosciences 50.2 (Feb. 2018), pp. 209–234. doi: 10.1007/s11004-017-9709-7. url: https://hal. science/hal-02976423. 278 [338] Jure Zbontar et al. “fastMRI: An Open Dataset and Benchmarks for Accelerated MRI.” In: 2018. arXiv: 1811.08839. [339] Frederik Zernike. “The Concept of Degree of Coherence and Its Application to Optical Problems.” In: Physica 5.8 (1938), pp. 785–795. issn: 00318914. doi: 10.1016/S0031- 8914(38)80203- 2. url: https://www. sciencedirect.com/science/article/pii/S0031891438802032. [340] Zhifang Zhan, Jian-Feng Cai, Di Guo, Yunsong Liu, Zhong Chen, and Xiaobo Qu. “Fast Multiclass Dictionaries Learning with Geometrical Directions in MRI Reconstruction.” In: IEEE Transactions on Biomedical Engineering 63.9 (2016), pp. 1850–1861. doi: 10.1109/TBME.2015.2503756. [341] Bingliang Zhang, Wenda Chu, Julius Berner, Chenlin Meng, Anima Anandkumar, and Yang Song. “Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing.” In: Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR). June 2025, pp. 20895– 20905. url: https://openaccess.thecvf.com/content/CVPR2025/ html/Zhang_Improving_Diffusion_Inverse_Problem_Solving_ with_Decoupled_Noise_Annealing_CVPR_2025_paper.html. [342] Cheng Zhang, Judith Bütepage, Hedvig Kjellström, and Stephan Mandt. “Advances in Variational Inference.” In: IEEE Transactions on Pattern Analysis and Machine Intelligence 41.8 (2019), pp. 2008–2026. doi: 10.1109/ TPAMI.2018.2889774. [343] Jian Zhang and Bernard Ghanem. “ISTA-Net: Interpretable OptimizationInspired Deep Network for Image Compressive Sensing.” In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, pp. 1828– 1837. doi: 10.1109/CVPR.2018.00196. [344] Jinwei Zhang, Hang Zhang, Alan Wang, Qihao Zhang, Mert Sabuncu, Pascal Spincemaille, Thanh D. Nguyen, and Yi Wang. “Extending LOUPE for K-Space Under-Sampling Pattern Optimization in Multi-Coil MRI.” In: Machine Learning for Medical Image Reconstruction: Third International Workshop, MLMIR 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 8, 2020, Proceedings. Lima, Peru: Springer-Verlag, 2020, pp. 91–101. isbn: 978-3-030-61597-0. doi: 10.1007/978-3-030-615987_9. url: https://doi.org/10.1007/978-3-030-61598-7_9. [345] Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Gool, and Radu Timofte. “Plug-and-Play Image Restoration With Deep Denoiser Prior.” In: IEEE Transactions on Pattern Analysis and Machine Intelligence PP (June 2021), pp. 1–1. doi: 10.1109/TPAMI.2021.3088914. [346] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. “Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising.” In: IEEE Transactions on Image Processing 26.7 (2017), pp. 3142–3155. doi: 10.1109/TIP.2017.2662206. 279 [347] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. “Learning Deep CNN Denoiser Prior for Image Restoration.” In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017, pp. 2808–2817. doi: 10.1109/CVPR.2017.300. [348] Qinsheng Zhang and Yongxin Chen. “Fast Sampling of Diffusion Models with Exponential Integrator.” In: The Eleventh International Conference on Learning Representations. 2023. url: https://openreview.net/ forum?id=Loek7hfb46P. [349] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.” In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, pp. 586–595. doi: 10.1109/CVPR.2018.00068. [350] Zhixing Zhang, Bichen Wu, Xiaoyan Wang, Yaqiao Luo, Luxin Zhang, Yinan Zhao, Peter Vajda, Dimitris Metaxas, and Licheng Yu. “AVID: Any-Length Video Inpainting with Diffusion Model.” In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024, pp. 7162–7172. doi: 10.1109/CVPR52733.2024.00684. [351] Zizhao Zhang, Adriana Romero, Matthew J. Muckley, Pascal Vincent, Lin Yang, and Michal Drozdzal. “Reducing Uncertainty in Undersampled MRI Reconstruction With Active Acquisition.” In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019, pp. 2049–2053. doi: 10.1109/CVPR.2019.00215. [352] Bingliang Zhang*, Zihui Wu*, Berthy T. Feng, Yang Song, Yisong Yue, and Katherine L. Bouman. STeP: A Framework for Solving Scientific Video Inverse Problems with Spatiotemporal Diffusion Priors. 2025. arXiv: 2504. 07549 [cs.CV]. url: https://arxiv.org/abs/2504.07549. [353] Lin Zhao, Xiao Chen, Eric Z. Chen, Yikang Liu, Dinggang Shen, Terrence Chen, and Shanhui Sun. JoJoNet: Joint-Contrast and Joint-Sampling-andReconstruction Network for Multi-Contrast MRI. 2022. arXiv: 2210.12548 [eess.IV]. url: https://arxiv.org/abs/2210.12548. [354] Ruiyang Zhao, Burhaneddin Yaman, Yuxin Zhang, Russell Stewart, Austin Dixon, Florian Knoll, Zhengnan Huang, Yvonne Lui, Michael Hansen, and Matthew Lungren. “fastMRI+, Clinical Pathology Annotations for Knee and Brain Fully Sampled Magnetic Resonance Imaging Data.” In: Scientific Data 9 (Apr. 2022), p. 152. doi: 10.1038/s41597-022-01255-z. [355] Guoan Zheng, Roarke Horstmeyer, and Changhuei Yang. “Wide-Field, High-Resolution Fourier Ptychographic Microscopy.” In: Nature Photonics 7 (July 2013), pp. 739–745. doi: 10.1038/nphoton.2013.187. [356] Hongkai Zheng, Wenda Chu, Austin Wang, Nikola Borislavov Kovachki, Ricardo Baptista, and Yisong Yue. “Ensemble Kalman Diffusion Guidance: A Derivative-free Method for Inverse Problems.” In: Transactions 280 on Machine Learning Research (2025). issn: 2835-8856. url: https : //openreview.net/forum?id=XPEEsKneKs. [357] Hongkai Zheng*, Wenda Chu*, Bingliang Zhang*, Zihui Wu*, Austin Wang, Berthy Feng, Caifeng Zou, Yu Sun, Nikola Borislavov Kovachki, Zachary E Ross, Katherine Bouman, and Yisong Yue. “InverseBench: Benchmarking Plug-and-Play Diffusion Models for Scientific Inverse Problems.” In: The Thirteenth International Conference on Learning Representations. 2025. url: https://openreview.net/forum?id=U3PBITXNG6. [358] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. MagicVideo: Efficient Video Generation With Latent Diffusion Models. 2023. arXiv: 2211.11018 [cs.CV]. url: https://arxiv.org/ abs/2211.11018. [359] Kevin Zhou and Roarke Horstmeyer. “Diffraction Tomography with a Deep Image Prior.” In: Optics Express 28 (Apr. 2020), pp. 12872–12896. doi: 10.1364/OE.379200. [360] Bo Zhu, Jeremiah Zhe Liu, Bruce Rosen, and Matthew Rosen. “Image Reconstruction by Domain Transform Manifold Learning.” In: Nature 555.7697 (Mar. 2018), pp. 487–492. doi: 10.1038/nature25988. [361] Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. “Denoising Diffusion Models for Plug-and-Play Image Restoration.” In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2023, pp. 1219–1229. doi: 10.1109/CVPRW59228.2023.00129. [362] Marcelo Victor Wust Zibetti, Florian Knoll, and Ravinder R. Regatte. “Alternating Learning Approach for Variational Networks and Undersampling Pattern in Parallel MRI Applications.” In: IEEE Transactions on Computational Imaging 8 (2022), pp. 449–461. doi: 10.1109/TCI.2022.3176129. [363] Jiaren Zou and Yue Cao. “Joint Optimization of k-t Sampling Pattern and Reconstruction of DCE MRI for Pharmacokinetic Parameter Estimation.” In: IEEE Transactions on Medical Imaging 41.11 (2022), pp. 3320–3331. doi: 10.1109/TMI.2022.3184261. [364] Kelly Zou, Simon Warfield, Aditya Bharatha, Clare Tempany, Michael Kaus, Steven Haker, William Wells, Ferenc Jolesz, and Ron Kikinis. “Statistical Validation of Image Segmentation Quality Based on a Spatial Overlap Index.” In: Academic Radiology 11 (Feb. 2004), pp. 178–89. doi: 10.1016/ S1076-6332(03)00671-8. [365] Zihao Zou, Jiaming Liu, Shirin Shoushtari, Yubo Wang, and Ulugbek S. Kamilov. “FLAIR: A Conditional Diffusion Framework with Applications to Face Video Restoration.” In: 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2025, pp. 5228–5238. doi: 10. 1109/WACV61041.2025.00511.