Reliable Learning and Control in Dynamic Environments: Towards Unified Theory and Learned Robotic Agility Citation Shi, Guanya (2023) Reliable Learning and Control in Dynamic Environments: Towards Unified Theory and Learned Robotic Agility. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/8rz4-7b35. https://resolver.caltech.edu/CaltechTHESIS:08052022-231458463 Abstract Recent breathtaking advances in machine learning beckon to their applications in a wide range of real-world autonomous systems. However, for safety-critical settings such as agile robotic control in hazardous environments, we must confront several key challenges before widespread deployment. Most importantly, the learning system must interact with the rest of the autonomous system (e.g., highly nonlinear and non-stationary dynamics) in a way that safeguards against catastrophic failures with formal guarantees. In addition, from both computational and statistical standpoints, the learning system must incorporate prior knowledge for efficiency and generalizability. This thesis presents progress towards establishing a unified framework that fundamentally connects learning and control. First, Part I motivates the benefit and necessity of such a unified framework by the Neural-Control Family, a family of nonlinear deep-learning-based control methods with not only stability and robustness guarantees but also new capabilities in agile robotic control. Then Part II discusses three unifying interfaces between learning and control: (1) online meta-adaptive control, (2) competitive online optimization and control, and (3) online learning perspectives on model predictive control. All interfaces yield settings that jointly admit both learning-theoretic and control-theoretic guarantees. Item Type: Thesis (Dissertation (Ph.D.)) Subject Keywords: Machine Learning, Control Theory, Robotics Degree Grantor: California Institute of Technology Division: Engineering and Applied Science Major Option: Control and Dynamical Systems Awards: Ben P.C. Chou Doctoral Prize in IST, 2022. Simoudis Discovery Prize, 2020/2021. Rising Stars in Data Science, Autumn 2021 cohort. Thesis Availability: Public (worldwide access) Research Advisor(s): Chung, Soon-Jo (co-advisor) Yue, Yisong (co-advisor) Thesis Committee: Burdick, Joel Wakeman (chair) Wierman, Adam C. Chung, Soon-Jo Yue, Yisong Defense Date: 21 July 2022 Funders: Funding Agency Grant Number DARPA UNSPECIFIED Raytheon UNSPECIFIED Center for Autonomous Systems and Technologies at Caltech UNSPECIFIED Record Number: CaltechTHESIS:08052022-231458463 Persistent URL: https://resolver.caltech.edu/CaltechTHESIS:08052022-231458463 DOI: 10.7907/8rz4-7b35 Related URLs: URL URL Type Description http://www.gshi.me Author Personal website https://ieeexplore.ieee.org/document/8794351 Publisher Article adapted for chapter 3 https://ieeexplore.ieee.org/document/9196800 Publisher Article adapted for chapter 4 https://ieeexplore.ieee.org/document/9508420 Publisher Article adapted for chapter 4 https://www.science.org/doi/10.1126/scirobotics.abm6597 Publisher Article adapted for chapter 5 https://ieeexplore.ieee.org/document/9290355 Publisher Article adapted for chapter 6 https://proceedings.mlr.press/v120/liu20a.html Publisher Article adapted for chapter 6 https://ieeexplore.ieee.org/document/9561483 Publisher Article adapted for chapter 7 https://papers.nips.cc/paper/2021/hash/52fc2aee802efbad698503d28ebd3a1f-Abstract.html Publisher Article adapted for chapter 9 https://papers.nips.cc/paper/2020/hash/ed46558a56a4a26b96a68738a0d28273-Abstract.html Publisher Article adapted for chapter 10 https://dl.acm.org/doi/10.1145/3508037 Publisher Article adapted for chapter 10 https://papers.nips.cc/paper/2020/hash/155fa09596c7e18e50b58eb7e0c6ccb4-Abstract.html Publisher Article adapted for chapter 11 https://papers.nips.cc/paper/2021/hash/298f587406c914fad5373bb689300433-Abstract.html Publisher Article adapted for chapter 11 https://dl.acm.org/doi/10.1145/3508038 Publisher Article adapted for chapter 11 https://arxiv.org/abs/2010.11637 arXiv Article adapted for chapter 11 ORCID: Author ORCID Shi, Guanya 0000-0002-9075-3705 Default Usage Policy: No commercial reproduction, distribution, display or performance rights in this work are provided. ID Code: 14994 Collection: CaltechTHESIS Deposited By: Guanya Shi Deposited On: 09 Aug 2022 23:46 Last Modified: 21 Jun 2023 23:47 Thesis Files PDF - Final Version See Usage Policy. 12MB Repository Staff Only: item control page

Reliable Learning and Control in Dynamic Environments: Towards Unified Theory and Learned Robotic Agility Thesis by Guanya Shi In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy CALIFORNIA INSTITUTE OF TECHNOLOGY Pasadena, California 2023 Defended July 21, 2022 ii © 2023 Guanya Shi ORCID: 0000-0002-9075-3705 All rights reserved iii ACKNOWLEDGEMENTS To my advisors, Soon-Jo Chung and Yisong Yue. It is extremely fortunate to simultaneously have two amazing advisors. Thank you for your guidance, patience, kindness, and encouragement. This dissertation would not have been possible without your support. I also have so many fond memories of you outside of research. For example, the wonderful group event at Soon-Jo’s house and my defense against Yisong when we played basketball at Grant Park. To other mentors and collaborators at Caltech: Adam Wierman, Anima Anandkumar, Kamyar Azizzadenesheli, Steven Low, Joel Burdick, Guannan Qu, and Rose Yu, outside of Caltech: Jean-Jacques Slotine, Byron Boots, Na Li, and Chuchu Fan, and at NVIDIA during my internship: Yuke Zhu, Stan Birchfield, Fabio Ramos, and Jonathan Tremblay. Thank you for your invaluable insights. To the ARCL Lab led by Prof. Soon-Jo Chung: Benjamin (Ben) Riviere, Michael (Mike) O’Connell, Kai Matsuka, Hiroyasu (Hiro) Tsukamoto, Ellande Tang, SooJean Han, Xichen Shi, Wolfgang Hönig, Connor Lee, Sorina Lupu, Kyunam Kim, Jimmy Ragan, Nikhil Ranganathan, John Lathrop, Fengze Xie, Yuta Takahashi, Yashwanth Kumar Nakka, Karena Cai, Rebecca Foust, Vincenzo Capuano, Patrick Spieler, Alexei Harvard, Matt Anderson, Lu Gan, and James Preiss. Thank you for all of the wonderful memories. To the Yue Crew led by Prof. Yisong Yue: Jeremy Bernstein, Geeling Chau, Victor Dorobantu, Alex Farhang, Yujia Huang, Uriah Israel, Ivan Jimenez Rodriguez, Francesca-Zhoufan Li, Kejun (Amy) Li, Yiheng Lin, Hao Liu, Jennifer Sun, Sabera Talukder, Cameron Voloshin, Xuefei (Julie) Wang, Jason Yang, Chris Yeh, Yuxin Chen, Hoang Le, Angie Liu, Joe Marino, Ellen Novoseller, Ugo Rosolia, Jialin Song, Yanan Sui, Eric Zhan, and Stephan Zheng. Thank you for all of the wonderful memories. To my undergraduate mentees at Caltech: Yiheng Lin, Chenkai Yu, Weici Pan, Yang Hu, Ruixiao Yang, Anya Vinogradsky, Alice Jin, Luis Pabon Madrid, and Nelson Badillo. Mentoring and teaching are always bidirectional for me. I’ve learned a lot from all of you. To all of the other friends and my Caltech cohort: Yuanyuan Shi, Tongxin Li, Sara Beery, Elizabeth Qian, Gautam Goel, Forte Shinko, Sahin Lale, Xiaomin Li, Zongyi Li, Navid Azizan, Jiawei Zhao, Shumao Zhang, Nian Guo, Chen Li, Kun iv Miao, Xin Su, Botao Hu, Fangzhu Zhao, Yukai Liu, Yalu Chen, Zhiquan Yuan, and many others. Thank you for making my Ph.D. journey memorable. In particular, I want to acknowledge our Wed and Fri basketball group. It is so much fun to play basketball twice a week. To the CMS department at Caltech, especially the CDS option. The department has offered me a fantastic environment for explorations and the access to worldclass faculty members, challenging but super interesting courses, and top-notch experimental facilities such as the Real Weather Wind Tunnel. In particular, I would like to acknowledge Richard Murray, Steve Brunton, and Katie Bouman for their help when I was on the academic job market. I also really appreciate taking excellent courses from Venkat Chandrasekaran, Richard Murray, Joel Burdick, John Doyle, Houman Owhadi, Andrew Stuart, and Kostia Zuev. To my family, especially my mom Chunling and my dad Yusheng. Thank you for your love. You always support me no matter what I decide to pursue, from high school, undergraduate studies at Tsinghua, to my Ph.D. at Caltech. Finally, to funding agencies: NSF, DARPA, Raytheon, and Caltech CAST, and prize donors: the Simoudis family for the Simoudis Discovery Prize and the Chou family for the Ben P.C. Chou Doctoral Prize. Thank you so much for your support on me and my research. v ABSTRACT Recent breathtaking advances in machine learning beckon to their applications in a wide range of real-world autonomous systems. However, for safety-critical settings such as agile robotic control in hazardous environments, we must confront several key challenges before widespread deployment. Most importantly, the learning system must interact with the rest of the autonomous system (e.g., highly nonlinear and non-stationary dynamics) in a way that safeguards against catastrophic failures with formal guarantees. In addition, from both computational and statistical standpoints, the learning system must incorporate prior knowledge for efficiency and generalizability. This thesis presents progress towards establishing a unified framework that fundamentally connects learning and control. First, Part I motivates the benefit and necessity of such a unified framework by the Neural-Control Family, a family of nonlinear deep-learning-based control methods with not only stability and robustness guarantees but also new capabilities in agile robotic control. Then Part II discusses three unifying interfaces between learning and control: (1) online meta-adaptive control, (2) competitive online optimization and control, and (3) online learning perspectives on model predictive control. All interfaces yield settings that jointly admit both learning-theoretic and control-theoretic guarantees. vi PUBLISHED CONTENT AND CONTRIBUTIONS Li, Tongxin, Ruixiao Yang, Guannan Qu, Guanya Shi, Chenkai Yu, Adam Wierman, and Steven Low (2022). Robustness and consistency in linear quadratic control with untrusted predictions. In: Proceedings of the ACM on Measurement and Analysis of Computing Systems 6.1, pp. 1–35. doi: 10.1145/3508038. G.S. contributed to the algorithm design and theoretical analysis. O’Connell, Michael, Guanya Shi, Xichen Shi, Kamyar Azizzadenesheli, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung (2022). Neural-Fly enables rapid learning for agile flight in strong winds. In: Science Robotics 7.66, eabm6597. doi: 10.1126/scirobotics.abm6597. The first two authors contributed equally and the order is alphabetic. G.S. co-led the project, designed the learning algorithm and co-designed the adaptive control method with theoretical analysis, and co-designed and co-conducted robotic experiments. Pan, Weici, Guanya Shi, Yiheng Lin, and Adam Wierman (2022). Online optimization with feedback delay and nonlinear switching cost. In: Proceedings of the ACM on Measurement and Analysis of Computing Systems 6.1, pp. 1–34. doi: 10.1145/3508037. G.S. designed the project and defined the research question, contributed to the algorithm design and proofs, and mentored W.P. in numerical experiments. Shi, Guanya, Wolfgang Hönig, Xichen Shi, Yisong Yue, and Soon-Jo Chung (2022). Neural-Swarm2: Planning and control of heterogeneous multirotor swarms using learned interactions. In: IEEE Transactions on Robotics 38.2, pp. 1063–1079. doi: 10.1109/TRO.2021.3098436. G.S. led the project, designed the learning, motion planning, and control algorithms with theoretical analysis, and designed and co-conducted the numerical and robotic experiments. Yu, Chenkai, Guanya Shi, Soon-Jo Chung, Yisong Yue, and Adam Wierman (2022). Competitive control with delayed imperfect information. In: American Control Conference (ACC). url: https://arxiv.org/abs/2010.11637. G.S. designed the project and defined the research problem, designed the predictive control algorithm and contributed to all proofs, and mentored C.Y. in numerical experiments. Lin, Yiheng, Yang Hu, Guanya Shi, Haoyuan Sun, Guannan Qu, and Adam Wierman (2021). Perturbation-based regret analysis of predictive control in linear time varying systems. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 34. Curran Associates, Inc., pp. 5174–5185. url: https : / / arxiv.org/abs/2106.10497. First five authors contributed equally. G.S. co-designed the theoretical analysis framework and contributed to all proofs. vii Nakka, Yashwanth Kumar, Anqi Liu, Guanya Shi, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung (2021). Chance-constrained trajectory optimization for safe exploration and learning of nonlinear Systems. In: IEEE Robotics and Automation Letters 6.2, pp. 389–396. doi: 10.1109/LRA.2020.3044033. G.S. contributed to the algorithm design and theoretical analysis, and helped Y.K.N. with the numerical experiments. Shi, Guanya, Kamyar Azizzadenesheli, Michael O’Connell, Soon-Jo Chung, and Yisong Yue (2021). Meta-adaptive nonlinear control: Theory and algorithms. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 34. Curran Associates, Inc., pp. 10013–10025. url: https://proceedings.neurips. cc / paper / 2021 / file / 52fc2aee802efbad698503d28ebd3a1f - Paper . pdf. G.S. led the project, designed the online meta-adaptive control algorithm with theoretical analysis, and designed and co-conducted numerical experiments. Shi, Guanya, Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, Fabio Ramos, Animashree Anandkumar, and Yuke Zhu (2021). Fast uncertainty quantification for deep object pose estimation. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 5200–5207. doi: 10.1109/ICRA48506. 2021.9561483. G.S. led the project, designed the uncertainty quantification algorithm, conducted numerical experiments, and co-conducted real-world manipulator experiments. Liu, Anqi, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue (2020). Robust regression for safe exploration in control. In: Learning for Dynamics and Control. PMLR, pp. 608–619. url: https://proceedings.mlr. press/v120/liu20a.html. G.S. co-designed the safe exploration algorithm and contributed to the numerical experiments. Shi, Guanya, Wolfgang Hönig, Yisong Yue, and Soon-Jo Chung (2020). NeuralSwarm: Decentralized close-proximity multirotor control using learned interactions. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 3241–3247. doi: 10.1109/ICRA40945.2020.9196800. G.S. led the project, designed the learning, motion planning, and control algorithms with theoretical analysis, and designed and co-conducted the numerical and robotic experiments. Shi, Guanya, Yiheng Lin, Soon-Jo Chung, Yisong Yue, and Adam Wierman (2020). Online optimization with memory and competitive control. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 33. Curran Associates, Inc. url: https://arxiv.org/abs/2002.05318. G.S. led the project, defined the research problem, and co-designed the core algorithm with theoretical analysis. Yu, Chenkai, Guanya Shi, Soon-Jo Chung, Yisong Yue, and Adam Wierman (2020). The power of predictions in online control. In: Advances in Neural Informa- viii tion Processing Systems (NeurIPS). Vol. 33. Curran Associates, Inc., pp. 1994– 2004. url: https : / / proceedings . neurips . cc / paper / 2020 / file / 155fa09596c7e18e50b58eb7e0c6ccb4-Paper.pdf. G.S. designed the project and defined the research problem, contributed to all proofs, and mentored C.Y. in numerical experiments. Shi, Guanya, Xichen Shi, Michael O’Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung (2019). Neural Lander: Stable drone landing control using learned dynamics. In: 2019 International Conference on Robotics and Automation (ICRA). IEEE, pp. 9784–9790. doi: 10.1109/ICRA.2019.8794351. G.S. led the project, designed the learning algorithm with theoretical analysis, co-designed the controller, and co-conducted the robotic experiments. ix TABLE OF CONTENTS Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v Published Content and Contributions . . . . . . . . . . . . . . . . . . . . . . vi Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii List of Illustrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Chapter I: Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Thesis Summary and Contributions . . . . . . . . . . . . . . . . . . 4 I Neural-Control Family: Deep-Learning-Based Nonlinear Control with Learned Robotic Agility Chapter II: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 A Mixed Model for Robot Dynamics and the Nonlinear Control Law 2.2 Example: Quadrotor Dynamics and Control . . . . . . . . . . . . . 2.3 Organization of Part I: Unifying Learning and Nonlinear Control . . 2.4 Preliminaries on Deep Learning . . . . . . . . . . . . . . . . . . . . 2.5 Preliminaries on Nonlinear Control Theory . . . . . . . . . . . . . . Chapter III: Neural-Lander . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Introduction and Related Work . . . . . . . . . . . . . . . . . . . . 3.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Dynamics Learning and Controller Design . . . . . . . . . . . . . . 3.4 Nonlinear Stability Analysis . . . . . . . . . . . . . . . . . . . . . . 3.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter IV: Neural-Swarm . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Learning of Swarm Aerodynamic Interaction . . . . . . . . . . . . . 4.5 Interaction-Aware Multi-Robot Planning . . . . . . . . . . . . . . . 4.6 Interaction-Aware Tracking Controller . . . . . . . . . . . . . . . . 4.7 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter V: Neural-Fly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Offline Meta-Learning . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Online Adaptive Control and Stability Analysis . . . . . . . . . . . . 9 11 11 14 18 20 23 26 27 29 30 30 34 43 44 47 48 51 57 63 64 79 80 83 88 89 94 x 5.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.7 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Chapter VI: Safe Exploration . . . . . . . . . . . . . . . . . . . . . . . . . . 117 6.1 Introduction and Related Work . . . . . . . . . . . . . . . . . . . . 118 6.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 6.3 Uncertainty Quantification under Domain Shift . . . . . . . . . . . . 124 6.4 Safe Planning and Control . . . . . . . . . . . . . . . . . . . . . . . 126 6.5 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Chapter VII: Discussion and Future Work . . . . . . . . . . . . . . . . . . . 132 7.1 Neural-Control for Other Robotic Systems . . . . . . . . . . . . . . 132 7.2 End-to-End Learning and Control Guarantees . . . . . . . . . . . . . 135 7.3 Model-Free and Model-Based Learning . . . . . . . . . . . . . . . . 135 7.4 Neurosymbolic Learning for Data-Efficient Decision Making . . . . 136 7.5 Other Residual Learning Methods . . . . . . . . . . . . . . . . . . . 137 II Unifying Interfaces Between Learning and Control Theory 139 Chapter VIII: Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 8.1 The Importance of Building Interfaces Between Learning and Control 141 8.2 Organization of Part II: Three Interfaces . . . . . . . . . . . . . . . 143 8.3 Preliminaries on Online Optimization and Learning . . . . . . . . . 144 8.4 Preliminaries on Online Optimal Control . . . . . . . . . . . . . . . 147 Chapter IX: Online Meta-Adaptive Control . . . . . . . . . . . . . . . . . . . 154 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 9.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 9.3 Online Meta-Adaptive Control (OMAC) Algorithm . . . . . . . . . . 159 9.4 Different Instantiations of OMAC and Theoretical Analysis . . . . . 162 9.5 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 9.6 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 Chapter X: Competitive Online Optimization and Control . . . . . . . . . . . 180 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 10.2 Problem Statement: OCO with Structured Memory . . . . . . . . . . 182 10.3 Algorithms and Theoretical Analysis . . . . . . . . . . . . . . . . . 185 10.4 Connections to Competitive Control . . . . . . . . . . . . . . . . . . 189 10.5 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 10.6 Extension: Nonlinear Switching Cost and Feedback Delay . . . . . . 196 10.7 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 Chapter XI: Online Learning Perspectives on Model Predictive Control . . . . 208 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 11.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 11.3 Dynamic Regret of MPC in LQ Systems . . . . . . . . . . . . . . . 216 11.4 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 11.5 Extension I: Delayed Inexact Predictions . . . . . . . . . . . . . . . 235 11.6 Extension II: Time-Variant Systems and General Costs . . . . . . . . 240 Chapter XII: Discussion and Future Work . . . . . . . . . . . . . . . . . . . 246 xi LIST OF ILLUSTRATIONS Number Page 3.1 Neural-Lander enables agile flight maneuver very close to the ground. 26 3.2 Training data trajectory. Part I (0 to 250 s) contains maneuvers at different heights (0.05 m to 1.50 m). Part II (250 s to 350 s) includes random 𝑥, 𝑦, and 𝑧 motions for maximum state-space coverage. . . . 35 3.3 DNN prediction performance. (a) Learned 𝑓𝑎,𝑧 compared to the ground effect model with respect to height 𝑧, with 𝑣 𝑧 = 𝑣 𝑥 = 𝑣 𝑦 = 0 m/s, 𝑅 = 𝐼, 𝑢 = 6400 RPM. Ground truth points are from hovering data at different heights. (b) Learned 𝑓𝑎,𝑧 with respect to rotation speed 𝑛 (𝑧 = 0.2 m, 𝑣 𝑧 = 0 m/s), compared to 𝐶𝑇 measured in the bench test. (c) Heatmaps of learned 𝑓𝑎,𝑧 versus 𝑧 and 𝑣 𝑧 . (Left) ReLU network with spectral normalization. (Right) ReLU network without spectral normalization. . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.4 Baseline Controller and Neural-Lander performance in take-off and landing. Means (solid curves) and standard deviations (shaded areas) of 10 trajectories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.5 Neural-Lander performance in take-off and landing with different DNN capacities. 1 layer means a linear mapping 𝑓ˆ𝑎 = 𝐴𝑥 + 𝑏; 0 layer means 𝑓ˆ𝑎 = 𝑏; Baseline means 𝑓ˆ𝑎 ≡ 0. . . . . . . . . . . . . . . . . . 38 3.6 Heatmaps of learned 𝑓ˆ𝑎,𝑧 and trajectory tracking performance. (a) Heatmaps of learned 𝑓ˆ𝑎,𝑧 versus 𝑥 and 𝑦, with other inputs fixed. (Left) ReLU network with spectral normalization. (Right) ReLU network without spectral normalization. (b) Tracking performance and statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.1 Overview of Neural-Swarm. We learn complex interaction between multirotors using heterogeneous deep sets and design an interactionaware nonlinear stable controller and a multi-robot motion planner (a). Our approach enables close-proximity flight (minimum vertical distance 24 cm) of heterogeneous aerial teams (16 robots) with significant lower tracking error compared to solutions that do not consider the interaction forces (b,c). . . . . . . . . . . . . . . . . . . . . . . . 43 xii 4.2 4.3 4.4 4.5 4.6 Illustration of Theorem 4.2. We first find a homeomorphism q𝐾 (·) between the original space and the latent space, and then find a continuous function 𝜌(·) ¯ such that 𝜌(q ¯ 𝐾 (·)) = ℎ(·). . . . . . . . . . . Example for Stage 1 of our motion planning with learned dynamics. Here, we have an initial solution for a small (blue) robot and plan for a large (orange) robot. The created search tree of the large robot is color-coded by the magnitude of the interaction force on the orange robot. During the search, we reject states that would cause a significant change in the interaction force for the blue robot (edges in blue). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hierarchy of control and planning blocks with information flow for commands and sensor feedback. We use different colors to represent heterogeneous neighbors. Note that the neighbors will influence the vehicle dynamics (dashed arrow). . . . . . . . . . . . . . . . . . . . 𝑓𝑎,𝑧 prediction from the trained {ρsmall , ρlarge , ϕsmall , ϕlarge , ϕenv } networks. Each heatmap gives the prediction of 𝑓𝑎,𝑧 of a vehicle in different horizontal and vertical (global) positions. The (global) position of neighboring drones are represented by drone icons. . . . . . . . . Example motion planning result for a three-robot swapping task in 2D (blue and orange: small robots; green: large robot). Top row: 𝑦𝑧state space plot, where the arrows indicate the velocities every second, and the circles show the robot collision boundary shape at the middle of the task. Bottom row: interaction force for each robot over time (dashed: desired limit per robot). Left: Sampling-based motion planning with neural network to compute trajectories where the large robots moves below the small robots. Middle: Refined trajectories using SCP (dashed) and tracked trajectories (solid). Right: Planned trajectories when ignoring interaction forces (dashed) and tracked trajectories (solid). In this case, a dangerous configuration is chosen where the large robot flies on top of the small robots, exceeding their disturbance limits of 5 g. . . . . . . . . . . . . . . . . . . . . . . . . 55 60 63 68 71 xiii 4.7 4.8 4.9 5.1 5.2 Motion planning results for different scenarios (e.g., SSL refers to two small robots and one large robot) comparing planning without neural network (BL) and planning with neural network (NN) over 5 trials. Top: Worst-case tracking error. Ignoring the interaction force can result in errors of over 10 cm. Bottom: Worst-case interaction force for small and large quadrotors. The baseline has significant violations of the interaction force bounds, e.g., the SL case might create interaction forces greater than 10 g for the small quadrotor. . . Flight test results comparing our solution with learned interaction compensation (NN) with the baseline (BL) in different scenarios. For each case, robots are initially arranged in a circle when viewed from above but at different 𝑧-planes and are tasked with moving linearly to the opposite side of the circle in their plane. For each swap, we compute the worst-case 𝑧-error of the lowest quadrotor and plot the data over six swaps. . . . . . . . . . . . . . . . . . . . . . . Generalization to a team of five multirotors. Three small multirotor move in a vertical ring and two large multirotor move in a horizontal ring. The maximum 𝑧-error of a small multirotor in the vertical ring with powerful motors is reduced from 10 cm to 5 cm and 𝑓𝑎 is predicted accurately. . . . . . . . . . . . . . . . . . . . . . . . . . . Visualization of wind effect for a drone flying in Caltech Real Weather Wind Tunnel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Agile flight through narrow gates. (A) Caltech Real Weather Wind Tunnel system, the quadrotor UAV, and the gate. In our flight tests, the UAV follows an agile trajectory through narrow gates, which are slightly wider than the UAV itself, under challenging wind conditions. (B-C) Trajectories used for the gate tests. In (B), the UAV follows a figure-8 through one gate, with wind speed 3.1 m/s or timevarying wind condition. In (C), the UAV follows an ellipse in the horizontal plane through two gates, with wind speed 3.1 m/s. (D-E) Long-exposure photos (with an exposure time of 5 s) showing one lap in two tasks. (F-I) High-speed photos (with a shutter speed of 1/200s) showing the moment the UAV passed through the gate and the interaction between the UAV and the wind. . . . . . . . . . . . . 72 73 74 79 82 xiv 5.3 Offline meta-learning and online adaptive control design. (A) The online adaptation block in our adaptive controller. Our controller leverages the meta-trained basis function 𝜙, which is a wind-invariant representation of the aerodynamic effects, and uses composite adaptation (that is, including tracking-error-based and prediction-errorbased adaptation) to update wind-specific linear weights 𝑎. ˆ The output of this block is the wind-effect force estimate, 𝑓ˆ = 𝜙 𝑎. ˆ (B) The illustration of our meta-learning algorithm DAIML. We collected data from wind conditions {𝑤 1 , · · · , 𝑤 𝐾 } and applied Algorithm 5.1 to train the 𝜙 net. (C) The diagram of our control method, where the grey part corresponds to (A). Interpreting the learned block as an aerodynamic force allows it to be incorporated into the feedback control easily. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 5.4 Training data collection. (A) The xyz position along a two-minute randomized trajectory for data collection with wind speed 8.3 km/h (3.7 m/s), in the Caltech Real Weather Wind Tunnel. (B) A typical 10-second trajectory of the inputs (velocity, attitude quaternion, and motor speed PWM command) and label (offline calculation of aerodynamic residual force) for our learning model, corresponding to the highlighted part in (A). (C) Histograms showing data distributions in different wind conditions. (C) Left: distributions of the 𝑥-component of the wind-effect force, 𝑓𝑥 . This shows that the aerodynamic effect changes as the wind varies. (C) Right: distributions of the pitch, a component of the state used as an input to the learning model. This shows that the shift in wind conditions causes a distribution shift in the input. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 xv 5.5 5.6 5.7 5.8 6.1 6.2 t-SNE plots showing the evolution of the linear weights (𝑎 ∗ ) during the training process. As the number of training epochs increases, the distribution of 𝑎 ∗ becomes more clustered with similar wind speed clusters near each other. The clustering also has a physical meaning: after training convergence, the right top part corresponds to a higher wind speed. This suggests that DAIML successfully learned a basis function 𝜙 shared by all wind conditions, and the wind-dependent information is contained in the linear weights. Compared to the case without the adversarial regularization term (using 𝛼 = 0 in Algorithm 5.1), the learned result using our algorithm is also more explainable, in the sense that the linear coefficients in different conditions are more disentangled. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Depiction of the trajectory tracking performance of each controller in several wind conditions. The baseline nonlinear controller can track the trajectory well, however, the performance substantially degrades at higher wind speeds. INDI, L1 , and Neural-Fly-Constant have similar performance and improve over the nonlinear baseline by estimating the aerodynamic disturbance force quickly. Neural-Fly and Neural-Fly-Transfer use a learned model of the aerodynamic effects and adapt the model in real time to achieve lower tracking error than the other methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Mean tracking errors of each lap in different wind conditions. This figure shows position tracking errors of different methods as wind speed increases. Solid lines show the mean error over 6 laps and the shade areas show standard deviation of the mean error on each lap. The grey area indicates the extrapolation region, where the wind speeds are not covered in training. Our primary method (NeuralFly) achieves state-of-the-art performance even with a strong wind disturbance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Outdoor flight setup and performance. Left: In outdoor experiments, a GPS module is deployed for state estimation, and a weather station records wind profiles. The maximum wind speed during the test was around 17 km/h (4.9 m/s). Right: Trajectory tracking performance of Neural-Fly. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Motivating example: drone landing trajectories with different speeds. 117 Episodic learning and control diagram. . . . . . . . . . . . . . . . . 121 xvi 6.3 7.1 7.2 8.1 8.2 8.3 9.1 9.2 10.1 10.2 11.1 Experimental results. Top: The pendulum task: (a)-(c) are the phase portraits of angle and angular velocity; Blue curve is tracking the desired trajectory with perfect knowledge of 𝑓 ; the worst-case possible trajectory is calculated according to the uncertainty bound; heatmap is the difference between predicted dynamics (the wind) and the ground truth; and (d) is the tracking error and the maximum density ratio. Bottom: The drone landing task: (e)-(g) are the phase portraits with height and velocity; heatmap is difference between the predicted ground effect) and the ground truth; (h) is the comparison with GPs in landing time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Detection results of the milk object of three deep learning models in two images. In the left image, these three models disagree much more than the right image. . . . . . . . . . . . . . . . . . . . . . . . 134 From Part I to future research (more data required from left to right). 136 Motivations of building the interface between learning and control, and the organization of Part II. . . . . . . . . . . . . . . . . . . . . . 141 A trajectory tracking problem to explain how MPC works. . . . . . . 150 An example illustrating why beyond regret. . . . . . . . . . . . . . . 151 The motivation of Chapter 9. . . . . . . . . . . . . . . . . . . . . . . 154 Drone experiment results. . . . . . . . . . . . . . . . . . . . . . . . 173 Controllable canonical form. . . . . . . . . . . . . . . . . . . . . . . 189 Numerical results of Optimistic ROBD in 1-d and 2-d systems, with different 𝜆. LC means the best linear controller in hindsight and OPT means the global optimal controller in hindsight. LC is numerically searched in stable linear controller space. We consider two different types of 𝑤 𝑡 : 𝑤 𝑡 is i.i.d. random/random walk, and also two different settings: 𝑤 𝑡 is known/unknown at step 𝑡. . . . . . . . . . . . . . . . . 195 The power of predictions in online tracking. The left five figures show the desired trajectory (blue) and the actual trajectories (orange). The rightmost figure shows the cost difference (regret) between MPC using 𝑘 predictions and the offline optimal policy. Note that the y-axis of the rightmost figure is in log-scale. . . . . . . . . . . . . . 234 xvii LIST OF TABLES Number Page 1.1 Number of deaths from different transportation methods every 109 miles in the United States. . . . . . . . . . . . . . . . . . . . . . . . 2 2.1 The summary of Part I. . . . . . . . . . . . . . . . . . . . . . . . . . 19 4.1 System identification of the used quadrotors. . . . . . . . . . . . . . 65 4.2 12 scenarios for data collection. . . . . . . . . . . . . . . . . . . . . 66 4.3 Ablation analysis. Top: 𝐿 = 4 and 𝐻 varies. Bottom: 𝐻 = 20 and 𝐿 varies. The error is the mean squared error (MSE) between 𝑓𝑎,𝑧 prediction and the ground truth. . . . . . . . . . . . . . . . . . . . . 70 5.1 Tracking error statistics in cm for different wind conditions. Two metrics are considered: root-mean-square (RMS) and mean. . . . . . 104 9.1 OMAC with convex loss. . . . . . . . . . . . . . . . . . . . . . . . . 162 9.2 OMAC with element-wise convex loss. . . . . . . . . . . . . . . . . 168 9.3 OMAC with bilinear model. . . . . . . . . . . . . . . . . . . . . . . 169 9.4 OMAC with deep learning. . . . . . . . . . . . . . . . . . . . . . . . 172 9.5 ACE results in pendulum (top) and drone (bottom) experiments from 10 random seeds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 1 Chapter 1 INTRODUCTION Since the 2010s, deep-learning-based decision making methods have made exciting progress in many domains such as playing real-time strategy games. For example, in 2016, AlphaGo (Silver et al., 2016) developed by DeepMind could master the game of Go and defeat the human champion with a large margin, which had been widely considered as a nearly impossible artificial intelligence task for a few decades. Later on, this model-free reinforcement learning methodology quickly generalized to multi-agent game settings: in 2019, another DeepMind team developed AlphaStar (Vinyals et al., 2019) and achieved grandmaster-level performance in StarCraft, which is a challenging real-time multi-agent strategy game. Besides playing games, deep-learning-based decision making methods have also shown fabulous progress in robotics. For instance, in 2022, a deep-learning-based policy (Wurman et al., 2022) developed by researchers at Sony AI outraced the world’s best drivers in the PlayStation game Gran Turismo, which faithfully reproduces the non-linear control challenges of real race cars while also encapsulating the complex multi-agent interactions. Such a novel learned capability also exists in real-world robotic control tasks. For example, in 2019, the OpenAI robotics team developed a deep-learning-based strategy which could solve non-trivial manipulation tasks, such as solving a Rubik’s cube only using one hand (Akkaya et al., 2019). Although those novel capabilities in decision making enabled by deep learning are fascinating, they are still limited in either game or highly controllable or non-safetycritical settings. In those settings, data collection is relatively cheap and theoretical guarantees are typically not required. For example, in the game of Go or StarCraft, a crazy or unexpected move by the AI agent is acceptable; similarly, for a robotic manipulator solving a Rubik’s cube, dropping the cube is not a big deal. However, this is not the case in real-world tasks involving high stakes, such as robotic control in hazardous environments, health care, and space exploration. In these tasks, computational resources are limited, data collection is expensive, and theoretical guarantees such as safety are extremely important. Unfortunately, deeplearning-based methods are not ready to be deployed in those settings. For example, 2 when visiting Caltech, an aerospace director said: “I would love to incorporate deep learning into the design, manufacturing, and operations of our aircraft. But I need some guarantees.” Such a concern is definitely not unfounded, because the aerospace industry has spent over 60 years making the airplane safer and safer such that the modern airplane is one of the safest transportation methods. In a recent talk “Can We Really Use Machine Learning in Safety-Critical Systems?”1 at UCLA IPAM, Prof. Richard Murray discussed the number of deaths from transportation every 109 miles in the United States (see Table 1.1). Human-driven car Buses and trains Airplane Self-driving car 7 0.1-0.4 0.07 ? Table 1.1: Number of deaths from different transportation methods every 109 miles in the United States. Based on this analysis, if we travel from LA to San Francisco, on average, taking a flight is 100 times safer than driving ourselves (also faster). Note that the question mark “?” in Table 1.1 indicates that the self-driving car industry has not tested enough to even get a statistically reliable number (i.e., the total mileage is much less than 109 miles), although there have already been multiple lethal accidents from deep-learning-based self-driving cars (Greenblatt, 2016). To summarize, with the unprecedented advances of modern machine learning comes the tantalizing possibility of smart data-driven autonomous systems across a broad range of real-world settings, but we are still begging for answers to the following question: For deep-learning-based autonomous systems, how do we ensure a comparable level of safety to human or classic methods while maintaining advantages from deep learning? To answer this question, we must confront several key challenges before the widespread deployment of machine learning: (1) Real-world systems have complex and timevarying uncertainties, which require robust and adaptive learning methods. (2) 1 Video link: https://youtu.be/Wi8Y---ce28 3 High-stakes tasks need control-theoretic guarantees such as safety and stability, which require a framework to unify learning and control theoretical results. (3) In order to be useful, such a theory needs to enable tractable system design. In light of these challenges, this thesis is to lay the groundwork for learning and control that enables the long-term autonomy of complex real-world systems, such as urban air mobility, space exploration, and medical robots. In pursuit of this vision, this thesis is centered around establishing a unified algorithmic and theoretical framework that can simultaneously reason over trade-offs in learning and control and using that theory to enable tractable system design to unlock new capabilities in autonomous systems. The proposed unified framework supported by rigorous yet practical theory advances the state of the art in that complex problems can be simplified greatly in a modularized way, making it much easier to reason about high-level goals. 1.1 Thesis Organization This thesis consists of two parts. In this section, we briefly introduce these two parts and their organizations. Part I: Neural-Control Family: Deep-Learning-Based Nonlinear Control with Learned Robotic Agility Part I will build a holistic view on Neural-Control Family, which is a family of deep-learning-based nonlinear controllers with not only theoretical guarantees but also learned novel capabilities in agile robotic control. A summary of Part I can be found in Table 2.1 in Chapter 2. In particular, Chapter 2 will give an overview about Neural-Control Family, define the research problem in general robotic systems and specific drone systems, and introduce necessary preliminaries. In Chapter 3, Chapter 4, Chapter 5, and Chapter 6, we will introduce four “children” in the family. Finally, Chapter 7 discusses limitations and future research directions. Part II: Unifying Interfaces Between Learning and Control Theory Part II will discuss three unifying interfaces between learning and control theory. Such interfaces deepen the fundamental connections between the two fields, enable much more efficient translation, and bring new perspectives and algorithmic principles. 4 A summary of Part II can be found in Fig. 8.1 in Chapter 8. In particular, Part II will focus on joint end-to-end learning and control theoretical guarantees in online decision-making problems. Chapter 8 will give an overview and introduce necessary preliminaries. In Chapters 9 to 11, we introduce the three interfaces, respectively. Finally, Chapter 12 discusses future research directions. 1.2 Thesis Summary and Contributions We summarize this thesis and its contributions from three aspects: robotics, theory, and algorithm. Robotics: Agile Control with Novel Capabilities and Certifiable Guarantees This thesis focuses on reliable agile robot control with new capabilities which have not been achieved by either pure control or learning methods. In particular, the Neural-Control Family in Part I demonstrates state-of-the-art flight control performance in the presence of unknown unsteady aerodynamics. Beyond flight control, the Neural-Control idea has been successfully adopted by many other researchers (e.g., for legged, underwater, soft robots), companies, and national agencies, including JPL and DARPA. For the first time, Neural-Lander (Chapter 3) enables agile maneuvers only a few millimeters from the ground; Neural-Swarm (Chapter 4) enables close-proximity flights (minimum distance 24cm) of a heterogeneous aerial swarm (16 drones), while prior works have to keep a safe distance of 60cm even with 2-3 drones; Neural-Fly (Chapter 5) achieves accurate adaptive tracking with an average error down to 3cm in gusty wind conditions up to 12m/s, which improves baselines by one order of magnitude. All these systems run Deep Neural Networks (DNNs) onboard in the control loop but with safety, stability, and robustness guarantees. The connection between these robotic capabilities and the unified theoretical and algorithmic framework is two-fold: (1) The unified framework allows tractable, efficient, and safe real-world deployment due to its modularity and end-to-end guarantees. For instance, Neural-Lander/Fly only needs 5/12-minute training data. (2) Pushing the boundaries of agile robot control requires and inspires new unified perspectives on control and learning. For example, for agile maneuvers in timevarying wind conditions, we have to extract common representations shared by all conditions, which requires and inspires the novel meta-adaptive control framework (Chapters 5 and 9). This thesis systematically introduces the unified framework and illustrates why it is necessary and sufficient for unlocking new capabilities in 5 real-world autonomous systems. Theory: Bridge Learning and Control in a Unified Framework The theoretical parts of this thesis are driven by the quest for a unified framework to characterize and safeguard learned control systems. This thesis is distinctive in three ways. First, it studies how to integrate advanced concepts such as deep learning and nonlinear control, for which unified learning-theoretic and control-theoretic analysis is still in its infancy. Second, it studies how to reconcile fairly fragmented analyses between the two fields (e.g., what does exponentially stable control imply about online regret?), which will deepen the fundamental connections and enable more efficient translation between the two fields. Third, it allows tractable real-world implementations with novel capabilities. From statistical learning with regularized DNNs to nonlinear stability and safety. From both computational and statistical standpoints, the learning methods must incorporate prior physics in real-world dynamical systems. For example, the Neural-Control Family in Part I only learns complex aerodynamics which is hard to model using classic approaches. A central question in this diagram is how to leverage the structure in prior physics and translate statistical learning results to stability and safety results in control. Chapter 3 connects generalization bounds of properly spectrally regularized DNNs with input-to-state stability (ISS) bounds in nonlinear control. The experiment found that poorly regularized DNNs without desired Lipschitz properties can cause drones to crash because small noise may destroy local stability. Chapter 4 and Chapter 5 extend this result to multi-agent and real-time adaptation settings. Chapter 6 then generalizes this idea to sequential control problems, by connecting uncertainty bounds under domain shift with safety bounds. These results not only provide sufficient conditions for stable and safe learning-based nonlinear control but also enable roboticists to reason more cleanly between learning and control performance. Meta-adaptive control. In order to have rapidly adaptable autonomous systems operating in changing environments (e.g., varying wind conditions for drones), it is crucial to extract common representations from all environments. However, existing theoretical results focus on either representation/meta-learning with i.i.d. data (i.e., no dynamics) or adaptive control in a single environment. Therefore, this thesis proposes a novel meta-adaptive control framework, where meta-learning optimizes a representation shared by all environments. Then the representation is seamlessly 6 used as basis functions for adaptive nonlinear control in low-dimensional space (Chapters 5 and 9). Meta-adaptive control provides both new theoretical results (the first end-to-end non-asymptotic guarantee for multi-task nonlinear control in Chapter 9) and new capabilities in robotics (Chapter 5), both beyond a simple stack of meta-learning and adaptive control. Beyond no-regret: competitive online control. The recent progress in learningtheoretic analyses for online control begs the question: What learning-theoretic guarantees do we need for real-world systems? Existing results focus on linear systems with classic no-regret guarantees. However, no-regret policies compare with the optimal static controller in a specific class (typically linear policy class), which could be arbitrarily suboptimal in real-world nonlinear or time-varying systems (see examples in Chapter 10). Therefore, this thesis studies competitive online control, which uses stronger metrics beyond regret, i.e., competitive ratio and dynamic regret (competitive difference). Those metrics directly compare with the global optimum, thus naturally suitable for real-world systems. This thesis designs competitive policies in time-varying and nonlinear systems, via novel reductions from online optimization to control (Chapters 10 and 11). Moreover, we show new fundamental limits via novel lower bounds, e.g., the impact of delay (Chapter 10). Understand MPC from learning perspectives. Another critical question is begged in online learning and control: Do established control methods have strong learning guarantees? In particular, Model Predictive Control (MPC) has been one of the most successful methods in industrial control since the 1980s. However, many learning theorists are studying RL algorithms, but few are analyzing MPC and why it is so powerful. To close this gap, this thesis proves the first non-asymptotic guarantee for MPC (Chapter 11), showing that MPC is near-optimal in the sense of dynamic regret in online LQR control with predictable disturbance. Then Chapter 11 further extends to settings with inexact predictions and LTV systems, in the competitive online control framework. These results found common ground for learning and control theory and imply fundamental algorithmic principles. Algorithm: Reliable Decision Making Methods with Certifiable Guarantees The algorithmic goal of this thesis is to translate the improved theoretical understand (in particular, the established unified abstractions for learning and control) into tractable algorithms that inherit both the flexibility and accuracy of learning and the certifiable guarantees of control. 7 Stable, robust, and adaptive deep-learning-based nonlinear control. As a powerful yet obscure black box, deep learning must incorporate principled regularizations for reliable and tractable real-world implementations. First, Part I uses Lipschitzconstrained DNNs to learn the residual dynamics, where the constraint is specified by the established unified framework between regularized learning and nonlinear stability. Such control-theoretic regularizations ensure the stability and robustness of deep-learning-based nonlinear controllers, and also improve generalization to unseen data (Chapters 3 to 5). The second type of regularization for learning is invariance. Neural-Swarm (Chapter 4) leverages the permutation-invariance of swarm dynamics to develop Heterogeneous Deep Sets for learning swarm interactions in a decentralized manner, which enables generalization from 1-3 robots in training to 5-16 in testing. Neural-Fly (Chapter 5) proposes the Domain Adversarially Invariant Meta-Learning (DAIML) algorithm, which uses an adversarial regularizer to train a domain-invariant representation for online adaptation. These principled regularizations are critical for efficient and safe real-world deployment. Safe exploration in nonlinear dynamical systems. Safety-critical tasks such as space exploration and agile drone landing are challenging because (1) there is no expert collecting data, and (2) there exists non-trivial domain shift (e.g., sim2real, exploring to unseen states). Therefore, we have to quantify uncertainty and design safe learning and exploration methods. Built on the connection between uncertainty bounds and safety bounds in the unified framework, Chapter 6 proposes the first safe exploration algorithm with robust end-to-end learning and control guarantees under domain shift, in both deterministic and stochastic settings. The key idea is to derive uncertainty bounds from distributionally robust learning and use them in robust trajectory optimization. References Akkaya, Ilge et al. (2019). Solving Rubik’s cube with a robot hand. In: arXiv preprint arXiv:1910.07113. Greenblatt, Nathan A. (2016). Self-driving cars and the law. In: IEEE Spectrum 53.2, pp. 46–51. Silver, David et al. (2016). Mastering the game of Go with deep neural networks and tree search. In: Nature 529.7587, pp. 484–489. Vinyals, Oriol et al. (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. In: Nature 575.7782, pp. 350–354. 8 Wurman, Peter R. et al. (2022). Outracing champion Gran Turismo drivers with deep reinforcement learning. In: Nature 602.7896, pp. 223–228. Part I Neural-Control Family: Deep-Learning-Based Nonlinear Control with Learned Robotic Agility 9 10 In this part of the thesis, we will build a holistic view on Neural-Control Family, which is a family of deep-learning-based nonlinear controllers with theoretical guarantees and learned new capabilities in robotics. The general idea is that we have a mixed dynamical system 𝑥¤ = 𝑓known (𝑥, 𝑢) + 𝑓unknown (𝑥, 𝑢) and our goal is to model 𝑓unknown using modern deep learning tools and design control laws. We will show the importance and necessity of building a unified framework of control and learning, e.g., incorporating control-theoretic constraints in learning. In Chapter 2, we will give an overview about the Neural-Control Family, which defines the research problem in general robotic systems and specific drone systems. In Chapter 3, Chapter 4, Chapter 5 and Chapter 6, we will introduce four “children” in the family. Finally, in Chapter 7, we will discuss future research directions. This part is mainly based on the following papers: Liu, Anqi, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue (2020). Robust regression for safe exploration in control. In: Learning for Dynamics and Control. PMLR, pp. 608–619. url: https://proceedings.mlr. press/v120/liu20a.html. Nakka, Yashwanth Kumar, Anqi Liu, Guanya Shi, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung (2021). Chance-constrained trajectory optimization for safe exploration and learning of nonlinear Systems. In: IEEE Robotics and Automation Letters 6.2, pp. 389–396. doi: 10.1109/LRA.2020.3044033. O’Connell, Michael, Guanya Shi, Xichen Shi, Kamyar Azizzadenesheli, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung (2022). Neural-Fly enables rapid learning for agile flight in strong winds. In: Science Robotics 7.66, eabm6597. doi: 10.1126/scirobotics.abm6597. Shi, Guanya, Wolfgang Hönig, Xichen Shi, Yisong Yue, and Soon-Jo Chung (2022). Neural-Swarm2: Planning and control of heterogeneous multirotor swarms using learned interactions. In: IEEE Transactions on Robotics 38.2, pp. 1063–1079. doi: 10.1109/TRO.2021.3098436. Shi, Guanya, Wolfgang Hönig, Yisong Yue, and Soon-Jo Chung (2020). NeuralSwarm: Decentralized close-proximity multirotor control using learned interactions. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 3241–3247. doi: 10.1109/ICRA40945.2020.9196800. Shi, Guanya, Xichen Shi, Michael O’Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung (2019). Neural Lander: Stable drone landing control using learned dynamics. In: 2019 International Conference on Robotics and Automation (ICRA). IEEE, pp. 9784–9790. doi: 10.1109/ICRA.2019.8794351. 11 Chapter 2 OVERVIEW Notations. In Part I, ∥ · ∥ indicates the 2-norm of a vector or the induced 2-norm of a matrix. 𝜆 min (·) and 𝜆 max (·) denote the minimum or the maximum eigenvalue of a real symmetric matrix. 𝑎 ◦ 𝑏 refers to the composition of two functions 𝑎 and 𝑏. (·) † is the Moore-Penrose inverse of a matrix. Finally, 𝑥¤ denotes the time derivative of a variable 𝑥, i.e., 𝑥¤ = 𝑑𝑥 𝑑𝑡 . 2.1 A Mixed Model for Robot Dynamics and the Nonlinear Control Law Often times in real-world robotic systems, there are complicated uncertainties in dynamics but we also have some prior knowledge. For example, as shown in later chapters, for a quadrotor, it is very easy and standard to model the rigid body dynamics in SE(3). However, it is in general much more difficult, if not impossible, to model the aerodynamics such as ground effect, down wash effect, air drag and wind effect (G. Shi, X. Shi, et al., 2019; G. Shi, Hönig, Yue, et al., 2020; G. Shi, Hönig, X. Shi, et al., 2022; O’Connell et al., 2022). To systematically and simultaneously study the known part and the unknown part in the dynamics, in this part of the thesis, we will consider a mixed model for robot dynamics in the form of the Euler-Lagrange equation. In particular, we consider the following robot dynamics model: 𝑀 (𝑞) 𝑞¥ + 𝐶 (𝑞, 𝑞) ¤ 𝑞¤ + 𝑔(𝑞) = 𝐵𝑢 + 𝑓 (𝑞, 𝑞, ¤ 𝑢, 𝑡) | {z } (2.1) unknown where 𝑞, 𝑞, ¤ 𝑞¥ ∈ R𝑛 are the 𝑛 dimensional position, velocity, and acceleration vectors, 𝑀 (𝑞) is the symmetric, positive definite inertia matrix, 𝐶 (𝑞, 𝑞) ¤ is the Coriolis 𝑚 matrix, 𝑔(𝑞) is the gravitational force vector, 𝑢 ∈ R is the control force and 𝐵 ∈ R𝑛×𝑚 is the actuation matrix. Most importantly, 𝑓 (𝑞, 𝑞, ¤ 𝑢, 𝑡) incorporates unmodeled dynamics, and it is potentially time-variant (i.e., 𝑓 depends on time 𝑡). Note that one important fact about the Euler-Lagrange equation in Eq. (2.1) is that 𝑀¤ − 2𝐶 is a skew-symmetric matrix. We will heavily use this fact for nonlinear stability analysis in the following chapters. 12 Tasks Stable and robust trajectory tracking. In Part I, we consider the trajectory tracking task, where we have a reference (or desired) trajectory 𝑞 𝑑 (𝑡) which we hope the robot can precisely track. In particular, our goal is to guarantee that the controller is (globally) 𝑠𝑡𝑎𝑏𝑙𝑒 and 𝑟𝑜𝑏𝑢𝑠𝑡. We say the closed-loop system is globally asymptotically stable if ∥𝑞 − 𝑞 𝑑 ∥ → 0 asymptotically wherever 𝑞 starts (i.e., whatever 𝑞(0) is). Furthermore, we say the closed-loop system is exponentially stable if ∥𝑞 − 𝑞 𝑑 ∥ → 0 exponentially. Note that exponential stability typically implies robustness in the sense of input-to-state stability (ISS) (Khalil, 2002). In the context of robotic trajectory tracking, it means that ∥𝑞 − 𝑞 𝑑 ∥ will exponentially converge to an error ball 𝜖, whose size is typically proportional to the size of extra disturbance on top of 𝑓 or the learning error (the gap between the estimation of 𝑓 and the true 𝑓 ) (Slotine and Li, 1991; O’Connell et al., 2022). Therefore, having exponential stability is critical for learning-based control because an exponentially stable controller can handle errors from imperfect learning. Safe motion planning and control. At this stage, readers may have the question “where is the reference trajectory 𝑞 𝑑 from?” The answer to this question is exactly the motion planning problem, whose goal is to find a path 𝑞 𝑑 (𝑡) for the robot to track. Typically, we want to make sure such a path is safe in the state space. Formally speaking, [𝑞 𝑑 ; 𝑞¤ 𝑑 ] ∈ S where S is some safe set. For example, for multi-robot planning we must consider collision and obstacle avoidance. It is worth noting that safety is not only important in planning, but also in feedback control. Once we have 𝑞 𝑑 , the next question is naturally how to design a controller such that the actual trajectory is also safe, i.e., [𝑞; 𝑞] ¤ ∈ S. Note that it is relatively easy if the tracking is perfect (i.e., 𝑞 = 𝑞 𝑑 ), but much more challenging if there exist uncertainties. Remarks and Assumptions Remark (nonlinear, time-variant, unknown and non-affine systems). Controlling the system in Eq. (2.1) is challenging in four folds. (1) The 𝑓 term is unknown and could be very complicated. (2) Both the known part and the unknown part ( 𝑓 (𝑞, 𝑞, ¤ 𝑢, 𝑡)) are highly nonlinear. (3) The 𝑓 term is in generally time-variant. Note that most of theories developed in Part I still hold if the nominal part (i.e., 𝑀, 𝐶, 𝑔, 𝐵) is also time-variant, and we assume they are time-invariant for simplicity 13 (also for all our experiments the nominal part is indeed time-invariant). (4) The system is non-affine in 𝑢, and in particular, the unknown term 𝑓 (𝑞, 𝑞, ¤ 𝑢, 𝑡) might depend on 𝑢. Controlling a non-affine linear system is notoriously challenging because 𝑢 often appears in both sides of a controller equation (e.g., 𝐵𝑢 = 𝜋(𝑥, 𝑢)). Assumption 2.1 (full actuation). In Part I, we focus on fully actuated systems, which means rank(𝐵) = 𝑛 and 𝑚 ≥ 𝑛. Generally speaking, guaranteed safe learning and control (especially adaptive control) in nonlinear systems with under actuation is very challenging (Lopez and Slotine, 2021) and left for future directions. We will revisit this point in Chapter 7. In the second part of this thesis (Part II), we will study learning and control in some under-actuated linear systems. Assumption 2.2 (full observation). In this thesis, we focus on fully observable settings, which means we can directly observe the state of the robot (𝑞 and 𝑞). ¤ The fully observable assumption matches robotic experiments in this thesis, because we deploy external motion capture (mocap) systems to localize the robot in SE(3). Nevertheless, note that some of control methods developed in Part I are robust for outdoor experiments without mocap systems, where we first use estimators (e.g., extended Kalman filter) to estimate robot state and then use these estimations for control (e.g., see outdoor experiments in Chapter 5). Having end-to-end guarantees in nonlinear partially observable systems is another interesting future research topic (see more discussions in Chapter 7). The Nonlinear Control Law In this subsection, we introduce the basic structure of controllers covered in Part I. Note that each child in the family (Neural-Lander/Swarm/Fly) has different downstream control structures but the high-level structures are consistent. We start by defining some notations. The composite velocity tracking error term 𝑠 and the reference velocity 𝑞¤𝑟 are defined such that 𝑠 = 𝑞¤ − 𝑞¤𝑟 = 𝑞¤̃ + Λ𝑞˜ (2.2) where 𝑞˜ = 𝑞 − 𝑞 𝑑 is the position tracking error and Λ is a positive definite gain matrix. Note when 𝑠 exponentially converges to an error ball around 0, 𝑞 will exponentially converge to a proportionate error ball around the desired trajectory 𝑞 𝑑 (𝑡) (see Lemma 2.3). Formulating our control law in terms of the composite velocity error 𝑠 simplifies the analysis and gain tuning without loss of rigor. 14 Our nonlinear control structure is given by 𝐵𝑢 = 𝑀 (𝑞) 𝑞¥𝑟 + 𝐶 (𝑞, 𝑞) ¤ 𝑞¤𝑟 + 𝑔(𝑞) | {z } nonlinear feedforward −𝐾 𝑠 |{z} − 𝑓ˆ |{z} , (2.3) linear feedback learned feedforward where 𝐾 is a positive definite control gain matrix and 𝑓ˆ is a learned term to “compensate” the unknown term 𝑓 in real time. Note that this nonlinear controller includes feedback on position and velocity errors and feedforward terms to account for known dynamics and desired acceleration, which allows good tracking of dynamic trajectories in the presence of nonlinearities (e.g., 𝑀 (𝑞) and 𝐶 (𝑞, 𝑞) ¤ are non-constant in drone attitude control). Moreover, this control law also “predicts” and then compensates the unknown dynamics via the 𝑓ˆ term. We will have different structures for 𝑓ˆ in different cases, e.g., a fixed regularized DNN in Neural-Lander (Chapter 3) and a real-time adaptive DNN in Neural-Fly (Chapter 5). 2.2 Example: Quadrotor Dynamics and Control In this section, we will provide a concrete and detailed example for the general robot dynamics model (Eq. (2.1)) in Section 2.1: quadrotor dynamics with unknown aerodynamics. Most of experiments in Part I also focus on drones. Consider states given by global position, 𝑝 ∈ R3 , velocity 𝑣 ∈ R3 , attitude rotation matrix 𝑅 ∈ SO(3), and body angular velocity 𝜔 ∈ R3 . Then dynamics of a quadrotor are 𝑝¤ = 𝑣, 𝑚 𝑣¤ = 𝑚𝑔 + 𝑅 𝑓𝑢 + 𝑓𝑎 , (2.4a) 𝑅¤ = 𝑅𝑆(𝜔), 𝐽 𝜔¤ = 𝐽𝜔 × 𝜔 + 𝜏 + 𝜏𝑎 , (2.4b) where 𝑚 is the mass, 𝐽 is the inertia matrix of the quadrotor, 𝑆(·) is the skewsymmetric mapping, 𝑔 is the gravity vector, 𝑓𝑢 = [0, 0, 𝑇] ⊤ and 𝜏 = [𝜏𝑥 , 𝜏𝑦 , 𝜏𝑧 ] ⊤ are the total thrust and body torques from four rotors predicted by the nominal model. In particular, typical quadrotor control input uses squared motor speeds 𝑢 = [𝑛21 , 𝑛22 , 𝑛23 , 𝑛24 ] ⊤ , and is linearly related to the total thrust and body torques: 𝑇   𝑐 𝑐𝑇 𝑐𝑇 𝑐𝑇  𝑛21     𝑇       𝜏𝑥   𝑐𝑇 𝑙arm 0 −𝑐𝑇 𝑙arm  𝑛22    = 0 · 𝜏  −𝑐 𝑙 0 𝑐𝑇 𝑙arm 0  𝑛23   𝑦  𝑇 arm        𝜏𝑧   −𝑐𝑄 𝑐𝑄 −𝑐𝑄 𝑐𝑄  𝑛24     |{z} | {z } |{z} 𝜂𝑜 𝐵0 𝑢 (2.5) 15 where 𝑐𝑇 and 𝑐𝑄 are rotor force and torque coefficients, 𝑙 arm denotes the length of rotor arm, and 𝜂 𝑜 is the output wrench. Note that 𝐵0 is an invertible matrix. Control task. In this thesis, we are interested in the position trajectory tracking problem for a quadrotor. Namely, there exists a time-variant desired position trajectory 𝑝 𝑑 (𝑡) and our goal is to design a controller such that the tracking error 𝑝˜ = 𝑝 − 𝑝 𝑑 is near-zero in the presence of unknown aerodynamic disturbance 𝑓𝑎 and 𝜏𝑎 . Remark (generalize to other aerial robots). Generally speaking, the model in Eq. (2.4) and Eq. (2.5) can also model other aerial vehicles with more than four actuators (i.e., to make sure the actuation matrix satisfying rank(𝐵0 ) ≥ 4), such as hexacopter, octocopter, and some hybrid vehicles (e.g., flying cars in (X. Shi et al., 2018)). Residual Aerodynamic Force and Torque The key difficulty of precise drone control is the influence of unknown disturbance forces 𝑓𝑎 = [ 𝑓𝑎,𝑥 , 𝑓𝑎,𝑦 , 𝑓𝑎,𝑧 ] ⊤ and torques 𝜏𝑎 = [𝜏𝑎,𝑥 , 𝜏𝑎,𝑦 , 𝜏𝑎,𝑧 ] ⊤ in Eq. (2.4), which originate from complex aerodynamic effects. For example, ground effect when the drone is flying close to the ground (or some other boundaries) (G. Shi, X. Shi, et al., 2019), downwash effect when one drone is flying below the other ones (G. Shi, Hönig, X. Shi, et al., 2022), and wind effect when the drone is flying in strong wind conditions (O’Connell et al., 2022). Generally speaking, the unknown force 𝑓𝑎 and torque 𝜏𝑎 depend on the drone’s states (position, velocity, attitude, angular velocity), rotor speeds (𝑛1 , · · · , 𝑛4 ), and external environmental conditions (e.g., wind conditions). To summarize, 𝑓𝑎 and 𝜏𝑎 are unknown, nonlinear, and potentially time-variant. In Part I’s experiments, we will use deep learning approaches to model 𝑓𝑎 and 𝜏𝑎 . We define the deep learned estimation as 𝑓ˆ𝑎 and 𝜏ˆ𝑎 , respectively. Different chapters have different learning structures (e.g., a fixed constrained DNN in Chapter 3 and an online adaptive DNN in Chapter 5). Hierarchical Control Structure and the Reduction to General Robot Dynamics Note that Eq. (2.4) cannot be directly converted to Eq. (2.1) because quadrotor is an underactuated system in SE(3). However, typical quadrotor control consists of a cascaded or hierarchical control structure which separates the design of the position controller, attitude controller, and thrust mixer (allocation). For both 16 position controller and attitude controller, we can reduce them to Eq. (2.1). Position controller. When designing the position controller, we assume that the global rotor force vector 𝑅 𝑓𝑢 can be directly controlled. In other words, we can cast the position dynamics in Eq. (2.4a) into the form of Eq. (2.1), by taking 𝑞 = 𝑝, 𝑀 (𝑞) = 𝑚𝐼, 𝐶 (𝑞, 𝑞) ¤ ≡ 0, 𝐵 = 𝐼, 𝑓 = 𝑓𝑎 , and 𝑢 = 𝑅 𝑓𝑢 . Corresponding to Eq. (2.3), the nonlinear position controller has the following structure: 𝑓𝑑 = 𝑚 𝑝¥𝑟 − 𝑚𝑔 | {z } −𝐾 𝑠 |{z} − 𝑓ˆ𝑎 |{z} (2.6) feedforward linear feedback learned feedforward where 𝑠 = 𝑝¤ − 𝑝¤𝑟 = 𝑝¤̃ +Λ 𝑝˜ is the composite velocity error (see details in Section 2.1), 𝑓𝑑 = (𝑅 𝑓𝑢 ) 𝑑 is the desired rotor force and 𝑓ˆ𝑎 is a learned term to compensate the unknown disturbance force 𝑓𝑎 . Note that this controller includes linear feedback on position and velocity errors and feedforward terms to account for gravity and desired acceleration, which allows agile tracking of dynamic trajectories. Moreover, the 𝑓ˆ𝑎 term predicts and compensates the unknown disturbance. Attitude controller. With the desired rotor force 𝑓𝑑 = (𝑅 𝑓𝑢 ) 𝑑 , consequently, desired total thrust 𝑇𝑑 and desired force direction 𝑘ˆ 𝑑 can be computed as ˆ and 𝑘ˆ 𝑑 = 𝑓𝑑 /∥ 𝑓𝑑 ∥ , 𝑇𝑑 = 𝑓𝑑 · 𝑘, (2.7) with 𝑘ˆ being the unit vector of rotor thrust direction (typically 𝑧-axis in quadrotors). Using 𝑘ˆ 𝑑 and fixing a desired yaw angle, desired attitude 𝑅𝑑 can be deduced (Morgan et al., 2016). Then we need to design a nonlinear attitude controller which uses the desired torque 𝜏𝑑 from rotors to track 𝑅𝑑 (𝑡). The quadrotor attitude dynamics Eq. (2.4b) is also a special case of the general robot dynamics model Eq. (2.1) (Slotine and Li, 1991; Murray et al., 2017), but it is worth noting that different choices of rotation errors will yield different controllers. For example, one nonlinear attitude controller is given by (X. Shi et al., 2018): 𝜏𝑑 = 𝐽 𝜔¤ 𝑟 − 𝐽𝜔 × 𝜔𝑟 −𝐾𝜔 (𝜔 − 𝜔𝑟 ) | {z }| {z } −𝜏ˆ𝑎 |{z} , (2.8) nonlinear feedforward linear feedback learned feedforward where the reference angular rate 𝜔𝑟 is designed similar to Eq. (2.2), so that when 𝜔 → 𝜔𝑟 , exponential trajectory tracking of a desired attitude 𝑅𝑑 (𝑡) is guaranteed. Similarly, 𝜏ˆ𝑎 is a learned term to compensate the unknown torque 𝜏𝑎 . Similar to the position controller, the attitude controller also includes a nonlinear feedforward term 17 and a linear feedback term, and a 𝜏ˆ𝑎 term to account for the unknown aerodynamic torque. Thrust mixer (allocation). Finally, with the desired total thrust 𝑇𝑑 and the desired torque 𝜏𝑑 , we can straightforwardly solve the rotor speed 𝑛1 , · · · , 𝑛4 using Eq. (2.5). Note that if the control limit (0 ≤ 𝑛𝑖 ≤ 𝑛, ¯ ∀𝑖) becomes infeasible, a thrust mixer typically compromises some maneuverability (especially yaw) to optimize the flight performance. Some thrust mixers also consider motor delay to some extent, i.e., 𝑛1 , · · · , 𝑛4 cannot immediately change 𝑇 and 𝜏. We will discuss more about delay compensation in Chapter 4. Remark (benefits and limitations of the hierarchical control structure). The aforementioned hierarchical controller can be summarized as the following steps: 1. The position controller accounts for the unknown aerodynamic force 𝑓𝑎 , and computes the desired rotor force 𝑓𝑑 . 2. Compute the desired total thrust 𝑇𝑑 and the desired attitude 𝑅𝑑 . 3. Given 𝑅𝑑 , the attitude controller accounts for the unknown aerodynamic torque 𝜏𝑎 , and computes the desired torque 𝜏𝑑 . 4. Given 𝑇𝑑 and 𝜏𝑑 , the thrust mixer computes the rotor speed 𝑛1 , · · · , 𝑛4 . This procedure has the following benefits: • It is supported by rigorous theories, because both position and attitude controllers follow the standard fully-actuated robot dynamics in Eq. (2.1). • It is relatively easy to implement and debug. In particular, we do not need to customize drones or flight controllers since most commonly-used off-the-shelf controllers (such as PX4) support the “offboard” control mode, which only needs a position control law (i.e., 𝑇𝑑 and 𝑅𝑑 ) and automatically tracks the reference attitude 𝑅𝑑 . In experiments, the hierarchical control law is robust even flying outdoors. • In practice, the attitude controller is running at a much higher frequency (e.g., 400Hz) than the position controller (e.g., 50Hz), which matches the key assumption made in the cascading structure (i.e., the attitude controller runs fast enough to track the desired attitude generated from the upstream position controller). 18 However, this cascading structure implicitly assumes that the cross-error term between position and attitude control is bounded, and the attitude controller can track arbitrary trajectories generated from the upstream controller. Therefore, such a cascading structure may not be proper for acrobatic maneuvers such as flips. Alternatively, we can also use a single optimization-based controller such as model predictive control (MPC) to directly compute motor speed commands from desired trajectories (Tal and Karaman, 2021; Hanover et al., 2021). Compared to the cascading controller structure, it is very challenging to have stable and robustness guarantees because of the nonlinear and underactuated nature. Moreover, the single optimization-based controller requires full custom flight controllers and drones. 2.3 Organization of Part I: Unifying Learning and Nonlinear Control In Part I, we will discuss four cases of Eq. (2.1) with the focus on the unknown term 𝑓 (𝑞, 𝑞, ¤ 𝑢, 𝑡). Meanwhile, we also introduce different corresponding drone dynamics in Eq. (2.4) and controllers. Neural-Control Family consists of these four different cases. Each case (chapter) will answer the following questions: 1. What is the main challenge with the unknown term 𝑓 ? 2. How to design the learning and control strategy with theoretical guarantees? 3. In real-world robotic experiments, how does the proposed learning and control method compare with baselines and state-of-the-art methods? 4. What new capabilities (in terms of robotic agility) does the proposed learning and control method bring? In other words, why is the proposed learning and control method crucial? Answers to these questions can be summarized in Table 2.1. Before diving into details, here we highlight how we unify learning and nonlinear control in each chapter: Neural-Lander (Chapter 3) considers the model 𝑓 = 𝑓 (𝑞, 𝑞, ¤ 𝑢). We use a deep neural network 𝑓ˆ(𝑞, 𝑞, ¤ 𝑢) to approximate 𝑓 . In this case, the challenge is to guarantee that the controller in Eq. (2.3) is stable and robust with a DNN term 𝑓ˆ. To that end, we propose a control-theoretic Lipschitz-constrained deep learning framework 19 NeuralLander (Chapter 3) Challenge Solution 𝑓 depends on 𝑢 Control-theoretic Lipschitz constraints for stability guarantees New capabilities Agile maneuvers very close to the ground NeuralSwarm (Chapter 4) 𝑓 depends on Encoding permutaheterogeneous tion invariance in neighbors in a learning and control swarm Close-proximity heterogeneous swarm control Neural-Fly (Chapter 5) 𝑓 depends on Invariant metatime-variant learning and comenvironmental posite adaptation conditions Precise flight in timevariant strong winds Safe Exploration (Chapter 6) Uncertainty quantifiNo precation under domain collected data shift Aggressive maneuvers with end-to-end safety guarantees Table 2.1: The summary of Part I. 20 to guarantee the nonlinear stability and robustness. This learning framework also improves sampling efficiency and generalizability. Neural-Swarm (Chapter 4) considers a multi-agent interactive setting, i.e., 𝑓 = 𝑓 (𝑞, 𝑞, ¤ neighbors’ states). In this case, we aim to design a decentralized learning and control method which can generalize to different numbers of agents. With this goal, we leveraged the permutation invariance property in the swarm system, and modeled the multi-agent interaction using heterogeneous deep sets. We also developed an interaction-aware motion planner and a delay compensator. Neural-Fly (Chapter 5) considers 𝑓 = 𝑓 (𝑞, 𝑞, ¤ 𝑤(𝑡)) where 𝑤 describes potentially time-variant environmental conditions (e.g., wind conditions for a drone). Our goal is to develop an efficient online adaptation strategy such that the controller can adapt in changing conditions in real time. Hence, we use 𝜙(𝑞, 𝑞)𝑎 ¤ to approximate 𝑓 , where 𝜙 is a pre-meta-trained representation shared by all conditions and 𝑎 is a condition-dependent linear coefficient which will be adapted by adaptive control in an online manner. In Neural-Lander/Swarm/Fly, we all need pre-collected data to train or pre-train our deep learning models. In Safe Exploration (Chapter 6), we have to learn and control from scratch without pre-collect data or human in the loop. The key challenge is to trade-off between exploration and safety such that we can gradually achieve an aggressive control goal with safety guarantees in the whole process. 2.4 Preliminaries on Deep Learning In this section, we will introduce some preliminaries which are heavily used in Part I. ReLU Deep Neural Networks In Part I, we often learn the unknown dynamics term 𝑓 in Eq. (2.1) using a DNN with Rectified Linear Units (ReLU) activation. In general, DNNs equipped with ReLU converge faster during training, demonstrate more robust behavior with respect to changes in hyperparameters, and have fewer vanishing gradient problems compared to other activation functions such as sigmoid (Krizhevsky et al., 2012). A ReLU deep neural network represents the functional mapping from the input 𝑥 to the output 𝑔(𝑥, 𝜃), parameterized by the DNN weights 𝜃 = 𝑊1 , · · · , 𝑊 𝐿+1 : 𝑔(𝑥, 𝜃) = 𝑊 𝐿+1 𝜎(𝑊 𝐿 (𝜎(𝑊 𝐿−1 (· · · 𝜎(𝑊1 𝑥) · · · )))), (2.9) where the activation function 𝜎(·) = max(·, 0) is called the element-wise ReLU 21 function. We omit the bias term of each layer for simplicity. Note that omitting the bias term does not change the Lipschitz property of the DNN so it will not influence our theoretical analysis. ReLU is less computationally expensive than tanh and sigmoid because it involves simpler mathematical operations. However, deep neural networks are usually trained by first-order gradient based optimization, which is highly sensitive on the curvature of the training objective and can be unstable (Salimans and Kingma, 2016). To alleviate this issue, we apply the spectral normalization technique (Miyato et al., 2018) and introduce the details as follows. Spectral Normalization and Constrained Training Spectral normalization stabilizes DNN training by constraining the Lipschitz constant of the objective function 𝑔(𝑥, 𝜃). Spectrally normalized DNNs have also been shown to generalize well (Bartlett et al., 2017), which is an indication of stability in machine learning. Mathematically, the Lipschitz constant of a function ∥𝑔∥ Lip is defined as the smallest value such that ∀ 𝑥, 𝑥 ′ : ∥𝑔(𝑥) − 𝑔(𝑥 ′)∥ 2 ≤ ∥𝑔∥ Lip . ∥𝑥 − 𝑥 ′ ∥ 2 (2.10) It is well known that the Lipschitz constant of a general differentiable function 𝑔 is the maximum spectral norm (i.e., induced 2-norm or maximum singular value) of its gradient over its domain: ∥𝑔∥ Lip = sup𝑥 ∥∇𝑔(𝑥)∥ 2 . In particular, if 𝑔 is an affine function 𝑔(𝑥) = 𝑊𝑥 + 𝑏, then ∥𝑔∥ Lip = ∥𝑊 ∥ 2 . The ReLU DNN in Eq. (2.9) is a composition of functions. Thus we can bound the Lipschitz constant of the network by constraining the spectral norm of each layer. Using the fact that the Lipschitz norm of ReLU activation function 𝜎(·) is equal to 1, with the inequality ∥𝑔𝑎 ◦ 𝑔𝑏 ∥ Lip ≤ ∥𝑔𝑎 ∥ Lip · ∥𝑔𝑏 ∥ Lip for two functions 𝑔𝑎 and 𝑔𝑏 , we can find the following bound on ∥𝑔∥ Lip : ∥𝑔∥ Lip ≤ ∥𝑊 𝐿+1 ∥ 2 · ∥𝜎∥ Lip · · · ∥𝑊1 ∥ 2 = 𝐿+1 Ö ∥𝑊𝑙 ∥ 2 . (2.11) 𝑙=1 Given the upper bound in Eq. (2.11), we propose the stochastic gradient descent (SGD) with spectral normalization algorithm (Algorithm 2.1). In Algorithm 2.1, 𝛾 > 0 is the intended upper bounded for the Lipschitz constant of the DNN 𝑔(𝑥, 𝜃), 𝜂 > 0 is the learning rate, and ℓ(𝑔(𝑥), 𝑦) could be any differentiable loss function, e.g., squared loss ∥𝑔(𝑥) − 𝑦∥ 2 . D is a dataset comprising 𝑁 input (𝑥) and label (𝑦) pairs. Essentially, Algorithm 2.1 enforces the spectral norm of each layer matrix 1 upper bounded by 𝛾 𝐿+1 . The following lemma formally bounds ∥𝑔∥ Lip : 22 Algorithm 2.1: SGD with Spectral Normalization Hyperparameter: Normalization constant 𝛾 > 0, learning rate 𝜂 > 0 𝑁 and a loss function ℓ(𝑔(𝑥), 𝑦) Input: Dataset D = {𝑥𝑖 , 𝑦𝑖 }𝑖=1 Initialize: Neural network parameter 𝜃 = 𝑊1 , · · · , 𝑊 𝐿+1 1 repeat 2 Randomly sample a batch 𝐷 from the dataset D 3 Gradient descent on 𝐷: 4 𝜃 ← 𝜃−𝜂· ∑︁ ∇𝜃 ℓ(𝑔(𝑥, 𝜃), 𝑦) (2.12) (𝑥,𝑦)∈𝐷 5 6 Spectral normalization of each layer: for 1 ≤ 𝑙 ≤ 𝐿 + 1 do 1 if ∥𝑊𝑙 ∥ 2 > 𝛾 𝐿+1 then 7 𝑊𝑙 ← 8 1 𝑊𝑙 · 𝛾 𝐿+1 , ∥𝑊𝑙 ∥ 2 (2.13) until convergence Lemma 2.1. Algorithm 2.1 guarantees that ∥𝑔∥ Lip ≤ 𝛾. Proof. As in Eq. (2.11), we have ∥𝑔∥ Lip ≤ Î 𝐿+1 𝑙=1 ∥𝑊𝑙 ∥ 2 ≤ (2.14) Î 𝐿+1 𝑙=1 𝛾 1 𝐿+1 = 𝛾. □ Remark (spectral norm computation). Note that Algorithm 2.1 needs to compute the spectral norm of 𝑊𝑙 , ∀𝑙 in each training epoch. Computing the largest singular value of a large matrix could be computationally heavy. However, one can use the power iteration method to estimate ∥𝑊𝑙 ∥ 2 (Miyato et al., 2018). Nevertheless, in this thesis we focus on learning the dynamics so the DNN is in general small (within 100K parameters) so we directly use numpy.linalg.norm(W,2) to compute the spectral norm. Remark (optimization and back propagation). In practice, we use the Adam optimizer to train the neural network 𝑔(𝑥, 𝜃). Note that the normalization step 1 𝑊𝑙 ← ∥𝑊𝑊𝑙𝑙∥ 2 · 𝛾 𝐿+1 is not backward propagated, which is different from the original implementation of spectral normalization in (Miyato et al., 2018). Miyato et al. (2018) normalizes each layer’s spectral norm to exactly 1 in each step via back propagation. Instead, after each optimization step, we “project” each layer to have 1 1 a 𝛾 𝐿+1 spectral norm if its norm is bigger than 𝛾 𝐿+1 . Alternatively, we can also 23 directly deploy the standard spectral normalization implementation in PyTorch with 1 a scaling factor 𝛾 𝐿+1 . In practice, we found our implementation is easier to tune and have slightly better performance. Remark (looseness of Eq. (2.11)). It is not hard to notice that the lower bound given in Eq. (2.11) could be loose, due to the nature of the composition inequality ∥𝑔𝑎 ◦ 𝑔𝑏 ∥ Lip ≤ ∥𝑔𝑎 ∥ Lip · ∥𝑔𝑏 ∥ Lip . For instance, in the linear case ∥𝑊1𝑊2 ∥ 2 ≤ ∥𝑊1 ∥ 2 ∥𝑊2 ∥ 2 is tight only if the first right-singular vector of 𝑊1 aligns with the first left-singular vector of 𝑊2 . In general, computing the exact global Lipschitz constant of a DNN is NP-hard, and there is a well-known trade-off between tightness and computational efficiency when computing a Lipschitz upper bound (Fazlyab et al., 2019). For example, Fazlyab et al. (2019) gives a tighter estimation based on SDP (semidefinite programming). However, those optimization-based methods are in general much more computationally heavy than spectral normalization and very hard to be incorporated within the training process. Moreover, since the DNNs implemented in this thesis are in general small, the looseness from spectral normalization is acceptable. Remark (robustness from spectral normalization). Although our motivation of introducing spectral normalization is to guarantee control stability, it is worth noting that another benefit from spectral normalization is that spectrally normalized DNN is robust to input disturbance. Namely, since 𝑔 is 𝛾−Lipschitz continuous, ∥𝑔(𝑥 + Δ𝑥) − 𝑔(𝑥)∥ is always bounded by 𝛾∥Δ𝑥∥. 2.5 Preliminaries on Nonlinear Control Theory Comparison Lemma In Lyapunov stability analysis, we often use the following comparison lemma to bound a solution of an ODE (ordinary differential equation). Lemma 2.2. [adopted from (Khalil, 2002)] Suppose that a continuously differentiable function 𝑎(𝑡) ∈ R≥0 ↦→ R satisfies the following inequality: 𝑎(𝑡) ¤ ≤ −𝑏 · 𝑎(𝑡) + 𝑐, ∀𝑡 ≥ 0, 𝑎(0) = 𝑎 0 where 𝑏 > 0. Then we have 𝑎(𝑡) ≤ 𝑎 0 𝑒 −𝑏𝑡 + 𝑐 (1 − 𝑒 −𝑏𝑡 ), 𝑏 ∀𝑡 ≥ 0. The first application of Lemma 2.2 will be analyzing the convergence of the position tracking error 𝑞˜ = 𝑞 − 𝑞 𝑑 when the composite velocity error 𝑠 = 𝑞¤̃ + Λ𝑞˜ converges. 24 Lemma 2.3. Suppose ∥𝑠(𝑡)∥ ≤ 𝑎(𝑡). Then we have −𝜆 min (Λ)𝑡 ∥ 𝑞(𝑡)∥ ˜ ≤ ∥ 𝑞(0)∥𝑒 ˜ + 𝑎(𝑡) (1 − 𝑒 −𝜆min (Λ)𝑡 ). 𝜆 min (Λ) ˜ 2 . Then we have Proof. Define a Lyapunov function as 𝑉 (𝑡) = 12 ∥ 𝑞∥ √ 𝑉¤ = −𝑞˜ ⊤ Λ𝑞˜ + 𝑞˜ ⊤ 𝑠 ≤ −𝜆 min (Λ)∥ 𝑞∥ ˜ 2 + 𝑎∥ 𝑞∥ ˜ = −2𝜆 min (Λ)𝑉 + 𝑎 2𝑉 . √ Define 𝑊 = 𝑉 and 𝑊¤ = 𝑉¤ √ . Then we have 2 𝑉 𝑎 𝑊¤ ≤ −𝜆 min (Λ)𝑊 + √ . 2 Applying Lemma 2.2 we have −𝜆 min (Λ)𝑡 ∥ 𝑞(𝑡)∥ ˜ ≤ ∥ 𝑞(0)∥𝑒 ˜ + 𝑎(𝑡) (1 − 𝑒 −𝜆min (Λ)𝑡 ). 𝜆 min (Λ) □ References Bartlett, Peter L., Dylan J. Foster, and Matus J. Telgarsky (2017). Spectrallynormalized margin bounds for neural networks. In: Advances in Neural Information Processing Systems, pp. 6240–6249. Fazlyab, Mahyar, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas (2019). Efficient and accurate estimation of Lipschitz constants for deep neural networks. In: Advances in Neural Information Processing Systems 32. Hanover, Drew, Philipp Foehn, Sihao Sun, Elia Kaufmann, and Davide Scaramuzza (2021). Performance, precision, and payloads: Adaptive nonlinear MPC for quadrotors. In: IEEE Robotics and Automation Letters 7.2, pp. 690–697. Khalil, Hassan K. (2002). Nonlinear systems. Pearson Education. Prentice Hall. isbn: 9780130673893. Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton (2012). Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems 25. Lopez, Brett T. and Jean-Jacques E. Slotine (2021). Universal adaptive control of nonlinear systems. In: IEEE Control Systems Letters 6, pp. 1826–1830. Miyato, Takeru, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida (2018). Spectral normalization for generative adversarial networks. In: arXiv preprint arXiv:1802.05957. 25 Morgan, Daniel, Giri P. Subramanian, Soon-Jo Chung, and Fred Y. Hadaegh (2016). Swarm assignment and trajectory optimization using variable-swarm, distributed auction assignment and sequential convex programming. In: The International Journal of Robotics Research 35.10, pp. 1261–1285. Murray, Richard M., Zexiang Li, and S. Shankar Sastry (Dec. 2017). A mathematical introduction to robotic manipulation. 1st ed. CRC Press. isbn: 978-1-315-136370. doi: 10.1201/9781315136370. url: https://www.taylorfrancis.com/ books/9781351469791. O’Connell, Michael, Guanya Shi, Xichen Shi, Kamyar Azizzadenesheli, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung (2022). Neural-Fly enables rapid learning for agile flight in strong winds. In: Science Robotics 7.66, eabm6597. doi: 10.1126/scirobotics.abm6597. Salimans, Tim and Diederik P. Kingma (2016). Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In: Advances in Neural Information Processing Systems, pp. 901–909. Shi, Guanya, Wolfgang Hönig, Xichen Shi, Yisong Yue, and Soon-Jo Chung (2022). Neural-Swarm2: Planning and control of heterogeneous multirotor swarms using learned interactions. In: IEEE Transactions on Robotics 38.2, pp. 1063–1079. doi: 10.1109/TRO.2021.3098436. Shi, Guanya, Wolfgang Hönig, Yisong Yue, and Soon-Jo Chung (2020). NeuralSwarm: Decentralized close-proximity multirotor control using learned interactions. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 3241–3247. doi: 10.1109/ICRA40945.2020.9196800. Shi, Guanya, Xichen Shi, Michael O’Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung (2019). Neural Lander: Stable drone landing control using learned dynamics. In: 2019 International Conference on Robotics and Automation (ICRA). IEEE, pp. 9784–9790. doi: 10.1109/ICRA.2019.8794351. Shi, Xichen, Kyunam Kim, Salar Rahili, and Soon-Jo Chung (2018). Nonlinear control of autonomous flying cars with wings and distributed electric propulsion. In: 2018 IEEE Conference on Decision and Control (CDC). IEEE, pp. 5326– 5333. Slotine, Jean-Jacques E. and Weiping Li (1991). Applied nonlinear control. Vol. 199. 1. Prentice Hall, Englewood Cliffs, NJ. Tal, Ezra and Sertac Karaman (May 2021). Accurate tracking of aggressive quadrotor trajectories using incremental nonlinear dynamic inversion and differential flatness. In: IEEE Transactions on Control Systems Technology 29.3, pp. 1203– 1218. issn: 1558-0865. doi: 10.1109/TCST.2020.3001117. 26 Chapter 3 NEURAL-LANDER rigid body dynamics in SE(3) aerodynamics (ground effect) Figure 3.1: Neural-Lander enables agile flight maneuver very close to the ground. In Chapter 2, we introduced a general mixed robot dynamics model 𝑀 (𝑞) 𝑞¥ + 𝐶 (𝑞, 𝑞) ¤ 𝑞¤ + 𝑔(𝑞) = 𝐵𝑢 + 𝑓 where 𝑓 is the unknown dynamics. We also introduced the corresponding drone dynamics model in Eq. (2.4). In this chapter, we focus on the case 𝑓 = 𝑓 (𝑞, 𝑞, ¤ 𝑢), and the main challenge is to ensure nonlinear stability and robustness when using a DNN to approximate 𝑓 . In drone experiments, the nominal part refers to the standard rigid body dynamics in SE(3) and 𝑓 refers to the unknown aerodynamics such as the ground effect when the drone is near-ground (see Fig. 3.1). This chapter is mainly based on the following paper1 2: Shi, Guanya, Xichen Shi, Michael O’Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung (2019). Neural Lander: Stable drone landing control using learned dynamics. In: 2019 International Conference on Robotics and Automation (ICRA). IEEE, pp. 9784–9790. doi: 10.1109/ICRA.2019.8794351. Abstract. Precise near-ground trajectory control is difficult for multi-rotor drones, due to the complex aerodynamic effects caused by interactions between multi-rotor airflow and the environment. Conventional control methods often fail to properly account for these complex effects and fall short in accomplishing smooth landing. 1 Summary video: https://youtu.be/FLLsG0S78ik 2 Simulator code: https://github.com/GuanyaShi/neural_lander_sim_1d 27 In this chapter, we present a novel deep-learning-based robust nonlinear controller (Neural-Lander) that improves control performance of a quadrotor during landing. Our approach combines a nominal dynamics model with a Deep Neural Network (DNN) that learns high-order interactions. We apply spectral normalization (SN) to constrain the Lipschitz constant of the DNN. Leveraging this Lipschitz property, we design a nonlinear feedback linearization controller using the learned model and prove system stability with disturbance rejection. To the best of our knowledge, this is the first DNN-based nonlinear feedback controller with stability guarantees that can utilize arbitrarily large neural nets. Experimental results demonstrate that the proposed controller significantly outperforms a Baseline Nonlinear Tracking Controller in both landing and cross-table trajectory tracking cases. We also empirically show that the DNN generalizes well to unseen data outside the training domain. 3.1 Introduction and Related Work Unmanned Aerial Vehicles (UAVs) require high precision control of aircraft positions, especially during landing and take-off. This problem is challenging largely due to complex interactions of rotor and wing airflows with the ground. The aerospace community has long identified such ground effect that can cause an increased lift force and a reduced aerodynamic drag. These effects can be both helpful and disruptive in flight stability (Cheeseman and Bennett, 1955), and the complications are exacerbated with multiple rotors. Therefore, performing automatic landing of UAVs is risk-prone, and requires expensive high-precision sensors as well as carefully designed controllers. Compensating for ground effect is a long-standing problem in the aerial robotics community. Prior work has largely focused on mathematical modeling (e.g. Nonaka and Sugizaki (2011)) as part of system identification (ID). These models are later used to approximate aerodynamics forces during flights close to the ground and combined with controller design for feed-forward cancellation (e.g. Danjun et al. (2015)). However, existing theoretical ground effect models are derived based on steady-flow conditions, whereas most practical cases exhibit unsteady flow. Alternative approaches, such as integral or adaptive control methods, often suffer from slow response and delayed feedback. Berkenkamp, Schoellig, et al. (2016) employs Bayesian Optimization for open-air control but not for take-off/landing. Given these limitations, the precision of existing fully automated systems for UAVs are still insufficient for landing and take-off, thereby necessitating the guidance of a human UAV operator during those phases. 28 To capture complex aerodynamic interactions without overly-constrained by conventional modeling assumptions, we take a machine-learning (ML) approach to build a black-box ground effect model using Deep Neural Networks (DNNs). However, incorporating such models into a UAV controller faces three key challenges. First, it is challenging to collect sufficient real-world training data, as DNNs are notoriously data-hungry. Second, due to high-dimensionality, DNNs can be unstable and generate unpredictable output, which makes the system susceptible to instability in the feedback control loop. Third, DNNs are often difficult to analyze, which makes it difficult to design provably stable DNN-based controllers. The aforementioned challenges pervade previous works using DNNs to capture high-order non-stationary dynamics. For example, Abbeel et al. (2010) and Punjani and Abbeel (2015) use DNNs to improve system ID of helicopter aerodynamics, but not for controller design. Other approaches aim to generate reference inputs or trajectories from DNNs (Bansal et al., 2016; Li et al., 2017; Zhou et al., 2017; Sánchez-Sánchez and Izzo, 2018). However, these approaches can lead to challenging optimization problems (Bansal et al., 2016), or heavily rely on well-designed closed-loop controller and require a large number of labeled training data (Li et al., 2017; Zhou et al., 2017; Sánchez-Sánchez and Izzo, 2018). A more classical approach of using DNNs is direct inverse control (Frye and Provence, 2014; Suprijono and Kusumoputro, 2017) but the non-parametric nature of a DNN controller also makes it challenging to guarantee stability and robustness to noise. Berkenkamp, Turchetta, et al. (2017) proposes a provably stable model-based Reinforcement Learning method based on Lyapunov analysis, but it requires a potentially expensive discretization step and relies on the native Lipschitz constant of the DNN. Contributions In this chapter, we propose a learning-based controller, Neural-Lander, to improve the precision of quadrotor landing with guaranteed stability. Our approach directly learns the ground effect on coupled unsteady aerodynamics and vehicular dynamics. We use deep learning for system ID of residual dynamics and then integrate it with nonlinear feedback linearization control. We train DNNs with layer-wise spectrally normalized weight matrices. We prove that the resulting controller is globally exponentially stable under bounded learning errors. This is achieved by exploiting the Lipschitz bound of spectrally normalized DNNs. It has earlier been shown that spectral normalization of DNNs leads to good 29 generalization, i.e. stability in a learning-theoretic sense (Miyato et al., 2018). It is intriguing that spectral normalization simultaneously guarantees stability both in a learning-theoretic and a control-theoretic sense. We evaluate Neural-Lander on trajectory tracking of quadrotor during take-off, landing and cross-table maneuvers. Neural-Lander is able to land a quadrotor much more accurately than a Baseline Nonlinear Tracking Controller with a pre-identified system. In particular, we show that compared to the baseline, Neural-Lander can decrease error in 𝑧 axis from 0.13 m to 0, mitigate 𝑥 and 𝑦 drifts by as much as 90%, in the landing case. Meanwhile, Neural-Lander can decrease 𝑧 error from 0.153 m to 0.027 m, in the cross-table trajectory tracking task. We also demonstrate that the learned model can handle temporal dependency, and is an improvement over the steady-state theoretical models. 3.2 Problem Statement General Robot Dynamics and Drone Dynamics Modelling As introduced in Chapter 2, we consider the following mixed robot dynamics model: 𝑀 (𝑞) 𝑞¥ + 𝐶 (𝑞, 𝑞) ¤ 𝑞¤ + 𝑔(𝑞) = 𝐵𝑢 + 𝑓 (𝑞, 𝑞, ¤ 𝑢) . | {z } (3.1) unknown For the specific drone dynamics, recall Eq. (2.4) in Chapter 2: 𝑝¤ = 𝑣, 𝑚 𝑣¤ = 𝑚𝑔 + 𝑅 𝑓𝑢 + 𝑓𝑎 , 𝑅¤ = 𝑅𝑆(𝜔), 𝐽 𝜔¤ = 𝐽𝜔 × 𝜔 + 𝜏 + 𝜏𝑎 . As discussed in Chapter 2, we can reduce the drone dynamics model to Eq. (3.1) using the cascading control structure. Therefore, for theoretical analysis we will focus on the general model in Eq. (3.1) for generality and rotational simplicity. Moreover, in drone experiments, as we mainly focus on landing and take-off tasks, the attitude dynamics is limited and the aerodynamic disturbance torque 𝜏𝑎 is negligible. Thus position dynamics and the aerodynamic force 𝑓𝑎 will our primary concern. Learning and Control Goals We first collect a data set D containing 𝑁 input and label pairs: 𝑥𝑖 = [𝑞𝑖 ; 𝑞¤𝑖 ; 𝑢𝑖 ], 𝑁 D = {𝑥𝑖 , 𝑦𝑖 = 𝑓 (𝑥𝑖 ) + 𝜖𝑖 }𝑖=1 , (3.3) where we define 𝑥 = [𝑞; 𝑞; ¤ 𝑢] for simplicity, and 𝑦𝑖 is a noisy measurement of 𝑓 (𝑥𝑖 ) with 𝜖𝑖 an extra disturbance. 30 With D, we aim to train a ReLU DNN 𝑓ˆ(𝑥, 𝜃) to approximate 𝑓 . Then, our goal is to design a stable and robust controller such that the tracking error ∥𝑞 − 𝑞 𝑑 ∥ is small. Training is done off-line, and the learned dynamics 𝑓ˆ is applied in the on-board controller in real-time to achieve smooth landing and take-off. 3.3 Dynamics Learning and Controller Design We apply Algorithm 2.1 to train the ReLU DNN 𝑓ˆ(𝑥, 𝜃) with a bounded Lipschitz constant 𝛾. We select 𝛾 such that 𝛾 · ∥𝐵† ∥ < 1 where 𝐵† is the Moore-Penrose inverse of 𝐵. Thus from Lemma 2.1, we have ∥ 𝑓ˆ∥ Lip ≤ 𝛾, 𝛾 · ∥𝐵† ∥ < 1. With 𝑓ˆ, the controller structure is given as Eq. (2.3): 𝐵𝑢 = 𝑀 (𝑞) 𝑞¥𝑟 + 𝐶 (𝑞, 𝑞) ¤ 𝑞¤𝑟 + 𝑔(𝑞) | {z } nonlinear feedforward −𝐾 𝑠 |{z} − 𝑓ˆ(𝑞, 𝑞, ¤ 𝑢) . | {z } (3.4) linear feedback learned feedforward Because of the dependency of 𝑓ˆ on 𝑢, the control synthesis problem in Eq. (3.4) is non-affine. Therefore, we propose the following fixed-point iteration method for solving Eq. (3.4):   𝑢 𝑘 = 𝐵† 𝑀 (𝑞) 𝑞¥𝑟 + 𝐶 (𝑞, 𝑞) ¤ 𝑞¤𝑟 + 𝑔(𝑞) − 𝐾 𝑠 − 𝑓ˆ(𝑞, 𝑞, ¤ 𝑢 𝑘−1 ) , (3.5) where 𝑢 𝑘 and 𝑢 𝑘−1 are the control input for current and previous time step in the discrete-time controller. In the next section, we prove the stability of the system and convergence of the control inputs in Eq. (3.5). 3.4 Nonlinear Stability Analysis In this section, the closed-loop tracking error analysis provides a direct correlation on how to tune the neural network 𝑓ˆ and controller parameter to improve control performance and robustness. Control Allocation as Contraction Mapping We first show that the control input 𝑢 𝑘 converges to the solution of Eq. (3.4) when all states are fixed. Lemma 3.1. Define mapping 𝑢 𝑘 = F (𝑢 𝑘−1 ) based on Eq. (3.5) and fix all current states. If 𝑓ˆ(𝑥) is 𝛾-Lipschitz continuous, and ∥𝐵† ∥ 2 · 𝛾 < 1; then F (·) is a contraction mapping, and 𝑢 𝑘 converges to unique solution of 𝑢 ∗ = F (𝑢 ∗ ). 31 Proof. ∀𝑢 1 , 𝑢 2 ∈ R𝑚 and given fixed states as 𝑞, 𝑞, ¤ 𝑞¤𝑟 , 𝑞¥𝑟 , 𝑠, then:   † ˆ ˆ ∥F (𝑢 1 ) − F (𝑢 2 )∥ 2 = 𝐵 𝑓 (𝑞, 𝑞, ¤ 𝑢 1 ) − 𝑓 (𝑞, 𝑞, ¤ 𝑢2) ≤ ∥𝐵† ∥ 2 · 𝛾 ∥𝑢 1 − 𝑢 2 ∥ 2 . 2 Thus, ∥𝐵† ∥ 2 · 𝛾 < 1 so F (·) is a contraction mapping. □ Stability of Learning-based Nonlinear Controller Before continuing to prove the stability of the full system, we make the following assumptions. Assumption 3.1 (boundedness of the desired trajectory). The desired states along the position trajectory 𝑞 𝑑 (𝑡), 𝑞¤ 𝑑 (𝑡), and 𝑞¥ 𝑑 (𝑡) are bounded. Assumption 3.2. One-step difference of control signal satisfies ∥𝑢 𝑘 − 𝑢 𝑘−1 ∥ ≤ 𝜌 ∥𝑠∥ with a small positive 𝜌. Namely, we assume the control signal is changing smoothly. Here we provide the intuition behind this assumption. From Eq. (3.5), we can derive the following approximate relation with Δ(·) 𝑘 = ∥(·) 𝑘 − (·) 𝑘−1 ∥:  Δ𝑢 𝑘 ≤ ∥𝐵† ∥ 𝛾Δ𝑢 𝑘−1 + 𝛾Δ𝑞 𝑘 + 𝛾Δ𝑞¤ 𝑘 + ∥𝐾 ∥Δ𝑠 𝑘  + Δ(𝑀 (𝑞) 𝑞¥𝑟 ) 𝑘 + Δ(𝐶 (𝑞, 𝑞) ¤ 𝑞¤𝑟 ) 𝑘 + Δ(𝑔(𝑞)) 𝑘 . Note that for drone position controller, 𝐶 = 0 and 𝑀 and 𝑔 are both constant (see more details in Chapter 2). Moreover, because update rate of the attitude controller (> 200 Hz) and motor speed control (> 5 kHz) are much higher than that of the position controller (≈ 50 Hz) and the desired trajectory 𝑞 𝑑 (𝑡) is not changing too fast, in practice, we can safely neglect Δ𝑠 𝑘 , Δ𝑞 𝑘 , Δ𝑞¤ 𝑘 and Δ𝑞¥𝑟,𝑘 in one update (Theorem 11.1 Khalil (2002)). It leads to:
Δ𝑢 𝑘 ≤ ∥𝐵† ∥ 𝛾Δ𝑢 𝑘−1 + 𝑐 , with 𝑐 being a small constant and ∥𝐵† ∥ · 𝛾 < 1, we can deduce that Δ𝑢 rapidly converges to a small ultimate bound between each position controller update. Assumption 3.3 (bounded learning error). The learning error of 𝑓ˆ(𝑥) over the compact sets 𝑥 ∈ X is upper bounded by 𝜖 𝑚 = sup𝑥∈X ∥ 𝑓 (𝑥) − 𝑓ˆ(𝑥)∥. DNNs have been shown to generalize well to the set of unseen events that are from almost the same distribution as training set (Zhang et al., 2016; He et al., 32 2016). This empirical observation is also theoretically studied in order to shed more light toward an understanding of the complexity of these models (Neyshabur, Bhojanapalli, McAllester, and Nathan Srebro, 2017; Bartlett et al., 2017; Dziugaite and Roy, 2017; Neyshabur, Bhojanapalli, McAllester, and Nati Srebro, 2017). Based on the above assumptions, we can now present our overall stability and robustness result. Theorem 3.1. Under Assumptions 3.1 to 3.3, for a time-varying 𝑞 𝑑 (𝑡), the controller defined in Eq. (3.5) with 𝜆 min (𝐾) > 𝛾𝜌 and 𝛾∥𝐵† ∥ < 1 achieves the exponential convergence of the tracking error ∥ 𝑞∥ ˜ = ∥𝑞 − 𝑞 𝑑 ∥ to the following error ball 𝜆 max (𝑀) · 𝜖𝑚 . 𝜆 min (𝑀)𝜆 min (Λ)(𝜆 min (𝐾) − 𝛾𝜌) Proof. Plugging Eq. (3.5) into Eq. (3.1), we have the following closed-loop dynamics 𝑀 𝑠¤ + 𝐶𝑠 + 𝐾 𝑠 = 𝑓 (𝑞, 𝑞, ¤ 𝑢 𝑘 ) − 𝑓ˆ(𝑞, 𝑞, ¤ 𝑢 𝑘−1 ). Note that this is a hybrid system in the sense that 𝑢 evolves in a discrete manner but other variables changes continuously. We define the Lyapunov function as 𝑉 (𝑠) = 12 𝑠⊤ 𝑀 𝑠, and after taking the derivative we have   1 ⊤ ¤ ⊤ ˆ ¤ ¤ 𝑢 𝑘 ) − 𝑓 (𝑞, 𝑞, ¤ 𝑢 𝑘−1 ) 𝑉 = 𝑠 𝑀 𝑠 + 𝑠 −𝐶𝑠 − 𝐾 𝑠 + 𝑓 (𝑞, 𝑞, 2   (1) ¤ 𝑢 𝑘 ) − 𝑓ˆ(𝑞, 𝑞, ¤ 𝑢 𝑘 ) + 𝑓ˆ(𝑞, 𝑞, ¤ 𝑢 𝑘 ) − 𝑓ˆ(𝑞, 𝑞, ¤ 𝑢 𝑘−1 ) = −𝑠⊤ 𝐾 𝑠 + 𝑠⊤ 𝑓 (𝑞, 𝑞, (2) ≤ −𝑠⊤ 𝐾 𝑠 + ∥𝑠∥ (𝜖 𝑚 + 𝛾∥𝑢 𝑘 − 𝑢 𝑘−1 ∥) (3) ≤ −𝑠⊤ 𝐾 𝑠 + 𝛾𝜌∥𝑠∥ 2 + 𝜖 𝑚 ∥𝑠∥ ≤ −(𝜆 min (𝐾) − 𝛾𝜌)∥𝑠∥ 2 + 𝜖 𝑚 ∥𝑠∥ where (1) is from the fact 𝑀¤ − 2𝐶 is skew-symmetric, (2) is from Assumption 3.3 and the Lipschitz property of 𝑓ˆ, and (3) is from Assumption 3.2. Note that 1 1 2 2 2 𝜆 max (𝑀)∥𝑠∥ ≥ 𝑉 ≥ 2 𝜆 min (𝑀)∥𝑠∥ , we have √︄ 2𝑉 2𝑉 𝑉¤ ≤ −(𝜆min (𝐾) − 𝛾𝜌) + 𝜖𝑚 . 𝜆 max (𝑀) 𝜆 min (𝑀) √︁ ¤ Then define 𝑊 (𝑡) = 𝑉 (𝑡). Note that 𝑊¤ = √𝑉 . Therefore we have 2 𝑉 𝜆 min (𝐾) − 𝛾𝜌 𝑊¤ ≤ − 𝑊 + 𝜖𝑚 𝜆 max (𝑀) √︄ 1 2𝜆 min (𝑀) . 33 Using the comparison lemma (Lemma 2.2), we have √︄  𝜆 min (𝐾) − 𝛾𝜌 𝜖 𝑚 𝜆 max (𝑀) 1 𝑊 ≤ exp − 𝑡 𝑊 (0) + . 𝜆 max (𝑀) 𝜆 min (𝐾) − 𝛾𝜌 2𝜆 min (𝑀) √︃ √ Recall that 𝑊 = 𝑉 ≥ ∥𝑠∥ 𝜆min2(𝑀) , so we finally have  √︄  𝜆 max (𝑀) 𝜆 min (𝐾) − 𝛾𝜌 𝜆max (𝑀) 𝑡 ∥𝑠(0)∥ + · 𝜖𝑚 ∥𝑠∥ ≤ exp − 𝜆 max (𝑀) 𝜆 min (𝑀) 𝜆 min (𝑀)(𝜆 min (𝐾) − 𝛾𝜌)  It can be shown that this leads to finite-gain L 𝑝 stability and input-to-state stability (ISS) (Chung et al., 2013). Further, with Lemma 2.3, we have ∥ 𝑞∥ ˜ exponentially converges to the following error ball 𝜆 max (𝑀) · 𝜖𝑚 . 𝜆 min (𝑀)𝜆 min (Λ)(𝜆 min (𝐾) − 𝛾𝜌) □ Interpretation of Theorem 3.1 Potential trade-off when tuning 𝛾. Note that Theorem 3.1 has two conditions on the Lipschitz constant of the DNN 𝑓ˆ: 𝜆 min (𝐾) > 𝛾𝜌 and 𝛾∥𝐵† ∥ < 1. The latter one is to make sure the fixed-point iteration method will converge, and the former one guarantees the closed-loop stability. They both suggest 𝛾 = ∥ 𝑓ˆ∥ Lip should be small and bigger 𝛾 will influence the controller robustness against big 𝜌 (i.e., bigger delay). However, the learning error 𝜖 𝑚 might increase if 𝛾 gets too small, especially when the ground-truth function 𝑓 has bad Lipschitz property. In that case, one should look for higher-order methods than the fixed-point iteration (e.g., Newton method or something like MPC) but those methods are in general much more involved and it is unclear how they influence the closed-loop stability analysis. Nevertheless, in our experiments we strictly followed 𝛾∥𝐵† ∥ < 1 and the learning performance is even better than the baseline method without Lipschitz constraints. An alternative assumption on ∥𝑢 𝑘 −𝑢 𝑘−1 ∥. Assumption 3.2 assumes ∥𝑢 𝑘 −𝑢 𝑘−1 ∥ ≤ 𝜌∥𝑠∥. Another assumption we can consider is ∥𝑢 𝑘 − 𝑢 𝑘−1 ∥ ≤ 𝜌¯ where 𝜌¯ is a small constant. In this case, the Lyapunov analysis will change to 𝑉¤ ≤ −𝜆 min (𝐾)∥𝑠∥ 2 + (𝜖 𝑚 + 𝛾 𝜌)∥𝑠∥ ¯ and the error ball for 𝑞˜ will be changed to 𝜆 max (𝑀) · (𝜖 𝑚 + 𝛾 𝜌). ¯ 𝜆 min (𝑀)𝜆 min (Λ)𝜆 min (𝐾) 34 With this alternative assumption, 𝛾 𝜌¯ can be viewed as an add-on term on the learning error. Similarly, bigger 𝛾 amplifies the delay constant 𝜌. ¯ Moreover, it is worth noting that with this alternative assumption we no longer need to require the gain condition about 𝜆 min (𝐾). 3.5 Experiments In our experiments, we evaluate both the generalization performance of our DNN as well as the overall control performance of Neural-Lander. The experimental setup is composed of a motion capture system with 17 cameras, a WiFi router for communication, and an Intel Aero drone, weighing 1.47 kg with an onboard Linux computer (2.56 GHz Intel Atom x7 processor, 4 GB DDR3 RAM). We retrofitted the drone with eight reflective infrared markers for accurate position, attitude and velocity estimation at 100Hz. The Intel Aero drone and the test space are shown in Fig. 3.1. In this section, we will focus on the drone dynamics given in Eq. (2.4) and Eq. (2.5). We apply the proposed nonlinear learning-based controller Eq. (3.5) to the quadrotor position dynamics Eq. (2.4a) via the reduction in Eq. (2.6). In particular, 𝑓ˆ𝑎 in Eq. (2.6) is a ReLU DNN depending on the rotor control signal 𝑢. We call this drone position controller Neural-Lander. See the original paper (Shi et al., 2019) for the detailed structure and analysis of Neural-Lander in quadrotor dynamics. Bench Test To identify a good nominal model, we first measured the mass 𝑚, diameter of the rotor 𝐷 𝑟 , the air density 𝜌𝑎 , and the gravity 𝑔. Then we performed bench test to determine the thrust constant, 𝑐𝑇 , as well as the non-dimensional thrust coefficient 𝐶𝑇 = 𝜌 𝑐𝑇𝐷 4 . Note that 𝐶𝑇 is a function of propeller speed 𝑛, and here we picked a 𝑎 𝑟 nominal value at 𝑛 = 2000 RPM. Real-World Flying Data and Preprocessing To estimate the disturbance force 𝑓𝑎 , an expert pilot manually flew the drone at different heights, and we collected training data consisting of sequences of state estimates and control inputs {( 𝑝, 𝑣, 𝑅, 𝑢), 𝑦} where 𝑢 = [𝑛21 ; 𝑛22 ; 𝑛23 ; 𝑛24 ] and 𝑦 is the observed value of 𝑓𝑎 . We utilized the relation 𝑓𝑎 = 𝑚 𝑣¤ − 𝑚𝑔 − 𝑅 𝑓𝑢 from Eq. (2.4a) to calculate 𝑓𝑎 , where 𝑓𝑢 is calculated based on the nominal 𝑐𝑇 from the bench test. For the acceleration 𝑣¤ , we used the fifth-order numerical differentiation method. Our training set is a 35 Part I Part II Figure 3.2: Training data trajectory. Part I (0 to 250 s) contains maneuvers at different heights (0.05 m to 1.50 m). Part II (250 s to 350 s) includes random 𝑥, 𝑦, and 𝑧 motions for maximum state-space coverage. single continuous trajectory with varying heights and velocities. The trajectory has two parts shown in Fig. 3.2. We aim to learn the ground effect through Part I of the training set, and other aerodynamics forces such as air drag through Part II. DNN Prediction Performance We train a deep ReLU network 𝑓ˆ𝑎 (𝑧, 𝑣, 𝑅, 𝑢), with 𝑧, 𝑣, 𝑅, 𝑢 corresponding to global height, global velocity, attitude, and control input. We build the ReLU network using PyTorch. Our ReLU network consists of four fully-connected hidden layers, with input and the output dimensions 12 and 3, respectively. We use spectral normalization (see Algorithm 2.1) to constrain the Lipschitz constant of the DNN. We compare the near-ground estimation accuracy our DNN model with existing 1D steady ground effect model (Cheeseman and Bennett, 1955; Danjun et al., 2015): 𝑇 (𝑛, 𝑧) = 𝑛2 𝑐 (𝑛) = 𝑛2 𝑐𝑇 (𝑛0 ) + 𝑓¯𝑎,𝑧 , 𝐷𝑟 2 𝑇 1 − 𝜇( 8𝑧 ) (3.6) where 𝑇 is the thrust generated by propellers, 𝑛 is the rotation speed, 𝑛0 is the idle RPM, and 𝜇 depends on the number and the arrangement of propellers (𝜇 = 1 for a single propeller, but must be tuned for multiple propellers). Note that 𝑐𝑇 is a function of 𝑛. Thus, we can derive 𝑓¯𝑎,𝑧 (𝑛, 𝑧) from 𝑇 (𝑛, 𝑧). Figure 3.3(a) shows the comparison between the estimation of 𝑓𝑎 from the DNN 𝑓ˆ𝑎 and the theoretical ground effect model Eq. (3.6) at different 𝑧 (assuming 𝑇 = 𝑚𝑔 when 𝑧 = ∞). We can see that our DNN can achieve much better estimates than the theoretical ground effect model. We further investigate the trend of 𝑓¯𝑎,𝑧 with respect to the rotation speed 𝑛. Fig. 3.3(b) shows the learned 𝑓ˆ𝑎,𝑧 over the rotation speed 𝑛 at a given height, in comparison with the 𝐶𝑇 measured from the bench test. We 36 (a) (b) (c) New Training set domain domain New Training set domain domain Figure 3.3: DNN prediction performance. (a) Learned 𝑓𝑎,𝑧 compared to the ground effect model with respect to height 𝑧, with 𝑣 𝑧 = 𝑣 𝑥 = 𝑣 𝑦 = 0 m/s, 𝑅 = 𝐼, 𝑢 = 6400 RPM. Ground truth points are from hovering data at different heights. (b) Learned 𝑓𝑎,𝑧 with respect to rotation speed 𝑛 (𝑧 = 0.2 m, 𝑣 𝑧 = 0 m/s), compared to 𝐶𝑇 measured in the bench test. (c) Heatmaps of learned 𝑓𝑎,𝑧 versus 𝑧 and 𝑣 𝑧 . (Left) ReLU network with spectral normalization. (Right) ReLU network without spectral normalization. observe that the increasing trend of the estimates 𝑓ˆ𝑎,𝑧 is consistent with bench test results for 𝐶𝑇 . To understand the benefits of SN, we compared 𝑓ˆ𝑎,𝑧 predicted by the DNNs trained both with and without SN as shown in Fig. 3.3(c). Note that 𝑣 𝑧 from −1 m/s to 1 m/s is covered in our training set, but −2 m/s to −1 m/s is not. We observe the following differences: 1. Ground effect: 𝑓ˆ𝑎,𝑧 increases as 𝑧 decreases, which is also shown in Fig. 3.3(a). 2. Air drag: 𝑓ˆ𝑎,𝑧 increases as the drone goes down (𝑣 𝑧 < 0) and it decreases as the drone goes up (𝑣 𝑧 > 0). 37 3. Generalization: the spectral normalized DNN is much smoother and can also generalize to new input domains not contained in the training set. In Bartlett et al. (2017), the authors theoretically show that spectral normalization can provide tighter generalization guarantees on unseen data, which is consistent with our empirical observation. We will connect generalization theory more tightly with our robustness guarantees in the future. Baseline Controller Baseline final error: 0.13 m final error: zero Baseline Baseline mean L1 error: 0.072 m mean L1 error: 0.007 m mean L1 error: 0.032 m mean L1 error: 0.021 m Baseline Figure 3.4: Baseline Controller and Neural-Lander performance in take-off and landing. Means (solid curves) and standard deviations (shaded areas) of 10 trajectories. We compared the Neural-Lander with a Baseline Nonlinear Tracking Controller. We implemented both a Baseline Controller similar to Eq. (2.6) with 𝑓ˆ𝑎 ≡ 0, as ∫𝑡 ˜ Though well as an integral controller variant with 𝑠 = 𝑝¤̃ + 2Λ 𝑝˜ + Λ2 0 𝑝(𝜏)𝑑𝜏. an integral gain can cancel steady-state error during set-point regulation, our flight results showed that the performance can be sensitive to the integral gain, especially during trajectory tracking. This can be seen in the demo video3. 3 Demo videos: https://youtu.be/FLLsG0S78ik 38 position (m) Setpoint Regulation Performance 1.0 0.8 0.6 0.4 0.2 0.0 4 layers (error: 0) 1 layer (error: 0.07m) 0 layer (error: 0.08m) Baseline (error: 0.12m) 0 2 4 6 8 time (s) 10 12 14 16 Figure 3.5: Neural-Lander performance in take-off and landing with different DNN capacities. 1 layer means a linear mapping 𝑓ˆ𝑎 = 𝐴𝑥 + 𝑏; 0 layer means 𝑓ˆ𝑎 = 𝑏; Baseline means 𝑓ˆ𝑎 ≡ 0. First, we tested the two controllers’ performance in take-off/landing, by commanding position setpoint 𝑝 𝑑 , from (0, 0, 0), to (0, 0, 1), then back to (0, 0, 0), with 𝑝¤ 𝑑 ≡ 0. From Fig. 3.4, we can conclude that there are two main benefits of our NeuralLander. (a) Neural-Lander can control the drone to precisely and smoothly land on the ground surface while the Baseline Controller struggles to achieve 0 terminal height due to the ground effect. (b) Neural-Lander can mitigate drifts in 𝑥 − 𝑦 plane, as it also learned about additional aerodynamics such as air drag. Second, we tested Neural-Lander performance with different DNN capacities. Fig. 3.5 shows that compared to the baseline ( 𝑓ˆ𝑎 ≡ 0), 1 layer model could decrease 𝑧 error but it is not enough to land the drone. 0 layer model generated significant error during take-off. In experiments, we observed the Neural-Lander without spectral normalization can even result in unexpected controller outputs leading to crash, which empirically implies the necessity of SN in training the DNN and designing the controller. Trajectory Tracking Performance To show that our algorithm can handle more complicated environments where physics-based modelling of dynamics would be substantially more difficult, we devise a task of tracking an elliptic trajectory very close to a table with a period of 10 seconds shown in Fig. 3.6. The trajectory is partially over the table with significant ground effects, and a sharp transition to free space at the edge of the table. We compared the performance of both Neural-Lander and Baseline Controller on this test. 39 (a) table desired trajectory (b) Mean X error 0.079m 0.126m Mean Y error 0.052m 0.061m Mean Z error 0.027m 0.153m Z variance 0.014m 0.026m Figure 3.6: Heatmaps of learned 𝑓ˆ𝑎,𝑧 and trajectory tracking performance. (a) Heatmaps of learned 𝑓ˆ𝑎,𝑧 versus 𝑥 and 𝑦, with other inputs fixed. (Left) ReLU network with spectral normalization. (Right) ReLU network without spectral normalization. (b) Tracking performance and statistics. In order to model the complex dynamics near the table, we manually flew the drone in the space close to the table to collect another data set. We trained a new ReLU DNN model with 𝑥-𝑦 positions as additional input features: 𝑓ˆ𝑎 ( 𝑝, 𝑣, 𝑅, 𝑢). Similar to the setpoint experiment, the benefit of spectral normalization can be seen in Fig. 3.6(a), where only the spectrally-normalized DNN exhibits a clear table boundary. Fig. 3.6(b) shows that Neural-Lander outperformed the Baseline Controller for tracking the desired position trajectory in all 𝑥, 𝑦, and 𝑧 axes. Additionally, NeuralLander showed a lower variance in height, even at the edge of the table, as the controller captured the changes in ground effects when the drone flew over the table. In summary, the experimental results with multiple ground interaction scenarios show that much smaller tracking errors are obtained by Neural-Lander, which 40 is essentially the nonlinear tracking controller with feedforward cancellation of a spectrally-normalized DNN. References Abbeel, Pieter, Adam Coates, and Andrew Y. Ng (2010). Autonomous helicopter aerobatics through apprenticeship learning. In: The International Journal of Robotics Research 29.13, pp. 1608–1639. Bansal, Somil, Anayo K. Akametalu, Frank J. Jiang, Forrest Laine, and Claire J. Tomlin (2016). Learning quadrotor dynamics using neural network for flight control. In: 2016 IEEE 55th Conference on Decision and Control (CDC). IEEE, pp. 4653–4660. Bartlett, Peter L., Dylan J. Foster, and Matus J. Telgarsky (2017). Spectrallynormalized margin bounds for neural networks. In: Advances in Neural Information Processing Systems, pp. 6240–6249. Berkenkamp, Felix, Angela P. Schoellig, and Andreas Krause (2016). Safe controller optimization for quadrotors with Gaussian Processes. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 493–496. url: https : //arxiv.org/abs/1509.01066. Berkenkamp, Felix, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause (2017). Safe model-based reinforcement learning with stability guarantees. In: Neural Information Processing Systems (NIPS). url: https://arxiv.org/ abs/1705.08551. Cheeseman, I. C. and W. E. Bennett (1955). The effect of ground on a helicopter rotor in forward flight. In: Citeseer. Chung, Soon-Jo, Saptarshi Bandyopadhyay, Insu Chang, and Fred Y. Hadaegh (2013). Phase synchronization control of complex networks of Lagrangian systems on adaptive digraphs. In: Automatica 49.5, pp. 1148–1161. Danjun, Li, Zhou Yan, Shi Zongying, and Lu Geng (2015). Autonomous landing of quadrotor based on ground effect modelling. In: 2015 34th Chinese Control Conference (CCC). IEEE, pp. 5647–5652. Dziugaite, Gintare Karolina and Daniel M. Roy (2017). Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In: arXiv preprint arXiv:1703.11008. Frye, Michael T. and Robert S. Provence (2014). Direct inverse control using an artificial neural network for the autonomous hover of a helicopter. In: 2014 IEEE International Conference on Systems, Man and Cybernetics (SMC). IEEE, pp. 4121–4122. 41 He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun (2016). Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778. Khalil, Hassan K. (2002). Nonlinear systems. Pearson Education. Prentice Hall. isbn: 9780130673893. Li, Qiyang, Jingxing Qian, Zining Zhu, Xuchan Bao, Mohamed K. Helwa, and Angela P. Schoellig (2017). Deep neural networks for improved, impromptu trajectory tracking of quadrotors. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 5183–5189. Miyato, Takeru, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida (2018). Spectral normalization for generative adversarial networks. In: arXiv preprint arXiv:1802.05957. Neyshabur, Behnam, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro (2017). A PAC-Bayesian approach to spectrally-normalized margin bounds for neural networks. In: arXiv preprint arXiv:1707.09564. Neyshabur, Behnam, Srinadh Bhojanapalli, David McAllester, and Nati Srebro (2017). Exploring generalization in deep learning. In: Advances in Neural Information Processing Systems, pp. 5947–5956. Nonaka, Kenichiro and Hirokazu Sugizaki (2011). Integral sliding mode altitude control for a small model helicopter with ground effect compensation. In: American Control Conference (ACC), 2011. IEEE, pp. 202–207. Punjani, Ali and Pieter Abbeel (2015). Deep learning helicopter dynamics models. In: 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 3223–3230. Sánchez-Sánchez, Carlos and Dario Izzo (2018). Real-time optimal control via Deep Neural Networks: Study on landing problems. In: Journal of Guidance, Control, and Dynamics 41.5, pp. 1122–1135. Shi, Guanya, Xichen Shi, Michael O’Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung (2019). Neural Lander: Stable drone landing control using learned dynamics. In: 2019 International Conference on Robotics and Automation (ICRA). IEEE, pp. 9784–9790. doi: 10.1109/ICRA.2019.8794351. Suprijono, Herwin and Benyamin Kusumoputro (2017). Direct inverse control based on neural network for unmanned small helicopter attitude and altitude control. In: Journal of Telecommunication, Electronic and Computer Engineering (JTEC) 9.2-2, pp. 99–102. Zhang, Chiyuan, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals (2016). Understanding deep learning requires rethinking generalization. In: arXiv preprint arXiv:1611.03530. 42 Zhou, Siqi, Mohamed K. Helwa, and Angela P. Schoellig (2017). Design of deep neural networks as add-on blocks for improving impromptu trajectory tracking. In: 2017 IEEE 56th Annual Conference on Decision and Control (CDC). IEEE, pp. 5201–5207. 43 Chapter 4 NEURAL-SWARM (a) Inputs 1 Dynamics Heterogeneous Deep Sets 2 Learned Nominal + Interaction Dynamics 𝛴 3 {𝜱$%&'', 𝜱$%&'', 𝜱)*+} 4 Environment (b) 𝝆'&./) Nonlinear Stable Controller Motion Planner (c) Large Robot Small Robot Ground Effect Minimum Vertical Distance: 𝟐𝟒𝐜𝐦 Figure 4.1: Overview of Neural-Swarm. We learn complex interaction between multirotors using heterogeneous deep sets and design an interaction-aware nonlinear stable controller and a multi-robot motion planner (a). Our approach enables closeproximity flight (minimum vertical distance 24 cm) of heterogeneous aerial teams (16 robots) with significant lower tracking error compared to solutions that do not consider the interaction forces (b,c). In Chapter 2, we introduced a general mixed robot dynamics model 𝑀 (𝑞) 𝑞¥ + 𝐶 (𝑞, 𝑞) ¤ 𝑞¤ + 𝑔(𝑞) = 𝐵𝑢 + 𝑓 where 𝑓 is the unknown dynamics. We also introduced the corresponding drone dynamics model in Eq. (2.4). In Chapter 3, we discussed the case 𝑓 = 𝑓 (𝑞, 𝑞, ¤ 𝑢) where the main challenge is to ensure nonlinear stability and robustness when using a DNN to approximate 𝑓 . In this chapter, we will study a multi-agent interactive setting, where 𝑓 = 𝑓 (𝑞, 𝑞, ¤ neighbors’ states) and the main challenge is to design a scalable learning and control method which can generalize to different numbers of agents. For example, in multirotor swarm experiments in Fig. 4.1, 𝑓 refers to complex aerodynamic interactions between multirotors. In addition to learning-based control methods, we will also discuss 44 how to deal with motor delays and design learning-based motion planners. This chapter is mainly based on the following papers1 2: Shi, Guanya, Wolfgang Hönig, Xichen Shi, Yisong Yue, and Soon-Jo Chung (2022). Neural-Swarm2: Planning and control of heterogeneous multirotor swarms using learned interactions. In: IEEE Transactions on Robotics 38.2, pp. 1063–1079. doi: 10.1109/TRO.2021.3098436. Shi, Guanya, Wolfgang Hönig, Yisong Yue, and Soon-Jo Chung (2020). NeuralSwarm: Decentralized close-proximity multirotor control using learned interactions. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp. 3241–3247. doi: 10.1109/ICRA40945.2020.9196800. Abstract. We present Neural-Swarm, a learning-based method for motion planning and control that allows heterogeneous multirotors in a swarm to safely fly in close proximity. Such operation for drones is challenging due to complex aerodynamic interaction forces, such as downwash generated by nearby drones and ground effect. Conventional planning and control methods neglect capturing these interaction forces, resulting in sparse swarm configuration during flight. Our approach combines a physics-based nominal dynamics model with learned Deep Neural Networks (DNNs) with strong Lipschitz properties. We make use of two techniques to accurately predict the aerodynamic interactions between heterogeneous multirotors: i) spectral normalization for stability and generalization guarantees of unseen data and ii) heterogeneous deep sets for supporting any number of heterogeneous neighbors in a permutation-invariant manner without reducing expressiveness. The learned residual dynamics benefit both the proposed interaction-aware multi-robot motion planning and the nonlinear tracking control design because the learned interaction forces reduce the modelling errors. Experimental results demonstrate that NeuralSwarm is able to generalize to larger swarms beyond training cases and significantly outperforms a baseline nonlinear tracking controller with up to three times reduction in worst-case tracking errors. 4.1 Introduction The ongoing commoditization of unmanned aerial vehicles (UAVs) requires robots to fly in much closer proximity to each other than before, which necessitates ad1 Summary video: https://youtu.be/Y02juH6BDxo 2 Data and code: https://github.com/aerorobotics/neural-swarm 45 vanced planning and control methods for large aerial swarms (Chung et al., 2018; Morgan, Subramanian, et al., 2016). For example, consider a search-and-rescue mission where an aerial swarm must enter and search a collapsed building. In such scenarios, close-proximity flight enables the swarm to navigate the building much faster compared to swarms that must maintain large distances from each other. Other important applications of close-proximity flight include manipulation, search, surveillance, and mapping. In many scenarios, heterogeneous teams with robots of different sizes and sensing or manipulation capabilities are beneficial due to their significantly higher adaptability. For example, in a search-and-rescue mission, larger UAVs can be used for manipulation tasks or to transport goods, while smaller ones are more suited for exploration and navigation. A major challenge of close-proximity control and planning is that small distances between UAVs create complex aerodynamic interactions. For instance, one multirotor flying above another causes the so-called downwash effect on the lower one, which is difficult to model using conventional model-based approaches (Jain et al., 2019). Without accurate downwash interaction modeling, a large safety distance between vehicles is necessary, thereby preventing a compact 3-D formation shape, e.g., 60 cm for the small Crazyflie 2.0 quadrotor (9 cm rotor-to-rotor) (Hönig et al., 2018). Moreover, the downwash is sometimes avoided by restricting the relative position between robots in the 2-D horizontal plane (Du et al., 2019). For heterogeneous teams, even larger and asymmetric safety distances are required (Debord et al., 2018). However, the downwash for two small Crazyflie quadrotors hovering 30 cm on top of each other is only 9 g, which is well within their thrust capabilities, and suggests that proper modeling of downwash and other interaction effects can lead to more precise motion planning and dense formation control. In this chapter, we present a learning-based approach, Neural-Swarm, which enhances the precision, safety, and density of close-proximity motion planning and control of heterogeneous multirotor swarms. In the example shown in Fig. 4.1, we safely operate the same drones with vertical distances less than half of those of prior work (Hönig et al., 2018). In particular, we train deep neural networks (DNNs) to predict the residual interaction forces that are not captured by the nominal models of free-space aerodynamics. To the best of our knowledge, this is the first model for aerodynamic interactions between two or more multirotors in flight. Our DNN architecture supports heterogeneous inputs in a permutation-invariant manner without reducing the expressiveness. The DNN only requires relative positions and veloci- 46 ties of neighboring multirotors as inputs, similar to the existing collision-avoidance techniques (Berg et al., 2009), which enables fully-decentralized computation. We use the predicted interaction forces to augment the nominal dynamics and derive novel methods to directly consider them during motion planning and as part of the multirotors’ controller. From a learning perspective, we leverage and extend two state-of-the-art tools to derive effective DNN models. First, we extend deep sets (Zaheer et al., 2017) to the heterogeneous case and prove its representation power. Our novel encoding is used to model interactions between heterogeneous vehicle types in an index-free or permutation-invariant manner, enabling better generalization to new formations and a varying number of vehicles. The second is spectral normalization (see details in Chapter 2 and Algorithm 2.1), which ensures the DNN is Lipschitz continuous and helps the DNN generalize well on test examples that lie outside the training set. We demonstrate that the interaction forces can be computationally efficiently and accurately learned such that a small 32-bit microcontroller can predict such forces in real-time. From a planning and control perspective, we derive novel methods that directly consider the predicted interaction forces. For motion planning, we use a twostage approach. In the first stage, we extend an existing kinodynamic samplingbased planner for a single robot to the interaction-aware multi-robot case. In the second stage, we adopt an optimization-based planner to refine the solutions of the first stage. Empirically, we demonstrate that our interaction-aware motion planner both avoids dangerous robot configurations that would saturate the multirotors’ motors and reduces the tracking error significantly. For the nonlinear control we leverage the Lipschitz continuity of our learned interaction forces to derive stability guarantees similar to Chapter 3. The controller can be used to reduce the tracking error of arbitrary desired trajectories, including ones that were not planned with an interaction-aware planner. We validate our approach using two to sixteen quadrotors of two different sizes, and we also integrate ground effect and other unmodeled dynamics into our model, by viewing the physical environment as a special robot. To our knowledge, our approach is the first that models interactions between two or more multirotor vehicles and demonstrates how to use such a model effectively and efficiently for motion planning and control of aerial teams. 47 4.2 Related Work The aerodynamic interaction force applied to a single UAV flying near the ground (ground effect), has been discussed in Chapter 3. The interaction between two rotor blades of a single multirotor has been studied in a lab setting to optimize the placement of rotors on the vehicle (Shukla and Komerath, 2018). However, it remains an open question how this influences the flight of two or more multirotors in close proximity. Interactions between two multirotors can be estimated using a propeller velocity field model (Jain et al., 2019). Unfortunately, this method is hard to generalize to the multi-robot or heterogeneous case and it only considers the stationary case, which is inaccurate for real flights. The use of DNNs to learn higher-order residual dynamics or control actions is gaining attention in the areas of control and reinforcement learning settings (Le et al., 2016; Taylor et al., 2019; Cheng et al., 2019; McKinnon and Schoellig, 2019; Saveriano et al., 2017; Johannink et al., 2019). For swarms, a common encoding approach is to discretize the whole space and employ convolutional neural networks (CNNs), which yields a permutation-invariant encoding. Another common encoding for robot swarms is a Graphic Neural Network (GNN) (Scarselli et al., 2008; Tolstaya et al., 2020). GNNs have been extended to heterogeneous graphs (Zhang et al., 2019), but it remains an open research question how such a structure would apply to heterogeneous robot teams. We extend a different architecture, which is less frequently used in robotics applications, called deep sets (Zaheer et al., 2017). Deep sets enable distributed computation without communication requirements. Compared to CNNs, our approach: i) requires less training data and computation; ii) is not restricted to a pre-determined resolution and input domain; and iii) directly supports the heterogeneous swarm. Compared to GNNs, we do not require any direct communication between robots. Deep sets have been used in robotics for heterogeneous (Riviére et al., 2020) teams. Our heterogeneous deep set extension has a more compact encoding and we prove its representation power. For motion planning, empirical models have been used to avoid harmful interactions (Morgan, Subramanian, et al., 2016; Morgan, Chung, et al., 2014; Hönig et al., 2018; Debord et al., 2018; Mellinger et al., 2012). Typical safe boundaries along multi-vehicle motions form ellipsoids (Hönig et al., 2018) or cylinders (Debord et al., 2018) along the motion trajectories. Estimating such shapes experimentally would potentially lead to many collisions and dangerous flight tests and those collision-free regions are in general conservative. In contrast, we use deep learning to estimate 48 the interaction forces accurately in heterogeneous multi-robot teams. This model allows us to directly control the magnitude of the interaction forces to accurately and explicitly control the risk, removing the necessity of conservative collision shapes. 4.3 Problem Statement Similar to Neural-Lander in Chapter 3, Neural-Swarm can generally apply to any robotic system and we will focus on multirotor UAVs in this chapter for notational simplicity. In this section, we first recap single multirotor dynamics given in Eq. (2.4) with an extra first-order motor delay model. Then, we generalize these dynamics for a swarm of multirotors. Finally, we formulate our objective as a variant of an optimal control problem and introduce our performance metric. Single Multirotor Dynamics We will keep using the multirotor model given in Eq. (2.4) but with extra delay modelling: 𝑝¤ = 𝑣, 𝑚 𝑣¤ = 𝑚𝑔 + 𝑅 𝑓𝑢 + 𝑓𝑎 , (4.1a) 𝑅¤ = 𝑅𝑆(𝜔), 𝐽 𝜔¤ = 𝐽𝜔 × 𝜔 + 𝜏 + 𝜏𝑎 , (4.1b) 𝑢¤ = −𝜆𝑢 + 𝜆𝑢 𝑐 (4.1c) [𝑇; 𝜏𝑥 ; 𝜏𝑦 ; 𝜏𝑧 ] = 𝐵0 𝑢, | {z } 𝜂𝑜 where 𝜂 𝑜 is the output wrench, 𝑢 is the control input to the system (i.e., squared motor speeds), 𝑢 𝑐 is the actual command signal we can control and 𝜆 > 0 is the time constant of the delay model. Compared to Eq. (2.4) in Chapter 2, in this chapter we consider a first-order delay model in Eq. (4.1c). Moreover, the aerodynamic residual terms 𝑓𝑎 and 𝜏𝑎 are mainly from the interaction between other multirotors and the environment. Heterogeneous Swarm Dynamics We now consider 𝑁 multirotor robots. We use 𝑥 (𝑖) = [ 𝑝 (𝑖) ; 𝑣 (𝑖) ; 𝑅 (𝑖) ; 𝜔 (𝑖) ] to denote the state of the 𝑖 th multirotor. We use 𝑥 (𝑖 𝑗) to denote the relative state component ⊤ between robot 𝑖 and 𝑗, e.g., 𝑥 (𝑖 𝑗) = [ 𝑝 ( 𝑗) − 𝑝 (𝑖) ; 𝑣 ( 𝑗) − 𝑣 (𝑖) ; 𝑅 (𝑖) 𝑅 ( 𝑗) ]. We use I (𝑖) to denote the type of the 𝑖 th robot, where robots with identical physical parameters such as 𝑚, 𝐽, and 𝐵0 are considered to be of the same type. We assume there are 𝐾 ≤ 𝑁 types of robots, i.e., I (·) is a surjective mapping from {1, · · · , 𝑁 } to (𝑖) {type1 , · · · , type𝐾 }. Let rtype be the set of the relative states of the type𝑘 neighbors 𝑘 49 of robot 𝑖: (𝑖) = {𝑥 (𝑖 𝑗) | 𝑗 ∈ neighbor(𝑖) and I ( 𝑗) = type 𝑘 }. rtype 𝑘 (4.2) The neighboring function neighbor(𝑖) is defined by an interaction volume function V. Formally, 𝑗 ∈ neighbor(𝑖) if 𝑝 ( 𝑗) ∈ V ( 𝑝 (𝑖) , I (𝑖), I ( 𝑗)), i.e., robot 𝑗 is a neighbor of 𝑖 if the position of 𝑗 is within the interaction volume of 𝑖. In this chapter, we design V as a cuboid based on observed interactions in experiments. The ordered sequence of all relative states grouped by robot type is   (𝑖) (𝑖) (𝑖) rI(𝑖) = rtype , r , · · · , r (4.3) type2 type𝐾 . 1 The dynamics of the 𝑖 th multirotor can be written in compact form:   0    (𝑖) (𝑖)   𝑓𝑎 (r )  I , 𝑥¤ (𝑖) = Φ (𝑖) (𝑥 (𝑖) , 𝑢 (𝑖) ) +   0    (𝑖) (𝑖)   𝜏𝑎 (r )  I   (4.4) where Φ (𝑖) (𝑥 (𝑖) , 𝑢 (𝑖) ) denotes the nominal dynamics of robot 𝑖, and 𝑓𝑎(𝑖) (·) and 𝜏𝑎(𝑖) (·) are the unmodeled force and torque of the 𝑖 th robot that are caused by interactions with neighboring robots or the environment (e.g., ground effect and air drag). Robots with the same type have the same nominal dynamics and unmodeled force and torque: Φ (𝑖) (·) = ΦI (𝑖) (·), 𝑓𝑎(𝑖) (·) = 𝑓𝑎I (𝑖) (·), 𝜏𝑎(𝑖) (·) = 𝜏𝑎I (𝑖) (·) ∀𝑖. (4.5) Note that the homogeneous case is a special case where 𝐾 = 1, i.e., Φ (𝑖) (·) = Φ(·), 𝑓𝑎(𝑖) (·) = 𝑓𝑎 (·), and 𝜏𝑎(𝑖) (·) = 𝜏𝑎 (·) ∀𝑖. Our system is heterogeneous in three ways: i) different robot types have heterogeneous nominal dynamics ΦI (𝑖) ; ii) different robot types have different unmodeled 𝑓𝑎I (𝑖) and 𝜏𝑎I (𝑖) ; and iii) the neighbors of each robot belong to 𝐾 different sets. We highlight that our heterogeneous model not only captures different types of robot, but also different types of environmental interactions, e.g., ground effect (G. Shi, X. Shi, et al., 2019) and air drag. This is achieved in a straightforward manner by viewing the physical environment as a special robot type. We illustrate this generalization in the following example. 50 Example 4.1 (small and large robots, and the environment). We consider a heterogeneous system as depicted in Fig. 4.1(a). Robot 3 (large robot) has three neighbors: robot 1 (small), robot 2 (small) and environment 4. For robot 3, we have large 𝑓𝑎(3) = 𝑓𝑎 large (rI(3) ) = 𝑓𝑎 (3) (3) (3) ), (rsmall , rlarge , renv (3) (3) (3) = {𝑥 (34) } rsmall = {𝑥 (31) , 𝑥 (32) }, rlarge = ∅, renv and a similar expression for 𝜏𝑎(3) . Interaction-Aware Motion Planning & Control Our goal is to move the heterogeneous team of robots from their start states to goal states, which can be framed as the following optimal control problem: 𝑁 ∫ 𝑡𝑓 ∑︁ min ∥𝑢 (𝑖) (𝑡)∥𝑑𝑡 (4.6) 𝑢 (𝑖) ,𝑥 (𝑖) ,𝑡 𝑓 𝑖=1 0    robot dynamics (4.4)       𝑢 (𝑖) (𝑡) ∈ U I (𝑖) ; 𝑥 (𝑖) (𝑡) ∈ X I (𝑖)     s.t. ∥ 𝑝 (𝑖 𝑗) ∥ ≥ 𝑟 (I (𝑖)I ( 𝑗))    I (𝑖) I (𝑖)   ∥ 𝑓𝑎(𝑖) ∥ ≤ 𝑓𝑎,max ; ∥𝜏𝑎(𝑖) ∥ ≤ 𝜏𝑎,max       𝑥 (𝑖) (0) = 𝑥 𝑠(𝑖) ; 𝑥 (𝑖) (𝑡 𝑓 ) = 𝑥 (𝑖) 𝑓  𝑖 ∈ [1, 𝑁] 𝑖 ∈ [1, 𝑁] 𝑖 < 𝑗, 𝑗 ∈ [2, 𝑁] 𝑖 ∈ [1, 𝑁] 𝑖 ∈ [1, 𝑁] where U (𝑘) is the control space for type 𝑘 robots, X (𝑘) is the free space for type 𝑘 (𝑘) robots, 𝑟 (𝑙 𝑘) is the minimum safety distance between type𝑙 and type 𝑘 robots, 𝑓𝑎,max (𝑘) is the maximum endurable interaction force for type 𝑘 robots, 𝜏𝑎,max is the maximum (𝑖) endurable interaction torque for type 𝑘 robots, 𝑥 𝑠 is the start state of robot 𝑖, and 𝑥 (𝑖) 𝑓 is the desired state of robot 𝑖. In contrast with the existing literature (Debord et al., 2018), we assume a tight spherical collision model and bound the interaction forces directly, eliminating the need of manually defining virtual collision shapes. For I (𝑖) I (𝑖) instance, larger 𝑓𝑎,max and 𝜏𝑎,max will yield denser and more aggressive trajectories. Also note that the time horizon 𝑡 𝑓 is a decision variable. Solving Eq. (4.6) in real-time in a distributed fashion is intractable due to the exponential growth of the decision space with respect to the number of robots. Thus, we focus on solving two common subproblems instead. First, we approximately solve Eq. (4.6) offline as an interaction-aware motion planning problem. Second, we formulate an interaction-aware controller that minimizes the tracking error online. This controller can use both predefined trajectories and planned trajectories from the interaction-aware motion planner. 51 Since interaction between robots might only occur for a short time period with respect to the overall flight duration but can cause significant deviation from the nominal trajectory, we consider the worst tracking error of any robot in the team as a success metric: (4.7) max ∥ 𝑝 (𝑖) (𝑡) − 𝑝 𝑑(𝑖) (𝑡)∥, 𝑖,𝑡 where 𝑝 𝑑(𝑖) (𝑡) is the desired trajectory for robot 𝑖. Note that this metric reflects the worst error out of all robots, because different robots in a team have various levels of task difficulty. For example, the two-drone swapping task is very challenging for the bottom drone due to the downwash effect, but relatively easier for the top drone. Minimizing Eq. (4.7) implies improved tracking performance and safety of a multirotor swarm during tight formation flight. 4.4 Learning of Swarm Aerodynamic Interaction We employ state-of-the-art deep learning methods to capture the unknown (or residual) dynamics caused by interactions of heterogeneous robot teams. In order to use the learned functions effectively for motion planning and control, we require that the DNNs have strong Lipschitz properties (for stability analysis), can generalize well to new test cases, and use compact encodings to achieve high computational and statistical efficiency. To that end, we introduce heterogeneous deep sets, a generalization of regular deep sets (Zaheer et al., 2017), and employ spectral normalization for strong Lipschitz properties (see more details in Chapter 2). In this section, we will first review the homogeneous learning architecture covered in prior work (Zaheer et al., 2017). Then we generalize them to the heterogeneous case with representation guarantees. Finally, we introduce our data collection procedures. Homogeneous Permutation-Invariant Neural Networks Recall that in the homogeneous case, all robots are with the same type (type1 ). Therefore, the input to functions 𝑓𝑎 or 𝜏𝑎 is a single set. The permutation-invariant aspect of 𝑓𝑎 or 𝜏𝑎 can be characterized as: (𝑖) (𝑖) (𝑖) (𝑖) 𝑓𝑎 (rtype ) = 𝑓𝑎 (𝜋(rtype )), 𝜏𝑎 (rtype ) = 𝜏𝑎 (𝜋(rtype )) 1 1 1 1 for any permutation 𝜋. Since the aim is to learn the function 𝑓𝑎 and 𝜏𝑎 using DNNs, we need to guarantee that the learned DNN is permutation-invariant. Therefore, we consider the following “deep sets” (Zaheer et al., 2017) architecture to approximate homogeneous 𝑓𝑎 and 𝜏𝑎 : 52 " # " # ˆ𝑎(𝑖) © ∑︁ ª 𝑓 (𝑖 𝑗) ≈ ρ ­­ ϕ(𝑥 ) ®® B (𝑖) , (4.8) (𝑖) 𝜏𝑎 (rtype ) 𝜏ˆ𝑎 (𝑖) (𝑖 𝑗) 1 «𝑥 ∈rtype1 ¬ where ϕ(·) and ρ(·) are two DNNs. The output of ϕ is a hidden state to represent “contributions” from each neighbor, and ρ is a nonlinear mapping from the summation of these hidden states to the total effect. (𝑖) ) 𝑓𝑎 (rtype 1 Obviously the network architecture in Eq. (4.8) is permutation-invariant due to the inner sum operation. We now show that this architecture is able to approximate any continuous permutation-invariant function. The following Lemma 4.1 and Theorem 4.1 are adopted from (Zaheer et al., 2017) and will be used and extended in the next section for the heterogeneous case. Lemma 4.1. Define ϕ̄(𝑧) = [1; 𝑧; · · · ; 𝑧 𝑀 ] ∈ R 𝑀+1 as a mapping from R to R 𝑀+1 , and X = {[𝑥1 ; · · · ; 𝑥 𝑀 ] ∈ [0, 1] 𝑀 |𝑥 1 ≤ · · · ≤ 𝑥 𝑀 } as a subset of [0, 1] 𝑀 . For Í𝑀 x = [𝑥1 ; · · · ; 𝑥 𝑀 ] ∈ X, define q(x) = 𝑚=1 ϕ̄(𝑥 𝑚 ). Then q(x) : X → R 𝑀+1 is a homeomorphism. Proof. The proof builds on the Newton-Girard formulae, which connect the moments of a sample set (sum-of-power) to the elementary symmetric polynomials (see Zaheer et al. (2017)). □ Theorem 4.1. Suppose ℎ(x) : [0, 1] 𝑀 → R is a permutation-invariant continuous function, i.e., ℎ(x) = ℎ(𝑥1 , · · · , 𝑥 𝑀 ) = ℎ(𝜋(𝑥 1 , · · · , 𝑥 𝑀 )) for any permutation 𝜋. Then there exist continuous functions 𝜌¯ : R 𝑀+1 → R and ϕ̄ : R → R 𝑀+1 such that ! 𝑀 ∑︁ ℎ(x) = 𝜌¯ ϕ̄(𝑥 𝑚 ) , ∀x ∈ [0, 1] 𝑀 . 𝑚=1 Proof. We choose ϕ̄(𝑧) = [1; 𝑧; · · · ; 𝑧 𝑀 ] and 𝜌(·) ¯ = ℎ(q−1 (·)), where q(·) is defined in Lemma 4.1. Note that since q(·) is a homeomorphism, q−1 (·) exists 𝑀+1 to X. Therefore, 𝜌¯ is also a continuous and it is a continuous function from Í R  𝑀 𝑀+1 function from R to R, and 𝜌¯ 𝑚=1 ϕ̄(𝑥 𝑚 ) = 𝜌(q(x)) ¯ = ℎ(q−1 (q(x))) = ℎ(x) for x ∈ X. Finally, note that for any x ∈ [0, 1] 𝑀 , there exists some permutation 𝜋 such that 𝜋(x) ∈ X. Then because both 𝜌(q(x)) ¯ and ℎ(x) are permutation-invariant, we have 𝜌¯ (q(x)) = 𝜌¯ (q(𝜋(x))) = ℎ(𝜋(x)) = ℎ(x) for all x ∈ [0, 1] 𝑀 . □ 53 Theorem 4.1 focuses on scalar valued permutation-invariant continuous functions with hypercubic input space [0, 1] 𝑀 , i.e., each element in the input set is a scalar. In contrast, our learning target function [ 𝑓𝑎 ; 𝜏𝑎 ] in Eq. (4.8) is a vector valued function with a bounded input space, and each element in the input set is also a vector. However, Theorem 4.1 can be generalized in a straightforward manner by the following corollary. Corollary 4.1. Suppose x (1) , x (2) , · · · , x (𝑀) are 𝑀 bounded vectors in R𝐷 1 , and ℎ(x (1) , · · · , x (𝑀) ) is a continuous permutation-invariant function from R 𝑀×𝐷 1 to R𝐷 2 , i.e., ℎ(x (1) , · · · , x (𝑀) ) = ℎ(x𝜋(1) , · · · , x𝜋(𝑀) ) for any permutation 𝜋. Then ℎ(x (1) , · · · , x (𝑀) ) can be approximated arbitrarily close in the proposed architecture in Eq. (4.8). Proof. First, there exists a bijection from the bounded vector space in R𝐷 1 to [0, 1] after discretization, with finite but arbitrary precision. Thus, Theorem 4.1 is applicable. Second, we apply Theorem 4.1 𝐷 2 times and stack 𝐷 2 scalar-valued functions to represent the vector-valued function with output space R𝐷 2 . Finally, because DNNs are universal approximators for continuous functions (Csáji et al., 2001), the proposed architecture in Eq. (4.8) can approximate any ℎ(x (1) , · · · , x (𝑀) ) arbitrarily close. □ Heterogeneous 𝐾-Group Permutation-Invariant DNN Different from the homogeneous setting, the inputs to functions 𝑓𝑎I (𝑖) and 𝜏𝑎I (𝑖) in Eq. (4.5) are 𝐾 different sets. First, we define permutation-invariance in the heterogeneous case. Intuitively, we expect that the following equality holds: 𝑓𝑎I (𝑖)  (𝑖) (𝑖) rtype , · · · , rtype 1 𝐾  = 𝑓𝑎I (𝑖)  (𝑖) (𝑖) 𝜋1 (rtype ), · · · , 𝜋 𝐾 (rtype ) 1 𝐾  for any permutations 𝜋1 , · · · , 𝜋 𝐾 (similarly for 𝜏𝑎I (𝑖) ). Formally, we define 𝐾-group permutation invariance as follows. (𝑘) Definition 4.1 (𝐾-group permutation invariance). Let x (𝑘) = [𝑥 1(𝑘) ; · · · ; 𝑥 𝑀 ] ∈ 𝑘 𝑀 (1) (𝐾) 𝑀 𝐾 𝑘 [0, 1] for 1 ≤ 𝑘 ≤ 𝐾, and x = [x ; · · · ; x ] ∈ [0, 1] , where 𝑀𝐾 = Í𝐾 𝑀𝐾 → R is 𝐾-group permutation-invariant if 𝑘=1 𝑀 𝑘 . ℎ(x) : R ℎ([x (1) ; · · · ; x (𝐾) ]) = ℎ([𝜋1 (x (1) ); · · · ; 𝜋 𝐾 (x (𝐾) )]) for any permutations 𝜋1 , 𝜋2 , · · · , 𝜋 𝐾 . 54 For example, ℎ(𝑥 1 , 𝑥2 , 𝑦 1 , 𝑦 2 ) = max{𝑥 1 , 𝑥2 }+2·max{𝑦 1 , 𝑦 2 } is a 2-group permutationinvariant function, because we can swap 𝑥 1 and 𝑥 2 or swap 𝑦 1 and 𝑦 2 , but if we large interchange 𝑥 1 and 𝑦 1 the function value may vary. In addition, the 𝑓𝑎 function in Example 4.1 is a 3-group permutation-invariant function. Similar to Lemma 4.1, in order to handle ambiguity due to permutation, we define X𝑀𝑘 = {[𝑥 1 ; · · · ; 𝑥 𝑀𝑘 ] ∈ [0, 1] 𝑀𝑘 |𝑥 1 ≤ · · · ≤ 𝑥 𝑀𝑘 } and n o X𝐾 = [x (1) ; · · · ; x (𝐾) ] ∈ [0, 1] 𝑀𝐾 |x (𝑘) ∈ X𝑀𝑘 , ∀𝑘 . Finally, we show how a 𝐾-group permutation-invariant function can be approximated via the following theorem. Theorem 4.2. ℎ(x) : [0, 1] 𝑀𝐾 → R is a 𝐾-group permutation-invariant continuous function if and only if it has the representation ! ! 𝑀𝑘 𝑀1 𝑀𝐾 𝐾 ∑︁ ∑︁ ∑︁ ∑︁ (1) (𝐾) (𝑘) ℎ(x) = 𝜌¯ ϕ̄1 (𝑥 𝑚 ) + · · · + ϕ̄𝐾 (𝑥 𝑚 ) = 𝜌¯ ϕ̄ 𝑘 (𝑥 𝑚 ) , ∀x ∈ [0, 1] 𝑀𝐾 𝑚=1 𝑚=1 𝑘=1 𝑚=1 for some continuous outer and inner functions 𝜌¯ : R𝐾+𝑀𝐾 → R and ϕ̄ 𝑘 : R → R𝐾+𝑀𝐾 for 1 ≤ 𝑘 ≤ 𝐾. Proof. The sufficiency follows from that ℎ(x) is 𝐾-group permutation-invariant by construction. For the necessary condition, we need to find continuous functions 𝜌¯ and {ϕ̄ 𝑘 } 𝐾𝑘=1 given ℎ. We define ϕ̄ 𝑘 (𝑥) : R → R𝐾+𝑀𝐾 as  1       𝑥   ϕ̄ 𝑘 (𝑥) = [0 𝑀1 ; · · · ; 0 𝑀𝑘−1 ;  .  ; 0 𝑀𝑘+1 ; · · · ; 0 𝑀𝐾 ]  ..   𝑀  𝑥 𝑘    where 0 𝑀𝑘 = [0; · · · ; 0] ∈ R 𝑀𝑘 +1 . Then q𝐾 (x) = 𝑀𝑘 𝐾 ∑︁ ∑︁ ϕ̄ 𝑘 (𝑥 𝑚(𝑘) ) 𝑘=1 𝑚=1 is a homeomorphism from X𝐾 ⊆ R 𝑀𝐾 to R𝐾+𝑀𝐾 from Lemma 4.1. We choose −1 𝜌¯ : R𝐾+𝑀𝐾 → R as 𝜌(·) ¯ = ℎ(q−1 𝐾 (·)) which is continuous, because both q𝐾 and ℎ are continuous. Then 𝜌(q ¯ 𝐾 (x)) = ℎ(x) for x ∈ X𝐾 . Finally, because i) (1) (𝐾) for all x = [x ; · · · ; x ] in [0, 1] 𝑀𝐾 there exist permutations 𝜋1 , · · · , 𝜋 𝐾 such that [𝜋1 (x (1) ); · · · ; 𝜋 𝐾 (x (𝐾) )] ∈ X𝐾 ; and ii) both 𝜌(q ¯ 𝐾 (x)) and ℎ(x) are 𝐾-group permutation-invariant, we have 𝜌(q ¯ 𝐾 (x)) = ℎ(x) for x ∈ [0, 1] 𝑀𝐾 . □ 55 Original Space 𝒳" … 1 Latent Space {𝒒" 𝐱 |𝐱 ∈ 𝒳" } 𝑀$ ($) 𝑥$ Continuous Map 𝜌̅ (𝒒" (𝐱)) = ℎ(𝐱) ($) 𝑥*+ 1 (") 𝑥$ 2 … (") 𝑀" + …+ … Type1 (") 𝑥- 𝑥*. Target Space Type𝐾 Homeomorphism 𝒒" 𝐱 Figure 4.2: Illustration of Theorem 4.2. We first find a homeomorphism q𝐾 (·) between the original space and the latent space, and then find a continuous function 𝜌(·) ¯ such that 𝜌(q ¯ 𝐾 (·)) = ℎ(·). Figure 4.2 depicts the key idea of Theorem 4.2. Moreover, we provide a 2-group permutation-invariant function example to highlight the roles of ϕ and 𝜌 in the heterogeneous case. Example 4.2 (2-group permutation-invariant function). Consider ℎ(𝑥 1 , 𝑥2 , 𝑦 1 , 𝑦 2 ) = max{𝑥 1 , 𝑥2 } + 2 · max{𝑦 1 , 𝑦 2 }, which is 2-group permutation-invariant. Then we define ϕ𝑥 (𝑥) = [𝑒 𝛼𝑥 ; 𝑥𝑒 𝛼𝑥 ; 0; 0], ϕ𝑦 (𝑦) = [0; 0; 𝑒 𝛼𝑦 ; 𝑦𝑒 𝛼𝑦 ] and 𝜌([𝑎; 𝑏; 𝑐; 𝑑]) = 𝑏/𝑎 + 2 · 𝑑/𝑐. Note that 𝑦 1 𝑒 𝛼𝑦1 + 𝑦 2 𝑒 𝛼𝑦2 𝑥1 𝑒 𝛼𝑥1 + 𝑥 2 𝑒 𝛼𝑥2 + 2 · , 𝑒 𝛼𝑥1 + 𝑒 𝛼𝑥2 𝑒 𝛼𝑦1 + 𝑒 𝛼𝑦2 which is asymptotically equal to max{𝑥 1 , 𝑥2 } + 2 · max{𝑦 1 , 𝑦 2 } as 𝛼 → +∞. 𝜌(ϕ𝑥 (𝑥1 ) + ϕ𝑥 (𝑥2 ) + ϕ𝑦 (𝑦 1 ) + ϕ𝑦 (𝑦 2 )) = Similar to the homogeneous case, Theorem 4.2 can generalize to vector-output functions with a bounded input space by applying the same argument as in Corollary 4.1. We propose the following heterogeneous deep set structure to model the heterogeneous functions 𝑓𝑎I (𝑖) and 𝜏𝑎I (𝑖) : " I (𝑖) (𝑖) # (𝑖) 𝑓𝑎 (rtype1 , · · · , rtype ) 𝐾 " # 𝐾 ∑︁ ˆ𝑎(𝑖) ª ©∑︁ 𝑓 ≈ ρI (𝑖) ­­ ϕI ( 𝑗) (𝑥 (𝑖 𝑗) ) ®® B (𝑖) . (𝑖) (𝑖) 𝜏𝑎I (𝑖) (rtype , · · · , r ) 𝜏ˆ𝑎 type𝐾 𝑘=1 (𝑖 𝑗) (𝑖) 1 « 𝑥 ∈rtype𝑘 ¬ (4.9) Example 4.3 (Use of 3-group permutation-invariant function for multirotors). For example, in the heterogeneous system provided by Example 4.1 (as depicted in Fig. 4.1(a)), we have " # " large (3) # (3) (3)   𝑓𝑎 (rsmall , rlarge , renv ) 𝑓𝑎(3) (31) (32) (34) ≈ ρ ϕ (𝑥 ) + ϕ (𝑥 ) + ϕ (𝑥 ) , = env large small small large (3) (3) (3) 𝜏𝑎(3) 𝜏𝑎 (rsmall , rlarge , renv ) 56 for the large robot 3, where ϕsmall captures the interaction with the small robot 1 and 2, and ϕenv captures the interaction with the environment 4, e.g., ground effect and air drag. The structure in Eq. (4.9) has many valuable properties: • Representation ability. Since Theorem 4.2 is necessary and sufficient, we do not lose approximation power by using this constrained framework, i.e., any 𝐾-group permutation-invariant function can be learned by Eq. (4.9). We demonstrate strong empirical performance using relatively compact DNNs for ρI (𝑖) and ϕI ( 𝑗) . • Computational and sampling efficiency and scalability. Since the input dimension of ϕI ( 𝑗) is always the same as the single vehicle case, the feedforward computational complexity of Eq. (4.9) grows linearly with the number of neighboring vehicles. Moreover, the number of neural networks (ρI (𝑖) and ϕI ( 𝑗) ) we need is 2𝐾, which grows linearly with the number of robot types. In practice, we found that one hour flight data is sufficient to accurately learn interactions between two to five multirotors. • Generalization to varying swarm size. Given learned ϕI ( 𝑗) and ρI (𝑖) functions, Eq. (4.9) can be used to predict interactions for any swarm size. In other words, we can accurately model swarm sizes (slightly) larger than those used for training. In practice, we found that our model can give good predictions for five multirotor swarms, despite only being trained on one to three multirotor swarms. Theoretical analysis on this generalizability is an interesting future research direction. Curriculum Learning Training DNNs in Eq. (4.9) to approximate 𝑓𝑎I (𝑖) and 𝜏𝑎I (𝑖) requires collecting close formation flight data. However, the downwash effect causes the nominally controlled multirotors (without compensation for the interaction forces) to move apart from each other. Thus, we use a curriculum/cumulative learning approach: first, we collect data for two multirotors without a DNN and learn a model. Second, we repeat the data collection using our learned model as a feed-forward term in our controller, which allows closer-proximity flight of the two vehicles. Third, we repeat the procedure with increasing number of vehicles, using the current best model. 57 Note that our data collection and learning are independent of the controller used and independent of the 𝑓𝑎I (𝑖) or 𝜏𝑎I (𝑖) compensation. In particular, if we actively compensate for the learned 𝑓𝑎I (𝑖) or 𝜏𝑎I (𝑖) , this will only affect 𝜂 𝑜 in Eq. (4.1) and not the observed 𝑓𝑎I (𝑖) or 𝜏𝑎I (𝑖) . 4.5 Interaction-Aware Multi-Robot Planning We approximately solve Eq. (4.6) offline by using two simplifications: i) we plan sequentially for each robot, treating other robots as dynamic obstacles with known trajectories, and ii) we use double-integrator dynamics plus learned interactions. Both simplifications are common for multi-robot motion planning with applications to multirotors (Morgan, Subramanian, et al., 2016; Luis and Schoellig, 2019). Such a motion planning approach can be easily distributed and is complete for planning instances that fulfill the well-formed infrastructure property (Čáp et al., 2015). However, the interaction forces Eq. (4.9) complicate motion planning significantly, because the interactions are highly nonlinear and robot dynamics are not independent from each other anymore. For example, consider a three-robot team with two small and one large robot as in Fig. 4.1(a). Assume that we already have valid trajectories for the two small robots and now plan a motion for the large robot. The resulting trajectory might result in a significant downwash force for the small robots if the large robot flies directly above the small ones. This strong interaction might invalidate the previous trajectories small and 𝜏 small . of the small robots or even violate their interaction force limits 𝑓𝑎,max 𝑎,max Furthermore, the interaction force is asymmetric and thus it is not sufficient to only consider the interaction force placed on the large robot. We solve this challenge by directly limiting the change of the interaction forces placed on all neighbors when we plan for a robot. This concept is similar to trust regions in sequential optimization (Foust et al., 2020). The simplified state is 𝑥 (𝑖) = [ 𝑝 (𝑖) ; 𝑣 (𝑖) ; 𝑓ˆ𝑎(𝑖) ] and the simplified dynamics Eq. (4.4) become:  𝑣 (𝑖)     (𝑖) ˆ(𝑖)  (𝑖) (𝑖) (𝑖) (𝑖) 𝑥¤ = 𝑓 (𝑥 , 𝑢 ) = 𝑢 + 𝑓𝑎  . (4.10)   (𝑖) ¤̂  𝑓  𝑎   These dynamics are still complex and nonlinear because of 𝑓ˆ𝑎(𝑖) , which is the learned interaction force represented by DNNs in Eq. (4.9). We include 𝑓ˆ𝑎(𝑖) in our state space to simplify the enforcement of the bound on the interaction force in Eq. (4.6). 58 We propose a novel hybrid two-stage planning algorithm, see Algorithm 4.1, leveraging the existing approaches while still highlighting the importance of considering interactive forces/torques in the planning. The portions of the pseudo-code in Algorithm 4.1 that significantly differ from the existing methods to our approach are highlighted. In Stage 1, we find initial feasible trajectories using a kinodynamic sampling-based motion planner. Note that any kinodynamic planner can be used for Stage 1. In Stage 2, we use sequential convex programming (SCP) (Morgan, Subramanian, et al., 2016; Foust et al., 2020; Dinh and Diehl, 2010) to refine the initial solution to reach the desired states exactly and to minimize our energy objective defined in Eq. (4.6). Intuitively, Stage 1 identifies the homeomorphism class of the solution trajectories and fixes 𝑡 𝑓 , while Stage 2 finds the optimal trajectories to the goal within that homeomorphism class. Both stages differ from similar methods in the literature (Morgan, Subramanian, et al., 2016), because they need to reason over the coupling of the robots caused by interaction forces 𝑓ˆ𝑎(𝑖) . Stage 1: Sampling-Based Planning using Interaction Forces For Stage 1, any kinodynamic single-robot motion planner can be extended. For the coupled multi-robot setting in this chapter, we modify AO-RRT (Hauser and Zhou, 2016), which is is a meta-algorithm that uses the rapidly-exploring random tree (RRT) algorithm as a subroutine. Sampling-based planner. Our adaption of RRT (Lines 3 to 15 in Algorithm 4.1) works as follows. First, a random state 𝑥rand is uniformly sampled from the state space (Line 6) and the closest state 𝑥near that is already in the search tree T is found (Line 7). This search can be done efficiently in logarithmic time by employing a specialized data structured such as a kd-tree (Bentley, 1975) and requires the user to define a distance function on the state space. Second, an action is uniformly sampled from the action space (Line 8) and the dynamics Eq. (4.4) are forward propagated for a fixed time period Δ𝑡 using 𝑥near as the initial condition, e.g., by using the Runge-Kutta method (Line 9). Note that this forward propagation directly considers the learned dynamics 𝑓ˆ𝑎(𝑖) . Third, the new state 𝑥 new is checked for validity with respect to i) the state space (which includes 𝑓ˆ𝑎(𝑖) ), ii) collisions with other robots, and iii) change and bound of the neighbor’s interaction forces (Line 10). The first validity check ensures that the interaction force of the robot itself is bounded, while the third check is a trust region and upper bound for the neighbor’s interaction forces. If 𝑥 new is valid, it is added as a child node of 𝑥near in the search tree T (Line 11). 59 Algorithm 4.1: Interaction-aware motion planning Input: 𝑥 0(𝑖) , 𝑥 (𝑖) 𝑓 , Δ𝑡    (𝑖) (𝑖) = 𝑢 0(𝑖) , 𝑢 1(𝑖) , 𝑢 2(𝑖) , . . . , 𝑥𝑇(𝑖)(𝑖) −1 Result: Xsol = 𝑥 0(𝑖) , 𝑥1(𝑖) , 𝑥2(𝑖) , . . . , 𝑥𝑇(𝑖)(𝑖) , Usol ⊲ Stage 1: Find 𝑡 𝑓 and initial trajectories that are close to the goal state (𝑖) ← ∞, X (𝑖) ← (), U (𝑖) ← () ∀𝑖 ∈ {1, . . . , 𝑁 } 1 𝑐 sol sol 2 repeat 3 foreach 𝑖 ∈ RandomShuffle({1, . . . , 𝑁 }) do 4 T = ({x0(𝑖) }, ∅) 5 repeat 6 𝑥 rand ← UniformSample(X I (𝑖) ) 7 𝑥 near ← FindClosest(T , 𝑥 rand ) 8 𝑢 rand ← UniformSample(U I (𝑖) ) ( 𝑗) 9 𝑥 new , 𝑐 ← Propagate(𝑥 near , 𝑢 rand , Δ𝑡, {Xsol | 𝑗 ≠ 𝑖}) ( 𝑗) 10 if StateValid(xnew , {Xsol | 𝑗 ≠ 𝑖}) and 𝑐 ≤ 𝑐 (𝑖) then 11 Add(T , 𝑥near → 𝑥 new ) 12 if ∥𝑥 new − 𝑥 (𝑖) 𝑓 ∥ ≤ 𝜀 then (𝑖) 13 𝑐 ←𝑐 (𝑖) (𝑖) 14 Xsol , Usol ← ExtractSolution(T , 𝑥new ) 15 break 16 until TerminationCondition1() (𝑖) (𝑖) (𝑖) (𝑖) 17 X , U ← PostProcess(Xsol , Usol ) sol sol ⊲ Stage 2: Refine trajectories sequentially; Based on SCP 18 repeat 19 foreach 𝑖 ∈ RandomShuffle({1, . . . , 𝑁 }) do (𝑖) (𝑖) (𝑖) (𝑖) 20 Xsol , Usol ← SolveCP(Eq. (4.14), {Xsol |∀𝑖}, {Usol |∀𝑖}) 21 until Converged() Finally, if 𝑥 new is within an 𝜀-distance to the goal 𝑥 (𝑖) 𝑓 , the solution can be extracted by following the parent pointers of each tree node starting from 𝑥new until the root node 𝑥0(𝑖) is reached (Line 15). We note that our RRT steering method departs from ones in the literature which either sample Δ𝑡, use a best-control approximation of the steer method in RRT, or use a combination of both Δ𝑡-sampling and best-control approximation (Hauser and Zhou, 2016). We are interested in a constant Δ𝑡 for our optimization formulation in Stage 2. In this case, a best-control approximation would lead to a probabilistic incomplete planner (Kunz and Stilman, 2015). We adopt a technique of goal biasing where we pick the goal state rather than 𝑥 rand in fixed intervals, in order to improve 60 10 Initial trajectory of neighbor Search tree Rejected motions Solution 8 Z | fa | [g] 6 4 2 0 Y Figure 4.3: Example for Stage 1 of our motion planning with learned dynamics. Here, we have an initial solution for a small (blue) robot and plan for a large (orange) robot. The created search tree of the large robot is color-coded by the magnitude of the interaction force on the orange robot. During the search, we reject states that would cause a significant change in the interaction force for the blue robot (edges in blue). the convergence speed. While RRT is probabilistically complete, it also almost surely converges to a suboptimal solution (Karaman and Frazzoli, 2011). AO-RRT remedies this shortcoming by planning in a state-cost space and using RRTs sequentially with a monotonically decreasing cost bound. The cost bound 𝑐 (𝑖) is initially set to infinity (Line 1) and the tree can only grow with states that have a lower cost associated with them (Line 10). Once a solution is found, the cost bound is decreased accordingly (Line 13) and the search is repeated using the new cost bound (Line 2). This approach is asymptotically optimal, but in practice the algorithm is terminated based on some condition, e.g., a timeout or a fixed number of iterations without improvements (Line 16). Modification of sampling-based planner. We extend AO-RRT to a sequential interaction-aware multi-robot planner by adding 𝑓ˆ𝑎(𝑖) and time to our state space and treating the other robots as dynamic obstacles. As cost, we use a discrete approximation of the objective in Eq. (4.6). For each AO-RRT outer-loop iteration with a fixed cost bound, we compute trajectories sequentially using a random permutation of the robots (Line 3). When we check the state for validity (Line 10), we also enforce that the new state is not in collision with the trajectories of the other robots and that their 61 interaction forces are bounded and within a trust region compared to their previous value, see Fig. 4.3 for visualization. Here, the red edges show motions that cause large (≈ 10 g) but admissible interaction forces on the orange robot, because the blue robot flies directly above it. The blue edges are candidate edges as computed in Line 9 and are not added to the search tree, because their motion would cause a violation of the interaction force trust region of the blue robot (condition in Line 10). Once the search tree contains a path to the goal region, a solution is returned (orange path). (𝑖) The output of the sequential planner (Line 15) is a sequence of states Xsol and (𝑖) actions Usol , each to be applied for a duration of Δ𝑡. Note that the sequences might have different lengths for each robot. Implicitly, the sequences also define 𝑡 𝑓 . Furthermore, the first element of each sequence is the robots’ start state and the last element is within a 𝜀-distance of the robots’ goal state. We postprocess this sequence of states to make it an appropriate input for the optimization, e.g., for uniform length (Line 17). In practice, we found that repeating the last state and adding null actions, or (virtual) tracking of the computed trajectories using a controller are efficient and effective postprocessing techniques. Other sampling-based methods can be used as foundation of the first stage as well, with similar changes in sequential planning, state-augmentation to include the interaction forces, state-validity checking, and postprocessing. Stage 2: Optimization-Based Motion Planning We employ sequential convex programming (SCP) for optimization. SCP is a local optimization method for nonconvex problems that leverages convex optimization. The key concept is to convexify the nonconvex portions of the optimization problem by linearizing around a prior solution. The resulting convex problem instance is solved and a new solution obtained. The procedure can be repeated until convergence criteria are met. Because of the local nature of this procedure, a good initial guess is crucial for high-dimensional and highly nonlinear system dynamics. In our case, we use the searched trajectories from Stage 1 in Section 4.5 as the initial guess. We first adopt a simple zero-order hold temporal discretization of the dynamics Eq. (4.10) using Euler integration: (𝑖) 𝑥 𝑘+1 = 𝑥 𝑘(𝑖) + 𝑥¤ 𝑘(𝑖) Δ𝑡. (4.11) 62 Second, we linearize 𝑥¤ 𝑘(𝑖) around prior states 𝑥¯ 𝑘(𝑖) and actions 𝑢¯ 𝑘(𝑖) : 𝑥¤ 𝑘(𝑖) ≈ 𝐴 𝑘 (𝑥 𝑘(𝑖) − 𝑥¯ 𝑘(𝑖) ) + 𝐵 𝑘 (𝑢 𝑘(𝑖) − 𝑢¯ 𝑘(𝑖) ) + 𝑓 (𝑖) ( 𝑥¯ 𝑘(𝑖) , 𝑢¯ 𝑘(𝑖) ), (4.12) where 𝐴 𝑘 and 𝐵 𝑘 are the partial derivative matrices of 𝑓 (𝑖) with respect to 𝑥 𝑘(𝑖) and 𝑢 𝑘(𝑖) evaluated at 𝑥¯ 𝑘(𝑖) , 𝑢¯ 𝑘(𝑖) . Because we encode 𝑓ˆ𝑎(𝑖) using fully-differentiable DNNs, the partial derivatives can be efficiently computed analytically, e.g., by using autograd in PyTorch (Paszke et al., 2019). ( 𝑗) Third, we linearize 𝑓ˆ𝑎 𝑗 ∈ neighbor(𝑖): around our prior states 𝑥¯ 𝑘(𝑖) for all neighboring robots ( 𝑗) ( 𝑗) ( 𝑗) 𝑓ˆ𝑎 ≈ 𝐶 𝑘 (𝑥 𝑘(𝑖) − 𝑥¯ 𝑘(𝑖) ) + 𝑓ˆ𝑎 (rI(𝑖) ( 𝑥¯ 𝑘(𝑖) )), (4.13) ( 𝑗) ( 𝑗) where 𝐶 𝑘 is the derivative matrix of 𝑓ˆ𝑎 (the learned interaction function of robot 𝑗, represented by DNNs) with respect to 𝑥 𝑘(𝑖) evaluated at 𝑥¯ 𝑘(𝑖) ; and rI(𝑖) ( 𝑥¯ 𝑘(𝑖) ) is the ordered sequence of relative states as defined in Eq. (4.3) but using the fixed prior state 𝑥¯ 𝑘(𝑖) rather than decision variable 𝑥 𝑘(𝑖) in Eq. (4.2). We now formulate a convex program, one per robot: min (𝑖) (𝑖) Xsol ,Usol 𝑇 ∑︁ ∥𝑢 𝑘(𝑖) ∥ 2 + 𝜆 1 ∥𝑥𝑇(𝑖) − 𝑥 (𝑖) 𝑓 ∥ ∞ + 𝜆2 𝛿 (4.14) 𝑡=0 subject to:    robot dynamics Eq. (4.11) and Eq. (4.12)       𝑢 𝑘(𝑖) ∈ U I (𝑖)       𝑥 (𝑖) ∈ X I (𝑖)    𝑘 (𝑖 𝑗) 𝛿(𝑖)  (𝑖 𝑗) ⟨ 𝑝¯ 𝑘 , 𝑝 𝑘 − 𝑝¯ 𝑘(𝑖) ⟩ ≥ 𝑟 (I (𝑖)I ( 𝑗)) ∥ 𝑝¯ 𝑘 ∥ 2      𝑥0(𝑖) = 𝑥 𝑠(𝑖)     ( 𝑗)   |𝐶 𝑘 (𝑥 𝑘(𝑖) − 𝑥¯ 𝑘(𝑖) )| ≤ 𝑏 𝑓 𝑎       |𝑥 𝑘(𝑖) − 𝑥¯ 𝑘(𝑖) | ≤ 𝑏 𝑥 ; |𝑢 𝑘(𝑖) − 𝑢¯ 𝑘(𝑖) | ≤ 𝑏 𝑢  𝑖 ∈ [1, 𝑁] 𝑖 ∈ [1, 𝑁] 𝑖 ∈ [1, 𝑁], 𝛿 ≥ 0 𝑖 < 𝑗, 𝑗 ∈ [2, 𝑁] 𝑖 ∈ [1, 𝑁] 𝑖 < 𝑗, 𝑗 ∈ [2, 𝑁] 𝑖 ∈ [1, 𝑁] where X𝛿I (𝑖) is the state space increased by 𝛿 in each direction, the linearized robot dynamics are similar to (Foust et al., 2020; Nakka et al., 2021), and the convexified inter-robot collision constraint is from (Morgan, Subramanian, et al., 2016). We use soft constraints for reaching the goal (with weight 𝜆1 ) and the state space (with weight 𝜆 2 ), and trust regions around 𝑥¯ 𝑘(𝑖) , 𝑢¯ 𝑘(𝑖) , and the neighbors’ interaction forces for numerical stability. Interaction forces are constrained in (4.14) because 𝑓ˆ𝑎(𝑖) is part of the state space X I (𝑖) . 63 We solve these convex programs sequentially and they converge to a locally optimal solution (Morgan, Subramanian, et al., 2016). For the first iteration, we linearize around the trajectory computed during Stage 1 of our motion planner while subsequent iterations linearize around the solution of the previous iteration (Lines 18 to 21 in Algorithm 4.1). It is possible to implement Algorithm 4.1 in a distributed fashion similar to prior work (Morgan, Subramanian, et al., 2016). 4.6 Interaction-Aware Tracking Controller Delay Compensation Desired Trajectory ($) ($) 𝐟) − Position Controller ($) 𝐟+, Thrust Mixing 𝝉) ($) Attitude Controller 𝐮/ Vehicle Dynamics ($) 𝝉-, Interaction Estimation Environment 𝐱 ($) Relative Neighbor States {𝐱 ($%) } Figure 4.4: Hierarchy of control and planning blocks with information flow for commands and sensor feedback. We use different colors to represent heterogeneous neighbors. Note that the neighbors will influence the vehicle dynamics (dashed arrow). Given arbitrary desired trajectories, including ones that have not been computed using the method presented in Section 4.5, we augment the hierarchical multirotor controller discussed in Chapter 2 with delay compensation. The overall control structure is shown in Fig. 4.4. First, we recap a multi-agent integral variant of the hierarchical control structure given in Chapter 2. For the 𝑖 th agent, we have the following control law: 𝑓𝑑(𝑖) = 𝑚 (𝑖) 𝑝¥𝑟(𝑖) − 𝑚 (𝑖) 𝑔 − 𝐾 (𝑖) 𝑠 (𝑖) − 𝑓ˆ𝑎(𝑖) 𝜏𝑑(𝑖) = 𝐽 (𝑖) 𝜔¤ 𝑟(𝑖) − 𝐽 (𝑖) 𝜔 (𝑖) × 𝜔𝑟(𝑖) − 𝐾𝜔(𝑖) (𝜔 (𝑖) − 𝜔𝑟(𝑖) ) − Γ𝜔(𝑖) ∫ (𝜔 (𝑖) − 𝜔𝑟(𝑖) )𝑑𝑡 − 𝜏ˆ𝑎(𝑖) . Since most of theoretical analyses are covered in Chapter 2 and Chapter 3. Here we highlight three main differences of the above controller from previous chapters: Decentralized, heterogeneous, and interaction-aware. Note that in this chapter, we consider a heterogeneous multi-agent setting, where the nominal dynamics (e.g., 64 𝑚 (𝑖) , 𝐽 (𝑖) , 𝐾 (𝑖) ) depend on the agent type I (𝑖). Moreover, the learned interaction term 𝑓ˆ𝑎(𝑖) and 𝜏ˆ𝑎(𝑖) are computed in real time in a decentralized manner: " # 𝐾 ∑︁ ©∑︁ ª 𝑓ˆ𝑎(𝑖) (𝑖 𝑗) ® ­ = ρ ϕ (𝑥 ) I ( 𝑗) I (𝑖) ­ ®. 𝜏ˆ𝑎(𝑖) 𝑘=1 𝑥 (𝑖 𝑗) ∈r (𝑖) type 𝑘 « ¬ Integral control. We also enhance the controller in Chapter 2 by two integral terms. In the position control law, the composite variable 𝑠 (𝑖) contains an integral ∫ term such that 𝑝¤̃ (𝑖) + 2Λ (𝑖) 𝑝˜ (𝑖) + Λ (𝑖)2 𝑝˜ (𝑖) . The attitude control law also contains ∫ an integral term Γ𝜔(𝑖) (𝜔 (𝑖) − 𝜔𝑟(𝑖) )𝑑𝑡. Adding integral terms makes the tracking error bound depend on the maximum time-derivative of the approximation error (i.e., 𝑑𝑡𝑑 ( 𝑓ˆ𝑎(𝑖) − 𝑓𝑎(𝑖) ) or 𝑑𝑡𝑑 ( 𝜏ˆ𝑎(𝑖) − 𝜏𝑎(𝑖) )). See more details in G. Shi, Hönig, et al. (2022). (𝑖) Delay compensation. With the desired output wrench 𝜂 𝑜,𝑑 = [𝑇𝑑(𝑖) ; 𝜏𝑑(𝑖) ] from the position and attitude controller, the next step is to compute the control command 𝑢 𝑐(𝑖) . Note that we can not directly control the motor speed 𝑢 (𝑖) because of the first order delay model in Eq. (4.1c). Therefore, we apply the following delay compensation method: (𝑖) 𝜂¤𝑜,𝑑 ª (𝑖) (𝑖)† © (𝑖) (4.15) 𝑢 𝑐 = 𝐵0 ­𝜂 𝑜,𝑑 + (𝑖) ® . 𝜆 ¬ « (𝑖) The key idea of Eq. (4.15) is to “predict” the future of 𝜂 𝑜,𝑑 and account for it (𝑖) according to the delay constant 𝜆 . If the delay constant is accurate, one can show that the actual output wrench 𝜂 𝑜(𝑖) will exponentially converge to the desired one (X. Shi et al., 2020). With small modelling errors on 𝜆 (𝑖) , it can robustly cancel out some effects from delays and improve tracking performance in practice. Furthermore, it can handle not only first-order motor delay, but also signal transport delays (X. Shi et al., 2020). In case of the small quadrotors used in our experiments, such delays are on the same order of magnitude as the motor delay, thus making Eq. (4.15) essential for improving the control performance. 4.7 Experiments We use quadrotors based on Bitcraze Crazyflie 2.0/2.1 (CF). Our small quadrotors are Crazyflie 2.X, which are small (9 cm rotor-to-rotor) and lightweight (34 g) products that are commercially available. Our large quadrotors use the Crazyflie 2.1 as control board on a larger frame with brushed motors (model: Parrot Mini Drone), see Table 4.1 for a summary of physical parameters. We use the Crazyswarm (Preiss 65 et al., 2017) package to control multiple Crazyflies simultaneously. Each quadrotor is equipped with a single reflective marker for position tracking at 100 Hz using a motion capture system. The nonlinear controller, extended Kalman filter, and neural network evaluation are running on-board the STM32 microcontroller. Table 4.1: System identification of the used quadrotors. Weight Max Thrust Diameter 𝜆 𝜓1 ( 𝑝, ˆ 𝑣ˆ ) 𝜓2 ( 𝑓ˆ, 𝑣ˆ ) 𝜓3 ( 𝑝, ˆ 𝑣ˆ ) Small Large 34 g 67 g 65 g 145 g 12 cm 16 11.09 − 39.08 𝑝ˆ − 9.53𝑣ˆ + 20.57 𝑝ˆ2 + 38.43 𝑝ˆ𝑣ˆ 0.5 + 0.12 𝑓ˆ − 0.41𝑣ˆ − 0.002 𝑓ˆ2 − 0.043 𝑓ˆ𝑣ˆ −9.86 + 3.02 𝑝ˆ + 26.72𝑣ˆ 19 cm 16 44.1.0 − 122.51 𝑝ˆ − 36.18𝑣ˆ + 53.11 𝑝ˆ2 + 107.68 𝑝ˆ𝑣ˆ 0.56 + 0.06 𝑓ˆ − 0.6𝑣ˆ − 0.0007 𝑓ˆ2 − 0.015 𝑓ˆ𝑣ˆ −29.91 + 8.1 𝑝ˆ + 65.2𝑣ˆ For the controller, we implement the delay compensation Eq. (4.15) by numerical differentiation. The baseline controller is identical (including the chosen gains) to our proposed controller except that the interaction force for the baseline is set to zero. The baseline controller is much more robust and efficient than the welltuned nonlinear controller in the Crazyswarm package, which cannot safely execute the close-proximity flight shown in Fig. 4.1(c) and requires at least 60 cm safety distance (Hönig et al., 2018). For data collection, we use the 𝜇SD card extension board and store binary encoded data roughly every 10 ms. Each dataset is timestamped using the on-board microsecond timer and the clocks are synchronized before takeoff using broadcast radio packets. The drift of the clocks of different Crazyflies can be ignored for our short flight times (less than 2 min). Calibration and System Identification of Different Robots Prior to learning the residual terms 𝑓𝑎(𝑖) and 𝜏𝑎(𝑖) , we first calibrate the nominal dynamics model Φ (𝑖) (𝑥, 𝑢). We found that existing motor thrust models (Bitcraze, 2015; Förster, 2015) are not very accurate, because they only consider a single motor 66 and ignore the effect of the battery state of charge. We calibrate each Crazyflie by mounting the whole quadrotor upside-down on a load cell (model TAL221) which is directly connected to the Crazyflie via a custom extension board using a 24-bit ADC (model HX711). The upside-down mounting avoids contamination of our measurements with downwash-related forces. We use a 100 g capacity load cell for the small quadrotor and a 500 g capacity load cell for the large quadrotor. We randomly generate desired PWM motor signals (identical for all 4 motors) and collect the current battery voltage, PWM signals, and measured force. We use this data to find three polynomial functions: 𝜓1 , 𝜓2 , and 𝜓3 . The first 𝑓ˆ = 𝜓1 ( 𝑝, ˆ 𝑣ˆ ) computes the force of a single rotor given the normalized PWM signal 𝑝ˆ and the normalized battery voltage 𝑣ˆ . This function is only required for the data collection preparation in order to compute 𝑓𝑎(𝑖) . The second 𝑝ˆ = 𝜓2 ( 𝑓ˆ, 𝑣ˆ ) computes the required PWM signal 𝑝ˆ given the desired force 𝑓ˆ and current battery voltage 𝑣ˆ . Finally, 𝑓ˆmax = 𝜓3 ( 𝑝, ˆ 𝑣ˆ ) computes the maximum achievable force 𝑓ˆmax , given the current PWM signal 𝑝ˆ and battery voltage 𝑣ˆ . The last two functions are important at runtime for outputting the correct force as well as for thrust mixing when motors are saturated (Faessler et al., 2017). We use the same measurement setup with the load cell to establish the delay model of 𝑇𝑑(𝑖) with a square wave PWM signal. While the delay model is slightly asymmetric in practice, we found that our symmetric model Eq. (4.1c) is a good approximation. All results are summarized in Table 4.1. We use the remaining parameters (𝐽, thrust-to-torque ratio) from the existing literature (Förster, 2015). Data Collection Table 4.2: 12 scenarios for data collection. S→S ρsmall (ϕenv + ϕsmall ) Scenario S Model ρsmall (ϕenv ) Scenario {S, L} → S ρsmall (ϕenv + ϕsmall + ϕlarge ) {L, L} → S ρsmall (ϕenv + ϕlarge + ϕlarge ) Scenario L→L Model ρlarge (ϕenv + ϕlarge ) {S, S} → L ρlarge (ϕenv + ϕsmall + ϕsmall ) Model L→S ρsmall (ϕenv +ϕlarge ) {S, S} → S ρsmall (ϕenv + ϕsmall + ϕsmall ) L S→L ρlarge (ϕenv ) ρlarge (ϕenv +ϕsmall ) {S, L} → L ρlarge (ϕenv + ϕsmall + ϕlarge ) {L, L} → S ρlarge (ϕenv + ϕlarge + ϕlarge ) Recall that in Eq. (4.9), we need to learn 2𝐾 neural networks for 𝐾 types of robots. In our experiments, we consider two types of quadrotors (small and large) and also the environment (mainly ground effect and air drag), as shown in Example 4.1. 67 Therefore, we have 5 neural networks to be learned: ρsmall , ρlarge , ϕsmall , ϕlarge , ϕenv , (4.16) where we do not have ρenv because the aerodynamical force acting on the environment is not interesting for our purpose. To learn these 5 neural networks, we fly the heterogeneous swarm in 12 different scenarios (see Table 4.2) to collect labeled 𝑓𝑎(𝑖) and 𝜏𝑎(𝑖) data for each robot. For instance, Example 4.1 (as depicted in Fig. 3.1(a)) corresponds to the “{S, S} → L” scenario in Table 4.2, where the large robot has two small robots and the environment as its neighbors. We utilize two types of data collection tasks: random walk and swapping. For random walk, we implement a simple reactive collision avoidance approach based on artificial potentials on-board each Crazyflie (Khatib, 1985). The host computer randomly selects new goal points within a cube for each vehicle at a fixed frequency. These goal points are used as an attractive force, while neighboring drones contribute a repulsive force. For swapping, the drones are placed in different horizontal planes on a cylinder and tasked to move to the opposite side. All the drones are vertically aligned for one time instance, causing a large interaction force. The random walk data helps us to explore the whole space quickly, while the swapping data ensures that we have data for a specific task of interest. Note that for both random walk and swapping, the drones also move close to the ground, to collect sufficient data for learning the ground effect. The collected data covers drone flying speeds from 0 to 2 m/s, where 7% are with relatively high speeds (≥0.5 m/s) to learn the aerodynamic drag. For both task types, we varied the scenarios listed in Table 4.2. To learn the 5 DNNs in Eq. (4.16), for each robot 𝑖 in each scenario, we collect the timestamped states 𝑥 (𝑖) = [ 𝑝 (𝑖) ; 𝑣 (𝑖) ; 𝑅 (𝑖) ; 𝜔 (𝑖) ]. We then compute 𝑦 (𝑖) as the observed value of 𝑓𝑎(𝑖) and 𝜏𝑎(𝑖) . We compute 𝑓𝑎(𝑖) and 𝜏𝑎(𝑖) using Eq. (4.4), where the nominal dynamics Φ (𝑖) is calculated based on our system identification. With Φ (𝑖) , 𝑦 (𝑖) is computed by 𝑥¤ (𝑖) − Φ (𝑖) , where 𝑥¤ (𝑖) is estimated by the five-point numerical differentiation method. Note that the control delay 𝜆 (𝑖) is also considered when we compute 𝑓𝑎(𝑖) and 𝜏𝑎(𝑖) . Our training data consists of sequences of  (𝑖) (𝑖) (𝑖) {rtype , · · · , rtype }, 𝑦 (𝑖) pairs, where rtype = {𝑥 (𝑖 𝑗) | 𝑗 ∈ neighbor(𝑖) and I ( 𝑗) = 1 𝐾 𝑘 type 𝑘 } is the set of the relative states of the type-𝑘 neighbors of 𝑖. We have the following loss function for robot 𝑖 in each scenario (see Table 4.2 for the detailed 68 model structure in each scenario): ρI (𝑖) 𝐾 ∑︁ 2 ∑︁ ϕI ( 𝑗) (𝑥 𝑘=1 𝑥 (𝑖 𝑗) ∈r (𝑖) type 𝑘 (𝑖 𝑗)  ) −𝑦 (𝑖) , (4.17) 2 and we stack all the robots’ data in all scenarios and train on them together. There are 1.4 million pairs in the full dataset. In practice, we found the unmodeled torque ∥𝜏𝑎(𝑖) ∥ is very small (smaller by two orders of magnitude than the feedback term in the attitude controller), so we only learn 𝑓𝑎(𝑖) . We compute the relative states from our collected data as 𝑥 (𝑖 𝑗) = [ 𝑝 ( 𝑗) − 𝑝 (𝑖) ; 𝑣 ( 𝑗) − 𝑣 (𝑖) ] ∈ R6 (i.e., relative position and relative velocity both in the world frame), since the attitude information 𝑅 and ω are not dominant for 𝑓𝑎(𝑖) . If the type of neighbor 𝑗 is “environment”, we set 𝑝 ( 𝑗) = 0 and 𝑣 ( 𝑗) = 0. In this work, we only learn the 𝑧-component of 𝑓𝑎(𝑖) since we found the other two components, 𝑥 and 𝑦, are much smaller and less varied, and do not significantly alter the nominal dynamics. In data collection, the rooted means and standard deviations of the squared values of 𝑓𝑎,𝑥 , 𝑓𝑎,𝑦 , and 𝑓𝑎,𝑧 are 1.6±2.5, 1.2±2.2, 5.0±8.9 grams, respectively (for reference, the weights of the small and large drones are 34 g and 67 g). Therefore, the output of our learning model in Eq. (4.9) is a scalar to approximate the 𝑧-component of the unmodeled force function 𝑓𝑎(𝑖) . Learning Results and Ablation Analysis Figure 4.5: 𝑓𝑎,𝑧 prediction from the trained {ρsmall , ρlarge , ϕsmall , ϕlarge , ϕenv } networks. Each heatmap gives the prediction of 𝑓𝑎,𝑧 of a vehicle in different horizontal and vertical (global) positions. The (global) position of neighboring drones are represented by drone icons. 69 Each scenario uses a trajectory with a duration around 1000 seconds. For each scenario, we equally split the total trajectory into 50 shorter pieces, where each one is about 20 seconds. Then we randomly choose 80% of these 50 trajectories for training and 20% for validation. Our DNN functions of ϕ (ϕsmall , ϕlarge , ϕenv ) have four layers with architecture 6 → 25 → 40 → 40 → 𝐻, and our ρ DNNs (ρsmall , ρlarge ) also have 𝐿 = 4 layers, with architecture 𝐻 → 40 → 40 → 40 → 1. We use the ReLU function as the activation operator, and we use PyTorch (Paszke et al., 2019) for training and implementation of spectral normalization (see Chapter 2) of all these five DNNs. During training we iterate all the data 20 times for error convergence. Note that 𝐻 is the dimension of the hidden state. To study the effect of 𝐻 on learning performance, we use three different values of 𝐻 and the mean validation errors for each 𝐻 are shown in Table 4.3. Meanwhile, we also study the influence of the number of layers by fixing 𝐻 = 20 and changing 𝐿, which is the number of layers of all ρ nets and ϕ nets. For 𝐿 = 3 or 𝐿 = 5, we delete or add a 40 → 40 layer for all ρ nets and ϕ nets, before their last layers. We repeat all experiments three times to get mean and standard deviation. As depicted in Table 4.3, we found that the average learning performance (mean validation error) is not sensitive to 𝐻, but larger 𝐻 results in higher variance, possibly because using a bigger hidden space (larger 𝐻) leads to a more flexible encoding that is harder to train reliably. In terms of the number of layers, four layers are significantly better than five (which tends to overfit data), and slighter better than three. To optimize performance, we finally choose 𝐻 = 20 and use four-layer neural networks, which can be efficiently evaluated on-board. We notice that 𝐻 and 𝐿 are the most important parameters, and the learning performance is not sensitive to other parameters such as the number of weights in intermediate layers. Figure 4.5 depicts the prediction of 𝑓𝑎,𝑧 , trained with flight data from the 12 scenarios listed in Table 4.2. The color encodes the magnitude of 𝑓ˆ𝑎,𝑧 for a single small multirotor positioned at different global (𝑦, 𝑧) coordinates. The big/small black drone icons indicate the (global) coordinates of neighboring big/small multirotors, and the dashed line located at 𝑧 = 0 represents the ground. All quadrotors are in the same 𝑥-plane. For example, in Fig. 4.5(e), one large quadrotor is hovering at (𝑦 = −0.1, 𝑧 = 0.5) and one small quadrotor is hovering at (𝑦 = 0.1, 𝑧 = 0.5). If we place a third small quadrotor at (𝑦 = 0, 𝑧 = 0.3), it would estimate 𝑓ˆ𝑎,𝑧 = −10 g as indicated by the red color in that part of the heatmap. Similarly, in Fig. 4.5(a) 70 the small multirotor only has the environment as a special neighbor. If the small multirotor is hovering at (𝑦 = 0, 𝑧 = 0.05), it would estimate 𝑓ˆ𝑎,𝑧 = 5 g, which is mainly from the ground effect. All quadrotors are assumed to be stationary except for Fig. 4.5(d), where the one neighbor is moving at 0.8 m/s. Table 4.3: Ablation analysis. Top: 𝐿 = 4 and 𝐻 varies. Bottom: 𝐻 = 20 and 𝐿 varies. The error is the mean squared error (MSE) between 𝑓𝑎,𝑧 prediction and the ground truth. 𝐻 Validation Error 10 6.70±0.05 20 6.42±0.18 40 6.63±0.35 𝐿 Validation Error 3 6.52±0.17 4 6.42±0.18 5 7.21±0.28 We observe that the interaction between quadrotors is non-stationary and sensitive to relative velocity. In Fig. 4.5(d), the vehicle’s neighbor is moving, and the prediction becomes significantly different from Fig. 4.5(c), where the neighbor is just hovering. To further understand the importance of relative velocity, we retrain neural networks neglecting relative velocity and the mean squared validation error degrades by 18%, from 6.42 to 7.60. We can also observe that the interactions are not a simple superposition of different pairs. For instance, Fig. 4.5(g) is significantly more complex than a simple superposition of Fig. 4.5(a) plus three (b), i.e., ρsmall (ϕenv ) + ρsmall (ϕsmall )+ρsmall (ϕsmall )+ρsmall (ϕsmall ). The maximum gap between Fig. 4.5(g) and the superposition version is 11.4 g. Moreover, we find that the ground effect and the downwash effect from a neighboring multirotor interact in an intriguing way. For instance, in Fig. 4.5(b), the downwash effect is “mitigated” as the vehicle gets closer to the ground. Finally, we observe that the large quadrotors cause significantly higher interaction forces than the small ones (see Fig. 4.5(e)), which further emphasizes the importance of our heterogeneous modeling. Note that in training we only have data from 1-3 vehicles (see Table 4.2). Our approach can generalize well to a larger swarm system. In Fig. 4.5, predictions for a 4-vehicle team (as shown in Fig. 4.5(g,h)) are still reliable. Moreover, our models work well in real flight tests with 5 vehicles (see Fig. 4.9) and even 16 vehicles (see Fig. 4.1). Motion Planning with Aerodynamics Coupling We implement Algorithm 4.1 in Python using PyTorch 1.5 (Paszke et al., 2019) for automatic gradient computation, CVXPY 1.0 (Diamond and Boyd, 2016) for 71 convex optimization, and GUROBI 9.0 (Gurobi Optimization, 2020) as underlying solver. To simulate the tracking performance of the planned trajectories, we also implement a nonlinear controller, which uses the planned controls as feed-forward term. We compare trajectories that were planned with a learned model of 𝑓𝑎,𝑧 with trajectories without such a model (i.e., 𝑓𝑎,𝑧 = 0) using Algorithm 4.1 with identical parameters. At test time, we track the planned trajectories with our controller, and forward propagate the dynamics with our learned model of 𝑓𝑎,𝑧 . Z Y 0.0 0.5 1.0 1.5 2.0 time [s] 2.5 3.0 fa [g] Y fa [g] fa [g] Y 0 −10 Planning without NN (Tracking) Z Planning with NN (Tracking) Z Planning with NN (Stage 1) 0.0 0.5 1.0 1.5 2.0 time [s] 2.5 3.0 0.0 0.5 1.0 1.5 2.0 time [s] 2.5 3.0 Figure 4.6: Example motion planning result for a three-robot swapping task in 2D (blue and orange: small robots; green: large robot). Top row: 𝑦𝑧-state space plot, where the arrows indicate the velocities every second, and the circles show the robot collision boundary shape at the middle of the task. Bottom row: interaction force for each robot over time (dashed: desired limit per robot). Left: Samplingbased motion planning with neural network to compute trajectories where the large robots moves below the small robots. Middle: Refined trajectories using SCP (dashed) and tracked trajectories (solid). Right: Planned trajectories when ignoring interaction forces (dashed) and tracked trajectories (solid). In this case, a dangerous configuration is chosen where the large robot flies on top of the small robots, exceeding their disturbance limits of 5 g. We visualize an example in Fig. 4.6, where two small and one large robots are tasked with exchanging positions. We focus on the 2D case in the 𝑦𝑧-plane to create significant interaction forces between the robots. The first stage of Algorithm 4.1 uses sampling-based motion planning to identify the best homeomorphism class where the small multirotors fly on top of the large multirotor (the interaction forces would require more total energy the other way around). However, the robots do not reach their goal state exactly and motions are jerky (Fig. 4.6, left). The second stage uses SCP to refine the motion plan such that robots reach their goal and minimize the total control effort (Fig. 4.6, middle). The planned trajectory can be tracked without significant error and the interaction forces are very small for the two small quadrotors and within the chosen bound of 10 g for the large quadrotor. We compare 72 this solution to one where we do not consider the interaction forces between robots by setting 𝑓𝑎,𝑧 = 0 in Algorithm 4.1. The planned trajectories tend to be shorter (Fig. 4.6, right, dashed lines) in that case. However, when tracking those trajectories, significant tracking errors occur and the interaction forces are outside their chosen bounds of 5 g for the small multirotors. Max z error [m] SL SS LL SSL 0.15 0.10 0.05 0.00 NN BL NN BL NN BL NN BL max | fa | [g] 15 10 5 0 SNN SBL LNN LBL SNN SBL LNN LBL SNN SBL LNN LBL SNN SBL LNN LBL Figure 4.7: Motion planning results for different scenarios (e.g., SSL refers to two small robots and one large robot) comparing planning without neural network (BL) and planning with neural network (NN) over 5 trials. Top: Worst-case tracking error. Ignoring the interaction force can result in errors of over 10 cm. Bottom: Worstcase interaction force for small and large quadrotors. The baseline has significant violations of the interaction force bounds, e.g., the SL case might create interaction forces greater than 10 g for the small quadrotor. We empirically evaluated the effect of planning with and without considering interaction forces in several scenarios, see Fig. 4.7. We found that ignoring the interaction forces results in significant tracking errors in all cases (top row). While this tracking error could be reduced when using our interaction-aware control law, the interaction forces are in some cases significantly over their desired limit. For example, in the small/large, small/small/large, and large/large cases, the worst-case interaction forces were consistently nearly double the limit (red line, bottom row). In practice, such large disturbances can cause instabilities or even a total loss of control, justifying the use of an interaction-aware motion planner. Control Performance in Flight Tests We study the flight performance improvements on swapping tasks with varying number of quadrotors. For each case, robots are initially arranged in a circle when viewed from above but at different 𝑧-planes and are tasked with moving linearly to the opposite side of the circle in their plane. During the swap, all vehicles align vertically at one point in time with vertical distances of 0.2 m to 0.3 m between neighbors. The tasks are similar, but not identical to the randomized swapping tasks 73 Max z error [m] used in data collection because different parameters (locations, transition times) are used. S2S S2L L2S L2L SS2S* SL2S* LL2S* SS2L SL2L NN BL NN BL NN BL NN BL NN BL NN BL NN BL NN BL NN BL 0.4 0.2 0.0 Figure 4.8: Flight test results comparing our solution with learned interaction compensation (NN) with the baseline (BL) in different scenarios. For each case, robots are initially arranged in a circle when viewed from above but at different 𝑧planes and are tasked with moving linearly to the opposite side of the circle in their plane. For each swap, we compute the worst-case 𝑧-error of the lowest quadrotor and plot the data over six swaps. Our results are summarized in Fig. 4.8 for various combinations of two and three multirotors, where we use “XY2Z” to denote the swap task with robots of type X and Y at the top and a robot of type Z at the bottom. We compute a box plot with median (green line) and first/third quartile (box) of the maximum 𝑧-error (repeated over 6 swaps). In some cases, the downwash force was so large that we upgraded the motors of the small quadrotor to improve the best-case thrust-to-weight ratio to 2.6. Such modified quadrotors are indicated as “S*”. We also verified that the 𝑥and 𝑦-error distributions are similar across the different controllers and omit those numbers for brevity. Our controller improves the median 𝑧-error in all cases and in most cases this improvement is statistically significant. For example, in the “L2S” case, where a large multirotor is on top of a small multirotor for a short period of time, the median 𝑧-error is reduced from 17 cm to 7 cm. To estimate the limits of our learning generalization, we test our approach on larger teams. First, we consider a team of five robots, where two large robots move on a circle in the horizontal plane and three small robots move on circle in the vertical plane such that the two circles form intertwined rings. In this case, the 𝑓𝑎,𝑧 prediction is accurate and the maximum 𝑧-error can be reduced significantly using our neural network prediction, see Fig. 4.9 for an example. Second, we consider a team of 16 robots moving on three intertwined rings as shown in Fig. 4.1(b,c). Here, two large and four small robots move on an ellipsoid in the horizontal plane, and five robots move on circles in different vertical planes. In this case, robots can 74 Height [m] 1.0 NN Desired BL 0.8 0.6 0.4 fa [g] 0 −10 Computed NN prediction 2 4 6 Time [s] 8 10 Figure 4.9: Generalization to a team of five multirotors. Three small multirotor move in a vertical ring and two large multirotor move in a horizontal ring. The maximum 𝑧-error of a small multirotor in the vertical ring with powerful motors is reduced from 10 cm to 5 cm and 𝑓𝑎 is predicted accurately. have significantly more neighbors (up to 15) compared to the training data (up to 2), making the prediction of 𝑓𝑎,𝑧 relatively less accurate. However, the maximum 𝑧-error of a small multirotor in one of the vertical rings with powerful motors is still reduced from 15 cm to 10 cm. We note that a conceptually-simpler method is to estimate and compensate for 𝑓𝑎,𝑧 online without learning. However, online estimation will not only introduce significant delays, but also be very noisy especially in close-proximity flight. Our learning-based method has no delay (because it directly predicts 𝑓𝑎,𝑧 at the current time step), and considerably mitigates the noise due to the use of spectral normalization and delay-free filtering in the training process. In experiments, we observe that the online estimation and compensate method would quickly crash the drone. References Bentley, Jon Louis (1975). Multidimensional binary search trees used for associative searching. In: Communications of the ACM 18.9, pp. 509–517. doi: 10.1145/ 361002.361007. url: http://doi.acm.org/10.1145/361002.361007. Berg, Jur van den, Stephen J. Guy, Ming C. Lin, and Dinesh Manocha (2009). Reciprocal n-body collision avoidance. In: International Symposium on Robotics Research. Vol. 70, pp. 3–19. doi: 10.1007/978- 3- 642- 19457- 3\_1. url: https://doi.org/10.1007/978-3-642-19457-3%5C_1. Bitcraze (2015). Crazyflie 2.0 thrust investigation. url: https://wiki.bitcraze. io/misc:investigations:thrust. 75 Čáp, Michal, Peter Novák, Alexander Kleiner, and Martin Selecký (2015). Prioritized planning algorithms for trajectory coordination of multiple mobile robots. In: IEEE Transactions on Automation Science and Engineering 12.3, pp. 835– 849. doi: 10.1109/TASE.2015.2445780. url: https://doi.org/10.1109/ TASE.2015.2445780. Cheng, Richard, Abhinav Verma, Gábor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick (2019). Control regularization for reduced variance reinforcement learning. In: International Conference on Machine Learning, pp. 1141–1150. url: http://proceedings.mlr.press/v97/cheng19a.html. Chung, Soon-Jo, Aditya A. Paranjape, Philip M. Dames, Shaojie Shen, and Vijay Kumar (2018). A survey on aerial swarm robotics. In: IEEE Transactions on Robotics 34.4, pp. 837–855. doi: 10.1109/TRO.2018.2857475. url: https: //doi.org/10.1109/TRO.2018.2857475. Csáji, Balázs Csanád et al. (2001). Approximation with artificial neural networks. In: Faculty of Sciences, Eötvös Loránd University, Hungary 24.48, p. 7. Debord, Mark, Wolfgang Hönig, and Nora Ayanian (2018). Trajectory planning for heterogeneous robot teams. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 7924–7931. doi: 10.1109/IROS.2018.8593876. url: https://doi.org/10.1109/IROS.2018.8593876. Diamond, Steven and Stephen Boyd (2016). CVXPY: A Python-embedded modeling language for convex optimization. In: Journal of Machine Learning Research 17.83, pp. 1–5. url: http://jmlr.org/papers/v17/15-408.html. Dinh, Quoc Tran and Moritz Diehl (2010). Local convergence of sequential convex programming for nonconvex optimization. In: Recent Advances in Optimization and Its Applications in Engineering. Springer, pp. 93–102. doi: 10.1007/9783-642-12598-0_9. url: http://link.springer.com/10.1007/978-3642-12598-0_9. Du, Xintong, Carlos E. Luis, Marijan Vukosavljev, and Angela P. Schoellig (2019). Fast and in sync: Periodic swarm patterns for quadrotors. In: IEEE International Conference on Robotics and Automation, pp. 9143–9149. doi: 10.1109/ICRA. 2019.8794017. url: https://doi.org/10.1109/ICRA.2019.8794017. Faessler, Matthias, Davide Falanga, and Davide Scaramuzza (2017). Thrust mixing, saturation, and body-rate control for accurate aggressive quadrotor flight. In: IEEE Robotics and Automation Letters 2.2, pp. 476–482. doi: 10.1109/LRA. 2016.2640362. url: https://doi.org/10.1109/LRA.2016.2640362. Förster, Julian (2015). System identification of the Crazyflie 2.0 nano quadrocopter. MA thesis. Zurich: ETH Zurich. doi: 10.3929/ethz-b-000214143. Foust, Rebecca, Soon-Jo Chung, and Fred Y. Hadaegh (2020). Optimal guidance and control with nonlinear dynamics using sequential convex programming. In: Journal of Guidance, Control, and Dynamics 43.4, pp. 633–644. doi: 10.2514/ 1.G004590. url: https://doi.org/10.2514/1.G004590. 76 Gurobi Optimization, LLC (2020). Gurobi optimizer reference manual. url: http: //www.gurobi.com. Hauser, Kris and Yilun Zhou (2016). Asymptotically optimal planning by feasible kinodynamic planning in a state-cost space. In: IEEE Transactions on Robotics 32.6, pp. 1431–1443. doi: 10.1109/TRO.2016.2602363. url: https://doi. org/10.1109/TRO.2016.2602363. Hönig, Wolfgang, James A. Preiss, T. K. Satish Kumar, Gaurav S. Sukhatme, and Nora Ayanian (2018). Trajectory planning for quadrotor swarms. In: IEEE Transactions on Robotics 34.4, pp. 856–869. doi: 10.1109/TRO.2018.2853613. url: https://doi.org/10.1109/TRO.2018.2853613. Jain, Karan P., Trey Fortmuller, Jaeseung Byun, Simo A. Mäkiharju, and Mark W. Mueller (2019). Modeling of aerodynamic disturbances for proximity flight of multirotors. In: International Conference on Unmanned Aircraft Systems, pp. 1261– 1269. doi: 10.1109/ICUAS.2019.8798116. url: https://doi.org/10. 1109/ICUAS.2019.8798116. Johannink, Tobias et al. (2019). Residual reinforcement learning for robot control. In: IEEE International Conference on Robotics and Automation, pp. 6023–6029. doi: 10.1109/ICRA.2019.8794127. url: https://doi.org/10.1109/ ICRA.2019.8794127. Karaman, Sertac and Emilio Frazzoli (2011). Sampling-based algorithms for optimal motion planning. In: International Journal of Robotics Research 30.7, pp. 846– 894. doi: 10.1177/0278364911406761. url: https://doi.org/10.1177/ 0278364911406761. Khatib, Oussama (1985). Real-time obstacle avoidance for manipulators and mobile robots. In: IEEE International Conference on Robotics and Automation, pp. 500– 505. doi: 10.1109/ROBOT.1985.1087247. url: https://doi.org/10. 1109/ROBOT.1985.1087247. Kunz, Tobias and Mike Stilman (2015). Kinodynamic RRTs with fixed time step and best-input extension are not probabilistically complete. In: International Workshop on the Algorithmic Foundations of Robotics, pp. 233–244. doi: 10. 1007/978-3-319-16595-0_14. url: https://doi.org/10.1007/978-3319-16595-0_14. Le, Hoang Minh, Andrew Kang, Yisong Yue, and Peter Carr (2016). Smooth imitation learning for online sequence prediction. In: International Conference on Machine Learning. Vol. 48, pp. 680–688. url: http://proceedings.mlr. press/v48/le16.html. Luis, Carlos E. and Angela P. Schoellig (2019). Trajectory generation for multiagent point-to-point transitions via distributed model predictive control. In: IEEE Robotics and Automation Letters 4.2, pp. 375–382. doi: 10.1109/LRA.2018. 2890572. url: https://doi.org/10.1109/LRA.2018.2890572. 77 McKinnon, Christopher D. and Angela P. Schoellig (2019). Learn fast, forget slow: safe predictive learning control for systems with unknown and changing dynamics performing repetitive tasks. In: IEEE Robotics and Automation Letters 4.2, pp. 2180–2187. doi: 10.1109/LRA.2019.2901638. url: https://doi.org/ 10.1109/LRA.2019.2901638. Mellinger, Daniel, Aleksandr Kushleyev, and Vijay Kumar (2012). Mixed-integer quadratic program trajectory generation for heterogeneous quadrotor teams. In: IEEE International Conference on Robotics and Automation, pp. 477–483. doi: 10.1109/ICRA.2012.6225009. url: https://doi.org/10.1109/ICRA. 2012.6225009. Morgan, Daniel, Soon-Jo Chung, and Fred Y. Hadaegh (2014). Model predictive control of swarms of spacecraft using sequential convex programming. In: Journal of Guidance, Control, and Dynamics 37.6, pp. 1725–1740. url: https://doi. org/10.2514/1.G000218. Morgan, Daniel, Giri P. Subramanian, Soon-Jo Chung, and Fred Y. Hadaegh (2016). Swarm assignment and trajectory optimization using variable-swarm, distributed auction assignment and sequential convex programming. In: The International Journal of Robotics Research 35.10, pp. 1261–1285. Nakka, Yashwanth Kumar, Anqi Liu, Guanya Shi, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung (2021). Chance-constrained trajectory optimization for safe exploration and learning of nonlinear Systems. In: IEEE Robotics and Automation Letters 6.2, pp. 389–396. doi: 10.1109/LRA.2020.3044033. Paszke, Adam et al. (2019). PyTorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8024– 8035. url: http://papers.nips.cc/paper/9015-pytorch-an-imperativestyle-high-performance-deep-learning-library. Preiss, James A., Wolfgang Hönig, Gaurav S. Sukhatme, and Nora Ayanian (2017). Crazyswarm: A large nano-quadcopter swarm. In: IEEE International Conference on Robotics and Automation, pp. 3299–3304. doi: 10 . 1109 / ICRA . 2017 . 7989376. url: https://doi.org/10.1109/ICRA.2017.7989376. Riviére, Benjamin, Wolfgang Hönig, Yisong Yue, and Soon-Jo Chung (2020). GLAS: Global-to-local safe autonomy synthesis for multi-robot motion planning with end-to-end learning. In: IEEE Robotics and Automation Letters 5.3, pp. 4249–4256. doi: 10.1109/LRA.2020.2994035. url: https://doi.org/ 10.1109/LRA.2020.2994035. Saveriano, Matteo, Yuchao Yin, Pietro Falco, and Dongheui Lee (2017). Dataefficient control policy search using residual dynamics learning. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4709–4715. doi: 10.1109/IROS.2017.8206343. url: https://doi.org/10.1109/IROS. 2017.8206343. 78 Scarselli, Franco, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini (2008). The graph neural network model. In: IEEE Transactions on Neural Networks 20.1, pp. 61–80. Shi, Guanya, Wolfgang Hönig, Xichen Shi, Yisong Yue, and Soon-Jo Chung (2022). Neural-Swarm2: Planning and control of heterogeneous multirotor swarms using learned interactions. In: IEEE Transactions on Robotics 38.2, pp. 1063–1079. doi: 10.1109/TRO.2021.3098436. Shi, Guanya, Xichen Shi, Michael O’Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung (2019). Neural Lander: Stable drone landing control using learned dynamics. In: 2019 International Conference on Robotics and Automation (ICRA). IEEE, pp. 9784–9790. doi: 10.1109/ICRA.2019.8794351. Shi, Xichen, Michael O’Connell, and Soon-Jo Chung (2020). Numerical predictive control for delay compensation. In: arXiv preprint arXiv:2009.14450. Shukla, Dhwanil and Narayanan Komerath (2018). Multirotor drone aerodynamic interaction investigation. In: Drones 2.4. issn: 2504-446X. doi: 10 . 3390 / drones2040043. url: https://www.mdpi.com/2504-446X/2/4/43. Taylor, Andrew J., Victor D. Dorobantu, Hoang M. Le, Yisong Yue, and Aaron D. Ames (2019). Episodic learning with control Lyapunov functions for uncertain robotic systems. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 6878–6884. doi: 10.1109/IROS40897.2019.8967820. url: https://doi.org/10.1109/IROS40897.2019.8967820. Tolstaya, Ekaterina, Fernando Gama, James Paulos, George Pappas, Vijay Kumar, and Alejandro Ribeiro (2020). Learning decentralized controllers for robot swarms with graph neural networks. In: The Conference on Robot Learning (CoRL), pp. 671–682. Zaheer, Manzil, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R. Salakhutdinov, and Alexander J. Smola (2017). Deep sets. In: Advances in Neural Information Processing Systems, pp. 3391–3401. Zhang, Chuxu, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V. Chawla (2019). Heterogeneous graph neural network. In: ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 793–803. 79 Chapter 5 NEURAL-FLY Figure 5.1: Visualization of wind effect for a drone flying in Caltech Real Weather Wind Tunnel. In Chapter 2, we introduced a general mixed robot dynamics model 𝑀 (𝑞) 𝑞¥ + 𝐶 (𝑞, 𝑞) ¤ 𝑞¤ +𝑔(𝑞) = 𝐵𝑢 + 𝑓 where 𝑓 is the unknown dynamics. We also introduced the corresponding drone dynamics model in Eq. (2.4). In Chapter 3 and Chapter 4, we discussed the case 𝑓 = 𝑓 (𝑞, 𝑞, ¤ 𝑢) and 𝑓 = 𝑓 (𝑞, 𝑞, ¤ neighbors’ states), respectively. In this chapter, we will move to a more challenging time-variant setting, where 𝑓 = 𝑓 (𝑞, 𝑞, ¤ 𝑤(𝑡)) and 𝑤(𝑡) is an unknown environmental condition. For example, 𝑤 in Fig. 5.1 is the unknown wind condition. In this case, the main challenge is to fast adapt the controller in changing environments in an efficient and reliable manner. This chapter is mainly based on the following paper1 2: O’Connell, Michael, Guanya Shi, Xichen Shi, Kamyar Azizzadenesheli, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung (2022). Neural-Fly enables rapid learning for agile flight in strong winds. In: Science Robotics 7.66, eabm6597. doi: 10.1126/scirobotics.abm6597. 1 Summary video: https://youtu.be/TuF9teCZX0U 2 Data and code: https://github.com/aerorobotics/neural-fly 80 Abstract. Executing safe and precise flight maneuvers in dynamic high-speed winds is important for the ongoing commoditization of uninhabited aerial vehicles (UAVs). However, since the relationship between various wind conditions and its effect on aircraft maneuverability is not well understood, it is challenging to design effective robot controllers using traditional control design methods. We present NeuralFly, a learning-based approach that allows rapid online adaptation by incorporating pre-trained representations through deep learning. Neural-Fly builds on two key observations that aerodynamics in different wind conditions share a common representation and that the wind-specific part lies in a low-dimensional space. To that end, Neural-Fly uses a proposed learning algorithm, Domain Adversarially Invariant Meta-Learning (DAIML), to learn the shared representation, only using 12 minutes of flight data. With the learned representation as a basis, Neural-Fly then uses a composite adaptation law to update a set of linear coefficients for mixing the basis elements. When evaluated under challenging wind conditions generated with the Caltech Real Weather Wind Tunnel with wind speeds up to 43.6 km/h (12.1 m/s), Neural-Fly achieves precise flight control with substantially smaller tracking error than state-of-the-art nonlinear and adaptive controllers. In addition to strong empirical performance, the exponential stability of Neural-Fly results in robustness guarantees. Finally, our control design extrapolates to unseen wind conditions, is shown to be effective for outdoor flights with only on-board sensors, and can transfer across drones with minimal performance degradation. 5.1 Introduction The commoditization of uninhabited aerial vehicles (UAVs) requires that the control of these vehicles become more precise and agile. For example, drone delivery requires transporting goods to a narrow target area in various weather conditions. Unmodeled and often complex aerodynamics (e.g., 𝑓𝑎 and 𝜏𝑎 discussed in Chapter 2) are among the most notable challenges to precise flight control. Flying in windy environments (as shown in Fig. 5.1) introduces even more complexity because of the unsteady aerodynamic interactions between the drone, the induced airflow, and the wind. These unsteady and nonlinear aerodynamic effects substantially degrade the performance of conventional UAV control methods that neglect to account for them in the control design. Prior approaches partially capture these effects with simple linear or quadratic air drag models, which limit the tracking performance in 81 agile flight and cannot be extended to external wind conditions (Foehn et al., 2021; Faessler et al., 2018). Although more complex aerodynamic models can be derived from computational fluid dynamics (Ventura Diaz and Yoon, 2018), such modelling is often computationally expensive, and is limited to steady non-dynamic wind conditions. Adaptive control addresses this problem by estimating linear parametric uncertainty in the dynamical model in real time to improve tracking performance. Recent state-of-the-art in quadrotor flight control has used adaptive control methods that directly estimate the unknown aerodynamic force without assuming the structure of the underlying physics, but relying on high-frequency and low-latency control (Tal and Karaman, 2021; Mallikarjunan et al., 2012; Pravitra et al., 2020; Hanover et al., 2021). In parallel, there has been increased interest in data-driven modeling of aerodynamics (e.g., Torrente et al. (2021) and Chapters 3 and 4), however existing approaches cannot effectively adapt in changing or unknown environments such as time-varying wind conditions. In this chapter, we present a data-driven approach called Neural-Fly, which is a deep-learning-based tracking controller that learns to quickly adapt to rapidlychanging wind conditions. Our method, depicted in Fig. 5.3, advances and offers insights into both adaptive flight control and deep-learning-based robot control. Our experimental demonstrates (e.g., Fig. 5.2) that Neural-Fly achieves centimeterlevel position-error tracking of an agile and challenging trajectory in dynamic wind conditions on a standard UAV. Our method has two main components: an offline learning phase and an online adaptive control phase used as real-time online learning. For the offline learning phase, we have developed Domain Adversarially Invariant Meta-Learning (DAIML) that learns a wind-condition-independent deep neural network (DNN) representation of the aerodynamics in a data-efficient manner. The output of the DNN is treated as a set of basis functions that represent the aerodynamic effects. This representation is adapted to different wind conditions by updating a set of linear coefficients that mix the output of the DNN. DAIML is data efficient and uses only 12 total minutes of flight data in just 6 different wind conditions to train the DNN. DAIML incorporates several key features which not only improve the data efficiency but also are informed by the downstream online adaptive control phase. In particular, DAIML uses spectral normalization (see Chapter 2) to control the Lipschitz property of the DNN to improve generalization to unseen data and provide closed-loop stability and robustness guarantees. DAIML also uses a discriminative network, which 82 Figure 5.2: Agile flight through narrow gates. (A) Caltech Real Weather Wind Tunnel system, the quadrotor UAV, and the gate. In our flight tests, the UAV follows an agile trajectory through narrow gates, which are slightly wider than the UAV itself, under challenging wind conditions. (B-C) Trajectories used for the gate tests. In (B), the UAV follows a figure-8 through one gate, with wind speed 3.1 m/s or time-varying wind condition. In (C), the UAV follows an ellipse in the horizontal plane through two gates, with wind speed 3.1 m/s. (D-E) Long-exposure photos (with an exposure time of 5 s) showing one lap in two tasks. (F-I) High-speed photos (with a shutter speed of 1/200s) showing the moment the UAV passed through the gate and the interaction between the UAV and the wind. ensures that the learned representation is wind-invariant and that the wind-dependent information is only contained in the linear coefficients that are adapted in the online control phase. For the online adaptive control phase, we have developed a regularized composite adaptive control law, which we derived from a fundamental understanding of how the learned representation interacts with the closed-loop control system and which we support with rigorous theory. The adaptation law updates the wind-dependent linear coefficients using a composite of the position tracking error term and the 83 aerodynamic force prediction error term. Such a principled approach effectively guarantees stable and fast adaptation to any wind condition and robustness against imperfect learning. Although this adaptive control law could be used with a number of learned models, the speed of adaptation is further aided by the concise representation learned from DAIML. Using Neural-Fly, we report an average improvement of 66 % over a nonlinear tracking controller, 42 % over an L1 adaptive controller, and 35 % over an Incremental Nonlinear Dynamics Inversion (INDI) controller. These results are all accomplished using standard quadrotor UAV hardware, while running the PX4’s default regulation attitude control. Our tracking performance is competitive even compared to related work without external wind disturbances and with more complex hardware (for example, Tal and Karaman (2021) requires a 10-time higher control frequency and onboard optical sensors for direct motor speed feedback). We also compare Neural-Fly with two variants of our method: Neural-Fly-Transfer, which uses a learned representation trained on data from a different drone, and NeuralFly-Constant, which only uses our adaptive control law with a trivial non-learning basis. Neural-Fly-Transfer demonstrates that our method is robust to changes in vehicle configuration and model mismatch. Neural-Fly-Constant, L1 , and INDI all directly adapt to the unknown dynamics without assuming the structure of the underlying physics, and they have similar performance. Furthermore, we demonstrate that our method enables a new set of capabilities that allow the UAV to fly through low-clearance gates following agile trajectories in gusty wind conditions (Fig. 5.2). 5.2 Related Work Precise Quadrotor Control Prior work on agile quadrotor control has achieved impressive results by considering aerodynamics (Tal and Karaman, 2021; Hanover et al., 2021; Torrente et al., 2021; Faessler et al., 2018). However, those approaches require specialized onboard hardware (Tal and Karaman, 2021), full custom flight control stacks (Tal and Karaman, 2021; Hanover et al., 2021), or cannot adapt to external wind disturbances (Torrente et al., 2021; Faessler et al., 2018). For example, state-of-the-art tracking performance has been demonstrated using incremental nonlinear dynamics inversion to estimate aerodynamic disturbance forces, with a root-mean-square tracking error of 6.6 cm and drone ground speeds up to 12.9 m/s (Tal and Karaman, 2021). However, Tal and Karaman (2021) relies on high-frequency control updates (500 Hz) and direct motor speed feedback using optical encoders to rapidly estimate external 84 A Online adaptation B Offline meta-learning Adaptation Training set set Prediction-based Tracking-based adaptation adaptation Wind-specific linear coefficients Adaptive Velocity control Dataset from K wind conditions Attitude Learned basis function Residual force prediction net PWM Domain adversarially invariant meta-learning (DAIML) net Wind-invariant representation Drone state Least squares SGD C Control diagram Online adaptation block Learned basis function Prediction-based adaptation Adaptive control Residual force error Nominal dynamics Vehicle plant Tracking-based adaptation Desired trajectory + Tracking error Feedforward + Model kinematics Gravity Feedback Drone state Flight control unit Figure 5.3: Offline meta-learning and online adaptive control design. (A) The online adaptation block in our adaptive controller. Our controller leverages the metatrained basis function 𝜙, which is a wind-invariant representation of the aerodynamic effects, and uses composite adaptation (that is, including tracking-error-based and prediction-error-based adaptation) to update wind-specific linear weights 𝑎. ˆ The output of this block is the wind-effect force estimate, 𝑓ˆ = 𝜙 𝑎. ˆ (B) The illustration of our meta-learning algorithm DAIML. We collected data from wind conditions {𝑤 1 , · · · , 𝑤 𝐾 } and applied Algorithm 5.1 to train the 𝜙 net. (C) The diagram of our control method, where the grey part corresponds to (A). Interpreting the learned block as an aerodynamic force allows it to be incorporated into the feedback control easily. disturbances. Both are challenging to deploy on standard systems. Hanover et al. (2021) simplifies the hardware setup and does not require optical motor speed sensors and has demonstrated state-of-the-art tracking performance. However, Hanover et al. (2021) relies on a high-rate L1 adaptive controller inside a model predictive controller and uses a racing drone with a fully customized control stack. Torrente et al. (2021) leverages an aerodynamic model learned offline and represented as Gaussian Processes. However, Torrente et al. (2021) cannot adapt to unknown 85 or changing wind conditions and provides no theoretical guarantees. Another recent work focuses on deriving simplified rotor-drag models that are differentially flat (Faessler et al., 2018). However, Faessler et al. (2018) focuses on horizontal, 𝑥𝑦−plane trajectories at ground speeds of 4 m/s without external wind, where the thrust is more constant than ours, achieves ∼6 cm tracking error, uses an attitude controller running at 4000 Hz, and is not extensible to faster flights as pointed out by Torrente et al. (2021). Adaptive Control Adaptive control theory has been extensively studied for online control and identification problems with parametric uncertainty, for example, unknown linear coefficients for mixing known basis functions (Slotine and Li, 1991; Ioannou and Sun, 1996; Krstic et al., 1995; Narendra and Annaswamy, 2012; Farrell and Polycarpou, 2006; Wise et al., 2006). There are three common aspects of adaptive control which must be addressed carefully in any well-designed system and which we address in Neural-Fly: designing suitable basis functions for online adaptation, stability of the closed-loop system, and persistence of excitation, which is a property related to robustness against disturbances. These challenges arise due to the coupling between the unknown underlying dynamics and the online adaptation. This coupling precludes naive combinations of online learning and control. For example, gradient-based parameter adaptation has well-known stability and robustness issues as discussed in Slotine and Li (1991). The basis functions play a crucial role in the performance of adaptive control, but designing or selecting proper basis functions might be challenging. A good set of basis functions should reflect important features of the underlying physics. In practice, basis functions are often designed using physics-informed modeling of the system, such as the nonlinear aerodynamic modeling in X. Shi, Spieler, et al. (2020). However, physics-informed modeling requires a tremendous amount of prior knowledge and human labor, and is often still inaccurate. Another approach is to use random features as the basis set, such as random Fourier features (Rahimi and Recht, 2007; Lale et al., 2021), which can model all possible underlying physics as long as the number of features is large enough. However, the high-dimensional feature space is not optimal for a specific system because many of the features might be redundant or irrelevant. Such suboptimality and redundancy not only increase the computational burden but also slow down the convergence speed of the adaptation process. 86 Given a set of basis functions, naive adaptive control designs may cause instability and fragility in the closed-loop system, due to the nontrivial coupling between the adapted model and the system dynamics. In particular, asymptotically stable adaptive control cannot guarantee robustness against disturbances and so exponential stability is desired. Even so, often, existing adaptive control methods only guarantee exponential stability when the desired trajectory is persistently exciting, by which information about all of the coefficients (including irrelevant ones) is constantly provided at the required spatial and time scales. In practice, persistent excitation requires either a succinct set of basis functions or perturbing the desired trajectory, which compromises tracking performance. Recent multirotor flight control methods, including INDI (Tal and Karaman, 2021) and L1 adaptive control, presented in (Mallikarjunan et al., 2012) and demonstrated inside a model predictive control loop in (Hanover et al., 2021), achieve good results by abandoning complex basis functions. Instead, these methods directly estimate the aerodynamic residual force vector. The residual force is observable, thus, these methods bypass the challenge of designing good basis functions and the associated stability and persistent excitation issues. However, these methods suffer from lag in estimating the residual force and encounter the the filter design performance trade of reduced lag versus amplified noise. Note that Neural-Fly-Constant only uses Neural-Fly’s composite adaptation law to estimate the residual force, and therefore, Neural-Fly-Constant also falls into this class of adaptive control structures. This chapter demonstrates that the inherent estimation lag in these existing methods limits performance on agile trajectories and in strong wind conditions. Neural-Fly solves the aforementioned issues of basis function design and adaptive control stability, using newly developed methods for meta-learning and composite adaptation that can be seamlessly integrated together. Neural-Fly uses DAIML and flight data to learn an effective and compact set of basis functions, represented as a DNN. The regularized composite adaptation law uses the learned basis functions to quickly respond to wind conditions. Neural-Fly enjoys fast adaptation because of the conciseness of the feature space, and it guarantees closed-loop exponential stability and robustness without assuming persistent excitation. Related to Neural-Fly, neural network based adaptive control has been researched extensively, but by and large was limited to shallow or single-layer neural networks without pretraining. Some early works focus on shallow or single-layer neural networks with unknown parameters which are adapted online (Farrell and Polycarpou, 87 2006; Nakanishi et al., 2002; Chen and Khalil, 1995; Johnson and Calise, 2003; Narendra and Mukhopadhyay, 1997). A recent work applies this idea to perform an impressive quadrotor flip (Bisheban and T. Lee, 2021). However, the existing neural network based adaptive control work does not employ multi-layer DNNs, and lacks a principled and efficient mechanism to pretrain the neural network before deployment. Instead of using shallow neural networks, recent trends in machine learning highly rely on DNNs due to their representation power (LeCun et al., 2015). In this chapter, we leverage modern deep learning advances to pretrain a DNN which represents the underlying physics compactly and effectively. Multi-Environment Deep Learning for Robot Control Recently, researchers have been addressing the data and computation requirements for DNNs to help the field progress towards the fast online-learning paradigm. In turn, this progress has been enabling adaptable DNN-based control in dynamic environments. The most popular learning scheme in dynamic environments is metalearning, or “learning-to-learn,” which aims to learn an efficient model from data across different tasks or environments (Finn et al., 2017; Hospedales et al., 2021). The learned model, typically represented as a DNN, ideally should be capable of rapid adaptation to a new task or an unseen environment given limited data. For robotic applications, meta-learning has shown great potential for enabling autonomy in highly-dynamic environments. For example, it has enabled quick adaptation against unseen terrain or slopes for legged robots (Nagabandi et al., 2018; Song et al., 2020), changing suspended payload for drones (Belkhale et al., 2021), and unknown operating conditions for wheeled robots (McKinnon and Schoellig, 2021). In general, learning algorithms typically can be decomposed into two phases: offline learning and online adaptation. In the offline learning phase, the goal is to learn a model from data collected in different environments, such that the model contains shared knowledge or features across all environment, for example, learning aerodynamic features shared by all wind conditions. In the online adaptation phase, the goal is to adapt the offline-learned model, given limited online data from a new environment or a new task, for example, fine tuning the aerodynamic features in a specific wind condition. There are two ways that the offline-learned model can be adapted. In the first class, the adaptation phase adapts the whole neural network model, typically using one or more gradient descent steps (Finn et al., 2017; Nagabandi et al., 2018; Belkhale 88 et al., 2021; Clavera et al., 2018). However, due to the notoriously data-hungry and high-dimensional nature of neural networks, for real-world robots it is still impossible to run such adaptation on-board as fast as the feedback control loop (e.g., ∼100Hz for quadrotor). Furthermore, adapting the whole neural network often lacks explainability and robustness and could generate unpredictable outputs that make the closed-loop unstable. In the second class (including Neural-Fly), the online adaptation only adapts a relatively small part of the learned model, for example, the last layer of the neural network (O’Connell et al., 2021; McKinnon and Schoellig, 2021; Richards et al., 2021; Peng et al., 2021). The intuition is that, different environments share a common representation (e.g., the wind-invariant representation in Fig. 5.3(A)), and the environment-specific part is in a low-dimensional space (e.g., the wind-specific linear weight in Fig. 5.3(A)), which enables the real-time adaptation as fast as the control loop. In particular, the idea of integrating meta-learning with adaptive control is first presented in O’Connell et al. (2021), later followed by Richards et al. (2021). However, the representation learned in O’Connell et al. (2021) is ineffective and the tracking performance in O’Connell et al. (2021) is similar as the baselines; Richards et al. (2021) focuses on a planar and fully-actuated rotorcraft simulation without experiment validation and there is no stability or robustness analysis. Neural-Fly instead learns an effective representation using a new meta-learning algorithm called DAIML, demonstrates state-of-the-art tracking performance on real drones, and achieves non-trivial stability and robustness guarantees. Another popular deep-learning approach for control in dynamic environments is robust policy learning via domain randomization (J. Lee et al., 2020; Tobin et al., 2017; Ramos et al., 2019). The key idea is to train the policy with random physical parameters such that the controller is robust to a range of conditions. For example, the quadrupedal locomotion controller in J. Lee et al. (2020) retains its robustness over challenging natural terrains. However, robust policy learning optimizes average performance under a broad range of conditions rather than achieving precise control by adapting to specific environments. 5.3 Problem Statement In this chapter, we consider the general robot dynamics model given in Eq. (2.1) in Chapter 2: 𝑀 (𝑞) 𝑞¥ + 𝐶 (𝑞, 𝑞) ¤ 𝑞¤ + 𝑔(𝑞) = 𝑢 + 𝑓 (𝑞, 𝑞, ¤ 𝑤) . (5.1) | {z } unknown 89 Most importantly, 𝑓 (𝑞, 𝑞, ¤ 𝑤) incorporates unmodeled dynamics, and 𝑤 ∈ R𝑚 represents the underlying environmental conditions, which is potentially time-variant. Specifically, in this chapter, 𝑤 represents the wind profile. Neural-Fly can be broken into two main stages, the offline meta-learning stage and the online adaptive control stage. These two stages build a model of the unknown dynamics of the form 𝑓 (𝑞, 𝑞, ¤ 𝑤) ≈ 𝜙(𝑞, 𝑞)𝑎(𝑤), ¤ (5.2) where 𝜙 is a basis or representation function shared by all wind conditions and captures the dependence of the unmodeled dynamics on the robot state, and 𝑎 is a set of linear coefficients that is updated for each condition. In the Appendix (Section 5.7), we prove that the decomposition 𝜙(𝑞, 𝑞)𝑎(𝑤) ¤ exists for any analytic function 𝑓 (𝑞, 𝑞, ¤ 𝑤). In the offline meta-learning stage, we learn 𝜙 as a DNN using our meta-learning algorithm DAIML. This stage results in learning 𝜙 as a wind-invariant representation of the unmodeled dynamics, which generalizes to new trajectories and new wind conditions. In the online adaptive control stage, we adapt the linear coefficients 𝑎 using adaptive control. Our adaptive control algorithm is a type of composite adaptation and was carefully designed to allow for fast adaptation while maintaining the global exponential stability and robustness of the closed loop system. The offline learning and online control architectures are illustrated in Fig. 5.3(B) and Fig. 5.3(A,C), respectively. 5.4 Offline Meta-Learning In this section, we will present the methodology and details of learning the representation function 𝜙. In particular, we will first introduce the goal of meta-learning, motivate the proposed algorithm DAIML by the observed domain shift problem from the collected dataset, and finally discuss key algorithmic details. For notional simplicity, we define 𝑥 = [𝑞; 𝑞]. ¤ The offline meta-learning phase optimizes an efficient representation 𝜙 given a multi-environment dataset D = {𝐷 𝑤 1 , · · · , 𝐷 𝑤 𝐾 }, where n o 𝑁𝑘 𝐷 𝑤 𝑘 = 𝑥 𝑘(𝑖) , 𝑦 𝑘(𝑖) = 𝑓 (𝑥 𝑘(𝑖) , 𝑤 𝑘 ) + 𝜖 𝑘(𝑖) 𝑖=1 is the collection of 𝑁 𝑘 noisy input-output pairs with wind condition 𝑤 𝑘 . (5.3) 90 Meta-Learning Goal Given the dataset, the goal of meta-learning is to learn a representation 𝜙(𝑥), such that for any wind condition 𝑤, there exists a latent variable 𝑎(𝑤) which allows 𝜙(𝑥)𝑎(𝑤) to approximate 𝑓 (𝑥, 𝑤) well. Formally, an optimal representation, 𝜙, solves the following optimization problem: min 𝜙,𝑎 1 ,··· ,𝑎 𝐾 𝑁𝑘 𝐾 ∑︁ ∑︁ 𝑦 𝑘(𝑖) − 𝜙(𝑥 𝑘(𝑖) )𝑎 𝑘 2 , (5.4) 𝑘=1 𝑖=1 where 𝜙(·) : R2𝑛 → R𝑛×ℎ is the representation function and 𝑎 𝑘 ∈ Rℎ is the latent linear coefficient. Note that the optimal weight 𝑎 𝑘 is specific to each wind condition, but the optimal representation 𝜙 is shared by all wind conditions. We use a deep neural network (DNN) to represent 𝜙. In the Appendix (Section 5.7), we prove that for any analytic function 𝑓 (𝑥, 𝑤), the structure 𝜙(𝑥)𝑎(𝑤) can approximate 𝑓 (𝑥, 𝑤) with an arbitrary precision, as long as the DNN 𝜙 has enough neurons. This result implies that the 𝜙 solved from the optimization in Eq. (5.4) is a reasonable representation of the unknown dynamics 𝑓 (𝑥, 𝑤). Domain Shift Problems One challenge of the optimization in Eq. (5.4) is the inherent domain shift in 𝑥 caused by the shift in 𝑤. Recall that during data collection we have a program flying the drone in different winds. The actual flight trajectories differ vastly from wind to wind because of the wind effect. Formally, the distribution of 𝑥 𝑘(𝑖) varies between 𝑘 because the underlying environment or context 𝑤 has changed. For example, as depicted by Fig. 5.4(C), the drone pitches into the wind, and the average degree of pitch depends on the wind condition. Note that pitch is only one component of the state 𝑥. The domain shift in the whole state 𝑥 is even more drastic. Such inherent shifts in 𝑥 bring challenges for deep learning. The DNN 𝜙 may memorize the distributions of 𝑥 in different wind conditions, such that the variation in the dynamics { 𝑓 (𝑥, 𝑤 1 ), 𝑓 (𝑥, 𝑤 2 ), · · · , 𝑓 (𝑥, 𝑤 𝐾 )} is reflected via the distribution of 𝑥, rather than the wind condition {𝑤 1 , 𝑤 2 , · · · , 𝑤 𝐾 }. In other words, the optimization in Eq. (5.4) may lead to over-fitting and may not properly find a wind-invariant representation 𝜙. To solve the domain shift problem, inspired by Ganin et al. (2016), we propose the following adversarial optimization framework: 𝑁𝑘  𝐾 ∑︁   ∑︁ 2 (𝑖) (𝑖) (𝑖) (5.5) max min 𝑦 𝑘 − 𝜙(𝑥 𝑘 )𝑎 𝑘 − 𝛼 · loss ℎ(𝜙(𝑥 𝑘 )), 𝑘 , ℎ 𝜙,𝑎 1 ,··· ,𝑎 𝐾 𝑘=1 𝑖=1 91 Figure 5.4: Training data collection. (A) The xyz position along a two-minute randomized trajectory for data collection with wind speed 8.3 km/h (3.7 m/s), in the Caltech Real Weather Wind Tunnel. (B) A typical 10-second trajectory of the inputs (velocity, attitude quaternion, and motor speed PWM command) and label (offline calculation of aerodynamic residual force) for our learning model, corresponding to the highlighted part in (A). (C) Histograms showing data distributions in different wind conditions. (C) Left: distributions of the 𝑥-component of the wind-effect force, 𝑓𝑥 . This shows that the aerodynamic effect changes as the wind varies. (C) Right: distributions of the pitch, a component of the state used as an input to the learning model. This shows that the shift in wind conditions causes a distribution shift in the input. where ℎ is another DNN that works as a discriminator to predict the environment index out of 𝐾 wind conditions, loss(·) is a classification loss function (e.g., the cross entropy loss), 𝛼 ≥ 0 is a hyperparameter to control the degree of regularization, 𝑘 is the wind condition index, and (𝑖) is the input-output pair index. Intuitively, ℎ and 𝜙 play a zero-sum max-min game: the goal of ℎ is to predict the index 𝑘 directly from 𝜙(𝑥) (achieved by the outer max); the goal of 𝜙 is to approximate the label 𝑦 𝑘(𝑖) while making the job of ℎ harder (achieved by the inner min). In other words, ℎ is a learned regularizer to remove the environment information contained in 𝜙. In our experiments, the output of ℎ is a 𝐾−dimensional vector for the classification 92 probabilities of 𝐾 conditions, and we use the cross entropy loss for loss(·), which is given as loss  ℎ(𝜙(𝑥 𝑘(𝑖) )), 𝑘  =− 𝐾 ∑︁ 𝛿 𝑘 𝑗 log  ℎ(𝜙(𝑥 𝑘(𝑖) )) ⊤ 𝑒 𝑗  (5.6) 𝑗=1 where 𝛿 𝑘 𝑗 = 1 if 𝑘 = 𝑗 and 𝛿 𝑘 𝑗 = 0 otherwise and 𝑒 𝑗 is the standard basis function. Design of the DAIML Algorithm Finally, we solve the optimization problem in Eq. (5.5) by the proposed algorithm DAIML (described in Algorithm 5.1 and illustrated in Fig. 5.3(B)), which belongs to the category of gradient-based meta-learning (Hospedales et al., 2021), but with least squares as the adaptation step. DAIML contains three steps: (i) The adaptation step (Line 4-6) solves an least squares problem as a function of 𝜙 on the adaptation set 𝐵𝑎 . (ii) The training step (Line 7) updates the learned representation 𝜙 on the training set 𝐵, based on the optimal linear coefficient 𝑎 ∗ solved from the adaptation step. (iii) The regularization step (Line 8-9) updates the discriminator ℎ on the training set. We emphasize important features of DAIML: (i) After the adaptation step, 𝑎 ∗ is a function of 𝜙. In other words, in the training step (Line 7), the gradient with respect to the parameters in the neural network 𝜙 will backpropagate through 𝑎 ∗ . Note that the least-square problem (Line 4) can be solved efficiently with a closed-form solution. Namely, DAIML is a Hessian-free meta-learning algorithm because of the closed-form inner loop. (ii) The normalization (Line 6) is to make sure ∥𝑎 ∗ ∥ ≤ 𝛾, which improves the robustness of our adaptive control design. We also use spectral normalization in training 𝜙, to control the Lipschitz property of the neural network and improve generalizability (G. Shi, X. Shi, et al., 2019; G. Shi, Hönig, et al., 2022; Bartlett et al., 2017). (iii) We train ℎ and 𝜙 in an alternating manner. In each iteration, we first update 𝜙 (Line 7) while fixing ℎ and then update ℎ (Line 9) while fixing 𝜙. However, the probability to update the discriminator ℎ in each iteration is 𝜂 ≤ 1 instead of 1, to improve the convergence of the algorithm (Goodfellow et al., 2014). We further motivate the algorithm design using Fig. 5.4 and Fig. 5.5. Figure 5.4(A,B) shows the input and label from one wind condition, and Fig. 5.4(C) shows the distributions of the pitch component in input and the 𝑥−component in label, in different wind conditions. The distribution shift in label implies the importance of meta-learning and adaptive control, because the aerodynamic effect changes 93 Algorithm 5.1: Domain Adversarially Invariant Meta-Learning (DAIML) Hyperparameter: 𝛼 ≥ 0, 0 < 𝜂 ≤ 1, 𝛾 > 0 Input: D = {𝐷 𝑤 1 , · · · , 𝐷 𝑤 𝐾 } Initialize: Neural networks 𝜙 and ℎ Result: Trained neural networks 𝜙 and ℎ 1 repeat 2 Randomly sample 𝐷 𝑤 𝑘 from D 3 Randomly sample two disjoint batches 𝐵𝑎 (adaptation set) and 𝐵 (training set) from 𝐷 𝑤 𝑘 2 Í 4 Solve the least squares problem 𝑎 ∗ (𝜙) = arg min𝑎 𝑖∈𝐵 𝑎 𝑦 𝑘(𝑖) − 𝜙(𝑥 𝑘(𝑖) )𝑎 5 6 7 if ∥𝑎 ∗ ∥ > 𝛾 then ∗ 𝑎 ∗ ← 𝛾 · ∥𝑎𝑎∗ ∥ ⊲ normalization Train DNN 𝜙 using stochastic gradient descent (SGD) and spectral normalization with loss   ∑︁  2 (𝑖) (𝑖) ∗ (𝑖) 𝑦 𝑘 − 𝜙(𝑥 𝑘 )𝑎 − 𝛼 · loss ℎ(𝜙(𝑥 𝑘 )), 𝑘 𝑖∈𝐵 8 9 10 if rand() ≤ 𝜂 then Train DNN ℎ using SGD with loss Í 𝑖∈𝐵 loss  ℎ(𝜙(𝑥 𝑘(𝑖) )), 𝑘  until convergence drastically as the wind condition switches. On the other hand, the distribution shift in input motivates the need of DAIML. Figure 5.5 depicts the evolution of the optimal linear coefficient (𝑎 ∗ ) solved from the adaptation step in DAIML, via the t-distributed stochastic neighbor embedding (t-SNE) dimension reduction, which projects the 12-dimensional vector 𝑎 ∗ into 2-d. The distribution of 𝑎 ∗ is more and more clustered as the number of training epochs increases. In addition, the clustering behavior in Fig. 5.5 has a concrete physical meaning: right top part of the t-SNE plot corresponds to a higher wind speed. These properties imply the learned representation 𝜙 is indeed shared by all wind conditions, and the linear weight 𝑎 contains the wind-specific information. Finally, note that 𝜙 with 0 training epoch reflects random features, which cannot decouple different wind conditions as cleanly as the trained representation 𝜙. Similarly, as shown in Fig. 5.5, if we ignore the adversarial regularization term (by setting 𝛼 = 0), different 𝑎 ∗ vectors in different conditions are less disentangled, which indicates that the learned representation might be less robust and explainable. 94 t-SNE plots of the linear coefficients (a * ) in the training process 0.0 m/s 1.3 m/s 2.5 m/s 3.7 m/s 4.9 m/s 6.1 m/s Training epoch 0 Training epoch 30 Training epoch 90 Without adversarial loss ( = 0) Training epoch 90 Figure 5.5: t-SNE plots showing the evolution of the linear weights (𝑎 ∗ ) during the training process. As the number of training epochs increases, the distribution of 𝑎 ∗ becomes more clustered with similar wind speed clusters near each other. The clustering also has a physical meaning: after training convergence, the right top part corresponds to a higher wind speed. This suggests that DAIML successfully learned a basis function 𝜙 shared by all wind conditions, and the wind-dependent information is contained in the linear weights. Compared to the case without the adversarial regularization term (using 𝛼 = 0 in Algorithm 5.1), the learned result using our algorithm is also more explainable, in the sense that the linear coefficients in different conditions are more disentangled. 5.5 Online Adaptive Control and Stability Analysis During the offline meta-training process, a least-squares fit is used to find a set of parameters 𝑎 that minimizes the force prediction error for each data batch. The leastsquares fit plays a crucial role in the offline training because it yields a Hessian-free and efficient meta-learning algorithm. However, during the online control phase, we are ultimately interested in minimizing the position tracking error. Thus, in this section, we propose a more sophisticated adaptation law for the linear coefficients based upon a Kalman-filter estimator. This formulation results in automatic gain tuning for the update law, which allows the controller to quickly estimate parameters with large uncertainty. We further boost this estimator into a composite adaptation law, that is the parameter update depends both on the prediction error in the dynamics model as well as on the tracking error, as illustrated in Fig. 5.3. This allows the system to quickly identify and adapt to new wind conditions without requiring persistent excitation. In turn, this enables online adaptation of the high dimensional learned models from DAIML. 95 Robust Composite Adaptive Controller Design Our online adaptive control algorithm can be summarized by the following control law, adaptation law, and covariance update equations, respectively. 𝑢 = 𝑀 (𝑞) 𝑞¥𝑟 + 𝐶 (𝑞, 𝑞) ¤ 𝑞¤𝑟 + 𝑔(𝑞) | {z } nominal nonlinear feedforward 𝑎¤̂ = −𝜆 𝑎ˆ |{z} ℓ2 regularization −𝐾 𝑠 |{z} (5.7) linear feedback learning-based feedforward −𝑃𝜙⊤ 𝑅 −1 (𝜙 𝑎ˆ − 𝑦) | {z } prediction error term 𝑃¤ = −2𝜆𝑃 + 𝑄 − 𝑃𝜙⊤ 𝑅 −1 𝜙𝑃. −𝜙(𝑞, 𝑞) ¤ 𝑎ˆ | {z } +𝑃𝜙⊤ 𝑠 | {z } (5.8) tracking error term (5.9) Note that the controller in Eq. (5.7) can be viewed as an augmentation of Eq. (2.3) ˆ 𝑎¤̂ as the online linear-parameter update, and 𝑃 as a in Chapter 2 with 𝑓ˆ = 𝜙 𝑎, covariance-like matrix used for automatic gain tuning. Recall that the composite error term 𝑠 = 𝑞¤̃ + Λ𝑞˜ (see Eq. (2.2)), so the tracking error term in Eq. (5.8) updates 𝑎ˆ based on the tracking error. The prediction error term in Eq. (5.8) instead updates 𝑎ˆ based on the mismatch between model prediction 𝜙 𝑎ˆ and 𝑦, which is the measured aerodynamic residual force. In this controller, 𝐾, Λ, 𝑅, 𝑄, and 𝜆 are all tunable positive definite gain matrices. The structure of this control law is illustrated in Fig. 5.3. Figure 5.3 also shows further quadrotor specific details for the implementation of our method. These blocks are discussed further in the “Implementation of Our Control Methods and Baselines” section. Some readers may note that the regularization term, prediction error term, and covariance update, when taken alone, are in the form of a Kalman-Bucy filter. This Kalman-Bucy filter can be derived as the optimal estimator that minimizes the variance of the parameter error (Kalman and Bucy, 1961). The Kalman-Bucy filter perspective provides intuition for tuning the adaptive controller: the damping gain 𝜆 corresponds to how quickly the environment returns to the nominal conditions, 𝑄 corresponds to how quickly the environment changes, and 𝑅 corresponds to the combined representation error 𝑑 and measurement noise for 𝑦. However, naively combining this parameter estimator with the controller can lead to instabilities in the closed-loop system behavior unless extra care is taken in constraining the learned model and tuning the gains. Thus, we have designed our adaptation law to include a tracking error term, making Eq. (5.8) a composite adaptation law, guaranteeing stability of the closed-loop system (see Theorem 5.1), and in turn simplifying the gain tuning process. The regularization term allows the stability result to be independent of the persistent excitation of the learned model 𝜙, which is particularly 96 relevant when using high-dimensional learned representations. The adaptation gain and covariance matrix, 𝑃, acts as automatic gain tuning for the adaptive controller, which allows the controller to quickly adapt to when a new mode in the learned model is excited. In terms of theoretical guarantees, the control law and adaptation law have been designed so that the closed-loop behavior of the system is robust to imperfect learning and time-varying wind conditions. Specifically, we formally define 𝑑 (𝑎, 𝑡) as the representation error and 𝜖 (𝑡) as the measurement noise as follows: Definition 5.1 (Representation error and measurement noise). We define the representation error 𝑑 (𝑎, 𝑡) as the gap between the ground-truth unknown dynamics 𝑓 and the learned representation 𝜙 multiplying the “true” linear parameter 𝑎(𝑡), namely: 𝑓 (𝑞, 𝑞, ¤ 𝑤(𝑡)) = 𝜙(𝑞, 𝑞)𝑎(𝑡) ¤ + 𝑑 (𝑎(𝑡), 𝑡). Note that 𝑑 (𝑎, 𝑡) depends on 𝑎. For example, if 𝜙 is perfect, there exists an oracle 𝑎 ∗ such that 𝑓 (𝑞, 𝑞, ¤ 𝑤(𝑡)) = 𝜙(𝑞, 𝑞)𝑎 ¤ ∗ (𝑤(𝑡)) and 𝑑 (𝑎 ∗ , 𝑡) = 0. On the other hand, we define the measurement noise as 𝜖 (𝑡) = 𝑦(𝑡) − 𝑓 (𝑞, 𝑞, ¤ 𝑤(𝑡)) which is the gap between the ground-truth unknown dynamics 𝑓 and its online noisy measurement 𝑦. Finally, we define the joint error 𝜖¯ as 𝜖¯ = 𝜖 + 𝑑. In the next subsection, our theory shows that the robot tracking error exponentially converges to an error ball whose size is upper bounded by ∥𝑑 (𝑎, 𝑡) + 𝜖 (𝑡)∥ = ∥ 𝜖¯ ∥ and ∥ 𝑎∥ ¤ (i.e., how fast the wind condition changes) for arbitrary 𝑎(𝑡). Stability Analysis Before discussing the main theorem and proof, let us consider the stability and robustness properties of the feedback controller without considering any specific adaptation law. Define the parameter estimation error 𝑎˜ = 𝑎ˆ − 𝑎. Taking the dynamics Eq. (5.1) and the control law Eq. (5.7),we find 𝑀 𝑠¤ + (𝐶 + 𝐾)𝑠 = 𝑓 − 𝜙 𝑎ˆ = −𝜙 𝑎˜ + 𝑑 (𝑎, 𝑡). (5.10) 97 As the stability analysis in Chapter 3, we can use the Lyapunov function 𝑉 = 𝑠⊤ 𝑀 𝑠 under the assumption of bounded 𝑎˜ to show that lim ∥𝑠∥ ≤ 𝑡→∞ sup𝑡 ∥𝑑 (𝑎, 𝑡) − 𝜙 𝑎∥𝜆 ˜ max (𝑀) 𝜆 min (𝐾)𝜆 min (𝑀) for arbitrary 𝑎(𝑡). (5.11) Taking this results alone, one might expect that any online estimator or learning algorithm will lead to good performance. However, the boundedness of 𝑎˜ is not guaranteed (Slotine and Li (1991) discuss this topic thoroughly). Even though ∥ 𝑎∥ ˜ is bounded, it may not converge and could be arbitrarily large. Therefore, in the theorem and proof below, we formally show that [𝑠; 𝑎] ˜ will jointly converge. Theorem 5.1 (Main stability and robustness guarantee). The adaptive controller given in Eqs. (5.7) to (5.9) guarantees that the tracking error 𝑞˜ exponentially converges to the following error ball:   ¤ lim ∥ 𝑞∥ ˜ ≤ sup 𝐶1 ∥𝑑 (𝑎(𝑡), 𝑡)∥ + 𝐶2 ∥𝜖 (𝑡)∥ + 𝐶3 (𝜆∥𝑎(𝑡)∥ + ∥ 𝑎(𝑡)∥) (5.12) 𝑡→∞ 𝑡 for arbitrary 𝑎(𝑡), where 𝐶1 , 𝐶2 , and 𝐶3 are three bounded constants depending on 𝜙, 𝑅, 𝑄, 𝐾, Λ, 𝑀 and 𝜆. Proof. Combining the closed-loop dynamics Eq. (5.10) and the adaptation law Eq. (5.8), we have the following joint closed-loop dynamics for the tracking error 𝑠 and the parameter error 𝑎: ˜ " #" # " #" # " # 𝑀 0 𝑠¤ 𝐶+𝐾 𝜙 𝑠 𝑑 + = ⊤ −1 . 0 𝑃−1 𝑎¤̃ −𝜙𝑇 𝜙⊤ 𝑅 −1 𝜙 + 𝜆𝑃−1 𝑎˜ 𝜙 𝑅 𝜖¯ − 𝜆𝑃−1 𝑎 − 𝑃−1 𝑎¤ (5.13) We also have the following 𝑃−1 dynamics from Eq. (5.9): 𝑑  −1  ¤ −1 = 2𝜆𝑃−1 − 𝑃−1 𝑄𝑃−1 + 𝜙⊤ 𝑅 −1 𝜙. 𝑃 = −𝑃−1 𝑃𝑃 (5.14) 𝑑𝑡 In this proof, we rely on the fact that 𝑃−1 is both uniformly positive definite and uniformly bounded, that is, there exists some positive definite constant matrices 𝐴 and 𝐵 such that 𝐴 ⪰ 𝑃−1 ⪰ 𝐵 (see a proof when 𝜙 is bounded and 𝑄 ≻ 0 in Dieci and Eirola (1994)). Now consider the Lyapunov function 𝑉 given by " #⊤ " #" # 𝑠 𝑀 0 𝑠 𝑉= . 𝑎˜ 0 𝑃−1 𝑎˜ | {z } M (5.15) 98 Plugging in Eqs. (5.13) and (5.14), this Lyapunov function has the following time derivative: " #⊤ " # " # " #⊤ " #" # ¤ 𝑠 𝑀 0 𝑠 ¤ 𝑠 𝑀 0 𝑠 𝑉¤ = 2 +
𝑎˜ 0 𝑃−1 𝑎¤̃ 𝑎˜ 0 𝑑𝑡𝑑 𝑃−1 𝑎˜ " #⊤ " #" # " #⊤ " # 𝑠 𝐶+𝐾 𝜙 𝑠 𝑠 𝑑 = −2 +2 + 𝑎˜ −𝜙𝑇 𝜙⊤ 𝑅 −1 𝜙 + 𝜆𝑃−1 𝑎˜ 𝑎˜ 𝜙⊤ 𝑅 −1 𝜖¯ − 𝜆𝑃−1 𝑎 − 𝑃−1 𝑎¤ #" # " #⊤ " 𝑠 𝑠 𝑀¤ 0
𝑑 −1 𝑎˜ 𝑎˜ 0 𝑑𝑡 𝑃 " #⊤ " #" # " #⊤ " # 𝑠 2𝐾 0 𝑠 𝑠 𝑑 =− +2 𝑎˜ 0 𝜙⊤ 𝑅 −1 𝜙 + 𝑃−1 𝑄𝑃−1 𝑎˜ 𝑎˜ 𝜙⊤ 𝑅 −1 𝜖¯ − 𝜆𝑃−1 𝑎 − 𝑃−1 𝑎¤ | {z } | {z } K 𝐷 (5.16) where we used the fact 𝑀¤ −2𝐶 is skew-symmetric again. Since 𝑀, 𝑃−1 , 𝐾 and 𝑄 are both uniformly bounded and positive definite, and 𝜙⊤ 𝑅 −1 𝜙 is positive semi-definite, there exists some 𝛼 > 0 such that K ⪰ 2𝛼M, ∀𝑡. (5.17) Therefore we have √︄ 𝑉¤ ≤ −2𝛼𝑉 + 2∥𝐷 ∥ 𝑉 , 𝜆min (M) ∀𝑡. (5.18) √ Focusing on 𝑉 and following the analysis in Chapter 3, we have √︁ √ ∥𝐷 ∥ 𝑉 ≤ 𝑉 (0)𝑒 −𝛼𝑡 + √︁ . 𝛼 𝜆 min (M) (5.19) Therefore the joint state [𝑠; 𝑎] ˜ exponentially converges to the following error ball " # 𝑠 ∥𝐷 ∥ lim ≤ sup . (5.20) 𝑡→∞ 𝑎˜ 𝑡 𝛼 · 𝜆 min (M) Finally, the relation between 𝑠 and 𝑞˜ (Lemma 2.3) yields that ∥ 𝑞∥ ˜ exponentially converges to ∥𝐷 ∥ lim ∥ 𝑞∥ ˜ ≤ sup (5.21) 𝑡→∞ 𝑡 𝛼 · 𝜆 min (M)𝜆 min (Λ) which completes the proof. □ 99 5.6 Experiments In this section, we first discuss the experimental platform for data collection and experiments. Second, we discuss the implementations of our method and baselines. Third, we discuss several experiments to quantitatively compare the closed-loop trajectory-tracking performance of our methods to a nonlinear baseline method and two state-of-the-art adaptive flight control methods, and we observe our methods reduce the average tracking error substantially. In order to demonstrate the new capabilities brought by our methods, we present agile flight results in gusty winds, where the UAV must quickly fly through narrow gates that are only slightly wider than the vehicle. Finally, we show our methods are also applicable in outdoor agile tracking tasks without external motion capture systems. Experimental Platform All of our experiments are conducted at Caltech’s Center for Autonomous Systems and Technologies (CAST). The experimental setup consists of an OptiTrack motion capture system with 12 infrared cameras for localization streaming position measurements at 50 Hz, a WiFi router for communication, the Caltech Real Weather Wind Tunnel for generating dynamic wind conditions, and a custom-built quadrotor UAV. For outdoor flight, the drone is also equipped with a Global Positioning System (GPS) module and an external antenna. We now discuss the design of the UAV and the wind condition in detail. UAV design. We built a quadrotor UAV for our primary data collection and all experiments, shown in Fig. 5.2(A). The quadrotor weighs 2.6 kg with a thrust to weight ratio of 2.2. The UAV is equipped with a Pixhawk flight controller running PX4, an open-source commonly used drone autopilot platform (Meier et al., 2012). The UAV incorporates a Raspberry Pi 4 onboard computer running a Linux operation system, which performs real-time computation and adaptive control and interfaces with the flight controller through MAVROS, an open-source set of communication drivers for UAVs. State estimation is performed using the built-in PX4 Extended Kalman Filter (EKF), which fuses inertial measurement unit (IMU) data with global position estimates from OptiTrack motion capture system (or the GPS module for outdoor flight tasks). The UAV platform features a wide-X configuration, measuring 85 cm in width, 75 cm in length, and 93 cm diagonally, and tilted motors for improved yaw authority. This general hardware setup is standard and similar to many quadrotors. To study the generalizability and robustness of our approach, we also use an Intel 100 Aero Ready to Fly drone for data collection. This dataset is used to train a representation of the wind effects on the Intel Aero drone, which we test on our custom UAV. The Intel Aero drone (weighing 1.4 kg) has a symmetric X configuration, 52 cm in width and 52 cm in length, without tilted motors. Wind condition design. To generate dynamic and diverse wind conditions for the data collection and experiments, we leverage the state-of-the-art Caltech Real Weather Wind Tunnel system (Fig. 5.2). The wind tunnel is a 3 m by 3 m array of 1296 independently controllable fans capable of generating wind conditions up to 43.6 km/h. The distributed fans are controlled in real-time by a Pythonbased Application Programming Interface (API). For data collection and flight experiments, we designed two types of wind conditions. For the first type, each fan has uniform and constant wind speed between 0 km/h and 43.6 km/h (12.1 m/s). The second type of wind follows a sinusoidal function in time, e.g., 30.6 + 8.6 sin(𝑡) km/h. Note that the training data only covers constant wind speeds up to 6.1 m/s. To visualize the wind, we use 5 smoke generators to indicate the direction and intensity of the wind condition (see examples in Fig. 5.2 and the video). Data Collection and Offline Meta-Training To learn an effective representation of the aerodynamic effects, we have a custombuilt drone follow a randomized trajectory for 2 minutes each in six different static wind conditions, with speeds ranging from 0 km/h to 22.0 km/h. However, in experiments we used wind speeds up to 43.6 km/h (12.1 m/s) to study how our methods extrapolate to unseen wind conditions (e.g., Fig. 5.7). The randomized trajectory follows a polynomial spline between 3 waypoints: the current position and two randomly generated target positions. This process allows us to generate a large amount of data using a trajectory very different from the trajectories used to test our method, such as the figure-8 in Fig. 5.2. By training and testing on different trajectories, we demonstrate that the learned model generalizes well to new trajectories. Along each trajectory, we collect time-stamped data [𝑞, 𝑞, ¤ 𝑢]. Next, we compute the acceleration 𝑞¥ by fifth-order numerical differentiation and compute the noisy label of 𝑓 using Eq. (5.1). The data is collected at 50 Hz with a total of 36, 000 data points. Figure 5.4(A) shows the data collection process, and Fig. 5.4(B) shows the inputs and labels of the training data, under one wind condition of 13.3 km/h (3.7 m/s). Figure 5.4(C) shows the distributions of input data (pitch) and 101 label data (𝑥−component of the aerodynamic force) in different wind conditions. Clearly, a shift in wind conditions causes distribution shifts in both input domain and label domain, which motivates the algorithm design of DAIML. The same data collection process is repeated on the Intel Aero drone, to study whether the learned representation can generalize to a different drone. On the collected datasets for both our custom drone and the Intel Aero drone, we apply the DAIML algorithm (Algorithm 5.1) to learn two representations 𝜙 of the wind effects. The learning process is done offline on a normal desktop computer, and depicted in Fig. 5.3(B). Figure 5.5 shows the evolution of the linear coefficients (𝑎 ∗ ) during the learning process, where DAIML learns a representation of the aerodynamic effects shared by all wind conditions, and the linear coefficient contains the wind-specific information. Moreover, the learned representation is explainable in the sense that the linear coefficients in different wind conditions are well disentangled (see Fig. 5.5). Implementation of Our Control Methods and Baselines We following the cascading control structure in Chapter 2 to apply our controller and baselines as position controllers for a drone. In particular, the position dynamics of a drone in Eq. (2.4a) can be reduced to Eq. (5.1) by taking 𝑀 = 𝑚𝐼, 𝐶 = 0, 𝑞 = 𝑝, 𝑢 = 𝑅 𝑓𝑢 ∈ R3 and 𝑓 = 𝑓𝑎 ∈ R3 . Therefore, we implemented all control methods in the position control loop in Python, and run it on the onboard Linux computer at 50 Hz. The PX4 was set to the offboard flight mode and received thrust and attitude commands from the position control loop. The built-in PX4 multicopter attitude controller was then executed at the default rate. The online inference of the learned representation is also in Python via PyTorch (Paszke et al., 2019). We implemented three baselines: • Globally exponentially-stabilizing nonlinear tracking controller for quadrotor control (Morgan et al., 2016; G. Shi, X. Shi, et al., 2019; X. Shi, Kim, et al., ∫ 2018). This controller uses an integral term 𝑞˜ to compensate 𝑓 . • Incremental nonlinear dynamics inversion (INDI) linear acceleration control (Tal and Karaman, 2021). In particular, we implemented the position and acceleration controller from Sections III.A and III.B in Tal and Karaman (2021). 102 • L1 adaptive control (Mallikarjunan et al., 2012; Hanover et al., 2021). We followed the adaptation law first presented in Pravitra et al. (2020), and augment the nonlinear baseline control with 𝑓ˆ = −𝑢 L1 . The primary difference between these baseline methods and Neural-Fly is how the controller compensates for the unmodelled residual force (that is, each baseline method has the same control structure, in Fig. 5.3(C), except for the estimation of the 𝑓ˆ). The integral gain in the first baseline is limited by the stability of the interaction with the position and velocity error feedback, leading to slow model correction. In contrast, both INDI and L1 decouple the adaptation rate from the PD gains, which allow for fast adaptation. However, both INDI and L1 directly estimate the unknown vector 𝑓 which might change very fast, thus limited by more fundamental design factors, such as system delay, measurement noise, and controller rate. We also implemented three variants of our methods, depending on the learned representation 𝜙: • Neural-Fly is our primary method using a representation learned from the dataset collected by the custom-built drone, which is the same drone used in experiments. • Neural-Fly-Transfer uses the Neural-Fly algorithm where the representation is trained using the dataset collected by the aforementioned Intel Aero drone. This variant is included to show the generalizability and robustness of our approach with drone transfer, i.e., using a different drone in experiments than data collection. • Neural-Fly-Constant uses the online adaptation algorithm from Neural-Fly, but the representation is an artificially designed constant mapping (𝜙 = 𝐼). Neural-Fly-Constant demonstrates the benefit of using a better representation learned from the proposed meta-learning method DAIML. Note that NeuralFly-Constant is a composite adaptation form of a Kalman-filter disturbance observer, that is a Kalman-filter augmented with a tracking error update term. Neural network architectures and training details. In practice, we found that in addition to the drone velocity, the aerodynamic effects also depend on the drone attitude and the rotor rotation speed. To that end, the input state 𝑥 to the deep neural network 𝜙 is a 11-d vector, consisting of the drone velocity (3-d), the drone 103 attitude represented as a quaternion (4-d), and the rotor speed commands as a pulse width modulation (PWM) signal (4-d) (see Figs. 5.3 and 5.4). The DNN 𝜙 has four fully-connected hidden layers, with an architecture 11 → 50 → 60 → 50 → 4 and Rectified Linear Units (ReLU) activation. We found that the three components of the wind-effect force, 𝑓𝑥 , 𝑓 𝑦 , 𝑓𝑧 , are highly correlated and sharing common features, so we use 𝜙 as the basis function for all the component. Therefore, the wind-effect force 𝑓 is approximated by 𝜙(𝑥) 0 0  𝑎 𝑥      𝑓 ≈ 0 𝜙(𝑥) 0  𝑎 𝑦  ,     0 0 𝜙(𝑥)   𝑎 𝑧   (5.22) where 𝑎 𝑥 , 𝑎 𝑦 , 𝑎 𝑧 ∈ R4 are the linear coefficients for each component of the windeffect force. Note that we explicitly include the PWM as an input to the 𝜙 network. The PWM information is a function of 𝑢, which makes the controller law Eq. (5.7) non-affine in 𝑢. We solve this issue by using the PWM from the last time step as an input to 𝜙, to compute the desired force 𝑢 at the current time step. Because we train 𝜙 using spectral normalization (see Algorithm 5.1), this method is stable and guaranteed to converge to a fixed point, as discussed in Chapter 3. Trajectory Tracking Performance We quantitatively compare the performance of the aforementioned control methods when the UAV follows a 2.5 m wide, 1.5 m tall figure-8 trajectory with a lap time of 6.28 s under constant, uniform wind speeds of 0 km/h, 15.1 km/h (4.2 m/s), 30.6 km/h (8.5 m/s), and 43.6 km/h (12.1 m/s) and under time-varying wind speeds of 30.6 + 8.6 sin(𝑡) km/h (8.5 + 2.4 sin(𝑡) m/s). The flight trajectory for each of the experiments is shown in Fig. 5.6, which includes a warm up lap and six 6.28 s laps. The nonlinear baseline integral term compensates for the mean model error within the first lap. As the wind speed increases, the aerodynamic force variation becomes larger and we notice a substantial performance degradation. INDI and L1 both improve over the nonlinear baseline, but INDI is more robust than L1 at high wind speeds. Neural-Fly-Constant outperforms INDI except during the two most challenging tasks: 43.6 km/h and sinusoidal wind speeds. The learning based methods, Neural-Fly and Neural-Fly-Transfer, outperform all other methods in all tests. Neural-Fly outperforms Neural-Fly- 104 Figure 5.6: Depiction of the trajectory tracking performance of each controller in several wind conditions. The baseline nonlinear controller can track the trajectory well, however, the performance substantially degrades at higher wind speeds. INDI, L1 , and Neural-Fly-Constant have similar performance and improve over the nonlinear baseline by estimating the aerodynamic disturbance force quickly. Neural-Fly and Neural-Fly-Transfer use a learned model of the aerodynamic effects and adapt the model in real time to achieve lower tracking error than the other methods. Table 5.1: Tracking error statistics in cm for different wind conditions. Two metrics are considered: root-mean-square (RMS) and mean. PP Wind PP 0 PP[m/s] PP Method P RMS Mean Nonlinear INDI L1 NF-Constant NF-Transfer NF 11.9 7.3 4.6 5.4 3.7 3.2 10.8 6.3 4.2 5.0 3.4 2.9 4.2 RMS 10.7 6.4 5.8 6.1 4.8 4.0 Mean 9.9 5.9 5.2 5.7 4.4 3.7 8.5 RMS 16.3 8.5 12.1 7.5 6.2 5.8 Mean 14.7 8.2 11.1 6.9 5.9 5.3 12.1 RMS 23.9 10.7 22.7 12.7 10.2 9.4 Mean 21.6 10.1 21.3 11.2 9.4 8.7 8.5 + 2.4 sin(𝑡) RMS 31.2 11.1 13.0 12.7 8.8 7.6 Mean 28.2 10.3 11.6 12.1 8.0 6.9 Transfer slightly, which is because the learned model was trained on data from the same drone and thus better matches the dynamics of the vehicle. In Table 5.1, we tabulate the root-mean-square position error and mean position error values over the six laps for each experiment. Figure 5.7 shows how the mean tracking error changes for each controller as the wind speed increases, and includes the standard deviation for the mean lap position error. In all cases, Neural-Fly and Neural-Fly-Transfer outperform the state-of-the-art baseline methods, including the 105 Mean tracking error in each lap Tracking error [cm] 31.6 Interpolation region Extrapolation region (not covered in training) 17.8 Nonlinear INDI L1 NF-Constant NF-Transfer NF PID 10.0 5.6 3.2 0.0 4.2 Wind speed [m/s] 8.5 12.1 Figure 5.7: Mean tracking errors of each lap in different wind conditions. This figure shows position tracking errors of different methods as wind speed increases. Solid lines show the mean error over 6 laps and the shade areas show standard deviation of the mean error on each lap. The grey area indicates the extrapolation region, where the wind speeds are not covered in training. Our primary method (Neural-Fly) achieves state-of-the-art performance even with a strong wind disturbance. 30.6 km/h, 43.6 km/h, and sinusoidal wind speeds all of which exceed the wind speed in the training data. All of these results presents a clear trend: adaptive control substantially outperforms the nonlinear baseline which relies on integralcontrol, and learning markedly improves adaptive control. Agile Flight Through Narrow Gates Precise flight control in dynamic and strong wind conditions has many applications, such as rescue and search, delivery, and transportation. In this section, we present a challenging drone flight task in strong winds, where the drone must follow agile trajectories through narrow gates, which are only slightly wider than the drone. The overall result is depicted in Fig. 5.2 and the video. As shown in Fig. 5.2(A), the gates used in our experiments are 110 cm in width, which is only slightly wider than the drone (85 cm wide, 75 cm long). To visualize the trajectory using long-exposure photography, our drone is deployed with four main light emitting diodes (LEDs) on its legs, where the two rear LEDs are red and the front two are white. There are also several small LEDs on the flight controller, the computer, and the motor controllers, which can be seen in the long-exposure shots. Task design. We tested our method on three different tasks. In the first task (see Fig. 5.2(B,D,F-I) and the video), the desired trajectory is a 3 m by 1.5 m figure-8 in the 𝑥 − 𝑧 plane with a lap time of 5 s. A gate is placed at the left bottom part 106 of the trajectory. The minimum clearance is about 10 cm (see Fig. 5.2(I), which requires that the controller precisely tracks the trajectory. The maximum speed and acceleration of the desired trajectory are 2.7 m/s and 5.0 m/s2 , respectively. The wind speed is 3.1 m/s. The second task (see Video 1) is the same as the first one, except that it uses a more challenging, time-varying wind condition, 3.1 + 1.8 sin( 2𝜋 5 𝑡)m/s. In the third task (see Fig. 5.2(C,E) and the video), the desired trajectory is a 3 m by 2.5 m ellipse in the 𝑥 − 𝑦 plane with a lap time of 5 s. We placed two gates on the left and right sides of the ellipse. As with the first task, the wind speed is 3.1 m/s. Performance. For all three tasks, we used our primary method, Neural-Fly, where the representation is learned using the dataset collected by the custom-built drone. Figure 5.2(D,E) are two long-exposure photos with an exposure time of 5 s, which is the same as the lap time of the desired trajectory. We see that our method precisely tracked the desired trajectories and flew safely through the gates (see the video). These long-exposure photos also captured the smoke visualization of the wind condition. We would like to emphasize that the drone is wider than the LED light region, since the LEDs are located on the legs (see Fig. 5.2(A)). Figure 5.2(F-I) are four high-speed photos with a shutter speed of 1/200s. These four photos captured the moment the drone passed through the gate in the first task, as well as the complex interaction between the drone and the wind. We see that the aerodynamic effects are complex and non-stationary and depend on the UAV attitude, the relative velocity, and aerodynamic interactions between the propellers and the wind. Outdoor Experiments We tested our algorithm outdoors in gentle breeze conditions (wind speeds measured up to 17 km/h). An onboard GPS receiver provided position information to the EKF, giving lower precision state estimation, and therefore less precise aerodynamic residual force estimation. Following the same aforementioned figure-8 trajectory, the controller reached 7.5 cm mean tracking error, shown in Fig. 5.8. Discussions State-of-the-art tracking performance. When measuring position tracking errors, we observe that our Neural-Fly method outperforms state-of-the-art flight controllers in all wind conditions. Furthermore, we observe a mean tracking error of 2.9 cm in 0 km/h wind, which is comparable with state-of-the-art tracking performance demonstrated on more aggressive racing drones (Tal and Karaman, 2021; Hanover 107 z [m] weather station 8.75 8.50 8.25 8.00 7.75 7.50 7.25 7.00 Outdoor tracking performance RMS error=8.2cm, mean error=7.5cm >50 30 10 3 2.0 1.5 1.0 x [m] 0.5 0.0 0.5 Tracking error [cm] GPS 1 Figure 5.8: Outdoor flight setup and performance. Left: In outdoor experiments, a GPS module is deployed for state estimation, and a weather station records wind profiles. The maximum wind speed during the test was around 17 km/h (4.9 m/s). Right: Trajectory tracking performance of Neural-Fly. et al., 2021) despite several architectural limitations such as limited control rate in offboard mode, a larger, less maneuverable vehicle, and without direct motor speed measurements. All our experiments were conducted using the standard PX4 attitude controller, with Neural-Fly implemented in an onboard, low cost, and “credit-card sized” Raspberry Pi 4 computer. Furthermore, Neural-Fly is robust to changes in vehicle configuration, as demonstrated by the similar performance of Neural-FlyTransfer. To understand the fundamental tracking-error limit, we estimate that the localization precision from the OptiTrack system is about 1 cm, which is a practical lower bound for the average tracking error in our system. This is based on the fact that the difference between the OptiTrack position measurement and the onboard EKF position estimate is around 1 cm. To achieve a tracking error of 1 cm, remaining improvements should focus on reducing code execution time, communication delays, and attitude tracking delay. We measured the combined code execution time and communication delay to be at least 15 ms and often as much as 30 ms. A faster implementation (such as using C++ instead of Python) and streamlined communication layer (such as using ROS2’s real-time features) could allow us to achieve tracking errors on the order of the localization accuracy. Attitude tracking delay can be substantially reduced through the use of a nonlinear attitude controller (e.g., X. Shi, Kim, et al. (2018)). 108 Our method is also directly extensible to attitude control because attitude dynamics match the Euler-Lagrange dynamics used in our derivations. However, further work is needed to understand the interaction of the learned dynamics with the cascaded control design when implementing a tracking attitude controller. Challenges caused by unknown and time-varying wind conditions. In the real world, the wind conditions are not only unknown but also constantly changing, and the vehicle must continuously adapt. We designed the the sinusoidal wind test to emulate unsteady or gusty wind conditions. Although our learned model is trained on static and approximately uniform wind condition data, Neural-Fly can quickly identify changing wind speed and maintains precise tracking even on the sinusoidal wind experiment. Moreover, in each of our experiments, the wind conditions were unknown to the UAV before starting the test yet were quickly identified by the adaptation algorithm. Our work demonstrated that it is possible to repeatably and quantitatively test quadrotor flight in time-varying wind. Our method separately learns the wind effect’s dependence on the vehicle state (i.e., the wind-invariant representation in Fig. 5.3(A)) and the wind condition (i.e., the wind-specific linear weight in Fig. 5.3(A)). This separation allows Neural-Fly to quickly adapt to the time-varying wind even as the UAV follows a dynamic trajectory, with an average tracking error below 8.7 cm in Table 5.1. Computational efficiency of our method. In the offline meta-learning phase, the proposed DAIML algorithm is able to learn an effective representation of the aerodynamic effect in a data efficient manner. This requires only 12 minutes of flight data at 50 Hz, for a total of 36,000 data points. The training procedure only takes 5 minutes on a normal desktop computer. In the online adaptation phase, our adaptive control method only takes 10 ms to compute on a compact onboard Linux computer (Raspberry Pi 4). In particular, the feedforward inference time via the learned basis function is about 3.5 ms and the adaptation update is about 3.0 ms, which implies the compactness of the learned representation. Generalization to new trajectories and new aircrafts. It is worth noting that our control method is orthogonal to the design of the desired trajectory 𝑞 𝑑 . In this chapter, we focus on the figure-8 trajectory which is a commonly used control benchmark. We also demonstrate our method flying a horizontal ellipse during the narrow gate demonstration Fig. 5.2. Note that our method supports any trajectory planners such as Foehn et al. (2021) or learning-based planners (Nakka et al., 2021; 109 Loquercio et al., 2021). In particular, for those planners which require a precise and agile downstream controller (e.g., for close-proximity flight or drone racing (Foehn et al., 2021; G. Shi, Hönig, et al., 2022)), our method immediately provides a solution and further pushes the boundary of these planners, because our stateof-the-art tracking capabilities enable tighter configurations and smaller clearances. However, further research is required to understand the coupling between planning and learning-based control near actuation limits. Future work will consider using Neural-Fly in a combined planning and control structure such as MPC, which will be able to handle actuation limits. The comparison between Neural-Fly and Neural-Fly-Transfer show that our approach is robust to changing vehicle design and the learned representation does not depend on the vehicle. This demonstrates the generalizability of the proposed method running on different quadrotors. Moreover, our control algorithm is formulated generally for all robotic systems described by the Euler-Langrange equation Eq. (5.1), including many types of aircraft such as X. Shi, Spieler, et al. (2020) and Kim et al. (2021). 5.7 Appendix The Expressiveness of the Learning Architecture In this section, we theoretically justify the decomposition 𝑓 (𝑥, 𝑤) ≈ 𝜙(𝑥)𝑎(𝑤). In particularly, we prove that any analytic function 𝑓¯(𝑥, 𝑤) : [−1, 1] 𝑛 × [−1, 1] 𝑚 → R ¯ and a 𝑤-dependant part 𝑎(𝑤) can be split into a 𝑤-invariant part 𝜙(𝑥) ¯ in the structure ¯ 𝑎(𝑤) ¯ 𝜙(𝑥) ¯ with arbitrary precision 𝜖, where 𝜙(𝑥) and 𝑎(𝑤) ¯ are two polynomials. Further, the dimension of 𝑎(𝑤) ¯ only scales polylogarithmically with 1/𝜖. We first introduce the following multivariate polynomial approximation lemma in the hypercube proved in Trefethen (2017). Lemma 5.1. (Multivariate polynomial approximation in the hypercube) Let 𝑓¯(𝑥, 𝑤) : [−1, 1] 𝑛 × [−1, 1] 𝑚 → R be a smooth function of [𝑥, 𝑤] ∈ [−1, 1] 𝑛+𝑚 for 𝑛, 𝑚 ≥ 1. Assume 𝑓¯(𝑥, 𝑤) is analytic for all [𝑥, 𝑤] ∈ C𝑛+𝑚 with ℜ(𝑥 12 +· · ·+𝑥 𝑛2 +𝑤 21 +· · ·+𝑤 2𝑚 ) ≥ −𝑡 2 for some 𝑡 > 0, where ℜ(·) denotes the real part of a complex number. Then 𝑓¯ has a uniformly and absolutely convergent multivariate Chebyshev series ∞ ∑︁ 𝑘 1 =0 ··· ∞ ∑︁ ∞ ∑︁ 𝑘 𝑛 =0 𝑙1 =0 ··· ∞ ∑︁ 𝑏 𝑘 1 ,··· ,𝑘 𝑛 ,𝑙1 ,··· ,𝑙 𝑚 𝑇𝑘 1 (𝑥 1 ) · · · 𝑇𝑘 𝑛 (𝑥 𝑛 )𝑇𝑙1 (𝑤 1 ) · · · 𝑇𝑙 𝑚 (𝑤 𝑚 ). 𝑙 𝑚 =0 Define 𝑠 = [𝑘 1 , · · · , 𝑘 𝑛 , 𝑙1 , · · · , 𝑙 𝑚 ]. The multivariate Chebyshev coefficients satisfy 110 the following exponential decay property:   𝑏 𝑠 = 𝑂 (1 + 𝑡) −∥𝑠∥ 2 . Note that this lemma shows that the truncated Chebyshev expansions C𝑝 = 𝑝 ∑︁ ··· 𝑘 1 =0 𝑝 ∑︁ 𝑝 ∑︁ ··· 𝑘 𝑛 =0 𝑙 1 =0 𝑝 ∑︁ 𝑏 𝑘 1 ,··· ,𝑘 𝑛 ,𝑙1 ,··· ,𝑙 𝑚 𝑇𝑘 1 (𝑥 1 ) · · · 𝑇𝑘 𝑛 (𝑥 𝑛 )𝑇𝑙1 (𝑤 1 ) · · · 𝑇𝑙 𝑚 (𝑤 𝑚 ) 𝑙 𝑚 =0 √ will converge to 𝑓¯ with the rate 𝑂 ((1 + 𝑡) −𝑝 𝑛+𝑚 ) for some 𝑡 > 0, i.e., √ ∥ 𝑓¯(𝑥, 𝑤) − C𝑝 (𝑥, 𝑤)∥ ≤ 𝑂 ((1 + 𝑡) −𝑝 𝑛+𝑚 ). sup [𝑥,𝑤]∈[−1,1] 𝑛+𝑚 Finally we are ready to present the following representation theorem. Theorem 5.2. 𝑓¯(𝑥, 𝑤) is a function satisfying the assumptions in Lemma 5.1. For ¯ any 𝜖 > 0, there exist ℎ ∈ Z+ , and two Chebyshev polynomials 𝜙(𝑥) : [−1, 1] 𝑛 → R1×ℎ and 𝑎(𝑤) ¯ : [−1, 1] 𝑚 → Rℎ×1 such that ¯ 𝑎(𝑤)∥ ∥ 𝑓¯(𝑥, 𝑤) − 𝜙(𝑥) ¯ ≤𝜖 sup [𝑥,𝑤]∈[−1,1] 𝑛+𝑚 and ℎ = 𝑂 ((log(1/𝜖)) 𝑚 ). Proof. First note that there exists 𝑝 = 𝑂  log(1/𝜖) √ 𝑛+𝑚  such that 𝑓¯(𝑥, 𝑤) − C𝑝 (𝑥, 𝑤) ≤ 𝜖 . sup [𝑥,𝑤]∈[−1,1] 𝑛+𝑚 To simplify the notation, define 𝑔(𝑥, 𝑘, 𝑙) = 𝑔(𝑥1 , · · · , 𝑥 𝑛 , 𝑘 1 , · · · , 𝑘 𝑛 , 𝑙1 , · · · , 𝑙 𝑚 ) = 𝑏 𝑘 1 ,··· ,𝑘 𝑛 ,𝑙1 ,··· ,𝑙 𝑚 𝑇𝑘 1 (𝑥1 ) · · · 𝑇𝑘 𝑛 (𝑥 𝑛 ) 𝑔(𝑤, 𝑙) = 𝑔(𝑤 1 , · · · , 𝑤 𝑚 , 𝑙1 , · · · , 𝑙 𝑚 ) = 𝑇𝑙1 (𝑤 1 ) · · · 𝑇𝑙 𝑛 (𝑤 𝑚 ). Then we have C𝑝 (𝑥, 𝑤) = 𝑝 ∑︁ 𝑝 ∑︁ 𝑘 1 ,··· ,𝑘 𝑛 =0 𝑙1 ,··· ,𝑙 𝑚 =0 𝑔(𝑥, 𝑘 1 , · · · , 𝑘 𝑛 , 𝑙1 , · · · , 𝑙 𝑚 )𝑔(𝑤, 𝑙1 , · · · , 𝑙 𝑚 ). 111 ¯ 𝑎(𝑤): Then we rewrite C𝑝 as C𝑝 (𝑥, 𝑤) = 𝜙(𝑥) ¯   Í𝑝  𝑘 1 ,··· ,𝑘 𝑛 =0 𝑔(𝑥, 𝑘 1 , · · · , 𝑘 𝑛 , 𝑙 = [0, 0, · · · , 0])    Í𝑝   𝑔(𝑥, 𝑘 , · · · , 𝑘 , 𝑙 = [1, 0, · · · , 0]) 1 𝑛   Í 𝑘 1 ,··· ,𝑘 𝑛 =0 ¯ ⊤ =  𝑘𝑝 ,··· ,𝑘 =0 𝑔(𝑥, 𝑘 1 , · · · , 𝑘 𝑛 , 𝑙 = [2, 0, · · · , 0])  , 𝜙(𝑥) 𝑛   1 ..   .    Í 𝑝    𝑘 1 ,··· ,𝑘 𝑛 =0 𝑔(𝑥, 𝑘 1 , · · · , 𝑘 𝑛 , 𝑙 = [ 𝑝, 𝑝, · · · , 𝑝])   𝑔(𝑤, 𝑙 = [0, 0, · · · , 0])       𝑔(𝑤, 𝑙 = [1, 0, · · · , 0])     𝑔(𝑤, 𝑙 = [2, 0, · · · , 0])  𝑎(𝑤) ¯ = .   ..   .     𝑔(𝑤, 𝑙 = [ 𝑝, 𝑝, · · · , 𝑝])    ¯ Note that the dimension of 𝜙(𝑥) and 𝑎(𝑤) ¯ is  𝑚 log(1/𝜖) 𝑚 ℎ = ( 𝑝 + 1) = 𝑂 1 + √ = 𝑂 ((log(1/𝜖)) 𝑚 ) . 𝑛+𝑚 □ Note that Theorem 5.2 can be generalized to vector-valued functions with bounded input space straightforwardly. Finally, since deep neural networks are universal approximators for polynomials (Yarotsky, 2017), Theorem 5.2 immediately guarantees the expressiveness of our learning structure, i.e., 𝜙(𝑥)𝑎(𝑤) can approximate 𝑓 (𝑥, 𝑤) with arbitrary precision, where 𝜙(𝑥) is a deep neural network and 𝑎ˆ includes the linear coefficients for all the elements of 𝑓 . In experiments, we show that a four-layer neural network can efficiently learn an effective representation for the underlying unknown dynamics 𝑓 (𝑥, 𝑤). References Bartlett, Peter L., Dylan J. Foster, and Matus J. Telgarsky (2017). Spectrallynormalized margin bounds for neural networks. In: Advances in Neural Information Processing Systems, pp. 6240–6249. Belkhale, Suneel, Rachel Li, Gregory Kahn, Rowan McAllister, Roberto Calandra, and Sergey Levine (2021). Model-based meta-reinforcement learning for flight with suspended payloads. In: IEEE Robotics and Automation Letters 6.2, pp. 1471–1478. Bisheban, Mahdis and Taeyoung Lee (July 2021). Geometric adaptive control with neural networks for a quadrotor in wind fields. In: IEEE Transactions on Control Systems Technology 29.4, pp. 1533–1548. issn: 1558-0865. doi: 10.1109/TCST. 2020.3006184. 112 Chen, Fu-Chuang and Hassan K. Khalil (1995). Adaptive control of a class of nonlinear discrete-time systems using neural networks. In: IEEE Transactions on Automatic Control 40.5, pp. 791–801. Clavera, Ignasi, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel (2018). Model-based reinforcement learning via meta-policy optimization. In: Conference on Robot Learning. Proceedings of Machine Learning Research, pp. 617–629. Dieci, Luca and Timo Eirola (Apr. 1994). Positive definiteness in the numerical solution of Riccati differential equations. In: Numerische Mathematik 67.3, pp. 303– 313. issn: 0945-3245. doi: 10.1007/s002110050030. url: https://doi. org/10.1007/s002110050030. Faessler, Matthias, Antonio Franchi, and Davide Scaramuzza (Apr. 2018). Differential flatness of quadrotor dynamics subject to rotor drag for accurate tracking of high-speed trajectories. In: IEEE Robotics and Automation Letters 3.2, pp. 620– 626. issn: 2377-3766. doi: 10.1109/LRA.2017.2776353. Farrell, Jay A. and Marios M. Polycarpou (2006). Adaptive approximation based control. John Wiley & Sons, Ltd. isbn: 978-0-471-78181-3. doi: 10 . 1002 / 0471781819.fmatter. url: https://onlinelibrary.wiley.com/doi/ abs/10.1002/0471781819.fmatter. Finn, Chelsea, Pieter Abbeel, and Sergey Levine (2017). Model-agnostic metalearning for fast adaptation of deep networks. In: International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 1126–1135. Foehn, Philipp, Angel Romero, and Davide Scaramuzza (2021). Time-optimal planning for quadrotor waypoint flight. In: Science Robotics 6.56. Ganin, Yaroslav, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky (2016). Domainadversarial training of neural networks. In: The Journal of Machine Learning Research 17.1, pp. 2096–2030. Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio (2014). Generative adversarial nets. In: Advances in Neural Information Processing Systems 27. Hanover, Drew, Philipp Foehn, Sihao Sun, Elia Kaufmann, and Davide Scaramuzza (2021). Performance, precision, and payloads: Adaptive nonlinear MPC for quadrotors. In: IEEE Robotics and Automation Letters 7.2, pp. 690–697. Hospedales, Timothy M., Antreas Antoniou, Paul Micaelli, and Amos J. Storkey (2021). Meta-learning in neural networks: A survey. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–1. doi: 10 . 1109 / TPAMI . 2021.3079209. Ioannou, Petros A. and Jing Sun (1996). Robust adaptive control. Vol. 1. PrenticeHall Upper Saddle River, NJ. 113 Johnson, Eric N. and Anthony J. Calise (2003). Limited authority adaptive flight control for reusable launch vehicles. In: Journal of Guidance, Control, and Dynamics 26.6, pp. 906–913. Kalman, R. E. and R. S. Bucy (Mar. 1961). New results in linear filtering and prediction theory. In: Journal of Basic Engineering 83.1, pp. 95–108. issn: 0021-9223. doi: 10.1115/1.3658902. url: https://asmedigitalcollection.asme. org/fluidsengineering/article/83/1/95/426820/New-Results-inLinear-Filtering-and-Prediction. Kim, Kyunam, Patrick Spieler, Elena-Sorina Lupu, Alireza Ramezani, and Soon-Jo Chung (2021). A bipedal walking robot that can fly, slackline, and skateboard. In: Science Robotics 6.59, eabf8136. doi: 10.1126/scirobotics.abf8136. Krstic, Miroslav, Petar V. Kokotovic, and Ioannis Kanellakopoulos (1995). Nonlinear and adaptive control design. John Wiley & Sons, Inc. Lale, Sahin, Kamyar Azizzadenesheli, Babak Hassibi, and Anima Anandkumar (Dec. 2021). Model learning predictive control in nonlinear dynamical systems. In: 2021 60th IEEE Conference on Decision and Control (CDC). ISSN: 25762370, pp. 757–762. doi: 10.1109/CDC45484.2021.9683670. LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton (2015). Deep learning. In: Nature 521.7553, pp. 436–444. Lee, Joonho, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter (2020). Learning quadrupedal locomotion over challenging terrain. In: Science Robotics 5.47. Loquercio, Antonio, Elia Kaufmann, René Ranftl, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza (2021). Learning high-speed flight in the wild. In: Science Robotics 6.59, eabg5810. doi: 10.1126/scirobotics.abg5810. url: https: //www.science.org/doi/abs/10.1126/scirobotics.abg5810. Mallikarjunan, Srinath, Bill Nesbitt, Evgeny Kharisov, Enric Xargay, Naira Hovakimyan, and Chengyu Cao (Aug. 2012). L1 adaptive controller for attitude control of multirotors. In: AIAA Guidance, Navigation, and Control Conference. Minneapolis, Minnesota: American Institute of Aeronautics and Astronautics. isbn: 978-1-60086-938-9. doi: 10.2514/6.2012-4831. url: https://arc. aiaa.org/doi/10.2514/6.2012-4831. McKinnon, Christopher D. and Angela P. Schoellig (2021). Meta learning with paired forward and inverse models for efficient receding horizon control. In: IEEE Robotics and Automation Letters 6.2, pp. 3240–3247. Meier, Lorenz, Petri Tanskanen, Lionel Heng, Gim Hee Lee, Friedrich Fraundorfer, and Marc Pollefeys (Aug. 2012). PIXHAWK -A micro aerial vehicle design for autonomous flight using onboard computer vision. In: Autonomous Robots 33.12, pp. 21–39. issn: 0929-5593. doi: 10 . 1007 / s10514 - 012 - 9281 - 4. url: https : / / graz . pure . elsevier . com / en / publications / pixhawk - a micro-aerial-vehicle-design-for-autonomous-flight-using. 114 Morgan, Daniel, Giri P. Subramanian, Soon-Jo Chung, and Fred Y. Hadaegh (2016). Swarm assignment and trajectory optimization using variable-swarm, distributed auction assignment and sequential convex programming. In: The International Journal of Robotics Research 35.10, pp. 1261–1285. Nagabandi, Anusha, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn (2018). Learning to adapt in dynamic, realworld environments through meta-reinforcement learning. In: arXiv preprint arXiv:1803.11347. Nakanishi, J., J.A. Farrell, and S. Schaal (Sept. 2002). A locally weighted learning composite adaptive controller with structure adaptation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems. Vol. 1, 882–889 vol.1. doi: 10.1109/IRDS.2002.1041502. Nakka, Yashwanth Kumar, Anqi Liu, Guanya Shi, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung (2021). Chance-constrained trajectory optimization for safe exploration and learning of nonlinear Systems. In: IEEE Robotics and Automation Letters 6.2, pp. 389–396. doi: 10.1109/LRA.2020.3044033. Narendra, Kumpati S. and Anuradha M. Annaswamy (2012). Stable adaptive systems. Courier Corporation. Narendra, Kumpati S. and Snehasis Mukhopadhyay (1997). Adaptive control using neural networks and approximate models. In: IEEE Transactions on Neural Networks 8.3, pp. 475–485. O’Connell, Michael, Guanya Shi, Xichen Shi, and Soon-Jo Chung (2021). Metalearning-based robust adaptive flight control under uncertain wind conditions. In: arXiv preprint arXiv:2103.01932. url: https://arxiv.org/abs/2103. 01932. Paszke, Adam et al. (2019). PyTorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8024– 8035. url: http://papers.nips.cc/paper/9015-pytorch-an-imperativestyle-high-performance-deep-learning-library. Peng, Matt, Banghua Zhu, and Jiantao Jiao (2021). Linear representation metareinforcement learning for instant adaptation. In: arXiv preprint arXiv:2101.04750. Pravitra, Jintasit, Kasey A. Ackerman, Chengyu Cao, Naira Hovakimyan, and Evangelos A. Theodorou (Oct. 2020). L1-adaptive MPPI architecture for robust and agile control of multirotors. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). ISSN: 2153-0866, pp. 7661–7666. doi: 10.1109/IROS45743.2020.9341154. Rahimi, Ali and Benjamin Recht (2007). Random features for large-scale kernel machines. In: Proceedings of the 20th International Conference on Neural Information Processing Systems, pp. 1177–1184. 115 Ramos, Fabio, Rafael Carvalhaes Possas, and Dieter Fox (2019). Bayessim: Adaptive domain randomization via probabilistic inference for robotics simulators. In: arXiv preprint arXiv:1906.01728. Richards, Spencer M., Navid Azizan, Jean-Jacques E. Slotine, and Marco Pavone (2021). Adaptive-control-oriented meta-learning for nonlinear systems. In: arXiv preprint arXiv:2103.04490. Shi, Guanya, Wolfgang Hönig, Xichen Shi, Yisong Yue, and Soon-Jo Chung (2022). Neural-Swarm2: Planning and control of heterogeneous multirotor swarms using learned interactions. In: IEEE Transactions on Robotics 38.2, pp. 1063–1079. doi: 10.1109/TRO.2021.3098436. Shi, Guanya, Xichen Shi, Michael O’Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung (2019). Neural Lander: Stable drone landing control using learned dynamics. In: 2019 International Conference on Robotics and Automation (ICRA). IEEE, pp. 9784–9790. doi: 10.1109/ICRA.2019.8794351. Shi, Xichen, Kyunam Kim, Salar Rahili, and Soon-Jo Chung (2018). Nonlinear control of autonomous flying cars with wings and distributed electric propulsion. In: 2018 IEEE Conference on Decision and Control (CDC). IEEE, pp. 5326– 5333. Shi, Xichen, Patrick Spieler, Ellande Tang, Elena-Sorina Lupu, Phillip Tokumaru, and Soon-Jo Chung (May 2020). Adaptive nonlinear control of fixed-wing VTOL with airflow vector sensing. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). Paris, France: IEEE, pp. 5321–5327. isbn: 978-172817-395-5. doi: 10 . 1109 / ICRA40945 . 2020 . 9197344. url: https : / / ieeexplore.ieee.org/document/9197344/. Slotine, Jean-Jacques E. and Weiping Li (1991). Applied nonlinear control. Vol. 199. 1. Prentice Hall, Englewood Cliffs, NJ. Song, Xingyou, Yuxiang Yang, Krzysztof Choromanski, Ken Caluwaerts, Wenbo Gao, Chelsea Finn, and Jie Tan (2020). Rapidly adaptable legged robots via evolutionary meta-learning. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 3769–3776. Tal, Ezra and Sertac Karaman (May 2021). Accurate tracking of aggressive quadrotor trajectories using incremental nonlinear dynamic inversion and differential flatness. In: IEEE Transactions on Control Systems Technology 29.3, pp. 1203– 1218. issn: 1558-0865. doi: 10.1109/TCST.2020.3001117. Tobin, Josh, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel (2017). Domain randomization for transferring deep neural networks from simulation to the real world. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 23–30. 116 Torrente, Guillem, Elia Kaufmann, Philipp Föhn, and Davide Scaramuzza (2021). Data-driven MPC for quadrotors. In: IEEE Robotics and Automation Letters 6.2, pp. 3769–3776. Trefethen, Lloyd (2017). Multivariate polynomial approximation in the hypercube. In: Proceedings of the American Mathematical Society 145.11, pp. 4837–4844. Ventura Diaz, Patricia and Steven Yoon (2018). High-fidelity computational aerodynamics of multi-rotor unmanned aerial vehicles. In: 2018 AIAA Aerospace Sciences Meeting, p. 1266. Wise, Kevin A., Eugene Lavretsky, and Naira Hovakimyan (2006). Adaptive control of flight: Theory, applications, and open problems. In: 2006 American Control Conference. Yarotsky, Dmitry (2017). Error bounds for approximations with deep ReLU networks. In: Neural Networks 94, pp. 103–114. 117 Chapter 6 SAFE EXPLORATION Figure 6.1: Motivating example: drone landing trajectories with different speeds. In Chapter 2, we introduced a general mixed robot dynamics model 𝑀 (𝑞) 𝑞¥ + 𝐶 (𝑞, 𝑞) ¤ 𝑞¤ + 𝑔(𝑞) = 𝐵𝑢 + 𝑓 where 𝑓 is the unknown dynamics. In Chapters 3 to 5, we discussed different settings for 𝑓 : 𝑓 depends on 𝑢, 𝑓 considers multi-agent interactions, and 𝑓 depends on time-variant environmental conditions. However, all these settings rely on pre-collected data to train or pre-train deep learning models. In general, data collection for learning robotic agility needs either human supervision or fine-tuned programs in well-controlled environments. For example, in NeuralLander (Chapter 3) an expert human pilot flied the drone for 5 min to collect data where the drone was flying aggressively and very close to the ground. In this chapter, we will discuss several key principles to safe learn an agile controller under uncertainty from scratch, i.e., without any pre-collected data. Figure 6.1 depicts an example, where the drone must start from extremely slow but safe landing, explore and learn new skills itself, and eventually achieve fast and safe landing. In this process, we must avoid catastrophic failure such as the fast and unsafe landing curve in Fig. 6.1. This chapter is mainly based on the following papers: Liu, Anqi, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue (2020). Robust regression for safe exploration in control. In: Learning for Dynamics and Control. PMLR, pp. 608–619. url: https://proceedings.mlr. press/v120/liu20a.html. 118 Nakka, Yashwanth Kumar, Anqi Liu, Guanya Shi, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung (2021). Chance-constrained trajectory optimization for safe exploration and learning of nonlinear Systems. In: IEEE Robotics and Automation Letters 6.2, pp. 389–396. doi: 10.1109/LRA.2020.3044033. Abstract. We study the problem of safe learning and exploration in sequential control problems. The goal is to safely collect data samples from operating in an environment, in order to learn to achieve a challenging control goal (e.g., an agile maneuver close to a boundary). A central challenge in this setting is how to quantify uncertainty in order to choose provably-safe actions that allow us to collect informative data and reduce uncertainty, thereby achieving both improved controller safety and optimality. To address this challenge, we present a deep robust regression model that is trained to directly predict the uncertainty bounds for safe exploration. We derive generalization bounds for learning, and connect them with safety and stability bounds in control. We demonstrate empirically that our robust regression approach can outperform conventional Gaussian process (GP) based safe exploration in settings where it is difficult to specify a good GP prior. 6.1 Introduction and Related Work A key challenge in data-driven design for robotic controllers (e.g., Chapters 3 to 5) is automatically and safely collecting training data. Consider safely landing a drone at fast landing speeds (beyond a human expert’s piloting abilities). The dynamics are both highly non-linear and poorly modeled as the drone approaches the ground, but such dynamics can be learnable given the appropriate training data (see Chapter 3). To collect such data autonomously, one must guarantee safety while operating in the environment, which is the problem of safe exploration. In the drone landing example, collecting informative training data requires the drone to land increasingly faster while not crashing. Figure 6.1 depicts an example, where the goal is to learn the most aggressive yet safe trajectory (orange), while not being overconfident and execute trajectories that crash (green); the initial nominal controller may only be able to execute very conservative trajectories (blue). In order to safely collect such informative training data, we need to overcome two difficulties. First, we must quantify the learning errors in out-of-sample data. Every step of data collection creates a shift in the training data distribution. More specifically, our setting is an instance of covariate shift, where the underlying true 119 physics stay constant, but the sampling of the state space is biased by the data collection (Chen et al., 2016). In order to leverage modern learning approaches, such as deep learning, we must reason about the impact of covariate shift when predicting on states not well represented by the training set. Second, we must reason about how to guarantee safety and stability when controlling using the current learned model. Our ultimate goal is to control the dynamical system with desired properties but staying safe and stable while data collection. The imperfect dynamical model’s error translate to possible control error, which must be quantified and controlled. In this chapter, we propose a deep robust regression approach for safe exploration in model-based control. We view exploration as a data shift problem, i.e., the “test” data in the proposed exploratory trajectory comes from a shifted distribution compared to the training set. Our approach explicitly learns to quantify uncertainty under such covariate shift, which we use to learn robust dynamics models to quantify uncertainty of entire trajectories for safe exploration. We analyze learning performance from both generalization and data perturbation perspectives. We use our robust regression analysis to derive stability bounds for control performance when learning robust dynamics models, which is used for safe exploration. We empirically show that our approach outperforms conventional safe exploration approaches with much less tuning effort in two scenarios: (a) inverted pendulum trajectory tracking under wind disturbance; and (b) fast drone landing using an aerodynamics simulation based on real-world flight data from Chapter 3. Related Work in Safe Exploration Most approaches for safe exploration use Gaussian processes (GPs) to quantify uncertainty (Sui, Gotovos, et al., 2015; Sui, Zhuang, et al., 2018; Kirschner et al., 2019; Akametalu et al., 2014; Berkenkamp, Schoellig, et al., 2016; Turchetta et al., 2016; Wachi et al., 2018; Berkenkamp, Turchetta, et al., 2017; Fisac et al., 2018). These methods are related to bandit algorithms (Bubeck, Cesa-Bianchi, et al., 2012) and typically employ upper confidence bounds (Auer, 2002) to balance exploration versus exploitation (Srinivas et al., 2010). However, GPs are sensitive to model (i.e., the kernel) selection, and thus are often not suitable for tasks that aim to gradually reach boundaries of safety sets in a highly non-linear environment. In the high-dimensional case and under finite information, GPs suffer from bad priors even more severely (Owhadi et al., 2015). One could blend GP-based modeling with general function approximations (such as 120 deep learning) (Berkenkamp, Turchetta, et al., 2017; Cheng, Orosz, et al., 2019), but the resulting optimization-based control problem can be challenging to solve. Other approaches either require having a safety model pre-specified upfront (Alshiekh et al., 2018), are restricted to relatively simple models (Moldovan and Abbeel, 2012), have no convergence guarantees during learning (Taylor et al., 2019), or have no safety guarantees (Garcia and Fernández, 2012). Related Work in Distribution Shift The study of learning under distribution shift has seen increasing interest, owing to the widespread practical issue of distribution mismatch. Our work frames uncertainty quantification through the lens of covariate shift (Liu and Ziebart, 2014) with rigorous guarantees. Dealing with domain shift is a fundamental challenge in deep learning, as highlighted by their vulnerability to adversarial inputs (Goodfellow et al., 2014), and the implied lack of robustness. Beyond robust estimation, the typical approaches are to either regularize (Srivastava et al., 2014; Wager et al., 2013; Le et al., 2016; Bartlett et al., 2017; Miyato et al., 2018; Benjamin et al., 2019; Cheng, Verma, et al., 2019) or synthesize an augmented dataset that anticipates the domain shift (Prest et al., 2012; Zheng et al., 2016; Stewart and Ermon, 2017). We also utilize spectral normalization (see Chapter 2) in conjunction with robust estimation. Related Work in Robust and Adaptive Control Robust control (Zhou and Doyle, 1998) and adaptive control (see the related work section in Chapter 5) are two classical frameworks to handle uncertainties in the dynamics. GPs have been combined with nonlinear MPC for online adaptation and uncertainty estimation (Ostafew et al., 2016). However, robust control suffers from large uncertainty set, and it is hard to jointly analyse learning and control convergence under the framework of model reference adaptive control. For example, in Chapter 5 we need to assume the representation error from the learned model is bounded by 𝑑, to be able to show exponential convergence to a bounded error ball. However, how to quantify 𝑑 remains unclear. Ours is the first to explicitly consider covariate shift in learning dynamics. We pick the region to estimate uncertainty carefully and adapt the controller to track safe proposed trajectory in data collection. 6.2 Problem Statement To simplify the notation, we define the robot state as 𝑥 = [𝑞; 𝑞] ¤ and rewrite the robotic dynamics model in Chapter 2 as 121 𝑥¤ = 𝑓0 (𝑥, 𝑢) | {z } + nominal dynamics 𝑓 (𝑥, 𝑢) | {z } + unknown dynamics 𝜖 (𝑡) |{z} (6.1) bounded noise where 𝜖 is a noise term and we assume ∥𝜖 ∥ ≤ 𝜖. ¯ In this chapter, on top of the dynamics model introduced in Chapter 2, we also consider a cost function 𝑐(𝑥, 𝑢), a terminal cost function 𝑐 𝑓 (𝑥, 𝑢) and a safety set S. Our goal is to optimally move a robot from its start state 𝑥0 to its goal state 𝑥 𝑓 at the terminal time 𝑡 𝑓 , which can be framed as the following time-optimal control problem: ∫ 𝑡𝑓 min s.t.              𝑐(𝑥(𝑡), 𝑢(𝑡))𝑑𝑡 + 𝑐 𝑓 (𝑥(𝑡 𝑓 ), 𝑢(𝑡 𝑓 )) 𝐽= 𝑢(𝑡),𝑥(𝑡),𝑡 𝑓 (6.2) 0 dynamics in (6.1) 𝑥(𝑡) ∈ S, 𝑢(𝑡) ∈ U, ∀𝑡 𝑥(0) = 𝑥 0 , ∥𝑥(𝑡 𝑓 ) − 𝑥 𝑓 ∥ ≤ 𝛿 𝑓 where U is the control feasible set and 𝛿 𝑓 is a threshold for the goal reaching. As discussed in Chapter 4, solving Eq. (6.2) is very challenging because: (1) 𝑓 in the dynamics is unknown and highly nonlinear; (2) the whole program is typically highly non-convex and 𝑡 𝑓 is also a decision variable; and (3) the safety constraint 𝑥 ∈ S further complifies the problem. In this chapter, we address these challenges by considering an episodic learning and control setting, which can be summarized in Fig. 6.2. We discuss the three important modules in Fig. 6.2 (i.e., model learning and uncertainty quantification, safe planning, and safe control) as follows. model learning and uncertainty quantification data safe trajectory generation safe controller to collect new data the next episode Figure 6.2: Episodic learning and control diagram. Model learning and uncertainty quantification under domain shift. The goal for the learning module in Fig. 6.2 is to estimate the residual unknown dynamics 𝑓 (𝑥, 𝑢) in a way that admits rigorous uncertainty estimates for safety certification. 122 The key challenge is that the training data (from previous episodes or trajectories) and test data (the next episode) are not sampled from the same distribution, which can be framed as covariate shift (Shimodaira, 2000). Generally speaking, covariate shift refers to distribution shift caused by the input variables 𝑃(𝑥, 𝑢), while keeping 𝑃(𝑦 = 𝑓 (𝑥, 𝑢)|𝑥, 𝑢) fixed. In our motivating safe landing example, there is a universal “true” aerodynamics model, but we typically only observe training data from a limited source data distribution 𝑃src (𝑥, 𝑢). Certifying safety of a proposed trajectory will inevitably cover (𝑥, 𝑢) that are not well-represented by the training, i.e., data from a target data distribution 𝑃trg (𝑥, 𝑢). In other words, the distribution of (𝑥, 𝑢) in a proposed trajectory (for the next episode) does not match the distribution of (𝑥, 𝑢) in the training data. The output of the learning module includes an estimate of the unknown dynamics, 𝑓ˆ(𝑥, 𝑢), and the uncertainty quantification of 𝑓ˆ(𝑥, 𝑢), i.e., bounding the learning error 𝑑 (𝑥, 𝑢) = 𝑓ˆ(𝑥, 𝑢) − 𝑓 (𝑥, 𝑢). Note that such an uncertainty quantification could be either deterministic in the worst case or stochastic in expectation. Moreover, most importantly, the uncertainty quantification should be with respect to the target data distribution 𝑃trg (𝑥, 𝑢) rather than the source. Safe trajectory generation. Given the learned model 𝑓ˆ(𝑥, 𝑢) and the quantified uncertainty from the learning module, the next step is to optimize (or generate) a 𝑡𝑓 safe and feasible reference trajectory {𝑥 𝑑 (𝑡), 𝑢 𝑑 (𝑡)}𝑡=0 from solving Eq. (6.2). However, as discussed before, solving Eq. (6.2) is typically intractable, so prior works typically solve surrogates of Eq. (6.2), e.g., using sequential convex programming (SCP, see Chapter 4 for an example). Another type of surrogate approximately solves Eq. (6.2) by searching or sampling (e.g., see the AO-RRT method discussed in Chapter 4). Note that the true dynamics 𝑓 is unknown when solving Eq. (6.2). Therefore, we must use 𝑓ˆ and 𝑑 to solve a robust optimal control problem considering all possible 𝑓 given 𝑓ˆ and the learning error 𝑑. A typical technical assumption one needs to make to start the episodic process is that, the safe trajectory generation problem is feasible in the beginning. Namely, we need to assume that the aforementioned robust optimal control problem is feasible when 𝑓ˆ = 𝑓ˆ(0) and 𝑑 = 𝑑 (0) where 𝑓ˆ(0) is the initial guess of the unknown dynamics (typically set to be 0) and 𝑑 (0) = 𝑓ˆ(0) − 𝑓 is the initial uncertainty bound (typically depending on the boundedness of 𝑓 and its Lipschitz properties). We make the same 123 assumption in this chapter. It is not hard to show that the safe exploration problem is ill-posed without such an assumption. Safe controller design to collect new data. The safe trajectory generator outputs 𝑡𝑓 a safe reference trajectory {𝑥 𝑑 (𝑡), 𝑢 𝑑 (𝑡)}𝑡=0 . Taking 𝑢 𝑑 (𝑡) alone, one might expect that it is enough to directly execute the open-loop control signal 𝑢 𝑑 (𝑡) without any feedback. However, the open-loop trajectory 𝑢 𝑑 (𝑡) might be arbitrarily fragile and not guaranteed to be safe because of higher order model uncertainty not covered in Eq. (6.2) (this is thoroughly discussed in Åström and Murray (2021)). Therefore, we still need to design a closed-loop tracking controller to guarantee the closed-loop safety. Note that such a controller must also be robust to the model uncertainty 𝑑. For example, the nonlinear tracking controllers developed in Chapter 3 and the adaptive controllers developed in Chapter 5 are both robust and safe in the sense that the tracking error ∥𝑥 − 𝑥 𝑑 (𝑡)∥ will be proportionally bounded by ∥𝑑 ∥. Another example of safe controller is based on robust Control Barrier Functions (CBFs) (Ames et al., 2016; Cheng, Orosz, et al., 2019), which can be viewed as a “safety filter” on top of 𝑢 𝑑 (𝑡). Trade-offs between exploration and exploitation. The exploration-exploitation trade-off is one of the most important algorithmic considerations in bandits (Bubeck, Cesa-Bianchi, et al., 2012) and active learning (Srinivas et al., 2010). Similarly, such a trade-off is equally important in the episodic learning and control setting. Namely, 𝑡𝑓 {𝑥 𝑑 (𝑡), 𝑢 𝑑 (𝑡)}𝑡=0 is not always trying to minimize the cost function in Eq. (6.2) (i.e., 𝑡𝑓 exploitation). Often times, we need to design {𝑥 𝑑 (𝑡), 𝑢 𝑑 (𝑡)}𝑡=0 such that they can gather new information to reduce the uncertainty 𝑑 (i.e., exploration), rather than just minimizing the cost given the current information. For more in-depth discussions please see Nakka et al. (2021). End-to-end theoretical guarantees: consistency and convergence. In the episodic learning and control setting, we pursue end-to-end guarantees in the sense of consistency and convergence. Note that we assume the initial episode is feasible so we have an initial bounded cost 𝐽 (0) < ∞. We pursue the following two guarantees: (1) all episodes are safe (𝑥 𝑑 ∈ S and 𝑥 ∈ S); and (2) 𝐽 (𝑖+1) < 𝐽 (𝑖) , which is weaker than lim𝑖→∞ 𝐽 (𝑖) = 𝐽 ∗ where 𝐽 ∗ is the optimal value of Eq. (6.2) with the perfect knowledge of 𝑓 . To have as strong guarantees as lim𝑖→∞ 𝐽 (𝑖) = 𝐽 ∗ , we need much more assumptions regarding the problem structure (e.g., system is linear and cost functions are quadratic). 124 Stochastic variant of the optimal control problem. Equation (6.2) considers a deterministic optimal control problem. In Nakka et al. (2021), we also consider a stochastic variant, where 𝑥 is a stochastic random variable governed by a stochastic differential equation (SDE). The safety requirement is changed to a chance constraint Pr(𝑥 ∈ S) ≥ 1 − 𝛿 𝑠 . In this case, the goal is to generate a deterministic control signal 𝑢 𝑑 such that it minimizes the cost 𝐽 in expectation and also satisfies the chance constraint for safety. We will focus on the deterministic optimal control problem in this chapter for simplicity and refer to Nakka et al. (2021) for more details in the stochastic case. 6.3 Uncertainty Quantification under Domain Shift Recall that the key challenge of the learning module in Fig. 6.2 is that the training data (from previous episodes or trajectories) and test data (the next episode) are not sampled from the same distribution. More specifically, we observe training data from a source data distribution 𝑃src (𝑥, 𝑢), and we need to quantify the model uncertainty in a target data distribution 𝑃trg (𝑥, 𝑢). In this section, we will present a robust regression framework for uncertainty quantification under domain shift. General intuition. We use robust regression (Chen et al., 2016) to estimate the residual dynamics under covariate shift. Robust regression is derived from a minimax estimation framework, where the estimator 𝑃(𝑦|𝑥, 𝑢) tries to minimize a loss function L on target data distribution, and the adversary 𝑄(𝑦|𝑥, 𝑢) tries to maximize the loss under source data constraints Γ: min max L. (6.3) 𝑃(𝑦|𝑥,𝑢) 𝑄(𝑦|𝑥,𝑢)∈Γ Using the minimax framework, we achieve robustness to the worst-case possible conditional distribution that is “compatible" with finite training data if the estimator reaches the Nash equilibrium by minimizing a loss function defined on target data distribution. Technical design choices. Our derivation hinges on a choice of loss function L and constraint set for the adversary Γ, from which one can derive a formal objective, a learning algorithm, and an uncertainty bound. We use a relative loss function defined as the difference in conditional log-loss between an estimator 𝑃(𝑦|𝑥, 𝑢) and a baseline conditional distribution 𝑃0 (𝑦|𝑥, 𝑢) on the target data distribution 𝑃trg (𝑥, 𝑢)𝑃(𝑦|𝑥, 𝑢): 125  relative loss L := E𝑃trg (𝑥,𝑢)𝑄(𝑦|𝑥,𝑢)  𝑃(𝑦|𝑥, 𝑢) − log . 𝑃0 (𝑦|𝑥, 𝑢) (6.4) To construct the constraint set Γ, we utilize statistical properties of the source data distribution 𝑃src (𝑥, 𝑢):  Γ := 𝑄(𝑦|𝑥, 𝑢)||E𝑃src (𝑥,𝑢)𝑄(𝑦|𝑥,𝑢) [Φ(𝑥, 𝑢, 𝑦)] − c| ≤ 𝜆 (6.5) where Φ(𝑥, 𝑢, 𝑦) correspond to the sufficient statistics of the estimation task, and Í c = 𝑛1 𝑖𝑛 Φ(𝑥𝑖 , 𝑢𝑖 , 𝑦𝑖 ) is a vector of sample mean of statistics in the source data. This constraint means the adversary cannot choose a distribution whose sufficient statistics deviate too far from the collected training data. The consequence of the above choices is that the solution has a parametric form: 𝑃src ( 𝑥,𝑢) 𝑇 𝜃 Φ(𝑥,𝑢,𝑦) 𝑃(𝑦|𝑥, 𝑢) ∝ 𝑃0 (𝑦|𝑥, 𝑢)𝑒 𝑃trg ( 𝑥,𝑢) . (6.6) This form has two useful properties. First, it is straightforward to compute gradients on 𝜃 using only the training data. One can also train deep neural networks by treating Φ(𝑥, 𝑢, 𝑦) as the last hidden layer, i.e, we learn a representation of the sufficient statistics. Second, this form yields a concrete uncertainty bound that can be used to certify safety. For specific choices of 𝑃0 and Φ, the uncertainty is Gaussian distributed, which can be useful for many stochastic control approaches that assume Gaussian uncertainty (see Nakka et al. (2021)). Moreover, our method generalizes naturally to multidimensional output setting, where we predict a multidimensional Gaussian. The learning performance (i.e., the learning error 𝑑 (𝑥, 𝑢)) of robust regression approach can be analyzed from two perspectives: generalization error under covariant shift and perturbation error based on Lipschitz continuity. The generalization error reflects the expected error on a target distribution given certain function class, bounded distribution discrepancy, and base distribution. The perturbation error reflects the maximum error if target data deviates from training but stays in a Lipschitz ball. These error bounds are compatible with deep neural networks whose Rademacher complexity and Lipschitz constant can be controlled and measured (e.g., spectrally normalized neutral networks). 126 Generalization error bound (informal). In this case, we bound the generalization error on the target data:   E𝑃trg (𝑥,𝑢,𝑦) ∥ 𝑓 (𝑥, 𝑢) − 𝑓ˆ(𝑥, 𝑢)∥ 2 ≤ ( 𝑃trg (𝑥, 𝑢) ) · 𝐶1 𝑥,𝑢∼𝑃src (𝑥) 𝑃src (𝑥, 𝑢) sup (6.7) with probability 1 − 𝛿𝑔 . 𝐶1 is from standard statistical learning theory and depends on the Rademacher complexity, base distribution 𝑃0 , the number of data points, and 𝑃trg (𝑥,𝑢) log(1/𝛿𝑔 ). Most importantly, 𝐶1 is multiplied by the density ratio 𝑃src (𝑥,𝑢) , which highlights the extra challenge from domain shifts. We refer to Liu, Shi, et al. (2020) for the formal statement and proof. Perturbation error bound (informal). Alternatively, if we assume that target data samples in a ball B(𝛿 𝑏 ) with diameter 𝛿 𝑏 from the source data, we can also bound sup𝑥,𝑢∈B(𝛿𝑏 ) [∥ 𝑓 (𝑥, 𝑢) − 𝑓ˆ(𝑥, 𝑢)∥ 2 ] based on the Lipschitz properties of the true dynamics 𝑓 and the learner 𝑓ˆ. We refer to Liu, Shi, et al. (2020) for the formal statement and proof. 6.4 Safe Planning and Control In this chapter, we solve the planning problem with a finite pool of reference trajectories: {𝑥 𝑑,1 (𝑡), · · · , 𝑥 𝑑,𝑁 (𝑡)}. These trajectories are safe (i.e., 𝑥 𝑑 ∈ S) but with decreasing margins. For example, 𝑥 𝑑,1 is very conservative (far away from the boundary of S, e.g., the slow trajectory in Fig. 6.1) and 𝑥 𝑑,𝑁 is very aggressive (close to the boundary of S, e.g., the fast and safe trajectory in Fig. 6.1). With 𝑓ˆ(𝑥, 𝑢) and the learning error bound (i.e., the bound of ∥𝑑 (𝑥, 𝑢)∥ = ∥ 𝑓ˆ(𝑥, 𝑢) − 𝑓 (𝑥, 𝑢)∥), we can apply the learning-based controller developed in Chapter 3. For a reference trajectory candidate 𝑥 𝑑 , we consider its neighborhood 𝑃trg . Define 𝑑¯ = sup𝑥∈𝑃trg ∥𝑑 ∥ as the worst-case learning error in the target domain. Then the tracking controller will guarantee ∥𝑥 − 𝑥 𝑑 ∥ ≤ 𝐶2 · ( 𝑑¯ + 𝜖). ¯ Therefore, for trajectory generation, we select the most aggressive trajectory in the pool such that the worst case closed-loop behavior is still safe. Namely, we select 𝑥 𝑑,𝑖 such that 𝑥 𝑑,𝑖 plus a 𝐶2 · ( 𝑑¯ + 𝜖) ¯ error tube remains safe. This trajectory-pool-based method is simple, efficient, and commonly used in practice. However, as expected, it needs heuristics to design and typically yields a sub-optimal solution. Therefore, we solve the full optimal control problem in Nakka et al. (2021). 127 6.5 Simulations We conduct simulation experiments on the inverted pendulum and drone landing. We use kernel density estimation to estimate the density ratios. We demonstrate that our approach can reliably and safely converge to optimal behavior. We also compare with a Gaussian process (GP) version of the proposed algorithm. In general, we find it is difficult to tune the GP kernel parameters, especially in the multidimensional output cases. Both inverted pendulum and drone can be described by Eq. (2.1): 𝑀 (𝑞) 𝑞+𝐶 ¥ (𝑞, 𝑞) ¤ 𝑞+ ¤ 𝑔(𝑞) = 𝐵𝑢 + 𝑓 (𝑞, 𝑞). ¤ For pendulum, 𝑓 is the unknown drag torque from an analytic quadratic air drag model; for the drone landing problem, 𝑓 is represented as a DNN learned in Chapter 3. For pendulum the safety set S is defined such that the angle is within [−1.5, 1.5] radius; for drone it is defined such that the landing velocity is small than 1 m/s. Figure 6.3: Experimental results. Top: The pendulum task: (a)-(c) are the phase portraits of angle and angular velocity; Blue curve is tracking the desired trajectory with perfect knowledge of 𝑓 ; the worst-case possible trajectory is calculated according to the uncertainty bound; heatmap is the difference between predicted dynamics (the wind) and the ground truth; and (d) is the tracking error and the maximum density ratio. Bottom: The drone landing task: (e)-(g) are the phase portraits with height and velocity; heatmap is difference between the predicted ground effect) and the ground truth; (h) is the comparison with GPs in landing time. Main experimental results are shown in Fig. 6.3. Figure 6.3(a) to (c) and (e) to (g) demonstrate the exploration process with selected desired trajectories, worstcase tracking trajectory under current dynamics model, tracking trajectories with the ground truth unknown dynamics model, and actual tracking trajectories. In 128 both tasks, the algorithm selects conservative trajectories to guarantee safety at the beginning, and gradually is able to select more aggressive trajectories while staying the worst-case safety. We also demonstrate the decaying tracking error in each iteration for the pendulum task in Fig. 6.3(d). We validate that our density ratio is always bounded along the exploration. We examine the drone landing time in Fig. 6.3(h) and compare against multitask GP models (Bonilla et al., 2008) with both RBF kernel and Matern kernel. Our approach outperforms all GP models. Modeling the ground effect is notoriously challenging (see Chapter 3), and the GP suffers from model misspecification, especially in the multidimensional setting (Owhadi et al., 2015). Besides, GP models are also more computationally expensive than our method in making predictions. In contrast, our approach can fit general non-linear function estimators such as deep neural networks adaptively to the available data efficiently, which leads to more flexible inductive bias and better fitting of the data and uncertainty quantification. References Akametalu, Anayo K., Jaime F. Fisac, Jeremy H. Gillula, Shahab Kaynama, Melanie N. Zeilinger, and Claire J. Tomlin (2014). Reachability-based safe learning with Gaussian processes. In: 53rd IEEE Conference on Decision and Control. IEEE, pp. 1424–1431. Alshiekh, Mohammed, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu (2018). Safe reinforcement learning via shielding. In: Thirty-Second AAAI Conference on Artificial Intelligence. Ames, Aaron D., Xiangru Xu, Jessy W. Grizzle, and Paulo Tabuada (2016). Control barrier function based quadratic programs for safety critical systems. In: IEEE Transactions on Automatic Control 62.8, pp. 3861–3876. Åström, Karl Johan and Richard M. Murray (2021). Feedback systems: An introduction for scientists and engineers. Princeton University Press. Auer, Peter (2002). Using confidence bounds for exploitation-exploration trade-offs. In: Journal of Machine Learning Research 3.Nov, pp. 397–422. Bartlett, Peter L., Dylan J. Foster, and Matus J. Telgarsky (2017). Spectrallynormalized margin bounds for neural networks. In: Advances in Neural Information Processing Systems, pp. 6240–6249. Benjamin, Ari S., David Rolnick, and Konrad Kording (2019). Measuring and regularizing networks in function space. In: International Conference on Learning Representations (ICLR). Berkenkamp, Felix, Angela P. Schoellig, and Andreas Krause (2016). Safe controller optimization for quadrotors with Gaussian Processes. In: IEEE International 129 Conference on Robotics and Automation (ICRA), pp. 493–496. url: https : //arxiv.org/abs/1509.01066. Berkenkamp, Felix, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause (2017). Safe model-based reinforcement learning with stability guarantees. In: Neural Information Processing Systems (NIPS). url: https://arxiv.org/ abs/1705.08551. Bonilla, Edwin V., Kian M. Chai, and Christopher Williams (2008). Multi-task Gaussian process prediction. In: Advances in Neural Information Processing Systems, pp. 153–160. Bubeck, Sébastien, Nicolo Cesa-Bianchi, et al. (2012). Regret analysis of stochastic and nonstochastic multi-armed bandit problems. In: Foundations and Trends® in Machine Learning 5.1, pp. 1–122. Chen, Xiangli, Mathew Monfort, Anqi Liu, and Brian D. Ziebart (2016). Robust covariate shift regression. In: Artificial Intelligence and Statistics, pp. 1270–1279. Cheng, Richard, Gábor Orosz, Richard M. Murray, and Joel W. Burdick (2019). Endto-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In: Conference on Artificial Intelligence (AAAI). Cheng, Richard, Abhinav Verma, Gábor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick (2019). Control regularization for reduced variance reinforcement learning. In: International Conference on Machine Learning, pp. 1141–1150. url: http://proceedings.mlr.press/v97/cheng19a.html. Fisac, Jaime F., Anayo K. Akametalu, Melanie N. Zeilinger, Shahab Kaynama, Jeremy Gillula, and Claire J. Tomlin (2018). A general safety framework for learning-based control in uncertain robotic systems. In: IEEE Transactions on Automatic Control. Garcia, Javier and Fernando Fernández (2012). Safe exploration of state and action spaces in reinforcement learning. In: Journal of Artificial Intelligence Research 45, pp. 515–564. Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy (2014). Explaining and harnessing adversarial examples. In: arXiv preprint arXiv:1412.6572. Kirschner, Johannes, Mojmír Mutny, Nicole Hiller, Rasmus Ischebeck, and Andreas Krause (2019). Adaptive and safe Bayesian optimization in high dimensions via one-dimensional subspaces. In: International Conference on Machine Learning (ICML). Le, Hoang M., Andrew Kang, Yisong Yue, and Peter Carr (2016). Smooth imitation learning for online sequence prediction. In: International Conference on Machine Learning (ICML). 130 Liu, Anqi, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue (2020). Robust regression for safe exploration in control. In: Learning for Dynamics and Control. PMLR, pp. 608–619. url: https://proceedings.mlr. press/v120/liu20a.html. Liu, Anqi and Brian Ziebart (2014). Robust classification under sample selection bias. In: Advances in Neural Information Processing Systems, pp. 37–45. Miyato, Takeru, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida (2018). Spectral normalization for generative adversarial networks. In: arXiv preprint arXiv:1802.05957. Moldovan, Teodor Mihai and Pieter Abbeel (2012). Safe exploration in Markov decision processes. In: International Conference on Machine Learning (ICML). Nakka, Yashwanth Kumar, Anqi Liu, Guanya Shi, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung (2021). Chance-constrained trajectory optimization for safe exploration and learning of nonlinear Systems. In: IEEE Robotics and Automation Letters 6.2, pp. 389–396. doi: 10.1109/LRA.2020.3044033. Ostafew, Chris J., Angela P. Schoellig, and Timothy D. Barfoot (2016). Robust constrained learning-based NMPC enabling reliable mobile robot path tracking. In: The International Journal of Robotics Research 35.13, pp. 1547–1563. Owhadi, Houman, Clint Scovel, and Tim Sullivan (2015). On the brittleness of Bayesian inference. In: SIAM Review 57.4, pp. 566–582. Prest, Alessandro, Christian Leistner, Javier Civera, Cordelia Schmid, and Vittorio Ferrari (2012). Learning object class detectors from weakly annotated video. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp. 3282–3289. Shimodaira, Hidetoshi (2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. In: Journal of Statistical Planning and Inference 90.2, pp. 227–244. Srinivas, Niranjan, Andreas Krause, Sham M. Kakade, and Matthias Seeger (2010). Gaussian process optimization in the bandit setting: No regret and experimental design. In: International Conference on Machine Learning (ICML). Srivastava, Nitish, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov (2014). Dropout: A simple way to prevent neural networks from overfitting. In: The Journal of Machine Learning Research 15.1, pp. 1929–1958. Stewart, Russell and Stefano Ermon (2017). Label-free supervision of neural networks with physics and domain knowledge. In: Thirty-First AAAI Conference on Artificial Intelligence. Sui, Yanan, Alkis Gotovos, Joel Burdick, and Andreas Krause (2015). Safe exploration for optimization with Gaussian processes. In: International Conference on Machine Learning, pp. 997–1005. 131 Sui, Yanan, Vincent Zhuang, Joel W. Burdick, and Yisong Yue (2018). Stagewise safe Bayesian optimization with Gaussian processes. In: International Conference on Machine Learning (ICML). Taylor, Andrew J., Victor D. Dorobantu, Hoang M. Le, Yisong Yue, and Aaron D. Ames (2019). Episodic learning with control Lyapunov functions for uncertain robotic systems. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 6878–6884. doi: 10.1109/IROS40897.2019.8967820. url: https://doi.org/10.1109/IROS40897.2019.8967820. Turchetta, Matteo, Felix Berkenkamp, and Andreas Krause (2016). Safe exploration in finite Markov decision processes with Gaussian processes. In: Advances in Neural Information Processing Systems, pp. 4312–4320. Wachi, Akifumi, Yanan Sui, Yisong Yue, and Masahiro Ono (2018). Safe exploration and optimization of constrained mdps using Gaussian processes. In: Thirty-Second AAAI Conference on Artificial Intelligence. Wager, Stefan, Sida Wang, and Percy S. Liang (2013). Dropout training as adaptive regularization. In: Advances in Neural Information Processing Systems, pp. 351– 359. Zheng, Stephan, Yang Song, Thomas Leung, and Ian Goodfellow (2016). Improving the robustness of deep neural networks via stability training. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4480– 4488. Zhou, Kemin and John Comstock Doyle (1998). Essentials of robust control. Vol. 104. Prentice Hall Upper Saddle River, NJ. 132 Chapter 7 DISCUSSION AND FUTURE WORK In Part I, through Chapters 2 to 6, we have built a holistic view of the NeuralControl Family, which is a family of deep-learning-based nonlinear controllers with theoretical guarantees and learned new capabilities in robotics. See Table 2.1 for a summary of four main members in the family: Neural-Lander, Neural-Swarm, Neural-Fly, and Safe Exploration. In this chapter, we will conclude Part I by discussing several important future research directions. 7.1 Neural-Control for Other Robotic Systems In Neural-Lander (Chapter 3), Neural-Swarm (Chapter 4), and Neural-Fly (Chapter 5), we all considered a mixed robot dynamics model in Eq. (2.1): 𝑀 (𝑞) 𝑞¥ + 𝐶 (𝑞, 𝑞) ¤ 𝑞¤ + 𝑔(𝑞) = 𝐵𝑢 + 𝑓 (𝑞, 𝑞, ¤ 𝑢, 𝑡) | {z } unknown which is fully observable (we can directly observe 𝑞 and 𝑞) ¤ and fully actuated (rank(𝐵) = dim(𝑞) = 𝑛). In this section, we will discuss how to generalize the Neural-Control methodology to other systems. Partially Observable Systems By and large, learning and control with end-to-end theoretical guarantees in partially observable systems is very challenging due to the following fundamental challenges. Coupling between estimation and control. In linear time-invariant (LTI) systems, the well-known separation principle (Åström and Murray, 2021) suggests that we can safely decouple the design of the state estimator and the design of the feedback controller. Moreover, one can show that in a LQG problem (a LTI system with Gaussian noise and quadratic costs), an optimal Kalman filter plus an optimal LQR controller is jointly optimal (Åström and Murray, 2021). The separation principle is also widely applied in practice, and often yields satisfactory performance. For example, in the outdoor flight experiment in Chapter 5 (see Fig. 5.8), we first used a Kalman filter to estimate the drone state, and then use our Neural-Fly method for control by treating the estimated state as the true state. 133 However, generally speaking, for either time-variant or nonlinear systems, the separation principle is no longer true. That being said, a good estimator plus a good controller does not necessarily imply a good policy. In the worst case, the overall policy could even be unstable! Therefore, one must jointly analyze the estimation and control in a loop. The analysis becomes further involved if either of them is data-driven. Fundamental limits of robustness. To apply data-driven methods in partially observable systems, we must consider how learning error propagates through the system and how to guarantee robustness against such an error. For fully observable systems, we achieve this by designing exponentially stable feedback controllers, which naturally guarantees robustness against imperfect learning (e.g., Theorem 3.1 and Theorem 5.1). However, unfortunately, robustness in partially observable system raises a lot more fundamental limits. For example, John Doyle constructed a simple counter example in his well-known paper in 1978 (Doyle, 1978), to show that an optimal LQG controller could be arbitrarily fragile in the sense of having an arbitrarily small gain margin. Those fundamental limits suggest that, when designing learning and control method for partially observable systems, we might have to trade-off between optimality and robustness. Uncertainty of vision-based deep learning models. One interesting future direction is to apply Neural-Control in vision-based control settings, i.e., only with raw vision data. However, deep-learning-based computer vision methods are typically overconfident, especially under domain shift. For example, we show that state-of-the-art deeplearning-based object pose detectors are very sensitive to the training data and could be overconfident (Shi et al., 2021). In Fig. 7.1, we trained three different deep pose detectors with different simulation data under different randomization, and tested them in two real-world images for the milk object detection task. In the left image, these three models highly disagree with each other but in the right image (the only change is the light condition) they almost give identical detection results. Figure 7.1 suggests that when designing learning and control methods with vision input, we must carefully consider domain shift (especially sim-to-real shift) issues and quantify the uncertainty of the deep learning model in principled ways. 134 Figure 7.1: Detection results of the milk object of three deep learning models in two images. In the left image, these three models disagree much more than the right image. Under-Actuated Systems The fully-actuated structure in Eq. (2.1) has two major benefits in learning and control: (1) We can freely command 𝐵𝑢 in R𝑛 , which allows us to design a globally exponentially stable controller for the nominal system. (2) The unknown term 𝑓 becomes a matched uncertainty term. Namely, we can compensate (or cancel) 𝑓 via the certainty-equivalence principle. For under-actuated systems, in general we need more structures for both (1) and (2). For example, if we know a Control Lyapunov Function (CLF) a priori, we can leverage the CLF for stabilization (e.g., Taylor et al. (2019)) or adaptive control (e.g., Lopez and Slotine (2021)). Similarly, Westenbroek et al. (2019) designed a particular reinforcement learning method assuming the system is feedback linearizable with a known relative degree. However, those methods typically need strong prior knowledge (e.g., a CLF). Robotic Systems with Constraints Another interesting future direction is to consider constraints. Here we briefly discuss two types of constraints. Control input constraint considers 𝑢 ∈ U where U is the control input feasible set. Generally speaking, dealing with such a constraint needs multi-step planning (e.g., via MPC) or reachability analysis. Holonomic constraint augments Eq. (2.1) to 𝑀 (𝑞) 𝑞¥ + 𝐶 (𝑞, 𝑞) ¤ 𝑞¤ + 𝑔(𝑞) = 𝐵𝑢 + 𝑓 (𝑞, 𝑞, ¤ 𝑢, 𝑡) +𝐽 ⊤ (𝑞)𝜆 | {z } unknown 𝐽 (𝑞) 𝑞¥ + 𝐽¤(𝑞) 𝑞¤ = 0 where 𝐽 is the Jacobian of the holonomic constraint and 𝜆 is the Lagrange multi- 135 plier (Murray et al., 2017). Such a system can be used to model legged robots and manipulators. 7.2 End-to-End Learning and Control Guarantees Our tracking error bounds in Theorem 3.1 and Theorem 5.1 assume known learning error bounds. More specifically, in Theorem 3.1 we need to know ∥ 𝑓ˆ − 𝑓 ∥ ∞ in a compact set, and in Theorem 5.1 we need to know that the learned representation 𝜙 can adapt sufficiently well in any new wind condition. One interesting and important future research direction is to establish an end-to-end guarantee, by quantitatively and directly bounding the tracking error as a function of the number of data points. However, this could be very challenging because: (1) Classic learning theory needs to assume i.i.d. data distribution in training and testing data, but the nature of dynamical systems yields neither identical nor independent data distribution. (2) Classic learning theory needs a non-vacuous nation of learning model capacity (e.g., coverage number or Rademacher complexity, see details in Vapnik (1999)), but modern neural networks are in general overparameterized, which makes classic generalization error bound more or less vacuous. By and large, end-to-end guarantees with “classic” learning method (e.g., least squares) in “simple” systems (e.g., LQR) are tractable, and have been attracting a lot of research attention recently (e.g., regret bounds for learning in LQR problems (Dean et al., 2020)). However, for nonlinear systems with “modern” learning methods, the analysis is much more complicated and in general more structured assumptions are required (e.g., we limited the spectral complexity of the DNN for end-to-end non-vacuous guarantees in the episodic setting in Chapter 6). 7.3 Model-Free and Model-Based Learning In Part I, we have been focusing on the model-based learning scheme, where we first learn the model 𝑓 , and then design a policy based on the learned model. The model-based methodology enjoys several properties: (1) It yields a clean interface between learning theory and control theory, which allows us to build a unified framework between control and learning. (2) Compared to the model-free approach, it is much easier to have rigorous safety and robustness guarantees. (3) Often times, model-based methods are more data-efficient and interpretable (Kaiser et al., 2019). However, compared to the model-based approach, model-free policy learning is potentially more flexible and more globally optimal. One future direction is to combine 136 model-free policy learning and model-based methods. In particular, one promising direction is encoding control-theoretic knowledge to model-free reinforcement learning. Most popular RL algorithms (e.g., TRPO, SAC) are universal for all tasks. In contrast, drastically different nonlinear control methods are developed for different systems/tasks, and their successes highly rely on structures inside these systems/tasks. It is important and interesting to encode these structures into RL algorithms in a principled manner. 7.4 Neurosymbolic Learning for Data-Efficient Decision Making Vision-based control Multi-robot coordination Neural-Lander/Fly 15K/36K data points Neural-Swarm 1.4M data points Symbolic structures from physics Human-robot interaction Neurosymbolic learning Figure 7.2: From Part I to future research (more data required from left to right). In Part I, we found that more data is needed as tasks get more involved (Fig. 7.2). Neural-Lander needs 15K data points (5-min flight) (Chapter 3); Moving to the multi-environment case, Neural-Fly needs 36K points (12-min) (Chapter 5); For the heterogeneous swarm case, Neural-Swarm needs 1.4M points (Chapter 4). One key observation is that having symbolic structures in black box learning is critical for learning in autonomous systems. For example, if we learn the full dynamics instead of the residual for drone landing, we need 1hour of data instead of 5min. Encoding permutation invariance and domain invariance also greatly improves sample efficiency for Neural-Swarm and Neural-Fly, respectively. However, these structures are from physics and relatively straightforward to discover. Neurosymbolic learning (Chaudhuri et al., 2021) is required when moving to more complex scenarios, including vision-based control, multi-robot coordination, and human-robot interaction (depicted in Fig. 7.2). For these settings, data is often very high-dimensional, and there is no clear symbolic prior such as the nominal dynamics in Neural-Lander. Therefore we need to develop a principled neurosymbolic learning framework that discovers symbolic priors from data and integrates these priors with control theory. For instance, in vision-based control, we can discover causal structures by learning a low-dimensional representation that causally relates to the control task, then integrating these structures with robust and optimal control 137 theory. Another example is multi-robot coordination with uncertainty: consider three drones carrying a payload in strong winds. Drones need to share common information (e.g., wind effect) and negotiate potential conflicts (e.g., which drone moves first). We need to disentangle the shared and conflict parts and incorporate them using game theory and hierarchical planning. 7.5 Other Residual Learning Methods In Part I, we have been focusing on residual learning in the dynamics level. Namely, we learn 𝑓unknown in 𝑥¤ = 𝑓known + 𝑓unknown . Note that there are at least two other residual learning methods: Action-level residual learning learns a residual action Δ𝑢 in 𝑢 = 𝑢 nominal + Δ𝑢. For example, Johannink et al. (2019) learns Δ𝑢 using deep RL methods on top of a manully designed 𝑢 nominal . Program-level residual learning aims to learn the residual information in a “program.” For example, Taylor et al. (2019) learns the projection of the residual dynamics on a Lyapunov function, rather than the residual dynamics itself; Amos et al. (2018) learns system parameters or cost functions via a differentiable MPC program; Zhou et al. (2017) learns reference signals as an add-on block for the nominal feedback controller. In the future, it will be very interesting to compare these different levels of residual learning and understand their trade-offs. References Amos, Brandon, Ivan Jimenez, Jacob Sacks, Byron Boots, and Zico Kolter (2018). Differentiable MPC for end-to-end planning and control. In: Advances in Neural Information Processing Systems 31. Åström, Karl Johan and Richard M. Murray (2021). Feedback systems: An introduction for scientists and engineers. Princeton University Press. Chaudhuri, Swarat, Kevin Ellis, Oleksandr Polozov, Rishabh Singh, Armando SolarLezama, Yisong Yue, et al. (2021). Neurosymbolic Programming. In: Foundations and Trends® in Programming Languages 7.3, pp. 158–243. Dean, Sarah, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu (2020). On the sample complexity of the linear quadratic regulator. In: Foundations of Computational Mathematics 20.4, pp. 633–679. Doyle, John C. (1978). Guaranteed margins for LQG regulators. In: IEEE Transactions on Automatic Control 23.4, pp. 756–757. 138 Johannink, Tobias et al. (2019). Residual reinforcement learning for robot control. In: IEEE International Conference on Robotics and Automation, pp. 6023–6029. doi: 10.1109/ICRA.2019.8794127. url: https://doi.org/10.1109/ ICRA.2019.8794127. Kaiser, Lukasz et al. (2019). Model-based reinforcement learning for Atari. In: arXiv preprint arXiv:1903.00374. Lopez, Brett T. and Jean-Jacques E. Slotine (2021). Universal adaptive control of nonlinear systems. In: IEEE Control Systems Letters 6, pp. 1826–1830. Murray, Richard M., Zexiang Li, and S. Shankar Sastry (Dec. 2017). A mathematical introduction to robotic manipulation. 1st ed. CRC Press. isbn: 978-1-315-136370. doi: 10.1201/9781315136370. url: https://www.taylorfrancis.com/ books/9781351469791. Shi, Guanya, Yifeng Zhu, Jonathan Tremblay, Stan Birchfield, Fabio Ramos, Animashree Anandkumar, and Yuke Zhu (2021). Fast uncertainty quantification for deep object pose estimation. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 5200–5207. doi: 10.1109/ICRA48506. 2021.9561483. Taylor, Andrew J., Victor D. Dorobantu, Hoang M. Le, Yisong Yue, and Aaron D. Ames (2019). Episodic learning with control Lyapunov functions for uncertain robotic systems. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 6878–6884. doi: 10.1109/IROS40897.2019.8967820. url: https://doi.org/10.1109/IROS40897.2019.8967820. Vapnik, Vladimir N. (1999). An overview of statistical learning theory. In: IEEE Transactions on Neural Networks 10.5, pp. 988–999. Westenbroek, Tyler, David Fridovich-Keil, Eric Mazumdar, Shreyas Arora, Valmik Prabhu, S. Shankar Sastry, and Claire J. Tomlin (2019). Feedback linearization for unknown systems via reinforcement learning. In: arXiv preprint arXiv:1910.13272. Zhou, Siqi, Mohamed K. Helwa, and Angela P. Schoellig (2017). Design of deep neural networks as add-on blocks for improving impromptu trajectory tracking. In: 2017 IEEE 56th Annual Conference on Decision and Control (CDC). IEEE, pp. 5201–5207. Part II Unifying Interfaces Between Learning and Control Theory 139 140 In this part of the thesis, we will discuss three unifying interfaces between learning and control theory. Such interfaces deepen the fundamental connections between the two fields, enable much more efficient translation, and bring new perspectives and algorithmic principles. In particular, Part II will focus on joint end-to-end learning and control theoretical guarantees in online learning and control problems. In Chapter 8, we will give an overview. In Chapters 9 to 11, we introduce the three interfaces, respectively. This part is mainly based on the following papers: Li, Tongxin, Ruixiao Yang, Guannan Qu, Guanya Shi, Chenkai Yu, Adam Wierman, and Steven Low (2022). Robustness and consistency in linear quadratic control with untrusted predictions. In: Proceedings of the ACM on Measurement and Analysis of Computing Systems 6.1, pp. 1–35. doi: 10.1145/3508038. Lin, Yiheng, Yang Hu, Guanya Shi, Haoyuan Sun, Guannan Qu, and Adam Wierman (2021). Perturbation-based regret analysis of predictive control in linear time varying systems. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 34. Curran Associates, Inc., pp. 5174–5185. url: https : / / arxiv.org/abs/2106.10497. Pan, Weici, Guanya Shi, Yiheng Lin, and Adam Wierman (2022). Online optimization with feedback delay and nonlinear switching cost. In: Proceedings of the ACM on Measurement and Analysis of Computing Systems 6.1, pp. 1–34. doi: 10.1145/3508037. Shi, Guanya, Kamyar Azizzadenesheli, Michael O’Connell, Soon-Jo Chung, and Yisong Yue (2021). Meta-adaptive nonlinear control: Theory and algorithms. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 34. Curran Associates, Inc., pp. 10013–10025. url: https://proceedings.neurips. cc / paper / 2021 / file / 52fc2aee802efbad698503d28ebd3a1f - Paper . pdf. Shi, Guanya, Yiheng Lin, Soon-Jo Chung, Yisong Yue, and Adam Wierman (2020). Online optimization with memory and competitive control. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 33. Curran Associates, Inc. url: https://arxiv.org/abs/2002.05318. Yu, Chenkai, Guanya Shi, Soon-Jo Chung, Yisong Yue, and Adam Wierman (2020). The power of predictions in online control. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 33. Curran Associates, Inc., pp. 1994– 2004. url: https : / / proceedings . neurips . cc / paper / 2020 / file / 155fa09596c7e18e50b58eb7e0c6ccb4-Paper.pdf. – (2022). Competitive control with delayed imperfect information. In: American Control Conference (ACC). url: https://arxiv.org/abs/2010.11637. 141 Chapter 8 OVERVIEW Notations. In Part II, ∥ · ∥ indicates the 2-norm of a vector or the induced 2-norm of a matrix. ∥ · ∥ 𝐹 denotes the Frobenius norm of a matrix and 𝜆 min (·) (𝜆 max (·)) denotes the minimum (maximum) eigenvalue of a real symmetric matrix. We use 𝜌(·) to denote the spectral radius of a matrix. vec(·) ∈ R𝑚𝑛 denotes the vectorization of a 𝑚 × 𝑛 matrix, and ⊗ denotes the Kronecker product. Finally, we use 𝑥 1:𝑡 to denote a sequence {𝑥 1 , 𝑥2 , · · · , 𝑥𝑡 }. The Importance of Building Interfaces Between Learning and Control Closed-loop stability Steady-state error … Control theoretician Sophistication of Control Theory 8.1 Adaptive nonlinear control Model predictive control … (Ch. 9) Meta-adaptive control (Ch. 10) Online optimization & control (Ch. 11) Online learning & MPC Deep representation learning No-regret online control … Sophistication of Learning Theory Algorithmic stability Sample complexity … Learning theoretician Figure 8.1: Motivations of building the interface between learning and control, and the organization of Part II. The motivation of Part II is depicted in Fig. 8.1. In y-axis, control theory becomes more and more sophisticated. For example, adaptive nonlinear control (Slotine and W. Li, 1991; Krstic et al., 1995) and model predictive control (Rawlings, 2000) have been heavily researched since the 1980s, because they can deal with highly nonlinear and time-variant systems. Typically, control theoreticians focus on concepts such as closed-loop stability, steady-state error, or robustness. 142 On the other hand, in x-axis, learning theory (in particular, for decision making) also gets more and more complicated. For example, online learning and control with no-regret guarantees (a learning theory concept) has attracted a lot of attention since 2011 (Abbasi-Yadkori and Szepesvári, 2011). Generally speaking, learning theoreticians are interested in pursuing learning-theoretic guarantees such as algorithmic stability or sample complexity. However, it is worth noting that the top left of Fig. 8.1 has limited learning theory. For instance, most adaptive nonlinear control methods focus on linear parametric uncertainty with known basis functions (see the introduction of Chapter 5). The nonparametric-uncertainty case is still very challenging under the framework of model reference adaptive control (MRAC, Åström and Wittenmark (2013)). Combining meta-learning or representation learning and adaptive control for the non-parametric case is a promising direction with recently developed exciting algorithms and experimental results (e.g., Chapter 5 and Richards et al. (2021)), but there is no joint or end-to-end learning and control theoretical analysis. Similarly, the bottom right of Fig. 8.1 has limited control theory. For example, most no-regret online control research focuses on linear time-invariant systems such as LQR with classic static regret guarantees. However, most real robotic systems are highly nonlinear or time-variant (e.g., considering the changing wind condition in Chapter 5). Since both learning and control perspectives have been developed to a mature level, now is the time to focus on the interface (i.e., the orange dashed line in Fig. 8.1) and deeply integrate learning and control theories. The benefit includes: Building interfaces deepens the fundamental connections and enables more efficient translation between the two fields. One of focuses in Part II is to study how to reconcile fairly fragmented analysis between the two fields. For example, for online control problems, what does exponentially stable control limply about online regret (a notion of algorithm stability)? In Chapter 11, we will show that the closed-loop exponential stability is the key reason behind MPC’s competitiveness (a even stronger guarantee in online learning than regret). In Chapter 9, we will show that exponential input-to-state stability (e-ISS) is the key to translate learningtheoretic bounds to control-theoretic bounds. These examples demonstrate that building interfaces can deepen understanding and translate rich results between the two fields. 143 Building interfaces brings new perspectives and new algorithms. Often times, interfaces between learning and control yield “learning-inspired control algorithms” or “control-inspired learning algorithms.” In Chapter 11, we analyze a well-known and very successful control algorithm, Model Predictive Control (MPC), from learning theory perspectives. Chapter 11 not only provides new learning-theoretic perspectives of MPC, but yields new algorithms, e.g., a new variant of MPC from a robustness-consistency trade-off perspective in online learning. Building interfaces also has educational values. As mentioned before, Model Predictive Control (MPC) has been one of the most successful methods in industrial control since the 1980s. However, many learning theorists are studying RL algorithms, but few are analyzing MPC and why it is so powerful. In particular, in his blog1, Prof. Ben Recht said: “So many theorists are spending a lot of time studying RL algorithms, but few in the ML community are analyzing MPC and why it’s so successful. We should rebalance our allocation of mental resources! . . . I’d urge the MPC crowd to connect more with the learning theory crowd to see if a common ground can be found to better understand how MPC works and how we might push its performance even farther.” Thus, building interfaces can find common ground for learning and control theory, and also provide unique educational values in the sense of diversifying researchers’ research visions and methodology. 8.2 Organization of Part II: Three Interfaces As shown in Fig. 8.1, in this thesis we will focus on the following three interfaces between online learning and control theory: Online meta-adaptive control (Chapter 9). In order to have rapidly adaptable autonomous systems operating in changing environments (e.g., varying wind conditions for drones), it is crucial to extract common representations from all environments. However, existing theoretical results focus on either representation/metalearning with i.i.d. data (i.e., no dynamics) or adaptive control in a single environment. Therefore, Chapter 9 proposes a novel meta-adaptive control framework, where meta-learning optimizes a representation shared by all environments. Then 1 Link: http://www.argmin.net/2020/06/29/tour-revisited/ 144 the representation is fine-tuned by adaptive nonlinear control in a low-dimensional space. Chapter 9 jointly analyzes outer-loop learning and inner-loop adaptive control under a unified framework, and provides the first end-to-end non-asymptotic guarantee for multi-task nonlinear control. Competitive online optimization and control (Chapter 10). The recent progress in learning-theoretic analyses for online control begs the question: What learningtheoretic guarantees do we need for real-world systems? Existing results focus on linear systems with classic no-regret guarantees. However, no-regret policies compare with the optimal static controller in a specific class (typically linear policy class), which could be arbitrarily suboptimal in real-world nonlinear or time-varying systems (see details in Chapter 10). Therefore, Chapter 10 builds interfaces between competitive online optimization and control, which uses stronger metrics beyond regret, i.e., competitive ratio and dynamic regret (competitive difference). Those metrics directly compare with the global optimum, thus naturally suitable for realworld systems. We have designed competitive policies in time-varying and nonlinear systems, via novel reductions from online optimization to control. Moreover, we show new fundamental limits via novel lower bounds, e.g., the impact of delay. Online learning perspectives on model predictive control (Chapter 11). Another critical question is begged in online learning and control: Do established control methods such as MPC have strong learning guarantees? To close this gap, Chapter 11 proves the first non-asymptotic guarantee for MPC, showing that MPC is nearoptimal in the sense of dynamic regret in online LQR control with predictable disturbance. Chapter 11 also extends to settings with delayed inexact predictions and LTV systems. These results found common ground for learning and control theory and imply fundamental algorithmic principles. 8.3 Preliminaries on Online Optimization and Learning In this section, we introduce some preliminaries on online optimization and learning, which will be heavily used in Part II. In online optimization and learning, an online player (or learner) iteratively makes decisions. Most importantly, at the time of each decision, the outcomes associated with the choices are unknown to the player. Formally, at time step 𝑡 ∈ [1, 𝑇], the online player picks an action 𝑥𝑡 in a convex action set K ⊂ R𝑑 and then a loss function 𝑓𝑡 : K → R is revealed. The player then incurs a loss of 𝑓𝑡 (𝑥𝑡 ). Finally, the player moves to the next round 𝑡 + 1 and makes a decision 𝑥𝑡+1 based on all previous information. Note that the losses 𝑓1:𝑇 can be 145 adversarially chosen, and even depend on the action taken by the decision maker. Convexity and Smoothness of Functions For analysis and algorithm design, the loss functions’ convexity and smoothness properties play a key role, so we recap related concepts. Definition 8.1 (Convex). A function 𝑓 : K → R is convex if for all 𝑥, 𝑦 in the relative interior of the domain of 𝑓 and 𝜆 ∈ (0, 1), we have 𝑓 (𝜆𝑥 + (1 − 𝜆)𝑦) ≤ 𝜆 𝑓 (𝑥) + (1 − 𝜆) 𝑓 (𝑦). Definition 8.2 (𝑚-strongly convex). A function 𝑓 : K → R is 𝑚-strongly convex with respect to a norm ∥ · ∥ if for all 𝑥, 𝑦 in the relative interior of the domain of 𝑓 and 𝜆 ∈ (0, 1), we have 𝑓 (𝜆𝑥 + (1 − 𝜆)𝑦) ≤ 𝜆 𝑓 (𝑥) + (1 − 𝜆) 𝑓 (𝑦) − 𝑚 𝜆(1 − 𝜆)∥𝑥 − 𝑦∥ 2 . 2 Intuitively, the 𝑚-strongly convex property requires extra “curvature” on top of the standard convexity. There are three commonly used equivalent conditions: • 𝑓 − 𝑚2 ∥𝑥∥ 2 is convex. • (when 𝑓 is differentiable) For all 𝑥, 𝑦 we have 𝑓 (𝑦) ≥ 𝑓 (𝑥) + ⟨∇ 𝑓 (𝑥), 𝑦 − 𝑥⟩ + 𝑚 2 2 ∥𝑦 − 𝑥∥ . • (when 𝑓 is twice differentiable) The Hessian is uniformly lower bounded by ∇2 𝑓 (𝑥) ⪰ 𝑚𝐼. Definition 8.3 (𝑙-strongly smooth). A function 𝑓 : K → R is 𝑙-strongly smooth with respect to a norm ∥ · ∥ if 𝑓 is everywhere differentiable and if for all 𝑥, 𝑦 we have 𝑙 𝑓 (𝑦) ≤ 𝑓 (𝑥) + ⟨∇ 𝑓 (𝑥), 𝑦 − 𝑥⟩ + ∥𝑦 − 𝑥∥ 2 . 2 Intuitively, compared to the second equivalent condition of the 𝑚-strongly convex property, the 𝑙-strongly smooth property is “limiting” the curvature of 𝑓 . Similarly, if 𝑓 is twice differentiable, 𝑓 is 𝑙-strongly smooth if and only if 𝑙 𝐼 ⪰ ∇2 𝑓 (𝑥). We say a function is well-conditioned if it is simultaneously strongly convex and strongly smooth. For example, a quadratic function 𝑓 (𝑥) = 12 𝑥 ⊤ 𝑄𝑥 is 𝜆 min (𝑄)strongly convex and 𝜆 max (𝑄)-strongly smooth. 146 Metrics Used in Online Learning Regret. A commonly used performance metric in online learning is (static) regret. The static regret of an online algorithm is defined by regret = 𝑇 ∑︁ 𝑓𝑡 (𝑥𝑡 ) − min ∗ 𝑇 ∑︁ 𝑓𝑡 (𝑥 ∗ ). 𝑥 ∈K 𝑡=1 (8.1) 𝑡=1 Using this metric, an algorithm performs well if its regret is sublinear as a function of 𝑇, i.e. regret = 𝑜(𝑇), because this implies that the algorithm performs as well as the best fixed strategy in hindsight on the average. Dynamic regret (competitive difference). Although the classic regret is the predominant metric used in previous work, it cannot fully capture the dynamic nature Í of 𝑓1:𝑇 . In particular, min𝑥 ∗ ∈K 𝑇𝑡=1 𝑓𝑡 (𝑥 ∗ ) could be a weak benchmark if the minimizer of 𝑓𝑡 is highly time-variant. For example, for robotic control applications, 𝑓𝑡 depends on some time-variant environmental conditions (e.g., the wind condition in Chapter 5), and then the optimal policy at each time step is also time-variant. Therefore, we also consider an alternative metric called dynamic regret (also as known as competitive difference): dynamic regret = 𝑇 ∑︁ 𝑓𝑡 (𝑥𝑡 ) − 𝑇 ∑︁ min 𝑓𝑡 (𝑥) (8.2) 𝑥∈K 𝑡=1 𝑡=1 which is the difference between the online algorithm cost and the offline optimal cost with the full knowledge of 𝑓1:𝑇 . Intuitively, pursuing sublinear convergence is too good to be true using such a strong metric, so typically the goal is to bound the dynamic regret as a function of path length, which measures the variation of 𝑓1:𝑇 (e.g., Y. Li et al. (2019)). Competitive ratio. As mentioned before, dynamic regret analysis typically offers problem-dependent bounds (i.e., depending on the path length). Therefore, we also consider a even stronger variant called competitive ratio: Í𝑇 𝑓𝑡 (𝑥𝑡 ) (8.3) competitive ratio = Í𝑇 𝑡=1 𝑡=1 min𝑥∈K 𝑓𝑡 (𝑥) which studies the ratio between the online algorithm cost and the offline optimal cost. Typically the theoretical goal is to have competitive ratios bounded by a constant (e.g., Lin et al. (2021)). 147 Online Gradient Descent (OGD) and the Regret Analysis In this subsection, we introduce a natural and canonical algorithm in online optimization, Online Gradient Descent (OGD), and analyze its regret. The OGD algorithm runs as follows: it initializes 𝑥 1 ∈ K. At time step 𝑡, it plays 𝑥𝑡 , observes the cost function 𝑓𝑡 , and then updates 𝑥𝑡+1 by 𝑥𝑡+1 = ΠK (𝑥𝑡 − 𝜂𝑡 ∇ 𝑓𝑡 (𝑥𝑡 )) where ΠK is the projection onto K, i.e., ΠK (𝑦) = arg min𝑥∈K ∥𝑥 − 𝑦∥. Namely, OGD makes the decision by a projected gradient descent step with a time-variant learning rate 𝜂𝑡 . We have the following well-known guarantee: Lemma 8.1 (Regret of OGD). Suppose 𝑓1:𝑇 (𝑥) is a sequence of differentiable convex cost functions from R𝑛 to R, and K is a convex set in R𝑛 with diameter 𝐷, i.e., ∀𝑥 1 , 𝑥2 ∈ K, ∥𝑥 1 − 𝑥 2 ∥ ≤ 𝐷. We denote by 𝐺 > 0 an upper bound on the norm of the gradients of 𝑓1:𝑇 over K, i.e., ∥∇ 𝑓𝑡 (𝑥)∥ ≤ 𝐺 for all 𝑡 ∈ [1, 𝑇] and 𝑥 ∈ K. Then OGD with learning rates {𝜂𝑡 = 𝐺𝐷√𝑡 } guarantees the following: 𝑇 ∑︁ 𝑓𝑡 (𝑥𝑡 ) − min ∗ 𝑇 ∑︁ 𝑓𝑡 (𝑥 ∗ ) ≤ 𝑥 ∈K 𝑡=1 𝑡=1 √ 3 𝐺𝐷 𝑇. 2 (8.4) The proof can be found in the Chapter 3.1 of Hazan (2019). Lower bounds. Perhaps surprisingly, the OGD algorithm attains a tight regret bound up to a small constant factor in the worst case. Formally speaking, one can construct a counterexample to prove that any online algorithm for online optimization with √ convex cost functions incurs a 𝐺 𝐷 𝑇 regret in the worst case. √ Logarithmic regret with strongly convex functions. Even though the 𝑇 dependence is unavoidable, it is possible to have much better regret bounds if the cost function has strong curvature, namely, being strongly convex. One can show that the same OGD algorithm with a different learning rate scheduling can achieve a logarithmic regret 𝑂 (log 𝑇) when 𝑓1:𝑇 is strongly convex. 8.4 Preliminaries on Online Optimal Control Similar to online learning and optimization, in online optimal control, an online policy (or agent) iteratively makes actions, and the goal is to minimize a sequence of cost functions. However, the main difference and extra challenge is from the underlying dynamics. Namely, in online optimal control, the policy can only decide 148 the action 𝑢 𝑡 instead of the state. The state 𝑥𝑡+1 is a (deterministic or stochastic) function of 𝑥𝑡 and 𝑢 𝑡 . Instead, in online optimization the player can directly select 𝑥𝑡 . Therefore, generally speaking, non-asymptotic optimality guarantees (e.g., regret, dynamic regret, competitive ratio) in online optimal control are much more challenging to obtain, and most existing optimal control theoretical results focus on stability, robustness, and asymptotic convergence rather than those learningtheoretic non-asymptotic guarantees. In this section, we briefly introduce two commonly studied optimal control problems: LQR and MPC. For more details, we refer to Anderson and Moore (2007), Kirk (2004), and Camacho and Alba (2013). Linear Quadratic Regulator (LQR) Here we consider a generalized stochastic version of the classic LQR problem. In particular, we consider a linear system initialized with 𝑥 0 ∈ R𝑛 and controlled by 𝑢 𝑡 ∈ R𝑚 , with dynamics 𝑥𝑡+1 = 𝐴𝑥𝑡 + 𝐵𝑢 𝑡 + 𝑤 𝑡 and cost 𝐽= 𝑇−1 ∑︁ (𝑥𝑡⊤ 𝑄𝑥𝑡 + 𝑢 𝑡⊤ 𝑅𝑢 𝑡 ) + 𝑥𝑇⊤ 𝑄 𝑓 𝑥𝑇 (8.5) 𝑡=0 where 𝑇 ≥ 1 is the total length of the control period, 𝑄 𝑓 is the terminal cost matrix, and 𝑤 0:𝑇−1 are i.i.d. with E[𝑤 𝑡 ] = 0, E[𝑤 𝑡 𝑤 𝑡⊤ ] = 𝑊. The goal of the controller is to minimize the expectation of the cost 𝐽 given 𝐴, 𝐵, 𝑄, 𝑅, 𝑄 𝑓 , 𝑥0 and 𝑊, i.e., min E𝑤 0:𝑇−1 [𝐽]. In the stochastic LQR problem, the controller choose 𝑢 𝑡 after knowing the current state 𝑥𝑡 but before knowing the current disturbance 𝑤 𝑡 . Or equivalently, at time step 𝑡 the controller knows 𝑥0 and 𝑤 0:𝑡−1 . Therefore, at time step 𝑡, we need to solve an optimal policy as a function of 𝑥0 and 𝑤 0:𝑡−1 , namely, 𝑢 𝑡∗ (𝑥 0 , 𝑤 0:𝑡−1 ). Now let us solve Eq. (8.5) via dynamic programming. Define 𝑉𝑡 (𝑧) as the optimal value function (or cost-to-go) from time step 𝑡 starting at 𝑥𝑡 = 𝑧. Namely: "𝑇−1 # ∑︁ ⊤ 𝑉𝑡 (𝑧) = min E (𝑥 𝜏⊤ 𝑄𝑥 𝜏 + 𝑢 ⊤ 𝜏 𝑅𝑢 𝜏 ) + 𝑥𝑇 𝑄 𝑓 𝑥𝑇 . 𝑢 𝑡:𝑇−1 𝜏=𝑡 Obviously we have 𝑉𝑇 (𝑧) = 𝑧 ⊤ 𝑄 𝑓 𝑧, and the optimal total cost is 𝑉0 (𝑥 0 ). Moreover, via dynamic programming, 𝑉𝑡 can be found by the following backward recursion 149 (for 𝑡 = 𝑇 − 1, · · · , 0):  𝑉𝑡 (𝑧) = 𝑧 ⊤ 𝑄𝑧 + min 𝑢 ⊤ 𝑅𝑢 + E𝑤 𝑡 [𝑉𝑡+1 ( 𝐴𝑧 + 𝐵𝑢 + 𝑤 𝑡 )] . 𝑢 After solving 𝑉0:𝑇 , the optimal policy at time step 𝑡 becomes  𝑢 𝑡∗ = arg min 𝑢 ⊤ 𝑅𝑢 + E𝑤 𝑡 [𝑉𝑡+1 ( 𝐴𝑧 + 𝐵𝑢 + 𝑤 𝑡 )] . 𝑢 Because the dynamics is linear and the cost is quadratic, one can show that the value function 𝑉𝑡 has a nice quadratic form 𝑉𝑡 (𝑧) = 𝑧⊤ 𝑃𝑡 𝑥𝑡 + 𝑞 𝑡 for 𝑡 = 0, · · · , 𝑇. And we have the following recursion: 𝑃𝑇 = 𝑄 𝑓 , 𝑞𝑇 = 0 𝑃𝑡 = 𝑄 + 𝐴⊤ 𝑃𝑡+1 𝐴 − 𝐴⊤ 𝑃𝐵(𝑅 + 𝐵⊤ 𝑃𝑡+1 𝐵) −1 𝐵⊤ 𝑃𝑡+1 𝐴 𝑞 𝑡 = 𝑞 𝑡+1 + Tr(𝑊 𝑃𝑡+1 ). Moreover, the optimal policy 𝑢 𝑡 is in a linear state feedback form: 𝑢 𝑡∗ = −𝐾𝑡 𝑥𝑡 where 𝐾𝑡 = (𝑅 + 𝐵⊤ 𝑃𝑡+1 𝐵) −1 𝐵⊤ 𝑃𝑡+1 𝐴. Finally, the optimal cost is 𝑉0 (𝑥 0 ) = 𝑥 0⊤ 𝑃0 𝑥 0 + 𝑞 0 = 𝑥0⊤ 𝑃0 𝑥 0 + 𝑇 ∑︁ Tr(𝑊 𝑃𝑡 ). 𝑡=1 Some remarks: • In classic (deterministic) LQR, 𝑊 = 0, so we have the same 𝑃𝑡 , 𝐾𝑡 while 𝑞 𝑡 = 0, ∀𝑡. Moreover, the optimal cost is 𝑥 0⊤ 𝑃0 𝑥 0 . • Due to the structure of the LQR system, the optimal policy 𝑢 𝑡∗ has a nice linear feedback form and it is independent of 𝑥 0 and 𝑊. • Infinite horizon. When 𝑇 → ∞, we choose to minimize an average cost, i.e., "𝑇−1 # ∑︁ 1 (𝑥𝑡⊤ 𝑄𝑥𝑡 + 𝑢 𝑡⊤ 𝑅𝑢 𝑡 ) + 𝑥𝑇⊤ 𝑄 𝑓 𝑥𝑇 . min lim E𝑤 0:𝑇−1 𝑇→∞ 𝑇 𝑡=0 In this case, the optimal average cost is given by Tr(𝑊 𝑃), where 𝑃 satisfies 𝑃 = 𝑄 + 𝐴⊤ 𝑃 𝐴 − 𝐴⊤ 𝑃𝐵(𝑅 + 𝐵⊤ 𝑃𝐵) −1 𝐵⊤ 𝑃 𝐴. (8.6) 150 Equation (8.6) is called discrete algebraic Riccati equation (DARE). Moreover, the optimal policy is a constant linear feedback policy 𝑢 𝑡∗ = −𝐾𝑥𝑡 where 𝐾 = (𝑅 + 𝐵⊤ 𝑃𝐵) −1 𝐵⊤ 𝑃 𝐴. • Predictions and delays. Note that in the standard LQR setting, at time step 𝑡 the policy knows 𝑥0 and 𝑤 0:𝑡−1 , i.e., knowing the current state 𝑥𝑡 but not knowing the current disturbance 𝑤 𝑡 . In Chapter 11, we will generalize it significantly to settings with predictions (knowing future 𝑤) and delays. • I.i.d. disturbance. The simple structure of the value function 𝑉𝑡 and the policy 𝑢 = −𝐾𝑡 𝑥 also relies on the fact that the disturbance 𝑤 is i.i.d. and zero-mean. In Chapter 11, we will also generalize it significantly to general stochastic disturbance and adversarial disturbance. Model Predictive Control (MPC) Since the 1980s, Model Predictive Control (MPC) has been one of the most influential and popular process control methods in industries. The key idea of MPC is straightforward: with a finite look-ahead window of the future, MPC optimizes a finite-time optimal control problem at each time step, but only implements/executes the current time slot and then optimizes again at the next time step, repeatedly. The second part “only implements the current time slot and reoptimizes at each time step” is one of the reasons MPC was not that popular before the 1980s, because iteratively solving complex optimal control problems at high frequency was such a luxury task before computational power took off. In this subsection, we use a trajectory tracking problem to explain how MPC works (visualized in Fig. 8.2). 𝑘 predictions unknown and adversarial robot trajectory 𝑥! desired trajectory Figure 8.2: A trajectory tracking problem to explain how MPC works. Suppose a robot is tracking a trajectory 𝑑1 , · · · , 𝑑𝑇 , and the robot’s dynamics follows 𝑥𝑡+1 = 𝑓𝑡 (𝑥𝑡 , 𝑢 𝑡 ) + 𝑤 𝑡 , 0 ≤ 𝑡 ≤ 𝑇 − 1 where 𝑥𝑡 , 𝑢 𝑡 , 𝑤 𝑡 are state, control input 151 and disturbance, respectively. At time step 𝑡, the cost the robot needs to pay is (𝑥𝑡 − 𝑑𝑡 ) ⊤ 𝑄(𝑥𝑡 − 𝑑𝑡 ) + 𝑢 𝑡⊤ 𝑅𝑢 𝑡 where 𝑄 penalizes the tracking error and 𝑅 penalizes the fuel cost (control energy). At each time step, the robot has 𝑘-step future predictions: 𝑑𝑡+1:𝑡+𝑘 (trajectory), 𝑓𝑡:𝑡+𝑘−1 (dynamics) and 𝑤 𝑡:𝑡+𝑘−1 (disturbance). At each time stpe, MPC solves the following 𝑘-step optimal control problem: min 𝑘 ∑︁ 𝑢 𝑡:𝑡+𝑘−1
⊤ (𝑥𝑡+𝑖 − 𝑑𝑡+𝑖 ) ⊤ 𝑄(𝑥𝑡+𝑖 − 𝑑𝑡+𝑖 ) + 𝑢 𝑡+𝑖−1 𝑅𝑢 𝑡+𝑖−1 + 𝑉 (𝑥𝑡+𝑘 ) 𝑖=1 s.t. 𝑥𝑡+𝑖+1 = 𝑓𝑡+𝑖 (𝑥𝑡+𝑖 , 𝑢 𝑡+𝑖 ) + 𝑤 𝑡+𝑖 , 𝑢 𝑡+𝑖 ∈ U, 𝑥𝑡+𝑖+1 ∈ X 0 ≤𝑖 ≤ 𝑘 −1 0 ≤𝑖 ≤ 𝑘 −1 where the terminal cost 𝑉 (·) regularizes the terminal state. Having a proper 𝑉 is critical for the stability and performance of MPC. Suppose the optimal solution ∗ , · · · , 𝑢∗ from the above optimization problem is 𝑢 𝑡|𝑡 . A key feature of MPC is 𝑡+𝑘−1|𝑡 ∗ is executed/used, and at the next step we need to that only the first solved action 𝑢 𝑡|𝑡 ∗ solve another optimization problem for 𝑢 𝑡+1|𝑡+1 . Why Beyond Regret in Online Control? We finish this section by discussing why we prefer metrics beyond regret in Part II. In Part II, we focus on non-asymptotic analyses of online control and learning problems. In Chapters 10 and 11, we use global optimality metrics beyond (static) regret, namely, dynamic regret and competitive ratio, as introduced in Section 8.3. The reader may wonder why we choose those two metrics rather than the classic regret, especially considering that no-regret online control has become an extremely popular research area since 2011. However, we will use the following example to show that we should consider metrics beyond regret in dynamic environments. … 𝑤( 𝑤$ 𝑤' 𝐽!"# 𝑢$∗ (𝑤$:' ) 𝑢(∗ (𝑤$:' ) 𝑢∗' (𝑤$:' ) 𝐽)∗∈+ 𝑢$ = 𝜋 ∗ (𝑤$:' ) 𝑢( = 𝜋 ∗ (𝑤$:' ) 𝑢 ' = 𝜋 ∗ (𝑤$:' ) Figure 8.3: An example illustrating why beyond regret. Figure 8.3 revisits the drone-flying-in-wind example in Chapter 5. At every time step, the drone experiences a new wind condition 𝑤 𝑡 . The first row in Fig. 8.3 is the benchmark we use in dynamic regret or competitive ratio, which compares 152 against the globally offline optimal policy’s cost 𝐽OPT given the full sequence 𝑤 1:𝑇 in hindsight. Note that the globally offline optimal policy might be highly nonlinear and adaptive. On the other hands, the second row in Fig. 8.3 is the benchmark we use in classic regret, which compares against a fixed and stationary offline optimal policy 𝜋 ∗ . Note that 𝜋 ∗ is in some known and fixed policy class Π. The predominant Π used in previous work is the linear controller class (e.g., Agarwal et al. (2019), Dean et al. (2020), and Cohen et al. (2018)). However, the cost of the optimal static policy 𝐽𝜋∗ ∈Π might be far from 𝐽OPT . In Chapter 10, we will show that this gap could be arbitrarily large even in simple 1-d systems. Therefore, achieving small regret may still mean having a significantly larger cost than optimal. Another reason we choose metrics beyond regret is that, it is in general intractable to design or characterize a reasonable comparator policy class Π in dynamic environments (i.e., time-variant systems). For example, MPC is exactly such a dynamic algorithm. If we use regret to analyze MPC, it is extremely hard (if not impossible) to define a reasonable comparator class Π. References Abbasi-Yadkori, Yasin and Csaba Szepesvári (2011). Regret bounds for the adaptive control of linear quadratic systems. In: Proceedings of the 24th Annual Conference on Learning Theory. JMLR Workshop and Conference Proceedings, pp. 1–26. Agarwal, Naman, Brian Bullins, Elad Hazan, Sham M. Kakade, and Karan Singh (2019). Online control with adversarial disturbances. In: International Conference on Machine Learning (ICML). Anderson, Brian D. O. and John B. Moore (2007). Optimal control: Linear quadratic methods. Courier Corporation. Åström, Karl J. and Björn Wittenmark (2013). Adaptive control. Courier Corporation. Camacho, Eduardo F. and Carlos Bordons Alba (2013). Model predictive control. Springer Science & Business Media. Cohen, Alon, Avinatan Hasidim, Tomer Koren, Nevena Lazic, Yishay Mansour, and Kunal Talwar (2018). Online linear quadratic control. In: International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 1029–1038. 153 Dean, Sarah, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu (2020). On the sample complexity of the linear quadratic regulator. In: Foundations of Computational Mathematics 20.4, pp. 633–679. Hazan, Elad (2019). Introduction to online convex optimization. In: arXiv preprint arXiv:1909.05207. Kirk, Donald E. (2004). Optimal control theory: An introduction. Courier Corporation. Krstic, Miroslav, Petar V. Kokotovic, and Ioannis Kanellakopoulos (1995). Nonlinear and adaptive control design. John Wiley & Sons, Inc. Li, Yingying, Xin Chen, and Na Li (2019). Online optimal control with linear dynamics and predictions: Algorithms and regret analysis. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 14858–14870. Lin, Yiheng, Yang Hu, Guanya Shi, Haoyuan Sun, Guannan Qu, and Adam Wierman (2021). Perturbation-based regret analysis of predictive control in linear time varying systems. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 34. Curran Associates, Inc., pp. 5174–5185. url: https : / / arxiv.org/abs/2106.10497. Rawlings, James B. (2000). Tutorial overview of model predictive control. In: IEEE Control Systems Magazine 20.3, pp. 38–52. Richards, Spencer M., Navid Azizan, Jean-Jacques E. Slotine, and Marco Pavone (2021). Adaptive-control-oriented meta-learning for nonlinear systems. In: arXiv preprint arXiv:2103.04490. Slotine, Jean-Jacques E. and Weiping Li (1991). Applied nonlinear control. Vol. 199. 1. Prentice Hall, Englewood Cliffs, NJ. 154 Chapter 9 ONLINE META-ADAPTIVE CONTROL Meta-/Representation Learning Metaadaptive Control Adaptive (Nonlinear) Control Figure 9.1: The motivation of Chapter 9. In order to have rapidly adaptable autonomous systems operating in changing environments (e.g., varying wind conditions for drones in Chapter 5), it is crucial to extract common representations from all environments. However, as shown in Fig. 9.1, existing theoretical results focus on either representation/meta-learning with i.i.d. data (i.e., no dynamics) or adaptive control in a single environment. In this chapter, we propose a novel meta-adaptive control framework, where meta-learning optimizes a representation shared by all environments. Then the representation is fine-tuned by adaptive nonlinear control in a low-dimensional space. Chapter 9 jointly analyzes outer-loop learning and inner-loop adaptive control under a unified framework, and provides the first end-to-end non-asymptotic guarantee for multitask nonlinear control. This chapter is mainly based on the following paper1: Shi, Guanya, Kamyar Azizzadenesheli, Michael O’Connell, Soon-Jo Chung, and Yisong Yue (2021). Meta-adaptive nonlinear control: Theory and algorithms. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 34. Curran Associates, Inc., pp. 10013–10025. url: https://proceedings.neurips. cc / paper / 2021 / file / 52fc2aee802efbad698503d28ebd3a1f - Paper . pdf. Abstract. We present an online multi-task learning approach for adaptive nonlinear control, which we call Online Meta-Adaptive Control (OMAC). The goal is to con1 Code and video: https://github.com/GuanyaShi/Online-Meta-Adaptive-Control 155 trol a nonlinear system subject to adversarial disturbance and unknown environmentdependent nonlinear dynamics, under the assumption that the environment-dependent dynamics can be well captured with some shared representation. Our approach is motivated by robot control, where a robotic system encounters a sequence of new environmental conditions that it must quickly adapt to. A key emphasis is to integrate online representation learning with established methods from control theory, in order to arrive at a unified framework that yields both control-theoretic and learning-theoretic guarantees. We provide instantiations of our approach under varying conditions, leading to the first non-asymptotic end-to-end convergence guarantee for multi-task nonlinear control. OMAC can also be integrated with deep representation learning. Experiments show that OMAC significantly outperforms conventional adaptive control approaches which do not learn the shared representation, in inverted pendulum and 6-DoF drone control tasks under varying wind conditions. 9.1 Introduction One important goal in autonomy and artificial intelligence is to enable autonomous robots to learn from prior experience to quickly adapt to new tasks and environments. Examples abound in robotics, such as a drone flying in different wind conditions (Chapter 5), a manipulator throwing varying objects (Zeng et al., 2020), or a quadruped walking over changing terrains (Lee et al., 2020). Though those examples provide encouraging empirical evidence, when designing such adaptive systems, two important theoretical challenges arise, as discussed below. First, from a learning perspective, the system should be able to learn an “efficient” representation from prior tasks, thereby permitting faster future adaptation, which falls into the categories of representation learning or meta-learning. Recently, a line of work has shown theoretically that learning representations (in the standard supervised setting) can significantly reduce sample complexity on new tasks (Du et al., 2020; Tripuraneni, Jin, et al., 2020; Maurer et al., 2016). Empirically, deep representation learning or meta-learning has achieved success in many applications (Bengio et al., 2013), including control, in the context of meta-reinforcement learning (Gupta et al., 2018; Finn, Abbeel, et al., 2017; Nagabandi et al., 2018). However, theoretical benefits (in the end-to-end sense) of representation learning or meta-learning for adaptive control remain unclear. Second, from a control perspective, the agent should be able to handle parametric 156 model uncertainties with control-theoretic guarantees such as stability and tracking error convergence, which is a common adaptive control problem (Slotine and W. Li, 1991; Karl J. Åström and Wittenmark, 2013). For classic adaptive control algorithms, theoretical analysis often involves the use of Lyapunov stability and asymptotic convergence (Slotine and W. Li, 1991; Karl J. Åström and Wittenmark, 2013). Moreover, many recent studies aim to integrate ideas from learning, optimization, and control theory to design and analyze adaptive controllers using learning-theoretic metrics. Typical results guarantee non-asymptotic convergence in finite time horizons, such as regret (Boffi et al., 2021; Simchowitz and Foster, 2020; Dean et al., 2020; Kakade et al., 2020) and dynamic regret (Chapter 11 and Y. Li et al. (2019)). However, these results focus on a single environment or task. A multi-task extension, especially whether and how prior experience could benefit the adaptation in new tasks, remains an open problem. Main contributions. In this chapter, we address both learning and control challenges in a unified framework and provide end-to-end guarantees. We derive a new method of Online Meta-Adaptive Control (OMAC) that controls uncertain nonlinear systems under a sequence of new environmental conditions. The underlying assumption is that the environment-dependent unknown dynamics can well be captured by a shared representation, which OMAC learns using a meta-adapter. OMAC then performs environment-specific updates using an inner-adapter. We provide different instantiations of OMAC under varying assumptions and conditions. In the jointly and element-wise convex cases, we show sublinear cumulative control error bounds, which to our knowledge is the first non-asymptotic convergence result for multi-task nonlinear control. Compared to standard adaptive control approaches that do not have a meta-adapter, we show that OMAC possesses both stronger guarantees and empirical performance. We finally show how to integrate OMAC with deep representation learning, which further improves empirical performance. 9.2 Problem Statement We consider the setting where a controller encounters a sequence of 𝑁 environments, with each environment lasting 𝑇 time steps. We use outer iteration to refer to the iterating over the 𝑁 environments, and inner iteration to refer to the 𝑇 time steps within an environment. We use superscripts (e.g., (𝑖) in 𝑥𝑡(𝑖) ) to denote the index of the outer iteration where 1 ≤ 𝑖 ≤ 𝑁, and subscripts (e.g., 𝑡 in 𝑥𝑡(𝑖) ) to denote the time 157 index of the inner iteration where 1 ≤ 𝑡 ≤ 𝑇. We use step(𝑖, 𝑡) to refer to the inner time step 𝑡 at the 𝑖 th outer iteration. We consider a discrete-time nonlinear control-affine system (Murray et al., 2017; LaValle, 2006) with environment-dependent uncertainty 𝑓 (𝑥, 𝑐). The dynamics model at the 𝑖 th outer iteration is: (𝑖) = 𝑓0 (𝑥𝑡(𝑖) ) + 𝐵(𝑥𝑡(𝑖) )𝑢 𝑡(𝑖) − 𝑓 (𝑥𝑡(𝑖) , 𝑐 (𝑖) ) + 𝑤 𝑡(𝑖) , 𝑥𝑡+1 1 ≤ 𝑡 ≤ 𝑇, (9.1) where the state 𝑥𝑡(𝑖) ∈ R𝑛 , the control 𝑢 𝑡(𝑖) ∈ R𝑚 , 𝑓0 : R𝑛 → R𝑛 is a known nominal dynamics model, 𝐵 : R𝑛 → R𝑛×𝑚 is a known state-dependent actuation matrix, 𝑐 (𝑖) ⊂ Rℎ is the unknown parameter that indicates an environmental condition, 𝑓 : R𝑛 × Rℎ → R𝑛 is the unknown 𝑐 (𝑖) -dependent dynamics model, and 𝑤 𝑡(𝑖) is disturbance, potentially adversarial. For simplicity we define 𝐵𝑡(𝑖) = 𝐵(𝑥𝑡(𝑖) ) and 𝑓𝑡(𝑖) = 𝑓 (𝑥𝑡(𝑖) , 𝑐 (𝑖) ). Interaction protocol. We study the following adaptive nonlinear control problem under 𝑁 unknown environments. At the beginning of outer iteration 𝑖, the environment first selects 𝑐 (𝑖) (adaptively and adversarially), which is unknown to the (𝑖) controller, and then the controller makes decision 𝑢 1:𝑇 under unknown dynamics (𝑖) (𝑖) (𝑖) 𝑓 (𝑥𝑡 , 𝑐 ) and potentially adversarial disturbances 𝑤 𝑡 . To summarize: 1. Outer iteration 𝑖. A policy encounters environment 𝑖 (𝑖 ∈ {1, . . . , 𝑁 }), associated with unobserved variable 𝑐 (𝑖) (e.g., the wind condition for a flying drone). Run inner loop (Step 2). 2. Inner loop. Policy interacts with environment 𝑖 for 𝑇 time steps, observing 𝑥𝑡(𝑖) and taking action 𝑢 𝑡(𝑖) , with state/action dynamics following (9.1). 3. Policy optionally observes 𝑐 (𝑖) at the end of the inner loop (used for some variants of the analysis). 4. Increment 𝑖 = 𝑖 + 1 and repeat from Step 1. We use average control error (ACE) as our performance metric: Definition 9.1 (Average control error). The average control error (ACE) of 𝑁 outer iterations (i.e., 𝑁 environments) with each lasting 𝑇 time steps, is defined as ACE = (𝑖) 1 Í 𝑁 Í𝑇 𝑖=1 𝑡=1 ∥𝑥 𝑡 ∥. 𝑇𝑁 158 ACE can be viewed as a non-asymptotic generalization of steady-state error in control (Karl Johan Åström and Murray, 2021). We make the following assumptions on the actuation matrix 𝐵, the nominal dynamics 𝑓0 , and disturbance 𝑤 𝑡(𝑖) : Assumption 9.1 (Full actuation, bounded disturbance, and e-ISS assumptions). We consider fully-actuated systems, i.e., for all 𝑥, rank(𝐵(𝑥)) = 𝑛. ∥𝑤 𝑡(𝑖) ∥ ≤ 𝑊, ∀𝑡, 𝑖. Moreover, the nominal dynamics 𝑓0 is exponentially input-to-state stable (e-ISS): let constants 𝛽, 𝛾 ≥ 0 and 0 ≤ 𝜌 < 1. For a sequence 𝑣 1:𝑡−1 ∈ R𝑛 , consider the dynamics 𝑥 𝑘+1 = 𝑓0 (𝑥 𝑘 ) + 𝑣 𝑘 , 1 ≤ 𝑘 ≤ 𝑡 − 1. 𝑥𝑡 satisfies: ∥𝑥𝑡 ∥ ≤ 𝛽𝜌 𝑡−1 ∥𝑥 1 ∥ + 𝛾 𝑡−1 ∑︁ 𝜌 𝑡−1−𝑘 ∥𝑣 𝑘 ∥. (9.2) 𝑘=1 With the e-ISS property in Assumption 9.1, we have the following bound that connects ACE with the average squared loss between 𝐵𝑡(𝑖) 𝑢 𝑡(𝑖) + 𝑤 𝑡(𝑖) and 𝑓𝑡(𝑖) . Lemma 9.1 (Connecting learning error with control error). Assume 𝑥 1(𝑖) = 0, ∀𝑖. The average control error (ACE) is bounded as: √︄ Í𝑁 Í𝑇 Í𝑁 Í𝑇 (𝑖) (𝑖) (𝑖) (𝑖) (𝑖) 2 𝛾 𝑖=1 𝑡=1 ∥𝐵𝑡 𝑢 𝑡 − 𝑓𝑡 + 𝑤 𝑡 ∥ 𝑖=1 𝑡=1 ∥𝑥 𝑡 ∥ ≤ . (9.3) 𝑇𝑁 1−𝜌 𝑇𝑁 Proof. Using the e-ISS property in Assumption 9.1, we have: 𝑁 𝑇 𝑁 𝑇 𝑡−1 1 ∑︁ ∑︁ (𝑖) 1 ∑︁ ∑︁ ∑︁ 𝑡−1−𝑘 (𝑖) (𝑖) ∥𝑥 ∥ ≤ 𝛾 𝜌 ∥𝐵 𝑘 𝑢 𝑘 − 𝑓 𝑘(𝑖) + 𝑤 𝑘(𝑖) ∥ 𝑇 𝑁 𝑖=1 𝑡=1 𝑡 𝑇 𝑁 𝑖=1 𝑡=1 𝑘=1 𝑁 𝑇−1 𝛾 1 ∑︁ ∑︁ (𝑖) (𝑖) ∥𝐵 𝑢 − 𝑓𝑡(𝑖) + 𝑤 𝑡(𝑖) ∥ 1 − 𝜌 𝑇 𝑁 𝑖=1 𝑡=1 𝑡 𝑡 v u √︂ t 𝑁 𝑇 (𝑏) 𝛾 1 ∑︁ ∑︁ (𝑖) (𝑖) ∥𝐵𝑡 𝑢 𝑡 − 𝑓𝑡(𝑖) + 𝑤 𝑡(𝑖) ∥ 2 , ≤ 1 − 𝜌 𝑇 𝑁 𝑖=1 𝑡=1 ! (𝑎) ≤ (9.4) where (𝑎) and (𝑏) are from geometric series and Cauchy-Schwarz inequality, respectively. □ In Lemma 9.1, we assume 𝑥 1(𝑖) = 0 for simplicity: the influence of non-zero and bounded 𝑥 1(𝑖) is a constant term in each outer iteration, from the e-ISS property (9.2). Remark on the e-ISS assumption and the ACE metric. Note that an exponentially stable linear system 𝑓0 (𝑥𝑡 ) = 𝐴𝑥𝑡 (i.e., the spectral radius of 𝐴 is < 1) satisfies 159 the exponential ISS (e-ISS) assumption. However, in nonlinear systems e-ISS is a stronger assumption than exponential stability. For both linear and nonlinear systems, the e-ISS property of 𝑓0 is usually achieved by applying some stable feedback controller to the system2, i.e., 𝑓0 is the closed-loop dynamics (Slotine and W. Li, 1991; Cohen et al., 2018). e-ISS assumption is standard in both online adaptive linear control (Cohen et al., 2018; Simchowitz and Foster, 2020) and nonlinear control (Boffi et al., 2021), and practical in robotic control such as drones (see Chapter 2). In ACE, we consider a regulation task, but it can also capture trajectory tracking task with time-variant nominal dynamics 𝑓0 under incremental stability assumptions (Boffi et al., 2021). We only consider the regulation task in this chapter for simplicity. Generality. We would like to emphasize the generality of our dynamics model (9.1). The nominal control-affine part can model general fully-actuated robotic systems via Euler-Langrange equations (Murray et al., 2017; LaValle, 2006), and the unknown part 𝑓 (𝑥, 𝑐) is nonlinear in 𝑥 and 𝑐. We only need to assume the disturbance 𝑤 𝑡(𝑖) is bounded, which is more general than stochastic settings in linear (Simchowitz and Foster, 2020; Dean et al., 2020) and nonlinear (Boffi et al., 2021) cases. For example, 𝑤 𝑡(𝑖) can model extra (𝑥, 𝑢, 𝑐)-dependent uncertainties or adversarial disturbances. Moreover, the environment sequence 𝑐 (1:𝑁) could also be adversarial. In term of the extension to under-actuated systems, all the results in this chapter hold for (𝑖) the matched uncertainty setting, i.e., in the form 𝑥𝑡+1 = 𝑓0 (𝑥𝑡(𝑖) ) + 𝐵(𝑥𝑡(𝑖) )(𝑢 𝑡(𝑖) − 𝑓 (𝑥𝑡(𝑖) , 𝑐 (𝑖) )) + 𝑤 𝑡(𝑖) where 𝐵(𝑥𝑡(𝑖) ) is not necessarily full rank (e.g., drone and inverted pendulum experiments in Section 9.5). Generalizing to other under-actuated systems is interesting future work. 9.3 Online Meta-Adaptive Control (OMAC) Algorithm The design of our online meta-adaptive control (OMAC) approach comprises four pieces: the policy class, the function class, the inner loop (within environment) adaptation algorithm A2 , and the outer loop (between environment) online learning algorithm A1 . Policy class. We focus on the class of certainty-equivalence controllers (Boffi et al., 2021; Mania et al., 2019; Simchowitz and Foster, 2020), which is a general class of model-based controllers that also includes linear feedback controllers commonly studied in online control (Mania et al., 2019; Simchowitz and Foster, 2020). After 3 𝑡+1 = 2 𝑥 𝑡 +2 sin 𝑥 𝑡 + 𝑢¯ 𝑡 . With a feedback controller 𝑢¯ 𝑡 = 𝑢 𝑡 −𝑥 𝑡 −2 sin 𝑥 𝑡 , the closed-loop dynamics is 𝑥 𝑡+1 = 12 𝑥 𝑡 + 𝑢 𝑡 , where 𝑓0 (𝑥) = 12 𝑥 is e-ISS. 2 For example, consider 𝑥 160 a model is learned from past data, a controller is designed by treating the learned model as the truth (Karl J. Åström and Wittenmark, 2013). Formally, at step(𝑖, 𝑡), the controller first estimates 𝑓ˆ𝑡(𝑖) (an estimation of 𝑓𝑡(𝑖) ) based on past data, and then executes 𝑢 𝑡(𝑖) = 𝐵𝑡(𝑖)† 𝑓ˆ𝑡(𝑖) , where (·) † is the pseudo inverse. Note that from Lemma 9.1, the average control error of the omniscient controller using 𝑓ˆ𝑡(𝑖) = 𝑓𝑡(𝑖) (i.e., the controller perfectly knows 𝑓 (𝑥, 𝑐)) is upper bounded as3: ACE (omniscient) ≤ 𝛾𝑊/(1 − 𝜌). ACE (omniscient) can be viewed as a fundamental limit of the certainty-equivalence policy class. Function class 𝐹 for representation learning. In OMAC, we need to define a function class 𝐹 (𝜙(𝑥; Θ̂), 𝑐) ˆ to compute 𝑓ˆ𝑡(𝑖) (i.e., 𝑓ˆ𝑡(𝑖) = 𝐹 (𝜙(𝑥𝑡(𝑖) ; Θ̂), 𝑐)), ˆ where 𝜙 (parameterized by Θ̂) is a representation shared by all environments, and 𝑐ˆ is an environment-specific latent state. From a theoretical perspective, the main consideration of the choice of 𝐹 (𝜙(𝑥), 𝑐) ˆ is on how it effects the resulting learning objective. For instance, 𝜙 represented by a Deep Neural Network (DNN) would lead to a highly non-convex learning objective. In this chapter, we focus on the setting Θ̂ ∈ R 𝑝 , 𝑐ˆ ∈ Rℎ , and 𝑝 ≫ ℎ, i.e., it is much more expensive to learn the shared representation 𝜙 (e.g., a DNN) than “fine-tuning” via 𝑐ˆ in a specific environment, which is consistent with meta-learning (Finn, Abbeel, et al., 2017) and representation learning (Du et al., 2020; Bengio et al., 2013) practices. Inner loop adaptive control. We take a modular approach in our algorithm design, in order to cleanly leverage state-of-the-art methods from online learning, representation learning, and adaptive control. When interacting with a single environment (for 𝑇 time steps), we keep the learned representation 𝜙 fixed, and use that representation for adaptive control by treating 𝑐ˆ as an unknown low-dimensional parameter. We can utilize any adaptive control method such as online gradient descent, velocity gradient, or composite adaptation (Hazan, 2019; Boffi et al., 2021; Slotine and W. Li, 1991). Outer loop online learning. We treat the outer loop (which iterates between environments) as an online learning problem, where the goal is to learn the shared representation 𝜙 that optimizes the inner loop adaptation performance. Theoreti3 This upper bound is tight. Consider a scalar system 𝑥 𝑡+1 = 𝑎𝑥 𝑡 + 𝑢 𝑡 − 𝑓 (𝑥 𝑡 ) + 𝑤 with |𝑎| < 1 and 𝑤 a constant. In this case 𝜌 = |𝑎|, 𝛾 = 1, and the omniscient controller 𝑢 𝑡 = 𝑓 (𝑥 𝑡 ) yields ACE = 𝛾|𝑤|/(1 − 𝜌). 161 cally, we can reason about the analysis by setting up a hierarchical or nested online learning procedure (adaptive control nested within online learning). Design goal. Our goal is to design a meta-adaptive controller that has low ACE, ideally converging to ACE (omniscient) as 𝑇, 𝑁 → ∞. In other words, OMAC should converge to performing as good as the omniscient controller with perfect knowledge of 𝑓 (𝑥, 𝑐). Algorithm 9.1: Online Meta-Adaptive Control (OMAC) algorithm Input: Meta-adapter A1 ; inner-adapter A2 ; model 𝐹 (𝜙(𝑥; Θ̂), 𝑐); ˆ Boolean ObserveEnv 1 for 𝑖 = 1, · · · , 𝑁 do 2 The environment selects 𝑐 (𝑖) 3 for 𝑡 = 1, · · · , 𝑇 do 4 Compute 𝑓ˆ𝑡(𝑖) = 𝐹 (𝜙(𝑥𝑡(𝑖) ; Θ̂ (𝑖) ), 𝑐ˆ𝑡(𝑖) ) 5 Execute 𝑢 𝑡(𝑖) = 𝐵𝑡(𝑖)† 𝑓ˆ𝑡(𝑖) //certainty-equivalence policy (𝑖) 6 Observe 𝑥𝑡+1 , 𝑦 𝑡(𝑖) = 𝑓 (𝑥𝑡(𝑖) , 𝑐 (𝑖) ) − 𝑤 𝑡(𝑖) , and ℓ𝑡(𝑖) ( Θ̂, 𝑐) ˆ = ∥𝐹 (𝜙(𝑥𝑡(𝑖) ; Θ̂), 𝑐) ˆ − 𝑦 𝑡(𝑖) ∥ 2 //𝑦 𝑡(𝑖) is a noisy measurement of 𝑓 and ℓ𝑡(𝑖) is the observed loss 7 Construct an inner cost function 𝑔𝑡(𝑖) ( 𝑐) ˆ by A2 //𝑔𝑡(𝑖) is a function of 𝑐ˆ (𝑖) (𝑖) 8 Inner-adaptation: 𝑐ˆ𝑡+1 ← A2 ( 𝑐ˆ𝑡(𝑖) , 𝑔1:𝑡 ) (𝑖) 9 if ObserveEnv then Observe 𝑐 //only used in some instantiations (𝑖) 10 Construct an outer cost function 𝐺 ( Θ̂) by A1 //𝐺 (𝑖) is a function of Θ̂ 11 Meta-adaptation: Θ̂ (𝑖+1) ← A1 ( Θ̂ (𝑖) , 𝐺 (1:𝑖) ) Algorithm 9.1 describes the OMAC algorithm. Since 𝜙 is environment-invariant and 𝑝 ≫ ℎ, we only adapt Θ̂ at the end of each outer iteration. On the other hand, because 𝑐 (𝑖) varies in different environments, we adapt 𝑐ˆ at each inner step. As shown in Algorithm 9.1, at step(𝑖, 𝑡), after applying 𝑢 𝑡(𝑖) , the controller observes (𝑖) (𝑖) = 𝑓𝑡(𝑖) − 𝑤 𝑡(𝑖) , the next state 𝑥𝑡+1 and computes: 𝑦 𝑡(𝑖) ≜ 𝑓0 (𝑥𝑡(𝑖) ) + 𝐵𝑡(𝑖) 𝑢 𝑡(𝑖) − 𝑥𝑡+1 which is a disturbed measurement of the ground truth 𝑓𝑡(𝑖) . We then define ℓ𝑡(𝑖) ( Θ̂, 𝑐) ˆ ≜ ∥𝐹 (𝜙(𝑥𝑡(𝑖) ; Θ̂), 𝑐) ˆ − 𝑦 𝑡(𝑖) ∥ 2 as the observed loss at step(𝑖, 𝑡), which is a squared loss between the disturbed measurement of 𝑓𝑡(𝑖) and the model prediction 𝐹 (𝜙(𝑥𝑡(𝑖) ; Θ̂), 𝑐). ˆ Instantiations. Depending on {𝐹 (𝜙(𝑥; Θ̂), 𝑐), ˆ A1 , A2 , ObserveEnv}, we consider three settings: • Convex case: The observed loss ℓ𝑡(𝑖) is convex with respect to Θ̂ and 𝑐. ˆ 162 • Element-wise convex case: Fixing Θ̂ or 𝑐, ˆ ℓ𝑡(𝑖) is convex with respect to the other. • Deep learning case: We use a DNN with weight Θ̂ to represent 𝜙. 9.4 Different Instantiations of OMAC and Theoretical Analysis Convex Case In this subsection, we focus on a setting where the observed loss ℓ𝑡(𝑖) ( Θ̂, 𝑐) ˆ = (𝑖) 2 ∥𝐹 (𝜙(𝑥; Θ̂), 𝑐) ˆ − 𝑦 𝑡 ∥ is convex with respect to Θ̂ and 𝑐. ˆ We provide the following example to illustrate this case and highlight its difference between conventional adaptive control (e.g., Boffi et al. (2021) and Karl J. Åström and Wittenmark (2013)). Example 9.1. We consider a model 𝐹 (𝜙(𝑥; Θ̂), 𝑐) ˆ = 𝑌1 (𝑥) Θ̂ +𝑌2 (𝑥) 𝑐ˆ to estimate 𝑓 : 𝑓ˆ𝑡(𝑖) = 𝑌1 (𝑥𝑡(𝑖) ) Θ̂ (𝑖) + 𝑌2 (𝑥𝑡(𝑖) ) 𝑐ˆ𝑡(𝑖) , (9.5) where 𝑌1 : R𝑛 → R𝑛×𝑝 , 𝑌2 : R𝑛 → R𝑛×ℎ are two known bases. Note that conventional adaptive control approaches typically concatenate Θ̂ and 𝑐ˆ and adapt on both at each time step, regardless of the environment changes (e.g., Boffi et al. (2021)). Since 𝑝 ≫ ℎ, such concatenation is computationally much more expensive than OMAC, which only adapts Θ̂ in outer iterations. Because ℓ𝑡(𝑖) ( Θ̂, 𝑐) ˆ is jointly convex with respect to Θ̂ and 𝑐, ˆ the OMAC algorithm in this case falls into the category of Nested Online Convex Optimization (Nested OCO) (Agarwal et al., 2021). The choice of 𝑔𝑡(𝑖) , 𝐺 (𝑖) , A1 , A2 and ObserveEnv are depicted in Table 9.1. Note that in the convex case OMAC does not need to know 𝑐 (𝑖) in the whole process (ObserveEnv = False). 𝐹 (𝜙(𝑥; Θ̂), 𝑐) ˆ (𝑖) 𝑔𝑡 ( 𝑐) ˆ (𝑖) 𝐺 ( Θ̂) A1 A2 ObserveEnv Any 𝐹 model such that ℓ𝑡(𝑖) ( Θ̂, 𝑐) ˆ is convex (e.g., Example 9.1) (𝑖) (𝑖) (𝑖) ∇𝑐ˆ ℓ ( Θ̂ , 𝑐ˆ𝑡 ) · 𝑐ˆ Í𝑇 𝑡 (𝑖) (𝑖) (𝑖) 𝑡=1 ∇Θ̂ ℓ𝑡 ( Θ̂ , 𝑐ˆ𝑡 ) · Θ̂ With a convex set K1 , A1 initializes Θ̂ (1) ∈ K1 and returns Θ̂ (𝑖+1) ∈ K1 , ∀𝑖. A1 has sublinear regret, i.e., the total regret of A1 is 𝑇 · 𝑜(𝑁) (e.g., online gradient descent) (𝑖) With a convex set K2 , ∀𝑖, A2 initializes 𝑐ˆ1(𝑖) ∈ K2 and returns 𝑐ˆ𝑡+1 ∈ K2 , ∀𝑡. A2 has sublinear regret, i.e., the total regret of A2 is 𝑁 · 𝑜(𝑇) (e.g., online gradient descent) False Table 9.1: OMAC with convex loss. 163 (𝑖) ∈ K2 As shown in Table 9.1, at the end of step(𝑖, 𝑡) we fix Θ̂ = Θ̂ (𝑖) and update 𝑐ˆ𝑡+1 (𝑖) (𝑖) (𝑖) using A2 ( 𝑐ˆ𝑡 , 𝑔1:𝑡 ), which is an OCO problem with linear costs 𝑔1:𝑡 . On the other hand, at the end of outer iteration 𝑖, we update Θ̂ (𝑖+1) ∈ K1 using A1 ( Θ̂ (𝑖) , 𝐺 (1:𝑖) ), which is another OCO problem with linear costs 𝐺 (1:𝑖) . From Agarwal et al. (2021), we have the following regret relationship: Lemma 9.2 (Nested OCO regret bound, Agarwal et al. (2021)). OMAC (Algorithm 9.1) specified by Table 9.1 has regret: 𝑁 ∑︁ 𝑇 ∑︁ 𝑖=1 𝑡=1 ≤ ℓ𝑡(𝑖) ( Θ̂ (𝑖) , 𝑐ˆ𝑡(𝑖) ) − min Θ∈K1 𝑁 ∑︁ 𝐺 (𝑖) ( Θ̂ (𝑖) ) − min Θ∈K1 𝑖=1 | 𝑁 ∑︁ 𝑖=1 𝑁 ∑︁ min 𝑐 (𝑖) ∈K2 𝑇 ∑︁ 𝑡=1 𝐺 (𝑖) (Θ) + the total regret of A1 ,𝑇 ·𝑜(𝑁) 𝑁 ∑︁ 𝑇 ∑︁ 𝑖=1 𝑡=1 𝑖=1 {z ℓ𝑡(𝑖) (Θ, 𝑐 (𝑖) ) } | 𝑔𝑡(𝑖) ( 𝑐ˆ𝑡(𝑖) ) − 𝑁 ∑︁ 𝑖=1 min 𝑐 (𝑖) ∈K2 𝑇 ∑︁ 𝑔𝑡(𝑖) (𝑐 (𝑖) ) . 𝑡=1 {z the total regret of A2 ,𝑁·𝑜(𝑇) } (9.6) Note that the total regret of A1 is 𝑇 · 𝑜(𝑁) because 𝐺 (𝑖) is scaled up by a factor of 𝑇. With Lemmas 9.1 and 9.2, we have the following guarantee for the average control error. Theorem 9.1 (OMAC ACE bound with convex loss). Assume the unknown dynamics model is 𝑓 (𝑥, 𝑐) = 𝐹 (𝜙(𝑥; Θ), 𝑐). Assume the true parameters Θ ∈ K1 and 𝑐 (𝑖) ∈ K2 , ∀𝑖. Then OMAC (Algorithm 9.1) specified by Table 9.1 ensures the following ACE guarantee: √︂ 𝛾 𝑜(𝑇) 𝑜(𝑁) ACE ≤ 𝑊2 + + . 1−𝜌 𝑇 𝑁 Proof. Since Θ ∈ K1 and 𝑐 (1:𝑁) ∈ K2 , applying Lemma 9.2 we have 𝑁 ∑︁ 𝑇 ∑︁ 𝑖=1 𝑡=1 ℓ𝑡(𝑖) ( Θ̂ (𝑖) , 𝑐ˆ𝑡(𝑖) ) − 𝑁 ∑︁ 𝑇 ∑︁ ℓ𝑡(𝑖) (Θ, 𝑐 (𝑖) ) ≤ 𝑇 · 𝑜(𝑁) + 𝑁 · 𝑜(𝑇). (9.7) 𝑖=1 𝑡=1 Recall that the definition of ℓ𝑡(𝑖) is ℓ𝑡(𝑖) ( Θ̂, 𝑐) ˆ = ∥𝐹 (𝜙(𝑥𝑡(𝑖) ; Θ̂), 𝑐) ˆ − 𝑦 𝑡(𝑖) ∥ 2 , and 𝑦 𝑡(𝑖) = 𝑓𝑡(𝑖) − 𝑤 𝑡(𝑖) . Therefore we have ℓ𝑡(𝑖) ( Θ̂ (𝑖) , 𝑐ˆ𝑡(𝑖) ) = ∥ 𝑓ˆ𝑡(𝑖) − 𝑓𝑡(𝑖) + 𝑤 𝑡(𝑖) ∥ 2 = ∥𝐵𝑡(𝑖) 𝑢 𝑡(𝑖) − 𝑓𝑡(𝑖) + 𝑤 𝑡(𝑖) ∥ 2 ℓ𝑡(𝑖) (Θ, 𝑐 (𝑖) ) = ∥𝑤 𝑡(𝑖) ∥ 2 ≤ 𝑊 2 . (9.8) 164 Then applying Lemma 9.1, we have √︄ Í𝑁 Í𝑇 (𝑖) (𝑖) (𝑖) (𝑖) 2 𝛾 𝑖=1 𝑡=1 ∥𝐵𝑡 𝑢 𝑡 − 𝑓𝑡 + 𝑤 𝑡 ∥ ACE ≤ 1−𝜌 𝑇𝑁 √︄ Í𝑁 Í𝑇 (𝑖) (𝑖) (𝑖) 𝛾 𝑖=1 𝑡=1 ℓ𝑡 ( Θ̂ , 𝑐ˆ𝑡 ) = 1−𝜌 𝑇𝑁 √︄ Í𝑁 Í𝑇 (𝑖) (𝑖) (𝑎) 𝑇 · 𝑜(𝑁) + 𝑁 · 𝑜(𝑇) + 𝑖=1 𝛾 𝑡=1 ℓ𝑡 (Θ, 𝑐 ) ≤ 1−𝜌 𝑇𝑁 √︂ 𝛾 𝑜(𝑇) 𝑜(𝑁) ≤ 𝑊2 + + , 1−𝜌 𝑇 𝑁 (9.9) where (𝑎) uses (9.7). □ To further understand Theorem 9.1 and compare OMAC with conventional adaptive control approaches, we provide the following corollary using the model in Example 9.1. Corollary 9.1. Suppose the unknown dynamics model is 𝑓 (𝑥, 𝑐) = 𝑌1 (𝑥)Θ +𝑌2 (𝑥)𝑐, where 𝑌1 : R𝑛 → R𝑛×𝑝 , 𝑌2 : R𝑛 → R𝑛×ℎ are known bases. We assume ∥Θ∥ ≤ 𝐾Θ and ∥𝑐 (𝑖) ∥ ≤ 𝐾𝑐 , ∀𝑖. Moreover, we assume ∥𝑌1 (𝑥)∥ ≤ 𝐾1 and ∥𝑌2 (𝑥)∥ ≤ 𝐾2 , ∀𝑥. In Table 9.1 we use 𝑓ˆ𝑡(𝑖) = 𝑌1 (𝑥𝑡(𝑖) ) Θ̂ (𝑖) + 𝑌2 (𝑥𝑡(𝑖) ) 𝑐ˆ𝑡(𝑖) , and Online Gradient Descent (OGD) (Hazan, 2019) for both A1 and A2 , with learning rates 𝜂¯ (𝑖) and 𝜂𝑡(𝑖) , respectively. We set K1 = {Θ̂ : ∥ Θ̂∥ ≤ 𝐾Θ } and K2 = { 𝑐ˆ : ∥ 𝑐∥ ˆ ≤ 𝐾𝑐 }. If we schedule the learning rates as: 𝜂¯ (𝑖) = 2𝐾𝑐 2𝐾Θ (𝑖) √ , 𝜂𝑡 = √ , 2 2 (4𝐾2 𝐾𝑐 + 4𝐾1 𝐾2 𝐾Θ + 2𝐾2𝑊 ) 𝑡 (4𝐾1 𝐾Θ + 4𝐾1 𝐾2 𝐾𝑐 + 2𝐾1𝑊 )𝑇 𝑖 | {z 𝐶1 } | {z } 𝐶2 then the average control performance is bounded as: √︄   𝛾 1 1 ACE (OMAC) ≤ 𝑊 2 + 3 𝐾Θ𝐶1 √ + 𝐾𝑐 𝐶2 √ . 1−𝜌 𝑁 𝑇 Moreover, the baseline adaptive control which uses OGD to adapt Θ̂ and 𝑐ˆ at each time step satisfies: √︄ √︃ √︃ 𝛾 1 2 2 2 ACE (baseline adaptive control) ≤ 𝑊 + 3 𝐾Θ + 𝐾𝑐 𝐶12 + 𝐶22 √ . 1−𝜌 𝑇 165 Proof. Define R (A1 ) as the total regret of the outer-adapter A1 , and R (A2 ) as the total regret of the √︃ inner-adapter A2 . Recall that in Theorem 9.1 we show that (A2 ) 𝑊 2 + R (A1𝑇)+R . Now we will prove Corollary 9.1 by 𝑁 analyzing R (A1 ) and R (A2 ), respectively. 𝛾 ACE (OMAC) ≤ 1−𝜌 Since the true dynamics 𝑓 (𝑥, 𝑐 (𝑖) ) = 𝑌1 (𝑥)Θ + 𝑌2 (𝑥)𝑐 (𝑖) , we have ℓ𝑡(𝑖) ( Θ̂, 𝑐) ˆ = ∥𝑌1 (𝑥𝑡(𝑖) ) Θ̂ + 𝑌2 (𝑥𝑡(𝑖) ) 𝑐ˆ − 𝑌1 (𝑥𝑡(𝑖) )Θ − 𝑌2 (𝑥𝑡(𝑖) )𝑐 (𝑖) + 𝑤 𝑡(𝑖) ∥ 2 . (9.10) Recall that 𝑔𝑡(𝑖) ( 𝑐) ˆ = ∇𝑐ˆℓ𝑡(𝑖) ( Θ̂ (𝑖) , 𝑐ˆ𝑡(𝑖) ) · 𝑐, ˆ which is convex (linear) w.r.t. 𝑐. ˆ The (𝑖) gradient of 𝑔𝑡 is upper bounded as   (𝑖) (𝑖) ⊤ (𝑖) (𝑖) (𝑖) (𝑖) (𝑖) (𝑖) (𝑖) (𝑖) ∥∇𝑐ˆ 𝑔𝑡 ∥ = 2𝑌2 (𝑥𝑡 ) 𝑌1 (𝑥𝑡 ) Θ̂ + 𝑌2 (𝑥𝑡 ) 𝑐ˆ𝑡 − 𝑌1 (𝑥𝑡 )Θ − 𝑌2 (𝑥𝑡 )𝑐 + 𝑤 𝑡 ≤ 2𝐾2 𝐾1 𝐾Θ + 2𝐾22 𝐾𝑐 + 2𝐾2 𝐾1 𝐾Θ + 2𝐾22 𝐾𝑐 + 2𝐾2𝑊 = 4𝐾1 𝐾2 𝐾Θ + 4𝐾22 𝐾𝑐 + 2𝐾2𝑊 . | {z } 𝐶2 From Lemma 8.1, using learning rates 𝜂𝑡(𝑖) = 𝐶2𝐾√𝑐𝑡 √2 (9.11) for all 𝑖, the regret of A2 at each outer iteration is upper bounded by 3𝐾𝑐 𝐶2 𝑇. Then the total regret of A2 is bounded as √ R (A2 ) ≤ 3𝐾𝑐 𝐶2 𝑁 𝑇 . (9.12) Í Now let us study A1 . Similarly, recall that 𝐺 (𝑖) ( Θ̂) = 𝑇𝑡=1 ∇Θ̂ ℓ𝑡(𝑖) ( Θ̂ (𝑖) , 𝑐ˆ𝑡(𝑖) ) · Θ̂, which is convex (linear) w.r.t. Θ̂. The gradient of 𝐺 (𝑖) is upper bounded as 𝑇   ∑︁ (𝑖) (𝑖) (𝑖) (𝑖) (𝑖) (𝑖) (𝑖) (𝑖) ⊤ (𝑖) (𝑖) ∥∇Θ̂ 𝐺 ∥ = 2𝑌1 (𝑥𝑡 ) 𝑌1 (𝑥𝑡 ) Θ̂ + 𝑌2 (𝑥𝑡 ) 𝑐ˆ𝑡 − 𝑌1 (𝑥𝑡 )Θ − 𝑌2 (𝑥𝑡 )𝑐 + 𝑤 𝑡 𝑡=1   ≤ 𝑇 2𝐾12 𝐾Θ + 2𝐾1 𝐾2 𝐾𝑐 + 2𝐾12 𝐾Θ + 2𝐾1 𝐾2 𝐾𝑐 + 2𝐾1𝑊
= 𝑇 4𝐾12 𝐾Θ + 4𝐾1 𝐾2 𝐾𝑐 + 2𝐾1𝑊 . | {z } 𝐶1 (9.13) From Lemma 8.1, using learning rates 𝜂¯ (𝑖) = bounded as 2𝐾Θ √ , the total regret of A 1 is upper 𝑇𝐶1 𝑖 √ R (A1 ) ≤ 3𝐾Θ𝑇𝐶1 𝑁. (9.14) Finally using Theorem 9.1 we have 𝛾 ACE (OMAC) ≤ 1−𝜌 √︂ 𝑊2 + √︄ 𝛾 ≤ 1−𝜌 R (A1 ) + R (A2 ) 𝑇𝑁 1 1 𝑊 2 + 3(𝐾Θ𝐶1 √ + 𝐾𝑐 𝐶2 √ ). 𝑁 𝑇 (9.15) 166 Now let us analyze ACE (baseline adaptive control). To simplify notations, we define 𝑌¯ (𝑥) = [𝑌1 (𝑥) 𝑌2 (𝑥)] : R𝑛 → R𝑛×( 𝑝+ℎ) and 𝛼ˆ = [ Θ̂; 𝑐] ˆ ∈ R 𝑝+ℎ . The baseline adaptive controller updates the whole vector 𝛼ˆ at every time step. We denote the (𝑖) (𝑖) (𝑖) (𝑖) (𝑖) ground truth parameter √︃ by 𝛼 = [Θ; 𝑐 ], and the estimation by 𝛼ˆ 𝑡 = [ Θ̂𝑡 ; 𝑐ˆ𝑡 ]. We have ∥𝛼 (𝑖) ∥ ≤ 𝐾Θ2 + 𝐾𝑐2 . Define K̄ = { 𝛼ˆ = [ Θ̂; 𝑐] ˆ : ∥ Θ̂∥ ≤ KΘ , ∥ 𝑐∥ ˆ ≤ K𝑐 }, 𝑝+ℎ which is a convex set in R . Note that the loss function for the baseline adaptive control is ℓ¯𝑡(𝑖) ( 𝛼) ˆ = ∥𝑌¯ (𝑥𝑡(𝑖) ) 𝛼ˆ − 𝑌1 (𝑥𝑡(𝑖) )Θ − 𝑌2 (𝑥𝑡(𝑖) )𝑐 (𝑖) + 𝑤 𝑡(𝑖) ∥ 2 . The gradient of ℓ¯𝑡(𝑖) is " # (𝑖) ⊤ 𝑌 (𝑥 ) 1 𝑡 ∇𝛼ˆ ℓ¯𝑡(𝑖) ( 𝛼) ˆ =2 (𝑌1 (𝑥𝑡(𝑖) ) Θ̂ + 𝑌2 (𝑥𝑡(𝑖) ) 𝑐ˆ − 𝑌1 (𝑥𝑡(𝑖) )Θ − 𝑌2 (𝑥𝑡(𝑖) )𝑐 (𝑖) + 𝑤 𝑡(𝑖) ), 𝑌2 (𝑥𝑡(𝑖) ) ⊤ (9.16) whose norm on K̄ is bounded by √︃ √︃ 4(𝐾12 + 𝐾22 )(2𝐾1 𝐾Θ + 2𝐾2 𝐾𝑐 + 𝑊) 2 = 𝐶12 + 𝐶22 . (9.17) √︃ 2 𝐾Θ2 +𝐾𝑐2 Therefore, from Lemma 8.1, running OGD on K̄ with learning rates √ 2 2 √ gives 𝐶1 +𝐶2 𝑡 the following guarantee at each outer iteration: 𝑇 ∑︁ √︃ √︃ √ (𝑖) (𝑖) (𝑖) (𝑖) 2 2 ¯ ¯ ℓ𝑡 ( 𝛼ˆ 𝑡 ) − ℓ𝑡 (𝛼 ) ≤ 3 𝐾Θ + 𝐾𝑐 𝐶12 + 𝐶22 𝑇 . (9.18) 𝑡=1 Finally, similar as (9.9) we have √︄ Í𝑁 Í𝑇 ¯(𝑖) (𝑖) ˆ𝑡 ) 𝛾 𝑖=1 𝑡=1 ℓ𝑡 ( 𝛼 ACE (baseline adaptive control) ≤ 1−𝜌 𝑇𝑁 v u √︃ √︃ tÍ √ 𝑁 2 + 𝐾 2 𝐶 2 + 𝐶 2 𝑇 + Í 𝑁 Í𝑇 ℓ¯(𝑖) (𝛼 (𝑖) ) 3 𝐾 𝑐 𝑖=1 𝑖=1 𝑡=1 𝑡 1 2 Θ 𝛾 ≤ 1−𝜌 𝑇𝑁 √︄ √︃ √︃ 𝛾 1 ≤ 𝑊 2 + 3 𝐾Θ2 + 𝐾𝑐2 𝐶12 + 𝐶22 √ . 1−𝜌 𝑇 (9.19) □ Note that ACE (baseline adaptive control) does not improve as 𝑁 increases (i.e., encountering more environments has no benefit). If 𝑝 ≫ ℎ, we potentially have 𝐾1 ≫ 𝐾2 and 𝐾Θ ≫ 𝐾𝑐 , so 𝐶1 ≫ 𝐶2 . Therefore, the ACE upper bound of OMAC is better than the baseline adaptation if 𝑁 > 𝑇, which is consistent with recent representation learning results (Du et al., 2020; Tripuraneni, Jin, et al., 2020). It is also 167 worth noting that the baseline adaptation is much more computationally expensive, because it needs to adapt both Θ̂ and 𝑐ˆ at each time step. Intuitively, OMAC improves convergence because the meta-adapter A1 approximates the environment-invariant part 𝑌1 (𝑥)Θ, which makes the inner-adapter A2 much more efficient. Element-wise Convex Case In this subsection, we consider the setting where the loss function ℓ𝑡(𝑖) ( Θ̂, 𝑐) ˆ is element-wise convex with respect to Θ̂ and 𝑐, ˆ i.e., when one of Θ̂ or 𝑐ˆ is fixed, (𝑖) ℓ𝑡 is convex with respect to the other one. For instance, 𝐹 could be function as depicted in Example 9.2. Outside the context of control, such bilinear models are also studied in representation learning (Du et al., 2020; Tripuraneni, Jin, et al., 2020) and factorization bandits (Wang et al., 2017; Kawale et al., 2015). Example 9.2. We consider a model 𝐹 (𝜙(𝑥; Θ̂), 𝑐) ˆ = 𝑌 (𝑥) Θ̂𝑐ˆ to estimate 𝑓 : 𝑓ˆ𝑡(𝑖) = 𝑌 (𝑥𝑡(𝑖) ) Θ̂ (𝑖) 𝑐ˆ𝑡(𝑖) , (9.20) ¯ , and 𝑐ˆ (𝑖) ∈ R ℎ . Note that the where 𝑌 : R𝑛 → R𝑛× 𝑝¯ is a known basis, Θ̂ (𝑖) ∈ R 𝑝×ℎ 𝑡 dimensionality of Θ̂ is 𝑝 = 𝑝ℎ. ¯ Conventional adaptive control typically views Θ̂𝑐ˆ as a vector in R 𝑝¯ and adapts it at each time step regardless of the environment changes (Boffi et al., 2021). Compared to the Convex Case, the element-wise convex case poses new challenges: i) the coupling between Θ̂ and 𝑐ˆ brings inherent non-identifiability issues; and ii) statistical learning guarantees typical need i.i.d. assumptions on 𝑐 (𝑖) and 𝑥𝑡(𝑖) (Du et al., 2020; Tripuraneni, Jin, et al., 2020). These challenges are further amplified by the underlying unknown nonlinear system (9.1). Therefore in this section we set ObserveEnv = True, i.e., the controller has access to the true environmental condition 𝑐 (𝑖) at the end of the 𝑖 th outer iteration, which is practical in many systems when 𝑐 (𝑖) has a concrete physical meaning, e.g., drones with wind disturbances (Chapter 5). The inputs to OMAC for the element-wise convex case are specified in Table 9.2. Compared to the convex case in Table 9.1, difference includes i) the cost functions 𝑔𝑡(𝑖) and 𝐺 (𝑖) are convex, not necessarily linear; and ii) since ObserveEnv = True, in 𝐺 (𝑖) we use the true environmental condition 𝑐 (𝑖) . With inputs specified in Table 9.2, Algorithm 9.1 has ACE guarantees in the following theorem. Theorem 9.2 (OMAC ACE bound with element-wise convex loss). Assume the unknown dynamics model is 𝑓 (𝑥, 𝑐) = 𝐹 (𝜙(𝑥; Θ), 𝑐). Assume the true parameter 168 𝐹 (𝜙(𝑥; Θ̂), 𝑐) ˆ 𝑔𝑡(𝑖) ( 𝑐) ˆ 𝐺 (𝑖) ( Θ̂) A1 & A2 ObserveEnv Any 𝐹 model such that ℓ𝑡(𝑖) ( Θ̂, 𝑐) ˆ is element-wise convex (e.g., Example 9.2) ℓ𝑡(𝑖) ( Θ̂ (𝑖) , 𝑐) ˆ Í𝑇 (𝑖) ℓ ( Θ̂, 𝑐 (𝑖) ) 𝑡=1 𝑡 Same as Table 9.1 True Table 9.2: OMAC with element-wise convex loss. Θ ∈ K1 and 𝑐 (𝑖) ∈ K2 , ∀𝑖. Then OMAC (Algorithm 9.1) specified by Table 9.2 ensures the following ACE guarantee: √︂ 𝑜(𝑇) 𝑜(𝑁) 𝛾 + . 𝑊2 + ACE ≤ 1−𝜌 𝑇 𝑁 Proof. For any Θ ∈ K1 and 𝑐 (1:𝑁) ∈ K2 , we have 𝑁 ∑︁ 𝑇 ∑︁ = = ℓ𝑡(𝑖) ( Θ̂ (𝑖) , 𝑐ˆ𝑡(𝑖) ) − 𝑖=1 𝑡=1 𝑁 ∑︁ 𝑇 h ∑︁ 𝑖=1 𝑡=1 𝑁 ∑︁ 𝑇 h ∑︁ ℓ𝑡(𝑖) (Θ, 𝑐 (𝑖) ) 𝑖=1 𝑡=1 ℓ𝑡(𝑖) ( Θ̂ (𝑖) , 𝑐ˆ𝑡(𝑖) ) − ℓ𝑡(𝑖) ( Θ̂ (𝑖) , 𝑐 (𝑖) ) i + 𝑁 ∑︁ 𝑇 h ∑︁ ℓ𝑡(𝑖) ( Θ̂ (𝑖) , 𝑐 (𝑖) ) − ℓ𝑡(𝑖) (Θ, 𝑐 (𝑖) ) i 𝑖=1 𝑡=1 i 𝑔𝑡(𝑖) (𝑐 𝑡(𝑖) ) − 𝑔𝑡(𝑖) (𝑐 (𝑖) ) + 𝑖=1 𝑡=1 | 𝑁 ∑︁ 𝑇 ∑︁ 𝑁 h ∑︁ i 𝐺 (𝑖) ( Θ̂ (𝑖) ) − 𝐺 (𝑖) (Θ) . 𝑖=1 {z ≤𝑜(𝑇) } | {z ≤𝑇 ·𝑜(𝑁) Then combining with Lemma 9.1 results in the ACE bound. } (9.21) □ Faster convergence with sub-Gaussian and environment diversity assumptions. Since the cost functions 𝑔𝑡(𝑖) and 𝐺 (𝑖) in Table 9.2 are not √︃ necessarily√strongly convex, √ 𝛾 𝑊 2 + 𝑂 (1/ 𝑇) + 𝑂 (1/ 𝑁) the ACE upper bound in Theorem 9.2 is typically 1−𝜌 (e.g., using OGD for both A1 and A2 ). However, for the bilinear model in Example 9.2, it is possible to achieve faster convergence with extra sub-Gaussian and environment diversity assumptions. With a bilinear model, the inputs to the OMAC algorithm are shown in Table 9.3. With extra assumptions on 𝑤 𝑡(𝑖) and the environment, we have the following ACE guarantees. Theorem 9.3 (OMAC ACE bound with bilinear model). Consider an unknown dynamics model 𝑓 (𝑥, 𝑐) = 𝑌 (𝑥)Θ𝑐 where 𝑌 : R𝑛 → R𝑛× 𝑝¯ is a known basis and 169 𝐹 (𝜙(𝑥; Θ̂), 𝑐) ˆ (𝑖) 𝑔𝑡 ( 𝑐) ˆ (𝑖) 𝐺 ( Θ̂) A1 A2 ObserveEnv The bilinear model in Example 9.2 ℓ𝑡(𝑖) ( Θ̂ (𝑖) , 𝑐) ˆ Í𝑖 Í𝑇 ( 𝑗) 2 𝜆∥ Θ̂∥ 𝐹 + 𝑗=1 𝑡=1 ℓ𝑡 ( Θ̂, 𝑐 ( 𝑗) ) with some 𝜆 > 0 Θ̂ (𝑖+1) = arg minΘ̂ 𝐺 (𝑖) ( Θ̂) (i.e., Ridge regression) Same as Table 9.1 True Table 9.3: OMAC with bilinear model. ¯ . Assume each component of the disturbance 𝑤 (𝑖) is 𝑅-sub-Gaussian, Θ ∈ R 𝑝×ℎ 𝑡 (𝑖) the true parameters ∥Θ∥ 𝐹 ≤ 𝐾Θ , ∥𝑐 ∥ ≤ 𝐾𝑐 , ∀𝑖, and ∥𝑌 (𝑥)∥ 𝐹 ≤ 𝐾𝑌 , ∀𝑥. Define ( 𝑗) ( 𝑗) ¯ and assume environment diversity: 𝑍𝑡 = 𝑐 ( 𝑗)⊤ ⊗ 𝑌 (𝑥𝑡 ) ∈ R𝑛× 𝑝ℎ 𝜆 min ( 𝑖 ∑︁ 𝑇 ∑︁ ( 𝑗)⊤ 𝑍𝑡 ( 𝑗) 𝑍𝑡 ) = Ω(𝑖). 𝑗=1 𝑡=1 Then OMAC (Algorithm 9.1) specified by Table 9.3 has the following ACE guarantee (with probability at least 1 − 𝛿): √︄   𝛾 𝑜(𝑇) log(𝑁𝑇/𝛿) log(𝑁) 2 ACE ≤ 𝑊 + +𝑂 . (9.22) 1−𝜌 𝑇 𝑁 √ If we relax the environment diversity condition to Ω( 𝑖), the last term becomes log(𝑁𝑇/𝛿) 𝑂( √ ). 𝑁 Proof. Note that in this case the available measurement of 𝑓 at the end of the outer iteration 𝑖 is: ( 𝑗) 𝑦𝑡 ( 𝑗) ( 𝑗) = 𝑌 (𝑥𝑡 )Θ𝑐 ( 𝑗) − 𝑤 𝑡 , 1 ≤ 𝑗 ≤ 𝑖, 1 ≤ 𝑡 ≤ 𝑇 . (9.23) Recall that the Ridge-regression estimation of Θ̂ is given by Θ̂ (𝑖+1) = arg min 𝜆∥ Θ̂∥ 2𝐹 + Θ̂ ( 𝑗) ( 𝑗) ∥𝑌 (𝑥𝑡 ) Θ̂𝑐 ( 𝑗) − 𝑦 𝑡 ∥ 2 𝑗=1 𝑡=1 = arg min 𝜆∥ Θ̂∥ 2𝐹 + Θ̂ ( 𝑗) 𝑖 ∑︁ 𝑇 ∑︁ 𝑇 𝑖 ∑︁ ∑︁ (9.24) ( 𝑗) ( 𝑗) ∥𝑍𝑡 vec( Θ̂) − 𝑦 𝑡 ∥ 2 . 𝑗=1 𝑡=1 ( 𝑗) ( 𝑗) ( 𝑗) ( 𝑗) Note that 𝑦 𝑡 = (𝑐 ( 𝑗)⊤ ⊗ 𝑌 (𝑥𝑡 )) · vec(Θ) − 𝑤 𝑡 = 𝑍𝑡 vec(Θ) − 𝑤 𝑡 . Define Í Í ( 𝑗)⊤ ( 𝑗) 𝑉𝑖 = 𝜆𝐼 + 𝑖𝑗=1 𝑇𝑡=1 𝑍𝑡 𝑍𝑡 . Then from the Theorem 2 of Abbasi-Yadkori, Pál, et al. (2011) we have √︄ √ 1 + 𝑖𝑇 · 𝑛𝐾𝑌2 𝐾𝑐2 /𝜆 ∥vec( Θ̂ (𝑖+1) − Θ)∥𝑉𝑖 ≤ 𝑅 𝑝ℎ ¯ log( ) + 𝜆𝐾Θ (9.25) 𝛿 170 for all 𝑖 with probability at least 1 − 𝛿. Note that the environment diversity condition implies: 𝑉𝑖 ≻ Ω(𝑖)𝐼. Finally we have 1 log(𝑖𝑇/𝛿) ∥ Θ̂ (𝑖+1) − Θ∥ 2𝐹 = ∥vec( Θ̂ (𝑖+1) − Θ)∥ 2 ≤ 𝑂 ( )𝑂 (log(𝑖𝑇/𝛿)) = 𝑂 ( ). 𝑖 𝑖 (9.26) Then with a fixed Θ̂ (𝑖+1) , in outer iteration 𝑖 + 1 we have 𝑔𝑡(𝑖+1) ( 𝑐) ˆ = ∥𝑌 (𝑥𝑡(𝑖+1) ) Θ̂ (𝑖+1) 𝑐ˆ − 𝑌 (𝑥𝑡(𝑖+1) )Θ𝑐 (𝑖+1) + 𝑤 𝑡(𝑖+1) ∥ 2 . (9.27) Since A2 gives sublinear regret, we have 𝑇 ∑︁ ∥𝑌 (𝑥𝑡(𝑖+1) ) Θ̂ (𝑖+1) 𝑐ˆ𝑡(𝑖+1) − 𝑌 (𝑥𝑡(𝑖+1) )Θ𝑐 (𝑖+1) + 𝑤 𝑡(𝑖+1) ∥ 2 𝑡=1 − min 𝑐∈K ˆ 2 𝑇 ∑︁ (9.28) ∥𝑌 (𝑥𝑡(𝑖+1) ) Θ̂ (𝑖+1) 𝑐ˆ − 𝑌 (𝑥𝑡(𝑖+1) )Θ𝑐 (𝑖+1) + 𝑤 𝑡(𝑖+1) ∥ 2 = 𝑜(𝑇). 𝑡=1 Note that min 𝑐∈K ˆ 2 ≤ 𝑇 ∑︁ 𝑇 ∑︁ ∥𝑌 (𝑥𝑡(𝑖+1) ) Θ̂ (𝑖+1) 𝑐ˆ − 𝑌 (𝑥𝑡(𝑖+1) )Θ𝑐 (𝑖+1) + 𝑤 𝑡(𝑖+1) ∥ 2 𝑡=1 ∥𝑌 (𝑥𝑡(𝑖+1) ) Θ̂ (𝑖+1) 𝑐 (𝑖+1) − 𝑌 (𝑥𝑡(𝑖+1) )Θ𝑐 (𝑖+1) + 𝑤 𝑡(𝑖+1) ∥ 2 (9.29) 𝑡=1 (𝑎) ≤ 𝑇𝑊 2 + 𝑇 · 𝐾𝑌2 · 𝑂 ( log(𝑖𝑇/𝛿) ) · 𝐾𝑐2 , 𝑖 where (𝑎) uses (9.26). Finally we have 𝑇 ∑︁ = 𝑡=1 𝑇 ∑︁ ∥ 𝑓ˆ𝑡(𝑖+1) − 𝑓𝑡(𝑖+1) + 𝑤 𝑡(𝑖+1) ∥ 2 ∥𝑌 (𝑥𝑡(𝑖+1) ) Θ̂ (𝑖+1) 𝑐ˆ𝑡(𝑖+1) − 𝑌 (𝑥𝑡(𝑖+1) )Θ𝑐 (𝑖+1) + 𝑤 𝑡(𝑖+1) ∥ 2 (9.30) 𝑡=1 (𝑏) ≤ 𝑜(𝑇) + 𝑇𝑊 2 + 𝑂 (𝑇 log(𝑖𝑇/𝛿) ) 𝑖 for all 𝑖 with probability at least 1 − 𝛿. (𝑏) is from (9.28) and (9.29). Then with 171 Lemma 9.1 we have (with probability at least 1 − 𝛿) √︄ Í𝑁 log(𝑖𝑇/𝛿) 2 ) 𝛾 𝑖=1 𝑜(𝑇) + 𝑇𝑊 + 𝑂 (𝑇 𝑖 ACE ≤ 1−𝜌 𝑇𝑁 v u t 𝑁 𝛾 𝑜(𝑇) 𝑂 (log(𝑁𝑇/𝛿)) ∑︁ 1 (9.31) 2 ≤ 𝑊 + + 1−𝜌 𝑇 𝑁 𝑖 𝑖=1 √︂ 𝑜(𝑇) 𝛾 log(𝑁𝑇/𝛿) log(𝑁) 𝑊2 + ≤ + 𝑂( ). 1−𝜌 𝑇 𝑁 √ If we relax the environment diversity condition to Ω( 𝑖), in (9.26) we will have log(𝑖𝑇/𝛿) 𝑂 (log(𝑁𝑇/𝛿)) Í 𝑁 1 √ ≤ 𝑂 ( √ ). Therefore in (9.31) the last term becomes 𝑖=1 𝑁 𝑖 𝑂 (log(𝑁𝑇/𝛿)) √ . 𝑁 𝑖 □ The sub-Gaussian assumption is widely used in statistical learning theory to obtain concentration bounds (Abbasi-Yadkori, Pál, et al., 2011; Du et al., 2020). The environment diversity assumption states that 𝑐 (𝑖) provides “new information” in Í Í ( 𝑗)⊤ ( 𝑗) every outer iteration such that the minimum eigenvalue of 𝑖𝑗=1 𝑇𝑡=1 𝑍𝑡 𝑍𝑡 Í𝑖 Í𝑇 ( 𝑗)⊤ ( 𝑗) grows linearly as 𝑖 increases. Note that we do not need 𝜆 min ( 𝑗=1 𝑡=1 𝑍𝑡 𝑍𝑡 ) √ to increase as 𝑇 goes up. Moreover, if we relax the condition to Ω( 𝑖), the ACE bound becomes worse than the general element-wise convex case (the last term is √ 𝑂 (1/ 𝑁)), which implies the importance of “linear” environment diversity Ω(𝑖). Task diversity has been shown to be important for representation learning (Du et al., 2020; Tripuraneni, Jordan, et al., 2020). Deep OMAC We now introduce deep OMAC, a deep representation learning based OMAC algorithm. Table 9.4 shows the instantiation. As shown in Table 9.4, Deep OMAC employs a DNN to represent 𝜙, and uses gradient descent for optimization. With the model4, 𝜙(𝑥; Θ̂) · 𝑐, ˆ the meta-adapter A1 updates the representation 𝜙 via deep learning, and the inner-adapter A2 updates a linear layer 𝑐. ˆ 9.5 Simulations To demonstrate the performance of OMAC, we consider two sets of experiments: • Inverted pendulum with external wind, gravity mismatch, and unknown ¤ 𝑐), damping. The continuous-time model is 𝑚𝑙 2 𝜃¥ − 𝑚𝑙 𝑔ˆ sin 𝜃 = 𝑢 + 𝑓 (𝜃, 𝜃, 4 This structure generalizes the meta-learning and adaptive control method covered in Chapter 5 172 𝐹 (𝜙(𝑥; Θ̂), 𝑐) ˆ 𝑔𝑡(𝑖) ( 𝑐) ˆ A1 A2 ObserveEnv 𝜙(𝑥; Θ̂) · 𝑐, ˆ where 𝜙(𝑥; Θ̂) : R𝑛 → R𝑛×ℎ is a neural network with weight Θ̂ ℓ𝑡(𝑖) ( Θ̂ (𝑖) , 𝑐) ˆ Í Gradient descent: Θ̂ (𝑖+1) ← Θ̂ (𝑖) − 𝜂∇Θ̂ 𝑇𝑡=1 ℓ𝑡(𝑖) ( Θ̂, 𝑐ˆ𝑡(𝑖) ) Same as Table 9.1 False Table 9.4: OMAC with deep learning. ¤ 𝜃¥ is the angular velocity/acceleration, 𝑚 is where 𝜃 is the pendulum angle, 𝜃/ the pendulum mass and 𝑙 is the length, 𝑔ˆ is the gravity estimation, 𝑐 is the ¤ 𝑐) unknown parameter that indicates the external wind condition, and 𝑓 (𝜃, 𝜃, is the unknown dynamics which depends on 𝑐, but also includes 𝑐-invariant terms such as damping and gravity mismatch. This model generalizes Liu et al. (2020) by considering damping and gravity mismatch. • 6-DoF quadrotor with 3-D wind disturbances. We consider a 6-DoF quadrotor model with unknown wind-dependent aerodynamic force 𝑓 (𝑥, 𝑐) ∈ R3 , where 𝑥 ∈ R13 is the drone state (including position, velocity, attitude, and angular velocity) and 𝑐 is the unknown parameter indicating the wind condition. We incorporate a realistic high-fidelity aerodynamic model from Shi et al. (2020). We consider 6 different controllers in the experiment: • No-adapt is simply using 𝑓ˆ𝑡(𝑖) = 0, and omniscient is using 𝑓ˆ𝑡(𝑖) = 𝑓𝑡(𝑖) . • OMAC (convex) is based on Example 9.1, where 𝑓ˆ𝑡(𝑖) = 𝑌1 (𝑥𝑡(𝑖) ) Θ̂ (𝑖) + 𝑌2 (𝑥𝑡(𝑖) ) 𝑐ˆ𝑡(𝑖) . We use random Fourier features (Rahimi and Recht, 2007; Boffi et al., 2021) to generate both 𝑌1 and 𝑌2 . We use OGD for both A1 and A2 in Table 9.1. • OMAC (bi-convex) is based on Example 9.2, where 𝑓ˆ𝑡(𝑖) = 𝑌 (𝑥𝑡(𝑖) ) Θ̂ (𝑖) 𝑐ˆ𝑡(𝑖) . Similarly, we use random Fourier features to generate 𝑌 . Although the theoretical result in the Element-wise Convex Case requires ObserveEnv = True, Í we relax this in our experiments and use 𝐺 (𝑖) ( Θ̂) = 𝑇𝑡=1 ℓ𝑡(𝑖) ( Θ̂, 𝑐ˆ𝑡(𝑖) ) in TaÍ ble 9.2, instead of 𝑇𝑡=1 ℓ𝑡(𝑖) ( Θ̂, 𝑐 (𝑖) ). We also deploy OGD for A1 and A2 . Baseline has the same procedure except with Θ̂ (𝑖) ≡ Θ̂ (1) , i.e., it calls the inner-adapter A2 at every step and does not update Θ̂, which is standard in adaptive control (Boffi et al., 2021; Karl J. Åström and Wittenmark, 2013). 173 • OMAC (deep learning) is based on Table 9.4. We use a DNN with spectral normalization (see details in Chapter 2) to represent 𝜙, and use the Adam optimizer (Kingma and Ba, 2014) for A1 . Same as other methods, A2 is also an OGD algorithm. Baseline ACE = 0.199m OMAC (convex) ACE = 0.125m OMAC (deep) ACE = 0.066m px pz 0.4 0.2 Position [m] OMAC (bi-convex) ACE = 0.074m 0.0 0.2 0.4 fx fx fz fz Force [N] 5 0 5 5 10 15 20 Time [s] 25 30 5 10 15 20 Time [s] 25 30 5 10 15 20 Time [s] 25 30 5 10 15 20 Time [s] 25 30 (a) xz-position trajectories (top) and force predictions (bottom) in the quadrotor experiment from one random seed. The wind condition is switched randomly every 2 s (indicated by the dotted vertical lines). Average control error for each 2s wind condition Average force prediction error (Newtons) for each 2s wind condition 7 0.5 6 0.4 5 No-adapt Baseline OMAC (convex) OMAC (bi-convex) OMAC (deep) Omniscient 4 0.3 3 0.2 2 0.1 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Wind condition index 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Wind condition index (b) The evolution of control error (left) and prediction error (right) in each wind condition. The solid lines average 10 random seeds and the shade areas show standard deviations. The performance OMAC improves as the number of wind conditions increases (especially the bi-convex and deep variants) while the baseline not. Figure 9.2: Drone experiment results. For all methods, we randomly switch the environment (wind) 𝑐 every 2 s. To make a fair comparison, except no-adapt or omniscient, all methods have the same learning rate for the inner-adapter A2 and the dimensions of 𝑐ˆ are also same (dim( 𝑐) ˆ = 20 for the pendulum and dim( 𝑐) ˆ = 30 for the drone). ACE results from 10 random seeds are depicted in Table 9.5. Figure 9.2 visualizes the drone experiment results. We observe that i) OMAC significantly outperforms the baseline; ii) OMAC adapts faster and faster as it encounters more environments but baseline cannot benefit from prior experience, especially for the bi-convex and deep versions (see Figure 174 unknown external wind no-adapt 0.663 ± 0.142 OMAC (bi-convex) 0.129 ± 0.044 no-adapt 0.374 ± 0.044 OMAC (bi-convex) 0.150 ± 0.019 baseline OMAC (convex) 0.311 ± 0.112 0.147 ± 0.047 OMAC (deep) omniscient 0.093 ± 0.027 0.034 ± 0.017 baseline OMAC (convex) 0.283 ± 0.043 0.251 ± 0.043 OMAC (deep) omniscient 0.141 ± 0.024 0.100 ± 0.018 Table 9.5: ACE results in pendulum (top) and drone (bottom) experiments from 10 random seeds. 9.2), and iii) Deep OMAC achieves the best ACE due to the representation power of DNN. We note that in the drone experiments the performance of OMAC (convex) is only marginally better than the baseline. This is because the aerodynamic disturbance force in the quadrotor simulation is a multiplicative combination of the relative wind speed, the drone attitude, and the motor speeds; thus, the superposition structure 𝑓ˆ𝑡(𝑖) = 𝑌1 (𝑥𝑡(𝑖) ) Θ̂ (𝑖) + 𝑌2 (𝑥𝑡(𝑖) ) 𝑐ˆ𝑡(𝑖) cannot easily model the unknown force, while the bi-convex and deep learning variants both learn good controllers. In particular, OMAC (bi-convex) achieves similar performance as the deep learning case with much fewer parameters. On the other hand, in the pendulum experiments, OMAC (convex) is relatively better because a large component of the 𝑐-invariant part in the unknown dynamics is in superposition with the 𝑐-dependent part. 9.6 Related Work Meta-learning and representation learning. Empirically, representation learning and meta-learning have shown great success in various domains (Bengio et al., 2013). In terms of control, meta-reinforcement learning is able to solve challenging mult-task RL problems (Gupta et al., 2018; Finn, Abbeel, et al., 2017; Nagabandi et al., 2018). We remark that learning representation for control also refers to learn state representation from rich observations (Lesort et al., 2018; Gelada et al., 2019; Zhang et al., 2019), but we consider dynamics representation in this chapter. On the theoretic side, Du et al. (2020), Tripuraneni, Jin, et al. (2020), and Tripuraneni, Jordan, et al. (2020) have shown representation learning reduces sample complexity on new tasks, and “task diversity” is critical. Consistently, we show OMAC enjoys better convergence theoretically (Corollary 9.1) and empirically, and Theorem 9.3 also implies the importance of environment diversity. Another relevant line of 175 theoretical work (Finn, Rajeswaran, et al., 2019; Denevi et al., 2019; Khodak et al., 2019) uses tools from online convex optimization to show guarantees for gradientbased meta-learning, by assuming closeness of all tasks to a single fixed point in parameter space. We eliminate this assumption by considering a hierarchical or nested online optimization procedure. Adaptive control and online control. There is a rich body of literature studying Lyapunov stability and asymptotic convergence in adaptive control theory (Slotine and W. Li, 1991; Karl J. Åström and Wittenmark, 2013). Recently, there has been increased interest in studying online adaptive control with non-asymptotic metrics (e.g., regret) from learning theory, largely for settings with linear systems such as online LQR or LQG with unknown dynamics (Simchowitz and Foster, 2020; Dean et al., 2020; Cohen et al., 2018; Abbasi-Yadkori and Szepesvári, 2011). The most relevant work (Boffi et al., 2021) gives the first regret bound of adaptive nonlinear control with unknown nonlinear dynamics and stochastic noise. Another relevant work studies online robust control of nonlinear systems with a mistake guarantee on the number of robustness failures (Ho et al., 2021). However, all these results focus on the single-task case. To our knowledge, we show the first non-asymptotic convergence result for multi-task adaptive control. On the empirical side, Chapter 5 and Richards et al. (2021) combine adaptive nonlinear control with meta-learning, yielding encouraging experimental results. Online matrix factorization. Our work bears affinity to online matrix factorization, particularly the bandit collaborative filtering setting (Kawale et al., 2015; Bresler et al., 2016; Wang et al., 2017). In this setting, one typically posits a linear lowrank projection as the target representation (e.g., a low-rank factorization of the user-item matrix), which is similar to our bilinear case. Setting aside the significant complexity introduced by nonlinear control, a key similarity comes from viewing different users as “tasks” and recommended items as “actions.” Prior work in this area has by and large not been able to establish strong regret bounds, in part due to the non-identifiability issue inherent in matrix factorization. In contrast, in our setting, one set of latent variables (e.g., the wind condition) has a concrete physical meaning that we are allowed to measure (ObserveEnv in Algorithm 9.1), thus side-stepping this non-identifiability issue. 176 References Abbasi-Yadkori, Yasin, Dávid Pál, and Csaba Szepesvári (2011). Improved algorithms for linear stochastic bandits. In: Neural Information Processing Systems. Vol. 11, pp. 2312–2320. Abbasi-Yadkori, Yasin and Csaba Szepesvári (2011). Regret bounds for the adaptive control of linear quadratic systems. In: Proceedings of the 24th Annual Conference on Learning Theory. JMLR Workshop and Conference Proceedings, pp. 1–26. Agarwal, Naman, Elad Hazan, Anirudha Majumdar, and Karan Singh (2021). A regret minimization approach to iterative learning control. In: arXiv preprint arXiv:2102.13478. Åström, Karl J. and Björn Wittenmark (2013). Adaptive control. Courier Corporation. Åström, Karl Johan and Richard M. Murray (2021). Feedback systems: An introduction for scientists and engineers. Princeton University Press. Bengio, Yoshua, Aaron Courville, and Pascal Vincent (2013). Representation learning: A review and new perspectives. In: IEEE Transactions on Pattern Analysis and Machine Intelligence 35.8, pp. 1798–1828. Boffi, Nicholas M., Stephen Tu, and Jean-Jacques E. Slotine (2021). Regret bounds for adaptive nonlinear control. In: Learning for Dynamics and Control. Proceedings of Machine Learning Research, pp. 471–483. Bresler, Guy, Devavrat Shah, and Luis Filipe Voloch (2016). Collaborative filtering with low regret. In: Proceedings of the 2016 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Science. Cohen, Alon, Avinatan Hasidim, Tomer Koren, Nevena Lazic, Yishay Mansour, and Kunal Talwar (2018). Online linear quadratic control. In: International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 1029–1038. Dean, Sarah, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu (2020). On the sample complexity of the linear quadratic regulator. In: Foundations of Computational Mathematics 20.4, pp. 633–679. Denevi, Giulia, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil (2019). Learning-to-learn stochastic gradient descent with biased regularization. In: International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 1566–1575. Du, Simon S., Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei (2020). Few-shot learning via learning the representation, provably. In: arXiv preprint arXiv:2002.09434. 177 Finn, Chelsea, Pieter Abbeel, and Sergey Levine (2017). Model-agnostic metalearning for fast adaptation of deep networks. In: International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 1126–1135. Finn, Chelsea, Aravind Rajeswaran, Sham Kakade, and Sergey Levine (2019). Online meta-learning. In: International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 1920–1930. Gelada, Carles, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G. Bellemare (2019). Deepmdp: Learning continuous latent space models for representation learning. In: International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 2170–2179. Gupta, Abhishek, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine (2018). Meta-reinforcement learning of structured exploration strategies. In: arXiv preprint arXiv:1802.07245. Hazan, Elad (2019). Introduction to online convex optimization. In: arXiv preprint arXiv:1909.05207. Ho, Dimitar, Hoang Le, John Doyle, and Yisong Yue (2021). Online robust control of nonlinear systems with large uncertainty. In: International Conference on Artificial Intelligence and Statistics. Kakade, Sham, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun (2020). Information theoretic regret bounds for online nonlinear control. In: arXiv preprint arXiv:2006.12466. Kawale, Jaya, Hung H. Bui, Branislav Kveton, Long Tran-Thanh, and Sanjay Chawla (2015). Efficient thompson sampling for online matrix-factorization recommendation. In: Advances in Neural Information Processing Systems, pp. 1297–1305. Khodak, Mikhail, Maria-Florina Balcan, and Ameet Talwalkar (2019). Adaptive gradient-based meta-learning methods. In: arXiv preprint arXiv:1906.02717. Kingma, Diederik P. and Jimmy Ba (2014). Adam: A method for stochastic optimization. In: arXiv preprint arXiv:1412.6980. LaValle, Steven M. (2006). Planning algorithms. Cambridge University Press. Lee, Joonho, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter (2020). Learning quadrupedal locomotion over challenging terrain. In: Science Robotics 5.47. Lesort, Timothée, Natalia Díaz-Rodríguez, Jean-Franois Goudou, and David Filliat (2018). State representation learning for control: An overview. In: Neural Networks 108, pp. 379–392. Li, Yingying, Xin Chen, and Na Li (2019). Online optimal control with linear dynamics and predictions: Algorithms and regret analysis. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 14858–14870. 178 Liu, Anqi, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue (2020). Robust regression for safe exploration in control. In: Learning for Dynamics and Control. PMLR, pp. 608–619. url: https://proceedings.mlr. press/v120/liu20a.html. Mania, Horia, Stephen Tu, and Benjamin Recht (2019). Certainty equivalence is efficient for linear quadratic control. In: arXiv preprint arXiv:1902.07826. Maurer, Andreas, Massimiliano Pontil, and Bernardino Romera-Paredes (2016). The benefit of multitask representation learning. In: Journal of Machine Learning Research 17.81, pp. 1–32. Murray, Richard M., Zexiang Li, and S. Shankar Sastry (Dec. 2017). A mathematical introduction to robotic manipulation. 1st ed. CRC Press. isbn: 978-1-315-136370. doi: 10.1201/9781315136370. url: https://www.taylorfrancis.com/ books/9781351469791. Nagabandi, Anusha, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn (2018). Learning to adapt in dynamic, realworld environments through meta-reinforcement learning. In: arXiv preprint arXiv:1803.11347. Rahimi, Ali and Benjamin Recht (2007). Random features for large-scale kernel machines. In: Proceedings of the 20th International Conference on Neural Information Processing Systems, pp. 1177–1184. Richards, Spencer M., Navid Azizan, Jean-Jacques E. Slotine, and Marco Pavone (2021). Adaptive-control-oriented meta-learning for nonlinear systems. In: arXiv preprint arXiv:2103.04490. Shi, Xichen, Patrick Spieler, Ellande Tang, Elena-Sorina Lupu, Phillip Tokumaru, and Soon-Jo Chung (May 2020). Adaptive nonlinear control of fixed-wing VTOL with airflow vector sensing. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). Paris, France: IEEE, pp. 5321–5327. isbn: 978-172817-395-5. doi: 10 . 1109 / ICRA40945 . 2020 . 9197344. url: https : / / ieeexplore.ieee.org/document/9197344/. Simchowitz, Max and Dylan Foster (2020). Naive exploration is optimal for online LQR. In: International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 8937–8948. Slotine, Jean-Jacques E. and Weiping Li (1991). Applied nonlinear control. Vol. 199. 1. Prentice Hall, Englewood Cliffs, NJ. Tripuraneni, Nilesh, Chi Jin, and Michael I. Jordan (2020). Provable meta-learning of linear representations. In: arXiv preprint arXiv:2002.11684. Tripuraneni, Nilesh, Michael I. Jordan, and Chi Jin (2020). On the theory of transfer learning: The importance of task diversity. In: arXiv preprint arXiv:2006.11650. 179 Wang, Huazheng, Qingyun Wu, and Hongning Wang (2017). Factorization bandits for interactive recommendation. In: Proceedings of the AAAI Conference on Artificial Intelligence. Zeng, Andy, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser (2020). Tossingbot: Learning to throw arbitrary objects with residual physics. In: IEEE Transactions on Robotics 36.4, pp. 1307–1319. Zhang, Amy, Zachary C. Lipton, Luis Pineda, Kamyar Azizzadenesheli, Anima Anandkumar, Laurent Itti, Joelle Pineau, and Tommaso Furlanello (2019). Learning causal state representations of partially observable environments. In: arXiv preprint arXiv:1906.10437. 180 C h a p t e r 10 COMPETITIVE ONLINE OPTIMIZATION AND CONTROL The recent progress in learning-theoretic analyses for online control begs the question: What learning-theoretic guarantees do we need for real-world systems? Existing results focus on linear systems with classic no-regret guarantees. However, no-regret policies compare with the optimal static controller in a specific class (typically linear policy class), which could be arbitrarily suboptimal in real-world nonlinear or time-varying systems. Therefore, this chapter builds interfaces between competitive online optimization and control, which uses stronger metrics beyond regret, i.e., competitive ratio and dynamic regret (competitive difference). Those metrics directly compare with the global optimum, thus naturally suitable for real-world systems. This chapter is mainly based on the following papers: Pan, Weici, Guanya Shi, Yiheng Lin, and Adam Wierman (2022). Online optimization with feedback delay and nonlinear switching cost. In: Proceedings of the ACM on Measurement and Analysis of Computing Systems 6.1, pp. 1–34. doi: 10.1145/3508037. Shi, Guanya, Yiheng Lin, Soon-Jo Chung, Yisong Yue, and Adam Wierman (2020). Online optimization with memory and competitive control. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 33. Curran Associates, Inc. url: https://arxiv.org/abs/2002.05318. 10.1 Introduction This chapter studies the problem of Online Convex Optimization (OCO) with memory, a variant of classic OCO (Hazan, 2019) where an online learner iteratively picks an action 𝑦 𝑡 and then suffers a convex loss 𝑔𝑡 (𝑦 𝑡−𝑝 , · · · , 𝑦 𝑡 ), depending on current and previous actions. Incorporating memory into OCO has seen increased attention recently, due to both its theoretical implications, such as to convex body chasing problems (Bubeck, Lee, et al., 2019; Argue et al., 2020; Sellke, 2020; Bubeck, Klartag, et al., 2020), and its wide applicability to settings such as data centers (M. Lin, Liu, et al., 2012), power systems (Li, Qu, et al., 2018b; Badiei et al., 2015; Kim and Giannakis, 2016), and electric vehicle charging (Kim and Giannakis, 2016; S. 181 Chen and Tong, 2012). Of particular relevance to this chapter is the considerable recent effort studying connections between OCO with memory with online control in dynamical systems, leading to online algorithms that enjoy sublinear static regret (N. Agarwal, Bullins, et al., 2019; N. Agarwal, Hazan, et al., 2019), low dynamic regret (Li, Qu, et al., 2018a; Li, X. Chen, et al., 2019), constant competitive ratio (Goel and Wierman, 2019), and the ability to boost weak controllers (N. Agarwal, Brukhim, et al., 2019). Prior work on OCO with memory is typically limited in one of two ways. First, algorithms with the strongest guarantees, a constant competitive ratio, are restricted to a special case known as Smoothed Online Convex Optimization (SOCO), or OCO with switching costs (N. Chen, Goel, et al., 2018; M. Lin, Liu, et al., 2012; Goel, Y. Lin, et al., 2019), which considers only one step of memory and assumes cost functions can be observed before actions are chosen. Second, algorithms proposed for the general case typically only enjoy sublinear static regret (Anava et al., 2015), which is a much weaker guarantee, because static regret compares to the offline optimal static solution while competitive ratio directly compares to the true offline optimal. It is known that algorithms that achieve sublinear static regret can be arbitrarily worse than the true offline optimal (Goel and Hassibi, 2020), and also may have unbounded competitive ratios (Andrew et al., 2013). The pursuit of general-purpose constant-competitive algorithms for OCO with memory remains open. This chapter is also motivated by establishing theoretical connections between online optimization and control (see more discussions in Chapter 8). Recently a line of work has shown the applicability of tools from online optimization for control, albeit in limited settings (N. Agarwal, Bullins, et al., 2019; N. Agarwal, Hazan, et al., 2019; Lale et al., 2020; Goel and Wierman, 2019). Deepening these connections can potentially be impactful since most prior work studies how to achieve sublinear regret compared to the best static linear controller (Dean et al., 2020; N. Agarwal, Hazan, et al., 2019; N. Agarwal, Bullins, et al., 2019; Cohen et al., 2018). However, the best static linear controller is a weak benchmark compared to the true optimal controller (Goel and Hassibi, 2020), which may be neither linear nor static. To achieve stronger guarantees, one must seek to bound either the competitive ratio (Goel and Wierman, 2019) or dynamic regret (Li, Qu, et al., 2018a; Li, X. Chen, et al., 2019), and connections to online optimization can provide such results. However, prior attempts either have significant caveats (e.g., bounds depend on the 182 path length of the instance (Li, Qu, et al., 2018a; Li, X. Chen, et al., 2019)) or only apply to very restricted control systems (e.g., invertible control actuation matrices and perfect knowledge of disturbances (Goel and Wierman, 2019)). As such, the potential to obtain constant-competitive policies for general control systems via online optimization remains unrealized. 10.2 Problem Statement: OCO with Structured Memory In this section, we first survey prior work on Online Convex Optimization (OCO) with memory and then introduce our new model of OCO with structured memory. OCO with Memory OCO with memory is a variant of classic OCO that was first introduced by Anava et al. (2015). In contrast to classic OCO, in OCO with memory, the loss function depends on previous actions in addition to the current action. At time step 𝑡, the online agent picks 𝑦 𝑡 ∈ K ⊂ R𝑑 and then a loss function 𝑔𝑡 : K 𝑝+1 → R is revealed. The agent incurs a loss of 𝑔𝑡 (𝑦 𝑡−𝑝:𝑡 ). Thus, 𝑝 quantifies the length of the memory in the loss function. Within this general model of OCO with memory, Anava et al. (2015) focus on developing policies with small policy regret, which is defined as: policy regret = 𝑇 ∑︁ 𝑔𝑡 (𝑦 𝑡−𝑝:𝑡 ) − min 𝑇 ∑︁ 𝑔𝑡 (𝑦, · · · , 𝑦). 𝑦∈K 𝑡=𝑝 𝑡=0 The main result presents a memory-based online gradient descent algorithm that √ achieves 𝑂 ( 𝑇) regret under some moderate assumptions on the diameter of K and the gradient of the loss functions. OCO with Switching Costs (SOCO) While the general form of OCO with memory was introduced only recently, specific forms of OCO problems involving memory have been studied for decades. The most prominent example is OCO with switching costs, often termed Smoothed Online Convex Optimization (SOCO) (M. Lin, Liu, et al., 2012; N. Chen, A. Agarwal, et al., 2015; N. Chen, Goel, et al., 2018; Goel and Wierman, 2019; Li, Qu, et al., 2018b; Goel, Y. Lin, et al., 2019). In SOCO, the loss function is separated into two pieces: 1. A hitting cost 𝑓𝑡 , which depends on only the current action 𝑦 𝑡 . 2. A switching cost 𝑐(𝑦 𝑡 , 𝑦 𝑡−1 ), which penalizes big changes in the action between rounds. 183 Often the hitting cost is assumed to be of the form ∥𝑦 𝑡 − 𝑣 𝑡 ∥ for some (squared) norm, motivated by tracking some unknown trajectory 𝑣 𝑡 , and the switching cost 𝑐 is a (squared) norm motivated by penalizing switching in proportion to the (squared) distance between the actions, e.g., a common choice 𝑐(𝑦 𝑡 , 𝑦 𝑡−1 ) = 12 ∥𝑦 𝑡 − 𝑦 𝑡−1 ∥ 22 . The goal of the online learner is to minimize its total cost over 𝑇 rounds: cost( 𝐴𝐿𝐺) = 𝑇 ∑︁ 𝑓𝑡 (𝑦 𝑡 ) + 𝑐(𝑦 𝑡 , 𝑦 𝑡−1 ). 𝑡=1 Under SOCO, results characterizing the policy regret are straightforward, and the goal is instead to obtain stronger results that characterize the competitive ratio. The competitive ratio is the worst-case ratio of total cost incurred by the online learner and the offline optimal. The cost of the offline optimal is defined as the minimal cost of an algorithm if it has full knowledge of the sequence { 𝑓𝑡 }, i.e.: cost(𝑂𝑃𝑇) = min 𝑇 ∑︁ 𝑓𝑡 (𝑦 𝑡 ) + 𝑐(𝑦 𝑡 , 𝑦 𝑡−1 ). 𝑦 1 ...𝑦 𝑇 𝑡=1 Using this, the competitive ratio is defined as: cost( 𝐴𝐿𝐺) . 𝑓1:𝑇 cost(𝑂𝑃𝑇) competitive ratio( 𝐴𝐿𝐺) = sup Bounds for competitive ratio are stronger than for policy regret, since the dynamic offline optimal can change its decisions on different time steps. In the context of SOCO, the first results bounding the competitive ratio focused on one-dimensional action sets (M. Lin, Wierman, et al., 2013; Bansal et al., 2015), but after a long series of papers there now exist algorithms that provide constant competitive ratios in high dimensional settings (N. Chen, Goel, et al., 2018; Goel and Wierman, 2019; Goel, Y. Lin, et al., 2019). Among different choices of switching cost 𝑐, we are particularly interested in 𝑐(𝑦 𝑡 , 𝑦 𝑡−1 ) = 12 ∥𝑦 𝑡 − 𝑦 𝑡−1 ∥ 22 due to the connection to quadratic costs in control problems. The state-of-the-art algorithm for this switching cost is Regularized Online Balanced Descent (ROBD), introduced by Goel, Y. Lin, et al. (2019), which achieves the lowest possible competitive ratio of any online algorithm. Other recent results study the case where 𝑐(𝑦 𝑡 , 𝑦 𝑡−1 ) = ∥𝑦 𝑡 − 𝑦 𝑡−1 ∥ (Bubeck, Lee, et al., 2019; Argue et al., 2020; Sellke, 2020; Bubeck, Klartag, et al., 2020). Variants of the problem with predictions (N. Chen, A. Agarwal, et al., 2015; N. Chen, Comden, et al., 2016; Li, Qu, et al., 2018b), non-convex cost functions 184 (Y. Lin et al., 2020), and constraints (Q. Lin et al., 2019; Yang et al., 2020) have been studied as well. OCO with Structured Memory Though competitive algorithms have been proposed for many SOCO instances, the SOCO setting has two limitations. First, the hitting cost 𝑓𝑡 is revealed before making action 𝑦 𝑡 , i.e., SOCO requires one step exact prediction of 𝑓𝑡 . Second, the switching cost in SOCO only depends on one previous action in the form 𝑐(𝑦 𝑡 , 𝑦 𝑡−1 ), so only one step of memory is considered. In this chapter, our goal is to derive competitive algorithms (as exist for SOCO) in more general settings where more than one step of memory is considered. Working with the general model of OCO with memory is too ambitious for this goal. Instead, we introduce a model of OCO with structured memory that generalizes SOCO, and is motivated by a nontrivial connection with online control (see Example 10.1). Specifically, we consider a loss function 𝑔𝑡 at time step 𝑡 that can be decomposed as the sum of a hitting cost function 𝑓𝑡 : R𝑑 → R+ ∪ {0} and a switching cost function 𝑐 : R𝑑×( 𝑝+1) → R+ ∪ {0}. Additionally, we assume that the switching cost has the form: 𝑝 ∑︁ 2 1 𝐶𝑖 𝑦 𝑡−𝑖 , 𝑐(𝑦 𝑡:𝑡−𝑝 ) = 𝑦 𝑡 − 2 2 𝑖=1 with known 𝐶𝑖 ∈ R𝑑×𝑑 , 𝑖 = 1, · · · , 𝑝. Note that SOCO is a special case 𝑝 = 1 and 𝐶1 = 𝐼. Example 10.1 (Relations between the structured memory and control). Intuitively, the hitting cost penalizes the agent for deviating from an optimal point sequence, while the switching cost captures the cost of implementing a control action. Specifically, suppose 𝑦 𝑡 is a robot’s position at 𝑡, and then the classic SOCO switching cost ∥𝑦 𝑡 − 𝑦 𝑡−1 ∥ 2 is approximately its “velocity.” Under our new switching cost, we can represent acceleration by ∥𝑦 𝑡 − 2𝑦 𝑡−1 + 𝑦 𝑡−2 ∥ 2 , and many other higher-order dynamics. In other word, the classic SOCO models the single integrator dynamics where Í𝑝 the control input can directly decide the robot’s velocity, but the 21 ∥𝑦 𝑡 − 𝑖=1 𝐶𝑖 𝑦 𝑡−𝑖 ∥ 22 structure can model higher-order systems, such as the double integrator dynamics where the control input can only directly decide the robot’s acceleration. We present a formal statement of this connection in Section 10.4. To summarize, we consider an online agent and an offline adversary interacting as follows in each time step 𝑡, and we assume 𝑦𝑖 is already fixed for 𝑖 = −𝑝, −( 𝑝 − 185 1), · · · , 0. 1. The adversary reveals a function ℎ𝑡 and a convex estimation set Ω𝑡 ⊆ R𝑑 . We assume ℎ𝑡 is both 𝑚-strongly convex and 𝑙-strongly smooth, and that arg min𝑦 ℎ𝑡 (𝑦) = 0. 2. The agent picks 𝑦 𝑡 ∈ R𝑑 . 3. The adversary picks 𝑣 𝑡 ∈ Ω𝑡 . 4. The agent incurs hitting cost 𝑓𝑡 (𝑦 𝑡 ) = ℎ𝑡 (𝑦 𝑡 − 𝑣 𝑡 ) and switching cost 𝑐(𝑦 𝑡:𝑡−𝑝 ). Notice that the hitting cost 𝑓𝑡 is revealed to the online agent in two separate steps. The geometry of 𝑓𝑡 (given by ℎ𝑡 whose minimizer is at 0) is revealed before the agent picks 𝑦 𝑡 . After 𝑦 𝑡 is picked, the minimizer 𝑣 𝑡 of 𝑓𝑡 is revealed. Unlike SOCO, due to the uncertainty about 𝑣 𝑡 , the agent cannot determine the exact value of the hitting cost it incurs at time step 𝑡 when determining its action 𝑦 𝑡 . To keep the problem tractable, we assume an estimation set Ω𝑡 , which contains all possible 𝑣 𝑡 ’s, is revealed to bound the uncertainty. The agent can leverage this information when picking 𝑦 𝑡 . SOCO is a special case where Ω𝑡 contains only one point, i.e., Ω𝑡 = {𝑣 𝑡 }, and then the agent has a precise estimate of the minimizer 𝑣 𝑡 when choosing its action (Goel and Wierman, 2019; Goel, Y. Lin, et al., 2019). Like SOCO, the offline optimal cost in the structured memory model is defined as Í cost(𝑂𝑃𝑇) = min𝑦1 ...𝑦𝑇 𝑇𝑡=1 𝑓𝑡 (𝑦 𝑡 ) + 𝑐(𝑦 𝑡:𝑡−𝑝 ) given the full sequence { 𝑓𝑡 }𝑇𝑡=1 . 10.3 Algorithms and Theoretical Analysis In OCO with structured memory, there is a key differentiation depending on whether the agent has knowledge of the hitting cost function (both ℎ𝑡 and 𝑣 𝑡 ) when choosing its action or not, i.e., whether the estimation set Ω𝑡 is a single point, 𝑣 𝑡 , or not. We deal with each case in turn in the following. Case 1: Exact Prediction of 𝑣 𝑡 (Ω𝑡 = {𝑣 𝑡 }) We first study the simplest case where Ω𝑡 = {𝑣 𝑡 }. Recall that Ω𝑡 is the convex set which contains all possible 𝑣 𝑡 and so, in this case, the agent has exact knowledge of the hitting cost when picking action. This assumption, while strict, is standard in the SOCO literature, e.g., Goel and Wierman (2019) and Goel, Y. Lin, et al. (2019). It is appropriate for situations where the cost function can be observed before choosing 186 Algorithm 10.1: Regularized OBD (ROBD), Goel, Y. Lin, et al. (2019) Parameter: 𝜆 1 ≥ 0, 𝜆2 ≥ 0 1 for 𝑡 = 1 to 𝑇 do Input: Hitting cost function 𝑓𝑡 , previous decision points 𝑦 𝑡−𝑝 , · · · , 𝑦 𝑡−1 2 𝑣 𝑡 ← arg min𝑦 𝑓𝑡 (𝑦) 3 𝑦 𝑡 ← arg min𝑦 𝑓𝑡 (𝑦) + 𝜆 1 𝑐(𝑦, 𝑦 𝑡−1:𝑡−𝑝 ) + 𝜆22 ∥𝑦 − 𝑣 𝑡 ∥ 22 Output: 𝑦 𝑡 an action, e.g., Li, Qu, et al. (2018b), Kim and Giannakis (2016), and Goel and Wierman (2019). Our main result in this setting is the following theorem, which shows that the ROBD algorithm (Algorithm 10.1), which is the state-of-the-art algorithm for SOCO, performs well in the more general case of structured memory. Note that, in this setting, the smoothness parameter 𝑙 of hitting cost functions is not involved in the competitive ratio bound. Theorem 10.1. Suppose the hitting cost functions are 𝑚−strongly convex and the Í𝑝 2 switching cost is given by 𝑐(𝑦 𝑡:𝑡−𝑝 ) = 21 𝑦 𝑡 − 𝑖=1 𝐶𝑖 𝑦 𝑡−𝑖 2 , where 𝐶𝑖 ∈ R𝑑×𝑑 and Í𝑝 ∥𝐶𝑖 ∥ 2 = 𝛼. The competitive ratio of ROBD with parameters 𝜆 1 and 𝜆 2 is upper 𝑖=1 bounded by: n𝑚 + 𝜆 o 𝜆1 + 𝜆2 + 𝑚 2 max , , 𝑚𝜆1 (1 − 𝛼2 )𝜆 1 + 𝜆2 + 𝑚 √ 𝑚+𝛼2 −1+ (𝑚+𝛼2 −1) 2 +4𝑚 2 · if 𝜆 1 > 0 and (1−𝛼 )𝜆1 +𝜆 2 +𝑚 > 0. If 𝜆 1 and 𝜆 2 satisfy 𝑚+𝜆 2 = 2 𝜆 1 , then the competitive ratio is: ! √︂  1 𝛼2 − 1  2 4 𝛼2 − 1 1+ + 1+ + . 2 𝑚 𝑚 𝑚 The proof of Theorem 10.1 is given in the appendix (Section 10.7). To get insight into Theorem 10.1, first consider the case when 𝛼 is a constant. In this case, the competitive ratio is of order 𝑂 (1/𝑚), which highlights that the challenging setting is when 𝑚 is small. It is easy to see that this upper bound is in fact tight. To see this, note that the case of SOCO with ℓ2 squared switching cost considered in Goel and Wierman (2019) and Goel, Y. Lin, et al. (2019) is a special case where 𝑝 = 1, 𝐶1 = 𝐼, 𝛼 = 1. Substituting these parameters into Theorem 10.1 gives exactly the same upper bound (including constants) as Goel, Y. Lin, et al. (2019), which has been shown to match a lower bound on the achievable cost of any online algorithm, including constant factors. On the other hand, if we instead assume that 𝑚 is a fixed 187
2 positive constant. The competitive ratio can be expressed as 1 + 𝑂 𝛼 . Therefore, the competitive ratio gets worse quickly as 𝛼 increases. This is also the best possible scaling, achievable via any online algorithm, as we show in the original paper (see Shi et al., 2020, Appendix D). Perhaps surprisingly, the memory length 𝑝 does not appear in the competitive ratio bound, which contradicts the intuition that the online optimization problem should get harder as the memory length increases. However, it is worth noting that 𝛼 becomes larger as 𝑝 increases, so the memory length implicitly impacts the competitive ratio. For example, an interesting form of switching cost is   𝑝 2 1 ∑︁ 𝑖 𝑝 𝑐(𝑦 𝑡:𝑡−𝑝 ) = (−1) 𝑦 𝑡−𝑖 , 2 2 𝑖=0 𝑖 which corresponds to the 𝑝 th derivative of 𝑦 and generalizes SOCO (𝑝 = 1). In this case, we have 𝛼 = 2 𝑝 − 1. Hence 𝛼 grows exponentially in 𝑝. Case 2: Inexact Prediction of 𝑣 𝑡 (𝑣 𝑡 ∈ Ω𝑡 ) For general Ω𝑡 , ROBD is no longer enough. It needs to be adapted to handle the uncertainty that results from the estimation set Ω𝑡 . Note that this uncertainty set is crucial for many applications, such as online control with adversarial disturbances in Section 10.4. Algorithm 10.2: Optimistic ROBD Parameter: 𝜆 ≥ 0 1 for 𝑡 = 1 to 𝑇 do Input: 𝑣 𝑡−1 , ℎ𝑡 , Ω𝑡 2 Initialize a ROBD instance with 𝜆 1 = 𝜆, 𝜆 2 = 0 3 Recover 𝑓𝑡−1 (𝑦) = ℎ𝑡−1 (𝑦 − 𝑣 𝑡−1 ) 4 𝑦ˆ 𝑡−1 ← ROBD( 𝑓𝑡−1 , 𝑦ˆ 𝑡−𝑝−1:𝑡−2 ) 5 𝑣˜ 𝑡 ← arg min𝑣∈Ω𝑡 min𝑦 ℎ𝑡 (𝑦 − 𝑣) + 𝜆𝑐(𝑦, 𝑦ˆ 𝑡−1:𝑡−𝑝 ) 6 Estimate 𝑓˜𝑡 (𝑦) = ℎ𝑡 (𝑦 − 𝑣˜ 𝑡 ) 7 𝑦 𝑡 ← ROBD( 𝑓˜𝑡 , 𝑦ˆ 𝑡−𝑝:𝑡−1 ) Output: 𝑦 𝑡 (the decision at time step 𝑡) To handle this additional complexity, we propose Optimistic ROBD (Algorithm 10.2). Optimistic ROBD is based on two key ideas. The first is to ensure that the algorithm tracks the sequence of actions it would have made if given observations of the true cost functions before choosing an action. To formalize this, we define the accurate sequence { 𝑦ˆ 1 , · · · , 𝑦ˆ𝑇 } to be the choices of ROBD (Algorithm 10.1) with 188 𝜆1 = 𝜆, 𝜆2 = 0 when each hitting cost 𝑓𝑡 is revealed before picking 𝑦ˆ 𝑡 . The goal of Optimistic ROBD (Algorithm 10.2) is to approximate the accurate sequence. In order to track the accurate sequence, the first step is to recover it up to time step 𝑡 − 1 at time step 𝑡. To do this, after we observe the previous minimizer 𝑣 𝑡−1 , we can compute the accurate choice of ROBD as if both ℎ𝑡−1 and 𝑣 𝑡−1 are observed before picking 𝑦 𝑡−1 . Therefore, Algorithm 10.2 can compute the accurate subsequence { 𝑦ˆ 1 , · · · , 𝑦ˆ 𝑡−1 } at time step 𝑡. Picking 𝑦 𝑡 based on the accurate sequence { 𝑦ˆ 1 , · · · , 𝑦ˆ 𝑡−1 } instead of the noisy sequence {𝑦 1 , · · · , 𝑦 𝑡−1 } ensures that the actions do not drift too far from the accurate sequence. The second key idea is to be optimistic by assuming the adversary will give it 𝑣 ∈ Ω𝑡 that minimizes the cost it will experience. Specifically, before 𝑣 𝑡 is revealed, the algorithm assumes it is the point in Ω𝑡 which minimizes the weighted sum ℎ𝑡 (𝑦 − 𝑣) + 𝜆𝑐(𝑦, 𝑦ˆ 𝑡−1:𝑡−𝑝 ) if ROBD is implemented with parameter 𝜆 to pick 𝑦. This ensures that additional cost is never taken unnecessarily, which could be exploited by the adversary. Note that min𝑦 ℎ𝑡 (𝑦 − 𝑣) + 𝜆𝑐(𝑦) is strongly convex with respect to 𝑣 (see Shi et al., 2020, Appendix E), so it is tractable even if Ω𝑡 is unbounded. The following theorem bounds the competitive ratio of Optimistic ROBD. Theorem 10.2 (The upper bound of the competitive ratio of Optimistic ROBD). Suppose the hitting cost functions are both 𝑚−strongly convex and 𝑙−strongly smooth Í𝑝 2 and the switching cost is given by 𝑐(𝑦 𝑡:𝑡−𝑝 ) = 21 𝑦 𝑡 − 𝑖=1 𝐶𝑖 𝑦 𝑡−𝑖 2 , where 𝐶𝑖 ∈ R𝑑×𝑑 Í𝑝 and 𝑖=1 ∥𝐶𝑖 ∥ 2 = 𝛼. For arbitrary 𝜂 > 0, the cost of Optimistic ROBD with parameter 𝜆 > 0, is upper bounded by 𝐾1 cost(𝑂𝑃𝑇) + 𝐾2 , where: o n1 𝜆+𝑚 , 𝐾1 = (1 + 𝜂) max , 𝜆 (1 − 𝛼2 )𝜆 + 𝑚 𝑇  𝑙 4𝛼2 𝑚  ∑︁ ∥𝑣 𝑡 − 𝑣˜ 𝑡 ∥ 2 𝐾2 = 𝜆 + − . 1+𝜂−𝜆 𝜂 𝜆 + 𝑚 𝑡=1 2 The proof of Theorem 10.2 is given in (Shi et al., 2020, Appendix E). This proof relies on the two key ideas we mentioned before. Note that we can choose 𝜂 to balance 𝐾1 and 𝐾2 and obtain a competitive ratio, in particular the smallest 𝜂 such that:  𝑙 4𝛼2 𝑚  𝜆 + − ≤ 0. 1+𝜂−𝜆 𝜂 𝜆+𝑚 Therefore, we have 𝜂 = 𝑂 (𝑙 + 𝛼2 ) and 𝐾2 ≤ 0. So the competitive ratio is upper bounded by:  n1 o 𝜆+𝑚 𝑂 (𝑙 + 𝛼2 ) max , . 𝜆 (1 − 𝛼2 )𝜆 + 𝑚 189 However, the reason we present Theorem 10.2 in terms of 𝐾1 and 𝐾2 is that, when the diameter of Ω𝑡 is small, we can pick a small 𝜂 so that the ratio coefficient 𝐾1 will be close to the competitive ratio of ROBD when 𝑣 𝑡 is known before picking 𝑦 𝑡 . This “beyond-the-worst-case” analysis is useful in many applications. 10.4 Connections to Competitive Control Goel and Wierman (2019) show a connection between SOCO and online control in the setting where disturbance is perfectly known at time step 𝑡 and the control actuation matrix 𝐵 is invertible, which leads to the only constant-competitive control policy as far as we know. Since the new proposed OCO with structured memory generalizes SOCO, one may expect its connects to more general dynamical systems. In this section, we present a nontrivial reduction from Input-Disturbed Squared Regulators (IDSRs) to OCO with structured memory, leading to the first constantcompetitive policy in online control with adversarial disturbance. Control Problem Setting Figure 10.1: Controllable canonical form. Input-disturbed systems. We focus on systems in controllable canonical form defined by: 𝑥𝑡+1 = 𝐴𝑥𝑡 + 𝐵(𝑢 𝑡 + 𝑤 𝑡 ), (10.1) where 𝑥𝑡 ∈ R𝑛 is the state, 𝑢 𝑡 ∈ R𝑑 is the control, and 𝑤 𝑡 ∈ R𝑑 is a potentially adversarial disturbance to the system. We further assume that ( 𝐴, 𝐵) is in controllable canonical form (see Fig. 10.1), where each ∗ represents a (possibly) non-zero entry, and the rows of 𝐵 with 1 are the same rows of 𝐴 with ∗ (Luenberger, 1967). It is well-known that any controllable system can be linearly transformed to the canonical form. This system is more restrictive than the general form in linear 190 systems. We call these Input-Disturbed systems, since the disturbance 𝑤 𝑡 is in the control input/action space. There are many corresponding real-world applications that are well-described by Input-Disturbed systems, e.g., external/disturbance force in robotics (see Part I). Squared regulator costs. We consider the following cost model for the controller: 𝑐 𝑡 (𝑥𝑡 , 𝑢 𝑡 ) = 𝑞𝑡 1 ∥𝑥𝑡 ∥ 22 + ∥𝑢 𝑡 ∥ 22 , 2 2 (10.2) where 𝑞 𝑡 is a positive scalar. The sequence 𝑞 0:𝑇 is picked by the adversary and revealed online. The objective of the controller is to minimize the total control Í cost 𝑇𝑡=0 𝑐 𝑡 (𝑥𝑡 , 𝑢 𝑡 ). We call this cost model the Squared Regulator model, which is a restriction of the classical quadratic cost model. This class of costs is general enough to address a fundamental trade-off in optimal control: the trade-off between the state cost and the control effort (Kirk, 2004). Disturbances. In the online control literature, a variety of assumptions have been made about the noise 𝑤 𝑡 . In most works, the assumption is that the exact noise 𝑤 𝑡 is not known before 𝑢 𝑡 is taken. Many assume 𝑤 𝑡 is drawn from a certain known distribution, e.g., N. Agarwal, Hazan, et al. (2019). Others assume 𝑤 𝑡 is chosen adversarially subject to ∥𝑤 𝑡 ∥ 2 being upper bounded by a constant 𝑊, e.g., N. Agarwal, Bullins, et al. (2019). In a closely related paper, Goel and Wierman (2019) connect SOCO with online control under the assumption that 𝑤 𝑡 can be observed before picking the control action 𝑢 𝑡 . In contrast, in this chapter we assume that the exact 𝑤 𝑡 is not observable before the agent picks 𝑢 𝑡 . Instead, we assume a convex estimation set 𝑊𝑡 (not necessarily bounded) that contains all possible 𝑤 𝑡 is revealed to the online agent to help the agent decide 𝑢 𝑡 . Our assumption is a generalization of Goel and Wierman (2019), where 𝑊𝑡 is a one-point set, and N. Agarwal, Bullins, et al. (2019), where 𝑊𝑡 is a ball of radius 𝑊 centered at 0. Our setting can also naturally model time-Lipschitz noise, where 𝑤 𝑡 is chosen adversarially subject to ∥𝑤 𝑡 − 𝑤 𝑡−1 ∥ 2 ≤ 𝜖, by picking 𝑊𝑡 as a sphere of radius 𝜖 centered at 𝑤 𝑡−1 , which has many real-applications such as smooth disturbances in robotics. Moreover, note that our setting is naturally adaptive because of the estimation set 𝑊𝑡 (e.g., controller may choose more aggressive action if 𝑊𝑡 is small), which is different from the classic H∞ control setting (Zhou and Doyle, 1998). Competitive ratio. Our goal is to develop policies with constant (small) competitive ratios. This is a departure from the bulk of the literature (N. Agarwal, Hazan, et al., 2019; N. Agarwal, Bullins, et al., 2019; Dean et al., 2020; Cohen et al., 191 2018), which focuses on designing policies that have low regret compared to the optimal linear controller. We show the optimal linear controller can have cost arbitrarily larger than the offline optimal, via an analytic example (see Shi et al., 2020, Appendix B). We again denote the offline optimal cost, with full knowledge of Í the sequence 𝑤 0:𝑇 , as cost(𝑂𝑃𝑇) = min𝑢0:𝑇 𝑇𝑡=0 𝑐 𝑡 (𝑥𝑡 , 𝑢 𝑡 ). For an online algorithm 𝐴𝐿𝐺, let cost( 𝐴𝐿𝐺) be its cost on the same disturbance sequence 𝑤 0:𝑇 . The competitive ratio is then the worst-case ratio of cost( 𝐴𝐿𝐺) and cost(𝑂𝑃𝑇) over any disturbance sequence, i.e., sup𝑤 0:𝑇 cost( 𝐴𝐿𝐺)/cost(𝑂𝑃𝑇). To the best of our knowledge, the only prior work that studies competitive algorithms for online control is Goel and Wierman (2019), which considers a very restricted system with invertible 𝐵 and known 𝑤 𝑡 at step 𝑡. A related line of online optimization research studies dynamic regret, or competitive difference, defined as the difference between online algorithm cost and the offline optimal (Li, X. Chen, et al., 2019; Li, Qu, et al., 2018a). For example, Li, X. Chen, et al. (2019) bound the dynamic regret of online control with time-varying convex costs with no noise. However, results for the dynamic regret depend on the path-length or variation budget, not just system properties. Bounding the competitive ratio is typically more challenging. A Reduction to OCO with Structured Memory Algorithm 10.3: Reduction from Online Control to OCO with Structured Memory Input: Transition matrix 𝐴 and control matrix 𝐵 Solver: OCO with structured memory algorithm ALG 1 for 𝑡 = 0 to 𝑇 − 1 do Observe: 𝑥𝑡 , 𝑊𝑡 , and 𝑞 𝑡:𝑡+𝑝−1 2 if 𝑡 > 0 then 3 𝑤 𝑡−1 ← 𝜓 (𝑥𝑡 − 𝐴𝑥𝑡−1 − 𝐵𝑢 𝑡−1 ) Í𝑝 4 𝜁𝑡−1 ← 𝑤 𝑡−1 + 𝑖=1 𝐶𝑖 𝜁𝑡−1−𝑖 5 𝑣 𝑡−1 ← −𝜁𝑡−1   2 Í𝑑 Í 𝑝 𝑖 6 Define ℎ𝑡 (𝑦) = 21 𝑖=1 𝑞 𝑦 (𝑖) 𝑡+ 𝑗 𝑗=1 Í𝑝 7 Define Ω𝑡 = {−𝑤 − 𝑖=1 𝐶𝑖 𝜁𝑡−𝑖 | 𝑤 ∈ 𝑊𝑡 } 8 Feed 𝑣 𝑡−1 , ℎ𝑡 , Ω𝑡 into ALG 9 Obtain ALG’s output 𝑦 𝑡 Í𝑝 10 𝑢 𝑡 ← 𝑦 𝑡 − 𝑖=1 𝐶𝑖 𝑦 𝑡−𝑖 Output: 𝑢 𝑡 Output: 𝑢𝑇 = 0 We now present a reduction from IDSR to OCO with structured memory. This reduction allows us to inherit the competitive ratio bounds on Optimistic ROBD 192 for this class of online control problems. Before presenting the reduction, we first introduce some important notations. The indices of non-zero rows in matrix 𝐵 in (10.1) are denoted as {𝑘 1 , · · · , 𝑘 𝑑 } := I. We define operator 𝜓 : R𝑛 → R𝑑 as:  ⊺ 𝜓(𝑥) = 𝑥 (𝑘 1 ) , · · · , 𝑥 (𝑘 𝑑 ) , which extracts the dimensions in I. Moreover, let 𝑝𝑖 = 𝑘 𝑖 − 𝑘 𝑖−1 for 1 ≤ 𝑖 ≤ 𝑛, where 𝑘 0 = 0. The controllability index of the canonical-form ( 𝐴, 𝐵) is defined as 𝑝 = max{𝑝 1 , · · · , 𝑝 𝑑 }. We assume that the initial state is zero, i.e., 𝑥0 = 0. In the reduction, we also need to use matrices 𝐶𝑖 ∈ R𝑑×𝑑 , 𝑖 = 1, · · · , 𝑝, which regroup the columns of 𝐴(I, :). We define 𝐶𝑖 for 𝑖 = 1, · · · , 𝑝 formally by constructing each of its columns. For 𝑗 = 1, · · · , 𝑑, if 𝑖 ≤ 𝑝 𝑗 , the 𝑗 th column of 𝐶𝑖 is the (𝑘 𝑗 + 1 − 𝑖) th column of 𝐴(I, :); otherwise, the 𝑗 th column of 𝐶𝑖 is 0. Formally, for 𝑖 ∈ {1, · · · , 𝑝}, 𝑗 ∈ {1, · · · , 𝑑}, we have: 𝐶𝑖 (:, 𝑗) =    𝐴(I, 𝑘 𝑗 + 1 − 𝑖)  if 𝑖 ≤ 𝑝 𝑗  0  otherwise. Based on coefficients 𝑞 0:𝑇 , we define: 𝑞 min = min 𝑝𝑖 ∑︁ 𝑞 𝑡+ 𝑗 , 0≤𝑡≤𝑇−1,1≤𝑖≤𝑑 𝑗=1 𝑞 max = max 𝑝𝑖 ∑︁ 𝑞 𝑡+ 𝑗 , 0≤𝑡≤𝑇−1,1≤𝑖≤𝑑 𝑗=1 where we assume 𝑞 𝑡 = 0 for all 𝑡 > 𝑇. Theorem 10.3. Consider IDSR where the cost function and dynamics are specified by (10.2) and (10.1). We assume the coefficients 𝑞 𝑡:𝑡+𝑝−1 are observable at step 𝑡. Any instance of IDSR in controllable canonical form can be reduced to an instance of OCO with structured memory by Algorithm 10.3. A proof of Theorem 10.3 are given in (Shi et al., 2020, Appendix G) and an example are given in Example 10.2. Notably, 𝑐𝑜𝑠𝑡 (𝑂𝑃𝑇) and 𝑐𝑜𝑠𝑡 ( 𝐴𝐿𝐺) remain unchanged in the reduction described by Algorithm 10.3. In fact, Algorithm 10.3, when instantiated with Optimistic ROBD, provides an efficient algorithm for online control. It only requires 𝑂 ( 𝑝) memory to compute the recursive sequences. 193 Example 10.2 (Reduction for a control system with dim(𝑥) = 5, dim(𝑢) = 2). To illustrate the reduction, consider the following example: 𝑥 (1)   0 1 0 0 0  𝑥 (1)  0  𝑡    𝑡+1       (2)   𝑥𝑡+1  𝑎 1 𝑎 2 𝑎 3 𝑎 4 𝑎 5  𝑥𝑡(2)  1     (3)   𝑥  =  0 0 0 1 0  𝑥 (3)  + 0  𝑡    𝑡+1       (4)   𝑥𝑡+1   0 0 0 0 1  𝑥𝑡(4)  0     (5)   𝑥  𝑏 1 𝑏 2 𝑏 3 𝑏 4 𝑏 5  𝑥 (5)  0  𝑡    𝑡+1   0  0 " (1) # " (1) # !  𝑢𝑡 𝑤𝑡 . 0 (2) + 𝑤 𝑡(2)  𝑢𝑡 0  1 (10.3) (3) (1) = 𝑥𝑡(4) , we can rewrite (10.3) in a more compact = 𝑥𝑡(2) , 𝑥𝑡+1 Notice that since 𝑥𝑡+1 form: # " # " # # " #" # " #" # " #" " (2) (2) (2) 𝑢 𝑡(1) 𝑤 𝑡(1) 0 𝑎 3 𝑥𝑡−2 𝑎 2 𝑎 5 𝑥𝑡(2) 𝑎 1 𝑎 4 𝑥𝑡−1 𝑥𝑡+1 + (2) . + (5) + (5) + (5) = 𝑢 𝑡(2) 𝑤𝑡 0 𝑏 3 𝑥𝑡−2 𝑏 2 𝑏 5 𝑥𝑡(5) 𝑥𝑡+1 𝑏 1 𝑏 4 𝑥𝑡−1 | {z } |{z} | {z } | {z } 𝑧 𝑡+1 𝐶1 𝐶3 𝐶2 (10.4) In this example 𝑝 1 = 2, 𝑝 2 = 3, I = {𝑘 1 , 𝑘 2 } = {2, 5} and thus 𝑝 = 3 and 𝑛 = 2. From (10.4) we have 𝑧𝑡+1 = 𝐶1 𝑧𝑡 + 𝐶2 𝑧𝑡−1 + 𝐶3 𝑧𝑡−2 + 𝑢 𝑡 + 𝑤 𝑡 . (10.5) Recall the definition of 𝑦 𝑡 and 𝜁𝑡 : 𝑦𝑡 = 𝑢𝑡 + 3 ∑︁ 𝐶𝑖 𝑦 𝑡−𝑖 , ∀𝑡 ≥ 0, 𝜁𝑡 = 𝑤 𝑡 + 𝑖=1 3 ∑︁ 𝐶𝑖 𝜁𝑡−𝑖 , ∀𝑡 ≥ 0. (10.6) 𝑖=1 Then the original system could be translated to the compact form: 𝑧𝑡+1 = 𝑦 𝑡 + 𝜁𝑡 . Í If the objective is given as 12 𝑇𝑡=0  2 (10.7) 2  𝑞 𝑡 ∥𝑥𝑡 ∥ + ∥𝑢 𝑡 ∥ , we have that  2 1  2 1 (1) ℎ𝑡 (𝑧) = (𝑞 𝑡+1 + 𝑞 𝑡+2 ) 𝑧 + (𝑞 𝑡+1 + 𝑞 𝑡+2 + 𝑞 𝑡+3 ) 𝑧 (2) . 2 2 Competitive Policy The reduction in Algorithm 10.3 immediately translates the competitive ratio guarantees in Section 10.3 into competitive policies. As Theorem 10.2 suggests, we can tune 𝜂 in Optimistic ROBD based on the quality of prediction. As a result, we present two forms of upper bounds for Algorithm 10.3 in Corollaries 10.1 and 10.2. Notably, Corollary 10.1 gives a tighter bound where good estimations are 194 available, while Corollary 10.2 gives a bound that does not depend on the quality of the estimations. In the first case, we assume that a good estimation of 𝑤 𝑡 is available before picking 𝑢 𝑡 . Specifically, we assume the diameter of set 𝑊𝑡 is upper bounded by 𝜖𝑡 at time step 𝑡, where 𝜖𝑡 is a small positive constant. We derive Corollary 10.1 by setting 𝜂 = 1 + 𝜆 in Theorem 10.2. Corollary 10.1. In IDSR, assume that coefficients 𝑞 𝑡:𝑡+𝑝−1 are observable at time Í𝑞 step 𝑡. Let 𝛼 = 𝑖=1 ∥𝐶𝑖 ∥ 2 . When the diameter of 𝑊𝑡 is upper bounded by 𝜖𝑡 at time step 𝑡, the total cost incurred by Algorithm 10.3 (using Optimistic ROBD with parameter 𝜆) in the online control problem is upper bounded by 𝐾1 cost(𝑂𝑃𝑇) +𝐾2 , where: n1 o 𝜆 + 𝑞 min 𝐾1 = (2 + 𝜆) · max , , 𝜆 (1 − 𝛼2 )𝜆 + 𝑞 min 𝑇−1 𝑞 4𝛼2 𝑞 min  ∑︁ 1 2 max + − · 𝜖 . 𝐾2 = 𝜆 2 1 + 𝜆 𝜆 + 𝑞 min 𝑡=0 2 𝑡 The residue term 𝐾2 in Corollary 10.1 becomes negligible when the total estimation Í 2 error 𝑇−1 𝑡=0 𝜖 𝑡 is small, leading to a pure competitive ratio guarantee. Further, if we ignore 𝐾2 , the coefficient 𝐾1 is only constant factor worse than the ratio we obtain when exact prediction of 𝑤 𝑡 is available. However, the bound in Corollary 10.1 can be significantly worse than the case where exact prediction is available when the diameter of 𝑊𝑡 is large or unbounded. Hence we introduce a second corollary that does not use any information about 𝑤 𝑡 when picking 𝑢 𝑡 . Specifically, we assume the diameter of set 𝑊𝑡 cannot be bounded, so the upper bound  10.1 is meaningless. By picking the parameter  given in 2Corollary 4𝛼 𝑚 𝑙 𝜂 such that 𝜆 1+𝜂−𝜆 + 𝜂 − 𝜆+𝑚 ≤ 0 in Theorem 10.2, we obtain the following result. Corollary 10.2. In IDSR, assume that coefficients 𝑞 𝑡:𝑡+𝑝−1 are observable at time Í𝑞 step 𝑡. Let 𝛼 = 𝑖=1 ∥𝐶𝑖 ∥ 2 . The competitive ratio of Algorithm 10.3, using Optimistic ROBD with 𝜆, is upper bounded by:  n1 o 𝜆 + 𝑞 min 2 𝑂 (𝑞 max + 4𝛼 ) max , . 𝜆 (1 − 𝛼2 )𝜆 + 𝑞 min Compared with Corollary 10.1, Corollary 10.2 gives an upper bound that is independent of the size of 𝑊𝑡 . It is also a pure constant competitive ratio, without any 195 additive term. However, the ratio is worse than the case where exact prediction of 𝑤 𝑡 is available, especially when 𝑞 max or 𝛼 is large. Contrasting no-regret and constant-competitive guarantees. The predominant benchmark used in previous work on online control via learning is static regret relative to the best linear controller in hindsight, i.e., 𝑢 𝑡 = −𝐾 ∗ 𝑥𝑡 (Cohen et al., 2018; Abeille and Lazaric, 2018; N. Agarwal, Bullins, et al., 2019; N. Agarwal, Hazan, et al., 2019; Dean et al., 2020; Abbasi-Yadkori et al., 2018). For example, N. Agarwal, Hazan, et al. (2019) achieve logarithmic regret under stochastic noise and √ strongly convex loss, and N. Agarwal, Bullins, et al. (2019) achieve 𝑂 ( 𝑇) regret under adversarial noise and convex loss. However, the cost of the optimal linear controller may be far from the true offline optimal cost. Goel and Hassibi (2020) recently show that there is a linear regret between the optimal offline linear policy and the true offline optimal policy in online LQR control. Thus, achieving small regret may still mean having a significantly larger cost than optimal. We illustrate this difference and our algorithm’s performance by a 1-d analytic example (see Shi et al., 2020, Appendix B), and also numerical experiments in higher dimensions (Section 10.5). In particular, we see that the optimal linear controller can be significantly more costly than the offline optimal controller and that Optimistic ROBD can significantly outperform the optimal linear controller. 10.5 (a) 1-d system, wt 20 normalized cost Simulations 15 U( 1, 1) 5 Op-ROBD (wt known at t) Op-ROBD (wt unknown at t) LC OPT 10 5 (b) 1-d, wt wt 1 + U( 0.2, 0.2) 4 20 3 15 0.5 1.0 1.5 2.0 U( 1, 1) (d) 2-d, wt 2 wt 1 + U( 0.2, 0.2) 2.5 2.0 10 1.5 5 1 0.0 (c) 2-d system, wt 25 0.0 0.5 1.0 1.5 2.0 0 1.0 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 Figure 10.2: Numerical results of Optimistic ROBD in 1-d and 2-d systems, with different 𝜆. LC means the best linear controller in hindsight and OPT means the global optimal controller in hindsight. LC is numerically searched in stable linear controller space. We consider two different types of 𝑤 𝑡 : 𝑤 𝑡 is i.i.d. random/random walk, and also two different settings: 𝑤 𝑡 is known/unknown at step 𝑡. In this section, we use simple numerical examples to illustrate the contrast between the best linear controller in hindsight and the optimal offline controller. We also test our algorithm, Optimistic ROBO, and then numerically illustrate that Optimistic ROBD can obtain near-optimal cost and outperform the offline optimal linear controller. 196 In the first example we consider a simple 1-d system, where the object function is Í200 2 2 𝑇 𝑡=0 8|𝑥 𝑡 | + |𝑢 𝑡 | and the dynamics is 𝑥 𝑡+1 = 2𝑥 𝑡 + 𝑢 𝑡 + 𝑤 𝑡 . For the sequence {𝑤 𝑡 } 𝑡=0 , we consider two cases, in the first case {𝑤 𝑡 }𝑇𝑡=0 is generated by 𝑤 𝑡 ∼ U (−1, 1) i.i.d., and in the second case the sequence is generated by 𝑤 𝑡+1 = 𝑤 𝑡 + 𝜓𝑡 where 𝜓𝑡 ∼ U (−0.2, 0.2) i.i.d.. The first case corresponds to unpredictable disturbances, where the estimation set 𝑊𝑡 = (−1, 1), and the second to smooth disturbances (i.e., a random walk), where 𝑊𝑡 = 𝑤 𝑡−1 + (−0.2, 0.2). For both types of {𝑤 𝑡 }𝑇𝑡=0 , we test Optimistic ROBD algorithms in two settings: 𝑤 𝑡 is known/unknown at step 𝑡. In the first setting, 𝑤 𝑡 is directly given to the algorithm, and in the latter setting, only 𝑊𝑡 is given at time step 𝑡. The results are shown in Fig. 10.2 (a-b). We see that if 𝑤 𝑡 is known at step 𝑡, Optimistic ROBD is much better than the best linear controller in hindsight, and almost matches the true optimal when 𝑤 𝑡 is smooth. In fact, when 𝑤 𝑡 is smooth, Optimistic ROBD is much better than the best linear controller even if it does not know 𝑤 𝑡 at step 𝑡. Even in the case when 𝑤 𝑡 ∼ U (−1, 1), and so is extremely unpredictable, Optimistic ROBD’s performance still matches the best linear controller, which uses perfect hindsight. Our second example considers a 2-d system with the following objective and dynamics: " # " # " # 200 ∑︁ 0 1 0 0 min 8∥𝑥𝑡 ∥ 22 + ∥𝑢 𝑡 ∥ 22 , s.t. 𝑥𝑡+1 = 𝑥𝑡 + 𝑢𝑡 + 𝑤𝑡 , 𝑢𝑡 −1 2 1 1 𝑡=0 where ( 𝐴, 𝐵) is the canonical form of double integrator dynamics. For this 2-d system, similarly, we test the performance of Optimistic ROBD with two types of 𝑤𝑡 . The results are shown in Fig. 10.2 (c-d) and reinforce the same observations we observed in the 1-d system. In particular, we see that the optimal linear controller can be significantly more costly than the offline optimal controller and that Optimistic ROBD can outperform the optimal linear controller, sometimes by a significant margin. 10.6 Extension: Nonlinear Switching Cost and Feedback Delay Even though the proposed OCO with structured memory setting significantly generalizes the classic SOCO setting, and enables new competitive policies in online control with adversarial disturbance, there are two limitations. 197 First, the online learner observes the hitting cost function 𝑓𝑡 either before (classic SOCO setting) or one step after (OCO with structured memory) picking the action 𝑦 𝑡 . However, in many applications, such as trajectory tracking problems in robotics, 𝑓𝑡 is revealed after a multi-round delay due to communication and process delays, i.e., multiple rounds of actions must be taken without feedback on their hitting costs. Delay is known to be very challenging in practice and even one-step delay requires non-trivial algorithmic modifications (i.e., from ROBO to Optimistic ROBD). The impact of multi-round delay has been recognized as a challenging open question for the design of online algorithms (Joulani et al., 2013; Shamir and Szlak, 2017) and broadly in applications. For example, Chapter 4 highlights that a three-step delay (around 30 milliseconds) can already cause catastrophic crashes in drone tracking control using a standard controller without algorithmic adjustments for delay. Second, either SOCO or OCO with structured memory allows only linear forms of switching cost functions, where the switching cost 𝑐 is some (squared) norm of a linear combination of current and previous actions. However, in many practical scenarios the costs to move from 𝑦 𝑡−1 to 𝑦 𝑡 are non-trivial nonlinear functions. For example, consider 𝑦 𝑡 ∈ R as the vertical velocity of a drone in a velocity control task. Hovering the drone (i.e., holding the position such that 𝑦 𝑡 = 0, ∀𝑡) is not free, due to gravity. In this case, the cost to move from 𝑦 𝑡−1 to 𝑦 𝑡 is (𝑦 𝑡 − 𝑦 𝑡−1 + 𝑔(𝑦 𝑡−1 )) 2 where the nonlinear term 𝑔(𝑦 𝑡−1 ) accounts for the gravity and aerodynamic drag. Such non-linearities create significant algorithmic challenges because (i) in contrast to the linear setting, small movement between decisions does not necessarily imply small switching cost (e.g., the aforementioned drone control example), and (ii) a small error in a decision can lead to large non-linear penalties in the switching cost in future steps, which is further amplified by the multi-round delay. Addressing such challenges is well-known to be a challenging open question for the design of online algorithms. Therefore, in this section we extend the proposed OCO with structured memory setting with a 𝑘-round multi-round feedback delay and nonlinear switching costs. In this extended setting, we introduce a new algorithm, Iterative Regularized Online Balanced Descent (iROBD) and prove that it maintains a dimension-free constant competitive ratio that is 𝑂 (𝐿 2𝑘 ), where 𝐿 is the Lipschitz constant of the non-linear switching cost and 𝑘 is the delay. This is the first constant competitive algorithm in the case of either feedback delay or nonlinear switching cost and we show, via lower bounds, that the dependencies on both 𝐿 and 𝑘 are tight. These lower bounds 198 further serve to emphasize how the algorithmic difficulties increase as the length of delay becomes longer. Moreover, we show that online optimization with feedback delay and nonlinear switching costs can be connected with linear control problems more general than IDSR and a class of nonlinear control problems. Thus iROBD immediately provides novel competitive policies in those cases. This section is mainly based on (Pan et al., 2022). An Online Optimization Model Considering Nonlinearity and Delay To incorporate feedback delay, we consider a situation where the online learner only knows the geometry of the hitting cost function at each round, i.e., ℎ𝑡 , but that the minimizer of 𝑓𝑡 is revealed only after a delay of 𝑘 steps, i.e., at time 𝑡 + 𝑘. This captures practical scenarios where the form of the loss function or tracking function is known by the online learner, but the target moves over time and measurement lag means that the position of the target is not known until some time after an action must be taken. To incorporate nonlinear (and potentially nonconvex) switching costs, we consider a known nonlinear function 𝛿 from R𝑑×𝑝 to R𝑑 in the switching cost. Specifically, we have 𝑐(𝑦 𝑡:𝑡−𝑝 ) = 1 ∥𝑦 𝑡 − 𝛿(𝑦 𝑡−1:𝑡−𝑝 )∥ 2 . 2 (10.8) In summary, we consider an online agent that interacts with the environment as follows: 1. The adversary reveals a function ℎ𝑡 , which is the geometry of the 𝑡 th hitting cost, and a point 𝑣 𝑡−𝑘 , which is the minimizer of the (𝑡 − 𝑘) th hitting cost. Assume that ℎ𝑡 is 𝑚-strongly convex and 𝑙-strongly smooth, and that arg min𝑦 ℎ𝑡 (𝑦) = 0. 2. The online learner picks 𝑦 𝑡 as its decision point at time step 𝑡 after observing ℎ𝑡 , 𝑣 𝑡−𝑘 . 3. The adversary picks the minimizer of the hitting cost at time step 𝑡: 𝑣 𝑡 . 4. The learner pays hitting cost 𝑓𝑡 (𝑦 𝑡 ) = ℎ𝑡 (𝑦 𝑡 − 𝑣 𝑡 ) and switching cost 𝑐(𝑦 𝑡:𝑡−𝑝 ) of the form (10.8). Iterative ROBD Algorithm Iterative ROBD (iROBD, Algorithm 10.4) is the first competitive algorithm for online optimization with multi-step delay and nonlinear switching costs. Generally 199 Algorithm 10.4: Iterative ROBD (iROBD) 1: Parameter: 𝜆 ≥ 0 2: Initialize a ROBD instance with 𝜆 1 = 𝜆, 𝜆 2 = 0 3: for 𝑡 = 1 to 𝑇 do 4: Input: ℎ𝑡 , 𝑣 𝑡−𝑘 5: Observe 𝑓𝑡−𝑘 (𝑦) = ℎ𝑡−𝑘 (𝑦 − 𝑣 𝑡−𝑘 ) 6: 𝑦ˆ 𝑡−𝑘 = ROBD( 𝑓𝑡−𝑘 , 𝑦ˆ 𝑡−𝑘−𝑝:𝑡−𝑘−1 ) 7: Initialize a temporary sequence 𝑠1:𝑡 8: 𝑠1:𝑡−𝑘 ← 𝑦ˆ 1:𝑡−𝑘 9: for 𝑖 = 𝑡 − 𝑘 + 1 to 𝑡 do 10: 𝑣˜ 𝑖 = arg min𝑣 min𝑦 ℎ𝑖 (𝑦 − 𝑣) + 𝜆𝑐(𝑦, 𝑠𝑖−1:𝑖−𝑝 ) 11: Set 𝑓˜𝑖 (𝑦) = ℎ𝑖 (𝑦 − 𝑣˜ 𝑖 ) 12: 𝑠𝑖 ← ROBD( 𝑓˜𝑖 , 𝑠𝑖−𝑝:𝑖−1 ) 13: end for 14: 𝑦 𝑡 = 𝑠𝑡 15: Output: 𝑦 𝑡 (the action at time step 𝑡) 16: end for speaking, the design of iROBD deals with the 𝑘-round delay via a novel iterative process of estimating the unknown cost function optimistically, i.e., iteratively assuming that the unknown cost functions will lead to minimal cost for the algorithm. This approach is different than a one-shot optimistic approach focused on the whole trajectory of unknown cost functions, and the iterative nature is crucial for bounding the competitive ratio. More interpretations can be found in Pan et al. (2022). iROBD enjoys the following competitive ratio bound: Theorem 10.4. Suppose the hitting costs are 𝑚-strongly convex and 𝑙-strongly smooth, and the switching cost is given by 𝑐(𝑦 𝑡:𝑡−𝑝 ) = 21 ∥𝑦 𝑡 − 𝛿(𝑦 𝑡−1:𝑡−𝑝 )∥ 2 , where 𝛿 : R𝑑×𝑝 → R𝑑 . If there is a 𝑘-round-delayed feedback on the minimizers, and for any 1 ≤ 𝑖 ≤ 𝑝 there exists a constant 𝐿 𝑖 > 0, such that for any given 𝑦 𝑡−1 , · · · , 𝑦 𝑡−𝑖−1 , 𝑦 𝑡−𝑖+1 , · · · , 𝑦 𝑡−𝑝 ∈ R𝑑 , we have: ∥𝜃 (𝑎) − 𝜃 (𝑏)∥ ≤ 𝐿 𝑖 ∥𝑎 − 𝑏∥, ∀𝑎, 𝑏 ∈ R𝑑 , where 𝜃 (𝑥) = 𝛿(𝑦 𝑡−1 , · · · , 𝑦 𝑡−𝑖−1 , 𝑥, 𝑦 𝑡−𝑖+1 , · · · , 𝑦 𝑡−𝑝 ), then the competitive ratio of iROBD(𝜆) is bounded by    𝑚+𝜆 1 2 2 𝑘 𝑂 (𝑙 + 2𝑝 𝐿 ) max , , 𝜆 𝑚 + (1 − 𝑝 2 𝐿 2 )𝜆 where 𝐿 = max𝑖 {𝐿 𝑖 }, 𝜆 > 0 and 𝑚 + (1 − 𝑝 2 𝐿 2 )𝜆 > 0. 200 The detailed analysis and a proof can be found in Pan et al. (2022). The key idea to our competitive ratio proof is to bound the error that accumulates in the iterations by leveraging a Lipschitz property on the nonlinear component of the switching cost. This analytic approach is novel and a contribution in its own right. Note that the competitive ratio of iROBD in Theorem 10.4 is 𝑂 (𝐿 2𝑘 ). In other words, feedback delay (𝑘) leads to an exponential degradation of the competitive ratio while memory (𝑝) and the Lipschitz constant of the nonlinear switching cost (𝐿) impact the competitive ratio only in a polynomial manner. Thus, one may wonder if this dependence is a function of the iROBD algorithm or if it is fundamental. In Pan et al. (2022), we provide a Ω(𝐿 2𝑘 ) to show that it is a fundamental limit. Connections to Competitive Control As expected, online optimization with feedback delay and nonlinear switching costs can be connected with much more general control problems than classic SOCO or OCO with structured memory settings. In particular, we show two examples here. For both examples, we can show exact reductions from them to online optimization with delay and nonlinearity, and iROBD immediately provides competitive policies. For detailed reductions and analysis, we refer to Pan et al. (2022). Linear systems with matched and unmatched disturbance. We consider min 𝑇 ∑︁ 𝑞𝑡 𝑢𝑡 𝑡=1 s.t. 2 2 ∥𝑥𝑡 ∥ + 𝑇−1 ∑︁ 1 ∥𝑢 𝑡 ∥ 2 2 𝑡=0 𝑥𝑡+1 = 𝐴𝑥𝑡 + 𝐵𝑢 𝑡 + 𝑤 𝑡 , where ( 𝐴, 𝐵) are in controllable canonical form and 𝑤 𝑡 is a potentially adversarial disturbance. Note that it considers both matched and unmatched disturbance while Section 10.4 only considers the matched disturbance setting 𝐵(𝑢 𝑡 + 𝑤 𝑡 ). Nonlinear systems with delay and time-variant costs. Consider the following class of online nonlinear control problems: min 𝑇 ∑︁ 𝑢𝑡 𝑡=1 s.t. 𝑓𝑡 (𝑥𝑡 ) + 𝑇−1 ∑︁ 1 ∥𝑢 𝑡 ∥ 2 2 𝑡=0 𝑥𝑡+1 = 𝐴𝑥𝑡 + 𝑢 𝑡 + 𝑔(𝑥𝑡 ) where { 𝑓𝑡 }𝑇𝑡=1 are time-variant well-conditioned costs (e.g., trajectory tracking costs), and 𝑔(𝑥𝑡 ) is the nonlinear dynamics term. At time step 𝑡, only 𝑓1:𝑡−𝑘 is known due to communication delays. Many robotic systems can be viewed as special cases of this 201 form, such as pendulum dynamics and quadrotor position dynamics (see Chapter 2). It is immediate to see that, by defining 𝑦 𝑡 = 𝑥𝑡 , this online control problem can be converted into an online optimization problem with hitting cost 𝑓𝑡 and nonlinear switching cost 𝑐(𝑦 𝑡 , 𝑦 𝑡−1 ) = 21 ∥𝑦 𝑡 − 𝐴𝑦 𝑡−1 − 𝑔(𝑦 𝑡−1 )∥ 2 . 10.7 Appendix Proof of Theorem 10.1 Our approach is to make use of strong convexity and properties of the hitting cost, the switching cost, and the regularization term to derive an inequality in the form of 𝐻𝑡 + 𝑀𝑡 + Δ𝜙𝑡 ≤ 𝐶 (𝐻𝑡∗ + 𝑀𝑡∗ ) for some positive constant 𝐶, where Δ𝜙𝑡 is the change Í in potential, which satisfies 𝑇𝑡=1 Δ𝜙𝑡 ≥ 0. We will give the formal definition of Δ𝜙𝑡 later. The constant 𝐶 is then an upper bound for the competitive ratio. By assumption, we have 𝑦𝑖 = 𝑦𝑖∗ for 𝑖 = 0, −1, · · · , −( 𝑝 − 1). For convenience, we define 𝜙𝑡 = 𝜆1 + 𝜆2 + 𝑚 2 𝑦 𝑡 − 𝑦 𝑡∗ . 2 Recall that we define 𝑣 𝑡 = arg min𝑦 𝑓𝑡 (𝑦). Since the function 2 𝑝 ∑︁ 𝜆2 𝜆1 ∥𝑦 − 𝑣 𝑡 ∥ 2 𝑦− 𝐶𝑖 𝑦 𝑡−𝑖 + 𝑔𝑡 (𝑦) = 𝑓𝑡 (𝑦) + 2 2 𝑖=1 is (𝑚 + 𝜆1 + 𝜆 2 )−strongly convex and ROBD selects 𝑦 𝑡 = arg min𝑦 𝑔𝑡 (𝑦), we see that 𝑔𝑡 (𝑦 𝑡 ) + 𝑚 + 𝜆1 + 𝜆2 2 𝑦 𝑡 − 𝑦 𝑡∗ ≤ 𝑔𝑡 (𝑦 𝑡∗ ), 2 which implies 𝐻𝑡 + 𝜆 1 𝑀𝑡 + 𝜙𝑡 − 𝑝 ∑︁ ∥𝐶𝑖 ∥ ! 𝜙𝑡−𝑖 𝛼 𝑖=1 2  𝑝 𝑝 ∑︁ ∥𝐶𝑖 ∥ 𝜆2 ∗ 2 © 𝜆 1 ∗ ∑︁ ª ∗ 𝐶𝑖 𝑦 𝑡−𝑖 − ≤ 𝐻𝑡 + 𝑦𝑡 − 𝑣𝑡 +­ 𝑦𝑡 − 𝜙𝑡−𝑖 ® . 2 2 𝛼 𝑖=1 𝑖=1 « ¬ (10.9)  In the following steps, we bound the second term in the right-hand side of (10.9) by 202 the switching cost of the offline optimal. 𝑝 ∑︁ ∥𝐶𝑖 ∥ 𝜙𝑡−𝑖 𝛼 𝑖=1 𝑝 = 𝜆 1 + 𝜆 2 + 𝑚 ∑︁ ∗ 2 ∥𝐶𝑖 ∥ · 𝑦 𝑡−𝑖 − 𝑦 𝑡−𝑖 2𝛼 𝑖=1 !2 𝑝 𝜆 1 + 𝜆 2 + 𝑚 ∑︁ ∗ ∥𝐶𝑖 ∥ · 𝑦 𝑡−𝑖 − 𝑦 𝑡−𝑖 ≥ 2𝛼2 𝑖=1 !2 𝑝 𝜆 1 + 𝜆 2 + 𝑚 ∑︁ ∗ 𝐶𝑖 𝑦 𝑡−𝑖 − 𝐶𝑖 𝑦 𝑡−𝑖 ≥ 2𝛼2 𝑖=1 2 𝑝 𝑝 ∑︁ 𝜆 1 + 𝜆 2 + 𝑚 ∑︁ ∗ ≥ 𝐶𝑖 𝑦 𝑡−𝑖 − 𝐶𝑖 𝑦 𝑡−𝑖 , 2𝛼2 𝑖=1 𝑖=1 (10.10a) (10.10b) (10.10c) where we use Jensen’s Inequality in (10.10a); the definition of the matrix norm in (10.10b); the triangle inequality in (10.10c). For notation convenience, we define 𝛿𝑡 = 𝑝 ∑︁ 𝑖=1 𝐶𝑖 𝑦 𝑡−𝑖 − 𝑝 ∑︁ 𝑖=1 ∗ . 𝐶𝑖 𝑦 𝑡−𝑖 203 Therefore, we obtain that 2 𝑝 𝑝 ∑︁ ∥𝐶𝑖 ∥ 𝜆 1 ∗ ∑︁ 𝐶𝑖 𝑦 𝑡−𝑖 − 𝑦𝑡 − 𝜙𝑡−𝑖 2 𝛼 𝑖=1 𝑖=1 2 𝑝 𝜆 1 ∗ ∑︁ 𝜆1 + 𝜆2 + 𝑚 ≤ 𝑦𝑡 − 𝐶𝑖 𝑦 𝑡−𝑖 − · ∥𝛿𝑡 ∥ 2 2 2 2𝛼 𝑖=1 ! 2 𝑝 ∑︁ 𝜆1 + 𝜆2 + 𝑚 𝜆1 ∗ ∗ 𝐶𝑖 𝑦 𝑡−𝑖 − 𝛿𝑡 − = 𝑦𝑡 − · ∥𝛿𝑡 ∥ 2 2 2 2𝛼 𝑖=1 (10.11a) 2 𝑝 𝑝 ∑︁ 𝜆 1 ∗ ∑︁ ∗ ∗ ∗ 𝑦 − 𝐶𝑖 𝑦 𝑡−𝑖 + 𝜆 1 𝑦 𝑡 − 𝐶𝑖 𝑦 𝑡−𝑖 · ∥𝛿𝑡 ∥ ≤ 2 𝑡 𝑖=1 𝑖=1 𝜆1 𝜆1 + 𝜆2 + 𝑚 ∥𝛿𝑡 ∥ 2 − ∥𝛿𝑡 ∥ 2 2 2𝛼2 2 𝑝 𝑝 ∑︁ 𝜆 1 ∗ ∑︁ ∗ ∗ ∗ = 𝑦 − 𝐶𝑖 𝑦 𝑡−𝑖 + 𝜆 1 𝑦 𝑡 − 𝐶𝑖 𝑦 𝑡−𝑖 · ∥𝛿𝑡 ∥ 2 𝑡 𝑖=1 𝑖=1 + (10.11b) (1 − 𝛼2 )𝜆 1 + 𝜆 2 + 𝑚 ∥𝛿𝑡 ∥ 2 2 2𝛼 2 2 𝑝 𝑝 ∑︁ ∑︁ 𝛼2𝜆21 𝜆1 ∗ ∗ ∗ ∗
𝑦𝑡 − ≤ 𝑦 − 𝐶𝑖 𝑦 𝑡−𝑖 + 𝐶𝑖 𝑦 𝑡−𝑖 2 𝑡 𝑖=1 2 (1 − 𝛼2 )𝜆 1 + 𝜆 2 + 𝑚 𝑖=1 − (1 − 𝛼2 )𝜆 1 + 𝜆 2 + 𝑚 (1 − 𝛼2 )𝜆 1 + 𝜆 2 + 𝑚 2 ∥𝛿 ∥ ∥𝛿𝑡 ∥ 2 − 𝑡 2𝛼2 2𝛼2 𝜆 1 (𝜆 1 + 𝜆 2 + 𝑚) = 𝑀 ∗, (1 − 𝛼2 )𝜆 1 + 𝜆 2 + 𝑚 𝑡 + (10.11c) where we use (10.10) in (10.11a); the triangle inequality in (10.11b); the AM-GM inequality in (10.11c). We also notice that since 𝑓𝑡 is 𝑚-strongly convex, the first term in the right-hand side of (10.9) can be bounded by 𝐻𝑡∗ + 𝜆2 ∗ 𝑚 + 𝜆2 ∗ 2 𝑦𝑡 − 𝑣𝑡 ≤ 𝐻𝑡 . 2 𝑚 (10.12) Substituting (10.11) and (10.12) into (10.9), we obtain that 𝐻𝑡 + 𝜆1 𝑀𝑡 + 𝜙𝑡 − 𝑞 ∑︁ ∥𝐶𝑖 ∥ 𝜙𝑡−𝑖 𝛼 𝑡=1 𝑚 + 𝜆2 ∗ 𝜆 1 (𝜆 1 + 𝜆 2 + 𝑚) ≤ 𝐻𝑡 + 𝑀𝑡∗ . 2 𝑚 (1 − 𝛼 )𝜆 1 + 𝜆 2 + 𝑚 (10.13) 204 Define Δ𝜙𝑡 = 𝜙𝑡 − 𝑇 ∑︁ Í𝑞 𝑡=1 Δ𝜙𝑡 = 𝑡=1 ∥𝐶𝑖 ∥ 𝛼 𝜙𝑡−𝑖 . We see that 𝑞 𝑞 𝑞−1 𝑞−1 1 ∑︁ © ∑︁ 1 ∑︁ © ∑︁ ª ª 𝐶 𝜙 − 𝐶 𝑗 ® 𝜙−𝑖 . ­ ­ 𝑗 ® 𝑇−𝑖 𝛼 𝑖=0 𝑗=𝑖+1 𝛼 𝑖=0 𝑗=𝑖+1 ¬ ¬ « « Since 𝜙𝑡 ≥ 0, ∀𝑡 and 𝜙0 = 𝜙−1 = · · · = 𝜙−𝑞+1 = 0, we have 𝑇 ∑︁ Δ𝜙𝑡 ≥ 0. (10.14) 𝑡=1 Summing (10.13) over timesteps 𝑡 = 1, 2, · · · , 𝑇, we see that 𝑇 ∑︁ (𝐻𝑡 + 𝜆 1 𝑀𝑡 ) + 𝑇 ∑︁ Δ𝜙𝑡 ≤ 𝑚 𝑡=1 𝑡=1 𝑡=1 𝑇  ∑︁ 𝑚 + 𝜆2 𝐻𝑡∗ +  𝜆 1 (𝜆 1 + 𝜆 2 + 𝑚) ∗ 𝑀 . (1 − 𝛼2 )𝜆 1 + 𝜆 2 + 𝑚 𝑡 Using (10.14), we obtain that 𝑇 ∑︁ (𝐻𝑡 + 𝜆 1 𝑀𝑡 ) ≤ 𝑡=1 𝑇  ∑︁ 𝑚 + 𝜆2 𝑚 𝐻𝑡∗ + 𝑡=1  𝜆 1 (𝜆 1 + 𝜆2 + 𝑚) ∗ 𝑀 , (1 − 𝛼2 )𝜆 1 + 𝜆 2 + 𝑚 𝑡 (10.15) which implies 𝑇 ∑︁ 𝑡=1 (𝐻𝑡 + 𝑀𝑡 ) ≤ 𝑇  ∑︁ 𝑚 + 𝜆2 𝑡=1 𝑚𝜆 1 𝐻𝑡∗ +  𝜆1 + 𝜆2 + 𝑚 ∗ 𝑀 . (1 − 𝛼2 )𝜆 1 + 𝜆 2 + 𝑚 𝑡 References Abbasi-Yadkori, Yasin, Nevena Lazic, and Csaba Szepesvári (2018). Regret bounds for model-free linear quadratic control. In: arXiv preprint arXiv:1804.06021. Abeille, Marc and Alessandro Lazaric (2018). Improved regret bounds for thompson sampling in linear quadratic control problems. In: International Conference on Machine Learning (ICML). Agarwal, Naman, Nataly Brukhim, Elad Hazan, and Zhou Lu (2019). Boosting for dynamical systems. In: arXiv preprint arXiv:1906.08720. Agarwal, Naman, Brian Bullins, Elad Hazan, Sham M. Kakade, and Karan Singh (2019). Online control with adversarial disturbances. In: International Conference on Machine Learning (ICML). Agarwal, Naman, Elad Hazan, and Karan Singh (2019). Logarithmic regret for online control. In: Neural Information Processing Systems (NeurIPS). Anava, Oren, Elad Hazan, and Shie Mannor (2015). Online learning for adversaries with memory: Price of past mistakes. In: Advances in Neural Information Processing Systems, pp. 784–792. 205 Andrew, Lachlan, Siddharth Barman, Katrina Ligett, Minghong Lin, Adam Meyerson, Alan Roytman, and Adam Wierman (2013). A tale of two metrics: Simultaneous bounds on competitiveness and regret. In: Conference on Learning Theory, pp. 741–763. Argue, C. J., Anupam Gupta, Guru Guruganesh, and Ziye Tang (2020). Chasing convex bodies with linear competitive ratio. In: SIAM Symposium on Discrete Algorithms (SODA). Badiei, Masoud, Na Li, and Adam Wierman (2015). Online convex optimization with ramp constraints. In: IEEE Conference on Decision and Control (CDC). Bansal, Nikhil, Anupam Gupta, Ravishankar Krishnaswamy, Kirk Pruhs, Kevin Schewior, and Cliff Stein (2015). A 2-competitive algorithm for online convex optimization with switching costs. In: Proceedings of the Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques (APPROX/RANDOM). Bubeck, Sébastien, Bo’az Klartag, Yin Tat Lee, Yuanzhi Li, and Mark Sellke (2020). Chasing nested convex bodies nearly optimally. In: SIAM Symposium on Discrete Algorithms (SODA). Bubeck, Sébastien, Yin Tat Lee, Yuanzhi Li, and Mark Sellke (2019). Competitively chasing convex bodies. In: ACM Symposium on Theory of Computing (STOC). Chen, Niangjun, Anish Agarwal, Adam Wierman, Siddharth Barman, and Lachlan Andrew (2015). Online convex optimization using predictions. In: Proceedings of ACM SIGMETRICS, pp. 191–204. Chen, Niangjun, Joshua Comden, Zhenhua Liu, Anshul Gandhi, and Adam Wierman (2016). Using predictions in online optimization: Looking forward with an eye on the past. In: Proceedings of ACM SIGMETRICS, pp. 193–206. Chen, Niangjun, Gautam Goel, and Adam Wierman (2018). Smoothed online convex optimization in high dimensions via online balanced descent. In: Proceedings of Conference On Learning Theory (COLT), pp. 1574–1594. Chen, Shiyao and Lang Tong (2012). iEMS for large scale charging of electric vehicles: Architecture and optimal online scheduling. In: 2012 IEEE Third International Conference on Smart Grid Communications (SmartGridComm). IEEE, pp. 629–634. Cohen, Alon, Avinatan Hasidim, Tomer Koren, Nevena Lazic, Yishay Mansour, and Kunal Talwar (2018). Online linear quadratic control. In: International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 1029–1038. Dean, Sarah, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu (2020). On the sample complexity of the linear quadratic regulator. In: Foundations of Computational Mathematics 20.4, pp. 633–679. 206 Goel, Gautam and Babak Hassibi (2020). The power of linear controllers in LQR control. In: arXiv preprint arXiv:2002.02574. Goel, Gautam, Yiheng Lin, Haoyuan Sun, and Adam Wierman (2019). Beyond online balanced descent: An optimal algorithm for smoothed online optimization. In: Neural Information Processing Systems (NeurIPS). Goel, Gautam and Adam Wierman (2019). An online algorithm for smoothed regression and LQR control. In: International Conference on Artificial Intelligence and Statistics (AISTATS). Hazan, Elad (2019). Introduction to online convex optimization. In: arXiv preprint arXiv:1909.05207. Joulani, Pooria, András György, and Csaba Szepesvári (2013). Online learning under delayed feedback. In: International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 1453–1461. Kim, Seung-Jun and Geogios B. Giannakis (2016). An online convex optimization approach to real-time energy pricing for demand response. In: IEEE Transactions on Smart Grid 8.6, pp. 2784–2793. Kirk, Donald E. (2004). Optimal control theory: An introduction. Courier Corporation. Lale, Sahin, Kamyar Azizzadenesheli, Babak Hassibi, and Anima Anandkumar (2020). Logarithmic regret bound in partially observable linear dynamical systems. In: arXiv preprint arXiv:2003.11227. Li, Yingying, Xin Chen, and Na Li (2019). Online optimal control with linear dynamics and predictions: Algorithms and regret analysis. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 14858–14870. Li, Yingying, Guannan Qu, and Na Li (2018a). Online optimization with predictions and switching costs: Fast algorithms and the fundamental limit. In: arXiv preprint arXiv:1801.07780. – (2018b). Using predictions in online optimization with switching costs: A fast algorithm and a fundamental limit. In: 2018 Annual American Control Conference (ACC). IEEE, pp. 3008–3013. Lin, Minghong, Zhenhua Liu, Adam Wierman, and Lachlan Andrew (2012). Online algorithms for geographical load balancing. In: Proceedings of the International Green Computing Conference (IGCC), pp. 1–10. Lin, Minghong, Adam Wierman, Lachlan Andrew, and Eno Thereska (2013). Dynamic right-sizing for power-proportional data centers. In: IEEE/ACM Transactions on Networking (TON) 21.5, pp. 1378–1391. 207 Lin, Qiulin, Hanling Yi, John Pang, Minghua Chen, Adam Wierman, Michael Honig, and Yuanzhang Xiao (2019). Competitive online optimization under inventory constraints. In: Proceedings of the ACM on Measurement and Analysis of Computing Systems 3.1, pp. 1–28. Lin, Yiheng, Gautam Goel, and Adam Wierman (2020). Online optimization with predictions and non-convex losses. In: Proceedings of ACM SIGMETRICS. Luenberger, David (1967). Canonical forms for linear multivariable systems. In: IEEE Transactions on Automatic Control 12.3, pp. 290–293. Pan, Weici, Guanya Shi, Yiheng Lin, and Adam Wierman (2022). Online optimization with feedback delay and nonlinear switching cost. In: Proceedings of the ACM on Measurement and Analysis of Computing Systems 6.1, pp. 1–34. doi: 10.1145/3508037. Sellke, Mark (2020). Chasing convex bodies optimally. In: SIAM Symposium on Discrete Algorithms (SODA). Shamir, Ohad and Liran Szlak (2017). Online learning with local permutations and delayed feedback. In: International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 3086–3094. Shi, Guanya, Yiheng Lin, Soon-Jo Chung, Yisong Yue, and Adam Wierman (2020). Online optimization with memory and competitive control. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 33. Curran Associates, Inc. url: https://arxiv.org/abs/2002.05318. Yang, Lin, Mohammad H. Hajiesmaili, Ramesh Sitaraman, Enrique Mallada, Wing S. Wong, and Adam Wierman (2020). Online inventory management with application to energy procurement in data centers. In: Proceedings of ACM Sigmetrics. Zhou, Kemin and John Comstock Doyle (1998). Essentials of robust control. Vol. 104. Prentice Hall Upper Saddle River, NJ. 208 C h a p t e r 11 ONLINE LEARNING PERSPECTIVES ON MODEL PREDICTIVE CONTROL Another critical question is begged in online learning and control: Do established control methods have strong learning guarantees? In particular, Model Predictive Control (MPC) has been one of the most successful methods in industrial control since the 1980s. However, many learning theorists are studying RL algorithms, but few are analyzing MPC and why it is so powerful. To close this gap, this chapter studies MPC from learning-theoretic perspectives, and proves the first nonasymptotic guarantee for MPC, showing that MPC is near-optimal in the sense of dynamic regret in online LQR control with predictable disturbance. This chapter also extends to settings with inexact predictions and linear time-variant (LTV) systems. These results found common ground for learning and control theory and imply fundamental algorithmic principles. This chapter is mainly based on the following papers1: Li, Tongxin, Ruixiao Yang, Guannan Qu, Guanya Shi, Chenkai Yu, Adam Wierman, and Steven Low (2022). Robustness and consistency in linear quadratic control with untrusted predictions. In: Proceedings of the ACM on Measurement and Analysis of Computing Systems 6.1, pp. 1–35. doi: 10.1145/3508038. Lin, Yiheng, Yang Hu, Guanya Shi, Haoyuan Sun, Guannan Qu, and Adam Wierman (2021). Perturbation-based regret analysis of predictive control in linear time varying systems. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 34. Curran Associates, Inc., pp. 5174–5185. url: https : / / arxiv.org/abs/2106.10497. Yu, Chenkai, Guanya Shi, Soon-Jo Chung, Yisong Yue, and Adam Wierman (2020). The power of predictions in online control. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 33. Curran Associates, Inc., pp. 1994– 2004. url: https : / / proceedings . neurips . cc / paper / 2020 / file / 155fa09596c7e18e50b58eb7e0c6ccb4-Paper.pdf. – (2022). Competitive control with delayed imperfect information. In: American Control Conference (ACC). url: https://arxiv.org/abs/2010.11637. 1 Blog post: https://www.gshi.me/blog/CompetitiveMPC/ 209 11.1 Introduction This chapter studies the effect of using predictions for online control in a linear dynamical system governed by 𝑥𝑡+1 = 𝐴𝑡 𝑥𝑡 + 𝐵𝑡 𝑢 𝑡 + 𝑤 𝑡 , where 𝑥𝑡 , 𝑢 𝑡 , and 𝑤 𝑡 are the state, control, and disturbance (or exogenous input), respectively. At each time step 𝑡, the controller incurs a cost 𝑐 𝑡 (𝑥𝑡 , 𝑢 𝑡 ). Recently, considerable effort has been made to leverage and integrate ideas from learning, optimization and control theory to study the design of optimal controllers under various performance criteria, such as static regret (Dean et al., 2020; Agarwal, Bullins, et al., 2019; Agarwal, Hazan, et al., 2019; Cohen et al., 2019; Hazan et al., 2020; D. J. Foster and Simchowitz, 2020; Simchowitz and D. Foster, 2020), dynamic regret (Y. Li et al., 2019; Goel and Hassibi, 2020) and competitive ratio (Chapter 10 and Goel and Wierman (2019)). However, the study of online convergence when incorporating predictions has been largely absent. Indeed, a key aspect of online control is considering the amount of available information when making decisions. Most recent studies focus on the basic setting where only past information, 𝑥 0 , 𝑤 0 , · · · , 𝑤 𝑡−1 , is available for 𝑢 𝑡 at every time step (Dean et al., 2020; Agarwal, Bullins, et al., 2019; D. J. Foster and Simchowitz, 2020). However, this basic setting does not effectively characterize situations where we have accurate predictions, e.g., when 𝑥 0 , 𝑤 0 , · · · , 𝑤 𝑡−1+𝑘 are available at step 𝑡. These types of accurate predictions are often available in many applications, including robotics (Baca et al., 2018), energy systems (Vazquez et al., 2016), and data center management (Lazic et al., 2018). Moreover, there are many practical algorithms that leverage predictions, such as the popular Model Predictive Control (MPC) (Baca et al., 2018; Camacho and Alba, 2013; Angeli, Casavola, et al., 2016; Angeli, Amrit, et al., 2011; Grüne and Pirkelmann, 2018; Grüne and Stieler, 2014). While there has been increased interest in studying online guarantees for control with predictions, to our knowledge, there has been no such study for the case of a finitetime horizon with disturbances. Several previous works studied the economic MPC problem by analyzing the asymptotic performance without disturbances (Angeli, Casavola, et al., 2016; Angeli, Amrit, et al., 2011; Grüne and Pirkelmann, 2018; Grüne and Stieler, 2014). Rosolia and Borrelli (2019) and Rosolia and Borrelli (2017) studied learning for MPC but focused on the episodic setting with asymptotic convergence guarantees. Y. Li et al. (2019) considered a linear system where finite predictions of costs are available, and analyzed the dynamic regret of their new algorithm; however, they neither consider disturbances nor study the more practically 210 relevant MPC approach. Goel and Hassibi (2020) characterized the offline optimal policy (i.e., with infinite predictions) and cost in LQR control with i.i.d. zero-mean stochastic disturbances, but those results do not apply to limited predictions or noni.i.d. disturbances. Other prior works analyze the power of predictions in online optimization (Lin, Goel, et al., 2020; Chen et al., 2015), but the connection to online control in dynamical systems is unclear. From this literature, fundamental questions about online control with predictions have emerged: 1. What are the cost-optimal and regret-minimizing policies when given 𝑘 predictions? What are the corresponding cost and regret of these policies? 2. What is the marginal benefit from each additional prediction used by the policy, and how many predictions are needed to achieve (near-)optimal performance? 3. How well does MPC with 𝑘 predictions perform compared to cost-optimal and regret-minimizing policies? Contributions and Organizations In Sections 11.2 to 11.4, we systematically address each of the questions above in the context of LQR systems with general stochastic and adversarial disturbances in the dynamics. In the stochastic case, we explicitly derive the cost-optimal and dynamic regret minimizing policies with 𝑘 predictions. In both the stochastic and adversarial cases, we derive (mostly tight) upper bounds for the optimal cost and minimum dynamic regret given access to 𝑘 predictions. We also show that the marginal benefit of an extra prediction exponentially decays as 𝑘 increases. We further show that MPC is near-optimal in terms of dynamic regret, and needs only 𝑂 (log 𝑇) predictions to achieve 𝑂 (1) dynamic regret (the same order as is needed by the dynamic regret minimizing policy) in both settings. Then in Section 11.5, we generalize the analysis framework and results from perfect predictions to settings with inexact predictions and delays. Section 11.5 also analyzes MPC’s competitive ratios. Finally in Section 11.6, we generalize the setting from standard LQR to Linear Time-Variant (LTV) systems with general well-conditioned costs. 211 Perhaps surprisingly, for all these settings (LQR, inexact predictions, or LTV), we have shown that classic MPC, which is a simple greedy policy (up to the prediction horizon), is near-optimal even with adversarial disturbances in the dynamics. Our results thus highlight the power of predictions to reduce the need for algorithmic sophistication. In that sense, our results somewhat mirror recent developments in the study of exploration strategies in online LQR control with unknown dynamics {𝐴, 𝐵}: after a decade’s research beginning with the work of Abbasi-Yadkori and Szepesvári (2011), Simchowitz and D. Foster (2020) recently show that naive exploration is optimal. Taken together with the result from Simchowitz and D. Foster (2020), we provide additional evidence for the idea that the structure of LQR allows simple algorithmic ideas to be effective, which sheds light on key algorithmic principles and fundamental limits in continuous control. 11.2 Problem Statement Throughout Sections 11.2 to 11.4, we consider the Linear Quadratic Regulator (LQR) optimal control problem with disturbances in the dynamics (introduced in Chapter 8). In particular, we consider a linear system initialized with 𝑥 0 ∈ R𝑛 and controlled by 𝑢 𝑡 ∈ R𝑑 , with dynamics 𝑥𝑡+1 = 𝐴𝑥𝑡 + 𝐵𝑢 𝑡 + 𝑤 𝑡 and cost 𝐽= 𝑇−1 ∑︁ (𝑥𝑡⊤ 𝑄𝑥𝑡 + 𝑢 𝑡⊤ 𝑅𝑢 𝑡 ) + 𝑥𝑇⊤ Qf 𝑥𝑇 , 𝑡=0 where 𝑇 ≥ 1 is the total length of the control period. The goal of the controller is to minimize the cost given 𝐴, 𝐵, 𝑄, 𝑅, Qf , 𝑥0 , and the characterization of the disturbance 𝑤 𝑡 . We make the following assumption about the system. Assumption 11.1. We assume 𝑄, Qf ⪰ 0, 𝑅 ≻ 0 and the pair ( 𝐴, 𝐵) is stabilizable, i.e., there exists a matrix 𝐾0 ∈ R𝑑×𝑛 such that 𝜌( 𝐴 − 𝐵𝐾0 ) < 1. Further, we assume the pair ( 𝐴, 𝑄) is detectable, i.e., ( 𝐴⊤ , 𝑄) is stabilizable, to guarantee stability of the closed-loop. Note that detectability of ( 𝐴, 𝑄) is more general than 𝑄 ≻ 0, i.e., 𝑄 ≻ 0 implies ( 𝐴, 𝑄) is detectable. For 𝑤 𝑡 , in the stochastic case, we assume {𝑤 𝑡 }𝑡=0,1,··· are sampled from a joint distribution with bounded cross-correlation,   i.e., E 𝑤 𝑡⊤ 𝑤 𝑡 ′ ≤ 𝑚 for any 𝑡, 𝑡 ′; in the adversarial case, we assume 𝑤 𝑡 is picked from a bounded set Ω and ∥𝑤 𝑡 ∥ ≤ 𝑟. Generality. These are standard assumptions in the literature, e.g., Dean et al. (2020), D. J. Foster and Simchowitz (2020), and Simchowitz and D. Foster (2020) 212 and it is worth noting that our notion of stochasticity is much more general than typically considered (Dean et al., 2020; Cohen et al., 2019; Cassel et al., 2020). We also note that many important problems can be straightforwardly converted to our model. For example, input-disturbed systems and the Linear Quadratic (LQ) tracking problem (B. D. O. Anderson and Moore, 2007). Example 11.1 (Linear quadratic tracking). The standard quadratic tracking problem Í is defined with dynamics 𝑥𝑡+1 = 𝐴𝑥𝑡 + 𝐵𝑢 𝑡 + 𝑤˜ 𝑡 and cost function 𝐽 = 𝑇−1 𝑡=0 (𝑥 𝑡+1 − ⊤ ⊤ 𝑇 𝑑𝑡+1 ) 𝑄(𝑥𝑡+1 − 𝑑𝑡+1 ) + 𝑢 𝑡 𝑅𝑢 𝑡 , where {𝑑𝑡 }𝑡=1 is the desired trajectory to track. To Í ⊤ ⊤ map this to our model, let 𝑥˜𝑡 = 𝑥𝑡 − 𝑑𝑡 . Then, we get 𝐽 = 𝑇−1 𝑡=0 𝑥˜𝑡+1 𝑄 𝑥˜𝑡+1 + 𝑢 𝑡 𝑅𝑢 𝑡 and 𝑥˜𝑡+1 = 𝐴𝑥˜𝑡 + 𝐵𝑢 𝑡 + 𝑤 𝑡 , which is an LQR control problem with disturbance 𝑤 𝑡 = 𝑤˜ 𝑡 + 𝐴𝑑𝑡 − 𝑑𝑡+1 in the dynamics. Predictions In the classic model, at each step 𝑡, the controller decides 𝑢 𝑡 after observing 𝑤 𝑡−1 and 𝑥𝑡 . In other words, 𝑢 𝑡 is a function of all the previous information: 𝑥 0 , 𝑥1 , . . . , 𝑥𝑡−1 and 𝑤 0 , 𝑤 1 , . . . , 𝑤 𝑡−1 , or equivalently, of 𝑥 0 , 𝑤 0 , 𝑤 1 , · · · , 𝑤 𝑡−1 . We describe this scenario via the following event sequence: 𝑥 0 𝑢 0 𝑤 0 𝑢 1 𝑤 1 · · · 𝑢𝑇−1 𝑤𝑇−1 , where each 𝑢 𝑡 denotes the decision of a control policy, each 𝑤 𝑡 denote the observation of a disturbance, and each decision may depend on previous events. However, in many real-world applications the controller may have some knowledge about future. In particular, at time step 𝑡, the controller may have predictions of immediate 𝑘 future disturbances and make decision 𝑢 𝑡 based on 𝑥 0 , 𝑤 0 , . . . , 𝑤 𝑡+𝑘−1 . In this case, the event sequence is given by: 𝑥 0 𝑤 0 · · · 𝑤 𝑘−1 𝑢 0 𝑤 𝑘 𝑢 1 𝑤 𝑘+1 · · · 𝑢𝑇−𝑘−1 𝑤𝑇−1 𝑢𝑇−𝑘 · · · 𝑢𝑇−1 . The existence of predictions is common in many applications such as disturbance estimation in robotics and model predictive control (MPC) (Camacho and Alba, 2013), which is a common approach for the LQ tracking problem. When given 𝑘 predictions of 𝑑𝑡 , the LQ tracking problem can be formulated as a LQR problem with 𝑘 − 1 predictions of future disturbances. Throughout Sections 11.2 to 11.4 we assume all the predictions are exact, and discuss inexact predictions in Section 11.5. 213 Disturbances The characteristics of the disturbances have a fundamental impact on the optimal control policy and cost. We consider two types of disturbance: stochastic disturbances, which are drawn from a joint distribution (not necessarily i.i.d.), and adversarial disturbances, which are chosen by an adversary to maximize the overall control cost of the policy. In the stochastic setting, we model the disturbance sequence {𝑤 𝑡 }𝑇−1 𝑡=0 as a discretetime stochastic process with joint distribution W which is known to the controller. Let 𝑊𝑡 = 𝑊𝑡 (𝑤 0 , . . . , 𝑤 𝑡−1 ) be the conditional distribution of 𝑤 𝑡 given 𝑤 0 , . . . , 𝑤 𝑡−1 . Then the cost of the optimal online policy with 𝑘 predictions is given by:       STO𝑇𝑘 = E min E · · · min E min 𝐽 . 𝑤 0 ∼𝑊0 ,...,𝑤 𝑘−1 ∼𝑊 𝑘−1 𝑢0 𝑤 𝑘 ∼𝑊 𝑘 𝑢𝑇−𝑘−1 𝑤 𝑇−1 ∼𝑊𝑇−1 𝑢𝑇−𝑘 ,...,𝑢𝑇−1 Note that the cost 𝐽 = 𝐽 (𝑥0 , 𝑢 0 , · · · , 𝑢𝑇−1 , 𝑤 0 , · · · , 𝑤𝑇−1 ). Two extreme cases are noteworthy: 𝑘 = 0 reduces to the classic case without prediction (in Chapter 8) and 𝑘 = 𝑇 reduces to the offline optimal. In the adversarial setting, each disturbance 𝑤 𝑡 is selected by an adversary from a bounded set Ω ⊆ R𝑛 in order to maximize the cost. The controller has no information about the disturbance except that it is in Ω. Similar to the stochastic setting, we define:       min 𝐽 . sup min sup · · · min ADV 𝑇𝑘 = sup 𝑤 0 ,...,𝑤 𝑘−1 ∈Ω 𝑢0 𝑤 𝑘 ∈Ω 𝑢𝑇−𝑘−1 𝑤 𝑇−1 ∈Ω 𝑢𝑇−𝑘 ,...,𝑢𝑇−1 This can be viewed as online H∞ control (Zhou and Doyle, 1998) with predictions. The average cost in an infinite horizon is particularly important in both control and learning communities to understand asymptotic behaviors. We use separate notation for it: 1 1 STO 𝑘 = lim STO𝑇𝑘 , ADV 𝑘 = lim ADV 𝑇𝑘 . 𝑇→∞ 𝑇 𝑇→∞ 𝑇 We emphasize that we do not have any constraints (like linearity) on the policy space, and both STO𝑇𝑘 and ADV 𝑇𝑘 are globally optimal with the corresponding type of disturbance. This point is important in light of recent results that show that linear policies cannot make use of predictions at all (see Chapter 10 and Goel and Hassibi (2020)), i.e., the cost of the best linear policy with infinite predictions (𝑘 = ∞) is asymptotically equal to that with no predictions (𝑘 = 0) in the setting with i.i.d. zero-mean stochastic disturbances. In this chapter, we explicitly derive the optimal policy for every 𝑘 > 0, which is nonlinear in general. 214 Model Predictive Control Algorithm 11.1: Model predictive control with 𝑘 predictions Parameter: {𝐴, 𝐵, 𝑄, 𝑅} and Q̃f ∈ R𝑛×𝑛 Input: 𝑥 0 , 𝑤 0 , . . . , 𝑤 𝑘−1 1 for 𝑡 = 0 to 𝑇 − 1 do Input: 𝑥𝑡 , 𝑤 𝑡+𝑘−1 // The controller now knows 𝑥0 , . . . , 𝑥𝑡 , 𝑤 0 , . . . , 𝑤 𝑡+𝑘−1 Í𝑡+𝑘−1 ⊤ ⊤ Q̃ 𝑥 2 (𝑢 𝑡 , . . . , 𝑢 𝑡+𝑘−1 ) = arg min𝑢 𝑖=𝑡 (𝑥𝑖 𝑄𝑥𝑖 + 𝑢𝑖⊤ 𝑅𝑢𝑖 ) + 𝑥𝑡+𝑘 f 𝑡+𝑘 subject to 𝑥𝑖+1 = 𝐴𝑥𝑖 + 𝐵𝑢𝑖 + 𝑤 𝑖 for 𝑖 = 𝑡, . . . , 𝑡 + 𝑘 − 1 Output: 𝑢 𝑡 As introduced in Chapter 8, Model predictive control (MPC) is perhaps the most common control policy for situations where predictions are available. MPC is a greedy algorithm with a receding horizon based on all available current predictions. Algorithm 11.1 provides a formal definition, and we additionally refer the reader to the book by Camacho and Alba (2013) for a literature review on MPC. We adopt a conventional definition of MPC as an online optimal control problem with a finite-time horizon with dynamics constraints. Note that other prior work on MPC sometimes considers other input and state constraints (Camacho and Alba, 2013). MPC is a practical algorithm in many scenarios like robotics (Baca et al., 2018), energy system (Vazquez et al., 2016) and data center cooling (Lazic et al., 2018). The existing theoretical studies of MPC focus on asymptotic stability and performance (Angeli, Casavola, et al., 2016; Angeli, Amrit, et al., 2011; Grüne and Pirkelmann, 2018; Grüne and Stieler, 2014; Rosolia and Borrelli, 2017). To our knowledge, we provide the first general, dynamic regret guarantee for MPC in this thesis. In Section 11.3, we study the performance of MPC in three different cases, where disturbances are i.i.d. zero-mean stochastic, generally stochastic, and adversarial. We define the performance of MPC in the stochastic and adversarial settings as follows: 1 MPCS𝑇𝑘 = E 𝐽 MPC𝑘 , MPCS 𝑘 = lim MPCS𝑇𝑘 , 𝑤 0 ,...,𝑤 𝑇−1 𝑇→∞ 𝑇 1 MPCA𝑇𝑘 = sup 𝐽 MPC𝑘 , MPCA 𝑘 = lim MPCA𝑇𝑘 , 𝑇→∞ 𝑇 𝑤 0 ,...,𝑤 𝑇−1 where 𝐽 MPC𝑘 is the cost of MPC given a specific disturbance sequence, i.e., 𝐽 MPC𝑘 (𝑤) = 𝐽 (𝑢, 𝑤) where for each 𝑡, 𝑢 𝑡 = 𝜙(𝑥𝑡 , 𝑤 𝑡 , . . . , 𝑤 𝑡+𝑘−1 ) and 𝜙(·) is the function that maps 𝑥𝑡 , 𝑤 𝑡 , . . . , 𝑤 𝑡+𝑘−1 to the policy 𝑢 𝑡 , as defined in Algorithm 11.1. By definition, MPCS𝑘 ≥ STO𝑘 and MPCA 𝑘 ≥ ADV 𝑘 for every 𝑘 ≥ 1 since they use the same information but the latter ones are defined to be optimal. 215 Dynamic Regret Throughout Sections 11.2 to 11.4, we use dynamic regret as the performance metric. In Section 11.5, we will also discuss another metric, competitive ratio. Regret is a standard metric in online learning and provides a bound on the cost difference between an online algorithm and the optimal static policy in a specific policy class given complete information. The predominant policy class is the linear policy class (𝑢 = 𝐾𝑥). We focus on the dynamic regret, which compares to the globally optimal offline policy without any constraint, rather than the optimal static offline policy in a specific class. Note that the globally optimal offline policy may be nonlinear or non-stationary. See more discussions about regret versus dynamic regret in Chapters 8 and 10. More specifically, we compare the cost of an online algorithm with 𝑘 predictions to that of the offline optimal (nonlinear) algorithm, i.e., one that has predictions of all disturbances. For MPC with 𝑘 predictions, we define its dynamic regret in the stochastic and adversarial settings, respectively, as:   ′ 𝑆 MPC 𝑘 𝑅𝑒𝑔 ( MPC 𝑘 ) = E 𝐽 (𝑤) − ′ min′ 𝐽 (𝑢 , 𝑤) , 𝑢 0 ,...,𝑢𝑇−1 (𝑤 0 ,··· ,𝑤 𝑇−1 )∼W   𝑅𝑒𝑔 𝐴 ( MPC 𝑘 ) = sup 𝐽 MPC𝑘 (𝑤) − ′ min′ 𝐽 (𝑢′, 𝑤) . 𝑤 0 ,··· ,𝑤 𝑇−1 ∈Ω 𝑢 0 ,...,𝑢𝑇−1 Again, as compared to (static) regret, dynamic regret does not have any restriction ′ on the policies 𝑢′0 , . . . , 𝑢𝑇−1 used for comparison and thus differs from other notions ′ ′ of regret where 𝑢 0 , . . . , 𝑢𝑇−1 are limited in special cases. For example, in the classic ′ ; and in the regret compared to the best offline linear form of regret, 𝑢′0 = · · · = 𝑢𝑇−1 controller (Agarwal, Bullins, et al., 2019; Cohen et al., 2019), 𝑢′𝑡 = −𝐾 ∗ 𝑥𝑡 . In Section 11.3, we obtain both upper bounds and lower bounds on dynamic regret. For lower bounds, we define the minimum possible regret that an algorithm with 𝑘 predictions can achieve (i.e., the regret of the algorithm that minimizes the regret):   ∗ 𝑅𝑒𝑔 𝑘𝑆 = E min E · · · min E min 𝐽 (𝑢, 𝑤) − ′ min′ 𝐽 (𝑢′, 𝑤) , 𝑢𝑇−𝑘−1 𝑤 𝑇−1 𝑢𝑇−𝑘 ,··· ,𝑢𝑇−1 𝑤 0 ,··· ,𝑤 𝑘−1 𝑢 0 𝑤 𝑘 𝑢 0 ,...,𝑢𝑇−1   ∗ 𝑅𝑒𝑔 𝑘𝐴 = sup min sup · · · min sup min 𝐽 (𝑢, 𝑤) − ′ min′ 𝐽 (𝑢′, 𝑤) . 𝑤 0 ,··· ,𝑤 𝑘−1 𝑢 0 𝑤𝑘 𝑢𝑇−𝑘−1 𝑤 𝑇−1 𝑢𝑇−𝑘 ,··· ,𝑢𝑇−1 𝑢 0 ,...,𝑢𝑇−1 Finally, we end our discussion of dynamic regret with a note highlighting an impor- 216 tant contrast between stochastic and adversarial settings. In the stochastic setting,   ∗ 𝑅𝑒𝑔 𝑘𝑆 = E min E · · · min E min 𝐽 (𝑢, 𝑤) − ′ min′ 𝐽 (𝑢′, 𝑤) 𝑤 0 ,··· ,𝑤 𝑘−1 𝑢 0 𝑤 𝑘 = E 𝑢𝑇−𝑘−1 𝑤 𝑇−1 min E · · · min E 𝑤 0 ,··· ,𝑤 𝑘−1 𝑢 0 𝑤 𝑘 𝑢𝑇−𝑘 ,··· ,𝑢𝑇−1 min 𝐽 (𝑢, 𝑤) − 𝑢𝑇−𝑘−1 𝑤 𝑇−1 𝑢𝑇−𝑘 ,··· ,𝑢𝑇−1 𝑢 0 ,...,𝑢𝑇−1 E min 𝐽 (𝑢′, 𝑤) ′ 𝑤 0 ,...,𝑤 𝑇−1 𝑢 0′ ,...,𝑢𝑇−1 = STO𝑇𝑘 − STO𝑇𝑇 . This equality still holds if we take arg min instead of min and thus the regret-optimal policy is the same as the cost-optimal policy. However, in the adversarial case, a ∗ similar reasoning gives an inequality: 𝑅𝑒𝑔 𝑘𝐴 ≥ ADV 𝑇𝑘 −ADV 𝑇𝑇 , and correspondingly, the regret-optimal and cost-optimal policies can be different. Similarly, for MPC, we have 𝑅𝑒𝑔 𝑆 ( MPC 𝑘 ) = MPCS𝑇𝑘 − STO𝑇𝑇 while 𝑅𝑒𝑔 𝐴 ( MPC 𝑘 ) ≥ MPCA𝑇𝑘 − ADV 𝑇𝑇 . 11.3 Dynamic Regret of MPC in LQ Systems Zero-Mean I.I.D. Disturbances We begin our analysis with the simplest of the three settings we consider: the disturbances 𝑤 𝑡 are independent and identically distributed with zero mean. Though i.i.d. zero-mean is a limited setting, it is still complex enough to study predictions and the first results characterizing the optimal policy with predictions appeared only recently (Goel and Hassibi, 2020; D. J. Foster and Simchowitz, 2020), focusing only on the optimal policy when 𝑘 → ∞. Before delving into our results, we first recap the classic Infinite Horizon Linear Quadratic Stochastic Regulator (see B. D. Anderson and Moore (2012) and Chapter 8), i.e., the case when 𝑘 = 0: Lemma 11.1 (B. D. Anderson and Moore (2012)). Let 𝑤 𝑡 be i.i.d. with zero mean and covariance matrix 𝑊. Then, the optimal control policy corresponding to STO0 is given by: 𝑢 𝑡 = −(𝑅 + 𝐵⊤ 𝑃𝐵) −1 𝐵⊤ 𝑃 𝐴𝑥𝑡 C −𝐾𝑥𝑡 , where 𝑃 is the solution of discrete-time algebraic Riccati equation (DARE) 𝑃 = 𝑄 + 𝐴⊤ 𝑃 𝐴 − 𝐴⊤ 𝑃𝐵(𝑅 + 𝐵⊤ 𝑃𝐵) −1 𝐵⊤ 𝑃 𝐴. (11.1) The corresponding closed-loop dynamics 𝐴 − 𝐵𝐾 is exponentially stable, i.e., 𝜌( 𝐴 − 𝐵𝐾) < 1. Further, the optimal cost is given by STO0 = Tr{𝑃𝑊 }. This result has been extensively studied in optimal control theory (Kirk, 2004; B. D. O. Anderson and Moore, 2007) as well as in reinforcement learning (Fazel 217 et al., 2018; Dean et al., 2020; Simchowitz and D. Foster, 2020). We want to emphasize two important properties of the optimal policy 𝑢 𝑡 = −𝐾𝑥𝑡 . First, the policy is linear in the state 𝑥𝑡 . In contrast, we show later that the optimal policy when 𝑘 ≠ 0 is, in general, nonlinear. Second, under the assumptions of our model, this policy is exponentially stable, i.e., 𝜌( 𝐴 − 𝐵𝐾) < 1. We leverage this to show the power of predictions later. Optimal policy with 𝑘 predictions. For notational simplicity, we define 𝐹 = 𝐴−𝐵𝐾 and 𝜆 = 1+𝜌(𝐹) < 1. From Gelfand’s formula, there exists a constant 𝑐(𝑛) such that 2 𝑘 𝑘 ∥𝐹 ∥ ≤ 𝑐(𝑛)𝜆 for all 𝑘 ≥ 1. We explicitly characterize the optimal policy with 𝑘 predictions in the following theorem. Theorem 11.1. Let 𝑤 𝑡 be i.i.d. with zero mean and covariance matrix 𝑊. Suppose the controller has 𝑘 ≥ 1 predictions. Then, the optimal control policy at each step 𝑡 is given by: 𝑘−1 ∑︁ © ª 𝑢 𝑡 = −(𝑅 + 𝐵⊤ 𝑃𝐵) −1 𝐵⊤ ­­𝑃 𝐴𝑥𝑡 + ( 𝐴⊤ − 𝐴⊤ 𝑃𝐻 ) 𝑖 𝑃𝑤 𝑡+𝑖 ®®, | {z } 𝑖=0 =𝐹 ⊤ « ¬ (11.2) where 𝑃 is the solution of DARE in Eq. (11.1). The cost under this policy is: ( ! ) 𝑘−1 ∑︁ STO 𝑘 = Tr 𝑃 − 𝑃( 𝐴 − 𝐻𝑃 𝐴) 𝑖 𝐻 ( 𝐴⊤ − 𝐴⊤ 𝑃𝐻) 𝑖 𝑃 𝑊 , (11.3) 𝑖=0 where 𝐻 = 𝐵(𝑅 + 𝐵⊤ 𝑃𝐵) −1 𝐵⊤ . Proof. Our proof technique closely follows that in Section 4.1 of Goel and Hassibi (2020). To begin, note that the definition of STO𝑇𝑘 has a structure of repeating min’s and E’s. Similarly to the analysis of Eq. (8.5) in Chapter 8, we use dynamic programming to compute the value iteratively. In particular, we apply backward induction to solve the optimal cost-to-go functions, from time step 𝑇 to the initial state. Given state 𝑥𝑡 and predictions 𝑤 𝑡 , . . . , 𝑤 𝑡+𝑘−1 , we define the cost-to-go function: 𝑉𝑡 (𝑥𝑡 ; 𝑤 𝑡:𝑡+𝑘−1 ) B min E min · · · E 𝑢 𝑡 𝑤 𝑡+𝑘 𝑢 𝑡+1 min 𝑇−1 ∑︁ 𝑤 𝑇−1 𝑢𝑇−𝑘 ,··· ,𝑢𝑇−1 (𝑥𝑖⊤ 𝑄𝑥𝑖 + 𝑢𝑖⊤ 𝑅𝑢𝑖 ) + 𝑥𝑇⊤ Qf 𝑥𝑇 𝑖=𝑡 (11.4)  ⊤ ⊤ = 𝑥𝑡 𝑄𝑥𝑡 + min 𝑢 𝑡 𝑅𝑢 𝑡 + E [𝑉𝑡+1 ( 𝐴𝑥𝑡 + 𝐵𝑢 𝑡 + 𝑤 𝑡 ; 𝑤 𝑡+1:𝑡+𝑘 )]  𝑢𝑡 𝑤 𝑡+𝑘 218 with 𝑉𝑇 (𝑥𝑇 ; . . . ) = 𝑥𝑇⊤ Qf 𝑥𝑇 . Note that E𝑤 𝑡+𝑘 has no effect for 𝑡 ≥ 𝑇 − 𝑘. This function measures the expected overall control cost from a given state to the end, assuming the controller makes the optimal decision at each time. We will show by backward induction that for every 𝑡 = 0, . . . , 𝑇, 𝑉𝑡 (𝑥𝑡 ; 𝑤 𝑡:𝑡+𝑘−1 ) = 𝑥𝑡⊤ 𝑃𝑡 𝑥𝑡 + 𝑣 𝑡⊤ 𝑥𝑡 + 𝑞 𝑡 , where 𝑃𝑡 , 𝑣 𝑡 , 𝑞 𝑡 are coefficients that may depend on 𝑤 𝑡:𝑡+𝑘−1 . This is clearly true for 𝑡 = 𝑇. Suppose this is true at 𝑡 + 1. Then, 𝑉𝑡 (𝑥; 𝑤 𝑡:𝑡+𝑘−1 )  = 𝑥 ⊤ 𝑄𝑥 + min 𝑢 ⊤ 𝑅𝑢 + ( 𝐴𝑥 + 𝐵𝑢 + 𝑤 𝑡 ) ⊤ 𝑃𝑡+1 ( 𝐴𝑥 + 𝐵𝑢 + 𝑤 𝑡 ) 𝑢  ⊤ + E [𝑣 𝑡+1 ] ( 𝐴𝑥 + 𝐵𝑢 + 𝑤 𝑡 ) + E [𝑞 𝑡+1 ] 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘 = 𝑥 𝑄𝑥 + ( 𝐴𝑥 + 𝑤 𝑡 ) 𝑃𝑡+1 ( 𝐴𝑥 + 𝑤 𝑡 ) + E [𝑣 𝑡+1 ] ⊤ ( 𝐴𝑥 + 𝑤 𝑡 ) + E [𝑞 𝑡+1 ] 𝑤 𝑤 𝑡+𝑘   𝑡+𝑘  ⊤ ⊤ ⊤ ⊤ + min 𝑢 (𝑅 + 𝐵 𝑃𝑡+1 𝐵)𝑢 + 𝑢 𝐵 2𝑃𝑡+1 𝐴𝑥 + 2𝑃𝑡+1 𝑤 𝑡 + E [𝑣 𝑡+1 ] . ⊤ ⊤ 𝑢 𝑤 𝑡+𝑘 The optimal 𝑢 is obtained by setting the derivative to be zero:   1 ∗ ⊤ −1 ⊤ 𝑢 = −(𝑅 + 𝐵 𝑃𝑡+1 𝐵) 𝐵 𝑃𝑡+1 𝐴𝑥 + 𝑃𝑡+1 𝑤 𝑡 + E [𝑣 𝑡+1 ] . 2 𝑤 𝑡+𝑘 (11.5) Let 𝐻𝑡 = 𝐵(𝑅 + 𝐵⊤ 𝑃𝑡+1 𝐵) −1 𝐵⊤ . Plugging 𝑢 ∗ back into 𝑉𝑡 , we have 𝑉𝑡 (𝑥; 𝑤 𝑡:𝑡+𝑘−1 ) = 𝑥 ⊤ 𝑄𝑥 + ( 𝐴𝑥 + 𝑤 𝑡 ) ⊤ 𝑃𝑡+1 ( 𝐴𝑥 + 𝑤 𝑡 ) + E [𝑣 𝑡+1 ] ⊤ ( 𝐴𝑥 + 𝑤 𝑡 ) + E [𝑞 𝑡+1 ] 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘 ⊤    1 1 E [𝑣 𝑡+1 ] 𝐻𝑡 𝑃𝑡+1 𝐴𝑥 + 𝑃𝑡+1 𝑤 𝑡 + E [𝑣 𝑡+1 ] − 𝑃𝑡+1 𝐴𝑥 + 𝑃𝑡+1 𝑤 𝑡 + 2 𝑤 𝑡+𝑘 2 𝑤 𝑡+𝑘
= 𝑥 ⊤ 𝑄 + 𝐴⊤ 𝑃𝑡+1 𝐴 − 𝐴⊤ 𝑃𝑡+1 𝐻𝑡 𝑃𝑡+1 𝐴 𝑥   ⊤ ⊤ ⊤ ⊤ ⊤ + 𝑥 ( 𝐴 − 𝐴 𝑃𝑡+1 𝐻𝑡 ) E [𝑣 𝑡+1 ] + 2( 𝐴 − 𝐴 𝑃𝑡+1 𝐻𝑡 )𝑃𝑡+1 𝑤 𝑡 𝑤 𝑡+𝑘 + 𝑤 𝑡⊤ (𝑃𝑡+1 − 𝑃𝑡+1 𝐻𝑡 𝑃𝑡+1 )𝑤 𝑡 + 𝑤 𝑡⊤ (𝐼 − 𝑃𝑡+1 𝐻𝑡 ) E [𝑣 𝑡+1 ] 𝑤 𝑡+𝑘 1 − E [𝑣 𝑡+1 ] ⊤ 𝐻𝑡 E [𝑣 𝑡+1 ] + E [𝑞 𝑡+1 ]. 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘 4 𝑤 𝑡+𝑘 Thus, the recursive formulae are given by: 𝑃𝑡 = 𝑄 + 𝐴⊤ 𝑃𝑡+1 𝐴 − 𝐴⊤ 𝑃𝑡+1 𝐻𝑡 𝑃𝑡+1 𝐴, (11.6a) 𝑣 𝑡 = ( 𝐴⊤ − 𝐴⊤ 𝑃𝑡+1 𝐻𝑡 ) E [𝑣 𝑡+1 ] + 2( 𝐴⊤ − 𝐴⊤ 𝑃𝑡+1 𝐻𝑡 )𝑃𝑡+1 𝑤 𝑡 , (11.6b) 𝑤 𝑡+𝑘 𝑞 𝑡 = 𝑤 𝑡⊤ (𝑃𝑡+1 − 𝑃𝑡+1 𝐻𝑡 𝑃𝑡+1 )𝑤 𝑡 + 𝑤 𝑡⊤ (𝐼 − 𝑃𝑡+1 𝐻𝑡 ) E [𝑣 𝑡+1 ] 𝑤 𝑡+𝑘 − 1 E [𝑣 𝑡+1 ] ⊤ 𝐻𝑡 E [𝑣 𝑡+1 ] + E [𝑞 𝑡+1 ]. 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘 4 𝑤 𝑡+𝑘 (11.6c) 219 As 𝑇 − 𝑡 → ∞, 𝑃𝑡 and 𝐻𝑡 converge to 𝑃 and 𝐻, respectively, where 𝑃 is the solution of discrete-time algebraic Riccati equation (DARE) 𝑃 = 𝑄 + 𝐴⊤ 𝑃 𝐴 − 𝐴⊤ 𝑃𝐻𝑃 𝐴, and 𝐻 = 𝐵(𝑅 + 𝐵⊤ 𝑃𝐵) −1 𝐵⊤ . Note that 𝑣𝑇 = 0 and 𝑞𝑇 = 0. Then, 𝑣𝑡 = 2 𝑘−1 ∑︁ ( 𝐴⊤ − 𝐴⊤ 𝑃𝐻) 𝑖+1 𝑃𝑤 𝑡+𝑖 , (11.7) 𝑖=0 𝑞 𝑡 = 𝑤 𝑡⊤ (𝑃 − 𝑃𝐻𝑃)𝑤 𝑡 + 𝑤 𝑡⊤ (𝐼 − 𝑃𝐻) E [𝑣 𝑡+1 ] − 𝑤 𝑡+𝑘 E [𝑣 𝑡+1 ] = 2 𝑘−1 ∑︁ 1 E [𝑣 𝑡+1 ] ⊤ 𝐻 E [𝑣 𝑡+1 ] + E [𝑞 𝑡+1 ], 𝑤 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘 4 𝑡+𝑘 (11.8) ( 𝐴⊤ − 𝐴⊤ 𝑃𝐻) 𝑖 𝑃𝑤 𝑡+𝑖 . (11.9) 𝑤 𝑡+𝑘 𝑖=1 Taking the expectation of 𝑞 𝑡 over all randomness, namely 𝑤 0 , 𝑤 1 , 𝑤 2 , . . . , we have E[𝑞 𝑡 ] = Tr{(𝑃 − 𝑃𝐻𝑃)𝑊 } − 𝑘−1 ∑︁  Tr 𝑃( 𝐴 − 𝐻𝑃 𝐴) 𝑖 𝐻 ( 𝐴⊤ − 𝐴⊤ 𝑃𝐻) 𝑖 𝑃𝑊 + E[𝑞 𝑡+1 ] 𝑖=1 ( = Tr 𝑃− 𝑘−1 ∑︁ ! ) 𝑖 ⊤ ⊤ 𝑖 𝑃( 𝐴 − 𝐻𝑃 𝐴) 𝐻 ( 𝐴 − 𝐴 𝑃𝐻) 𝑃 𝑊 + E[𝑞 𝑡+1 ], (11.10) 𝑖=0 where in the first equality we use E[𝑤 𝑡 ] = 0 and the independence of the disturbances. Thus, as 𝑇 → ∞, in each time step, a constant cost is incurred and the average cost STO 𝑘 is exactly this value. 1 1 1 STO𝑇𝑘 = lim E[𝑉0 (𝑥 0 ; 𝑤 0:𝑘−1 )] = lim E[𝑞 0 ] 𝑇→∞ 𝑇 𝑇→∞ 𝑇 𝑇→∞ 𝑇 ( ! ) 𝑇−1 𝑘−1 ∑︁ ∑︁ 1 = lim E[𝑞 𝑡 ] − E[𝑞 𝑡+1 ] = Tr 𝑃 − 𝑃( 𝐴 − 𝐻𝑃 𝐴) 𝑖 𝐻 ( 𝐴⊤ − 𝐴⊤ 𝑃𝐻) 𝑖 𝑃 𝑊 . 𝑇→∞ 𝑇 𝑡=0 𝑖=0 STO 𝑘 = lim The explicit form of the optimal control policy is obtained by combining Eqs. (11.5) and (11.9). □ Roughly speaking, the proof is based on an analysis of quadratic cost-to-go functions in the form 𝑉𝑡 (𝑥𝑡 ) = 𝑥𝑡⊤ 𝑃𝑡 𝑥𝑡 + 𝑣 𝑡⊤ 𝑥𝑡 + 𝑞 𝑡 (Chapter 8’s analysis does not have the linear term 𝑣 𝑡⊤ 𝑥𝑡 ). Note that 𝐴 − 𝐻𝑃 𝐴 = 𝐴 − 𝐵(𝑅 + 𝐵⊤ 𝑃𝐵) −1 𝐵⊤ 𝑃 𝐴 = 𝐴 − 𝐵𝐾 = 𝐹. Thus, the online optimal cost STO 𝑘 with 𝑘 predictions approaches the offline optimal cost STO∞ by an exponential rate. In other words, STO 𝑘 /STO∞ = 1 + 𝑂 (∥𝐹 𝑘 ∥ 2 ) = 1 + 𝑂 (𝜆2𝑘 ). Two extreme cases of our result are noteworthy. When 𝑘 = 0, it reduces to the classic Lemma 11.1. When 𝑘 → ∞, it reduces to the offline optimal case derived by Goel and Hassibi (2020). 220 MPC’s performance. As might be expected, since the disturbances are i.i.d., future disturbances have no dependence on the current. As a result, MPC in Algorithm 11.1 with Qf = 𝑃 gives the optimal policy, as shown below. Theorem 11.2. In Algorithm 11.1, let Q̃f = 𝑃. Then, the MPC policy with 𝑘 predictions is always given by Eq. (11.2) (no matter 𝑤 𝑡 is i.i.d. stochastic, generally stochastic, or adversarial). Moreover, assuming i.i.d. disturbance with zero mean, the MPC policy is optimal (both cost-optimal and dynamic-regret-optimal). Proof. Due to the greedy nature, MPC policy is given by the solution of a length-𝑘 optimal control problem, given deterministic and known 𝑤 𝑡 , · · · , 𝑤 𝑡+𝑘−1 . In other words, we want to derive the optimal policy (𝑢 𝑡 , . . . , 𝑢 𝑡+𝑘−1 ) that minimizes 𝑡+𝑘−1 ∑︁ ⊤ (𝑥𝑖⊤ 𝑄𝑥𝑖 + 𝑢𝑖⊤ 𝑅𝑢𝑖 ) + 𝑥𝑡+𝑘 𝑃𝑥𝑡+𝑘 , 𝑖=𝑡 where 𝑥𝑖+1 = 𝐴𝑥𝑖 + 𝐵𝑢𝑖 + 𝑤 𝑖 , given 𝑥𝑡 , 𝑤 𝑡 , . . . , 𝑤 𝑡+𝑘−1 . Define the cost-to-go function at time 𝑖 given 𝑥𝑖 , 𝑤𝑖 , . . . , 𝑤 𝑡+𝑘−1 : 𝑉𝑖 (𝑥𝑖 ; 𝑤 𝑖:𝑡+𝑘−1 ) = min 𝑡+𝑘−1 ∑︁ 𝑢 𝑖:𝑡+𝑘−1 ⊤ (𝑥 ⊤𝑗 𝑄𝑥 𝑗 + 𝑢 ⊤𝑗 𝑅𝑢 𝑗 ) + 𝑥𝑡+𝑘 𝑃𝑥𝑡+𝑘 𝑗=𝑖 = 𝑥𝑖⊤ 𝑄𝑥𝑖 + min (𝑢𝑖⊤ 𝑅𝑢𝑖 + 𝑉𝑖+1 ( 𝐴𝑥𝑖 + 𝐵𝑢𝑖 + 𝑤 𝑖 ; 𝑤 𝑖+1:𝑡+𝑘−1 )). 𝑢 𝑖 ⊤ 𝑃𝑥 Note that 𝑉𝑡+𝑘 (𝑥𝑡+𝑘 ) = 𝑥𝑡+𝑘 𝑡+𝑘 . Similar to the proof of Theorem 11.1, we can inductively show that 𝑉𝑖 (𝑥𝑖 ; 𝑤 𝑖:𝑡+𝑘−1 ) = 𝑥𝑖⊤ 𝑃𝑥𝑖 + 𝑣 𝑖⊤ 𝑥𝑖 + 𝑞𝑖 for some 𝑣 𝑖 and 𝑞𝑖 . Note that the second-degree coefficient no longer depends on the index 𝑖 as in the previous proof because we start from 𝑃, the solution of DARE. We then have the followings equations that parallel with Equations (11.5) and (11.7): 𝑣𝑖 = 2 𝑡+𝑘−𝑖−1 ∑︁ 𝐹⊤ 𝑗+1 𝑃𝑤 𝑖+ 𝑗 , 𝑗=0 𝑢𝑖∗ = −(𝑅 + 𝐵⊤ 𝑃𝐵) −1 𝐵⊤  1 𝑃 𝐴𝑥𝑖 + 𝑃𝑤 𝑖 + 𝑣 𝑖+1 2  𝑡+𝑘−𝑖−1 ∑︁ © ª 𝑗 = −(𝑅 + 𝐵⊤ 𝑃𝐵) −1 𝐵⊤ ­𝑃 𝐴𝑥𝑖 + 𝐹 ⊤ 𝑃𝑤 𝑖+ 𝑗 ®. 𝑗=0 « ¬ The case 𝑖 = 𝑡 gives: 𝑘−1 ∑︁ © ª 𝑗 𝑢 𝑡∗ = −(𝑅 + 𝐵⊤ 𝑃𝐵) −1 𝐵⊤ ­𝑃 𝐴𝑥𝑡 + 𝐹 ⊤ 𝑃𝑤 𝑡+ 𝑗 ®, 𝑗=0 « ¬ which is the MPC policy at time step 𝑡, and is same as Equation (11.2). □ 221 General Stochastic Disturbance In this section, we consider a general form of stochastic disturbance, more general than typically considered in this context (Dean et al., 2020; Cohen et al., 2019; Cassel et al., 2020). Suppose the disturbance sequence {𝑤 𝑡 }𝑡=0,1,2,... is sampled from a joint distribution W such that the trace of the cross-correlation of each pair is uniformly   bounded, i.e., there exist 𝑚 > 0 such that for all 𝑡, 𝑡 ′ ≥ 1, E 𝑤 𝑡⊤ 𝑤 𝑡 ′ ≤ 𝑚. Optimal policy with 𝑘 predictions. In the case of general stochastic disturbances, we cannot obtain as clean a form for STO 𝑘 as in the i.i.d. case in Theorem 11.1. However, the marginal benefit of having an extra prediction decays with the same (exponential) rate and the optimal policy is similar to that in Theorem 11.1, but with some additional terms that characterize the expected future disturbances given the current information, as shown in the following theorem. Theorem 11.3. The optimal control policy with general stochastic disturbance is given by:   ∞ 𝑘−1 ∑︁ ∑︁ ⊤𝑖 ⊤ −1 ⊤ ⊤𝑖 𝐹 𝑃𝜇𝑡+𝑖|𝑡+𝑘−1 , (11.11) 𝑢 𝑡 = −(𝑅 + 𝐵 𝑃𝐵) 𝐵 𝑃 𝐴𝑥𝑡 + 𝐹 𝑃𝑤 𝑡+𝑖 + 𝑖=0 𝑖=𝑘 where 𝜇𝑡 ′ |𝑡 = E[𝑤 𝑡 ′ | 𝑤 0 , . . . , 𝑤 𝑡 ]. Under this policy, the marginal benefit of obtaining an extra prediction decays exponentially fast in the existing number 𝑘 of predictions. Formally, for 𝑘 ≥ 1, STO 𝑘 − STO 𝑘+1 = 𝑂 (∥𝐹 𝑘 ∥ 2 ) = 𝑂 (𝜆2𝑘 ).2 Proof. Similar to the proof of Theorem 11.1, we assume 𝑉𝑡 (𝑥𝑡 ; 𝑤 0:𝑡+𝑘−1 ) = 𝑥𝑡⊤ 𝑃𝑡 𝑥𝑡 + 𝑥𝑡⊤ 𝑣 𝑡 + 𝑞 𝑡 , where 𝑉𝑡 has a similar definition as in Eq. (11.4) but may further depend on 𝑤 0 , . . . , 𝑤 𝑡−1 because the disturbance sequence is no longer Markovian. In this case, 𝑃𝑡 , 𝑣 𝑡 and 𝑞 𝑡 still satisfy the recursive forms in Eq. (11.6). However, the expected values of 𝑤 𝑡 and 𝑣 𝑡 are different since we have a more general distribution now. Let 𝑇 − 𝑡 → ∞, 𝜇𝑡 ′ |𝑡 = E[𝑤 𝑡 ′ | 𝑤 0 , . . . , 𝑤 𝑡 ] and 𝐹 = 𝐴 − 𝐻𝑃 𝐴. Then, 𝑣 𝑡𝑘 = 2 𝑘−1 ∑︁ 𝑖=0 𝐹⊤ 𝑖+1 𝑃𝑤 𝑡+𝑖 + 2 ∞ ∑︁ 𝐹⊤ 𝑖+1 𝑃𝜇𝑡+𝑖|𝑡+𝑘−1 , (11.12) 𝑖=𝑘  𝑘  1  𝑘 ⊤  𝑘   𝑘  𝑞 𝑡𝑘 = 𝑤 𝑡⊤ (𝑃 − 𝑃𝐻𝑃)𝑤 𝑡 + 𝑤 𝑡⊤ (𝐼 − 𝑃𝐻) E 𝑣 𝑡+1 − E 𝑣 𝑡+1 𝐻 E 𝑣 𝑡+1 + E 𝑞 𝑡+1 , 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘 4 𝑤 𝑡+𝑘 2 We say that 𝑓 (𝑘) = 𝑂 (𝑔(𝑘)) if ∃𝐶 > 0, ∀𝑘 ≥ 1, | 𝑓 (𝑘)| ≤ 𝐶 𝑔(𝑘); Ω() is similar except that the last “≤” is replaced by “≥”; Θ() means both 𝑂 () and Ω(). This is stronger than the standard definition where 𝑓 (𝑘) = 𝑂 (𝑔(𝑘)) if ∃𝐶 > 0, 𝑘 ∗ > 0, ∀𝑘 ≥ 𝑘 ∗ , | 𝑓 (𝑘)| ≤ 𝐶 𝑔(𝑘). 222 where the superscript 𝑘 denotes the number of predictions. The optimal policy in this case has the same form as Eq. (11.5). Plugging Eq. (11.12) into it, we obtain the optimal policy in the theorem. Further,     𝑘   𝑘   𝑘+1  𝑘+1 ⊤ (11.13a) E 𝑞𝑡 − 𝑞𝑡 = E 𝑤 𝑡 (𝐼 − 𝑃𝐻) E 𝑣 𝑡+1 − E 𝑣 𝑡+1 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘+1    𝑘+1  ⊤  𝑘+1   𝑘 ⊤  𝑘  1 + E E 𝑣 𝑡+1 𝐻 E 𝑣 𝑡+1 − E 𝑣 𝑡+1 𝐻 E 𝑣 𝑡+1 𝑤 𝑡+𝑘+1 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘 4 𝑤 𝑡+𝑘+1 (11.13b)  𝑘  𝑘+1 + E 𝑞 𝑡+1 − 𝑞 𝑡+1 , (11.13c) where the expectation E is taken over all randomness. Part (11.13a) is zero because  𝑘  E 𝑣 𝑡+1 = 𝑤 𝑡+𝑘 E 𝑤 𝑡+𝑘 ,𝑤 𝑡+𝑘+1  𝑘+1  𝑣 𝑡+1 .      𝑘+1   𝑘  ⊤  𝑘+1   𝑘  1 Part (11.13b) = E E 𝑣 − E 𝑣 𝑡+1 𝐻 E 𝑣 𝑡+1 − E 𝑣 𝑡+1 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘+1 𝑤 𝑡+𝑘 4 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘+1 𝑡+1  ⊤  = E 𝑧 𝑘,𝑡 𝐻𝑧 𝑘,𝑡 , 𝑤 𝑡+𝑘 where 𝑧 𝑘,𝑡 = 𝐹 ⊤𝑘 𝑃(𝑤 𝑡+𝑘 − 𝜇𝑡+𝑘 |𝑡+𝑘−1 ) + ∞ ∑︁ 𝐹 ⊤ 𝑃(𝜇𝑡+𝑖|𝑡+𝑘 − 𝜇𝑡+𝑖|𝑡+𝑘−1 ). 𝑖 𝑖=𝑘+1 Note that 𝑧 𝑘,𝑡 = 𝐹 ⊤ 𝑧 𝑘−1,𝑡+1 = 𝐹 ⊤ 𝑘 𝑧0,𝑡+𝑘 . Thus,  1  𝑘 E 𝑞 0 − 𝑞 0𝑘+1 𝑇→∞ 𝑇 𝑇−1  1 ∑︁  ⊤ = lim E 𝑧 𝑘,𝑡 𝐻𝑧 𝑘,𝑡 𝑇→∞ 𝑇 𝑡=0 STO 𝑘 − STO 𝑘+1 = lim 𝑇−1 i 1 ∑︁ h ⊤ 𝑘 ⊤𝑘 E 𝑧 0,𝑡+𝑘 𝐹 𝐻𝐹 𝑧0,𝑡+𝑘 = lim 𝑇→∞ 𝑇 𝑡=0 𝑇−1 o 1 ∑︁ n 𝑘 𝑘  Tr 𝐹 𝐻𝐹 ⊤ E 𝑧 0,𝑡+𝑘 𝑧⊤ = lim 0,𝑡+𝑘 𝑇→∞ 𝑇 𝑡=0 2 𝑇−1  1 ∑︁   Tr E 𝑧0,𝑡+𝑘 𝑧⊤ 0,𝑡+𝑘 𝑇→∞ 𝑇 𝑡=0 ≤ 𝐹 𝑘 ∥𝐻 ∥ lim 223 where in the last line we use the fact that if 𝐴 is symmetric, Tr{𝐴𝐵} n then h io ≤ 𝜆max ( 𝐴)Tr{𝐵}. Finally we just need to show the last item Tr E 𝑧 0,𝑡+𝑘 𝑧⊤ is 0,𝑡+𝑘 uniformly bounded for all 𝑡. This is straightforward because the cross-correlation of each disturbance pair is uniformly bounded, i.e., there exists 𝑚 > 0 such that for   all 𝑡, 𝑡 ′ ≥ 1, E 𝑤 𝑡⊤ 𝑤 𝑡 ′ ≤ 𝑚. ∞ n h io ∑︁    𝑗 ⊤ Tr E 𝑃𝐹 𝑖 𝐹 ⊤ 𝑃(𝜇𝑡+ 𝑗 |𝑡 − 𝜇𝑡+ 𝑗 |𝑡−1 )(𝜇𝑡+𝑖|𝑡 − 𝜇𝑡+𝑖|𝑡−1 ) ⊤ Tr E 𝑧0,𝑡 𝑧0,𝑡 = = ≤ ≤ 𝑖, 𝑗=0 ∞ ∑︁ 𝑖, 𝑗=0 ∞ ∑︁ 𝑖, 𝑗=0 ∞ ∑︁ 𝑖, 𝑗=0 n h io 𝑗 ⊤ ⊤ Tr 𝑃𝐹 𝑖 𝐹 ⊤ 𝑃 E 𝜇𝑡+ 𝑗 |𝑡 𝜇𝑡+𝑖|𝑡 − 𝜇𝑡+ 𝑗 |𝑡−1 𝜇𝑡+𝑖|𝑡−1 h i ⊤ ⊤ 𝐹 𝑖 𝐹 𝑗 ∥𝑃∥ 2 E 𝑤 𝑡+ 𝑤 − 𝑤 𝑤 𝑗 𝑡+𝑖 𝑡+ 𝑗 𝑡+𝑖 𝑐𝜆 𝑐𝜆 ∥𝑃∥ 2 2𝑚 = 2 𝑖 𝑗 𝑐2 ∥𝑃∥ 2 𝑚 2 (1 − 𝜆) n h io for some constant 𝑐 from Gelfand’s formula. Thus Tr E 𝑧 0,𝑡 𝑧⊤ is bounded by a 0,𝑡 constant independent of 𝑡. Thus, STO 𝑘 − STO 𝑘+1 = 𝑂 (∥𝐹 𝑘 ∥ 2 ). □ Roughly speaking, this proof leverages a novel difference analysis of cost-to-go functions. Note that for some distributions, STO 𝑘 may approach STO∞ much faster than exponential rate. It is even possible that STO 𝑘 = STO∞ for finite 𝑘, as we show in Example 11.2 below. On the other hand, there are scenarios where STO 𝑘 approaches STO∞ in an exactly exponential manner, as we show in Example 11.3 below. Example 11.2. Define the joint distribution W such that with probability 12 , all 𝑤 𝑡 = 𝑤, and otherwise all 𝑤 𝑡 = −𝑤. In this case, one prediction is equivalent to infinite predictions since it is enough to distinguish these two scenarios with only 𝑤 0 . As a result, STO1 = STO∞ . Example 11.3. Suppose the system is 1-d (𝑛 = 𝑑 = 1) and the disturbance is i.i.d. with zero mean, i.e., the setting of Theorem 11.1. Then, according to Eq. (11.3), as 224 long as 𝐹, 𝑃, 𝐻, 𝑊 are non-zero, STO 𝑘 − STO∞ = ∞ ∑︁ 𝐹 2𝑖 𝑃2 𝐻𝑊 = Θ(𝐹 2𝑘 ). 𝑖=𝑘 MPC’s performance. The comparison between the MPC policy in Eq. (11.2) and the optimal policy in Eq. (11.11) reveals that MPC is a truncation of the optimal policy and is no longer optimal because MPC is a greedy policy without considering future dependence on current information. Nevertheless, it is still a near-optimal policy, as characterized by the following results. Theorem 11.4. MPCS 𝑘 − MPCS 𝑘+1 = 𝑂 (∥𝐹 𝑘 ∥ 2 ) = 𝑂 (𝜆2𝑘 ). Moreover, in Example 11.3, MPCS 𝑘 − MPCS 𝑘+1 = Θ(∥𝐹 𝑘 ∥ 2 ). Proof. To recursively calculate the value of 𝐽 MPC𝑘 , we define: 𝑉𝑡MPC𝑘 (𝑥𝑡 ; 𝑤 0:𝑡+𝑘−1 ) = 𝑇−1 ∑︁ (𝑥𝑖⊤ 𝑄𝑥𝑖 + 𝑢𝑖⊤ 𝑅𝑢𝑖 ) + 𝑥𝑇⊤ Qf 𝑥𝑇 𝑖=𝑡 = 𝑥𝑡⊤ 𝑄𝑥𝑡 + 𝑢 𝑡⊤ 𝑅𝑢 𝑡 + 𝑉𝑡+1 ( 𝐴𝑥𝑡 + 𝐵𝑢 𝑡 + 𝑤 𝑡 ; 𝑤 0:𝑡+𝑘 ) as the cost-to-go function with MPC as the policy, i.e., 𝑢 𝑡 is the control at time step 𝑡 from the MPC policy with 𝑘 predictions. Similar to the previous proofs, we assume 𝑉𝑡MPC𝑘 (𝑥) = 𝑥 ⊤ 𝑃𝑡 𝑥 + 𝑥 ⊤ 𝑣 𝑡 + 𝑞 𝑡 (which turns out to be correct by induction) and 𝑇 − 𝑡 → ∞ so that 𝑃𝑡 = 𝑃. Then, 𝑉𝑡MPC𝑘 (𝑥𝑡 ; 𝑤 0:𝑡+𝑘−1 ) = 𝑥𝑡⊤ 𝑄𝑥𝑡 + 𝑢 𝑡⊤ 𝑅𝑢 𝑡 + ( 𝐴𝑥𝑡 + 𝐵𝑢 𝑡 + 𝑤 𝑡 ) ⊤ 𝑃( 𝐴𝑥𝑡 + 𝐵𝑢 𝑡 + 𝑤 𝑡 ) + ( 𝐴𝑥𝑡 + 𝐵𝑢 𝑡 + 𝑤 𝑡 ) ⊤ 𝑣 𝑡+1 + 𝑞 𝑡+1 = 𝑢 𝑡⊤ (𝑅 + 𝐵⊤ 𝑃𝐵)𝑢 𝑡 + 2𝑢 𝑡⊤ 𝐵⊤ (𝑃 𝐴𝑥𝑡 + 𝑃𝑤 𝑡 + 𝑣 𝑡+1 /2) + 𝑥𝑡⊤ 𝑄𝑥𝑡 + ( 𝐴𝑥𝑡 + 𝑤 𝑡 ) ⊤ 𝑃( 𝐴𝑥𝑡 + 𝑤 𝑡 ) + ( 𝐴𝑥𝑡 + 𝑤 𝑡 ) ⊤ 𝑣 𝑡+1 + 𝑞 𝑡+1 . (11.14) 225 Let 𝐹 = 𝐴 − 𝐻𝑃 𝐴. Plugging in the formula of 𝑢 𝑡 in Theorem 11.2, we have !⊤ ! 𝑘−1 𝑘−1 ∑︁ ∑︁ 1 1 𝑖 𝑖 𝑉𝑡MPC𝑘 (𝑥𝑡 ; 𝑤 0:𝑡+𝑘−1 ) = 𝑣 𝑡+1 − 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 𝐻 𝑣 𝑡+1 − 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 2 2 𝑖=1 𝑖=1  ⊤   1 1 − 𝑃 𝐴𝑥𝑡 + 𝑃𝑤 𝑡 + 𝑣 𝑡+1 𝐻 𝑃 𝐴𝑥𝑡 + 𝑃𝑤 𝑡 + 𝑣 𝑡+1 2 2 + 𝑥𝑡⊤ 𝑄𝑥𝑡 + ( 𝐴𝑥𝑡 + 𝑤 𝑡 ) ⊤ 𝑃( 𝐴𝑥𝑡 + 𝑤 𝑡 ) + ( 𝐴𝑥𝑡 + 𝑤 𝑡 ) ⊤ 𝑣 𝑡+1 + 𝑞 𝑡+1 = 𝑥𝑡⊤ (𝑄 + 𝐴⊤ 𝑃 𝐴 − 𝐴⊤ 𝑃𝐻𝑃 𝐴)𝑥𝑡 + 𝑥𝑡⊤ (𝐹 ⊤ 𝑣 𝑡+1 + 2𝐹 ⊤ 𝑃𝑤 𝑡 ) !⊤ ! 𝑘−1 𝑘−1 ∑︁ ∑︁ 1 1 𝑖 𝑖 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 𝐻 𝑣 𝑡+1 − 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 + 𝑣 𝑡+1 − 2 2 𝑖=1 𝑖=1  ⊤   1 1 − 𝑃𝑤 𝑡 + 𝑣 𝑡+1 𝐻 𝑃𝑤 𝑡 + 𝑣 𝑡+1 + 𝑤 𝑡⊤ 𝑃𝑤 𝑡 + 𝑤 𝑡⊤ 𝑣 𝑡+1 + 𝑞 𝑡+1 2 2 = 𝑥𝑡⊤ 𝑃𝑥𝑡 + 𝑥𝑡⊤ 𝑣 𝑡 + 𝑞 𝑡 . Thus, ⊤ ⊤ 𝑣 𝑡 = 𝐹 𝑣 𝑡+1 + 2𝐹 𝑃𝑤 𝑡 = 2 ∞ ∑︁ 𝐹⊤ 𝑖+1 𝑃𝑤 𝑡+𝑖 . 𝑖=0 Then, we can plug 𝑣 𝑡+1 into 𝑞 𝑡 : ! !⊤ ∞ ∞ ∑︁ ∑︁ 𝑖 𝑖 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 𝑞 𝑡 = 𝑞 𝑡+1 + 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 𝐻 𝑖=𝑘 𝑖=𝑘 − ∞ ∑︁ !⊤ 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 𝑖 𝑖=0 𝐻 ∞ ∑︁ ! 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 + 𝑤 𝑡⊤ 𝑃𝑤 𝑡 + 2𝑤 𝑡⊤ 𝑖 ∞ ∑︁ ! 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 . 𝑖 𝑖=1 𝑖=0 (11.15) Note that Eq. (11.15) is for MPC with 𝑘 predictions. With the disturbance sequence {𝑤 𝑡 } fixed, we can compare the per-step cost of MPC with 𝑘 predictions and that with 𝑘 + 1 predictions: !⊤ ! ∞ ∞ ∑︁ ∑︁ 𝑖 𝑖 𝑘 𝑘+1 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 𝐻 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 − 𝑞 𝑡+1 + 𝑞 𝑡𝑘 − 𝑞 𝑡𝑘+1 = 𝑞 𝑡+1 𝑖=𝑘 − ∞ ∑︁ 𝑖=𝑘+1 𝑖=𝑘 !⊤ ⊤𝑖 𝐹 𝑃𝑤 𝑡+𝑖 𝐻 ∞ ∑︁ ! ⊤𝑖 𝐹 𝑃𝑤 𝑡+𝑖 𝑖=𝑘+1 𝑘 𝑘+1 ⊤ = 𝑞 𝑡+1 − 𝑞 𝑡+1 + 𝑤 𝑡+𝑘 𝑃𝐹 𝑘 𝐻𝐹 ⊤𝑘 𝑃𝑤 𝑡+𝑘 + 2 ∞ ∑︁ 𝑖=1 ! ⊤𝑖 𝐹 𝑃𝑤 𝑡+𝑖+𝑘 . (11.16) 226 Thus, " !# ∞ ∑︁  𝑘  𝑘 𝑖 𝑘 𝑘+1 ⊤ E 𝑞 𝑡 − 𝑞 𝑡𝑘+1 − (𝑞 𝑡+1 − 𝑞 𝑡+1 ) = E 𝑤 𝑡+𝑘 𝑃𝐹 𝑘 𝐻𝐹 ⊤ 𝑃𝑤 𝑡+𝑘 + 2 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖+𝑘 𝑖=1 ( = Tr 𝑃𝐹 𝑘 𝐻𝐹 ⊤𝑘  ⊤  𝑃 E 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘 + 2 ∞ ∑︁ !) ⊤𝑖  ⊤  𝐹 𝑃 E 𝑤 𝑡+𝑖+𝑘 𝑤 𝑡+𝑘 𝑖=1 n 𝑘 = Tr 𝑃𝐹 𝐻𝐹 ⊤𝑘 o 𝑍 𝑘,𝑡 ,     ⊤ +2 Í∞ 𝐹 ⊤ 𝑖 𝑃 E 𝑤 ⊤ . Note that 𝑍 where 𝑍 𝑘,𝑡 = 𝑃 E 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘 𝑤 𝑡+𝑖+𝑘 𝑘,𝑡 = 𝑍 𝑘−1,𝑡+1 . 𝑖=1 𝑡+𝑘  1  𝑘 E 𝑞 0 − 𝑞 0𝑘+1 𝑇→∞ 𝑇 𝑇−1 o 1 ∑︁ n 𝑘 𝑘 = lim Tr 𝑃𝐹 𝐻𝐹 ⊤ 𝑍 𝑘,𝑡 𝑇→∞ 𝑇 𝑡=0 MPCS 𝑘 − MPCS 𝑘+1 = lim 𝑇−1 1 ∑︁ 2  ∥𝑃∥∥𝐻 ∥ 𝐹 𝑘 Tr 𝑍 𝑘,𝑡 , ≤ lim 𝑇→∞ 𝑇 𝑡=0 where in the last line we use the fact that if 𝐴 is symmetric, then Tr{𝐴𝐵} ≤ ∥ 𝐴∥Tr{𝐵}. Similarly to the last part in the proof of Theorem 11.3, now we just need  to show the last term Tr 𝑍 𝑘,𝑡 is uniformly bounded for all 𝑡. Again, this is because the cross-correlation of each disturbance pair is uniformly bounded. " # ∞ ∑︁ ∑︁     ⊤ ⊤ ∥𝑃∥ 𝐹 𝑖 E Tr 𝑍 𝑘,𝑡 ≤ ∥𝑃∥Tr E 𝑤 𝑡+𝑘 𝑤 𝑡+𝑘 𝜎 𝑗 (𝑤 𝑡+𝑖+𝑘 𝑤 𝑡+𝑘 ) +2 𝑖=1 ≤ ∥𝑃∥𝑚 + 2 ∞ ∑︁ 𝑖=1 𝑐𝜆𝑖 ∥𝑃∥𝑚 = ∥𝑃∥𝑚 + 2𝑐 𝑗 𝜆 ∥𝑃∥𝑚 1−𝜆 where 𝑐 is some constant, and in the first line, we use the fact that Tr{𝐴𝐵} ≤  Í ∥ 𝐴∥ 𝑗 𝜎 𝑗 (𝐵) with 𝜎 𝑗 (·) denoting the 𝑗-th singular value. Thus, Tr 𝑍 𝑘,𝑡 is uniformly bounded. Therefore, MPCS 𝑘 − MPCS 𝑘+1 = 𝑂 (∥𝐹 𝑘 ∥ 2 ). □ In other words, the marginal benefit for the MPC algorithm of an extra prediction decays exponentially fast, paralleling the result for optimal policy in Eq. (11.11). Besides, the dynamic regret of MPC (nearly) matches the order of the optimal dynamic regret, as shown in Theorem 11.5 and Theorem 11.6. Theorem 11.5 (Main result). 𝑅𝑒𝑔 𝑆 ( MPC 𝑘 ) = MPCS𝑇𝑘 − STO𝑇𝑇 = 𝑂 (∥𝐹 𝑘 ∥ 2𝑇 + 1) = 𝑂 (𝜆2𝑘 𝑇 +1), where the second term results from the difference between finite/infinite horizons. 227 Proof. To calculate the dynamic regret, we cannot simply let 𝑇 − 𝑡 → ∞ as we did before Eq. (11.14) in the proof of Theorem 11.4 and instead need to handle the expressions in a more delicate manner. In particular, we need to rigorously analyze the impact of finite horizon. Let Δ𝑡 = 𝑃𝑡 − 𝑃. 𝑉𝑡MPC𝑘 (𝑥𝑡 ; 𝑤 0:𝑡+𝑘−1 ) = 𝑢 𝑡⊤ (𝑅 + 𝐵⊤ 𝑃𝑡+1 𝐵)𝑢 𝑡 + 2𝑢 𝑡⊤ 𝐵⊤ (𝑃𝑡+1 𝐴𝑥𝑡 + 𝑃𝑡+1 𝑤 𝑡 + 𝑣 𝑡+1 /2) + 𝑥𝑡⊤ 𝑄𝑥𝑡 + ( 𝐴𝑥𝑡 + 𝑤 𝑡 ) ⊤ 𝑃𝑡+1 ( 𝐴𝑥𝑡 + 𝑤 𝑡 ) + ( 𝐴𝑥𝑡 + 𝑤 𝑡 ) ⊤ 𝑣 𝑡+1 + 𝑞 𝑡+1 = 𝑢 𝑡⊤ (𝑅 + 𝐵⊤ 𝑃𝐵)𝑢 𝑡 + 2𝑢 𝑡⊤ 𝐵⊤ (𝑃 𝐴𝑥𝑡 + 𝑃𝑤 𝑡 + 𝑣 𝑡+1 /2) + 𝑥𝑡⊤ 𝑄𝑥𝑡 + ( 𝐴𝑥𝑡 + 𝑤 𝑡 ) ⊤ 𝑃( 𝐴𝑥𝑡 + 𝑤 𝑡 ) + ( 𝐴𝑥𝑡 + 𝑤 𝑡 ) ⊤ 𝑣 𝑡+1 + 𝑞 𝑡+1 + 𝑢 𝑡⊤ 𝐵⊤ Δ𝑡+1 𝐵𝑢 𝑡 + 2𝑢 𝑡⊤ 𝐵⊤ Δ𝑡+1 ( 𝐴𝑥𝑡 + 𝑤 𝑡 ) + ( 𝐴𝑥𝑡 + 𝑤 𝑡 ) ⊤ Δ𝑡+1 ( 𝐴𝑥𝑡 + 𝑤 𝑡 ). Plugging in the MPC policy as in Theorem 11.2, we have: 𝑉𝑡MPC𝑘 (𝑥𝑡 ; 𝑤 0:𝑡+𝑘−1 ) = 𝑥𝑡⊤ (𝑄 + 𝐴⊤ 𝑃 𝐴 − 𝐴⊤ 𝑃𝐻𝑃 𝐴)𝑥𝑡 + 𝑥𝑡⊤ (𝐹 ⊤ 𝑣 𝑡+1 + 2𝐹 ⊤ 𝑃𝑤 𝑡 ) !⊤ ! 𝑘−1 𝑘−1 ∑︁ ∑︁ 1 1 𝑖 𝑖 + 𝑣 𝑡+1 − 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 𝐻 𝑣 𝑡+1 − 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 2 2 𝑖=1 𝑖=1 ⊤    1 1 − 𝑃𝑤 𝑡 + 𝑣 𝑡+1 𝐻 𝑃𝑤 𝑡 + 𝑣 𝑡+1 + 𝑤 𝑡⊤ 𝑃𝑤 𝑡 + 𝑤 𝑡⊤ 𝑣 𝑡+1 + 𝑞 𝑡+1 2 2 ! ! ⊤ 𝑘−1 𝑘−1 ∑︁ ∑︁ 𝑖 𝑖 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 + 𝐹𝑥𝑡 + 𝑤 𝑡 − 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 Δ𝑡+1 𝐹𝑥𝑡 + 𝑤 𝑡 − 𝑖=0 𝑖=0 ⊤ ⊤ = 𝑥𝑡 (𝑄 + 𝐴 𝑃 𝐴 − 𝐴⊤ 𝑃𝐻𝑃 𝐴 + 𝐹 ⊤ Δ𝑡+1 𝐹)𝑥𝑡 + 𝑥𝑡⊤ 𝐹 ⊤ 𝑣 𝑡+1 + 2𝐹 ⊤ 𝑃𝑤 𝑡 + 2𝐹 ⊤ Δ𝑡+1 𝑤 𝑡 − 𝑘−1 ∑︁ !! ⊤𝑖 𝐹 𝑃𝑤 𝑡+𝑖 𝑖=0 !⊤ ! 1 1 ⊤𝑖 ⊤𝑖 + 𝑣 𝑡+1 − 𝐹 𝑃𝑤 𝑡+𝑖 𝐻 𝑣 𝑡+1 − 𝐹 𝑃𝑤 𝑡+𝑖 2 2 𝑖=1 𝑖=1  ⊤   1 1 − 𝑃𝑤 𝑡 + 𝑣 𝑡+1 𝐻 𝑃𝑤 𝑡 + 𝑣 𝑡+1 + 𝑤 𝑡⊤ 𝑃𝑤 𝑡 + 𝑤 𝑡⊤ 𝑣 𝑡+1 + 𝑞 𝑡+1 2 2 ! ! ⊤ 𝑘−1 𝑘−1 ∑︁ ∑︁ 𝑖 𝑖 + 𝑤𝑡 − 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 Δ𝑡+1 𝑤 𝑡 − 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 . 𝑘−1 ∑︁ 𝑖=0 𝑘−1 ∑︁ 𝑖=0 Comparing this with the induction hypothesis 𝑉𝑡MPC𝑘 = 𝑥𝑡⊤ (𝑃 + Δ𝑡 )𝑥𝑡 + 𝑥𝑡⊤ 𝑣 𝑡 + 𝑞 𝑡 , we obtain the recursive formulae for Δ𝑡 , 𝑣 𝑡 , 𝑞 𝑡 . Δ𝑡 = 𝐹 ⊤ Δ𝑡+1 𝐹 = 𝐹 ⊤ 𝑇−𝑡 Δ𝑇 𝐹 𝑇−𝑡 = 𝐹 ⊤ 𝑇−𝑡 (Qf − 𝑃)𝐹 𝑇−𝑡 . 228 This implies that 𝑃𝑡 converges to 𝑃 exponentially fast, i.e., ∥Δ𝑡 ∥ = 𝑂 (∥𝐹 𝑇−𝑡 ∥ 2 ) = 𝑂 (𝜆2(𝑇−𝑡) ). ! 𝑘−1 ∑︁ 𝑖 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 𝑣 𝑡 = 𝐹 ⊤ 𝑣 𝑡+1 + 2𝐹 ⊤ 𝑃𝑤 𝑡 + 2𝐹 ⊤ Δ𝑡+1 𝑤 𝑡 − 𝑖=0 =2 𝑇−𝑡−1 ∑︁ 𝐹⊤ 𝑗+1 𝑃𝑤 𝑡+ 𝑗 + 𝐹 ⊤ 𝑗+1 Δ𝑡+ 𝑗+1 𝑤 𝑡+ 𝑗 − 𝑗=0 =2 𝑇−𝑡−1 ∑︁ 𝑘−1 ∑︁ !! 𝐹 ⊤ 𝑃𝑤 𝑡+ 𝑗+𝑖 𝑖 𝑖=0 𝐹⊤ 𝑖+1 𝑃𝑤 𝑡+𝑖 + 2 𝑇−𝑡−1 ∑︁ 𝐹⊤ 𝑗+1 Δ𝑡+ 𝑗+1 𝑤 𝑡+ 𝑗 − ! 𝐹 ⊤ 𝑃𝑤 𝑡+ 𝑗+𝑖 . 𝑖 𝑖=0 𝑗=0 𝑖=0 𝑘−1 ∑︁ Denote the second term by 2𝑑𝑡 . We have 𝑑𝑡 = 𝑇−𝑡−1 ∑︁ 𝐹 ⊤ 𝑗+1 Δ𝑡+ 𝑗+1 𝑤 𝑡+ 𝑗 − 𝑗=0 = 𝑇−𝑡−1 ∑︁ 𝑘−1 ∑︁ ! ⊤𝑖 𝐹 𝑃𝑤 𝑡+ 𝑗+𝑖 𝑖=0 𝑂 (𝜆 𝑗 𝜆2(𝑇−𝑡− 𝑗) ) = 𝑂 (𝜆𝑇−𝑡 ). 𝑗=0 𝑑𝑡𝑘 − 𝑑𝑡𝑘+1 = 𝑇−𝑡−𝑘−1 ∑︁ 𝐹⊤ 𝑗+1 Δ𝑡+ 𝑗+1 𝐹 ⊤ 𝑃𝑤 𝑡+ 𝑗+𝑘 𝑘 (11.17) 𝑗=0 = 𝑇−𝑡−𝑘−1 ∑︁ 𝑂 (𝜆 𝑗 𝜆2(𝑇−𝑡− 𝑗) ∥𝐹 𝑘 ∥) = 𝑂 (𝜆𝑇−𝑡+𝑘 ∥𝐹 𝑘 ∥). 𝑗=0 Finally, we have a formula for 𝑞 𝑡 that parallels Eq. (11.15): 𝑞 𝑡 = 𝑞 𝑡+1 + 𝑑𝑡+1 + 𝑇−𝑡−1 ∑︁ !⊤ 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 𝑖 𝐻 𝑑𝑡+1 + 𝑇−𝑡−1 ∑︁ − 𝑑𝑡+1 + !⊤ ⊤𝑖 𝐹 𝑃𝑤 𝑡+𝑖 𝐻 𝑑𝑡+1 + 𝑖=0 + 𝑤 𝑡⊤ 𝑃𝑤 𝑡 + 2𝑤 𝑡⊤ 𝑑𝑡+1 + 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 𝑖 𝑖=𝑘 𝑖=𝑘 𝑇−𝑡−1 ∑︁ ! 𝑇−𝑡−1 ∑︁ 𝑖=0 𝑇−𝑡−1 ∑︁ 𝑖=1 ! 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖 . 𝑖 ! ⊤𝑖 𝐹 𝑃𝑤 𝑡+𝑖 229 Taking the difference between 𝑘 and 𝑘 + 1 predictions, we have 𝑘 𝑘+1 𝑞 𝑡𝑘 − 𝑞 𝑡𝑘+1 − (𝑞 𝑡+1 − 𝑞 𝑡+1 ) ⊤ 𝑘 𝑘+1 ⊤ 𝑘 𝑘+1 = (𝑤 𝑡+𝑘 𝑃𝐹 𝑘 + (𝑑𝑡+1 − 𝑑𝑡+1 ) )𝐻 𝑑𝑡+1 + 𝑑𝑡+1 +𝐹 ⊤𝑘 𝑃𝑤 𝑡+𝑘 + 2 𝑇−𝑡−𝑘−1 ∑︁ ! 𝐹 ⊤ 𝑖+𝑘 𝑃𝑤 𝑡+𝑖+𝑘 𝑖=1 ⊤ = (𝑤 𝑡+𝑘 𝑃𝐹 𝑘 + 𝑂 (𝜆𝑇−𝑡 ∥𝐹 𝑘 ∥))𝐻 𝑇−𝑡 𝑂 (𝜆 )+𝐹 ⊤𝑘 𝑃𝑤 𝑡+𝑘 + 2 𝑇−𝑡−𝑘−1 ∑︁ (11.18) ! 𝐹⊤ 𝑖+𝑘 𝑃𝑤 𝑡+𝑖+𝑘 , 𝑖=1 and thus   𝑘 𝑘+1 E 𝑞 𝑡𝑘 − 𝑞 𝑡𝑘+1 − (𝑞 𝑡+1 − 𝑞 𝑡+1 ) = 𝑂 (∥𝐹 𝑘 ∥(𝜆𝑇−𝑡 + ∥𝐹 𝑘 ∥)). ∑︁   𝑘  𝑇−1  𝑇 𝑘 𝑘+1 E 𝑞0 − 𝑞0 = E 𝑞 𝑡𝑘 − 𝑞 𝑡𝑘+1 − (𝑞 𝑡+1 − 𝑞 𝑡+1 ) = 𝑡=0 𝑇−1 ∑︁ 𝑂 (∥𝐹 𝑘 ∥(𝜆𝑇−𝑡 + ∥𝐹 𝑘 ∥)) 𝑡=0 = 𝑂 (∥𝐹 𝑘 ∥ 2𝑇 + ∥𝐹 𝑘 ∥).   E 𝑣 0𝑘 − 𝑣𝑇0 = 2(𝑑0𝑘 − 𝑑0𝑇 ) = 𝑂 (𝜆𝑇+𝑘 ∥𝐹 𝑘 ∥).   E 𝐽 MPC𝑘 − E 𝐽 MPC𝑇 = E 𝑉0𝑘 (𝑥 0 ) − 𝑉0𝑇 (𝑥 0 )   = E 𝑥 0⊤ (𝑣 0𝑘 − 𝑣𝑇0 ) + (𝑞 0𝑘 + 𝑞𝑇0 ) = 𝑂 (∥𝐹 𝑘 ∥ 2𝑇 + ∥𝐹 𝑘 ∥). (11.19) By definition, 𝐽 MPC𝑇 is the cost of MPC policy given all future disturbances before making any decisions. It almost equals to min𝑢 𝐽, the optimal policy given all future disturbances, except that during optimization, MPC assumes the final-step cost to be 𝑥𝑇⊤ 𝑃𝑥𝑇 instead of 𝑥𝑇⊤ Qf 𝑥𝑇 . This will incur at most constant extra cost, i.e., 𝐽 MPC𝑇 − min 𝐽 = 𝑂 (𝑃 − Qf ) = 𝑂 (1). (11.20) 𝑢 By Eqs. (11.19) and (11.20), 𝑅𝑒𝑔 𝑆 ( MPC 𝑘 ) = E 𝐽 MPC𝑘 − E min 𝐽 = 𝑂 (∥𝐹 𝑘 ∥ 2𝑇 + ∥𝐹 𝑘 ∥ + 1) = 𝑂 (∥𝐹 𝑘 ∥ 2𝑇 + 1). 𝑢 □ 230 ∗ Theorem 11.6. The optimal dynamic regret 𝑅𝑒𝑔 𝑘𝑆 = STO𝑇𝑘 − STO𝑇𝑇 = 𝑂 (∥𝐹 𝑘 ∥ 2𝑇 + ∗ 1) = 𝑂 (𝜆2𝑘 𝑇 + 1) and there exist 𝐴, 𝐵, 𝑄, 𝑅, Qf , 𝑥0 , and W such that 𝑅𝑒𝑔 𝑘𝑆 = Θ(∥𝐹 𝑘 ∥ 2 (𝑇 − 𝑘)). ∗ Proof. The first part follows from Theorem 11.5 and that fact that 𝑅𝑒𝑔 𝑘𝑆 ≤ 𝑅𝑒𝑔 𝑆 ( MPC 𝑘 ). The second part is shown by Example 11.3, i.e., suppose 𝑛 = 𝑑 = 1 and the disturbance are i.i.d. and zero-mean. Additionally, let Qf = 𝑃 and 𝑥 0 = 0. In this case, MPC has not only the same policy but also the same cost as the optimal control policy. Also, 𝑃𝑡 = 𝑃 for all 𝑡. To calculate the total cost, we follow the approach used in the proof of Theorem 11.1. Since 𝑇 is finite now, we have a similar (to Eq. (11.7)) but different form of 𝑣 𝑡 : 𝑣𝑡 = 2 min{𝑘−1,𝑇−𝑡−1} ∑︁ 𝐹⊤ 𝑖+1 𝑃𝑤 𝑡+𝑖 . 𝑖=0 Thus, ( E[𝑞 𝑡 ] = Tr 𝑃− min{𝑘−1,𝑇−𝑡−1} ∑︁ ! ) ⊤𝑖 𝑃𝐹 𝑖 𝐻𝐹 𝑃 𝑊 + E[𝑞 𝑡+1 ]. 𝑖=0 E[𝑞 0 ] = Tr (𝑇−1 ∑︁ 𝑃− min{𝑘−1,𝑇−𝑡−1} ∑︁ 𝑡=0 ! ) 𝑃𝐹 𝑖 𝐻𝐹 ⊤ 𝑃 𝑊 . 𝑖 𝑖=0 Let 𝑞 𝑡𝑘 denote 𝑞 𝑡 in the scenario of 𝑘 predictions. 𝑅𝑒𝑔 𝑆∗ (𝑇−𝑘−1 𝑇−𝑡−1 ) ∑︁ ∑︁  𝑘  𝑖 = E 𝑞 0 − 𝑞𝑇0 = Tr 𝑃𝐹 𝑖 𝐻𝐹 ⊤ 𝑃𝑊 𝑡=0 n𝑖=𝑘 o 𝑘 ≥ (𝑇 − 𝑘)Tr 𝑃𝐹 𝑘 𝐻𝐹 ⊤ 𝑃𝑊 = Ω(∥𝐹 𝑘 ∥ 2 (𝑇 − 𝑘)). On the other hand, 𝑅𝑒𝑔 𝑆∗ (∞ ) ∑︁   𝑘 𝑖 ⊤𝑖 𝑇 = E 𝑞 0 − 𝑞 0 ≤ (𝑇 − 𝑘)Tr 𝑃𝐹 𝐻𝐹 𝑃𝑊 = 𝑂 (∥𝐹 𝑘 ∥ 2 (𝑇 − 𝑘)). 𝑖=𝑘 ∗ Therefore, 𝑅𝑒𝑔 𝑆 = Θ(∥𝐹 𝑘 ∥ 2 (𝑇 − 𝑘)). □ Note that, in the stochastic case, the regret-optimal policy is the same as the cost∗ optimal policy, i.e., the policy for STO𝑇𝑘 is the same as 𝑅𝑒𝑔 𝑘𝑆 . 231 Adversarial Disturbance We now move from stochastic to adversarial disturbances. In this case, the disturbances are chosen from a bounded set Ω ⊆ R𝑛 by an adversary in order to maximize the controller’s cost. Maintaining small regret is more challenging in adversarial models than in stochastic ones, so one may expect weaker bounds. Perhaps surprisingly, we obtain bounds with the same order. Optimal policy with 𝑘 predictions. In the adversarial setting, the cost of the optimal policy, defined with a sequence of min’s and sup’s, is the equilibrium value of a two-player zero-sum game. In general, it is impossible to give an analytical expression of either ADV 𝑘 or the corresponding optimal policy. However, we prove the following result that is structurally similar to the results from the stochastic setting, highlighting the exponential improvement from predictions. Theorem 11.7. For 𝑘 ≥ 1, ADV 𝑘 − ADV 𝑘+1 = 𝑂 (∥𝐹 𝑘 ∥ 2 ) = 𝑂 (𝜆2𝑘 ). Proof. This proof is based on Theorem 11.8. It turns out that the behavior of the MPC policy and its cost is easier to analyze than the optimal one, especially in the adversarial setting. ADV 𝑘 − ADV 𝑘+1 ≤ ADV 𝑘 − ADV∞ ≤ MPCA 𝑘 − ADV∞ = ∞ ∑︁ MPCA𝑖 − MPCA𝑖+1 . 𝑖=𝑘 By Theorem 11.8, MPCA𝑖 − MPCA𝑖+1 ≤ 𝑂  𝐹 𝑖 2   ≤𝑂 𝐹 𝑘 2 𝐹 𝑖−𝑘 2   ≤𝑂 𝐹 𝑘 2 2(𝑖−𝑘) 𝜆  . Thus, ADV 𝑘 − ADV 𝑘+1 ≤ 𝑂 𝐹 𝑘 2 ∞ ∑︁ ! 𝜆2(𝑖−𝑘) = 𝑂 (∥𝐹 𝑘 ∥ 2 ). 𝑖=𝑘 □ Similarly to Example 11.2 for the stochastic case, in the adversarial setting, the optimal cost with 𝑘 predictions may approach the offline optimal cost (under infinite predictions) much faster than exponential rate, and it is possible that ADV 𝑘 = ADV∞ for finite 𝑘, as shown in Example 11.4. Example 11.4. Let 𝐴 = 𝐵 = 𝑄 = 𝑅 = 1 and Ω = [−1, 1]. In this case, one prediction is enough to leverage the full power of prediction. Formally, we have 232 ADV1 = ADV∞ = 1. In other words, for all 𝑘 ≥ 1, ADV 𝑘 = 1. The optimal control policy (as 𝑇 → ∞) is a piecewise function:    −(𝑥 + 𝑤)   √   ∗ 𝑢 (𝑥, 𝑤) = −(𝑥 + 𝑤) + 3−2 5 (𝑥 + 𝑤 − 1)  √     −(𝑥 + 𝑤) + 3−2 5 (𝑥 + 𝑤 + 1)  , −1 ≤ 𝑥 + 𝑤 ≤ 1 ,𝑥 + 𝑤 > 1 . , 𝑥 + 𝑤 < −1 The proof leverages two different cost-to-go functions for the min player and the sup player (see Yu et al., 2020, Appendix C.2). Note that with adversarial disturbances, the optimal policy could be much more complex than stochastic cases. Unlike Example 11.4, where the optimal policy is piecewise linear with only 3 pieces, for other values of 𝐴, 𝐵, 𝑄, 𝑅, this function may have many more pieces. MPC’s performance. Under adversarial disturbances, MPC is suboptimal, e.g., in Example 11.4. However, its dynamic regret bound turn out to be the same as those in the stochastic setting, as shown in following theorems. Theorem 11.8. MPCA 𝑘 − MPCA 𝑘+1 = 𝑂 (∥𝐹 𝑘 ∥ 2 ) = 𝑂 (𝜆2𝑘 ). Proof. Note that Eq. (11.16) in the proof of Theorem 11.4 does not rely on the type of disturbance, i.e., Eq. (11.16) holds for adversarial disturbance as well. Let 𝑟 = sup𝑤∈Ω ∥𝑤∥ 2 . ! ∞ ∑︁ 𝑖 𝑘 𝑘 𝑘+1 ⊤ 𝐹 ⊤ 𝑃𝑤 𝑡+𝑖+𝑘 𝑞 𝑡𝑘 − 𝑞 𝑡𝑘+1 − (𝑞 𝑡+1 − 𝑞 𝑡+1 ) = 𝑤 𝑡+𝑘 𝑃𝐹 𝑘 𝐻𝐹 ⊤ 𝑃𝑤 𝑡+𝑘 + 2 𝑖=1 ≤ ∥𝑤 𝑡+𝑘 ∥∥𝑃∥∥𝐻 ∥ 𝐹 𝑘 2 ∥𝑃∥∥𝑤 𝑡+𝑘 ∥ + 2 ∞ ∑︁ 𝑖=1 ≤ 𝐹𝑘 2 1+2 ∞ ∑︁ ! 𝐹𝑖 ∥𝐻 ∥∥𝑃∥ 2𝑟 2 𝑖=1 ≤ 𝐹 𝑘 2   𝑐𝜆 ∥𝐻 ∥∥𝑃∥ 2𝑟 2 1+2 1−𝜆 ! 𝐹 𝑖 ∥𝑃∥∥𝑤 𝑡+𝑖+𝑘 ∥ 233 for some constant 𝑐. 1 (max 𝑞 0𝑘 − max 𝑞 0𝑘+1 ) 𝑤 𝑤 𝑇→∞ 𝑇 1 ≤ lim max (𝑞 0𝑘 − 𝑞 0𝑘+1 ) 𝑇→∞ 𝑇 𝑤 𝑇−1 1 ∑︁ 𝑘 𝑘+1 max (𝑞 𝑡𝑘 − 𝑞 𝑡𝑘+1 − (𝑞 𝑡+1 − 𝑞 𝑡+1 )) ≤ lim 𝑤 𝑇→∞ 𝑇 𝑡=0   𝑐𝜆 𝑘 2 ∥𝐻 ∥∥𝑃∥ 2𝑟 2 = 𝑂 (∥𝐹 𝑘 ∥ 2 ). ≤ 𝐹 1+2 1−𝜆 MPCA 𝑘 − MPCA 𝑘+1 = lim □ Theorem 11.9 (Main result). 𝑅𝑒𝑔 𝐴 ( MPC 𝑘 ) = 𝑂 (∥𝐹 𝑘 ∥ 2𝑇 + 1) = 𝑂 (𝜆2𝑘 𝑇 + 1). Proof. We follow the notations in the proof of Theorem 11.5. Eq. (11.18) does not rely on the type of disturbance, so it holds for adversarial disturbance as well. By Eq. (11.18) and the fact that 𝑤 𝑡 is bounded, we have 𝑘 𝑘+1 − 𝑞 𝑡+1 ) = 𝑂 (∥𝐹 𝑘 ∥(𝜆𝑇−𝑡 + ∥𝐹 𝑘 ∥)), 𝑞 𝑡𝑘 − 𝑞 𝑡𝑘+1 − (𝑞 𝑡+1 where the constant in the Big-Oh notation does not depend on the disturbance sequence 𝑤. Thus, max (𝑞 0𝑘 − 𝑞𝑇0 ) ≤ 𝑤 𝑇−1 ∑︁ 𝑘 𝑘+1 max (𝑞 𝑡𝑘 − 𝑞 𝑡𝑘+1 − (𝑞 𝑡+1 − 𝑞 𝑡+1 )) = 𝑂 (∥𝐹 𝑘 ∥ 2𝑇 + ∥𝐹 𝑘 ∥). 𝑤 𝑡=0 By Eq. (11.17) and the boundedness of 𝑤 𝑡 , max (𝑣 0𝑘 − 𝑣𝑇0 ) = 2 max (𝑑0𝑘 − 𝑑0𝑇 ) = 𝑂 (𝜆𝑇+𝑘 ∥𝐹 𝑘 ∥). 𝑤 𝑤 max (𝐽 MPC𝑘 − 𝐽 MPC𝑇 ) = max (𝑉0𝑘 (𝑥 0 ) − 𝑉0𝑇 (𝑥 0 )) ≤ max (𝑥 0⊤ (𝑣 0𝑘 − 𝑣𝑇0 )) + max (𝑞 0𝑘 − 𝑞𝑇0 ) 𝑤 𝑤 𝑤 𝑤 𝑘 2 𝑘 = 𝑂 (∥𝐹 ∥ 𝑇 + ∥𝐹 ∥). As Eq. (11.20), 𝐽 MPC𝑇 − min𝑢 𝐽 = 𝑂 (1). Thus, 𝑅𝑒𝑔 𝐴 ( MPC 𝑘 ) = max (𝐽 MPC𝑘 − min 𝐽) ≤ max (𝐽 MPC𝑘 − 𝐽 𝑀 𝑃𝐶𝑇 ) + max (𝐽 MPC𝑇 − min 𝐽) 𝑤 𝑢 𝑤 𝑢 𝑤 𝑘 2 𝑘 𝑘 2 = 𝑂 (∥𝐹 ∥ 𝑇 + ∥𝐹 ∥ + 1) = 𝑂 (∥𝐹 ∥ 𝑇 + 1). □ 234 This dynamic regret is linear in the horizon 𝑇 if we fix the number of predictions. However, if 𝑘 is a super-constant function of 𝑇, i.e., an increasing function of 𝑇 that is not upper-bounded by a constant, then the regret is sub-linear. Furthermore, if we log 𝑇 let 𝑘 = 2 log(1/𝜆) , then 𝑅𝑒𝑔 𝐴 ( MPC 𝑘 ) = 𝑂 (1). In other words, we can get constant regret with 𝑂 (log 𝑇) predictions, even with adversarial disturbances. Finally, as implied by the following result, the 𝑂 (log 𝑇) horizon cannot be improved since even the regret minimizing algorithm needs the same order of predictions to reach constant regret, depicted by the following theorem. ∗ Theorem 11.10. 𝑅𝑒𝑔 𝑘𝐴 = 𝑂 (∥𝐹 𝑘 ∥ 2𝑇 + 1) = 𝑂 (𝜆2𝑘 𝑇 + 1). Moreover, there exist 𝐴, ∗ 𝐵, 𝑄, 𝑅, Qf , 𝑥 0 , and Ω such that 𝑅𝑒𝑔 𝑘𝐴 = Ω(∥𝐹 𝑘 ∥ 2 (𝑇 − 𝑘)). Proof. The first part of the theorem follows from Theorem 11.9 and the fact that ∗ 𝑅𝑒𝑔 𝑘𝐴 ≤ 𝑅𝑒𝑔 𝐴 ( MPC 𝑘 ). We reduce the second part of this theorem to the second part of Theorem 11.6. Since the proof of Theorem 11.6 works for any fixed distribution of 𝑤 𝑡 (with finite second moment), we can restrict that distribution to have bounded support. Denote this bounded support by Ω. Then, we have   ∗ 𝑅𝑒𝑔 𝑘𝐴 = sup min sup · · · min sup min 𝐽 (𝑢, 𝑤) − ′ min′ 𝐽 (𝑢′, 𝑤) 𝑢𝑇−𝑘−1 𝑤 𝑇−1 𝑢𝑇−𝑘 ,··· ,𝑢𝑇−1 𝑢 0 ,...,𝑢𝑇−1 𝑤 0 ,··· ,𝑤 𝑘−1 𝑢 0 𝑤 𝑘   ≥ E min E · · · min E min 𝐽 (𝑢, 𝑤) − ′ min′ 𝐽 (𝑢′, 𝑤) 𝑤 0 ,··· ,𝑤 𝑘−1 𝑢 0 𝑤 𝑘 𝑢𝑇−𝑘−1 𝑤 𝑇−1 𝑢𝑇−𝑘 ,··· ,𝑢𝑇−1 𝑢 0 ,...,𝑢𝑇−1 ∗ = 𝑅𝑒𝑔 𝑘𝑆 = Θ(∥𝐹 𝑘 ∥ 2 (𝑇 − 𝑘)). □ 11.4 Simulations k=0 k=1 k=2 k=4 k=8 5 5 5 5 5 cost difference 1000 10 0.100 -4 -2 -5 2 4 -4 -2 -5 2 4 -4 -2 -5 2 4 -4 -2 -5 2 4 -4 -2 -5 2 4 0.001 10-5 10-7 5 10 15 k Figure 11.1: The power of predictions in online tracking. The left five figures show the desired trajectory (blue) and the actual trajectories (orange). The rightmost figure shows the cost difference (regret) between MPC using 𝑘 predictions and the offline optimal policy. Note that the y-axis of the rightmost figure is in log-scale. To illustrate our theoretical results, we test MPC with different numbers of predictions in a Linear Quadratic (LQ) tracking problem, where the desired trajectory is 235 given by: " # 8 sin(𝑡/3) cos(𝑡/3) 𝑑𝑡 = . 8 sin(𝑡/3) We consider following double integrator dynamics: 𝑝 𝑡+1 = 𝑝 𝑡 + 𝑣 𝑡 + ℎ𝑡 , 𝑣 𝑡+1 = 𝑣 𝑡 + 𝑢 𝑡 + 𝜂𝑡 , where 𝑝 𝑡 ∈ R2 is the position, 𝑣 𝑡 is the velocity, 𝑢 𝑡 is the control, and ℎ𝑡 , 𝜂𝑡 ∼ U[−1, 1] 2 are i.i.d. noises. The objective is to minimize 𝑇−1 ∑︁ ∥ 𝑝 𝑡 − 𝑑𝑡 ∥ 2 + ∥𝑢 𝑡 ∥ 2 , 𝑡=0 where we let 𝑇 = 200. This problem can be converted to the standard LQR with     disturbance 𝑤 𝑡 by letting 𝑥𝑡 = 𝑣𝑝𝑡𝑡 and 𝑤˜ 𝑡 = 𝜂ℎ𝑡𝑡 and then using the reduction in the LQ tracking example (Example 11.1). Note that after the reduction, the disturbances are the combination of a deterministic trajectory and i.i.d. noises, which corresponds to the general stochastic case discussed in Section 11.3. Figure 11.1 shows the tracking results with MPC using different numbers of predictions. We see that the regret exponentially decreases as the number of predictions increases, which is consistent with our theoretical results. 11.5 Extension I: Delayed Inexact Predictions Throughout Sections 11.2 to 11.4, we focus on LQR systems with exact predictions of disturbance 𝑤 𝑡 . In this section, we will discuss the influence from delayed inexact predictions. Problem Statement In this section, we use the following model to study the setting with delayed and inexact predictions. Formally, at each step 𝑡, the revealed information is: 𝑥 0 , 𝑢 0 , . . . , 𝑢 𝑡−1 , 𝑤 0 , . . . , 𝑤 𝑡−𝑑−1 , 𝑤ˆ 𝑡−𝑑|𝑡 , . . . , 𝑤ˆ 𝑇−1|𝑡 , or equivalently, 𝑥 0 , 𝑢 0 , . . . , 𝑢 𝑡−1 , 𝑥1 , . . . , 𝑥𝑡−𝑑 , 𝑤ˆ 𝑡−𝑑|𝑡 , . . . , 𝑤ˆ 𝑇−1|𝑡 , where 𝑑 ≥ 0 is the length of feedback delay, and 𝑤ˆ 𝑠|𝑡 is the prediction of 𝑤 𝑠 at time 𝑡. Namely, at time step 𝑡, the controller must make the decision 𝑢 𝑡 without knowing 𝑥𝑡−𝑑+1 , · · · , 𝑥𝑡 , but with noisy predictions 𝑤ˆ 𝑡−𝑑|𝑡 , · · · , 𝑤ˆ 𝑇−1|𝑡 . 236 We define 𝑒 𝑠|𝑡 = 𝑤 𝑠 − 𝑤ˆ 𝑠|𝑡 as the estimation error, and we assume that the predictor satisfies 𝑒 𝑡−𝑑+𝑖|𝑡 ≤ 𝜖𝑖 ∥𝑤 𝑡−𝑑+𝑖 ∥ for all 𝑖 ≥ 0 and 𝑡 ≥ 0, where 𝜖𝑖 is a given parameter that measures the prediction quality at 𝑖 steps into the unknown. In this section, we focus on the most challenging adversarial case where both 𝑤 𝑡 and 𝑒 𝑠|𝑡 are adversarial. Although predictions are available for every time step, those for far future may have bad quality, i.e., 𝜖𝑖 is typically large for large 𝑖. Therefore, a good control policy may not use all predictions in the same way: using only the predictions with smaller estimation error may yield better performance. For example, if 𝜖𝑖 > 1, we can simply let 𝑤ˆ 𝑡−𝑑+𝑖|𝑡 = 0, which is a better prediction with 𝜖𝑖 = 1. Competitive ratio. In this section, we measure the performance using a even stronger metric competitive ratio, which bounds the worst-case ratio of the cost of an online policy (Alg, i.e., the MPC policy) to the cost of the optimal offline policy (Opt) with perfect knowledge of {𝑤 𝑡 }𝑇−1 𝑡=0 . Formally, we study the so-called weak competitive ratio, which allows for an additive horizon-independent constant factor. We say that a policy is (weakly) 𝑐-competitive if, given 𝐴, 𝐵, 𝑄, 𝑅, Qf and 𝑟, for any adversarially and adaptively chosen disturbances 𝑤 𝑡 and predictions errors 𝑒 𝑠|𝑡 , we have Alg ≤ 𝑐 Opt + 𝜅, where 𝜅 is a constant, independent of 𝑇. We say that the algorithm is constant competitive when 𝑐 is a constant independent of 𝑇. We use competitive ratio in this section to avoid a path-length term in the bound (a dynamic regret bound typically includes a path-length term, as discussed in Chapter 8). Under the competitive ratio metric, in general, the error bound assumption ( 𝑒 𝑡−𝑑+𝑖|𝑡 ≤ 𝜖𝑖 ∥𝑤 𝑡−𝑑+𝑖 ∥) has to be multiplicative rather than additive. Consider the case where all disturbances 𝑤 𝑡 are zero, but there are nonzero prediction errors. The optimal offline policy incurs zero cost, while any online policy that uses the (wrong) predictions incurs nonzero cost, hence leading to an infinite ratio. A myopic variant of MPC. In this section we consider a straightforward myopic extension of the classic MPC algorithm to deal with delayed inexact predictions. In the case without delay (𝑑 = 0), suppose the controller uses 𝑘 predictions. At each time 𝑡, the controller optimizes based on 𝑥𝑡 , 𝑤ˆ 𝑡|𝑡 , . . . , 𝑤ˆ 𝑡+𝑘−1|𝑡 :  𝑡+𝑘−1  ∑︁ ⊤ ⊤ ⊤ (𝑢 𝑡 , . . . , 𝑢 𝑡+𝑘−1 ) = arg min (𝑥𝑖 𝑄𝑥𝑖 + 𝑢𝑖 𝑅𝑢𝑖 ) + 𝑥𝑡+𝑘 Q̃f 𝑥𝑡+𝑘 , 𝑢 𝑖=𝑡 s.t. 𝑥𝑖+1 = 𝐴𝑥𝑖 + 𝐵𝑢𝑖 + 𝑤ˆ 𝑖|𝑡 , ∀𝑖 = 𝑡, . . . , 𝑡 + 𝑘 − 1. This optimization is myopic in the sense that it assumes that the length of the problem is 𝑘 instead of 𝑇 and treats predicted future disturbances as true disturbances. Again, 237 the terminal cost matrix Q̃f may or may not be the same as the terminal cost matrix Qf of the original problem, and can be viewed as a hyperparameter. Similarly, 𝑘 is also a hyperparameter. Larger 𝑘 is not necessarily better because the predictions in the far future may have very large errors. In this section, similar to Algorithm 11.1, we let Q̃f = 𝑃, where 𝑃 is the solution of the discrete algebraic Riccati equation (DARE). Now let us discuss the extension of this myopic MPC policy to the case with delay (𝑑 > 0). When 𝑘 ≥ 𝑑, the extension is perhaps straightforward. Here, although the controller does not know the current state 𝑥𝑡 , it knows 𝑥𝑡−𝑑 and 𝑤ˆ 𝑡−𝑑|𝑡 , . . . , 𝑤ˆ 𝑡−1|𝑡 . Thus, it can estimate the current state. This means that it is possible to simply use this estimation, 𝑥ˆ𝑡|𝑡 , as a replacement for 𝑥𝑡 in the algorithm, which yields the following: ⊤ −1 ⊤ 𝑢 𝑡 = −(𝑅 + 𝐵 𝑃𝐵) 𝐵  𝑃 𝐴𝑥ˆ𝑡|𝑡 + 𝑘−𝑑−1 ∑︁ ⊤𝑖  𝐹 𝑃 𝑤ˆ 𝑡+𝑖|𝑡 . (11.21) 𝑖=0 When 𝑘 < 𝑑, the extension is not as obvious. In this setting, the quality of the predictions is poor enough that it is better not to use the predictions to estimate the current state. Thus, one cannot simply estimate the current state and run classic MPC. In this case, the key is to view (classic) MPC from a different perspective: MPC locally solves an optimal control problem by treating known disturbances (predictions) as exact, and treating unknown disturbances as zero. Following this philosophy, in the case when predictions are not enough to be used to estimate the current state, we can instead assume that unknown disturbances are exactly zero. The following theorem derives the optimal policy under this “optimistic” assumption. Theorem 11.11. Suppose there are 𝑑 delays and 𝑘 exact predictions with 𝑘 < 𝑑. The myopic MPC policy assumes all used predictions are exact and other disturbances (with unused predictions) are zero. Its policy at time 𝑡 is: ⊤ −1 ⊤  𝑢 𝑡 = −(𝑅 + 𝐵 𝑃𝐵) 𝐵 𝑃 𝐴 𝐴 𝑑−𝑘 𝑥ˆ𝑡−𝑑+𝑘 |𝑡 + 𝑑−𝑘−1 ∑︁  𝑖 𝐴 𝐵𝑢 𝑡−1−𝑖 . (11.22) 𝑖=0 Proof. See Yu et al. (2022). □ In other words, the policy in (11.22) first obtains the greedy estimation 𝑥ˆ𝑡−𝑑+𝑘 |𝑡 using predictions 𝑤ˆ 𝑡−𝑑|𝑡 , . . . , 𝑤ˆ 𝑡−𝑑+𝑘−1|𝑡 , and then estimates the current state by treating 𝑤 𝑡−𝑑+𝑘 = · · · = 𝑤 𝑡−1 = 0. In fact, instead of treating them as zero, we can 238 impose other values or distributions on those disturbances. This would generalize Theorem 11.11 to a broader class of policies. To summarize the two cases above (𝑘 ≥ 𝑑 and 𝑘 < 𝑑), the myopic generalization of MPC we study in this section is described as follows. Suppose we want to use 𝑘 predictions. If 𝑘 ≥ 𝑑, then we estimate the current state 𝑥𝑡 and apply (11.21). If 𝑘 < 𝑑, then we estimate the state at time 𝑡 − 𝑑 + 𝑘 and apply (11.22). In fact, the two cases coincide when 𝑘 = 𝑑. Main Results Our main result provides bounds on the competitive ratio for the myopic MPC-like policy in the case of inexact delayed predictions. We present our general result below and then discuss the special cases of (i) exact predictions and no delay, (ii) inexact predictions and no delay, and (iii) delay but no access to predictions. The special cases illustrate the contrast between inexact and exact predictions as well as the impact of delay. All proofs and more detailed discussions can be found in Yu et al. (2022). Theorem 11.12 (General result). Suppose there are 𝑑 steps of delays and the controller uses 𝑘 predictions. Define 𝑐 = ∥𝑃∥∥𝑃−1 ∥(1 + ∥𝐹 ∥). When 𝑘 ≥ 𝑑,   Alg ≤ 2 𝑘−1 Í 𝑑−𝑖 𝑖−𝑑 𝑘−𝑑  𝑐 𝜖𝑖 ∥ 𝐴 ∥ + 𝑐 𝜖𝑖 ∥𝐹 ∥ + ∥𝐹 ∥ 𝑖=0 𝑖=𝑑 + 1 Opt + 𝑂 (1). ∥𝐻 ∥ −1𝜆min (𝑃−1 − 𝐹𝑃−1 𝐹 ⊤ − 𝐻) 𝑑−1 Í When 𝑘 ≤ 𝑑,   Alg ≤ 2 𝑑−1 Í 𝑑−𝑖 𝑑−𝑖 𝑐  𝜖𝑖 ∥ 𝐴 ∥ + 𝑐 ∥𝐴 ∥ + 1 𝑖=0 𝑖=𝑘 + 1 Opt + 𝑂 (1). ∥𝐻 ∥ −1𝜆min (𝑃−1 − 𝐹𝑃−1 𝐹 ⊤ − 𝐻) 𝑘−1 Í  The 𝑂 (1) is with respect to 𝑇. It may depend on the system parameters 𝐴, 𝐵, 𝑄, 𝑅, Qf and the range of disturbances 𝑟, but not on 𝑇. When Qf = 𝑃 the 𝑂 (1) is zero. The two cases in Theorem 11.12 correspond to the two cases in the algorithm: when predictions are of high enough quality to allow estimation of the current state and when they are not. In the first case, we see that the quality of predictions in the near future has more impact, especially when 𝜌( 𝐴) > 1. In the second case, we see that the amount of delay 𝑑 exponentially increases the bound if 𝜖𝑖 > 0 and 𝜌( 𝐴) > 1. 239 Theorem 11.13 (Exact predictions without delay). Suppose there are 𝑘 exact predictions and no feedback delay. Then:   ∥𝐹 𝑘 ∥ 2 ∥𝐻 ∥ Opt + 𝑂 (1). Alg ≤ 1 + 𝜆 min (𝑃−1 − 𝐹𝑃−1 𝐹 ⊤ − 𝐻) In other words, Theorem 11.13 gives a competitive ratio result for the standard setting discussed in Sections 11.2 and 11.3. In particular, the competitive ratio exponentially decreases to 1 as 𝑘 goes up. Theorem 11.14 (Inexact predictions without delay). Suppose there are 𝑘 inexact predictions and no feedback delay. Then,  2  ∥𝐻 ∥ 𝑐 Í 𝑘−1 𝜖𝑖 ∥𝐹 𝑖 ∥ + ∥𝐹 𝑘 ∥  𝑖=0 Alg ≤ + 1 Opt + 𝑂 (1). 𝜆 min (𝑃−1 − 𝐹𝑃−1 𝐹 ⊤ − 𝐻) Theorem 11.14 differs from the previous one in that the controller can minimize the bound with respect to 𝑘 according to the prediction quality. We characterize this optimization in the following result in 1-d systems. Corollary 11.1. Suppose there are 𝑘 inexact predictions and no feedback delay. Assume 𝑛 = 𝑚 = 1. Given non-decreasing {𝜖𝑖 }, to minimize the competitive ratio bound in Theorem 11.14, the optimal number 𝑘 of predictions to use is such that: 𝜖 𝑘−1 < 1 − |𝐹 | < 𝜖𝑘 . 1 + |𝐹 | Finally, we discuss the case 𝑑 > 0 and 𝑘 = 0 (i.e., delay without predictions). Theorem 11.15 (Delay without predictions). Suppose there are 𝑑 delays and no predictions are available. Then the competitive ratio is bounded by  2  ∥𝐻 ∥ 𝑐 Í𝑑 ∥ 𝐴𝑖 ∥ + 1 𝑖=1 Alg ≤ 𝜆 min (𝑃−1 − 𝐹𝑃−1 𝐹 ⊤ − 𝐻)  + 1 Opt + 𝑂 (1). Beyond Myopic MPC: A Robustness-Consistency Perspective So far, in this section we have been focusing on a myopic MPC policy. Namely, the policy will assume that all predictions 𝑤ˆ 𝑡+𝑖|𝑡 are true disturbances. Intuitively, when designing an online control or learning algorithm with predictions, we want to balance “consistency,” which measures the competitive ratio when 240 predictions are accurate, and “robustness,” which bounds the competitive ratio when predictions are inaccurate. In online learning and algorithm community, such a balance is called the robustness-consistency trade-off (commonly discussed in online caching, ski-rental, online set cover and online matching problems). Obviously, the classic (myopic) MPC approach is a consistent algorithm, i.e., its competitive ratio is small when predictions are accurate. However, it might not be a robust algorithm, if the predictions are inaccurate. Therefore, we propose a 𝛽-confident algorithm (see more details in T. Li et al. (2022)) such that we can optimally balance between robustness and consistency: ! 𝑇−1 ∑︁
𝜏−𝑡 𝑢 𝑡 = −(𝑅 + 𝐵⊤ 𝑃𝐵) −1 𝐵⊤ 𝑃 𝐴𝑥𝑡 + 𝛽 𝐹⊤ 𝑃 𝑤ˆ 𝜏 (11.23) 𝜏=𝑡 where the main difference from the classic myopic MPC is that there is a tunable parameter 𝛽 ∈ [0, 1]. In T. Li et al. (2022), we proposed a self-tuning algorithm to optimally tune this 𝛽 parameter in an online manner, based on the prediction quality of all previous predictions. 11.6 Extension II: Time-Variant Systems and General Costs In all previous sections, we analyzed the non-asymptotic performance (using dynamic regret or competitive ratio as the metric) of Model Predictive Control (MPC) in time-invariant LQ systems. Namely, the dynamics is 𝑥𝑡+1 = 𝐴𝑥𝑡 + 𝐵𝑢 𝑡 + 𝑤 𝑡 and the cost is 𝑥𝑡⊤ 𝑄𝑥𝑡 + 𝑢 𝑡⊤ 𝑅𝑢 𝑡 , where 𝐴, 𝐵, 𝑄, 𝑅 are time-invariant matrices. Naturally, the reader might wonder how much our strong guarantees for MPC rely on the structure of the simple LQ system. In particular, does the exponential decaying property (i.e., the dynamic regret or competitive ratio bound exponentially improves as the number of predictions increases) still hold in more complicated systems? In this section, we give an affirmative answer to the question above, by briefly introducing the results in Lin, Hu, et al. (2021) for linear time-variant (LTV) systems with general time-variant well-conditioned costs. 241 Problem Statement and Assumptions We consider a finite-horizon discrete-time online control problem with LTV dynamics, time-varying costs, and disturbances, namely min 𝑇 ∑︁ ( 𝑓𝑡 (𝑥𝑡 ) + 𝑐 𝑡 (𝑢 𝑡−1 )) 𝑥 0:𝑇 ,𝑢 0:𝑇−1 𝑡=1 s.t. 𝑥𝑡 = 𝐴𝑡−1 𝑥𝑡−1 + 𝐵𝑡−1 𝑢 𝑡−1 + 𝑤 𝑡−1 , 𝑡 = 1, . . . , 𝑇, (11.24) 𝑥0 = 𝑥(0), where 𝑥𝑡 ∈ R𝑛 , 𝑢 𝑡 ∈ R𝑚 , and 𝑤 𝑡 ∈ R𝑛 denote the state, the control action, and the disturbance of the system at time steps 𝑡 = 1, . . . , 𝑇, and 𝑥(0) ∈ R𝑛 is a given initial state. By convention, the hitting cost function 𝑓𝑡 : R𝑛 → R+ and control cost function 𝑐 𝑡 : R𝑚 → R+ are assumed to be time-varying and well-conditioned. Define the tuple 𝜗𝑡 := ( 𝐴𝑡 , 𝐵𝑡 , 𝑤 𝑡 , 𝑓𝑡+1 , 𝑐 𝑡+1 ). We assume that the algorithm has access to the exact predictions of disturbances, cost functions and dynamical matrices in the future 𝑘 time steps (which are timevarying), i.e., the event sequence is 𝑥 0 , 𝜗0 , 𝜗1 , . . . , 𝜗𝑘−1 , 𝑢 0 , 𝜗𝑘 , 𝑢 1 , 𝜗𝑘+1 , . . . , 𝑢𝑇−𝑘−1 , 𝜗𝑇−1 , 𝑢𝑇−𝑘 , 𝑢𝑇−𝑘+1 , . . . , 𝑢𝑇−1 . Here we assume all predictions are exact. Assumption 11.2. We assume all cost functions 𝑓𝑡 (·), 𝑐 𝑡 (·) are twice continuously differentiable and well-conditioned (i.e., both strongly convex and strongly smooth, see definitions in Chapter 8). Moreover, we assume they are non-negative and 𝑓𝑡 (0) = 0, 𝑐 𝑡 (0) = 0. We assume 𝐴𝑡 , 𝐵𝑡 and 𝐵𝑡† are uniformly bounded, and the LTV system {𝐴𝑡 , 𝐵𝑡 } are uniformly (𝑑, 𝛿)-controllable (the controllability index is 𝑑 and the controllability matrix’s minimum singular value is uniformly lower bounded by 𝛿). Main Results Suppose all above assumptions hold, we have the following informal theorem for the performance of MPC: Theorem 11.16 (Informal, see the formal version in Lin, Hu, et al. (2021)). Suppose the terminal cost 𝑉 (·) in MPC is either the indicator function of the origin, or a non-negative convex function that is twice continuously differentiable and satisfies 242 𝑉 (0) = 0. If the prediction window 𝑘 ≥ 𝑑 is sufficiently large (larger than a constant depending on the system parameter, but not on 𝑇), we have: Alg − Opt = 𝑂 (𝜆¯ 𝑘 𝑇 + 1) where 0 ≤ 𝜆¯ < 1. Moreover, if the terminal cost is the indicator function of the origin, and the prediction window is sufficiently large, we have:   Alg ≤ 1 + 𝑂 ( 𝜆¯ 𝑘 ) · Opt. Generally speaking, Theorem 11.16 shows that the exponentially decaying property still holds even in highly time-variant systems, as long as the prediction window is large enough. Proof sketch. In all previous sections, we analyzed the dynamic regret or competitive ratio of MPC via direct methods. Namely, we directly characterize the MPC policy’s structure and the offline optimal policy’s structure, and then compare their difference. However, it is very hard (if not impossible) to apply the direct method in time-variant systems because such characterizations are in general intractable. Therefore, to prove Theorem 11.16, Lin, Hu, et al. (2021) uses a novel proof framework based on a perturbation bound that characterizes how a small change to the system parameters impacts the optimal trajectory. However, even under the perturbation analysis framework, analyzing LTV systems directly is still hard, so we instead develop a novel reduction from LTV systems to fully-actuated systems, i.e., systems where the controller can steer the system to any state in the whole space R𝑛 freely at every time step. This special case is a form of online optimization called smoothed online convex optimization (SOCO). Such a reduction is inspired by Chapter 10. References Abbasi-Yadkori, Yasin and Csaba Szepesvári (2011). Regret bounds for the adaptive control of linear quadratic systems. In: Proceedings of the 24th Annual Conference on Learning Theory. JMLR Workshop and Conference Proceedings, pp. 1–26. Agarwal, Naman, Brian Bullins, Elad Hazan, Sham M. Kakade, and Karan Singh (2019). Online control with adversarial disturbances. In: International Conference on Machine Learning (ICML). Agarwal, Naman, Elad Hazan, and Karan Singh (2019). Logarithmic regret for online control. In: Neural Information Processing Systems (NeurIPS). 243 Anderson, Brian D. O. and John B. Moore (2007). Optimal control: Linear quadratic methods. Courier Corporation. – (2012). Optimal filtering. Courier Corporation. Angeli, David, Rishi Amrit, and James B. Rawlings (2011). On average performance and stability of economic model predictive control. In: IEEE Transactions on Automatic Control 57.7, pp. 1615–1626. Angeli, David, Alessandro Casavola, and Francesco Tedesco (2016). Theoretical advances on economic model predictive control with time-varying costs. In: Annual Reviews in Control 41, pp. 218–224. Baca, Tomas, Daniel Hert, Giuseppe Loianno, Martin Saska, and Vijay Kumar (2018). Model predictive trajectory tracking and collision avoidance for reliable outdoor deployment of unmanned aerial vehicles. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 6753– 6760. Camacho, Eduardo F. and Carlos Bordons Alba (2013). Model predictive control. Springer Science & Business Media. Cassel, Asaf, Alon Cohen, and Tomer Koren (2020). Logarithmic regret for learning linear quadratic regulators efficiently. In: arXiv preprint arXiv:2002.08095. Chen, Niangjun, Anish Agarwal, Adam Wierman, Siddharth Barman, and Lachlan Andrew (2015). Online convex optimization using predictions. In: Proceedings of ACM SIGMETRICS, pp. 191–204. Cohen, Alon, Tomer Koren, and Yishay √ Mansour (2019). Learning linear-quadratic regulators efficiently with only 𝑇 regret. In: International Conference on Machine Learning (ICML). Dean, Sarah, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu (2020). On the sample complexity of the linear quadratic regulator. In: Foundations of Computational Mathematics 20.4, pp. 633–679. Fazel, Maryam, Rong Ge, Sham M. Kakade, and Mehran Mesbahi (2018). Global convergence of policy gradient methods for the linear quadratic regulator. In: arXiv preprint arXiv:1801.05039. Foster, Dylan J. and Max Simchowitz (2020). Logarithmic regret for adversarial online control. In: arXiv preprint arXiv:2003.00189. Goel, Gautam and Babak Hassibi (2020). The power of linear controllers in LQR control. In: arXiv preprint arXiv:2002.02574. Goel, Gautam and Adam Wierman (2019). An online algorithm for smoothed regression and LQR control. In: International Conference on Artificial Intelligence and Statistics (AISTATS). 244 Grüne, Lars and Simon Pirkelmann (2018). Economic model predictive control for time-varying system: Performance and stability results. In: Optimal Control Applications and Methods. Grüne, Lars and Marleen Stieler (2014). Asymptotic stability and transient optimality of economic MPC without terminal conditions. In: Journal of Process Control 24.8, pp. 1187–1196. Hazan, Elad, Sham M. Kakade, and Karan Singh (2020). The nonstochastic control problem. In: Conference on Algorithmic Learning Theory (ALT). Kirk, Donald E. (2004). Optimal control theory: An introduction. Courier Corporation. Lazic, Nevena, Craig Boutilier, Tyler Lu, Eehern Wong, Binz Roy, Moonkyung Ryu, and Greg Imwalle (2018). Data center cooling using model-predictive control. In: Advances in Neural Information Processing Systems, pp. 3814–3823. Li, Tongxin, Ruixiao Yang, Guannan Qu, Guanya Shi, Chenkai Yu, Adam Wierman, and Steven Low (2022). Robustness and consistency in linear quadratic control with untrusted predictions. In: Proceedings of the ACM on Measurement and Analysis of Computing Systems 6.1, pp. 1–35. doi: 10.1145/3508038. Li, Yingying, Xin Chen, and Na Li (2019). Online optimal control with linear dynamics and predictions: Algorithms and regret analysis. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 14858–14870. Lin, Yiheng, Gautam Goel, and Adam Wierman (2020). Online optimization with predictions and non-convex losses. In: Proceedings of ACM SIGMETRICS. Lin, Yiheng, Yang Hu, Guanya Shi, Haoyuan Sun, Guannan Qu, and Adam Wierman (2021). Perturbation-based regret analysis of predictive control in linear time varying systems. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 34. Curran Associates, Inc., pp. 5174–5185. url: https : / / arxiv.org/abs/2106.10497. Rosolia, Ugo and Francesco Borrelli (2017). Learning model predictive control for iterative tasks. A data-driven control framework. In: IEEE Transactions on Automatic Control 63.7, pp. 1883–1896. – (2019). Sample-based learning model predictive control for linear uncertain systems. In: arXiv preprint arXiv:1904.06432. Simchowitz, Max and Dylan Foster (2020). Naive exploration is optimal for online LQR. In: International Conference on Machine Learning. Proceedings of Machine Learning Research, pp. 8937–8948. Vazquez, Sergio, Jose Rodriguez, Marco Rivera, Leopoldo G. Franquelo, and Margarita Norambuena (2016). Model predictive control for power converters and drives: Advances and trends. In: IEEE Transactions on Industrial Electronics 64.2, pp. 935–947. 245 Yu, Chenkai, Guanya Shi, Soon-Jo Chung, Yisong Yue, and Adam Wierman (2020). The power of predictions in online control. In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 33. Curran Associates, Inc., pp. 1994– 2004. url: https : / / proceedings . neurips . cc / paper / 2020 / file / 155fa09596c7e18e50b58eb7e0c6ccb4-Paper.pdf. – (2022). Competitive control with delayed imperfect information. In: American Control Conference (ACC). url: https://arxiv.org/abs/2010.11637. Zhou, Kemin and John Comstock Doyle (1998). Essentials of robust control. Vol. 104. Prentice Hall Upper Saddle River, NJ. 246 C h a p t e r 12 DISCUSSION AND FUTURE WORK In Part II, through Chapters 9 to 11, we have introduced three unifying interfaces between learning and control theory. See Fig. 8.1 in Chapter 8 for a summary. In this chapter, we will conclude Part II by discussing several important future research directions. Provably Safe Lifelong Learning in Real-World Autonomous Systems As one of the most challenging open problems in AI, lifelong learning considers systems that can continually learn many tasks over a lifetime. Lifelong learning already poses several fundamental problems (e.g., catastrophic forgetting), let alone considering the safety-critical real-world setting (e.g., aircraft control with selfimprovement over a lifetime). Note that Chapters 5 and 9 can be viewed as a “one-shot” and supervised approximation of lifelong learning, where a representation shared by several tasks is learned first (with supervision) and then adapted by adaptive control. Towards unsupervised and continual lifelong learning with safety guarantees, the goal is to systematically address the following questions: How to distill knowledge from previous tasks without supervision? How to design an “event-triggered” mechanism to selectively and safely transfer the distilled knowledge to new tasks? Encoding Control-Theoretic Knowledge to Reinforcement Learning Most popular RL algorithms (e.g., TRPO, SAC) are universal for all tasks. In contrast, drastically different control methods are developed for different systems/tasks, and their successes highly rely on structures inside these systems/tasks. For example, Chapter 11 shows that the strong non-asymptotic guarantees of MPC are not only from the receding horizon nature, but also from the structure of the underlying dynamical systems and the exponential closed-loop stability property. One interesting and important research direction is to encode these structures and algorithmic principles into black-box RL algorithms, which will potentially make RL algorithms much more data-efficient, robust, interpretable, and safe. 247 Sample Complexity Analysis for Nonlinear Systems Existing sample complexity results for dynamical systems focus on linear dynamics, but most real-world systems are highly nonlinear. Significantly more challenges appear from nonlinearity. For example, it is often intractable to characterize the closed-loop behavior of the optimal policy. Chapter 10 serves as an initial step by considering LTV systems and some particular classes of nonlinear systems. One important yet challenging future research topic is to have end-to-end guarantees for uncertain nonlinear systems. Hierarchical and Layered Learning and Control Having different levels of abstractions is crucial for autonomous systems, but how to design each level in a data-driven manner is unclear. As an initial step, Chapters 5 and 9 build on a 2-layer structure. Namely, an outer loop optimizes a shared representation and an inner loop fine-tunes the representation in a low dimensional space. The future direction is to study general algorithmic principles and convergence properties in general multi-layer learning and control settings.